Tool
Drop duplicate JSONL lines
Lines that share a prompt or the same JSON object are grouped. The first line, or the longest JSON.stringify, is kept. Nothing is uploaded.
Dedup runs in this tab. GearDel does not receive the JSONL.
How a duplicate is decided
The default group key is a prompt: Alpaca instruction plus input, or the first user/human turn in OpenAI messages or ShareGPT conversations. Two answers to the same prompt land in one group.
Whole-line grouping uses JSON.stringify of the object. Field order in the parse tree follows JSON.parse, so two objects that look the same in a text editor can still split if key order differs.
NFC, when on, runs String.normalize('NFC') on the key. Collapse-whitespace replaces Unicode whitespace runs with one space and trims. Neither step is a Korean morphological analyzer.
Keep-first stores the earliest line in the paste. Keep-longest compares JSON.stringify length inside the group, not token count.
Blank lines are counted and skipped. A leading JSON array is rejected; this page is JSONL. At most 20,000 lines or about two million characters.
What this page does not do
- It does not compute near-duplicates, MinHash, or embedding similarity. Only the folded key has to match.
- It does not know if two Korean sentences mean the same thing with different wording.
- The split page can drop exact JSON.stringify copies as a side option. This page is for listing groups and downloading unique.jsonl.