Tool
Korean encoding damage and mixed speech levels
Paste or JSONL strings are scanned for U+FFFD, combining or compatibility jamo, non-NFC Hangul, zero-width marks, UTF-8 misread as Latin-1, and mixed formal/polite/plain endings in replies. Nothing leaves this tab.
Scanning runs in this tab. GearDel does not receive the paste. Recovered previews stay here too.
What is counted
U+FFFD, Hangul combining jamo (U+1100–11FF and extended blocks), compatibility jamo (U+3130–318F), and halfwidth Hangul (U+FFA0–FFDC) are counted from code points.
A string that is not equal to String.normalize('NFC') is flagged. Isolated combining jamo often come from NFD or broken syllables.
Zero-width and bidi marks (including U+200B and U+FEFF) and C0 controls other than tab/newline/CR are counted. They are easy to miss in a viewer.
If every code unit is 0–255 and those bytes are valid UTF-8 that decodes to Hangul, the page shows a recovered preview. That is a Latin-1 misread heuristic, not a guaranteed history of the file. EUC-KR reverse mapping is not attempted.
JSONL mode walks string fields. Speech-level tags (합니다체, 해요체, 반말) run on Alpaca output or assistant/gpt turns only. A line is mixed when two of those endings appear in that reply. Sentence splits are punctuation and newlines. This is not a parser of every Korean ending.
What this page does not do
- It does not score dataset quality, contamination, or PIPA compliance. Counts are pattern hits, not a grade.
- It does not run a morphological analyzer. Short replies and quoted speech can look mixed or miss an ending.
- Recovered text is a preview. Do not treat it as a lossless repair of CP949, EUC-KR, or double-encoded files.
- At most 20,000 JSONL lines or about two million characters.