Tools
Browser tools for Korean AI data
These pages run in the tab. Pastes and local files are not uploaded. GearDel still does not host dataset archives. Follow the groups in order for a Korean SFT file you already have.
Bring data in
Move between tables and JSONL, then fold Hangul.
CSV/TSV to instruction JSONL
Map header columns to instruction, input, and output, then write one JSON object per row. The table never leaves this tab.
Flatten JSONL to CSV or TSV
Each JSON object becomes one table row. Use top-level keys as columns, or Alpaca and the first system/user/assistant turns. Nested values are JSON strings. Nothing leaves this tab.
Korean NFC, NFD, and jamo
Text that looks the same can use different code points. This page shows String.normalize output and counts syllables, combining jamo, and compatibility jamo.
Korean characters and bytes
Dropping spaces and counting UTF-8 or EUC-KR bytes are different questions. Korean ‘word count’ is not the same as spaces, so this page does not invent one.
Inspect
Check format, fields, excerpts, citations, Hangul quality, and tokens in this tab.
Instruction JSONL checker and converter
One JSON object per line. Bad lines keep their line number and a reason. Conversion stays in this tab; GearDel never receives the file.
JSONL field viewer and row sampler
Top-level keys are counted. The first n lines, or n lines from a seed hash, are kept. This page does not validate or convert formats. Lines never leave this tab.
Open catalog excerpts, not the files
Only excerpts already stored on the catalog row are shown. The count is the length of the samples array. GearDel does not host archives. Quality checks run on those excerpts only.
Build a citation from catalog fields
Only title, year, provider, original URL, license label, commercial mark, and last-checked date are used. Author lists are not in the catalog, so BibTeX has no author field.
Korean encoding damage and mixed speech levels
Paste or JSONL strings are scanned for U+FFFD, combining or compatibility jamo, non-NFC Hangul, zero-width marks, UTF-8 misread as Latin-1, and mixed formal/polite/plain endings in replies. Nothing leaves this tab.
Token count for Korean text
encode() uses js-tiktoken ranks. o200k_base maps to GPT-4o, GPT-4.1, o1, o3, o4-mini, and GPT-5. cl100k_base maps to GPT-3.5-turbo, GPT-4, and GPT-4-turbo. GPT-6 ranks are not in the library.
Clean
Mask PII patterns, drop copies, cut long rows.
Mask Korean phone, RRN, and email
Korean phone numbers, 13-digit resident-ID patterns, and emails in paste or JSONL strings are replaced with tokens or a partial mask. Nothing leaves this tab.
Drop duplicate JSONL lines
Lines that share a prompt or the same JSON object are grouped. The first line, or the longest JSON.stringify, is kept. Nothing is uploaded.
Filter JSONL by token length
Each line’s prompt, prompt plus response, or JSON.stringify is encoded with js-tiktoken o200k or cl100k. Lines outside min/max are dropped. This is not a Llama, Gemma, or GPT-6 vocabulary.
Split and eval
Seed-split, then check overlap with eval prompts.
Split JSONL into train, val, and test
A seed and FNV-1a place each group in a 0–9999 bucket. The same instruction or first user turn stays in one split. Lines never leave this tab.
Whether train JSONL overlaps eval prompts
Train prompt keys are compared to eval lines (plain text or JSONL) after NFC and whitespace folding. This page does not invent a contamination rate for a public benchmark.
Train wrap
Wrap published chat strings, then count tokens.
Wrap a chat template and count tokens
Published ChatML, Llama 3, Gemma, and Qwen wrapper strings are prepended to the messages. The token number is encode() of that wrapped string with o200k or cl100k, not a GPT-6 vocabulary.