Alpaca, ShareGPT, and OpenAI messages

Alpaca

One object with instruction, optional input, and output. Empty input is an empty string, not a missing key you have to guess. Multi-turn chat does not fit. If a trainer expects this shape, convert extra turns only when you accept dropping all but the first user turn and the first assistant turn.

{"instruction":"다음 문장을 높임말로 바꾸세요.","input":"밥 먹었니","output":"식사하셨어요?"}

ShareGPT

conversations (sometimes conversation) is an array of from/value. human and user map to the user role; gpt, assistant, chatgpt, and model map to assistant; system stays system. Several turns stay several turns. Tool or function turns are not this page’s format.

{"conversations":[{"from":"human","value":"KMMLU는 학습용인가요?"},{"from":"gpt","value":"평가용 벤치마크입니다."}]}

OpenAI messages

messages is an array of role system|user|assistant with string content. Fine-tuning chat JSONL uses this object per line. Image parts and tool calls need a different schema; GearDel’s JSONL checker rejects those lines instead of dropping fields silently.

{"messages":[{"role":"user","content":"서울의 수도는?"},{"role":"assistant","content":"대한민국의 수도는 서울입니다."}]}

After the shape is fixed

Convert a spreadsheet on CSV→JSONL, check lines on JSONL, sample fields on JSONL sample, mask phone/RRN/email patterns on PII mask, scan Hangul damage on data quality, open catalog excerpts on try samples, flatten JSONL on JSONL to CSV, cite catalog fields on cite, drop copies on JSONL dedup, filter token length on token length, then partition with a seed hash on train/val/test. Wrap ChatML or Llama 3 strings and count that wrap on chat wrap. Scoring sets such as KLUE stay on eval vs train.

Instruction and corpus rows are in Korean LLM training datasets (63 rows). Pick a task on pick.