Tool
Filter JSONL by token length
Each line’s prompt, prompt plus response, or JSON.stringify is encoded with js-tiktoken o200k or cl100k. Lines outside min/max are dropped. This is not a Llama, Gemma, or GPT-6 vocabulary.
Encoding runs in this tab. Tokenizer rank files load from this site into the browser. GearDel does not receive the JSONL.
What is counted
Prompt is Alpaca instruction plus input, or the first user/human turn. Sample concatenates that prompt with Alpaca output or assistant/gpt turns, separated by a blank line. JSON uses JSON.stringify of the object.
encode() is js-tiktoken o200k_base or cl100k_base — the same encodings as the token-count page. Chat markup such as <|im_start|> is not inserted here. GPT-6 ranks are not in the library.
Min and max are inclusive token counts of that encoded string. 0–4096 is only a default interval, not a model context window.
A field that cannot be extracted is counted as missing and skipped. Each encoded string is clipped at 200,000 code points, same as the token-count page.
Blank lines are counted and skipped. A leading JSON array is rejected. At most 20,000 lines or about two million characters.
What this page does not do
- It does not implement Llama, Gemma, Claude, Gemini, or a Korean SentencePiece tokenizer. Those numbers would differ.
- It does not wrap ChatML or Llama 3 headers before counting. Use the chat-wrap page if you need that string.
- It does not estimate GPU memory or dollar cost.