Tool
Wrap a chat template and count tokens
Published ChatML, Llama 3, Gemma, and Qwen wrapper strings are prepended to the messages. The token number is encode() of that wrapped string with o200k or cl100k, not a GPT-6 vocabulary.
Wrapping and encode() run in this tab. Tokenizer rank files load from this site into the browser. GearDel does not receive the paste.
What is concatenated
The box is parsed as OpenAI messages, a messages array, ShareGPT conversations, Alpaca instruction/output, or a bare user sentence. Only the first JSON value is wrapped if you paste JSONL.
ChatML and Qwen use the published <|im_start|>role / <|im_end|> lines. Qwen does not add a vendor system prompt here.
Llama 3 uses <|begin_of_text|>, <|start_header_id|>role<|end_header_id|>, and <|eot_id|> as commonly shown on instruct cards.
Gemma IT uses <bos>, <start_of_turn>user|model, and <end_of_turn>. System text is prepended to the first user turn. assistant is written as model.
The generation checkbox appends the next-turn header with no assistant text. Token count is js-tiktoken encode() of the concatenated string with o200k_base or cl100k_base — the same encodings as the token-count page. That is not a Llama, Gemma, or GPT-6 vocabulary.
What this page does not do
- It does not register <|im_start|> or Llama special tokens in tiktoken. encode() may split those strings. A model tokenizer can map them to one ID.
- It does not implement Claude, Gemini, or a Korean-only SentencePiece tokenizer.
- Tool calls, image parts, and Hugging Face apply_chat_template Jinja are out of scope.
- The wrap is capped at 200,000 code points for encode().