Notes
- Each model requires its own tokenizer download before tokenization can run.
- Tokenization rules differ by model, so the same input can produce very different token counts.
- Long inputs may start with a partial token preview for readability.
How to Use
- Select a model (Llama 3.2, Qwen 2.5, Qwen 3.6, or Gemma 4).
- Click Download to fetch that model’s tokenizer (first time only).
- Type or paste your text, then click Re-tokenize or wait for auto refresh.
- Check the color token output preview, and switch between original text and token text display.
- Expand Token Detail Preview when you need the detailed token list.
What Can This Tool Do?
When writing prompts, testing LLMs, or studying tokenizer behavior across models, you often want to know exactly how a piece of text gets broken into tokens. This tool displays the results directly so you can compare them at a glance.
Why Does Token Segmentation Matter?
Tokens affect how much context a model can use. Understanding segmentation helps you see why a prompt grows longer or why certain phrases consume more tokens than expected — making it easier to optimize your text.
Use Cases
- Prompt planning: estimate token usage before sending requests.
- Model comparison: switch between Llama, Qwen, and Gemma to compare segmentation differences.
- Edge-case debugging: inspect unexpected splits and special characters in the detail preview.
Supported Models
Currently supported:
- Llama 3.2
- Qwen 2.5
- Qwen 3.6
- Gemma 4
For longer texts, the view will initially show a partial token preview for readability.