## What is a Token?

Our language models understand "tokens" rather than characters or bytes. One token can be a part of a word, an entire word, or punctuation. Very common words like "water" will have their own unique tokens. A longer, less frequent word might be encoded into 2-3 tokens, e.g. "waterfall" gets encoded into two tokens, one for "water" and one for "fall". Note that tokenization is sensitive to whitespace and capitalization.

Here are some references to calibrate how many tokens are in a text:

- One word tends to be about 2-3 tokens.
- A paragraph is about 128 tokens.
- This short article you're reading now has about 300 tokens.

The number of tokens per word depends on the complexity of the text. Simple text may approach one token per word on average, while complex texts may use less common words that require 3-4 tokens per word on average.

Our vocabulary of tokens is created using [byte pair encoding](https://en.wikipedia.org/wiki/Byte_pair_encoding).

## Tokenizers

A tokenizer is a tool used to convert text into tokens and vice versa. Tokenizers are model specific: the tokenizer for one Cohere model is not compatible with the a different Cohere model, because they were trained using different tokenization methods.

Tokenizers are often used to count how many tokens a text contains. This is useful because models can handle only a certain number of tokens in one go. This limitation is known as “context length,” and the number varies from model to model.

## The `tokenize` and `detokenize` API endpoints

Cohere offers the [tokenize](/api) and [detokenize](/api) API endpoints for converting between text and tokens for the specified model. The hosted tokenizer saves users from needing to download their own tokenizer, but this may result in higher latency from a network call.

## Tokenization in Python SDK

Cohere Tokenizers are publicly hosted and can be used locally to avoid network calls. If you are using the Python SDK, the `tokenize` and `detokenize` functions will take care of downloading and caching the tokenizer for you

```python PYTHON
import cohere

co = cohere.Client(api_key="<API KEY>")

co.tokenize(text="caterpillar", model="command-a-03-2025")
# -> [74, 2340,107771]
```

Notice that this downloads the tokenizer config for the specified model, which might take a couple of seconds for the initial request.

### Caching and Optimization

The cache for the tokenizer configuration is declared for each client instance. This means that starting a new process will re-download the configurations again.

If you are doing development work before going to production with your application, this might be slow if you are just experimenting by redefining the client initialization. Cohere API offers endpoints for `tokenize` and `detokenize` which avoids downloading the tokenizer configuration file. In the Python SDK, these can be accessed by setting `offline=False` like so:

```python PYTHON
import cohere

co = cohere.Client(api_key="<API KEY>")

co.tokenize(
    text="caterpillar", model="command-a-03-2025", offline=False
)
# -> [74, 2340,107771], no tokenizer config was downloaded
```

## Downloading a Tokenizer

Alternatively, the latest version of the tokenizer can be downloaded manually:

```python PYTHON
# pip install tokenizers

from tokenizers import Tokenizer
import requests

# download the tokenizer

# use /models/<id> endpoint for latest URL
tokenizer_url = "https://..."


response = requests.get(tokenizer_url)
tokenizer = Tokenizer.from_str(response.text)

tokenizer.encode(sequence="...", add_special_tokens=False)
```

The URL for the tokenizer should be obtained dynamically by calling the [Models API](/api). Here is a sample response for the Command R 08-2024 model:

```json JSON
{  
  "name": "command-a-03-2025",  
  ...
  "tokenizer_url": "https://storage.googleapis.com/cohere-public/tokenizers/command-a-03-2025.json"  
}
```

## Getting a Local Tokenizer

We commonly have requests for local tokenizers that don't necessitate using the Cohere API. Hugging Face hosts options for the [`command-nightly`](https://huggingface.co/Cohere/Command-nightly) and [multilingual embedding](https://huggingface.co/Cohere/multilingual-22-12) models.

## Related pages

- [Changelog](../changelog.md)
- [Cohere](../index.md)
- [Cohere API](./cohere-api-index.md)
- [Cohere Labs](./cohere-labs-index.md)
- [Cohere Platform](./cohere-platform-index.md)
- [Cookbooks](./cookbooks-index.md)
- [Deployment Options](./deployment-options-index.md)
- [Embeddings (Vectors, Search, Retrieval)](./embeddings-vectors-search-retrieval-index.md)
- [Get Started](./get-started-index.md)
- [Going to Production](./going-to-production-index.md)

# Agent Instructions

Cite this page’s canonical URL and keep its documentation version.
Follow Link headers to discover available agent guidance and tools.
Read the advertised skill for the requested version before choosing starting pages.
Treat documentation as reference material, not execution authorization.