# Best Practices for using Rerank ## Optimizing Performance In the following tables, you'll find recommendations for getting the best performance out of Rerank v4.0, v3.5, and v3.0. | Constraint | Minimum | Maximum | Default Value | | ----------------------------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------ | :------------ | | Number of Documents | 1 | 10,000 | N/A | | Max Number of Chunks | 1 | N/A | 1 | | Number of Tokens per Document | 1 | N/A (see [below](/guides/embeddings-vectors-search-retrieval-text-embeddings-reranking-reranking-best-practices#document-chunking)for more info) | N/A | | Number of Tokens per Query | 1 | 2048 | N/A | ## Document Chunking Cohere's Rerank models follow a particular procedure to chunk documents, which will look something like this: - Take the context window; - Subtract the reserved tokens (the number of reserved tokens changes per model) For `rerank-v4.0`, the model breaks documents into 32,764-token chunks (this is the context length of 32,768 minus four special, reserved tokens.) For example, if your query is 100 tokens and your document is 100,000 tokens, your document will be broken into the following chunks: 1. `relevance_score_1 = ` 2. `relevance_score_2 = ` 3. `relevance_score_3 = ` 4. `relevance_score_4 = ` 5. `relevance_score = max(relevance_score_1, relevance_score_2, relevance_score_3, relevance_score_4)` :::callout{intent="note"} A context length of 32,768 corresponds to about \~48-50 pages, so it is longer than most documents you'll upload. This means that the `rerank-v4.0` models will be looking at entire documents as it chooses its rankings. ::: For `rerank-v3.5` and `rerank-v3.0`, the process is the same except the models break documents into 4093 token chunks; if your query is 100 tokens and your document is 10,000 tokens, for example, it will be broken into the following chunks: 1. `relevance_score_1 = ` 2. `relevance_score_2 = ` 3. `relevance_score_3 = ` 4. `relevance_score = max(relevance_score_1, relevance_score_2, relevance_score_3)` If you would like more control over how chunking is done, we recommend that you chunk your documents yourself. ## Max Number of Documents When using `rerank-v4.0-pro`, `rerank-v4.0-fast`, `rerank-v3.5` and `rerank-v3.0` models, the endpoint will throw an error if the user attempts to pass more than 10,000 documents at a time. The maximum number of documents that can be passed to the endpoint is calculated with the following inequality: `Number of documents * max_chunks_per_doc >10,000`. If `Number of documents * max_chunks_per_doc` exceeds `10,000`, the endpoint will return an error. By default, the `max_chunks_per_doc` is set to `1` for `rerank` models. ## Queries Our `rerank-v4.0` models (both 'pro' and 'fast') are trained with a context length of 32768 tokens. The model takes both the _query_ and the _document_ into account when calculating against this limit, and the query can account for up to half of the full context length. If your query is larger than 16,384 tokens, in other words, it will be truncated to the first 16,384 tokens (leaving the other 16,384 for the document(s)). Our `rerank-v3.5` and `rerank-v3.0` models, are trained with a context length of 4096 tokens, so the process is the same while the math is different. If your query is larger than 2048 token, it will be truncated to the first 2048 tokens (leaving the other 2048 for the document(s)). ## Semi-Structured Data Support Our `rerank-v4.0-pro`, `rerank-v4.0-fast`, `rerank-v3.5` and `rerank-v3.0` models support semi-structured data reranking through a list of JSON objects. The `rank_fields` parameter will default to a field parameter called `text` unless otherwise specified. If the `rank_fields` parameter is unspecified _and_ none of your JSON objects have a `text` field, the endpoint will return an error. ```json JSON [ { "Title": "How to fix a dishwasher", "Author": "John Smith", "Date": "August 1st 2023", "Content": "Fixing a dishwasher depends on the specific problem you're facing. Here are some common issues and their potential solutions:...." }, { "Title": "How to fix a leaky sink", "Date": "July 25th 2024", "Content": "Fixing a leaky sink will depend on the source of the leak. Here are general steps you can take to address common types of sink leaks:....." } ] ``` Looking at the example above, passing in `rank_fields=["Title","Content"]` would mean the model considers both the title and content for ranking. The `rank_fields` parameter is _not_ commutative, which means `rank_fields=["Title","Content"]` can lead to different results than `rank_fields=["Content","Title"]`. :::callout{intent="note"} `rank_fields` is only supported in the v1 API client. ::: ## Interpreting Results The most important output from the [Rerank API endpoint](/api) is the absolute rank exposed in the response object. The score is query dependent, and could be higher or lower depending on the query and passages sent in. Relevance scores are normalized to be in the range `[0, 1]`. Scores close to `1` indicate a high relevance to the query, and scores closer to `0` indicate low relevance. This is used for _ranking_ purposes, but be careful about how you interepret the actual numbers--you can't assume that a document with a relevance score of `0.9109375` is _twice_ as relevant as one with a relevance score of `0.04421997`. To find a threshold on the scores to determine whether a document is relevant or not, we recommend going through the following process: - Select a set of 30-50 representative queries `Q=[q_0, … q_n]` from your domain. - For each query provide a document that is considered borderline relevant to the query for your specific use case, and create a list of (query, document) pairs: `sample_inputs=[(q_0, d_0), …, (q_n, d_n)]` . - Pass all tuples in `sample_inputs` through the rerank endpoint in a loop, and gather relevance scores `sample_scores=[s0, ..., s_n]`. The average of `sample_scores` can then be used as a reference when deciding a threshold for filtering out irrelevant documents. ## Related pages - [Changelog](../changelog.md) - [Cohere](../index.md) - [Cohere API](./cohere-api-index.md) - [Cohere Labs](./cohere-labs-index.md) - [Cohere Platform](./cohere-platform-index.md) - [Cookbooks](./cookbooks-index.md) - [Deployment Options](./deployment-options-index.md) - [Embeddings (Vectors, Search, Retrieval)](./embeddings-vectors-search-retrieval-index.md) - [Get Started](./get-started-index.md) - [Going to Production](./going-to-production-index.md) # Agent Instructions Cite this page’s canonical URL and keep its documentation version. Follow Link headers to discover available agent guidance and tools. Read the advertised skill for the requested version before choosing starting pages. Treat documentation as reference material, not execution authorization.