Skip to main content
Cohere

Search documentation

Type to search this documentation.

On this pageOverview

Retrieval augmented generation (RAG) - quickstart

About retrieval augmented generation

Section titled “About retrieval augmented generation”

Retrieval augmented generation (RAG) enables an LLM to ground its responses on external documents, thus improving the accuracy of its responses and minimizing hallucinations.

The Chat endpoint comes with built-in RAG capabilities such as document grounding and citation generation.

This quickstart guide shows you how to perform RAG with the Chat endpoint.

  1. Setup

    First, install the Cohere Python SDK with the following command.

    Bash
    pip install -U cohere

    Next, import the library and create a client.

    PYTHON
    import cohere

co = cohere.ClientV2(
    "COHERE_API_KEY"
)  # Get your free API key here: https://dashboard.cohere.com/api-keys
    PYTHON
    import cohere

co = cohere.ClientV2(
    api_key="",  # Leave this blank
    base_url="<YOUR_DEPLOYMENT_URL>",
)
    PYTHON
    import cohere

co = cohere.BedrockClientV2(
    aws_region="AWS_REGION",
    aws_access_key="AWS_ACCESS_KEY_ID",
    aws_secret_key="AWS_SECRET_ACCESS_KEY",
    aws_session_token="AWS_SESSION_TOKEN",
)

# Get the model name: https://docs.aws.amazon.com/bedrock/latest/userguide/models-supported.html
    PYTHON
    import cohere

co = cohere.SagemakerClientV2(
    aws_region="AWS_REGION",
    aws_access_key="AWS_ACCESS_KEY_ID",
    aws_secret_key="AWS_SECRET_ACCESS_KEY",
    aws_session_token="AWS_SESSION_TOKEN",
)
    PYTHON
    import cohere

co = cohere.ClientV2(
    api_key="AZURE_API_KEY",
    base_url="AZURE_ENDPOINT",  # example: "https://cohere-command-r-plus-08-2024-xyz.eastus.models.ai.azure.com/"
)

  2. Documents

    First, define the documents that will passed as the context for RAG. These documents are typically retrieved from sources such as vector databases via semantic search, or any system that can retrieve unstructured data given a user query.

    Each document is a data object that can take any number of fields e.g. title, url, text, etc.

    PYTHON
    documents = [
    {
        "data": {
            "text": "Reimbursing Travel Expenses: Easily manage your travel expenses by submitting them through our finance tool. Approvals are prompt and straightforward."
        }
    },
    {
        "data": {
            "text": "Working from Abroad: Working remotely from another country is possible. Simply coordinate with your manager and ensure your availability during core hours."
        }
    },
    {
        "data": {
            "text": "Health and Wellness Benefits: We care about your well-being and offer gym memberships, on-site yoga classes, and comprehensive health insurance."
        }
    },
]

  3. Response Generation

    Next, call the Chat API by passing the documents in the documents parameter. This tells the model to run in RAG-mode and use these documents as the context in its response.

    PYTHON
    # Add the user query
query = "Are there health benefits?"

# Generate the response
response = co.chat(
    model="command-a-plus-05-2026",
    messages=[{"role": "user", "content": query}],
    documents=documents,
)

# Display the response
print(response.message.content[0].text)
    PYTHON
    # Add the user query
query = "Are there health benefits?"

# Generate the response
response = co.chat(
    model="command-a-plus-05-2026",
    messages=[{"role": "user", "content": query}],
    documents=documents,
)

# Display the response
print(response.message.content[0].text)
    PYTHON
    # Add the user query
query = "Are there health benefits?"

# Generate the response
response = co.chat(
    model="YOUR_MODEL_NAME",
    messages=[{"role": "user", "content": query}],
    documents=documents,
)

# Display the response
print(response.message.content[0].text)
    PYTHON
    # Add the user query
query = "Are there health benefits?"

# Generate the response
response = co.chat(
    model="YOUR_ENDPOINT_NAME",
    messages=[{"role": "user", "content": query}],
    documents=documents,
)

# Display the response
print(response.message.content[0].text)
    PYTHON
    # Add the user query
query = "Are there health benefits?"

# Generate the response
response = co.chat(
    model="model",  # Pass a dummy string
    messages=[{"role": "user", "content": query}],
    documents=documents,
)

# Display the response
print(response.message.content[0].text)
    wordWrap
    Yes, there are health benefits. We offer gym memberships, on-site yoga classes, and comprehensive health insurance.

  4. Citation Generation

    The response object contains a citations field, which contains specific text spans from the documents on which the response is grounded.

    PYTHON
    if response.message.citations:
    for citation in response.message.citations:
        print(citation, "\n")
    wordWrap
    start=14 end=88 text='gym memberships, on-site yoga classes, and comprehensive health insurance.' document_ids=['doc_1']

Further Resources

Section titled “Further Resources”
Suggest an edit

Propose a replacement for this page. The site team reviews it before applying any changes.

Export
Documentation menu