Retrieval Augmented Generation (RAG)
Retrieval Augmented Generation (RAG) enables an LLM to ground its responses on external documents, thus improving the accuracy of its responses and minimizing hallucinations.
The Chat endpoint comes with built-in RAG capabilities such as document grounding and citation generation.
This quickstart guide shows you how to perform RAG with the Chat endpoint.
Setup
First, install the Cohere Python SDK with the following command.
Bash pip install -U cohereNext, import the library and create a client.
PYTHON import cohere co = cohere.Client( "COHERE_API_KEY" ) # Get your free API key here: https://dashboard.cohere.com/api-keysPYTHON import cohere co = cohere.Client( api_key="", # Leave this blank base_url="<YOUR_DEPLOYMENT_URL>", )PYTHON import cohere co = cohere.BedrockClient( aws_region="AWS_REGION", aws_access_key="AWS_ACCESS_KEY_ID", aws_secret_key="AWS_SECRET_ACCESS_KEY", aws_session_token="AWS_SESSION_TOKEN", ) # Get the model name: https://docs.aws.amazon.com/bedrock/latest/userguide/models-supported.htmlPYTHON import cohere co = cohere.SagemakerClient( aws_region="AWS_REGION", aws_access_key="AWS_ACCESS_KEY_ID", aws_secret_key="AWS_SECRET_ACCESS_KEY", aws_session_token="AWS_SESSION_TOKEN", )PYTHON import cohere co = cohere.Client( api_key="AZURE_API_KEY", base_url="AZURE_ENDPOINT", # example: "https://cohere-command-r-plus-08-2024-xyz.eastus.models.ai.azure.com/" )Documents
First, define the documents that will passed as the context for RAG. These documents are typically retrieved from sources such as vector databases via semantic search, or any system that can retrieve unstructured data given a user query.
Each document can take any number of fields e.g.
title,url,text, etc.PYTHON documents = [ { "text": "Reimbursing Travel Expenses: Easily manage your travel expenses by submitting them through our finance tool. Approvals are prompt and straightforward." }, { "text": "Health and Wellness Benefits: We care about your well-being and offer gym memberships, on-site yoga classes, and comprehensive health insurance." }, ]Response Generation
Next, call the Chat API by passing the documents in the
documentsparameter. This tells the model to run in RAG-mode and use these documents as the context in its response.PYTHON message = "Are there fitness-related benefits?" response = co.chat( model="command-a-plus-05-2026", message=message, documents=documents, ) print(response.text)PYTHON message = "Are there fitness-related benefits?" response = co.chat( model="command-a-plus-05-2026", message=message, documents=documents, ) print(response.text)PYTHON message = "Are there fitness-related benefits?" response = co.chat( model="YOUR_MODEL_NAME", message=message, documents=documents ) print(response.text)PYTHON message = "Are there fitness-related benefits?" response = co.chat( model="YOUR_ENDPOINT_NAME", message=message, documents=documents ) print(response.text)PYTHON message = "Are there fitness-related benefits?" response = co.chat(message=message, documents=documents) print(response.text)wordWrap Yes, we offer gym memberships, on-site yoga classes, and comprehensive health insurance.Citation Generation
The response object contains a
citationsfield, which contains specific text spans from the documents on which the response is grounded.PYTHON if response.citations: for citation in response.citations: print(citation, "\n")wordWrap start=14 end=88 text='gym memberships, on-site yoga classes, and comprehensive health insurance.' document_ids=['doc_1']