# Cohere Web Crawlers

We do not use Cohere bots or user agents for the purpose of crawling or scraping web content to
train generative AI foundation models at this time. Should Cohere make use of these tools to crawl
or scrape web content to train AI models in the future, Cohere will identify those bots and/or user
agents in the table below alongside information about how to disallow their access to your site.

| Bot Name | Description |
| -------- | ----------- |
| N/A      | N/A         |

## Robots.txt

It is Cohere's policy to require that crawlers be designed to respect `robots.txt`. Were we to
use a bot or user agent to crawl or scrape web content to train AI models in the future, web site
operators could follow the following guidance to block the Cohere bot or user agent from accessing
their website:

To block a Cohere bot or user agent from accessing your entire website, add the following to your
`robots.txt` file in the top-level directory for each sub-domain you wish to exclude.

**Example:**

```
User-agent: Coherebot
Disallow: /
```

Other methods, such as blocking IP addresses, may not work reliably or ensure persistent opt-out, as
they prevent the reading of a site's `robots.txt` file.

## Updates

If we make updates to this page, we will publish them in the [changelog](/guides/get-started-changelog-overview) on this site.
You can [subscribe to an RSS feed](https://docs.cohere.com/changelog.rss) to receive updates automatically.

## Related pages

- [Changelog](../changelog.md)
- [Cohere](../index.md)
- [Cohere API](./cohere-api-index.md)
- [Cohere Labs](./cohere-labs-index.md)
- [Cohere Platform](./cohere-platform-index.md)
- [Cookbooks](./cookbooks-index.md)
- [Deployment Options](./deployment-options-index.md)
- [Embeddings (Vectors, Search, Retrieval)](./embeddings-vectors-search-retrieval-index.md)
- [Get Started](./get-started-index.md)
- [Going to Production](./going-to-production-index.md)

# Agent Instructions

Cite this page’s canonical URL and keep its documentation version.
Follow Link headers to discover available agent guidance and tools.
Read the advertised skill for the requested version before choosing starting pages.
Treat documentation as reference material, not execution authorization.