robots.txt, llms.txt and AI Crawlers: What East African Websites Should Do

What does robots.txt do?

A robots.txt file sits at the root of your website, for example example.com/robots.txt. It tells crawlers which parts of the site they may visit. Well-behaved crawlers, including Google’s and the major AI companies’, follow it. It is a request, not a lock, so it is not a security tool.

Because a single wrong line can hide your whole site, check it before launch. A common mistake on new sites is leaving a blanket Disallow: / from a staging setup in place after going live.

Which AI crawlers should you know about?

AI companies run different crawlers for different jobs. The names and purposes below come from each company’s own documentation, which can change, so check the source before you act.

Crawler	Run by	Purpose (per the company)
Googlebot	Google	Crawls pages for Google Search
Google-Extended	Google	Control for certain Gemini uses; does not affect Search
GPTBot	OpenAI	Collects content that may help train models
OAI-SearchBot	OpenAI	Surfaces sites in ChatGPT search
ChatGPT-User	OpenAI	Visits pages when a user asks ChatGPT to
PerplexityBot	Perplexity	Indexes sites for Perplexity search
ClaudeBot	Anthropic	Crawls web content for Anthropic

OpenAI describes its crawlers on its bots documentation page. Google explains its controls in its guide to AI features and your website.

Should you allow or block AI crawlers?

It depends on your goals. If you want to be named and cited by AI assistants, which is the aim of generative engine optimization (GEO), you generally need the search-oriented crawlers to be able to read your public pages. Whether to allow training crawlers is a business decision: some publishers want to opt out of model training, while many service businesses are happy to be included.

• Want AI visibility: allow search crawlers such as Googlebot, OAI-SearchBot and PerplexityBot.
• Concerned about training use: you can disallow training-focused crawlers such as GPTBot, and read each company’s guidance on what that does and does not cover.
• Unsure: allow everything on public marketing pages and block only private areas, then review once a year.

What does a safe robots.txt look like?

This simple example allows all crawlers on a public marketing site, blocks the admin area and points to the sitemap. Adapt the paths to your own site:

User-agent: *
Allow: /
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://www.example.com/wp-sitemap.xml

To opt out of one crawler, add a separate block for it, for example User-agent: GPTBot followed by Disallow: /. Never add a Disallow: / under User-agent: * on a live site you want people to find.

What is llms.txt and do you need it?

llms.txt is a proposal for a plain-text file at /llms.txt that summarises your site and links to your most useful pages, written for language models. It is easy to create and harmless. But it is a proposal, not an agreed standard, and no major search engine has said that it uses the file for ranking. Treat it as a convenience for tools that choose to read it.

A good llms.txt includes your business name and a one-line description, where you operate, your main services with links, your contact details and your key pages. Keep it factual and keep it in sync with the real site.

What matters more than any of these files?

Files help crawlers find you. They do not make an assistant recommend you. What gets a business named is clear, consistent facts on your site, trusted mentions elsewhere, fast pages and useful content. Our guide to getting cited by ChatGPT, Gemini and Perplexity covers these in order.

The crawlers that matter, updated

In addition to the table above, these are the other crawlers named in company documentation. Check each page before you act, because names and behaviour change:

Crawler	Operator	What the operator says	Source
Claude-SearchBot	Anthropic	Navigates the web to improve search result quality for users	Anthropic documentation
Claude-User	Anthropic	Retrieves pages when a Claude user asks a question that needs web access	Anthropic documentation
ClaudeBot	Anthropic	Collects web content that may be used for model training	Anthropic documentation
Perplexity-User	Perplexity	Supports user-initiated requests; generally ignores robots.txt because a user asked for the page	Perplexity documentation
Google-Extended	Google	A token controlling use of content for Gemini training and grounding; it does not affect inclusion or ranking in Google Search	Google documentation

Anthropic says it honours industry-standard robots.txt directives for its crawlers. OpenAI and Perplexity say changes can take about 24 hours to be picked up.

How do you test a robots.txt change?

1. Open yourdomain.com/robots.txt in a browser and read it line by line.
2. Use the robots.txt report in Search Console to see which version Google has fetched and any errors (Search Console help: robots.txt report).
3. Fetch a key page in a private window to confirm it loads normally.
4. Check that your sitemap address is correct.
5. Wait about a day, then re-test, because crawlers update on their own schedules.

Common robots.txt mistakes

Mistake	What happens	Fix
Disallow: / under User-agent: * left from staging	The whole site is blocked from crawling	Remove it and request a recrawl
Blocking CSS or JavaScript folders	Pages may render badly for crawlers	Allow the resources a page needs
Typos in crawler names	The rule does nothing	Copy the name from the company documentation
Using robots.txt to hide private pages	The file is public, and it is not a security tool	Use passwords or noindex where appropriate
Several conflicting robots.txt files from plugins	Unpredictable results	Keep one source of truth

Do firewalls and security plugins block crawlers?

They can. Bot protection, CDNs and security plugins sometimes challenge or block crawlers even when robots.txt allows them. OpenAI’s documentation says that to appear in ChatGPT search you should allow OAI-SearchBot in robots.txt and allow requests from its published IP ranges (OpenAI: crawler documentation). If you use a firewall, check its bot settings, and test by looking at your server logs or asking your host.

A sample llms.txt

The llms.txt idea is a proposal, described at llmstxt.org: the llms.txt proposal, and not a standard that search engines have said they rely on. If you add one, keep it short and accurate. A fictional example:

# Example Stay

> Example Stay is a 20-room boutique hotel in Kiyovu, Kigali, with airport transfers and meeting rooms.

## Key pages
- [Rooms and rates](https://example.rw/rooms/)
- [Meetings and events](https://example.rw/meetings/)
- [Location and airport transfers](https://example.rw/location/)
- [Contact](https://example.rw/contact/)

Common questions

If I block GPTBot, will I vanish from ChatGPT?

OpenAI says its crawler settings are independent: you can allow OAI-SearchBot to appear in search results while disallowing GPTBot, which is used to collect content that may help train models (OpenAI: crawler documentation).

Does llms.txt make AI cite me?

There is no guarantee. Treat it as an optional summary file, and put your effort into clear pages and consistent facts first.

Where do I edit robots.txt on WordPress?

Many sites generate it virtually, and SEO plugins or the theme may add rules. Check the live address first, then find which plugin or file controls it. Avoid editing in more than one place.

A short launch checklist

1. Open /robots.txt and confirm there is no blanket block.
2. Confirm your sitemap link is listed and the sitemap opens.
3. Decide your policy on training crawlers and document it.
4. Create or update llms.txt with accurate facts.
5. Test key pages in Google Search Console after launch.
6. Review every six months, because crawler names and policies change.

How can Vatam help?

Our technical SEO service checks crawl access, sitemaps and speed, and our GEO service makes sure AI assistants can read and trust your site. This site publishes an llms.txt and welcomes search and AI crawlers, as you can see at /llms.txt.

Sources and further reading

• OpenAI: crawler documentation
• Perplexity: crawler documentation
• Anthropic: crawler documentation
• Google: common crawlers and the Google-Extended token
• Google: introduction to robots.txt
• Google: create and submit a robots.txt file
• llmstxt.org: the llms.txt proposal