AI Crawler robots.txt Builder

Jaspreet NagiUpdated 20 October 2026Free tool

What it does

Choose which AI crawlers may read your site and copy a ready robots.txt. The default keeps you visible in ChatGPT, Claude, Perplexity and Google while opting out of AI model training. It runs in your browser and nothing you type is saved.

Pick the bots

Search engines and AI search bots decide whether you can be found and cited. Training bots decide whether your pages are used to build future models.

My site runs on WordPress (keep the admin area private for every bot)

Other folders to keep privateOne path per line, starting with /. Leave empty if you have none.

Sitemap URLOptional, but it helps every crawler find your pages.

Your robots.txt

Replace the file at yoursite.com/robots.txt with this, or paste it into your SEO plugin's robots.txt editor.

How to use the builder

Start with a preset. Get cited, skip training is the setting most businesses want.
Change any single bot with its Allow or Block switch.
Keep the WordPress box ticked if your site runs on WordPress, add any other private folders, and paste your sitemap URL.
Copy the result or download it as a file, then replace your current robots.txt.

The three kinds of AI bot

Most AI companies run more than one crawler, and each one does a different job. Blocking the wrong one is the most common mistake: a site blocks GPTBot to stay out of training, then also blocks OAI-SearchBot and disappears from ChatGPT search.

AI search botsIndex pages so assistants can cite and link them.Allow to be cited

User-triggered fetchersOpen one page because a person asked. Some ignore robots.txt.Allow to be read

Training botsCollect pages to train future models. No effect on search.Your choice

Three kinds of AI bot. Only the first two affect whether AI answers can use and link your pages today.

Every bot in the builder

The list below comes from each company's own documentation, checked on 20 October 2026. It is updated when a company adds or renames a crawler, and the changes are logged on AI Search Updates.

User agent	Company	Type	What it does
Googlebot	Google	Search engine	Google Search, including AI Overviews and AI Mode
Bingbot	Microsoft	Search engine	Bing and Microsoft Copilot
OAI-SearchBot	OpenAI	AI search	Shows sites in ChatGPT search answers
Claude-SearchBot	Anthropic	AI search	Indexes pages for search answers in Claude
PerplexityBot	Perplexity	AI search	Surfaces and links sites in Perplexity answers
Applebot	Apple	AI search	Search features in Siri, Spotlight and Safari
Meta-WebIndexer	Meta	AI search	Search results and citations in Meta AI
ChatGPT-User	OpenAI	User-triggered	Opens a page when a ChatGPT user asks. OpenAI says robots.txt may not apply
Claude-User	Anthropic	User-triggered	Opens a page when a Claude user asks
Perplexity-User	Perplexity	User-triggered	Opens a page for a user question. Perplexity says it generally ignores robots.txt
Meta-ExternalFetcher	Meta	User-triggered	Fetches a link at a user's request. Meta says it may bypass robots.txt
GPTBot	OpenAI	Training	Collects pages to train OpenAI models
ClaudeBot	Anthropic	Training	Collects pages to train Claude models
Google-Extended	Google	Training token	Controls Gemini training and grounding. Google says it has no effect on Search
Applebot-Extended	Apple	Training token	Controls Apple model training. Apple says it does not crawl pages
Meta-ExternalAgent	Meta	Training	Collects pages to train Meta AI models
CCBot	Common Crawl	Training	Builds the open Common Crawl archive that many AI labs train on

Where to put the file

WordPress with Yoast SEO or Rank Math: both include a robots.txt editor in their settings. Paste the result there.
WordPress without an SEO plugin: WordPress serves a virtual robots.txt. Upload a real robots.txt file to the site's root folder and it takes over.
Shopify: edit the robots.txt.liquid template in your theme. Shopify adds its own default rules, so add yours alongside them.
Any other site: upload robots.txt to the root, so it opens at yoursite.com/robots.txt.

I co-founded wptaskify, so weigh this suggestion with that in mind: if your WordPress site is connected to it, you can ask Claude or ChatGPT to show and edit robots.txt for you. The manual routes above work just as well.

Check your firewall too

robots.txt only matters if the bot can reach your site. Cloudflare started blocking known AI crawlers by default on new domains on 1 July 2025, and some security plugins do the same. If you want to be cited, open your CDN or firewall settings and make sure the AI search bots you allowed here are not blocked there.

What robots.txt can't do

It is a request. Well-behaved crawlers follow it, and nothing forces the rest. To stop a bot for certain, block it in your firewall or CDN.

User-triggered fetchers may skip it. OpenAI, Perplexity and Meta all say some of their user-requested fetches may not follow robots.txt.

It does not remove pages that are already indexed. Use a noindex tag for that, and leave the page crawlable so Google can see the tag.

It does not hide private content. Anyone can read your robots.txt, so never list secret folders in it. Protect them with a password.

Questions people ask

Will blocking GPTBot remove me from ChatGPT?

No. OpenAI treats them separately. GPTBot is for training. OAI-SearchBot decides whether you appear in ChatGPT search answers, so keep that one allowed.

Does blocking Google-Extended hurt my Google rankings?

No. Google says Google-Extended does not affect inclusion in Google Search and is not a ranking signal. AI Overviews and AI Mode use Googlebot, so blocking Google-Extended does not remove you from them either.

How long do changes take?

Crawlers fetch robots.txt again every so often, usually within a day. Perplexity says changes can take up to 24 hours to show in its systems.

Is my data stored?

No. The builder runs in your browser and nothing you type is sent anywhere.

Sources

Want your site set up for AI search?We check crawler access, schema and content so AI answers can find and cite you.

See AEO & GEO services