Free Tool

Robots.txt Generator for AI

Control which participating crawlers can access your site, including GPTBot, ClaudeBot, PerplexityBot, Google-Extended, Amazonbot and Applebot-Extended. Choose settings that match your content, business and data-use preferences; crawler access alone does not guarantee AI-search visibility.

Choose a preset

23 crawlers allowed
AI Assistants
GPTBotTraining
OpenAI (ChatGPT) - Powers ChatGPT knowledge and responses
ClaudeBotTraining
Anthropic (Claude) - Powers Claude AI assistant
anthropic-aiTraining
Anthropic - Anthropic general crawler
Google-ExtendedTraining
Google (Gemini) - Google Gemini AI training - separate from Googlebot
AmazonbotTraining
Amazon (Alexa) - Amazon AI assistants and Alexa
Applebot-ExtendedTraining
Apple (Siri) - Apple AI features and Siri
BytespiderTraining
ByteDance (TikTok) - TikTok and ByteDance AI products
cohere-aiTraining
Cohere - Cohere enterprise AI products
meta-externalagentTraining
Meta (Facebook) - Meta AI training and features
AI Search Engines
OAI-SearchBot
OpenAI (Search) - OpenAI real-time web search
PerplexityBot
Perplexity AI - Powers Perplexity AI search
YouBot
You.com - You.com AI search engine
AI Training Crawlers
CCBotTraining
Common Crawl - Creates datasets used for AI model training
DiffbotTraining
Diffbot - AI-powered web data extraction
omgiliTraining
Webz.io - Web data aggregation for AI
Traditional Search Engines
Applebot
Apple (Search) - Apple Spotlight and Siri search
Googlebot
Google Search - Google traditional search indexing
Bingbot
Microsoft Bing - Bing search engine crawler
DuckDuckBot
DuckDuckGo - DuckDuckGo search crawler
Slurp
Yahoo Search - Yahoo search engine crawler
Social Preview Bots
Twitterbot
X (Twitter) - Twitter/X link preview generation
LinkedInBot
LinkedIn - LinkedIn link preview generation
facebookexternalhit
Facebook - Facebook link preview generation
robots.txt
# robots.txt - generated by Think SEO
# https://think-seo.co.uk/free-tools/robots-txt-generator

# Default rule - allow all search engines
User-agent: *
Allow: /
Disallow: /admin/
Disallow: /private/

Sitemap: https://yourdomain.com/sitemap.xml

How to deploy

  1. 1Download or copy the generated file
  2. 2Upload to your website's root directory
  3. 3Verify at yoursite.com/robots.txt
  4. 4Test with Google Search Console

Need a full AI SEO audit?

Get a complete technical SEO report including AI crawler access, schema markup, Core Web Vitals and more.

Frequently Asked Questions

What is a robots.txt file?

A robots.txt file is a plain text file placed in your website's root directory that tells web crawlers which pages or sections of your site they can or cannot access. It follows the Robots Exclusion Protocol and is the first file most crawlers check when visiting your site.

Why should I control AI crawler access?

Some crawlers support model training, web search or product features, while others have different roles. Allowing a crawler does not guarantee inclusion in any AI response. Blocking a crawler may be appropriate for proprietary content; the right choice depends on your content, legal and commercial preferences.

What's the difference between Googlebot and Google-Extended?

Googlebot crawls your site for traditional Google Search indexing. Google-Extended is a separate crawler used specifically for Google's Gemini AI products. You can block Google-Extended (to prevent AI training) while still allowing Googlebot (to maintain Google Search rankings).

What's the difference between robots.txt and llms.txt?

robots.txt gives participating crawlers access instructions. llms.txt is an optional plain-text convention for systems that choose to support it. They are different tools, and neither replaces useful crawlable content, accurate business information or standard SEO fundamentals.

How does robots.txt affect AI search visibility?

Blocking a specific crawler can prevent that crawler from accessing pages, subject to its published behaviour. It does not by itself determine Google Search or Google AI feature eligibility: Googlebot controls Google Search crawling, while Google-Extended is a separate control. Allowing a crawler is not a ranking, citation or recommendation guarantee.

Chat with us on WhatsApp