Robots.txt Generator for AI
Control which participating crawlers can access your site, including GPTBot, ClaudeBot, PerplexityBot, Google-Extended, Amazonbot and Applebot-Extended. Choose settings that match your content, business and data-use preferences; crawler access alone does not guarantee AI-search visibility.
Choose a preset
# robots.txt - generated by Think SEO # https://think-seo.co.uk/free-tools/robots-txt-generator # Default rule - allow all search engines User-agent: * Allow: / Disallow: /admin/ Disallow: /private/ Sitemap: https://yourdomain.com/sitemap.xml
How to deploy
- 1Download or copy the generated file
- 2Upload to your website's root directory
- 3Verify at yoursite.com/robots.txt
- 4Test with Google Search Console
Frequently Asked Questions
What is a robots.txt file?
A robots.txt file is a plain text file placed in your website's root directory that tells web crawlers which pages or sections of your site they can or cannot access. It follows the Robots Exclusion Protocol and is the first file most crawlers check when visiting your site.
Why should I control AI crawler access?
Some crawlers support model training, web search or product features, while others have different roles. Allowing a crawler does not guarantee inclusion in any AI response. Blocking a crawler may be appropriate for proprietary content; the right choice depends on your content, legal and commercial preferences.
What's the difference between Googlebot and Google-Extended?
Googlebot crawls your site for traditional Google Search indexing. Google-Extended is a separate crawler used specifically for Google's Gemini AI products. You can block Google-Extended (to prevent AI training) while still allowing Googlebot (to maintain Google Search rankings).
What's the difference between robots.txt and llms.txt?
robots.txt gives participating crawlers access instructions. llms.txt is an optional plain-text convention for systems that choose to support it. They are different tools, and neither replaces useful crawlable content, accurate business information or standard SEO fundamentals.
How does robots.txt affect AI search visibility?
Blocking a specific crawler can prevent that crawler from accessing pages, subject to its published behaviour. It does not by itself determine Google Search or Google AI feature eligibility: Googlebot controls Google Search crawling, while Google-Extended is a separate control. Allowing a crawler is not a ranking, citation or recommendation guarantee.