Free Robots.txt Generator
Create a clear starting robots.txt file for ordinary search engines, social platforms and AI crawlers. Always review the result against your own website before publishing it.
What does robots.txt control?
A robots.txt file gives participating crawlers instructions about which site paths they may access. It does not prevent indexing by itself or replace page-level noindex directives.