Robots.txt Generator
Generate a robots.txt file for your website visually. Control which pages search engine crawlers can access. Free, no sign-up.
One path per line, e.g. /admin/ or /private/
robots.txt output
Frequently asked
questions
Robots.txt is a text file placed at the root of your website (e.g., example.com/robots.txt) that tells search engine crawlers which pages or sections they should not visit. It's part of the Robots Exclusion Protocol (REP).
No — robots.txt only controls crawling, not indexing. A page that isn't crawled can still be indexed if it has inbound links. To prevent indexing, use the noindex meta tag or X-Robots-Tag header instead.
Well-behaved bots from Google, Bing, and most legitimate crawlers respect robots.txt. However, malicious bots and scrapers typically ignore it. Robots.txt is a courtesy instruction, not a security mechanism.