Free SEO Tool
Robots.txt Builder
Tell search engines what to crawl. Pick a preset, add your own rules and sitemap, and copy a clean, valid robots.txt in seconds.
Configure
Preset
Crawlers
Disallow paths
Allow paths optional exceptions
Sitemap & crawl-delay
Your robots.txt
Upload the file to the root of your domain so it lives at https://yourdomain.com/robots.txt. Test it in Google Search Console before relying on it.
Not sure what to block?
A misconfigured robots.txt can quietly hide your best pages from Google. Get a free technical SEO audit and we'll check crawlability end-to-end.
Get my free technical audit →What is a robots.txt file?
A robots.txt file lives at the root of your domain and tells search-engine crawlers which parts of your site they may or may not request. Used well, it keeps low-value pages out of crawl budget; used carelessly, it can accidentally hide pages you want to rank.
Does Disallow remove a page from Google?
No. Disallow stops crawling, but a blocked URL can still appear in results if other sites link to it. To keep a page out of the index, allow crawling and use a noindex meta tag instead.
Should I block AI bots?
It's your call. Blocking training crawlers like GPTBot won't affect your Google Search rankings, but it opts your content out of some AI datasets.