Free robots.txt generator
Build a valid robots.txt in seconds, and control whether AI crawlers like GPTBot, ClaudeBot, and PerplexityBot can read your site. Free, no signup.
Allow them so ChatGPT, Claude, and Perplexity can find and cite you. Block them to keep your content out of AI training and answers.
User-agent: * Allow: / Disallow: /admin Disallow: /cart
Save this as robots.txt in your site root (yourdomain.com/robots.txt).
A robots.txt file is a plain text file at the root of your domain that tells crawlers which parts of your site they can and cannot access. In 2026 it does double duty: it guides Google and Bing, and it is also where you decide whether AI crawlers like OpenAI's GPTBot and Anthropic's ClaudeBot are allowed to read and learn from your content. This generator handles both, then you copy or download the file and drop it at yourdomain.com/robots.txt.
How to use this robots.txt generator
Pick a default policy: allow all crawlers (the right choice for almost every site that wants to be found) or block all. Add your sitemap URL so crawlers can discover your pages faster. If you have private areas like an admin or cart, add them under Disallow paths.
Then set your AI crawler rules. Toggle each one on to allow it or off to block it, and use the output below. When you are happy, copy the file or download it and upload it to your site root.
Should you allow or block AI crawlers?
This is the question robots.txt did not used to answer. If you want ChatGPT, Claude, Perplexity, and Gemini to find, cite, and recommend your brand, allow their crawlers. Blocking them keeps your content out of AI answers, which for most businesses is the opposite of what you want now that buyers research in AI.
There are real reasons to block: protecting proprietary content, or keeping a site out of model training. But understand the trade-off. A blocked site cannot be cited. We default the AI crawlers to allowed for exactly this reason, getting cited by AI is the whole game.
Common robots.txt mistakes
The most common one is accidentally blocking your whole site with Disallow: / left over from a staging environment. The second is blocking CSS and JS folders, which stops Google rendering your pages properly. The third is forgetting the sitemap line. This tool avoids all three by default.
Remember that robots.txt is a request, not a lock. Well-behaved crawlers respect it; it is not a security measure. For truly private content, use authentication, not robots.txt.
Want to know if AI is actually crawling you?
RankCite tracks which AI crawlers hit your site and whether ChatGPT, Claude, and Perplexity cite you, alongside your Google Search Console and GA4 data in one dashboard.
Start free for 7 daysRobots.txt Generator: FAQ
What is a robots.txt file?+
A robots.txt file is a plain text file at the root of your domain (yourdomain.com/robots.txt) that tells web crawlers which parts of your site they may access. It is the first thing most crawlers, including Google and AI bots, check when they visit.
How do I block AI crawlers like ChatGPT from my site?+
Add a rule for each AI crawler with Disallow: /. This generator does it for you: toggle off GPTBot, ClaudeBot, PerplexityBot, Google-Extended, and others, and the correct rules are written automatically. Note that blocking them means AI engines cannot cite your content.
Should I allow AI crawlers?+
For most businesses, yes. If you want to be found and recommended in ChatGPT, Claude, Perplexity, and Gemini answers, their crawlers need access. Blocking them protects content from AI training and answers but removes you from AI search entirely.
Where do I put the robots.txt file?+
In your site's root directory, so it is reachable at yourdomain.com/robots.txt. It must be at the root; a robots.txt in a subfolder is ignored.
Is this robots.txt generator free?+
Yes, completely free with no signup. Build your file, copy or download it, and you are done.
Does robots.txt help my SEO?+
Indirectly. A correct robots.txt lets crawlers reach the pages that matter and points them to your sitemap, while keeping low-value pages out of the crawl. A broken one can hide your whole site from search, so getting it right matters.
More free tools
All toolsPreview how your brand shows up across ChatGPT, Claude, Perplexity, and Gemini. Free, instant, no signup.
Build and validate an llms.txt that tells AI crawlers like ChatGPT and Claude which pages to read. With examples and a validator.
Generate valid JSON-LD for FAQ, Article, Product, Organization, Breadcrumb, How-To, and more. Copy, download, test.