Choose which AI crawlers may search, fetch or train on your site, get an AI recommendation, and export a clean robots.txt. Free, no signup.
Free · live preview · AI policy advisorUser-agent: *)
robots.txt.Your robots.txt file tells crawlers which parts of your site they may request. AI companies now run several different crawlers, and they do different jobs: AI search crawlers such as OAI-SearchBot, Claude-SearchBot and PerplexityBot build the indexes behind AI answers; assistant fetchers such as ChatGPT-User and Claude-User load a page when a person asks about it; and training crawlers such as GPTBot, ClaudeBot and CCBot collect content that may be used to train models. Blocking all of them with one rule is rarely what a site owner wants, because blocking search crawlers can remove you from AI answers while blocking training crawlers does not.
This AI crawler robots.txt generator lets you set each crawler to Allow, Block or Not mentioned, groups them by purpose, and writes a tidy file with shared rules merged together. You can add private paths such as /admin/ or /cart/ for the crawlers you allow, add your sitemap, and keep any existing rules for other bots. Everything updates live as you click.
The AI policy advisor turns a plain description of your site and goals into a recommended setting for every crawler, with a short reason for each, and loads it into the controls so you can adjust it. The advisor can only choose from the crawlers listed on this page, and it is told that robots.txt is voluntary. If AI is not enabled, the presets and manual controls work the same way.
The robots.txt reader fetches your live file safely, applies the same longest-match, allow-wins-ties rules that major search engines use, and shows whether each AI crawler is currently allowed or blocked and by which rule, so you can see the effect of your file before and after the change. Two names in the list, Google-Extended and Applebot-Extended, are control tokens rather than separate crawlers; blocking them limits how the companies use your content for AI but does not change your normal Google or Apple search presence.
Add a group with "User-agent: GPTBot" followed by "Disallow: /". Set GPTBot to Block in the generator and it writes the correct lines for you.
Blocking training crawlers or the Google-Extended token does not change normal Google Search ranking. Blocking AI search crawlers can remove you from the AI answers that use their index, so choose deliberately.
GPTBot collects content for model training, OAI-SearchBot builds the index for ChatGPT search, and ChatGPT-User fetches a page when a person asks ChatGPT about it. They can be set independently.
No. Reputable operators say they follow it, but some crawlers have been reported to ignore it and user-triggered fetchers may be handled differently. For a firm block, add server or firewall rules too.
They are control tokens, not separate crawlers. They let you opt your content out of certain AI uses by Google and Apple without changing how your pages appear in normal search.
Most major crawlers do not honor Crawl-delay, so this tool does not add it. Use server rate limiting if a crawler is overloading your site.
Yes. Read your current file, tick "Keep the other rules" and the generator will carry over rules for bots that are not in its list and your sitemap lines.
The description is sent to the configured AI provider to produce a recommendation and is not saved by this tool. Do not include private information.
AI Crawler Robots.txt Generator & Policy Advisor: Control How AI Interacts With Your Site
Managing how search engines and automated web crawlers access your digital content has always been a core foundation of technical SEO. However, the rapid expansion of Artificial Intelligence (AI) technologies has fundamentally reshaped web traffic dynamics. Modern web crawlers no longer just index pages for traditional search result pages; they fetch real-time content for interactive AI assistants and scrape vast repositories of web data to train large language models (LLMs).
Blocking all automated agents with a single blanket directive is rarely optimal. Misconfiguring your robots.txt file can inadvertently strip your brand from AI-powered search engines (like ChatGPT Search or Perplexity) while failing to protect your intellectual property from training scrapers.
This interactive tool suite—combining an AI Crawler Robots.txt Generator, an AI Policy Advisor, and a Live Robots.txt Reader—is designed to give webmasters, SEO strategists, and site owners precise, granular control over AI user agents.
To configure your robots.txt effectively, you must understand that AI companies deploy distinct web crawlers for specific tasks. Blocking a company’s training bot does not necessarily mean you are blocking their search engine, and vice versa.
AI crawlers generally fall into three functional categories:
┌─────────────────────────────────────────┐
│ AI Web Crawlers │
└────────────────────┬────────────────────┘
│
┌──────────────────────────────┼──────────────────────────────┐
│ │ │
┌───────┴────────┐ ┌────────┴────────┐ ┌────────┴────────┐
│ AI Search │ │ Assistant │ │ Training │
│ Crawlers │ │ Fetchers │ │ Crawlers │
└───────┬────────┘ └────────┬────────┘ └────────┬────────┘
│ │ │
• OAI-SearchBot • ChatGPT-User • GPTBot
• Claude-SearchBot • Claude-User • ClaudeBot
• PerplexityBot • CCBot
Primary Examples: OAI-SearchBot, Claude-SearchBot, PerplexityBot
Purpose: These bots build and update search indexes that power generative AI answers and conversational discovery platforms.
Impact: Allowing these crawlers ensures your site's latest articles, products, or insights appear as cited sources when users ask questions on AI platforms. Blocking them removes your content from conversational search engines.
Primary Examples: ChatGPT-User, Claude-User
Purpose: These are real-time fetchers triggered directly by an end-user request. When a user pastes your URL into an AI chat interface and asks for a summary or critique, these fetchers load the page on demand.
Impact: Blocking assistant fetchers prevents users from interacting with your content inside AI chat interfaces, even if they explicitly request it.
Primary Examples: GPTBot, ClaudeBot, CCBot (Common Crawl)
Purpose: These scrapers systematically gather large datasets across the web to train future generations of AI models.
Impact: Blocking training bots prevents your site’s intellectual property from being included in offline training data without affecting your presence in real-time search results.
Important Note on Control Tokens:
User agents such as
Google-ExtendedandApplebot-Extendedare control tokens rather than separate web crawlers. AddingDisallowdirectives for these tokens signals to Google and Apple that you opt out of having your content used for generative AI development (e.g., Gemini or Apple Intelligence features) without impacting your core ranking or indexing in traditional Google or Apple Search.
This utility simplifies technical syntax creation while adhering to standard web protocol specifications (RFC 9309).
| Feature | Description | Webmaster Benefit |
| Granular Controls | Set individual crawlers to Allow, Block, or Not Mentioned. | Avoids accidental sitewide indexing blocks. |
| Live Rule Merging | Automatically combines shared rules and deduplicates paths. | Keeps your robots.txt lightweight and readable. |
| Private Path Shielding | Easily append disallow rules for sensitive paths (e.g., /admin/, /cart/). |
Protects backend resources across allowed bots. |
| Sitemap Integration | Injects dynamic XML sitemap declarations into the generated code. | Guides legitimate search bots to your key URLs. |
| Live Robots.txt Reader | Fetches and parses your live site's file using standard longest-match logic. | Tests rule updates before deploying them live. |
If you are unsure which setup suits your business model, the built-in AI Policy Advisor bridges the gap between high-level business strategy and technical execution.
Input Your Site Profile: Enter a brief plain-text description of your website, business model, and content goals (e.g., "We are a digital news publisher looking to appear in AI search engines but block training scrapers").
Automated Recommendation: The advisor analyzes your goals and selects recommended states (Allow, Block, or Omit) for every recognized AI bot, complete with a contextual rationale for each choice.
Control Injection: Recommended settings are instantly loaded into the interactive manual controls, allowing you to review, fine-tune, and inspect the syntax before publishing.

Decide whether your primary goal is maximizing reach (allowing search crawlers) or protecting proprietary data (blocking training scrapers).
Toggle the controls for each agent group. Add private paths that should never be crawled by any agent, such as internal search pages or checkout flows:
Plaintext
User-agent: * Disallow: /admin/ Disallow: /cart/ Disallow: /search/
Copy the cleanly formatted output generated by the tool and save it as robots.txt in the root directory of your web host (e.g., [https://example.com/robots.txt](https://example.com/robots.txt)).
Paste your live site URL into the Robots.txt Reader. The engine applies standard longest-match and allow-wins-ties rules (the exact parsing logic used by major search engines) to verify whether each AI crawler is correctly allowed or blocked by your active ruleset.
Robots.txt is Voluntary: Directives in robots.txt serve as instructions rather than technical enforcement walls. While reputable vendors (OpenAI, Anthropic, Google, Perplexity) strictly respect robots.txt rules, malicious scrapers may ignore them. Use authentication or server-level firewalls for sensitive data.
Avoid Conflicting Directives: Always review root User-agent: * rules. A sitewide Disallow: / under the wildcard agent will block all bots unless explicit Allow exceptions are set for specific crawlers.
Keep XML Sitemaps Visible: Ensure your XML sitemaps remain publicly accessible to allowed crawlers to streamline re-indexing after rule updates.
No. Blocking GPTBot only stops OpenAI from using your content for future model training. To remove your site from real-time ChatGPT search citations, you must block OAI-SearchBot.
Google-Extended affect my rankings on Google Search?No. Google-Extended is strictly used to manage access for Google's AI models (such as Gemini). Blocking it does not impact your site's visibility, indexing, or ranking in traditional Google Web Search.
robots.txt rules?Most major AI crawlers cache robots.txt files and update them every few hours to a day. Changes are generally reflected shortly after the updated file is fetched from your server.
Executive Summary & Final Verdict
Finding the right balance between brand visibility and data protection is now a critical task for site managers. The key takeaway is simple: Do not rely on single, blanket rules.
To preserve search visibility: Keep search crawlers like
OAI-SearchBot,Claude-SearchBot, andPerplexityBotset to Allow.To safeguard intellectual property: Toggle training bots like
GPTBot,ClaudeBot, andCCBotto Block.To manage platform AI usage: Add opt-out control tokens like
Google-ExtendedandApplebot-Extendedwithout jeopardizing your main search rankings.Use the AI Crawler Robots.txt Generator & Reader above to build, refine, and verify your file syntax in real time. Safeguard your site's data while maintaining full discovery across modern AI platforms today!