Free tool, runs in your browser
Generate robots.txt and llms.txt, with AI crawler controls
Choose which paths search engines should skip, decide bot by bot whether AI companies may crawl you for training, search or on-demand fetches, and download a file that is ready to deploy. There is an llms.txt builder too.
robots.txt
User-agent: *
Disallow: /api/
robots.txt is not access control
Anyone can read this file at /robots.txt, and crawlers follow it voluntarily. Never list secret or admin paths here to hide them: that advertises them. Anything private needs a login and a server-side check.
Place the file at the root of your domain, for example https://your-app.example/robots.txt. Allowed AI crawlers fall under the * group, so they follow the same path rules as everyone else. Everything runs in your browser. Nothing you type is sent anywhere, and Download saves the file directly from this page.
llms.txt
# Your project name
- Add a project or site name. It is the only required part.
Follows the llmstxt.org layout: an H1 name, a blockquote summary, optional paragraphs, then H2 sections of links. Sections or links without a name and a full URL are left out. Serve it at /llms.txt.
llms.txt is a proposed convention, not a standard. Adoption by AI tools is limited and not guaranteed, and it does not replace robots.txt or a sitemap. Only link to pages you are happy to be public.
Training, search and user-triggered bots are different
AI companies now run several crawlers each, and blocking one does not block the others. Deciding what to allow is easier once you know which job each one does.
| Type | What it does | What blocking it means |
|---|---|---|
| Training (GPTBot, ClaudeBot, CCBot, Google-Extended, Applebot-Extended) | Collects pages that may be used to train AI models. Google-Extended and Applebot-Extended are opt-out tokens read by the normal crawler, not separate crawlers | Your content is less likely to end up in future model training. It does not remove you from normal search results |
| Search (OAI-SearchBot, Claude-SearchBot, PerplexityBot) | Indexes pages so an AI assistant can find and cite them | You are less likely to be linked as a source in that assistant's answers |
| User-triggered (ChatGPT-User, Claude-User, Perplexity-User) | Fetches one page because a person asked the assistant to read it | The assistant cannot open your page when a user asks. Some vendors say these fetches may not follow robots.txt |
Rules that cause trouble
- Blocking /_next/, /static/ or other folders of JavaScript and CSS. Google renders pages like a browser and needs those files. Without them your pages can look empty or broken to Google.
- Using Disallow to remove a page from Google. A disallowed page can still be indexed if other sites link to it. Use a noindex meta tag or header instead, and leave the page crawlable so Google can see it.
- Relative sitemap addresses. Sitemap lines must be full URLs, such as https://your-app.example/sitemap.xml.
- Leaving a staging rule of Disallow: / on the live site. It asks every crawler to skip the whole site.
What llms.txt is, and what it is not
llms.txt is a proposed convention from llmstxt.org: a Markdown file at /llms.txt that gives language models a short summary of your site and links to the pages that matter most. The format is simple. An H1 with the name, a blockquote summary, optional paragraphs, then H2 sections containing lists of links. A section called Optional marks links that can be skipped when a shorter context is needed.
It is not a standard, and it does not control crawling. Adoption by AI tools is limited and not guaranteed, so treat it as a cheap extra rather than a ranking lever. Google says Google Search does not use llms.txt, including for its AI features, and that the file neither helps nor hurts rankings. It never replaces robots.txt or a sitemap.
# Acme Invoices
> Invoicing for freelancers, with a REST API and Stripe payouts.
## Docs
- [Quickstart](https://acme.example/docs/quickstart.md): Create an account and send a first invoice
## Optional
- [Changelog](https://acme.example/changelog)Check what is live
After deploying, fetch both files from your own domain and read them back.
curl -s https://your-app.example/robots.txt
curl -s https://your-app.example/llms.txtFrequently asked questions
Is anything I type sent to a server?
No. Both generators run entirely in your browser. Copy uses your clipboard and Download creates the file on your device. Nothing is sent, stored or logged.
Will blocking GPTBot or ClaudeBot remove my site from ChatGPT or Claude?
Not by itself. Those are the training crawlers. OpenAI and Anthropic run separate search and user-triggered agents with their own names, which you control separately. Blocking only the training bots is what the "Allow search, block AI training" preset does.
Does blocking Google-Extended hurt my Google rankings?
Google says Google-Extended does not affect inclusion or ranking in Google Search. It only controls whether content Google crawls may be used for Gemini model training and grounding.
Can robots.txt protect my admin pages or API?
No. It is a public file that polite crawlers choose to follow. Protect admin pages and APIs with authentication and server-side authorization. Listing a sensitive path in robots.txt makes it easier to find.
Do I need an llms.txt file?
No. It is optional and adoption is limited, and Google says Google Search ignores it. If you have documentation or product pages that are useful to AI assistants, it costs little to add. Only link to pages that are already public.
Sources
- 1.RFC 9309: Robots Exclusion Protocol
- 2.Google Search Central: Introduction to robots.txt
- 3.Google Search Central: Google's common crawlers (Google-Extended)
- 4.Google Search Central: Optimizing for generative AI features (llms.txt)
- 5.OpenAI: Overview of OpenAI crawlers
- 6.Anthropic: Does Anthropic crawl data from the web?
- 7.Perplexity: Perplexity crawlers
- 8.llmstxt.org: The /llms.txt file