To block AI crawlers in your robots.txt file, add a User-agent line with the bot name followed by Disallow: /. This page gives you the exact code for GPTBot, ClaudeBot, Google-Extended, PerplexityBot, and 10 more AI scrapers — plus a free generator that builds your complete robots.txt in 30 seconds.
The Complete List of AI Crawlers to Block in 2026
New AI bots appear regularly. Here is the current list of known AI training and scraping crawlers, their operators, and whether blocking them is recommended:
| Bot Name (User-agent) | Operated By | Purpose | Block Recommended? |
|---|---|---|---|
| GPTBot | OpenAI (ChatGPT) | Crawls content to train GPT models | Yes — if you don’t want content in ChatGPT training |
| ClaudeBot | Anthropic (Claude) | Crawls content to train Claude AI | Yes — same reason |
| Google-Extended | Google (Gemini / Bard) | Trains Google’s Gemini AI models | Yes — separate from Googlebot for search |
| PerplexityBot | Perplexity AI | Indexes content for AI search answers | Yes — answers your content without linking back |
| CCBot | Common Crawl | Builds open dataset used to train many AI models | Yes — feeds dozens of AI training projects |
| Bytespider | ByteDance (TikTok) | Crawls for AI and content recommendations | Yes — aggressive crawler with high server load |
| omgili / omgilibot | Webhose / Bright Data | Content aggregation and AI datasets | Yes |
| Diffbot | Diffbot | Structured data extraction for AI | Yes |
| FacebookBot | Meta | Trains Meta AI models | Optional — also used for link previews |
| anthropic-ai | Anthropic | Alternative Claude crawler identifier | Yes — block alongside ClaudeBot |
| cohere-ai | Cohere | Trains Cohere language models | Yes |
| YouBot | You.com | AI-powered search engine crawler | Optional |
Exact Robots.txt Code to Block All AI Crawlers
Copy and paste this block into your robots.txt file. It blocks all major AI training and scraping bots while leaving Google Search (Googlebot) fully operational:
# Block OpenAI GPTBot
User-agent: GPTBot
Disallow: /
# Block Anthropic ClaudeBot
User-agent: ClaudeBot
Disallow: /
User-agent: anthropic-ai
Disallow: /
# Block Google AI training (keeps Googlebot for search)
User-agent: Google-Extended
Disallow: /
# Block Perplexity AI
User-agent: PerplexityBot
Disallow: /
# Block Common Crawl (feeds many AI datasets)
User-agent: CCBot
Disallow: /
# Block ByteDance / TikTok crawler
User-agent: Bytespider
Disallow: /
# Block Diffbot
User-agent: Diffbot
Disallow: /
# Block Cohere AI
User-agent: cohere-ai
Disallow: /
# Block omgili
User-agent: omgili
Disallow: /
User-agent: omgilibot
Disallow: /Want this built automatically? Our free Robots.txt Generator includes an AI crawler blocking option — tick the checkbox and your complete file is ready to copy in seconds.
How to Add This to Your WordPress Robots.txt
Method 1: RankMath SEO (Easiest)
Go to RankMath → General Settings → Edit robots.txt. Paste the AI crawler block above your existing rules. Click Save. Done.
Method 2: Hostinger File Manager
Log into Hostinger hPanel → File Manager → public_html. Find robots.txt, click Edit, paste the block at the top of the file, save. Visit yourdomain.com/robots.txt to confirm the changes are live.
Method 3: Use the Generator
Use our free Robots.txt Generator. Select “Block AI Crawlers”, choose your other settings, and copy the output. It builds the entire file including your sitemap URL and standard WordPress rules automatically.
Does Blocking AI Crawlers Hurt Your SEO?
No — blocking AI crawlers does not affect your Google search rankings. The key is understanding that Googlebot (used for Google Search) is completely separate from Google-Extended (used for Gemini AI training). Blocking Google-Extended has no effect on how your site ranks in Google Search results.
The same applies to OpenAI, Anthropic, and other AI companies — their AI training crawlers are entirely separate from any search or discovery function. Blocking them only prevents your content from being used in AI training datasets.
Frequently Asked Questions
What is the robots.txt code to block GPTBot?
To block GPTBot, add these two lines to your robots.txt file: User-agent: GPTBot on the first line, then Disallow: / on the second. This tells OpenAI’s crawler it is not permitted to access any page on your site.
How do I block ClaudeBot in WordPress?
Add User-agent: ClaudeBot followed by Disallow: / in your WordPress robots.txt file. Also add a separate block for anthropic-ai as Anthropic uses both identifiers. Edit the file via RankMath → General Settings → Edit robots.txt, or directly in Hostinger File Manager under public_html.
Does blocking AI crawlers affect Google Search ranking?
No. Googlebot (search crawler) and Google-Extended (AI training crawler) are completely separate. Blocking Google-Extended only prevents your content from being used to train Gemini AI — it has zero effect on your Google Search rankings. The same applies to all other AI bots.
What is GPTBot, ClaudeBot, and PerplexityBot blocking policy for 2026?
In 2026, all major AI companies — OpenAI (GPTBot), Anthropic (ClaudeBot), and Perplexity (PerplexityBot) — honour robots.txt disallow rules. If you add Disallow: / under their user-agent, they will stop crawling your site. This is a voluntary compliance policy; there is no legal requirement, but all major players currently follow it.
Should I block all AI crawlers or just some?
Block all AI training crawlers if you do not want your content used to train AI models without compensation. However, consider keeping PerplexityBot if you want to appear in Perplexity AI answers — it can drive referral traffic. Google-Extended is safe to block since it only affects Gemini training, not search.
How do I check if AI crawlers are visiting my site?
Check your server access logs in Hostinger hPanel → Logs. Search for GPTBot, ClaudeBot, or CCBot in the log file. Alternatively, install a plugin like WP Security Audit Log or check Cloudflare analytics if you use it — it shows bot traffic by user-agent string.
Build your complete blocking robots.txt in 30 seconds with our free Robots.txt Generator. No code knowledge needed — just tick the AI crawler option and copy the output.