Your WordPress robots.txt file tells search engine crawlers which pages to crawl and which to skip. Get it wrong and Google wastes its crawl budget on your login pages, admin area, and search results instead of your actual content. This guide gives you the exact robots.txt setup for WordPress in 2026, three ways to edit it, and a free one-click generator.
What Is a Robots.txt File in WordPress?
A robots.txt file is a plain text file that sits at the root of your website at yourdomain.com/robots.txt. It uses a simple set of rules to tell crawlers — Google, Bing, and others — what they are and are not allowed to access. WordPress does not create this file automatically by default. Instead, it generates a virtual robots.txt in memory, which means you need to either use a plugin or create the file manually to take full control.
The file uses two main commands: Disallow (block this path) and Allow (permit this path). A User-agent line specifies which crawler the rule applies to. User-agent: * means the rule applies to all crawlers.
The Best Robots.txt for WordPress in 2026
Here is the exact robots.txt setup recommended for most WordPress sites. Copy this, replace yourdomain.com with your actual domain, and you are done:
User-agent: *
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /search/
Disallow: /?s=
Disallow: /cart/
Disallow: /checkout/
Disallow: /my-account/
Allow: /wp-admin/admin-ajax.php
# Block AI training crawlers
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: PerplexityBot
Disallow: /
User-agent: Bytespider
Disallow: /
Sitemap: https://yourdomain.com/sitemap_index.xmlDo not want to type this manually? Use our free WordPress Robots.txt Generator to build your robots.txt in 30 seconds with one click — no code required.
What Each Line Does
| Rule | What It Does | Why It Matters |
|---|---|---|
Disallow: /wp-admin/ | Blocks the WordPress admin area | Prevents Google from wasting crawl budget on login pages |
Disallow: /wp-includes/ | Blocks core WordPress files | These files have no SEO value and should not be indexed |
Disallow: /search/ and /?s= | Blocks WordPress search result pages | Duplicate content — Google penalises sites that index search pages |
Allow: /wp-admin/admin-ajax.php | Allows AJAX requests | Required for plugins that use AJAX — without this, some features break |
Sitemap: | Points crawlers to your sitemap | Helps Google discover all your pages faster |
3 Ways to Edit Robots.txt in WordPress
Method 1: RankMath (Recommended)
If you use RankMath SEO, it has a built-in robots.txt editor. Go to RankMath → General Settings → Edit robots.txt. Paste your rules and save. RankMath writes the changes directly to your actual robots.txt file on the server.
Method 2: Yoast SEO
In Yoast, go to SEO → Tools → File editor. You will see the robots.txt editor there. This only appears if WordPress has write access to the root directory, which it does on most shared hosting setups including Hostinger.
Method 3: FTP or File Manager
Log in to your Hostinger hPanel, open File Manager, navigate to public_html, and look for an existing robots.txt file. If one exists, edit it. If not, create a new file named robots.txt and paste your rules. This is the most reliable method as it bypasses WordPress entirely.
Where Is the Robots.txt File in WordPress?
Your WordPress robots.txt file is located in the root directory of your website — the same folder as wp-config.php. On most hosting setups this is public_html/robots.txt. You can verify it is working by visiting yourdomain.com/robots.txt in your browser. If you see plain text with your rules, it is working correctly.
If you see a default WordPress virtual robots.txt (usually just User-agent: * and Disallow: on separate lines with nothing else), you do not have a physical robots.txt file yet — use one of the methods above to create one.
Common WordPress Robots.txt Mistakes to Avoid
- Blocking /wp-content/uploads/ — Never do this. It blocks your images from being indexed in Google Images, which can be a source of traffic.
- Using Disallow: / on everything — This blocks your entire site from Google. Only use this on staging sites.
- Forgetting the sitemap line — Always include your sitemap URL. It is the most important line for SEO.
- Leaving the file blank — A blank robots.txt is fine, but you miss the opportunity to block wasted crawl budget and AI scrapers.
- Blocking CSS and JS — Google needs to render your pages to assess quality. Never block stylesheet or script files.
Skip the manual setup entirely with our free Robots.txt Generator for WordPress. Select your options, copy the output, and paste it into RankMath or your file manager. Takes under 60 seconds.
Frequently Asked Questions
Does WordPress automatically create a robots.txt file?
WordPress generates a virtual robots.txt in memory if no physical file exists. This default version is very basic — it only blocks wp-admin and allows everything else. You should replace it with a proper robots.txt using RankMath, Yoast, or by creating the file manually in your root directory.
Should I block wp-admin in robots.txt?
Yes. Always block /wp-admin/ in robots.txt, but add an exception for /wp-admin/admin-ajax.php. This prevents Google from crawling your login and admin pages while keeping AJAX-powered features working correctly for your visitors.
Where do I find the robots.txt file in WordPress?
Your robots.txt file is at yourdomain.com/robots.txt in the browser, and on the server it is in public_html/robots.txt — the same root folder as wp-config.php. If you are on Hostinger, access it via hPanel → File Manager → public_html.
Does robots.txt affect SEO?
Yes. A properly configured robots.txt improves SEO by directing Google’s crawl budget toward your important pages and away from admin pages, search results, and duplicate content. It does not directly boost rankings, but it prevents crawl budget waste which becomes important as your site grows.
Should I block AI crawlers in my WordPress robots.txt?
Yes, if you do not want your content used to train AI models. Add User-agent: GPTBot followed by Disallow: / for each AI bot you want to block. See our full guide on how to block AI crawlers in robots.txt for the complete list of bots and exact code.