WordPress Robots TXT: Best Setup Guide for Bloggers (2026)

f you run a WordPress site, getting your wordpress robots txt file right is one of those small tasks that actually matters for SEO. When it is set up wrong, Google wastes time crawling admin pages and search results instead of your real content — and that eats into the crawl budget that should go toward your posts and pages.

This guide shows you exactly what your WordPress robots.txt should look like, how to add it in three different ways, and how to test it once it is done. If you just want to generate the file without doing it manually, use our free Robots.txt Generator — it builds the correct file for your setup in about 30 seconds.

What WordPress Robots.txt Should Block (and Why)

WordPress creates a lot of URLs that are not useful to search engines. If you leave robots.txt at its default state, Google wastes time crawling these pages. Over time that eats into the crawl budget that should go toward your posts and pages.

Here are the paths you should block for a standard WordPress blog:

Admin and login pages — these should never appear in search results. Block /wp-admin/ entirely. The only exception is admin-ajax.php, which needs to stay accessible for plugins that use it on the front end.

Duplicate content paths — WordPress generates /feed/, /trackback/, and /?s= (search results) URLs automatically. None of these should be indexed. Search result pages are especially problematic because they produce hundreds of thin pages with near-duplicate content.

Staging and utility paths — /wp-includes/ files are WordPress core assets. There is no SEO value in Google indexing your jQuery files.

Tag and category archives (optional) — if your site has hundreds of tags with only one or two posts each, blocking them prevents thin page problems. Skip this if your categories are well-organized with substantial content.

The Best WordPress Robots TXT Template for Blogs

This is the template I recommend for a standard WordPress blog. It covers the most common issues without being overly aggressive.

User-agent: *
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /feed/
Disallow: /trackback/
Disallow: /?s=
Disallow: /search/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://yoursite.com/sitemap_index.xml

Replace yoursite.com with your actual domain. If you use RankMath, your sitemap URL is typically https://yoursite.com/sitemap_index.xml. If you use Yoast, it is https://yoursite.com/sitemap.xml.

Do not want to write this manually or worry about getting the sitemap URL wrong? The Robots.txt Generator fills everything in for you based on your site type and SEO plugin choice.

What this template does not block: your pages, posts, categories, tags (unless you have the thin tag problem), images, and media files. These should all stay accessible so Google can crawl and index them normally.

wordpress robots txt file setup in dashboard

How to Add WordPress Robots txt to (3 Methods)

WordPress does not give you a wordpress robots txt file by default. There are three ways to add or edit one, depending on your setup.

Method 1 — Manual via FTP or File Manager

This is the most direct method. It works on any WordPress hosting.

  1. Log in to your hosting control panel and open File Manager, or connect via FTP using FileZilla
  2. Navigate to the root of your site — this is the folder that contains wp-config.php and the wp-content folder
  3. Check if a robots.txt file already exists. If it does, right-click and edit it. If not, create a new file called robots.txt
  4. Paste your robots.txt content and save

One thing to know: WordPress auto-generates a virtual robots.txt if you do not have a real one. The auto-generated one is very basic — it only blocks /wp-admin/. Creating a real file overrides that virtual one FileZilla FTP client.

Method 2 — Via RankMath SEO

If you use RankMath (which this site uses), you can edit wordpress robots txt directly from the WordPress dashboard without touching any files.

  1. In your WordPress dashboard, go to RankMath → General Settings → Edit robots.txt
  2. You will see the current robots.txt content in a text editor
  3. Paste or edit the content and click Save Changes

This is the easiest method if you are already using RankMath. Changes take effect immediately.

Method 3 — Via WPCode Lite

WPCode Lite is better suited for injecting scripts, but if you need to manage robots.txt as part of a broader code management workflow, RankMath’s built-in editor is cleaner for this specific task. Stick with Method 2 if you have RankMath installed.

how to add wordpress robots txt three methods

Should You Block AI Crawlers on Your WordPress Site?

This has become a real question in 2025 and 2026. AI companies like OpenAI, Anthropic, Apple, and Meta are running crawlers that scrape content for training data. These crawlers are separate from Google and Bing — blocking them does not affect your search rankings at all.

Whether to block them is a personal decision. Some site owners are fine with their content being used for AI training. Others prefer to opt out.

If you want to block the major AI crawlers, add these lines to your robots.txt:

User-agent: GPTBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: anthropic-ai
Disallow: /

User-agent: Applebot-Extended
Disallow: /

These go above your general User-agent: * block. Each one targets a specific crawler by name. Blocking these does not remove your site from AI products that have already crawled it — it only prevents future crawls Google’s robots.txt documentation .

If you want a clean file that handles both standard WordPress optimization and AI crawler blocking together, generate it with the free Robots.txt Generator — it has a toggle for AI crawlers so you do not have to add these lines manually.

How to Test Your Robots.txt After Adding It

Once your robots.txt is live, you want to confirm Google can read it and that you have not accidentally blocked something important.

Step 1 — View your live file Go to https://yoursite.com/robots.txt in your browser. You should see the plain text file. If you see a 404 or a WordPress page, the file was not saved to the right location.

Step 2 — Use Google Search Console In GSC, go to Settings → robots.txt (or search “robots.txt” in the GSC navigation). Google shows you the last time it fetched your file and flags any issues it found.

Step 3 — Check that important pages are not blocked In GSC, go to URL Inspection and enter one of your important pages — a blog post or your homepage. Under “Coverage,” it will tell you if that page is blocked by robots.txt. If it is, something in your robots.txt is too broad Google Search Console .

Common mistake: Using Disallow: /wp-content/uploads/ by accident. This blocks all your media files from being crawled, which can hurt image search traffic. Unless you have a specific reason to block uploads, leave that path out.

test wordpress robots txt in google search console

Wrapping Up

Your WordPress robots txt file is small — usually under 20 lines — but it has a real effect on how Google crawls your site. A properly set up file stops crawl budget waste, keeps admin pages out of search results, and points Google directly to your sitemap.

The template in this guide covers the most common WordPress setup. If you have a WooCommerce store, a membership site, or a multisite installation, your robots.txt will need a few extra lines for those specific paths.

The fastest way to get this right without second-guessing yourself: use the Robots.txt Generator to build the file for your exact setup, then add it to WordPress using one of the three methods above.

Once it is live, verify it in Google Search Console and check that your main pages are still accessible. That is the entire job done.