Robots.txt disallow all is the command that blocks every search engine crawler from your entire website. Adding Disallow: / under User-agent: * in your robots.txt file tells Google, Bing, and all other crawlers to stop visiting every page you own. This is one of the most powerful commands in robots.txt — and the most dangerous on a live site. This guide covers exactly when robots.txt disallow all is appropriate, the correct syntax for every scenario, the SEO consequences, and how to safely remove it when you are ready to launch.
What Does Robots.txt Disallow All Mean?
The robots.txt disallow all command uses two lines to block every crawler from accessing every page on your site:
User-agent: *
Disallow: /User-agent: * means “all crawlers”. Disallow: / means “block access to the root directory and everything inside it” — which is every page, post, image, and file on your domain. Together, these two lines instruct Google, Bing, and all other compliant crawlers to crawl nothing on your website.
This is different from using noindex meta tags. The noindex tag tells crawlers “crawl this page but do not include it in search results”. Robots.txt disallow all tells crawlers “do not even visit this page”. The distinction matters enormously for how quickly pages are removed from Google’s index, as we explain below.
When Should You Use Robots.txt Disallow All?
There are exactly four legitimate use cases for robots.txt disallow all. Outside of these situations, it should never appear on a live website:
| Use Case | Why Disallow All Is Correct Here | When to Remove It |
|---|---|---|
| Staging / development site | Prevents Google from indexing a half-built site with duplicate content | Before launching the live site |
| Site under construction | Stops crawlers from indexing placeholder pages and coming soon content | When you publish real content |
| Password-protected intranet | Internal sites behind a login have no need for public search indexing | If site ever goes public |
| Archived / retired domain | Old domains kept alive for redirects should not compete in search | If the domain is repurposed |
Never use robots.txt disallow all on a live public website that you want to appear in Google search results. It will cause your rankings to drop to zero within weeks as Googlebot stops crawling and eventually de-indexes your pages.
Robots.txt Disallow All: Correct Syntax for Every Situation
Block all crawlers from all pages (full disallow all)
User-agent: *
Disallow: /Block all crawlers except Google (allow Googlebot only)
User-agent: Googlebot
Disallow:
User-agent: *
Disallow: /An empty Disallow: means “allow everything”. This lets Googlebot crawl normally while blocking all other crawlers.
Block all crawlers except Google and Bing
User-agent: Googlebot
Disallow:
User-agent: Bingbot
Disallow:
User-agent: *
Disallow: /Block a specific section only (not full disallow all)
If you only want to block a specific folder — like a staging subfolder or an admin area — use a path instead of /:
User-agent: *
Disallow: /staging/
Disallow: /wp-admin/
Disallow: /private/Want these generated automatically without writing the syntax manually? Use our free Robots.txt Generator — select the paths you want to block and copy the correctly formatted output in seconds.
What Happens to SEO When You Use Robots.txt Disallow All?
The SEO impact of robots.txt disallow all depends on whether it is on a staging subdomain or your main live domain:
| Scenario | SEO Impact | Timeline |
|---|---|---|
| New site, never indexed, disallow all | None — Google has nothing to de-index | Immediate when removed |
| Staging subdomain (staging.yourdomain.com) | None to main domain — subdomains are independent | No impact on live site |
| Live site, already ranked, disallow all added | Rankings drop to zero within 4–8 weeks as Googlebot stops crawling | Pages begin disappearing from index in 1–2 weeks |
| Live site, disallow all removed after brief period | Rankings usually recover within 4–8 weeks of removal | Full recovery takes 1–3 months depending on site age |
| Live site, disallow all left on for 6+ months | Severe — domain authority drops, backlinks lose value, full recovery uncertain | Can take 6–12 months to fully recover |
According to Google’s official robots.txt documentation, crawlers respect robots.txt on a voluntary basis, and Google checks the robots.txt file before every crawl. This means that once you add disallow all, Google will typically stop visiting your pages within days — and previously indexed pages will begin dropping out of search results within 1–2 weeks.
How to Set Up Disallow All on a WordPress Staging Site
The most common legitimate use of robots.txt disallow all is on a WordPress staging or development site. Here is the correct setup process:
Step 1: Confirm your staging URL is separate from your live domain
Your staging site should be on a separate subdomain (e.g. staging.yourdomain.com) or a completely different domain. If your staging site is a subfolder of your live domain (e.g. yourdomain.com/staging/), use a path-specific disallow instead of a full disallow all, to avoid accidentally blocking your live content.
Step 2: Add the disallow all rule via RankMath
In WordPress, go to RankMath → General Settings → Edit robots.txt. Replace any existing content with:
User-agent: *
Disallow: /Save the changes. Visit yourstagingdomain.com/robots.txt in your browser to confirm the file is live and contains the correct rules.
Step 3: Verify in Google Search Console
If your staging domain is added to Google Search Console, go to Settings → robots.txt to confirm Google has picked up the disallow all rule. You can also use the GSC robots.txt tester to verify that specific URLs on your staging site are being blocked correctly.
How to Remove Robots.txt Disallow All When You Go Live
When your site is ready to launch and you want Google to start crawling, removing robots.txt disallow all is a three-step process:
- Replace the disallow all with your proper live robots.txt. Go to RankMath → General Settings → Edit robots.txt. Replace
Disallow: /with your full live site robots.txt rules — blocking only wp-admin, search pages, and other non-content URLs. Use our free Robots.txt Generator to build the correct file instantly. - Submit your sitemap to Google Search Console. Go to GSC → Sitemaps and submit your sitemap URL (usually
yourdomain.com/sitemap_index.xmlfor RankMath users). This tells Google where all your pages are so it can start crawling them quickly. - Request indexing for your key pages. In GSC, go to URL Inspection for each of your most important pages and click Request Indexing. This speeds up the crawl after the disallow all is lifted.
After removing robots.txt disallow all, Google typically starts crawling your site within 24–72 hours. Pages begin appearing in search results within 1–2 weeks for a brand new site, or faster if the site was previously indexed and is returning after a brief period of being blocked.
Robots.txt Disallow All vs Noindex: What Is the Difference?
Many site owners confuse robots.txt disallow all with the noindex meta tag. They produce different outcomes:
| Robots.txt Disallow All | Noindex Meta Tag | |
|---|---|---|
| What it does | Prevents crawlers from visiting the page | Allows crawlers to visit but tells them not to index it |
| Removes from Google index? | Indirectly — Google eventually de-indexes pages it cannot crawl | Directly — Google removes the page from its index on next crawl |
| How quickly it takes effect | 1–2 weeks for pages to disappear from index | Typically within 1–2 crawl cycles (days to weeks) |
| Best for | Blocking an entire site or section during development | Removing specific pages (thin content, tag pages, search results) |
| Works on disallowed pages? | N/A — if a page is disallowed, crawlers cannot read its meta tags | Only works if the page can be crawled |
| Recommended for live sites | No — use only for staging or development | Yes — safe to use on specific pages you want de-indexed |
Important: You cannot use both simultaneously for the same pages. If you block a page with robots.txt disallow all, Google cannot read its noindex tag. If you want a page removed from Google’s index on a live site, use noindex via RankMath, not robots.txt disallow all.
Common Robots.txt Disallow All Mistakes and How to Fix Them
- Leaving disallow all on a live site after launch. This is the most common and damaging mistake. If you migrated from a staging setup and forgot to update robots.txt, check yourdomain.com/robots.txt right now. If you see
Disallow: /, fix it immediately. - Using disallow all to hide thin content instead of noindex. If you have low-quality pages you want out of Google’s index, use RankMath’s noindex option per page, not a blanket robots.txt disallow all.
- Not testing after making changes. Always visit yourdomain.com/robots.txt after any edit to confirm the file looks correct. A misplaced character can corrupt the entire file silently.
- Blocking a subfolder when you meant to block a subdomain.
Disallow: /staging/blocks the subfolder path. It has no effect onstaging.yourdomain.com, which is a separate subdomain with its own robots.txt. - Forgetting to remove it before submitting a site to AdSense. Google AdSense reviewers check whether Googlebot can access your content during the approval review. A robots.txt disallow all will cause automatic rejection of your AdSense application.
Need to build a proper robots.txt file without disallow all for your live site? Our free Robots.txt Generator builds the correct file for WordPress in 60 seconds — with the right disallow rules for wp-admin, search pages, and AI crawlers, and nothing that will accidentally block your entire site.
Frequently Asked Questions
What does robots.txt disallow all mean?
Robots.txt disallow all means adding User-agent: * and Disallow: / to your robots.txt file, which instructs all search engine crawlers to stop visiting every page on your website. It is the most aggressive blocking command available in robots.txt and should only be used on staging, development, or private sites that you do not want indexed by Google.
Will robots.txt disallow all remove my site from Google?
Not immediately. Robots.txt disallow all prevents Googlebot from crawling your pages, but pages that are already indexed can remain in Google search results for weeks or months. As Googlebot stops visiting and refreshing these pages, they gradually disappear from the index. Full de-indexing typically takes 4–8 weeks. If you need immediate removal, use the URL Removal tool in Google Search Console alongside robots.txt disallow all.
How do I block all robots except Google?
Add a specific allow rule for Googlebot before the disallow all rule: User-agent: Googlebot followed by Disallow: (empty, meaning allow all), then on a new block: User-agent: * followed by Disallow: /. Googlebot reads its own specific rule first and ignores the wildcard block. All other crawlers only match the wildcard rule and are blocked.
Does robots.txt disallow all hurt SEO?
On a staging site that has never been indexed, no — there is nothing to lose. On a live site that is already ranking, yes — significantly. Rankings begin dropping within 2–4 weeks as Googlebot stops crawling, and pages progressively disappear from Google’s index. Recovery after removing the disallow all typically takes 1–3 months, longer if the disallow was in place for 6+ months.
How do I check if my site has robots.txt disallow all?
Visit yourdomain.com/robots.txt in your browser. If you see Disallow: / under User-agent: *, your entire site is blocked from crawlers. Fix it immediately by going to RankMath → General Settings → Edit robots.txt and replacing the disallow all with your correct live site rules. Then go to Google Search Console → URL Inspection and request re-indexing of your key pages.
Is robots.txt disallow all the same as blocking all robots?
Yes. User-agent: * with Disallow: / is the standard way to block all robots in robots.txt. It applies to Google, Bing, DuckDuckGo, and every other compliant crawler. Note that malicious bots and scrapers may ignore robots.txt entirely — the disallow all command only affects crawlers that voluntarily respect the robots exclusion protocol, which all major search engines do.