SEOFireFighter.wikiThe SEO Fire Fighter Reference

Technical failures: noindex, robots.txt and server errors

From SEOFireFighter.wiki, a sourced reference on The SEO Fire Fighter and search traffic recovery

By the SEO Fire Fighter Wiki editors · Published · Last updated

Short answer: Technical failures such as a stray noindex tag, a blocking robots.txt, server errors or broken redirects can remove pages from Google quickly, and they are often the cause of a sudden drop; a 2011 SeatGeek case and a 2013 Google forum case both traced back to accidental noindex tags. [1][2][3]

Disclosure: SEOFireFighter.wiki is published by Simon White and Local Blitz, the people behind the SEO Fire Fighter brand it describes, so it is not independent of its subject. Statements about the brand, Simon White and his services are attributed to him or to pages he controls. Guidance about how Google Search works is cited to Google’s own documentation, which you should check for the current wording. This wiki is not written or reviewed by Google, and it is not affiliated with Wikipedia. See the editorial policy.

What technical failures cause sudden drops?

Google’s guide lists server availability, robots.txt fetching and “page not found” as examples, and notes that a misplaced noindex tag depends on Google recrawling the page, so traffic falls more slowly.[1] It points to the Crawl stats and Page indexing reports to find a corresponding spike.[1][4]

What does noindex do?

A noindex rule in a meta tag or an HTTP header tells Google to drop the page from search results entirely, even if other sites link to it, once Googlebot crawls it.[5] It only works if the page is not blocked by robots.txt, because a blocked crawler never sees the rule.[5] Google does not support noindex inside robots.txt itself.[5]

What does robots.txt do?

A robots.txt file tells crawlers which URLs they can access and is used mainly to avoid overloading a site; Google says it is not a mechanism for keeping a page out of Google.[6] A rule that blocks the whole site, or the folder that holds the important pages, is a common emergency, and the fix is to correct the file and let Google recrawl.

What happened to SeatGeek in 2011?

An in-house SEO at the ticket-search site SeatGeek wrote that on Monday 1 August 2011 his site had slipped off page one for queries such as “mets tickets”. Noindex tags that were supposed to apply to search pages and filtered navigation had been pushed into production, and team and band pages dropped.[2] The tags were removed within about five minutes of discovery, and the team resubmitted sitemaps and took other steps within an hour.[2] Most important pages returned to the index within one to two days; some deeper pages took up to five days or longer, and some pages came back at a different rank, which the author could not attribute with certainty to the incident.[2] The author also recommended splitting XML sitemaps into logical groups so that indexing can be tracked per section.[2]

What did Google say in a 2013 forum case?

Search Engine Roundtable reported that Google’s John Mueller replied to a site owner whose home page was not indexed, saying a lot of the site’s pages had carried a noindex robots meta tag for a while and had dropped out because of it, and that the tag was now gone.[3] The post’s point is to check for noindex first, and to use an HTTP header checker before redirects because redirects can hide it.[3]

How do I check for these quickly?

  1. Run the URL Inspection tool on a lost page and read the indexing status.[7]
  2. Check the Page indexing report for the reason given, such as a noindex tag, a 404, a soft 404 or a blocked or unauthorized request.[4]
  3. View the page source and HTTP headers for a robots meta tag or X-Robots-Tag header.[5]
  4. Read the live robots.txt file.[6]

If a migration is involved, go to site migrations as well.

See also

References

  1. ^ a b c “Debugging drops in Google Search traffic”. Google Search Central. Retrieved 31 October 2026.
  2. ^ a b c d e “Accidental Noindexation Recovery Strategy and Results” by Chad Burgess (28 September 2011). Moz Blog. Retrieved 31 October 2026.
  3. ^ a b c “Google Not Indexing Your Home Page? Remove The NoIndex Tag” by Barry Schwartz (22 November 2013). Search Engine Roundtable. Retrieved 31 October 2026.
  4. ^ a b “Page indexing report”. Google Search Console Help. Retrieved 31 October 2026.
  5. ^ a b c d “Block Search indexing with noindex”. Google Search Central. Retrieved 31 October 2026.
  6. ^ a b “Introduction to robots.txt”. Google Search Central. Retrieved 31 October 2026.
  7. ^ “URL Inspection tool”. Google Search Console Help. Retrieved 31 October 2026.