How to Audit a Website for Technical SEO Problems in Python and Catch Redirect Loops, noindex Traps, Canonical Conflicts, and Sitemap Errors
Build a Python crawler that finds redirect loops, hidden noindex tags, canonical conflicts and sitemap mistakes, test it on a practice shop with 15 planted problems, and use it to fix them.
A web page can be well written, fast and useful, and still never show up in Google. When that happens, the cause is often not the page itself. A link to it leads to a 404. A redirect sends the crawler in circles. A robots.txt rule keeps the crawler out. A stray noindex tag asks Google to leave the page out. A canonical link names a different address, or a sitemap lists the wrong URLs. Most of this stays out of sight, because a browser shows you the page and hides the plumbing around it. A noindex tag, a canonical link and a sitemap entry never show on screen at all.
Table Of Content
- The terms used in this tutorial
- Prerequisites
- Step 1: Set Up the Project and Run the Practice Shop
- Create the folder and the virtual environment
- How the pieces fit
- The practice shop
- Start it and look at it
- The problems planted in the shop
- Step 2: Fetch a URL and Follow Redirects by Hand
- The idea
- The code
- Run it
- What to check and what goes wrong
- Step 3: Read the Page With html.parser
- The idea
- The code
- Run it
- What to check
- Step 4: Teach the Crawler to Read robots.txt
- What robots.txt is, and what it is not
- The rules the matcher must follow
- The code
- Run it
- Do you need your own matcher? Python has one
- What Python 3.13.14 does
- Google’s examples and random rule sets
- When robots.txt cannot be read
- Real robots.txt files
- The verdict
- Step 5: Decide Whether a Page May Be Indexed
- The idea
- The code
- Run it
- Step 6: Read the Sitemap
- The idea
- The code
- Run it
- Step 7: Put It Together in a Crawler and Read the Report
- How the crawl works
- The checks
- The code
- Run it on the broken shop
- The errors
- The warnings
- The info lines, and what is missing on purpose
- Run it on the repaired shop
- Use the repairs as a to-do list
- Step 8: Test the Auditor So You Can Trust a Clean Report
- The idea
- Common Mistakes and How to Avoid Them
- What This Auditor Does Not Do
- Check the Whole Thing and Keep Going
- Sources and Further Reading
In this tutorial you will build a small Python program that checks the plumbing for you. It crawls a site, follows every redirect one hop at a time, reads each page’s HTML, applies robots.txt the way Google’s documentation describes it, decides whether each page may be indexed, compares what it found with the sitemap, and prints a sorted list of problems. You will test it on a practice shop that has 15 problems planted in it (they produce 20 errors and warnings), read the report, fix the shop, and watch the report go quiet.
One caution first. Fixing technical problems does not promise a place in search results. Google’s page on HTTP status codes says that for Google Search, an HTTP 2xx (success) status code “doesn’t guarantee indexing.” What these checks do is remove the avoidable reasons for a good page to be missed or passed over.
This tutorial was prompted by freeCodeCamp’s November 3 guide by Saar Twito, How to Find and Fix Common Technical SEO Issues on Your Website, which works through the same families of problem on a Node.js practice site with curl and the browser. I rebuilt the idea from scratch in Python: a different practice site, a crawler that finds the problems for you, tests, and measurements of my own.
By the end you will be able to:
- tell apart three things that are easy to mix up: a URL that crawlers are asked to stay out of (robots.txt), a page that can be crawled but asks not to be indexed (
noindex), and a page that names a different URL as its preferred address (canonical); - fetch a URL and follow its redirects by hand, so that chains and loops show up as numbers instead of as silent extra requests;
- read the title, description, robots meta tags, canonical links and anchors of a page with Python’s
html.parser; - write a robots.txt matcher that follows Google’s documented rules (longest rule wins,
*and$wildcards, what an unreadable file means), and know where Python’s own parser agrees and where it does not; - decide from the status code, robots.txt, the robots meta tag and the
X-Robots-Tagheader whether a page may be indexed, and spot the trap where anoindexis hidden by a robots.txt block; - cross-check a sitemap against what the crawl found;
- turn all of it into one crawler with error, warning and info levels and an exit code you can use in a deploy pipeline;
- test the crawler with a golden test and 12 planted bugs, so that you can trust a clean report.
Everything below was run on Python 3.13.14 on Windows 11 on November 4, 2026, with requests 2.34.2 and pytest 9.2.0. Every output block is copied from those runs.
The terms used in this tutorial
- Crawler. A program that fetches pages by following links. Googlebot is Google’s.
- Crawling and indexing. Crawling is fetching a page. Indexing is the later step where a search engine decides to store the page so that it can appear in results. A page can be crawled and not indexed.
- HTTP status code. The three-digit number a server sends first: 200 means success, 3xx means “look somewhere else”, 4xx means the request was wrong or the page is missing, and 5xx means the server failed.
- Redirect. A 3xx answer with a
Locationheader that names another URL. A hop is one such answer. A chain is two or more in a row, and a loop is a chain that comes back to a URL it already visited. - robots.txt. A text file at the root of a site that asks crawlers to stay out of some paths.
- noindex. An instruction, in a meta tag or an HTTP header, asking search engines not to show the page in results.
- Canonical URL. The address you want search engines to treat as the main one when the same content can be reached at several addresses.
- Sitemap. An XML file that lists the URLs you want search engines to know about.
- Soft 404. A page that shows an error message to people but answers with status 200.
- Orphan page. A page that no other page links to, so a crawler that follows links never finds it.
Prerequisites
- Python 3.10 or newer. The code uses
str | Noneannotations. I used 3.13.14, and the comparison with Python’s built-in robots.txt parser in Step 4 describes that exact version. - A terminal and curl. curl comes with macOS, most Linux systems and Windows 10 (version 1803 and later) and Windows 11. In Windows PowerShell, type
curl.exe, because plaincurlis an alias for a different command there. - Basic Python. Functions, lists, dictionaries and dataclasses. HTML and URL basics help, and each idea is explained when it first appears.
- About two hours. Every script runs in seconds. The test suite takes about ten seconds and the planted-bug check about a minute.
- Permission. Run the crawler only against sites you run or have permission to audit. It sends one request at a time and says what it is in its User-Agent, but it is still a crawler.
Step 1: Set Up the Project and Run the Practice Shop
Create the folder and the virtual environment
A virtual environment is a private copy of Python and its packages for one project. Create one, activate it, and install the only two packages you need. On Windows PowerShell activate with seo-venv\Scripts\Activate.ps1. On Linux and macOS use source seo-venv/bin/activate.
mkdir seo-audit
cd seo-audit
python -m venv seo-venv
python -m pip install requests==2.34.2 pytest==9.2.0
mkdir lab
cd labEverything from here on happens inside the lab folder. requests is the HTTP client. Everything else in this tutorial comes from Python’s standard library. Check the versions before you go on:
python -c "import sys, requests; print(sys.version.split()[0], 'requests', requests.__version__)"3.13.14 requests 2.34.2How the pieces fit
You will create these files, one per step. Each module does one job and can be run on its own, so you can see its output before you move on.
shop.pyis the practice website (this step).fetch.pyfetches a URL and follows redirects by hand (Step 2).parse.pyreads titles, canonicals, robots meta tags and links from HTML (Step 3).robots.pydecides what robots.txt allows (Step 4).indexing.pydecides whether a page may be indexed (Step 5).sitemap.pylists the URLs in a sitemap (Step 6).audit.pycrawls the site, runs the checks and prints the report (Step 7).test_audit.pyandmutants.pytest the tools (Step 8).
The practice shop
You need a site with known problems, because a crawler is only trustworthy if you have seen it find problems that you put there on purpose. shop.py serves a made-up shop, Maple Trail Outfitters, with about 25 URLs. Save the code below as shop.py. You do not have to study every line. Read it with two things in mind. build(base, fixed) returns a dictionary from a URL path to the status, headers and body the server sends, and every place where the broken and the repaired shop differ is a branch on fixed, so the repairs are written right next to the problems.
# shop.py
"""Practice site for the technical SEO audit tutorial (Python 3.13, standard library only).
python shop.py the broken Maple Trail Outfitters site on http://127.0.0.1:8080
python shop.py --fixed the repaired copy
python shop.py --fixed --port 8081
"""
import argparse
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
HOST = "127.0.0.1"
SHOP = "Maple Trail Outfitters"
def page(title, body, description=None, canonical=None, robots=None, second_canonical=None):
"""Build a small HTML page. A field that is None is left out of the <head> entirely."""
head = ['<meta charset="utf-8">']
if title is not None:
head.append(f"<title>{title}</title>")
if description is not None:
head.append(f'<meta name="description" content="{description}">')
for href in (canonical, second_canonical):
if href:
head.append(f'<link rel="canonical" href="{href}">')
if robots:
head.append(f'<meta name="robots" content="{robots}">')
return ("<!doctype html>\n<html lang=\"en\">\n<head>\n" + "\n".join(head)
+ "\n</head>\n<body>\n" + body + "\n</body>\n</html>\n")
def build(base, fixed):
"""Return {request target: (status, headers, body)} for the whole site."""
routes = {}
def ok(path, title, description, body, canonical="self", **extra):
canonical = base + path.split("?")[0] if canonical == "self" else canonical
routes[path] = (200, {}, page(title, f"<h1>{title}</h1>\n{body}", description, canonical, **extra))
def redirect(path, code, target):
routes[path] = (code, {"Location": target}, "")
filler = "<p>" + "Our guides explain how to size, fit, wear and care for trail gear. " * 12 + "</p>"
nav = [
'<a href="/boots/">Boots</a>', '<a href="/packs/">Packs</a>', '<a href="/sale">Sale</a>',
'<a href="/boots/">Winter boots</a>' if fixed else '<a href="/old-boots">Winter boots</a>',
'<a href="/about">About</a>', '<a href="/contact">Contact</a>', '<a href="/private/">Staff area</a>',
'<a href="/members/offers">Member offers</a>', '<a href="/boots/?sessionid=abc123">Boots (saved cart)</a>',
'<a href="/search">Search</a>' if fixed else '<a onclick="openSearch()">Search</a>',
'' if fixed else '<a href="javascript:openMenu()">Menu</a>',
'' if fixed else '<span onclick="location=\'/search\'">Find a store</span>',
'' if fixed else '<a href="/gone">Clearance</a>',
'<a href="/shipping">Shipping</a>', '<a href="/returns">Returns</a>', '<a href="/press">Press</a>',
'<a href="/guide/fix-404-errors">Guides</a>',
]
ok("/", f"{SHOP}: Hiking Boots and Packs", "Boots, packs and trail gear for every season.", "\n".join(nav))
ok("/boots/", f"Hiking Boots | {SHOP}", "Waterproof and lightweight hiking boots.",
'<a href="/boots/trail-runner">Trail Runner</a>\n<a href="/boots/summit-pro">Summit Pro</a>\n'
'<a href="/boots/trail-runner?color=red">Trail Runner in red</a>\n'
+ ('<a href="/size-guide">Size guide</a>' if fixed else ""))
ok("/boots/?sessionid=abc123", f"Hiking Boots | {SHOP}", "Waterproof and lightweight hiking boots.", "",
canonical=base + "/boots/")
ok("/boots/trail-runner", f"Trail Runner Boots | {SHOP}", "A light boot for fast days on dry trails.", "")
ok("/boots/trail-runner?color=red", f"Trail Runner Boots | {SHOP}", "A light boot for fast days on dry trails.", "",
canonical=base + "/boots/trail-runner")
if fixed:
ok("/boots/summit-pro", f"Summit Pro Boots | {SHOP}", "A stiff, warm boot for snow and scrambling.", "")
else:
ok("/boots/summit-pro", f"Trail Runner Boots | {SHOP}", None, "")
ok("/packs/", f"Packs | {SHOP}", "Day packs and multi-day packs.", "",
canonical=base + ("/packs/" if fixed else "/packs"))
redirect("/packs", 301, "/packs/")
ok("/sale", f"Sale | {SHOP}", "Last season's gear at lower prices.", "", robots="noindex")
if fixed:
redirect("/old-boots", 301, "/boots/")
else:
redirect("/old-boots", 302, "/boots-2024")
redirect("/boots-2024", 301, "/boots/")
redirect("/loop-a", 301, "/loop-b")
redirect("/loop-b", 301, "/loop-a")
gone_body = "<h1>Page not found</h1>\n<p>Sorry, we could not find that page.</p>"
routes["/gone"] = (404 if fixed else 200, {}, page(f"Page not found | {SHOP}", gone_body, None, None))
routes["404"] = (404, {}, page(f"Page not found | {SHOP}", gone_body, None, None))
if fixed:
ok("/contact", f"Contact | {SHOP}", "Write to us or call the shop.", "")
ok("/search", f"Search | {SHOP}", "Search boots, packs and guides.", "")
ok("/about", f"About | {SHOP}", "Who we are and where the shop is.",
'<a href="/guide.pdf">Care guide (PDF)</a>' + ("" if fixed else '\n<a href="/loop-a">Old page</a>'),
**({} if fixed else {"second_canonical": base + "/company"}))
routes["/guide.pdf"] = (200, {"Content-Type": "application/pdf", "X-Robots-Tag": "noindex"}, "%PDF-1.4\n% practice file\n")
ok("/guide/fix-404-errors", f"How to Fix a 404 Not Found Error | {SHOP} Guides", "Why a page returns 404 and how to fix it.", filler)
ok("/members/offers", f"Member Offers | {SHOP}", "Offers for members.", "")
ok("/private/", f"Staff Area | {SHOP}", "Staff tools.", "", robots="noindex")
ok("/size-guide", f"Size Guide | {SHOP}", "How to measure your foot.", "")
routes["/press"] = (200, {} if fixed else {"X-Robots-Tag": "noindex"},
page(f"Press | {SHOP}", "<h1>Press</h1>", "Press kit and contacts.", base + "/press"))
if fixed:
ok("/shipping", f"Shipping | {SHOP}", "Delivery times and costs.", "")
ok("/returns", f"Returns | {SHOP}", "How to return an order.", "")
else:
routes["/shipping"] = (200, {}, page(None, "<h1>Shipping</h1>", "", base + "/shipping"))
ok("/returns", f"Returns | {SHOP}", "How to return an order.", "", canonical="/returns")
listed = ["/", "/boots/", "/boots/trail-runner", "/boots/summit-pro", "/packs/", "/about", "/press", "/shipping",
"/guide/fix-404-errors", "/members/offers"]
listed += ["/returns", "/size-guide", "/search", "/contact"] if fixed else ["/sale", "/old-boots", "/ghost", "/orphan"]
if not fixed:
ok("/orphan", f"Orphan | {SHOP}", "A page nothing links to.", "")
entries = "".join(f"<url><loc>{base}{path}</loc></url>" for path in listed)
routes["/sitemap.xml"] = (200, {"Content-Type": "application/xml"},
'<?xml version="1.0" encoding="UTF-8"?>\n'
f'<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">{entries}</urlset>\n')
rules = ["User-agent: *"] + ([] if fixed else ["Disallow: /private/"]) + [
"Disallow: /members/", "Allow: /members/offers", "Disallow: /*?sessionid=", "", f"Sitemap: {base}/sitemap.xml"]
routes["/robots.txt"] = (200, {"Content-Type": "text/plain; charset=utf-8"}, "\n".join(rules) + "\n")
return routes
class Handler(BaseHTTPRequestHandler):
routes = {}
def _respond(self, send_body):
status, headers, body = self.routes.get(self.path, self.routes["404"])
data = body.encode("utf-8")
self.send_response(status)
headers = {"Content-Type": "text/html; charset=utf-8", **headers}
for name, value in headers.items():
self.send_header(name, value)
self.send_header("Content-Length", str(len(data)))
self.end_headers()
if send_body:
self.wfile.write(data)
def do_GET(self):
self._respond(True)
def do_HEAD(self):
self._respond(False)
def log_message(self, *args):
pass
def make_server(fixed=False, port=0):
"""Start listening and return the server; the routes need the real port for absolute canonical URLs."""
server = ThreadingHTTPServer((HOST, port), Handler)
base = f"http://{HOST}:{server.server_port}"
server.RequestHandlerClass = type("SiteHandler", (Handler,), {"routes": build(base, fixed)})
server.base = base
return server
if __name__ == "__main__":
parser = argparse.ArgumentParser()
parser.add_argument("--fixed", action="store_true")
parser.add_argument("--port", type=int, default=8080)
args = parser.parse_args()
server = make_server(args.fixed, args.port)
print(f"{'fixed' if args.fixed else 'broken'} site on {server.base} (Ctrl+C to stop)", flush=True)
try:
server.serve_forever()
except KeyboardInterrupt:
passThe server binds to a port first and builds its pages afterwards, because canonical links must be absolute URLs and so need the real port number. It listens on 127.0.0.1, the numeric address, rather than localhost, so that name resolution cannot get in the way.
Start it and look at it
Open a second terminal for the shop and leave it running. The broken shop listens on port 8080. Start the repaired copy on port 8081 in a third terminal, so that you can audit both.
python shop.py
python shop.py --fixed --port 8081Now look at the broken shop the way a crawler does, with curl. The -i flag prints the response headers as well as the body:
curl -i http://127.0.0.1:8080/old-bootsHTTP/1.0 302 Found
Server: BaseHTTP/0.6 Python/3.13.14
Date: Wed, 04 Nov 2026 20:39:34 GMT
Content-Type: text/html; charset=utf-8
Location: /boots-2024
Content-Length: 0The first line is the status line: protocol, status code 302 and its reason phrase. The headers follow, and the one that matters is Location: /boots-2024, which tells the client to ask again at that address. The body is empty (Content-Length: 0). Your Date line will differ. Next, the raw HTML of one product page:
curl -s http://127.0.0.1:8080/boots/summit-pro<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>Trail Runner Boots | Maple Trail Outfitters</title>
<link rel="canonical" href="http://127.0.0.1:8080/boots/summit-pro">
</head>
<body>
<h1>Trail Runner Boots | Maple Trail Outfitters</h1>
</body>
</html>This page has a title and a canonical link, and no meta description. Its title is also a copy of another product’s title, which you cannot see from one page alone. A crawler can see it, because it reads all pages. One more file, the one crawlers read first:
curl -s http://127.0.0.1:8080/robots.txtUser-agent: *
Disallow: /private/
Disallow: /members/
Allow: /members/offers
Disallow: /*?sessionid=
Sitemap: http://127.0.0.1:8080/sitemap.xmlThe problems planted in the shop
These are the 15 problems and the report code that should flag each one. The last row lists five pages that look suspicious but are fine. A good audit stays quiet about them.
| Page | What is wrong | Report code |
|---|---|---|
/contact |
The home page links to a page that does not exist | http-error |
/old-boots |
A 302 and then a 301 before the real page; also listed in the sitemap | redirect-chain, sitemap-redirect |
/loop-a, /loop-b |
Two redirects that point at each other | redirect-loop |
/gone |
An error message with status 200 | soft-404 |
/private/ |
A noindex tag, but robots.txt blocks the folder |
noindex-blocked |
/sale |
A noindex tag, but listed in the sitemap |
sitemap-noindex |
/press |
An X-Robots-Tag: noindex header that nobody sees in the HTML, and listed in the sitemap |
sitemap-noindex |
/packs/ |
Its canonical is /packs, which redirects back |
canonical-redirect, sitemap-canonical |
/about |
Two different canonical links | canonical-conflict |
/returns |
A relative canonical, and missing from the sitemap | canonical-relative, not-in-sitemap |
/boots/summit-pro |
Another product’s title, and no description | duplicate-title, no-description |
/shipping |
No title, and an empty description | no-title, no-description |
/ |
A link without href, a javascript: link and a clickable span |
link-not-crawlable |
/ghost |
Listed in the sitemap, but it does not exist | sitemap-error |
/orphan |
Listed in the sitemap, but no page links to it | orphan |
| Five look-alikes | /members/offers (allowed by a longer rule), /boots/?sessionid=abc123 (blocked on purpose), /boots/trail-runner?color=red (canonical to the real page), /guide.pdf (noindex on purpose) and /guide/fix-404-errors (a long guide that has “404” in its title) |
no problem |
Step 2: Fetch a URL and Follow Redirects by Hand
The idea
The first question about any URL is what the server answers. The status code decides almost everything that follows. Google’s page on HTTP status codes explains what it does with each class: “Google doesn’t use the content from URLs that return 4xx status codes”, and a 5xx makes Google’s crawlers slow down.
Redirects need special care. If you let requests follow them for you, you only see the final page and the chain behind it disappears. So you will switch automatic redirects off with allow_redirects=False and follow each hop yourself, recording it. Two rules keep the loop safe. First, keep a set of URLs already visited, so that a loop is recognized the moment it comes back to one. Second, stop after 10 hops, because the same page says: “By default, Google’s crawlers follow up to 10 redirect hops.”
The status code of a redirect also carries a message. The same page says that for a 301 “Google systems use the redirect as a strong signal that the redirect target should be processed”, while for a 302 they use it “as a weak signal”. Google’s page on redirects puts it in terms of what searchers see: permanent redirects “Show the new redirect target in search results”, and temporary redirects “Show the source page in search results”. A 308 counts as a 301 and a 307 counts as a 302.
The code
Save this as fetch.py. The Fetched dataclass is the record every other module will use. It keeps the list of hops, the final URL and status, the response headers we care about, the body as text, and an error string for the cases where no normal answer exists.
# fetch.py
"""Fetch a URL and follow redirects by hand, so that every hop is visible.
python fetch.py URL
"""
import sys
from dataclasses import dataclass, field
from urllib.parse import urljoin
import requests
UA = "SeoAuditTutorial/1.0 (audits sites its owner runs)"
REDIRECT_CODES = {301, 302, 303, 307, 308}
MAX_HOPS = 10 # Google's crawlers follow up to 10 redirect hops
MAX_BYTES = 5_000_000 # stop reading a body after this many bytes
@dataclass
class Fetched:
url: str # the URL that was asked for
final_url: str = "" # where the last response came from
status: int = 0 # status of the last response, 0 if there was none
hops: list = field(default_factory=list) # (status, from_url, to_url) for every redirect
content_type: str = ""
x_robots: list = field(default_factory=list) # one string per X-Robots-Tag header line
body: str = ""
error: str = "" # "redirect loop", "more than 10 redirect hops", ...
def fetch(url, session=None, max_hops=MAX_HOPS, max_bytes=MAX_BYTES):
session = session or requests.Session()
result = Fetched(url=url, final_url=url)
seen = {url}
current = url
while True:
try:
response = session.get(current, allow_redirects=False, timeout=10, stream=True,
headers={"User-Agent": UA})
except requests.RequestException as exc:
result.error = f"request failed: {type(exc).__name__}"
return result
result.final_url, result.status = current, response.status_code
location = response.headers.get("Location")
if response.status_code in REDIRECT_CODES and location:
response.close()
if len(result.hops) >= max_hops:
result.error = f"more than {max_hops} redirect hops"
return result
target = urljoin(current, location)
result.hops.append((response.status_code, current, target))
if target in seen:
result.error = "redirect loop"
return result
seen.add(target)
current = target
continue
result.content_type = response.headers.get("Content-Type", "")
result.x_robots = response.raw.headers.getlist("X-Robots-Tag")
data = b""
for chunk in response.iter_content(65536):
data += chunk
if len(data) >= max_bytes:
break
response.close()
if any(kind in result.content_type.lower() for kind in ("text", "xml", "json")):
result.body = data.decode("utf-8", errors="replace")
return result
if __name__ == "__main__":
fetched = fetch(sys.argv[1])
for status, source, target in fetched.hops:
print(f"{status} {source} -> {target}")
print(f"final: {fetched.status} {fetched.final_url}" + (f" ({fetched.error})" if fetched.error else ""))A few details are worth understanding. stream=True with iter_content lets the function stop reading after 5,000,000 bytes, so one enormous file cannot use all your memory. response.raw.headers.getlist("X-Robots-Tag") keeps repeated headers as separate lines, which Step 5 needs. The body is only decoded for text, XML and JSON content types, so a PDF response never turns into text in memory. The User-Agent says what the program is.
Run it
Try four URLs: a normal page, the two-hop chain, the loop and the missing page.
python fetch.py http://127.0.0.1:8080/boots/
python fetch.py http://127.0.0.1:8080/old-boots
python fetch.py http://127.0.0.1:8080/loop-a
python fetch.py http://127.0.0.1:8080/contactfinal: 200 http://127.0.0.1:8080/boots/
302 http://127.0.0.1:8080/old-boots -> http://127.0.0.1:8080/boots-2024
301 http://127.0.0.1:8080/boots-2024 -> http://127.0.0.1:8080/boots/
final: 200 http://127.0.0.1:8080/boots/
301 http://127.0.0.1:8080/loop-a -> http://127.0.0.1:8080/loop-b
301 http://127.0.0.1:8080/loop-b -> http://127.0.0.1:8080/loop-a
final: 301 http://127.0.0.1:8080/loop-b (redirect loop)
final: 404 http://127.0.0.1:8080/contactRead the four results one after another. /boots/ answers directly with 200. /old-boots takes a 302 to /boots-2024 and then a 301 to /boots/, so two hops. /loop-a goes to /loop-b and straight back, and the function stops with redirect loop instead of running forever. /contact answers 404.
What to check and what goes wrong
A chain is not an error. Google follows up to 10 hops. But a chain usually means that links were never updated after a page moved, and each hop is one more round trip for every visitor and every crawler. The usual fix is to link straight to the final URL and to keep the old address as a single permanent redirect. The mistake to avoid in your own code is calling requests.get(url) with its default of following redirects: it works, and it hides exactly what you are looking for.
Step 3: Read the Page With html.parser
The idea
An HTML document has a head, which holds information about the page, and a body, which holds the visible content. Search-related tags live mostly in the head: the <title>, the meta description, meta robots tags and the <link rel="canonical">. Python’s standard library has html.parser.HTMLParser, which reads HTML as a stream of events: a start tag, an end tag, some text. You subclass it and decide what to remember.
The parser below remembers eight things. The title and the first <h1>. The meta description, where “missing” (None) and “empty” (an empty string) are different results. Every robots or googlebot meta tag. Every canonical link in the head. Every <a> element, with its href. A count of other elements that have an onclick attribute. And the amount of visible text in the body, which Step 7 uses to tell an error page from a long article.
Two details come from Google’s documentation. Google’s page on links says: “Google can’t reliably extract URLs from <a> elements that don’t have an href attribute or other tags that perform as links because of script events.” That is why anchors without an href and elements with onclick are counted. And the robots meta tag page says: “Google Search doesn’t enforce placement of meta robots in the HTML head and will respect robots meta tags in the body section of an HTML document as well.” My first version read robots tags only from the head. After reading that note I changed it to accept them anywhere, and a test now covers it. The canonical link is different: the documentation shows it in the head, so the parser reads it there only.
The code
Save this as parse.py.
# parse.py
"""Pull the SEO-relevant parts out of an HTML page with the standard library's html.parser.
python parse.py URL
"""
import sys
from dataclasses import dataclass, field
from html.parser import HTMLParser
@dataclass
class Anchor:
href: str | None # None when the <a> element has no href attribute at all
rel: str
onclick: bool
@dataclass
class PageInfo:
title: str | None = None # None means there is no <title> element
description: str | None = None # None means there is no meta description
robots: list = field(default_factory=list) # (meta name, content) for name="robots" and name="googlebot"
canonicals: list = field(default_factory=list) # href of every <link rel="canonical"> inside <head>
anchors: list = field(default_factory=list)
clickable_non_links: int = 0 # elements other than <a> that have an onclick attribute
h1: str | None = None
text_length: int = 0 # characters of visible text in <body>
class _Parser(HTMLParser):
def __init__(self):
super().__init__()
self.info = PageInfo()
self.in_head = False
self.in_body = False
self.skip = 0 # depth inside <script> and <style>
self.capture = None # "title" or "h1" while we are inside one
self.buffer = []
def handle_starttag(self, tag, attrs):
attrs = {name: (value or "") for name, value in attrs}
in_head_area = self.in_head or not self.in_body
if tag == "head":
self.in_head = True
elif tag == "body":
self.in_head, self.in_body = False, True
elif tag in ("script", "style"):
self.skip += 1
elif tag == "title" and self.info.title is None and in_head_area:
self.capture, self.buffer = "title", []
elif tag == "h1" and self.info.h1 is None:
self.capture, self.buffer = "h1", []
elif tag == "meta":
name = attrs.get("name", "").lower()
if name in ("robots", "googlebot"): # Google respects these even when they sit in the body
self.info.robots.append((name, attrs.get("content", "").lower()))
elif name == "description" and self.info.description is None and in_head_area:
self.info.description = " ".join(attrs.get("content", "").split())
elif tag == "link" and self.in_head and "canonical" in attrs.get("rel", "").lower().split():
self.info.canonicals.append(attrs.get("href", "").strip())
if tag == "a":
href = attrs.get("href") if "href" in attrs else None
self.info.anchors.append(Anchor(href, attrs.get("rel", ""), "onclick" in attrs))
elif "onclick" in attrs:
self.info.clickable_non_links += 1
def handle_endtag(self, tag):
if tag == "head":
self.in_head = False
elif tag in ("script", "style"):
self.skip = max(0, self.skip - 1)
elif tag == self.capture:
text = " ".join("".join(self.buffer).split())
if tag == "title":
self.info.title = text
else:
self.info.h1 = text
self.capture = None
def handle_data(self, data):
if self.capture:
self.buffer.append(data)
if self.in_body and not self.skip:
self.info.text_length += len(data.strip())
def parse_page(html):
parser = _Parser()
parser.feed(html)
parser.close()
return parser.info
if __name__ == "__main__":
from fetch import fetch
fetched = fetch(sys.argv[1])
info = parse_page(fetched.body)
print(f"status {fetched.status}")
print(f"title {info.title!r}")
print(f"description {info.description!r}")
print(f"robots meta {info.robots}")
print(f"canonicals {info.canonicals}")
print(f"links {len(info.anchors)} <a> elements, {info.clickable_non_links} other clickable elements")Run it
Parse four pages: the product with the copied title, the page with two canonicals, the page with no title, and the home page.
python parse.py http://127.0.0.1:8080/boots/summit-pro
python parse.py http://127.0.0.1:8080/about
python parse.py http://127.0.0.1:8080/shipping
python parse.py http://127.0.0.1:8080/status 200
title 'Trail Runner Boots | Maple Trail Outfitters'
description None
robots meta []
canonicals ['http://127.0.0.1:8080/boots/summit-pro']
links 0 <a> elements, 0 other clickable elements
status 200
title 'About | Maple Trail Outfitters'
description 'Who we are and where the shop is.'
robots meta []
canonicals ['http://127.0.0.1:8080/about', 'http://127.0.0.1:8080/company']
links 2 <a> elements, 0 other clickable elements
status 200
title None
description ''
robots meta []
canonicals ['http://127.0.0.1:8080/shipping']
links 0 <a> elements, 0 other clickable elements
status 200
title 'Maple Trail Outfitters: Hiking Boots and Packs'
description 'Boots, packs and trail gear for every season.'
robots meta []
canonicals ['http://127.0.0.1:8080/']
links 16 <a> elements, 1 other clickable elementsThe first page has a title but description None, which means there is no description tag at all. The second page lists two canonical URLs, /about and /company. The third has title None and description '', the empty tag, which is a different mistake. The home page has 16 anchors and one other clickable element, which you will meet again in the report.
What to check
Compare each line with what curl showed you in Step 1. If the parser disagrees with what you see in the raw HTML, fix the parser first. Every later step trusts it.
Step 4: Teach the Crawler to Read robots.txt
What robots.txt is, and what it is not
robots.txt asks crawlers to stay out of parts of a site. It is a request and not a lock. RFC 9309, the standard for it, says so directly: “These rules are not a form of access authorization.” It also controls crawling and not indexing. Google’s page on canonicalization warns: “Google may still index URLs that are disallowed in robots.txt without their content.”
The rules the matcher must follow
- A file is made of groups. A group starts with one or more
User-agentlines and continues withAllowandDisallowrules. A crawler obeys the group with the most specific user agent that matches it, and otherwise the*group. - A rule is a path prefix.
Disallow: /members/covers everything below that folder.*matches any run of characters and a final$means “the URL ends here”. - When several rules match, the most specific one wins. Google’s page says: “crawlers use the most specific rule based on the length of the rule path. In case of conflicting rules, including those with wildcards, Google uses the least restrictive rule.” RFC 9309 agrees on the first part: “The most specific match found MUST be used.”
- If nothing matches, the URL is allowed, and
/robots.txtitself is always allowed.
The last piece is what happens when the file cannot be read. RFC 9309 says that if a server answers with a 400 to 499 status, the crawler “MAY access any resources on the server”. Google goes further and says it treats all 4xx errors except 429 “as if a valid robots.txt file didn’t exist”. For a server error it is stricter: “For the first 12 hours, Google stops crawling the site but keeps trying to fetch the robots.txt file.” After that it uses the last good copy for up to 30 days. My loader takes the simple, cautious reading of that: a 5xx, a 429 or a failed connection blocks everything.
The code
Save this as robots.py. compile_pattern turns a rule into a regular expression. Robots._parse reads groups. Robots.match picks the longest matching rule and lets Allow win a tie, and load_robots applies the status-code rules above.
# robots.py
"""A robots.txt matcher that follows RFC 9309 and the wildcard rules in Google's documentation.
python robots.py SITE_URL compare this matcher with Python's urllib.robotparser on a few URLs
"""
import re
import sys
from dataclasses import dataclass, field
from urllib.parse import urlsplit
from fetch import fetch
@dataclass
class Rule:
allow: bool
pattern: str
regex: re.Pattern
@property
def length(self):
return len(self.pattern)
def __str__(self):
return f"{'Allow' if self.allow else 'Disallow'}: {self.pattern}"
def compile_pattern(pattern):
"""'*' matches any run of characters, a final '$' means "the URL ends here"; otherwise it is a prefix match."""
anchored = pattern.endswith("$")
if anchored:
pattern = pattern[:-1]
body = ".*".join(re.escape(piece) for piece in pattern.split("*"))
return re.compile(body + ("$" if anchored else ""), re.DOTALL)
@dataclass
class Group:
agents: list
rules: list = field(default_factory=list)
class Robots:
def __init__(self, text="", state="parsed"):
self.state = state # "parsed", "allow-all" (4xx) or "disallow-all" (5xx, unreachable)
self.text = text
self.groups = []
self.sitemaps = []
self._parse(text)
def _parse(self, text):
group, in_rules = None, False
for line in text.lstrip(chr(0xFEFF)).splitlines():
line = line.split("#", 1)[0].strip()
name, colon, value = line.partition(":")
if not colon:
continue
name, value = name.strip().lower(), value.strip()
if name == "user-agent":
if group is None or in_rules:
group, in_rules = Group([]), False
self.groups.append(group)
group.agents.append(value.lower())
elif name in ("allow", "disallow") and group is not None:
in_rules = True
if value: # an empty Disallow allows everything
group.rules.append(Rule(name == "allow", value, compile_pattern(value)))
elif name == "sitemap":
self.sitemaps.append(value)
def match(self, url, agent="googlebot"):
"""Return (allowed, the rule that decided it or None)."""
parts = urlsplit(url)
target = (parts.path or "/") + (f"?{parts.query}" if parts.query else "")
if target == "/robots.txt":
return True, None # the robots.txt URL itself is implicitly allowed
if self.state == "disallow-all":
return False, None
if self.state == "allow-all":
return True, None
agent = agent.lower()
groups = [g for g in self.groups if agent in g.agents] or [g for g in self.groups if "*" in g.agents]
best = None
for rule in (rule for group in groups for rule in group.rules):
if not rule.regex.match(target):
continue
if best is None or rule.length > best.length or (rule.length == best.length and rule.allow and not best.allow):
best = rule
return (True, None) if best is None else (best.allow, best)
def can_fetch(self, url, agent="googlebot"):
return self.match(url, agent)[0]
def load_robots(origin, session=None):
"""Fetch /robots.txt and apply the status-code rules from RFC 9309 section 2.3.1."""
fetched = fetch(origin + "/robots.txt", session=session, max_hops=5)
if fetched.error.startswith(("redirect", "more than")):
return Robots(state="allow-all") # too many redirects: treated as "unavailable"
if fetched.error or fetched.status == 429 or fetched.status >= 500:
return Robots(state="disallow-all") # unreachable: assume everything is disallowed
if fetched.status >= 400:
return Robots(state="allow-all") # unavailable: the crawler may access anything
return Robots(fetched.body)
if __name__ == "__main__":
from urllib.robotparser import RobotFileParser
origin = sys.argv[1].rstrip("/")
ours = load_robots(origin)
stdlib = RobotFileParser()
stdlib.parse(ours.text.splitlines())
print(f"{'URL':<28}{'this matcher':<14}{'urllib.robotparser':<20}rule used")
for path in ["/boots/", "/private/", "/members/account", "/members/offers", "/boots/?sessionid=abc123"]:
allowed, rule = ours.match(origin + path)
theirs = stdlib.can_fetch("Googlebot", origin + path)
flag = "" if allowed == theirs else " <-- differs"
print(f"{path:<28}{'allowed' if allowed else 'blocked':<14}{'allowed' if theirs else 'blocked':<20}{rule or '-'}{flag}")Run it
python robots.py http://127.0.0.1:8080URL this matcher urllib.robotparser rule used
/boots/ allowed allowed -
/private/ blocked blocked Disallow: /private/
/members/account blocked blocked Disallow: /members/
/members/offers allowed allowed Allow: /members/offers
/boots/?sessionid=abc123 blocked blocked Disallow: /*?sessionid=Read the rows against the file you saw in Step 1. /members/account is blocked by Disallow: /members/. /members/offers matches both Disallow: /members/ and Allow: /members/offers, and the Allow rule is longer, so it wins. The session-id URL is caught by the wildcard rule. The last column of each row also shows what Python’s built-in urllib.robotparser says, which leads to a question.
Do you need your own matcher? Python has one
What Python 3.13.14 does
I expected the answer to be yes. I believed that urllib.robotparser used the first matching rule and had no wildcard support, so I wrote tests that documented those differences. Two of them failed. The reason is in the source of Python 3.13.14, which you can print yourself:
python -c "import inspect, urllib.robotparser as r; print(inspect.getsource(r.RuleLine.applies_to)); print(inspect.getsource(r.RobotFileParser.read))" def applies_to(self, filename):
# If the filename matches the rule, return the matching length plus 1.
# If it does not match, return 0.
if self.matcher is not None:
m = self.matcher(filename)
if m:
return m.end() + 1
else:
if self.fullmatch:
if filename == self.path:
return len(self.path) + 1
else:
if filename.startswith(self.path):
return len(self.path) + 1
return 0
def read(self):
"""Reads the robots.txt URL and feeds it to the parser."""
try:
f = urllib.request.urlopen(self.url)
except urllib.error.HTTPError as err:
if err.code in (401, 403):
self.disallow_all = True
elif err.code >= 400 and err.code < 500:
self.allow_all = True
err.close()
else:
raw = f.read()
self.parse(raw.decode("utf-8", "surrogateescape").splitlines())In this version the parser does support * and $, and it does choose the rule that matches the most. But look at the comment: “If the filename matches the rule, return the matching length plus 1.” It scores a rule by how much of the URL the rule matched, while Google’s page scores it by the length of the rule. For plain prefixes those are the same number. For wildcard rules they are not. Also look at read(): a 401 or 403 sets disallow_all, and any other 4xx, including 429, sets allow_all. Google’s documented handling is different on 403 and on 429.
Google’s examples and random rule sets
Measure how much that matters. compare_stdlib.py runs Google’s six documented examples through both matchers and then 20,000 random rule sets. Save it:
# compare_stdlib.py
"""Compare robots.py with Python's urllib.robotparser. python compare_stdlib.py"""
import random
import sys
from urllib.robotparser import RobotFileParser
from robots import Robots
# (rules, path, what Google's documentation says about whether the path may be crawled)
GOOGLE_EXAMPLES = [
("allow: /p | disallow: /", "/page", True),
("allow: /folder | disallow: /folder", "/folder/page", True),
("allow: /page | disallow: /*.htm", "/page.htm", False),
("allow: /page | disallow: /*.ph", "/page.php5", True),
("allow: /$ | disallow: /", "/", True),
("allow: /$ | disallow: /", "/page.htm", False),
]
def both(rules, path):
text = "user-agent: *\n" + rules
stdlib = RobotFileParser()
stdlib.parse(text.splitlines())
return Robots(text).can_fetch("https://example.com" + path), stdlib.can_fetch("anybot", "https://example.com" + path)
print("=== Google's documented examples")
print(f"{'rules':<38}{'URL path':<14}{'Google':<9}{'robots.py':<11}urllib.robotparser")
for rules, path, google in GOOGLE_EXAMPLES:
mine, theirs = both(rules.replace(" | ", "\n"), path)
mark = "" if theirs == google else " <-- differs from Google"
print(f"{rules:<38}{path:<14}{google!s:<9}{mine!s:<11}{theirs}{mark}")
print("=== 20,000 random rule sets")
random.seed(7)
letters = "ab/.?="
total = same = 0
shown = []
for _ in range(20000):
rules = []
for _ in range(random.randint(1, 3)):
pattern = "/" + "".join(random.choice(letters) for _ in range(random.randint(0, 4)))
if random.random() < 0.4:
at = random.randrange(1, len(pattern) + 1)
pattern = pattern[:at] + "*" + pattern[at:]
if random.random() < 0.2:
pattern += "$"
rules.append(random.choice(["allow", "disallow"]) + ": " + pattern)
path = "/" + "".join(random.choice(letters) for _ in range(random.randint(0, 6)))
mine, theirs = both("\n".join(rules), path)
total += 1
same += mine == theirs
if mine != theirs and len(shown) < 3:
shown.append((" | ".join(rules), path, mine, theirs))
print(f"same answer: {same} of {total} ({100 * same / total:.1f} percent)")
for rules, path, mine, theirs in shown:
print(f" {rules:<34}{path:<10}robots.py={mine!s:<6}urllib={theirs}")
sys.exit(0)python compare_stdlib.py=== Google's documented examples
rules URL path Google robots.py urllib.robotparser
allow: /p | disallow: / /page True True True
allow: /folder | disallow: /folder /folder/page True True True
allow: /page | disallow: /*.htm /page.htm False False False
allow: /page | disallow: /*.ph /page.php5 True True False <-- differs from Google
allow: /$ | disallow: / / True True True
allow: /$ | disallow: / /page.htm False False False
=== 20,000 random rule sets
same answer: 19888 of 20000 (99.4 percent)
disallow: /* | allow: /= /=?. robots.py=True urllib=False
allow: /?a?. | allow: /* | disallow: /*///.a robots.py=False urllib=True
disallow: /. | allow: /* | disallow: /b*/bbb= robots.py=False urllib=TruePython’s parser gives Google’s documented answer in five of the six examples. The exception is the fourth: with allow: /page and disallow: /*.ph, the URL /page.php5 is allowed according to Google, because the two rules are the same length and the least restrictive one wins. Python blocks it, because the wildcard rule matched more of the URL. Over 20,000 random rule sets, the two agree 99.4 percent of the time. Those rule sets are synthetic and full of wildcards, so the survey below is the better guide to how often the difference matters on real files.
When robots.txt cannot be read
The status-code behavior is easier to see directly. robots_status.py starts a tiny local server that answers robots.txt with different statuses and asks both parsers about two URLs:
# robots_status.py
"""What happens when robots.txt itself cannot be read? python robots_status.py"""
import threading
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
from urllib.robotparser import RobotFileParser
from robots import load_robots
STATUS = {"code": 200}
class Handler(BaseHTTPRequestHandler):
def do_GET(self):
body = b"User-agent: *\nDisallow: /private/\n" if STATUS["code"] == 200 else b""
self.send_response(STATUS["code"])
self.send_header("Content-Length", str(len(body)))
self.end_headers()
self.wfile.write(body)
def log_message(self, *args):
pass
server = ThreadingHTTPServer(("127.0.0.1", 0), Handler)
threading.Thread(target=server.serve_forever, daemon=True).start()
base = f"http://127.0.0.1:{server.server_port}"
print(f"{'robots.txt answers':<20}{'robots.py state':<17}{'robots.py':<22}urllib.robotparser")
print(f"{'':<37}{'/boots/':<11}{'/private/':<11}{'/boots/':<11}/private/")
for code in (200, 404, 403, 429, 500):
STATUS["code"] = code
mine = load_robots(base)
stdlib = RobotFileParser(base + "/robots.txt")
stdlib.read()
row = [("allowed" if mine.can_fetch(base + path) else "blocked") for path in ("/boots/", "/private/")]
theirs = [("allowed" if stdlib.can_fetch("googlebot", base + path) else "blocked") for path in ("/boots/", "/private/")]
print(f"HTTP {code:<15}{mine.state:<17}{row[0]:<11}{row[1]:<11}{theirs[0]:<11}{theirs[1]}")
server.shutdown()python robots_status.pyrobots.txt answers robots.py state robots.py urllib.robotparser
/boots/ /private/ /boots/ /private/
HTTP 200 parsed allowed allowed allowed blocked
HTTP 404 allow-all allowed allowed allowed allowed
HTTP 403 allow-all allowed allowed blocked blocked
HTTP 429 disallow-all blocked blocked allowed allowed
HTTP 500 disallow-all blocked blocked blocked blockedOn a 403 Python blocks everything while Google, as documented, behaves as if there were no file. On a 429 Python allows everything while Google treats 429 as a server error and slows down. The 200, 404 and 500 rows agree.
Real robots.txt files
What about real files? survey_robots.py fetches the robots.txt of 64 well-known sites, builds test URLs from each file’s own rules, and counts where the two matchers disagree. Run it, and expect different numbers on another day, because sites change their files:
# survey_robots.py
"""Fetch robots.txt from well-known sites and see where Python's parser and robots.py disagree. python survey_robots.py"""
from concurrent.futures import ThreadPoolExecutor
from urllib.robotparser import RobotFileParser
import requests
from robots import Robots
SITES = """nytimes.com washingtonpost.com theguardian.com bbc.com cnn.com reuters.com apnews.com bloomberg.com
wsj.com ft.com economist.com forbes.com businessinsider.com npr.org usatoday.com latimes.com wired.com
theverge.com techcrunch.com arstechnica.com engadget.com cnet.com zdnet.com theatlantic.com newyorker.com
time.com vox.com axios.com politico.com nbcnews.com foxnews.com buzzfeed.com huffpost.com independent.co.uk
dailymail.co.uk telegraph.co.uk en.wikipedia.org github.com stackoverflow.com reddit.com medium.com quora.com
nature.com sciencedirect.com arxiv.org imdb.com yelp.com tripadvisor.com goodreads.com substack.com
bleepingcomputer.com theregister.com techradar.com tomshardware.com pcmag.com gizmodo.com mashable.com
slashdot.org dev.to news.ycombinator.com wordpress.org cloudflare.com openai.com anthropic.com""".split()
UA = "SeoAuditTutorial/1.0 (robots.txt survey)"
def get(domain):
try:
response = requests.get(f"https://{domain}/robots.txt", headers={"User-Agent": UA}, timeout=15)
return domain, response.status_code, response.content.decode("utf-8", errors="replace")
except requests.RequestException:
return domain, 0, ""
def test_urls(robots):
"""One URL per rule: the rule's own path, with each * replaced by 'x' and the $ removed, plus 'z' when it is a prefix."""
urls = set()
for group in robots.groups:
for rule in group.rules:
path = rule.pattern.replace("*", "x")
urls.add("https://example.com" + path.removesuffix("$"))
if not path.endswith("$"):
urls.add("https://example.com" + path + "z")
return urls
def matching_rules(robots, path, agent="googlebot"):
groups = [g for g in robots.groups if agent in g.agents] or [g for g in robots.groups if "*" in g.agents]
return [str(rule) for group in groups for rule in group.rules if rule.regex.match(path)]
with ThreadPoolExecutor(8) as pool:
results = list(pool.map(get, SITES))
statuses = {}
for _, status, _ in results:
statuses[status] = statuses.get(status, 0) + 1
print(f"sites asked: {len(SITES)}; HTTP statuses:", dict(sorted(statuses.items())))
parsed = wild = differ_files = urls_total = urls_differ = 0
examples = []
for domain, status, text in results:
if status != 200 or "user-agent" not in text.lower():
continue
parsed += 1
robots = Robots(text)
if any("*" in r.pattern.rstrip("*") or r.pattern.endswith("$") for g in robots.groups for r in g.rules):
wild += 1
stdlib = RobotFileParser()
stdlib.parse(text.splitlines())
changed = 0
for url in sorted(test_urls(robots)):
urls_total += 1
allowed, rule = robots.match(url, "googlebot")
theirs = stdlib.can_fetch("googlebot", url)
if allowed != theirs:
if not changed:
path = url.removeprefix("https://example.com")
examples.append((domain, path, allowed, theirs, matching_rules(robots, path)))
changed += 1
urls_differ += changed
differ_files += bool(changed)
print(f"robots.txt files read: {parsed}; with a wildcard or $ rule: {wild}")
print(f"URLs built from the files' own rules: {urls_total}; answered differently: {urls_differ}; files with a difference: {differ_files}")
for domain, path, allowed, theirs, rules in examples[:3]:
print(f"{domain}: {path}")
print(f" robots.py: {'allowed' if allowed else 'blocked'} urllib.robotparser: {'allowed' if theirs else 'blocked'}")
print(" rules that match: " + " | ".join(rules))python survey_robots.pysites asked: 64; HTTP statuses: {0: 1, 200: 60, 403: 2, 418: 1}
robots.txt files read: 60; with a wildcard or $ rule: 49
URLs built from the files' own rules: 9058; answered differently: 223; files with a difference: 6
nytimes.com: /athletic/live-blogs/discuss/z
robots.py: allowed urllib.robotparser: blocked
rules that match: Disallow: /athletic/*/discuss/* | Allow: /athletic/live-blogs/discuss/
bloomberg.com: /account/newslettersz
robots.py: allowed urllib.robotparser: blocked
rules that match: Allow: /account/newsletters | Disallow: /account/*
wired.com: /services.min.js
robots.py: blocked urllib.robotparser: allowed
rules that match: Allow: /*.js | Disallow: /services.min.jsOn November 4, 2026, 60 files could be read (the others answered 403, 418 or failed), and 49 of them used a wildcard or a $. Of 9,058 URLs built from the files’ own rules, 223 were answered differently, in 6 files. The three examples show the pattern. In the New York Times and Bloomberg files, a long plain Allow and a shorter wildcard Disallow both match. Google’s rule gives the Allow, because it is the longer rule, and Python gives the Disallow, because the wildcard rule matches the whole URL. In Wired’s file a wildcard Allow and a plain Disallow tie on matched length. Python lets the Allow win, while Google’s longer-rule logic keeps the Disallow.
The verdict
So Python’s parser is a good quick check, and closer to Google than I expected. For an auditor that should reproduce what Google’s documentation says, the 100 lines in robots.py are worth having, and they give you one place to change if you disagree. Neither one is Google’s own code, so treat a difference on a rule that really matters to you as a reason to test in Search Console.
Step 5: Decide Whether a Page May Be Indexed
The idea
A page that can be fetched is not necessarily a page that may be indexed. Two mechanisms say “do not index this”:
- The robots meta tag, such as
<meta name="robots" content="noindex">. Google’s page saysnoindexmeans “Do not show this page, media, or resource in search results”, andnoneis “Equivalent to noindex, nofollow”. The name can berobotsfor every crawler orgooglebotfor Google only. - The
X-Robots-TagHTTP header, which accepts the same rules and may start with a crawler name and a colon, as inX-Robots-Tag: googlebot: noindex. For files that are not HTML, the page says: “To block indexing of non-HTML resources, such as PDF files, video files, or image files, use the X-Robots-Tag response header instead.” The header is easy to miss, because nothing in the page’s HTML shows it.
If the two disagree, Google’s page says: “In the case of conflicting robots rules, the more restrictive rule applies.” Taking the union of all directives gives that result.
Now the trap. Both mechanisms are read when a URL is crawled. The same page warns: “If a page is disallowed from crawling through the robots.txt file, then any information about indexing or serving rules will not be found and will therefore be ignored.” So a page that is both blocked in robots.txt and marked noindex does not get what the owner wanted. The owner probably blocked it to keep it out of search, but robots.txt does not do that, and the noindex that would is never read. The page continues: “If indexing or serving rules must be followed, the URLs containing those rules cannot be disallowed from crawling.”
The code
Save this as indexing.py. split_directives turns a rule list into a set, meta_directives and header_directives collect the rules for Googlebot from the two places, and indexing_state combines them with the status code and the robots.txt verdict. The header parser has to tell a crawler prefix (googlebot: noindex) from a rule that has a value (max-snippet: 50), which is what VALUE_DIRECTIVES is for.
# indexing.py
"""Decide whether Google may index a fetched page, and say why not.
python indexing.py URL
"""
import sys
from dataclasses import dataclass, field
# These directives take a value after a colon, so a colon does not mean "a crawler name comes first".
VALUE_DIRECTIVES = {"max-snippet", "max-image-preview", "max-video-preview", "unavailable_after"}
def split_directives(text):
"""'noindex, max-snippet: 50' -> {'noindex', 'max-snippet'}. 'none' is shorthand for noindex plus nofollow."""
found = {token.split(":")[0].strip() for token in text.lower().split(",")} - {""}
if "none" in found:
found |= {"noindex", "nofollow"}
return found
def meta_directives(info, agent="googlebot"):
found = set()
for name, content in info.robots:
if name in ("robots", agent):
found |= split_directives(content)
return found
def header_directives(lines, agent="googlebot"):
"""X-Robots-Tag lines, one string per header line. A line may start with a crawler name and a colon."""
found = set()
for line in lines:
first, colon, rest = line.partition(":")
crawler = first.strip().lower()
if colon and "," not in first and crawler not in VALUE_DIRECTIVES:
if crawler != agent:
continue # this line is addressed to a different crawler
line = rest
found |= split_directives(line)
return found
@dataclass
class Indexing:
indexable: bool
reasons: list = field(default_factory=list)
noindex_source: str = "" # "meta robots", "X-Robots-Tag header", both, or ""
def indexing_state(fetched, info, blocked):
reasons, sources = [], []
if fetched.error:
reasons.append(fetched.error)
elif not 200 <= fetched.status < 300:
reasons.append(f"HTTP {fetched.status}")
if info is not None and "noindex" in meta_directives(info):
sources.append("meta robots")
if "noindex" in header_directives(fetched.x_robots):
sources.append("X-Robots-Tag header")
if sources:
reasons.append("noindex via " + " and ".join(sources))
if blocked:
reasons.append("blocked by robots.txt")
return Indexing(not reasons, reasons, " and ".join(sources))
if __name__ == "__main__":
from fetch import fetch
from parse import parse_page
from robots import load_robots
url = sys.argv[1]
origin = url.split("/", 3)[0] + "//" + url.split("/", 3)[2]
fetched = fetch(url)
info = parse_page(fetched.body) if "html" in fetched.content_type else None
state = indexing_state(fetched, info, not load_robots(origin).can_fetch(url))
print(f"{url}\n indexable: {state.indexable}")
for reason in state.reasons:
print(f" - {reason}")Run it
python indexing.py http://127.0.0.1:8080/boots/
python indexing.py http://127.0.0.1:8080/sale
python indexing.py http://127.0.0.1:8080/press
python indexing.py http://127.0.0.1:8080/private/
python indexing.py http://127.0.0.1:8080/members/offershttp://127.0.0.1:8080/boots/
indexable: True
http://127.0.0.1:8080/sale
indexable: False
- noindex via meta robots
http://127.0.0.1:8080/press
indexable: False
- noindex via X-Robots-Tag header
http://127.0.0.1:8080/private/
indexable: False
- noindex via meta robots
- blocked by robots.txt
http://127.0.0.1:8080/members/offers
indexable: TrueA normal product list is indexable. /sale is held back by a meta tag you could find in the HTML. /press is held back by a header you cannot see in the page, which is the kind of problem a crawler finds and a person does not. /private/ shows the trap: two reasons, and the first one never reaches Google. /members/offers is indexable, because the longer Allow rule wins.
Step 6: Read the Sitemap
The idea
A sitemap is a list of the URLs you want found. Google’s page says what it is for: “When creating a sitemap, you’re telling search engines about which URLs you prefer to show in search results. These are the canonical URLs.” Two limits apply to one file: “All formats limit a single sitemap to 50MB (uncompressed) or 50,000 URLs.” A bigger site splits its URLs over several files and lists those in a sitemap index file. Submitting is “merely a hint”, as Google puts it, so a sitemap does not replace links and does not force anything.
That is why a sitemap needs checking. Every URL in it says “index this”, so a URL that answers 404, redirects, or carries noindex contradicts the sitemap. The file format is simple XML, and the protocol at sitemaps.org adds that “The file itself must be UTF-8 encoded.”
The code
Save this as sitemap.py. It reads one or more sitemap files, follows sitemap index files (up to 20 files in total), collects every <loc> and notes problems: a file that cannot be fetched, invalid XML, or a size over the limits. The XML comes from the site being audited, which is your own site here. If you ever point the tool at files you do not control, read our guide to parsing untrusted XML safely first.
# sitemap.py
"""Read a sitemap (or a sitemap index) and list the URLs it asks Google to index.
python sitemap.py SITE_URL
"""
import sys
import xml.etree.ElementTree as ET
from dataclasses import dataclass, field
from fetch import fetch
MAX_URLS = 50_000 # per sitemap file (Google: 50,000 URLs)
MAX_BYTES = 50 * 1024 * 1024 # per sitemap file, uncompressed (Google: 50MB)
@dataclass
class SitemapResult:
urls: list = field(default_factory=list)
files: list = field(default_factory=list) # (sitemap URL, HTTP status, number of URLs)
problems: list = field(default_factory=list)
def local_name(tag):
return tag.rsplit("}", 1)[-1]
def read_sitemaps(entry_urls, session=None, max_files=20):
result, queue, seen = SitemapResult(), list(entry_urls), set()
while queue and len(seen) < max_files:
url = queue.pop(0)
if url in seen:
continue
seen.add(url)
fetched = fetch(url, session=session, max_bytes=MAX_BYTES + 1)
if fetched.error or fetched.status != 200:
result.files.append((url, fetched.status, 0))
result.problems.append(f"{url}: {fetched.error or 'HTTP ' + str(fetched.status)}")
continue
data = fetched.body.encode("utf-8")
try:
root = ET.fromstring(data)
except ET.ParseError as exc:
result.files.append((url, fetched.status, 0))
result.problems.append(f"{url}: not valid XML ({exc})")
continue
locs = [el.text.strip() for el in root.iter() if local_name(el.tag) == "loc" and el.text]
if local_name(root.tag) == "sitemapindex":
queue.extend(locs)
else:
result.urls.extend(locs)
result.files.append((url, fetched.status, len(locs)))
if len(locs) > MAX_URLS:
result.problems.append(f"{url}: {len(locs)} URLs, over the {MAX_URLS:,} limit")
if len(data) > MAX_BYTES:
result.problems.append(f"{url}: {len(data):,} bytes, over the 50MB limit")
return result
if __name__ == "__main__":
from robots import load_robots
origin = sys.argv[1].rstrip("/")
robots = load_robots(origin)
sitemap = read_sitemaps(robots.sitemaps or [origin + "/sitemap.xml"])
for url, status, count in sitemap.files:
print(f"sitemap file {url} HTTP {status} {count} URLs")
for problem in sitemap.problems:
print("PROBLEM", problem)
for url in sitemap.urls:
page = fetch(url)
note = f" -> {page.final_url}" if page.hops else ""
print(f"{page.status or '---'} {url.removeprefix(origin) or '/'}{note}")Run it
python sitemap.py http://127.0.0.1:8080sitemap file http://127.0.0.1:8080/sitemap.xml HTTP 200 14 URLs
200 /
200 /boots/
200 /boots/trail-runner
200 /boots/summit-pro
200 /packs/
200 /about
200 /press
200 /shipping
200 /guide/fix-404-errors
200 /members/offers
200 /sale
200 /old-boots -> http://127.0.0.1:8080/boots/
404 /ghost
200 /orphanOne file with 14 URLs. Each line shows the status the URL answers: /old-boots is a redirect, /ghost is a 404, and /orphan answers 200 but nothing links to it. This listing shows status only. Whether /sale or /press are noindex needs the checks from Step 5, and the full audit combines them.
Step 7: Put It Together in a Crawler and Read the Report
How the crawl works
The crawler in audit.py starts at the home page and visits every same-site URL it finds in a link, one request at a time, up to a limit. For each URL it stores a Page: what fetch returned, what parse_page found, the indexing verdict and whether robots.txt blocks it. Three design decisions are worth knowing.
- URLs that robots.txt blocks are still fetched once. Googlebot would not fetch them, but you are auditing your own site, and the simplest way to find a
noindexthat Google cannot read is to read it yourself. Their links are not followed. - A redirect target counts as discovered. A page reached only through a redirect is not an orphan.
- Checks reuse fetched pages.
Crawl.inspectreturns a stored page, or fetches a URL that the crawl did not visit, such as a canonical target or a sitemap entry, and remembers it.
The checks
Each check is a function that returns a list of Issue objects. There are three levels: error for something that probably keeps a page out of search or breaks a visit, warning for something that weakens a page or sends mixed signals, and info for facts that are worth knowing and are often deliberate.
| Report code | What it flags | Level |
|---|---|---|
http-error, fetch-failed |
A crawled URL answers 4xx or 5xx, or cannot be fetched | error |
redirect-loop, redirect-limit |
A redirect chain that comes back on itself, or runs past 10 hops | error |
redirect-chain |
Two or more hops before the final page | warning |
soft-404 |
Status 200, under 600 characters of text, and “not found”, “404” and similar words in the title or heading | warning |
noindex-blocked |
A noindex on a URL that robots.txt blocks |
error |
canonical-conflict, canonical-relative, canonical-redirect |
Two different canonicals, a relative one, or one that points at a redirecting URL | warning |
canonical-error, canonical-noindex |
A canonical that points at a missing or noindex page | error |
no-title, no-description, duplicate-title |
Missing or empty text, or a title shared by two indexable pages | warning |
link-not-crawlable |
An <a> with no real href, or a clickable element that is not a link |
warning |
sitemap-error, sitemap-blocked, sitemap-noindex, sitemap-file |
The sitemap lists a URL that answers an error, is blocked, or is noindex, or the file itself is broken | error |
sitemap-redirect, sitemap-canonical, orphan |
A listed URL redirects, names another canonical, or has no incoming link | warning |
blocked, noindex, canonicalized, no-canonical, not-in-sitemap |
Facts to review, usually deliberate | info |
Two of the rules come straight from Google’s canonicalization page. It lists the ways to signal a preferred URL, “in order of how strongly they can influence canonicalization”: redirects and rel="canonical" annotations are strong signals, and inclusion in a sitemap is “a weak signal”. And it recommends absolute canonical URLs: “Even though relative paths are supported by Google, they can cause problems in the long run.” A canonical that points at a redirect, a duplicate title and a soft 404 are my own checks. Treat them as prompts to look, because the documentation does not set a rule for them.
The code
Save this as audit.py.
# audit.py
"""Crawl a site you run and report technical SEO problems.
python audit.py URL [--limit 200] [--json] [--strict]
The exit code is 1 when an error is found (with --strict, also when a warning is found).
"""
import argparse
import json
import re
import sys
from collections import defaultdict
from dataclasses import asdict, dataclass, field
from urllib.parse import urldefrag, urljoin, urlsplit
import requests
from fetch import fetch
from indexing import indexing_state
from parse import parse_page
from robots import load_robots
from sitemap import read_sitemaps
ERROR, WARN, INFO = "ERROR", "WARN", "INFO"
NOT_FOUND = re.compile(r"not found|404|no longer available|does not exist|doesn't exist", re.I)
SOFT_404_MAX_TEXT = 600 # an error page is short; a guide about 404 errors is not
@dataclass
class Issue:
severity: str
code: str
url: str
message: str
@dataclass
class Page:
url: str
fetched: object = None
info: object = None
state: object = None
blocked: bool = False
rule: object = None
@dataclass
class Crawl:
start: str
origin: str
robots: object
session: object
pages: dict = field(default_factory=dict) # URL asked for -> Page
links: dict = field(default_factory=lambda: defaultdict(set)) # target URL -> URLs of pages that link to it
extra: dict = field(default_factory=dict) # pages looked at but not crawled
def inspect(self, url):
"""Return the Page for a URL, fetching it if the crawl has not seen it."""
if url in self.pages:
return self.pages[url]
if url not in self.extra:
self.extra[url] = load_page(url, self.robots, self.session)
return self.extra[url]
def load_page(url, robots, session):
allowed, rule = robots.match(url)
fetched = fetch(url, session)
info = parse_page(fetched.body) if "html" in fetched.content_type.lower() else None
return Page(url, fetched, info, indexing_state(fetched, info, not allowed), not allowed, rule)
def crawl(start, limit=200, session=None):
session = session or requests.Session()
start = urldefrag(start)[0]
parts = urlsplit(start)
origin = f"{parts.scheme}://{parts.netloc}"
start = start if parts.path else start + "/"
site = Crawl(start, origin, load_robots(origin, session), session)
queue = [start]
while queue and len(site.pages) < limit:
url = queue.pop(0)
if url in site.pages:
continue
page = site.pages[url] = load_page(url, site.robots, session)
if page.blocked:
continue # fetched once to see what Google cannot see, but its links are not followed
targets = [page.fetched.final_url] if page.fetched.hops and not page.fetched.error else []
if page.info and not page.fetched.hops:
for anchor in page.info.anchors:
href = (anchor.href or "").strip()
if href and not href.startswith("#") and not href.lower().startswith(("javascript:", "mailto:", "tel:")):
targets.append(urldefrag(urljoin(page.fetched.final_url, href))[0])
for target in targets:
if target.startswith(origin + "/") and target != url:
site.links[target].add(url)
queue.append(target)
return site
def short(site, url):
return url.removeprefix(site.origin) or "/"
def served(page):
"""True when this URL itself answered 200 with an HTML page that Google is allowed to fetch."""
return page.info is not None and page.fetched.status == 200 and not page.fetched.hops and not page.blocked
def canonical_of(page):
found = [href for href in page.info.canonicals if href]
return urljoin(page.fetched.final_url, found[0]) if found else None
def looks_like_error(page):
"""A short page whose title or heading says "not found" (a soft 404 candidate)."""
label = page.info.title or page.info.h1 or ""
return page.info.text_length < SOFT_404_MAX_TEXT and bool(NOT_FOUND.search(label) or NOT_FOUND.search(page.info.h1 or ""))
def eligible(page):
"""A page that should appear in search under its own URL."""
return (served(page) and page.state.indexable and not looks_like_error(page)
and canonical_of(page) in (None, page.fetched.final_url))
def check_fetch(site):
issues = []
for url, page in site.pages.items():
f = page.fetched
sources = sorted(site.links.get(url, ()))
via = f" (linked from {short(site, sources[0])}{' and others' if len(sources) > 1 else ''})" if sources else ""
codes = " > ".join(str(code) for code, _, _ in f.hops)
if f.error and "redirect" in f.error:
code = "redirect-loop" if "loop" in f.error else "redirect-limit"
issues.append(Issue(ERROR, code, url, f"{f.error} after {len(f.hops)} hops ({codes}){via}"))
elif f.error:
issues.append(Issue(ERROR, "fetch-failed", url, f.error + via))
elif f.status >= 400:
issues.append(Issue(ERROR, "http-error", url, f"HTTP {f.status}{via}"))
elif len(f.hops) >= 2:
issues.append(Issue(WARN, "redirect-chain", url, f"{len(f.hops)} hops ({codes}) to {short(site, f.final_url)}{via}"))
return issues
def check_soft_404(site):
issues = []
for url, page in site.pages.items():
if served(page) and looks_like_error(page):
label = page.info.title or page.info.h1
issues.append(Issue(WARN, "soft-404", url, f'HTTP 200, but it reads like an error page: "{label}"'))
return issues
def check_indexing(site):
issues = []
for url, page in site.pages.items():
source = page.state.noindex_source
if page.blocked and source:
issues.append(Issue(ERROR, "noindex-blocked", url, f"{page.rule} hides the noindex ({source}) from Google"))
elif page.blocked:
issues.append(Issue(INFO, "blocked", url, f"blocked by robots.txt ({page.rule})"))
elif source and page.fetched.status == 200:
issues.append(Issue(INFO, "noindex", url, f"noindex via {source}"))
return issues
def check_canonical(site):
issues = []
for url, page in site.pages.items():
if not served(page):
continue
found = page.info.canonicals
resolved = {urljoin(page.fetched.final_url, href) for href in found if href}
if not resolved:
if eligible(page):
issues.append(Issue(INFO, "no-canonical", url, "no rel=canonical link"))
continue
if len(resolved) > 1:
names = ", ".join(sorted(short(site, r) for r in resolved))
issues.append(Issue(WARN, "canonical-conflict", url, f"{len(resolved)} different canonical URLs: {names}"))
continue
if any(href and not urlsplit(href).netloc for href in found):
issues.append(Issue(WARN, "canonical-relative", url, f"relative canonical {found[0]!r}; Google recommends absolute URLs"))
target = next(iter(resolved))
if target == page.fetched.final_url:
continue
other = site.inspect(target)
f = other.fetched
if f.hops:
issues.append(Issue(WARN, "canonical-redirect", url, f"canonical {short(site, target)} redirects to {short(site, f.final_url)}"))
elif f.error or f.status >= 400:
issues.append(Issue(ERROR, "canonical-error", url, f"canonical {short(site, target)} answers {f.error or f.status}"))
elif other.state.noindex_source:
issues.append(Issue(ERROR, "canonical-noindex", url, f"canonical {short(site, target)} is noindex ({other.state.noindex_source})"))
else:
issues.append(Issue(INFO, "canonicalized", url, f"canonical points to {short(site, target)}"))
return issues
def check_text(site):
issues, titles = [], defaultdict(list)
for url, page in site.pages.items():
if not eligible(page):
continue
title = (page.info.title or "").strip()
if title:
titles[title.lower()].append((url, title))
else:
issues.append(Issue(WARN, "no-title", url, "missing or empty <title>"))
if not (page.info.description or "").strip():
issues.append(Issue(WARN, "no-description", url, "missing or empty meta description"))
for same in titles.values():
for url, title in same if len(same) > 1 else []:
others = ", ".join(short(site, other) for other, _ in same if other != url)
issues.append(Issue(WARN, "duplicate-title", url, f'same title as {others}: "{title}"'))
return issues
def check_links(site):
issues = []
for url, page in site.pages.items():
if not served(page):
continue
dead = [a for a in page.info.anchors
if a.href is None or a.href.strip() in ("", "#") or a.href.strip().lower().startswith("javascript:")]
extra = page.info.clickable_non_links
if dead or extra:
issues.append(Issue(WARN, "link-not-crawlable", url,
f"{len(dead)} <a> without a real href and {extra} other element(s) that navigate only through onclick"))
return issues
def check_sitemap(site, sitemap):
issues, listed = [], set()
declared = bool(site.robots.sitemaps)
for problem in sitemap.problems:
issues.append(Issue(ERROR if declared else INFO, "sitemap-file" if declared else "no-sitemap", site.start, problem))
for url in sitemap.urls:
url = urldefrag(url)[0]
listed.add(url)
page = site.inspect(url)
f = page.fetched
if f.error or f.status >= 400:
issues.append(Issue(ERROR, "sitemap-error", url, f"listed in the sitemap, but it answers {f.error or f.status}"))
elif f.hops:
issues.append(Issue(WARN, "sitemap-redirect", url, f"listed in the sitemap, but it redirects to {short(site, f.final_url)}"))
elif page.blocked:
issues.append(Issue(ERROR, "sitemap-blocked", url, f"listed in the sitemap, but robots.txt blocks it ({page.rule})"))
elif page.state.noindex_source:
issues.append(Issue(ERROR, "sitemap-noindex", url, f"listed in the sitemap, but it is noindex via {page.state.noindex_source}"))
elif page.info and canonical_of(page) not in (None, f.final_url):
issues.append(Issue(WARN, "sitemap-canonical", url, f"listed in the sitemap, but its canonical is {short(site, canonical_of(page))}"))
if f.status == 200 and not f.hops and url != site.start and url not in site.links:
issues.append(Issue(WARN, "orphan", url, "listed in the sitemap, but no crawled page links to it"))
for url, page in site.pages.items():
if eligible(page) and url not in listed:
issues.append(Issue(INFO, "not-in-sitemap", url, "indexable page that the sitemap does not list"))
return issues
def run_audit(start, limit=200, session=None):
site = crawl(start, limit, session)
sitemap = read_sitemaps(site.robots.sitemaps or [site.origin + "/sitemap.xml"], site.session)
issues = []
for check in (check_fetch, check_soft_404, check_indexing, check_canonical, check_text, check_links):
issues += check(site)
issues += check_sitemap(site, sitemap)
return site, sitemap, issues
def report(site, sitemap, issues):
order = {ERROR: 0, WARN: 1, INFO: 2}
print(f"SEO audit of {site.start}")
print(f"URLs fetched: {len(site.pages)} sitemap URLs: {len(sitemap.urls)} robots.txt: {site.robots.state}")
for issue in sorted(issues, key=lambda i: (order[i.severity], i.code, i.url)):
print(f"{issue.severity:<5} {issue.code:<19} {short(site, issue.url):<30} {issue.message}")
counts = {severity: sum(1 for i in issues if i.severity == severity) for severity in order}
print(f"errors: {counts[ERROR]} warnings: {counts[WARN]} info: {counts[INFO]}")
def main(argv=None):
parser = argparse.ArgumentParser(description="Technical SEO audit of a site you run.")
parser.add_argument("url")
parser.add_argument("--limit", type=int, default=200, help="maximum number of URLs to fetch")
parser.add_argument("--json", action="store_true", help="print the issues as JSON")
parser.add_argument("--strict", action="store_true", help="exit 1 on warnings too")
args = parser.parse_args(argv)
site, sitemap, issues = run_audit(args.url, args.limit)
if args.json:
print(json.dumps([asdict(issue) for issue in issues], indent=2))
else:
report(site, sitemap, issues)
failed = any(i.severity == ERROR for i in issues) or (args.strict and any(i.severity == WARN for i in issues))
return 1 if failed else 0
if __name__ == "__main__":
sys.exit(main())Notice how eligible(page) decides which pages the text checks apply to: pages that answered 200 themselves, may be fetched, are indexable, do not look like error pages and do not point to another canonical. That one function is why /boots/trail-runner?color=red does not count as a duplicate of /boots/trail-runner, and why a redirecting URL does not share its target’s title.
Run it on the broken shop
python audit.py http://127.0.0.1:8080SEO audit of http://127.0.0.1:8080/
URLs fetched: 20 sitemap URLs: 14 robots.txt: parsed
ERROR http-error /contact HTTP 404 (linked from /)
ERROR noindex-blocked /private/ Disallow: /private/ hides the noindex (meta robots) from Google
ERROR redirect-loop /loop-a redirect loop after 2 hops (301 > 301) (linked from /about)
ERROR sitemap-error /ghost listed in the sitemap, but it answers 404
ERROR sitemap-noindex /press listed in the sitemap, but it is noindex via X-Robots-Tag header
ERROR sitemap-noindex /sale listed in the sitemap, but it is noindex via meta robots
WARN canonical-conflict /about 2 different canonical URLs: /about, /company
WARN canonical-redirect /packs/ canonical /packs redirects to /packs/
WARN canonical-relative /returns relative canonical '/returns'; Google recommends absolute URLs
WARN duplicate-title /boots/summit-pro same title as /boots/trail-runner: "Trail Runner Boots | Maple Trail Outfitters"
WARN duplicate-title /boots/trail-runner same title as /boots/summit-pro: "Trail Runner Boots | Maple Trail Outfitters"
WARN link-not-crawlable / 2 <a> without a real href and 1 other element(s) that navigate only through onclick
WARN no-description /boots/summit-pro missing or empty meta description
WARN no-description /shipping missing or empty meta description
WARN no-title /shipping missing or empty <title>
WARN orphan /orphan listed in the sitemap, but no crawled page links to it
WARN redirect-chain /old-boots 2 hops (302 > 301) to /boots/ (linked from /)
WARN sitemap-canonical /packs/ listed in the sitemap, but its canonical is /packs
WARN sitemap-redirect /old-boots listed in the sitemap, but it redirects to /boots/
WARN soft-404 /gone HTTP 200, but it reads like an error page: "Page not found | Maple Trail Outfitters"
INFO blocked /boots/?sessionid=abc123 blocked by robots.txt (Disallow: /*?sessionid=)
INFO canonicalized /boots/trail-runner?color=red canonical points to /boots/trail-runner
INFO noindex /guide.pdf noindex via X-Robots-Tag header
INFO noindex /press noindex via X-Robots-Tag header
INFO noindex /sale noindex via meta robots
INFO not-in-sitemap /returns indexable page that the sitemap does not list
errors: 6 warnings: 14 info: 6The errors
Read it from the top, because the order is the order of urgency. The six errors:
http-error /contact: the home page links to a missing page.noindex-blocked /private/: the robots.txt ruleDisallow: /private/hides thenoindexfrom Google.redirect-loop /loop-a: two redirects that point at each other, found through a link on the About page.sitemap-error /ghost,sitemap-noindex /pressand/sale: the sitemap asks for a page that does not exist and for two pages that ask not to be indexed. The/pressone comes from a header and is invisible in the HTML.
The warnings
The 14 warnings group by theme. Redirects: the /old-boots chain, which is also listed in the sitemap. Canonicals: /about has two, /packs/ points at a URL that redirects back, and /returns uses a relative one. Text: two pages share a title, two pages lack a description, and one of those also lacks a title. Links: the home page has two anchors without a usable href and one clickable span. And structure: /orphan is listed but nothing links to it, and /gone reads like an error page but answers 200.
The info lines, and what is missing on purpose
The six info lines are facts and not faults. A session-id URL is blocked on purpose, a variant is canonicalized to the real product, a PDF, the sale page and the press page are noindex (the press page is the accident that the sitemap error already reported), and /returns is an indexable page that the sitemap forgot. Review them once.
Just as important is what is not in the report. The long guide with “404” in its title is not called a soft 404, because it has far more than 600 characters of text. /members/offers is not reported as blocked, because the longer Allow rule wins. And /boots/trail-runner?color=red is not called a duplicate. Those are the false alarms a crawler this simple could easily raise, and the golden test in Step 8 makes sure it does not.
Run it on the repaired shop
python audit.py http://127.0.0.1:8081SEO audit of http://127.0.0.1:8081/
URLs fetched: 19 sitemap URLs: 14 robots.txt: parsed
INFO blocked /boots/?sessionid=abc123 blocked by robots.txt (Disallow: /*?sessionid=)
INFO canonicalized /boots/trail-runner?color=red canonical points to /boots/trail-runner
INFO noindex /guide.pdf noindex via X-Robots-Tag header
INFO noindex /private/ noindex via meta robots
INFO noindex /sale noindex via meta robots
errors: 0 warnings: 0 info: 5No errors and no warnings. The five info lines are the deliberate ones: the blocked session URL, the canonicalized variant, and three pages that are noindex on purpose. Notice that /private/ is now an info line and not an error: the Disallow: /private/ rule is gone, so Google can read the noindex.
Use the repairs as a to-do list
Open shop.py again and read every fixed branch next to the problem it repairs. The fixes are small: link to the final URL instead of an old one, remove the sitemap entries that do not belong, make the canonical absolute and self-referential, drop the header that nobody meant to send, give every page a title and a description, and return a real 404 for a page that is gone. To practice the loop yourself, copy shop.py, repair one problem, restart it, run the audit and watch exactly that line disappear.
Step 8: Test the Auditor So You Can Trust a Clean Report
The idea
A report that says “no problems” is only useful if the tool can find problems. You already have good test data: the broken shop has a known list of problems, and the fixed shop has none. The test suite below uses that in three ways.
- A golden test compares the full set of (code, path) pairs reported for the broken shop with the expected set. It fails on a missed problem and on a false alarm.
- The fixed shop must have no errors and no warnings, and the exit codes must be right: 1 for the broken shop, and 0 for the fixed shop even with
--strict. - Unit tests check the pieces: Google’s six precedence examples and eight pattern examples, the RFC 9309 longest-match example, status handling for robots.txt, the hop limit (a chain of exactly 10 hops works, and one of 11 stops), directive parsing, and how the parser treats a canonical in the body and a robots meta tag in the body.
Save this as test_audit.py:
# test_audit.py
import threading
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
from urllib.robotparser import RobotFileParser
import pytest
import audit
from fetch import fetch
from indexing import header_directives, meta_directives, split_directives
from parse import parse_page
from robots import Robots, load_robots
from shop import make_server
def serve(routes):
"""A tiny server for one test: {path: (status, headers, body)}."""
class Handler(BaseHTTPRequestHandler):
def do_GET(self):
status, headers, body = routes.get(self.path, (404, {}, "not found"))
data = body.encode()
self.send_response(status)
for name, value in headers.items():
self.send_header(name, value)
self.send_header("Content-Length", str(len(data)))
self.end_headers()
self.wfile.write(data)
def log_message(self, *args):
pass
server = ThreadingHTTPServer(("127.0.0.1", 0), Handler)
threading.Thread(target=server.serve_forever, daemon=True).start()
server.base = f"http://127.0.0.1:{server.server_port}"
return server
@pytest.fixture(scope="module")
def sites():
servers = {"broken": make_server(False), "fixed": make_server(True)}
for server in servers.values():
threading.Thread(target=server.serve_forever, daemon=True).start()
yield {name: server.base for name, server in servers.items()}
for server in servers.values():
server.shutdown()
server.server_close()
@pytest.fixture(scope="module")
def results(sites):
found = {}
for name, base in sites.items():
site, sitemap, issues = audit.run_audit(base)
found[name] = {"site": site, "issues": issues,
"pairs": {(i.code, i.url.removeprefix(base) or "/") for i in issues}}
return found
BROKEN_ERRORS = {
("http-error", "/contact"), ("noindex-blocked", "/private/"), ("redirect-loop", "/loop-a"),
("sitemap-error", "/ghost"), ("sitemap-noindex", "/press"), ("sitemap-noindex", "/sale"),
}
BROKEN_WARNINGS = {
("canonical-conflict", "/about"), ("canonical-redirect", "/packs/"), ("canonical-relative", "/returns"),
("duplicate-title", "/boots/summit-pro"), ("duplicate-title", "/boots/trail-runner"),
("link-not-crawlable", "/"), ("no-description", "/boots/summit-pro"), ("no-description", "/shipping"),
("no-title", "/shipping"), ("orphan", "/orphan"), ("redirect-chain", "/old-boots"),
("sitemap-canonical", "/packs/"), ("sitemap-redirect", "/old-boots"), ("soft-404", "/gone"),
}
INFOS = {
("blocked", "/boots/?sessionid=abc123"), ("canonicalized", "/boots/trail-runner?color=red"),
("noindex", "/guide.pdf"), ("noindex", "/sale"),
}
def test_broken_site_reports_exactly_the_planted_problems(results):
expected = BROKEN_ERRORS | BROKEN_WARNINGS | INFOS | {("noindex", "/press"), ("not-in-sitemap", "/returns")}
assert results["broken"]["pairs"] - expected == set(), "false positives"
assert expected - results["broken"]["pairs"] == set(), "missed problems"
def test_fixed_site_has_no_errors_or_warnings(results):
severities = {i.severity for i in results["fixed"]["issues"]}
assert severities == {audit.INFO}
assert results["fixed"]["pairs"] == INFOS | {("noindex", "/private/")}
def test_exit_codes(sites):
assert audit.main([sites["broken"]]) == 1
assert audit.main([sites["fixed"], "--strict"]) == 0
def test_negative_controls_are_not_flagged(results):
pairs = results["broken"]["pairs"]
assert ("soft-404", "/guide/fix-404-errors") not in pairs # long page about 404 errors
assert ("duplicate-title", "/boots/trail-runner?color=red") not in pairs # canonicalized to the real page
assert not any(path == "/members/offers" for _, path in pairs) # allowed by the longer Allow rule
assert "http://" + "127.0.0.1" in next(iter(results["broken"]["site"].pages))
@pytest.mark.parametrize("rules,path,allowed", [
("allow: /p\ndisallow: /", "/page", True),
("allow: /folder\ndisallow: /folder", "/folder/page", True),
("allow: /page\ndisallow: /*.htm", "/page.htm", False),
("allow: /page\ndisallow: /*.ph", "/page.php5", True),
("allow: /$\ndisallow: /", "/", True),
("allow: /$\ndisallow: /", "/page.htm", False),
])
def test_google_precedence_examples(rules, path, allowed):
assert Robots("user-agent: *\n" + rules).can_fetch("https://example.com" + path) is allowed
@pytest.mark.parametrize("pattern,path,matches", [
("/*.php", "/folder/filename.php?parameters", True), ("/*.php", "/windows.PHP", False),
("/*.php$", "/filename.php", True), ("/*.php$", "/filename.php?parameters", False),
("/fish*", "/fish.html", True), ("/fish*", "/catfish", False), ("/fish/", "/fish", False),
("/fish*.php", "/fishheads/catfish.php?parameters", True),
])
def test_google_pattern_examples(pattern, path, matches):
assert Robots(f"user-agent: *\ndisallow: {pattern}").can_fetch("https://example.com" + path) is (not matches)
RFC_EXAMPLE = "User-Agent: foobot\nAllow: /example/page/\nDisallow: /example/page/disallowed.gif\n"
def test_rfc_9309_longest_match():
robots = Robots(RFC_EXAMPLE)
assert robots.can_fetch("https://example.com/example/page/disallowed.gif", "foobot") is False
assert robots.can_fetch("https://example.com/example/page/other.gif", "foobot") is True
def test_python_313_stdlib_parser_agrees_on_the_rfc_example_and_on_wildcards():
stdlib = RobotFileParser()
stdlib.parse(RFC_EXAMPLE.splitlines())
assert stdlib.can_fetch("foobot", "https://example.com/example/page/disallowed.gif") is False
stdlib = RobotFileParser()
stdlib.parse(["User-agent: *", "Disallow: /*?sessionid="])
assert stdlib.can_fetch("Googlebot", "https://example.com/boots/?sessionid=1") is False
def test_python_313_stdlib_parser_measures_wildcard_specificity_by_matched_length():
text = "\n".join(["user-agent: *", "allow: /page", "disallow: /*.ph"])
stdlib = RobotFileParser()
stdlib.parse(text.splitlines())
assert stdlib.can_fetch("anybot", "https://example.com/page.php5") is False
assert Robots(text).can_fetch("https://example.com/page.php5") is True # Google's documented answer
def test_most_specific_user_agent_group_wins_and_groups_do_not_mix():
robots = Robots("User-agent: *\nDisallow: /\n\nUser-agent: googlebot\nDisallow: /private/\n")
assert robots.can_fetch("https://example.com/boots/", "googlebot") is True
assert robots.can_fetch("https://example.com/private/x", "googlebot") is False
assert robots.can_fetch("https://example.com/boots/", "otherbot") is False
def test_robots_txt_url_is_always_allowed():
assert Robots(state="disallow-all").can_fetch("https://example.com/robots.txt") is True
@pytest.mark.parametrize("status,state", [(200, "parsed"), (404, "allow-all"), (403, "allow-all"),
(500, "disallow-all"), (429, "disallow-all")])
def test_robots_txt_status_codes(status, state):
server = serve({"/robots.txt": (status, {}, "User-agent: *\nDisallow: /x\n")})
assert load_robots(server.base).state == state
server.shutdown()
def test_python_stdlib_parser_treats_403_as_disallow_all():
server = serve({"/robots.txt": (403, {}, "")})
stdlib = RobotFileParser(server.base + "/robots.txt")
stdlib.read()
assert stdlib.can_fetch("Googlebot", server.base + "/anything") is False
assert load_robots(server.base).can_fetch(server.base + "/anything") is True
server.shutdown()
def test_unreachable_robots_txt_disallows_everything():
server = serve({})
base = server.base
server.shutdown()
server.server_close()
assert load_robots(base).state == "disallow-all"
def test_fetch_follows_every_hop_and_stops_on_a_loop(sites):
chain = fetch(sites["broken"] + "/old-boots")
assert [code for code, _, _ in chain.hops] == [302, 301] and chain.status == 200
assert chain.final_url == sites["broken"] + "/boots/"
loop = fetch(sites["broken"] + "/loop-a")
assert loop.error == "redirect loop" and len(loop.hops) == 2
def test_fetch_gives_up_after_ten_hops():
routes = {f"/{kind}/{n}": (301, {"Location": f"/{kind}/{n + 1}"}, "") for kind, last in (("a", 10), ("b", 11)) for n in range(last)}
routes.update({"/a/10": (200, {"Content-Type": "text/plain"}, "done"), "/b/11": (200, {"Content-Type": "text/plain"}, "done")})
server = serve(routes)
ten = fetch(server.base + "/a/0")
assert len(ten.hops) == 10 and ten.error == "" and ten.body == "done"
eleven = fetch(server.base + "/b/0")
assert eleven.error == "more than 10 redirect hops" and len(eleven.hops) == 10
server.shutdown()
def test_fetch_reports_a_connection_failure():
server = serve({})
base = server.base
server.shutdown()
server.server_close()
assert fetch(base + "/").error.startswith("request failed")
def test_parse_reads_head_fields_and_ignores_a_canonical_in_the_body():
info = parse_page('<html><head><title> Hi\n there </title><meta name="Description" content="d">'
'<meta name="robots" content="NoIndex"><link rel="alternate canonical" href="/a"></head>'
'<body><link rel="canonical" href="/b"><h1>Head</h1><script>var x = 1;</script>'
'<a href="/x">x</a><a>y</a><span onclick="go()">z</span></body></html>')
assert info.title == "Hi there" and info.description == "d" and info.h1 == "Head"
assert info.robots == [("robots", "noindex")] and info.canonicals == ["/a"]
assert [a.href for a in info.anchors] == ["/x", None] and info.clickable_non_links == 1
assert info.text_length == len("Head") + 1 + 1 + 1
def test_parse_reads_a_robots_meta_tag_that_sits_in_the_body():
info = parse_page('<html><head></head><body><meta name="robots" content="noindex"><p>x</p></body></html>')
assert info.robots == [("robots", "noindex")]
def test_parse_marks_a_missing_title_and_description_as_none():
info = parse_page("<html><head></head><body>text</body></html>")
assert info.title is None and info.description is None and info.canonicals == []
def test_directives():
assert split_directives("none") == {"none", "noindex", "nofollow"}
assert header_directives(["googlebot: noindex"]) == {"noindex"}
assert header_directives(["otherbot: noindex"]) == set()
assert header_directives(["noindex, nofollow"]) == {"noindex", "nofollow"}
assert header_directives(["unavailable_after: 25 Jun 2030 15:00:00 PST"]) == {"unavailable_after"}
assert header_directives(["max-snippet: 50, noindex"]) == {"max-snippet", "noindex"}
info = parse_page('<head><meta name="robots" content="nofollow"><meta name="googlebot" content="noindex"></head>')
assert meta_directives(info) == {"nofollow", "noindex"}
assert meta_directives(info, "otherbot") == {"nofollow"}python -m pytest -q --tb=short -p no:cacheprovider test_audit.py..................................... [100%]
37 passed in 10.02sAll 37 tests pass. (The -p no:cacheprovider option only stops pytest from creating a cache folder.) Passing tests prove little by themselves, though, because a test that never fails could be testing nothing. The second safety net is to break the code on purpose. mutants.py plants one bug at a time in a copy of the code, such as making Disallow win a tie, dropping loop detection, or letting a canonical in the body count, and checks that the tests notice.
# mutants.py
"""Plant one bug at a time in a copy of the code and check that the tests notice. python mutants.py"""
import shutil
import subprocess
import sys
from pathlib import Path
HERE = Path(__file__).resolve().parent
WORK = HERE.parent / "mutant-copy"
# (what the bug does, file, text to find, replacement)
MUTANTS = [
("robots: on equal length Disallow wins", "robots.py", "(rule.length == best.length and rule.allow and not best.allow)", "(rule.length == best.length and not rule.allow and best.allow)"),
("robots: first matching rule wins", "robots.py", "rule.length > best.length", "False"),
("robots: a trailing $ is ignored", "robots.py", 'anchored = pattern.endswith("$")', "anchored = False"),
("robots: a 404 robots.txt blocks everything", "robots.py", ' if fetched.status >= 400:\n return Robots(state="allow-all")', ' if fetched.status >= 400:\n return Robots(state="disallow-all")'),
("fetch: redirect loops are not detected", "fetch.py", "if target in seen:", "if False:"),
("fetch: allows an eleventh hop", "fetch.py", "if len(result.hops) >= max_hops:", "if len(result.hops) > max_hops:"),
("parse: a canonical in the body counts", "parse.py", 'elif tag == "link" and self.in_head and', 'elif tag == "link" and'),
("indexing: none is not noindex", "indexing.py", 'if "none" in found:', "if False:"),
("indexing: another crawler's header applies", "indexing.py", "if crawler != agent:", "if False:"),
("audit: soft-404 ignores page length", "audit.py", "SOFT_404_MAX_TEXT = 600", "SOFT_404_MAX_TEXT = 10 ** 9"),
("audit: redirecting URLs count as pages", "audit.py", "and not page.fetched.hops and not page.blocked", "and not page.blocked"),
("audit: sitemap noindex check dropped", "audit.py", "elif page.state.noindex_source:\n issues.append(Issue(ERROR, \"sitemap-noindex\"", "elif False:\n issues.append(Issue(ERROR, \"sitemap-noindex\""),
]
def run_tests(folder):
result = subprocess.run([sys.executable, "-m", "pytest", "-q", "-x", "--tb=no", "-p", "no:cacheprovider"],
cwd=folder, capture_output=True, text=True)
return result.returncode, (result.stdout.strip().splitlines() or ["no output"])[-1]
def fresh_copy():
shutil.rmtree(WORK, ignore_errors=True)
shutil.copytree(HERE, WORK, ignore=shutil.ignore_patterns("out", "__pycache__", "mutants.py", "run_all.py"))
if __name__ == "__main__":
fresh_copy()
code, line = run_tests(WORK)
print(f"{'unchanged code':<46} {'tests pass' if code == 0 else 'TESTS FAIL'} ({line})")
caught = 0
for name, filename, old, new in MUTANTS:
fresh_copy()
path = WORK / filename
text = path.read_text(encoding="utf-8")
assert text.count(old) == 1, f"{name}: expected one match in {filename}, found {text.count(old)}"
path.write_text(text.replace(old, new), encoding="utf-8", newline="\n")
code, line = run_tests(WORK)
caught += code != 0
print(f"{name:<46} {'caught' if code else 'SURVIVED'} ({line})")
print(f"caught {caught} of {len(MUTANTS)}")
shutil.rmtree(WORK, ignore_errors=True)python mutants.pyunchanged code tests pass (37 passed in 10.11s)
robots: on equal length Disallow wins caught (1 failed, 5 passed in 0.86s)
robots: first matching rule wins caught (1 failed in 0.77s)
robots: a trailing $ is ignored caught (1 failed, 8 passed in 0.85s)
robots: a 404 robots.txt blocks everything caught (1 failed, 24 passed in 1.36s)
fetch: redirect loops are not detected caught (1 failed in 0.78s)
fetch: allows an eleventh hop caught (1 failed, 31 passed in 7.01s)
parse: a canonical in the body counts caught (1 failed, 33 passed in 10.08s)
indexing: none is not noindex caught (1 failed, 36 passed in 10.07s)
indexing: another crawler's header applies caught (1 failed, 36 passed in 10.08s)
audit: soft-404 ignores page length caught (1 failed in 0.77s)
audit: redirecting URLs count as pages caught (1 failed in 0.77s)
audit: sitemap noindex check dropped caught (1 failed in 0.76s)
caught 12 of 12All 12 planted bugs are caught. Do not skip this step when you change the tool. If you add a check, add a mutant for it. A bug that survives is a place where your tests are too weak.
Common Mistakes and How to Avoid Them
- Blocking in robots.txt a page you want out of search. That stops Google from reading the
noindex. Let the page be crawled and keep thenoindex. If it must also be unreachable for people, protect it with a login, because robots.txt is not access control. - Listing a URL in the sitemap that says
noindex, redirects or answers an error. The sitemap should list the canonical, indexable version of each page and nothing else. - Serving an error page with status 200. People see “not found”, and the status says “here is a page”. Send a real 404 or 410.
- Writing canonical links as relative paths, or pointing them at a URL that redirects. Use the full final URL.
- Treating info lines as faults. A blocked session URL and a noindex PDF are usually right. The level exists so that you can ignore what is deliberate.
- Auditing someone else’s site, or crawling fast. Audit your own sites. This tool is sequential and honest about its User-Agent, and a limit on the number of URLs is a good default for any crawler you write.
- Believing a clean audit means “indexed”. It means the avoidable technical reasons are gone. Search Console is where Google reports on its own view of your URLs.
What This Auditor Does Not Do
- It reads the HTML the server sends. Google can run JavaScript, but this tool does not, so a page that builds its links or content in the browser looks emptier to it than it does to a visitor.
- It reads robots meta tags and
X-Robots-Tagrules for Googlebot only. It ignores values such asunavailable_afterand the rules for other crawlers. - It does not check hreflang, structured data, images,
nofollowlinks,<base href>, page speed or mobile layout. - It does not read compressed sitemaps, and it does not enforce the robots.txt size limit. Google’s page says it enforces a limit of 500 kibibytes and ignores content after it.
- For a server error on robots.txt it blocks everything, while Google keeps using its last good copy of the file for a while.
Check the Whole Thing and Keep Going
To confirm the whole tutorial works end to end, start both shops, run python audit.py http://127.0.0.1:8080 and expect 6 errors, 14 warnings and 6 info lines, then run python audit.py http://127.0.0.1:8081 and expect 0, 0 and 5. Then run the tests and the planted-bug check and expect 37 passed and 12 of 12 caught.
The exit code is what makes this useful in a deploy pipeline. audit.py exits with 1 when it finds an error, and with --strict also when it finds a warning. Run it against a staging site after each deploy and let a failed run stop the release. --json prints the issues as JSON for other tools to read.
Good next steps:
- Run the audit on your own site, read every error, and decide for each warning whether it is a mistake or a choice.
- Add one check at a time, with a planted problem in
shop.py, a test and a mutant. Real audit tools are largely long lists of checks like these. - Read the pages in the sources list below. They are short and they are the actual rules.
- Combine this with Search Console. The auditor finds what you can fix on your side, and Search Console tells you what Google saw.
- If you care about who may read your site, our posts on the robots.txt training signal and AI agents and robots.txt show where the file stops being enough.
Sources and Further Reading
- Google Search Central, How Google interprets the robots.txt specification (precedence, wildcards, status codes, size limit).
- Google Search Central, How HTTP status codes affect Google’s crawlers (4xx, 5xx, redirect hops, soft 404).
- Google Search Central, Robots meta tag, data-nosnippet, and X-Robots-Tag specifications.
- Google Search Central, How to specify a canonical URL with
rel="canonical"and other methods. - Google Search Central, Redirects and Google Search.
- Google Search Central, Link best practices for Google.
- Google Search Central, Build and submit a sitemap.
- IETF, RFC 9309: Robots Exclusion Protocol.
- sitemaps.org Protocol.
- Python documentation, urllib.robotparser and html.parser.
- freeCodeCamp, How to Find and Fix Common Technical SEO Issues on Your Website, the guide that prompted this tutorial.
No Comment! Be the first one.