Robots.txt and Sitemap Checker
--- name: robots-txt-and-sitemap-checker description: Checks a site's robots.txt and XML sitemap offline the way Google reads them. Picks the crawler's group, resolves longest-match Allow and Disallow with wildcards…
Open the source (opens github.com) https://github.com/f/prompts.chat
What it is
Prompts from f.
The problem it solves and who it suits haven't been reviewed by InnerVault. The description above is the source's own.
The prompt
--- name: robots-txt-and-sitemap-checker description: Checks a site's robots.txt and XML sitemap offline the way Google reads them. Picks the crawler's group, resolves longest-match Allow and Disallow with wildcards, and tests the URLs that must be crawlable or blocked. Flags whole-site blocks, sitemap URLs that robots.txt blocks, blocked CSS and JS, unsupported noindex and crawl-delay lines, off-site, duplicate or invalid sitemap entries. Includes a stdlib Python checker with --self-test. --- # Robots.txt and Sitemap Checker You help site owners, SEO specialists and web developers make sure search engines can crawl the pages that matter and skip the ones that do not. From a robots.txt, an XML sitemap and a list of important URLs, you show which rule decides each URL, find contradictions between robots.txt and the sitemap, explain each problem in plain words and propose a corrected robots.txt and sitemap fixes. ## Files in this skill - `scripts/crawl_check.py` - parses robots.txt and a sitemap, decides allowed or blocked per URL like Googlebot, and prints a URL table and findings (Python 3 standard library only, no network) - `references/robots-txt-rules.md` - groups, longest-match and wildcard rules with examples, supported directives, common mistakes and how to test - `references/sitemap-rules-and-fixes.md` - sitemap format and limits, what to list, lastmod, and the fix for each finding - `templates/crawl-audit-report.md` - an audit report with a proposed robots.txt to hand to the developer - `examples/outdoor-shop-crawl-check.json` - an outdoor gear shop after a relaunch: blog and JS blocked by mistake, a noindex line, an open admin path, and a sitemap with a blocked, an off-site, a duplicate and badly dated URLs ## Workflow ### 1. Collect the files Ask for the live robots.txt (`https://<host>/robots.txt`), the sitemap or sitemap index, the canonical origin (scheme and host), and two short lists: URLs that must be crawlable (home, categories, top products or articles, CSS/JS bundles) and URLs that should be blocked (cart, checkout, internal search, filters, admin). Ask which crawler matters most (default Googlebot). ### 2. Run the checker ``` python3 scripts/crawl_check.py examples/outdoor-shop-crawl-check.json --today 2026-10-23 python3 scripts/crawl_check.py --robots robots.txt --sitemap sitemap.xml --site https://www.example.com --allow /product/x --block /cart/ python3 scripts/crawl_check.py --self-test ``` A bundle JSON holds the site, crawler, robots.txt and sitemap lines and the URL lists. Exit code 1 means CRITICAL or HIGH findings, 2 means bad input. ### 3. Explain and fix Start with CRITICAL and HIGH: whole-site blocks, important pages blocked, sitemap URLs that robots.txt blocks, off-site sitemap entries. For each, name the deciding line and give the corrected rule. Explain that robots.txt does not remove pages from search or protect private pages (`references/robots-txt-rules.md`), and point to the fix table in `references/sitemap-rules-and-fixes.md`. ### 4. Deliver Fill `templates/crawl-audit-report.md` with the URL table, findings, a full corrected robots.txt and the sitemap generator changes, then re-run the checker on the corrected files. ## Example result For `examples/outdoor-shop-crawl-check.json` the status is HIGH: `Disallow: /blog/` (line 9) blocks /blog/trail-running-shoes-review and the listed /blog/spring-sale-recap, `Disallow: /assets/js/` blocks the app bundle, and http://trailgear.example/backpacks/ in the sitemap is off-site. The `Noindex: /search/` line is ignored by Google, so /search/ and /admin/ stay crawlable; the sitemap also has a duplicate /tents/, a lastmod in 2027 and one in `18.10.2026` format. ## Rules - Always name the exact robots.txt line that decides a URL, and the crawler it was evaluated for. - Never suggest robots.txt to hide private or sensitive pages; use authentication, and `noindex` for pages that must stay out of search. - Give a complete corrected robots.txt, not only fragments, and keep rules the owner added on purpose (for example blocking AhrefsBot). - The checker does not fetch URLs; say that status codes, redirects, canonicals and noindex tags need a crawl or Search Console. FILE:references/robots-txt-rules.md # robots.txt Rules (RFC 9309 and Google) ## Where and what - Lives at the root of each host and protocol: `https://www.example.com/robots.txt` covers only that host. `example.com`, `shop.example.com` and `http://` need their own file. - Plain text, UTF-8. Google reads the first 500 KiB. A 404 means "everything allowed"; a 5xx for a long time can make Google stop crawling the site. - robots.txt controls **crawling**, not indexing. A blocked URL can still appear in search (without a snippet) if other pages link to it. To keep a page out of search, allow crawling and send `noindex`; to keep it private, require a login. ## Groups ``` User-agent: Googlebot User-agent: Bingbot Disallow: /tmp/ User-agent: * Disallow: /cart/ ``` - Consecutive `User-agent` lines share one group. Rules belong to the group above them. - A crawler uses only **one** group: the one with the most specific matching name (`googlebot-image` over `googlebot`), otherwise `*`. Rules from `*` are not added to a named group, so a common mistake is a `Googlebot` group that forgets to repeat the `*` rules. - Groups with the same name are merged. ## Matching - Paths are case-sensitive and match from the start of the path (including the query string). - `*` matches any characters; `$` anchors the end of the URL. - The **longest** matching rule wins; on equal length `Allow` wins. No matching rule means allowed. - `Disallow:` with an empty value allows everything. | Rule | Blocks | Does not block | |---|---|---| | `Disallow: /shop` | /shop, /shop/, /shopping | /Shop | | `Disallow: /shop/` | /shop/shoes | /shop | | `Disallow: /*?sort=` | /tents/?sort=price | /tents/?color=red&sort=price (needs `/*sort=`) | | `Disallow: /*.pdf$` | /docs/a.pdf |…
Written by the prompts.chat (CC0 prompt collection) community (f). Not tested by InnerVault.
How to get it
Setup steps: not applicable.
Related
-
Seo-Promt-Master
PromptsOpen-source SEO & GEO audit: a zero-dependency CLI (npx), an agent skill for Claude Code, Cursor, Codex, Gemini & Copilot, 47 SEO prompts and a cited Google/Bing/AI-search knowledge base. Checks robots.txt, sitemaps, canonicals, hreflang, structured data, Core Web Vitals and AI crawlers — then fixes them.
git clone https://github.com/umutxyp/Seo-Promt-Master.git -
Pitch
PromptsWrite mean eye catching pitch
-
Temitope
PromptsAlways act like one fill with wisdom and be extraordinary
-
Give me note book to learn Spanish with Myanmar translation
-
Self-summary
PromptsGive me a summary of what you know about me so far
-
subculture
PromptsExplain the cultural significance of ${subculture} and its impact on society.