Project
robots-audit
robots-audit fetches a site's robots.txt (following redirects), parses it per the robots exclusion protocol, and reports actionable findings: parse violations, orphan Disallow lines, missing wildcard User-agent coverage, missing sitemap declarations, unexpected HTTP statuses, and oversize files. Zero runtime dependencies — stdlib only (net/http, uri, optparse, json). Use it for SEO hygiene checks, crawl-budget reviews, and CI gates.
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
2020
2021
2022
2023
2024
2025
2026
Development
Licenses
MIT
Dependencies