robots.txt tester
Fetches a site's robots.txt and checks whether a path is allowed for a given user agent, using longest-match rules.
A domain or full URL both work.
Fetches a site's robots.txt and checks whether a path is allowed for a given user agent, using longest-match rules.
A domain or full URL both work.