All processing happens locally in your browser. No data is sent to any server.
Robots.txt Directives
User-agent
Specifies which crawler the rules apply to. Use * for all bots.
Disallow
Blocks access to specified paths. Empty value allows all.
Allow
Explicitly allows paths (useful for exceptions within disallowed directories).
Sitemap
Points to XML sitemap location. Can have multiple entries.
Crawl-delay
Seconds to wait between requests. Not universally supported.
Wildcards
* matches any sequence, $ matches end of URL.