robots.txt Generator

Create common robots.txt rules for crawl access and blocked paths.

robotscrawlerseotxt

Privacy note: this tool runs in your browser; inputs are not uploaded.

Rules

Preset

Paths to disallow

One path per line. Enabled for the custom preset.

User-agent: *
Allow: /