🤖

Robots.txt Generator

Build a robots.txt file with allow/disallow rules per crawler and a sitemap link.

Rules

Allow all by default Block all crawlers

Sitemap URL (optional)

Crawl-delay seconds (optional, non-standard)

Block common AI crawlers (GPTBot, CCBot…)

robots.txt