Configure crawl rules with presets (including AI bot blocking), generate XML sitemaps, and download production-ready files.
//admin//api//private/# Generated by The IT Hustle -- www.the-it-hustle.com # robots.txt User-agent: * Disallow: /admin/ Disallow: /api/ Disallow: /private/ Allow: / Sitemap: https://example.com/sitemap.xml
robots.txt in your site's root directoryyour-site.com/robots.txt