AI crawlers

Cecil generates a robots.txt file that allows all crawlers. To block AI crawlers (or some of them), add your own rules by creating a page with the robots path:

pages/robots.md

---
layout: robots
output: txt
---
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

The body of the page is appended to the default rules (see default pages).