# Spendify marketing site robots.txt
#
# IMPORTANT: this file must stay the ONLY source of crawler directives for
# spendify.money. Cloudflare's "Managed robots.txt" setting -- dashboard path:
# AI Crawl Control -> Signals -> "Managed robots.txt" -- PREPENDS a
# Cloudflare-authored block ahead of this file, zone-wide (it applied to ~51
# hostnames here, not just the marketing site). That managed block is
# all-or-nothing: the per-crawler switches under AI Crawl Control -> Security
# control EDGE BLOCKING only and cannot grant an exception to it.
#
# Its default blocklist includes `Amazonbot` -- Amazon's own crawler -- so with
# the setting on, the site looks dead to Amazon while looking perfectly healthy
# to everyone else. That matters because the Activate application form states
# AWS uses "automated means to collect publicly available information from your
# website as part of evaluating your credit application" (#1168). Note the
# rejection AWS actually recorded cited only its email/domain rule and never
# mentioned the website, so treat the Amazonbot block as a real defect we fixed,
# not as the proven cause of that rejection.
#
# So: keep the Cloudflare managed setting OFF and express our preferences here.
# Re-enabling it silently re-blocks Amazonbot. See #1168.

# Content Signals (https://contentsignals.org/) -- we allow search indexing and
# reference-style reuse, and we do not consent to AI training.
User-agent: *
Content-Signal: search=yes,ai-train=no,use=reference
Allow: /

# --- AI training crawlers: blocked -------------------------------------------
# Mirrors Cloudflare's managed blocklist, DELIBERATELY OMITTING Amazonbot.
User-agent: Applebot-Extended
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: GPTBot
Disallow: /

User-agent: meta-externalagent
Disallow: /

# --- Explicitly allowed ------------------------------------------------------
# Amazonbot is Amazon's search/retail crawler, not an AI-training crawler, and
# AWS's Activate verification depends on it reaching the site. Do not block it.
User-agent: Amazonbot
Allow: /

# Everything else may crawl the whole site.
User-agent: *
Allow: /

Sitemap: https://spendify.money/sitemap-index.xml