# robots.txt for Sundvolden Hotel
# Source: PKND Tech Studio
# Stance: allow search engines and AI search crawlers, block AI training crawlers.
# Reference: https://www.rfc-editor.org/rfc/rfc9309.html
#
# RFC 9309 selects exactly ONE group per crawler (most specific match).
# Groups do NOT inherit from *, so each group states its own rules.
# No comment or blank line appears between User-agent: lines — a comment-only
# line is an `emptyline` per the ABNF and splits the group on legacy parsers.

# ── Search engines + AI citation agents (full access, minus internals) ──
User-agent: Googlebot
User-agent: bingbot
User-agent: Slurp
User-agent: DuckDuckBot
User-agent: YandexBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Applebot
User-agent: Amzn-SearchBot
User-agent: Amzn-User
User-agent: Meta-ExternalFetcher
User-agent: DuckAssistBot
User-agent: MistralAI-User
User-agent: MistralAI-Index
User-agent: YouBot
Disallow: /api/
Allow: /

# ── AI training / scraping crawlers (opt-out) ───────────────────────────
# DECISION — Google-Extended: blocking it also costs a live answer surface.
# Opts out of Gemini model training AND Gemini-app grounding/citations. Google
# Search, AI Overviews and AI Mode are unaffected either way. To exit AI
# features while staying in Search, use Search Console "Search generative AI
# controls" (June 2026) rather than robots.txt.
# DECISION — Amazonbot: blocking it may also cost a live answer surface.
# Content is withheld from Amazon product/service improvement and from Amazon
# AI model training. Whether that ALSO costs a live answer surface is not
# settled: Amazon assigns Alexa to two separate tokens (Amzn-SearchBot for
# search eligibility, Amzn-User for live queries), yet describes Amazonbot only
# as improving "our products and services" — wider than training, and without
# the "does not crawl content for generative AI model training" limit Amazon
# publishes for the other two. Flagged as citation-costing because the live
# cost cannot be ruled out, not because it is proven.
User-agent: GPTBot
User-agent: Google-Extended
User-agent: Google-CloudVertexBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Applebot-Extended
User-agent: Bytespider
User-agent: Amazonbot
User-agent: cohere-ai
User-agent: Meta-ExternalAgent
User-agent: FacebookBot
User-agent: MistralAI-Training
User-agent: FirecrawlAgent
User-agent: Diffbot
User-agent: Claude-Web
User-agent: anthropic-ai
User-agent: ImagesiftBot
User-agent: Omgilibot
User-agent: Omgili
Disallow: /

# ── Fallback for unnamed crawlers ───────────────────────────────────────
User-agent: *
Disallow: /api/
Allow: /

# ── Sitemaps ────────────────────────────────────────────────────────────
# Non-XML resources are intentionally NOT declared here — Sitemap: requires
# XML, and a non-XML target is a permanent Search Console parse error that
# masks real sitemap failures. Files such as llms.txt are discovered at
# their own well-known root path.
Sitemap: https://sundvolden.no/sitemap.xml