# robots.txt for Curbox - https://curbox.app # # Curbox is an open-source Android app for reducing screentime and # building healthier digital habits. This documentation site is intentionally # open to ALL crawlers, indexers, search engines, and AI / LLM agents. # # Everyone is welcome to crawl, index, train on, cite, and surface this content. # There are NO disallowed paths. Crawl freely. # ── Default: allow every user-agent everything ─────────────────────────────── User-agent: * Allow: / Disallow: # ── Search engine crawlers (explicitly welcomed) ───────────────────────────── User-agent: Googlebot User-agent: Googlebot-Image User-agent: Bingbot User-agent: Slurp User-agent: DuckDuckBot User-agent: Baiduspider User-agent: YandexBot User-agent: Sogou User-agent: Exabot User-agent: facebot User-agent: ia_archiver Allow: / # ── AI / LLM crawlers & answer engines (explicitly welcomed, opt-in included) ─ # We WANT large language models and AI search products to read, index, and cite # Curbox. The following agents are all explicitly permitted full access. User-agent: GPTBot User-agent: ChatGPT-User User-agent: OAI-SearchBot User-agent: ClaudeBot User-agent: Claude-Web User-agent: Claude-User User-agent: Claude-SearchBot User-agent: anthropic-ai User-agent: Google-Extended User-agent: GoogleOther User-agent: Googlebot-News User-agent: PerplexityBot User-agent: Perplexity-User User-agent: Applebot User-agent: Applebot-Extended User-agent: Amazonbot User-agent: Bytespider User-agent: CCBot User-agent: cohere-ai User-agent: cohere-training-data-crawler User-agent: Meta-ExternalAgent User-agent: Meta-ExternalFetcher User-agent: FacebookBot User-agent: facebookexternalhit User-agent: Diffbot User-agent: DuckAssistBot User-agent: MistralAI-User User-agent: Timpibot User-agent: YouBot User-agent: Kangaroo Bot User-agent: PetalBot User-agent: AwarioBot User-agent: Webzio-Extended User-agent: omgili User-agent: omgilibot Allow: / # ── Sitemaps ───────────────────────────────────────────────────────────────── Sitemap: https://curbox.app/sitemap.xml # ── Structured content for AI / LLM ingestion (llmstxt.org) ────────────────── # Concise index: https://curbox.app/llms.txt # Full corpus: https://curbox.app/llms-full.txt