# robots.txt for Colorado Web Impressions # Canonical: https://coloradowebimpressions.com/robots.txt # --------------------------------------------------------------- # Default policy — allow all reputable crawlers # Note: Googlebot, Bingbot, DuckDuckBot, Applebot, etc. inherit this # group automatically (no per-bot overrides needed). Per the robots.txt # spec only ONE group applies per user-agent, so each named group below # re-asserts the same Disallow rules to keep hygiene consistent. # --------------------------------------------------------------- User-agent: * Allow: / # Block post-conversion confirmation page (no SEO value, distorts goals) Disallow: /thank-you Disallow: /thank-you/ # Block transactional unsubscribe handler (token-gated, no SEO value) Disallow: /unsubscribe Disallow: /unsubscribe/ # Block internal pre-launch QA tool (noindex on-page, also blocked here) Disallow: /launch-checklist Disallow: /content-checklist Disallow: /launch-checklist/ Disallow: /content-checklist/ # Collapse tracking-parameter URLs onto their canonical equivalents Disallow: /*?*utm_ Disallow: /*?*gclid= Disallow: /*?*fbclid= Disallow: /*?*msclkid= Disallow: /*?*mc_cid= Disallow: /*?*mc_eid= # --------------------------------------------------------------- # AI / LLM crawlers — explicitly allowed (we publish llms.txt). # Each group repeats the same Disallows because robots.txt does NOT # inherit rules across user-agent groups. # --------------------------------------------------------------- User-agent: GPTBot Allow: / Disallow: /thank-you Disallow: /thank-you/ Disallow: /launch-checklist Disallow: /content-checklist Disallow: /*?*utm_ Disallow: /*?*gclid= Disallow: /*?*fbclid= User-agent: OAI-SearchBot Allow: / Disallow: /thank-you Disallow: /thank-you/ Disallow: /launch-checklist Disallow: /content-checklist User-agent: ChatGPT-User Allow: / User-agent: ClaudeBot Allow: / Disallow: /thank-you Disallow: /thank-you/ Disallow: /launch-checklist Disallow: /content-checklist Disallow: /*?*utm_ Disallow: /*?*gclid= Disallow: /*?*fbclid= User-agent: Claude-Web Allow: / User-agent: anthropic-ai Allow: / User-agent: PerplexityBot Allow: / Disallow: /thank-you Disallow: /thank-you/ Disallow: /launch-checklist Disallow: /content-checklist Disallow: /*?*utm_ Disallow: /*?*gclid= Disallow: /*?*fbclid= User-agent: Perplexity-User Allow: / User-agent: Google-Extended Allow: / Disallow: /thank-you Disallow: /thank-you/ User-agent: Applebot-Extended Allow: / Disallow: /thank-you Disallow: /thank-you/ User-agent: CCBot Allow: / Disallow: /thank-you Disallow: /thank-you/ User-agent: Bytespider Allow: / Disallow: /thank-you Disallow: /thank-you/ User-agent: meta-externalagent Allow: / Disallow: /thank-you Disallow: /thank-you/ # --------------------------------------------------------------- # Block known abusive / scraping bots (SEO tool harvesters) # --------------------------------------------------------------- User-agent: AhrefsBot Disallow: / User-agent: SemrushBot Disallow: / User-agent: MJ12bot Disallow: / User-agent: DotBot Disallow: / User-agent: PetalBot Disallow: / # --------------------------------------------------------------- # Discovery aids # --------------------------------------------------------------- # Human-friendly site map: https://coloradowebimpressions.com/sitemap # CSV export of sitemap: https://coloradowebimpressions.com/sitemap.csv # llms.txt: https://coloradowebimpressions.com/llms.txt # llms-full.txt: https://coloradowebimpressions.com/llms-full.txt # Both sitemaps declared so Google/Bing pick up the full feed AND the # curated AI-priority feed. Order doesn't matter to crawlers. Sitemap: https://coloradowebimpressions.com/sitemap.xml Sitemap: https://coloradowebimpressions.com/ai-sitemap.xml Sitemap: https://coloradowebimpressions.com/image-sitemap.xml # news-sitemap.xml is declared dynamically only when fresh news articles exist.