# --- START: Custom Rules --- # --- AI Crawlers --- User-agent: GPTBot User-agent: ChatGPT-User User-agent: OAI-SearchBot User-agent: GoogleOther User-agent: Google-CloudVertexBot User-agent: Anthropic-Bot User-agent: anthropic-ai User-agent: ClaudeBot User-agent: Claude-SearchBot User-agent: Claude-User User-agent: Claude-Code User-agent: PerplexityBot User-agent: Perplexity-User User-agent: CCBot User-agent: YouBot User-agent: NeevaBot User-agent: Amazonbot User-agent: bedrockbot User-agent: amazon-kendra User-agent: cohere-ai User-agent: cohere-training User-agent: GrokBot User-agent: xAI-Grok User-agent: Grok-DeepSearch User-agent: FacebookBot User-agent: Meta-ExternalAgent User-agent: Meta-ExternalFetcher User-agent: Applebot User-agent: Applebot-Extended User-agent: AzureAI-SearchBot User-agent: Bytespider User-agent: Diffbot User-agent: AI2Bot User-agent: Ai2Bot-Dolma User-agent: ApifyBot User-agent: Cloudflare-AutoRAG User-agent: Omgilibot # --- Search Engines --- User-agent: Googlebot User-agent: Bingbot User-agent: Yandex User-agent: DuckDuckBot User-agent: Baiduspider User-agent: Bravebot User-agent: PetalBot Allow: / Disallow: /cart Disallow: /carts/ Disallow: /checkout Disallow: /checkouts/ Disallow: /account Disallow: /orders Disallow: /admin Disallow: /password # --- END: Custom Rules --- User-agent: * Disallow: /cart Disallow: /checkout Disallow: /orders Disallow: /account Disallow: /search Disallow: /login Disallow: /register Disallow: /*?*_sid= Disallow: /*?*_pos= Disallow: /*?*_ss= Disallow: /*?*variant_error= # Legacy Magento URLs Disallow: /catalog/product/view/ Disallow: /catalogsearch/result/ Disallow: /catalogsearch/result/index/?q Disallow: /customer/account/ User-agent: * /a/downloads/-/* /admin /cart /orders /checkouts/ /checkout /94796316984/checkouts /94796316984/orders /carts /account /collections/*sort_by* /*/collections/*sort_by* /collections/*+* /collections/*%2B* /collections/*%2b* /*/collections/*+* /*/collections/*%2B* /*/collections/*%2b* */collections/*filter*&*filter* /blogs/*+* /blogs/*%2B* /blogs/*%2b* /*/blogs/*+* /*/blogs/*%2B* /*/blogs/*%2b* /*?*oseid=* /*preview_theme_id* /*preview_script_id* /policies/ /*/policies/ /*/*?*ls=*&ls=* /*/*?*ls%3D*%3Fls%3D* /*/*?*ls%3d*%3fls%3d* /search /apple-app-site-association /.well-known/shopify/monorail /cdn/wpm/*.js /recommendations/products /*/recommendations/products /services/login_with_shop Sitemap: https://www.asedeals.com/sitemap.xml User-agent: adsbot-google /checkouts/ /checkout /carts /orders /94796316984/checkouts /94796316984/orders /*?*oseid=* /*preview_theme_id* /*preview_script_id* /cdn/wpm/*.js /services/login_with_shop User-agent: Nutch / User-agent: AhrefsBot 10 /a/downloads/-/* /admin /cart /orders /checkouts/ /checkout /94796316984/checkouts /94796316984/orders /carts /account /collections/*sort_by* /*/collections/*sort_by* /collections/*+* /collections/*%2B* /collections/*%2b* /*/collections/*+* /*/collections/*%2B* /*/collections/*%2b* */collections/*filter*&*filter* /blogs/*+* /blogs/*%2B* /blogs/*%2b* /*/blogs/*+* /*/blogs/*%2B* /*/blogs/*%2b* /*?*oseid=* /*preview_theme_id* /*preview_script_id* /policies/ /*/policies/ /*/*?*ls=*&ls=* /*/*?*ls%3D*%3Fls%3D* /*/*?*ls%3d*%3fls%3d* /search /apple-app-site-association /.well-known/shopify/monorail /cdn/wpm/*.js /services/login_with_shop Sitemap: https://www.asedeals.com/sitemap.xml User-agent: AhrefsSiteAudit 10 /a/downloads/-/* /admin /cart /orders /checkouts/ /checkout /94796316984/checkouts /94796316984/orders /carts /account /collections/*sort_by* /*/collections/*sort_by* /collections/*+* /collections/*%2B* /collections/*%2b* /*/collections/*+* /*/collections/*%2B* /*/collections/*%2b* */collections/*filter*&*filter* /blogs/*+* /blogs/*%2B* /blogs/*%2b* /*/blogs/*+* /*/blogs/*%2B* /*/blogs/*%2b* /*?*oseid=* /*preview_theme_id* /*preview_script_id* /policies/ /*/policies/ /*/*?*ls=*&ls=* /*/*?*ls%3D*%3Fls%3D* /*/*?*ls%3d*%3fls%3d* /search /apple-app-site-association /.well-known/shopify/monorail /cdn/wpm/*.js /services/login_with_shop Sitemap: https://www.asedeals.com/sitemap.xml User-agent: MJ12bot 10 User-agent: Pinterest 1 # Allow ChatGPT Crawlers User-agent: OAI-SearchBot Allow: / User-agent: ChatGPT-User Allow: / # Allow Perplexity Crawler User-agent: PerplexityBot Allow: / # Allow Gemini AI (Google-Extended) Crawler User-agent: Google-Extended Allow: / # Allow Anthropic and Claude AI User-agent: Anthropic-ai Allow: / User-agent: ClaudeBot Allow: / User-agent: Claude-User Allow: / User-agent: Claude-SearchBot Allow: / # Allow Abblebot (Apple's web crawler) User-agent: Applebot Allow: / # Allow Common Crawl bot (CCBot) User-agent: CCBot Allow: / # Allow Microsoft's bots (BingBot, Copilot, and MSRBot) User-agent: Copilot Allow: / User-agent: MSRBot Allow: / User-agent: Bingbot Allow: / # Allow Facebook's bot (FacebookExternalHit) User-agent: facebookexternalhit Allow: / # Allow Byte Spider User-agent: ByteSpider Allow: / Sitemap: https://www.asedeals.com/sitemap.xml