User-agent: * Disallow: /admin/ # ---- 分页/排序/筛选组合页：重复内容，每个 URL 唯一导致边缘缓存必然回源（2026-09-10 加）---- Disallow: /*/page- Disallow: /*/sort- Disallow: /*/limit- Disallow: /*page= Disallow: /*sort= Disallow: /*limit= Disallow: /*mfilter # ===================================================================== # AI 策略：允许"被 AI 搜索到"，拒绝"被拿去训练模型" # 2026-09-03 制定。改动前请先读 网站运维/CLAUDE.md # ===================================================================== # ---- 允许：AI 搜索索引 / 用户提问时实时取回（这些放行才会被 AI 引用）---- User-agent: OAI-SearchBot Allow: / # ---- 分页/排序/筛选组合页：重复内容，每个 URL 唯一导致边缘缓存必然回源（2026-09-10 加）---- Disallow: /*/page- Disallow: /*/sort- Disallow: /*/limit- Disallow: /*page= Disallow: /*sort= Disallow: /*limit= Disallow: /*mfilter Crawl-delay: 3 User-agent: ChatGPT-User Allow: / # ---- 分页/排序/筛选组合页：重复内容，每个 URL 唯一导致边缘缓存必然回源（2026-09-10 加）---- Disallow: /*/page- Disallow: /*/sort- Disallow: /*/limit- Disallow: /*page= Disallow: /*sort= Disallow: /*limit= Disallow: /*mfilter User-agent: Claude-SearchBot Allow: / # ---- 分页/排序/筛选组合页：重复内容，每个 URL 唯一导致边缘缓存必然回源（2026-09-10 加）---- Disallow: /*/page- Disallow: /*/sort- Disallow: /*/limit- Disallow: /*page= Disallow: /*sort= Disallow: /*limit= Disallow: /*mfilter Crawl-delay: 3 User-agent: Claude-User Allow: / # ---- 分页/排序/筛选组合页：重复内容，每个 URL 唯一导致边缘缓存必然回源（2026-09-10 加）---- Disallow: /*/page- Disallow: /*/sort- Disallow: /*/limit- Disallow: /*page= Disallow: /*sort= Disallow: /*limit= Disallow: /*mfilter User-agent: PerplexityBot Allow: / # ---- 分页/排序/筛选组合页：重复内容，每个 URL 唯一导致边缘缓存必然回源（2026-09-10 加）---- Disallow: /*/page- Disallow: /*/sort- Disallow: /*/limit- Disallow: /*page= Disallow: /*sort= Disallow: /*limit= Disallow: /*mfilter Crawl-delay: 3 User-agent: Perplexity-User Allow: / # ---- 分页/排序/筛选组合页：重复内容，每个 URL 唯一导致边缘缓存必然回源（2026-09-10 加）---- Disallow: /*/page- Disallow: /*/sort- Disallow: /*/limit- Disallow: /*page= Disallow: /*sort= Disallow: /*limit= Disallow: /*mfilter User-agent: DuckAssistBot Allow: / # ---- 分页/排序/筛选组合页：重复内容，每个 URL 唯一导致边缘缓存必然回源（2026-09-10 加）---- Disallow: /*/page- Disallow: /*/sort- Disallow: /*/limit- Disallow: /*page= Disallow: /*sort= Disallow: /*limit= Disallow: /*mfilter User-agent: Amzn-SearchBot Allow: / Crawl-delay: 10 # ---- 分页/排序/筛选组合页：重复内容，每个 URL 唯一导致边缘缓存必然回源（2026-09-10 加）---- Disallow: /*/page- Disallow: /*/sort- Disallow: /*/limit- Disallow: /*page= Disallow: /*sort= Disallow: /*limit= Disallow: /*mfilter User-agent: meta-webindexer Allow: / # ---- 分页/排序/筛选组合页：重复内容，每个 URL 唯一导致边缘缓存必然回源（2026-09-10 加）---- Disallow: /*/page- Disallow: /*/sort- Disallow: /*/limit- Disallow: /*page= Disallow: /*sort= Disallow: /*limit= Disallow: /*mfilter User-agent: Applebot Allow: / # ---- 分页/排序/筛选组合页：重复内容，每个 URL 唯一导致边缘缓存必然回源（2026-09-10 加）---- Disallow: /*/page- Disallow: /*/sort- Disallow: /*/limit- Disallow: /*page= Disallow: /*sort= Disallow: /*limit= Disallow: /*mfilter # ---- 拒绝：抓取内容用于训练大模型 ---- User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: anthropic-ai Disallow: / User-agent: Claude-Web Disallow: / User-agent: Google-Extended Disallow: / User-agent: Applebot-Extended Disallow: / User-agent: CCBot Disallow: / User-agent: Bytespider Disallow: / User-agent: meta-externalagent Disallow: / User-agent: FacebookBot Disallow: / User-agent: Amazonbot Disallow: / User-agent: AI2Bot Disallow: / User-agent: Diffbot Disallow: / User-agent: ImagesiftBot Disallow: / User-agent: Omgilibot Disallow: / User-agent: Omgili Disallow: / User-agent: Timpibot Disallow: / User-agent: Webzio-Extended Disallow: / User-agent: cohere-ai Disallow: / User-agent: PanguBot Disallow: / User-agent: Kangaroo Bot Disallow: / # ---- 拒绝：SEO / 数据转售爬虫，对销售无价值且吃带宽 ---- User-agent: AhrefsBot Disallow: / User-agent: SemrushBot Disallow: / User-agent: MJ12bot Disallow: / User-agent: DotBot Disallow: / User-agent: DataForSeoBot Disallow: / User-agent: Barkrowler Disallow: / User-agent: BrightEdge Disallow: / User-agent: PetalBot Disallow: / User-agent: SeobilityBot Disallow: / User-agent: Qwantbot Disallow: / # ---- 商品数据抓取（2026-09-10 加）：83.99.206.0/24 全段扫全库，已同时在 CF 封锁 ---- User-agent: GeedoShopProductFinder Disallow: / User-agent: GeedoShopProductImageFetcher Disallow: / # ---- 中国系爬虫（2026-09-10 加）：面向海外客户的站点，这些搜索引擎无业务价值；已同时在 CF 按 UA 拦截 ---- # 注意：stepperonline.cn 是独立 DocumentRoot、不用这份 robots，百度在那边照常收录 User-agent: Baiduspider Disallow: / User-agent: Sogou web spider Disallow: / User-agent: 360Spider Disallow: / User-agent: HaoSouSpider Disallow: / User-agent: YisouSpider Disallow: / User-agent: YoudaoBot Disallow: / User-agent: EasouSpider Disallow: / User-agent: JikeSpider Disallow: / User-agent: Sohu-Search-Robot Disallow: / User-agent: Dola-SearchBot Disallow: / User-agent: Qwenbot Disallow: /