webcrawler-deep-crawl
适合整站索引、文档抓取与大规模数据收集。
大规模爬取可能造成目标站点负载或被封禁。
从起始 URL 递归深度爬取站点并返回每页结构化内容与元数据。
从几个起始 URL 出发，递归把整个网站的每一页抓下来并把内容做成 LLM-ready 的纯文本、Markdown 或 HTML，同时返回 title、description、author、语言、canonical、OG、以及页面内外链等元数据。常用场景是抓取文档站、博客或知识库，或为 RAG/向量数据库构建语料时使用。特别之处在于支持深度/页数限制、包含/排除 URL 规则和去除 boilerplate（导航/页脚）等清理，直接产出可供模型索引的结构化记录。
▸ 展开 SKILL.md 英文原文
Deep-crawl any website from start URLs, return per-page LLM-ready text/markdown/HTML plus metadata (title, description, author, language, canonical URL, OG) and in-scope outbound links. Use when user mentions deep crawl website, recursive crawl, crawl a whole site, scrape entire website, scrape docs site, scrape documentation, scrape knowledge base, scrape blog, build RAG corpus, build vector database from website, knowledge base for chatbot, GPT knowledge files, llms.txt, sitemap crawl, BFS crawl, scrape with depth or page limit, include exclude URL globs, remove boilerplate, strip navigation header footer, website to markdown, website to text, multi-page extraction, bulk page scraping, clean markdown from URL, docs site to markdown corpus, site to clean corpus. Also applies to building RAG pipelines, indexing a customer site, syncing docs into a vector store, generating training corpora from any docs hub, or expanding a single start URL into a clean corpus of every reachable in-scope
帮我安装这个 skill：https://raw.githubusercontent.com/browser-act/skills/main/solutions/search-research/webcrawler-deep-crawl/SKILL.mdcurl -fsSL "https://raw.githubusercontent.com/browser-act/skills/main/solutions/search-research/webcrawler-deep-crawl/SKILL.md"# Website Deep Crawl
> Input: one or more start URLs (+ optional scope, depth, page-count, globs, removal selectors). Output: per-page records `{url, crawl, metadata, text, markdown, html, outboundLinks}` for every page reached within scope.
## Language
All process output to user (progress updates, process notifications) follows the user's language.
## Objective
From a small set of start URLs, breadth-first crawl every reachable in-scope page, strip boilerplate (navigation, header, footer, cookie banners, etc.), and emit per-page LLM-ready content (text / markdown / HTML) plus structured metadata — suitable for feeding RAG pipelines, vector databases, or chatbot knowledge bases.
## Pre