Terminal
Terminal/Skill
Bulk extracts many pages from one site or section by following its links, such as everything under a docs path.
By Firecrawl
Works with
npx skills add firecrawl/cli --skill firecrawl-crawlBulk extract content from a website. Crawls pages following links up to a depth/limit.
Prerequisite: crawl requires authentication (no keyless free tier); without credentials the CLI prompts an interactive login.
# Crawl a docs section
firecrawl crawl "<url>" --include-paths /docs --limit 50 --wait -o .firecrawl/crawl.json
# Full crawl with depth limit
firecrawl crawl "<url>" --max-depth 3 --wait --progress -o .firecrawl/crawl.json
# Check status of a running crawl
firecrawl crawl <job-id>
Run firecrawl crawl --help for the full option list.
Done when: the crawl reaches a terminal status and the saved output under .firecrawl/ contains the expected pages.
--wait when you need the results immediately. It has no default timeout; use --timeout <seconds> to bound polling. Without --wait, crawl returns a job ID for async polling.--include-paths whenever the request names a section — crawl only the pages you need.firecrawl credit-usage before large crawls (credit-usage requires authentication).