Terminal
Terminal/Skill
Pulls named fields from a web page, a list of URLs, or a directory into clean rows, starting with free JSON APIs before rendered scraping.
By Clay
Works with
npx skills add clay-run/clay-skill-creator --skill scrape-any-websiteThe insight: the rendered page is the most expensive, most brittle place to get its own data. Most "scraping" jobs are an API call or a static fetch in disguise — the pretty page is a thin front-end over JSON, or its URLs follow a pattern you can build instead of crawl. So this skill works DOWN a ladder — API → URL pattern → static fetch → rendered/anti-bot scrape — and only pays for the rung the target actually requires. The bottom rung's CSS selectors rot quarterly; every trip down the ladder is maintenance debt you should decline when a higher rung works.
Nothing here ships with a value. Each one is the installer's, not the author's: ask for it, never substitute a plausible default, and where an answer does not exist say which step becomes unavailable rather than guessing. Where a default IS defensible it is named below, and using it means saying so in the output.
| Input | What the installer supplies | If it is missing |
|---|---|---|
| The target | one URL, a list of them, or an index page to expand | no default |
| The fields | exactly what comes out per row, named like table headers | if they cannot name fields they want reading, not scraping — offer a text pull instead of guessing at a schema |
| Volume and recurrence | one page, hundreds, or a recurring pull | ask. One page is an ad-hoc call; hundreds or a recurring cadence belongs in a table or workflow, and the skill should say so rather than looping |
| Legitimacy | that the pages are public | public pages only — no login walls, no paywalled content, and no personal-data harvesting beyond what a page publicly presents |
If an answer sheet is present beside this skill, load it and ask only for what it does not cover. A partial sheet is normal; a value it is missing gets asked for on its own rather than restarting the interview. Say which values came from the sheet before using them — a sheet applied silently is a wrong field nobody catches. If there is no sheet, say nothing about sheets — the check is a file lookup, not a question, so run the interview as though the feature did not exist rather than reporting an absence. At delivery, offer to save the answers back (identifiers only — never a token or a password), private and never published — and phrase the offer so it explains itself: "want me to save your answers to a file, so the next person on your team doesn't have to answer these again?"
Run clay whoami; echo "exit_code=$?". If it fails or Clay tools are missing, run the
Clay plugin's setup skill, restart if it says to, and re-run this skill. Tell the
user which workspace you're in. Confirm the three actions exist in the catalog and
read their LIVE costs (clay workflows actions list) — verified costs and configs are
in references/scraper-ladder.md, and they drift.
Test each rung cheaply before falling to the next
(references/scraper-ladder.md §Rung tests):
http-api-v2,
parameterize query/page. Structured JSON out, no selectors to rot.scrape-website for plain HTML — text, links,
or html output; outputFields: ["links"] is the canonical "give me every link so
I can filter" move for index pages.zenrows-run-scrape with js_render /
premium_proxy / anti_bot as the target demands, plus css_extractor to return
named fields instead of raw HTML. Last rung, most brittle — selectors are
maintenance debt with a shelf life.State the chosen rung, why, and the cost arithmetic (pages × rung cost) — get explicit approval before any multi-page run.
css_extractor returns
column-major parallel arrays — recolumnize to rows before delivering.references/scraper-ladder.md §Failure shapes) and report them as such.Rows with the named fields + source URL, plus a run summary: pages fetched, rows extracted, pages that failed and HOW (404 / blocked / empty / challenge), rung used, credits spent (measured from usage metadata). A field the page doesn't contain stays empty with a note — never inferred from the model's own knowledge of the site.
Ask: "Scrape the speaker list off this conference site into name / title / company."
Rung test: the speakers page loads its grid from /api/speakers?page=1 (DevTools,
two minutes) → rung 1: http-api-v2 GET, paginated 1..4, free, structured JSON —
names/titles/companies land as clean rows with source URLs; 2 pages returned 404
past the last page → reported as end-of-pagination, run stopped.
Counter-example: "get every portfolio company off this VC's site" — no API found,
but portfolio pages live at /companies/a … /companies/z → rung 2 URL
interpolation (26 URLs) + scrape-website links output (~26 credits, approved) —
Zenrows never fired, no selectors to maintain.