Recipes (beta)¶

A recipe is a ready-made scraper for a well-known site. You send a small input - a search query, a repository name - and get back clean JSON with a stable schema, instead of raw HTML to parse yourself. ScrapeNest maintains the scraper: when the site changes, we fix the recipe; your integration does not change.

How it works¶

Every recipe publishes one or more contracts. A contract is the stable part: the JSON Schema of the input you send and of the output you receive. Behind a contract, ScrapeNest runs an implementation (impl) - the actual scraper. Implementations change whenever the site changes; contracts do not. A new contract is only published for a breaking output change, and an old contract keeps working until its announced sunset date.

Every output is validated against its contract's output schema before it is delivered. If a site change breaks a recipe, you get a clear outcome (layout_changed) and no charge - never silently wrong data.

Catalog¶

All recipes are in beta. In the console they are grouped by category, with search; GET /v1/recipes returns each recipe's category.

Search engines

Video

E-commerce

App stores & browser extensions

Safari extensions are sold on the App Store, so the app-store-* recipes cover them. Review recipes never return the reviewer's name or profile.

Social & communities (public content only; no logged-in data)

Social recipes return what anyone can see without an account. Networks that require a login to read (X, Instagram, Facebook, LinkedIn, TikTok, Threads) are not in the catalog.

Code hosting

Developer tools

News (publishers' own RSS feeds: headline, link, summary, author, date, image - never article text)

Knowledge & reference

Wikipedia and Stack Overflow content is CC BY-SA: attribute it if you republish it.

The console's Recipes page and GET /v1/recipes list what your account can run, with each recipe's status (beta / ga) and health:

• Operational - our continuous test runs (every 15 minutes) pass.
• Degraded - test runs are failing; we are on it. Runs still work when the site allows it, and failed runs are not billed.
• No data yet - a new recipe without test results yet.

Run a recipe¶

curl -X POST https://api.scrapenest.com/v1/recipes/github-issues/runs \
  -H "X-API-Key: $SCRAPENEST_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"input": {"owner": "cli", "repo": "cli", "state": "open"}}'
# -> 201 {"job_id": "...", "status": "queued", "recipe": {"slug": "github-issues", "contract": 1, "impl": 3}}

curl https://api.scrapenest.com/v1/jobs/<job_id>/result -H "X-API-Key: $SCRAPENEST_API_KEY"

# client.recipes is available from the SDK release that follows 1.2.0.
from scrapenest import ScrapeNestClient

client = ScrapeNestClient(api_key="sn_live_...")
result = client.recipes.run_and_wait("github-issues", {"owner": "cli", "repo": "cli"})
if result.outcome == "ok":
    for issue in result.output["issues"]:
        print(issue["number"], issue["title"])

GET /v1/recipes/{slug} returns each contract's input_schema, output_schema, an example input and output, the engine it runs on and its credits_per_unit. Pass contract in the run request to pin a contract; otherwise the newest one runs. Pass an idempotency_token to make retries safe: the same token returns the original job instead of a new one.

Getting the result¶

A run is an ordinary job: it appears in your job history, and you can follow it the same way.

• Poll: GET /v1/jobs/{job_id}/result returns 409 result_not_ready while the job runs, then the structured result.
• Webhook: the job.completed event of a recipe job carries a data.recipe block with the outcome and, when it is at most 256 KB, the result itself (otherwise a result_url).

{
  "job_id": "7473a7e8-39bd-4fca-ab9b-e1e5293d6ed3",
  "status": "succeeded",
  "recipe": {"slug": "github-issues", "contract": 1, "impl": 3, "outcome": "ok"},
  "captured_at": "2026-10-10T08:00:05+00:00",
  "output": {"issues": [{"number": 11873, "title": "...", "state": "open", "labels": ["bug"], "...": "..."}], "pages_fetched": 1, "captured_at": "..."}
}

Outcomes and billing¶

You pay only when the recipe gives a definitive answer:

The price of one unit is the engine's credit weight times the recipe's multiplier, rounded up (credits_per_unit in GET /v1/recipes/{slug}). Recipes billed per page (such as github-issues) reserve credits for the maximum pages you ask for (max_pages) and charge only for the pages actually fetched; the rest is released when the job finishes. Your plan caps the pages a single run may fetch:

Inside a recipe¶

Every recipe is written in a small declarative language: no custom code per site. Here is most of hackernews-search:

slug: hackernews-search
contract: 1
tier: light
steps:
  - request:
      method: GET
      url: "https://hn.algolia.com/api/v1/search?query={{ input.q | urlencode }}&tags={{ input.type | urlencode }}"
assertions:
  page_ok: [{ status: 200, jsonpath_exists: "$.hits" }]
  empty: [{ status: 200, body_matches: '"nbHits":\s*0\b' }]
  rate_limited: [{ status: 429 }]
extract:
  items:
    type: records
    container_jsonpath: ["$.hits[*]"]
    fields:
      title: { jsonpaths: ["$.title"], required: true, transform: [html_text] }
      points: { jsonpaths: ["$.points"], default: 0 }
output:
  results: { from: items }
  captured_at: { meta: captured_at }

• steps fetch the data. The Light engine runs an HTTP request, optionally followed by paginate or follow. The browser engines run navigate, click, fill, scroll, wait_for_selector, dismiss_consent and similar steps.
• assertions decide what the response means: a real answer, an empty result, not found, blocked or rate-limited. That decision is the outcome you are billed on.
• extract reads fields with CSS, XPath or JSONPath. Each locator is a fallback chain.
• output maps the extracted data onto the contract's output schema.

Inputs are templated only into URLs, request bodies and form values, never into selectors, scripts or hosts. Each recipe also ships with golden test cases, which are replayed offline against captured pages before every release and run against the live site every 15 minutes as canaries.

Errors¶