Skip to content

Extract structured data

Goal: turn a web page into clean JSON without downloading and parsing HTML yourself.

ScrapeNest runs extraction hooks inside the job. You describe what you want with CSS selectors, XPath, regular expressions, or JSONPath, and the results come back as a JSON artifact.

Minimal example

Extract a page title and all product prices:

from scrapenest import ScrapeNestClient

client = ScrapeNestClient(api_key="sn_live_...", base_url="https://api.scrapenest.com")

result = client.scrape(
    job_type="light",
    target_url="https://example.com/products",
    artifact_options={"include_extraction": True},
    extraction={
        "hooks": [
            {"hook_id": "title", "type": "css", "selector": "h1"},
            {"hook_id": "prices", "type": "css", "selector": ".price", "all_matches": True},
        ]
    },
)
print(result.status)  # "succeeded"

curl -X POST "https://api.scrapenest.com/v1/jobs" \
  -H "X-API-Key: sn_live_..." \
  -H "Content-Type: application/json" \
  -d '{
    "job_type": "light",
    "target_url": "https://example.com/products",
    "artifact_options": {"include_extraction": true},
    "extraction": {"hooks": [
      {"hook_id": "title", "type": "css", "selector": "h1"},
      {"hook_id": "prices", "type": "css", "selector": ".price", "all_matches": true}
    ]}
  }'

Read the results

The job produces a json artifact containing one entry per hook. Download it like any artifact:

import json

job = client.jobs.get(result.job_id)
art = next(a for a in job.artifacts if a.artifact_type == "json")

data = json.loads(client.artifacts.download_text(art.artifact_id))
for hook in data["hooks"]:
    print(hook["hook_id"], hook["status"], hook["result"])

Extracting rows

When the values belong to rows - products, listings, search results - use a records hook rather than one hook per field.

One hook per field returns a separate flat list for each field, and those lists line up only if every field matches exactly once per row. A product without a sale price, a listing with no rating: the shorter list slides out of step, and every later value belongs to the wrong row with nothing in the response to say so.

A records hook resolves each field inside its own container, so rows stay aligned by construction and a missing field is an explicit null:

result = client.scrape(
    job_type="light",
    target_url="https://example.com/products",
    artifact_options={"include_extraction": True},
    extraction={
        "hooks": [
            {
                "hook_id": "products",
                "type": "records",
                "container": ".product-card",
                "fields": {
                    "name":  {"selector": "h3 a"},
                    "url":   {"selector": "h3 a", "attribute": "href"},
                    "price": {"selector": ".price", "transform": ["number"]},
                    "sale":  {"selector": ".sale-price", "transform": ["number"]},
                },
            }
        ]
    },
)

curl -X POST "https://api.scrapenest.com/v1/jobs" \
  -H "X-API-Key: sn_live_..." \
  -H "Content-Type: application/json" \
  -d '{
    "job_type": "light",
    "target_url": "https://example.com/products",
    "artifact_options": {"include_extraction": true},
    "extraction": {"hooks": [{
      "hook_id": "products",
      "type": "records",
      "container": ".product-card",
      "fields": {
        "name":  {"selector": "h3 a"},
        "price": {"selector": ".price", "transform": ["number"]},
        "sale":  {"selector": ".sale-price", "transform": ["number"]}
      }
    }]}
  }'

The result is a list of objects, one per container:

[
  {"name": "Box of Chocolate Candy", "price": 9.99, "sale": 7.99},
  {"name": "Dark Energy Potion",     "price": 4.50, "sale": null}
]

Use required: true on a field to drop rows that lack it (counted as skipped_records), or default to fill them in.

Variations

Grab an attribute, not text - get href/src instead of element text:

{"hook_id": "links", "type": "css", "selector": "a.product", "attribute": "href", "all_matches": true}

Extract from a JSON API - use JSONPath on a light job against an API endpoint:

{"hook_id": "names", "type": "jsonpath", "path": "$.data[*].name"}

Pick a node by its content - XPath matches on what the node says, not where it sits, so it survives re-ordering:

{"hook_id": "qty", "type": "xpath", "path": "//div[contains(@class,'card')][.//h4[normalize-space()='BTC']]//span[@class='qty']"}

Pull IDs with a regex - capture a group from the raw source:

{"hook_id": "skus", "type": "regex", "pattern": "SKU-(\\d+)", "group": 1, "all_matches": true}

Extract after rendering - for JavaScript-built pages, switch to standard/stealth and combine with wait_until:

client.scrape(
    job_type="standard",
    target_url="https://example.com/app",
    wait_until="networkidle",
    artifact_options={"include_extraction": True},
    extraction={"hooks": [{"hook_id": "rows", "type": "css", "selector": "tr.item", "all_matches": True}]},
)

Tips

  • A hook that runs but matches nothing never fails the job - always check each hook's status. A malformed hook is rejected at submit time with a 422, so it never costs a job.
  • Check truncated before treating a result as complete: it is true when the page held more matches than max_results returned, and total_matched says how many there were.
  • Limits: up to 25 hooks per job, 50 results per hook by default (raise with max_results, up to 1000). For larger crawls, paginate.
  • Prefer light for static/server-rendered pages - it's the fastest and cheapest tier.

See also