Artifacts & Extraction¶

Every scraping job produces one or more artifacts - the outputs you actually consume. This page lists the artifact types, the manifest that describes them, how to download them, and how to pull structured data out of a page with extraction hooks.

Artifact types¶

Which artifacts a job produces depends on its artifact_options and tier.

Response metadata - HTTP status, response headers, and timing - is not a separate artifact. It lives in the manifest (see The manifest).

The manifest¶

Each job emits a manifest: the forensics record for the run. It describes every artifact produced (ids, types, sizes, integrity hashes) and, at the top level, captures the job's outcome, the target's HTTP response, and timing.

{
  "manifest_version": "1.0.0",
  "job_id": "3d7d1e6e-2b8e-47c2-8bbd-9c2a1a3f9b10",
  "engine": "stealth",
  "target_url": "https://example.com",
  "final_url": "https://example.com/",
  "status": "completed",
  "outcome": "success",
  "response": {
    "http_status": 200,
    "headers": { "content-type": "text/html; charset=utf-8" }
  },
  "timing": {
    "started_at": "2026-06-06T12:00:03Z",
    "finished_at": "2026-06-06T12:00:05Z",
    "duration_ms": 1250
  },
  "credits": { "weight": 30, "charged": true, "amount": 30 },
  "artifacts": [
    {
      "artifact_id": "7d6cbb9f-2f31-4e9c-8c3a-5e9b0f2a14e1",
      "artifact_type": "html",
      "mime_type": "text/html",
      "checksum_sha256": "…",
      "byte_size": 18342,
      "created_at": "2026-06-06T12:00:05Z"
    }
  ]
}

The same artifact list is returned inline on the job object (GET /v1/jobs/{jobId}) and, in the SDK, as Job.artifacts.

Status vs. outcome¶

The manifest separates two things that are easy to conflate:

A blocked job has status: "completed" (it ran) but outcome: "blocked", plus a structured block:

{
  "status": "completed",
  "outcome": "blocked",
  "block": { "blocked": true, "signal": "anti_bot_challenge", "http_status": 403 },
  "response": { "http_status": 403, "headers": { … } },
  "credits": { "weight": 30, "charged": false, "amount": 0 }
}

Response headers in the manifest are sanitized: cookies, set-cookie, authorization, and similar credential-bearing headers are redacted.

Downloading artifacts¶

Artifacts are never public. You exchange an artifact_id for a short-lived presigned URL, then fetch the bytes from that URL.

job = client.jobs.get("JOB_ID")
art = next(a for a in job.artifacts if a.artifact_type == "html")

html = client.artifacts.download_text(art.artifact_id)

# Exchange the id for a presigned URL (optionally set ttl_seconds, 60–86400)
curl "https://api.scrapenest.com/v1/artifacts/ARTIFACT_ID/download?ttl_seconds=600" \
  -H "X-API-Key: sn_live_..."
# → {"download_url": "https://...", "expires_at": "2026-06-06T12:15:05Z"}

curl -L "PRESIGNED_DOWNLOAD_URL" -o result.html

Integrity: every artifact carries a checksum_sha256. Verify it after download if you need tamper-evidence.

Extraction hooks¶

Instead of downloading HTML and parsing it yourself, let ScrapeNest extract structured data during the job. Add extraction.hooks to the payload and set artifact_options.include_extraction: true. Results come back as an extraction artifact.

A hook has an id, a type, and type-specific fields:

"extraction": {
  "hooks": [
    {"hook_id": "title", "type": "css", "selector": "h1"},
    {"hook_id": "image", "type": "css", "selector": "img.hero", "attribute": "src"},
    {"hook_id": "skus", "type": "regex", "pattern": "SKU-(\\d+)", "group": 1, "all_matches": true},
    {"hook_id": "price", "type": "jsonpath", "path": "$.product.price"}
  ]
}

Common fields (all hook types)¶

Every hook also accepts these optional fields, regardless of type:

Scalar hook types (css, xpath, regex, jsonpath) also take all_matches, which defaults to true: a hook returns every match unless you set it to false to keep only the first.

Hook types¶

{"hook_id": "links", "type": "css", "selector": "a.product", "attribute": "href"}

{"hook_id": "qty", "type": "xpath", "path": "//div[contains(@class,'card')][.//h4[normalize-space()='BTC']]//span[@class='qty']"}

{"hook_id": "ids", "type": "regex", "pattern": "id=(\\d+)", "group": 1}

{"hook_id": "names", "type": "jsonpath", "path": "$.data[*].name"}

{
  "hook_id": "products",
  "type": "records",
  "container": ".product-card",
  "fields": {
    "name":  {"selector": "h3 a"},
    "url":   {"selector": "h3 a", "attribute": "href"},
    "price": {"selector": ".price", "transform": ["number"]},
    "stock": {"selector": ".stock", "default": "unknown"}
  }
}

{
  "hook_id": "stories",
  "type": "records",
  "container_path": "//tr[@class='athing']",
  "fields": {
    "title": {"path": ".//span[@class='titleline']/a"},
    "score": {"path": "following-sibling::tr[1]//span[@class='score']", "transform": ["number"]}
  }
}

Transforms¶

transform takes a list of names, applied in order to each extracted value. A transform that cannot be applied yields null rather than failing the hook, so the value falls through to the field's default (records) or comes back as null.

Result format¶

The extraction artifact contains one entry per hook:

{
  "hooks": [
    {
      "hook_id": "title",
      "type": "css",
      "status": "succeeded",
      "duration_ms": 4,
      "result": ["Example Domain"],
      "total_matched": 1,
      "truncated": false
    },
    {
      "hook_id": "products",
      "type": "records",
      "status": "succeeded",
      "duration_ms": 12,
      "result": [
        {"name": "Box of Chocolate Candy", "price": 9.99},
        {"name": "Dark Energy Potion", "price": null}
      ],
      "total_matched": 2,
      "truncated": false,
      "skipped_records": 0
    }
  ]
}

Limits¶

Hooks are validated when the job is submitted: a malformed hook returns 422 rather than costing a job. A hook that runs but matches nothing never fails the job - check each hook's status in the result.

Next steps¶

• Extract structured data - extraction hooks in a full recipe.
• Job Parameters - artifact_options and extraction in context.
• Data Retention & Holds - how long artifacts live.