Observability & Reliability¶

ScrapeNest is designed for visibility. We provide the tools you need to monitor the performance and reliability of your scraping fleet.

Job Lifecycle & Reliability¶

ScrapeNest manages the entire lifecycle of every scraping job on a durable orchestration layer, so job state survives transient failures.

• Deterministic Retries: If a job fails due to a transient network error or target site timeout, it is automatically retried with exponential backoff.
• Durable Execution: If a ScrapeNest component restarts mid-job, the job's orchestration state is preserved and resumes once the system is healthy - accepted jobs are not lost.
• Traceability: Every job is traced throughout its lifecycle for internal diagnostics.

Monitoring Performance¶

You can track your scraping fleet's performance directly in the Customer Console Dashboard.

Key Metrics¶

• Success Rate: The percentage of jobs that complete with a 200 OK status.
• Median Latency (p50/p90): The time it takes from submission to job completion.
• Error Breakdown: A categorized view of job failures (e.g., http_403, timeout, stealth_blocked).
• Data Throughput: The total volume of bytes downloaded and artifacts generated.

Error Handling & Debugging¶

When a job fails, we provide detailed diagnostic information in the job's manifest.

• Status Code: The raw HTTP status code received from the target.
• Failure Reason: A human-readable description of the error (e.g., browser_crashed, navigation_timeout).
• Console Logs: For Standard and Stealth jobs, we include the browser console logs as a searchable artifact.
• Network HAR: A full HTTP Archive (HAR) of the browser's network requests for deep debugging.

Real-Time Notifications¶

Don't poll; get notified. Use Webhooks to receive real-time events:

• job.completed - a job reached a terminal state. The payload's status is succeeded or failed (with a failure_reason).
• artifact.ready - an artifact is uploaded and ready, with a presigned download_url.

See the Events Reference for every event and payload, and Webhooks end-to-end for a complete integration.