Observability & Reliability¶

ScrapeNest is designed for visibility. We provide the tools you need to monitor the performance and reliability of your scraping fleet.

Job Lifecycle & Reliability¶

ScrapeNest uses Temporal, a robust workflow orchestration engine, to manage the entire lifecycle of every scraping job.

• Deterministic Retries: If a job fails due to a transient network error or target site timeout, Temporal automatically retries it with exponential backoff.
• Workflow Persistence: If a ScrapeNest component fails, the workflow remains active in Temporal and resumes once the system is restored.
• Traceability: Every job can be traced throughout its lifecycle via our internal tracing system (OpenTelemetry).

Monitoring Performance¶

You can track your scraping fleet's performance directly in the Customer Console Dashboard.

Key Metrics¶

• Success Rate: The percentage of jobs that complete with a 200 OK status.
• Median Latency (p50/p90): The time it takes from submission to job completion.
• Error Breakdown: A categorized view of job failures (e.g., http_403, timeout, stealth_blocked).
• Data Throughput: The total volume of bytes downloaded and artifacts generated.

Error Handling & Debugging¶

When a job fails, we provide detailed diagnostic information in the Metadata Artifact of the job.

• Status Code: The raw HTTP status code received from the target.
• Failure Reason: A human-readable description of the error (e.g., browser_crashed, navigation_timeout).
• Console Logs: For Standard and Stealth jobs, we include the browser console logs as a searchable artifact.
• Network HAR: A full HTTP Archive (HAR) of the browser's network requests for deep debugging.

Real-Time Notifications¶

Don't poll; get notified. Use Webhooks to receive real-time events:

• job.completed — a job reached a terminal state. The payload's status is succeeded or failed (with a failure_reason).
• artifact.ready — an artifact is uploaded and ready, with a presigned download_url.

See the Events Reference for every event and payload, and Webhooks end-to-end for a complete integration.