Monitoring¶

A monitor runs a scrape on a recurring cron cadence and, optionally, watches the page for changes. ScrapeNest fires the monitor, creates a normal job each time, and records the outcome so you have a full run history. Add a change-detection block to also get notified (webhook + email) when the watched content changes, with a before/after diff.

Monitors run on a durable scheduling engine, so runs happen exactly once per window with predictable catch-up behavior - not a best-effort timer.

How billing works¶

A monitor run is a normal job. It consumes credits by engine weight (Light 1, Standard 5, Stealth 30) exactly like a manual submission, and it is billed only on a successful, delivered result. There is no separate charge for running a monitor.

If a run cannot be created because you are out of credits, or because the engine is no longer included in your plan, the run is skipped - it costs nothing and appears in the run history as skipped_quota or skipped_not_allowed. You can subscribe to the schedule.run_skipped webhook to be notified.

Plan limits¶

Monitoring is a paid capability. Your plan controls whether you can create monitors, how many you can keep, and the minimum interval between runs:

The minimum interval is a guardrail: a monitor that fires more often than your plan allows is rejected at creation time, not silently throttled. The engine you choose must also be included in your plan (for example, Stealth requires Pro or higher).

Create a monitor¶

from scrapenest import ScrapeNestClient

client = ScrapeNestClient(api_key="sn_live_...", base_url="https://api.scrapenest.com")

monitor = client.monitors.create(
    name="hourly-homepage",
    cron="0 * * * *",            # top of every hour
    timezone="Europe/Paris",     # IANA timezone
    job_type="light",
    target_url="https://example.com",
)
print(monitor.id, monitor.status, monitor.next_run_at)

curl -X POST "https://api.scrapenest.com/v1/monitors" \
  -H "X-API-Key: sn_live_..." \
  -H "Content-Type: application/json" \
  -d '{
    "name": "hourly-homepage",
    "cron": "0 * * * *",
    "timezone": "Europe/Paris",
    "job_type": "light",
    "target_url": "https://example.com"
  }'

To watch the page for changes, add a detection block to the request - see Change detection.

Cron and timezone¶

cron is a standard 5-field expression (minute hour day-of-month month day-of-week). It is evaluated in the timezone you provide (any IANA name, e.g. Europe/Paris or UTC), including daylight-saving transitions. A few examples:

Overlap policy¶

If a run is still executing when the next fire time arrives, overlap_policy decides what happens:

• skip (default) - do not start the next run until the current one finishes.
• buffer_one - queue at most one pending run.
• allow - start the next run regardless.

Manage monitors¶

# List and iterate
for m in client.monitors.iter():
    print(m.name, m.cron, m.status)

# Pause and resume (keeps the monitor, stops firing)
client.monitors.pause(monitor.id)
client.monitors.resume(monitor.id)

# Update the definition
client.monitors.update(
    monitor.id,
    name="hourly-homepage",
    cron="0 */2 * * *",
    job_type="light",
    target_url="https://example.com",
)

# Delete
client.monitors.delete(monitor.id)

If you downgrade your plan below what a monitor requires (too many monitors, an engine your new plan does not include, or an interval that is now too frequent), the affected monitors are automatically paused and you receive a notification. Re-enable them after upgrading, or edit them to fit the new plan.

Run history¶

Every fire is recorded, whether it created a job or was skipped:

runs = client.monitors.runs(monitor.id, limit=20)
for run in runs.items:
    print(run.fired_at, run.status, run.job_id)

Run statuses:

• minted - a job was created (job_id points to it; follow it with the Jobs API or webhooks).
• delivered - the job ran and returned content that was checked for changes.
• blocked - the target denied or challenged the scrape (for example anti-bot, HTTP 403), so there was nothing to check. The detail names the signal and status code.
• failed - the scrape errored and returned no content.
• skipped_quota - out of credits this period; nothing was charged.
• skipped_not_allowed - the engine is not included in your current plan.
• error - an unexpected failure creating the run.

Monitor health¶

Each monitor reports a detection.health.status so you can tell at a glance whether it is working:

• ok - the last run delivered content and was compared. Healthy.
• selector_missing - the page loaded but your selector no longer matches. Update the selector.
• blocked - recent runs were blocked by the target site. Nothing could be checked for changes.
• failed - recent scrapes errored.
• null - no run has been evaluated yet (pending the first run).

Blocked and failed runs are never charged and never count as a change. To stop a monitor from running - and reporting - checks that cannot succeed, a monitor whose scrapes do not deliver for 3 consecutive runs is paused automatically and you receive a notification (webhook and email). Try a higher engine tier (for example stealth) or adjust the target, then resume the monitor. Resuming clears the streak.

In the console¶

The Monitoring section of the console lists your monitors, lets you create and edit them, pause or resume with one click, and drill into the run history and detected changes for each one.