Rate Limits & Quotas¶

ScrapeNest enforces limits at two levels - your organization (plan-wide) and each API key. Together they protect platform stability and your budget. You set the per-key limits when you create or update a key.

Organization limits¶

Two limits are set by your plan and apply across every key in your organization:

Plan	Max concurrent jobs	Monthly credit pool
Free	2	1,000
Starter	10	150,000
Pro	50	1,000,000
Business	100	4,000,000
Enterprise	Custom	Committed

Concurrency caps how many jobs run at once across your organization; the monthly credit pool is your included usage (see Billing & Usage). Request throughput itself is governed per API key (below), not by a separate org-wide requests-per-minute cap.

Monitoring limits apply to monitors: your plan caps how many monitors you can keep and the minimum interval between runs (Starter 5 / 1 hour, Pro 25 / 15 minutes, Business 100 / 5 minutes, Enterprise unlimited / 60 seconds). Free plans cannot create monitors. A monitor that fires more often than your plan allows is rejected at creation, and each fired run consumes normal credits.

Per-key limits¶

When creating an API key (Console → Developer → API Keys, or POST /v1/keys), you can scope tighter limits to that specific key. This is ideal for isolating a noisy integration or a third party.

Setting	Description
`rate_limit_rpm`	Requests per minute allowed for this key.
`daily_quota`	Maximum requests per day for this key.
`max_concurrent_jobs`	Maximum jobs this key may run simultaneously.
`enforcement_mode`	`block` rejects requests over the limit with `429`; `observe` only meters and logs them (useful for sizing limits before enforcing).

If you do not set them, a new key defaults to 600 rpm and a 50,000/day quota. The effective concurrency for a key is also capped by your plan's org-wide concurrency limit.

curl -X POST "https://api.scrapenest.com/v1/keys" \
  -H "X-API-Key: sn_live_..." \
  -H "Content-Type: application/json" \
  -d '{
    "name": "nightly-batch",
    "scopes": ["jobs.create", "jobs.read"],
    "rate_limit_rpm": 120,
    "daily_quota": 50000,
    "max_concurrent_jobs": 10,
    "enforcement_mode": "block"
  }'

The effective limit on any request is the most restrictive of the org and key limits.

Handling 429s¶

When a request exceeds an enforced limit, the API returns 429 Too Many Requests. Inspect these headers:

X-RateLimit-Limit - the total request limit for the window.
X-RateLimit-Remaining - requests remaining in the current window.
X-RateLimit-Reset - when the window resets.
Retry-After - seconds to wait before retrying (when present).

Retry strategy¶

Back off exponentially on 429, preferring Retry-After/X-RateLimit-Reset over a fixed delay, and always pair retries with an idempotency_token. The Python SDK errors guide has a ready-made retry helper.

Pace large crawls

Most 429s come from bursting. When paginating or crawling, spread submissions and use max_concurrent_jobs to self-limit.