Authentication & Access Control¶
ScrapeNest employs a defense-in-depth approach to security, utilizing two distinct authentication layers: Human access via the Customer Console (OIDC/SSO) and machine access via API Keys.
1. Customer Console (Human Access)¶
Console login is implemented using Keycloak with the OIDC Authorization Code flow (with PKCE).
- Identity Provider: Keycloak realm
scrapenest. - Session Security: HttpOnly secure cookies; no tokens are persisted in
localStorage. - MFA Enforcement: Required for all accounts with
org_ownerororg_adminroles.
Role-Based Access Control (RBAC)¶
Your permissions are derived from organization-level role claims. ScrapeNest enforces a deny-by-default policy.
| Role | Scope | Permissions |
|---|---|---|
Owner (org_owner) |
Organization | Full control: billing, team management, security settings, legal holds. |
Admin (org_admin) |
Organization | Management of jobs, API keys, webhooks, and retention policies. |
Member (org_member) |
Organization | Create and monitor scraping jobs; view artifacts. |
Read-Only (org_readonly) |
Organization | View job status and history; cannot submit new jobs. |
2. API Keys (Machine Access)¶
For automated integrations, authenticate using the X-API-Key header.
Key Management & Security¶
- Hashing at Rest: ScrapeNest stores only Argon2id hashes of your keys. We cannot retrieve a lost key.
- Least Privilege Scopes: When creating a key, restrict its capabilities to the minimum required.
jobs:write: Submit new scraping jobs.jobs:read: Check job status and list history.artifacts:read: Download job results.webhooks:read: List webhook deliveries.
- Rotation: We recommend rotating production keys every 90 days. The API supports a grace period where both old and new keys remain valid.
- Revocation: Leaked keys can be revoked instantly via the Console or the API.
3. Network Security: IP Allowlisting¶
ScrapeNest provides two layers of IP restriction to prevent unauthorized API access, even if a key is compromised.
Organization-Level Allowlist¶
Restricts all API access for the organization (Console and Keys) to a set of CIDR blocks. This is the recommended baseline for enterprise customers.
API Key-Level Allowlist¶
Restricts a specific API key to a set of CIDR blocks. Useful for isolating specific integrations (e.g., a worker node in a specific VPC).
4. Troubleshooting¶
403 missing_org_context¶
The request did not include a valid organization ID. When using API keys, ensure the key belongs to the intended organization. When using the Console, ensure an organization is selected in the switcher.
403 forbidden¶
Your current role (e.g., org_readonly) does not have the required permission (e.g., jobs:write) for the requested action.
401 unauthorized¶
The API key is invalid, revoked, or expired.