MCP Server¶
The ScrapeNest MCP server gives AI assistants and agents the ability to scrape, extract, screenshot, and monitor web pages through your ScrapeNest account, and to run ready-made recipes that return schema-validated JSON for well-known sites. It returns clean Markdown instead of raw HTML, so results fit in context windows without burning tokens on navigation, scripts, and ads.
The hosted server is at https://mcp.scrapenest.com/mcp and speaks the Model Context Protocol over HTTP. You do not need to install a package or configure an API key for this connection: add the URL to your client and sign in with your ScrapeNest account.
Connect¶
The server lives at one URL:
You need a ScrapeNest account. For the hosted connection, there is no package to install, no API key to create, and no scopes to pick. Your account's existing roles and scopes are what the server enforces, and jobs bill to your own organization.
Claude Code¶
Then run /mcp inside Claude Code and pick scrapenest to authenticate. Your browser opens, you sign in, and the tools appear.
Claude Desktop and Claude on the web¶
- Open Settings > Connectors.
- Click Add custom connector.
- Paste
https://mcp.scrapenest.com/mcpand confirm. - Click Connect. Your browser opens the ScrapeNest sign-in page.
- Sign in and approve the connection.
The ScrapeNest tools appear in the tool picker.
Other MCP clients¶
The hosted server works with MCP-capable CLI agents and web agents. Add https://mcp.scrapenest.com/mcp through your agent's MCP or connector settings, then sign in with your ScrapeNest account. Claude, Codex, OpenCode, Cursor, Windsurf, Cline, Mistral Le Chat, DeepSeek, and Z.ai all use the same hosted endpoint.
To inspect the authentication metadata the server advertises:
The connection uses remote MCP over HTTP with OAuth 2.1. If your agent is not listed above, add the same endpoint through its MCP or connector settings. If you need help with a client-specific setup, contact support and tell us which agent you use.
Signing in again¶
Access tokens expire, and your client will re-run the sign-in when that happens. If tools stop working after a while, re-authenticate: in Claude Code run /mcp, elsewhere reconnect the ScrapeNest connector.
Your first scrape¶
Once connected, ask your assistant:
Scrape https://example.com and summarize the main content.
The assistant calls the scrape tool, which submits a job to ScrapeNest, waits for the result, and returns the page as clean Markdown. By default it uses the Light engine (1 credit). For JavaScript-heavy pages, ask for Standard or Stealth when your plan includes it:
Scrape https://example.com/dashboard using the standard engine.
Tools¶
Scraping¶
| Tool | What it does |
|---|---|
scrape(url, engine) |
Fetch a page and return clean Markdown. |
extract(url, selectors, engine) |
Pull fields with CSS selectors or XPath. With a container, it returns aligned records. |
crawl(urls, engine) |
Scrape many pages concurrently. Markdown for each, for RAG ingestion. |
screenshot(url, engine, full_page) |
Capture a PNG screenshot. |
get_job(job_id) |
Check status, outcome, and credit cost of a job. |
get_artifact(job_id, artifact_type) |
Re-download a screenshot or HTML from a previous job. |
list_jobs(status, engine, tag, ...) |
Filter your job history by status, engine, tag, URL, or date range. |
usage(from_date, to_date) |
Throughput stats, success rate, and job counts for a time window. |
compare(url_a, url_b, engine) |
Scrape two URLs and return a unified diff of their Markdown. |
resubmit(job_id, engine) |
Retry a previous job by ID. Optionally override the engine. |
Monitoring¶
| Tool | What it does |
|---|---|
create_monitor(name, cron, target_url, ...) |
Create a recurring scrape with optional change detection. |
list_monitors(status) |
List all monitors, optionally filtered by status. |
get_monitor(monitor_id) |
Full config, detection block, and schedule. |
pause_monitor(monitor_id) |
Pause a monitor without deleting it. |
resume_monitor(monitor_id) |
Resume a paused monitor. |
delete_monitor(monitor_id) |
Remove a monitor permanently. |
monitor_changes(monitor_id, limit) |
List recently detected changes. |
get_change_diff(monitor_id, change_id) |
Fetch the actual unified diff for a detected change. |
monitor_runs(monitor_id, limit) |
List fire history: minted, skipped, or errored. |
update_monitor(monitor_id, ...) |
Replace a monitor's full configuration. Read it with get_monitor first. |
Recipes (beta - ready-made scrapers returning schema-validated JSON; only definitive answers are billed):
| Tool | Description |
|---|---|
list_recipes() |
The catalog with status, health and billing unit. |
get_recipe(slug) |
Input and output JSON Schemas, examples, engine and credits per unit. |
run_recipe(slug, input, contract, wait) |
Run a recipe and wait for its structured result, or return the job id with wait=false. |
get_recipe_result(job_id) |
The structured result of an earlier run. |
Engines¶
Every scraping tool accepts an engine parameter. Engine access depends on your plan:
| Engine | Cost | Best for |
|---|---|---|
| light (default) | 1 credit | APIs, static HTML, high volume |
| standard | 5 credits | JavaScript-heavy pages, SPAs, interactions |
| stealth | 30 credits | Sites with bot defenses (Cloudflare, DataDome) |
Browser options (Standard and Stealth only)¶
dismiss_cookies=True- dismiss cookie/consent banners automatically. Prefer this over writing click actions.wait_until/navigation_timeout_ms- navigation control. Use"domcontentloaded"with a higher timeout for tracker-heavy sites.locale(e.g."fr-FR") - pin language and consent variant.actions- interactions to run before capture:click,fill,wait_for_selector,wait_for_timeout,evaluate.
Stealth engine restrictions
The Stealth engine takes its viewport and locale from the browser fingerprint. Passing viewport_width, viewport_height, or locale to a Stealth scrape raises an error rather than weakening the fingerprint.
Examples¶
Extract structured data¶
Extract the product name, price, and rating from https://example.com/product/42 using CSS selectors .product-title, .price, .rating
The assistant calls extract with your selectors and returns structured JSON.
Use a recipe¶
List the open bug issues on cli/cli with their labels.
For well-known sites, the assistant calls list_recipes, reads the input schema with get_recipe("github-issues"), then calls run_recipe. The result is JSON validated against the recipe's output contract, with no selectors involved, and runs that get blocked or break are not billed.
Set up a price monitor¶
Monitoring starts on the Starter plan. On Free, use scrape or compare for one-off checks.
Create a monitor called "competitor-pricing" that checks https://competitor.com/pricing every hour using the standard engine, with change detection on the .price-table selector.
The assistant calls create_monitor with a cron expression (0 * * * *), the URL, engine, and a detection block targeting your selector.
Check for changes¶
What changes has the competitor-pricing monitor detected recently?
The assistant calls monitor_changes and returns the diffs.
Compare two pages¶
Compare the pricing pages of competitor.com and our-product.com.
The assistant calls compare with both URLs, scrapes them in parallel, and returns a unified diff showing the differences.
Check your usage¶
How many jobs did I run this week and what's the success rate?
The assistant calls usage with a date range and returns totals, success rate, and average duration.
Find failed jobs¶
Show me all failed stealth jobs from the last 24 hours.
The assistant calls list_jobs with status="failed", engine="stealth", and a date range.
See what changed¶
Show me the actual diff for the latest change on my competitor-pricing monitor.
The assistant calls monitor_changes to find the change ID, then get_change_diff to fetch the unified diff.
Retry a failed job¶
That last scrape failed. Resubmit it with the stealth engine instead.
The assistant calls resubmit with the job ID and engine="stealth" to retry on a harder engine.
Check monitor health¶
How has my competitor-pricing monitor been firing? Any skipped runs?
The assistant calls monitor_runs and shows you the fire history with statuses.
Change a monitor's schedule¶
Change competitor-pricing to run every 30 minutes instead of hourly.
The assistant calls get_monitor to read the current config, then update_monitor with the cron changed to */30 * * * *.
Crawl for RAG¶
Crawl these 5 documentation pages and give me a summary of each: [urls]
The assistant calls crawl with all URLs and gets back Markdown for each page.
Limits and defaults¶
These are fixed by the hosted server. There is nothing to configure on your side.
| Setting | Value | Meaning |
|---|---|---|
| Default engine | light |
Used when a tool call does not name an engine. |
| Job wait | 90 seconds | How long a tool waits for a job before returning. |
| Markdown cap | 40,000 characters | Per-result truncation, to protect your context window. |
| Data residency | EU only | Identical to the API - MCP jobs are normal ScrapeNest jobs. |
Troubleshooting¶
Sign-in loop, or tools disappear after a while¶
Your access token expired. Re-authenticate: in Claude Code run /mcp and select scrapenest; in Claude Desktop or on the web, reconnect the ScrapeNest connector.
"This account is not linked to a workspace"¶
You signed in with a ScrapeNest account that has no organization yet. The server creates one for you on the free plan on first use, but the org only appears in a freshly issued token - reconnect the server once and try again.
Tool calls time out¶
The server waits 90 seconds for a job. Heavy JavaScript rendering or a Stealth job on a slow target can exceed that. Submit the work and poll instead: the tool returns a job id, and get_job reports status and outcome once it finishes.
Markdown is truncated¶
Results are capped at 40,000 characters to protect your context window. Use extract with CSS selectors or XPath to pull only the fields you need, which returns structured JSON instead of a wall of text.
401 Unauthorized¶
Your client is calling the server without a token, or with an expired one. Reconnect the server so it runs the OAuth flow again. If your client does not support remote MCP servers with OAuth, it cannot connect - see Other MCP clients above.
Next steps¶
- Python SDK - use the API directly from Python code.
- Monitoring - deeper dive on monitors and change detection.
- Worker Tiers - when to use Light vs Standard vs Stealth.
- Billing - how credits work.