MCP Server¶

The ScrapeNest MCP server gives AI assistants and agents the ability to scrape, extract, screenshot, and monitor web pages through your ScrapeNest account. It returns clean Markdown instead of raw HTML, so results fit in context windows without burning tokens on navigation, scripts, and ads.

It works with any client that speaks the Model Context Protocol: Claude Desktop, Claude Code, Cursor, Windsurf, Cline, or your own agent.

Install¶

pip install scrapenest-mcp

The package installs as scrapenest-mcp; the entry point is scrapenest-mcp. Requires Python 3.10+.

Prerequisites¶

You need a ScrapeNest API key. If you don't have one:

Open the Customer Console and go to Settings > API Keys.
Click Create New Key.
Select the scopes the key needs (at minimum jobs.create, jobs.read, and artifacts.read; add schedules.* for monitoring tools).
Copy the key now - it cannot be retrieved again.

Connect to Claude Desktop¶

Add the server to your Claude Desktop configuration file:

macOSWindows

File: ~/Library/Application Support/Claude/claude_desktop_config.json

{
  "mcpServers": {
    "scrapenest": {
      "command": "scrapenest-mcp",
      "env": {
        "SCRAPENEST_API_KEY": "sn_live_..."
      }
    }
  }
}

File: %APPDATA%\Claude\claude_desktop_config.json

{
  "mcpServers": {
    "scrapenest": {
      "command": "scrapenest-mcp",
      "env": {
        "SCRAPENEST_API_KEY": "sn_live_..."
      }
    }
  }
}

Restart Claude Desktop after saving. The ScrapeNest tools appear in the tool picker.

Connect to Claude Code¶

Add the server to your project or user settings:

claude mcp add scrapenest -- scrapenest-mcp

Then set the API key in your environment:

export SCRAPENEST_API_KEY="sn_live_..."

Or add it to your .claude/settings.json:

{
  "env": {
    "SCRAPENEST_API_KEY": "sn_live_..."
  }
}

Connect to Cursor / Windsurf¶

Both editors support MCP servers through their settings. Add the same configuration block:

{
  "mcpServers": {
    "scrapenest": {
      "command": "scrapenest-mcp",
      "env": {
        "SCRAPENEST_API_KEY": "sn_live_..."
      }
    }
  }
}

In Cursor, go to Settings > MCP Servers. In Windsurf, go to Settings > Cascade > MCP.

Your first scrape¶

Once connected, ask your assistant:

Scrape https://example.com and summarize the main content.

The assistant calls the scrape tool, which submits a job to ScrapeNest, waits for the result, and returns the page as clean Markdown. By default it uses the Light engine (1 credit). For JavaScript-heavy pages, ask for Standard or Stealth:

Scrape https://example.com/dashboard using the standard engine.

Tools¶

Scraping¶

Tool	What it does
`scrape(url, engine)`	Fetch a page and return clean Markdown.
`extract(url, selectors, engine)`	Pull specific fields with CSS selectors. Returns `{name: [values]}`.
`crawl(urls, engine)`	Scrape many pages concurrently. Markdown for each, for RAG ingestion.
`screenshot(url, engine, full_page)`	Capture a PNG screenshot.
`get_job(job_id)`	Check status, outcome, and credit cost of a job.
`get_artifact(job_id, artifact_type)`	Re-download a screenshot or HTML from a previous job.
`list_jobs(status, engine, tag, ...)`	Filter your job history by status, engine, tag, URL, or date range.
`usage(from_date, to_date)`	Throughput stats, success rate, and job counts for a time window.
`compare(url_a, url_b, engine)`	Scrape two URLs and return a unified diff of their Markdown.
`resubmit(job_id, engine)`	Retry a previous job by ID. Optionally override the engine.

Monitoring¶

Tool	What it does
`create_monitor(name, cron, target_url, ...)`	Create a recurring scrape with optional change detection.
`list_monitors(status)`	List all monitors, optionally filtered by status.
`get_monitor(monitor_id)`	Full config, detection block, and schedule.
`pause_monitor(monitor_id)`	Pause a monitor without deleting it.
`resume_monitor(monitor_id)`	Resume a paused monitor.
`delete_monitor(monitor_id)`	Remove a monitor permanently.
`monitor_changes(monitor_id, limit)`	List recently detected changes.
`get_change_diff(monitor_id, change_id)`	Fetch the actual unified diff for a detected change.
`monitor_runs(monitor_id, limit)`	List fire history: minted, skipped, or errored.
`update_monitor(monitor_id, ...)`	Replace a monitor's full configuration. Read it with `get_monitor` first.

Engines¶

Every scraping tool accepts an engine parameter:

Engine	Cost	Best for
light (default)	1 credit	APIs, static HTML, high volume
standard	5 credits	JavaScript-heavy pages, SPAs, interactions
stealth	30 credits	Sites with bot defenses (Cloudflare, DataDome)

Browser options (Standard and Stealth only)¶

dismiss_cookies=True - dismiss cookie/consent banners automatically. Prefer this over writing click actions.
wait_until / navigation_timeout_ms - navigation control. Use "domcontentloaded" with a higher timeout for tracker-heavy sites.
locale (e.g. "fr-FR") - pin language and consent variant.
actions - interactions to run before capture: click, fill, wait_for_selector, wait_for_timeout, evaluate.

Stealth engine restrictions

The Stealth engine takes its viewport and locale from the browser fingerprint. Passing viewport_width, viewport_height, or locale to a Stealth scrape raises an error rather than weakening the fingerprint.

Examples¶

Extract structured data¶

Extract the product name, price, and rating from https://example.com/product/42 using CSS selectors .product-title, .price, .rating

The assistant calls extract with your selectors and returns structured JSON.

Set up a price monitor¶

Create a monitor called "competitor-pricing" that checks https://competitor.com/pricing every hour using the standard engine, with change detection on the .price-table selector.

The assistant calls create_monitor with a cron expression (0 * * * *), the URL, engine, and a detection block targeting your selector.

Check for changes¶

What changes has the competitor-pricing monitor detected recently?

The assistant calls monitor_changes and returns the diffs.

Compare two pages¶

Compare the pricing pages of competitor.com and our-product.com.

The assistant calls compare with both URLs, scrapes them in parallel, and returns a unified diff showing the differences.

Check your usage¶

How many jobs did I run this week and what's the success rate?

The assistant calls usage with a date range and returns totals, success rate, and average duration.

Find failed jobs¶

Show me all failed stealth jobs from the last 24 hours.

The assistant calls list_jobs with status="failed", engine="stealth", and a date range.

See what changed¶

Show me the actual diff for the latest change on my competitor-pricing monitor.

The assistant calls monitor_changes to find the change ID, then get_change_diff to fetch the unified diff.

Retry a failed job¶

That last scrape failed. Resubmit it with the stealth engine instead.

The assistant calls resubmit with the job ID and engine="stealth" to retry on a harder engine.

Check monitor health¶

How has my competitor-pricing monitor been firing? Any skipped runs?

The assistant calls monitor_runs and shows you the fire history with statuses.

Change a monitor's schedule¶

Change competitor-pricing to run every 30 minutes instead of hourly.

The assistant calls get_monitor to read the current config, then update_monitor with the cron changed to */30 * * * *.

Crawl for RAG¶

Crawl these 5 documentation pages and give me a summary of each: [urls]

The assistant calls crawl with all URLs and gets back Markdown for each page.

Configuration¶

These environment variables configure the MCP server:

Variable	Default	Purpose
`SCRAPENEST_API_KEY`	(required)	Your ScrapeNest API key
`SCRAPENEST_BASE_URL`	`https://api.scrapenest.com`	API base URL
`SCRAPENEST_DEFAULT_ENGINE`	`light`	Default engine when none is specified
`SCRAPENEST_SYNC_TIMEOUT`	`90`	Seconds to wait for a job to finish
`SCRAPENEST_MAX_MARKDOWN_CHARS`	`40000`	Truncation limit for returned Markdown
`SCRAPENEST_VERIFY_TLS`	`1`	Set `0` only for self-signed local stacks

Troubleshooting¶

"API key not set"¶

Set SCRAPENEST_API_KEY in the env block of your MCP config, or export it in your shell before starting the MCP client.

Tool calls time out¶

Increase SCRAPENEST_SYNC_TIMEOUT. The default 90 seconds is enough for most pages, but heavy JavaScript rendering or Stealth jobs on slow targets may take longer.

Markdown is truncated¶

The server caps Markdown output at 40,000 characters by default to protect context window size. Increase SCRAPENEST_MAX_MARKDOWN_CHARS if you need the full content, or use extract with CSS selectors to pull only the fields you need.

Connection refused / server not found¶

Make sure scrapenest-mcp is installed in the Python environment your MCP client uses. Run which scrapenest-mcp to verify. If you installed it in a virtual environment, either activate it before starting the client or use the full path in your config:

{
  "mcpServers": {
    "scrapenest": {
      "command": "/path/to/venv/bin/scrapenest-mcp",
      "env": { "SCRAPENEST_API_KEY": "sn_live_..." }
    }
  }
}

Next steps¶

Python SDK - use the API directly from Python code.
Monitoring - deeper dive on monitors and change detection.
Worker Tiers - when to use Light vs Standard vs Stealth.
Billing - how credits work.