Skip to content

Work With Protected Targets

Goal: tune a job for targets that block default HTTP clients or standard browser automation.

ScrapeNest's Stealth tier combines a hardened browser profile, fingerprint controls, helper extensions, and managed proxy routing. Most tuning comes down to tier selection, visitor profile, timing, and diagnostics. See Anti-Blocking for the underlying controls.

Escalate the tier first

Pick the lowest tier that works - it's the cheapest and fastest:

  1. light - try first for APIs and static pages.
  2. standard - when the site needs JavaScript rendering.
  3. stealth - when you hit blocks, challenges, or stealth_blocked failures.

Minimal stealth example

from scrapenest import ScrapeNestClient

client = ScrapeNestClient(api_key="sn_live_...", base_url="https://api.scrapenest.com")

result = client.scrape(
    job_type="stealth",
    target_url="https://protected.example.com",
    os_name="macos",
    browser_extensions=["ublock", "isdcac"],
    timeout=60,
)

curl -X POST "https://api.scrapenest.com/v1/jobs" \
  -H "X-API-Key: sn_live_..." \
  -H "Content-Type: application/json" \
  -d '{
    "job_type": "stealth",
    "target_url": "https://protected.example.com",
    "os_name": "macos",
    "browser_extensions": ["ublock", "isdcac"]
  }'

Tuning checklist

Match the target's audience - set an OS profile, locale, and timezone consistent with the expected visitor:

client.scrape(
    job_type="stealth",
    target_url="https://protected.example.com",
    os_name="windows",
    locale="fr-FR",
    timezone_id="Europe/Paris",
)

Block noisy resources - fewer requests means a smaller detection surface and faster loads:

artifact_options={"include_html": True, "block_images": True, "block_media": True}

Handle overlays - the isdcac extension handles many consent banners; for the rest, add an action:

actions=[{"type": "click", "selector": "#onetrust-accept-btn-handler"}]

Give it time - protected sites often run interstitial challenges. Allow a longer wait:

client.scrape(job_type="stealth", target_url=url, wait_until="networkidle", timeout=90)

Built-in controls

  • Managed proxy rotation across residential and datacenter IPs.
  • Browser fingerprint controls - Canvas, WebGL, AudioContext, WebRTC.
  • TLS/HTTP impersonation so handshakes match modern browser traffic.
  • Session stickiness so all requests in a job share one IP.

To route through your own proxy instead of the managed pool, pass a proxy object (Stealth tier). ScrapeNest chains egress through your proxy rather than bypassing Proxy Guard. See Custom egress.

When a job still fails

A blocked job ends with status="failed" and failure_reason="stealth_blocked". Then:

  • Confirm you're on stealth, not standard.
  • Vary os_name and add wait_until="networkidle" with a higher timeout.
  • Inspect the job manifest (and HAR/console if enabled) for the block reason - see Observability.
  • Pace your requests; bursts against one domain invite blocks. See Rate Limits.

See also