Web Scraping API

Start an async web crawl

Start a breadth-first same-site crawl and get a job id back immediately; poll /v1/web/jobs/{job_id} for the collected pages (url, title, markdown excerpt, status).

Read as Markdown Use with an AI agent

Endpoint

GET/v1/web/crawllive · proven5 credits

Try in Playground

Parameters

Query parameters

Name	Required	Description	Example
`url`	yes	Seed URL to crawl (http/https, public hosts only)	`https://example.com`
`limit`	no	Maximum pages to fetch (default 10, cap 50)	`10`
`max_depth`	no	Link hops from the seed page (default 2, cap 3)	`2`
`include_paths`	no	Comma-separated glob patterns; a discovered path must match at least one. * matches any characters, ^ and $ anchor, everything else is literal	`^/blog/.*`
`exclude_paths`	no	Comma-separated glob patterns; a discovered path matching any of them is skipped	`/tag/,/author/`
`formats`	no	Comma-separated subset of markdown,text,links (default markdown)	`markdown`
`only_main_content`	no	Render only the main content region of each page (default true)	`true`
`dry_run`	no	Read a zero-credit estimate without fetching sources, running AI, or reserving credits. Cache status is a snapshot, not a guarantee at execution.	`1`

Examples

Make the request

curl

curl "https://www.monocrawl.com/v1/web/crawl?url=https%3A%2F%2Fexample.com" \
  -H "x-api-key: mn_your_key_here"

TypeScript

typescript

const key = process.env.MONOCRAWL_API_KEY;
if (!key) throw new Error('Set MONOCRAWL_API_KEY on your server.');
const res = await fetch(
  "https://www.monocrawl.com/v1/web/crawl?url=https%3A%2F%2Fexample.com",
  { headers: { "x-api-key": key } },
);
const body = await res.json();

if (!body.success) {
  // one error shape for every endpoint — see /docs/errors
  throw new Error(`${body.error.type}: ${body.error.message}`);
}

console.log(body.data, "credits left:", body.credits_remaining);

Python

python

import os
import requests

res = requests.get(
    "https://www.monocrawl.com/v1/web/crawl?url=https%3A%2F%2Fexample.com",
    headers={"x-api-key": os.environ["MONOCRAWL_API_KEY"]},
    timeout=60,
)
body = res.json()

if not body["success"]:
    # one error shape for every endpoint — see /docs/errors
    raise RuntimeError(f"{body['error']['type']}: {body['error']['message']}")

print(body["data"], "credits left:", body["credits_remaining"])

Response

Response fields and example

This example is illustrative, not a captured live response. Variable-cost operations may settle a charge different from the list price below. A successful response puts the platform payload in data and reports the exact credits used, remaining balance, request id and cache status beside it.

200 · application/json

{
  "success": true,
  "platform": "web",
  "endpoint": "/v1/web/crawl",
  "data": {
    "id": "job_example",
    "status": "queued"
  },
  "credits_used": 5,
  "credits_remaining": 99,
  "request_id": "req_…",
  "cached": false
}

The example is illustrative and shows a documented subset of data. The fields below are optional across supported sources; nullable fields can also be absent. Preserve unknown values and accept additional fields.

Schema basis: reviewed response shaper. See schema coverage and validation limits.

Download the data JSON Schema. Validate response.data, not the whole envelope. A valid shape does not establish that every field or source record was returned.

Field inside data	Type	Meaning
`id`	`string \| null`	Exact source identifier; preserve it as a string.
`kind`	`string \| null`	kind
`status`	`string \| null`	status
`created_at`	`string \| null`	created at
`started_at`	`string \| null`	started at
`completed_at`	`string \| null`	completed at
`poll`	`string \| null`	poll
`refund_status`	`string \| null`	refund status
`credits_charged`	`number \| null`	credits charged; null is unknown.
`credits_refunded`	`number \| null`	credits refunded; null is unknown.
`attempts`	`number \| null`	attempts; null is unknown.
`progress`	`object \| null`	Nested response fields; optional unless explicitly documented.
`result`	`object \| null`	Nested response fields; optional unless explicitly documented.
`error`	`string \| null`	Failure details if present.

This response accepts asynchronous work. Poll web/jobs/get for the result and per-page failures.

Failures use the typed error envelope. A confirmed uncharged or refunded failure reports zero; a pending reconciliation can report an unknown charge. Read credits_used and error.details.billing_status, and keep request_id for recovery. Response contract · Error reference

Start an async web crawl

Query parameters

Make the request

Response fields and example

More Web Scraping reference

First call in under a minute