API reference
Batch Web Scraping API
Scrape up to 25 URLs in one background job and get a job id back immediately; individual URL failures are reported per item and never fail the batch.
Endpoint
GET/v1/web/batch-scrapelive · proven3 creditsParameters
Query parameters
| Name | Required | Description | Example |
|---|---|---|---|
urls | yes | Comma-separated list or JSON array of URLs to scrape (max 25 per job) | https://example.com,https://www.iana.org |
formats | no | Comma-separated subset of markdown,text,links (default markdown) | markdown |
only_main_content | no | Render only the main content region of each page (default true) | true |
dry_run | no | Read a zero-credit estimate without fetching sources, running AI, or reserving credits. Cache status is a snapshot, not a guarantee at execution. | 1 |
Examples
Make the request
curl "https://www.monocrawl.com/v1/web/batch-scrape?urls=https%3A%2F%2Fexample.com%2Chttps%3A%2F%2Fwww.iana.org" \ -H "x-api-key: mn_your_key_here"
TypeScript
const key = process.env.MONOCRAWL_API_KEY;
if (!key) throw new Error('Set MONOCRAWL_API_KEY on your server.');
const res = await fetch(
"https://www.monocrawl.com/v1/web/batch-scrape?urls=https%3A%2F%2Fexample.com%2Chttps%3A%2F%2Fwww.iana.org",
{ headers: { "x-api-key": key } },
);
const body = await res.json();
if (!body.success) {
// one error shape for every endpoint — see /docs/errors
throw new Error(`${body.error.type}: ${body.error.message}`);
}
console.log(body.data, "credits left:", body.credits_remaining);Python
import os
import requests
res = requests.get(
"https://www.monocrawl.com/v1/web/batch-scrape?urls=https%3A%2F%2Fexample.com%2Chttps%3A%2F%2Fwww.iana.org",
headers={"x-api-key": os.environ["MONOCRAWL_API_KEY"]},
timeout=60,
)
body = res.json()
if not body["success"]:
# one error shape for every endpoint — see /docs/errors
raise RuntimeError(f"{body['error']['type']}: {body['error']['message']}")
print(body["data"], "credits left:", body["credits_remaining"])Response
Response fields and example
This example is illustrative, not a captured live response. Variable-cost operations may settle a charge different from the list price below. A successful response puts the platform payload in data and reports the exact credits used, remaining balance, request id and cache status beside it.
{
"success": true,
"platform": "web",
"endpoint": "/v1/web/batch-scrape",
"data": {
"id": "job_example",
"status": "queued"
},
"credits_used": 3,
"credits_remaining": 99,
"request_id": "req_…",
"cached": false
}The example is illustrative and shows a documented subset of data. The fields below are optional across supported sources; nullable fields can also be absent. Preserve unknown values and accept additional fields.
Schema basis: reviewed response shaper. See schema coverage and validation limits.
Download the data JSON Schema. Validate response.data, not the whole envelope. A valid shape does not establish that every field or source record was returned.
| Field inside data | Type | Meaning |
|---|---|---|
id | string | null | Exact source identifier; preserve it as a string. |
kind | string | null | kind |
status | string | null | status |
created_at | string | null | created at |
started_at | string | null | started at |
completed_at | string | null | completed at |
poll | string | null | poll |
refund_status | string | null | refund status |
credits_charged | number | null | credits charged; null is unknown. |
credits_refunded | number | null | credits refunded; null is unknown. |
attempts | number | null | attempts; null is unknown. |
progress | object | null | Nested response fields; optional unless explicitly documented. |
result | object | null | Nested response fields; optional unless explicitly documented. |
error | string | null | Failure details if present. |
This response accepts asynchronous work. Poll web/jobs/get for the result and per-page failures.
Failures use the typed error envelope. A confirmed uncharged or refunded failure reports zero; a pending reconciliation can report an unknown charge. Read credits_used and error.details.billing_status, and keep request_id for recovery. Response contract · Error reference