API reference

Website Crawler API

Start a breadth-first same-site crawl and get a job id back immediately; poll /v1/web/jobs/{job_id} for the collected pages (url, title, markdown excerpt, status).

Read as Markdown Use with an AI agent

Endpoint

GET/v1/web/crawllive · proven5 credits

Try in Playground

Parameters

Query parameters

Useful optional inputs

limit — Bound the returned rows to what your task needs.
A smaller limit does not necessarily reduce source calls or price; inspect the quote.
dry_run — Read a free estimate before fetching sources or scheduling paid work.
Zero credits; cache status and availability are snapshots rather than guarantees.

Name	Required	Description	Example
`url`	yes	Seed URL to crawl (http/https, public hosts only)	`https://example.com`
`limit`	no	Maximum pages to fetch (default 10, cap 50)	`10`
`max_depth`	no	Link hops from the seed page (default 2, cap 3)	`2`
`include_paths`	no	Comma-separated glob patterns; a discovered path must match at least one. * matches any characters, ^ and $ anchor, everything else is literal	`^/blog/.*`
`exclude_paths`	no	Comma-separated glob patterns; a discovered path matching any of them is skipped	`/tag/,/author/`
`formats`	no	Comma-separated subset of markdown,text,links (default markdown) Accepted values (comma-separated): markdown, text, links. Case, spaces, hyphens and underscores are interchangeable.	`markdown`
`only_main_content`	no	Render only the main content region of each page (default true)	`true`
`dry_run`	no	Read a zero-credit estimate without fetching sources, running AI, or reserving credits. Cache status is a snapshot, not a guarantee at execution. Accepted values: 1, 0, true, false. Case, spaces, hyphens and underscores are interchangeable.	`1`

Examples

Make the request

curl

curl "https://www.monocrawl.com/v1/web/crawl?url=https%3A%2F%2Fexample.com" \
  -H "x-api-key: mn_your_key_here"

TypeScript

typescript

const key = process.env.MONOCRAWL_API_KEY;
if (!key) throw new Error('Set MONOCRAWL_API_KEY on your server.');
const res = await fetch(
  "https://www.monocrawl.com/v1/web/crawl?url=https%3A%2F%2Fexample.com",
  { headers: { "x-api-key": key } },
);
const body = await res.json();

if (!body.success) {
  // one error shape for every endpoint — see /docs/errors
  throw new Error(`${body.error.type}: ${body.error.message}`);
}

console.log(body.data, "credits left:", body.credits_remaining);

Python

python

import os
import requests

res = requests.get(
    "https://www.monocrawl.com/v1/web/crawl?url=https%3A%2F%2Fexample.com",
    headers={"x-api-key": os.environ["MONOCRAWL_API_KEY"]},
    timeout=60,
)
body = res.json()

if not body["success"]:
    # one error shape for every endpoint — see /docs/errors
    raise RuntimeError(f"{body['error']['type']}: {body['error']['message']}")

print(body["data"], "credits left:", body["credits_remaining"])

Response

Response fields and example

This example is illustrative, not a captured live response. Variable-cost operations may settle a charge different from the list price below. A successful response puts the platform payload in data and reports the exact credits used, remaining balance, request id and cache status beside it.

200 · application/json

{
  "success": true,
  "platform": "web",
  "endpoint": "/v1/web/crawl",
  "data": {
    "job_id": "job_example",
    "check": "/v1/web/jobs/job_example",
    "status": "queued",
    "kind": "crawl",
    "params": {
      "url": "https://example.com/",
      "limit": 10,
      "max_depth": 2,
      "formats": "markdown",
      "only_main_content": true
    }
  },
  "credits_used": 5,
  "credits_remaining": 99,
  "request_id": "req_…",
  "cached": false
}

The example is illustrative and shows a documented subset of data. The fields below are optional across supported sources; nullable fields can also be absent. Preserve unknown values and accept additional fields.

Schema basis: reviewed response shaper. See schema coverage and validation limits.

Download the data JSON Schema. Validate response.data, not the whole envelope. A valid shape does not establish that every field or source record was returned.

Field inside data	Type	Meaning
`job_id`	`string`	Owner-scoped job identifier. Use check to read its status and result.
`check`	`string`	Relative GET path for this job. Authenticate with the same owning account.
`status`	`string`	Job status at enqueue time; this is not a completed crawl or batch result.
`kind`	`string`	Stored job kind: crawl or batch_scrape.
`created_at`	`string \| null`	Source date or timestamp; absent dates remain unknown.
`credits_charged`	`number \| null`	Credits charged when the job was accepted.
`params`	`object`	Resolved job parameters, including applied defaults and limits.
`_warnings`	`array`	Nested response fields; optional unless explicitly documented.

This is an enqueue receipt, not the completed pages. Read check with the same owning account until the job is completed, failed or cancelled.

The status response uses id and contains result and per-page failures. The enqueue response uses job_id/check; web/agent uses id/poll instead.

Failures use the typed error envelope. A confirmed uncharged or refunded failure reports zero; a pending reconciliation can report an unknown charge. Read credits_used and error.details.billing_status, and keep request_id for recovery. Response contract · Error reference

Website Crawler API

Query parameters

Make the request

Response fields and example

Useful next calls

First call in under a minute