Web Scraping API

Map URLs on a site

Discover a site's URLs from its sitemap.xml (one level of sitemap indexes) with same-origin page links as fallback.

Read as Markdown Use with an AI agent

Endpoint

GET/v1/web/maplive · proven3 credits

Try in Playground

Parameters

Query parameters

Name	Required	Description	Example
`url`	yes	Site URL to map (http/https, public hosts only)	`https://example.com`
`search`	no	Only return URLs containing this text (case-insensitive)	`blog`
`limit`	no	Maximum URLs to return (default 100, cap 500)	`100`
`sitemap`	no	Sitemap use: include (default: sitemap first, page links as fallback), only, or skip	`include`
`include_subdomains`	no	Also include URLs on subdomains of the site	`false`
`ignore_query_parameters`	no	Strip query strings before deduplicating URLs	`false`
`dry_run`	no	Read a zero-credit estimate without fetching sources, running AI, or reserving credits. Cache status is a snapshot, not a guarantee at execution.	`1`

Examples

Make the request

curl

curl "https://www.monocrawl.com/v1/web/map?url=https%3A%2F%2Fexample.com" \
  -H "x-api-key: mn_your_key_here"

TypeScript

typescript

const key = process.env.MONOCRAWL_API_KEY;
if (!key) throw new Error('Set MONOCRAWL_API_KEY on your server.');
const res = await fetch(
  "https://www.monocrawl.com/v1/web/map?url=https%3A%2F%2Fexample.com",
  { headers: { "x-api-key": key } },
);
const body = await res.json();

if (!body.success) {
  // one error shape for every endpoint — see /docs/errors
  throw new Error(`${body.error.type}: ${body.error.message}`);
}

console.log(body.data, "credits left:", body.credits_remaining);

Python

python

import os
import requests

res = requests.get(
    "https://www.monocrawl.com/v1/web/map?url=https%3A%2F%2Fexample.com",
    headers={"x-api-key": os.environ["MONOCRAWL_API_KEY"]},
    timeout=60,
)
body = res.json()

if not body["success"]:
    # one error shape for every endpoint — see /docs/errors
    raise RuntimeError(f"{body['error']['type']}: {body['error']['message']}")

print(body["data"], "credits left:", body["credits_remaining"])

Response

Response fields and example

This example is illustrative, not a captured live response. Variable-cost operations may settle a charge different from the list price below. A successful response puts the platform payload in data and reports the exact credits used, remaining balance, request id and cache status beside it.

200 · application/json

{
  "success": true,
  "platform": "web",
  "endpoint": "/v1/web/map",
  "data": {
    "_meta": {
      "billing": {
        "charged": 0,
        "list_price": 0,
        "reason": "example"
      }
    },
    "_warnings": [
      "example"
    ],
    "count": 0,
    "discovery": {
      "document_limit_bytes": 0,
      "documents_fetched": 0,
      "max_documents": 0,
      "max_sitemaps": 0,
      "time_limit_ms": 0
    },
    "items": [],
    "limit": 0,
    "origin": "example",
    "sources": [
      "example"
    ],
    "url": "https://example.com/example"
  },
  "credits_used": 3,
  "credits_remaining": 99,
  "request_id": "req_…",
  "cached": false
}

The example is illustrative and shows a documented subset of data. The fields below are optional across supported sources; nullable fields can also be absent. Preserve unknown values and accept additional fields.

Schema basis: observed response shapes, not an exhaustive field specification. See schema coverage and validation limits.

Download the data JSON Schema. Validate response.data, not the whole envelope. A valid shape does not establish that every field or source record was returned.

Field inside data	Type	Meaning
`_meta`	`object \| null`	Optional meta supplied by this operation. Null means unknown.
`_meta.billing`	`object \| null`	Optional billing supplied by this operation. Null means unknown.
`_meta.billing.charged`	`number \| null`	Optional charged supplied by this operation. Null means unknown.
`_meta.billing.list_price`	`number \| null`	Optional list price supplied by this operation. Null means unknown.
`_meta.billing.reason`	`string \| null`	Optional reason supplied by this operation. Null means unknown.
`_warnings`	`array \| null`	Optional warnings supplied by this operation. Null means unknown.
`count`	`number \| null`	Number of records in this response, not lifetime or total matches.
`discovery`	`object \| null`	Optional discovery supplied by this operation. Null means unknown.
`discovery.document_limit_bytes`	`number \| null`	Optional document limit bytes supplied by this operation. Null means unknown.
`discovery.documents_fetched`	`number \| null`	Optional documents fetched supplied by this operation. Null means unknown.
`discovery.max_documents`	`number \| null`	Optional max documents supplied by this operation. Null means unknown.
`discovery.max_sitemaps`	`number \| null`	Optional max sitemaps supplied by this operation. Null means unknown.
`discovery.time_limit_ms`	`number \| null`	Optional time limit ms supplied by this operation. Null means unknown.
`items`	`array \| null`	Source array was empty in the reviewed captures; item fields remain unspecified.
`limit`	`number \| null`	Optional limit supplied by this operation. Null means unknown.
`origin`	`string \| null`	Optional origin supplied by this operation. Null means unknown.
`sources`	`array \| null`	Optional sources supplied by this operation. Null means unknown.
`url`	`string \| null`	Original source URL when supplied.

Optional fields reviewed from preserved successful responses. Additive fields and source variations remain allowed; validate the fields your workflow requires.

The example is invented and illustrates types only; zero and empty values are not claims about a real result.

Failures use the typed error envelope. A confirmed uncharged or refunded failure reports zero; a pending reconciliation can report an unknown charge. Read credits_used and error.details.billing_status, and keep request_id for recovery. Response contract · Error reference

Map URLs on a site

Query parameters

Make the request

Response fields and example

More Web Scraping reference

First call in under a minute