Web Scraping reference
Web Scraping API guide
Map a public site, retrieve selected pages and extract structured page content. Start with bounded single-page reads before using asynchronous crawls or browser sessions.
Quickstart
Start with Web Scraping
Map a public site, retrieve selected pages and extract structured page content. Start with bounded single-page reads before using asynchronous crawls or browser sessions.
These requests use your API key and can spend credits. Check the current price on the endpoint page; each new page is a separate request. Later examples use illustrative identifiers: replace them with values from your earlier response.
1. Discover site URLs
Mapping discovers URLs. It does not fetch the content of every discovered page. Parameters and current cost →
curl "https://www.monocrawl.com/v1/web/map?url=https%3A%2F%2Fexample.com&limit=10" \ -H "x-api-key: mn_your_key_here"
2. Retrieve a selected page
Replace the example URL with a returned public page and choose the output formats your application needs. Parameters and current cost →
curl "https://www.monocrawl.com/v1/web/scrape?url=https%3A%2F%2Fexample.com&formats=markdown%2Cmetadata" \ -H "x-api-key: mn_your_key_here"
3. Read structured page content
Use the same selected page if you need the extractor’s fields. This is another request; reuse saved scrape output when it already answers the task. Parameters and current cost →
curl "https://www.monocrawl.com/v1/web/extract?url=https%3A%2F%2Fexample.com" \ -H "x-api-key: mn_your_key_here"
Filters and coverage
Choose filters with the source behavior in mind
| Control | Behavior and limits |
|---|---|
| Fetch mode | direct reads static HTML by default. Non-direct fetch modes require deployment enablement, approved targets and resource/spending controls. They can be refused; requesting browser mode does not guarantee JavaScript rendering for any URL. Endpoint reference → |
| Mapping | sitemap can be include, only or skip. URL discovery, include_subdomains and query-string deduplication affect the set, not whether every page was retrieved. Endpoint reference → |
| Output formats | Select the declared formats. only_main_content strips page chrome; keep the final URL and metadata alongside extracted text. Endpoint reference → |
| Jobs | Crawl and batch-scrape enqueue bounded jobs. Use the documented HTTP method, keep job_id, and inspect jobs/get for results. A timeout or client disconnect is not a cancellation. Endpoint reference → |
Source availability, ordering and field coverage can change. Inspect warnings and partial-result metadata alongside the rows. See runnable workflows, metric semantics, computed fields and the pagination matrix.
For a stateful browser workflow, see browser sessions. A browser execution runs page JavaScript; it is not a general Node.js process.
Choose an endpoint
Match the operation to your task
| Your task | Endpoint | When to use it |
|---|---|---|
| A document file | parse | Use the document parser for supported PDF, text and structured text formats. |
| Several pages on one site | crawl | Starts an asynchronous job; read job state and limits instead of treating the enqueue response as finished data. |
| Known URLs in a job | batch-scrape | This web job is distinct from the generic POST /v1/batch wrapper. |
| Read job progress | jobs/get | Use the returned job_id; job creation is not available through MCP call_endpoint. |
Explore all documented operations by purpose below. Each endpoint reference contains its accepted parameters and current price.
Other operations
Accounts and audiences
Operations
Web Scraping
Each row is a registry operation: expand it for parameters, a curl built from the registry's own examples, and a Markdown copy of the whole entry. The badge on the right is the operation's state, read at request time — we publish all three rather than rounding up. Recorded proof is historical, not a guarantee of current availability.
- live · proven
- a real upstream response has been served through this API and recorded
- wired · unproven
- a real-data route is registered, but qualifying proof is absent; permission or capacity restrictions may apply
- sample · free
- no live upstream — /v1 refuses it, /sandbox returns a marked sample
Web Scraping
16 operationsElsewhere
The same registry, other shapes
- Web Scraping overview — supported data categories and use cases for this platform.
- Full endpoint reference — every platform on one page.
/openapi.jsonor/openapi.yaml— the machine-readable contract, for Postman, Insomnia or a client generator.GET /v1/utility/endpoints?platform=web— the same rows to your code at runtime, for 0 credits.