Jobs, webhooks & monitors

Async jobs

Long-running operations (site crawls, batch scrapes) run as jobs: the enqueueing call charges up front and returns a job id immediately; GET /v1/web/jobs lists yours and GET /v1/web/jobs/get?id=… returns one with its status and, when finished, its results. The refund contract is strict: a failed job refunds its full charge, exactly once — and if the ledger is briefly unavailable at refund time, the refund is durably marked due and retried until it lands rather than being dropped. Cancelling a job reports what the ledger actually did, never an assumed refund. A worker dying mid-job is detected and treated as a failure, with the same refund path.

Webhooks

Register up to 5 HTTPS endpoints per account in the dashboard, each with its own signing secret and event filter. Deliveries are signed so you can prove they came from us:

import crypto from 'node:crypto';

// rawBody is the original Buffer (or unmodified UTF-8 string), BEFORE JSON parsing.
export function verifyMonocrawlSignature(rawBody, header, secret, nowMs = Date.now()) {
  const match = /^t=(\d+),v1=([0-9a-f]{64})$/.exec(header ?? '');
  if (!match || !secret) return false;
  const [, timestamp, signature] = match;
  const seconds = Number(timestamp);
  if (!Number.isSafeInteger(seconds) || Math.abs(nowMs / 1000 - seconds) > 300) return false;
  const expected = crypto.createHmac('sha256', secret)
    .update(timestamp + '.').update(rawBody).digest();
  const received = Buffer.from(signature, 'hex');
  return received.length === expected.length && crypto.timingSafeEqual(expected, received);
}

Delivery is at-least-once. Successive retry delays are 1 minute, 5 minutes, 30 minutes, 2 hours and 12 hours, then mark the delivery failed (about 14 hours 36 minutes total, plus processing and worker scheduling). Dedupe on the delivery id. These signatures and JSON envelopes describe the JSON format; Slack and Discord destinations receive formatted messages instead. An endpoint that fails 10 consecutive deliveries is auto-disabled (with the reason recorded and visible in the dashboard) rather than being hammered forever; re-enable it after fixing your receiver. Respond 2xx within 10 seconds; do the work after acknowledging.

Monitors

See the dedicated monitor payloads, billing payloads and receiver implementation. For saved member sweeps and their jobs, read Cohorts.

Every monitor can optionally filter results against plain-English relevance criteria, whether created in the dashboard, through the API or through MCP. Confirmed matches can be delivered while undecided items remain pending. Alerts link to the saved matching finding, its explanation, supporting passage and source. See relevance filtering and uncertainty and reading alerts and grouped results.

Monitors follow subjects, review feeds, pages or endpoint data on a schedule. Depending on the monitor kind, they retain findings, change records and numeric timeseries. GET /v1/monitors lists yours; monitors/create, monitors/get, monitors/runs, monitors/timeseries, monitors/update and monitors/delete cover the basic lifecycle — the parameter tables live in the endpoint reference. Scheduled runs are billed according to the monitor kind and estimate, then settled against retrieval. Successful checks can cost credits even when no findings match. See the full monitor guide for previews, destinations, caps and findings.