Prepare an integration for production

Use the reusable request client

Download the Node.js client. It sends direct JSON GET requests to the six example platforms. It includes a deadline covering headers and body, up to three attempts with the same idempotency key, bounded backoff, Retry-After handling, local credit accounting and a stop on unresolved billing. The workflow script demonstrates its use.

import { createClient } from './monocrawl-client.mjs';
import { randomUUID } from 'node:crypto';
import { appendFile } from 'node:fs/promises';

// Keep this journal private. A production service should use its job store.
const journal = record => appendFile('monocrawl-job.jsonl',
  JSON.stringify(record) + '\n', { mode: 0o600 });
const key = randomUUID();
const client = createClient({
  apiKey: process.env.MONOCRAWL_API_KEY,
  budget: 20,
  onRequest: request => journal({ event: 'checkpoint', ...request }),
  onEvent: async event => console.log(JSON.stringify(event)),
});
const specResponse = await fetch('https://www.monocrawl.com/openapi.json');
if (!specResponse.ok) throw new Error('Catalogue unavailable');
const spec = await specResponse.json();
const operation = spec.paths['/v1/tiktok/profile'].get;
if (operation['x-production-available'] !== true) throw new Error('Check availability');
const result = await client.get('tiktok/profile', { handle: 'nasa' }, {
  quote: operation['x-credits'], idempotencyKey: key,
});
await journal({ event: 'completed', idempotency_key: key, result });
console.log({ request_id: result.request_id, credits: client.spent });

The sample client intentionally runs sequentially. It is not a distributed scheduler or a durable job store. Persist request identity and completed results before adding workers or automatic process restarts. A fresh CLI run generates new keys; it cannot infer which earlier requests completed.

Decide what is safe to retry

The example refuses to sleep more than 30 seconds for one retry; it returns control so a job scheduler can reschedule longer waits. Its per-attempt timeout is 65 seconds, covering the API’s current 60-second function ceiling with some network margin. Adapt the overall job deadline to your workload. Aborting fetch does not cancel server-side work.

Send Idempotency-Key on the first request. Ordinary keys retain results for 24 hours; unresolved recovery records can remain protected longer. Do not rely on replay after expiry. Replays return the original body, including the original credits_used and balance, with x-idempotent-replay:true. Count that logical request once, not once per transport attempt.

These rules describe direct requests. POST /v1/batch has a separate contract and does not provide this direct-request replay protection. Retrying a whole batch can repeat successful items. See error and retry reference.

Set cost and concurrency boundaries

Use a dedicated key with an explicit cumulative credit limit. Check the current endpoint price before each new logical request; reserve enough for the whole job, including pages, detail calls and possible successful retries. A local estimate cannot enforce a shared wallet’s ceiling under concurrent workers. The API-key cap is enforced server-side.

Respect account rate and concurrency limits; begin with one in-flight request and increase only within the documented limits. Keep retries bounded, and prevent several workers from retrying the same job simultaneously.

Use the normal response cache where acceptable. Cache hits cost zero and report cached:true. Forced freshness can cost credits and has separate limits. Keep original filters and locale in your own cache keys, and distinguish fetched_at from the source publication time. See pagination and caching.

Treat successful partial results explicitly

Validate the envelope and the fields your application needs. HTTP 200 is not a complete-data guarantee. Bundles can expose partial, complete, legs and warnings; a failed component can leave usable rows elsewhere. Persist those diagnostics with the data and report gaps to the user rather than silently displaying a complete result.

Keep cursors with all original filters. Stop on repeated tokens, a page budget or an explicit exhaustion signal. Deduplicate by stable source ID and retain records without IDs separately. Neither an empty page nor a missing cursor proves complete historical coverage.

Use endpoint schemas for types and computed-field definitions for denominators. Do not turn missing metrics into zero or source-provided rankings into chronological order.

Log enough to explain a job

Record your job ID, endpoint path, attempt number, HTTP status, error type/reason, request_id, latency, cached/replay flags and confirmed credits_used. Keep unknown billing separately. Persist idempotency keys with job state, and restrict access to the request journal.

Keep API keys out of URLs and logs. Do not log raw headers, full response bodies or search terms by default. Store any needed parameters and collected content in your protected job data, with retention appropriate to your application. Use request_id when asking support about a specific outcome.

Before enabling a scheduled workload, complete the integration testing path: offline fixtures, explicit sandbox samples, then a small live smoke test with a capped key.