Pagination & caching

Follow the endpoint's continuation contract

For a generated list of endpoint controls and optional response fields, use the pagination matrix. The guidance below explains how to interpret them.

Many list endpoints return data.items and data.cursor. Where supported, pass that cursor back as the cursor query parameter while retaining the original filters. Treat it as opaque: copy the returned value unchanged, as its format can change. Do not decode, edit or construct it yourself, or assume every endpoint uses this pattern. Previously issued valid cursors remain usable for continuation.

null means no usable next cursor was supplied, not proof that all history was retrieved. Check has_more and warnings where present:has_more: null means unknown; false describes the returned source's continuation state or bounded surface, not necessarily the entire platform history.

For an endpoint using items and cursor, the loop looks like this:

const key = process.env.MONOCRAWL_API_KEY;
if (!key) throw new Error('Set MONOCRAWL_API_KEY on your server.');
let cursor = null;
const seen = new Set();
// Bound collection: each successful uncached page can cost credits.
for (let page = 0; page < 10; page++) {
  const url = new URL('https://www.monocrawl.com/v1/tiktok/search');
  url.searchParams.set('query', 'coffee brewing');
  if (cursor) url.searchParams.set('cursor', cursor);

  const res = await fetch(url, { headers: { 'x-api-key': key } });
  const body = await res.json();
  if (!body.success) throw new Error(body.error.message);

  for (const item of body.data.items ?? []) console.log(item);
  cursor = body.data.cursor;
  if (!cursor || seen.has(cursor)) break;
  seen.add(cursor);
}
// Stopping (including at the page cap) is not proof of complete history.

Each page is a separate request under the endpoint's billing rules. Some endpoints are deliberately single-page and say so in their description (instagram/search-top returns Instagram’s one ranked page; instagram/reels-trending returns a fresh small batch per call).

Bundles and unsupported continuation

Social full-profile bundles expose posts, posts_cursor and posts_pagination.next_params. Use all the supplied next parameters together; some continuations require more than one token. A null next_params means no usable next request is offered, not that every post has been collected. The requested posts count is a desired page size; the source may return fewer or more.complete: true describes successful components and one bounded source page, not full history. Check partial, legs and warnings.

YouTube's channel bundle uses recent_videos, cursor and recent_videos_pagination. Other composites can have named collections without continuation. A fallback may not support a primary source's cursor or sort; an unsupported request can be refused before collection rather than restarting the first page. The endpoint reference and returned metadata take precedence over the generic loop above.

The response cache: repeat reads are free

Eligible public responses may enter a shared response cache. A repeat of the same endpoint with the same parameters can be served from a reusable entry within its fresh-cache lifetime. A cache hit costs 0 credits net; the response says "cached": true, the x-cache header says HIT, and your request log marks the row as a hit. A 3-credit search re-checked inside the window costs nothing, with zero code on your side. Paid endpoints still need credits in your balance: with none left, a request is refused even when its answer is cached.

Identical concurrent requests can share one upstream fetch: successful waiters receive a free cache hit. A waiting request may show a temporary reservation and matching refund in its ledger. This is best-effort, not an exactly-once guarantee. Entries may be evicted or exceed the size limit; a shared-store outage, lock expiry or wait timeout can cause a separate billed fetch. Use the idempotency contract for retries of the same logical direct request.

The default fresh-cache lifetimes are below. Endpoint overrides can change them; eviction can remove an entry sooner:

What never enters the cache: errors, synthetic fallbacks, sandbox samples, and anything that depends on who is asking — your balance, your monitors, jobs and reports are always live and never shared. Cache keys are canonicalized: ?a=1&b=2 and ?b=2&a=1 hit the same entry, and the page cursor is part of the key, so different page cursors do not collide. A normal HIT can be as old as its permitted fresh-cache lifetime. A marked outage fallback has the separate limit below.

A reference answer during a temporary outage

The public directories amazon/categories, target/categories and sephora/brands may return their last successful cached answer when a live retrieval temporarily fails or all eligible sources are temporarily paused after repeated failures. An expired entry is considered only after that failure; it is not used for an ordinary repeat read.

These entries can remain available for at most one additional hour after their existing fresh-cache lifetime, with total retention capped at 24 hours from the original retrieval. Eviction or a missing entry can still mean an error. Serving an older entry does not reset its retrieval time or extend its retention.

This response costs 0 credits net and carries cached: true, stale: true, a warning, the original retrieval time and the age of the data. The headers identify it as x-cache: STALE; x-cache-age gives its age in seconds and x-cache-fetched-at gives the original retrieval timestamp. Treat the contents as an older reference answer, not confirmation of a current source value.

This fallback does not apply to search, personal profiles, live prices, availability or private responses. It does not replace invalid-input, not-found or other definitive answers. Monitor and observer retrievals, including internal shared fetches, do not use expired entries.

Empty results cost zero credits

A successful normalized list or search page with no records costs 0 credits, including an empty page carrying a continuation token. The cursor is retained. Any reservation is refunded, and X-Credits-Used: 0 agrees with credits_used: 0. An absent transcript is also free. A populated lookup remains billed even when optional fields are null. Supplier usage still counts toward service limits.

Asking for a fresh answer

Add fresh=1 (or send Cache-Control: no-cache) to skip the cache and request a new retrieval. A successful uncached retrieval is billed under the endpoint’s rules, the response carries x-cache-fresh: applied, and an eligible answer can update the shared cache. So that this cannot become a way to burn upstream quota, each key may ask for a fresh answer up to 30 times a minute. Beyond that the response is an uncharged 429 with x-cache-fresh: rejected and retry-after. If production cannot verify the shared allowance, it returns an uncharged 503. Add max_age=<seconds> (minimum 30) to accept a cached answer only if it is at most that old: a hit is still free, anything older is fetched at the endpoint price.

Requesting fresh=1 or Cache-Control: no-cache always rules out a stale outage fallback. Admitted fresh calls run their own retrieval; they do not join an older request. Your allowance is checked before wallet or supplier work. Your max_age limit also applies while waiting for a shared result and to outage fallbacks: if the saved answer is older, the request returns the live outcome rather than that entry.

Streaming long searches (SSE)

The streaming guide includes a request example, an incremental parser and disconnect recovery rules.

The multi-source search endpoints (search/all, search/creators) accept stream=1 and answer text/event-stream: a plan event with the leg list, source-started / source-completed per leg as they actually run (with latency, upstream cost and items found), a cost rollup, enrichment-completed events when comment enrichment is on — then a result event carrying the same envelope a plain call returns. Billing, receipts and your usage log are identical either way. The HTTP stream can start with 200 and end with an error envelope, so inspect the final event’s success field. A transport exception may emit an error event instead; a dropped stream is not proof of success or failure. Disconnecting is not a supported cancellation mechanism.

Upstream caching on metered routes

Some flagship-upstream endpoints additionally accept cache_max_age-style recency controls at the platform layer; where an endpoint supports one, its parameter table on the endpoint reference lists it. When you need a new retrieval, request fresh=1 and check x-cache-fresh andcached. This bypasses Monocrawl’s cache when honoured, not every upstream’s cache, and cannot guarantee the source has updated its data.