Cohorts and keyword sweeps
Save a bounded set of communities or accounts, run a keyword sweep, and inspect outcomes for each member. A completed sweep is a sample, not exhaustive history.
Choose supported sweep members
| Platform | Identifier | Search behavior |
|---|---|---|
| Subreddit, such as rust | Keyword search ordered by new. | |
| Hacker News | Username | Search with an author restriction. |
| GitHub | Owner or owner/repository | Search within the user or repository. |
| Bluesky | Account handle | Search posts from the account. |
These four platforms have sweep adapters. Members on other accepted catalogue platforms can be stored but report ok:false during a sweep. Inspect sweepable_count and warnings first. Membership does not verify identity or merge people across platforms.
Matching is case-insensitive substring matching over returned item text. A multi-word keyword can also match when every word occurs somewhere in the item. This is not semantic relevance or exact-phrase-only matching. Source filters also constrain the retrieved window.
1. Create and inspect a cohort
The current canonical REST operations below use GET, including named creation and deletion operations. They have effects despite the method: invoke explicitly from your server, not from prefetching links. Use header authentication and check the current price before creation or submission.
curl -G 'https://www.monocrawl.com/v1/cohorts/create' \
-H "x-api-key: $MONOCRAWL_API_KEY" \
-H 'Idempotency-Key: REPLACE_WITH_SAVED_CREATE_KEY' \
--data-urlencode 'name=Rust discussions' \
--data-urlencode 'members=[{"platform":"reddit","identifier":"rust"},{"platform":"github","identifier":"rust-lang/rust"}]'Check success and the credit receipt, then save data.id as cohort_id. The cohort includes name, members, member_count, sweepable_count and created_at. Inspect it with cohorts/get and cohort_id. Replace example IDs below with returned IDs.
2. Submit once, then poll
curl -G 'https://www.monocrawl.com/v1/cohorts/queries' \ -H "x-api-key: $MONOCRAWL_API_KEY" \ -H 'Idempotency-Key: REPLACE_WITH_SAVED_QUERY_KEY' \ --data-urlencode 'cohort_id=YOUR_COHORT_ID' \ --data-urlencode 'keyword=async runtime'
Save data.id as query_id and retain job_id. Poll cohort-queries/get with query_id at a bounded interval. A lost submission response does not prove no work was queued: retain the original key and identical parameters while reconciling. A new submission can duplicate work and spend.
| Status | Next step |
|---|---|
| queued / running | Wait and poll the saved query. A client deadline stops waiting, not server execution. |
| completed | Read results and inspect per-member failures and warnings. |
| failed | Inspect error and the associated job refund_status; due means pending, not already credited. |
curl -G 'https://www.monocrawl.com/v1/cohort-queries/get' \ -H "x-api-key: $MONOCRAWL_API_KEY" --data-urlencode 'query_id=YOUR_QUERY_ID'
3. Read retained result pages
curl -G 'https://www.monocrawl.com/v1/cohort-queries/results' \ -H "x-api-key: $MONOCRAWL_API_KEY" \ --data-urlencode 'query_id=YOUR_QUERY_ID' --data-urlencode 'limit=25'
| Field under data | Meaning |
|---|---|
| ready / status | HTTP 200 can carry ready:false and empty items while unfinished or failed. That does not mean no matches. |
| items / count / total | This page, its length and the retained total, not every match on the source. |
| cursor | Pass unchanged with the same query_id. Null ends the retained pages. |
| counts | members, swept, ok, failed, returned and matched summarize bounded work. |
| per_member | Inspect ok, returned/matched details, errors and truncation. |
| _warnings | Time, source-window and result-size limitations. Preserve alongside exports. |
{
"success": true,
"credits_used": 0,
"data": {
"query_id": "cq_example",
"status": "completed",
"ready": true,
"items": [],
"count": 0,
"total": 0,
"cursor": null,
"counts": {
"members": 2,
"swept": 2,
"ok": 1,
"failed": 1,
"returned": 10,
"matched": 0
},
"_warnings": [
"One member could not be swept; inspect per_member."
]
}
}The example omits per_member rows for brevity. Completion can include failed or unattempted members. Retain original IDs and URLs; an empty bounded sample does not prove a topic was never discussed.
Limits, membership and cleanup
| Control | Current behavior |
|---|---|
| Membership | 200 cohorts/account; up to 100 supplied members/cohort, normalized and deduplicated. |
| Pending work | 25 queued/running queries/account. |
| Sweep | 60-second worker budget; at most one source call/member requesting up to 25 items. |
| Results | Up to 500 matches and a 1,000,000-byte result bound; warnings disclose truncation. |
| Result pages | Default 25, maximum 100 rows. |
cohorts/members replaces membership rather than appending. cohort-queries/delete with query_id deletes a query; cohorts/delete with cohort_id removes the cohort and its queries and attempts cancellation of queued/running work. Inspect cancellation/refund fields. Export needed results before deleting.
Resources are account-owned; missing and unowned resources return not found. No date-complete archive or fixed retention period is promised. Generic MCP call_endpoint does not provide cohort management or sweeps; use authenticated REST. MCP documentation and endpoint discovery can still explain them.