The article pipeline, map and machinery

Gather, don't decide

Ten fetch services sweep the literature every 20 minutes and return one plain record each. They only collect — nothing here judges relevance or writes to the library.

Seen it already?

The cheapest gate, run first. Matched by normalized DOI, then title. A hit ends the journey before any model is touched.

Does it belong here?

A keyword prefilter settles almost everything for free. Only genuinely borderline records cost the one relevance model call, which returns a verdict and a one-line reason.

Explain it plainly

The kept article gets the second and last model call: one request returns an 80–120 word summary and four study facts (design, sample, population, key finding) together.

Two doors out

Where the article lands is decided entirely by the verdict it earned back in Step 03.

LibraryFetchJob — the heartbeat

The recurring form of everything in Parts I–II. Each run is deliberately small: at most 3 rotation cycles (a few minutes of work), an 80-year publication window, and a 5-million target that in practice means never done. Coverage comes from query rotation: 677 curated queries in config/comprehensive_queries.yml, visited round-robin across all 10 sources, each source keeping its own position in the list.

LibraryFetchService.run(
  resume:     true,       # pick up the checkpoint exactly where the last run stopped
  target:     5_000_000,  # effectively "never done"
  years:      80,
  max_cycles: 3,          # bounded work per run, then yield until the next tick
  install_signals: false
)

Checkpoint. Progress lives in storage/library_fetch.json — a persistent volume, so it survives deploys — rewritten after every source pull, so a killed process resumes mid-rotation. Progress is measured as new ids above a baseline (immune to concurrent deletes). A run stops early when a full pass through every query saves nothing (stalled), and a source that fails 8 times in a row is disabled for the rest of the run.

TopicsStatsWarmJob — keep topic pages instant

Topic pages show heavy rollups (article counts, participants, study designs, top journals). Recomputing those per visitor would be brutal, so stats persist on the topic row in stats_data. Every 6 hours this job finds topics whose stats are older than 24 hours and calls refresh_stats!, pausing 2 seconds between topics. Visitors always read a precomputed answer.

TrialFetchJob — clinical trials

Sweeps ClinicalTrials.gov with 25 curated seed terms (psychedelic, contemplative, and dual-use compounds), up to 200 trials per term. Every trial passes a code-only ClinicalTrial.relevant? gate before storage, and rows are upserted by NCT id so the daily run is idempotent. Trials surface on topic pages and the trials index.

SavedSearchDigestJob — the weekly email

For every confirmed, opted-in user with saved searches: each follow collects articles newer than its last_sent_at (first digest looks back 7 days), then one rolled-up email goes out only if anything matched. The watermark is stamped forward even on a quiet week, so old articles are never re-sent later. One user's failure never sinks the run.

TopicEvidenceRefreshJob — State of the Evidence

Walks every topic and asks one question: evidence_due? — does this topic have at least 8 qualifying studies, and is its synthesis missing, failed, or older than 30 days? Due topics get refresh_evidence!, which only enqueues a pending synthesis; the actual model call runs one at a time in MetaAnalysisJob. The full anatomy of that synthesis is Part VI.

StudyDetailsBackfillJob — the patient extractor

New articles get their study-at-a-glance facts (design, sample size, interventions, dose, duration) extracted at fetch time. This cycle works the backlog: articles from before those fields existed. Each run takes a 10-minute time-boxed slice of the remaining scope, one model call per article, then yields until the next tick.

Checkpoint. Progress lives in the database: every non-erroring pass stamps study_details_checked_at, and the scope excludes stamped rows. A paper the model finds nothing in is stamped too — a nil answer is a final answer, paid for once, never re-asked. Only an article whose extraction raises (an API hiccup) stays unstamped for the next run to retry. A deploy costs nothing: the next tick resumes at the exact row where the container died. When the backlog drains, each run becomes a single cheap count. Serialized by its own advisory lock, same pattern as the heartbeat.

Check what the model hid

The queue at /articles/relevance_admin lists every article with relevance_flag = true, newest verdicts first. Each row shows the model's one-line rationale, any llm_error note, and when it was checked. Filters: full-text search, provider, source, checked-before/after dates. A dedicated Recheck toggle narrows to rows whose rationale starts recheck: model now says relevant — articles where a later pass changed the model's mind, queued for a human tiebreak instead of silently unhiding.

Rescue, or remove

Bulk mode: select-all checkboxes drive Unflag selected (one update_all with the same audit stamps, though no note) and Delete selected. Every rescue lands on a second page, /articles/relevance_unflagged_admin — the audit trail of who unflagged what, when, and why.

A question arrives

Door one: a reader types a question into the tool. Rate-limited to 5 per minute, capped at 300 characters, normalized to lowercase so "Does MDMA help PTSD?" and "does mdma help ptsd" are the same question. Door two: Cycle 05 (or topics:refresh_evidence) asks each due topic's deterministic question — "What does the research say about psilocybin?" — so a topic always reuses one synthesis row instead of minting duplicates.

Has this been asked before?

Code picks the studies, not the model

The model never chooses its own evidence. Topic door: a study qualifies only when the topic name or a known alias appears in its title or as an exact keyword (loose full-text mentions are excluded), it must have an abstract, and the topic needs at least 8 qualifying studies or no synthesis is attempted. Selection is the 15 most-cited plus 10 most recent, deduped, capped at 25 — established evidence anchors, fresh work fills in. Tool door: the question is expanded to keywords by regex (no model), searched through the same weighted index as the library, capped at 25.

One grounded question

DeepSeek deepseek-v4-flash at temperature 0.2, up to 4,000 output tokens, thinking disabled. The system prompt is a hard grounding contract:

Code strips what the model can't prove

Every field of the model's JSON passes deterministic gates before anything is stored:

parse_json(raw)          # strip ``` fences, take first "{" to last "}", JSON.parse or nil

clean_evidence_row(row)  # per study row:
  return nil unless valid_ids.include?(row["article_id"])  # hallucinated citation → dropped
  direction   = DIRECTIONS.include?(dir) ? dir : "unclear"   # invented category → coerced
  sample_size = digits_only(row["sample_size"])              # "n=24 adults" → 24
  design    ||= "unspecified"

clean_strings(list)      # trim, drop blanks, cap every list at 6 items
article_count            # recomputed from the rows that SURVIVED, not the model's claim

EvidenceStrength.for(ma) # strength tier COMPUTED from the surviving rows:
  # classify each design (meta-analysis > RCT > observational > case/theoretical),
  # then grade on volume + design + directional agreement + sample size.
  # High needs a meta-analysis or 3+ consistent, adequately-powered RCTs.

The one that matters most: a citation of any article that wasn't in the provided set simply ceases to exist. The model cannot cite its way outside the library.

The strength of evidence is not something the model rates: it is computed from the surviving evidence rows, so it is a transparent function of the studies (their number, designs, agreement, and sample sizes) rather than a self-assessment that always hugged the low end.

Where it lands

The row flips to complete and the polling page swaps in the result. From there it appears in the tool's Recent analyses (last 10), as the lead section of its topic page ("State of the evidence", with the synthesized date and a copy-link anchor), in reader bookmarks, and through copy-summary / copy-APA-citations buttons. Every evidence row links back to its real article in the library — the same rows Part II built.