Public benchmark
Synthea to OMOP CDM 5.4, scored against the official ETL
The public Synthea sample (111 patients, 5,059 encounters, 3,387 conditions) is mapped by a deterministic baseline, compiled to SQL, executed, and compared row by row with OHDSI's own ETL-Synthea SQL run on the same files. No model call anywhere in it.
Run 2026-11-14 · 9.5 s end to end · 0 model calls · node v22.23.3 · latest.json
Table by table
| Table | Reference rows | Our rows | Agreement | What the number means |
|---|---|---|---|---|
person | 111 | 111 | 100.0% | gender, race, ethnicity and birth date match the reference on all 111 persons |
visit_occurrence | 4,781 | 5,059 | 94.5% | all 4,781 reference visits are present; the official ETL folds 278 inpatient and same-day ER rows into stays, the baseline spec does not roll up yet |
condition_occurrence | 1,323 | 3,387 | 100% code-keyed 98.4% label-only | concept_id agreement on the 1,323 rows the reference keeps; the reference drops codes missing from the vocabulary shard, the baseline keeps them with concept 0 and lists them |
Independent check: of the 70 condition codes in OHDSI's Eunomia Synthea27Nj CDM, 58 also occur in this sample; the baseline picks the same concept_id for 58 of them code-keyed and 57 label-only.
Where it falls short, from the same run
Visit rollup: 94.5% against a 95% gate
5,059 visits against 4,781. The official ETL merges inpatient stays and same-day emergency rows; the baseline mapping spec has one visit per encounter. Every reference visit exists in ours, so nothing is lost: the extra rows are the merge.
2,059 of 3,387 condition rows carry concept 0 (60.8%)
68 of 189 distinct codes cannot be mapped code-keyed (73 label-only): they are absent from the 2,294-concept vocabulary shard used here or sit outside the Condition domain. Each one is listed with its reason in latest.json under code_keyed.unmapped. These are the codes a model or a reviewer would pick up next.
4 data-quality checks fail, and they trace to two causes
condition_occurrence.condition_concept_id(standardConceptRecordCompleteness): 2,059 of 3,387 rows (60.8%), threshold 5%visit_occurrence.visit_concept_id(standardConceptRecordCompleteness): 71 of 5,059 rows (1.4%), threshold 0%condition_occurrence.condition_source_concept_id(sourceConceptRecordCompleteness): 3,387 of 3,387 rows (100.0%), threshold 10%condition_occurrence.condition_source_value(sourceValueCompleteness): 68 of 189 distinct codes (36.0%), threshold 10%
What was run
- Source: Synthea sample CSV, sha256
990d52064037bbf8… - Reference: OHDSI ETL-Synthea v5.4.0 SQL for person, visit_occurrence and condition_occurrence (ported to SQLite), run on the same CSV and the same vocabulary shard.
- Vocabulary: the 2,294-concept shard shipped inside OHDSI's Eunomia Synthea27Nj dataset. 140 of the 189 condition codes in the sample exist in it.
- Two mapping modes: code-keyed looks each source code up in the vocabulary; label-only strips the codes and matches on the text label alone, the situation of a local code list.
- Data-quality checks: the check types and thresholds come from OHDSI's published DataQualityDashboard tables; 256 checks were instantiated on person, visit_occurrence and condition_occurrence, with the results in DQD's own JSON shape.
Read this before you quote a number
- The vocabulary is a 2,294-concept shard, about 230 standard Condition concepts. Label-only agreement of 98.4% is easy against a search space that small. Against the full Athena vocabulary it will be lower, and that is the number to measure next.
- Code-keyed agreement of 100% is a lookup against the same vocabulary the reference uses; it shows the compiled SQL reproduces the official ETL's concept logic, not that mapping is solved.
- The visit rollup of the official ETL is not reproduced (above).
- DQD check types are re-implemented here (256 instances on three tables), not run through the DQD R package. Seven standard Athena concepts missing from the shard (no-matching-concept, gender, race and type concepts) are added before the checks run.
- A passing data-quality run shows the tables are well-formed and plausible; it does not show the mapping is clinically right.
How many AI calls a 5,000-code list needs
Deterministic tiers (exact code, exact name, token search) settle what they can before any model is asked. On a 5,000-label stress list built by rewriting shard concept names (word order, dropped words, typos, abbreviations, noise words): 3,511 codes mapped without a model, 21 of them to the wrong concept (0.6%), 1,489 left for the model tier, which is 15 calls at 100 codes per call. A list keyed by vocabulary codes (230 Condition codes) leaves 2 codes: 1 call. The stress list is synthetic; a real local list will differ.
Re-run it
Node 22.13 or newer. The command downloads the two public inputs, builds the reference with the official SQL, runs the baseline and prints the scorecard in about 10 seconds.
curl -sSfLO https://omopforge.grain64.com/downloads/omopforge-proof.tar.gz
tar xzf omopforge-proof.tar.gz && cd omopforge-proof
make proofResults land in proof/results/latest.json. make test runs the test suite (spec validator, compiler golden files for four SQL dialects, mapper, data-quality checks, SQLite, PostgreSQL and DuckDB executing the same compiled spec).
Run it on your own tables
The same engine takes your schema and a synthetic sample, proposes the mappings, writes the ETL for SQLite, PostgreSQL, SQL Server or DuckDB, runs it and scores it. Leave an email for the day it is open.