Add diagnostic-delay concept module (cohort + delay measures + data-quality report) - #2173
Draft
developer-rpai wants to merge 1 commit into
Draft
developer-rpai wants to merge 1 commit into
developer-rpai wants to merge 1 commit into
Conversation
…uality report) New `mimic-iv/concepts/diagnosticdelay/` module: - suspected_sepsis_cohort.sql: one row per ICU stay for adults admitted via the ED with >= 1 suspected-infection episode (antibiotic + microbiology culture pair, per suspicion_of_infection). - diagnostic_delay.sql: delay in hours between admission (hospital/ICU) and first recognition of infection (suspected-infection time, first antibiotic, first culture); onset-window classification (pre_admission / present_on_admission / hospital_onset); and temporal-plausibility flags for out-of-order or missing timestamps. - data_quality_report.py: renders a Markdown/JSON data-quality report (cohort coverage, missingness, flag prevalence, delay distributions) from the two tables above; runs on DuckDB or PostgreSQL. - README.md documents the cohort definition, measures, validation, and limitations. ICD codes carry no timestamp in MIMIC, so diagnosis time is proxied by the clinically observable recognition events used in the Sepsis-3 definition (see MIT-LCP#1843); ordering checks surface implausible values of the kind reported in MIT-LCP#2168 instead of silently averaging them in. Validated without credentialed MIMIC-IV access: sqlglot parse (BigQuery dialect, same as CI), sqlfluff 4.1.0 with the repo config, transpile to DuckDB/PostgreSQL via mimic_utils, execution of the DuckDB build on hand-built synthetic fixtures (16 assertions), and a run of the report script against the fixture database. End-to-end run on real MIMIC-IV data is still outstanding.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this is
A new concept module,
mimic-iv/concepts/diagnosticdelay/, for reproducible cohort extraction and diagnostic-delay measurement on MIMIC-IV:suspected_sepsis_cohort.sql— one row per ICU stay: adults admitted via the emergency department with at least one suspected-infection episode (antibiotic + microbiology culture pair, persuspicion_of_infection), with anchor timestamps and demographics.diagnostic_delay.sql— delay in hours between hospital/ICU admission and first infection recognition (suspected-infection time, first antibiotic, first culture); an onset-window classification (pre_admission/present_on_admission/hospital_onset); and temporal-plausibility flags (1 = implausible, NULL = unassessable) for out-of-order or missing timestamps.data_quality_report.py— generates a Markdown/JSON data-quality report (cohort coverage, missingness, flag prevalence, delay distributions with median/p25/p75) from the two tables; runs on DuckDB or PostgreSQL with portable SQL.README.md+ concept-index entry — documents the cohort definition, measures, validation, and limitations.Why
ICD codes in MIMIC carry no timestamp, so "when was this diagnosed relative to admission?" is a recurring community question (e.g. #1843 on diagnosis timing for sepsis/AKI research). This module answers a tractable version of it by proxying diagnosis time with the clinically observable recognition events already used in the Sepsis-3 definition. The plausibility flags address the class of problem raised in #2168 (implausible values invisible to anyone who doesn't check before aggregating): flagged stays can be investigated rather than silently averaged in.
Validation (what ran, what didn't)
I do not have credentialed MIMIC-IV access, so validation was done without real data — an end-to-end run on MIMIC-IV (v3.1) is still outstanding:
sqlglotparse check of both queries in the BigQuery dialect (.github/scripts/check_sql_syntax.py) — pass.sqlflufflint with the repo's.sqlfluffconfig (sqlfluff 4.1.0, same as CI) — pass, no violations.mimic_utilstranspiler; both parse in the target dialects.patients,admissions,icustays, derivedsuspicion_of_infection): 16 assertions covering a standard present-on-admission case, a hospital-onset case, a pre-admission-suspicion case (flagged), multiple episodes per stay, pediatric exclusion, non-ED admission exclusion, a stay with no suspected infection (excluded), and a NULLicu_outtimecase (flag unassessable) — all pass, delays match hand computation.data_quality_report.pyagainst the fixture database (with a raw-schema coverage denominator): Markdown/JSON outputs verified against hand-computed values (4/5 = 80% capture, flag counts, medians/p25/p75).Notes for reviewers
mimic-iv/concepts/sources are touched. The transpiledconcepts_postgres/concepts_duckdboutputs are intentionally not included — the regenerate-dialects bot produces those on merge. Theduckdb.sql/postgres-make-concepts.sqlbuild lists are likewise left for maintainers to extend.mimiciv_hosp+mimiciv_icu+ derived concepts (no MIMIC-IV-ED module), so it runs on the standard build. ED arrival time is deliberately not used; the README notes this as a limitation.