Skip to content

Add diagnostic-delay concept module (cohort + delay measures + data-quality report) - #2173

Draft
developer-rpai wants to merge 1 commit into
MIT-LCP:mainfrom
developer-rpai:feature/diagnostic-delay-cohort
Draft

developer-rpai wants to merge 1 commit into
MIT-LCP:mainfrom
developer-rpai:feature/diagnostic-delay-cohort

Conversation

@developer-rpai

Copy link
Copy Markdown

What this is

A new concept module, mimic-iv/concepts/diagnosticdelay/, for reproducible cohort extraction and diagnostic-delay measurement on MIMIC-IV:

  • suspected_sepsis_cohort.sql — one row per ICU stay: adults admitted via the emergency department with at least one suspected-infection episode (antibiotic + microbiology culture pair, per suspicion_of_infection), with anchor timestamps and demographics.
  • diagnostic_delay.sql — delay in hours between hospital/ICU admission and first infection recognition (suspected-infection time, first antibiotic, first culture); an onset-window classification (pre_admission / present_on_admission / hospital_onset); and temporal-plausibility flags (1 = implausible, NULL = unassessable) for out-of-order or missing timestamps.
  • data_quality_report.py — generates a Markdown/JSON data-quality report (cohort coverage, missingness, flag prevalence, delay distributions with median/p25/p75) from the two tables; runs on DuckDB or PostgreSQL with portable SQL.
  • README.md + concept-index entry — documents the cohort definition, measures, validation, and limitations.

Why

ICD codes in MIMIC carry no timestamp, so "when was this diagnosed relative to admission?" is a recurring community question (e.g. #1843 on diagnosis timing for sepsis/AKI research). This module answers a tractable version of it by proxying diagnosis time with the clinically observable recognition events already used in the Sepsis-3 definition. The plausibility flags address the class of problem raised in #2168 (implausible values invisible to anyone who doesn't check before aggregating): flagged stays can be investigated rather than silently averaged in.

Validation (what ran, what didn't)

I do not have credentialed MIMIC-IV access, so validation was done without real data — an end-to-end run on MIMIC-IV (v3.1) is still outstanding:

  1. sqlglot parse check of both queries in the BigQuery dialect (.github/scripts/check_sql_syntax.py) — pass.
  2. sqlfluff lint with the repo's .sqlfluff config (sqlfluff 4.1.0, same as CI) — pass, no violations.
  3. Transpiled both queries BigQuery → DuckDB and BigQuery → PostgreSQL with the repo's mimic_utils transpiler; both parse in the target dialects.
  4. Executed the DuckDB build against hand-built synthetic fixtures mirroring the MIMIC-IV schema (patients, admissions, icustays, derived suspicion_of_infection): 16 assertions covering a standard present-on-admission case, a hospital-onset case, a pre-admission-suspicion case (flagged), multiple episodes per stay, pediatric exclusion, non-ED admission exclusion, a stay with no suspected infection (excluded), and a NULL icu_outtime case (flag unassessable) — all pass, delays match hand computation.
  5. Ran data_quality_report.py against the fixture database (with a raw-schema coverage denominator): Markdown/JSON outputs verified against hand-computed values (4/5 = 80% capture, flag counts, medians/p25/p75).

Notes for reviewers

  • Only mimic-iv/concepts/ sources are touched. The transpiled concepts_postgres/concepts_duckdb outputs are intentionally not included — the regenerate-dialects bot produces those on merge. The duckdb.sql/postgres-make-concepts.sql build lists are likewise left for maintainers to extend.
  • The module depends only on mimiciv_hosp + mimiciv_icu + derived concepts (no MIMIC-IV-ED module), so it runs on the standard build. ED arrival time is deliberately not used; the README notes this as a limitation.
  • Draft PR — happy to adjust naming, column choices, or the onset-window thresholds to fit maintainer conventions.

…uality report)

New `mimic-iv/concepts/diagnosticdelay/` module:

- suspected_sepsis_cohort.sql: one row per ICU stay for adults admitted
  via the ED with >= 1 suspected-infection episode (antibiotic +
  microbiology culture pair, per suspicion_of_infection).
- diagnostic_delay.sql: delay in hours between admission (hospital/ICU)
  and first recognition of infection (suspected-infection time, first
  antibiotic, first culture); onset-window classification
  (pre_admission / present_on_admission / hospital_onset); and
  temporal-plausibility flags for out-of-order or missing timestamps.
- data_quality_report.py: renders a Markdown/JSON data-quality report
  (cohort coverage, missingness, flag prevalence, delay distributions)
  from the two tables above; runs on DuckDB or PostgreSQL.
- README.md documents the cohort definition, measures, validation,
  and limitations.

ICD codes carry no timestamp in MIMIC, so diagnosis time is proxied by
the clinically observable recognition events used in the Sepsis-3
definition (see MIT-LCP#1843); ordering checks surface implausible values of
the kind reported in MIT-LCP#2168 instead of silently averaging them in.

Validated without credentialed MIMIC-IV access: sqlglot parse (BigQuery
dialect, same as CI), sqlfluff 4.1.0 with the repo config, transpile to
DuckDB/PostgreSQL via mimic_utils, execution of the DuckDB build on
hand-built synthetic fixtures (16 assertions), and a run of the report
script against the fixture database. End-to-end run on real MIMIC-IV
data is still outstanding.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant