Add a multi-reference version of the existing one-to-one indicator comparison report by jingjtang · Pull Request #7 · cmu-delphi/analysis-notebooks

jingjtang · 2026-07-31T19:35:52Z

The new report evaluates one candidate indicator against multiple reference indicators in a single render. Each diagnostic remains pairwise.

Preview

Rendered HTML preview

Main Changes

Added multi-reference parameters for reference sources, signals, names, and optional CSV inputs.
Reused the core logic from indicator_comparison.qmd for:
- exploratory data checks
- coverage and missingness
- coefficient of variation comparison
- spatial correlation over locations
- temporal correlation within locations
- lagged correlation analysis
- geographic agreement at optimal lag
Added matrix-based Spearman correlation helpers to compute candidate-vs-multiple-reference correlations more efficiently than repeatedly rendering one-to-one reports.
Added a reference-agreement context check to show agreement among selected reference indicators.
Standardized table formatting and plot labels for automated reports with variable numbers of references.

Validation

Rendered successfully with:

#!/usr/bin/env Rscript

options(repos = c(
  delphi = "https://cmu-delphi.r-universe.dev",
  CRAN = "https://cloud.r-project.org"
))

if (!require("pacman")) install.packages("pacman")
pacman::p_load(quarto, here, fs)

data_dir <- here::here("indicator_analysis", "data", "multi_comparison_render")

candidate_csv <- fs::path_abs(fs::path(data_dir, "candidate_doctor_visits.csv"))
reference_csvs <- fs::path_abs(c(
  fs::path(data_dir, "ref_quidel.csv"),
  fs::path(data_dir, "ref_hospital_admissions.csv"),
  fs::path(data_dir, "ref_chng_inpatient.csv"),
  fs::path(data_dir, "ref_chng_outpatient.csv")
))

missing_files <- c(candidate_csv, reference_csvs)[!file.exists(c(candidate_csv, reference_csvs))]
if (length(missing_files) > 0) {
  stop(sprintf("Missing CSV snapshots:\n%s", paste(missing_files, collapse = "\n")))
}

old_wd <- setwd(here::here("indicator_analysis"))
on.exit(setwd(old_wd), add = TRUE)

quarto::quarto_render(
  input = "indicator_multi_comparison.qmd",
  output_file = "indicator_multi_comparison.html",
  cache_refresh = TRUE,
  execute_params = list(
    candidate_source = "doctor-visits",
    candidate_indicator = "smoothed_adj_cli",
    candidate_name = "Doctor Visits: Smoothed Adj CLI",
    candidate_csv = candidate_csv,
    reference_sources = c("quidel", "hospital-admissions", "chng", "chng"),
    reference_indicators = c(
      "covid_ag_smoothed_pct_positive",
      "smoothed_covid19_from_claims",
      "7dav_inpatient_covid",
      "7dav_outpatient_covid"
    ),
    reference_names = c(
      "Quidel: Smoothed % Positive COVID Antigen",
      "Hospital Admissions: Smoothed COVID-19 from Claims",
      "CHNG: 7-day Avg Inpatient COVID",
      "CHNG: 7-day Avg Outpatient COVID"
    ),
    reference_csvs = reference_csvs,
    geo_type = "state",
    time_type = "day",
    start_day = "2020-09-01",
    end_day = "2023-03-01",
    max_locations_plot = 12
  )
)

JavierMtzRdz

Thanks for this! It improves almost all sections. Since it doesn't conflict with anything, we can merge once you add some instructions on how to use this to the READMEs and address the suggestions and issues. Those marked as minor suggestions (most of my comments) are more open recommendations. Then, eventually, we could replace indicator_correlation.qmdwith this.

If you also include the processing that uses only the necessary parameter depending on the indicator source and handle edge cases with either one location or many candidates, I'm happy to replace indicator_correlation.qmd with this, but that is not necessary. We can leave the full replacement for later.

JavierMtzRdz · 2026-07-31T22:05:38Z

+  candidate_source: doctor-visits
+  candidate_indicator: smoothed_adj_cli
+  candidate_name: "Doctor Visits: Smoothed Adj CLI"
+  candidate_csv: !expr "NULL"
+  # Reference Indicators
+  reference_sources: !expr 'c("quidel", "hospital-admissions", "chng", "chng")'
+  reference_indicators: !expr 'c("covid_ag_smoothed_pct_positive", "smoothed_covid19_from_claims", "7dav_inpatient_covid", "7dav_outpatient_covid")'
+  reference_names: !expr 'c("Quidel: Smoothed % Positive COVID Antigen", "Hospital Admissions: Smoothed COVID-19 from Claims", "CHNG: 7-day Avg Inpatient COVID", "CHNG: 7-day Avg Outpatient COVID")'


suggestion: I'm inclined to leave these parameters empty to ensure the user provides all the information required for the indicator they are interested in. However, the required indicators vary depending on whether the user is calling an indicator from the V5 API, the new API, or providing a CSV file. The last version of the correlation notebook has some of this.

Agreed. I now left the YAML params empty.

JavierMtzRdz · 2026-07-31T23:55:13Z

+        overlap_metric = factor(overlap_metric, levels = c("Rows loaded", "Locations loaded", "Overlap rows", "Overlap locations", "Overlap dates", "Overlap percent"))
+    )
+
+p_overlap <- ggplot(overlap_long, aes(x = overlap_metric, y = reference_label)) +


question: Is there a reason to prefer a plot over a table for this information?

I changed the overlap metrics from a plot to a table and kept only the date-span plot.

JavierMtzRdz · 2026-08-01T00:02:06Z

I'm aiming to keep the rendered files separate from the notebooks and code. Alternatively, you could include the rendering codes in the script/{post/pre}_render.R scripts to generate the output and then add it to the gh branch.

Before merging into main, delete this file and include the code to generate it within the pre_render script and the YAML. I am adding an action to handle this rendering when a notebook is modified.

Done. I removed the rendered HTML from the PR and added the report generation to scripts/pre_render.R and _quarto.yml.

JavierMtzRdz · 2026-08-01T00:13:58Z

Thanks for this. It looks great!

JavierMtzRdz · 2026-08-01T00:16:19Z

I've also added you to the repo, so there's no need to create forks.

JavierMtzRdz

Thank you for addressing my comments. I have added a few minor points.
Before merging, please remove the rendered example, merge main into this branch to avoid conflicts, add code to render it in script/pre_render.R, add documentation on how to use it in 'indicator_analysis/README.md, and mention it in the main README.md`.

JavierMtzRdz · 2026-08-15T04:19:06Z


-wrap_legend_label <- function(x, width = 26) {
-    stringr::str_wrap(x, width = width)
+wrap_legend_label <- function(x) {


Suggestion: It looks like the function now does the opposite of what it did before. I'd rename it accordingly.

Good point. I renamed the helper to clean_legend_label()

JavierMtzRdz · 2026-08-15T04:26:05Z

Before merging into main, delete this file and include the code to generate it within the pre_render script and the YAML. I am adding an action to handle this rendering when a notebook is modified.

JavierMtzRdz · 2026-08-15T04:41:40Z

+    theme(axis.text.x = element_text(angle = 35, hjust = 1))
+```
+
+### Quantile Trends


Instead of just breaking or printing a message, I'd recommend implementing dynamic metadata sections. This allows you to manage the visibility of sections using rules, but it's not essential.

JavierMtzRdz · 2026-08-15T04:58:03Z

 ggplot(coverage, aes(x = time_value, y = locations, color = availability)) +
    geom_step(direction = "mid", linewidth = 0.5) +
-    facet_wrap(~wrap_reference(reference_name), ncol = 1) +
+    facet_wrap(~stringr::str_wrap(reference_name, width = 30), ncol = 1) +


minor suggestion: use scale = "free_y"

JavierMtzRdz · 2026-08-15T05:15:11Z

 #| fig-height: 5.5
+knitr::opts_current$set(fig.height = scaled_fig_height(n_distinct(lag_summary$reference_label), base = 3.5, per_item = 0.55, min_height = 5.5, max_height = 13))
+
 ggplot(lag_summary, aes(x = lag, y = reference_label, fill = mean_cor)) +


issue: This plot and the map indicate that the maximum correlation for the empty indicator is -21.

Good catch. I now filter to finite correlations before selecting the optimal lag, so empty references should no longer produce invalid values.

jingjtang added 2 commits July 23, 2026 12:20

Add files via upload

b74b08f

add 1-to-many qmd

896e950

JavierMtzRdz requested changes Aug 1, 2026

View reviewed changes

Address review feedback

5b09d0d

jingjtang requested a review from JavierMtzRdz August 10, 2026 13:51

JavierMtzRdz approved these changes Aug 15, 2026

View reviewed changes

jingjtang added 2 commits August 16, 2026 06:51

Merge remote-tracking branch 'origin/main' into one-to-many-corr

48958c0

Add multi-reference indicator comparison report

a7861ac

Uh oh!

Conversation

jingjtang commented Jul 31, 2026

Preview

Main Changes

Validation

Uh oh!

JavierMtzRdz left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

JavierMtzRdz commented Aug 1, 2026

Uh oh!

JavierMtzRdz left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

JavierMtzRdz left a comment •

edited

Loading