Add bundle_strep (Group A + Group B Strep) and the ABCs Strep sources

[MadiBal]

Adds bundle_gas, plus the two ABCs Strep sources it needs. Supersedes #187.

Stacked on #188. Base is gas-respiratory, so this diff excludes the epic_resp_infections refresh. Merge #188 first and GitHub retargets this to main automatically.

@DanWeinberger — two shape questions for you in Open questions below, plus a data-quality oddity. @anshap32 — the ABCs findings in What checking Group A turned up are the substantive part.

---

What's here

bundle_gas — 13 long-format parquets in dist/, one per contributing source standard file. Every parquet shares the same shape:

Column	Notes
geography	state name, or "United States" for the national total
geography_fips	matching FIPS code ("00" national), kept so these join to FIPS-keyed data
time	ISO YYYY-mm-dd period end
measure	which measure the row reports
value	the plotting column

plus each source's dimension columns (age, sex, race_ethnicity, onset, population_group) and suppressed for Epic.

Parquet	Source	Rows
epic_gas.parquet	epic_resp_infections	39,312
nnds_stss.parquet	nnds	24,856
abcs_gas{,_counts,_syndromes,_resistance,_emm}.parquet	abcs_gas	644 / 84 / 107 / 206 / 1,128
abcs_gbs{,_counts,_syndromes,_resistance,_serotypes,_alph}.parquet	abcs_gbs	812 / 140 / 243 / 719 / 1,260 / 286

abcs_gbs (new) — ABCs Group B Streptococcus, CDC dataset 95m5-agj4, national, annual 1997–2024. ABCs publishes one dataset per pathogen and only Group A was being read in. GBS is organized differently, so the ingest isn't a copy: infant onset timing is central (early-onset 0–6 days vs late-onset 7–89 days), a population_group column replaces sex, and serotypes plus ALPH typing stand in for emm types.

abcs_gas — brought onto main-line development from the stale gas branch and substantially rewritten (below). Note data/abcs is pneumococcus, a different pathogen despite the similar name.

---

What checking Group A turned up

Re-running the existing abcs_gas ingest against current CDC data does not just add a year. Three real problems:

1. CDC restructured the 2026 release, changing meaning. emm types was one topic whose viewby held the type; now each type is its own topic (emm 1, emm 12, …), viewby holds that type's isolate count, and value is a proportion despite units reading "Percent". The old parsing keyed on the vanished topic and produced an empty emm file. Separately, syndromes went from percent-of-cases (1997+) to rate per 100,000 (2001+), so those columns are now abcs_gas_rate_syndrome_* — carrying rates under a pct_ name would mislead silently.

   Verified against the old release: 1997 emm 1 arrives as viewby=99, value=0.225; per-type counts sum to 440; 99/440 = 0.225 = the 22.5 percent previously reported on 440 isolates.

2. A 100× scale error. Every "Percent" value is now a proportion — 2023 clindamycin arrives as 0.264 where the old release said 26.4. Caught by diffing against the old committed file; resistance and emm now both reproduce the old values exactly.

3. The ≥65 age band was being silently dropped. Under the C locale dcf_process runs in, that non-ASCII literal can't be translated for comparison, so all 28 rows vanished. Labels are now stripped to ASCII before matching.

Each of these now fails loudly: the script stop()s if "Percent" values exceed 1.5, if the emm or syndrome topics stop matching, or if no 65+ rows survive — rather than writing an empty or 100×-wrong file.

Also picked up: GAS 2024 data (the repo's raw file was from May 2025), linezolid resistance, and a survivals count.

Counts are split into their own data_counts.csv.gz in both sources. Folded in, they left columns ~85–93% NA — sparse enough that vroom's type guessing infers logical and blanks out real values. This was observed, not theoretical: 140 real GBS values silently became NA before the split.

---

Open questions

1. Parquet shape — @DanWeinberger. These are fully long (one row per measure). Looking at bundle_childhood_immunizations/overall_rates_by_source.parquet and bundle_youth_wellbeing/yrbss_state_age_demographics.parquet, those are noticeably less tall, so it's unclear whether the Epic and NNDSS outputs should be long, semi-wide, or just a parquet copy of the standard file. Happy to reshape once there's a house rule.

2. NNDSS long vs wide, and negative values — @DanWeinberger. NNDSS publishes STSS as a cumulative year-to-date count that resets each MMWR year (national 2024 runs 5 → 647 across weeks 1–52), so build.R de-accumulates it into stss_cases_weekly and keeps stss_cases_cumulative alongside. Two questions: should those be two measure levels or two columns, and what should happen to the 27 negative weekly increments? They come from NNDSS revising earlier weeks downward. Currently left as reported rather than clamped to zero, with build.R logging the count — clamping is a one-line change.

3. 13 parquets is a lot. One per source standard file, following the earlier "keep the upstream split" decision. Reasonable place to push back if consolidation is preferred.

---

Verification

• dcf::dcf_process clean on all three projects; bundle rebuild is deterministic (re-running produces no diff).
• emm percents sum to 100.00 ± 0.02 in all 28 years, and per-type counts reconcile to the typed total. GBS serotypes sum to 99.7–100.2, ALPH to 99.8–100.2; resistance within 0–100.
• All 11 standard files parse with zero vroom problems.
• dcf_check: 0 undocumented columns on abcs_gas (was 36 — the gas-branch measure_info used generic keys matching no actual column name), abcs_gbs, and epic_resp_infections.
• Bundle measure_info.json documents all 78 columns across 13 parquets — 0 undocumented, 0 falling back to STANDARD_VARS defaults — and all 97 measure levels resolve against the contributing sources. Includes a _bundle block mapping each parquet to its source file.
• geography ↔ geography_fips is 1:1 in every parquet, national always "00".
• docs/ and resources/ regenerated (14 bundles, 46 sources); every bundle_gas column renders with a name and a description or value list.

Caveats worth carrying forward

• ABCs covers a catchment of roughly 35 million people, not the whole US — rates are observed within the catchment and "national" counts are CDC's population-weighted extrapolations.
• value mixes units within a parquet; always filter or facet by measure first.
• Group A and Group B are different diseases with different at-risk populations — don't pool them.
• Three different strep-toxic-shock series now exist (NNDSS counts, ABCs rate per 100,000 from 2001, and the older ABCs percent-of-cases) and are not interchangeable.

🤖 Generated with Claude Code

[MadiBal]

Updated per review feedback — force-pushed and retargeted to main

The feedback checked out against the data, including one item that was a genuine bug. Rebuilt on current main (#188 having merged) as two clean commits, so the diff no longer contains the add-then-delete churn of the previous version.

1. Consolidated into data/abcs ✅

Group A (9y49-tura) and Group B (95m5-agj4) now run from the existing abcs source's single ingest.R, with each download tracked separately in process.json. The pneumococcal data.csv.gz / uad.csv.gz are left structurally untouched. data/abcs_gas and data/abcs_gbs are gone.

One correction to the premise: "group A and group B have the same format" is true of the raw CDC files but only partly of the standardized outputs. Rates, counts and resistance merge cleanly and now sit in single files keyed by a pathogen column with organism-agnostic measure names (abcs_rate_cases, not abcs_gas_rate_cases — per-organism prefixes would force half of every row to NA). But syndromes can't merge (Group A is a rate per 100,000, Group B a percent of cases) and neither can typing (emm types vs capsular serotypes are different concepts). So: 8 strep files, down from 11.

2. The Group B counts bug was real ✅

You were right. The old output kept onset but dropped age, so the infant rows and the all-ages row were indistinguishable:

1997  <1     Early-onset    2,600
1997  <1     Late-onset     1,300
1997  Total  Overall       16,600   (= 1,600 deaths + 15,000 survivals)

Different populations — summing them was an accident waiting to happen. As you spotted, CDC's viewby fields conflate two things (Infants, early-onset disease), so both are now decomposed explicitly, the same way the rates path already did. Duplicate index rows are asserted against after every pivot.

population_group is gone entirely, for the same reason — its five levels conflated an age band with onset timing, and collapsing to age alone would map both infant levels onto <1 and create duplicate keys. It decomposes to age + onset throughout, which also removes the embedded commas that were forcing CSV quoting.

3. Not-reported flags ✅

abcs_not_reported_flag_* marks values CDC never published, distinct from confidentiality suppression. Nothing is imputed, so a flagged row must not be read as a zero. One flag per group of columns that go missing together — N_deaths and N_survivals share one, as you suggested, since they're absent on exactly the same 56 of 84 rows.

The NA patterns turned out highly structured (1–6 distinct patterns per file), so this is ~15 flags, not a column explosion. Reasons are documented per flag: death rates aren't published by sex/race, tetracycline and linezolid are on the Group A panel only, CDC itemises different emm types each year, and so on.

The ingest now errors if any column has NAs without a covering flag, so a future change in what CDC publishes fails loudly instead of shipping undocumented gaps.

4. Ages standardized ✅

Total as the all-ages aggregate — matching the pneumococcal output already in this folder, and your wording. Bands cleaned to <1 / 1 / 2-4 / 5-17 / 18-34 / 35-49 / 50-64 / 65+ for rates and the coarser <1 / 18-64 / 65+ for resistance and typing, as you asked. The typo "65+ years olds" is gone.

Two notes: this diverges from CLAUDE.md's Overall — local consistency within the folder won, flag it if you'd rather go the other way. And your "weird they don't have 2-17" is a real source quirk (ABCs concentrates Group B reporting on neonates and older adults); it's documented rather than fixed, along with the fact that ABCs' 18-64/65+ split doesn't line up with the 18-49/50-64 bands used elsewhere in this repo.

Verification

The restructure changed arrangement only. For every one of the 11 previous outputs, the melted (time, measure, value) triples are identical old vs new — same count, same measures, same values. Plus:

• emm percents still sum to 100.00 ± 0.02 in all 28 years; per-type counts still reconcile to the typed total
• 2015/2023 Group A resistance still reads 13.1/14.9/17 and 26.4/27.4/32.1
• Group B serotypes and ALPH still sum to ~100 per group-year; all percents within 0–100
• every file parses with zero vroom problems; no duplicate index rows
• two consecutive runs produce identical hashes across all 20 outputs
• dcf_check("abcs"): 0 undocumented columns (measure_info went 7 → 106 entries, which also closed a pre-existing gap where pop was undocumented)
• bundle: all 86 columns documented, 0 via STANDARD_VARS defaults, all 86 measure levels resolving

Bundle changes

Ten parquets instead of thirteen, mirroring the consolidated sources — which also answers the "13 parquets is a lot" question I'd raised. not_reported is carried through, and I caught a bug in my own first attempt at that: a grouped flag is 1 wherever any covered column is missing, so propagating it per-measure over-marked rows that did have values (420 flagged vs 280 actual). In long format the correct per-row flag is simply whether that row's value is absent; verified to equal is.na(value) in every parquet.

Two incidental fixes

• Dropped the unused library(cdlTools) from abcs/ingest.R — never invoked there (geography comes from the all_fips merge) but it blocked the ingest anywhere that package isn't installed.
• The pneumococcal data.csv.gz is rewritten with last-digit float differences in pop/rate_IPD only — N_IPD and pct_IPD are identical on all 176 differing lines — because the committed copy was generated in a different environment. Structurally unchanged, as asked.

Still open for @DanWeinberger

1. Is age + onset the right shape for the ABCs strep files, or would you prefer a single age-like category? Flagged by the reviewer as your call. Decomposing is what keeps the index unique — collapsing onset into age gives two <1 rows per year.
2. Parquet shape — these are fully long; bundle_childhood_immunizations/overall_rates_by_source.parquet and bundle_youth_wellbeing/yrbss_state_age_demographics.parquet are less tall.
3. The 27 negative NNDSS weekly increments — from CDC revising earlier weeks downward. Currently left as reported and logged; clamping to zero is a one-line change.
4. New: the Total vs Overall aggregate-label divergence from CLAUDE.md.

Also worth knowing for the state-level question that came up separately: NNDSS carries STSS and nothing else for Group A — invasive Group A disease generally and Group B disease aren't nationally notifiable, so there's no broader NNDSS series to draw on. 647 STSS cases in 2024 against ABCs' ~41,400 estimated invasive Group A cases.

🤖 Generated with Claude Code

[DanWeinberger]

(1) On Bundle files, we name the time variable as 'date'
2) I think the bundles could be further consolidated. Rather than have a separate table for each of the stratification variables, you can stack these and just include a 'Total' category if that particular slice is not stratified for that variable (so you would have a column for age, syndrome, resistance, etc all on same file. likewise can combined GAS and GBS on same file if they have same structure; just have a column for the specific pathogen, similar to what we have for flu/covid/rsv

3. Parquet files should generally be long. The immunization one makes sense to be slightly wider--that ensures that value, the CIs, the percent surveyed, and the sample size are kept together on 1 line, which makes sense for interpretation and for labeling. One way to think of it is if there is information that should be kept together for a tooltip on a graph, then it should be on 1 line on te parquet file

4. For the negative values--follow the same convention we used for the measles bundle. I don't remember what we settled on for that. I think maybe we maintain the negative values for transparency but on the plots, cut the axis to 0.

5. total vs overall label: use what we are using for the other parts of the database

6. Age+ onset: I would keep a separate column for both but also add a 'total' row for each age group, adding the early+late.

[MadiBal]

Updated for @DanWeinberger's review — force-pushed, rebased on current main

All six points are in. I checked each against the repo before changing anything, which turned up two useful things: one point was already compliant, and one couldn't be followed literally without creating contradictory data. Details below.

Ten parquets are now three.

Parquet	Rows	Grain
abcs_strep.parquet	6,101	national, annual — all eight ABCs topics stacked
epic_gas.parquet	39,312	state + national, quarterly, by age
nnds_stss.parquet	24,856	state + national, weekly (MMWR)

---

1. date, not time ✅

Confirmed the split: standard files use time (38 of 40 sampled; none use date) and bundles rename it in build.R, exactly as bundle_measles and bundle_respiratory do. Renamed in the dist files only. Also added year, plus week for NNDSS, following bundle_measles which carries date/year/week together.

Worth flagging: your convention isn't uniform yet — 46 of 64 existing dist parquets still use time, only 18 use date. Might be worth a separate repo-wide sweep.

2. Consolidated ✅

All eight ABCs topics stack into one file with a named column per stratification and "Total" wherever a row isn't stratified on that dimension, so every column is populated and filterable without NA handling:

geography, geography_fips, date, year, pathogen,
age, sex, race_ethnicity, onset,                        <- stratifications
syndrome, antibiotic, emm_type, serotype, alph_type,    <- which entity
measure, value, n_type, n_isolates, not_reported

Epic and NNDSS stay separate — different geography grain, time resolution and measure.

One nuance on "group A and group B have the same format": true of the raw CDC files, only partly true of the outputs. Rates, counts and resistance share a schema and merge cleanly. But Group A syndromes are a rate per 100,000 while Group B's are a percent of cases, and emm types and capsular serotypes are different concepts. So they share the file but keep distinct measure levels and entity columns rather than being forced into common columns.

3. Long, with tooltip info on one line ✅

Took the pneumococcus_by_geography.parquet value, value_N pattern as the model. n_isolates (denominator) and n_type (emm numerator) sit on the same row as the percentage, so a tooltip reads "emm1: 22.5% (99 of 440 isolates)" from one line.

Cases, deaths and survivals stayed as separate measure levels — by your rule they're three plottable series, not metadata for a single value.

4. Negative values — already matching measles ✅

Checked bundle_measles: it de-accumulates with value - lag(value, default = 0), exactly as this does, and keeps the negatives (11 rows, min −21.44) with no clamping. So this was already compliant; I've just documented it explicitly, including that plots should cut the y axis at 0. Nothing changed in the data.

5. Total vs Overall ✅

Total outnumbers Overall 40 to 3 across bundle dist files (only bundle_maternal_health uses Overall). Now Total for sex, race_ethnicity and onset as well as age.

Flagging that CLAUDE.md documents Overall — it's the outlier, and probably the reason this keeps recurring. Worth correcting the doc.

6. Onset totals — done for counts, deliberately not for rates ⚠️

This is the one I couldn't implement as written. Adding a Total row summing early + late is valid for counts and wrong for rates, because CDC labels both "Per 100,000 population" while using different denominators:

1997 Group B	Value	Denominator
<1 age band	115.7	per 100,000 infants (3,900 cases ÷ ~3.9M births × 100,000 ≈ 100 ✓)
early-onset + late-onset	0.70 + 0.40 = 1.10	per 100,000 general population (3,900 ÷ ~272M × 100,000 ≈ 1.4 ✓)

They differ in all 28 years, mean absolute difference 66.3. Summing would have put 1.10 next to CDC's published 115.7 as two contradictory infant rates.

So:

• Counts — derived age = "<1" / onset = "Total" row added, = early + late (1997: 3,900). Genuinely new information, since CDC publishes the parts but no combined infant figure. Summing is valid because counts carry no denominator.
• Rates — no derived total. The onset-specific rates moved to their own measure, rate_cases_by_onset, so the two denominators never share a column. The two are mutually exclusive by row and the ingest asserts it.

This also fixes a latent defect: the previous strep_rates already mixed both denominators under one abcs_rate_cases column. Shout if you'd rather have it a different way — a rate_denominator column was the alternative.

---

Verification

The consolidation changes arrangement only. Every value multiset from the previous ten parquets is reproduced exactly — rates, counts, resistance, both syndrome sets, emm percents and per-type counts, serotypes, ALPH, and the isolate denominators — with the 28 derived count rows as the only addition.

Also asserted: the two rate measures never co-occur on a row and CDC's 115.7 survives; the derived infant count equals early + late in all 28 years with the all-ages row untouched; no Overall survives; dimension columns have zero NAs; no dist file has a time column; not_reported == is.na(value); no duplicate index rows; zero vroom parse problems; dcf_check clean on abcs (107 entries); all 34 bundle columns documented with 17 measure levels resolving.

Two fixes found along the way

• The ingest's own unflagged-NA guard caught that splitting the rate measure left both columns sparse. That's mutual exclusivity, not a reporting gap, so the guard now takes an exclusive declaration and asserts it rather than demanding a flag that would misdescribe the gap.
• The bundle parquets weren't byte-reproducible: vroom attaches spec and problems attributes that ride through dplyr into the parquet as R metadata, so identical data produced different bytes on each build. Stripped before writing — a forced re-download now reproduces all 13 output hashes exactly.

Still open

• antibiotic is the column name for the resistance entity rather than resistance, since it holds a drug name. Easy to rename.
• Whether Epic and NNDSS should also fold into one file with a source column — I kept them separate given the grain differences, but bundle_measles/bundle_respiratory do stack sources that way, so say the word.

🤖 Generated with Claude Code

[anshap32]

A few small updates necessary in the standard data in the abcs data folder:

• gas_emm.csv and gas_syndromes.csv shouldn't have any NAs. Fill these in with 0 (and make sure flag corresponds to these 0s).
• gbs_alph.csv and gbs_serotypes also still has NAs and needs additional flags. there should be separate flags for each variables -- for example, row 19 in gbs_serotypes has some NAs and some 0s and the flag is 1, so that doesn't tell me which are true 0s vs which aren't reported.
• strep_counts.csv also still has NAs
• in strep_rates.csv, abcs_rate_cases_by_onset and abcs_rate_cases have NAs but no flag. add flag and set NAs to 0. I think this is due to issue 6 you highlighted above. I think I'm in favor of having a denominator column rather than splitting them out, but this is pretty arbitrary
• strep_resistance.csv also still has NAs

[MadiBal]

Standard-data updates: no NAs, one flag per measure

All five points are in. Every strep standard file now has zero missing values, and each measure carries its own flag so a 0 can be told apart from an unreported cell.

File	NAs before	NAs now
strep_rates	1,400	0
strep_counts	168	0
strep_resistance	419	0
gas_syndromes	13	0
gbs_serotypes	280	0
gbs_alph	14	0
gas_emm	76	0
gbs_syndromes	0	0

Per-measure flags

80 flags across the eight files, covering all 80 measures. The flag name is derivable from the measure — abcs_gbs_pct_serotype_ia pairs with abcs_not_reported_flag_gbs_pct_serotype_ia. Every measure gets one, including those CDC always reports (whose flag is all zeros), so you never have to work out whether a flag exists for the column you want.

Your gbs_serotypes row 19 example is exactly why the grouped flags couldn't work. It now reads:

serotype	value	flag
ia, ib, iii, v	25.8, 8.1, 48.4, 17.7	0	reported
ii, iv, non_typeable, vi	0	0	true zero
vi_vii_viii_or_ix, vi_viii	0	1	not reported

Each column answers for itself instead of one serotype_vi_grouping flag saying only "something here is missing".

The rate denominator — folded back together, as you preferred

abcs_rate_cases_by_onset is gone. strep_rates now carries a single abcs_rate_cases plus a rate_denominator column:

• "Stratum population" — per 100,000 of the group the row describes (728 rows)
• "Population" — per 100,000 of the whole population regardless of stratum, which is how CDC reports the infant onset rates (224 rows)

The underlying hazard is unchanged: 1997 Group B reads 115.7 for the <1 band against 0.70 + 0.40 = 1.10 for early + late onset, differing in all 28 years with mean absolute difference 66.3. The distinction just rides on the index now instead of on two measure names.

I used "Population" rather than "Total population" so the level can't be confused with the "Total" not-applicable fill — those two sitting side by side in the same column was too easy to misread.

One caveat, stated plainly

A zero-filled cell asserts a value CDC never published. "0% of Group B isolates resistant to tetracycline" is not true — tetracycline isn't on the Group B panel. So a consumer that ignores the flag now gets a materially wrong number where before it got a blank, and the direction of the error is consistent: reading flagged rows as zeros understates resistance, syndrome rates and type shares.

The per-measure flags make that fully recoverable, and it's called out in both READMEs and in every flag's measure_info entry. Flagging it because it's the one thing that got riskier rather than safer in this round.

Bundle changes to match

• not_reported is read off the source flag rather than inferred from a missing value.
• rate_denominator joins the stratification columns ("Total" where the row isn't a rate).
• The mutually-exclusive-measure handling is gone, since there's no longer a split measure.
• n_isolates and n_type are blanked where their source flag is set rather than carried as zeros — a denominator of 0 would read as broken in a tooltip ("22.5% of 0 isolates").

Verification

All 80 measures check out three ways: unflagged values are identical to the previous non-NA values, each flag count equals the previous NA count for that column, and every flagged cell reads 0. The folded rate measure reproduces the union of the two split columns exactly (952 values) with CDC's published 115.7 intact.

Also asserted: zero NAs anywhere; 1:1 measure/flag pairing in every file; flags binary; no rate_denominator level a prefix of another; the derived infant count still equals early + late in all 28 years with the all-ages row untouched; dcf_check clean on abcs (185 entries); all 35 bundle columns documented with 16 measure levels resolving; and all 13 outputs reproducing identical hashes through a forced re-download.

🤖 Generated with Claude Code

[anshap32]

Questions for @DanWeinberger

• if a variable doesn't have any missing values, using gas_emm for example, do we need a flag? Some of the variables had missing values and some did not, but currently all have flags.
• if we only have one value for things like age group or onset, do we need those variables? example is gas_syndromes, both age and onset are total. gbs_syndromes doesn't have any missing values so I think it doesn't need any flags?
• should we separate out the <1 ratesfrom the other counts into two separate datasets? they have different denominators, currently indicated by a denominator variable

Questions for @MadiBal:

• do you know what the difference between clindamycin** and clindamycin*** is? It looks like just age groups, but want to make sure we can combine
• Did you confirm number of isolates in strep resistance is the denominator? It is missing in a lot of years.
• Some of the measure info long descriptions are quite long. For example, we don't need to say "Every measure in this source has its own flag, including measures CDC always reports (whose flag is all zeros), so a consumer never has to work out whether a flag exists for the column it cares about - and a shared flag could not distinguish a genuine zero from an unreported one where both occur on the same row" or "Reading a flagged 0 as a measured zero understates resistance, syndrome rates and type shares, so filter on this column before aggregating."
• I would read through the measure info long descriptions to confirm accuracy. For example, it says "Reasons a cell goes unreported are structural: an antibiotic off that pathogen's susceptibility panel, an emm type or serotype CDC pooled into "other" that year, a rate not broken out for that stratification." From what I can see, there aren't 'other' categories (except for race). Did you use the values in Topic == Other in GAS? I didn't see them

[DanWeinberger]

If a variable does not have missing values, you do not need a flag if there is only 1 group, you don't need the variable. however, this might cause issues when combining things in bundles if some of the datasets have the variables and others don't. the place to deal with that would be on the script creating the bundles I think it is OK to have outcomes with different definitions on a bundle file as long as there is a variable indicating what the outcome is

…

--- Dan Weinberger Professor of Epidemiology Yale School of Public Health

On Tue, Aug 11, 2026 at 9:42 AM anshap32 ***@***.***> wrote: *anshap32* left a comment (PopHIVE/Ingest#189) <#189 (comment)> Questions for @DanWeinberger <https://github.com/DanWeinberger> - if a variable doesn't have any missing values, using gas_emm for example, do we need a flag? Some of the variables had missing values and some did not, but currently all have flags. - if we only have one value for things like age group or onset, do we need those variables? example is gas_syndromes, both age and onset are total. gbs_syndromes doesn't have any missing values so I think it doesn't need any flags? - should we separate out the <1 ratesfrom the other counts into two separate datasets? they have different denominators, currently indicated by a denominator variable Questions for @MadiBal <https://github.com/MadiBal>: - do you know what the difference between clindamycin** and clindamycin*** is? It looks like just age groups, but want to make sure we can combine - Did you confirm number of isolates in strep resistance is the denominator? It is missing in a lot of years. - Some of the measure info long descriptions are quite long. For example, we don't need to say "Every measure in this source has its own flag, including measures CDC always reports (whose flag is all zeros), so a consumer never has to work out whether a flag exists for the column it cares about - and a shared flag could not distinguish a genuine zero from an unreported one where both occur on the same row" or "Reading a flagged 0 as a measured zero understates resistance, syndrome rates and type shares, so filter on this column before aggregating." - I would read through the measure info long descriptions to confirm accuracy. For example, it says "Reasons a cell goes unreported are structural: an antibiotic off that pathogen's susceptibility panel, an emm type or serotype CDC pooled into "other" that year, a rate not broken out for that stratification." From what I can see, there aren't 'other' categories (except for race). Did you use the values in Topic == Other in GAS? I didn't see them — Reply to this email directly, view it on GitHub <#189?email_source=notifications&email_token=ANPQWPEFDYPDLEAKRFRJFWD5JMPFFA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTKMRVGM4TOMZSGQ22M4TFMFZW63VHNVSW45DJN5XKKZLWMVXHJLDGN5XXIZLSL5RWY2LDNM#issuecomment-5253973245>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ANPQWPGKVKA3NEW2E6GUHMD5JMPFFAVCNFSNUABGKJSXA33TNF2G64TZHMYTAMJYGA3DSNZUG45US43TOVSTWNJQHA2DMMJVGI2DLILWAI> . Triage notifications, keep track of coding agent tasks and review pull requests on the go with GitHub Mobile for iOS <https://github.com/notifications/mobile/ios/ANPQWPEGQADLUOBDO64GFR35JMPFFA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTKMRVGM4TOMZSGQ22M4TFMFZW63VHNVSW45DJN5XKKZLWMVXHJKTGN5XXIZLSL5UW64Y> and Android <https://github.com/notifications/mobile/android/ANPQWPBV4X257PVCQSOTDL35JMPFFA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTKMRVGM4TOMZSGQ22M4TFMFZW63VHNVSW45DJN5XKKZLWMVXHJLTGN5XXIZLSL5QW4ZDSN5UWI>. Download it today! You are receiving this because you were mentioned.Message ID: ***@***.***>

[anshap32]

More comments on bundle for @MadiBal

• We should probably rename the bundle to something like bundle_strep, as it is no longer just group A.
• There are still many NAs in abcs_strep.parquet. I think we will likely need multiple missing flags -- for example, I see instances of NAs in n_type but not the others, and the not_reported flag is 0.

[MadiBal]

Went through all six. Pushed as 9fd5112.

1. The asterisks are footnote markers, not separate measures. From CDC's metadata: ** is "small numbers may make year to year changes difficult to interpret", *** is "information available to display differs by individual bacterium". It's a year split, not an age one (GBS adults 18–64 use *** through 2018, ** after), and the series doesn't jump across it. Fine to combine. The ingest was already stripping them.

2. No, couldn't confirm it. CDC leaves the column descriptions blank, so all we have is that it sits next to the percentages. Coverage is thinner than we'd written too: GAS only from 2006, and GBS gives one overall count against five strata, so 910 of 1,176 resistance rows have no denominator. I've reworded it as the apparent denominator. @anshap32 can you check it against the ABCs report?

3. Cut both sentences. Average long_description is down from ~790 characters to ~270. The repeated catchment and scaling text now lives once in _sources.

4. Half right. GAS does have a Topic == "Other" and we're using it (abcs_gas_emm_pct_other), and the percents sum to 100 including it. But no "other" for serotypes, you're right. CDC just regrouped the rare ones over time (VI, then "VI, VIII", then "VI, VII, VIII or IX"). Fixed. Also caught while in there: the Epic age description was copy-pasted from ABCs, N_cases still said onset "Overall", and 12 orphaned flag entries were left over from the old design.

5. Done, bundle_strep. Docs and manifest regenerated.

6. Two different things were sharing one NA: either the measure has no such companion (a case rate has no isolate denominator), or CDC didn't publish it. That's the n_type blank with the flag at 0 you were seeing. Each companion now carries a status instead:

	reported	not_reported	not_applicable
n_isolates	2,504	910	2,687
n_type	550	38	5,513

not_reported is now value_not_reported, since that's all it described. build.R asserts the two stay consistent.

Title and description still need updating for the rename, will do that next.

[anshap32]

1. Makes sense
2. I would recommend looking into the data dictionary/data page on CDC web page to figure out what it means. In the future, this is something that should be double checked, rather than left to AI to determine.
3. Can you read through the measure info? A lot of the information just moved but still doesn't need to be there. I would recommend looking through all the variables. A lot still sound very clearly written by AI, so I don't 100% trust it. As best practice, we can let the skill generate the measure info but you should be reading through and editing each one if necessary. See data dictionary for examples: https://pophive.github.io/Ingest/#
4. I don't quite understand your response. Did you confirm that Topic == "Other" does actually correspond to emm = other? It isn't apparent to me that it is from the data.
5. There are still many NAs in abcs_strep. Please recode them to 0, as we do in the data. There should not be any NAs in our files as it can mess with the figure generation on the webpage. Please confirm that this change is propagated through the entire file.

[anshap32]

Sorry, I was wrong about replacing the NAs with 0s! You can either leave them as NAs or code them something like -999.