Skip to content

Fix silent column collisions and accounting-style negatives in ingest - #248

Merged
shivamlalakiya merged 4 commits into
mainfrom
f3-ingest-hygiene
Sep 26, 2026
Merged

shivamlalakiya merged 4 commits into
mainfrom
f3-ingest-hygiene

Conversation

@shivamlalakiya

Copy link
Copy Markdown
Contributor

What

Two bugs in the ingest helpers:

  1. map_columns let a mapping send two different source columns to the
    same target name. The rename went through silently and produced a
    DataFrame with two identically named output columns, which could
    still pass a required= check because the column name existed, just
    twice. It now raises a ValueError naming the colliding source
    columns before the rename happens.

  2. _to_amount (shared by the CiviCRM, Raiser's Edge, and NPSP gift
    readers, since the latter two build on civicrm_contributions_to_features
    rather than reimplementing amount parsing) stripped parentheses along
    with every other non-numeric character, so an accounting-style
    negative like "($50.00)" parsed as positive 50 instead of -50. It
    now converts a parenthesised amount to a leading minus sign first.

Why

Both are silent-corruption bugs: a mis-mapped column passes validation
and a refund or credit note gets summed into a donor's lifetime giving
as a positive gift.

How tested

Added a regression test for each bug and confirmed both failed before
the fix:

  • tests/test_map_columns.py::test_raises_on_collision_naming_the_colliding_sources
  • tests/test_civicrm.py::test_to_amount_parses_signed_and_formatted_values
    (parametrized over "($50.00)", "(50)", "-25.50", "$1,000.00")

Ran the full targeted suite for the touched readers to confirm nothing
else changed meaning:

python -m pytest tests/test_map_columns.py tests/test_civicrm.py tests/test_raisers_edge.py tests/test_npsp.py
163 passed

make ci and make riskcov both pass locally (2200 passed, 30 skipped,
98.35% coverage against a 92% floor; 98% risk-tier coverage against a
93% floor).

CHANGELOG.md updated under [Unreleased].

map_columns let two source columns map to the same target name and
silently produced duplicate-named output columns that could still pass
a required= check. It now raises a ValueError naming the colliding
sources.

_to_amount, shared by the CiviCRM, Raiser's Edge and NPSP readers, was
parsing accounting-style negatives like "($50.00)" as positive because
it stripped parentheses before the digits without checking for the
minus sign they imply. It now converts a parenthesised amount to a
leading minus sign first.
@shivamlalakiya
shivamlalakiya merged commit 4de6a1a into main Sep 26, 2026
15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant