Fix silent column collisions and accounting-style negatives in ingest - #248
Merged
Merged
Conversation
map_columns let two source columns map to the same target name and silently produced duplicate-named output columns that could still pass a required= check. It now raises a ValueError naming the colliding sources. _to_amount, shared by the CiviCRM, Raiser's Edge and NPSP readers, was parsing accounting-style negatives like "($50.00)" as positive because it stripped parentheses before the digits without checking for the minus sign they imply. It now converts a parenthesised amount to a leading minus sign first.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Two bugs in the ingest helpers:
map_columnslet a mapping send two different source columns to thesame target name. The rename went through silently and produced a
DataFrame with two identically named output columns, which could
still pass a
required=check because the column name existed, justtwice. It now raises a
ValueErrornaming the colliding sourcecolumns before the rename happens.
_to_amount(shared by the CiviCRM, Raiser's Edge, and NPSP giftreaders, since the latter two build on
civicrm_contributions_to_featuresrather than reimplementing amount parsing) stripped parentheses along
with every other non-numeric character, so an accounting-style
negative like
"($50.00)"parsed as positive 50 instead of -50. Itnow converts a parenthesised amount to a leading minus sign first.
Why
Both are silent-corruption bugs: a mis-mapped column passes validation
and a refund or credit note gets summed into a donor's lifetime giving
as a positive gift.
How tested
Added a regression test for each bug and confirmed both failed before
the fix:
tests/test_map_columns.py::test_raises_on_collision_naming_the_colliding_sourcestests/test_civicrm.py::test_to_amount_parses_signed_and_formatted_values(parametrized over
"($50.00)","(50)","-25.50","$1,000.00")Ran the full targeted suite for the touched readers to confirm nothing
else changed meaning:
make ciandmake riskcovboth pass locally (2200 passed, 30 skipped,98.35% coverage against a 92% floor; 98% risk-tier coverage against a
93% floor).
CHANGELOG.md updated under
[Unreleased].