Skip to content

fix(units): scale amounts with exact integer arithmetic - #208

Open
mehmetkr-31 wants to merge 3 commits into
tempoxyz:mainfrom
mehmetkr-31:fix/parse-units-exact-scaling
Open

fix(units): scale amounts with exact integer arithmetic#208
mehmetkr-31 wants to merge 3 commits into
tempoxyz:mainfrom
mehmetkr-31:fix/parse-units-exact-scaling

Conversation

@mehmetkr-31

Copy link
Copy Markdown

Problem

parse_units scales with d * (10**decimals). That multiplication is evaluated in the active decimal context, whose default precision is 28 significant digits, so an amount longer than that is rounded before the integrality check runs. The rounded value is itself integral, so result != int(result) does not fire and a silently wrong base-unit amount is returned.

>>> from mpp._units import parse_units
>>> parse_units("999999999999999999999999999999", 0)
1000000000000000000000000000000          # exact: 999999999999999999999999999999

>>> parse_units("12345678901234567890.123456789012345678", 18)
12345678901234567890123456790000000000   # exact: 12345678901234567890123456789012345678

The second case overstates the amount by 987,654,322 base units. No exception is raised in either case, so a caller cannot detect it.

The threshold is 28 significant digits in total, not 28 decimal places, so it is reachable from ordinary inputs: an 18-decimal token needs only 10 integer digits before the value crosses the limit.

Fix

Scale from the decimal's own digit tuple with integer arithmetic, which is exact at any length. 10**decimals is still applied, but never through the rounding context.

Everything else is unchanged on purpose:

  • the fractional base units error keeps its condition and message
  • validation of empty / non-string / non-finite / negative input is untouched
  • exponent-form input ("1e5") keeps its current behavior — see the question below

Tests

Two cases added to TestParseUnitsEdgeCases, both verified to fail on main and pass here:

  • amounts past the context precision scale exactly
  • a long amount that genuinely produces fractional base units still raises

uv run pytest → 802 passed, 41 skipped. ruff check and ruff format --check clean.

One question, deliberately left out of this PR

parse_units("1e5", 6) currently returns 100000000000. The module docstring says the function matches "the parseUnits behavior in the TypeScript SDK (viem's parseUnits)", and viem validates its input against a plain decimal pattern that rejects exponent notation. If that divergence is real, it is a separate cross-SDK conformance question and I did not want to change two behaviors in one PR — happy to open a follow-up if you want it aligned.

Disclosure: I used an AI assistant while investigating and preparing this change; the analysis and conclusions are my own to defend.

🤖 Generated with Claude Code

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: e3ea34f284

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/mpp/_units.py Outdated
@mehmetkr-31

Copy link
Copy Markdown
Author

Good catch — fixed in 6ce85aa.

Measured on the flagged input, parse_units("1e-10000000", 6):

before this branch this branch @ e3ea34f this branch @ 6ce85aa
result returned 0 ValueError ValueError
time 0.0004s 9.3733s 0.0004s

unscaled has exactly len(digits) digits, so a divisor carrying at least that many zeros cannot divide it. Testing that from the exponent decides the fractional case without building the divisor, which is now only materialized when it can actually divide — bounded by the digit count of the amount itself. 1e-1000000000 is also immediate. A zero significand returns early so "0e-10000000" stays 0 instead of being swept into the rejection.

One correction to the premise, since it changes what the regression was. On main these inputs did not "just raise ValueError" — the product underflows below the context Emin of -999999, so d * (10**decimals) becomes 0E-1000026, which compares equal to int(res), the fractional check does not fire, and the function returns 0. So the previous behavior was a third silently-wrong amount, not an early rejection; what e3ea34f regressed was the cost of rejecting it, not the rejection itself. Both are covered by tests now.

@mehmetkr-31

Copy link
Copy Markdown
Author

The red conformance / python conformance check here is not caused by this PR — it fails the same way on every current PR that touches a protocol path, including my unrelated #209, which does not go near _units.py.

File "/home/runner/work/pympp/pympp/mpp-tools/conformance/scripts/vector_runner.py", line 29, in <module>
    from deepdiff import DeepDiff
ModuleNotFoundError: No module named 'deepdiff'

Cause. The reusable workflow is pinned at 9247105 (#173), but the suite it drives is checked out from a moving ref — the job log shows conformance-ref: main. mpp-tools has since moved the runner off a system pip install:

 install-scripts:
-	python3 -m pip install -q -r requirements.txt
+	uv sync --locked --python 3.12

with every Makefile target now going through uv run --locked python. Runner dependencies live in conformance/.venv, but the pinned workflow still calls python3 scripts/vector_runner.py. make install-runner succeeds — the log shows the venv created and 11 packages installed — and the runner then starts under an interpreter that cannot see them.

The last green vector step ran at 2026-08-08T00:27 under the same pin; the uv migration landed later that day.

Fix. Bumping both pins in .github/workflows/conformance.yml to 34ae1055 is enough. I verified the interface is unchanged at that revision: adapter and conformance-ref on the conformance workflow, protocol-paths and the conformance_ref output on the policy workflow, so SHA pinning is preserved. I have the commit ready but cannot open it as a PR — GitHub blocks fork PRs that modify .github/workflows/, so it needs to come from someone with write access.

Worth considering separately: a pinned workflow driving a main checkout can drift again. Pinning conformance-ref, or having the reusable workflow invoke the runner through the suite's own Makefile, would close that.

Also, on fork PRs generally: changelog.yml checks out ref: ${{ github.head_ref }} with no repository:, so it looks for the branch in this repo and fails for any fork contribution with "A branch or tag with the name ... could not be found" — that is why the changelog check is red on #209. Not something a contributor can work around from their side.

mehmetkr-31 and others added 2 commits August 9, 2026 12:14
`parse_units` scaled with `d * (10**decimals)`, which is evaluated in the
active decimal context. That context defaults to 28 significant digits, so
an amount longer than that was rounded to a different value before the
integrality check ran. Because the rounded value is itself integral, the
check passed and a silently wrong base-unit amount was returned:

    parse_units("999999999999999999999999999999", 0)
    # -> 1000000000000000000000000000000  (off by 1)

    parse_units("12345678901234567890.123456789012345678", 18)
    # -> ...790000000000  (off by 987654322 base units)

Scale from the decimal's own digit tuple instead, so the result is exact at
any length. The fractional-base-unit error and every other documented
behavior are unchanged.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Review feedback: scaling a value such as "1e-10000000" produced a divisor
with ten million digits before the fractional-base-unit check could reject
it — about 9.4s and ~4MB for an amount that never scales cleanly.

Decide divisibility from the exponent first. `unscaled` has exactly
`len(digits)` digits, so a divisor carrying at least that many zeros cannot
divide it, and the divisor is only built when it can. The pathological
inputs now fail in well under a millisecond.

Also return early for a zero significand so "0e-10000000" stays 0 rather
than being rejected by that check.

Note for the record: on main these inputs did not raise at all. The product
underflowed below the context Emin and compared equal to its own int, so
`parse_units("1e-10000000", 6)` returned 0 — another silently wrong amount
that this branch already fixes.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@mehmetkr-31
mehmetkr-31 force-pushed the fix/parse-units-exact-scaling branch from 6ce85aa to e50f148 Compare August 9, 2026 09:14
@mehmetkr-31

Copy link
Copy Markdown
Author

Correction to my previous comment: the pin bump had already landed before I posted it. #210 moved both pins to 29fee62d at 2026-08-08T23:30Z, which runs the suite through uv run --locked python and resolves the deepdiff failure. My comment's "needs to come from someone with write access" is stale — nothing is needed there, and I have dropped the branch I had prepared.

The rest of the diagnosis stands, and it explains why this PR was still red: it was based on 7f7164a, which predates #210, so it carried the old pin. I have rebased onto 9793ef5; conformance should now resolve on its own.

The two structural notes are unaffected: a workflow pinned by SHA while conformance-ref tracks main can drift apart again, and changelog.yml still checks out github.head_ref without a repository:, so it cannot pass on any fork PR.

`decimals` was never validated. A negative value divides the amount rather
than scaling it, and only reports the loss when the division leaves a
remainder:

    parse_units("100", -2)   # -> 1        an amount charged 100x too low
    parse_units("1000", -3)  # -> 1
    parse_units("7", -1)     # -> ValueError

That inconsistency is the dangerous part: the raising case makes it look
like the input is validated. mpp-go rejects negative decimals outright in
tempoxyz/mpp-go#87; do the same here.

`transform_units` guarded the type with `isinstance(decimals, int)`, which
accepts `bool`, so a request carrying `"decimals": true` scaled the amount
by 10 instead of being rejected. Exclude `bool` explicitly.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@mehmetkr-31

Copy link
Copy Markdown
Author

Pushed d427f42, which extends this PR to the other way parse_units returns a silently wrong amount. Same function, same failure mode, so it seemed better here than in a PR that would conflict with this one — say the word if you would rather have it split.

decimals was never validated. A negative value divides the amount instead of scaling it, and only reports the loss when the division leaves a remainder:

>>> parse_units("100", -2)
1                      # an amount charged 100x too low
>>> parse_units("1000", -3)
1
>>> parse_units("7", -1)
ValueError: ... produces fractional base units

The inconsistency is what makes it harmful: the raising case makes the input look validated. Reached through transform_units, {"amount": "100", "decimals": -2} became {"amount": "1"}.

This is already fixed on the Go side — tempoxyz/mpp-go#87 rejects negative decimals in ParseUnits for the same reason — so this brings the Python SDK to parity rather than inventing a rule.

Also excluded bool from the transform_units type check. bool is an int subclass, so {"decimals": true} was accepted and scaled the amount by 10.

810 passed, 41 skipped; ruff clean.

One thing I left alone. A very large decimals is accepted and scales fine in parse_units, but transform_units then calls str() on the result and trips CPython's integer-to-string limit:

ValueError: Exceeds the limit (4300 digits) for integer string conversion

So {"amount": "1", "decimals": 100000} fails with an error about string conversion rather than about the input. mpp-go does not bound decimals from above either, so I did not want to invent a ceiling unilaterally — happy to add one, or to convert that into a domain error, if you have a preference.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant