fix(r): correct two int64 corruption bugs affecting list<int64> conversion - #933
Merged
Merged
Conversation
…rsion Fixes two independent bugs that both corrupt int64 values, uncovered while investigating meztez/bigrquerystorage#86 (which found that converting a list<int64> array to a vctrs list_of<integer64> ptype silently produced wrong values instead of erroring or converting correctly): 1. as_nanoarrow_array.list() built its child array via unlist(x, recursive = FALSE, use.names = FALSE). unlist() strips the integer64 class from each element and concatenates the underlying doubles as a plain numeric vector, reinterpreting the raw int64 bit pattern as a double bit pattern -- corrupting every non-trivial int64 value before the child array was even built. Switched to do.call(c, x), which dispatches to c.integer64 (and other S3 c() methods) and preserves the class and underlying bit pattern. 2. nanoarrow_materialize_int64() read the NANOARROW_TYPE_INT64 buffer via buffer_views[1].data.as_int32 + raw_src_offset instead of ...data.as_int64 + raw_src_offset. Because the offset was added using int32-sized pointer arithmetic instead of int64-sized, any conversion of an int64 array/slice with a nonzero starting offset (for example, the per-row child slices produced when materializing a list<int64> column) read from the wrong memory location, whereas offset-0 arrays (the common case exercised by prior tests) were unaffected. This bug predates the list-of-int64 issue and has existed since apache#293 first added int64 support. Added regression tests for both bugs in test-as-array.R and test-convert-array.R; confirmed each new test fails against the pre-fix code and passes after the fix. Full existing test suite (r/tests/testthat) still passes (1408 passed, 0 failed).
meztez
added a commit
to meztez/bigrquerystorage
that referenced
this pull request
Sep 4, 2026
…d nanoarrow fix) Extends int64_ptype() to also recurse into vctrs_list_of ptypes with a nested (non-primitive) schema, so REPEATED INT64 (BigQuery's array-of-int64) columns are converted directly to bit64::integer64 instead of going through R's double first. This closes the remaining gap noted in #86: previously only scalar/struct-nested INT64 was lossless. This relies on the nanoarrow fix proposed in apache/arrow-nanoarrow#933 (closes apache/arrow-nanoarrow#932), which is not yet merged upstream. Until it lands and a release is cut, DESCRIPTION's Remotes field points bigrquerystorage's nanoarrow dependency at the patched fork/branch (meztez/arrow-nanoarrow@fix-932-list-int64-ptype) so this can be developed and tested against main. This should NOT be submitted to CRAN until nanoarrow releases with the fix included -- CRAN does not accept Remotes- based dependencies. - Bump version to 1.2.3.9000 (dev) - Require nanoarrow (>= 0.9.0.9000) and add a Remotes entry pointing at the patched fork - int64_ptype(): recurse into vctrs_list_of ptypes when the underlying schema is actually nested (guard against non-list vctrs_list_of ptypes like blob/BYTES, which have no schema children) - Update/extend test-bigint.R: repeated int64 columns are now asserted to be lossless end-to-end (previously only asserted that bigint was honored on the already-lossy double), plus a regression test guarding the blob/BYTES edge case - NEWS.md: documented as a dev-only, CRAN-pending entry
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #933 +/- ##
=======================================
Coverage 79.30% 79.30%
=======================================
Files 106 106
Lines 16223 16223
Branches 1882 1882
=======================================
Hits 12865 12865
Misses 2177 2177
Partials 1181 1181 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #932.
While investigating why converting a
list<int64>array to avctrs::list_of(ptype = bit64::integer64())target silently produced wrong values (not just a lossy-precision warning), I found two separate, independent bugs, both inr/:1.
as_nanoarrow_array.list()corruptedinteger64list elements while building the child arrayIt built the flattened child vector with:
unlist()strips theinteger64S3 class from each element before concatenating, so the elements are concatenated as plainnumeric, reinterpreting the raw 64-bit integer bit pattern as a double bit pattern. The resulting child array is corrupted before conversion even runs. Fixed by usingdo.call(c, x), which dispatches toc.integer64(and any other registeredc()S3 method) and preserves the class/underlying representation.2.
nanoarrow_materialize_int64()used the wrong buffer view for pointer arithmeticraw_src_offsetis added todata.as_int32, so the offset is scaled by 4 bytes instead of 8. Any int64 array/slice with a nonzero starting offset — such as the per-row child slices nanoarrow builds internally when materializing alist<int64>column — reads from the wrong memory location. Offset-0 conversions (the case covered by existing tests) are unaffected, which is presumably why this slipped through since it was introduced in #293.Both bugs needed fixing to correctly round-trip a
list<int64>throughbit64::integer64; either one alone still corrupts values.Testing
Added regression tests to
test-as-array.Randtest-convert-array.R. I confirmed both new tests fail against the pre-fix code and pass after the fix (see repro below), and that the full existingr/tests/testthatsuite still passes (1408 passed, 0 failed).Minimal repro of bug 1 (pre-fix):
After this PR, this returns
list(c(9223372036854775295, 2), 3)as expected.Originally reported against a downstream package: meztez/bigrquerystorage#86.