Skip to content

fix(r): correct two int64 corruption bugs affecting list<int64> conversion - #933

Merged
paleolimbot merged 1 commit into
apache:mainfrom
meztez:fix-932-list-int64-ptype
Sep 4, 2026
Merged

fix(r): correct two int64 corruption bugs affecting list<int64> conversion#933
paleolimbot merged 1 commit into
apache:mainfrom
meztez:fix-932-list-int64-ptype

Conversation

@meztez

@meztez meztez commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

Fixes #932.

While investigating why converting a list<int64> array to a vctrs::list_of(ptype = bit64::integer64()) target silently produced wrong values (not just a lossy-precision warning), I found two separate, independent bugs, both in r/:

1. as_nanoarrow_array.list() corrupted integer64 list elements while building the child array

It built the flattened child vector with:

child <- unlist(x, recursive = FALSE, use.names = FALSE)

unlist() strips the integer64 S3 class from each element before concatenating, so the elements are concatenated as plain numeric, reinterpreting the raw 64-bit integer bit pattern as a double bit pattern. The resulting child array is corrupted before conversion even runs. Fixed by using do.call(c, x), which dispatches to c.integer64 (and any other registered c() S3 method) and preserves the class/underlying representation.

2. nanoarrow_materialize_int64() used the wrong buffer view for pointer arithmetic

case NANOARROW_TYPE_INT64:
  memcpy(result + dst->offset,
         src->array_view->buffer_views[1].data.as_int32 + raw_src_offset,
         dst->length * sizeof(int64_t));

raw_src_offset is added to data.as_int32, so the offset is scaled by 4 bytes instead of 8. Any int64 array/slice with a nonzero starting offset — such as the per-row child slices nanoarrow builds internally when materializing a list<int64> column — reads from the wrong memory location. Offset-0 conversions (the case covered by existing tests) are unaffected, which is presumably why this slipped through since it was introduced in #293.

Both bugs needed fixing to correctly round-trip a list<int64> through bit64::integer64; either one alone still corrupts values.

Testing

Added regression tests to test-as-array.R and test-convert-array.R. I confirmed both new tests fail against the pre-fix code and pass after the fix (see repro below), and that the full existing r/tests/testthat suite still passes (1408 passed, 0 failed).

Minimal repro of bug 1 (pre-fix):

library(nanoarrow)
schema <- na_list(na_int64())
arr <- as_nanoarrow_array(
  list(bit64::as.integer64(c("9223372036854775295", "2")), bit64::as.integer64("3")),
  schema = schema
)
to <- vctrs::new_list_of(list(), ptype = bit64::integer64())
convert_array(arr, to = to)
#> [[1]]
#> integer64
#> [1] <NA> 0
#>
#> [[2]]
#> integer64
#> [1] 0

After this PR, this returns list(c(9223372036854775295, 2), 3) as expected.

Originally reported against a downstream package: meztez/bigrquerystorage#86.

…rsion

Fixes two independent bugs that both corrupt int64 values, uncovered while
investigating meztez/bigrquerystorage#86 (which
found that converting a list<int64> array to a vctrs list_of<integer64>
ptype silently produced wrong values instead of erroring or converting
correctly):

1. as_nanoarrow_array.list() built its child array via
   unlist(x, recursive = FALSE, use.names = FALSE). unlist() strips the
   integer64 class from each element and concatenates the underlying
   doubles as a plain numeric vector, reinterpreting the raw int64 bit
   pattern as a double bit pattern -- corrupting every non-trivial int64
   value before the child array was even built. Switched to
   do.call(c, x), which dispatches to c.integer64 (and other S3 c()
   methods) and preserves the class and underlying bit pattern.

2. nanoarrow_materialize_int64() read the NANOARROW_TYPE_INT64 buffer via
   buffer_views[1].data.as_int32 + raw_src_offset instead of
   ...data.as_int64 + raw_src_offset. Because the offset was added using
   int32-sized pointer arithmetic instead of int64-sized, any conversion
   of an int64 array/slice with a nonzero starting offset (for example,
   the per-row child slices produced when materializing a list<int64>
   column) read from the wrong memory location, whereas offset-0 arrays
   (the common case exercised by prior tests) were unaffected. This bug
   predates the list-of-int64 issue and has existed since apache#293 first
   added int64 support.

Added regression tests for both bugs in test-as-array.R and
test-convert-array.R; confirmed each new test fails against the
pre-fix code and passes after the fix. Full existing test suite
(r/tests/testthat) still passes (1408 passed, 0 failed).
meztez added a commit to meztez/bigrquerystorage that referenced this pull request Sep 4, 2026
…d nanoarrow fix)

Extends int64_ptype() to also recurse into vctrs_list_of ptypes with a
nested (non-primitive) schema, so REPEATED INT64 (BigQuery's array-of-int64)
columns are converted directly to bit64::integer64 instead of going through
R's double first. This closes the remaining gap noted in #86: previously
only scalar/struct-nested INT64 was lossless.

This relies on the nanoarrow fix proposed in apache/arrow-nanoarrow#933
(closes apache/arrow-nanoarrow#932), which is not yet merged upstream.
Until it lands and a release is cut, DESCRIPTION's Remotes field points
bigrquerystorage's nanoarrow dependency at the patched fork/branch
(meztez/arrow-nanoarrow@fix-932-list-int64-ptype) so this can be developed
and tested against main. This should NOT be submitted to CRAN until
nanoarrow releases with the fix included -- CRAN does not accept Remotes-
based dependencies.

- Bump version to 1.2.3.9000 (dev)
- Require nanoarrow (>= 0.9.0.9000) and add a Remotes entry pointing at the
  patched fork
- int64_ptype(): recurse into vctrs_list_of ptypes when the underlying
  schema is actually nested (guard against non-list vctrs_list_of ptypes
  like blob/BYTES, which have no schema children)
- Update/extend test-bigint.R: repeated int64 columns are now asserted to
  be lossless end-to-end (previously only asserted that bigint was honored
  on the already-lossy double), plus a regression test guarding the
  blob/BYTES edge case
- NEWS.md: documented as a dev-only, CRAN-pending entry
@codecov-commenter

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 79.30%. Comparing base (f55f85b) to head (6124762).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #933   +/-   ##
=======================================
  Coverage   79.30%   79.30%           
=======================================
  Files         106      106           
  Lines       16223    16223           
  Branches     1882     1882           
=======================================
  Hits        12865    12865           
  Misses       2177     2177           
  Partials     1181     1181           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@paleolimbot paleolimbot left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you!

@paleolimbot
paleolimbot merged commit 02824cc into apache:main Sep 4, 2026
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

R: converting list<int64> to list_of<integer64> (vctrs list-of) produces wrong values

3 participants