feat: append array views to arrays by rustyconover · Pull Request #930 · apache/arrow-nanoarrow

rustyconover · 2026-09-02T18:12:43Z

Summary

add ArrowArrayAppendArrayView() as a core array-building API
append primitive, nested, union, list-view, and run-end encoded values from an ArrowArrayView
preserve source slicing semantics and report unsupported sliced run-end encoded inputs
add direct native coverage for primitive, nested/sliced, union, and run-end encoded arrays

Context

This extracts the generic array-view appender from #928 as requested in review. It is independently useful and will be the prerequisite for a separate dictionary-delta decoding PR; #928 no longer contains the appender or decoder changes.

Validation

native CMake build passed
native CTest suite: 334/334 passed (4 optional codec tests skipped)

codecov-commenter · 2026-09-02T18:21:02Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 78.06%. Comparing base (f55f85b) to head (0603bc1).

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #930      +/-   ##
==========================================
- Coverage   79.30%   78.06%   -1.24%     
==========================================
  Files         106      106              
  Lines       16223    16484     +261     
  Branches     1882     1884       +2     
==========================================
+ Hits        12865    12869       +4     
- Misses       2177     2434     +257     
  Partials     1181     1181

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

paleolimbot

Thank you!

Some things to take care of here but I think this is great functionality.

I can follow up with R/Python bindings, which are mostly useful because testing this stuff in C++ is a pain. I think this may also make a few of the R and Python constructors quite a bit simpler.

paleolimbot · 2026-09-02T20:52:15Z

+ArrowErrorCode ArrowArrayAppendArrayView(struct ArrowArray* dst,
+                                         const struct ArrowArrayView* src,
+                                         struct ArrowError* error) {
+  if (src->storage_type == NANOARROW_TYPE_RUN_END_ENCODED) {


At the top of this function we should sanity check that the append makes sense, because I am not sure that all combinations of spurious comparisons will generate a reasonable error here. ArrowArrayView only gives you storage type, so we probably should name the function accordingly (ArrowArrayAppendStorageFromArrayView()) to make it a bit more obvious to the caller that they're on their own with respect to whether they should do this or not.

Also a note that this needs to check src->dictionary, which I don't believe is reflected in the storage type (for better or worse).

Addressed in bab29ad. The API is now ArrowArrayAppendStorageFromArrayView, with recursive preflight checks for compatible storage, child layouts, fixed-size widths, and dictionary presence. Dictionary values are deliberately caller-managed: both source and destination must be dictionary-encoded, and the API docs state that values are not copied. Follow-up 693718d narrows fixed-size layout equality checks to fixed-size-to-fixed-size appends and covers fixed-to-variable binary/list conversion.

paleolimbot · 2026-09-02T20:54:55Z

+    if (src->offset != 0) {
+      ArrowErrorSet(error, "Can't append a sliced run-end encoded array view");
+      return ENOTSUP;
+    }


No need to get too into the weeds supporting this one, but we do have a utility that I think can resolve the first run:

arrow-nanoarrow/src/nanoarrow/common/inline_buffer.h

Lines 53 to 54 in f55f85b

static inline int64_t ArrowResolveChunk32(int32_t index, const int32_t* offsets,

int32_t lo, int32_t hi) {

Addressed in bab29ad. Sliced run-end encoded views now resolve the first run with ArrowResolveChunk32 or ArrowResolveChunk64 (with the int16 fallback), clip the boundary runs to the slice, and rewrite run ends relative to the destination.

paleolimbot · 2026-09-02T20:57:15Z

+static ArrowErrorCode ArrowArrayAppendArrayViewElement(struct ArrowArray* dst,
+                                                       const struct ArrowArrayView* src,
+                                                       int64_t i,
+                                                       struct ArrowError* error) {


While we are here, I think it is worth the effort to special case primitive fixed-size appends of the same storage type (e.g., int32 getting appended to an int32). These should be much faster because they are just appending buffers to each other (the nulls bitmap is a bit more complex...may have to go through a uint8 buffer for the bitmap append if either the source or the target has any nulls).

Addressed in 4974cd7. Matching byte-aligned primitive storage takes a bulk buffer-copy path. Validity is copied in bounded uint8 chunks so sliced bitmaps and all four source/destination bitmap-presence combinations are handled.

paleolimbot · 2026-09-02T21:02:36Z

+      NANOARROW_RETURN_NOT_OK(ArrowArrayAppendInt(
+          dst->children[0],
+          run_end_offset + ArrowArrayViewGetIntUnsafe(src->children[0], i)));


These (and below) should use NANOARROW_RETURN_NOT_OK_WITH_ERROR()...we are in general very careful about ensuring that the contents of error are populated if an error code is returned.

Addressed in bab29ad and refined in e56f47b. Direct append and buffer operations use NANOARROW_RETURN_NOT_OK_WITH_ERROR so failures populate the error; calls to helpers that already populate it use the plain propagation macro so their more specific message is preserved.

paleolimbot · 2026-09-02T21:09:12Z

+static ArrowErrorCode AppendArrayViewForTest(const struct ArrowArrayView* src,
+                                             struct ArrowArray* dst,
+                                             struct ArrowError* error) {


In addition to these lovely nanoarrow only tests, an Arrow C++ gated test I think would help cover some of the things that can happen here in a way that can be parameterized to cover more cases. Perhaps one test per type family (unigned int, signed int, strings, binaries, lists), testing against the Arrow C++ builder for the matrix of source and destination types? Arrow C++ makes asserting the equality a bit easier too.

Addressed in e56f47b. Added Arrow C++-gated source/destination matrices for signed integers, unsigned integers, string/large_string, binary/large_binary, and list/large_list, with Arrow C++ equality checks on the imported result.

paleolimbot · 2026-09-02T21:15:16Z

+TEST(ArrayTest, ArrayAppendArrayViewPrimitiveTypes) {
+  struct ArrowArray array;
+
+  ASSERT_EQ(ArrowArrayInitFromType(&array, NANOARROW_TYPE_INT64), NANOARROW_OK);


Some things that I am not sure are covered here are:

Appending an integer that is out of range to a narrower integer type (should pass on the error from the integer appender)

Appending too many elements to a narrower list type (you can use a list of Null type to avoid allocating huge buffers), which should pass on an error from the list element finisher

Dictionary encoded source or target

Addressed in e56f47b. Added tests for integer narrowing overflow, int32 list-offset overflow using a Null child, successful dictionary storage copying with caller-managed dictionary values, and rejection when only one side is dictionary-encoded.

Add Arrow C++ conversion matrices plus sliced validity, overflow, dictionary, and sliced run-end tests. Preserve detailed errors produced by nested append helpers.

Avoid reading past short int8 or int32 input arrays when an append begins in a partially filled bitmap byte.

rustyconover · 2026-09-04T11:51:34Z

CI follow-up: b4f4adc fixes the shared failure behind clang-tidy and the four Valgrind jobs. ArrowBitmapAppendInt8Unsafe could read more values than supplied when a short append began in a partially filled byte; the same issue existed in the int32 variant. Both helpers now bound the partial-byte copy, with regression tests for short unaligned appends. Native, Arrow C++, AddressSanitizer, and local clang-tidy checks pass.

Ensure clang-tidy can prove that all values passed through the bitmap packing path are initialized.

rustyconover · 2026-09-04T13:13:12Z

clang-tidy follow-up: the Valgrind failures are resolved by b4f4adc, but clang-tidy 18 could not prove that ArrowBitsUnpackInt8 initialized every element later read from the scratch buffer. Commit 0603bc1 explicitly zero-initializes that buffer; this preserves behavior while guaranteeing analyzer-visible initialization.

feat: append array views to arrays

58d145c

rustyconover mentioned this pull request Sep 2, 2026

feat(ipc): add dictionary batch write support #928

Open

paleolimbot reviewed Sep 2, 2026

View reviewed changes

rustyconover added 5 commits September 3, 2026 09:00

refactor: validate array view storage appends

bab29ad

perf: bulk append matching fixed-width storage

4974cd7

test: expand array view appender coverage

e56f47b

Add Arrow C++ conversion matrices plus sliced validity, overflow, dictionary, and sliced run-end tests. Preserve detailed errors produced by nested append helpers.

fix: allow fixed-to-variable storage appends

693718d

fix: bound partial bitmap appends

b4f4adc

Avoid reading past short int8 or int32 input arrays when an append begins in a partially filled bitmap byte.

fix: initialize validity append scratch buffer

0603bc1

Ensure clang-tidy can prove that all values passed through the bitmap packing path are initialized.

	static inline int64_t ArrowResolveChunk32(int32_t index, const int32_t* offsets,
	int32_t lo, int32_t hi) {

Uh oh!

Conversation

rustyconover commented Sep 2, 2026

Summary

Context

Validation

Uh oh!

codecov-commenter commented Sep 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

paleolimbot left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rustyconover Sep 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rustyconover commented Sep 4, 2026

Uh oh!

rustyconover commented Sep 4, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

codecov-commenter commented Sep 2, 2026 •

edited

Loading

rustyconover Sep 3, 2026 •

edited

Loading