FrametoTable for Arrow backend by arnavdham · Pull Request #980 · AIDASoft/podio

arnavdham · 2026-06-23T00:28:24Z

Description
This PR isolates and implements a frameToTable conversion layer to map PODIO Frames and collection data types to Apache Arrow Tables. The implementation maps each PODIO Frame to a single row in an Arrow Table, with individual collections mapped to List<Struct<...>> columns.

Proposed Changes
Core Translation & Registries

ArrowTypeRegistry: Thread-safe global registry mapping PODIO collection type name strings to their corresponding Apache Arrow DataTypes.
ArrowConverterRegistry: Thread-safe global registry mapping PODIO collection type name strings to callback functions that serialize collection buffers into Arrow arrays.
ArrowFrameConverter: Implements convertFrameToTable, translating:
- Regular collections (delegated to registered converters).
- Subset collections (translated to reference coordinate structs with collectionID and index).
- Frame parameters (bundled into a frame_parameters struct column containing maps of string keys to list values).

Code Generation Integration

ArrowMapper.h.jinja2 & ArrowMapper.cc.jinja2: Templates that generate schema definition maps and register serializers dynamically for custom EDMs.
macros/arrow.jinja2: Jinja2 serialization macros that recursively parse complex types, fixed-size arrays, and one-to-one/one-to-many relationship structures to build Arrow array representations.
cpp_generator.py: Extracts structural metadata and triggers the generation of ArrowMapper files.

arnavdham · 2026-06-23T02:07:29Z

Related Issue: #981

arnavdham · 2026-06-24T10:43:43Z

@jmcarcell Please take a look. I'm not sure why some of the pretests fail.

/cvmfs/sft.cern.ch/lcg/views/LCG_104/x86_64-el9-gcc13-opt/include/arrow/scalar.h:39:10: fatal error: arrow/util/decimal.h: No such file or directory
39 | #include "arrow/util/decimal.h"
The following tests FAILED:
6 - read_python_frame_root (Not Run)
17 - read_python_frame_rntuple (Not Run)
40 - write_python_frame_root (Timeout)
41 - write_python_frame_rntuple (Timeout)
48 - read_python_frame_sio (Not Run)
54 - write_python_frame_sio (Timeout)

veprbl · 2026-06-24T21:21:03Z

I think, there is a missing symlink on the CVMFS:

% ls /cvmfs/sft.cern.ch/lcg/views/LCG_104/x86_64-el9-gcc13-opt/include/arrow/util/decimal.h
ls: cannot access /cvmfs/sft.cern.ch/lcg/views/LCG_104/x86_64-el9-gcc13-opt/include/arrow/util/decimal.h: No such file or directory
% ls /cvmfs/sft.cern.ch/lcg/releases/arrow/11.0.0-27569/x86_64-el9-gcc13-opt/include/arrow/util/decimal.h
/cvmfs/sft.cern.ch/lcg/releases/arrow/11.0.0-27569/x86_64-el9-gcc13-opt/include/arrow/util/decimal.h

I will try contacting the LCG support.

jmcarcell

@jmcarcell Please take a look. I'm not sure why some of the pretests fail.

* /cvmfs/sft.cern.ch/lcg/views/LCG_104/x86_64-el9-gcc13-opt/include/arrow/scalar.h:39:10: fatal error: arrow/util/decimal.h: No such file or directory
  39 | #include "arrow/util/decimal.h"

* The following tests FAILED:
  6 - read_python_frame_root (Not Run)
  17 - read_python_frame_rntuple (Not Run)
  40 - write_python_frame_root (Timeout)
  41 - write_python_frame_rntuple (Timeout)
  48 - read_python_frame_sio (Not Run)
  54 - write_python_frame_sio (Timeout)

Timeouts happen from time to time, simply ignore them. The workflow for LCG_104 is fixed now. You will need to rebase to pick #979 to avoid issues with sanitizers. I have made some comments inline and I have a few others:

Is mapping a Frame to a row in Arrow the natural way of doing this? What are the alternatives?
Values are copied through the object layer, did you think about this differently? There is arrow::Buffer::Wrap, maybe that could work?

jmcarcell · 2026-06-28T11:13:16Z

+#include <nlohmann/json.hpp>
+
+TEST_CASE("ArrowFrameConverter - convertFrameToTable Verification", "[arrow][converter]") {
+  podio::Frame originalFrame;


There are multiple tests that use the header write_frame.h and the function makeFrame, maybe that would be useful not to have to create it yourself, and it's quite complete. Eventually, when doing the roundtrip from Arrow to Frame, you could also use the stuff in read_frame.h in a similar way.

Thank you! I wasn't aware that this existed.

jmcarcell · 2026-06-28T11:30:16Z

+    auto* collectionBuilder = static_cast<arrow::ListBuilder*>(builder.get());
+    auto* objectBuilder = static_cast<arrow::StructBuilder*>(collectionBuilder->value_builder());
+
+    auto checkStatus = [](const arrow::Status& s, const std::string& msg) {


This is defined once per type, not ideal.

A comment explaining what changed would be better than "fixed"; in this case the fix is not trivial, a new file with the utility was made and that utility is being called. It's hard to follow otherwise.

jmcarcell · 2026-06-28T11:31:09Z

+    auto* collectionBuilder = static_cast<arrow::ListBuilder*>(builder.get());
+    auto* objectBuilder = static_cast<arrow::StructBuilder*>(collectionBuilder->value_builder());
+
+    auto checkStatus = [](const arrow::Status& s, const std::string& msg) {


Does every operation have to be run through checkStatus? Is that standard?

I think, this mimics ARROW_RETURN_NOT_OK. As far as I know there isn't an existing one that would throw an exception for us.

veprbl · 2026-06-29T01:35:38Z

+  checkStatus(builder_{{ relation.name }}_collId->Append(relObj_{{ relation.name }}.getObjectID().collectionID), "Failed to append relation collectionID");
+  checkStatus(builder_{{ relation.name }}_index->Append(relObj_{{ relation.name }}.getObjectID().index), "Failed to append relation index");
+} else {
+  checkStatus(builder_{{ relation.name }}->AppendNull(), "Failed to append null relation struct");


Do we need this case? It's possible that the values for invalid objects (-1, -1) already serve as a fine marker for this.

veprbl · 2026-06-29T01:35:48Z

+            "is_mult": is_mult,
+        }
+
+    def _arrow_field(self, name, type_expr, nullable=True):


We shouldn't need nullable, definitely not by default.

Appears fixed now.

Co-authored-by: Juan Miguel Carceller <22276694+jmcarcell@users.noreply.github.com>

arnavdham · 2026-06-29T02:29:08Z

@jmcarcell Please take a look. I'm not sure why some of the pretests fail.
* /cvmfs/sft.cern.ch/lcg/views/LCG_104/x86_64-el9-gcc13-opt/include/arrow/scalar.h:39:10: fatal error: arrow/util/decimal.h: No such file or directory
  39 | #include "arrow/util/decimal.h"

* The following tests FAILED:
  6 - read_python_frame_root (Not Run)
  17 - read_python_frame_rntuple (Not Run)
  40 - write_python_frame_root (Timeout)
  41 - write_python_frame_rntuple (Timeout)
  48 - read_python_frame_sio (Not Run)
  54 - write_python_frame_sio (Timeout)
Timeouts happen from time to time, simply ignore them. The workflow for LCG_104 is fixed now. You will need to rebase to pick #979 to avoid issues with sanitizers. I have made some comments inline and I have a few others:

Is mapping a Frame to a row in Arrow the natural way of doing this? What are the alternatives?

Values are copied through the object layer, did you think about this differently? There is arrow::Buffer::Wrap, maybe that could work?

In the future, we plan to map a Category to an Arrow Table, where each row in that table represents one individual Frame belonging to that category.
It keeps all collections in a given category grouped together in a single self-contained table, similar to how other formats (like ROOT RNTuple) structure events.

Alternative: We could store one flat Arrow Table per collection type (e.g., a HitsTable, a TracksTable) with an event_id column.But, it would require complex joins/filters to reconstruct a single frame, adding overhead.

Using arrow::Buffer::Wrap for zero-copy doesn't work directly here because:
AOS vs. SOA Layout: PODIO stores collection elements contiguously as Array-of-Structs (AOS), whereas Arrow's StructArray requires a columnar Struct-of-Arrays (SOA) layout with separate memory buffers for each field.
Might work for primitive members but not for complex structs.

veprbl · 2026-06-29T18:13:32Z

+std::shared_ptr<arrow::Schema> schema() {
+  const auto& types = typeMap();
+  static const auto instance = arrow::schema({
+{% for datatype in datatypes %}
+      arrow::field("{{ datatype.class.bare_type }}", types.at("{{ datatype.class.full_type }}")),
+{% endfor %}
+      arrow::field("frame_parameters", frameParametersType()),
+  });
+  return instance;
+}
+
+std::shared_ptr<arrow::Schema>
+schemaWithMetadata(std::shared_ptr<const arrow::KeyValueMetadata> metadata) {
+  return schema()->WithMetadata(std::move(metadata));
+}
+


Time to remove this? (also in the header file)

I think in the future for the Writer we could use this at some point(not sure about the exact usecase right now)

This schema is not usable. We would be refactoring or reusing schema generation from frameToTable instead.

have removed schema,frameparametersType and the related test.

veprbl · 2026-06-29T18:16:07Z

+    return()
+  endif()
+
+  add_library(${CORE_LIB}ArrowMapper SHARED ${SOURCES} ${HEADERS})


Since this becomes a part of the public interface, I wanted to check: Do we want to rename ArrowMapper/ARROW_MAPPER to just Arrow/ARROW? I get that SioBlocks is not just Sio, but the "Mapper" is not conveying much information? (We can keep ArrowMapper.{cc,h} as-is for clarity)

Sure. we can do this.

arnavdham · 2026-07-01T01:43:56Z

@jmcarcell @veprbl I have made changes according to all comments. Let me know if anything is still needed.

veprbl

I have some more nitpicks on unused things.

veprbl · 2026-07-03T14:20:53Z

+namespace arrow {
+class Schema;
+class KeyValueMetadata;
 }


This is now also unused.

veprbl · 2026-07-03T14:21:02Z

-#include <arrow/type.h>
-#include <arrow/util/key_value_metadata.h>
-
 #include <memory>


Also unused.

veprbl · 2026-07-03T14:22:39Z

-schemaWithMetadata(std::shared_ptr<const arrow::KeyValueMetadata> metadata) {
-  return schema()->WithMetadata(std::move(metadata));
-}
+bool registerTypes();


And we don't need to expose this library initialization. We don't expect to call this externally?

Have removed this from the .h file but now the ArrowMapper.h is empty entirely. Do I just remove the file entirely?

veprbl · 2026-07-03T14:30:56Z

+  void checkStatus(const arrow::Status& status, const std::string& msg) {
+    podio::arrow_utils::checkStatus(status, msg);
+  }


Suggested change

void checkStatus(const arrow::Status& status, const std::string& msg) {

podio::arrow_utils::checkStatus(status, msg);

}

using checkStatus = podio::arrow_utils::checkStatus;

Or something like that.

veprbl · 2026-07-03T14:31:16Z

+namespace {
+void checkStatus(const arrow::Status& status, const std::string& msg) {
+  podio::arrow_utils::checkStatus(status, msg);
+}


veprbl · 2026-07-03T14:32:17Z

+  std::cout << "--- ARROW TABLE JSON ---" << std::endl;
+  std::cout << arrowJson.dump(4) << std::endl;
+  std::cout << "--- FRAME DIRECT JSON ---" << std::endl;
+  std::cout << frameJson.dump(4) << std::endl;
+


Not sure if tests are meant to output to stdout.

Yes I did that to check outputs. Removed now

veprbl · 2026-07-03T14:33:41Z

+  ArrowTypeRegistry.cc
+  ArrowConverterRegistry.cc


Do these go below instead?

veprbl · 2026-07-03T14:34:38Z

+            "class_full_type": datatype["class"].full_type,
+            "class_bare_type": datatype["class"].bare_type,


Unused now?

veprbl

LGTM

arnavdham changed the title ~~Feature/frame to table~~ FrametoTable for Arrow backend Jun 23, 2026

arnavdham mentioned this pull request Jun 23, 2026

frameToTable In-Memory Conversion to Apache Arrow #981

Open

arnavdham marked this pull request as ready for review June 24, 2026 04:06

jmcarcell reviewed Jun 28, 2026

View reviewed changes

Comment thread include/podio/utilities/ArrowConverterRegistry.h Outdated

veprbl reviewed Jun 29, 2026

View reviewed changes

arnavdham and others added 7 commits June 29, 2026 07:41

frame to Table implementation

a7729d1

pre commit and ci/cd pipeline failures

cb0804e

fixes.

1fa2236

clang-tidy

7609a5e

Adding unittest for JSON Equivalence

283b3e1

Apply suggestions from code review

fbe6b85

Co-authored-by: Juan Miguel Carceller <22276694+jmcarcell@users.noreply.github.com>

Addressing all comments

3d3eac1

arnavdham force-pushed the feature/frameToTable branch from 0bf0945 to 3d3eac1 Compare June 29, 2026 02:18

pre commit issues with only .h

37398b1

veprbl reviewed Jun 29, 2026

View reviewed changes

arnavdham added 2 commits June 30, 2026 01:55

arrowMapper->Arrow

e160321

removing schema and framparametersType

657af59

veprbl mentioned this pull request Jun 30, 2026

Build libedm4eicArrow if PODIO supports it key4hep/EDM4hep#503

Open

veprbl reviewed Jul 3, 2026

View reviewed changes

arnavdham added 2 commits July 4, 2026 08:21

Addressing comments.

2ecbe96

removing arrowmapper.h

64ad735

veprbl approved these changes Jul 6, 2026

View reviewed changes

		"class_full_type": datatype["class"].full_type,
		"class_bare_type": datatype["class"].bare_type,

Uh oh!

Conversation

arnavdham commented Jun 23, 2026

Uh oh!

arnavdham commented Jun 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

arnavdham commented Jun 24, 2026

Uh oh!

veprbl commented Jun 24, 2026

Uh oh!

jmcarcell left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

arnavdham commented Jun 29, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

arnavdham commented Jul 1, 2026

Uh oh!

veprbl left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

arnavdham commented Jun 23, 2026 •

edited

Loading

jmcarcell left a comment •

edited

Loading