perf: speed up Parquet reading via LRU shard cache#5619
Draft
benjaminhuth wants to merge 8 commits into
Draft
Conversation
Replace the per-event Arrow dataset scan with an LRU shard cache. The old path decompressed the full shard row group (~750 MB) on every readEvent() call. The new path decompresses each shard once on first access and serves subsequent events with an in-memory Table::Slice, reducing per-event I/O cost by ~100x for sequential processing. Key changes: - Add Acts::LRUCache<Key,Value> template utility with O(1) get/put/evict - ParquetDatasetReader builds an event→shard index at construction (binary search replaces multi-file dataset filter pushdown in the hot path) - On cache miss, loadShard() uses the Dataset API on the single shard file so missing columns are still materialized as nulls (schema evolution) - ParquetReader::Config gains shardCacheCapacity (default 1; set to numThreads for parallel runs) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
github-actions
Bot
added
Component - Core
Contributor
Member
There was a problem hiding this comment.
Pedantic comment of the day: LruCache!
Add `keepParticlesWithoutHits` (default false) to `ColliderMLRelease1InputConverter::Config`. When false, pre-scan the hits table's `particle_id` column into an unordered_set before the particle loop so only particles referenced by at least one hit are constructed and inserted into the output container. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The loadShard() function uses the Arrow Dataset scanner, which requires compute functions (e.g. make_struct) to be registered. This was broken when ensureComputeInitialized() was removed during the LRU cache refactor. Restore it as a std::call_once guard inside loadShard(). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
asVector() returned std::vector<uint32_t>, causing a heap allocation on every Barcode comparison. Since the size is always exactly 5, switch to std::array<uint32_t, 5> and remove the old accessor. Update all call sites (CsvParticleWriter, CsvSimHitWriter, BarcodeTests) accordingly. This eliminates malloc/free overhead in every sorted Barcode container operation (flat_multimap sort in invertIndexMultimap, TrackTruthMatcher map lookups, ParticleSelector binary search), yielding ~1.8× overall speedup on a 100-event ColliderML PU200 benchmark. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
github-actions
Bot
added
Component - Fatras
The particleTruthHitCount map is built by iterating all hits in the measurement-particles map, incrementing a counter per SimBarcode key. Using std::map gave O(N log N) insertions with Barcode comparisons; switching to std::unordered_map (SimBarcode already has std::hash) makes the accumulation loop O(N) and lookups O(1). Yields ~3× speedup on TrackTruthMatcher (~10% overall on a 100-event ColliderML PU200 benchmark). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
runColliderMLTruthTracking() now takes particlesDir/hitsDir instead of inputDir and only returns the Sequencer (no perf writers), following the parquet-reading perf refactor. Update the test to match and drop the ROOT output assertions for writers that are no longer added. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Replace the per-event Arrow dataset scan with an LRU shard cache. The old path decompressed the full shard row group (~750 MB for ColliderML) on every readEvent() call. The new path decompresses each shard once on first access and serves subsequent events with an in-memory Table::Slice, reducing per-event I/O cost by ~100x for sequential processing.