Avoid unchecked dataset size/product overflow and centralize allocation checks

[rmems]

Tracks LIM-1307.

User description

Motivation

• Prevent unchecked multiplication of dataset extents (and use of Dataset::size) that can overflow or panic and allow hostile files to bypass budget checks.
• Unify and harden allocation charging and pre-checks so large declared shapes or capacity computations are rejected safely before any heavy HDF5 read calls.

Description

• Added safe helpers dataset_element_count, checked_shape_product, charge_allocation, check_allocation, allocation_size_overflow, and dataset_extent_overflow to compute element counts and verify/charge allocation sizes without allowing unchecked overflow.
• Replaced direct uses of Dataset::size() with dataset_element_count(...) and updated callers to use charge_allocation(...) or check_allocation(...) where appropriate, including in read_edges, read_tensor, charge_elements, read_string_scalar_validated, and charge_strings.
• Improved charge_strings to pre-check minimum allocation requirements via check_allocation and only call expensive VLEN sizing when necessary, while preserving the previous fast-path when there is no byte limit.
• Added test utilities and fixtures: decode_hex_fixture helper and tests/fixtures/overflowing_extents.nir.hex to reproduce overflowing extent scenarios.
• Added unit tests checked_shape_product_preserves_scalar_and_zero_extent_semantics and allocation_overflow_fails_closed_with_and_without_a_byte_limit, and a regression test overflowing_dataset_extents_are_rejected_before_reading (gated to 64-bit) under the HDF5-enabled integration tests.

Testing

Final Cloud validation also passed cargo fmt --check, locked default/serde/all-feature tests, strict locked all-target/all-feature Clippy, all-feature docs, targeted allocation-overflow unit tests, and the release --test hdf5_untrusted overflowing_dataset_extents_are_rejected_before_reading regression. The fixture is text-encoded so Cloud publication succeeds; it is decoded in a Rust temporary-directory helper. Published head: ffbf29338e1a2a84819524100983fe3c78deae18; base: b4dc78355e662556bf4ce6b3892072ed01856981. The remote commit carries the required Codex co-author trailer.

• Ran the crate test suite with cargo test and the HDF5 feature with cargo test --features hdf5, and all tests (including the new unit and HDF5 regression tests on a 64-bit host) passed.
• The new regression fixture overflowing_extents.nir.hex exercises a dataset whose declared shape product overflows usize and confirms the reader rejects it before any read_raw is attempted.
• Property and existing tests exercising the ReadBudget behavior were run and continue to pass after the changes.

---

Codex Task

---

CodeAnt-AI Description

Reject overflowing dataset sizes before reading data

What Changed

• Files with dataset shapes whose element count overflows are rejected before HDF5 data is read
• Decoded tensor and string allocation sizes are checked for overflow instead of wrapping or being accepted
• Bounded reads continue to return read-limit errors, while unbounded reads fail with a clear invalid-tensor error
• Added regression coverage for overflowing extents and allocation-size failures

Impact

✅ Safer reads of hostile files
✅ Fewer allocation overflows
✅ Clearer errors for impossible dataset sizes

💡 Usage Guide

Checking Your Pull Request

Every time you make a pull request, our system automatically looks through it. We check for security issues, mistakes in how you're setting up your infrastructure, and common code problems. We do this to make sure your changes are solid and won't cause any trouble later.

Talking to CodeAnt AI

Got a question or need a hand with something in your pull request? You can easily get in touch with CodeAnt AI right here. Just type the following in a comment on your pull request, and replace "Your question here" with whatever you want to ask:

@codeant-ai ask: Your question here

This lets you have a chat with CodeAnt AI about your pull request, making it easier to understand and improve your code.

Example

@codeant-ai ask: Can you suggest a safer alternative to storing this secret?

Preserve Org Learnings with CodeAnt

You can record team preferences so CodeAnt AI applies them in future reviews. Reply directly to the specific CodeAnt AI suggestion (in the same thread) and replace "Your feedback here" with your input:

@codeant-ai: Your feedback here

This helps CodeAnt AI learn and adapt to your team's coding style and standards.

Example

@codeant-ai: Do not flag unused imports.

Retrigger review

Ask CodeAnt AI to review the PR again, by typing:

@codeant-ai: review

Check Your Repository Health

To analyze the health of your code repository, visit our dashboard at https://app.codeant.ai. This tool helps you identify potential issues and areas for improvement in your codebase, ensuring your repository maintains high standards of code health.

[codeant-ai]

🤖 CodeAnt AI — Review Status

Status	Commit	Started (UTC)	Finished (UTC)
✅ Reviewed your PR	ffbf293	Sep 21, 2026 · 05:14	05:16

[codeant-ai]

---

Thanks for using CodeAnt! 🎉

We're free for open-source projects. if you're enjoying it, help us grow by sharing.

Share on X ·
Reddit ·
LinkedIn

[linear-code]

LIM-1307

[deepsource-io]

DeepSource Code Review

We reviewed changes in b4dc783...ffbf293 on this pull request. Below is the summary for the review, and you can see the individual issues we found as inline review comments.

See full review on DeepSource ↗

PR Report Card

Overall Grade

Security   Reliability   Complexity   Hygiene

Code Review Summary

Analyzer	Status	Updated (UTC)	Details
Rust		Sep 21, 2026 5:14a.m.	Review ↗
Secrets		Sep 21, 2026 5:14a.m.	Review ↗

---

Important

AI Review is run only on demand for your team. We're only showing results of static analysis review right now. To trigger AI Review, comment @deepsourcebot review on this thread.

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-09-21T05:17:27.817398Z	ffbf293	PR opened
🔒 Security Review	✅ Completed 2026-09-21T05:18:30.447834Z	ffbf293	PR opened

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[amazon-q-developer]

The changes comprehensively address overflow vulnerabilities in dataset extent handling. The implementation correctly replaces unchecked Dataset::size() calls with a safe dataset_element_count() helper that uses checked multiplication, properly distinguishes between bounded and unbounded error reporting, and includes comprehensive tests including a regression test with a crafted overflow fixture. The code follows Rust security best practices and maintains backward compatibility.

---

You can now have the agent implement changes and create commits directly on your pull request's source branch. Simply comment with /q followed by your request in natural language to ask the agent to make changes.

[codacy-production]

Up to standards ✅

🟢 Issues 0 issues

Results:
0 new issues

View in Codacy

🟢 Metrics 20 complexity · 2 duplication

Metric	Results
Complexity	20
Duplication	2

View in Codacy

_{NEW Get contextual insights on your PRs based on Codacy's metrics, along with PR and Jira context, without leaving GitHub. Enable AI reviewer
TIP This summary will be updated as you push new changes.}

[coderabbitai]

Understand this PR’s impact

Explore downstream dependencies and potential security impact with Blast Radius.

View blast radius →

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Essentials

Run ID: 258f73e4-4cbb-400b-b11c-5b9edc4820f4

📥 Commits

Reviewing files that changed from the base of the PR and between b4dc783 and ffbf293.

📒 Files selected for processing (4)

• src/io/hdf5_read.rs
• tests/common/mod.rs
• tests/fixtures/overflowing_extents.nir.hex
• tests/hdf5_untrusted.rs

Included review availability: 1 review is currently available. Your included PR review attempts over the past 7 days set your current allowance at 4 reviews per hour.

---

📝 Summary

Summary by CodeRabbit

• Bug Fixes
  • Improved handling of datasets with extremely large or overflowing dimensions.
  • Reads now return clear, structured errors when allocation limits or tensor dimensions cannot be safely represented.
  • String data size validation now accounts for all required storage components, including variable-length payloads.
  • Added regression coverage for scalar, zero-sized, bounded, and unbounded dataset reads.

Walkthrough

The HDF5 reader now uses checked shape products and explicit allocation accounting. Overflow produces structured errors for bounded and unbounded reads. Tests cover scalar, zero-extent, string, and overflowing-extents datasets.

Changes

HDF5 allocation validation

Layer / File(s)	Summary
Checked shape products src/io/hdf5_read.rs	Dataset element counts now handle scalar and zero-extent shapes. Shape multiplication overflow now produces read-limit errors for bounded reads and invalid-tensor errors for unbounded reads.
Allocation accounting and unit coverage src/io/hdf5_read.rs	Tensor and string allocation checks use checked counts. Variable-length strings account for descriptors, headers, and payloads. Fixed strings account for capacity and width. Tests cover scalar, zero-extent, bounded, and unbounded cases.
Overflow fixture regression tests/common/mod.rs, tests/fixtures/overflowing_extents.nir.hex, tests/hdf5_untrusted.rs	The test helper decodes hexadecimal fixtures. The regression fixture contains overflowing dataset extents. The test verifies InvalidTensor for unbounded reads and ReadLimitExceeded for bounded reads.

Priority: ➖ Normal

Estimated code review effort: 3 (Moderate) | ~30 minutes

Change: Bug fix

Sequence Diagram(s)

sequenceDiagram
  participant Test
  participant decode_hex_fixture
  participant HDF5Fixture
  participant hdf5_read
  Test->>decode_hex_fixture: Decode fixture bytes
  decode_hex_fixture->>HDF5Fixture: Read overflowing-extents fixture
  decode_hex_fixture-->>Test: Create temporary HDF5 file
  Test->>hdf5_read: Perform bounded or unbounded read
  hdf5_read-->>Test: Return ReadLimitExceeded or InvalidTensor

Loading

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 75.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 20 functions across 3 files. (1 skipped: …	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Title check	✅ Passed	The title clearly and concisely describes the main changes: preventing unchecked dataset size and product overflow while centralizing allocation checks.
Description check	✅ Passed	The description is directly related to the changeset and explains the motivation, implementation, tests, and security impact of the allocation and dataset-size checks.

Full details: Docstring Coverage

Explanation

Docstring coverage is 75.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 20 functions across 3 files. (1 skipped: 1 unsupported.)

• Fix all pre-merge checks with AI

✨ Finishing Touches 💡 1

📝 Generate docstrings 💡

• Commit to this branch
• Create a new PR

✨ Simplify code

• Commit to this branch
• Create a new PR

---

_{Comment @coderabbitai help to get the list of available commands.}

[codecov]

Codecov Report

❌ Patch coverage is 86.31579% with 13 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
src/io/hdf5_read.rs	86.31%	13 Missing ⚠️

📢 Thoughts on this report? Let us know!