Vectorize _split_overlaps, fixes #602 - #669
Open
steps-re wants to merge 1 commit into
Open
Conversation
For naive and UTC timestamps, frequency borders are uniform, so all segments are now computed at once with integer nanosecond arithmetic: count borders inside each record, repeat rows, and fill segment bounds with numpy, replacing the per-row apply that built a pd.date_range list for every record before exploding. Timezones with variable UTC offset keep the row-wise splitting for correctness (calendar days are not always 24h there), but only for records that actually contain a border, everything else passes through unchanged. Measured on 100k synthetic records: 4.5s to 0.008s (~500x) for typical staypoint durations at both day and hour granularity, 45x in the extreme case where records explode into 39 segments each. Split results are identical to the previous implementation across naive/UTC/CET data, ns and us datetime units, on-border starts, and zero-duration records. Input datetime dtype (including non-nano units) is now preserved in the output.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This vectorizes
_split_overlaps, addressing #602.For naive and UTC timestamps, frequency borders are uniform, so all segments are computed at once with integer nanosecond arithmetic: count the borders inside each record, repeat rows with numpy, and fill the segment bounds. Timezones with a variable UTC offset keep the row-wise splitting for correctness (calendar days are not always 24h there), but only for records that actually contain a border. Everything else passes through unchanged.
Measured on 100k synthetic records: 4.5s to 0.008s for typical staypoint durations, at both day and hour granularity. The worst case I tried (records spanning 39 segments each, 3.9M output rows) is still 45x faster.
Split results are identical to the previous implementation across naive, UTC, and CET data, ns and us datetime units, on-border starts, and zero-duration records. Input datetime dtype (including non-nano units) is now preserved in the output.
One note from testing: the existing
_get_timescrashes for tz-aware data whenceillands on a DST spring-forward gap (for example Europe/Zurich, hourly granularity, records around 2023-03-26 02:00). That crash predates this PR. The new vectorized path avoids it for naive and UTC data, the remaining row-wise path still has it. Happy to file it as a separate issue.Built with AI assistance and verified locally: full test suite passes (the single test_tutorial failure on my machine is unrelated and present on a clean checkout).