fix(pchronicle): non-loopback serve and lazy Directory discovery - #128
Merged
Conversation
Stop recursive object-store listing that hit max_files on large prefixes, classify Directory vs Dataset from shallow markers, and let catalog serve bind/listen before discovery finishes. Co-authored-by: Cursor <cursoragent@cursor.com>
Consolidated multiple lines of code into single lines for improved clarity and readability in the `discovery.rs` and `generic.rs` files. This change enhances the overall code structure without altering functionality.
…ry logic Added a new `Directory` variant to the `Candidate` enum to represent navigational directories in the catalog. Updated the discovery logic to classify immediate child directories and dataset sources separately, ensuring that loose files are not registered as sources. Enhanced the sorting and counting of sources in the catalog to accommodate the new directory type. Updated documentation to reflect these changes.
…ehavior Added support for writing and reading relative bytes in the DatasetLocation, enabling recursive listing of importable JSON objects. Improved the import process to clear existing prefixes in object stores before writing new data, ensuring a clean slate for imports. Updated documentation to clarify the behavior of the replace mode for object-store datasets, emphasizing that it clears the destination prefix before writing. Added tests to verify the new import behavior and ensure correct handling of existing data.
… modules Enhanced the formatting of code in the `location.rs`, `exchange.rs`, `sync.rs`, and `tests.rs` files for better clarity. This includes consolidating multiple lines into single lines where appropriate and ensuring consistent indentation. No functional changes were made, focusing solely on code aesthetics and maintainability.
…toryline manifest handling Added a new module for tracking index build progress, allowing for better visibility during long-running operations. Enhanced the `ChronicleManifest` to support a new storyline format and added methods for writing and loading storyline manifests. Updated the `commit_pending_content` function to conditionally build indexes based on the new options. Improved the discovery logic to classify storyline datasets correctly and ensure proper handling of their metadata. This update aims to improve the user experience during data imports and management.
…zation Improved the handling of timestamps in the Storyline format by adding lenient parsing methods for various timestamp string formats. Introduced a new module for sanitizing JSON input to handle non-standard tokens like NaN and Infinity, ensuring compatibility with scientific and Python-generated data. Updated the StorylineTimestamp struct to support optional timestamps and refined the deserialization process. This update aims to enhance robustness and flexibility in timestamp processing across different formats.
Added a `test_client` function to create a reqwest client that ignores proxy settings, enhancing the reliability of echo tests. Updated existing test cases to utilize this new client, ensuring consistent behavior across different test scenarios.
…_at timestamps Updated the test case to demonstrate that invalid non-null created_at values are now soft-accepted, allowing for the parsing of input without errors. Adjusted assertions to verify that timestamps are correctly handled as None in the resulting stories.
…ents Removed the suppression of child directory flags in the discovery process and clarified comments regarding the treatment of unknown Lance sidecars. Adjusted test cases to reflect changes in directory structure handling, ensuring accurate representation of file sources during discovery.
… directory discovery issues
Updated test cases to create directories under the agent mount instead of the root directory. This change ensures that shallow directory discovery correctly identifies sources and maintains the integrity of the test environment. Additionally, clarified comments regarding the placement of the broken Storyline marker to enhance understanding of the directory structure during tests.
Refactored the test setup to ensure that event files are correctly relocated beside their respective agent directories. This change enhances shallow directory discovery by maintaining a consistent dataset structure. Updated assertions to reflect the new file paths in routed SQL queries, ensuring accurate test outcomes.
…et handling Updated test cases to utilize the new examples_corpus function for dataset paths, enhancing clarity and consistency in dataset management across tests. This change ensures that tests reference the flat multi-format corpus, improving shallow directory discovery and overall test reliability.
Introduced a new `--home-link TEXT=PATH` option to the CLI, allowing users to specify additional navigation links on the homepage. This feature supports same-origin relative paths, enhancing user experience by providing quick access to relevant resources. Updated the server configuration to include these links and ensured they are correctly parsed and validated. Additionally, added tests to verify the functionality and integration of home links within the warehouse configuration.
Introduced a new `home-preview.html` file to enhance the user interface of the web application, providing a structured layout for the homepage. Updated the `home.css` file to refine styles, including adjustments to the terminal and tab components for better visual alignment and user experience. Additionally, added a comment in the tests to clarify the purpose of the `examples_corpus` function.
…bility Modified the homepage heading in `home-preview.html` to include a line break for better visual separation. Updated CSS styles in `home.css` to remove the maximum width constraint on the heading and adjusted font size and line height for enhanced readability.
Deleted the obsolete `home-preview.html` file to streamline the project structure. Updated `home.css` to improve button hover effects for better user interaction. This cleanup aligns with ongoing efforts to refine the web application's user interface.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
catalog://pins so ECS/LAN listeners and pins work.chronicle.manifest/CURRENT/ events markers → Dataset; plain prefixes only probe immediate children (no loose-file Sources).servediscovery in the background so--listencan accept connections before large prefixes finish classifying; update RFCs 0013/0015 accordingly.Test plan
just test persisting-pchroniclejust test persisting-pchronicle-cli(loopback/pin tests flipped to accept non-loopback)pchronicle serve --catalog-config … --listen 127.0.0.1:8081should bind withoutmax_filesrecursive-list failurecatalog://<non-loopback-ip>:portand confirm it is accepted*/events.lancestill appears in catalog/projection inventoryMade with Cursor