Cuvs lucene move#2353
Draft
imotov wants to merge 120 commits into
Draft
Conversation
…s-lucene#2) Fixes NVIDIA/cuvs-lucene#1 Things included in this PR: - Add the spotless maven plugin configuration. - Add the gitignore file. - Add a source license-header file that is used by the spotless plugin (to add/update license headers automatically if missing). - Update the readme file. Authors: - Vivek Narang (https://github.com/narangvivek10) - Gil Forsyth (https://github.com/gforsyth) Approvers: - Corey J. Nolet (https://github.com/cjnolet) - MithunR (https://github.com/mythrocks) URL: NVIDIA/cuvs-lucene#2
Fixes NVIDIA/cuvs-lucene#8 Authors: - Vivek Narang (https://github.com/narangvivek10) Approvers: - Corey J. Nolet (https://github.com/cjnolet) URL: NVIDIA/cuvs-lucene#9
Fixes NVIDIA/cuvs-lucene#5 Main changes in this PR: - Adapt to the cuVS-Java API updates: Mainly replace the use of `Dataset` with the new `CuVSMatrix` - Disable the tests that use byte vector encoding, as we do not support it. Authors: - Vivek Narang (https://github.com/narangvivek10) Approvers: - Corey J. Nolet (https://github.com/cjnolet) URL: NVIDIA/cuvs-lucene#10
Add CI files and workflows
This PR adds a few additional needed changes to enable CI. - change permission to make `.sh` files executable - removes unnecessary configuration from `.pre-commit-config.yaml`, `CODEOWNERS`, `dependencies.yaml` - updates all `pre-commit` hooks - adds `rapids-dependency-file-generator` hook (to validated `dependencies.yaml`) - fixes all issues found by `pre-commit` hooks (broken shell scripts, trailing whitespace, etc.) - fixes versioning in `pom.xml` Authors: - Dante Gama Dessavre (https://github.com/dantegd) - James Lamb (https://github.com/jameslamb) Approvers: - James Lamb (https://github.com/jameslamb) - Corey J. Nolet (https://github.com/cjnolet) URL: NVIDIA/cuvs-lucene#15
* updates `rapids-dependency-file-generator`, to pull in changes from rapidsai/dependency-file-generator#163 * removes unnecessary error-trapping in shell scripts Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Kyle Edwards (https://github.com/KyleFromNVIDIA) URL: NVIDIA/cuvs-lucene#19
Contributes to rapidsai/build-planning#208 * uses CUDA 13.0.0 to build and test (using the same patterns from the `cuvs-java` tests, in NVIDIA#1273) ## Notes for Reviewers This switches GitHub Actions workflows to the `cuda13.0` branch from here: rapidsai/shared-workflows#413 A future round of PRs will revert that back to `branch-25.10`, once all of RAPIDS supports CUDA 13. Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Jake Awe (https://github.com/AyodeAwe) - Ben Frederickson (https://github.com/benfred) - rhdong (https://github.com/rhdong) URL: NVIDIA/cuvs-lucene#20
Contributes to rapidsai/build-planning#208 Now that rapidsai/shared-workflows#413 is merged, this converts all GitHub Actions references from `@cuda13.0` back to `branch-25.10`. ## Notes for Reviewers This is safe to admin-merge because the change is a no-op... configs on those 2 branches are identical.
…cene#22) This PR configures this repo (adds a `.github/release.yml` file) for automatic release notes generation. See https://docs.github.com/en/repositories/releasing-projects-on-github/automatically-generated-release-notes. This will fix such issues as hotfix releases including wrong content (e.g [cugraph v25.04.01](https://github.com/rapidsai/cugraph/releases/tag/v25.04.01) contains 50+ PRs from the full v25.04.00 release when it should only contain the single hotfix PR (#5017).) Example auto-generated release notes: https://github.com/rapidsai/literate-octo-potato/releases/tag/v25.10.00a
… prefiltering (NVIDIA/cuvs-lucene#14) Refactoring, CI fixes (pulling libcuvs from pypi if not found), prefiltering support. Added tests: * TestCuVSGaps (for missing vectors in documents) * TestCuVSDeletedDocuments (for deleted vectors, that will leverage prefiltering) * TestMerge (dedicated test for testing merges) Authors: - Ishan Chattopadhyaya (https://github.com/chatman) - Vivek Narang (https://github.com/narangvivek10) Approvers: - Mike Sarahan (https://github.com/msarahan) - Corey J. Nolet (https://github.com/cjnolet) URL: NVIDIA/cuvs-lucene#14
Introducing a new Codec that uses CAGRA for building the index on GPU and serializing to Lucene-compatible HNSW index segments. The Lucene-compatible segments are searchable via the `Lucene99HnswVectorsReader` (which is the default in Lucene 10.x). Note: This is based on top of NVIDIA/cuvs-lucene#14 and should be rebased once that is merged. TODO: - Benchmarks and more tests - Further refactoring to split the `CuVSVectorsFormat` into GPU and CPU-specific formats. Fixes NVIDIA/cuvs-lucene#13 Authors: - Vivek Narang (https://github.com/narangvivek10) - Puneet Ahuja (https://github.com/punAhuja) - Ishan Chattopadhyaya (https://github.com/chatman) Approvers: - Ishan Chattopadhyaya (https://github.com/chatman) - Corey J. Nolet (https://github.com/cjnolet) - Kyle Edwards (https://github.com/KyleFromNVIDIA) URL: NVIDIA/cuvs-lucene#16
Fixes NVIDIA/cuvs-lucene#7 Authors: - Vivek Narang (https://github.com/narangvivek10) Approvers: - Corey J. Nolet (https://github.com/cjnolet) URL: NVIDIA/cuvs-lucene#24
…n to give detailed stack traces and information on tests (NVIDIA/cuvs-lucene#25) Fixes NVIDIA/cuvs-lucene#26 Authors: - Vivek Narang (https://github.com/narangvivek10) Approvers: - Corey J. Nolet (https://github.com/cjnolet) URL: NVIDIA/cuvs-lucene#25
…cene upgrades (NVIDIA/cuvs-lucene#29) As per [last night's comment on the Solr Jira issue](https://issues.apache.org/jira/browse/SOLR-17892?focusedCommentId=18024203&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-18024203) we realized that exposing the Lucene Codecs in the SPI registry prevents any calling application from being able to upgrade the codec because, under the hood, the next version of Lucene moves the older codec (Lucene101) into a different package name. As a result, we should only provide the relevant vector format, reader, and writer, and the calling application needs to wrap them into its own FilterCodec. For the sake of the current tests, we have decided to move the codec classes into the tests so that they are not bundled in the artifacts. This issue is a blocker for the inclusion of cuvs Lucene in Solr or any other Lucene-based search engine. Authors: - Vivek Narang (https://github.com/narangvivek10) - Ishan Chattopadhyaya (https://github.com/chatman) Approvers: - Ishan Chattopadhyaya (https://github.com/chatman) - MithunR (https://github.com/mythrocks) URL: NVIDIA/cuvs-lucene#29
…cuvs-lucene#30) We were inadvertently copying over the vectors from a list to an array before creating the CuVSMatrix instance. This is memory inefficient. Authors: - Ishan Chattopadhyaya (https://github.com/chatman) Approvers: - Corey J. Nolet (https://github.com/cjnolet) URL: NVIDIA/cuvs-lucene#30
…VIDIA/cuvs-lucene#31) * cuvs-java 25.10 is released, use that from Maven Central * There's a blocker bug for Solr integration where single vector indexes cause a Solr crash. Adding a graceful handling for that. * Updated README.md with a more accurate description of the project. Authors: - Ishan Chattopadhyaya (https://github.com/chatman) Approvers: - Corey J. Nolet (https://github.com/cjnolet) URL: NVIDIA/cuvs-lucene#31
## Description Supports rollout of new branching strategy. https://docs.rapids.ai/notices/rsn0047/ xref: rapidsai/build-planning#224
Contributes to rapidsai/build-planning#224 ## Notes for Reviewers This is safe to admin-merge because the change is a no-op... configs on those 2 branches are identical.
…vs-lucene#28) Fixes NVIDIA/cuvs-lucene#27 Authors: - Vivek Narang (https://github.com/narangvivek10) Approvers: - Ishan Chattopadhyaya (https://github.com/chatman) - Corey J. Nolet (https://github.com/cjnolet) URL: NVIDIA/cuvs-lucene#28
This commit fixes the publication of cuvs-lucene artifacts to Github. The Github actions build-logs indicate that while the JAR is built correctly, it doesn't seem to be uploaded properly as a Github artifact. It appears to be the result of a minor typo in the artifact's base dir. This commit fixes the typo, and allows the artifact to be published. Authors: - MithunR (https://github.com/mythrocks) Approvers: - Nate Rock (https://github.com/rockhowse) URL: NVIDIA/cuvs-lucene#34
…onversion from CAGRA to HNSW (NVIDIA/cuvs-lucene#140) The natural number of HNSW layers is ln(segment_size)/ln(graph_degree), which is often around 6, yet we had a constraint that was limiting us to a maximum of 3 HNSW layers. This change effectively makes the maximum number of layers unbounded. Authors: - https://github.com/nvzm123 - Corey J. Nolet (https://github.com/cjnolet) Approvers: - Corey J. Nolet (https://github.com/cjnolet) URL: NVIDIA/cuvs-lucene#140
This reverts commit 69bb9da.
…26.10 Update to 26.10
After the upgrade to 26.10 in NVIDIA/cuvs-lucene#176, the build is failing due to the absence of the rapidsai/ci-conda:26.10-cuda12.9.1-ubuntu24.04-py3.13 image on Docker Hub. We also need to update the CUDA version to 12.9.2. Authors: - Igor Motov (https://github.com/imotov) Approvers: - Bradley Dice (https://github.com/bdice) URL: NVIDIA/cuvs-lucene#179
Contributor
Author
|
/ok to test 4b41d07 |
imotov
added
improvement
Contributor
Author
|
/ok to test 150b1de |
Contributor
Author
|
/ok to test 39aa307 |
Contributor
Author
|
/ok to test 3025018 |
Contributor
Author
|
/ok to test 085107c |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
19 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Moves https://github.com/NVIDIA/cuvs-lucene into
java/cuvs-lucenefolder of cuVs.This is a preliminary PR to validate some of the CI scripts. We'll need to rerun parts of the process and rebase the PR when we're ready to make the move.
Known issues:
#<some number here>in commit history are replaced withNVIDIA/cuvs-lucene#<some number here>, this creates a nice navigable link but creates two links on each historic issue - one pointing tocuvs-lucenecommit and another to copiedcuvscommit.cuvs-lucene-codeownersor move ownership tocuvs-java-codeowners.Open Questions:
mainorrelease/26.08as well?cuvs-lucenerepo post move?Fixes NVIDIA/cuvs-lucene#168