From 7ec2547476400af523ecce3294cc40f7e521d806 Mon Sep 17 00:00:00 2001 From: Kurt Heiss Date: Tue, 7 Jul 2026 16:01:31 -0700 Subject: [PATCH 1/6] docs(26.05): split Kurt extraction reconcile updates Port Kurt-owned extraction workflow, API reference, support matrix, and nav/IA reconcile changes from PR 2311 onto 26.05. Includes the Greptile fixes for ASRParams and mkdocstrings module paths. --- .../extraction/agentic-retrieval-concept.md | 2 +- docs/docs/extraction/audio-video.md | 2 + docs/docs/extraction/concepts.md | 30 +- docs/docs/extraction/custom-metadata.md | 125 ---- docs/docs/extraction/customize-extend.md | 66 ++ docs/docs/extraction/embedding.md | 8 +- ...egrations-langchain-llamaindex-haystack.md | 22 - docs/docs/extraction/multimodal-extraction.md | 21 +- .../nemo-retriever-api-reference.md | 4 +- docs/docs/extraction/nimclient.md | 588 ------------------ .../prerequisites-support-matrix.md | 62 +- docs/docs/extraction/starter-kits.md | 24 + .../extraction/workflow-agentic-retrieval.md | 23 +- .../extraction/workflow-document-ingestion.md | 22 +- .../extraction/workflow-e2e-blueprints.md | 2 +- docs/mkdocs.yml | 40 +- 16 files changed, 204 insertions(+), 837 deletions(-) delete mode 100644 docs/docs/extraction/custom-metadata.md create mode 100644 docs/docs/extraction/customize-extend.md delete mode 100644 docs/docs/extraction/integrations-langchain-llamaindex-haystack.md delete mode 100644 docs/docs/extraction/nimclient.md create mode 100644 docs/docs/extraction/starter-kits.md diff --git a/docs/docs/extraction/agentic-retrieval-concept.md b/docs/docs/extraction/agentic-retrieval-concept.md index a06431a359..b2bfd09cfd 100644 --- a/docs/docs/extraction/agentic-retrieval-concept.md +++ b/docs/docs/extraction/agentic-retrieval-concept.md @@ -7,4 +7,4 @@ NeMo Retriever Library focuses on document ingestion, embeddings, vector stores, **Related** - [Semantic retrieval](vdbs.md#semantic-retrieval) -- Framework examples: [LangChain, LlamaIndex, Haystack](integrations-langchain-llamaindex-haystack.md) +- Framework examples: [Starter kits](starter-kits.md) diff --git a/docs/docs/extraction/audio-video.md b/docs/docs/extraction/audio-video.md index c9031ce413..6a6cfa21a6 100644 --- a/docs/docs/extraction/audio-video.md +++ b/docs/docs/extraction/audio-video.md @@ -100,6 +100,8 @@ Instead of running the pipeline locally, you can call Parakeet through [build.nv 2. Run inference from Python with the hosted gRPC endpoint and credentials from that page (the example below uses the default hosted gRPC hostname; confirm values in the **Get API Key** flow for your deployment). Pass hosted endpoint, function ID, and API key through `ASRParams` (`audio_endpoints`, `function_id`, `auth_token`). + For parameter details, refer to the [Python API guide](nemo-retriever-api-reference.md). + ```python from nemo_retriever import create_ingestor from nemo_retriever.params.models import ASRParams diff --git a/docs/docs/extraction/concepts.md b/docs/docs/extraction/concepts.md index 4418682052..bb6ce80998 100644 --- a/docs/docs/extraction/concepts.md +++ b/docs/docs/extraction/concepts.md @@ -2,40 +2,40 @@ These terms appear throughout NeMo Retriever Library documentation. -## Job +## Job { #job } -An **ingestion job** is a unit of work you run on input content (documents, audio, video, and other supported types). You submit jobs through the **ingestor Python API** (for example `Ingestor` task chains such as `.extract(...)`) or the **`retriever ingest` CLI**—not by posting a standalone JSON job document. Default tasks target strong recall; customize behavior with task keyword arguments (including chunking and splitting on `.extract()`) or custom UDF-style operations ([NeMo Retriever graph](https://github.com/NVIDIA/NeMo-Retriever/tree/main/nemo_retriever/src/nemo_retriever/graph#nemo-retriever-graph)). Results are structured metadata and annotations (Ray Dataset, pandas `DataFrame`, or similar). +An **ingestion job** is a unit of work you run on input content (documents, audio, video, and other supported types). You submit jobs through the **ingestor Python API** (for example `Ingestor` task chains such as `.extract(...)`) or the **`retriever ingest` CLI**—not by posting a standalone JSON job document. Default tasks target strong recall; customize behavior with task keyword arguments (including chunking and splitting on `.extract()`) or custom UDF-style operations. For UDFs and other extension paths, refer to [Customize & extend](customize-extend.md). Results are structured metadata and annotations (Ray Dataset, pandas `DataFrame`, or similar). -## Pipeline and tasks +## Pipeline and tasks { #pipeline-and-tasks } -NeMo Retriever Library does **not** run one static pipeline on every document. You configure **tasks** such as parsing, chunking, embedding, storage, and filtering per job. Related topics: [Extending/Customizing NeMo Retriever Library with custom code](https://github.com/NVIDIA/NeMo-Retriever/tree/main/nemo_retriever/src/nemo_retriever/graph#nemo-retriever-graph). +NeMo Retriever Library does **not** run one static pipeline on every document. You configure **tasks** such as parsing, chunking, embedding, storage, and filtering per job. For UDFs, custom graph stages, and other extension paths, refer to [Customize & extend](customize-extend.md). -## Extraction metadata +## Extraction metadata { #extraction-metadata } Output is a **Ray Dataset** (Ray Data) or **pandas** `DataFrame` listing extracted objects (text regions, tables, images, and so on), processing notes, and timing or trace data. Field-level detail is in the [metadata reference](content-metadata.md). -## Embeddings and retrieval +## Embeddings and retrieval { #embeddings-and-retrieval } -Optionally, the library can compute **embeddings** for extracted content and store vectors in [LanceDB](https://lancedb.com/) for downstream semantic search in your application. For upload and retrieval APIs, see [Vector databases](vdbs.md). For multimodal (VLM) embedding options, see [Multimodal embeddings (VLM)](embedding.md). +Optionally, the library can compute **embeddings** for extracted content and store vectors in [LanceDB](https://lancedb.com/) for downstream semantic search in your application. For upload and retrieval APIs, refer to [Vector databases](vdbs.md). For multimodal (VLM) embedding options, refer to [Multimodal embeddings (VLM)](embedding.md). ## Chunking { #chunking } Chunking is built into the `.extract()` task and depends on **content type**: - **PDF, DOCX, and PPTX** — Text is grouped using built-in **page** boundaries (one chunk per page where the format has pages). -- **Plain text (`.txt`) and HTML** — Formats without natural page breaks are split into segments of **1024 tokens** by default, using the [Llama 3.2 1B tokenizer](https://huggingface.co/meta-llama/Llama-3.2-1B) so chunk boundaries stay aligned with the default embedding tokenizer. The NeMo Retriever container image bundles this tokenizer, so default text chunking does not require a Hugging Face access token. See [Token-based splitting](#token-based-splitting) and [Environment variables](environment-config.md) for overrides and other runtimes. -- **Audio and video** — Media is split into **segments** for decoding and ASR using ffmpeg-based rules (configurable **size**, **time**, or **frame** split modes in the media chunking stage). With the Parakeet ASR path, you can optionally emit **sentence-like segments** using `extract_audio_params={"segment_audio": True}`; see [Speech and audio extraction](audio-video.md#speech-and-audio-extraction). +- **Plain text (`.txt`) and HTML** — Formats without natural page breaks are split into segments of **1024 tokens** by default, using the [Llama 3.2 1B tokenizer](https://huggingface.co/meta-llama/Llama-3.2-1B) so chunk boundaries stay aligned with the default embedding tokenizer. The NeMo Retriever container image bundles this tokenizer, so default text chunking does not require a Hugging Face access token. Refer to [Token-based splitting](#token-based-splitting) and [Environment variables](environment-config.md) for overrides and other runtimes. +- **Audio and video** — Media is split into **segments** for decoding and ASR using ffmpeg-based rules (configurable **size**, **time**, or **frame** split modes in the media chunking stage). With the Parakeet ASR path, you can optionally emit **sentence-like segments** using `extract_audio_params={"segment_audio": True}`; refer to [Speech and audio extraction](audio-video.md#speech-and-audio-extraction). -For PDF parallelism before Ray processing (large files), see [PDF pre-splitting for parallel ingest](nemo-retriever-api-reference.md#pdf-pre-splitting-for-parallel-ingest). +For PDF parallelism before Ray processing (large files), refer to [PDF pre-splitting for parallel ingest](nemo-retriever-api-reference.md#pdf-pre-splitting-for-parallel-ingest). ### Token-based splitting { #token-based-splitting } -Token-based splitting uses the Llama 3.2 1B tokenizer (default `meta-llama/Llama-3.2-1B`) with configurable `max_tokens` and `overlap_tokens` when you add an explicit `.split(...)` stage or when the pipeline applies the default text segmentation for unstructured text. In the shipped NeMo Retriever container, tokenizer assets are included locally, so you do not need `HF_ACCESS_TOKEN` for this default path. If your runtime loads the tokenizer from the Hugging Face Hub instead (for example, some library-only installs), set `HF_ACCESS_TOKEN` or pass `hf_access_token` in task params when the Hub requires it. Details appear in the [Python API guide](nemo-retriever-api-reference.md). +Token-based splitting uses the Llama 3.2 1B tokenizer (default `meta-llama/Llama-3.2-1B`) with configurable `max_tokens` and `overlap_tokens` when you add an explicit `.split(...)` stage or when the pipeline applies the default text segmentation for unstructured text. In the shipped NeMo Retriever container, tokenizer assets are included locally, so you do not need `HF_ACCESS_TOKEN` for this default path. If your runtime loads the tokenizer from the Hugging Face Hub instead (for example, some library-only installs), set `HF_ACCESS_TOKEN` or pass `hf_access_token` in task params when the Hub requires it. For parameter details, refer to the [Python API guide](nemo-retriever-api-reference.md). -## Deployment modes +## Deployment modes { #deployment-modes } -- **Library mode** — Run without the full container stack where appropriate; see [Deployment options](deployment-options.md). -- **Kubernetes / Helm (self-hosted)** — See [Deploy (Helm chart)](https://github.com/NVIDIA/NeMo-Retriever/blob/main/nemo_retriever/helm/README.md) and [deployment options](deployment-options.md) for running the full microservices pipeline on your infrastructure. -- **Notebooks** — [Jupyter examples](notebooks/index.md) for experimentation and RAG demos. +- **Library mode** — Run without the full container stack where appropriate; refer to [Deployment options](deployment-options.md). +- **Kubernetes / Helm (self-hosted)** — Refer to [Deploy (Helm chart)](https://github.com/NVIDIA/NeMo-Retriever/blob/main/nemo_retriever/helm/README.md) and [deployment options](deployment-options.md) for running the full microservices pipeline on your infrastructure. +- **Notebooks** — [Jupyter examples](starter-kits.md) for experimentation and RAG demos. For a concise comparison, refer to [Deployment options](deployment-options.md). diff --git a/docs/docs/extraction/custom-metadata.md b/docs/docs/extraction/custom-metadata.md deleted file mode 100644 index 45ffa34d29..0000000000 --- a/docs/docs/extraction/custom-metadata.md +++ /dev/null @@ -1,125 +0,0 @@ -# Custom metadata and filtering - -Use this documentation to attach per-document metadata during ingestion and to narrow [LanceDB](vdbs.md) search results in [NeMo Retriever Library](overview.md). Implementation details live in the package [Vector DB operators and LanceDB](https://github.com/NVIDIA/NeMo-Retriever/tree/main/nemo_retriever/src/nemo_retriever/vdb#metadata-filtering) README. - -## On this page { #on-this-page } - -- [Attach metadata at ingestion](#attach-metadata-at-ingestion) -- [Best practices](#best-practices) -- [Filter results during retrieval](#filter-results-during-retrieval) -- [How metadata is stored](#how-metadata-is-stored) - -## Attach metadata at ingestion { #attach-metadata-at-ingestion } - -Pass a **sidecar metadata table** on `vdb_upload` so selected columns are merged into each chunk's `content_metadata` before LanceDB upload. All three parameters must be set together: - -| Parameter | Purpose | -|-----------|---------| -| `meta_dataframe` | Path to CSV, JSON, or Parquet, or an in-memory `pandas.DataFrame` | -| `meta_source_field` | Column that identifies each document (must match ingest paths or basenames per `meta_join_key`) | -| `meta_fields` | Non-empty list of column names to copy into `content_metadata` | - -Optional `meta_join_key` controls how rows are matched to documents: `auto` (try full path then basename), `source_id` (full path), or `source_name` (basename only). - -For parameter details, refer to the [Python API guide](nemo-retriever-api-reference.md). - -```python -import pandas as pd -from nemo_retriever import create_ingestor - -meta_df = pd.DataFrame( - { - "source": ["data/woods_frost.pdf", "data/multimodal_test.pdf"], - "meta_a": ["alpha", "bravo"], - "meta_b": [10, 20], - } -) - -hostname = "localhost" -table_name = "nemo_retriever_collection" -lancedb_uri = "s3://your-bucket/lancedb" - -ingestor = ( - create_ingestor(run_mode="service", base_url=f"http://{hostname}:7670") - .files(["data/woods_frost.pdf", "data/multimodal_test.pdf"]) - .extract( - extract_text=True, - extract_tables=True, - extract_charts=True, - extract_images=True, - text_depth="page" - ) - .embed() - .vdb_upload( - vdb_op="lancedb", - vdb_kwargs={"lancedb_uri": lancedb_uri, "table_name": table_name}, - meta_dataframe=meta_df, - meta_source_field="source", - meta_fields=["meta_a", "meta_b"], - ) -) -results = ingestor.ingest_async().result() -``` - -Set `hostname`, `table_name`, and a **remote** `lancedb_uri` (for example `s3://bucket/path`) to match your deployment—the retriever service rejects local filesystem paths. The client uploads in-memory sidecar metadata to the service before ingest; do not pass a raw local file path as `meta_dataframe` on the REST spec. For local LanceDB directories, use `run_mode="batch"` instead (refer to [Vector databases](vdbs.md)). For a step-by-step walkthrough with additional fields such as category, department, and timestamp, refer to [Vector DB operators and LanceDB — Metadata filtering](https://github.com/NVIDIA/NeMo-Retriever/tree/main/nemo_retriever/src/nemo_retriever/vdb#metadata-filtering). - -## Best practices { #best-practices } - -- Plan metadata structure before ingestion. -- Test filter expressions with small datasets first. -- Consider performance implications of complex filters. -- Validate metadata during ingestion. -- Handle missing metadata fields gracefully. -- Log invalid filter expressions. - -## Filter results during retrieval { #filter-results-during-retrieval } - -You can use custom metadata to filter documents during retrieval operations. For **predicate pushdown**, pass a `where` SQL predicate through [`Retriever.query`](nemo-retriever-api-reference.md) (refer to [Vector databases](vdbs.md)) or chain `.where(...)` on a native LanceDB `table.search(...)` query. Application-side filtering on returned hits does not change what the database evaluates—raise `top_k` if matches might sit outside the first neighbors. - -### Example filter ideas - -Typical keys to filter on include `category`, `department`, `priority`, and `timestamp` (use comparable ISO-8601 strings for time ranges). Encode predicates in LanceDB SQL against your table columns (often the serialized `metadata` string), or inspect parsed hit metadata after search as in the example below. - -### Example: Use a Filter Expression in Search - -After ingestion is complete and documents are uploaded to LanceDB with metadata, you can narrow results in the database with a **`where`** clause, or in Python on the returned hits. - -**Native LanceDB (SQL pushdown):** connect, embed the query yourself (same model as ingestion), then chain `.where("")` on `table.search(...)` so filtering happens before the `limit`. Exact SQL depends on how `metadata` is stored; refer to [LanceDB metadata filtering](https://docs.lancedb.com/search/filtering#filtering-with-sql). - -```python -import lancedb - -# pseudocode — replace YOUR_VECTOR and YOUR_PREDICATE with real values. -db = lancedb.connect("./lancedb_data") -table = db.open_table("nemo_retriever_collection") -# table.search(YOUR_VECTOR, vector_column_name="vector").where(YOUR_PREDICATE).limit(10).to_list() -``` - -**`Retriever.query` + `where`:** LanceDB applies the predicate before ranking. For post-filter logic in Python, use a wider `top_k` first. - -```python -from nemo_retriever.retriever import Retriever - -retriever = Retriever( - vdb_kwargs={"uri": "./lancedb_data", "table_name": "nemo_retriever_collection"}, - embed_kwargs={ - "model_name": "nvidia/llama-nemotron-embed-1b-v2", - "embed_model_name": "nvidia/llama-nemotron-embed-1b-v2", - }, -) - -hits = retriever.query( - "this is expensive", - top_k=16, - vdb_kwargs={"where": "metadata LIKE '%\"department\":\"Engineering\"%'"}, -) -``` - -For a runnable end-to-end flow (ingest, `Retriever.query`, and both filter modes), refer to [nemo_retriever_retriever_query_metadata_filter.ipynb](https://github.com/NVIDIA/NeMo-Retriever/blob/main/examples/nemo_retriever_retriever_query_metadata_filter.ipynb). - -When you ingest through the **retriever service**, upload the sidecar with [`POST /v1/ingest/sidecar`](https://github.com/NVIDIA/NeMo-Retriever/blob/main/nemo_retriever/src/nemo_retriever/service/routers/ingest.py#L1040-L1129) (multipart file; response [`SidecarUploadResponse`](https://github.com/NVIDIA/NeMo-Retriever/blob/main/nemo_retriever/src/nemo_retriever/service/models/responses.py#L60-L68)), then pass the returned `sidecar_id` as `meta_dataframe_id` with `meta_source_field` and `meta_fields` in `pipeline.vdb_upload_params` on [`POST /v1/ingest`](https://github.com/NVIDIA/NeMo-Retriever/blob/main/nemo_retriever/src/nemo_retriever/service/models/requests.py#L15-L32) ([`PipelineSpec`](https://github.com/NVIDIA/NeMo-Retriever/blob/main/nemo_retriever/src/nemo_retriever/service/models/pipeline_spec.py#L55-L78)). Request and response shapes, form fields, and auth headers are in the service OpenAPI UI at `/docs` (or `/openapi.json`) on your retriever base URL (for example `http://localhost:7670/docs` after `retriever service start`). Do not send a raw local path as `meta_dataframe` on the service spec. - -## How metadata is stored { #how-metadata-is-stored } - -- [Vector databases](vdbs.md) — canonical LanceDB upload and retrieval guide -- [nemo_retriever_retriever_query_metadata_filter.ipynb](https://github.com/NVIDIA/NeMo-Retriever/blob/main/examples/nemo_retriever_retriever_query_metadata_filter.ipynb) — runnable notebook for sidecar metadata at ingest and filtered `Retriever.query` diff --git a/docs/docs/extraction/customize-extend.md b/docs/docs/extraction/customize-extend.md new file mode 100644 index 0000000000..31fb87e7bc --- /dev/null +++ b/docs/docs/extraction/customize-extend.md @@ -0,0 +1,66 @@ +# Customize & extend + +NeMo Retriever Library ships with defaults tuned for strong recall on common document types. When those defaults are not enough, you can extend the library at several levels—from task keyword arguments on the fluent ingestor API through custom graph operators and vector-database adapters. + +Use this page to choose an extension path and find the detailed guides in the repository. + +The following table maps common needs to the right section: + +| If you need to… | Start here | +|-----------------|------------| +| Tune extraction, chunking, embedding, or upload without new code | [Start with task configuration](#start-with-task-configuration) | +| Add a small Python transformation between pipeline stages | [User-defined functions (UDFs)](#user-defined-functions-udfs) | +| Build or reuse operators stage-by-stage | [Custom graph pipelines](#custom-graph-pipelines) | +| Store vectors in a backend other than LanceDB | [Custom vector databases](#custom-vector-databases) | + +## On this page { #on-this-page } + +- [Start with task configuration](#start-with-task-configuration) +- [User-defined functions (UDFs)](#user-defined-functions-udfs) +- [Custom graph pipelines](#custom-graph-pipelines) +- [Custom vector databases](#custom-vector-databases) +- [Related Topics](#related-topics) + +## Start with task configuration { #start-with-task-configuration } + +Most customization does not require new code. Chain tasks on `create_ingestor(...)` and pass keyword arguments to control extraction, chunking, embedding, and storage—for example `extract_method`, chunking and splitting options on `.extract()`, `embed_modality` on `.embed()`, and `vdb_op` / `vdb_kwargs` on `.vdb_upload()`. + +For parameter details, refer to the [Python API guide](nemo-retriever-api-reference.md). For chunking behavior and pipeline concepts, refer to [Concepts](concepts.md). + +## User-defined functions (UDFs) { #user-defined-functions-udfs } + +A **user-defined function (UDF)** wraps your Python logic as a first-class pipeline stage. In the graph model, `UDFOperator` turns a plain callable into an operator you can chain with built-in stages—for example to normalize HTML, apply a custom split, or call an external service between extract and embed steps. + +Use UDFs when you need a small, self-contained transformation that is not covered by task keyword arguments. + +### Repository guides + +- [NeMo Retriever graph README — `UDFOperator`](https://github.com/NVIDIA/NeMo-Retriever/tree/main/nemo_retriever/src/nemo_retriever/graph#using-udfoperator) — API, lifecycle, and when to use `UDFOperator` versus a custom operator class +- [NimClient and custom NIM endpoints](https://github.com/NVIDIA/NeMo-Retriever/blob/main/nemo_retriever/developer_docs/nimclient.md#nimclient-and-custom-nim-endpoints) — call custom or self-hosted NIM microservices from UDF stages + +## Custom graph pipelines { #custom-graph-pipelines } + +When you need to compose pipelines stage-by-stage, reuse operators across workflows, or run the same graph in-process or with Ray Data, use the **graph execution model** instead of (or alongside) the fluent `GraphIngestor` API. + +The graph package provides `AbstractOperator`, executors (`InprocessExecutor`, `RayDataExecutor`), and operator chaining with `>>`. Built-in ingestion operators live under `nemo_retriever.operators`; you can add your own operators or UDF stages anywhere in the chain. + +For the full guide—including custom operator classes, executors, and graph shape constraints—refer to the [NeMo Retriever graph README](https://github.com/NVIDIA/NeMo-Retriever/tree/main/nemo_retriever/src/nemo_retriever/graph#nemo-retriever-graph). + +## Custom vector databases { #custom-vector-databases } + +The supported user path for vector storage is **[LanceDB](vdbs.md)** (`vdb_op="lancedb"`). That page covers upload, semantic retrieval, metadata filtering, and LanceDB deployment characteristics. + +To integrate a different vector store, implement the [`VDB`](https://github.com/NVIDIA/NeMo-Retriever/blob/main/nemo_retriever/src/nemo_retriever/common/vdb/adt_vdb.py) interface and wire it through graph [`IngestVdbOperator`](https://github.com/NVIDIA/NeMo-Retriever/blob/main/nemo_retriever/src/nemo_retriever/operators/vdb.py) / [`RetrieveVdbOperator`](https://github.com/NVIDIA/NeMo-Retriever/blob/main/nemo_retriever/src/nemo_retriever/operators/vdb.py). NVIDIA validates the first-party LanceDB operator; you are responsible for testing and maintaining other backends. + +### Repository guides + +- [Vector DB package (source)](https://github.com/NVIDIA/NeMo-Retriever/tree/main/nemo_retriever/src/nemo_retriever/common/vdb) — `VDB` abstract base and LanceDB reference implementation + +Partner and blueprint integrations (Elasticsearch, Pinecone, Teradata, and others) are summarized on [Vector databases — Vector database partners](vdbs.md#vector-database-partners). + +## Related Topics { #related-topics } + +- [Concepts — Pipeline and tasks](concepts.md#pipeline-and-tasks) +- [Vector databases](vdbs.md) +- [Multimodal embeddings (VLM)](embedding.md) +- [Python API guide](nemo-retriever-api-reference.md) diff --git a/docs/docs/extraction/embedding.md b/docs/docs/extraction/embedding.md index e42574a519..b6a139fac3 100644 --- a/docs/docs/extraction/embedding.md +++ b/docs/docs/extraction/embedding.md @@ -12,8 +12,6 @@ The model can embed documents in the form of an image, text, or a combination of Documents can then be retrieved given a user query in text form. The model supports images that contain text, tables, charts, and infographics. -Parameter details for `.extract()` and `.embed()` appear in the [Python API guide](nemo-retriever-api-reference.md). - ## Example with Default Text-Based Embedding When you use the multimodal model, by default, all extracted content (text, tables, charts) is treated as plain text. @@ -21,6 +19,8 @@ The following example provides a strong baseline for retrieval. - The `embed` method is called with no arguments. +For parameter details, refer to the [Python API guide](nemo-retriever-api-reference.md). + ```python from nemo_retriever import create_ingestor @@ -42,6 +42,8 @@ The following example enables the multimodal model to capture the spatial and st - The `embed` method is configured with `embed_modality="text_image"` to embed the extracted tables and charts as images. - This configuration is more accurate than text only, with a performance cost. +For parameter details, refer to the [Python API guide](nemo-retriever-api-reference.md). + ```python from nemo_retriever import create_ingestor @@ -65,6 +67,8 @@ The following example extracts and embeds each page as an image. - The `embed` method processes the page images. +For parameter details, refer to the [Python API guide](nemo-retriever-api-reference.md). + ```python from nemo_retriever import create_ingestor diff --git a/docs/docs/extraction/integrations-langchain-llamaindex-haystack.md b/docs/docs/extraction/integrations-langchain-llamaindex-haystack.md deleted file mode 100644 index 7ee0dda650..0000000000 --- a/docs/docs/extraction/integrations-langchain-llamaindex-haystack.md +++ /dev/null @@ -1,22 +0,0 @@ -# Integrate with LangChain, LlamaIndex, and Haystack - -NeMo Retriever Library is commonly used **behind** retrieval-augmented generation (RAG) apps built with popular orchestration frameworks. - -## Jupyter examples (LangChain and LlamaIndex) - -The repository includes notebooks that demonstrate multimodal RAG patterns: - -- [Multimodal RAG with LangChain](https://github.com/NVIDIA/NeMo-Retriever/blob/main/examples/langchain_multimodal_rag.ipynb) -- [Multimodal RAG with LlamaIndex](https://github.com/NVIDIA/NeMo-Retriever/blob/main/examples/llama_index_multimodal_rag.ipynb) - -These are also linked from [Jupyter Notebooks](notebooks/index.md) and the [FAQ](faq.md). - -## Haystack - -Haystack-related extraction modes may appear in API tables as **deprecated** in favor of current pipeline options. For up-to-date integration patterns, prefer the Python API and CLI docs, and check [Release notes](releasenotes.md) for migration notes. - -## Related - -- [Use the Python API](nemo-retriever-api-reference.md) -- [Use the CLI](https://github.com/NVIDIA/NeMo-Retriever/tree/main/nemo_retriever/docs/cli) -- [Chunking](concepts.md#chunking), [Upload data](vdbs.md), [Filter search](custom-metadata.md) diff --git a/docs/docs/extraction/multimodal-extraction.md b/docs/docs/extraction/multimodal-extraction.md index 5e6a5a4fb5..f44aee36cf 100644 --- a/docs/docs/extraction/multimodal-extraction.md +++ b/docs/docs/extraction/multimodal-extraction.md @@ -27,7 +27,7 @@ NeMo Retriever Library accepts multiple document and media types. A current list For PDFs, NeMo Retriever Library typically uses **pdfium**-based extraction with configurable depth and paths. Scanned or mixed pages may use hybrid, OCR-oriented, or Nemotron Parse methods. For `extract_method` options such as `pdfium`, `pdfium_hybrid`, `ocr`, and `nemotron_parse`, refer to the [Python API reference](nemo-retriever-api-reference.md). !!! note - `extract_method="nemotron_parse"` requires the Nemotron Parse NIM client dependencies. Install them with the `nemotron-parse` extra, for example `pip install "nemo-retriever[nemotron-parse]"`, before running PDF extraction through Nemotron Parse. + `extract_method="nemotron_parse"` requires the Nemotron Parse NIM client dependencies. Install them with the `nemotron-parse` extra, for example `pip install "nemo-retriever[nemotron-parse]"`, before running PDF extraction through Nemotron Parse. This path does not produce chart modality rows; for chart detection, refer to [Charts and infographics](#charts-and-infographics). **Related** @@ -49,7 +49,18 @@ NeMo Retriever Library detects tables as structured page elements, processes the Charts and infographic regions are classified with other page layout elements (tables, text blocks, titles) and processed through layout detection and OCR. `extract_charts` and `extract_infographics` are enabled by default. Outputs use the same metadata schema as other extracted objects. -Chart-labeled PDF regions are **not** routed through the Omni caption stage; they remain on the layout-and-OCR path. For scope and validation guidance, see [Image captioning](prerequisites-support-matrix.md#image-captioning-2605). +!!! important "Chart modality requires the default layout path" + [Nemotron Parse v1.2](https://huggingface.co/nvidia/NVIDIA-Nemotron-Parse-v1.2) semantic classes do not include `Chart` or `Infographic`. The model labels regions as `Text`, `Table`, `Picture`, `Caption`, `List-item`, `Section-header`, and similar types instead. + + When you set `extract_method="nemotron_parse"`: + + - The pipeline does not produce `chart` or `infographic` modality rows, even when `extract_charts=True` or `extract_infographics=True`. + - Chart- and infographic-filtered retrieval (for example, queries scoped to figure or chart content) returns no hits. + - Chart-heavy and infographic-heavy pages are typically emitted as `Picture` or other non-chart modalities. + + For chart and infographic detection and modality-specific retrieval, use the default **pdfium** layout path (page-elements detection and OCR), not `extract_method="nemotron_parse"`. + +For how chart-labeled PDF regions interact with captioning, refer to [Image captioning](#image-captioning). For natural-language infographic descriptions, optionally enable [image captioning](#image-captioning) and set `caption_infographics=True` when you need VLM captions on infographic regions. @@ -63,7 +74,7 @@ For natural-language infographic descriptions, optionally enable [image captioni Scanned PDFs and image-only pages rely on OCR and hybrid paths that combine native text extraction with OCR when needed. For extract methods such as `ocr` and `pdfium_hybrid`, refer to the [Python API reference](nemo-retriever-api-reference.md). -OCR artifacts depend on how you deploy. **Helm / NIM:** the production chart uses **Nemotron OCR v1** (`nvcr.io/nim/nvidia/nemotron-ocr-v1:1.3.0`). **Local Hugging Face inference:** the default engine is **Nemotron OCR v2**, which operates in **multilingual** mode by default. For CLI flags and API parameters, see [Nemotron OCR v2 — language mode](https://github.com/NVIDIA/NeMo-Retriever/blob/main/nemo_retriever/docs/cli/README.md#nemotron-ocr-v2-language-mode). For Kubernetes defaults and the Helm-vs-local split, see [OCR artifacts (Helm vs local Hugging Face)](prerequisites-support-matrix.md#nemotron-ocr-v2-language-mode) in the support matrix. +When you run extraction locally with Hugging Face weights, the default OCR engine is **Nemotron OCR v2**, which operates in **multilingual** mode by default. For CLI flags and API parameters, refer to [Nemotron OCR v2 — language mode](https://github.com/NVIDIA/NeMo-Retriever/blob/main/nemo_retriever/docs/cli/README.md#nemotron-ocr-v2-language-mode). For Kubernetes deployment, refer to [OCR NIM configuration](https://github.com/NVIDIA/NeMo-Retriever/blob/main/nemo_retriever/helm/README.md#ocr-nim-configuration) in the Helm chart README. **Related** @@ -77,11 +88,13 @@ Image captioning generates natural-language descriptions for unstructured image **Captioning is optional** — enable it in your ingest configuration (for example, the `caption` API or pipeline flag) when you need natural-language descriptions of image content. Reasoning traces are disabled by default for captioning. +Chart-classified PDF regions stay on the layout/OCR path; only non-chart image regions and optional infographics (`caption_infographics=True`) receive Omni captions. + **Related** - [Multimodal embeddings (VLM)](embedding.md) - [Metadata reference](content-metadata.md) -- [Image captioning](prerequisites-support-matrix.md#image-captioning-2605) +- [Image captioning](prerequisites-support-matrix.md#image-captioning) ## Metadata and content schema { #metadata-and-content-schema } diff --git a/docs/docs/extraction/nemo-retriever-api-reference.md b/docs/docs/extraction/nemo-retriever-api-reference.md index da21b30a40..1b057431d3 100644 --- a/docs/docs/extraction/nemo-retriever-api-reference.md +++ b/docs/docs/extraction/nemo-retriever-api-reference.md @@ -1,10 +1,10 @@ # NeMo Retriever API Reference -## PDF pre-splitting for parallel ingest +## PDF pre-splitting for parallel ingest { #pdf-pre-splitting-for-parallel-ingest } Large PDFs are split into page batches before Ray processing so extraction can run in parallel. This happens on the default ingest path; you do not need extra configuration for typical workloads. -To tune splitter throughput from the CLI, use `--pdf-split-batch-size` (Ray actor batch size for the splitter stage). See [Text chunking and PDF page batches](https://github.com/NVIDIA/NeMo-Retriever/tree/main/nemo_retriever/docs/cli#text-chunking-and-pdf-page-batches) in the CLI reference. +To tune splitter throughput from the CLI, use `--pdf-split-batch-size` (Ray actor batch size for the splitter stage). Refer to [Text chunking and PDF page batches](https://github.com/NVIDIA/NeMo-Retriever/tree/main/nemo_retriever/docs/cli#text-chunking-and-pdf-page-batches) in the CLI reference. **Python client (`pdf_split_config`):** Only `create_ingestor(run_mode="service")` implements `.pdf_split_config(pages_per_chunk=...)`, which records page-chunking settings in the request pipeline spec for the remote gateway. Local graph ingest (`run_mode="inprocess"` or `"batch"`) raises `NotImplementedError` if you call this method; PDFs are split automatically on the default ingest path without client-side configuration. diff --git a/docs/docs/extraction/nimclient.md b/docs/docs/extraction/nimclient.md deleted file mode 100644 index 755db2b366..0000000000 --- a/docs/docs/extraction/nimclient.md +++ /dev/null @@ -1,588 +0,0 @@ -# NimClient Usage Guide for NeMo Retriever Library - -The `NimClient` class provides a unified interface for connecting to and interacting with NVIDIA NIM Microservices. -This documentation demonstrates how to create custom NIM integrations for use in [NeMo Retriever Library](overview.md) pipelines and User Defined Functions (UDFs). - -The NimClient architecture consists of two main components: - -1. **NimClient**: The client class that handles communication with NIM endpoints via gRPC or HTTP protocols -2. **ModelInterface**: An abstract base class that defines how to format input data, parse output responses, and process inference results for specific models - -For advanced usage patterns, refer to the existing model interfaces in `nemo_retriever/src/nemo_retriever/api/internal/primitives/nim/model_interface/`. - - -## Quick Start - -For ingest and pipeline APIs used with NimClient in UDFs, refer to the [Python API guide](nemo-retriever-api-reference.md). - -### Basic NimClient Creation - -```python -from nemo_retriever.api.util.nim import create_inference_client -from nemo_retriever.api.internal.primitives.nim import ModelInterface - -# Create a custom model interface (refer to examples below) -model_interface = MyCustomModelInterface() - -# Define endpoints (gRPC, HTTP) -endpoints = ("grpc://my-nim-service:8001", "http://my-nim-service:8000") - -# Create the client -client = create_inference_client( - endpoints=endpoints, - model_interface=model_interface, - auth_token="your-ngc-api-key", # Optional - infer_protocol="grpc", # Optional: "grpc" or "http" - timeout=120.0, # Optional: request timeout - max_retries=5 # Optional: retry attempts -) - -# Perform inference -data = {"input": "your input data"} -results = client.infer(data, model_name="your-model-name") -``` - -### Using Environment Variables - -```python -import os -from nemo_retriever.api.util.nim import create_inference_client - -# Use environment variables for configuration -auth_token = os.getenv("NGC_API_KEY") -grpc_endpoint = os.getenv("NIM_GRPC_ENDPOINT", "grpc://localhost:8001") -http_endpoint = os.getenv("NIM_HTTP_ENDPOINT", "http://localhost:8000") - -client = create_inference_client( - endpoints=(grpc_endpoint, http_endpoint), - model_interface=model_interface, - auth_token=auth_token -) -``` - -## Creating Custom Model Interfaces - -To integrate a new NIM, you need to create a custom `ModelInterface` subclass that implements the required methods. - -### Basic Model Interface Template - -```python -from typing import Dict, Any, List, Tuple, Optional -import numpy as np -from nemo_retriever.api.internal.primitives.nim import ModelInterface - -class MyCustomModelInterface(ModelInterface): - """ - Custom model interface for My Custom NIM. - """ - - def __init__(self, model_name: str = "my-custom-model"): - """Initialize the model interface.""" - self.model_name = model_name - - def name(self) -> str: - """Return the name of this model interface.""" - return "MyCustomModel" - - def prepare_data_for_inference(self, data: Dict[str, Any]) -> Dict[str, Any]: - """ - Prepare and validate input data before formatting. - - Parameters - ---------- - data : dict - Raw input data - - Returns - ------- - dict - Validated and prepared data - """ - # Validate required fields - if "input_text" not in data: - raise KeyError("Input data must include 'input_text'") - - # Ensure input is in the expected format - if not isinstance(data["input_text"], str): - raise ValueError("input_text must be a string") - - return data - - def format_input( - self, - data: Dict[str, Any], - protocol: str, - max_batch_size: int, - **kwargs - ) -> Tuple[List[Any], List[Dict[str, Any]]]: - """ - Format input data for the specified protocol. - - Parameters - ---------- - data : dict - Prepared input data - protocol : str - Communication protocol ("grpc" or "http") - max_batch_size : int - Maximum batch size for processing - **kwargs : dict - Additional parameters - - Returns - ------- - tuple - (formatted_batches, batch_data_list) - """ - if protocol == "http": - return self._format_http_input(data, max_batch_size, **kwargs) - elif protocol == "grpc": - return self._format_grpc_input(data, max_batch_size, **kwargs) - else: - raise ValueError("Invalid protocol. Must be 'grpc' or 'http'") - - def _format_http_input( - self, - data: Dict[str, Any], - max_batch_size: int, - **kwargs - ) -> Tuple[List[Dict[str, Any]], List[Dict[str, Any]]]: - """Format input for HTTP protocol.""" - input_text = data["input_text"] - - # Create HTTP payload - payload = { - "model": kwargs.get("model_name", self.model_name), - "input": input_text, - "max_tokens": kwargs.get("max_tokens", 512), - "temperature": kwargs.get("temperature", 0.7), - } - - # Return as single batch - return [payload], [{"original_input": input_text}] - - def _format_grpc_input( - self, - data: Dict[str, Any], - max_batch_size: int, - **kwargs - ) -> Tuple[List[np.ndarray], List[Dict[str, Any]]]: - """Format input for gRPC protocol.""" - input_text = data["input_text"] - - # Convert to numpy array for gRPC - text_array = np.array([[input_text.encode("utf-8")]], dtype=np.object_) - - return [text_array], [{"original_input": input_text}] - - def parse_output( - self, - response: Any, - protocol: str, - data: Optional[Dict[str, Any]] = None, - **kwargs - ) -> Any: - """ - Parse the raw model response. - - Parameters - ---------- - response : Any - Raw response from the model - protocol : str - Communication protocol used - data : dict, optional - Original batch data - **kwargs : dict - Additional parameters - - Returns - ------- - Any - Parsed response data - """ - if protocol == "http": - return self._parse_http_response(response) - elif protocol == "grpc": - return self._parse_grpc_response(response) - else: - raise ValueError("Invalid protocol. Must be 'grpc' or 'http'") - - def _parse_http_response(self, response: Dict[str, Any]) -> str: - """Parse HTTP response.""" - if isinstance(response, dict): - # Extract the generated text from response - if "choices" in response: - return response["choices"][0].get("text", "") - elif "output" in response: - return response["output"] - else: - raise RuntimeError("Unexpected response format") - return str(response) - - def _parse_grpc_response(self, response: np.ndarray) -> str: - """Parse gRPC response.""" - if isinstance(response, np.ndarray): - # Decode bytes response - return response.flatten()[0].decode("utf-8") - return str(response) - - def process_inference_results( - self, - output: Any, - protocol: str, - **kwargs - ) -> Any: - """ - Post-process the parsed inference results. - - Parameters - ---------- - output : Any - Parsed output from parse_output - protocol : str - Communication protocol used - **kwargs : dict - Additional parameters - - Returns - ------- - Any - Final processed results - """ - # Apply any final processing (e.g., filtering, formatting) - if isinstance(output, str): - return output.strip() - return output -``` - -## Real-World Examples - -### Text Generation Model Interface - -```python -class TextGenerationModelInterface(ModelInterface): - """Interface for text generation NIMs (e.g., LLaMA, GPT-style models).""" - - def name(self) -> str: - return "TextGeneration" - - def prepare_data_for_inference(self, data: Dict[str, Any]) -> Dict[str, Any]: - if "prompt" not in data: - raise KeyError("Input data must include 'prompt'") - return data - - def format_input(self, data: Dict[str, Any], protocol: str, max_batch_size: int, **kwargs): - prompt = data["prompt"] - - if protocol == "http": - payload = { - "model": kwargs.get("model_name", "llama-2-7b-chat"), - "messages": [{"role": "user", "content": prompt}], - "max_tokens": kwargs.get("max_tokens", 512), - "temperature": kwargs.get("temperature", 0.7), - "top_p": kwargs.get("top_p", 0.9), - "stream": False - } - return [payload], [{"prompt": prompt}] - else: - raise ValueError("Only HTTP protocol supported for this model") - - def parse_output(self, response: Any, protocol: str, data: Optional[Dict[str, Any]] = None, **kwargs): - if protocol == "http" and isinstance(response, dict): - choices = response.get("choices", []) - if choices: - return choices[0].get("message", {}).get("content", "") - return str(response) - - def process_inference_results(self, output: Any, protocol: str, **kwargs): - return output.strip() if isinstance(output, str) else output -``` - -### Image Analysis Model Interface - -```python -import base64 -from nemo_retriever.api.util.image_processing.transforms import numpy_to_base64 - -class ImageAnalysisModelInterface(ModelInterface): - """Interface for image analysis NIMs (e.g., vision models).""" - - def name(self) -> str: - return "ImageAnalysis" - - def prepare_data_for_inference(self, data: Dict[str, Any]) -> Dict[str, Any]: - if "images" not in data: - raise KeyError("Input data must include 'images'") - - # Ensure images is a list - if not isinstance(data["images"], list): - data["images"] = [data["images"]] - - return data - - def format_input(self, data: Dict[str, Any], protocol: str, max_batch_size: int, **kwargs): - images = data["images"] - prompt = data.get("prompt", "Describe this image.") - - # Convert images to base64 if needed - base64_images = [] - for img in images: - if isinstance(img, np.ndarray): - base64_images.append(numpy_to_base64(img)) - elif isinstance(img, str) and img.startswith("data:image"): - # Already base64 encoded - base64_images.append(img.split(",")[1]) - else: - base64_images.append(str(img)) - - # Batch images - batches = [base64_images[i:i + max_batch_size] - for i in range(0, len(base64_images), max_batch_size)] - - payloads = [] - batch_data_list = [] - - for batch in batches: - if protocol == "http": - messages = [] - for img_b64 in batch: - messages.append({ - "role": "user", - "content": f'{prompt} ' - }) - - payload = { - "model": kwargs.get("model_name", "llava-1.5-7b-hf"), - "messages": messages, - "max_tokens": kwargs.get("max_tokens", 512), - "temperature": kwargs.get("temperature", 0.1) - } - payloads.append(payload) - batch_data_list.append({"images": batch, "prompt": prompt}) - - return payloads, batch_data_list - - def parse_output(self, response: Any, protocol: str, data: Optional[Dict[str, Any]] = None, **kwargs): - if protocol == "http" and isinstance(response, dict): - choices = response.get("choices", []) - return [choice.get("message", {}).get("content", "") for choice in choices] - return [str(response)] - - def process_inference_results(self, output: Any, protocol: str, **kwargs): - if isinstance(output, list): - return [result.strip() for result in output] - return output -``` - -## Using NimClient in UDFs - -### Basic UDF with NimClient - -```python -from nemo_retriever.api.internal.primitives.ingest_control_message import IngestControlMessage -from nemo_retriever.api.util.nim import create_inference_client -import os - -def analyze_document_with_nim(control_message: IngestControlMessage) -> IngestControlMessage: - """UDF that uses a custom NIM to analyze document content.""" - - # Create NIM client - model_interface = TextGenerationModelInterface() - client = create_inference_client( - endpoints=( - os.getenv("ANALYSIS_NIM_GRPC", "grpc://analysis-nim:8001"), - os.getenv("ANALYSIS_NIM_HTTP", "http://analysis-nim:8000") - ), - model_interface=model_interface, - auth_token=os.getenv("NGC_API_KEY"), - infer_protocol="http" - ) - - # Get the document DataFrame - df = control_message.get_payload() - - # Process each document - for idx, row in df.iterrows(): - if row.get("content"): - # Prepare analysis prompt - prompt = f"Analyze the following document content and provide a summary: {row['content'][:1000]}" - - # Perform inference - try: - results = client.infer( - data={"prompt": prompt}, - model_name="llama-2-7b-chat", - max_tokens=256, - temperature=0.3 - ) - - # Add analysis to metadata - if results: - analysis = results[0] if isinstance(results, list) else results - df.at[idx, "custom_analysis"] = analysis - - except Exception as e: - print(f"NIM inference failed: {e}") - df.at[idx, "custom_analysis"] = "Analysis failed" - - # Update the control message with processed data - control_message.payload(df) - return control_message -``` - -### Advanced UDF with Batching - -```python -def batch_image_analysis_udf(control_message: IngestControlMessage) -> IngestControlMessage: - """UDF that performs batched image analysis using NIM.""" - - # Create image analysis client - model_interface = ImageAnalysisModelInterface() - client = create_inference_client( - endpoints=( - os.getenv("VISION_NIM_GRPC", "grpc://vision-nim:8001"), - os.getenv("VISION_NIM_HTTP", "http://vision-nim:8000") - ), - model_interface=model_interface, - auth_token=os.getenv("NGC_API_KEY") - ) - - df = control_message.get_payload() - - # Collect all images for batch processing - image_rows = [] - images = [] - - for idx, row in df.iterrows(): - if "image_data" in row and row["image_data"]: - image_rows.append(idx) - images.append(row["image_data"]) - - if images: - try: - # Batch process all images - results = client.infer( - data={ - "images": images, - "prompt": "Describe the content and key elements in this image." - }, - model_name="llava-1.5-7b-hf", - max_tokens=200 - ) - - # Apply results back to DataFrame - for idx, result in zip(image_rows, results): - df.at[idx, "image_description"] = result - - except Exception as e: - print(f"Batch image analysis failed: {e}") - for idx in image_rows: - df.at[idx, "image_description"] = "Analysis failed" - - control_message.payload(df) - return control_message -``` - -## Configuration and Best Practices - -### Environment Variables - -Set these environment variables for your NIM endpoints: - -```bash -# NIM endpoints -export MY_NIM_GRPC_ENDPOINT="grpc://my-nim-service:8001" -export MY_NIM_HTTP_ENDPOINT="http://my-nim-service:8000" - -# Authentication -export NGC_API_KEY="your-ngc-api-key" - -# Optional: timeouts and retries -export NIM_TIMEOUT=120 -export NIM_MAX_RETRIES=5 -``` - -### Performance Optimization - -1. **Use gRPC when possible**: Generally faster than HTTP for high-throughput scenarios -2. **Batch processing**: Process multiple items together to reduce overhead -3. **Connection reuse**: Create NimClient instances once and reuse them -4. **Appropriate timeouts**: Set reasonable timeouts based on your model's response time -5. **Error handling**: Always handle inference failures gracefully - -### Error Handling - -```python -def robust_nim_udf(control_message: IngestControlMessage) -> IngestControlMessage: - """UDF with comprehensive error handling.""" - - try: - client = create_inference_client( - endpoints=(grpc_endpoint, http_endpoint), - model_interface=model_interface, - auth_token=auth_token, - timeout=60.0, - max_retries=3 - ) - except Exception as e: - print(f"Failed to create NIM client: {e}") - return control_message - - df = control_message.get_payload() - - for idx, row in df.iterrows(): - try: - results = client.infer(data=input_data, model_name="my-model") - df.at[idx, "nim_result"] = results - except TimeoutError: - print(f"NIM request timed out for row {idx}") - df.at[idx, "nim_result"] = "timeout" - except Exception as e: - print(f"NIM inference failed for row {idx}: {e}") - df.at[idx, "nim_result"] = "error" - - control_message.payload(df) - return control_message -``` - -## Troubleshooting - -### Common Issues - -* **Connection Errors** – Verify NIM service is running and endpoints are correct -* **Authentication Failures** – Check NGC_API_KEY is valid and properly set -* **Timeout Errors** – Increase timeout values or check NIM service performance -* **Format Errors** – Ensure your ModelInterface formats data correctly for your NIM -* **Memory Issues** – Use appropriate batch sizes to avoid memory exhaustion - -### NIM Triton Limit Memory - -If you encounter memory issues, try increasing the `NIM_TRITON_CUDA_MEMORY_POOL_MB` parameter. This adjustment typically does not affect performance. - -If memory issues persist, you can reduce the `NIM_TRITON_RATE_LIMIT` value — even down to 1. However, lowering this parameter affects performance. - -### Debugging Tips - -```python -import logging - -# Enable debug logging -logging.getLogger("nemo_retriever.api.internal.primitives.nim").setLevel(logging.DEBUG) - -# Test your model interface separately -model_interface = MyCustomModelInterface() -test_data = {"input": "test"} - -# Test data preparation -prepared = model_interface.prepare_data_for_inference(test_data) -print(f"Prepared data: {prepared}") - -# Test input formatting -formatted, batch_data = model_interface.format_input(prepared, "http", 1) -print(f"Formatted input: {formatted}") -``` - -## Related Topics - -- [Extending/Customizing NeMo Retriever Library with custom code](https://github.com/NVIDIA/NeMo-Retriever/tree/main/nemo_retriever/src/nemo_retriever/graph#nemo-retriever-graph) diff --git a/docs/docs/extraction/prerequisites-support-matrix.md b/docs/docs/extraction/prerequisites-support-matrix.md index b8910f2ea8..fb4315daab 100644 --- a/docs/docs/extraction/prerequisites-support-matrix.md +++ b/docs/docs/extraction/prerequisites-support-matrix.md @@ -2,7 +2,7 @@ Before you begin using [NeMo Retriever Library](overview.md), confirm your software stack, deployment hardware, and—if you use them—advanced features (audio and video, Nemotron Parse, VLM image captioning, reranking) against the guidance in this page. -## Software Requirements +## Software Requirements { #software-requirements } - Linux operating systems (Ubuntu 22.04 or later recommended) - [CUDA Toolkit](https://developer.nvidia.com/cuda-downloads) (NVIDIA Driver >= `580`, CUDA >= `13.0`) @@ -11,8 +11,7 @@ Before you begin using [NeMo Retriever Library](overview.md), confirm your softw - For audio and video, `ffmpeg` and `ffprobe` must be on `PATH` (for example `sudo apt-get install -y --no-install-recommends ffmpeg` on Debian/Ubuntu). `ffmpeg-python` and `nemo-retriever[multimedia]` do not install these binaries. - On Helm with package-repo access, set `service.installFfmpeg=true`. For - air-gapped clusters, see [Air-gapped and disconnected deployment](deployment-options.md#air-gapped-deployment). + For container and Kubernetes guidance, refer to [Audio and video](audio-video.md). - For PDF extraction with `extract_method="nemotron_parse"`, install the Nemotron Parse client dependencies with `pip install "nemo-retriever[nemotron-parse]"` (pulls `open-clip-torch`, which provides the `open_clip` module required by the Nemotron Parse @@ -23,7 +22,7 @@ Before you begin using [NeMo Retriever Library](overview.md), confirm your softw When you use UV, create the environment with Python 3.12 — for example, `uv venv --python 3.12`. This matches the `requires-python` metadata in the library packages. -## Hardware Requirements +## Hardware Requirements { #hardware-requirements } The full ingestion pipeline is designed to consume significant CPU and memory resources to achieve maximal parallelism. Resource usage scales up to the limits of your deployed system. @@ -60,32 +59,32 @@ For production deployments processing large volumes of documents, consider: Ensure your deployment environment meets these specifications before running the full pipeline. Resource-constrained environments may experience performance degradation. -## Core and Advanced Pipeline Features +## Core and Advanced Pipeline Features { #core-and-advanced-pipeline-features } The NeMo Retriever Library extraction core pipeline features run on a single A10G or better GPU. -### Default Helm NIMs +### Default Helm NIMs { #default-helm-nims } -The production Helm chart enables these NIM microservices **by default** (for example via `nimOperator.*.enabled=true`): +The production Helm chart enables these NIM microservices **by default** (for example through `nimOperator.*.enabled=true`): | Helm flag | NIM | Role | |-----------|-----|------| | `page_elements` | [nemotron-page-elements-v3](https://huggingface.co/nvidia/nemotron-page-elements-v3) | Page layout and element detection | | `table_structure` | [nemotron-table-structure-v1](https://huggingface.co/nvidia/nemotron-table-structure-v1) | Table structure extraction | -| `ocr` | [nemotron-ocr-v1](https://huggingface.co/nvidia/nemotron-ocr-v1) | Image OCR | +| `ocr` | [nemotron-ocr-v2](https://huggingface.co/nvidia/nemotron-ocr-v2) | Image OCR | | `vlm_embed` | [llama-nemotron-embed-vl-1b-v2](https://huggingface.co/nvidia/llama-nemotron-embed-vl-1b-v2) | Multimodal (VL) embedding | ### OCR artifacts (Helm vs local Hugging Face) { #nemotron-ocr-v2-language-mode } !!! note - **Helm / NIM:** The production chart deploys **Nemotron OCR v1** under `nimOperator.ocr` (`nvcr.io/nim/nvidia/nemotron-ocr-v1:1.3.0`). For image defaults and upgrade notes, see [OCR NIM configuration](https://github.com/NVIDIA/NeMo-Retriever/blob/26.05/nemo_retriever/helm/README.md#ocr-nim-configuration) in the Helm chart README. + **Helm / NIM:** The production chart deploys **Nemotron OCR v2** under `nimOperator.ocr` (`nvcr.io/nim/nvidia/nemotron-ocr-v2:1.4.0`). For image defaults and upgrade notes, refer to [OCR NIM configuration](https://github.com/NVIDIA/NeMo-Retriever/blob/main/nemo_retriever/helm/README.md#ocr-nim-configuration) in the Helm chart README. - **Local Hugging Face inference:** When you deploy locally with HuggingFace model weights (for example `pip install "nemo-retriever[local]"` and GPU inference without remote OCR NIM URLs), the default OCR engine is **Nemotron OCR v2**, which runs in **multilingual** mode by default. For CLI flags and API parameters, see [Nemotron OCR v2 — language mode](https://github.com/NVIDIA/NeMo-Retriever/blob/main/nemo_retriever/docs/cli/README.md#nemotron-ocr-v2-language-mode). Remote OCR NIM endpoints use their own model and language behavior; local OCR language selectors are not sent on remote requests. + **Local Hugging Face inference:** When you deploy locally with HuggingFace model weights (for example `pip install "nemo-retriever[local]"` and GPU inference without remote OCR NIM URLs), the default OCR engine is **Nemotron OCR v2**, which runs in **multilingual** mode by default. For CLI flags and API parameters, refer to [Nemotron OCR v2 — language mode](https://github.com/NVIDIA/NeMo-Retriever/blob/main/nemo_retriever/docs/cli/README.md#nemotron-ocr-v2-language-mode). Remote OCR NIM endpoints use their own model and language behavior; local OCR language selectors are not sent on remote requests. Default OCR NIM container for release Helm deployments: -- **Image:** `nvcr.io/nim/nvidia/nemotron-ocr-v1:1.3.0` +- **Image:** `nvcr.io/nim/nvidia/nemotron-ocr-v2:1.4.0` Default VL embedder container and model for release deployments: @@ -94,54 +93,43 @@ Default VL embedder container and model for release deployments: ### Optional Helm NIMs (not auto-wired) { #optional-helm-nims-not-auto-wired-by-default } -These NIM microservices are **optional** for the default extraction pipeline. The retriever service does **not** call them until you enable the matching pipeline stage (reranker, Nemotron Parse, caption, or audio). For **26.05 production**, disable keys you do not need (see [Recommended minimal install (26.05)](https://github.com/NVIDIA/NeMo-Retriever/blob/26.05/nemo_retriever/helm/README.md#recommended-minimal-install-2605)). Set `nimOperator..enabled=true` when you want that NIM reconciled. Chart keys are in the [NeMo Retriever Helm chart README](https://github.com/NVIDIA/NeMo-Retriever/blob/26.05/nemo_retriever/helm/README.md#nim-operator-sub-stack). +These NIM microservices are **optional** for the default extraction pipeline. The retriever service does **not** call them until you enable the matching pipeline stage (reranker, Nemotron Parse, caption, or audio). In production, disable keys you do not need (refer to [Recommended minimal install](https://github.com/NVIDIA/NeMo-Retriever/blob/main/nemo_retriever/helm/README.md#recommended-minimal-install-2605)). Set `nimOperator..enabled=true` when you want that NIM reconciled. Chart keys are in the [NeMo Retriever Helm chart README](https://github.com/NVIDIA/NeMo-Retriever/blob/main/nemo_retriever/helm/README.md#nim-operator-sub-stack). | Helm flag | NIM | Role | |-----------|-----|------| | `rerankqa` | [llama-nemotron-rerank-vl-1b-v2](https://huggingface.co/nvidia/llama-nemotron-rerank-vl-1b-v2) | Reranking for improved retrieval accuracy | | `nemotron_parse` | [nemotron-parse](https://huggingface.co/nvidia/NVIDIA-Nemotron-Parse-v1.2) | Optional PDF `extract_method="nemotron_parse"` (default PDF extraction uses **pdfium**) | -| `nemotron_3_nano_omni_30b_a3b_reasoning` | [nemotron-3-nano-omni-30b-a3b-reasoning](https://huggingface.co/nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16) | Supported image captioning for 26.05 when you enable the caption stage | +| `nemotron_3_nano_omni_30b_a3b_reasoning` | [nemotron-3-nano-omni-30b-a3b-reasoning](https://huggingface.co/nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16) | Supported image captioning when you enable the caption stage | | `audio` | [parakeet-1-1b-ctc-en-us](https://huggingface.co/nvidia/parakeet-ctc-1.1b) | [Audio and video](audio-video.md) transcription | -### Image captioning (26.05) { #image-captioning-2605 } +### Image captioning { #image-captioning } -For 26.05, use **`nemotron_3_nano_omni_30b_a3b_reasoning`** when you enable the caption stage (hosted model ID `nvidia/nemotron-3-nano-omni-30b-a3b-reasoning`). The Helm key is in the [optional NIMs](#optional-helm-nims-not-auto-wired-by-default) table above. - -!!! important "PDF chart regions are not captioned by Omni" - - When **nemotron-page-elements-v3** classifies a PDF region as **chart**, that region is processed through layout detection and OCR—not the Omni caption stage. Enabling the caption NIM and the `caption` pipeline stage does **not** send chart-labeled figures to `/v1/chat/completions`. - - The caption stage covers: - - - Unstructured content in the `images` column (standalone image files and page-element regions **not** classified as table, chart, or infographic) - - Optional infographic regions when you set `caption_infographics=True` on `CaptionParams` (the VLM caption is stored in `caption`, separate from OCR `text`) - - To validate caption traffic during ingest, inspect metadata such as `page_elements_v3_counts_by_label`. If the figure is labeled `chart`, expect no Omni chat-completions requests for that region even when captioning is enabled. +Use **`nemotron_3_nano_omni_30b_a3b_reasoning`** when you enable the caption stage (hosted model ID `nvidia/nemotron-3-nano-omni-30b-a3b-reasoning`). The Helm key is in the [optional NIMs](#optional-helm-nims-not-auto-wired-by-default) table above. Optional features listed in the table above require additional GPU support, disk space, and feature-specific system dependencies beyond the four default NIMs. For published NIM model IDs and deployment-specific constraints, use the product support matrices linked under [Related Topics](#related-topics) below. -## Model Hardware Requirements +## Model Hardware Requirements { #model-hardware-requirements } NeMo Retriever Library supports the following GPU hardware given system constraints in the table. - **HF model weights** — approximate Hugging Face checkpoint footprint (files such as `model*.safetensors`, `weights.pth`, or other published weight bundles in the model repository). Values are rounded from the current public file listing and can change when the repository is updated. -- **NIM disk space** — approximate container and on-disk model cache for self-hosted NIM microservices (not the same as HF download size). For Nemotron 3 Nano Omni captioning, see the [NVIDIA NIM for Vision Language Models support matrix](https://docs.nvidia.com/nim/vision-language-models/latest/support-matrix.html#nemotron-3-nano-omni-30b-a3b-reasoning). +- **NIM disk space** — approximate container and on-disk model cache for self-hosted NIM microservices (not the same as HF download size). For Nemotron 3 Nano Omni captioning, refer to the [NVIDIA NIM for Vision Language Models support matrix](https://docs.nvidia.com/nim/vision-language-models/latest/support-matrix.html#nemotron-3-nano-omni-30b-a3b-reasoning). Model repositories and NIM references are linked in [Core and Advanced Pipeline Features](#core-and-advanced-pipeline-features) above. -**B200 and audio/video extraction (26.05):** The [audio and video](audio-video.md) transcription path (self-hosted Parakeet ASR via `nimOperator.audio`) is **not supported on B200** or other Blackwell GPUs. Core PDF and multimodal extraction on B200 is unchanged. See footnote ⁴ below. +**B200, H200 NVL, and audio/video extraction:** The [audio and video](audio-video.md) transcription path (self-hosted Parakeet ASR through `nimOperator.audio`) is **not supported on B200**, other Blackwell GPUs, or **H200 NVL**. Core PDF and multimodal extraction on those GPUs is unchanged. Refer to footnote ⁴ below. | Feature | HF Model Weights | GPU Option | [RTX Pro 6000](https://www.nvidia.com/en-us/data-center/rtx-pro-6000-blackwell-server-edition/) | [B200](https://www.nvidia.com/en-us/data-center/dgx-b200/) | [H200 NVL](https://www.nvidia.com/en-us/data-center/h200/) | [H100](https://www.nvidia.com/en-us/data-center/h100/) | [A100 80GB](https://www.nvidia.com/en-us/data-center/a100/) | A100 40GB | [A10G](https://aws.amazon.com/ec2/instance-types/g5/) | L40S | [RTX PRO 4500 Blackwell](https://www.nvidia.com/en-us/products/workstations/professional-desktop-gpus/rtx-pro-4500/) | |---------|------------------|------------|--------|--------|--------|--------|--------|--------|--------|--------|------------------------| | GPU | — | Memory | 96GB | 180GB | 141GB | 80GB | 80GB | 40GB | 24GB | 48GB | 32GB GDDR7 (GB203) | | Core Features | ~4.8 GiB combined: embed VL 1b ~3.1 GiB; page-elements ~0.41 GiB; table-structure ~0.81 GiB; OCR ~0.51 GiB | Total GPUs | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | | Core Features | — | Total Disk Space | ~150GB | ~150GB | ~150GB | ~150GB | ~150GB | ~150GB | ~150GB | ~150GB | ~150GB | -| Audio/video extraction (parakeet-1-1b-ctc-en-us) | ~4.0 GiB (`model.safetensors`; the repo also ships `parakeet-ctc-1.1b.nemo` of similar size—use one format to avoid roughly doubling disk use) | Additional Dedicated GPUs | Not supported⁴ | Not supported⁴ | 1¹ | 1¹ | 1¹ | 1¹ | 1¹ | 1¹ | Not supported⁴ | -| | — | Additional Disk Space | Not supported⁴ | Not supported⁴ | ~37GB¹ | ~37GB¹ | ~37GB¹ | ~37GB¹ | ~37GB¹ | ~37GB¹ | Not supported⁴ | -| nemotron-parse | ~3.5 GiB | Additional Dedicated GPUs | Not supported | 1 | Not supported | 1 | 1 | 1 | 1 | 1 | Not supported² | -| nemotron-parse | — | Additional Disk Space | Not supported | ~16GB | Not supported | ~16GB | ~16GB | ~16GB | ~16GB | ~16GB | Not supported² | +| Audio/video extraction (parakeet-1-1b-ctc-en-us) | ~4.0 GiB (`model.safetensors`; the repo also ships `parakeet-ctc-1.1b.nemo` of similar size—use one format to avoid roughly doubling disk use) | Additional Dedicated GPUs | Not supported⁴ | Not supported⁴ | Not supported⁴ | 1¹ | 1¹ | 1¹ | 1¹ | 1¹ | Not supported⁴ | +| | — | Additional Disk Space | Not supported⁴ | Not supported⁴ | Not supported⁴ | ~37GB¹ | ~37GB¹ | ~37GB¹ | ~37GB¹ | ~37GB¹ | Not supported⁴ | +| nemotron-parse | ~3.5 GiB | Additional Dedicated GPUs | Not supported | 1 | Not supported | 1 | 1 | 1 | 1 | 1 | 1 | +| nemotron-parse | — | Additional Disk Space | Not supported | ~16GB | Not supported | ~16GB | ~16GB | ~16GB | ~16GB | ~16GB | ~16GB | | Omni caption (nemotron-3-nano-omni-30b-a3b-reasoning) | ~62 GiB (BF16); ~33 GiB (FP8); ~21 GiB (NVFP4) | Additional Dedicated GPUs | 1 | 1 | 1 | 1 | 1 | Not supported | Not supported | 2 | Not supported³ | | Omni caption (nemotron-3-nano-omni-30b-a3b-reasoning) | — | Additional Disk Space (HF) | ~21–62GB | ~21–62GB | ~21–62GB | ~21–62GB | ~21–62GB | Not supported | Not supported | ~21–62GB | Not supported³ | | Omni caption (nemotron-3-nano-omni-30b-a3b-reasoning) | — | Additional Disk Space (NIM) | ~80GB | ~80GB | ~80GB | ~80GB | ~80GB | Not supported | Not supported | ~80GB | Not supported³ | @@ -150,17 +138,15 @@ Model repositories and NIM references are linked in [Core and Advanced Pipeline ¹ On other supported GPUs, Parakeet ASR (`parakeet-1-1b-ctc-en-us:1.5.0`) may require a runtime TensorRT engine build (no prebuilt profile in the chart image). -⁴ On **B200** and other **Blackwell** GPUs (compute capability 12.0), including RTX PRO 6000 Blackwell and RTX PRO 4500 Blackwell, self-hosted [audio/video extraction](audio-video.md) via Parakeet ASR (`parakeet-1-1b-ctc-en-us:1.5.0`, `nimOperator.audio`) is **not supported**. Core PDF and multimodal extraction on Blackwell is unchanged. Video workflows that depend on Parakeet for speech transcription are affected the same way. `NIMService` for `nimOperator.audio` may stay not Ready or enter `CrashLoopBackOff` while building the Riva/TensorRT engine (for example ONNX Runtime IR version, cuDNN visibility, or FP8 tactic errors). Use a non-Blackwell dedicated GPU, [hosted Parakeet on build.nvidia.com](audio-video.md#parakeet-hosted-inference-build-nvidia), or set `nimOperator.audio.enabled=false`. - -² Nemotron Parse fails to start on 32GB. +⁴ Self-hosted [audio/video extraction](audio-video.md) through Parakeet ASR (`parakeet-1-1b-ctc-en-us:1.5.0`, `nimOperator.audio`) is **not supported** on **B200**, other **Blackwell** GPUs (compute capability 12.0), including RTX PRO 6000 Blackwell and RTX PRO 4500 Blackwell, or **H200 NVL**. Core PDF and multimodal extraction on those GPUs is unchanged. Video workflows that depend on Parakeet for speech transcription are affected the same way. `NIMService` for `nimOperator.audio` may stay not Ready or enter `CrashLoopBackOff` while building the Riva/TensorRT engine (for example ONNX Runtime IR version, cuDNN visibility, or FP8 tactic errors). Use a supported dedicated GPU (for example H100 or A100), [hosted Parakeet on build.nvidia.com](audio-video.md#parakeet-hosted-inference-build-nvidia), or set `nimOperator.audio.enabled=false`. -³ Opt-in Omni captioning uses the [nemotron-3-nano-omni-30b-a3b-reasoning](https://docs.api.nvidia.com/nim/reference/nvidia-nemotron-3-nano-omni-30b-a3b-reasoning) NIM (`nvcr.io/nim/nvidia/nemotron-3-nano-omni-30b-a3b-reasoning:1.7.0-variant`). BF16 requires at least 80 GB total GPU memory; see the [VLM NIM support matrix](https://docs.nvidia.com/nim/vision-language-models/latest/support-matrix.html#nemotron-3-nano-omni-30b-a3b-reasoning). L40S requires two GPUs. A100 40GB, A10G, and RTX PRO 4500 are below the minimum. +³ Opt-in Omni captioning uses the [nemotron-3-nano-omni-30b-a3b-reasoning](https://docs.api.nvidia.com/nim/reference/nvidia-nemotron-3-nano-omni-30b-a3b-reasoning) NIM (`nvcr.io/nim/nvidia/nemotron-3-nano-omni-30b-a3b-reasoning:1.7.0-variant`). BF16 requires at least 80 GB total GPU memory; refer to the [VLM NIM support matrix](https://docs.nvidia.com/nim/vision-language-models/latest/support-matrix.html#nemotron-3-nano-omni-30b-a3b-reasoning). L40S requires two GPUs. A100 40GB, A10G, and RTX PRO 4500 are below the minimum. \* GPUs with less than 80GB VRAM cannot run the reranker concurrently with the core pipeline. To perform recall testing with the reranker on these GPUs, shut down the core pipeline NIM microservices and run only the embedder, reranker, and your vector database. -## Related Topics +## Related Topics { #related-topics } - [Troubleshooting](troubleshoot.md) - [Release Notes](releasenotes.md) diff --git a/docs/docs/extraction/starter-kits.md b/docs/docs/extraction/starter-kits.md new file mode 100644 index 0000000000..08b366a6eb --- /dev/null +++ b/docs/docs/extraction/starter-kits.md @@ -0,0 +1,24 @@ +# Starter Kits for NeMo Retriever Library + +To get started using [NeMo Retriever Library](overview.md), you can try one of the ready-made notebooks that are available. + +## Dataset Downloads for Benchmarking + +If you plan to run benchmarking or evaluation tests, you must download the [Benchmark Datasets (Bo20, Bo767, Bo10k)](https://github.com/NVIDIA/NeMo-Retriever/blob/main/evaluation/digital_corpora_download.ipynb) from Digital Corpora. This is a prerequisite for all benchmarking operations. + +## Getting Started + +To get started with the basics, try one of the following guides or notebooks: + +- [Quickstart: retriever CLI](../reference/retriever-cli-quickstart.md) +- [Workflow: Ingest documents](workflow-document-ingestion.md) +- [Adding Custom Metadata for Filtered Search/Retrieval](https://github.com/NVIDIA/NeMo-Retriever/blob/main/examples/nemo_retriever_retriever_query_metadata_filter.ipynb) — also summarized on [Vector databases — Metadata and filtering](vdbs.md#metadata-and-filtering) + + +For more advanced scenarios, try one of the following notebooks: + +- [Build a Custom Vector Database Operator](https://github.com/NVIDIA/NeMo-Retriever/blob/main/examples/building_vdb_operator.ipynb) +- [Try Enterprise RAG Blueprint](https://build.nvidia.com/nvidia/multimodal-pdf-data-extraction-for-enterprise-rag) +- [Evaluate bo767 retrieval recall accuracy with NeMo Retriever Library](https://github.com/NVIDIA/NeMo-Retriever/blob/main/evaluation/bo767_recall.ipynb) +- [Multimodal RAG with LangChain](https://github.com/NVIDIA/NeMo-Retriever/blob/main/examples/langchain_multimodal_rag.ipynb) +- [Multimodal RAG with LlamaIndex](https://github.com/NVIDIA/NeMo-Retriever/blob/main/examples/llama_index_multimodal_rag.ipynb) diff --git a/docs/docs/extraction/workflow-agentic-retrieval.md b/docs/docs/extraction/workflow-agentic-retrieval.md index 78b48cdc47..d29b36331b 100644 --- a/docs/docs/extraction/workflow-agentic-retrieval.md +++ b/docs/docs/extraction/workflow-agentic-retrieval.md @@ -4,11 +4,30 @@ NeMo Retriever Library provides ingestion, embedding, storage, and retrieval building blocks (jobs, chunking, vector stores, reranking) that you orchestrate in application code or frameworks. +## MCP access for agents + +`retriever service start` mounts a FastMCP HTTP endpoint at `/mcp` by default. Agents can use that endpoint to call the running service for health checks, pipeline introspection, document ingestion, job status, VectorDB query, and answer generation. If service auth is enabled, the MCP endpoint uses the same bearer-token middleware as the REST API. + +For local stdio-based agents, run the MCP server as a shim that points at an existing retriever service: + +```bash +retriever service mcp-stdio \ + --service-url http://localhost:7670 \ + --api-token "$NEMO_RETRIEVER_API_TOKEN" +``` + +For remote agents, expose the retriever service URL and configure the agent to connect to: + +```text +https:///mcp +``` + +The `ingest_documents` MCP tool accepts either paths visible to the MCP server process or inline `content_base64` document bytes. Use inline base64 for remote agents whose local files are not present on the service host. + **Where to go next** Use these pages together with your orchestration layer: -- [Semantic retrieval](vdbs.md#semantic-retrieval), [Custom metadata and filtering](custom-metadata.md), and [Evaluate on your data](evaluate-on-your-data.md) for retrieval quality and reranking notes +- [Semantic retrieval](vdbs.md#semantic-retrieval), [Metadata and filtering](vdbs.md#metadata-and-filtering), and [Evaluate on your data](evaluate-on-your-data.md) for retrieval quality, reranking, and evaluation guidance - [Agentic retrieval (concept)](agentic-retrieval-concept.md) -- [Evaluate on your data](evaluate-on-your-data.md), which includes retrieval evaluation guidance - [Release notes](releasenotes.md), which may mention agentic retrieval updates diff --git a/docs/docs/extraction/workflow-document-ingestion.md b/docs/docs/extraction/workflow-document-ingestion.md index 853de9e1a0..cba0164d63 100644 --- a/docs/docs/extraction/workflow-document-ingestion.md +++ b/docs/docs/extraction/workflow-document-ingestion.md @@ -2,7 +2,7 @@ This page covers extracting content from documents and turning that content into a searchable vector collection in one place so you can scroll and search (for example with Ctrl+F) instead of jumping across multiple short workflow stubs. -## Ingest and extract +## Ingest and extract { #ingest-and-extract } Document ingestion is the step where NeMo Retriever Library reads your files (PDFs, Office documents, images, and other [supported formats](multimodal-extraction.md#supported-file-types-and-formats)), runs extraction and optional enrichment, and returns structured content you can embed and index. @@ -14,9 +14,9 @@ Follow these steps: Pipeline concepts and stage overview appear in [Key concepts](concepts.md). Default chunking behavior is summarized under [Chunking](concepts.md#chunking). -`create_ingestor(...)` returns a `GraphIngestor`, which chains `.extract()`, `.embed()`, and `.vdb_upload()` into one graph. The Python example below stops after `.embed()` so you can inspect chunks first; append `.vdb_upload(vdb_op="lancedb", vdb_kwargs={...})` before `.ingest()` to write directly to LanceDB (refer to [Vector databases](vdbs.md)). For directory-scale corpus ingest, the `graph_pipeline` CLI below is the canonical path used in evaluation workflows. +`create_ingestor(...)` returns a `GraphIngestor`, which chains `.extract()`, `.embed()`, and `.vdb_upload()` into one graph. The Python example below stops after `.embed()` so you can inspect chunks first; append `.vdb_upload(vdb_op="lancedb", vdb_kwargs={...})` before `.ingest()` to write directly to LanceDB (refer to [Vector databases](vdbs.md)). -## Choose how you call the library +## Choose how you call the library { #choose-how-you-call-the-library } The following examples match the [NeMo Retriever Library README](https://github.com/NVIDIA/NeMo-Retriever/blob/main/nemo_retriever/README.md). They assume a checkout of the [NeMo Retriever](https://github.com/NVIDIA/NeMo-Retriever) repository and the `batch` run mode with local GPU inference unless you configure remote NIMs. @@ -47,20 +47,4 @@ result = ingestor.ingest() # ``pandas.DataFrame`` (``batch`` and ``inprocess``) Run the above with your working directory at the repository root (so `data/multimodal_test.pdf` resolves), or adjust `documents` to the absolute path of the test PDF. -### Ingest a test corpus (CLI) - -`graph_pipeline` is the canonical ingestion script used throughout the [QA evaluation guide](https://github.com/NVIDIA/NeMo-Retriever/blob/main/nemo_retriever/src/nemo_retriever/evaluation/README.md#step-1-ingest-and-embed-pdfs-nemo-retriever). Point it at a **directory** of PDFs to produce a ready-to-query LanceDB table. - -!!! note "Corpus size and LanceDB indexing" - - LanceDB's default IVF index needs enough chunks to train its partitions (often on the order of tens of chunks). A single small PDF can be insufficient; use a directory with enough documents for your index settings. Replace `/your-example-dir` with your corpus path. - -```bash -python -m nemo_retriever.examples.graph_pipeline \ - /your-example-dir \ - --vdb-kwargs-json '{"uri":"lancedb","table_name":"nemo-retriever"}' -``` - -For build.nvidia.com hosted inference, set [`NVIDIA_API_KEY`](api-keys.md#nvidia-api-key) and pass the `--*-invoke-url` / `--embed-invoke-url` options shown in the [README remote inference section](https://github.com/NVIDIA/NeMo-Retriever/blob/main/nemo_retriever/README.md#ingest-a-test-corpus-cli). - **Next:** [Semantic retrieval](vdbs.md#semantic-retrieval) when serving queries (also refer to [Evaluate on your data](evaluate-on-your-data.md) for reranking and quality checks). diff --git a/docs/docs/extraction/workflow-e2e-blueprints.md b/docs/docs/extraction/workflow-e2e-blueprints.md index 16aa4bb3d6..93203a6374 100644 --- a/docs/docs/extraction/workflow-e2e-blueprints.md +++ b/docs/docs/extraction/workflow-e2e-blueprints.md @@ -5,4 +5,4 @@ Use these external resources for end-to-end RAG implementations with NeMo Retrie - [Enterprise RAG - multimodal PDF data extraction](https://build.nvidia.com/nvidia/multimodal-pdf-data-extraction-for-enterprise-rag) - [NVIDIA AI Blueprints catalog](https://build.nvidia.com/explore/discover) -For framework-specific integration patterns, see [Framework integrations](integrations-langchain-llamaindex-haystack.md). +For framework-specific integration patterns, refer to [Starter kits](starter-kits.md). diff --git a/docs/mkdocs.yml b/docs/mkdocs.yml index a944ad6c8b..57a3ea6d78 100644 --- a/docs/mkdocs.yml +++ b/docs/mkdocs.yml @@ -84,8 +84,9 @@ nav: - "Authentication and API keys": extraction/api-keys.md - "3. Deployment options": - "Compare deployment options": extraction/deployment-options.md + - "OpenShift deployment (Helm)": https://github.com/NVIDIA/NeMo-Retriever/blob/main/nemo_retriever/helm/openshift.md - "Helm chart (Kubernetes)": https://github.com/NVIDIA/NeMo-Retriever/tree/main/nemo_retriever/helm - - "Docker Compose (unsupported, developer)": https://github.com/NVIDIA/NeMo-Retriever/blob/main/nemo_retriever/docker.md + - "Docker service image": https://github.com/NVIDIA/NeMo-Retriever/blob/main/nemo_retriever/docker.md - "4. Core workflows": - "Workflow: Ingest documents into a searchable VDB collection": extraction/workflow-document-ingestion.md - "Workflow: Audio & video ingestion": extraction/audio-video.md @@ -95,26 +96,22 @@ nav: # Single vector-DB page (vdbs.md). Deep links: in-page "On this page" TOC and redirects # (for example extraction/vector-db-partners.md → vdbs.md#vector-database-partners). - "Vector databases": extraction/vdbs.md - - "7. Retrieval & ranking": - - "Custom metadata and filtering": extraction/custom-metadata.md - - "8. Deployment & operations": + - "7. Deployment & operations": - "Ray and distributed ingest": extraction/ray-logging.md - - "9. Customize & extend": - - Extending/Customizing NeMo Retriever Library with custom code: https://github.com/NVIDIA/NeMo-Retriever/tree/main/nemo_retriever/src/nemo_retriever/graph#nemo-retriever-graph - - "NimClient and custom NIM endpoints": extraction/nimclient.md - - "10. Integrations & ecosystem": - - "Framework integrations": extraction/integrations-langchain-llamaindex-haystack.md - - "Starter kits": extraction/notebooks/index.md - - "11. Evaluation & benchmarks": + - "8. Customize & extend": + - "Customize & extend": extraction/customize-extend.md + - "9. Integrations & ecosystem": + - "Starter kits": extraction/starter-kits.md + - "10. Evaluation & benchmarks": - "Evaluate on your own documents": extraction/evaluate-on-your-data.md - - "12. Reference": + - "11. Reference": - "API guide": extraction/nemo-retriever-api-reference.md # TODO: after nv-ingest code removal, update this link when CLI docs are relocated. - "CLI reference": https://github.com/NVIDIA/NeMo-Retriever/tree/main/nemo_retriever/docs/cli - "Quickstart: retriever CLI": reference/retriever-cli-quickstart.md - Environment variables: extraction/environment-config.md - "Metadata reference": extraction/content-metadata.md - - "13. Support & community": + - "12. Support & community": - Troubleshooting: extraction/troubleshoot.md - FAQ: extraction/faq.md - Contributing: extraction/contributing.md @@ -159,8 +156,11 @@ plugins: extraction/hosted-nims-when-to-use.md: extraction/deployment-options.md extraction/releasenotes-nv-ingest.md: extraction/releasenotes.md extraction/ngc-api-key.md: extraction/api-keys.md - extraction/notebooks.md: extraction/notebooks/index.md + extraction/notebooks/index.md: extraction/starter-kits.md + extraction/notebooks.md: extraction/starter-kits.md extraction/data-store.md: extraction/vdbs.md + extraction/custom-metadata.md: extraction/vdbs.md#metadata-and-filtering + extraction/integrations-langchain-llamaindex-haystack.md: extraction/starter-kits.md extraction/nemoretriever-parse.md: extraction/multimodal-extraction.md#text-and-layout-extraction extraction/supported-file-types.md: extraction/multimodal-extraction.md#supported-file-types-and-formats extraction/text-layout-extraction.md: extraction/multimodal-extraction.md#text-and-layout-extraction @@ -185,7 +185,11 @@ plugins: extraction/chunking.md: extraction/concepts.md#chunking extraction/quickstart-library-mode.md: extraction/deployment-options.md extraction/workflow-video-ocr.md: extraction/audio-video.md - extraction/user-defined-stages.md: https://github.com/NVIDIA/NeMo-Retriever/tree/main/nemo_retriever/src/nemo_retriever/graph#nemo-retriever-graph + extraction/user-defined-stages.md: extraction/customize-extend.md + extraction/user-defined-functions/index.md: extraction/customize-extend.md#user-defined-functions-udfs + extraction/user-defined-functions.md: extraction/customize-extend.md#user-defined-functions-udfs + extraction/customize-and-extend.md: extraction/customize-extend.md + extraction/nimclient.md: https://github.com/NVIDIA/NeMo-Retriever/blob/main/nemo_retriever/developer_docs/nimclient.md#nimclient-and-custom-nim-endpoints - site-urls markdown_extensions: @@ -207,11 +211,11 @@ markdown_extensions: - admonition - footnotes -# MkDocs 1.6+: exclude legacy duplicate pages (still in repo for parity). +# MkDocs 1.6+: exclude suite landing and legacy duplicate pages (still in repo for parity). # extraction/chunking.md — removed from nav; content is under concepts.md (redirect_maps keeps old URLs). -# Root index.md is not in nav; redirect_maps sends index.md → extraction/overview.md so /latest/ (Docs Hub tile) resolves. -# Use /index.md in exclude_docs only (bare index.md would exclude every index.md, e.g. extraction/notebooks/index.md). +# Use /index.md (docs root only); bare index.md would exclude every index.md (e.g. under subfolders). exclude_docs: | + /index.md extraction/chunking.md extraction/helm.md extraction/choose-your-path.md From 1bbaef3a2a7a9d5e009ceda9e0d6d530181041a9 Mon Sep 17 00:00:00 2001 From: Kurt Heiss Date: Wed, 8 Jul 2026 08:55:16 -0700 Subject: [PATCH 2/6] docs(26.05): fix agentic MCP path and OCR Helm default Address PR 2313 review: remove the inaccurate MCP section from workflow-agentic-retrieval (no /mcp, mcp-stdio, or ingest_documents on the 26.05 retriever service) and document the supported Python API + VDB retrieval orchestration path instead. Correct support matrix OCR references to match the chart default (nemotron-ocr-v1:1.3.0), not v2. --- .../prerequisites-support-matrix.md | 6 +++--- .../extraction/workflow-agentic-retrieval.md | 20 +------------------ 2 files changed, 4 insertions(+), 22 deletions(-) diff --git a/docs/docs/extraction/prerequisites-support-matrix.md b/docs/docs/extraction/prerequisites-support-matrix.md index fb4315daab..678ff9c523 100644 --- a/docs/docs/extraction/prerequisites-support-matrix.md +++ b/docs/docs/extraction/prerequisites-support-matrix.md @@ -71,20 +71,20 @@ The production Helm chart enables these NIM microservices **by default** (for ex |-----------|-----|------| | `page_elements` | [nemotron-page-elements-v3](https://huggingface.co/nvidia/nemotron-page-elements-v3) | Page layout and element detection | | `table_structure` | [nemotron-table-structure-v1](https://huggingface.co/nvidia/nemotron-table-structure-v1) | Table structure extraction | -| `ocr` | [nemotron-ocr-v2](https://huggingface.co/nvidia/nemotron-ocr-v2) | Image OCR | +| `ocr` | [nemotron-ocr-v1](https://huggingface.co/nvidia/nemotron-ocr-v1) | Image OCR | | `vlm_embed` | [llama-nemotron-embed-vl-1b-v2](https://huggingface.co/nvidia/llama-nemotron-embed-vl-1b-v2) | Multimodal (VL) embedding | ### OCR artifacts (Helm vs local Hugging Face) { #nemotron-ocr-v2-language-mode } !!! note - **Helm / NIM:** The production chart deploys **Nemotron OCR v2** under `nimOperator.ocr` (`nvcr.io/nim/nvidia/nemotron-ocr-v2:1.4.0`). For image defaults and upgrade notes, refer to [OCR NIM configuration](https://github.com/NVIDIA/NeMo-Retriever/blob/main/nemo_retriever/helm/README.md#ocr-nim-configuration) in the Helm chart README. + **Helm / NIM:** The production chart deploys **Nemotron OCR v1** under `nimOperator.ocr` (`nvcr.io/nim/nvidia/nemotron-ocr-v1:1.3.0`). For image defaults and upgrade notes, refer to [OCR NIM configuration](https://github.com/NVIDIA/NeMo-Retriever/blob/main/nemo_retriever/helm/README.md#ocr-nim-configuration) in the Helm chart README. **Local Hugging Face inference:** When you deploy locally with HuggingFace model weights (for example `pip install "nemo-retriever[local]"` and GPU inference without remote OCR NIM URLs), the default OCR engine is **Nemotron OCR v2**, which runs in **multilingual** mode by default. For CLI flags and API parameters, refer to [Nemotron OCR v2 — language mode](https://github.com/NVIDIA/NeMo-Retriever/blob/main/nemo_retriever/docs/cli/README.md#nemotron-ocr-v2-language-mode). Remote OCR NIM endpoints use their own model and language behavior; local OCR language selectors are not sent on remote requests. Default OCR NIM container for release Helm deployments: -- **Image:** `nvcr.io/nim/nvidia/nemotron-ocr-v2:1.4.0` +- **Image:** `nvcr.io/nim/nvidia/nemotron-ocr-v1:1.3.0` Default VL embedder container and model for release deployments: diff --git a/docs/docs/extraction/workflow-agentic-retrieval.md b/docs/docs/extraction/workflow-agentic-retrieval.md index d29b36331b..d3685fb4de 100644 --- a/docs/docs/extraction/workflow-agentic-retrieval.md +++ b/docs/docs/extraction/workflow-agentic-retrieval.md @@ -4,25 +4,7 @@ NeMo Retriever Library provides ingestion, embedding, storage, and retrieval building blocks (jobs, chunking, vector stores, reranking) that you orchestrate in application code or frameworks. -## MCP access for agents - -`retriever service start` mounts a FastMCP HTTP endpoint at `/mcp` by default. Agents can use that endpoint to call the running service for health checks, pipeline introspection, document ingestion, job status, VectorDB query, and answer generation. If service auth is enabled, the MCP endpoint uses the same bearer-token middleware as the REST API. - -For local stdio-based agents, run the MCP server as a shim that points at an existing retriever service: - -```bash -retriever service mcp-stdio \ - --service-url http://localhost:7670 \ - --api-token "$NEMO_RETRIEVER_API_TOKEN" -``` - -For remote agents, expose the retriever service URL and configure the agent to connect to: - -```text -https:///mcp -``` - -The `ingest_documents` MCP tool accepts either paths visible to the MCP server process or inline `content_base64` document bytes. Use inline base64 for remote agents whose local files are not present on the service host. +Wire agentic loops by calling those building blocks from your framework or application: ingest and index with the [Python API](nemo-retriever-api-reference.md) or [Workflow: Ingest documents](workflow-document-ingestion.md), then expose [semantic retrieval](vdbs.md#semantic-retrieval) and [metadata filtering](vdbs.md#metadata-and-filtering) as agent tools. Refer to [Starter kits](starter-kits.md) for LangChain and LlamaIndex multimodal RAG notebooks. **Where to go next** From 69b415c2264fa600249bd977e0b9c088022a153f Mon Sep 17 00:00:00 2001 From: Kurt Heiss Date: Wed, 8 Jul 2026 09:44:10 -0700 Subject: [PATCH 3/6] docs(26.05): align extraction docs byte-for-byte with main Match main with no exceptions per review directive. Reverts the 26.05-only deltas so the PR-touched docs are identical to main: restore MCP agentic section, OCR v2 references in the support matrix, and the main module paths in audio-video and the API reference (mkdocstrings + ASRParams import). --- docs/docs/extraction/audio-video.md | 4 ++-- .../nemo-retriever-api-reference.md | 4 ++-- .../prerequisites-support-matrix.md | 6 +++--- .../extraction/workflow-agentic-retrieval.md | 20 ++++++++++++++++++- 4 files changed, 26 insertions(+), 8 deletions(-) diff --git a/docs/docs/extraction/audio-video.md b/docs/docs/extraction/audio-video.md index 6a6cfa21a6..3f2df2cf9c 100644 --- a/docs/docs/extraction/audio-video.md +++ b/docs/docs/extraction/audio-video.md @@ -73,7 +73,7 @@ Use the following procedure to run the NIM on your own infrastructure. Self-host ```python from nemo_retriever import create_ingestor - from nemo_retriever.params.models import ASRParams + from nemo_retriever.common.params.models import ASRParams ingestor = ( create_ingestor(run_mode="batch") @@ -104,7 +104,7 @@ Instead of running the pipeline locally, you can call Parakeet through [build.nv ```python from nemo_retriever import create_ingestor - from nemo_retriever.params.models import ASRParams + from nemo_retriever.common.params.models import ASRParams ingestor = ( create_ingestor(run_mode="batch") diff --git a/docs/docs/extraction/nemo-retriever-api-reference.md b/docs/docs/extraction/nemo-retriever-api-reference.md index 1b057431d3..2841b9799f 100644 --- a/docs/docs/extraction/nemo-retriever-api-reference.md +++ b/docs/docs/extraction/nemo-retriever-api-reference.md @@ -13,6 +13,6 @@ To tune splitter throughput from the CLI, use `--pdf-split-batch-size` (Ray acto filters: - "!^pdf_split_config$" -::: nemo_retriever.retriever +::: nemo_retriever.graph.retriever -::: nemo_retriever.params +::: nemo_retriever.common.params diff --git a/docs/docs/extraction/prerequisites-support-matrix.md b/docs/docs/extraction/prerequisites-support-matrix.md index 678ff9c523..fb4315daab 100644 --- a/docs/docs/extraction/prerequisites-support-matrix.md +++ b/docs/docs/extraction/prerequisites-support-matrix.md @@ -71,20 +71,20 @@ The production Helm chart enables these NIM microservices **by default** (for ex |-----------|-----|------| | `page_elements` | [nemotron-page-elements-v3](https://huggingface.co/nvidia/nemotron-page-elements-v3) | Page layout and element detection | | `table_structure` | [nemotron-table-structure-v1](https://huggingface.co/nvidia/nemotron-table-structure-v1) | Table structure extraction | -| `ocr` | [nemotron-ocr-v1](https://huggingface.co/nvidia/nemotron-ocr-v1) | Image OCR | +| `ocr` | [nemotron-ocr-v2](https://huggingface.co/nvidia/nemotron-ocr-v2) | Image OCR | | `vlm_embed` | [llama-nemotron-embed-vl-1b-v2](https://huggingface.co/nvidia/llama-nemotron-embed-vl-1b-v2) | Multimodal (VL) embedding | ### OCR artifacts (Helm vs local Hugging Face) { #nemotron-ocr-v2-language-mode } !!! note - **Helm / NIM:** The production chart deploys **Nemotron OCR v1** under `nimOperator.ocr` (`nvcr.io/nim/nvidia/nemotron-ocr-v1:1.3.0`). For image defaults and upgrade notes, refer to [OCR NIM configuration](https://github.com/NVIDIA/NeMo-Retriever/blob/main/nemo_retriever/helm/README.md#ocr-nim-configuration) in the Helm chart README. + **Helm / NIM:** The production chart deploys **Nemotron OCR v2** under `nimOperator.ocr` (`nvcr.io/nim/nvidia/nemotron-ocr-v2:1.4.0`). For image defaults and upgrade notes, refer to [OCR NIM configuration](https://github.com/NVIDIA/NeMo-Retriever/blob/main/nemo_retriever/helm/README.md#ocr-nim-configuration) in the Helm chart README. **Local Hugging Face inference:** When you deploy locally with HuggingFace model weights (for example `pip install "nemo-retriever[local]"` and GPU inference without remote OCR NIM URLs), the default OCR engine is **Nemotron OCR v2**, which runs in **multilingual** mode by default. For CLI flags and API parameters, refer to [Nemotron OCR v2 — language mode](https://github.com/NVIDIA/NeMo-Retriever/blob/main/nemo_retriever/docs/cli/README.md#nemotron-ocr-v2-language-mode). Remote OCR NIM endpoints use their own model and language behavior; local OCR language selectors are not sent on remote requests. Default OCR NIM container for release Helm deployments: -- **Image:** `nvcr.io/nim/nvidia/nemotron-ocr-v1:1.3.0` +- **Image:** `nvcr.io/nim/nvidia/nemotron-ocr-v2:1.4.0` Default VL embedder container and model for release deployments: diff --git a/docs/docs/extraction/workflow-agentic-retrieval.md b/docs/docs/extraction/workflow-agentic-retrieval.md index d3685fb4de..d29b36331b 100644 --- a/docs/docs/extraction/workflow-agentic-retrieval.md +++ b/docs/docs/extraction/workflow-agentic-retrieval.md @@ -4,7 +4,25 @@ NeMo Retriever Library provides ingestion, embedding, storage, and retrieval building blocks (jobs, chunking, vector stores, reranking) that you orchestrate in application code or frameworks. -Wire agentic loops by calling those building blocks from your framework or application: ingest and index with the [Python API](nemo-retriever-api-reference.md) or [Workflow: Ingest documents](workflow-document-ingestion.md), then expose [semantic retrieval](vdbs.md#semantic-retrieval) and [metadata filtering](vdbs.md#metadata-and-filtering) as agent tools. Refer to [Starter kits](starter-kits.md) for LangChain and LlamaIndex multimodal RAG notebooks. +## MCP access for agents + +`retriever service start` mounts a FastMCP HTTP endpoint at `/mcp` by default. Agents can use that endpoint to call the running service for health checks, pipeline introspection, document ingestion, job status, VectorDB query, and answer generation. If service auth is enabled, the MCP endpoint uses the same bearer-token middleware as the REST API. + +For local stdio-based agents, run the MCP server as a shim that points at an existing retriever service: + +```bash +retriever service mcp-stdio \ + --service-url http://localhost:7670 \ + --api-token "$NEMO_RETRIEVER_API_TOKEN" +``` + +For remote agents, expose the retriever service URL and configure the agent to connect to: + +```text +https:///mcp +``` + +The `ingest_documents` MCP tool accepts either paths visible to the MCP server process or inline `content_base64` document bytes. Use inline base64 for remote agents whose local files are not present on the service host. **Where to go next** From 14b31abdf203e6a67607ecae73d470a6f3f758ef Mon Sep 17 00:00:00 2001 From: Kurt Heiss Date: Wed, 8 Jul 2026 09:57:12 -0700 Subject: [PATCH 4/6] docs(26.05): remove unsupported MCP workflow from agentic retrieval 26.05 retriever service does not expose /mcp, mcp-stdio, or ingest_documents. Document the supported Python API, VDB retrieval, and starter-kit orchestration path instead. --- .../extraction/workflow-agentic-retrieval.md | 20 +------------------ 1 file changed, 1 insertion(+), 19 deletions(-) diff --git a/docs/docs/extraction/workflow-agentic-retrieval.md b/docs/docs/extraction/workflow-agentic-retrieval.md index d29b36331b..d0772c1651 100644 --- a/docs/docs/extraction/workflow-agentic-retrieval.md +++ b/docs/docs/extraction/workflow-agentic-retrieval.md @@ -4,25 +4,7 @@ NeMo Retriever Library provides ingestion, embedding, storage, and retrieval building blocks (jobs, chunking, vector stores, reranking) that you orchestrate in application code or frameworks. -## MCP access for agents - -`retriever service start` mounts a FastMCP HTTP endpoint at `/mcp` by default. Agents can use that endpoint to call the running service for health checks, pipeline introspection, document ingestion, job status, VectorDB query, and answer generation. If service auth is enabled, the MCP endpoint uses the same bearer-token middleware as the REST API. - -For local stdio-based agents, run the MCP server as a shim that points at an existing retriever service: - -```bash -retriever service mcp-stdio \ - --service-url http://localhost:7670 \ - --api-token "$NEMO_RETRIEVER_API_TOKEN" -``` - -For remote agents, expose the retriever service URL and configure the agent to connect to: - -```text -https:///mcp -``` - -The `ingest_documents` MCP tool accepts either paths visible to the MCP server process or inline `content_base64` document bytes. Use inline base64 for remote agents whose local files are not present on the service host. +Wire agentic loops in your orchestration framework or application code: ingest and index with the [Python API](nemo-retriever-api-reference.md) or [Workflow: Ingest documents](workflow-document-ingestion.md), then expose [semantic retrieval](vdbs.md#semantic-retrieval) and [metadata filtering](vdbs.md#metadata-and-filtering) as agent tools. Refer to [Starter kits](starter-kits.md) for LangChain and LlamaIndex multimodal RAG notebooks. **Where to go next** From 6d7dc44988064fef5d260b0935b86b75a24d6c92 Mon Sep 17 00:00:00 2001 From: Kurt Heiss Date: Wed, 8 Jul 2026 10:00:56 -0700 Subject: [PATCH 5/6] docs(26.05): correct Helm OCR default in support matrix Align nimOperator.ocr with the 26.05 chart default (nemotron-ocr-v1:1.3.0). Keep local Hugging Face OCR v2 paragraph unchanged. --- docs/docs/extraction/prerequisites-support-matrix.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/docs/extraction/prerequisites-support-matrix.md b/docs/docs/extraction/prerequisites-support-matrix.md index fb4315daab..2639059507 100644 --- a/docs/docs/extraction/prerequisites-support-matrix.md +++ b/docs/docs/extraction/prerequisites-support-matrix.md @@ -71,20 +71,20 @@ The production Helm chart enables these NIM microservices **by default** (for ex |-----------|-----|------| | `page_elements` | [nemotron-page-elements-v3](https://huggingface.co/nvidia/nemotron-page-elements-v3) | Page layout and element detection | | `table_structure` | [nemotron-table-structure-v1](https://huggingface.co/nvidia/nemotron-table-structure-v1) | Table structure extraction | -| `ocr` | [nemotron-ocr-v2](https://huggingface.co/nvidia/nemotron-ocr-v2) | Image OCR | +| `ocr` | [nemotron-ocr-v1](https://huggingface.co/nvidia/nemotron-ocr-v1) | Image OCR | | `vlm_embed` | [llama-nemotron-embed-vl-1b-v2](https://huggingface.co/nvidia/llama-nemotron-embed-vl-1b-v2) | Multimodal (VL) embedding | ### OCR artifacts (Helm vs local Hugging Face) { #nemotron-ocr-v2-language-mode } !!! note - **Helm / NIM:** The production chart deploys **Nemotron OCR v2** under `nimOperator.ocr` (`nvcr.io/nim/nvidia/nemotron-ocr-v2:1.4.0`). For image defaults and upgrade notes, refer to [OCR NIM configuration](https://github.com/NVIDIA/NeMo-Retriever/blob/main/nemo_retriever/helm/README.md#ocr-nim-configuration) in the Helm chart README. + **Helm / NIM:** The production chart deploys **Nemotron OCR v1** under `nimOperator.ocr` (`nvcr.io/nim/nvidia/nemotron-ocr-v1:1.3.0`). For image defaults and upgrade notes, refer to [OCR NIM configuration](https://github.com/NVIDIA/NeMo-Retriever/blob/26.05/nemo_retriever/helm/README.md#ocr-nim-configuration) in the Helm chart README. **Local Hugging Face inference:** When you deploy locally with HuggingFace model weights (for example `pip install "nemo-retriever[local]"` and GPU inference without remote OCR NIM URLs), the default OCR engine is **Nemotron OCR v2**, which runs in **multilingual** mode by default. For CLI flags and API parameters, refer to [Nemotron OCR v2 — language mode](https://github.com/NVIDIA/NeMo-Retriever/blob/main/nemo_retriever/docs/cli/README.md#nemotron-ocr-v2-language-mode). Remote OCR NIM endpoints use their own model and language behavior; local OCR language selectors are not sent on remote requests. Default OCR NIM container for release Helm deployments: -- **Image:** `nvcr.io/nim/nvidia/nemotron-ocr-v2:1.4.0` +- **Image:** `nvcr.io/nim/nvidia/nemotron-ocr-v1:1.3.0` Default VL embedder container and model for release deployments: From cb23e1d97d01c0d1fc7e98a3ba070eff5462c238 Mon Sep 17 00:00:00 2001 From: Kurt Heiss Date: Wed, 8 Jul 2026 10:54:24 -0700 Subject: [PATCH 6/6] docs(26.05): scope extraction docs to 26.5.0 behavior Revert main-only information-architecture and code-path churn that did not apply to the 26.05 release (restored custom-metadata, integrations, and nimclient pages; dropped main-only customize-extend and starter-kits pages; reverted nemo_retriever.common.params / graph.retriever module paths and blob/main link retargeting back to the 26.05 surface). Re-apply only genuine June-July documentation fixes valid for 26.5.0: - faq.md, multimodal-extraction.md: chart modality requires the default pdfium layout path; nemotron_parse does not emit chart/infographic rows - prerequisites-support-matrix.md: H200 NVL unsupported for self-hosted Parakeet ASR; nemotron-parse supported on RTX PRO 4500 - troubleshoot.md: replace stale save_to_disk OOM guidance with supported batching / output-sink / return_results guidance Docs-only; no runtime, test, or chart changes. --- .../extraction/agentic-retrieval-concept.md | 2 +- docs/docs/extraction/audio-video.md | 6 +- docs/docs/extraction/concepts.md | 30 +- docs/docs/extraction/custom-metadata.md | 125 ++++ docs/docs/extraction/customize-extend.md | 66 -- docs/docs/extraction/embedding.md | 8 +- docs/docs/extraction/faq.md | 9 +- ...egrations-langchain-llamaindex-haystack.md | 22 + docs/docs/extraction/multimodal-extraction.md | 4 +- .../nemo-retriever-api-reference.md | 8 +- docs/docs/extraction/nimclient.md | 588 ++++++++++++++++++ .../prerequisites-support-matrix.md | 48 +- docs/docs/extraction/starter-kits.md | 24 - docs/docs/extraction/troubleshoot.md | 10 +- .../extraction/workflow-agentic-retrieval.md | 5 +- .../extraction/workflow-document-ingestion.md | 22 +- .../extraction/workflow-e2e-blueprints.md | 2 +- docs/mkdocs.yml | 40 +- 18 files changed, 843 insertions(+), 176 deletions(-) create mode 100644 docs/docs/extraction/custom-metadata.md delete mode 100644 docs/docs/extraction/customize-extend.md create mode 100644 docs/docs/extraction/integrations-langchain-llamaindex-haystack.md create mode 100644 docs/docs/extraction/nimclient.md delete mode 100644 docs/docs/extraction/starter-kits.md diff --git a/docs/docs/extraction/agentic-retrieval-concept.md b/docs/docs/extraction/agentic-retrieval-concept.md index b2bfd09cfd..a06431a359 100644 --- a/docs/docs/extraction/agentic-retrieval-concept.md +++ b/docs/docs/extraction/agentic-retrieval-concept.md @@ -7,4 +7,4 @@ NeMo Retriever Library focuses on document ingestion, embeddings, vector stores, **Related** - [Semantic retrieval](vdbs.md#semantic-retrieval) -- Framework examples: [Starter kits](starter-kits.md) +- Framework examples: [LangChain, LlamaIndex, Haystack](integrations-langchain-llamaindex-haystack.md) diff --git a/docs/docs/extraction/audio-video.md b/docs/docs/extraction/audio-video.md index 3f2df2cf9c..c9031ce413 100644 --- a/docs/docs/extraction/audio-video.md +++ b/docs/docs/extraction/audio-video.md @@ -73,7 +73,7 @@ Use the following procedure to run the NIM on your own infrastructure. Self-host ```python from nemo_retriever import create_ingestor - from nemo_retriever.common.params.models import ASRParams + from nemo_retriever.params.models import ASRParams ingestor = ( create_ingestor(run_mode="batch") @@ -100,11 +100,9 @@ Instead of running the pipeline locally, you can call Parakeet through [build.nv 2. Run inference from Python with the hosted gRPC endpoint and credentials from that page (the example below uses the default hosted gRPC hostname; confirm values in the **Get API Key** flow for your deployment). Pass hosted endpoint, function ID, and API key through `ASRParams` (`audio_endpoints`, `function_id`, `auth_token`). - For parameter details, refer to the [Python API guide](nemo-retriever-api-reference.md). - ```python from nemo_retriever import create_ingestor - from nemo_retriever.common.params.models import ASRParams + from nemo_retriever.params.models import ASRParams ingestor = ( create_ingestor(run_mode="batch") diff --git a/docs/docs/extraction/concepts.md b/docs/docs/extraction/concepts.md index bb6ce80998..4418682052 100644 --- a/docs/docs/extraction/concepts.md +++ b/docs/docs/extraction/concepts.md @@ -2,40 +2,40 @@ These terms appear throughout NeMo Retriever Library documentation. -## Job { #job } +## Job -An **ingestion job** is a unit of work you run on input content (documents, audio, video, and other supported types). You submit jobs through the **ingestor Python API** (for example `Ingestor` task chains such as `.extract(...)`) or the **`retriever ingest` CLI**—not by posting a standalone JSON job document. Default tasks target strong recall; customize behavior with task keyword arguments (including chunking and splitting on `.extract()`) or custom UDF-style operations. For UDFs and other extension paths, refer to [Customize & extend](customize-extend.md). Results are structured metadata and annotations (Ray Dataset, pandas `DataFrame`, or similar). +An **ingestion job** is a unit of work you run on input content (documents, audio, video, and other supported types). You submit jobs through the **ingestor Python API** (for example `Ingestor` task chains such as `.extract(...)`) or the **`retriever ingest` CLI**—not by posting a standalone JSON job document. Default tasks target strong recall; customize behavior with task keyword arguments (including chunking and splitting on `.extract()`) or custom UDF-style operations ([NeMo Retriever graph](https://github.com/NVIDIA/NeMo-Retriever/tree/main/nemo_retriever/src/nemo_retriever/graph#nemo-retriever-graph)). Results are structured metadata and annotations (Ray Dataset, pandas `DataFrame`, or similar). -## Pipeline and tasks { #pipeline-and-tasks } +## Pipeline and tasks -NeMo Retriever Library does **not** run one static pipeline on every document. You configure **tasks** such as parsing, chunking, embedding, storage, and filtering per job. For UDFs, custom graph stages, and other extension paths, refer to [Customize & extend](customize-extend.md). +NeMo Retriever Library does **not** run one static pipeline on every document. You configure **tasks** such as parsing, chunking, embedding, storage, and filtering per job. Related topics: [Extending/Customizing NeMo Retriever Library with custom code](https://github.com/NVIDIA/NeMo-Retriever/tree/main/nemo_retriever/src/nemo_retriever/graph#nemo-retriever-graph). -## Extraction metadata { #extraction-metadata } +## Extraction metadata Output is a **Ray Dataset** (Ray Data) or **pandas** `DataFrame` listing extracted objects (text regions, tables, images, and so on), processing notes, and timing or trace data. Field-level detail is in the [metadata reference](content-metadata.md). -## Embeddings and retrieval { #embeddings-and-retrieval } +## Embeddings and retrieval -Optionally, the library can compute **embeddings** for extracted content and store vectors in [LanceDB](https://lancedb.com/) for downstream semantic search in your application. For upload and retrieval APIs, refer to [Vector databases](vdbs.md). For multimodal (VLM) embedding options, refer to [Multimodal embeddings (VLM)](embedding.md). +Optionally, the library can compute **embeddings** for extracted content and store vectors in [LanceDB](https://lancedb.com/) for downstream semantic search in your application. For upload and retrieval APIs, see [Vector databases](vdbs.md). For multimodal (VLM) embedding options, see [Multimodal embeddings (VLM)](embedding.md). ## Chunking { #chunking } Chunking is built into the `.extract()` task and depends on **content type**: - **PDF, DOCX, and PPTX** — Text is grouped using built-in **page** boundaries (one chunk per page where the format has pages). -- **Plain text (`.txt`) and HTML** — Formats without natural page breaks are split into segments of **1024 tokens** by default, using the [Llama 3.2 1B tokenizer](https://huggingface.co/meta-llama/Llama-3.2-1B) so chunk boundaries stay aligned with the default embedding tokenizer. The NeMo Retriever container image bundles this tokenizer, so default text chunking does not require a Hugging Face access token. Refer to [Token-based splitting](#token-based-splitting) and [Environment variables](environment-config.md) for overrides and other runtimes. -- **Audio and video** — Media is split into **segments** for decoding and ASR using ffmpeg-based rules (configurable **size**, **time**, or **frame** split modes in the media chunking stage). With the Parakeet ASR path, you can optionally emit **sentence-like segments** using `extract_audio_params={"segment_audio": True}`; refer to [Speech and audio extraction](audio-video.md#speech-and-audio-extraction). +- **Plain text (`.txt`) and HTML** — Formats without natural page breaks are split into segments of **1024 tokens** by default, using the [Llama 3.2 1B tokenizer](https://huggingface.co/meta-llama/Llama-3.2-1B) so chunk boundaries stay aligned with the default embedding tokenizer. The NeMo Retriever container image bundles this tokenizer, so default text chunking does not require a Hugging Face access token. See [Token-based splitting](#token-based-splitting) and [Environment variables](environment-config.md) for overrides and other runtimes. +- **Audio and video** — Media is split into **segments** for decoding and ASR using ffmpeg-based rules (configurable **size**, **time**, or **frame** split modes in the media chunking stage). With the Parakeet ASR path, you can optionally emit **sentence-like segments** using `extract_audio_params={"segment_audio": True}`; see [Speech and audio extraction](audio-video.md#speech-and-audio-extraction). -For PDF parallelism before Ray processing (large files), refer to [PDF pre-splitting for parallel ingest](nemo-retriever-api-reference.md#pdf-pre-splitting-for-parallel-ingest). +For PDF parallelism before Ray processing (large files), see [PDF pre-splitting for parallel ingest](nemo-retriever-api-reference.md#pdf-pre-splitting-for-parallel-ingest). ### Token-based splitting { #token-based-splitting } -Token-based splitting uses the Llama 3.2 1B tokenizer (default `meta-llama/Llama-3.2-1B`) with configurable `max_tokens` and `overlap_tokens` when you add an explicit `.split(...)` stage or when the pipeline applies the default text segmentation for unstructured text. In the shipped NeMo Retriever container, tokenizer assets are included locally, so you do not need `HF_ACCESS_TOKEN` for this default path. If your runtime loads the tokenizer from the Hugging Face Hub instead (for example, some library-only installs), set `HF_ACCESS_TOKEN` or pass `hf_access_token` in task params when the Hub requires it. For parameter details, refer to the [Python API guide](nemo-retriever-api-reference.md). +Token-based splitting uses the Llama 3.2 1B tokenizer (default `meta-llama/Llama-3.2-1B`) with configurable `max_tokens` and `overlap_tokens` when you add an explicit `.split(...)` stage or when the pipeline applies the default text segmentation for unstructured text. In the shipped NeMo Retriever container, tokenizer assets are included locally, so you do not need `HF_ACCESS_TOKEN` for this default path. If your runtime loads the tokenizer from the Hugging Face Hub instead (for example, some library-only installs), set `HF_ACCESS_TOKEN` or pass `hf_access_token` in task params when the Hub requires it. Details appear in the [Python API guide](nemo-retriever-api-reference.md). -## Deployment modes { #deployment-modes } +## Deployment modes -- **Library mode** — Run without the full container stack where appropriate; refer to [Deployment options](deployment-options.md). -- **Kubernetes / Helm (self-hosted)** — Refer to [Deploy (Helm chart)](https://github.com/NVIDIA/NeMo-Retriever/blob/main/nemo_retriever/helm/README.md) and [deployment options](deployment-options.md) for running the full microservices pipeline on your infrastructure. -- **Notebooks** — [Jupyter examples](starter-kits.md) for experimentation and RAG demos. +- **Library mode** — Run without the full container stack where appropriate; see [Deployment options](deployment-options.md). +- **Kubernetes / Helm (self-hosted)** — See [Deploy (Helm chart)](https://github.com/NVIDIA/NeMo-Retriever/blob/main/nemo_retriever/helm/README.md) and [deployment options](deployment-options.md) for running the full microservices pipeline on your infrastructure. +- **Notebooks** — [Jupyter examples](notebooks/index.md) for experimentation and RAG demos. For a concise comparison, refer to [Deployment options](deployment-options.md). diff --git a/docs/docs/extraction/custom-metadata.md b/docs/docs/extraction/custom-metadata.md new file mode 100644 index 0000000000..45ffa34d29 --- /dev/null +++ b/docs/docs/extraction/custom-metadata.md @@ -0,0 +1,125 @@ +# Custom metadata and filtering + +Use this documentation to attach per-document metadata during ingestion and to narrow [LanceDB](vdbs.md) search results in [NeMo Retriever Library](overview.md). Implementation details live in the package [Vector DB operators and LanceDB](https://github.com/NVIDIA/NeMo-Retriever/tree/main/nemo_retriever/src/nemo_retriever/vdb#metadata-filtering) README. + +## On this page { #on-this-page } + +- [Attach metadata at ingestion](#attach-metadata-at-ingestion) +- [Best practices](#best-practices) +- [Filter results during retrieval](#filter-results-during-retrieval) +- [How metadata is stored](#how-metadata-is-stored) + +## Attach metadata at ingestion { #attach-metadata-at-ingestion } + +Pass a **sidecar metadata table** on `vdb_upload` so selected columns are merged into each chunk's `content_metadata` before LanceDB upload. All three parameters must be set together: + +| Parameter | Purpose | +|-----------|---------| +| `meta_dataframe` | Path to CSV, JSON, or Parquet, or an in-memory `pandas.DataFrame` | +| `meta_source_field` | Column that identifies each document (must match ingest paths or basenames per `meta_join_key`) | +| `meta_fields` | Non-empty list of column names to copy into `content_metadata` | + +Optional `meta_join_key` controls how rows are matched to documents: `auto` (try full path then basename), `source_id` (full path), or `source_name` (basename only). + +For parameter details, refer to the [Python API guide](nemo-retriever-api-reference.md). + +```python +import pandas as pd +from nemo_retriever import create_ingestor + +meta_df = pd.DataFrame( + { + "source": ["data/woods_frost.pdf", "data/multimodal_test.pdf"], + "meta_a": ["alpha", "bravo"], + "meta_b": [10, 20], + } +) + +hostname = "localhost" +table_name = "nemo_retriever_collection" +lancedb_uri = "s3://your-bucket/lancedb" + +ingestor = ( + create_ingestor(run_mode="service", base_url=f"http://{hostname}:7670") + .files(["data/woods_frost.pdf", "data/multimodal_test.pdf"]) + .extract( + extract_text=True, + extract_tables=True, + extract_charts=True, + extract_images=True, + text_depth="page" + ) + .embed() + .vdb_upload( + vdb_op="lancedb", + vdb_kwargs={"lancedb_uri": lancedb_uri, "table_name": table_name}, + meta_dataframe=meta_df, + meta_source_field="source", + meta_fields=["meta_a", "meta_b"], + ) +) +results = ingestor.ingest_async().result() +``` + +Set `hostname`, `table_name`, and a **remote** `lancedb_uri` (for example `s3://bucket/path`) to match your deployment—the retriever service rejects local filesystem paths. The client uploads in-memory sidecar metadata to the service before ingest; do not pass a raw local file path as `meta_dataframe` on the REST spec. For local LanceDB directories, use `run_mode="batch"` instead (refer to [Vector databases](vdbs.md)). For a step-by-step walkthrough with additional fields such as category, department, and timestamp, refer to [Vector DB operators and LanceDB — Metadata filtering](https://github.com/NVIDIA/NeMo-Retriever/tree/main/nemo_retriever/src/nemo_retriever/vdb#metadata-filtering). + +## Best practices { #best-practices } + +- Plan metadata structure before ingestion. +- Test filter expressions with small datasets first. +- Consider performance implications of complex filters. +- Validate metadata during ingestion. +- Handle missing metadata fields gracefully. +- Log invalid filter expressions. + +## Filter results during retrieval { #filter-results-during-retrieval } + +You can use custom metadata to filter documents during retrieval operations. For **predicate pushdown**, pass a `where` SQL predicate through [`Retriever.query`](nemo-retriever-api-reference.md) (refer to [Vector databases](vdbs.md)) or chain `.where(...)` on a native LanceDB `table.search(...)` query. Application-side filtering on returned hits does not change what the database evaluates—raise `top_k` if matches might sit outside the first neighbors. + +### Example filter ideas + +Typical keys to filter on include `category`, `department`, `priority`, and `timestamp` (use comparable ISO-8601 strings for time ranges). Encode predicates in LanceDB SQL against your table columns (often the serialized `metadata` string), or inspect parsed hit metadata after search as in the example below. + +### Example: Use a Filter Expression in Search + +After ingestion is complete and documents are uploaded to LanceDB with metadata, you can narrow results in the database with a **`where`** clause, or in Python on the returned hits. + +**Native LanceDB (SQL pushdown):** connect, embed the query yourself (same model as ingestion), then chain `.where("")` on `table.search(...)` so filtering happens before the `limit`. Exact SQL depends on how `metadata` is stored; refer to [LanceDB metadata filtering](https://docs.lancedb.com/search/filtering#filtering-with-sql). + +```python +import lancedb + +# pseudocode — replace YOUR_VECTOR and YOUR_PREDICATE with real values. +db = lancedb.connect("./lancedb_data") +table = db.open_table("nemo_retriever_collection") +# table.search(YOUR_VECTOR, vector_column_name="vector").where(YOUR_PREDICATE).limit(10).to_list() +``` + +**`Retriever.query` + `where`:** LanceDB applies the predicate before ranking. For post-filter logic in Python, use a wider `top_k` first. + +```python +from nemo_retriever.retriever import Retriever + +retriever = Retriever( + vdb_kwargs={"uri": "./lancedb_data", "table_name": "nemo_retriever_collection"}, + embed_kwargs={ + "model_name": "nvidia/llama-nemotron-embed-1b-v2", + "embed_model_name": "nvidia/llama-nemotron-embed-1b-v2", + }, +) + +hits = retriever.query( + "this is expensive", + top_k=16, + vdb_kwargs={"where": "metadata LIKE '%\"department\":\"Engineering\"%'"}, +) +``` + +For a runnable end-to-end flow (ingest, `Retriever.query`, and both filter modes), refer to [nemo_retriever_retriever_query_metadata_filter.ipynb](https://github.com/NVIDIA/NeMo-Retriever/blob/main/examples/nemo_retriever_retriever_query_metadata_filter.ipynb). + +When you ingest through the **retriever service**, upload the sidecar with [`POST /v1/ingest/sidecar`](https://github.com/NVIDIA/NeMo-Retriever/blob/main/nemo_retriever/src/nemo_retriever/service/routers/ingest.py#L1040-L1129) (multipart file; response [`SidecarUploadResponse`](https://github.com/NVIDIA/NeMo-Retriever/blob/main/nemo_retriever/src/nemo_retriever/service/models/responses.py#L60-L68)), then pass the returned `sidecar_id` as `meta_dataframe_id` with `meta_source_field` and `meta_fields` in `pipeline.vdb_upload_params` on [`POST /v1/ingest`](https://github.com/NVIDIA/NeMo-Retriever/blob/main/nemo_retriever/src/nemo_retriever/service/models/requests.py#L15-L32) ([`PipelineSpec`](https://github.com/NVIDIA/NeMo-Retriever/blob/main/nemo_retriever/src/nemo_retriever/service/models/pipeline_spec.py#L55-L78)). Request and response shapes, form fields, and auth headers are in the service OpenAPI UI at `/docs` (or `/openapi.json`) on your retriever base URL (for example `http://localhost:7670/docs` after `retriever service start`). Do not send a raw local path as `meta_dataframe` on the service spec. + +## How metadata is stored { #how-metadata-is-stored } + +- [Vector databases](vdbs.md) — canonical LanceDB upload and retrieval guide +- [nemo_retriever_retriever_query_metadata_filter.ipynb](https://github.com/NVIDIA/NeMo-Retriever/blob/main/examples/nemo_retriever_retriever_query_metadata_filter.ipynb) — runnable notebook for sidecar metadata at ingest and filtered `Retriever.query` diff --git a/docs/docs/extraction/customize-extend.md b/docs/docs/extraction/customize-extend.md deleted file mode 100644 index 31fb87e7bc..0000000000 --- a/docs/docs/extraction/customize-extend.md +++ /dev/null @@ -1,66 +0,0 @@ -# Customize & extend - -NeMo Retriever Library ships with defaults tuned for strong recall on common document types. When those defaults are not enough, you can extend the library at several levels—from task keyword arguments on the fluent ingestor API through custom graph operators and vector-database adapters. - -Use this page to choose an extension path and find the detailed guides in the repository. - -The following table maps common needs to the right section: - -| If you need to… | Start here | -|-----------------|------------| -| Tune extraction, chunking, embedding, or upload without new code | [Start with task configuration](#start-with-task-configuration) | -| Add a small Python transformation between pipeline stages | [User-defined functions (UDFs)](#user-defined-functions-udfs) | -| Build or reuse operators stage-by-stage | [Custom graph pipelines](#custom-graph-pipelines) | -| Store vectors in a backend other than LanceDB | [Custom vector databases](#custom-vector-databases) | - -## On this page { #on-this-page } - -- [Start with task configuration](#start-with-task-configuration) -- [User-defined functions (UDFs)](#user-defined-functions-udfs) -- [Custom graph pipelines](#custom-graph-pipelines) -- [Custom vector databases](#custom-vector-databases) -- [Related Topics](#related-topics) - -## Start with task configuration { #start-with-task-configuration } - -Most customization does not require new code. Chain tasks on `create_ingestor(...)` and pass keyword arguments to control extraction, chunking, embedding, and storage—for example `extract_method`, chunking and splitting options on `.extract()`, `embed_modality` on `.embed()`, and `vdb_op` / `vdb_kwargs` on `.vdb_upload()`. - -For parameter details, refer to the [Python API guide](nemo-retriever-api-reference.md). For chunking behavior and pipeline concepts, refer to [Concepts](concepts.md). - -## User-defined functions (UDFs) { #user-defined-functions-udfs } - -A **user-defined function (UDF)** wraps your Python logic as a first-class pipeline stage. In the graph model, `UDFOperator` turns a plain callable into an operator you can chain with built-in stages—for example to normalize HTML, apply a custom split, or call an external service between extract and embed steps. - -Use UDFs when you need a small, self-contained transformation that is not covered by task keyword arguments. - -### Repository guides - -- [NeMo Retriever graph README — `UDFOperator`](https://github.com/NVIDIA/NeMo-Retriever/tree/main/nemo_retriever/src/nemo_retriever/graph#using-udfoperator) — API, lifecycle, and when to use `UDFOperator` versus a custom operator class -- [NimClient and custom NIM endpoints](https://github.com/NVIDIA/NeMo-Retriever/blob/main/nemo_retriever/developer_docs/nimclient.md#nimclient-and-custom-nim-endpoints) — call custom or self-hosted NIM microservices from UDF stages - -## Custom graph pipelines { #custom-graph-pipelines } - -When you need to compose pipelines stage-by-stage, reuse operators across workflows, or run the same graph in-process or with Ray Data, use the **graph execution model** instead of (or alongside) the fluent `GraphIngestor` API. - -The graph package provides `AbstractOperator`, executors (`InprocessExecutor`, `RayDataExecutor`), and operator chaining with `>>`. Built-in ingestion operators live under `nemo_retriever.operators`; you can add your own operators or UDF stages anywhere in the chain. - -For the full guide—including custom operator classes, executors, and graph shape constraints—refer to the [NeMo Retriever graph README](https://github.com/NVIDIA/NeMo-Retriever/tree/main/nemo_retriever/src/nemo_retriever/graph#nemo-retriever-graph). - -## Custom vector databases { #custom-vector-databases } - -The supported user path for vector storage is **[LanceDB](vdbs.md)** (`vdb_op="lancedb"`). That page covers upload, semantic retrieval, metadata filtering, and LanceDB deployment characteristics. - -To integrate a different vector store, implement the [`VDB`](https://github.com/NVIDIA/NeMo-Retriever/blob/main/nemo_retriever/src/nemo_retriever/common/vdb/adt_vdb.py) interface and wire it through graph [`IngestVdbOperator`](https://github.com/NVIDIA/NeMo-Retriever/blob/main/nemo_retriever/src/nemo_retriever/operators/vdb.py) / [`RetrieveVdbOperator`](https://github.com/NVIDIA/NeMo-Retriever/blob/main/nemo_retriever/src/nemo_retriever/operators/vdb.py). NVIDIA validates the first-party LanceDB operator; you are responsible for testing and maintaining other backends. - -### Repository guides - -- [Vector DB package (source)](https://github.com/NVIDIA/NeMo-Retriever/tree/main/nemo_retriever/src/nemo_retriever/common/vdb) — `VDB` abstract base and LanceDB reference implementation - -Partner and blueprint integrations (Elasticsearch, Pinecone, Teradata, and others) are summarized on [Vector databases — Vector database partners](vdbs.md#vector-database-partners). - -## Related Topics { #related-topics } - -- [Concepts — Pipeline and tasks](concepts.md#pipeline-and-tasks) -- [Vector databases](vdbs.md) -- [Multimodal embeddings (VLM)](embedding.md) -- [Python API guide](nemo-retriever-api-reference.md) diff --git a/docs/docs/extraction/embedding.md b/docs/docs/extraction/embedding.md index b6a139fac3..e42574a519 100644 --- a/docs/docs/extraction/embedding.md +++ b/docs/docs/extraction/embedding.md @@ -12,6 +12,8 @@ The model can embed documents in the form of an image, text, or a combination of Documents can then be retrieved given a user query in text form. The model supports images that contain text, tables, charts, and infographics. +Parameter details for `.extract()` and `.embed()` appear in the [Python API guide](nemo-retriever-api-reference.md). + ## Example with Default Text-Based Embedding When you use the multimodal model, by default, all extracted content (text, tables, charts) is treated as plain text. @@ -19,8 +21,6 @@ The following example provides a strong baseline for retrieval. - The `embed` method is called with no arguments. -For parameter details, refer to the [Python API guide](nemo-retriever-api-reference.md). - ```python from nemo_retriever import create_ingestor @@ -42,8 +42,6 @@ The following example enables the multimodal model to capture the spatial and st - The `embed` method is configured with `embed_modality="text_image"` to embed the extracted tables and charts as images. - This configuration is more accurate than text only, with a performance cost. -For parameter details, refer to the [Python API guide](nemo-retriever-api-reference.md). - ```python from nemo_retriever import create_ingestor @@ -67,8 +65,6 @@ The following example extracts and embeds each page as an image. - The `embed` method processes the page images. -For parameter details, refer to the [Python API guide](nemo-retriever-api-reference.md). - ```python from nemo_retriever import create_ingestor diff --git a/docs/docs/extraction/faq.md b/docs/docs/extraction/faq.md index 6b3010db52..3bca3bd8be 100644 --- a/docs/docs/extraction/faq.md +++ b/docs/docs/extraction/faq.md @@ -21,16 +21,17 @@ For more information, refer to [Vector databases](vdbs.md). For images that `nemoretriever-page-elements-v3` does not classify as tables, charts, or infographics, you can use our VLM caption task to create a dense caption of the detected image. That caption is then embedded along with the rest of your content. -For chart-labeled PDF regions and other caption scope limits, see [Are PDF chart or figure regions captioned when Omni is enabled?](#are-pdf-chart-or-figure-regions-captioned-when-omni-is-enabled). For more information, refer to [Extract Captions from Images](nemo-retriever-api-reference.md). +For chart-labeled PDF regions and other caption scope limits, refer to [Are PDF chart or figure regions captioned when Omni is enabled?](#are-pdf-chart-or-figure-regions-captioned-when-omni-is-enabled). For more information, refer to [Extract Captions from Images](nemo-retriever-api-reference.md). ## Are PDF chart or figure regions captioned when Omni is enabled? -No. Chart-labeled PDF regions are not routed through Omni captioning. See [Image captioning](prerequisites-support-matrix.md#image-captioning-2605) for scope, validation, and what the caption stage covers. +No. Chart-labeled PDF regions are not routed through Omni captioning. Refer to [Charts and infographics](multimodal-extraction.md#charts-and-infographics) and [Image captioning](multimodal-extraction.md#image-captioning) for caption scope and validation. ## When should I consider advanced visual parsing? -For scanned documents, or documents with complex layouts, -you can use [nemotron-parse](https://build.nvidia.com/nvidia/nemotron-parse) as an alternate PDF extraction method by setting `extract_method="nemotron_parse"`. +For scanned documents, or documents with complex layouts, +you can use [nemotron-parse](https://build.nvidia.com/nvidia/nemotron-parse) as an alternate PDF extraction method by setting `extract_method="nemotron_parse"`. +Nemotron Parse does not produce chart modality rows. For chart detection and chart-filtered retrieval, use the default **pdfium** layout path instead (refer to [Charts and infographics](multimodal-extraction.md#charts-and-infographics)). For more information, refer to [Nemotron Parse](https://build.nvidia.com/nvidia/nemotron-parse). ## Why are the environment variables different between library mode and self-hosted mode? diff --git a/docs/docs/extraction/integrations-langchain-llamaindex-haystack.md b/docs/docs/extraction/integrations-langchain-llamaindex-haystack.md new file mode 100644 index 0000000000..7ee0dda650 --- /dev/null +++ b/docs/docs/extraction/integrations-langchain-llamaindex-haystack.md @@ -0,0 +1,22 @@ +# Integrate with LangChain, LlamaIndex, and Haystack + +NeMo Retriever Library is commonly used **behind** retrieval-augmented generation (RAG) apps built with popular orchestration frameworks. + +## Jupyter examples (LangChain and LlamaIndex) + +The repository includes notebooks that demonstrate multimodal RAG patterns: + +- [Multimodal RAG with LangChain](https://github.com/NVIDIA/NeMo-Retriever/blob/main/examples/langchain_multimodal_rag.ipynb) +- [Multimodal RAG with LlamaIndex](https://github.com/NVIDIA/NeMo-Retriever/blob/main/examples/llama_index_multimodal_rag.ipynb) + +These are also linked from [Jupyter Notebooks](notebooks/index.md) and the [FAQ](faq.md). + +## Haystack + +Haystack-related extraction modes may appear in API tables as **deprecated** in favor of current pipeline options. For up-to-date integration patterns, prefer the Python API and CLI docs, and check [Release notes](releasenotes.md) for migration notes. + +## Related + +- [Use the Python API](nemo-retriever-api-reference.md) +- [Use the CLI](https://github.com/NVIDIA/NeMo-Retriever/tree/main/nemo_retriever/docs/cli) +- [Chunking](concepts.md#chunking), [Upload data](vdbs.md), [Filter search](custom-metadata.md) diff --git a/docs/docs/extraction/multimodal-extraction.md b/docs/docs/extraction/multimodal-extraction.md index f44aee36cf..c2f5581a90 100644 --- a/docs/docs/extraction/multimodal-extraction.md +++ b/docs/docs/extraction/multimodal-extraction.md @@ -74,7 +74,7 @@ For natural-language infographic descriptions, optionally enable [image captioni Scanned PDFs and image-only pages rely on OCR and hybrid paths that combine native text extraction with OCR when needed. For extract methods such as `ocr` and `pdfium_hybrid`, refer to the [Python API reference](nemo-retriever-api-reference.md). -When you run extraction locally with Hugging Face weights, the default OCR engine is **Nemotron OCR v2**, which operates in **multilingual** mode by default. For CLI flags and API parameters, refer to [Nemotron OCR v2 — language mode](https://github.com/NVIDIA/NeMo-Retriever/blob/main/nemo_retriever/docs/cli/README.md#nemotron-ocr-v2-language-mode). For Kubernetes deployment, refer to [OCR NIM configuration](https://github.com/NVIDIA/NeMo-Retriever/blob/main/nemo_retriever/helm/README.md#ocr-nim-configuration) in the Helm chart README. +OCR artifacts depend on how you deploy. **Helm / NIM:** the production chart uses **Nemotron OCR v1** (`nvcr.io/nim/nvidia/nemotron-ocr-v1:1.3.0`). **Local Hugging Face inference:** the default engine is **Nemotron OCR v2**, which operates in **multilingual** mode by default. For CLI flags and API parameters, see [Nemotron OCR v2 — language mode](https://github.com/NVIDIA/NeMo-Retriever/blob/main/nemo_retriever/docs/cli/README.md#nemotron-ocr-v2-language-mode). For Kubernetes defaults and the Helm-vs-local split, see [OCR artifacts (Helm vs local Hugging Face)](prerequisites-support-matrix.md#nemotron-ocr-v2-language-mode) in the support matrix. **Related** @@ -94,7 +94,7 @@ Chart-classified PDF regions stay on the layout/OCR path; only non-chart image r - [Multimodal embeddings (VLM)](embedding.md) - [Metadata reference](content-metadata.md) -- [Image captioning](prerequisites-support-matrix.md#image-captioning) +- [Image captioning](prerequisites-support-matrix.md#image-captioning-2605) ## Metadata and content schema { #metadata-and-content-schema } diff --git a/docs/docs/extraction/nemo-retriever-api-reference.md b/docs/docs/extraction/nemo-retriever-api-reference.md index 2841b9799f..da21b30a40 100644 --- a/docs/docs/extraction/nemo-retriever-api-reference.md +++ b/docs/docs/extraction/nemo-retriever-api-reference.md @@ -1,10 +1,10 @@ # NeMo Retriever API Reference -## PDF pre-splitting for parallel ingest { #pdf-pre-splitting-for-parallel-ingest } +## PDF pre-splitting for parallel ingest Large PDFs are split into page batches before Ray processing so extraction can run in parallel. This happens on the default ingest path; you do not need extra configuration for typical workloads. -To tune splitter throughput from the CLI, use `--pdf-split-batch-size` (Ray actor batch size for the splitter stage). Refer to [Text chunking and PDF page batches](https://github.com/NVIDIA/NeMo-Retriever/tree/main/nemo_retriever/docs/cli#text-chunking-and-pdf-page-batches) in the CLI reference. +To tune splitter throughput from the CLI, use `--pdf-split-batch-size` (Ray actor batch size for the splitter stage). See [Text chunking and PDF page batches](https://github.com/NVIDIA/NeMo-Retriever/tree/main/nemo_retriever/docs/cli#text-chunking-and-pdf-page-batches) in the CLI reference. **Python client (`pdf_split_config`):** Only `create_ingestor(run_mode="service")` implements `.pdf_split_config(pages_per_chunk=...)`, which records page-chunking settings in the request pipeline spec for the remote gateway. Local graph ingest (`run_mode="inprocess"` or `"batch"`) raises `NotImplementedError` if you call this method; PDFs are split automatically on the default ingest path without client-side configuration. @@ -13,6 +13,6 @@ To tune splitter throughput from the CLI, use `--pdf-split-batch-size` (Ray acto filters: - "!^pdf_split_config$" -::: nemo_retriever.graph.retriever +::: nemo_retriever.retriever -::: nemo_retriever.common.params +::: nemo_retriever.params diff --git a/docs/docs/extraction/nimclient.md b/docs/docs/extraction/nimclient.md new file mode 100644 index 0000000000..755db2b366 --- /dev/null +++ b/docs/docs/extraction/nimclient.md @@ -0,0 +1,588 @@ +# NimClient Usage Guide for NeMo Retriever Library + +The `NimClient` class provides a unified interface for connecting to and interacting with NVIDIA NIM Microservices. +This documentation demonstrates how to create custom NIM integrations for use in [NeMo Retriever Library](overview.md) pipelines and User Defined Functions (UDFs). + +The NimClient architecture consists of two main components: + +1. **NimClient**: The client class that handles communication with NIM endpoints via gRPC or HTTP protocols +2. **ModelInterface**: An abstract base class that defines how to format input data, parse output responses, and process inference results for specific models + +For advanced usage patterns, refer to the existing model interfaces in `nemo_retriever/src/nemo_retriever/api/internal/primitives/nim/model_interface/`. + + +## Quick Start + +For ingest and pipeline APIs used with NimClient in UDFs, refer to the [Python API guide](nemo-retriever-api-reference.md). + +### Basic NimClient Creation + +```python +from nemo_retriever.api.util.nim import create_inference_client +from nemo_retriever.api.internal.primitives.nim import ModelInterface + +# Create a custom model interface (refer to examples below) +model_interface = MyCustomModelInterface() + +# Define endpoints (gRPC, HTTP) +endpoints = ("grpc://my-nim-service:8001", "http://my-nim-service:8000") + +# Create the client +client = create_inference_client( + endpoints=endpoints, + model_interface=model_interface, + auth_token="your-ngc-api-key", # Optional + infer_protocol="grpc", # Optional: "grpc" or "http" + timeout=120.0, # Optional: request timeout + max_retries=5 # Optional: retry attempts +) + +# Perform inference +data = {"input": "your input data"} +results = client.infer(data, model_name="your-model-name") +``` + +### Using Environment Variables + +```python +import os +from nemo_retriever.api.util.nim import create_inference_client + +# Use environment variables for configuration +auth_token = os.getenv("NGC_API_KEY") +grpc_endpoint = os.getenv("NIM_GRPC_ENDPOINT", "grpc://localhost:8001") +http_endpoint = os.getenv("NIM_HTTP_ENDPOINT", "http://localhost:8000") + +client = create_inference_client( + endpoints=(grpc_endpoint, http_endpoint), + model_interface=model_interface, + auth_token=auth_token +) +``` + +## Creating Custom Model Interfaces + +To integrate a new NIM, you need to create a custom `ModelInterface` subclass that implements the required methods. + +### Basic Model Interface Template + +```python +from typing import Dict, Any, List, Tuple, Optional +import numpy as np +from nemo_retriever.api.internal.primitives.nim import ModelInterface + +class MyCustomModelInterface(ModelInterface): + """ + Custom model interface for My Custom NIM. + """ + + def __init__(self, model_name: str = "my-custom-model"): + """Initialize the model interface.""" + self.model_name = model_name + + def name(self) -> str: + """Return the name of this model interface.""" + return "MyCustomModel" + + def prepare_data_for_inference(self, data: Dict[str, Any]) -> Dict[str, Any]: + """ + Prepare and validate input data before formatting. + + Parameters + ---------- + data : dict + Raw input data + + Returns + ------- + dict + Validated and prepared data + """ + # Validate required fields + if "input_text" not in data: + raise KeyError("Input data must include 'input_text'") + + # Ensure input is in the expected format + if not isinstance(data["input_text"], str): + raise ValueError("input_text must be a string") + + return data + + def format_input( + self, + data: Dict[str, Any], + protocol: str, + max_batch_size: int, + **kwargs + ) -> Tuple[List[Any], List[Dict[str, Any]]]: + """ + Format input data for the specified protocol. + + Parameters + ---------- + data : dict + Prepared input data + protocol : str + Communication protocol ("grpc" or "http") + max_batch_size : int + Maximum batch size for processing + **kwargs : dict + Additional parameters + + Returns + ------- + tuple + (formatted_batches, batch_data_list) + """ + if protocol == "http": + return self._format_http_input(data, max_batch_size, **kwargs) + elif protocol == "grpc": + return self._format_grpc_input(data, max_batch_size, **kwargs) + else: + raise ValueError("Invalid protocol. Must be 'grpc' or 'http'") + + def _format_http_input( + self, + data: Dict[str, Any], + max_batch_size: int, + **kwargs + ) -> Tuple[List[Dict[str, Any]], List[Dict[str, Any]]]: + """Format input for HTTP protocol.""" + input_text = data["input_text"] + + # Create HTTP payload + payload = { + "model": kwargs.get("model_name", self.model_name), + "input": input_text, + "max_tokens": kwargs.get("max_tokens", 512), + "temperature": kwargs.get("temperature", 0.7), + } + + # Return as single batch + return [payload], [{"original_input": input_text}] + + def _format_grpc_input( + self, + data: Dict[str, Any], + max_batch_size: int, + **kwargs + ) -> Tuple[List[np.ndarray], List[Dict[str, Any]]]: + """Format input for gRPC protocol.""" + input_text = data["input_text"] + + # Convert to numpy array for gRPC + text_array = np.array([[input_text.encode("utf-8")]], dtype=np.object_) + + return [text_array], [{"original_input": input_text}] + + def parse_output( + self, + response: Any, + protocol: str, + data: Optional[Dict[str, Any]] = None, + **kwargs + ) -> Any: + """ + Parse the raw model response. + + Parameters + ---------- + response : Any + Raw response from the model + protocol : str + Communication protocol used + data : dict, optional + Original batch data + **kwargs : dict + Additional parameters + + Returns + ------- + Any + Parsed response data + """ + if protocol == "http": + return self._parse_http_response(response) + elif protocol == "grpc": + return self._parse_grpc_response(response) + else: + raise ValueError("Invalid protocol. Must be 'grpc' or 'http'") + + def _parse_http_response(self, response: Dict[str, Any]) -> str: + """Parse HTTP response.""" + if isinstance(response, dict): + # Extract the generated text from response + if "choices" in response: + return response["choices"][0].get("text", "") + elif "output" in response: + return response["output"] + else: + raise RuntimeError("Unexpected response format") + return str(response) + + def _parse_grpc_response(self, response: np.ndarray) -> str: + """Parse gRPC response.""" + if isinstance(response, np.ndarray): + # Decode bytes response + return response.flatten()[0].decode("utf-8") + return str(response) + + def process_inference_results( + self, + output: Any, + protocol: str, + **kwargs + ) -> Any: + """ + Post-process the parsed inference results. + + Parameters + ---------- + output : Any + Parsed output from parse_output + protocol : str + Communication protocol used + **kwargs : dict + Additional parameters + + Returns + ------- + Any + Final processed results + """ + # Apply any final processing (e.g., filtering, formatting) + if isinstance(output, str): + return output.strip() + return output +``` + +## Real-World Examples + +### Text Generation Model Interface + +```python +class TextGenerationModelInterface(ModelInterface): + """Interface for text generation NIMs (e.g., LLaMA, GPT-style models).""" + + def name(self) -> str: + return "TextGeneration" + + def prepare_data_for_inference(self, data: Dict[str, Any]) -> Dict[str, Any]: + if "prompt" not in data: + raise KeyError("Input data must include 'prompt'") + return data + + def format_input(self, data: Dict[str, Any], protocol: str, max_batch_size: int, **kwargs): + prompt = data["prompt"] + + if protocol == "http": + payload = { + "model": kwargs.get("model_name", "llama-2-7b-chat"), + "messages": [{"role": "user", "content": prompt}], + "max_tokens": kwargs.get("max_tokens", 512), + "temperature": kwargs.get("temperature", 0.7), + "top_p": kwargs.get("top_p", 0.9), + "stream": False + } + return [payload], [{"prompt": prompt}] + else: + raise ValueError("Only HTTP protocol supported for this model") + + def parse_output(self, response: Any, protocol: str, data: Optional[Dict[str, Any]] = None, **kwargs): + if protocol == "http" and isinstance(response, dict): + choices = response.get("choices", []) + if choices: + return choices[0].get("message", {}).get("content", "") + return str(response) + + def process_inference_results(self, output: Any, protocol: str, **kwargs): + return output.strip() if isinstance(output, str) else output +``` + +### Image Analysis Model Interface + +```python +import base64 +from nemo_retriever.api.util.image_processing.transforms import numpy_to_base64 + +class ImageAnalysisModelInterface(ModelInterface): + """Interface for image analysis NIMs (e.g., vision models).""" + + def name(self) -> str: + return "ImageAnalysis" + + def prepare_data_for_inference(self, data: Dict[str, Any]) -> Dict[str, Any]: + if "images" not in data: + raise KeyError("Input data must include 'images'") + + # Ensure images is a list + if not isinstance(data["images"], list): + data["images"] = [data["images"]] + + return data + + def format_input(self, data: Dict[str, Any], protocol: str, max_batch_size: int, **kwargs): + images = data["images"] + prompt = data.get("prompt", "Describe this image.") + + # Convert images to base64 if needed + base64_images = [] + for img in images: + if isinstance(img, np.ndarray): + base64_images.append(numpy_to_base64(img)) + elif isinstance(img, str) and img.startswith("data:image"): + # Already base64 encoded + base64_images.append(img.split(",")[1]) + else: + base64_images.append(str(img)) + + # Batch images + batches = [base64_images[i:i + max_batch_size] + for i in range(0, len(base64_images), max_batch_size)] + + payloads = [] + batch_data_list = [] + + for batch in batches: + if protocol == "http": + messages = [] + for img_b64 in batch: + messages.append({ + "role": "user", + "content": f'{prompt} ' + }) + + payload = { + "model": kwargs.get("model_name", "llava-1.5-7b-hf"), + "messages": messages, + "max_tokens": kwargs.get("max_tokens", 512), + "temperature": kwargs.get("temperature", 0.1) + } + payloads.append(payload) + batch_data_list.append({"images": batch, "prompt": prompt}) + + return payloads, batch_data_list + + def parse_output(self, response: Any, protocol: str, data: Optional[Dict[str, Any]] = None, **kwargs): + if protocol == "http" and isinstance(response, dict): + choices = response.get("choices", []) + return [choice.get("message", {}).get("content", "") for choice in choices] + return [str(response)] + + def process_inference_results(self, output: Any, protocol: str, **kwargs): + if isinstance(output, list): + return [result.strip() for result in output] + return output +``` + +## Using NimClient in UDFs + +### Basic UDF with NimClient + +```python +from nemo_retriever.api.internal.primitives.ingest_control_message import IngestControlMessage +from nemo_retriever.api.util.nim import create_inference_client +import os + +def analyze_document_with_nim(control_message: IngestControlMessage) -> IngestControlMessage: + """UDF that uses a custom NIM to analyze document content.""" + + # Create NIM client + model_interface = TextGenerationModelInterface() + client = create_inference_client( + endpoints=( + os.getenv("ANALYSIS_NIM_GRPC", "grpc://analysis-nim:8001"), + os.getenv("ANALYSIS_NIM_HTTP", "http://analysis-nim:8000") + ), + model_interface=model_interface, + auth_token=os.getenv("NGC_API_KEY"), + infer_protocol="http" + ) + + # Get the document DataFrame + df = control_message.get_payload() + + # Process each document + for idx, row in df.iterrows(): + if row.get("content"): + # Prepare analysis prompt + prompt = f"Analyze the following document content and provide a summary: {row['content'][:1000]}" + + # Perform inference + try: + results = client.infer( + data={"prompt": prompt}, + model_name="llama-2-7b-chat", + max_tokens=256, + temperature=0.3 + ) + + # Add analysis to metadata + if results: + analysis = results[0] if isinstance(results, list) else results + df.at[idx, "custom_analysis"] = analysis + + except Exception as e: + print(f"NIM inference failed: {e}") + df.at[idx, "custom_analysis"] = "Analysis failed" + + # Update the control message with processed data + control_message.payload(df) + return control_message +``` + +### Advanced UDF with Batching + +```python +def batch_image_analysis_udf(control_message: IngestControlMessage) -> IngestControlMessage: + """UDF that performs batched image analysis using NIM.""" + + # Create image analysis client + model_interface = ImageAnalysisModelInterface() + client = create_inference_client( + endpoints=( + os.getenv("VISION_NIM_GRPC", "grpc://vision-nim:8001"), + os.getenv("VISION_NIM_HTTP", "http://vision-nim:8000") + ), + model_interface=model_interface, + auth_token=os.getenv("NGC_API_KEY") + ) + + df = control_message.get_payload() + + # Collect all images for batch processing + image_rows = [] + images = [] + + for idx, row in df.iterrows(): + if "image_data" in row and row["image_data"]: + image_rows.append(idx) + images.append(row["image_data"]) + + if images: + try: + # Batch process all images + results = client.infer( + data={ + "images": images, + "prompt": "Describe the content and key elements in this image." + }, + model_name="llava-1.5-7b-hf", + max_tokens=200 + ) + + # Apply results back to DataFrame + for idx, result in zip(image_rows, results): + df.at[idx, "image_description"] = result + + except Exception as e: + print(f"Batch image analysis failed: {e}") + for idx in image_rows: + df.at[idx, "image_description"] = "Analysis failed" + + control_message.payload(df) + return control_message +``` + +## Configuration and Best Practices + +### Environment Variables + +Set these environment variables for your NIM endpoints: + +```bash +# NIM endpoints +export MY_NIM_GRPC_ENDPOINT="grpc://my-nim-service:8001" +export MY_NIM_HTTP_ENDPOINT="http://my-nim-service:8000" + +# Authentication +export NGC_API_KEY="your-ngc-api-key" + +# Optional: timeouts and retries +export NIM_TIMEOUT=120 +export NIM_MAX_RETRIES=5 +``` + +### Performance Optimization + +1. **Use gRPC when possible**: Generally faster than HTTP for high-throughput scenarios +2. **Batch processing**: Process multiple items together to reduce overhead +3. **Connection reuse**: Create NimClient instances once and reuse them +4. **Appropriate timeouts**: Set reasonable timeouts based on your model's response time +5. **Error handling**: Always handle inference failures gracefully + +### Error Handling + +```python +def robust_nim_udf(control_message: IngestControlMessage) -> IngestControlMessage: + """UDF with comprehensive error handling.""" + + try: + client = create_inference_client( + endpoints=(grpc_endpoint, http_endpoint), + model_interface=model_interface, + auth_token=auth_token, + timeout=60.0, + max_retries=3 + ) + except Exception as e: + print(f"Failed to create NIM client: {e}") + return control_message + + df = control_message.get_payload() + + for idx, row in df.iterrows(): + try: + results = client.infer(data=input_data, model_name="my-model") + df.at[idx, "nim_result"] = results + except TimeoutError: + print(f"NIM request timed out for row {idx}") + df.at[idx, "nim_result"] = "timeout" + except Exception as e: + print(f"NIM inference failed for row {idx}: {e}") + df.at[idx, "nim_result"] = "error" + + control_message.payload(df) + return control_message +``` + +## Troubleshooting + +### Common Issues + +* **Connection Errors** – Verify NIM service is running and endpoints are correct +* **Authentication Failures** – Check NGC_API_KEY is valid and properly set +* **Timeout Errors** – Increase timeout values or check NIM service performance +* **Format Errors** – Ensure your ModelInterface formats data correctly for your NIM +* **Memory Issues** – Use appropriate batch sizes to avoid memory exhaustion + +### NIM Triton Limit Memory + +If you encounter memory issues, try increasing the `NIM_TRITON_CUDA_MEMORY_POOL_MB` parameter. This adjustment typically does not affect performance. + +If memory issues persist, you can reduce the `NIM_TRITON_RATE_LIMIT` value — even down to 1. However, lowering this parameter affects performance. + +### Debugging Tips + +```python +import logging + +# Enable debug logging +logging.getLogger("nemo_retriever.api.internal.primitives.nim").setLevel(logging.DEBUG) + +# Test your model interface separately +model_interface = MyCustomModelInterface() +test_data = {"input": "test"} + +# Test data preparation +prepared = model_interface.prepare_data_for_inference(test_data) +print(f"Prepared data: {prepared}") + +# Test input formatting +formatted, batch_data = model_interface.format_input(prepared, "http", 1) +print(f"Formatted input: {formatted}") +``` + +## Related Topics + +- [Extending/Customizing NeMo Retriever Library with custom code](https://github.com/NVIDIA/NeMo-Retriever/tree/main/nemo_retriever/src/nemo_retriever/graph#nemo-retriever-graph) diff --git a/docs/docs/extraction/prerequisites-support-matrix.md b/docs/docs/extraction/prerequisites-support-matrix.md index 2639059507..1486d5ceb1 100644 --- a/docs/docs/extraction/prerequisites-support-matrix.md +++ b/docs/docs/extraction/prerequisites-support-matrix.md @@ -2,7 +2,7 @@ Before you begin using [NeMo Retriever Library](overview.md), confirm your software stack, deployment hardware, and—if you use them—advanced features (audio and video, Nemotron Parse, VLM image captioning, reranking) against the guidance in this page. -## Software Requirements { #software-requirements } +## Software Requirements - Linux operating systems (Ubuntu 22.04 or later recommended) - [CUDA Toolkit](https://developer.nvidia.com/cuda-downloads) (NVIDIA Driver >= `580`, CUDA >= `13.0`) @@ -11,7 +11,8 @@ Before you begin using [NeMo Retriever Library](overview.md), confirm your softw - For audio and video, `ffmpeg` and `ffprobe` must be on `PATH` (for example `sudo apt-get install -y --no-install-recommends ffmpeg` on Debian/Ubuntu). `ffmpeg-python` and `nemo-retriever[multimedia]` do not install these binaries. - For container and Kubernetes guidance, refer to [Audio and video](audio-video.md). + On Helm with package-repo access, set `service.installFfmpeg=true`. For + air-gapped clusters, see [Air-gapped and disconnected deployment](deployment-options.md#air-gapped-deployment). - For PDF extraction with `extract_method="nemotron_parse"`, install the Nemotron Parse client dependencies with `pip install "nemo-retriever[nemotron-parse]"` (pulls `open-clip-torch`, which provides the `open_clip` module required by the Nemotron Parse @@ -22,7 +23,7 @@ Before you begin using [NeMo Retriever Library](overview.md), confirm your softw When you use UV, create the environment with Python 3.12 — for example, `uv venv --python 3.12`. This matches the `requires-python` metadata in the library packages. -## Hardware Requirements { #hardware-requirements } +## Hardware Requirements The full ingestion pipeline is designed to consume significant CPU and memory resources to achieve maximal parallelism. Resource usage scales up to the limits of your deployed system. @@ -59,13 +60,13 @@ For production deployments processing large volumes of documents, consider: Ensure your deployment environment meets these specifications before running the full pipeline. Resource-constrained environments may experience performance degradation. -## Core and Advanced Pipeline Features { #core-and-advanced-pipeline-features } +## Core and Advanced Pipeline Features The NeMo Retriever Library extraction core pipeline features run on a single A10G or better GPU. -### Default Helm NIMs { #default-helm-nims } +### Default Helm NIMs -The production Helm chart enables these NIM microservices **by default** (for example through `nimOperator.*.enabled=true`): +The production Helm chart enables these NIM microservices **by default** (for example via `nimOperator.*.enabled=true`): | Helm flag | NIM | Role | |-----------|-----|------| @@ -78,9 +79,9 @@ The production Helm chart enables these NIM microservices **by default** (for ex !!! note - **Helm / NIM:** The production chart deploys **Nemotron OCR v1** under `nimOperator.ocr` (`nvcr.io/nim/nvidia/nemotron-ocr-v1:1.3.0`). For image defaults and upgrade notes, refer to [OCR NIM configuration](https://github.com/NVIDIA/NeMo-Retriever/blob/26.05/nemo_retriever/helm/README.md#ocr-nim-configuration) in the Helm chart README. + **Helm / NIM:** The production chart deploys **Nemotron OCR v1** under `nimOperator.ocr` (`nvcr.io/nim/nvidia/nemotron-ocr-v1:1.3.0`). For image defaults and upgrade notes, see [OCR NIM configuration](https://github.com/NVIDIA/NeMo-Retriever/blob/26.05/nemo_retriever/helm/README.md#ocr-nim-configuration) in the Helm chart README. - **Local Hugging Face inference:** When you deploy locally with HuggingFace model weights (for example `pip install "nemo-retriever[local]"` and GPU inference without remote OCR NIM URLs), the default OCR engine is **Nemotron OCR v2**, which runs in **multilingual** mode by default. For CLI flags and API parameters, refer to [Nemotron OCR v2 — language mode](https://github.com/NVIDIA/NeMo-Retriever/blob/main/nemo_retriever/docs/cli/README.md#nemotron-ocr-v2-language-mode). Remote OCR NIM endpoints use their own model and language behavior; local OCR language selectors are not sent on remote requests. + **Local Hugging Face inference:** When you deploy locally with HuggingFace model weights (for example `pip install "nemo-retriever[local]"` and GPU inference without remote OCR NIM URLs), the default OCR engine is **Nemotron OCR v2**, which runs in **multilingual** mode by default. For CLI flags and API parameters, see [Nemotron OCR v2 — language mode](https://github.com/NVIDIA/NeMo-Retriever/blob/main/nemo_retriever/docs/cli/README.md#nemotron-ocr-v2-language-mode). Remote OCR NIM endpoints use their own model and language behavior; local OCR language selectors are not sent on remote requests. Default OCR NIM container for release Helm deployments: @@ -93,33 +94,44 @@ Default VL embedder container and model for release deployments: ### Optional Helm NIMs (not auto-wired) { #optional-helm-nims-not-auto-wired-by-default } -These NIM microservices are **optional** for the default extraction pipeline. The retriever service does **not** call them until you enable the matching pipeline stage (reranker, Nemotron Parse, caption, or audio). In production, disable keys you do not need (refer to [Recommended minimal install](https://github.com/NVIDIA/NeMo-Retriever/blob/main/nemo_retriever/helm/README.md#recommended-minimal-install-2605)). Set `nimOperator..enabled=true` when you want that NIM reconciled. Chart keys are in the [NeMo Retriever Helm chart README](https://github.com/NVIDIA/NeMo-Retriever/blob/main/nemo_retriever/helm/README.md#nim-operator-sub-stack). +These NIM microservices are **optional** for the default extraction pipeline. The retriever service does **not** call them until you enable the matching pipeline stage (reranker, Nemotron Parse, caption, or audio). For **26.05 production**, disable keys you do not need (see [Recommended minimal install (26.05)](https://github.com/NVIDIA/NeMo-Retriever/blob/26.05/nemo_retriever/helm/README.md#recommended-minimal-install-2605)). Set `nimOperator..enabled=true` when you want that NIM reconciled. Chart keys are in the [NeMo Retriever Helm chart README](https://github.com/NVIDIA/NeMo-Retriever/blob/26.05/nemo_retriever/helm/README.md#nim-operator-sub-stack). | Helm flag | NIM | Role | |-----------|-----|------| | `rerankqa` | [llama-nemotron-rerank-vl-1b-v2](https://huggingface.co/nvidia/llama-nemotron-rerank-vl-1b-v2) | Reranking for improved retrieval accuracy | | `nemotron_parse` | [nemotron-parse](https://huggingface.co/nvidia/NVIDIA-Nemotron-Parse-v1.2) | Optional PDF `extract_method="nemotron_parse"` (default PDF extraction uses **pdfium**) | -| `nemotron_3_nano_omni_30b_a3b_reasoning` | [nemotron-3-nano-omni-30b-a3b-reasoning](https://huggingface.co/nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16) | Supported image captioning when you enable the caption stage | +| `nemotron_3_nano_omni_30b_a3b_reasoning` | [nemotron-3-nano-omni-30b-a3b-reasoning](https://huggingface.co/nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16) | Supported image captioning for 26.05 when you enable the caption stage | | `audio` | [parakeet-1-1b-ctc-en-us](https://huggingface.co/nvidia/parakeet-ctc-1.1b) | [Audio and video](audio-video.md) transcription | -### Image captioning { #image-captioning } +### Image captioning (26.05) { #image-captioning-2605 } -Use **`nemotron_3_nano_omni_30b_a3b_reasoning`** when you enable the caption stage (hosted model ID `nvidia/nemotron-3-nano-omni-30b-a3b-reasoning`). The Helm key is in the [optional NIMs](#optional-helm-nims-not-auto-wired-by-default) table above. +For 26.05, use **`nemotron_3_nano_omni_30b_a3b_reasoning`** when you enable the caption stage (hosted model ID `nvidia/nemotron-3-nano-omni-30b-a3b-reasoning`). The Helm key is in the [optional NIMs](#optional-helm-nims-not-auto-wired-by-default) table above. + +!!! important "PDF chart regions are not captioned by Omni" + + When **nemotron-page-elements-v3** classifies a PDF region as **chart**, that region is processed through layout detection and OCR—not the Omni caption stage. Enabling the caption NIM and the `caption` pipeline stage does **not** send chart-labeled figures to `/v1/chat/completions`. + + The caption stage covers: + + - Unstructured content in the `images` column (standalone image files and page-element regions **not** classified as table, chart, or infographic) + - Optional infographic regions when you set `caption_infographics=True` on `CaptionParams` (the VLM caption is stored in `caption`, separate from OCR `text`) + + To validate caption traffic during ingest, inspect metadata such as `page_elements_v3_counts_by_label`. If the figure is labeled `chart`, expect no Omni chat-completions requests for that region even when captioning is enabled. Optional features listed in the table above require additional GPU support, disk space, and feature-specific system dependencies beyond the four default NIMs. For published NIM model IDs and deployment-specific constraints, use the product support matrices linked under [Related Topics](#related-topics) below. -## Model Hardware Requirements { #model-hardware-requirements } +## Model Hardware Requirements NeMo Retriever Library supports the following GPU hardware given system constraints in the table. - **HF model weights** — approximate Hugging Face checkpoint footprint (files such as `model*.safetensors`, `weights.pth`, or other published weight bundles in the model repository). Values are rounded from the current public file listing and can change when the repository is updated. -- **NIM disk space** — approximate container and on-disk model cache for self-hosted NIM microservices (not the same as HF download size). For Nemotron 3 Nano Omni captioning, refer to the [NVIDIA NIM for Vision Language Models support matrix](https://docs.nvidia.com/nim/vision-language-models/latest/support-matrix.html#nemotron-3-nano-omni-30b-a3b-reasoning). +- **NIM disk space** — approximate container and on-disk model cache for self-hosted NIM microservices (not the same as HF download size). For Nemotron 3 Nano Omni captioning, see the [NVIDIA NIM for Vision Language Models support matrix](https://docs.nvidia.com/nim/vision-language-models/latest/support-matrix.html#nemotron-3-nano-omni-30b-a3b-reasoning). Model repositories and NIM references are linked in [Core and Advanced Pipeline Features](#core-and-advanced-pipeline-features) above. -**B200, H200 NVL, and audio/video extraction:** The [audio and video](audio-video.md) transcription path (self-hosted Parakeet ASR through `nimOperator.audio`) is **not supported on B200**, other Blackwell GPUs, or **H200 NVL**. Core PDF and multimodal extraction on those GPUs is unchanged. Refer to footnote ⁴ below. +**B200, H200 NVL, and audio/video extraction:** The [audio and video](audio-video.md) transcription path (self-hosted Parakeet ASR via `nimOperator.audio`) is **not supported on B200**, other Blackwell GPUs, or **H200 NVL**. Core PDF and multimodal extraction on those GPUs is unchanged. See footnote ⁴ below. | Feature | HF Model Weights | GPU Option | [RTX Pro 6000](https://www.nvidia.com/en-us/data-center/rtx-pro-6000-blackwell-server-edition/) | [B200](https://www.nvidia.com/en-us/data-center/dgx-b200/) | [H200 NVL](https://www.nvidia.com/en-us/data-center/h200/) | [H100](https://www.nvidia.com/en-us/data-center/h100/) | [A100 80GB](https://www.nvidia.com/en-us/data-center/a100/) | A100 40GB | [A10G](https://aws.amazon.com/ec2/instance-types/g5/) | L40S | [RTX PRO 4500 Blackwell](https://www.nvidia.com/en-us/products/workstations/professional-desktop-gpus/rtx-pro-4500/) | |---------|------------------|------------|--------|--------|--------|--------|--------|--------|--------|--------|------------------------| @@ -138,15 +150,15 @@ Model repositories and NIM references are linked in [Core and Advanced Pipeline ¹ On other supported GPUs, Parakeet ASR (`parakeet-1-1b-ctc-en-us:1.5.0`) may require a runtime TensorRT engine build (no prebuilt profile in the chart image). -⁴ Self-hosted [audio/video extraction](audio-video.md) through Parakeet ASR (`parakeet-1-1b-ctc-en-us:1.5.0`, `nimOperator.audio`) is **not supported** on **B200**, other **Blackwell** GPUs (compute capability 12.0), including RTX PRO 6000 Blackwell and RTX PRO 4500 Blackwell, or **H200 NVL**. Core PDF and multimodal extraction on those GPUs is unchanged. Video workflows that depend on Parakeet for speech transcription are affected the same way. `NIMService` for `nimOperator.audio` may stay not Ready or enter `CrashLoopBackOff` while building the Riva/TensorRT engine (for example ONNX Runtime IR version, cuDNN visibility, or FP8 tactic errors). Use a supported dedicated GPU (for example H100 or A100), [hosted Parakeet on build.nvidia.com](audio-video.md#parakeet-hosted-inference-build-nvidia), or set `nimOperator.audio.enabled=false`. +⁴ Self-hosted [audio/video extraction](audio-video.md) via Parakeet ASR (`parakeet-1-1b-ctc-en-us:1.5.0`, `nimOperator.audio`) is **not supported** on **B200**, other **Blackwell** GPUs (compute capability 12.0), including RTX PRO 6000 Blackwell and RTX PRO 4500 Blackwell, or **H200 NVL**. Core PDF and multimodal extraction on those GPUs is unchanged. Video workflows that depend on Parakeet for speech transcription are affected the same way. `NIMService` for `nimOperator.audio` may stay not Ready or enter `CrashLoopBackOff` while building the Riva/TensorRT engine (for example ONNX Runtime IR version, cuDNN visibility, or FP8 tactic errors). Use a supported dedicated GPU (for example H100 or A100), [hosted Parakeet on build.nvidia.com](audio-video.md#parakeet-hosted-inference-build-nvidia), or set `nimOperator.audio.enabled=false`. -³ Opt-in Omni captioning uses the [nemotron-3-nano-omni-30b-a3b-reasoning](https://docs.api.nvidia.com/nim/reference/nvidia-nemotron-3-nano-omni-30b-a3b-reasoning) NIM (`nvcr.io/nim/nvidia/nemotron-3-nano-omni-30b-a3b-reasoning:1.7.0-variant`). BF16 requires at least 80 GB total GPU memory; refer to the [VLM NIM support matrix](https://docs.nvidia.com/nim/vision-language-models/latest/support-matrix.html#nemotron-3-nano-omni-30b-a3b-reasoning). L40S requires two GPUs. A100 40GB, A10G, and RTX PRO 4500 are below the minimum. +³ Opt-in Omni captioning uses the [nemotron-3-nano-omni-30b-a3b-reasoning](https://docs.api.nvidia.com/nim/reference/nvidia-nemotron-3-nano-omni-30b-a3b-reasoning) NIM (`nvcr.io/nim/nvidia/nemotron-3-nano-omni-30b-a3b-reasoning:1.7.0-variant`). BF16 requires at least 80 GB total GPU memory; see the [VLM NIM support matrix](https://docs.nvidia.com/nim/vision-language-models/latest/support-matrix.html#nemotron-3-nano-omni-30b-a3b-reasoning). L40S requires two GPUs. A100 40GB, A10G, and RTX PRO 4500 are below the minimum. \* GPUs with less than 80GB VRAM cannot run the reranker concurrently with the core pipeline. To perform recall testing with the reranker on these GPUs, shut down the core pipeline NIM microservices and run only the embedder, reranker, and your vector database. -## Related Topics { #related-topics } +## Related Topics - [Troubleshooting](troubleshoot.md) - [Release Notes](releasenotes.md) diff --git a/docs/docs/extraction/starter-kits.md b/docs/docs/extraction/starter-kits.md deleted file mode 100644 index 08b366a6eb..0000000000 --- a/docs/docs/extraction/starter-kits.md +++ /dev/null @@ -1,24 +0,0 @@ -# Starter Kits for NeMo Retriever Library - -To get started using [NeMo Retriever Library](overview.md), you can try one of the ready-made notebooks that are available. - -## Dataset Downloads for Benchmarking - -If you plan to run benchmarking or evaluation tests, you must download the [Benchmark Datasets (Bo20, Bo767, Bo10k)](https://github.com/NVIDIA/NeMo-Retriever/blob/main/evaluation/digital_corpora_download.ipynb) from Digital Corpora. This is a prerequisite for all benchmarking operations. - -## Getting Started - -To get started with the basics, try one of the following guides or notebooks: - -- [Quickstart: retriever CLI](../reference/retriever-cli-quickstart.md) -- [Workflow: Ingest documents](workflow-document-ingestion.md) -- [Adding Custom Metadata for Filtered Search/Retrieval](https://github.com/NVIDIA/NeMo-Retriever/blob/main/examples/nemo_retriever_retriever_query_metadata_filter.ipynb) — also summarized on [Vector databases — Metadata and filtering](vdbs.md#metadata-and-filtering) - - -For more advanced scenarios, try one of the following notebooks: - -- [Build a Custom Vector Database Operator](https://github.com/NVIDIA/NeMo-Retriever/blob/main/examples/building_vdb_operator.ipynb) -- [Try Enterprise RAG Blueprint](https://build.nvidia.com/nvidia/multimodal-pdf-data-extraction-for-enterprise-rag) -- [Evaluate bo767 retrieval recall accuracy with NeMo Retriever Library](https://github.com/NVIDIA/NeMo-Retriever/blob/main/evaluation/bo767_recall.ipynb) -- [Multimodal RAG with LangChain](https://github.com/NVIDIA/NeMo-Retriever/blob/main/examples/langchain_multimodal_rag.ipynb) -- [Multimodal RAG with LlamaIndex](https://github.com/NVIDIA/NeMo-Retriever/blob/main/examples/llama_index_multimodal_rag.ipynb) diff --git a/docs/docs/extraction/troubleshoot.md b/docs/docs/extraction/troubleshoot.md index 4d0bc9c8d9..54223f0e45 100644 --- a/docs/docs/extraction/troubleshoot.md +++ b/docs/docs/extraction/troubleshoot.md @@ -77,11 +77,15 @@ ulimit -u 10000 ## Out-of-Memory (OOM) Error when Processing Large Datasets When you process a very large dataset with thousands of documents, you might encounter an Out-of-Memory (OOM) error. -This happens because, by default, NeMo Retriever Library stores the results from every document in system memory (RAM). +This happens because NeMo Retriever Library materializes extraction results in system memory (RAM) while the job runs. If the total size of the results exceeds the available memory, the process fails. -To resolve this issue, use the `save_to_disk` method. -For details, refer to [Working with Large Datasets: Saving to Disk](nemo-retriever-api-reference.md). +To reduce memory pressure, try one or more of the following: + +- Process documents in smaller batches instead of submitting the entire corpus in one job. +- Route outputs to a sink (for example, `.vdb_upload(...)`, `.webhook(...)`, or `.store(...)`) so results are written out instead of held in memory until the job finishes. +- In `run_mode="service"`, pass `return_results=False` to `.ingest(...)` when you do not need the full result payload returned to the client. For parameter details, refer to the [Python API guide](nemo-retriever-api-reference.md). +- Increase available host or pod memory for the ingest workload. diff --git a/docs/docs/extraction/workflow-agentic-retrieval.md b/docs/docs/extraction/workflow-agentic-retrieval.md index d0772c1651..78b48cdc47 100644 --- a/docs/docs/extraction/workflow-agentic-retrieval.md +++ b/docs/docs/extraction/workflow-agentic-retrieval.md @@ -4,12 +4,11 @@ NeMo Retriever Library provides ingestion, embedding, storage, and retrieval building blocks (jobs, chunking, vector stores, reranking) that you orchestrate in application code or frameworks. -Wire agentic loops in your orchestration framework or application code: ingest and index with the [Python API](nemo-retriever-api-reference.md) or [Workflow: Ingest documents](workflow-document-ingestion.md), then expose [semantic retrieval](vdbs.md#semantic-retrieval) and [metadata filtering](vdbs.md#metadata-and-filtering) as agent tools. Refer to [Starter kits](starter-kits.md) for LangChain and LlamaIndex multimodal RAG notebooks. - **Where to go next** Use these pages together with your orchestration layer: -- [Semantic retrieval](vdbs.md#semantic-retrieval), [Metadata and filtering](vdbs.md#metadata-and-filtering), and [Evaluate on your data](evaluate-on-your-data.md) for retrieval quality, reranking, and evaluation guidance +- [Semantic retrieval](vdbs.md#semantic-retrieval), [Custom metadata and filtering](custom-metadata.md), and [Evaluate on your data](evaluate-on-your-data.md) for retrieval quality and reranking notes - [Agentic retrieval (concept)](agentic-retrieval-concept.md) +- [Evaluate on your data](evaluate-on-your-data.md), which includes retrieval evaluation guidance - [Release notes](releasenotes.md), which may mention agentic retrieval updates diff --git a/docs/docs/extraction/workflow-document-ingestion.md b/docs/docs/extraction/workflow-document-ingestion.md index cba0164d63..853de9e1a0 100644 --- a/docs/docs/extraction/workflow-document-ingestion.md +++ b/docs/docs/extraction/workflow-document-ingestion.md @@ -2,7 +2,7 @@ This page covers extracting content from documents and turning that content into a searchable vector collection in one place so you can scroll and search (for example with Ctrl+F) instead of jumping across multiple short workflow stubs. -## Ingest and extract { #ingest-and-extract } +## Ingest and extract Document ingestion is the step where NeMo Retriever Library reads your files (PDFs, Office documents, images, and other [supported formats](multimodal-extraction.md#supported-file-types-and-formats)), runs extraction and optional enrichment, and returns structured content you can embed and index. @@ -14,9 +14,9 @@ Follow these steps: Pipeline concepts and stage overview appear in [Key concepts](concepts.md). Default chunking behavior is summarized under [Chunking](concepts.md#chunking). -`create_ingestor(...)` returns a `GraphIngestor`, which chains `.extract()`, `.embed()`, and `.vdb_upload()` into one graph. The Python example below stops after `.embed()` so you can inspect chunks first; append `.vdb_upload(vdb_op="lancedb", vdb_kwargs={...})` before `.ingest()` to write directly to LanceDB (refer to [Vector databases](vdbs.md)). +`create_ingestor(...)` returns a `GraphIngestor`, which chains `.extract()`, `.embed()`, and `.vdb_upload()` into one graph. The Python example below stops after `.embed()` so you can inspect chunks first; append `.vdb_upload(vdb_op="lancedb", vdb_kwargs={...})` before `.ingest()` to write directly to LanceDB (refer to [Vector databases](vdbs.md)). For directory-scale corpus ingest, the `graph_pipeline` CLI below is the canonical path used in evaluation workflows. -## Choose how you call the library { #choose-how-you-call-the-library } +## Choose how you call the library The following examples match the [NeMo Retriever Library README](https://github.com/NVIDIA/NeMo-Retriever/blob/main/nemo_retriever/README.md). They assume a checkout of the [NeMo Retriever](https://github.com/NVIDIA/NeMo-Retriever) repository and the `batch` run mode with local GPU inference unless you configure remote NIMs. @@ -47,4 +47,20 @@ result = ingestor.ingest() # ``pandas.DataFrame`` (``batch`` and ``inprocess``) Run the above with your working directory at the repository root (so `data/multimodal_test.pdf` resolves), or adjust `documents` to the absolute path of the test PDF. +### Ingest a test corpus (CLI) + +`graph_pipeline` is the canonical ingestion script used throughout the [QA evaluation guide](https://github.com/NVIDIA/NeMo-Retriever/blob/main/nemo_retriever/src/nemo_retriever/evaluation/README.md#step-1-ingest-and-embed-pdfs-nemo-retriever). Point it at a **directory** of PDFs to produce a ready-to-query LanceDB table. + +!!! note "Corpus size and LanceDB indexing" + + LanceDB's default IVF index needs enough chunks to train its partitions (often on the order of tens of chunks). A single small PDF can be insufficient; use a directory with enough documents for your index settings. Replace `/your-example-dir` with your corpus path. + +```bash +python -m nemo_retriever.examples.graph_pipeline \ + /your-example-dir \ + --vdb-kwargs-json '{"uri":"lancedb","table_name":"nemo-retriever"}' +``` + +For build.nvidia.com hosted inference, set [`NVIDIA_API_KEY`](api-keys.md#nvidia-api-key) and pass the `--*-invoke-url` / `--embed-invoke-url` options shown in the [README remote inference section](https://github.com/NVIDIA/NeMo-Retriever/blob/main/nemo_retriever/README.md#ingest-a-test-corpus-cli). + **Next:** [Semantic retrieval](vdbs.md#semantic-retrieval) when serving queries (also refer to [Evaluate on your data](evaluate-on-your-data.md) for reranking and quality checks). diff --git a/docs/docs/extraction/workflow-e2e-blueprints.md b/docs/docs/extraction/workflow-e2e-blueprints.md index 93203a6374..16aa4bb3d6 100644 --- a/docs/docs/extraction/workflow-e2e-blueprints.md +++ b/docs/docs/extraction/workflow-e2e-blueprints.md @@ -5,4 +5,4 @@ Use these external resources for end-to-end RAG implementations with NeMo Retrie - [Enterprise RAG - multimodal PDF data extraction](https://build.nvidia.com/nvidia/multimodal-pdf-data-extraction-for-enterprise-rag) - [NVIDIA AI Blueprints catalog](https://build.nvidia.com/explore/discover) -For framework-specific integration patterns, refer to [Starter kits](starter-kits.md). +For framework-specific integration patterns, see [Framework integrations](integrations-langchain-llamaindex-haystack.md). diff --git a/docs/mkdocs.yml b/docs/mkdocs.yml index 57a3ea6d78..a944ad6c8b 100644 --- a/docs/mkdocs.yml +++ b/docs/mkdocs.yml @@ -84,9 +84,8 @@ nav: - "Authentication and API keys": extraction/api-keys.md - "3. Deployment options": - "Compare deployment options": extraction/deployment-options.md - - "OpenShift deployment (Helm)": https://github.com/NVIDIA/NeMo-Retriever/blob/main/nemo_retriever/helm/openshift.md - "Helm chart (Kubernetes)": https://github.com/NVIDIA/NeMo-Retriever/tree/main/nemo_retriever/helm - - "Docker service image": https://github.com/NVIDIA/NeMo-Retriever/blob/main/nemo_retriever/docker.md + - "Docker Compose (unsupported, developer)": https://github.com/NVIDIA/NeMo-Retriever/blob/main/nemo_retriever/docker.md - "4. Core workflows": - "Workflow: Ingest documents into a searchable VDB collection": extraction/workflow-document-ingestion.md - "Workflow: Audio & video ingestion": extraction/audio-video.md @@ -96,22 +95,26 @@ nav: # Single vector-DB page (vdbs.md). Deep links: in-page "On this page" TOC and redirects # (for example extraction/vector-db-partners.md → vdbs.md#vector-database-partners). - "Vector databases": extraction/vdbs.md - - "7. Deployment & operations": + - "7. Retrieval & ranking": + - "Custom metadata and filtering": extraction/custom-metadata.md + - "8. Deployment & operations": - "Ray and distributed ingest": extraction/ray-logging.md - - "8. Customize & extend": - - "Customize & extend": extraction/customize-extend.md - - "9. Integrations & ecosystem": - - "Starter kits": extraction/starter-kits.md - - "10. Evaluation & benchmarks": + - "9. Customize & extend": + - Extending/Customizing NeMo Retriever Library with custom code: https://github.com/NVIDIA/NeMo-Retriever/tree/main/nemo_retriever/src/nemo_retriever/graph#nemo-retriever-graph + - "NimClient and custom NIM endpoints": extraction/nimclient.md + - "10. Integrations & ecosystem": + - "Framework integrations": extraction/integrations-langchain-llamaindex-haystack.md + - "Starter kits": extraction/notebooks/index.md + - "11. Evaluation & benchmarks": - "Evaluate on your own documents": extraction/evaluate-on-your-data.md - - "11. Reference": + - "12. Reference": - "API guide": extraction/nemo-retriever-api-reference.md # TODO: after nv-ingest code removal, update this link when CLI docs are relocated. - "CLI reference": https://github.com/NVIDIA/NeMo-Retriever/tree/main/nemo_retriever/docs/cli - "Quickstart: retriever CLI": reference/retriever-cli-quickstart.md - Environment variables: extraction/environment-config.md - "Metadata reference": extraction/content-metadata.md - - "12. Support & community": + - "13. Support & community": - Troubleshooting: extraction/troubleshoot.md - FAQ: extraction/faq.md - Contributing: extraction/contributing.md @@ -156,11 +159,8 @@ plugins: extraction/hosted-nims-when-to-use.md: extraction/deployment-options.md extraction/releasenotes-nv-ingest.md: extraction/releasenotes.md extraction/ngc-api-key.md: extraction/api-keys.md - extraction/notebooks/index.md: extraction/starter-kits.md - extraction/notebooks.md: extraction/starter-kits.md + extraction/notebooks.md: extraction/notebooks/index.md extraction/data-store.md: extraction/vdbs.md - extraction/custom-metadata.md: extraction/vdbs.md#metadata-and-filtering - extraction/integrations-langchain-llamaindex-haystack.md: extraction/starter-kits.md extraction/nemoretriever-parse.md: extraction/multimodal-extraction.md#text-and-layout-extraction extraction/supported-file-types.md: extraction/multimodal-extraction.md#supported-file-types-and-formats extraction/text-layout-extraction.md: extraction/multimodal-extraction.md#text-and-layout-extraction @@ -185,11 +185,7 @@ plugins: extraction/chunking.md: extraction/concepts.md#chunking extraction/quickstart-library-mode.md: extraction/deployment-options.md extraction/workflow-video-ocr.md: extraction/audio-video.md - extraction/user-defined-stages.md: extraction/customize-extend.md - extraction/user-defined-functions/index.md: extraction/customize-extend.md#user-defined-functions-udfs - extraction/user-defined-functions.md: extraction/customize-extend.md#user-defined-functions-udfs - extraction/customize-and-extend.md: extraction/customize-extend.md - extraction/nimclient.md: https://github.com/NVIDIA/NeMo-Retriever/blob/main/nemo_retriever/developer_docs/nimclient.md#nimclient-and-custom-nim-endpoints + extraction/user-defined-stages.md: https://github.com/NVIDIA/NeMo-Retriever/tree/main/nemo_retriever/src/nemo_retriever/graph#nemo-retriever-graph - site-urls markdown_extensions: @@ -211,11 +207,11 @@ markdown_extensions: - admonition - footnotes -# MkDocs 1.6+: exclude suite landing and legacy duplicate pages (still in repo for parity). +# MkDocs 1.6+: exclude legacy duplicate pages (still in repo for parity). # extraction/chunking.md — removed from nav; content is under concepts.md (redirect_maps keeps old URLs). -# Use /index.md (docs root only); bare index.md would exclude every index.md (e.g. under subfolders). +# Root index.md is not in nav; redirect_maps sends index.md → extraction/overview.md so /latest/ (Docs Hub tile) resolves. +# Use /index.md in exclude_docs only (bare index.md would exclude every index.md, e.g. extraction/notebooks/index.md). exclude_docs: | - /index.md extraction/chunking.md extraction/helm.md extraction/choose-your-path.md