feat: Knowledge table optimization (#2202)

[jason-dong03]

• fix: remove page_size cap and fetch all files to fix Knowledge table pagination past 100 files

• T1-1 to T1-3 deliverable

• file size sort, added check for file_size and chunk_count -10k limit -> 500 file limit
  -changed hardcoded page size to be controlled by React State; removed client side pagination from AG Grid and wired up custom UI mimicing AG Grid UI

• feat: T1-4 server-side sort + T1-6 native AG Grid pagination

• Add sortBy/sortOrder state wired to useListFiles (T1-4)
• Map AG Grid colId to backend sort field names (size->file_size, etc.)
• onSortChanged resets to page 1 on sort change via paginationGoToFirstPage()
• Restore AG Grid native pagination UI with pageSize:10000 (T1-6)
• T1-2: le=100 backend cap already removed from files.py

• v2 API Migration

Switched to composite aggregation pagination under v2

• Update page.tsx

• fix: remove page_size cap and fetch all files to fix Knowledge table pagination past 100 files

• T1-1 to T1-3 deliverable

• file size sort, added check for file_size and chunk_count -10k limit -> 500 file limit
  -changed hardcoded page size to be controlled by React State; removed client side pagination from AG Grid and wired up custom UI mimicing AG Grid UI

• feat: T1-4 server-side sort + T1-6 native AG Grid pagination

• Add sortBy/sortOrder state wired to useListFiles (T1-4)
• Map AG Grid colId to backend sort field names (size->file_size, etc.)
• onSortChanged resets to page 1 on sort change via paginationGoToFirstPage()
• Restore AG Grid native pagination UI with pageSize:10000 (T1-6)
• T1-2: le=100 backend cap already removed from files.py

• v2 API Migration

Switched to composite aggregation pagination under v2

• Address review feedback: log _get_file_count failures, restore search pagination, mark total as approximate

• Git merge fix issues

• fix git HEAD

• v2 files: add approximate counts & cursor parse

Frontend: switch to /api/v2/files, include is_approximate in ListFilesResponse, and disable pagination buttons; removed a debug console.log. Backend API: add _parse_after_key helper to validate JSON-encoded after_key and return 400 for invalid values; use FastAPI Query pattern parameter. FileServiceV2: add embedding_model to composite sort fields, make _get_file_count return (count, is_approximate) and log on failure, use raw bucket count to detect final page, and propagate is_approximate in responses so the UI can display approximate totals.

• Refactored component

-moved custom footer into its own component

• fix merge issues

• PR Fixes

-changed styling to match Tailwind

• footer is now always active, showing regardless of servertotal or wildcardquery -global sort fix

• Fix search pagination: route all data through useListFiles, restore search_files page_size cap, reset page on search change

• Update page.tsx

• update page.tsx

• style: ruff autofix (auto)

• Lint backend fix

• Fix: remove exception detail from error responses to prevent information exposure (git advanced security fix)

---

Summary by CodeRabbit

• New Features

  • Added faster, server-side file listing and search with cursor-based pagination.
  • Added filtering by connector, MIME type, owner, and filename.
  • Added server-side sorting for supported file metadata.
  • Added a pagination footer with page-size controls, navigation, and result counts.
  • Added approximate total result information for file listings.

• Bug Fixes

  • Improved sorting consistency when file size or chunk count values are missing.
  • Added validation and clearer error handling for invalid pagination cursors.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: d5be2f80-c558-427b-81ef-54c469794a13

📥 Commits

Reviewing files that changed from the base of the PR and between 2a49c24 and 2811d38.

📒 Files selected for processing (1)

• frontend/app/knowledge/page.tsx

🚧 Files skipped from review as they are similar to previous changes (1)

• frontend/app/knowledge/page.tsx

---

Walkthrough

This PR adds v2 file listing and search APIs with composite cursor pagination and approximate totals. The Knowledge page uses server-side pagination and sorting with cursor caching and pagination controls.

Changes

File listing pagination

Layer / File(s)	Summary
File aggregation service src/services/file_service_v2.py	FileServiceV2 adds filtered composite aggregation, cursor pagination, approximate cardinality totals, and a separate chunk_count terms-aggregation path.
V2 API exposure src/api/v2/*, src/app/routes/internal.py, src/app/container.py, src/dependencies.py	The v2 routes validate cursors and parameters, call FileServiceV2, map failures to HTTP responses, and remain registered alongside v1 routes.
Frontend pagination integration frontend/app/api/queries/useListFiles.ts, frontend/app/knowledge/page.tsx, frontend/components/knowledge-pagination-footer.tsx	The Knowledge page sends cursor and sort parameters, caches cursors, resets pagination when inputs change, and renders pagination controls.
Existing file sorting support src/api/files.py, src/services/file_service.py, src/api/v2/__init__.py	The file API documentation describes server-side aggregation behavior. Missing numeric sort values use 0 in the existing file service.

Estimated code review effort: 4 (Complex) | ~60 minutes

Sequence Diagram(s)

sequenceDiagram
  participant KnowledgePage
  participant useListFiles
  participant FilesV2API
  participant FileServiceV2
  participant OpenSearch
  KnowledgePage->>useListFiles: request page with sort and after_key
  useListFiles->>FilesV2API: GET /api/v2/files
  FilesV2API->>FileServiceV2: pass filters, sorting, and cursor
  FileServiceV2->>OpenSearch: execute aggregation
  OpenSearch-->>FileServiceV2: return files and after_key
  FileServiceV2-->>FilesV2API: return files and approximate total
  FilesV2API-->>useListFiles: return paginated response
  useListFiles-->>KnowledgePage: render rows and pagination footer

Loading

Possibly related issues

• langflow-ai/openrag#2142 — Covers server-side pagination and sorting for file listings in the same frontend and backend areas.

Possibly related PRs

• langflow-ai/openrag#2187 — Both changes modify frontend/app/knowledge/page.tsx for file-listing data-source behavior.
• langflow-ai/openrag#2198 — Both changes modify Knowledge file and search pagination with different pagination mechanisms.

Suggested reviewers: vchen7629

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly summarizes the main change: optimizing Knowledge table pagination and sorting.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch knowledge-table-pagination

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

React Doctor found 1 new issue in 1 file · 1 warning · score 85 / 100 (Great) · 1 fixed · vs main

1 warning

app/knowledge/page.tsx

• ⚠️ L559 State updates chained through effects no-chain-state-updates

_{Reviewed by React Doctor for commit 2811d38. See inline comments for fixes.}

[coderabbitai]

Actionable comments posted: 9

🧹 Nitpick comments (3)

src/api/v2/files.py (2)

77-86: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Log the traceback now that the response omits the error detail.

The 500 body no longer carries detail, which is correct. But the log line records only str(e), so the stack trace is lost. Use logger.exception or pass exc_info=True so the failure remains diagnosable. The same applies to the search_files handler at Lines 116-125.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/api/v2/files.py` around lines 77 - 86, Update the exception logging in
the file-listing handler and the search_files handler to include traceback
information while keeping the 500 response body free of error details. Replace
the current logger.error calls with logger.exception or equivalent
exc_info-enabled logging, preserving the existing authentication-error handling
and response behavior.

---

44-58: 🗄️ Data Integrity & Integration | 🔵 Trivial | ⚡ Quick win

Add response model typing to the new v2 endpoints.

Both v2 handlers return a bare JSONResponse, so FastAPI publishes no schema for them. The frontend already declares the shape in ListFilesResponse (frontend/app/api/queries/useListFiles.ts Lines 20-27). Declare a Pydantic response model and register it with response_model when the route is added in src/app/routes/internal.py. That makes the is_approximate and after_key contract explicit and detects field drift.

As per path instructions: "Verify dependency injection via src/dependencies.py, response model typing, and correct HTTP status codes."

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/api/v2/files.py` around lines 44 - 58, Define a Pydantic response model
for the payload returned by list_files, including the is_approximate and
after_key fields matching frontend ListFilesResponse, and register it as
response_model when the v2 route is added in internal.py. Verify the handler’s
dependency injection remains wired through _get_file_service and
get_current_user, and preserve the endpoint’s intended HTTP status codes.

Source: Path instructions

src/services/file_service_v2.py (1)

337-362: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Extract the shared metadata sub-aggregation.

The file_metadata top_hits block and the chunk_count sub-aggregation are identical here and at Lines 283-308. FileService._build_file_aggregation holds a third copy. A future field addition must be applied in three places. Define one module-level constant and reuse it in both builders.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/services/file_service_v2.py` around lines 337 - 362, Extract the repeated
file metadata and chunk-count sub-aggregations into one module-level constant,
then reuse it in the aggregation builders at the current block, the matching
block around the earlier query, and FileService._build_file_aggregation.
Preserve the existing top_hits fields, sorting, and chunk-count configuration
while ensuring all builders reference the shared definition.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@frontend/app/knowledge/page.tsx`:
- Around line 348-352: Gate non-first-page requests in the knowledge page’s
pagination flow so they require a cached composite cursor before using afterKey;
update the reactive cursor state when after_key is cached, and pass cursor
availability plus total-approximation state to KnowledgePaginationFooter. In
frontend/app/knowledge/page.tsx lines 348-352, 480-497, and 1128-1136, make
these request, state, and prop changes. In
frontend/components/knowledge-pagination-footer.tsx lines 3-10, 22-23, and
92-99, add the props, stop deriving final-page status solely from totalPages,
disable Next until the current response has a valid cursor, and disable
count-based controls when totals are approximate.
- Around line 377-387: The effectiveData flow must sort non-wildcard search
results before slicing the requested page, using the active sortBy/sortOrder
values; preserve wildcard server ordering by disabling or removing local
comparators for server-sorted columns in wildcard mode.

In `@src/api/v2/files.py`:
- Around line 20-23: Move FileServiceV2 construction out of _get_file_service:
instantiate and retain it in main.py’s lifespan block, expose a dependency
provider through src/dependencies.py, and update the route to inject that
provider instead of importing or constructing FileServiceV2 locally.
- Around line 89-99: Update the page_size Query parameter in search_files to
enforce the same upper bound as list_files, using le=500 while preserving its
existing minimum and default values.

In `@src/services/file_service_v2.py`:
- Around line 44-94: Update list_files so the composite-aggregation path rejects
page values greater than 1 when after_key is None, instead of silently returning
page 1 while reporting the requested page; preserve normal page-1 requests and
cursor-based pagination, and leave the chunk_count path unchanged.
- Around line 394-401: Update the exception path in the file-count retrieval
method to return an unknown or approximate total, such as `(0, True)`, instead
of `(0, False)`. Preserve the warning log and successful aggregation result,
ensuring the API does not treat a failed count as an exact total that disables
pagination.
- Around line 96-128: Add is_approximate: False to the fallback return
dictionary in the file-listing method’s opensearch exception path, keeping it
consistent with the success return and ensuring failed empty results are
reported as exact.
- Around line 313-336: The chunk-count terms aggregation in
_build_terms_aggregation_for_chunk_count must not request unbounded offset +
page_size buckets; enforce a safe maximum (or use cursor pagination) before
assigning the terms size, while preserving page slicing. Update nearby
documentation and any global/accurate-order claims to describe the results as
approximate unless the bounded aggregation configuration guarantees otherwise,
consistent with is_approximate=True.
- Around line 255-272: The _build_composite_aggregation method currently omits
documents missing the selected sort field or filename tie-breaker. Add
missing_bucket: true to the terms definitions for both the primary sort source
and the filename_tiebreak source, preserving existing field and order settings
so missing file_size, embedding_model, mimetype, and owner documents remain in
results.

---

Nitpick comments:
In `@src/api/v2/files.py`:
- Around line 77-86: Update the exception logging in the file-listing handler
and the search_files handler to include traceback information while keeping the
500 response body free of error details. Replace the current logger.error calls
with logger.exception or equivalent exc_info-enabled logging, preserving the
existing authentication-error handling and response behavior.
- Around line 44-58: Define a Pydantic response model for the payload returned
by list_files, including the is_approximate and after_key fields matching
frontend ListFilesResponse, and register it as response_model when the v2 route
is added in internal.py. Verify the handler’s dependency injection remains wired
through _get_file_service and get_current_user, and preserve the endpoint’s
intended HTTP status codes.

In `@src/services/file_service_v2.py`:
- Around line 337-362: Extract the repeated file metadata and chunk-count
sub-aggregations into one module-level constant, then reuse it in the
aggregation builders at the current block, the matching block around the earlier
query, and FileService._build_file_aggregation. Preserve the existing top_hits
fields, sorting, and chunk-count configuration while ensuring all builders
reference the shared definition.

🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 69fc7988-0f2f-460e-8f8e-b26bcf557c0d

📥 Commits

Reviewing files that changed from the base of the PR and between 5b565bb and b245948.

📒 Files selected for processing (9)

• frontend/app/api/queries/useListFiles.ts
• frontend/app/knowledge/page.tsx
• frontend/components/knowledge-pagination-footer.tsx
• src/api/files.py
• src/api/v2/__init__.py
• src/api/v2/files.py
• src/app/routes/internal.py
• src/services/file_service.py
• src/services/file_service_v2.py