feat(os-index): implement OS getIndicesStats and getClusterHealth for Phase 1 portlet display

[fabrizzio-dotCMS]

Summary

• Root cause: In Phase 1 (dual-write), OSIndexAPIImpl.getIndicesStats() and OSIndexAPIImpl.getClusterHealth() were unimplemented stubs returning empty maps. OS shadow indices appeared in the admin indices portlet with all metrics as n/a (count, size, replicas, health).
• IndexAPIImpl.getIndicesStats() routed exclusively to the read provider (ES in Phase 1) via router.read(), so OS stats were never consulted even if implemented.

Changes

OSIndexAPIImpl — implements two previously stubbed methods:

• getIndicesStats(): queries OS cluster via indices().stats(cluster_xxx.*), extracts primaries.docs().count() and primaries.store().sizeInBytes() per index, returns vendor-neutral IndexStats map
• getClusterHealth(): queries OS cluster via cluster().health(cluster_xxx.*) at Indices level, returns vendor-neutral ClusterIndexHealth map
• Errors are caught and logged — degraded portlet beats a hard failure

IndexAPIImpl.getIndicesStats() — changed from router.read() (ES-only) to the same dual-write aggregation pattern used by getClusterHealth(): merges ES and OS maps in Phase 1/2, OS entry wins on key collision.

Tests

• ContentletIndexAPIImplMigrationIT (integration, live ES + OS clusters):

  • getIndicesStats Phase 1: ES(T0) and OS(T1) both present in merged map with valid data
  • getIndicesStats Phase 0: ES only (auto-skipped on single-cluster profile)
  • getClusterHealth Phase 1: ES(T0) and OS(T1) both present with non-null status and positive shard count

• OSIndexAPIImplIntegrationTest (integration, live OS cluster):

  • getIndicesStats: creates two indices, verifies doc count ≥ 0, raw size ≥ 0, size string non-empty
  • getClusterHealth: creates an index, verifies status non-empty, shards > 0, replicas ≥ 0

Test plan

• Run ContentletIndexAPIImplMigrationIT with FEATURE_FLAG_OPEN_SEARCH_PHASE=1 — ES(T0) and OS(T1) indices should both appear in getIndicesStats and getClusterHealth results
• Verify admin indices portlet (/dotAdmin/#/dot-layout-grid) shows real count/size/replicas/health for OS shadow indices in Phase 1 instead of n/a

Closes #35820

🤖 Generated with Claude Code

[claude]

Claude finished @fabrizzio-dotCMS's task in 3m 33s —— View job

---

Claude PR Review

• Read changed files
• Analyze diagnostic code in IndexAPIImpl.java
• Analyze diagnostic code in OSIndexAPIImpl.java
• Review test changes
• Post review

---

Scope drift — PR metadata is stale

The title and body describe a temporary Phase 1 diagnostic ("debug code will be removed after root cause is confirmed"), but commit 28f79abd already reverted the JSP diagnostic and subsequent commits (e5e615d5, 0788346c, 7dbf16b6) turned this into a real implementation of OS getIndicesStats / getClusterHealth plus aggregation in IndexAPIImpl. This is now production code that affects the maintenance portlet. Title (debug(index-portlet): …), body, and any release-notes line should be updated to reflect that scope before merge — reviewers and a future bisect-author will be misled otherwise.

Findings

1. toHumanReadableSize — locale bug and missing units — OSIndexAPIImpl.java:425-430

if (bytes < 1_048_576L) return String.format("%.1fkb", bytes / 1_024.0);

• String.format without an explicit Locale follows the JVM default — on a server with LANG=es_* / de_* this produces "1,2kb" instead of "1.2kb", which then collides with any consumer that parses the string. Pass Locale.ROOT.
• No tb / pb branches. A 2 TB index renders as "2048.0gb". ES's ByteSizeValue.toString() (used in ESIndexAPI.java:170) handles those. Since the two providers' results are merged into one map by IndexAPIImpl.getIndicesStats(), this creates inconsistent formatting between rows in the same portlet table. Either reuse ByteSizeValue/equivalent or extend this helper.

2. ExpandWildcard.Open excludes closed indices — OSIndexAPIImpl.java:369

The ES path (ESIndexAPI.java:144) hits /<prefix>*/_stats with no wildcard filter — defaults are open+hidden depending on cluster setting. The OS path explicitly restricts to Open, so any closed OS index will silently produce no row in the portlet. Confirm this is the intended behavior or align both. getClusterHealth doesn't set a wildcard at all — also inconsistent with the stats call right above it.

3. No allowNoIndices(true) — OSIndexAPIImpl.java:367-369, 401-402

Fresh cluster / pre-bootstrap path: a call before any index matches <prefix>* can throw IndexNotFoundException from the OS client. It's caught by the broad Exception handler, but produces an Logger.error stack trace per portlet load until indices exist. getClosedIndexes() in the same class sets allowNoIndices(true) explicitly — worth doing here too.

4. Silent merge collision — IndexAPIImpl.java:146-148

final Map<String, IndexStats> merged = new HashMap<>(esImpl.getIndicesStats());
merged.putAll(osImpl.getIndicesStats());

The javadoc says "On key collision the OS entry wins." In Phase 1/2 ES holds T0 names and OS holds T1 names, so collisions shouldn't happen — but if naming ever overlaps (misconfig, shared prefix across clusters) ES data is dropped without any signal. A Logger.warn when esStats.keySet() ∩ osStats.keySet() ≠ ∅ would make this debuggable. Same point applies to getClusterHealth() which already uses the identical pattern.

5. Broad catch (Exception e) → return empty map — OSIndexAPIImpl.java:390-393, 419-422

Renders as "no data" in the portlet, which is what was happening before this PR for a different reason. A transient network blip + cached empty map gives the operator the same misleading screen the ghost-index investigation was chasing. Matches existing patterns in the file, so not a blocker — but worth narrowing to OS client exceptions, or at least logging at warn with a hint that the table will be incomplete.

6. Test assertion strength — OSIndexAPIImplIntegrationTest.java

assertTrue("Document count must be non-negative", liveStats.documentCount() >= 0) and sizeRaw() >= 0 — these are tautological for long-typed counters. They confirm the path runs without throwing, which is fine, but they don't verify the docs/bytes are actually being read from the response. After createIndex, the index is empty, so checking documentCount() == 0 and sizeRaw() > 0 (an empty index still has segment overhead) would be tighter. Not blocking.

7. Nit — fully-qualified IndexStats — OSIndexAPIImpl.java

The OS client also has a class named IndexStats (org.opensearch.client.opensearch.indices.stats.IndexStats), so the long com.dotcms.content.index.domain.IndexStats references are intentional disambiguation. Fine as-is; flagging only so future cleanup doesn't break it by adding the import.

---

Tests cover the right cases (Phase 0 ES-only, Phase 1 dual-merge for both stats and health). Routing aggregation in IndexAPIImpl follows the same shape as the existing listIndices / getClusterHealth / getIndices aggregating-read methods documented in the class header.
· Branch