feat(aicore): reactive credential reload and proactive secret watcher

[tiagoek]

Description

Addresses credential rotation failures in long-running agent pods.

When BTP rotates a service binding, kubelet updates the mounted secret volume within ~2 minutes. However, agent pods are never restarted — LiteLLM's cached OAuth token (up to 12h lifetime) eventually expires and the refresh attempt fails because the process still holds the old client_secret. This PR makes credential rotation transparent to agents without requiring pod restarts.

1. Reactive credential reload on AuthenticationError

completion() and acompletion() intercept litellm.AuthenticationError, reload credentials from the mounted secret volume via set_aicore_config(), and retry the call once. If the retry also fails, the error propagates normally.

2. Proactive secret watcher (watch_aicore_config)

A daemon thread polls the secret directory mtime every 60 seconds. On change (kubelet performs an atomic symlink swap on rotation), calls set_aicore_config() proactively — before the cached OAuth token expires. Avoids the 401 entirely rather than recovering from it.

Typical usage:

from sap_cloud_sdk.aicore import set_aicore_config, watch_aicore_config

set_aicore_config()
watch_aicore_config()   # starts daemon thread — call once at startup

3. LangGraph / ChatLiteLLM compatibility (patch_litellm_for_credential_rotation)

LangGraph agents use ChatLiteLLM which calls litellm.completion directly, bypassing our completion() wrapper. patch_litellm_for_credential_rotation() wraps litellm.completion and litellm.acompletion at the module level so all callers get transparent reload on 401 — including ChatLiteLLM.

from sap_cloud_sdk.aicore import (
    set_aicore_config,
    patch_litellm_for_credential_rotation,
    watch_aicore_config,
)

set_aicore_config()
patch_litellm_for_credential_rotation()   # for LangGraph / ChatLiteLLM agents
watch_aicore_config()

Related Issues

• Stacked on: this PR is the base; feat(aicore): transparent proxy routing and BTP Destination Service mode #271 stacks on top

Type of Change

• New feature (non-breaking change that adds functionality)
• Bug fix (non-breaking change that fixes an issue)
• Breaking change (fix or feature that would cause existing functionality to change)
• Documentation update
• Code refactoring
• Dependency update

How to Test

python -m pytest tests/aicore/unit/ -v
# 111 passed

Key test files:

• tests/aicore/unit/test_aicore_watcher.py — proactive watcher (8 tests)
• tests/aicore/unit/test_credential_rotation_flow.py — env update contract (6 tests)
• tests/aicore/unit/test_completion.py — reactive reload (sync + async)
• tests/aicore/unit/test_langgraph_compat.py — LangGraph / ChatLiteLLM compatibility (15 tests)

Checklist

• I have read the Contributing Guidelines
• I have verified that my changes solve the issue
• I have added/updated automated tests to cover my changes
• All tests pass locally
• I have verified that my code follows the Code Guidelines
• I have updated documentation (if applicable)
• I have added type hints for all public APIs
• My code does not contain sensitive information (credentials, tokens, etc.)
• I have followed Conventional Commits for commit messages

Breaking Changes

None. All changes are additive:

• watch_aicore_config() — new public function, opt-in
• patch_litellm_for_credential_rotation() — new public function, opt-in, idempotent
• Retry on AuthenticationError in completion() — same exception type propagates if retry also fails; no contract change

[tiagoek]

Production validation — proactive credential reload (watch_aicore_config)

Test suite (cloud-sdk-python)

All 70 unit tests pass on branch feat/aicore-transparent-tls:

tests/aicore/unit/test_aicore_watcher.py          8 passed
tests/aicore/unit/test_credential_rotation_flow.py 6 passed
tests/aicore/unit/test_aicore.py                  (existing, green)
tests/aicore/unit/test_completion.py              (existing, green)

Integration into autonomous-documentation-org

The vendor shim (vendor/sap_cloud_sdk_aicore/) was synced with watch_aicore_config + _get_secret_dir_mtime.

Finding (critical): Starlette 1.3.1 does NOT propagate lifespan events to mounted sub-apps — verified with a live ASGI simulation:

# Starlette 1.3.1 — Mount does not trigger FastAPI sub-app lifespan
outer = Starlette(routes=[Mount('/sub', app=fastapi_sub_app)])
# result: started = []  (sub-app _lifespan never called)

watch_aicore_config() was therefore moved to app/main.py at module level alongside set_aicore_config() — the same proven path that loads credentials on every boot.

Production proof (Kyma managed runtime, auto-doc-dev-eu12)

Thread count in PID 1 before and after deploying commit a72f5e0:

Pod	Image	Threads in PID 1
Pre-fix (6ad5b0854983)	0.3.3-20260825... (no watcher call)	14
Post-fix (1858f9060d95)	0.3.3-20260825200836_fd446f2+1	15

The extra thread is aicore-secret-watcher (daemon, polls /etc/secrets/appfnd/aicore/aicore-instance/ mtime every 30 s). It will call set_aicore_config() on the next kubelet symlink-swap rotation without requiring a pod restart.

autonomous-documentation-org unit tests: 286 passed, 1 skipped — coverage 90.51%

[tiagoek]

E2E validation — reactive credential reload (live pod test)

Environment: Kyma managed runtime, namespace auto-doc-dev-eu12, pod autonomous-documentation-ee13df69-1858f9060d95 (image 0.3.3-20260825)

Test script

tests/integration/e2e_llm_reactive_reload.py — run via kubectl exec in the pod where the secret volume is mounted.

Execution output

[1] Real secret loaded from file: 263a99c5...
[2] Env poisoned with invalid secret
INFO sap_cloud_sdk_aicore.completion: AI Core credentials reloading after authentication failure
INFO sap_cloud_sdk_aicore: Loaded AICORE_CLIENT_SECRET from file: /etc/secrets/appfnd/aicore/aicore-instance/clientsecret
INFO sap_cloud_sdk_aicore: AI Core configuration has been set successfully
[3] Post-call secret: (cleared by _clear_client_secret — expected PR#257)
[4] LLM response: 'OK'
[OK] Test 1 PASSED — reactive reload intercepted the failure, retried, LLM responded
[5] LiteLLM aicore/client namespace attrs: [...in_memory_llm_clients_cache...]
[6] AICORE_CLIENT_SECRET in env: (absent — cleared after token acquisition)
[OK] Test 2 PASSED — os.environ is the sole credential store; no per-model client object
INFO sap_cloud_sdk_aicore.completion: AICORE_CLIENT_SECRET cleared from environment after token acquisition (AFSDK-4291)

What this proves

Scenario	Result
Invalid AICORE_CLIENT_SECRET in env → LiteLLM call	AuthenticationError caught by handler
reload_aicore_credentials() reads from mounted file	All 5 AICORE_* vars reloaded from /etc/secrets/appfnd/aicore/aicore-instance/
Retry with restored credentials	LLM responded 'OK'
PR#257 _clear_client_secret() behaviour	Secret cleared from env after token acquisition — confirmed expected
LiteLLM in_memory_llm_clients_cache	Memory cache keyed by model identifier, not by credential values — os.environ read fresh on every OAuth token request

[pedro-hca]

[thiagob]

Would be possible to add a unit test with lang graph?

Checking the template I don't see our completion or acompletion being used. Would be important to map if there is any in the agent itself that LoBs will have to do, or if it is just a simper version bump.

[tiagoek]

improving!

[tiagoek]

LangGraph agents use ChatLiteLLM which calls litellm.completion directly and bypasses our completion() wrapper, so the reactive 401 handler wouldn't fire for them

To cover that gap I added patch_litellm_for_credential_rotation(), it wraps litellm.completion and litellm.acompletion at the module level so ALL callers (including ChatLiteLLM) get the same transparent reload on 401. Combined with watch_aicore_config(), LangGraph agents are fully covered