Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 25 additions & 1 deletion .github/workflows/build_container.yml
Original file line number Diff line number Diff line change
Expand Up @@ -200,6 +200,18 @@ jobs:
# discovery excludes this same prefix so no test runs in both shards.
test_filter: >-
code.api.v4_0_0.Dynamic
- shard: 10
name: "PropGatedPublicEndpoint (read_*_requires_role=true)"
# JsonSchemaValidationPublicPropTrueTest / AuthenticationTypeValidationPublicPropTrueTest
# need read_json_schema_validation_requires_role /
# read_authentication_type_validation_requires_role forced true, which is baked into
# Http4s400's ResourceDoc error list at object-init time -- a value only one JVM can
# observe. Every other shard boots with these props unset (false, the default) and
# the pom.xml default tagsToExclude skips this tag there. This shard is the only place
# the true branch is exercised; see the "Run tests" step's shard-10 special case below.
test_filter: >-
code.api.v4_0_0.JsonSchemaValidationPublicPropTrueTest
code.api.v4_0_0.AuthenticationTypeValidationPublicPropTrueTest

services:
redis:
Expand Down Expand Up @@ -373,6 +385,18 @@ jobs:
FILTER="${FILTER}${EXTRAS}"
fi

# Shard 10: PropGatedPublicEndpoint tests need read_json_schema_validation_requires_role /
# read_authentication_type_validation_requires_role forced true and the
# PropGatedPublicEndpoint tag included (pom.xml's default tagsToExclude skips it
# everywhere else). These two OBP_* env vars and the tag override apply only to this
# shard's mvn invocation below.
EXTRA_MVN_ARGS=""
if [ "${{ matrix.shard }}" = "10" ]; then
export OBP_READ_JSON_SCHEMA_VALIDATION_REQUIRES_ROLE="true"
export OBP_READ_AUTHENTICATION_TYPE_VALIDATION_REQUIRES_ROLE="true"
EXTRA_MVN_ARGS="-DtagsToInclude=PropGatedPublicEndpoint -Dtest.tagsToExclude="
fi

# `mvn process-resources scalatest:test` — run process-resources (copies the
# dynamically-generated props from src/main/resources onto the classpath at
# target/classes/props) then the scalatest:test goal. This SKIPS the
Expand All @@ -391,7 +415,7 @@ jobs:
set +e
MAVEN_OPTS="-Xmx3G -Xss2m -XX:MaxMetaspaceSize=1G" \
timeout 1500 mvn process-resources scalatest:test -pl obp-commons,obp-api -DfailIfNoTests=false \
-DwildcardSuites="$FILTER" \
-DwildcardSuites="$FILTER" $EXTRA_MVN_ARGS \
> maven-build-shard${{ matrix.shard }}.log 2>&1
rc=$?
set -e
Expand Down
26 changes: 25 additions & 1 deletion .github/workflows/build_pull_request.yml
Original file line number Diff line number Diff line change
Expand Up @@ -194,6 +194,18 @@ jobs:
# discovery excludes this same prefix so no test runs in both shards.
test_filter: >-
code.api.v4_0_0.Dynamic
- shard: 10
name: "PropGatedPublicEndpoint (read_*_requires_role=true)"
# JsonSchemaValidationPublicPropTrueTest / AuthenticationTypeValidationPublicPropTrueTest
# need read_json_schema_validation_requires_role /
# read_authentication_type_validation_requires_role forced true, which is baked into
# Http4s400's ResourceDoc error list at object-init time -- a value only one JVM can
# observe. Every other shard boots with these props unset (false, the default) and
# the pom.xml default tagsToExclude skips this tag there. This shard is the only place
# the true branch is exercised; see the "Run tests" step's shard-10 special case below.
test_filter: >-
code.api.v4_0_0.JsonSchemaValidationPublicPropTrueTest
code.api.v4_0_0.AuthenticationTypeValidationPublicPropTrueTest

services:
redis:
Expand Down Expand Up @@ -377,6 +389,18 @@ jobs:
FILTER="${FILTER}${EXTRAS}"
fi

# Shard 10: PropGatedPublicEndpoint tests need read_json_schema_validation_requires_role /
# read_authentication_type_validation_requires_role forced true and the
# PropGatedPublicEndpoint tag included (pom.xml's default tagsToExclude skips it
# everywhere else). These two OBP_* env vars and the tag override apply only to this
# shard's mvn invocation below.
EXTRA_MVN_ARGS=""
if [ "${{ matrix.shard }}" = "10" ]; then
export OBP_READ_JSON_SCHEMA_VALIDATION_REQUIRES_ROLE="true"
export OBP_READ_AUTHENTICATION_TYPE_VALIDATION_REQUIRES_ROLE="true"
EXTRA_MVN_ARGS="-DtagsToInclude=PropGatedPublicEndpoint -Dtest.tagsToExclude="
fi

# `mvn process-resources scalatest:test` — run process-resources (copies the
# dynamically-generated props from src/main/resources onto the classpath at
# target/classes/props) then the scalatest:test goal. This SKIPS the
Expand All @@ -395,7 +419,7 @@ jobs:
set +e
MAVEN_OPTS="-Xmx3G -Xss2m -XX:MaxMetaspaceSize=1G" \
timeout 1500 mvn process-resources scalatest:test -pl obp-commons,obp-api -DfailIfNoTests=false \
-DwildcardSuites="$FILTER" \
-DwildcardSuites="$FILTER" $EXTRA_MVN_ARGS \
> maven-build-shard${{ matrix.shard }}.log 2>&1
rc=$?
set -e
Expand Down
7 changes: 5 additions & 2 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
- Commit messages, code comments, and PR titles/descriptions: no AI/tool names (Claude, GPT, Copilot, etc.), no AI-typical filler phrasing ("Certainly!", "I'll help you with..."), no emoji, no "AI-generated"/"LLM" labels. Use plain Conventional Commits style (`fix:`, `feat:`, `refactor:`, ...) and set commit author/committer to the actual person directing the work.
- **Goal is full http4s migration** — eliminate Lift Web and all deprecated libraries entirely. Treat Lift code as temporary scaffolding to be removed, not maintained. When fixing bugs or adding features, always prefer the http4s path.
- **Versioning is tech-agnostic** — API version numbers reflect API signature changes (new/changed fields, new behaviour), never the underlying framework. A framework migration (Lift → http4s) happens in-place at the existing version; it does not justify a version bump.
- **`APIMethodsXYZ.scala` (Lift) files are the source of truth for migration.** The commented-out Lift ResourceDocs and endpoints inside each `APIMethodsXYZ.scala` are the canonical reference for what the http4s version should match: URL templates, verb casing, summaries, descriptions, example bodies, error lists, tags. **Do NOT edit these files to make the parity audit pass.** The audit compares http4s against the Lift source-of-truth — when it flags a diff, the fix is to either (a) update http4s to match Lift, or (b) document the difference at the http4s site as a known intentional drift (e.g. a placeholder rename for `ResourceDocMatcher` middleware, or an upstream-driven case-class shape change). Rewriting the Lift comments to match http4s runs the comparison backwards and destroys the historical record. See `scripts/check_lift_http4s_resource_doc_parity.py` for the audit, and `scripts/rehydrate_resource_docs.py` / `scripts/restore_resource_doc_bodies.py` for the canonical Lift → http4s restoration tools.
- **`scripts/resource_doc_baseline/lift_resource_docs_vX_Y_Z.json` is the source of truth for migration.** The 12 `APIMethodsXYZ.scala` files that used to hold this as commented-out Lift `ResourceDoc` text have been deleted (they had shrunk to thin runtime shims plus ~60,000 lines of dead comments — see git history for their last content, or `scripts/resource_doc_baseline/README.md` for the full story). The JSON baseline is the canonical reference for what the http4s version should match: URL templates, verb casing, summaries, descriptions, example bodies, error lists, tags — each field stored as the literal, unevaluated Scala source snippet it always was. **Do NOT hand-edit this JSON to make the parity audit pass**, for the same reason you never edited the old Lift comments for that purpose: it's the historical record the audit compares http4s against. When the audit flags a diff, the fix is to either (a) update http4s to match the baseline, or (b) if it's a reviewed, intentional difference, add a digest-bound entry to `scripts/resource_doc_baseline/parity_allowlist.json` (see that directory's README for the exact workflow — use `allowlist_helper.py`, don't hand-compute digests). See `scripts/check_lift_http4s_resource_doc_parity.py` for the audit (now reads the JSON baseline on the Lift side, live `.scala` on the http4s side), and `scripts/rehydrate_resource_docs.py` / `scripts/restore_resource_doc_bodies.py` for the canonical baseline → http4s restoration tools (also JSON-sourced now).

## Architecture (Onboarding)

Expand Down Expand Up @@ -274,7 +274,7 @@ Perf note: integration tests are DB/HTTP-bound (~0.4 s/test) on both frameworks;

### Shard assignment

Shards are defined per-matrix-entry in `.github/workflows/build_pull_request.yml` and `.github/workflows/build_container.yml` (both files carry an identical 9-shard matrix — update both when reshaping). Shard 8 runs a **catch-all**: any `.scala` test file whose package is not covered by shards 1–7 and 9 is appended automatically at runtime — new packages are never silently skipped. Extras are printed in the step log under `"Catch-all extras added to shard 8"`. Shard 1 (`code.api.v4_0_0` non-Dynamic) is itself discovered at runtime rather than hand-listed — see the "Run tests" step's `matrix.shard = 1` branch — specifically so a newly added class in that package can never fall through both shard 1 and the catch-all.
Shards are defined per-matrix-entry in `.github/workflows/build_pull_request.yml` and `.github/workflows/build_container.yml` (both files carry an identical 10-shard matrix — update both when reshaping). Shard 8 runs a **catch-all**: any `.scala` test file whose package is not covered by shards 1–7, 9, and 10 is appended automatically at runtime — new packages are never silently skipped. Extras are printed in the step log under `"Catch-all extras added to shard 8"`. Shard 1 (`code.api.v4_0_0` non-Dynamic) is itself discovered at runtime rather than hand-listed — see the "Run tests" step's `matrix.shard = 1` branch — specifically so a newly added class in that package can never fall through both shard 1 and the catch-all.

| Package prefix | Shard |
|---|---|
Expand All @@ -288,6 +288,9 @@ Shards are defined per-matrix-entry in `.github/workflows/build_pull_request.yml
| `code.connector`, `code.util`, `code.api.Authentication*`, `code.api.dauthTest`, `code.api.DirectLoginTest`, `code.api.gateWayloginTest`, `code.api.OBPRestHelperTest`, `code.entitlement`, `code.bankaccountcreation`, `code.bankconnectors`, `code.container`, `code.management`, `code.metrics`, `code.concurrency` | 8 |
| anything else | **8** (catch-all) |
| `code.api.v4_0_0.Dynamic*` | 9 |
| `code.api.v4_0_0.JsonSchemaValidationPublicPropTrueTest`, `code.api.v4_0_0.AuthenticationTypeValidationPublicPropTrueTest` (tagged `PropGatedPublicEndpoint`) | 10 |

Shard 10 is a special case: it's the only shard that overrides pom.xml's default `tagsToExclude` (which otherwise skips `PropGatedPublicEndpoint` everywhere) and sets `OBP_READ_JSON_SCHEMA_VALIDATION_REQUIRES_ROLE`/`OBP_READ_AUTHENTICATION_TYPE_VALIDATION_REQUIRES_ROLE=true`, because those two props are baked into `Http4s400`'s `ResourceDoc` error lists at object-init time — a single JVM can only ever observe one value of each, so the `true` branch needs its own shard while every other shard (which boots with the props unset, i.e. `false`) exercises the default branch. `run_tests_parallel.sh` (local runner) mirrors this with a dedicated sequential step after its 4 main shards — see that script's own comment near `PropGatedPublicEndpoint`.

To explicitly move a package to a different shard, add it to that shard's `test_filter` block — it will be excluded from the catch-all automatically. `run_tests_parallel.sh` (local runner) uses a coarser 4-shard layout that folds all 9 CI shards' coverage into 4 wildcardSuites groups — see its own header comment for the mapping.

Expand Down
4 changes: 2 additions & 2 deletions FAQ.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@

## How do I create new endpoints in OBP API?

See lazy val getCustomersForUser in /src/main/scala/code/api/v3_0_0/APIMethods300.scala as a code example
See lazy val getCustomersForUser in /src/main/scala/code/api/v3_0_0/Http4s300.scala as a code example


## How do I minimise merge conflicts in my fork?
Expand All @@ -30,7 +30,7 @@ In more detail:

2) Each version's endpoints are defined as native http4s `HttpRoutes[IO]` in `code.api.vX.Http4sXxx` (e.g. `Http4s300`) and exposed via `wrappedRoutesVXxxServices`. These are wired, in priority order, into `Http4sApp.baseServices` (see `obp-api/src/main/scala/code/api/util/http4s/Http4sApp.scala`). Each version's routes are wrapped by `Http4sApp.gate`, which returns `HttpRoutes.empty` when the version is disabled.

The matching Lift `OBPAPI3_0_0.scala` / `APIMethods300.scala` files are retained only as commented-out source-of-truth and as the `allResourceDocs` registry used by the resource-docs aggregation — they no longer serve requests.
`OBPAPI3_0_0.scala` is retained only as a thin re-export (`Implementations3_0_0 = Http4s300.Implementations3_0_0`) used by the resource-docs aggregation — it no longer serves requests. The original Lift `APIMethods300.scala` has been deleted; its ResourceDoc text lives on in `scripts/resource_doc_baseline/lift_resource_docs_v3_0_0.json` (see that directory's README).

3) Per-endpoint enable/disable is enforced at request time by `ResourceDocMiddleware`, which reads the Props for explicitly enabled/disabled endpoints (`api_enabled_endpoints` / `api_disabled_endpoints`) and also performs authentication, role checks, and entity resolution.

Expand Down
22 changes: 21 additions & 1 deletion obp-api/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,15 @@
<packaging>jar</packaging>
<name>Open Bank Project API</name>

<properties>
<!-- Referenced (not hardcoded) in the scalatest-maven-plugin config below, so a
command-line -Dtest.tagsToExclude= override actually takes effect: Maven only lets
-D override a POM-declared value when the config element is a ${property}
expression, not a fixed string literal. Default excludes PropGatedPublicEndpoint
(see run_tests_parallel.sh / the CI job that overrides this to include it). -->
<test.tagsToExclude>code.external,GetBanksPerf,PropGatedPublicEndpoint</test.tagsToExclude>
</properties>

<dependencies>
<dependency>
<groupId>com.tesobe</groupId>
Expand Down Expand Up @@ -595,7 +604,18 @@
InaccessibleObjectException that only reproduces in whichever environment
is missing it. -->
<argLine>-Drun.mode=test -XX:MaxMetaspaceSize=1g -Xms1g -Xmx2g --add-opens java.base/java.lang=ALL-UNNAMED --add-opens java.base/java.lang.reflect=ALL-UNNAMED --add-opens java.base/java.lang.invoke=ALL-UNNAMED --add-opens java.base/java.io=ALL-UNNAMED --add-opens java.base/java.util=ALL-UNNAMED --add-opens java.base/java.util.concurrent=ALL-UNNAMED --add-opens java.base/java.util.jar=ALL-UNNAMED --add-opens java.base/java.security=ALL-UNNAMED --add-opens java.base/sun.reflect.generics.reflectiveObjects=ALL-UNNAMED</argLine>
<tagsToExclude>code.external,GetBanksPerf</tagsToExclude>
<!-- PropGatedPublicEndpoint: JsonSchemaValidationPublicPropTrueTest /
AuthenticationTypeValidationPublicPropTrueTest need
read_json_schema_validation_requires_role /
read_authentication_type_validation_requires_role forced true, which is baked
into Http4s400's ResourceDoc at object-init time; a value only one JVM can
observe. Excluded here by default (every ordinary shard/local/IDE run boots with
the props unset, i.e. false) and run separately with -Dtest.tagsToExclude=
-DtagsToInclude=PropGatedPublicEndpoint plus the two OBP_READ_*_REQUIRES_ROLE
env vars set true (see run_tests_parallel.sh and the CI job that runs them);
see the ${test.tagsToExclude} property declared above for why this is not a
plain literal. -->
<tagsToExclude>${test.tagsToExclude}</tagsToExclude>
<testFailureIgnore>${maven.test.failure.ignore}</testFailureIgnore>
</configuration>
<executions>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -1925,7 +1925,7 @@ curl -s https://api.example.com/obp/v5.1.0/resource-docs/v5.1.0/obp | \

**Method 3: Check Source Code**

Operation IDs are defined in the API implementation files (e.g., `APIMethods510.scala`):
Operation IDs are defined in the API implementation files (e.g., `Http4s510.scala`):

```scala
lazy val getBanks: OBPEndpoint = {
Expand Down Expand Up @@ -3234,7 +3234,6 @@ For comprehensive use case examples and implementation guides, see the dedicated

```xml
<!-- Enable specific components -->
<logger name="code.api.v5_1_0.APIMethods510" level="DEBUG"/>
<logger name="code.bankconnectors.Connector" level="TRACE"/>
<logger name="code.api.util.RateLimiting" level="DEBUG"/>
```
Expand Down
Loading
Loading