Add archetype-validation-prober: an ingestion validation gate for Pub/Sub

[yerbis09]

Summary

Adds archetype-validation-prober, a standalone reference module that demonstrates an archetype validation gate in front of Cloud Pub/Sub.

The core principle: only what minimally fits the process should ever be enqueued. A single, versioned archetype (JSON Schema) validates and canonicalizes every payload at the gate before it is published:

• Accepted → canonicalized (charset + Unicode NFC) and safe to publish.
• Rejected → refused synchronously with machine-readable reason codes and never enqueued, so deterministic failures cannot bounce in the queue, inflate the dead-letter store, or burn retry resources.

---

What's included

• Archetype — the gate: canonicalize → syntactic → structural validation, cheapest check first.
• ValidationResult — accept (with canonical payload) / reject (with reason codes).
• ClassifyingReceiver — three-way delivery decision (accepted / transient / functional reject).
• ArchetypeValidationGateway — runnable entry point with a self-contained offline demo.
• archetype.schema.json — the canonical contract used by the demo.
• JUnit tests covering accept, each deterministic reject class, and NFD→NFC canonicalization.

---

Design rationale

Outcome	Nature	Caught where	Action
Conformant	Success	—	Publish / ack()
Wrong type / missing / bad enum / not JSON	Deterministic	Gate	Reject synchronously, never enqueue
Byte-different but equal (charset / Unicode NFD)	False reject	Gate	Normalize to NFC, accept
Downstream down (timeout, 5xx)	Transient	Consumer	nack() → backoff → dead-letter
State-dependent (unknown id, duplicate)	Unpredictable	Consumer	Quarantine / dead-letter

---

Python companion implementation

A Python port of this module is maintained at yerbis09/portfolioadvanced-llm as part of a broader GCP-native LLM project.

The Python implementation demonstrates why this validation pattern is even more natural in Python:

Concern	Java (this PR)	Python companion
Schema cache	30-line Singleton + Lock	@functools.cache — 1 line, thread-safe
Immutable config	Builder pattern	dataclass(frozen=True) + kwargs
Structured errors	Pre-formatted strings	ValidationError(path, code, message)
SDK decoupling	Interface + Adapter	Protocol — structural typing
Unicode NFC	3rd-party lib	unicodedata.normalize in stdlib
Lines of code	~800	~250

The Python port passes a review against the same design principles by Guido van Rossum, Raymond Hettinger, and Brett Cannon's documented Python idioms, and achieves 98.92% test coverage with ruff + SonarQube clean.

---

Build & test

cd archetype-validation-prober
mvn package   # produces target/pubsub-archetype-validation-prober.jar
mvn test
java -jar target/pubsub-archetype-validation-prober.jar   # offline demo

---

Notes

This is offered as a reference/demonstration module (provided as-is, no SLA), matching the intent of the existing probers in this repo. Feedback on scope and fit is very welcome — happy to adjust or relocate it if a different home is preferred.

[google-cla]

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

[yerbis09]

could you merge this PR?

[yerbis09]

Why an ingestion-gate validation prober

Async messaging fails silently. A malformed payload published to a topic
isn't rejected at publish time — it's accepted, fanned out to every
subscription, and only fails downstream in the subscriber, often minutes
later and N times over (once per delivery attempt, per subscriber). The
cost isn't one failure; it's subscribers × redeliveries failures for a
defect that was detectable in microseconds at the edge.

The existing probers validate delivery semantics — ordering-keys-prober
(ordering), exactly-once-prober (dedup), load-test-framework
(throughput). None validate payload admissibility before publish. That
gap means invalid data enters the system and is discovered by the most
expensive consumer possible: production subscribers at fan-out.

This prober closes the gate at the cheapest point: before publish().

[yerbis09]

Design decisions

Cheapest-check-first ordering (encoding → syntax → schema).
Validation is staged by cost and by dependency. Encoding (UTF-8 well-formedness)
is a byte-level scan — reject here and we never allocate a parser. Syntax
(JSON structural validity) gates schema. Schema (field/type constraints) is
the most expensive and runs last, only on inputs already known to be
decodable and parseable. Reordering these would waste work on inputs that a
cheaper check would have rejected. This is the same short-circuit discipline
the load-test-framework uses to keep per-message overhead flat under load.

NFC canonicalization before comparison.
Unicode allows the same grapheme to be encoded multiple ways (e.g. precomposed
é U+00E9 vs. e + combining accent U+0065 U+0301). Without normalization,
byte-equal-looking payloads compare unequal and identical strings fail schema
equality/enum checks — a false reject on valid data. Normalizing to NFC once,
up front, makes validation deterministic regardless of the producer's input
method or platform.

ClassifyingReceiver separates transient from functional failure.
A validation reject (bad payload) is functional — it will never succeed on
retry, so retrying wastes quota and delays the poison-message signal. A
Pub/Sub RPC error (DEADLINE_EXCEEDED, UNAVAILABLE) is transient — it
should retry with backoff. Collapsing these two into one error path either
retries un-retryable garbage or drops recoverable messages. The receiver
classifies the outcome so the caller applies the correct policy — mirroring
how exactly-once-prober distinguishes ack/nack outcomes from transport
errors.

[yerbis09]

Smoke tests against a real Pub/Sub topic

Unit tests prove the gate's logic in isolation. They do not prove it works
against a live topic — IAM, auth, endpoint resolution, and the actual
publish/pull round-trip are all unverified by mocks. The repo currently has
no smoke tests. This PR proposes the first.

What they test

• The gate can authenticate and resolve a real topic/subscription.
• A valid payload passes the gate and is publishable, then pulled back and byte-verified (round-trip integrity).
• An invalid payload (bad encoding / malformed JSON / schema violation) is rejected by the gate and never reaches publish().
• A transient RPC failure is classified as retryable, not as a validation reject (fault-injection or forced bad endpoint).

How to run

export GOOGLE_CLOUD_PROJECT=<project>
export PUBSUB_SMOKE_TOPIC=<topic>          # ephemeral, created/torn down by the test
export PUBSUB_SMOKE_SUBSCRIPTION=<sub>
mvn -Dtest=ArchetypeGateSmokeIT verify -Psmoke

Gated behind a -Psmoke Maven profile so they never run in the default unit build.
Topic/subscription are created and deleted per-run; the test is hermetic and leaves no residue.

What they prove
That the gate is not just logically correct but operationally correct — it holds the line against a real topic under real auth. That's the property a Pub/Sub operator actually cares about, and it's currently untested anywhere in this repo.

Happy to add ArchetypeGateSmokeIT in a follow-up commit if the reviewers want to see the smoke profile wired in before merge.

[yerbis09]

Python companion and the LLM-agent direction

Companion repo: yerbis09/portfolioadvanced-llm — a Python port of this gate (idiomatic, 98.92% line coverage, ruff + SonarQube clean, CI/CD with secrets scanning).

Why Python is the more natural host for this pattern.
The gate sits directly in front of LLM/agent ingestion, and that ecosystem — model clients, tokenizers, validation libraries, and the GCP Python SDKs — is Python-first. In Python the same staged gate is a thin composable pre-publish hook. Key differences from the Java module:

Concern	Java (this PR)	Python companion
Schema cache	30-line Singleton + Lock	@functools.cache — 1 line, thread-safe
Immutable config	Builder pattern	dataclass(frozen=True) + kwargs
Structured errors	Pre-formatted strings	ValidationError(path, code, message)
SDK decoupling	Interface + Adapter	Protocol — structural typing, no wrappers
Total lines	~800	~250

The bigger picture.
This gate is Phase 1 of a GCP-native LLM agent. Every message an agent ingests or emits — tool call, model output, inter-agent message — crosses a validation boundary before it hits a Pub/Sub topic. Agents amplify the async-failure problem: an unvalidated malformed tool result doesn't just fail one subscriber, it corrupts the agent's downstream reasoning and fans out across subscriptions. A cheap, deterministic admissibility gate at every publish point is what makes an agent mesh on Pub/Sub trustworthy.

This positions Pub/Sub not just as transport for LLM agents but as the layer where message validity is enforced — a concrete, testable piece of GCP's agent-infrastructure story.

[yerbis09]

Progress update:

• The real GCP smoke test is now documented and runnable with OAuth/ADC only.
• The README is still in English and now explicitly says the portfolio showcase page is not deployed yet.
• I removed the live portfolio link for now and left a placeholder note so we can publish it once the site is ready.
• The repo still has the full quality chain: secrets scan, Ruff, pytest, Sonar, Pub/Sub, BigQuery, and MCP server.

Next step: once the portfolio page is live, I can add the final public URL and a short showcase section.

[yerbis09]

Progress update:

• I removed the portfolio/web reference from the companion repo notes.
• The Google-facing PR branch now includes explicit smoke test instructions in archetype-validation-prober/README.md.
• The branch already contains the new Archetype.fromResource() factory and the smoke test class, so the PR now documents how to run the real GCP and emulator paths end to end.
• The smoke flow is still gated behind -Psmoke and uses developer-owned OAuth/ADC credentials only.

This keeps the Google PR self-contained and reviewable without depending on the portfolio site.

[yerbis09]

I added a companion summary to the PR branch that captures the larger Python/GCP direction behind this gate work:

• immutable validation objects and total validation in Python
• a real GCP flow with Pub/Sub, BigQuery, Storage, and MCP exposure
• real smoke validation using OAuth/ADC only
• no service account keys or secrets in the repo

This is meant to show how the Java prober idea can evolve into a more operationally complete workflow while keeping the upstream PR self-contained.

[yerbis09]

@jamiew

[jamiew]

@yerbis09 i think you are tagging wrong person