Fix/graph import streams - #41
Open
olavgg wants to merge 7 commits into
Open
Conversation
withoutAClientCertificateTheServerRejectsTheHandshake failed about nine runs in ten, so `./gradlew build` could not go green on any branch. It was not a broken build: the test was racing. It asserted that a client with no certificate is refused with an SSLHandshakeException, or something caused by one, or a message containing "SSL". That is a TLS 1.2 shape. Under TLS 1.3 the client finishes its side of the handshake before the server has looked at the certificate it never sent, so the refusal usually arrives on the first read instead, as "IOException: HTTP/1.1 header parser received no bytes". Both shapes occur and which one turns up is a matter of timing, which is exactly why it failed intermittently rather than always. The assertion is now just that the call fails. What mutual TLS has to guarantee is that an uncertificated client gets nothing back, not which layer reports it, and keystoreAndTruststoreCompleteTheMutualTlsHandshake against the same server already proves a failure here is the missing certificate rather than an unreachable server. The javadoc records both observed shapes so nobody tightens it back to one of them. Verified with 20 consecutive runs, previously 1 pass in 10. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Olav Gjerde <olav@intellistream.ai>
Until now nothing bounded what one caller could send. This puts a ceiling on every axis and answers each one with a status code a client can act on: what clears by waiting is a 429 with Retry-After, and nothing else is. - Field and batch caps in the API model: description, metadata, labels, related resources, GeoJSON size, datapoint value length, items per request, nodes and relations per create, datapoints per collection. Enforced on create and on update alike, so a cap cannot be grown past one append at a time. - Request body size, 4 MiB by default and 16 MiB for datapoint inserts, checked on Content-Length before the body is read and counted as a chunked body is consumed. Over the cap is a 413 problem document. The file upload and download stream and are exempt; the body-cache filter learns the same exemption, so a multi-gigabyte download is no longer held in memory and a raw upload is parsed straight off the stream. - Per-minute rate limits per organization and per user, with separate read and write budgets. A POST that only reads (filter, search, byids, list, count, check, fetch-related, fetch-nearest, aggregate, latest) is charged as a read. The MCP tools spend the same budget. - Daily ingest quotas per organization: events, nodes, relationships, datapoints and write-request bytes, reset at 00:00 UTC. - Lifetime ceilings on how large an organization may grow, answered with a 403 and no Retry-After. Off unless a deployment turns them on, since switching them on applies to every tenant at once. - Caps on live WebSocket connections per organization and per user, and on subscriptions multiplexed onto one socket; the first closes the socket with a reason frame, the second refuses only that subscription. Deployment-wide defaults live under datahub.limits. A one-row tenant_limits table in each tenant's schema (migration V38) overrides any of them for that tenant, live, within the cache TTL and with no restart. Jackson's stream constraints are raised so a legal body cannot be refused for one long string, and the OpenAPI description gains a Limits section. Around the API: the console shows a 413 or 429 as a message naming the limit rather than a generic failure; the Java SDK's ingest result tells a tenant-limit 403 apart from a permission one and never spools it; the nginx examples gain a per-IP request zone for traffic that arrives before a caller is identified, and keep their body ceiling above the API's own so an oversized request comes back as the platform's message rather than the proxy's error page; the systemd config comments say where to raise a limit for one tenant instead of the deployment. Two smaller changes ride along because they touch the same files: the console drops the Feign paths it no longer calls (the governance controllers, the label service, and the byids proxy, all of which the browser now reaches on the API directly), and fetch-nearest resolves an externalId the way fetch-related always has, instead of answering a 500 to a request its own form declares valid. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Olav Gjerde <olav@intellistream.ai>
…int ingestion Captures the plan to take TimeseriesService.insertDatapoints()/deleteDatapoints() off the per-request PostgreSQL read (unpooled SimpleDriverDataSource) by caching the small timeseries metadata the hot path needs in Valkey, including cache-key tenant scoping, the invalidation points on save/update/delete, and the write-ACL staleness considerations. Design note only, no behavior change. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Olav Gjerde <olav@intellistream.ai>
AGENTS.md states that every source file carries an SPDX identifier as its first line, and the Java tree does. The eight Python files did not, so the rule held everywhere except the one language where nothing enforced it. That matters more than tidiness here: the repository is public, and a file with no licence line is a file whose licence a reader has to infer from its neighbours. These are all platform tooling and demo seeding, so they take the AGPL header like the rest of the tree. Header only; no behaviour changes. Placed after the shebang where there is one, so the scripts stay executable. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Olav Gjerde <olav@intellistream.ai>
GraphEventNeo4jListener was replaced when resource mirroring moved to the Postgres outbox: the node and relationship properties it used to build are built in GraphNodeProperties now, and applying them is ResourceGraphApplier's job. Three comments still named it. PolicyService's publishPolicyUpsert carried more than a stale name. Its javadoc explained why the graph action had to be CREATE rather than UPDATE, and left a backlog note about teaching the consumer's UPDATE branch to accept full resources. There is no consumer and no action any more: the method queues one idempotent upsert, so the workaround and the follow-up are both gone. Comments only, no behaviour change. Signed-off-by: Olav Gjerde <olav@intellistream.ai>
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
…on message Code scanning flags an exception message written into an HTTP response as error information exposed to an external user. The two limit refusals, the daily quota 429 and the lifetime ceiling 403, did exactly that, in the RestControllerAdvice and in the body-size filter's own copy of the 429, through getMessage(). Nothing sensitive was in that message: both exceptions compose it from a metric name and a number the platform chose, and the sentence is the answer the caller is meant to read. But a scanner cannot tell a composed sentence from a leaked one when both travel through Throwable.getMessage(), and neither can the next person to add a catch block. LimitException now keeps the sentence as its own field behind detail(), and the advice and the filter read that. The message stays on the exception for logs, and the wire body is unchanged. Verified: LimitExceptionHandlerTest (new) pins the detail and the Retry-After on both refusals; RequestBodySizeLimitFilterTest and IngestQuotaServiceTest pass. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Olav Gjerde <olav@intellistream.ai>
POST /resources/import reads its file segment by segment off the raw
request stream and documents a 512 MB ceiling, but the two filters that
sit in front of every request did not know it: RequestBodySizeLimitFilter
refused anything over datahub.limits.max-body-bytes (4 MiB by default)
with a 413 before the controller ran, and CachingBodyFilter wrapped the
request in a ContentCachingRequestWrapper and the response in a
ContentCachingResponseWrapper, so an export was buffered whole in memory
on the way out. Both filters already exempted the file upload and
download through one shared list, StreamingFileEndpoints, precisely so
they could not disagree about what streams.
That list is now StreamingEndpoints and also matches POST /resources/import
and GET /resources/export/{id}. An import is therefore consumed as it
arrives, bounded only by the codec's own 512 MB and two-million-object
limits, and is not charged to the daily ingest byte quota, like a file
upload; the nodes and relationships it creates still count against theirs
through the ordinary create path. An export streams back without being
held in memory.
The nginx examples gain matching locations: /resources/import with no
body cap and no request buffering, /resources/export/ with response
buffering off, both with the /files timeouts. The README's api row now
lists them and says 20 MB for the REST body cap, which is what the site
files set; it said 64 MB.
Verified: RequestBodySizeLimitFilterTest (two new cases) and the new
StreamingEndpointsTest pass; nginx -t succeeds on both site files under
nginx 1.26 with a self-signed certificate and stubbed upstream hosts.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Olav Gjerde <olav@intellistream.ai>
olavgg
force-pushed
the
fix/graph-import-streams
branch
from
September 3, 2026 10:29
f2b129a to
a4a482a
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR depends on #40
What this changes
How it was verified
Checklist
git commit -s), per CONTRIBUTING.md./gradlew buildpasses(see AGENTS.md for which of the two documentation sites it belongs in)