From 766c68ef978bb25efccf6070959203fabdf907bc Mon Sep 17 00:00:00 2001 From: Olav Gjerde Date: Wed, 2 Sep 2026 14:05:48 +0200 Subject: [PATCH 1/2] docs: export and import a graph component as one file Platform branch feat/resource-graph-export-import adds GET /resources/export/{id} and POST /resources/import: a whole connected component, keyed by externalId, as one gzip-compressed binary file, with import replaying the file through the ordinary create pipeline in segments of 50 000 objects and skipping whatever already exists so a re-upload resumes where a failed one stopped. Nothing here covered it. Adds a section to the resources reference (what the file carries and does not, the access each side needs, the skip rules, the segment semantics, the summary body, the status codes, and a curl pair since no client wraps the endpoints), a row in the client-coverage table, and the transfer ceilings on the limits page. One caution worth reading before trusting the 512 MB figure: the import upload is not exempt from the general request-body cap, which defaults to 4 MiB and is deployment-wide, so that is what a default deployment accepts. Written from the filter code rather than the endpoint's own description. Verified: npm run build succeeds. Co-Authored-By: Claude Fable 5.1 Signed-off-by: Olav Gjerde --- docs/reference/limits.md | 21 +++++++ docs/reference/resources.md | 111 ++++++++++++++++++++++++++++++++++++ 2 files changed, 132 insertions(+) diff --git a/docs/reference/limits.md b/docs/reference/limits.md index ddbf590..0bb6750 100644 --- a/docs/reference/limits.md +++ b/docs/reference/limits.md @@ -19,6 +19,7 @@ second. | [Rate limit](#rate-limits) | `429` + `Retry-After` | `.../errors/rate-limit-exceeded` | Wait the seconds it names | | [Daily ingest quota](#daily-ingest-quotas) | `429` + `Retry-After` | `.../errors/ingest-quota-exceeded` | Wait until 00:00 UTC | | [Lifetime ceiling](#lifetime-ceilings) | `403`, no `Retry-After` | `.../errors/tenant-limit-reached` | Ask for it to be raised | +| [Graph transfer](#graph-transfer) | `400` on export, `413` on import | an `error` body, not a problem document | It does not: the component is too large to move as one file | Every `type` above is prefixed `https://intellistream.ai/errors/`, and the `429` and `413` bodies are [RFC 9457](https://www.rfc-editor.org/rfc/rfc9457) problem documents served as @@ -97,6 +98,26 @@ them not being real is not. A `413` is **terminal**. The same request will never become acceptable by being sent again, so split the batch instead of retrying it. +`POST /resources/import` is under the 4 MiB cap like everything else, even though the +[graph file format](./resources#graph-transfer) itself allows up to 512 MB. A deployment that +imports larger graphs raises `datahub.limits.max-body-bytes`, which is deployment-wide, and +the ceiling on any reverse proxy in front of it. + +## Graph transfer {#graph-transfer} + +[Export and import](./resources#graph-transfer) of a graph component have ceilings of their +own, fixed in the file format rather than set by the deployment: + +| Limit | Cap | Answered with | +| --- | --- | --- | +| Nodes in one component or file | 2 000 000 | `400` on export, `413` on import | +| Relationships in one component or file | 2 000 000 | `400` on export, `413` on import | +| Compressed file size on import | 512 MB | `413` | + +These answer with the `{ "error": { "code", "message" } }` body the resource endpoints use, +not a problem document, and none of them clears by waiting: a component over the cap cannot +be exported as one file at all. + ## Rate limits {#rate-limits} Counted per organisation and per user in a fixed one-minute window, with separate budgets for diff --git a/docs/reference/resources.md b/docs/reference/resources.md index ee20fbc..52e9140 100644 --- a/docs/reference/resources.md +++ b/docs/reference/resources.md @@ -775,6 +775,116 @@ let nearest = api.resources.fetch_nearest( +## Export and import a graph {#graph-transfer} + +Two endpoints move a whole connected component between tenants or environments as one file. +Export starts from one resource, walks outward with no depth limit, and writes every reachable +node and every relationship between them; import recreates them somewhere else. The file +references everything by `externalId` and never by numeric id, which is what makes it +portable: numeric ids are database identities and do not survive the transfer. + +| Endpoint | Body | Returns | +| --- | --- | --- | +| `GET /resources/export/{id}` | none; `id` is the numeric id of the resource to start from | The file, `application/octet-stream`, as an attachment named `.dhgraph` | +| `POST /resources/import` | the file, verbatim, as `application/octet-stream` | A JSON summary of what was created and what was skipped | + +The file is a gzip-compressed binary, streamed on the way out and decoded incrementally on the +way in, so neither side holds it whole in memory. Treat it as opaque: the format is versioned +by the server and is not part of the API contract. + +**What the file carries.** Per node: `externalId`, `name`, `description`, `source`, `isRoot`, +`labels`, `metadata`, the data set it belongs to (by that data set's `externalId`) and, for an +`ASSET`, a point `geoLocation`; other geometries are not carried. Per relationship: both +endpoints by `externalId`, the type, `description`, `metadata` and the data set. A data set +node inside the component is exported as a node like any other, ahead of everything that +references it. + +**What it does not carry.** A time-series's `unit` and `valueType`, so a time-series node in +the file cannot be created on import and is reported instead; datapoints; events; files. + +### Export {#graph-export} + +Export needs read access to the data set of the starting resource, and nothing more: the walk +is [gated on the starting node only](./datasets#access-control), so the file holds every node +the component reaches. A component of more than **2 000 000 nodes** or **2 000 000 +relationships** is refused with a `400` naming the limit, and nothing is exported partially. + +| Status | Meaning | +| --- | --- | +| `200` | The file. | +| `404` | No such resource, or the caller may not read it. | +| `400` | The component is over the export limit. | + +### Import {#graph-import} + +Import replays the file through the same pipeline as [create](#create-resources-and-relations), +so everything a create does, an import does: the [naming policy](./external-ids#the-naming-policy) +is applied, the data set ACLs are checked, and each committed segment is published to the +message bus and mirrored into the graph. The caller needs write access to every data set a +node or a relationship lands in, and the all-data-sets grant for a node that arrives with no +data set and for any `DATASET` node in the file. A denial is a `403`. + +**Skipped, not refused.** A node whose `externalId` already exists in the tenant is left as it +is, and so is a relationship already present between the same two endpoints with the same +type. Importing a file back into the tenant it came from is therefore a no-op, and +re-uploading after a failure is safe. Time-series nodes are skipped and listed by +`externalId`, and the relationships touching them are skipped with them; to keep those, create +the series through [`/timeseries`](./timeseries#create-a-series) first and import the same file +again. A data set reference is resolved by `externalId` against the data sets in the file and +those already in the tenant; one that resolves nowhere is dropped, and the node is created +without a data set. + +**Segments.** The upload is committed as it streams in, one transaction per 50 000 objects, +nodes first and relationships after, so memory stays flat however large the file. Each segment +is atomic on its own: a failure keeps the segments already committed and rejects the rest. +Because import skips what already exists, re-upload the same file once the cause is fixed and +it fast-forwards through the committed segments and resumes where it stopped. The response +counts the segments committed. + +```json +{ + "nodesCreated": 4210, + "relationsCreated": 4209, + "nodesSkippedExisting": 3, + "nodesSkippedTimeseries": ["pump_1_vibration", "pump_1_temperature"], + "relationsSkipped": 2, + "dataSetReferencesDropped": 0, + "segments": 1, + "warnings": [] +} +``` + +`warnings` carries [naming-policy warnings](./external-ids#the-naming-policy) exactly as a +create does, and a naming-policy refusal is the same `400` problem body a create returns. + +| Status | Meaning | +| --- | --- | +| `200` | The summary above, also when everything was skipped. | +| `400` | Not a readable graph file, a naming-policy refusal, or a value that failed validation. | +| `403` | A data set the caller may not write to. | +| `413` | Over a transfer limit: more than 2 000 000 nodes or relationships in the file, or a file larger than 512 MB. Nothing is imported. | + +:::caution The general request-body cap applies first +`POST /resources/import` is not exempt from the [request body size](./limits#request-body-size) +cap, which is 4 MiB unless the deployment raises `datahub.limits.max-body-bytes`. A larger +file is refused with that cap's own `413` (`.../errors/request-too-large`) before the import +reads it, so the 512 MB figure above is the format's ceiling, not what a default deployment +accepts. The upload's size also counts against the tenant's daily ingest byte quota. +::: + +No client wraps the pair. Call them over HTTP with the bearer token the client already holds: + +```bash +curl -fsS -H "Authorization: Bearer $TOKEN" \ + -o plant_oslo.dhgraph "$API/resources/export/5677892" + +curl -fsS -H "Authorization: Bearer $TOKEN" \ + -H "Content-Type: application/octet-stream" \ + --data-binary @plant_oslo.dhgraph "$API/resources/import" +``` + +Against the [rate-limit](./limits#rate-limits) budget, export is a read and import a write. + ## The `/assets` endpoints {#assets} An **asset** is the node type that can be a navigation root and the only one that carries a @@ -850,6 +960,7 @@ gives you the same asset back. | Filter | `resources().filter` | `resources.filter` | `resources.filter` | | Traverse (`fetch-related`) | `resources().fetchRelated` | `resources.fetch_related` | `resources.fetch_related` | | Nearest N (`fetch-nearest`) | `resources().fetchNearest` | `resources.fetch_nearest` | `resources.fetch_nearest` | +| [Export / import a graph](#graph-transfer) | HTTP only | HTTP only | HTTP only | Relations have their own client surface in all three clients — `edges()` in Java, `edges` in Python and Rust. [Edges → client coverage](./edges#client-coverage) From b32c6ecac5dfa1860146cec5672b0b4cfb38831d Mon Sep 17 00:00:00 2001 From: Olav Gjerde Date: Thu, 3 Sep 2026 09:56:59 +0200 Subject: [PATCH 2/2] docs: the graph import streams past the request-body cap Platform branch fix/graph-import-streams adds POST /resources/import and GET /resources/export/{id} to the streaming exemption the file endpoints already had, so the import is consumed as it arrives, bounded only by the file format's own 512 MB and two-million-object ceilings, and is not charged to the daily ingest byte quota. The shipped nginx examples gain matching locations. The previous commit on this branch documented the cap as it then stood. That caution is replaced by the rule as it is now. Co-Authored-By: Claude Fable 5.1 Signed-off-by: Olav Gjerde --- docs/reference/limits.md | 6 +----- docs/reference/resources.md | 13 ++++++------- 2 files changed, 7 insertions(+), 12 deletions(-) diff --git a/docs/reference/limits.md b/docs/reference/limits.md index 0bb6750..680c7e7 100644 --- a/docs/reference/limits.md +++ b/docs/reference/limits.md @@ -84,6 +84,7 @@ them not being real is not. | `POST /timeseries/data` | 16 MiB | | Everything else | 4 MiB | | `PUT /files` and `GET /files/download/**` | exempt, they stream | +| `POST /resources/import` and `GET /resources/export/{id}` | exempt, they stream; the [file format](#graph-transfer) has its own ceilings | ```json { @@ -98,11 +99,6 @@ them not being real is not. A `413` is **terminal**. The same request will never become acceptable by being sent again, so split the batch instead of retrying it. -`POST /resources/import` is under the 4 MiB cap like everything else, even though the -[graph file format](./resources#graph-transfer) itself allows up to 512 MB. A deployment that -imports larger graphs raises `datahub.limits.max-body-bytes`, which is deployment-wide, and -the ceiling on any reverse proxy in front of it. - ## Graph transfer {#graph-transfer} [Export and import](./resources#graph-transfer) of a graph component have ceilings of their diff --git a/docs/reference/resources.md b/docs/reference/resources.md index 52e9140..83e4364 100644 --- a/docs/reference/resources.md +++ b/docs/reference/resources.md @@ -864,13 +864,12 @@ create does, and a naming-policy refusal is the same `400` problem body a create | `403` | A data set the caller may not write to. | | `413` | Over a transfer limit: more than 2 000 000 nodes or relationships in the file, or a file larger than 512 MB. Nothing is imported. | -:::caution The general request-body cap applies first -`POST /resources/import` is not exempt from the [request body size](./limits#request-body-size) -cap, which is 4 MiB unless the deployment raises `datahub.limits.max-body-bytes`. A larger -file is refused with that cap's own `413` (`.../errors/request-too-large`) before the import -reads it, so the 512 MB figure above is the format's ceiling, not what a default deployment -accepts. The upload's size also counts against the tenant's daily ingest byte quota. -::: +Like a file upload, the import is exempt from the general +[request body size](./limits#request-body-size) cap and from the daily ingest byte quota: it +is consumed as it arrives, and the 512 MB ceiling is checked on the bytes as they stream in. +The nodes and relationships it creates count against their own daily quotas through the +ordinary create path. A reverse proxy in front of the API has to pass the upload through +unbuffered and uncapped, as the shipped nginx examples do for `/resources/import`. No client wraps the pair. Call them over HTTP with the bearer token the client already holds: