diff --git a/docs/reference/limits.md b/docs/reference/limits.md index ddbf590..680c7e7 100644 --- a/docs/reference/limits.md +++ b/docs/reference/limits.md @@ -19,6 +19,7 @@ second. | [Rate limit](#rate-limits) | `429` + `Retry-After` | `.../errors/rate-limit-exceeded` | Wait the seconds it names | | [Daily ingest quota](#daily-ingest-quotas) | `429` + `Retry-After` | `.../errors/ingest-quota-exceeded` | Wait until 00:00 UTC | | [Lifetime ceiling](#lifetime-ceilings) | `403`, no `Retry-After` | `.../errors/tenant-limit-reached` | Ask for it to be raised | +| [Graph transfer](#graph-transfer) | `400` on export, `413` on import | an `error` body, not a problem document | It does not: the component is too large to move as one file | Every `type` above is prefixed `https://intellistream.ai/errors/`, and the `429` and `413` bodies are [RFC 9457](https://www.rfc-editor.org/rfc/rfc9457) problem documents served as @@ -83,6 +84,7 @@ them not being real is not. | `POST /timeseries/data` | 16 MiB | | Everything else | 4 MiB | | `PUT /files` and `GET /files/download/**` | exempt, they stream | +| `POST /resources/import` and `GET /resources/export/{id}` | exempt, they stream; the [file format](#graph-transfer) has its own ceilings | ```json { @@ -97,6 +99,21 @@ them not being real is not. A `413` is **terminal**. The same request will never become acceptable by being sent again, so split the batch instead of retrying it. +## Graph transfer {#graph-transfer} + +[Export and import](./resources#graph-transfer) of a graph component have ceilings of their +own, fixed in the file format rather than set by the deployment: + +| Limit | Cap | Answered with | +| --- | --- | --- | +| Nodes in one component or file | 2 000 000 | `400` on export, `413` on import | +| Relationships in one component or file | 2 000 000 | `400` on export, `413` on import | +| Compressed file size on import | 512 MB | `413` | + +These answer with the `{ "error": { "code", "message" } }` body the resource endpoints use, +not a problem document, and none of them clears by waiting: a component over the cap cannot +be exported as one file at all. + ## Rate limits {#rate-limits} Counted per organisation and per user in a fixed one-minute window, with separate budgets for diff --git a/docs/reference/resources.md b/docs/reference/resources.md index ee20fbc..83e4364 100644 --- a/docs/reference/resources.md +++ b/docs/reference/resources.md @@ -775,6 +775,115 @@ let nearest = api.resources.fetch_nearest( +## Export and import a graph {#graph-transfer} + +Two endpoints move a whole connected component between tenants or environments as one file. +Export starts from one resource, walks outward with no depth limit, and writes every reachable +node and every relationship between them; import recreates them somewhere else. The file +references everything by `externalId` and never by numeric id, which is what makes it +portable: numeric ids are database identities and do not survive the transfer. + +| Endpoint | Body | Returns | +| --- | --- | --- | +| `GET /resources/export/{id}` | none; `id` is the numeric id of the resource to start from | The file, `application/octet-stream`, as an attachment named `.dhgraph` | +| `POST /resources/import` | the file, verbatim, as `application/octet-stream` | A JSON summary of what was created and what was skipped | + +The file is a gzip-compressed binary, streamed on the way out and decoded incrementally on the +way in, so neither side holds it whole in memory. Treat it as opaque: the format is versioned +by the server and is not part of the API contract. + +**What the file carries.** Per node: `externalId`, `name`, `description`, `source`, `isRoot`, +`labels`, `metadata`, the data set it belongs to (by that data set's `externalId`) and, for an +`ASSET`, a point `geoLocation`; other geometries are not carried. Per relationship: both +endpoints by `externalId`, the type, `description`, `metadata` and the data set. A data set +node inside the component is exported as a node like any other, ahead of everything that +references it. + +**What it does not carry.** A time-series's `unit` and `valueType`, so a time-series node in +the file cannot be created on import and is reported instead; datapoints; events; files. + +### Export {#graph-export} + +Export needs read access to the data set of the starting resource, and nothing more: the walk +is [gated on the starting node only](./datasets#access-control), so the file holds every node +the component reaches. A component of more than **2 000 000 nodes** or **2 000 000 +relationships** is refused with a `400` naming the limit, and nothing is exported partially. + +| Status | Meaning | +| --- | --- | +| `200` | The file. | +| `404` | No such resource, or the caller may not read it. | +| `400` | The component is over the export limit. | + +### Import {#graph-import} + +Import replays the file through the same pipeline as [create](#create-resources-and-relations), +so everything a create does, an import does: the [naming policy](./external-ids#the-naming-policy) +is applied, the data set ACLs are checked, and each committed segment is published to the +message bus and mirrored into the graph. The caller needs write access to every data set a +node or a relationship lands in, and the all-data-sets grant for a node that arrives with no +data set and for any `DATASET` node in the file. A denial is a `403`. + +**Skipped, not refused.** A node whose `externalId` already exists in the tenant is left as it +is, and so is a relationship already present between the same two endpoints with the same +type. Importing a file back into the tenant it came from is therefore a no-op, and +re-uploading after a failure is safe. Time-series nodes are skipped and listed by +`externalId`, and the relationships touching them are skipped with them; to keep those, create +the series through [`/timeseries`](./timeseries#create-a-series) first and import the same file +again. A data set reference is resolved by `externalId` against the data sets in the file and +those already in the tenant; one that resolves nowhere is dropped, and the node is created +without a data set. + +**Segments.** The upload is committed as it streams in, one transaction per 50 000 objects, +nodes first and relationships after, so memory stays flat however large the file. Each segment +is atomic on its own: a failure keeps the segments already committed and rejects the rest. +Because import skips what already exists, re-upload the same file once the cause is fixed and +it fast-forwards through the committed segments and resumes where it stopped. The response +counts the segments committed. + +```json +{ + "nodesCreated": 4210, + "relationsCreated": 4209, + "nodesSkippedExisting": 3, + "nodesSkippedTimeseries": ["pump_1_vibration", "pump_1_temperature"], + "relationsSkipped": 2, + "dataSetReferencesDropped": 0, + "segments": 1, + "warnings": [] +} +``` + +`warnings` carries [naming-policy warnings](./external-ids#the-naming-policy) exactly as a +create does, and a naming-policy refusal is the same `400` problem body a create returns. + +| Status | Meaning | +| --- | --- | +| `200` | The summary above, also when everything was skipped. | +| `400` | Not a readable graph file, a naming-policy refusal, or a value that failed validation. | +| `403` | A data set the caller may not write to. | +| `413` | Over a transfer limit: more than 2 000 000 nodes or relationships in the file, or a file larger than 512 MB. Nothing is imported. | + +Like a file upload, the import is exempt from the general +[request body size](./limits#request-body-size) cap and from the daily ingest byte quota: it +is consumed as it arrives, and the 512 MB ceiling is checked on the bytes as they stream in. +The nodes and relationships it creates count against their own daily quotas through the +ordinary create path. A reverse proxy in front of the API has to pass the upload through +unbuffered and uncapped, as the shipped nginx examples do for `/resources/import`. + +No client wraps the pair. Call them over HTTP with the bearer token the client already holds: + +```bash +curl -fsS -H "Authorization: Bearer $TOKEN" \ + -o plant_oslo.dhgraph "$API/resources/export/5677892" + +curl -fsS -H "Authorization: Bearer $TOKEN" \ + -H "Content-Type: application/octet-stream" \ + --data-binary @plant_oslo.dhgraph "$API/resources/import" +``` + +Against the [rate-limit](./limits#rate-limits) budget, export is a read and import a write. + ## The `/assets` endpoints {#assets} An **asset** is the node type that can be a navigation root and the only one that carries a @@ -850,6 +959,7 @@ gives you the same asset back. | Filter | `resources().filter` | `resources.filter` | `resources.filter` | | Traverse (`fetch-related`) | `resources().fetchRelated` | `resources.fetch_related` | `resources.fetch_related` | | Nearest N (`fetch-nearest`) | `resources().fetchNearest` | `resources.fetch_nearest` | `resources.fetch_nearest` | +| [Export / import a graph](#graph-transfer) | HTTP only | HTTP only | HTTP only | Relations have their own client surface in all three clients — `edges()` in Java, `edges` in Python and Rust. [Edges → client coverage](./edges#client-coverage)