From efbbf97ed3df3f8af65b777b6d10fdf41e25f650 Mon Sep 17 00:00:00 2001 From: Olav Gjerde Date: Wed, 2 Sep 2026 14:05:48 +0200 Subject: [PATCH 1/7] docs: export and import a graph component as one file Platform branch feat/resource-graph-export-import adds GET /resources/export/{id} and POST /resources/import: a whole connected component, keyed by externalId, as one gzip-compressed binary file, with import replaying the file through the ordinary create pipeline in segments of 50 000 objects and skipping whatever already exists so a re-upload resumes where a failed one stopped. Nothing here covered it. Adds a section to the resources reference (what the file carries and does not, the access each side needs, the skip rules, the segment semantics, the summary body, the status codes, and a curl pair since no client wraps the endpoints), a row in the client-coverage table, and the transfer ceilings on the limits page. One caution worth reading before trusting the 512 MB figure: the import upload is not exempt from the general request-body cap, which defaults to 4 MiB and is deployment-wide, so that is what a default deployment accepts. Written from the filter code rather than the endpoint's own description. Verified: npm run build succeeds. Co-Authored-By: Claude Fable 5.1 Signed-off-by: Olav Gjerde --- docs/reference/limits.md | 21 +++++++ docs/reference/resources.md | 111 ++++++++++++++++++++++++++++++++++++ 2 files changed, 132 insertions(+) diff --git a/docs/reference/limits.md b/docs/reference/limits.md index ddbf590..0bb6750 100644 --- a/docs/reference/limits.md +++ b/docs/reference/limits.md @@ -19,6 +19,7 @@ second. | [Rate limit](#rate-limits) | `429` + `Retry-After` | `.../errors/rate-limit-exceeded` | Wait the seconds it names | | [Daily ingest quota](#daily-ingest-quotas) | `429` + `Retry-After` | `.../errors/ingest-quota-exceeded` | Wait until 00:00 UTC | | [Lifetime ceiling](#lifetime-ceilings) | `403`, no `Retry-After` | `.../errors/tenant-limit-reached` | Ask for it to be raised | +| [Graph transfer](#graph-transfer) | `400` on export, `413` on import | an `error` body, not a problem document | It does not: the component is too large to move as one file | Every `type` above is prefixed `https://intellistream.ai/errors/`, and the `429` and `413` bodies are [RFC 9457](https://www.rfc-editor.org/rfc/rfc9457) problem documents served as @@ -97,6 +98,26 @@ them not being real is not. A `413` is **terminal**. The same request will never become acceptable by being sent again, so split the batch instead of retrying it. +`POST /resources/import` is under the 4 MiB cap like everything else, even though the +[graph file format](./resources#graph-transfer) itself allows up to 512 MB. A deployment that +imports larger graphs raises `datahub.limits.max-body-bytes`, which is deployment-wide, and +the ceiling on any reverse proxy in front of it. + +## Graph transfer {#graph-transfer} + +[Export and import](./resources#graph-transfer) of a graph component have ceilings of their +own, fixed in the file format rather than set by the deployment: + +| Limit | Cap | Answered with | +| --- | --- | --- | +| Nodes in one component or file | 2 000 000 | `400` on export, `413` on import | +| Relationships in one component or file | 2 000 000 | `400` on export, `413` on import | +| Compressed file size on import | 512 MB | `413` | + +These answer with the `{ "error": { "code", "message" } }` body the resource endpoints use, +not a problem document, and none of them clears by waiting: a component over the cap cannot +be exported as one file at all. + ## Rate limits {#rate-limits} Counted per organisation and per user in a fixed one-minute window, with separate budgets for diff --git a/docs/reference/resources.md b/docs/reference/resources.md index ee20fbc..52e9140 100644 --- a/docs/reference/resources.md +++ b/docs/reference/resources.md @@ -775,6 +775,116 @@ let nearest = api.resources.fetch_nearest( +## Export and import a graph {#graph-transfer} + +Two endpoints move a whole connected component between tenants or environments as one file. +Export starts from one resource, walks outward with no depth limit, and writes every reachable +node and every relationship between them; import recreates them somewhere else. The file +references everything by `externalId` and never by numeric id, which is what makes it +portable: numeric ids are database identities and do not survive the transfer. + +| Endpoint | Body | Returns | +| --- | --- | --- | +| `GET /resources/export/{id}` | none; `id` is the numeric id of the resource to start from | The file, `application/octet-stream`, as an attachment named `.dhgraph` | +| `POST /resources/import` | the file, verbatim, as `application/octet-stream` | A JSON summary of what was created and what was skipped | + +The file is a gzip-compressed binary, streamed on the way out and decoded incrementally on the +way in, so neither side holds it whole in memory. Treat it as opaque: the format is versioned +by the server and is not part of the API contract. + +**What the file carries.** Per node: `externalId`, `name`, `description`, `source`, `isRoot`, +`labels`, `metadata`, the data set it belongs to (by that data set's `externalId`) and, for an +`ASSET`, a point `geoLocation`; other geometries are not carried. Per relationship: both +endpoints by `externalId`, the type, `description`, `metadata` and the data set. A data set +node inside the component is exported as a node like any other, ahead of everything that +references it. + +**What it does not carry.** A time-series's `unit` and `valueType`, so a time-series node in +the file cannot be created on import and is reported instead; datapoints; events; files. + +### Export {#graph-export} + +Export needs read access to the data set of the starting resource, and nothing more: the walk +is [gated on the starting node only](./datasets#access-control), so the file holds every node +the component reaches. A component of more than **2 000 000 nodes** or **2 000 000 +relationships** is refused with a `400` naming the limit, and nothing is exported partially. + +| Status | Meaning | +| --- | --- | +| `200` | The file. | +| `404` | No such resource, or the caller may not read it. | +| `400` | The component is over the export limit. | + +### Import {#graph-import} + +Import replays the file through the same pipeline as [create](#create-resources-and-relations), +so everything a create does, an import does: the [naming policy](./external-ids#the-naming-policy) +is applied, the data set ACLs are checked, and each committed segment is published to the +message bus and mirrored into the graph. The caller needs write access to every data set a +node or a relationship lands in, and the all-data-sets grant for a node that arrives with no +data set and for any `DATASET` node in the file. A denial is a `403`. + +**Skipped, not refused.** A node whose `externalId` already exists in the tenant is left as it +is, and so is a relationship already present between the same two endpoints with the same +type. Importing a file back into the tenant it came from is therefore a no-op, and +re-uploading after a failure is safe. Time-series nodes are skipped and listed by +`externalId`, and the relationships touching them are skipped with them; to keep those, create +the series through [`/timeseries`](./timeseries#create-a-series) first and import the same file +again. A data set reference is resolved by `externalId` against the data sets in the file and +those already in the tenant; one that resolves nowhere is dropped, and the node is created +without a data set. + +**Segments.** The upload is committed as it streams in, one transaction per 50 000 objects, +nodes first and relationships after, so memory stays flat however large the file. Each segment +is atomic on its own: a failure keeps the segments already committed and rejects the rest. +Because import skips what already exists, re-upload the same file once the cause is fixed and +it fast-forwards through the committed segments and resumes where it stopped. The response +counts the segments committed. + +```json +{ + "nodesCreated": 4210, + "relationsCreated": 4209, + "nodesSkippedExisting": 3, + "nodesSkippedTimeseries": ["pump_1_vibration", "pump_1_temperature"], + "relationsSkipped": 2, + "dataSetReferencesDropped": 0, + "segments": 1, + "warnings": [] +} +``` + +`warnings` carries [naming-policy warnings](./external-ids#the-naming-policy) exactly as a +create does, and a naming-policy refusal is the same `400` problem body a create returns. + +| Status | Meaning | +| --- | --- | +| `200` | The summary above, also when everything was skipped. | +| `400` | Not a readable graph file, a naming-policy refusal, or a value that failed validation. | +| `403` | A data set the caller may not write to. | +| `413` | Over a transfer limit: more than 2 000 000 nodes or relationships in the file, or a file larger than 512 MB. Nothing is imported. | + +:::caution The general request-body cap applies first +`POST /resources/import` is not exempt from the [request body size](./limits#request-body-size) +cap, which is 4 MiB unless the deployment raises `datahub.limits.max-body-bytes`. A larger +file is refused with that cap's own `413` (`.../errors/request-too-large`) before the import +reads it, so the 512 MB figure above is the format's ceiling, not what a default deployment +accepts. The upload's size also counts against the tenant's daily ingest byte quota. +::: + +No client wraps the pair. Call them over HTTP with the bearer token the client already holds: + +```bash +curl -fsS -H "Authorization: Bearer $TOKEN" \ + -o plant_oslo.dhgraph "$API/resources/export/5677892" + +curl -fsS -H "Authorization: Bearer $TOKEN" \ + -H "Content-Type: application/octet-stream" \ + --data-binary @plant_oslo.dhgraph "$API/resources/import" +``` + +Against the [rate-limit](./limits#rate-limits) budget, export is a read and import a write. + ## The `/assets` endpoints {#assets} An **asset** is the node type that can be a navigation root and the only one that carries a @@ -850,6 +960,7 @@ gives you the same asset back. | Filter | `resources().filter` | `resources.filter` | `resources.filter` | | Traverse (`fetch-related`) | `resources().fetchRelated` | `resources.fetch_related` | `resources.fetch_related` | | Nearest N (`fetch-nearest`) | `resources().fetchNearest` | `resources.fetch_nearest` | `resources.fetch_nearest` | +| [Export / import a graph](#graph-transfer) | HTTP only | HTTP only | HTTP only | Relations have their own client surface in all three clients — `edges()` in Java, `edges` in Python and Rust. [Edges → client coverage](./edges#client-coverage) From 0b02a8d73db4e0969d1d5feba00857194439c6e7 Mon Sep 17 00:00:00 2001 From: Olav Gjerde Date: Wed, 2 Sep 2026 14:06:43 +0200 Subject: [PATCH 2/7] docs: fetch-nearest starts from an external id too Platform branch fix/fetch-nearest-external-id makes POST /resources/fetch-nearest honour the externalId its form always declared; an externalId-only request used to reach the repository with a null id and come back as a 500. The page carried a caution saying the field was accepted but not read, and a table row saying numeric id only. Both are now wrong for the endpoint and for the Java form, so the caution goes and the row matches fetchRelated's. The Python and Rust clients still build the request from a numeric id alone, so the resolve-with-by_ids advice stays, scoped to them, and the example comments say which client the constraint belongs to rather than blaming the endpoint. Verified: npm run build succeeds. Co-Authored-By: Claude Fable 5.1 Signed-off-by: Olav Gjerde --- docs/reference/resources.md | 16 +++++++--------- 1 file changed, 7 insertions(+), 9 deletions(-) diff --git a/docs/reference/resources.md b/docs/reference/resources.md index 52e9140..294dd0d 100644 --- a/docs/reference/resources.md +++ b/docs/reference/resources.md @@ -719,7 +719,7 @@ connecting them back to the start. | Field | Default | Meaning | | --- | --- | --- | -| `id` | — | Where to start. **Numeric id only** — see below. | +| `id` / `externalId` | — | Where to start. Supply exactly one, as for `fetchRelated`. | | `endLabels` | — | Labels that qualify as a match, e.g. `["TIMESERIES"]`. The walk continues past them. | | `limit` | `10` | How many matching end-nodes to return. | | `relationshipTypes` | all | Which edge types the walk may follow. | @@ -729,18 +729,16 @@ That is the difference worth internalising: with `fetchRelated` you pick a radiu out what is inside it, which on an unfamiliar graph is a guess. With `fetch-nearest` you name what you are looking for and how many you want, and the radius follows. -:::caution `externalId` is accepted but not read -The request form carries an `externalId` field, but this endpoint starts from `id` only — -sending an external id alone gets you a `404`. Resolve it to a numeric id with `byIds` first. -`fetchRelated` takes either. -::: +The endpoint resolves an `externalId` for you, and so does the Java form. The Python and Rust +clients build the request from a numeric `id` only, so there resolve an external id with +`by_ids` first when that is all you have. ```java FetchNearestResourcesForm form = new FetchNearestResourcesForm(); -form.setId(5677892L); // numeric id, not external id +form.setExternalId("pump_1"); // or form.setId(5677892L) form.setEndLabels(List.of("TIMESERIES")); form.setLimit(10); form.setExcludedLabels(List.of("POLICY")); @@ -753,7 +751,7 @@ ResourceNetwork nearest = client.resources().fetchNearest(form); ```python nearest = client.resources.fetch_nearest( - 5677892, # numeric id, not external id + 5677892, # the Python client takes the numeric id end_labels=["TIMESERIES"], limit=10, excluded_labels=["POLICY"]) @@ -766,7 +764,7 @@ nearest = client.resources.fetch_nearest( use intellistream_datahub_sdk::resources::FetchNearestResourcesForm; let nearest = api.resources.fetch_nearest( - &FetchNearestResourcesForm::from_id(5677892) // numeric id, not external id + &FetchNearestResourcesForm::from_id(5677892) // the Rust client takes the numeric id .with_end_labels(vec!["TIMESERIES".into()]) .with_limit(10) .with_excluded_labels(vec!["POLICY".into()])).await?; From 0953f3f3ddacfb1e87c64a9f1aee494492dea1af Mon Sep 17 00:00:00 2001 From: Olav Gjerde Date: Wed, 2 Sep 2026 14:07:28 +0200 Subject: [PATCH 3/7] docs: an event's time cannot be updated Platform PR #325 (fix(api): event time cannot be updated, so remove it from the update form) takes eventTime out of the event update form. The columnar store partitions events by their time and a row cannot move between partitions, so the api used to accept the field, answer 200 echoing the new value, and then fail to apply it; the field is gone from the form, and with the strict request reader an update naming it is a 400. The update section still listed eventTime among the updatable fields, said it was set from an ISO-8601 string, and counted it among the fields setNull is refused on. All three are replaced by the rule as it stands: eventTime is fixed at creation, an update naming it is refused, and the way to fix a mis-timed event is to delete and rewrite it or to write a correcting event. Verified: npm run build succeeds. Co-Authored-By: Claude Fable 5.1 Signed-off-by: Olav Gjerde --- docs/reference/events.md | 22 ++++++++++++++-------- 1 file changed, 14 insertions(+), 8 deletions(-) diff --git a/docs/reference/events.md b/docs/reference/events.md index c96f1b0..a356fb5 100644 --- a/docs/reference/events.md +++ b/docs/reference/events.md @@ -642,7 +642,7 @@ this" be expressed distinctly from "leave it alone": | Verb | Applies to | Effect | | --- | --- | --- | | `set` | every field | Replace the value. | -| `setNull: true` | nullable fields only | Clear the value. `externalId`, `type` and `eventTime` are not nullable, so asking to clear any of them is a `400`. | +| `setNull: true` | nullable fields only | Clear the value. `externalId` and `type` are not nullable, so asking to clear either is a `400`. | | `add` | `metadata`, `relatedResources` | Merge entries in, keeping the rest. | | `remove` | the same collections | Take entries out, keeping the rest. A `relatedResources` entry matches on either side, so you can remove by `id` or by `externalId` whichever you have. | @@ -661,13 +661,19 @@ this" be expressed distinctly from "leave it alone": ``` Updatable fields are `externalId`, `description`, `type`, `subType`, `status`, `source`, -`dataSetId`, `metadata`, `eventTime` and `relatedResources`. `eventTime` is set from an -ISO-8601 string. Sending both `set` and `setNull` for one field is -a `400` — the request is contradictory, so it is refused rather than resolved by precedence. - -`setNull` is refused on the three fields a create cannot omit: `externalId`, `type` and -`eventTime`. Clearing `type` used to be accepted, and it left the event unreadable by any -client that models `type` as required, so the read failed rather than the write. `dataSetId` +`dataSetId`, `metadata` and `relatedResources`. Sending both `set` and `setNull` for one +field is a `400` — the request is contradictory, so it is refused rather than resolved by +precedence. + +`eventTime` is **fixed at creation** and is not an update field at all: an update naming it +is a `400` that names the field, the same answer as for any field the form does not have. The +store partitions events by their time, and a row cannot move between partitions. An event +recorded against the wrong moment is deleted and written again, or corrected by a follow-up +event, which the caution below recommends anyway. + +`setNull` is refused on `externalId` and `type`, the two fields a create cannot omit and an +update can name. Clearing `type` would leave the event unreadable by any client that models +`type` as required, so the write is refused rather than the read failing later. `dataSetId` is the one field here that genuinely is nullable: `setNull` detaches the event from its data set, and naming a `dataSetId` that no data set has is a `400` rather than a stored dangling reference. From d8e12ad3c19c845f631b6506c746c6907277cc71 Mon Sep 17 00:00:00 2001 From: Olav Gjerde Date: Wed, 2 Sep 2026 14:08:12 +0200 Subject: [PATCH 4/7] docs: a body naming a field the api does not have is a 400 Platform PR #324 (feat(api): reject request bodies naming fields the api does not have) switches the request-body reader from dropping unknown properties to refusing them: a 400 problem document of type .../errors/unreadable-request-body with one errors entry per offender, each a JSON Pointer plus the names its position accepts. Malformed JSON answers with the same type, and a line and column. Two pages here still described the old behaviour as a trap. events.md said a client sending the retired relatedResourceIds names got a 200 with its relations silently dropped; datasets.md said the api drops unknown keys, so a filter still carrying writeProtected looked like it was narrowing and was not. Both now say the request is refused and link to a new section on the client page that documents the response shape once, since it applies to every endpoint that reads a JSON body. Verified: npm run build succeeds. Co-Authored-By: Claude Fable 5.1 Signed-off-by: Olav Gjerde --- docs/reference/client.md | 34 ++++++++++++++++++++++++++++++++++ docs/reference/datasets.md | 5 +++-- docs/reference/events.md | 4 ++-- 3 files changed, 39 insertions(+), 4 deletions(-) diff --git a/docs/reference/client.md b/docs/reference/client.md index 2c663a1..387a0a5 100644 --- a/docs/reference/client.md +++ b/docs/reference/client.md @@ -447,3 +447,37 @@ Two responses are worth recognising by shape: Both shapes, and the rules behind them, are in [External ids & naming](./external-ids). + +### Unknown fields are refused {#unknown-fields} + +A request body naming a field the endpoint does not have is a `400`, not a silent success. A +typo, or a field that has since been retired, would otherwise be dropped and answered `200`, +telling you a change was applied when nothing happened. The body is an +[RFC 9457](https://www.rfc-editor.org/rfc/rfc9457) problem document of +`type: ".../errors/unreadable-request-body"`, with one `errors` entry per offender, each +located by a JSON Pointer and listing the names accepted at that position: + +```json +{ + "type": "https://intellistream.ai/errors/unreadable-request-body", + "title": "Bad Request", + "status": 400, + "detail": "Unknown field: eventTime", + "errors": [ + { + "detail": "Unknown field", + "pointer": "#/items/0/update/eventTime", + "allowedFields": ["dataSetId", "description", "externalId", "metadata", + "relatedResources", "source", "status", "subType", "type"] + } + ] +} +``` + +Every offender in the body is reported at once, at whatever depth it sits, so several stale +fields cost one round trip rather than one each. A body that cannot be parsed at all, +malformed JSON or a value of the wrong shape, answers with the same `type` and a `detail` +naming the problem, plus `line` and `column` where the parser can say. + +The clients only ever send fields they declare, so this reaches you when you build a body by +hand, or keep an old field name in one. diff --git a/docs/reference/datasets.md b/docs/reference/datasets.md index 0bcd83e..60b5f3f 100644 --- a/docs/reference/datasets.md +++ b/docs/reference/datasets.md @@ -157,8 +157,9 @@ let matches = api.datasets.filter(&DatasetFilter::from_filter(criteria)).await?; :::note There is no `writeProtected` or `deactivated` -Both were removed server-side as inert. The api drops unknown keys silently, so a filter still -carrying one looked like it was narrowing and was not. +Both were removed server-side as inert. A filter or a create still carrying either is refused +as an [unknown field](./client#unknown-fields), so a stale client fails on the first call +rather than filtering on nothing. ::: ## Search {#search} diff --git a/docs/reference/events.md b/docs/reference/events.md index a356fb5..4b3d982 100644 --- a/docs/reference/events.md +++ b/docs/reference/events.md @@ -56,8 +56,8 @@ HTTP caller should expect the quotes. `relatedResources` replaced a `relatedResourceIds` / `relatedResourceExternalIds` pair. The two were independent inputs and drifted: a mismatched pair was unioned into an event describing both resources, and a patch setting only the external ids left the stored ids stale. There are no -aliases, and events ignore unknown properties, so a client still sending the old field names gets -a `200` with its relations silently dropped. Java SDK users get a compile break on the removed +aliases: a client still sending the old field names is refused with a `400` naming them as +[unknown fields](./client#unknown-fields). Java SDK users get a compile break on the removed setters instead. Supply an `id`, an `externalId`, or both. The server resolves whichever side you left out and From ce79d37ab26bae86b714cf6ac4e977a8af86d13a Mon Sep 17 00:00:00 2001 From: Olav Gjerde Date: Wed, 2 Sep 2026 14:08:54 +0200 Subject: [PATCH 5/7] docs: say which licence the client libraries carry Platform PR #365 (chore: license the client libraries Apache-2.0) relicenses datahub-java-sdk and datahub-api-model from AGPL to Apache-2.0, aligning them with the Python and Rust SDKs, which were Apache-2.0 already. The reason is the one a developer evaluating the SDK needs to hear: a client library gets linked into someone else's application, and a copyleft one carries its terms into that application. Nothing on this site mentioned a licence at all, so a reader who checked the POM before this change would have chosen the REST API over the SDK, or something else. Adds a short section to the landing page: the SDKs and the wire model are Apache-2.0, the platform is AGPL-3.0, and which side each licence covers. Verified: npm run build succeeds. Co-Authored-By: Claude Fable 5.1 Signed-off-by: Olav Gjerde --- docs/intro.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/docs/intro.md b/docs/intro.md index 79ba64d..72ea2cb 100644 --- a/docs/intro.md +++ b/docs/intro.md @@ -26,6 +26,14 @@ datapoint, or jump to [high-throughput ingestion](/guides/ingest-timeseries) to - **Files** — attach documents and images to your assets. - **Subscriptions** — tail live data over a streaming connection. +## Licence + +The three SDKs, and the Java wire-contract model the Java SDK depends on, are +[Apache-2.0](https://www.apache.org/licenses/LICENSE-2.0). They were placed under that licence +so that linking one into your application carries no copyleft obligation into your code. The +platform they talk to is [AGPL-3.0](https://www.gnu.org/licenses/agpl-3.0.html); that licence +covers the server, and using the server over its API is not what it restricts. + ## Where to go next - **[Tutorial](/tutorial)** — build a small metrics agent end to end: create series, ingest on a schedule, and survive API outages. From 6dc16b38d19a710955c4fd119c25245fbe2160a8 Mon Sep 17 00:00:00 2001 From: Olav Gjerde Date: Wed, 2 Sep 2026 14:41:13 +0200 Subject: [PATCH 6/7] docs: language and precision review of reference, guides and advanced Apply the audience-language review to the reference pages, the guides and the advanced scenarios. docs/industries is untouched: another open branch rewrites those pages. Register: reference pages now describe what the API is rather than what it became. The "What changed" notes and the "used to", "now", "new" and "previously" narration are gone from events, resources, datasets, timeseries, edges, client, limits and external-ids. The one behaviour that breaks an existing request (search.name and search.description) keeps a one-line caution. Marketing phrases and long sentences were cut or split. Precision, checked against the platform and SDK sources of 2026-09-02: - Events page with cursor/nextCursor; the _ "after" field that external-ids documented does not exist. An undecodable cursor, or one sent under a different sort, is a 400 (malformed-cursor), not a restart from page one. The old relatedResourceIds field names are a 400 as unknown fields, not a silent 200. The filter limit defaults to 1 000. No event endpoint is HTTP-only. - /assets/search applies its filter block; the search limit spans all node types and POLICY nodes are searched; GET /functions/{id} answers 404 for an unreadable function; traversal nodes list the fields the graph mirror holds; relatedResources is empty on /resources/{id}, byids, filter and search. - Subscriptions: the listen path, the Bearer header on the upgrade, the wire frame (subscriptionExternalId, messages[].messageId, payload.eventAction, items[].datapoints with ISO-8601 UTC timestamps) and what SubscriptionRetriever filters on. - Time series: the RetrieveFilter fields, the four aggregate names, the granularity units, the read-side timestamp types, and that values cross the wire as strings. Python's Page is not a list subclass. - Units: the seeded Celsius id is temperature_deg_c; the unit object's fields; unit is free text and unitExternalId the catalogue key. - MCP: edge_create only upper-cases, edge_create_type and label_create snake-upper-case. Contributor content removed. - Files: the one server path rule, per-client id types, failure statuses, no size cap on PUT /files. Datasets: GET /datasets/policies and the grant cache TTLs. Client: PROJECT_NAME row dropped (read by every SDK, used by none); EntraID.md linked. Limits: /events/update and /events/delete do not enforce the items cap, so the page says so. Runnable examples: every non-industry guide and advanced page that reads data opens with the same "Needs a sandbox" banner, one tip follows it in one order, and pages that ended on a write now read it back. Predictive maintenance works at the seed's 1 Hz, with the window, the band and the alarm level derived from the healthy data; k-means uses k=2 over four assets; attach-files downloads the id it uploaded; ingestion builds its readings in Java and Rust and verifies the count; model-assets-graph creates the series it links and reads the graph back through fetchRelated; the seed helpers tolerate a 409 on a second run. Naming: "data set", "time series", "organization", "catalogue" and "relationship" in prose, outside identifiers, endpoint paths and code. Not done: docs/industries, which another open branch owns; the sweep of "edge" to "relationship" across the rest of resources.md and edges.md, since both pages are about the /edges surface and the change would touch most of their lines; the claim that sorting events by subType or status cannot be paged, which the code no longer supports but which needs a platform-side answer on nullable keyset boundaries; and the Rust SearchForm, which always serialises name and description as null and may trip the unknown-field rejection, an SDK matter rather than a docs one. Co-Authored-By: Claude Fable 5.1 Signed-off-by: Olav Gjerde --- docs/advanced/asset-health-score.mdx | 60 +++++------ docs/advanced/data-cleaning-lineage.mdx | 26 +++-- docs/advanced/demand-forecasting.mdx | 25 +++-- docs/advanced/fraud-classification.mdx | 24 +++-- docs/advanced/generate-sample-data.mdx | 14 ++- docs/advanced/kmeans-clustering.mdx | 16 +-- docs/advanced/lstm-anomaly-detection.mdx | 14 ++- docs/advanced/lstm-forecasting.mdx | 14 ++- docs/advanced/machine-learning-gently.mdx | 2 +- docs/advanced/oxygen-crash-early-warning.mdx | 25 +++-- docs/advanced/pca-process-monitoring.mdx | 13 ++- docs/advanced/predictive-maintenance.mdx | 61 +++++++---- docs/advanced/random-forest-soft-sensor.mdx | 15 ++- docs/advanced/sustained-alarm-window.mdx | 2 +- docs/advanced/xgboost-failure-prediction.mdx | 10 +- docs/guides/attach-files.mdx | 64 +++++++++-- docs/guides/correlate-alarms.mdx | 24 ++--- docs/guides/ingest-timeseries.mdx | 87 +++++++++++---- docs/guides/model-assets-graph.mdx | 105 ++++++++++++++----- docs/guides/query-and-aggregate.mdx | 22 ++-- docs/guides/realtime-subscriptions.mdx | 55 +++++++++- docs/guides/seed-a-sandbox.mdx | 17 ++- docs/guides/work-with-units.mdx | 29 +++-- docs/intro.md | 8 +- docs/mcp-server.mdx | 44 ++++---- docs/quickstart.mdx | 15 +-- docs/reference/client.md | 25 ++--- docs/reference/datasets.md | 64 ++++++----- docs/reference/edges.md | 45 ++++---- docs/reference/events.md | 58 ++++------ docs/reference/external-ids.md | 41 ++++---- docs/reference/files.md | 55 +++++++++- docs/reference/limits.md | 35 +++---- docs/reference/resources.md | 77 ++++++-------- docs/reference/subscriptions.md | 48 ++++++--- docs/reference/timeseries.md | 62 ++++++----- docs/reference/units.md | 22 +++- docs/tutorial.mdx | 4 +- 38 files changed, 825 insertions(+), 502 deletions(-) diff --git a/docs/advanced/asset-health-score.mdx b/docs/advanced/asset-health-score.mdx index 4d00892..a7e2ac4 100644 --- a/docs/advanced/asset-health-score.mdx +++ b/docs/advanced/asset-health-score.mdx @@ -20,23 +20,22 @@ can rank a whole fleet at a glance and a dashboard can show green/amber/red. It' lightweight cousin of [predictive maintenance](/advanced/predictive-maintenance): no model to train, just a transparent, tunable index. -:::tip New to machine learning? -No background needed. Skim the [gentle primer](/advanced/machine-learning-gently) for the -ideas in plain language — model, feature, training, and the algorithm itself — and use -[Generate sample data](/advanced/generate-sample-data) for a sandbox to run this against. -::: - -:::tip Need data to run this? +:::info Needs a sandbox This reads several pump signals. [Generate a sandbox](/advanced/generate-sample-data) first — section A ingests `pump_07_bearing_temp_c`, `pump_07_oil_pressure_kpa` and a stand-in `pump_07_vibration_anomaly` (or run [predictive maintenance](/advanced/predictive-maintenance) to produce the real one). ::: +:::tip New to machine learning? +No background needed. Skim the [gentle primer](/advanced/machine-learning-gently) for the +ideas in plain language — model, feature, training, and the algorithm itself. +::: + ## 1. Pull the latest value of each signal -Take the most recent reading (or short average) of each contributing series for the -asset. +Take the most recent reading of each contributing series for the asset. The seed writes +these hourly, so ask for the latest datapoint rather than a short window that may be empty. @@ -47,13 +46,8 @@ import intellistream_datahub_sdk, pandas as pd client = intellistream_datahub_sdk.DataHubClient.from_env() def latest(external_id): - rf = intellistream_datahub_sdk.RetrieveFilter( - ts=external_id, - start=pd.Timestamp.now(tz="UTC") - pd.Timedelta(minutes=15), - end=pd.Timestamp.now(tz="UTC"), - aggregates=["avg"], granularity="15m") - pts = client.timeseries.retrieve_datapoints(rf)[0].get_datapoints() - return float(pts[-1].average) + pts = client.timeseries.retrieve_latest_datapoints([external_id])[0].get_datapoints() + return float(pts[-1].value) signals = { "vibration": latest("pump_07_vibration_anomaly"), # 0..~1, from the anomaly model @@ -66,13 +60,13 @@ signals = { ```java -// The same retrieve, per signal; the scoring arithmetic below is shown in Python. +// Java has no latest-datapoint call: read the last day and take the newest point. +// The scoring arithmetic below is shown in Python. var filter = new RetrieveFilter(); filter.setExternalId("pump_07_bearing_temp_c"); -filter.setStart(ZonedDateTime.now().minusMinutes(15)); +filter.setStart(ZonedDateTime.now().minusHours(24)); filter.setEnd(ZonedDateTime.now()); -filter.setAggregates(List.of("avg")); -filter.setGranularity("15m"); +filter.setLimit(100); var request = new DataRetriever(); request.setItems(List.of(filter)); @@ -84,21 +78,14 @@ double bearingTemp = Double.parseDouble(pts.get(pts.size() - 1).getValue()); ```rust -// The same retrieve, per signal; the scoring arithmetic below is shown in Python. -use intellistream_datahub_sdk::generic::{DataWrapper, RetrieveFilter}; -use chrono::Utc; - -let filter = RetrieveFilter { - external_id: Some("pump_07_bearing_temp_c".into()), - start: Some(Utc::now() - chrono::Duration::minutes(15)), - end: Some(Utc::now()), - aggregates: Some(vec!["avg".into()]), - granularity: Some("15m".into()), - ..Default::default() -}; -let s = api.time_series - .retrieve_datapoints(&DataWrapper::from(vec![filter])).await? +// The latest datapoint, per signal; the scoring arithmetic below is shown in Python. +use intellistream_datahub_sdk::generic::{DataWrapper, IdAndExtId}; + +let latest = api.time_series + .retrieve_latest_datapoint(&DataWrapper::from(vec![ + IdAndExtId::from_external_id("pump_07_bearing_temp_c")])).await? .get_items().remove(0); +let bearing_temp = latest.datapoints.last().and_then(|p| p.value).unwrap_or(f64::NAN); ``` @@ -156,6 +143,11 @@ if band == "critical": event_time=pd.Timestamp.now(tz="UTC"), metadata={"asset": "pump_07", "score": str(score), "worst_signal": max(parts, key=parts.get)})]) + +# read the score back +stored = client.timeseries.retrieve_latest_datapoints(["pump_07_health_score"])[0].get_datapoints() +print(f"health {stored[-1].value:.1f} ({band})") +assert abs(stored[-1].value - score) < 0.01, "the score was not written" ``` Run it across the fleet on a schedule and you have a single ranked health view — diff --git a/docs/advanced/data-cleaning-lineage.mdx b/docs/advanced/data-cleaning-lineage.mdx index 03679b0..8ebed74 100644 --- a/docs/advanced/data-cleaning-lineage.mdx +++ b/docs/advanced/data-cleaning-lineage.mdx @@ -14,10 +14,14 @@ linked in the graph, so the whole `raw → … → result` chain is traceable bo **Stack:** the SDK for data and graph, plus `pandas`, `numpy`. ::: -:::tip New to this? -No data-science background needed — see the [gentle primer](/advanced/machine-learning-gently) -for any unfamiliar term, and [generate a sandbox](/advanced/generate-sample-data) to run -this against. +:::info Needs a sandbox +This reads two raw sensors that already exist. [Generate a sandbox](/advanced/generate-sample-data) +first — section K ingests `engine_temperature_raw` and `engine_vibration_raw`, defects and all. +::: + +:::tip New to machine learning? +No data-science background needed. Skim the [gentle primer](/advanced/machine-learning-gently) +for any unfamiliar term. ::: **The idea in one paragraph.** Real data never goes straight from sensor to model. It's @@ -90,8 +94,9 @@ how they *relate*. That's step 2. Model each transformation as a **function node** and link it up: each input series is `processed_by` the function, which `produces` an output series. Done across the pipeline, -this builds a branching, ten-deep lineage DAG in one call. (`resources.create` is the -same call in Java and Rust — see [model assets as a graph](/guides/model-assets-graph).) +this builds a branching lineage DAG of 18 nodes (8 functions and 10 series) in one call. +(`resources.create` is the same call in Java and Rust — see +[model assets as a graph](/guides/model-assets-graph).) ```python def fn(ext_id, name): @@ -127,7 +132,10 @@ client.resources.create(functions, edges) ``` That graph branches (each cleaned signal feeds two features), converges (four features -into one vector), and runs ten nodes deep from raw sensor to health score. +into one vector), and runs nine nodes deep from raw sensor to health score: +`engine_temperature_raw` → `clean_temp_fn` → `engine_temperature_clean` → `roll_mean_fn` → +`engine_temp_roll_mean` → `assemble_fn` → `engine_feature_vector` → `score_fn` → +`engine_health_score`. ## 3. The payoff — trace it backward @@ -137,8 +145,9 @@ ultimately came from. No guessing, no stale wiki page. ```python back = client.resources.fetch_related(external_id="engine_health_score", depth=12) -raw_sources = [n.external_id for n in back.nodes if n.external_id.endswith("_raw")] +raw_sources = {n.external_id for n in back.nodes if n.external_id.endswith("_raw")} print("this score derives from:", raw_sources) # engine_temperature_raw, engine_vibration_raw +assert raw_sources == {"engine_temperature_raw", "engine_vibration_raw"}, raw_sources ``` ## 4. The payoff — trace it forward (impact analysis) @@ -155,6 +164,7 @@ affected = [n.external_id for n in fwd.nodes print("affected by the bad temperature sensor:", affected) # engine_temperature_clean, engine_temp_roll_mean, engine_temp_roc, # engine_feature_vector, engine_health_score — but NOT the vibration features +assert "engine_health_score" in affected and "engine_vib_rms" not in affected, affected ``` This is the same [blast-radius traversal](/guides/correlate-alarms) used for asset diff --git a/docs/advanced/demand-forecasting.mdx b/docs/advanced/demand-forecasting.mdx index fa499c5..c3d7c08 100644 --- a/docs/advanced/demand-forecasting.mdx +++ b/docs/advanced/demand-forecasting.mdx @@ -19,17 +19,16 @@ hour. The pattern is the same regardless of domain — past values plus calendar predict the next ones — and the forecast becomes a **new series** you can chart against actuals and alert on. -:::tip New to machine learning? -No background needed. Skim the [gentle primer](/advanced/machine-learning-gently) for the -ideas in plain language — model, feature, training, and the algorithm itself — and use -[Generate sample data](/advanced/generate-sample-data) for a sandbox to run this against. -::: - -:::tip Need data to run this? +:::info Needs a sandbox This reads a load history that already exists. [Generate a sandbox](/advanced/generate-sample-data) first — section G ingests the `feeder_f12_load_mw` curve this build forecasts. ::: +:::tip New to machine learning? +No background needed. Skim the [gentle primer](/advanced/machine-learning-gently) for the +ideas in plain language — model, feature, training, and the algorithm itself. +::: + ## 1. Load the history Pull a long, regularly-sampled history of the quantity you want to forecast. @@ -50,7 +49,7 @@ rf = intellistream_datahub_sdk.RetrieveFilter( points = client.timeseries.retrieve_datapoints(rf)[0].get_datapoints() demand = pd.Series([float(p.average) for p in points], - index=pd.to_datetime([p.timestamp for p in points])).sort_index() + index=pd.to_datetime([p.timestamp for p in points], utc=True)).sort_index() demand = demand.asfreq("1h").interpolate() # regular hourly grid ``` @@ -166,6 +165,14 @@ client.timeseries.create([intellistream_datahub_sdk.TimeSeries( external_id="feeder_f12_load_mw_forecast", name="Feeder F12 load — 48h forecast", unit="mw", value_type="float")]) client.timeseries.insert_from_lists( timestamps=forecast.index, values=forecast.to_numpy(), ts="feeder_f12_load_mw_forecast") + +# read the horizon back: one point per forecast hour +stored = client.timeseries.retrieve_datapoints(intellistream_datahub_sdk.RetrieveFilter( + ts="feeder_f12_load_mw_forecast", + start=forecast.index[0], end=forecast.index[-1] + pd.Timedelta(minutes=1), + limit=1000))[0].get_datapoints() +print(f"{len(stored)} forecast hours stored") +assert len(stored) == HORIZON, f"expected {HORIZON}, got {len(stored)}" ``` With the forecast stored as a series, a [threshold rule](/guides/detect-events) on it @@ -184,7 +191,7 @@ actual, is set to exceed a limit, hours before it happens. ## Further reading - **Gradient boosting** — [Wikipedia](https://en.wikipedia.org/wiki/Gradient_boosting) · [scikit-learn](https://scikit-learn.org/stable/modules/ensemble.html#histogram-based-gradient-boosting) -- **Time-series forecasting** — [Wikipedia](https://en.wikipedia.org/wiki/Time_series) +- **Time series forecasting** — [Wikipedia](https://en.wikipedia.org/wiki/Time_series) - **The ideas in plain language** — [Machine learning, gently](/advanced/machine-learning-gently) ## See also diff --git a/docs/advanced/fraud-classification.mdx b/docs/advanced/fraud-classification.mdx index 031d554..1d98c65 100644 --- a/docs/advanced/fraud-classification.mdx +++ b/docs/advanced/fraud-classification.mdx @@ -13,8 +13,7 @@ combined with behavioural features, feeding a supervised classifier · **Stack:* for data and traversal, plus `networkx`, `pandas`, `scikit-learn`. ::: -This is the capstone: it fuses the two things the rest of the docs treat separately — the -**knowledge graph** and **machine learning**. Money laundering doesn't look suspicious one +Money laundering doesn't look suspicious one payment at a time; it looks suspicious in the *shape* of the network — funds fanning out through mules and looping back — combined with behaviour like rapid pass-through. A rules engine flags thousands of alerts a day; a classifier that scores each one by its network @@ -24,17 +23,16 @@ The technique generalises to any "score an entity by its connections plus its be problem — [wafer-lot risk](/industries/manufacturing-process/semiconductor), [insurance rings](/industries/financial-services/insurance-fraud), [telecom abuse](/industries/technology-operations/network). -:::tip New to machine learning? -No background needed. Skim the [gentle primer](/advanced/machine-learning-gently) for the -ideas in plain language — model, feature, training, and the algorithm itself — and use -[Generate sample data](/advanced/generate-sample-data) for a sandbox to run this against. -::: - -:::tip Need data to run this? +:::info Needs a sandbox This walks a transfer graph that already exists. [Generate a sandbox](/advanced/generate-sample-data) first — section F creates the flagged ring around `account_77310`. ::: +:::tip New to machine learning? +No background needed. Skim the [gentle primer](/advanced/machine-learning-gently) for the +ideas in plain language — model, feature, training, and the algorithm itself. +::: + ## 1. Pull the account's network For a given account, [walk the money-flow graph](/guides/correlate-alarms) to get its @@ -210,6 +208,14 @@ if risk > 0.7: "in_loop": str(feat["in_loop"]), "pass_through": f"{feat['pass_through']:.2f}", })]) + +# read the score back +pts = client.timeseries.retrieve_datapoints(intellistream_datahub_sdk.RetrieveFilter( + ts=f"{alert}_aml_risk", + start=pd.Timestamp.now(tz="UTC") - pd.Timedelta(minutes=5), + end=pd.Timestamp.now(tz="UTC")))[0].get_datapoints() +print(f"{len(pts)} risk score(s) stored, latest {pts[-1].value:.2f}") +assert len(pts) >= 1, "the risk score was not written" ``` The risk score is now a live series the investigations dashboard ranks on, and each diff --git a/docs/advanced/generate-sample-data.mdx b/docs/advanced/generate-sample-data.mdx index ead0398..3b5f6bd 100644 --- a/docs/advanced/generate-sample-data.mdx +++ b/docs/advanced/generate-sample-data.mdx @@ -23,8 +23,12 @@ import intellistream_datahub_sdk, numpy as np, pandas as pd client = intellistream_datahub_sdk.DataHubClient.from_env() def ingest(external_id, index, values, unit=None, name=None): - client.timeseries.create([intellistream_datahub_sdk.TimeSeries( - external_id=external_id, name=name or external_id, unit=unit or "value", value_type="float")]) + try: + client.timeseries.create([intellistream_datahub_sdk.TimeSeries( + external_id=external_id, name=name or external_id, unit=unit or "value", value_type="float")]) + except intellistream_datahub_sdk.DataHubException as e: + if e.status_code != 409: # a second run finds the series already there + raise client.timeseries.insert_from_lists(timestamps=index, values=np.asarray(values), ts=external_id) print(f"ingested {len(values):,} points → {external_id}") ``` @@ -34,11 +38,13 @@ def ingest(external_id, index, values, unit=None, name=None): For [predictive maintenance](/advanced/predictive-maintenance), [LSTM anomaly detection](/advanced/lstm-anomaly-detection) and [health scoring](/advanced/asset-health-score): a vibration signal that runs healthy, -then develops a fault — rising amplitude and impulsiveness over the final stretch. +then develops a fault — rising amplitude and impulsiveness over the final stretch. It is +written at **one reading a second**, which is the rate the predictive-maintenance page's +`SAMPLE_HZ` and window length assume. ```python def degrading_vibration(hours=24, fs_per_hour=3600): - n = hours * fs_per_hour + n = hours * fs_per_hour # one reading a second idx = pd.date_range(end=pd.Timestamp.now(tz="UTC"), periods=n, freq="1s") t = np.linspace(0, 1, n) healthy = 0.6 * np.sin(2 * np.pi * 50 * np.arange(n) / fs_per_hour) # base tone diff --git a/docs/advanced/kmeans-clustering.mdx b/docs/advanced/kmeans-clustering.mdx index ffdeb47..d1a8cf6 100644 --- a/docs/advanced/kmeans-clustering.mdx +++ b/docs/advanced/kmeans-clustering.mdx @@ -19,15 +19,14 @@ centre. That one idea answers three different operational questions, all covered (cohorts), and *which parts of the network belong together?* (communities). The trick is always the same: turn the thing you want to group into a **feature vector**. -:::tip Need data to run this? +:::info Needs a sandbox These steps read series (and a graph for section 3) that already exist. -[Generate a sandbox](/advanced/generate-sample-data) first. +[Generate a sandbox](/advanced/generate-sample-data) first: sections I and F. ::: :::tip New to machine learning? No background needed. Skim the [gentle primer](/advanced/machine-learning-gently) for the -ideas in plain language — model, feature, training, and the algorithm itself — and use -[Generate sample data](/advanced/generate-sample-data) for a sandbox to run this against. +ideas in plain language — model, feature, training, and the algorithm itself. ::: ## 1. Asset cohorts — group assets that behave alike @@ -60,7 +59,8 @@ assets = ["pump_07", "pump_08", "pump_11", "pump_19"] # ...your fleet X = pd.DataFrame([signature(a) for a in assets], index=assets) Xs = StandardScaler().fit_transform(X) -km = KMeans(n_clusters=4, n_init=10, random_state=0).fit(Xs) +# k must be well below the number of assets, or every asset is its own cluster +km = KMeans(n_clusters=2, n_init=10, random_state=0).fit(Xs) X["cohort"] = km.labels_ ``` @@ -119,6 +119,10 @@ for asset, d in zip(assets, dist): type="peer_outlier", status="open", event_time=pd.Timestamp.now(tz="UTC"), metadata={"asset": asset, "cohort": str(int(km.labels_[assets.index(asset)]))})]) + +outliers = list(client.events.filter(intellistream_datahub_sdk.EventFilter( + basic_filter=intellistream_datahub_sdk.BasicEventFilter(type="peer_outlier"), limit=100))) +print(f"{len(outliers)} peer outlier(s) on record") ``` ## 2. Operating regimes — group an asset's *states* @@ -191,7 +195,7 @@ the k that separates clusters best: ```python from sklearn.metrics import silhouette_score scores = {k: silhouette_score(Xs, KMeans(k, n_init=10, random_state=0).fit_predict(Xs)) - for k in range(2, 8)} + for k in range(2, len(Xs))} # the silhouette needs fewer clusters than points best_k = max(scores, key=scores.get) ``` diff --git a/docs/advanced/lstm-anomaly-detection.mdx b/docs/advanced/lstm-anomaly-detection.mdx index 4446b40..52ce6f3 100644 --- a/docs/advanced/lstm-anomaly-detection.mdx +++ b/docs/advanced/lstm-anomaly-detection.mdx @@ -20,15 +20,14 @@ unmistakable. An **LSTM autoencoder** learns to reconstruct that normal joint be when a real event arrives, it can't reconstruct it well, and the reconstruction error spikes. It needs **no examples of failure** — only normal operation. -:::tip Need data to run this? +:::info Needs a sandbox These steps read series that already exist. [Generate a sandbox](/advanced/generate-sample-data) first — section J ingests the correlated drilling channels (with an injected kick) this build watches. ::: :::tip New to machine learning? No background needed. Skim the [gentle primer](/advanced/machine-learning-gently) for the -ideas in plain language — model, feature, training, and the algorithm itself — and use -[Generate sample data](/advanced/generate-sample-data) for a sandbox to run this against. +ideas in plain language — model, feature, training, and the algorithm itself. ::: ## 1. Load the normal multivariate window @@ -51,7 +50,7 @@ def load(external_id, start, end): rf = intellistream_datahub_sdk.RetrieveFilter(ts=external_id, start=start, end=end, limit=100_000) pts = client.timeseries.retrieve_datapoints(rf)[0].get_datapoints() return pd.Series([float(p.value) for p in pts], - index=pd.to_datetime([p.timestamp for p in pts])).sort_index() + index=pd.to_datetime([p.timestamp for p in pts], utc=True)).sort_index() start = pd.Timestamp.now(tz="UTC") - pd.Timedelta(hours=1) # a stretch of normal running end = pd.Timestamp.now(tz="UTC") - pd.Timedelta(minutes=12) # ends before the recent kick @@ -164,6 +163,13 @@ if error[-3:].mean() > THRESHOLD: type="kick_detected", status="critical", event_time=score_index[-1], metadata={"rig": "rig_deepwater_1", "score": f"{error[-1]:.3f}", "model": "lstm_ae_v1"})]) + +# read the score back: one point per scored second +stored = client.timeseries.retrieve_datapoints(intellistream_datahub_sdk.RetrieveFilter( + ts="rig_dw1_anomaly_score", start=score_index[0], end=pd.Timestamp.now(tz="UTC"), + limit=10_000))[0].get_datapoints() +print(f"{len(stored)} scores stored, latest {stored[-1].value:.3f} against a limit of {THRESHOLD:.3f}") +assert len(stored) == len(error), f"expected {len(error)}, got {len(stored)}" ``` ## Where to take it further diff --git a/docs/advanced/lstm-forecasting.mdx b/docs/advanced/lstm-forecasting.mdx index 3650d01..de5a35e 100644 --- a/docs/advanced/lstm-forecasting.mdx +++ b/docs/advanced/lstm-forecasting.mdx @@ -21,15 +21,14 @@ where rate, pressure and water-cut interact over time. We'll forecast a well's oil rate, but the pattern is the same for any signal with memory. -:::tip Need data to run this? +:::info Needs a sandbox These steps read a series that already exists. [Generate a sandbox](/advanced/generate-sample-data) first — section B ingests the decline curve this build forecasts. ::: :::tip New to machine learning? No background needed. Skim the [gentle primer](/advanced/machine-learning-gently) for the -ideas in plain language — model, feature, training, and the algorithm itself — and use -[Generate sample data](/advanced/generate-sample-data) for a sandbox to run this against. +ideas in plain language — model, feature, training, and the algorithm itself. ::: ## 1. Load the history @@ -54,7 +53,7 @@ rf = intellistream_datahub_sdk.RetrieveFilter( pts = client.timeseries.retrieve_datapoints(rf)[0].get_datapoints() rate = pd.Series([float(p.average) for p in pts], - index=pd.to_datetime([p.timestamp for p in pts])).sort_index().asfreq("1d").interpolate() + index=pd.to_datetime([p.timestamp for p in pts], utc=True)).sort_index().asfreq("1d").interpolate() scaler = MinMaxScaler() scaled = scaler.fit_transform(rate.to_numpy().reshape(-1, 1)) @@ -155,6 +154,13 @@ client.timeseries.create([intellistream_datahub_sdk.TimeSeries( external_id="well_a12_oil_rate_forecast", name="Well A-12 oil-rate forecast", unit="bpd", value_type="float")]) client.timeseries.insert_from_lists( timestamps=index, values=forecast, ts="well_a12_oil_rate_forecast") + +# read the horizon back: one point per forecast day +stored = client.timeseries.retrieve_datapoints(intellistream_datahub_sdk.RetrieveFilter( + ts="well_a12_oil_rate_forecast", + start=index[0], end=index[-1] + pd.Timedelta(minutes=1), limit=1000))[0].get_datapoints() +print(f"{len(stored)} forecast days stored") +assert len(stored) == HORIZON, f"expected {HORIZON}, got {len(stored)}" ``` The decline curve is now a stored series — chart it against actuals, and a diff --git a/docs/advanced/machine-learning-gently.mdx b/docs/advanced/machine-learning-gently.mdx index 6eeee54..56c18c6 100644 --- a/docs/advanced/machine-learning-gently.mdx +++ b/docs/advanced/machine-learning-gently.mdx @@ -19,7 +19,7 @@ The SDK does the first and last parts (you already know those). The middle part "learn a pattern" bit — is what people call *machine learning*. Here's all you need to know about it to follow along. -## The five words that unlock everything +## Five terms - **Model** — a thing that has *learned a pattern* from past data and can apply it to new data. Think of it as a function you didn't write by hand; you showed it examples and it diff --git a/docs/advanced/oxygen-crash-early-warning.mdx b/docs/advanced/oxygen-crash-early-warning.mdx index d654433..4055dea 100644 --- a/docs/advanced/oxygen-crash-early-warning.mdx +++ b/docs/advanced/oxygen-crash-early-warning.mdx @@ -23,18 +23,17 @@ The technique — a classifier trained on a **forward-looking label** — genera "predict the incident ahead of time" problem: a transformer about to trip, a server about to breach SLO, a line about to jam. -:::tip New to machine learning? -No background needed. Skim the [gentle primer](/advanced/machine-learning-gently) for the -ideas in plain language — model, feature, training, and the algorithm itself — and use -[Generate sample data](/advanced/generate-sample-data) for a sandbox to run this against. -::: - -:::tip Need data to run this? +:::info Needs a sandbox This trains on pen-oxygen history. [Generate a sandbox](/advanced/generate-sample-data) first — section H ingests `pen_h_07_dissolved_oxygen_mg_l` and `pen_h_07_water_temp_c` with the crash episodes the model learns. ::: +:::tip New to machine learning? +No background needed. Skim the [gentle primer](/advanced/machine-learning-gently) for the +ideas in plain language — model, feature, training, and the algorithm itself. +::: + ## 1. Load the signals that precede a crash Pull the pen's history at a few-minute cadence — the oxygen itself plus the drivers that @@ -53,7 +52,7 @@ def load(external_id, days=90): aggregates=["avg"], granularity="5m", limit=100_000) pts = client.timeseries.retrieve_datapoints(rf)[0].get_datapoints() return pd.Series([float(p.average) for p in pts], - index=pd.to_datetime([p.timestamp for p in pts])).sort_index() + index=pd.to_datetime([p.timestamp for p in pts], utc=True)).sort_index() do = load("pen_h_07_dissolved_oxygen_mg_l") temp = load("pen_h_07_water_temp_c") @@ -96,7 +95,7 @@ from sklearn.metrics import classification_report split = int(len(frame) * 0.8) train, test = frame.iloc[:split], frame.iloc[split:] -# class_weight balances the rare positive class +# sample_weight up-weights the rare positive class sample_weight = np.where(train["will_crash"] == 1, 20.0, 1.0) clf = HistGradientBoostingClassifier(max_iter=300, learning_rate=0.05) clf.fit(train[FEATURES], train["will_crash"], sample_weight=sample_weight) @@ -126,6 +125,14 @@ if prob > 0.6: type="oxygen_crash_predicted", status="warning", event_time=latest.index[-1], metadata={"pen": "pen_h_07", "risk": f"{prob:.2f}", "lead_minutes": "45"})]) + +# read the risk back +pts = client.timeseries.retrieve_datapoints(intellistream_datahub_sdk.RetrieveFilter( + ts="pen_h_07_crash_risk", + start=latest.index[-1] - pd.Timedelta(minutes=1), + end=pd.Timestamp.now(tz="UTC")))[0].get_datapoints() +print(f"{len(pts)} risk value(s) stored, latest {pts[-1].value:.2f}") +assert len(pts) >= 1, "the risk score was not written" ``` The risk score is now a live series, and the event gives the farm a 45-minute head diff --git a/docs/advanced/pca-process-monitoring.mdx b/docs/advanced/pca-process-monitoring.mdx index 4863fe5..d6d5438 100644 --- a/docs/advanced/pca-process-monitoring.mdx +++ b/docs/advanced/pca-process-monitoring.mdx @@ -21,15 +21,14 @@ and reduces it to a few components; two statistics — **Hotelling's T²** and t prediction error (Q / SPE)** — then flag when live data no longer fits, and a contribution check points at the sensor responsible. -:::tip Need data to run this? +:::info Needs a sandbox These steps read correlated process sensors that already exist. [Generate a sandbox](/advanced/generate-sample-data) first — section C ingests them, fault and all. ::: :::tip New to machine learning? No background needed. Skim the [gentle primer](/advanced/machine-learning-gently) for the -ideas in plain language — model, feature, training, and the algorithm itself — and use -[Generate sample data](/advanced/generate-sample-data) for a sandbox to run this against. +ideas in plain language — model, feature, training, and the algorithm itself. ::: ## 1. Learn normal operation @@ -55,7 +54,7 @@ def series(tag, start, end): aggregates=["avg"], granularity="1m", limit=100_000) pts = client.timeseries.retrieve_datapoints(rf)[0].get_datapoints() return pd.Series([float(p.average) for p in pts], - index=pd.to_datetime([p.timestamp for p in pts])).sort_index() + index=pd.to_datetime([p.timestamp for p in pts], utc=True)).sort_index() s = pd.Timestamp.now(tz="UTC") - pd.Timedelta(days=14) # a clean stretch of normal running e = pd.Timestamp.now(tz="UTC") - pd.Timedelta(days=1) # ends before the recent fault window @@ -153,6 +152,12 @@ if t2(znew)[-1] > T2_LIMIT or spe(znew)[-1] > SPE_LIMIT: metadata={"unit": "crude_unit_1", "t2": f"{t2(znew)[-1]:.1f}", "spe": f"{spe(znew)[-1]:.1f}", "top_contributor": culprit})]) + +# the seeded fault sits in the last ten hours, so the deviation must be on record +found = list(client.events.filter(intellistream_datahub_sdk.EventFilter( + basic_filter=intellistream_datahub_sdk.BasicEventFilter(type="process_deviation"), limit=10))) +assert found, "the seeded fault should have raised a process_deviation event" +print(f"{len(found)} process deviation(s) on record, latest names {found[-1].metadata['top_contributor']}") ``` You can also publish T² and SPE as their own series for a live "process health" chart — diff --git a/docs/advanced/predictive-maintenance.mdx b/docs/advanced/predictive-maintenance.mdx index eb73299..5c73e30 100644 --- a/docs/advanced/predictive-maintenance.mdx +++ b/docs/advanced/predictive-maintenance.mdx @@ -13,13 +13,6 @@ vibration and warns you when it starts to drift · **Stack:** the SDK for data i plus three Python libraries — `numpy`, `scipy`, `scikit-learn`. ::: -:::tip New to this? -No machine-learning background needed. If a word here is unfamiliar, the -[gentle primer](/advanced/machine-learning-gently) explains it in one line. And to run -this end-to-end, [generate a sandbox](/advanced/generate-sample-data) first (section A -creates the vibration signal used below). -::: - **The idea in one paragraph.** A failing bearing vibrates differently long before it breaks — but not *louder*, so a simple "is it above X?" alarm misses it. The difference is in the *texture* of the vibration: tiny repeated shocks, energy shifting into certain @@ -27,12 +20,18 @@ frequencies. We'll let a model look at lots of examples of the machine running * learn what healthy texture looks like, and then raise a flag when new data stops looking like it. Nothing here needs labelled failures — only normal running. -The [quick examples](/industries/transport-logistics/aerospace) raise an alarm on a raw threshold; this is -the grown-up version that *learns* normal first. +The [quick examples](/industries/transport-logistics/aerospace) raise an alarm on a raw threshold; this +version learns normal first. -:::tip Need data to run this? +:::info Needs a sandbox These steps read a vibration series that already exists. [Generate a sandbox](/advanced/generate-sample-data) -first — section A ingests the degrading `pump_07_vibration_mm_s` this build scores. +first — section A ingests the degrading `pump_07_vibration_mm_s` this build scores, one reading +a second. +::: + +:::tip New to machine learning? +No background needed. If a word here is unfamiliar, the +[gentle primer](/advanced/machine-learning-gently) explains it in one line. ::: ## Step 1 — Get a stretch of healthy data @@ -52,7 +51,7 @@ client = intellistream_datahub_sdk.DataHubClient.from_env() def load_series(external_id, start, end): rf = intellistream_datahub_sdk.RetrieveFilter(ts=external_id, start=start, end=end, limit=100_000) points = client.timeseries.retrieve_datapoints(rf)[0].get_datapoints() - idx = pd.to_datetime([p.timestamp for p in points]) + idx = pd.to_datetime([p.timestamp for p in points], utc=True) return pd.Series([float(p.value) for p in points], index=idx).sort_index() healthy = load_series( @@ -103,8 +102,8 @@ let series = api.time_series ## Step 2 — Turn the raw wiggle into a few meaningful numbers A model can't learn much from one raw vibration value — it has no context. So we chop the -signal into short **windows** (say, two seconds each) and, for each window, calculate a -few numbers that capture its *texture*. These numbers are called **features**. We use +signal into short **windows** (two minutes each, at the seed's one reading a second) and, +for each window, calculate a few numbers that capture its *texture*. These numbers are called **features**. We use four, and here's what each one means in plain terms: - **RMS** — roughly the overall "loudness" or energy of the window. @@ -121,8 +120,8 @@ four, and here's what each one means in plain terms: from scipy.stats import kurtosis from scipy.fft import rfft, rfftfreq -SAMPLE_HZ = 1000 # the sensor reports 1000 readings per second -WINDOW = SAMPLE_HZ * 2 # work in 2-second windows +SAMPLE_HZ = 1 # the seed writes one reading per second +WINDOW = SAMPLE_HZ * 120 # work in 2-minute windows def features(window, fs=SAMPLE_HZ): x = window - window.mean() # remove the average so we see the wobble @@ -130,7 +129,7 @@ def features(window, fs=SAMPLE_HZ): crest = np.max(np.abs(x)) / rms if rms else 0.0 # biggest peak vs. average spectrum = np.abs(rfft(x)) # the FFT: how much of each frequency freqs = rfftfreq(len(x), 1 / fs) - band = spectrum[(freqs >= 120) & (freqs <= 180)] # the bearing's fault frequency range + band = spectrum[(freqs >= 0.1) & (freqs <= 0.4)] # the fault band; the healthy tone sits below it band_energy = float(np.sum(band**2)) return [rms, kurtosis(x), crest, band_energy] # four numbers describe this window @@ -142,8 +141,10 @@ def feature_matrix(series): X_healthy = feature_matrix(healthy) # one row of four features per window ``` -So each two-second window is now just four numbers. A week of healthy data becomes a big -table of "this is what normal looks like." +At one reading a second the highest frequency a window can resolve is 0.5 Hz, so the fault +band sits below that; a real 1 kHz sensor would put it at the bearing's fault frequencies +instead. Each two-minute window is now just four numbers. Eighteen hours of healthy data +become a table of 540 rows that says "this is what normal looks like." ## Step 3 — Let the model learn "normal" @@ -164,6 +165,9 @@ model.fit(scaler.transform(X_healthy)) # learn what heal def anomaly_score(X): # the model returns "how normal"; we flip the sign so higher = more abnormal return -model.decision_function(scaler.transform(X)) + +# the alarm level comes from the healthy data itself: the score 99% of normal windows stay under +ALARM = float(np.percentile(anomaly_score(X_healthy), 99)) ``` ## Step 4 — Score new data, and write the answer back @@ -181,18 +185,29 @@ X_recent = feature_matrix(recent) scores = anomaly_score(X_recent) score_index = recent.index[WINDOW::WINDOW][:len(scores)] # timestamp each window's score -client.timeseries.create([intellistream_datahub_sdk.TimeSeries( - external_id="pump_07_vibration_anomaly", name="Pump 07 vibration anomaly score", unit="score", value_type="float")]) +try: + client.timeseries.create([intellistream_datahub_sdk.TimeSeries( + external_id="pump_07_vibration_anomaly", name="Pump 07 vibration anomaly score", unit="score", value_type="float")]) +except intellistream_datahub_sdk.DataHubException as e: + if e.status_code != 409: # the seed's stand-in series may already be there + raise client.timeseries.insert_from_lists( timestamps=score_index, values=scores, ts="pump_07_vibration_anomaly") -ALARM = 0.04 # healthy scores sit near/below 0; a sustained positive score is abnormal -if scores[-5:].mean() > ALARM: # sustained, not a single spike +if scores[-5:].mean() > ALARM: # sustained over the last ten minutes, not a single spike client.events.create([intellistream_datahub_sdk.Event( external_id=f"degradation_predicted_pump_07_{int(pd.Timestamp.now().timestamp())}", type="degradation_predicted", status="open", event_time=score_index[-1], metadata={"asset": "pump_07", "score": f"{scores[-1]:.3f}", "model": "iforest_v1"})]) + +# read the scores back: the seed's fault ramps in over the last few hours, so the latest hour is abnormal +stored = client.timeseries.retrieve_datapoints(intellistream_datahub_sdk.RetrieveFilter( + ts="pump_07_vibration_anomaly", start=score_index[0], end=pd.Timestamp.now(tz="UTC"), + limit=1000))[0].get_datapoints() +print(f"{len(stored)} scores stored, latest {stored[-1].value:.3f} against an alarm level of {ALARM:.3f}") +assert len(stored) >= len(scores), f"expected {len(scores)} scores, found {len(stored)}" +assert scores[-5:].mean() > ALARM, "the seeded fault should score as abnormal" ``` That's the whole loop. The anomaly score is now a first-class series — chart it beside the diff --git a/docs/advanced/random-forest-soft-sensor.mdx b/docs/advanced/random-forest-soft-sensor.mdx index 6619751..c1b053d 100644 --- a/docs/advanced/random-forest-soft-sensor.mdx +++ b/docs/advanced/random-forest-soft-sensor.mdx @@ -20,15 +20,14 @@ the lab value in real time from the cheap online sensors that *are* measured eve second. Random Forest is a great fit — robust, little tuning, and an out-of-bag score that estimates accuracy without a separate test set. -:::tip Need data to run this? +:::info Needs a sandbox These steps read process sensors and sparse lab samples that already exist. [Generate a sandbox](/advanced/generate-sample-data) first — section D ingests both. ::: :::tip New to machine learning? No background needed. Skim the [gentle primer](/advanced/machine-learning-gently) for the -ideas in plain language — model, feature, training, and the algorithm itself — and use -[Generate sample data](/advanced/generate-sample-data) for a sandbox to run this against. +ideas in plain language — model, feature, training, and the algorithm itself. ::: ## 1. Assemble lab targets against sensor features @@ -49,7 +48,7 @@ def series(external_id, start, end, granularity="5m"): aggregates=["avg"], granularity=granularity, limit=100_000) pts = client.timeseries.retrieve_datapoints(rf)[0].get_datapoints() return pd.Series([float(p.average) for p in pts], - index=pd.to_datetime([p.timestamp for p in pts])).sort_index() + index=pd.to_datetime([p.timestamp for p in pts], utc=True)).sort_index() start, end = pd.Timestamp.now(tz="UTC") - pd.Timedelta(days=14), pd.Timestamp.now(tz="UTC") SENSORS = ["cdu_1_top_temp_c", "cdu_1_reflux_ratio", "cdu_1_feed_bpd", "cdu_1_pressure_kpa"] @@ -139,6 +138,14 @@ client.timeseries.create([intellistream_datahub_sdk.TimeSeries( unit="ppm", value_type="float")]) client.timeseries.insert_from_lists( timestamps=recent.index, values=virtual, ts="cdu_1_product_sulfur_ppm_soft") + +# read the virtual sensor back: one point per sensor row +stored = client.timeseries.retrieve_datapoints(intellistream_datahub_sdk.RetrieveFilter( + ts="cdu_1_product_sulfur_ppm_soft", + start=recent.index[0], end=recent.index[-1] + pd.Timedelta(minutes=1), + limit=1000))[0].get_datapoints() +print(f"{len(stored)} soft-sensor values stored, latest {stored[-1].value:.2f} ppm") +assert len(stored) == len(virtual), f"expected {len(virtual)}, got {len(stored)}" ``` When the next lab sample lands, append it to the training set and re-fit — the soft diff --git a/docs/advanced/sustained-alarm-window.mdx b/docs/advanced/sustained-alarm-window.mdx index eb52ade..39c6165 100644 --- a/docs/advanced/sustained-alarm-window.mdx +++ b/docs/advanced/sustained-alarm-window.mdx @@ -16,7 +16,7 @@ that is a question about a stretch of time. This page builds that rule end to en signal, replay it to prove the rule behaves, then run the same code against a live subscription. -:::info Seed the data first +:::info Needs a sandbox Run generator **L** on [Generate sample data](/advanced/generate-sample-data#l-a-noisy-alarm-four-transients-and-one-real-excursion). It writes `pump_p101_discharge_bar`: six hours at one reading a minute, four brief spikes well above the limit, and one eighty-minute excursion just above it. Seeding is Python, as diff --git a/docs/advanced/xgboost-failure-prediction.mdx b/docs/advanced/xgboost-failure-prediction.mdx index 05e185b..0d6c5f6 100644 --- a/docs/advanced/xgboost-failure-prediction.mdx +++ b/docs/advanced/xgboost-failure-prediction.mdx @@ -19,15 +19,14 @@ features, handles missing values natively, trains fast, and tells you which feat drive the prediction. We'll predict an electric submersible pump (ESP) failure seven days out, but the recipe fits any "label the history, predict the window" problem. -:::tip Need data to run this? +:::info Needs a sandbox These steps read series and failure events that already exist. [Generate a sandbox](/advanced/generate-sample-data) first — section E ingests the labelled failure history this build trains on. ::: :::tip New to machine learning? No background needed. Skim the [gentle primer](/advanced/machine-learning-gently) for the -ideas in plain language — model, feature, training, and the algorithm itself — and use -[Generate sample data](/advanced/generate-sample-data) for a sandbox to run this against. +ideas in plain language — model, feature, training, and the algorithm itself. ::: ## 1. Engineer one feature row per asset per day @@ -179,6 +178,11 @@ if risk > 0.6: event_time=pd.Timestamp.now(tz="UTC"), metadata={"pump": "pump_esp_a12", "risk": f"{risk:.2f}", "top_drivers": ", ".join(import_)})]) + +# read the risk back +stored = client.timeseries.retrieve_latest_datapoints(["pump_esp_a12_failure_risk"])[0].get_datapoints() +print(f"7-day failure risk {stored[-1].value:.2f}") +assert abs(stored[-1].value - risk) < 0.01, "the risk was not written" ``` ## Where to take it further diff --git a/docs/guides/attach-files.mdx b/docs/guides/attach-files.mdx index 03e4cb8..ded59b9 100644 --- a/docs/guides/attach-files.mdx +++ b/docs/guides/attach-files.mdx @@ -9,12 +9,14 @@ import TabItem from '@theme/TabItem'; Not everything is a number. A pump has a calibration certificate, a plant has a piping diagram, a daily run produces a CSV export. Store these as **files** — -organized in folders, tagged with metadata, and linked to the dataset they belong to. +organized in folders, tagged with metadata, and linked to the data set they belong to. ## Upload a file -Give the file a destination path and some metadata. The Java client uploads raw -bytes; the Python and Rust clients upload a local file to a destination folder. +Give the file a destination path and some metadata. The server stores one full path, name +included; the Java client sends the `path` you give it as-is, and the Python and Rust +clients join `destination_path` and `name` into the same thing. The Java client uploads raw +bytes; the Python and Rust clients read a local file. @@ -27,7 +29,7 @@ byte[] content = Files.readAllBytes(Path.of("calibration_a12.pdf")); var uploaded = client.files().upload( FileUploadRequest.builder() - .path("certificates/2026/calibration_a12.pdf") // destination + .path("/certificates/2026/calibration_a12.pdf") // destination, name included .content(content) .contentType("application/pdf") .externalId("calibration_pump_esp_a12") @@ -72,10 +74,11 @@ let uploaded = api.files.upload_file(upload).await?; -:::tip Link a file to its asset -Set `dataSetId` (or the `external_id`/metadata) so a file is discoverable from the -dataset and asset it documents — the calibration certificate next to the pump it -certifies. +:::tip Link a file to what it documents +Set `dataSetId` so a file is discoverable from the data set it documents. To point it at the +resource it documents, the calibration certificate at the pump it certifies, the Python and +Rust clients take `related_resources`, a list of numeric resource ids; the Java client has no +field for it. ::: ## List a directory @@ -90,15 +93,24 @@ path and size. var listing = client.files().list("/certificates/2026"); listing.getItems().forEach(node -> System.out.println(node.getType() + " " + node.getName() + " " + node.getSize() + " bytes")); + +boolean present = listing.getItems().stream() + .anyMatch(node -> "calibration_a12.pdf".equals(node.getName())); +if (!present) { + throw new AssertionError("upload not found in /certificates/2026"); +} ``` ```python -for node in client.files.list_directory_by_path("/certificates/2026"): +listing = client.files.list_directory_by_path("/certificates/2026") +for node in listing: print(node.type, node.name, node.size) +assert "calibration_a12.pdf" in [node.name for node in listing], "upload not found" + roots = client.files.list_root_directory() # top of the tree ``` @@ -110,6 +122,9 @@ let listing = api.files.list_directory_by_path("/certificates/2026").await?; for node in listing.get_items() { println!("{} {} {} bytes", node.r#type.as_deref().unwrap_or(""), node.name, node.size); } + +assert!(listing.get_items().iter().any(|node| node.name == "calibration_a12.pdf"), + "upload not found in /certificates/2026"); ``` @@ -117,13 +132,40 @@ for node in listing.get_items() { ## Download -The Java client downloads a file's raw bytes by id: +Download by the id the upload echoed. Python and Rust take it as an integer, Java as a +string (the endpoint also accepts an external id in that position). + + + ```java -byte[] bytes = client.files().download("99"); +long id = uploaded.getItems().iterator().next().getId(); +byte[] bytes = client.files().download(String.valueOf(id)); Files.write(Path.of("calibration_a12.pdf"), bytes); ``` + + + +```python +from pathlib import Path + +download = client.files.download(uploaded[0].id) +Path("calibration_a12.pdf").write_bytes(download.content) +``` + + + + +```rust +let id = uploaded.get_items()[0].id.expect("the upload echo carries the id"); +let download = api.files.download(id).await?; +std::fs::write("calibration_a12.pdf", &download.bytes)?; +``` + + + + ## Delete diff --git a/docs/guides/correlate-alarms.mdx b/docs/guides/correlate-alarms.mdx index 03dced6..7911a0b 100644 --- a/docs/guides/correlate-alarms.mdx +++ b/docs/guides/correlate-alarms.mdx @@ -9,8 +9,7 @@ import TabItem from '@theme/TabItem'; Two alarms fire within seconds of each other. Are they one incident or two? A flat lookup can't tell you — but the **relationship graph** can. If both alarmed sensors -sit under the same subsystem, it's almost certainly one root cause, not two -coincidences. +sit under the same subsystem, the two alarms share a subsystem. `fetchRelated` walks the graph outward from a node and returns the connected sub-graph. Walk the neighbourhood of each alarm and intersect the two: a shared @@ -29,9 +28,10 @@ chain; a single-hop read stops at the first neighbour. ## 1. From alarms to resources -An alarm is an [event](./detect-events.mdx) that references the resource it concerns -(`relatedResources`). Start from the two resources the alarms point at — -here `sensor_a` and `sensor_b`. +An alarm is an [event](./detect-events.mdx) that names the resource it concerns. The two +alarms the sandbox seeds carry the sensor in `metadata.resource`, so the ids are given here: +`sensor_a` and `sensor_b`. In production an alarm carries the resource in `relatedResources`, +and a read of the event gives you the id. ## 2. Walk each alarm's neighbourhood @@ -104,8 +104,9 @@ Set shared = nb.nodes().stream() shared.remove("sensor_a"); shared.remove("sensor_b"); -if (!shared.isEmpty()) { - System.out.println("Both alarms are part of: " + shared); // e.g. [cooling_system] +System.out.println("Both alarms are part of: " + shared); // [cooling_system, skid_1] +if (!shared.contains("cooling_system")) { + throw new AssertionError("expected cooling_system, found " + shared); } ``` @@ -117,8 +118,8 @@ a_nodes = {n.external_id for n in na.nodes} shared = {n.external_id for n in nb.nodes} & a_nodes shared -= {"sensor_a", "sensor_b"} -if shared: - print("Both alarms are part of:", shared) # e.g. {'cooling_system'} +print("Both alarms are part of:", shared) # {'cooling_system', 'skid_1'} +assert "cooling_system" in shared, f"expected cooling_system, found {shared}" ``` @@ -135,9 +136,8 @@ let mut shared: HashSet<&str> = nb.nodes().iter() shared.remove("sensor_a"); shared.remove("sensor_b"); -if !shared.is_empty() { - println!("Both alarms are part of: {:?}", shared); // e.g. {"cooling_system"} -} +println!("Both alarms are part of: {:?}", shared); // {"cooling_system", "skid_1"} +assert!(shared.contains("cooling_system"), "expected cooling_system, found {:?}", shared); ``` diff --git a/docs/guides/ingest-timeseries.mdx b/docs/guides/ingest-timeseries.mdx index 385efde..e0a63bf 100644 --- a/docs/guides/ingest-timeseries.mdx +++ b/docs/guides/ingest-timeseries.mdx @@ -8,18 +8,20 @@ import TabItem from '@theme/TabItem'; # High-throughput ingestion A common job: push a steady, high-volume stream of sensor readings in so a dashboard -or alerting rule can act on them. The SDK does the heavy lifting — it **chunks the -data into batches and sends them concurrently**, retrying transient failures. - -:::note How it works -Datapoints are split into batches (default **10,000** per request — the store is -optimised for large batches), sent concurrently up to a bounded in-flight limit, and -transient failures (HTTP 429/5xx, network) are retried. The Java client returns an -`IngestResult` summarising what landed and what didn't. +or alerting rule can act on them. The SDK **chunks the data into batches and sends them +concurrently**, retrying transient failures. + +:::info Needs a sandbox +This guide writes into `engine_temperature`, which section **A** of +[Seed a sandbox](/guides/seed-a-sandbox) creates. ::: -:::caution Stay inside the ingest caps -The defaults already fit, so this only matters if you have tuned something: +:::note How it works, and the caps it fits +Datapoints are split into batches of **10,000** per request (the store is optimised for +large batches), sent concurrently up to a bounded in-flight limit, and transient failures +(HTTP 429/5xx, network) are retried. The Java client returns an `IngestResult` summarising +what landed and what didn't. The defaults already fit the caps, so the table only matters if +you have tuned something: | Cap | Value | | --- | --- | @@ -41,9 +43,19 @@ and both retried for you. See [Limits & quotas](/reference/limits). `ingest` groups datapoints by series external id and returns an `IngestResult`. ```java +import java.time.Instant; +import java.util.*; + var client = DatahubClient.fromEnv(); -// readings: Map> grouped by time-series external id +// a million readings, one per second from the start of 2026 +Instant start = Instant.parse("2026-01-01T00:00:00Z"); +List points = new ArrayList<>(1_000_000); +for (int i = 0; i < 1_000_000; i++) { + points.add(Datapoint.of(start.plusSeconds(i), 90 + 5 * Math.random())); +} +Map> readings = Map.of("engine_temperature", points); + IngestResult result = client.timeseries().ingest(readings, IngestOptions.builder() .batchSize(10_000) // datapoints per request @@ -52,6 +64,10 @@ IngestResult result = client.timeseries().ingest(readings, .build()); System.out.printf("ingested %,d, failed %,d%n", result.succeeded(), result.failed()); +if (!result.isComplete()) { + result.errors().forEach(e -> System.err.println(e.statusCode() + ": " + e.message())); + throw new IllegalStateException("ingest incomplete"); +} ``` @@ -60,34 +76,67 @@ System.out.printf("ingested %,d, failed %,d%n", result.succeeded(), result.faile `insert_from_lists` takes whole arrays (NumPy / pandas) and batches them for you. ```python -import numpy as np, pandas as pd +import intellistream_datahub_sdk, numpy as np, pandas as pd -client = DataHubClient.from_env() +client = intellistream_datahub_sdk.DataHubClient.from_env() +# a million readings, one per second from the start of 2026 client.timeseries.insert_from_lists( timestamps=pd.date_range("2026-01-01", periods=1_000_000, freq="s", tz="UTC"), - values=np.random.rand(1_000_000), + values=90 + 5 * np.random.rand(1_000_000), ts="engine_temperature") + +# read the first hour back: 3 600 readings +rf = intellistream_datahub_sdk.RetrieveFilter( + ts="engine_temperature", + start=pd.Timestamp("2026-01-01 00:00", tz="UTC"), + end=pd.Timestamp("2026-01-01 01:00", tz="UTC"), + limit=10_000) +points = client.timeseries.retrieve_datapoints(rf)[0].get_datapoints() +print(f"{len(points)} readings in the first hour") +assert len(points) >= 3600, len(points) ``` -`insert_datapoints` auto-batches large inputs (chunks above ~100k points). +`insert_datapoints` auto-batches large inputs (chunks at the 100 000-point collection cap). ```rust +use chrono::{Duration, TimeZone, Utc}; use intellistream_datahub_sdk::create_api_service; -use intellistream_datahub_sdk::generic::{DataWrapper, DatapointsCollection}; +use intellistream_datahub_sdk::generic::{DataWrapper, DatapointString, DatapointsCollection, RetrieveFilter}; let api = create_api_service(); +// a million readings, one per second from the start of 2026 +let start = Utc.with_ymd_and_hms(2026, 1, 1, 0, 0, 0).unwrap(); +let readings: Vec = (0..1_000_000) + .map(|i| DatapointString::from_datetime( + start + Duration::seconds(i), + &format!("{:.2}", 90.0 + (i % 50) as f64 / 10.0))) + .collect(); + let mut dw = DataWrapper::new(); dw.add_item(DatapointsCollection { external_id: Some("engine_temperature".into()), - datapoints: readings, // Vec + datapoints: readings, ..Default::default() }); api.time_series.insert_datapoints(&mut dw).await?; + +// read the first hour back: 3 600 readings +let first_hour = RetrieveFilter { + external_id: Some("engine_temperature".into()), + start: Some(start), + end: Some(start + Duration::hours(1)), + limit: Some(10_000), + ..Default::default() +}; +let points = api.time_series.retrieve_datapoints(&DataWrapper::from(vec![first_hour])).await?; +let n = points.get_items()[0].datapoints.len(); +println!("{n} readings in the first hour"); +assert!(n >= 3600, "expected 3600 readings, got {n}"); ``` @@ -105,12 +154,12 @@ The `ingest` knobs let you trade throughput against load on the server: | `failFast` | `false` | Abort on the first failed batch instead of collecting errors. | When `failFast` is off, inspect `result.errors()` for the per-batch failures. See the -[Time-series reference](/reference/timeseries#ingestresult) for the full result shape. +[Time series reference](/reference/timeseries#ingestresult) for the full result shape. ## Then chart the trend Once the data is in, roll it up to hourly or daily buckets for a dashboard — see -[Query & aggregate time-series](./query-and-aggregate.mdx). +[Query & aggregate time series](./query-and-aggregate.mdx). :::warning Ordering Batches are sent in parallel, so there is **no cross-batch ordering guarantee** — diff --git a/docs/guides/model-assets-graph.mdx b/docs/guides/model-assets-graph.mdx index b383f56..ca8fe55 100644 --- a/docs/guides/model-assets-graph.mdx +++ b/docs/guides/model-assets-graph.mdx @@ -112,57 +112,114 @@ let graph = api.resources.create( -## Attach time-series to a node +## Attach a time series to a node -A machine's sensors are time-series. Tie a series to the machine that produces it -with a relation, exactly as above (`press_07` → `produces` → `press_07_oil_temp`), -then [ingest its datapoints](./ingest-timeseries.mdx). +A machine's sensors are time series. Create the series, tie it to the machine that produces +it with a relationship (`press_07` → `PRODUCES` → `press_07_oil_temp`), then +[ingest its datapoints](./ingest-timeseries.mdx). + + + + +```java +Timeseries oilTemp = new Timeseries() + .setExternalId("press_07_oil_temp") + .setName("Press 07 oil temperature"); +oilTemp.setUnit("celsius"); +oilTemp.setUnitExternalId("temperature_deg_c"); +client.timeseries().create(List.of(oilTemp)); + +RelForm produces = new RelForm(); +produces.setRelationshipType("PRODUCES"); +produces.setFromExternalId("press_07"); +produces.setToExternalId("press_07_oil_temp"); +client.edges().create(List.of(produces)); +``` + + + + +```python +client.timeseries.create([intellistream_datahub_sdk.TimeSeries( + external_id="press_07_oil_temp", name="Press 07 oil temperature", + unit="celsius", unit_external_id="temperature_deg_c")]) + +client.edges.create([ + intellistream_datahub_sdk.RelForm.by_external_ids("press_07", "press_07_oil_temp", "PRODUCES")]) +``` + + + + +```rust +use intellistream_datahub_sdk::relations::RelForm; +use intellistream_datahub_sdk::timeseries::TimeSeries; + +let mut oil_temp = TimeSeries::new("press_07_oil_temp", "Press 07 oil temperature"); +oil_temp.unit = Some("celsius".into()); +oil_temp.unit_external_id = Some("temperature_deg_c".into()); +api.time_series.create_one(&oil_temp).await?; + +api.edges.create(&vec![ + RelForm::by_external_ids("press_07", "press_07_oil_temp", "PRODUCES"), +]).await?; +``` + + + ## Read the graph back -Fetch nodes by external id; each resource carries its outgoing edges. +Fetch a node's neighbourhood with `fetchRelated`; the returned network carries the nodes it +found and the edges between them. (`byIds` returns the node alone: `relatedResources` is +empty on that read.) One hop out from the press finds the two edges created above. ```java -var some = client.resources().byIds(List.of( - IdCollection.createFromExternalId("press_07"))); +ResourceNetwork net = client.resources().fetchRelated("press_07", 1); +net.edges().forEach(edge -> + System.out.println(edge.getStart() + " -" + edge.getType() + "-> " + edge.getEnd())); -Resource press = some.getItems().iterator().next(); -press.getRelations().forEach(edge -> - System.out.println(edge.getType() + " -> " + edge.getEnd())); +if (net.edges().size() != 2) { + throw new AssertionError("expected CONTAINS and PRODUCES, found " + net.edges().size()); +} ``` ```python -press = client.resources.by_ids(["press_07"])[0] -for edge in press.relations: - print(edge.relationship_type, "->", edge.end) +net = client.resources.fetch_related(external_id="press_07", depth=1) +for edge in net.edges: + print(edge.start, edge.relationship_type, edge.end) + +types = {edge.relationship_type for edge in net.edges} +assert types == {"CONTAINS", "PRODUCES"}, types ``` ```rust -use intellistream_datahub_sdk::generic::IdAndExtId; +use intellistream_datahub_sdk::resources::RelatedResourcesForm; -let some = api.resources.by_ids(&vec![IdAndExtId::from_external_id("press_07")]).await?; -if let Some(press) = some.nodes().first() { - for edge in press.relations.iter().flatten() { - println!("{} -> {}", edge.relationship_type, edge.end); - } +let net = api.resources.fetch_related( + &RelatedResourcesForm::from_external_id("press_07").with_depth(1)).await?; +for edge in net.edges() { + println!("{:?} -{:?}-> {:?}", edge.start, edge.relationship_type, edge.end); } + +assert_eq!(net.edges().len(), 2, "expected CONTAINS and PRODUCES"); ``` -:::tip Group with datasets -Relations model *structure*. To slice assets by ownership, environment or tenant — -"everything in the Oslo plant" — also put them in a **dataset** (see the -[Datasets reference](/reference/datasets)). A resource can sit in a graph and a -dataset at once. +:::tip Group with data sets +Relationships model *structure*. To slice assets by ownership, environment or tenant — +"everything in the Oslo plant" — also put them in a **data set** (see the +[Data sets reference](/reference/datasets)). A resource can sit in a graph and a +data set at once. ::: diff --git a/docs/guides/query-and-aggregate.mdx b/docs/guides/query-and-aggregate.mdx index dea8cc0..b67832e 100644 --- a/docs/guides/query-and-aggregate.mdx +++ b/docs/guides/query-and-aggregate.mdx @@ -1,11 +1,11 @@ --- sidebar_position: 2 -title: Query & aggregate time-series +title: Query & aggregate time series --- import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; -# Query & aggregate time-series +# Query & aggregate time series Read raw datapoints back for a window, then roll them up to fixed buckets (hourly averages, daily maxima) for a chart or report. Aggregates and the bucket @@ -143,11 +143,14 @@ for series in result.get_items() { -:::note Common aggregates -`avg`, `min`, `max`, `sum`. Granularity is a number plus a unit — -`30s`, `5m`, `1h`, `1d`. The Python and Rust datapoint types expose each requested -aggregate as its own field — `min`/`max`/`sum` by name, and `avg` read back on -`.average` (`dp.average`, `dp.max`, …). +:::note The aggregates and granularities +The aggregates are `avg`, `sum`, `min` and `max`, lower-case; a name outside that set is +dropped, not rejected. Granularity is a number plus a unit: `s`, `m`, `h`, `d`, `w`, `mo`, `y`, +or the words `sec`, `min`, `hour`, `day`, `week`, `month`, `year` and their plurals (`30s`, +`5m`, `1h`, `1d`). Bare `m` is a minute; a month is `mo`. The Python and Rust datapoint types +expose each requested aggregate as its own field: `min`/`max`/`sum` by name, and `avg` read +back on `.average` (`dp.average`, `dp.max`, …). +[Retrieve datapoints →](/reference/timeseries#retrieve-datapoints) ::: :::warning Java aggregate reads @@ -159,8 +162,9 @@ aggregates — or in Java, fetch raw datapoints and aggregate them in your appli ## Paging large windows -A wide window can exceed one response. When the result carries a cursor, pass it -back to fetch the next page. +A wide window can exceed one response. Each returned collection carries a `nextCursor` when +there is more (`getNextCursor()` in Java, `next_cursor` in Python and Rust); pass it back as +`cursor` to fetch the next page. diff --git a/docs/guides/realtime-subscriptions.mdx b/docs/guides/realtime-subscriptions.mdx index be5a22a..66d8ee4 100644 --- a/docs/guides/realtime-subscriptions.mdx +++ b/docs/guides/realtime-subscriptions.mdx @@ -7,14 +7,16 @@ import TabItem from '@theme/TabItem'; # Consume live data -Push-based monitoring: a **subscription** names a set of time-series, and a live +Push-based monitoring: a **subscription** names a set of time series, and a live connection delivers each new datapoint as it lands — no polling. A control-room dashboard, an alerting worker, or a downstream pipeline drives the loop and **acks** -what it has handled. Anything unacked is redelivered, so a crash never loses data. +what it has handled. Anything unacked is redelivered, so a crash is followed by redelivery. :::info Needs a sandbox This guide reads data it does not create. [Seed a sandbox](/guides/seed-a-sandbox) writes it -in one go. +in one go. Nothing arrives on the socket until something writes to the series, so once the +listener below is running, re-run section **A** of the seed in a second shell and watch the +messages arrive. ::: ## 1. Create a subscription @@ -137,6 +139,53 @@ while let Some(result) = listener.next().await { +## 3. Count what arrives + +Re-run section **A** of the seed in a second shell. Each batch it writes to +`engine_temperature` or `engine_rpm` becomes a message here, so the count climbs. + + + + +```java +import java.util.concurrent.atomic.AtomicInteger; + +AtomicInteger seen = new AtomicInteger(); +try (var stream = client.subscriptions().listen(List.of("engine_room")) + .stream((SubscriptionMessage msg) -> + System.out.println(seen.incrementAndGet() + " messages so far"))) { + Thread.sleep(60_000); // long enough to re-run the seed in another shell +} +``` + + + + +```python +seen = 0 +with client.subscriptions.listen(["engine_room"]) as listener: + for msg in listener: + listener.ack([msg.message_id]) + seen += 1 + print(f"{seen} messages so far") +``` + + + + +```rust +let mut seen = 0; +let mut listener = api.subscriptions.listen(&["engine_room"]).await?; +while let Some(Ok(msg)) = listener.next().await { + listener.ack(&[msg.message_id.as_str()]).await?; + seen += 1; + println!("{seen} messages so far"); +} +``` + + + + ## Change the interest set at runtime A long-lived listener doesn't need to reconnect to follow more (or fewer) series — diff --git a/docs/guides/seed-a-sandbox.mdx b/docs/guides/seed-a-sandbox.mdx index cb27c13..991c6b5 100644 --- a/docs/guides/seed-a-sandbox.mdx +++ b/docs/guides/seed-a-sandbox.mdx @@ -12,13 +12,13 @@ against a sandbox rather than read. :::info What needs seeding, and what does not | Guide | Needs | | --- | --- | -| [High-throughput ingestion](/guides/ingest-timeseries) | Nothing. It writes its own data | +| [High-throughput ingestion](/guides/ingest-timeseries) | **A** (the series must exist) | | [Query & aggregate](/guides/query-and-aggregate) | **A** the engine series | | [Model assets as a graph](/guides/model-assets-graph) | Nothing. It creates the graph it reads | | [Consume live data](/guides/realtime-subscriptions) | **A** the engine series | | [Turn readings into events](/guides/detect-events) | **A** the engine series, which include a hot stretch so the rule actually fires | | [Attach files to assets](/guides/attach-files) | Nothing. It uploads what it then reads | -| [Work with units](/guides/work-with-units) | Nothing. It creates the series it converts | +| [Work with units](/guides/work-with-units) | Nothing. It creates the series it tags | | [Correlate alarms](/guides/correlate-alarms) | **B** the cooling system, its two sensors and their alarms | ::: @@ -32,14 +32,21 @@ import intellistream_datahub_sdk, numpy as np, pandas as pd client = intellistream_datahub_sdk.DataHubClient.from_env() def ingest(external_id, index, values, unit=None, name=None): - client.timeseries.create([intellistream_datahub_sdk.TimeSeries( - external_id=external_id, name=name or external_id, - unit=unit or "value", value_type="float")]) + try: + client.timeseries.create([intellistream_datahub_sdk.TimeSeries( + external_id=external_id, name=name or external_id, + unit=unit or "value", value_type="float")]) + except intellistream_datahub_sdk.DataHubException as e: + if e.status_code != 409: # a second run finds the series already there + raise client.timeseries.insert_from_lists( timestamps=index, values=np.asarray(values), ts=external_id) print(f"ingested {len(values):,} points → {external_id}") ``` +A second run of any block below answers `409` on the duplicate external id; the helper +catches that and goes on to write the datapoints, which dedup on `(series, timestamp)`. + ## A. An engine, with the last twenty minutes running hot Three series for one engine: a day of readings at one a minute. The temperature ends in a diff --git a/docs/guides/work-with-units.mdx b/docs/guides/work-with-units.mdx index 6a8ac07..78ce800 100644 --- a/docs/guides/work-with-units.mdx +++ b/docs/guides/work-with-units.mdx @@ -7,9 +7,9 @@ import TabItem from '@theme/TabItem'; # Work with units of measure -Units are shared reference data — `celsius`, `bar`, `m/s`, `usd`. Tagging a -time-series with a unit makes its values self-describing: a chart can label its axis, -and a consumer knows that `248.6` is bar, not psi. Browse the catalogue, then +Units are shared reference data: `temperature_deg_c`, `pressure_bar`, `mass_flow_rate_kghr`. +Tagging a time series with a unit makes its values self-describing: a chart can label its +axis, and a consumer knows that `248.6` is bar, not psi. Browse the catalogue, then reference a unit by its external id when you create a series. ## Browse the catalogue @@ -44,8 +44,11 @@ for u in api.units.list().await?.get_items() { ## Tag a series with a unit -The everyday use: pass a unit's external id when you create a time-series. The value -type stays numeric; the unit just describes what the numbers mean. +A series carries two unit fields. `unit` is required free text of at most 64 characters, the +label a chart shows (`bar`, `°C`); it is never checked against the catalogue. `unitExternalId` +is optional and is the catalogue key (`pressure_bar`); set it when a consumer should be able to +resolve the unit's symbol, quantity and conversion. The value type stays numeric; the unit +describes what the numbers mean. @@ -53,6 +56,7 @@ type stays numeric; the unit just describes what the numbers mean. ```java var series = Timeseries.of("wellhead_pressure_bar").name("Wellhead pressure"); series.setUnit("bar"); +series.setUnitExternalId("pressure_bar"); client.timeseries().create(series); ``` @@ -65,7 +69,8 @@ import intellistream_datahub_sdk client.timeseries.create([intellistream_datahub_sdk.TimeSeries( external_id="wellhead_pressure_bar", name="Wellhead pressure", - unit="bar")]) + unit="bar", + unit_external_id="pressure_bar")]) ``` @@ -76,6 +81,7 @@ use intellistream_datahub_sdk::timeseries::TimeSeries; let mut ts = TimeSeries::new("wellhead_pressure_bar", "Wellhead pressure"); ts.unit = Some("bar".into()); +ts.unit_external_id = Some("pressure_bar".into()); api.time_series.create_one(&ts).await?; ``` @@ -120,11 +126,12 @@ let by_id = api.units.by_ids(&DataWrapper::from(vec![IdAndExtId::from_id(7)])).a -:::note Unit external ids are snake_case -The seeded unit catalogue uses snake_case throughout — `m_s` for metres per second, `deg_c` -for degrees Celsius — so reference them that way and browse the catalogue to find the exact -id. That is a property of the catalogue, not a rule about external ids in general: the ones -*you* create are [stored exactly as you send them](/reference/external-ids). +:::note Unit external ids are `_` in snake_case +The seeded unit catalogue names each unit by its quantity and symbol: `temperature_deg_c` for +degrees Celsius, `pressure_bar` for bar, `mass_flow_rate_kghr` for kilograms per hour. Browse +the catalogue to find the exact id. That is a property of the catalogue, not a rule about +external ids in general: the ones *you* create are +[stored exactly as you send them](/reference/external-ids). ::: See the [Units reference](/reference/units) for the full method list. diff --git a/docs/intro.md b/docs/intro.md index 72ea2cb..cc25b2d 100644 --- a/docs/intro.md +++ b/docs/intro.md @@ -6,7 +6,7 @@ title: DataHub SDK # DataHub SDK -A thin, fast client for the **DataHub Platform** — manage resources, time-series, +A thin client for the **DataHub Platform** — manage resources, time series, events, files, and stream live data from your own application. The SDK is available for **Java**, **Python** and **Rust**. Pick your language once @@ -20,7 +20,7 @@ datapoint, or jump to [high-throughput ingestion](/guides/ingest-timeseries) to ## What you can do -- **Time-series** — create series, ingest datapoints in parallel, query raw values and aggregates. +- **Time series** — create series, ingest datapoints in parallel, query raw values and aggregates. - **Resources** — model assets and their relationships as a graph. - **Events** — record and query operational events. - **Files** — attach documents and images to your assets. @@ -37,6 +37,6 @@ covers the server, and using the server over its API is not what it restricts. ## Where to go next - **[Tutorial](/tutorial)** — build a small metrics agent end to end: create series, ingest on a schedule, and survive API outages. -- **[Examples](/guides/ingest-timeseries)** — task recipes: ingestion, graph modeling, querying & aggregation, live consumption, and event detection. -- **[Industry scenarios](/industries/oil-and-gas/production)** — end-to-end walkthroughs for oil & gas, energy grids, finance, and IT operations. +- **[Guides](/guides/ingest-timeseries)** — task recipes: ingestion, graph modeling, querying & aggregation, live consumption, and event detection. +- **[Industry scenarios](/industries/overview)** — end-to-end walkthroughs across ten sectors, from oil & gas and energy to finance and healthcare. - **[API reference](/reference/client)** — every service, method and option, in Java, Python and Rust. diff --git a/docs/mcp-server.mdx b/docs/mcp-server.mdx index fb9fd3a..53bee61 100644 --- a/docs/mcp-server.mdx +++ b/docs/mcp-server.mdx @@ -11,7 +11,7 @@ against your data model directly instead of you hand-writing an integration laye | Server | Endpoint | Exposes | | --- | --- | --- | -| API | `/mcp` (port 8081 locally) | **37 tools** across datasets, resources, relationships, timeseries, events, labels and units | +| API | `/mcp` (port 8081 locally) | **37 tools** across data sets, resources, relationships, time series, events, labels and units | | Analysis | the analysis service's `/mcp` (port 8082 locally) | **one tool**, [`analysis_related_series`](#the-analysis-server) | The important part is what they *aren't*: each MCP endpoint is an ordinary Spring endpoint on @@ -52,12 +52,11 @@ Three things are checked before any tool body runs: `OrganizationValidator` used for REST, so an agent cannot reach another tenant's data even by asking for an id it happens to know. -Past those gates, per-dataset [access grants](/reference/datasets#access-control) apply +Past those gates, per-data-set [access grants](/reference/datasets#access-control) apply exactly as on REST: rows in data sets the identity cannot read are omitted, writes it lacks a grant for are refused. -The practical consequence, and the reason this design is worth the trouble: **an agent -inherits the permissions of whoever it is acting for.** If you want an agent that can read +The practical consequence: **an agent inherits the permissions of whoever it is acting for.** If you want an agent that can read but not write, issue it a token that can read but not write. There is no second permission model to keep in sync. @@ -123,19 +122,19 @@ Calling a tool is the same request with a `tools/call` body: Named `_` in snake_case, deliberately, so the listing reads the same to you and to the model. -### Datasets +### Data sets | Tool | What it does | | --- | --- | -| `dataset_list` | List datasets visible to the caller. The catalogue is usually small | +| `dataset_list` | List data sets visible to the caller | | `dataset_search` | Full-text search, returns ids needed by `timeseries_create` | -| `dataset_create` | Create one dataset, returns the server-assigned id | -| `dataset_update` | Update one dataset by id. Only provided fields change | +| `dataset_create` | Create one data set, returns the server-assigned id | +| `dataset_update` | Update one data set by id. Only provided fields change | | `dataset_delete` | Delete by id or externalId. Does **not** cascade to members | ### Resources -A resource is any node in the graph that is not a dataset or a timeseries: equipment, work +A resource is any node in the graph that is not a data set or a time series: equipment, work items, documents. | Tool | What it does | @@ -171,14 +170,14 @@ neighbour ("the time series measuring this pump"). | `edge_get` | Fetch a single edge by numeric id | | `edge_delete` | Delete edges by id. Nodes on either side are untouched | -### Timeseries +### Time series | Tool | What it does | | --- | --- | | `timeseries_list` | Browse when you do not yet have a search term | | `timeseries_search` | Full-text search over name, externalId and description, ranked by relevance | | `timeseries_get` | Look up by externalId or numeric id | -| `timeseries_create` | Create one timeseries, returns the assigned id. Needs a `unit` or a `unitExternalId` (see `unit_list`) | +| `timeseries_create` | Create one time series, returns the assigned id. Needs a `unit` or a `unitExternalId` (see `unit_list`) | | `timeseries_update` | Update common fields by id | | `timeseries_delete` | Delete the definition. Existing datapoints are not removed | | `timeseries_get_latest` | Most recent datapoint, for "what is it reading now?" | @@ -207,7 +206,7 @@ for when the agent only has words to go on. | `label_create` | Create one label, coerced to `SCREAMING_SNAKE_CASE` | | `label_update` | Update description, i18n code or colour by id | | `unit_list` | Units of measure known to DataHub, e.g. Celsius | -| `unit_get` | Look up one unit by externalId, e.g. `celsius` | +| `unit_get` | Look up one unit by externalId, e.g. `temperature_deg_c` | ## The analysis server @@ -314,18 +313,13 @@ and daily ingest quota as your own jobs, and hit the same field, batch and body- agent that lists in a loop can rate-limit the pipeline it shares a tenant with. The refusal comes back as a `429` carrying `Retry-After`, which is worth teaching the agent to honour. -**Deletes do not cascade.** Deleting a dataset leaves its resources and timeseries in place; -deleting a timeseries leaves its datapoints. This is deliberate, but it means an agent +**Deletes do not cascade.** Deleting a data set leaves its resources and time series in place; +deleting a time series leaves its datapoints. This is deliberate, but it means an agent tidying up will need more than one call, and a half-finished tidy leaves orphans. -**Name coercion is server-side.** Relationship types and labels are normalised to -`SCREAMING_SNAKE_CASE`, so `derived from` becomes `DERIVED_FROM`. Agents that construct a -name and then search for the string they constructed will miss. - -## Adding a tool - -Annotate a public method on a Spring bean in the API server's `mcp.tools` package with `@Tool`, reference -`McpResultConverter` from it, and register the bean in the `ToolCallbackProvider`. If it is -not registered there, the server does not advertise it. Spring AI builds the schema from the -parameter types and the `@ToolParam` descriptions, so those descriptions are what the model -reads: write them for the model, not for a colleague. +**Name coercion is server-side, and not uniform.** `edge_create_type` and `label_create` +normalise a name to `SCREAMING_SNAKE_CASE`, so `derived from` becomes `DERIVED_FROM`, and the +labels given to `resource_create` are canonicalised the same way when stored. `edge_create` +only upper-cases its `relationshipType`, so `derived from` becomes `DERIVED FROM`, a different +type from `DERIVED_FROM`. Agents that construct a name and then search for the string they +constructed will miss; have the agent write relationship types as `SCREAMING_SNAKE_CASE`. diff --git a/docs/quickstart.mdx b/docs/quickstart.mdx index 3a9b5c6..2989537 100644 --- a/docs/quickstart.mdx +++ b/docs/quickstart.mdx @@ -57,10 +57,9 @@ intellistream-datahub-sdk = "0.1" :::tip The name is long on purpose, and you can shorten it -`datahub-sdk` was already taken on PyPI by an unrelated project, and the bare `datahub` -neighbourhood belongs to a much better-known metadata platform, so the package is qualified -with the organisation and the Rust crate takes the same name. Alias it if you would rather -not type it: +`datahub-sdk` was already taken on PyPI by an unrelated project, and the `datahub` name is +taken, so the package is qualified with the organization and the Rust crate takes the same +name. Alias it if you would rather not type it: ```python import intellistream_datahub_sdk as dh @@ -173,16 +172,20 @@ for dp in client.timeseries.retrieve_datapoints(rf)[0].get_datapoints(): ```rust +use chrono::Utc; use intellistream_datahub_sdk::generic::{DataWrapper, RetrieveFilter}; let filter = RetrieveFilter { external_id: Some("engine_temperature".into()), - limit: Some(100), + start: Some(Utc::now() - chrono::Duration::hours(1)), + end: Some(Utc::now()), ..Default::default() }; let points = api.time_series.retrieve_datapoints(&DataWrapper::from(vec![filter])).await?; for c in points.get_items() { - println!("{} points", c.datapoints.len()); + for p in &c.datapoints { + println!("{} = {}", p.timestamp, p.value); + } } ``` diff --git a/docs/reference/client.md b/docs/reference/client.md index 387a0a5..aedc069 100644 --- a/docs/reference/client.md +++ b/docs/reference/client.md @@ -80,7 +80,7 @@ let api = blocking::create_api_service(); | Service | Java | Python | Rust | | --- | --- | --- | --- | | Resources | `client.resources()` | `client.resources` | `api.resources` | -| Time-series | `client.timeseries()` | `client.timeseries` | `api.time_series` | +| Time series | `client.timeseries()` | `client.timeseries` | `api.time_series` | | Datasets | `client.datasets()` | `client.datasets` | `api.datasets` | | Events | `client.events()` | `client.events` | `api.events` | | Units | `client.units()` | `client.units` | `api.units` | @@ -97,7 +97,6 @@ bearer token **or** OAuth2 client-credentials (the SDK fetches and refreshes the | `BASE_URL` | API base URL (required) | | `TOKEN` | Static bearer token | | `CLIENT_ID` / `CLIENT_SECRET` / `TOKEN_URI` | OAuth2 client-credentials (all three) | -| `PROJECT_NAME` | Optional project/tenant hint | `fromEnv()` / `from_env()` / `create_api_service()` read these from the environment, falling back to a `.env` file in the working directory (real environment variables win). @@ -117,8 +116,6 @@ When the claim is present but names an organization this deployment holds no ten (never onboarded, or since removed), every call fails **`403`** with an `application/problem+json` body of `type: ".../errors/unknown-tenant"` naming the refused `organizationId`. Retrying never helps: an administrator has to register the organization. -This previously surfaced as a `500`, so retry logic that keys on 5xx should be told to give -up on it. | Variable | Java builder | Python kwarg | Rust setter | When you need it | | --- | --- | --- | --- | --- | @@ -135,8 +132,8 @@ up on it. | The identity provider refused the token | Calls succeed, then start failing part-way through a run | Get a new token | The second one catches long-running processes. The API checks your token locally (signature, -expiry, issuer) and separately reads your dataset grants from the identity provider's UserInfo -endpoint, so a token can pass the first check and still be refused by the second: it is unexpired, +expiry, issuer) and separately reads your data set grants from the identity provider's UserInfo +endpoint. A token can pass the first check and still be refused by the second: it is unexpired, but the session behind it has ended, because an idle or maximum session lifetime elapsed or somebody signed out. The response carries `WWW-Authenticate: Bearer error="invalid_token"` and a problem+json body with `type: ".../errors/token-rejected"`. @@ -249,9 +246,10 @@ starts from a fresh request. The identity provider must be configured to trust the external issuer, and the external identity must map to a real user on that side. For Keycloak that means an Identity Provider with **JWT Authorization Grant** enabled (Keycloak 26.5+), a client with the matching capability, and a -linked user carrying the roles and tenant claim. See `EntraID.md` in the platform repository for -the full walkthrough, including the audience and assertion-lifetime settings that trip up a first -attempt. +linked user carrying the roles and tenant claim. See +[`EntraID.md`](https://github.com/IntelliStream-DataHub/datahub-platform/blob/master/EntraID.md) +in the platform repository for the full walkthrough, including the audience and +assertion-lifetime settings that trip up a first attempt. ::: ### From HashiCorp Vault (Java) @@ -269,8 +267,8 @@ DatahubConfig cfg = DatahubConfig.fromVaultAppRoleEnv("datahub/sdk"); / ## Durable ingest buffering Optional and **off by default**. When enabled, datapoint and event ingestion that can't reach the -API — or is rejected with an auth failure (HTTP 401/403, e.g. an expired or rotated token) — spools -to disk and is flushed automatically on the next ingest call, so neither a transient outage nor a +API, or is rejected with an auth failure (HTTP 401/403, e.g. an expired or rotated token), spools +to disk and is flushed automatically on the next ingest call. Neither a transient outage nor a credential hiccup loses data or raises. The buffer is a segmented, compressed log (gzip in Java, zstd in Rust/Python) bounded on two axes, either of which may be left unset; an unset axis defaults to **72 hours** / **5 GiB** once buffering is on: @@ -347,7 +345,7 @@ A [lifetime ceiling](./limits#lifetime-ceilings) answers `403` too, and that one missing grant is fixed out of band and the data then flushes; a ceiling never becomes acceptable by being replayed, so spooling it would fill the buffer with data the server refuses every time. The client matches the problem `type`, so an ordinary permission `403` -still buffers exactly as before. +is buffered. ::: :::note Retries are idempotent @@ -442,8 +440,7 @@ Two responses are worth recognising by shape: configured [naming policy](./external-ids#the-naming-policy). Nothing was created; the `violations` array names each one and suggests a replacement. - **A `warnings` array beside `items` on a `2xx`** — the write succeeded, and the ids in it - are in a data steward's queue. The field is absent when empty, so existing code is - unaffected. + are in a data steward's queue. The field is absent when empty. Both shapes, and the rules behind them, are in [External ids & naming](./external-ids). diff --git a/docs/reference/datasets.md b/docs/reference/datasets.md index 60b5f3f..9440742 100644 --- a/docs/reference/datasets.md +++ b/docs/reference/datasets.md @@ -1,29 +1,29 @@ --- sidebar_position: 5 -title: Datasets +title: Data sets --- import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; -# Datasets +# Data sets -Logical groupings of resources and time-series. Datasets can be nested (a dataset can -belong to a parent dataset), and that hierarchy is live in queries: filtering time-series -by a dataset also matches everything beneath it. +Logical groupings of resources and time series. Data sets can be nested (a data set can +belong to a parent data set), and that hierarchy is live in queries: filtering time series +by a data set also matches everything beneath it. [Filter series →](./timeseries#filter-series) -The hierarchy is built from `BELONGS_TO` edges, and the server enforces that: a relation -pointing at a dataset must be `BELONGS_TO`, and a dataset can only claim a time-series -that isn't already in another dataset. -[Edge rules →](./resources#create-resources-and-relations) +The hierarchy is built from `BELONGS_TO` relationships, and the server enforces that: a +relationship pointing at a data set must be `BELONGS_TO`, and a data set can only claim a +time series that isn't already in another data set. +[Relationship rules →](./resources#create-resources-and-relations) :::note External ids are stored exactly as you send them -The server does not rewrite a dataset external id: `Plant-A` stays `Plant-A`. Some clients -*derive* one from the name as a convenience (the Rust `Dataset::new` below), and that -derivation is snake_case — but it is a client-side default, not a server rule. +The server does not rewrite a data set external id: `Plant-A` stays `Plant-A`. The Rust +`Dataset::new` derives one from the name, in snake_case; that is a client-side default, +not a server rule. Uniqueness and lookup both ignore case, so `plant_a` collides with `PLANT_A` and either -spelling finds the same dataset. Datasets are also subject to the +spelling finds the same data set. Data sets are also subject to the [naming policy](./external-ids#the-naming-policy) if an administrator has set one. [External ids & naming →](./external-ids) ::: @@ -156,10 +156,9 @@ let matches = api.datasets.filter(&DatasetFilter::from_filter(criteria)).await?; -:::note There is no `writeProtected` or `deactivated` -Both were removed server-side as inert. A filter or a create still carrying either is refused -as an [unknown field](./client#unknown-fields), so a stale client fails on the first call -rather than filtering on nothing. +:::note There is no `writeProtected` or `deactivated` criterion +A filter carrying either, or any other unknown field, is refused with a `400` as an +[unknown field](./client#unknown-fields). ::: ## Search {#search} @@ -186,12 +185,6 @@ ever removes matches: the phrase decides what the candidates are. No match is an empty list, not a `404`. -:::note What changed -`filter` used to be accepted and silently ignored here, as it was on the resource and event -searches. All four searches now apply it, as one query rather than a phrase pass followed by a -narrowing pass. -::: - ## Access control {#access-control} Access to a data set is administered in Keycloak (or the directory behind it), not in @@ -214,19 +207,22 @@ Two consequences worth knowing when you code against this: - **A missing grant is a `403`** with an `application/problem+json` body naming the `dataSetId` and the `permission` (read or write) you lack. List, filter and search endpoints never 403 on grants: rows in data sets you cannot read are silently - omitted instead. Edge reads hide rather than refuse too: reading an edge needs read - on both endpoints' data sets, and one you may not read is a `404` from - `GET /edges/{id}` and omitted from `/edges/byids`, as if it did not exist. + omitted instead. +- **Relationship reads hide rather than refuse too.** Reading a relationship needs read + on both endpoints' data sets. One you may not read is a `404` from `GET /edges/{id}` + and omitted from `/edges/byids`, as if it did not exist. - **Managing a data set itself is stricter.** Creating, updating or deleting a data set - (as opposed to the data in it) requires the `/datasets/*/write` grant or `DATAHUB_ADMIN`; - grants on individual data sets are never enough, deliberately: a data set is the unit + (as opposed to the data in it) requires the `/datasets/*/write` grant or `DATAHUB_ADMIN`. + Grants on individual data sets are never enough, deliberately: a data set is the unit access is granted on, so renaming or re-parenting one changes what existing grants - cover. The rule follows the node, not the endpoint: a `DATASET`- or `POLICY`-labelled + cover. +- **The rule follows the node, not the endpoint.** A `DATASET`- or `POLICY`-labelled node reached through `/resources` answers the same way. The `403` detail spells this out. -The API reads grants from the identity provider's UserInfo endpoint, not from the token, -so a changed grant takes effect within about a minute, without a new token. +The API reads grants from the identity provider's UserInfo endpoint, not from the token, and +caches them: the grant cache refreshes after 45 seconds, behind a 10-second in-process cache. +A changed grant therefore takes effect within about a minute, without a new token. ## What each client covers {#client-coverage} @@ -239,7 +235,7 @@ so a changed grant takes effect within about a minute, without a new token. | Search | `datasets().search` | `datasets.search` | `datasets.search` | | Update | `datasets().update` | `datasets.update` | `datasets.update` | | Delete | `datasets().delete` | `datasets.delete` | `datasets.delete` | -| Access policies | HTTP | `datasets.policies` | `datasets.policies` | +| Policies (`GET /datasets/policies`) | HTTP | `datasets.policies` | `datasets.policies` | -All three clients now cover the whole surface bar the access-policy read in Java, which still -goes through the endpoint directly. +Java has no policies call; use `GET /datasets/policies` directly. It returns every policy in +the tenant that a data set can be associated with. diff --git a/docs/reference/edges.md b/docs/reference/edges.md index 48e27a0..28393e3 100644 --- a/docs/reference/edges.md +++ b/docs/reference/edges.md @@ -14,7 +14,7 @@ resources can be connected many ways, but only once each way. Most edges are born with their nodes: `POST /resources/create` takes `nodes` and `relations` together and writes them in one transaction. The `/edges` endpoints are for everything after that — linking resources that already exist, reading an edge back, cutting one without -touching its endpoints, and managing the relationship-type catalog. +touching its endpoints, and managing the relationship-type catalogue. [Create resources and relations →](./resources#create-resources-and-relations) ## The edge object {#body} @@ -25,7 +25,7 @@ touching its endpoints, and managing the relationship-type catalog. | `start` | number | Id of the `from` node. | | `end` | number | Id of the `to` node. | | `type` | string | The relationship type name, upper-cased (`CONTAINS`, `FLOWS_TO`). | -| `relationshipTypeId` | number | The type's id in the [catalog](#types). | +| `relationshipTypeId` | number | The type's id in the [catalogue](#types). | | `description` | string | Prose. | | `metadata` | map<string, string> | Flat key/value. | @@ -55,7 +55,7 @@ POST /edges/create Name each end by external id (`fromExternalId`/`toExternalId`) or by numeric id (`fromId`/`toId`), and the relation by `relationshipType` or `relationshipTypeId`. A type name you haven't used before is created for you, so [pre-registering types](#types) is only -for seeding the catalog or attaching a description. +for seeding the catalogue or attaching a description. The batch is **all-or-nothing** — one relation the server won't take and none of them are written. Success is a `201` with the created edges under `items`. @@ -118,39 +118,37 @@ println!("{:?}", created.get_items()[0].id); | Status | Means | | --- | --- | -| `400` | An end doesn't exist (the message names which), the relation has no type, or it breaks one of the dataset or time-series rules below. | +| `400` | An end doesn't exist (the message names which), the relation has no type, or it breaks one of the data set or time series rules below. | | `403` | You can't write one of the two resources. Both ends are checked, so linking something *into* a data set needs write access on that data set too. | | `409` | The two are already connected that way. `(start, end, type)` is unique — one relation per pair per type. | -:::note Edges into datasets and time-series are validated +:::note Edges into data sets and time series are validated The same two rules the [graph create](./resources#create-resources-and-relations) enforces apply here: -- A relation **to a dataset** must use the `BELONGS_TO` relationship type — that is the - relation dataset membership is built from, and anything else is a `400`. -- A **dataset → time-series** edge is accepted only when the series has no dataset yet, or - already belongs to that very dataset. A series in a *different* dataset is a `400`: a - time-series has one dataset. +- A relation **to a data set** must use the `BELONGS_TO` relationship type — that is the + relation data set membership is built from, and anything else is a `400`. +- A **data set → time series** edge is accepted only when the series has no data set yet, or + already belongs to that very data set. A series in a *different* data set is a `400`: a + time series has one data set. ::: ## Look up {#look-up} -`GET /edges/{id}` returns a single edge; an id that doesn't exist is a `404`. Older backends -answered `200` with an empty `items[]` here, so code that has to work against both should -check the count rather than the status. +`GET /edges/{id}` returns a single edge; an id that doesn't exist is a `404`. `POST /edges/byids` takes several ids and answers with a **graph**: the edges under `relations` and the resources at both ends under `nodes`, so you don't need a follow-up call to resolve endpoints. Unlike the single lookup, ids that match nothing are **silently omitted** — compare what comes back against what you asked for. -Both lookups are gated by dataset grants: reading an edge requires **read access to the data +Both lookups are gated by data set grants: reading an edge requires **read access to the data sets of both endpoints**, mirroring the write rule, because an edge reveals both ends. A denied edge behaves exactly like a missing one: `byids` omits it just as it omits an unknown id, and -the single lookup answers `404`, so an edge you may not read is indistinguishable from one that -does not exist — don't infer that two resources are unlinked from a missing edge. The MCP -`edge_get` tool follows the same rule. -[Dataset access control →](./datasets#access-control) +the single lookup answers `404`. An edge you may not read is therefore indistinguishable from +one that does not exist, so don't infer that two resources are unlinked from a missing edge. +The MCP `edge_get` tool follows the same rule. +[Data set access control →](./datasets#access-control) @@ -242,9 +240,9 @@ api.edges.delete(&vec![IdAndExtId::from_id(341)]).await?; ## Relationship types {#types} -Every edge carries a type, and the types are a per-tenant catalog: `GET /edges/types` lists +Every edge carries a type, and the types are a per-tenant catalogue: `GET /edges/types` lists them, `POST /edges/types/create` registers names up front. Registering is optional — a type -is created the first time an edge uses its name — so reach for it when you want the catalog +is created the first time an edge uses its name — so reach for it when you want the catalogue seeded before anyone writes, or a `description`/`i18nCode` attached to a type. A type is `{ id, name, description, i18nCode }`. @@ -302,7 +300,7 @@ normalised name, so `flows_to` and `FLOWS_TO` are the same type either way. The practical rule: write the type name the way you want it stored, `FLOWS_TO`, and the two paths agree. In Java, `RelForm.setName("Flows To")` snake-upper-cases client-side and lines the -create path up with the catalog; `setRelationshipType` passes the string through. Python and +create path up with the catalogue; `setRelationshipType` passes the string through. Python and Rust send what you give them. A name with no letter or digit in it (blank, or symbols only) is a `400`, and names registered @@ -311,9 +309,8 @@ through `types/create` are capped at 128 characters. :::caution A duplicate type name takes the whole batch with it `POST /edges/types/create` has no find-or-create: it saves a fresh type unconditionally, so a name that already exists (matched case-insensitively) collides on the unique name hash and -comes back as a **`409`** naming the conflict — not the "existing ones returned unchanged" the -endpoint used to advertise. You cannot use it to look up the id of a type you did not just -create; read `GET /edges/types` for that. +comes back as a **`409`** naming the conflict. You cannot use it to look up the id of a type +you did not just create; read `GET /edges/types` for that. Every form in a batch is saved in one transaction, so a single duplicate rolls the valid new types back alongside it. Treat the `409` as *"nothing in this batch was created"* rather than diff --git a/docs/reference/events.md b/docs/reference/events.md index 4b3d982..7e6c449 100644 --- a/docs/reference/events.md +++ b/docs/reference/events.md @@ -53,16 +53,14 @@ HTTP caller should expect the quotes. ::: :::note One list, not two parallel ones -`relatedResources` replaced a `relatedResourceIds` / `relatedResourceExternalIds` pair. The two -were independent inputs and drifted: a mismatched pair was unioned into an event describing both -resources, and a patch setting only the external ids left the stored ids stale. There are no -aliases: a client still sending the old field names is refused with a `400` naming them as -[unknown fields](./client#unknown-fields). Java SDK users get a compile break on the removed -setters instead. +A client still sending the retired `relatedResourceIds` / `relatedResourceExternalIds` names +is refused with a `400` naming them as [unknown fields](./client#unknown-fields). Supply an `id`, an `externalId`, or both. The server resolves whichever side you left out and returns both, so a read always gives you the pair. Sending both when they name *different* -resources is a `400` rather than a guess about which one you meant. +resources is a `400` rather than a guess about which one you meant. The field names +`relatedResourceIds` and `relatedResourceExternalIds` are unknown to the API and are refused +with a `400` (`type: ".../errors/unreadable-request-body"`), like any other unknown field. ::: ## Create {#create} @@ -122,7 +120,7 @@ that quietly relates to nothing. Create, update and delete each take at most **10 000 events** per request, and one event carries at most 10 000 characters of `description`, 256 metadata entries and 100 `relatedResources`. Past any of those is a `400`. [Limits & quotas](./limits) has the rest, -including the daily and lifetime ceilings on how many events an organisation may hold. +including the daily and lifetime ceilings on how many events an organization may hold. :::note Event ids are time-ordered UUID v7 The ingestion paths stamp every event that has no `id` with a **UUID v7** before sending — @@ -219,8 +217,8 @@ let events = api.events.filter(&filter).await?; -`limit` defaults to **100** and is capped at **10 000**; a zero or negative value falls back -to 100 rather than returning nothing. Whatever you ask for, the result is intersected with +`limit` defaults to **1 000** and is capped at **10 000**; a zero or negative value falls back +to the default rather than returning nothing. Whatever you ask for, the result is intersected with the data sets your token may read — a filter can never widen access, so an empty page can mean "no matches" or "none you may see", and the two are not distinguished. @@ -268,8 +266,7 @@ case-insensitively, so `"SHIFT_REPORT_1*"` is the case-insensitive way to ask th ::: **A parent data set stands in for its children.** Naming one covers everything beneath it in the -`BELONGS_TO` hierarchy, which is the same expansion access control applies to a grant, so the two -now agree. +`BELONGS_TO` hierarchy, the same expansion access control applies to a grant. An `externalId` that names no data set contributes nothing. That can only ever narrow the result — a typo gives you too few events, never events you should not see. @@ -278,10 +275,9 @@ result — a typo gives you too few events, never events you should not see. Omit the field (or send `null`) for **no data set restriction**. An explicit empty list means **narrow to no data sets**, which matches nothing. -The distinction matters if you build the filter programmatically: code that collects data set -references into a list and always sets the field will silently return zero events when that list -comes back empty, rather than the unrestricted result the same code returns for every other -filter field. +The distinction matters if you build the filter programmatically. Code that collects data set +references into a list and always sets the field silently returns zero events when that list +comes back empty. For every other filter field the same code returns the unrestricted result. ::: :::note `eventTime.max` is exclusive; the other maxima are inclusive @@ -354,13 +350,13 @@ To walk past the first page, echo back the `nextCursor` the response carried: ``` The cursor is **opaque** — base64 of a versioned encoding carrying the sort, the boundary value -and the id — so do not build or parse one. A cursor that does not decode restarts the walk from -the first page rather than failing, which is obviously wrong to a caller, where guessing at half -a position would silently skip or repeat the rows around the boundary. +and the id — so do not build or parse one. A cursor that does not decode is refused with a +`400` of `type: ".../errors/malformed-cursor"` rather than guessed at: half a position would +silently skip or repeat the rows around the boundary. Send it with the **same** `sort` that produced it: a cursor is a position in one particular -order. Continuing it under another is refused, though today the refusal arrives as an empty -response rather than a clean 400. Sorting by `subType` or `status` cannot be paged at all — +order. Continuing it under another is refused with the same `400`, which names both sorts. +Sorting by `subType` or `status` cannot be paged at all — both columns are nullable, and a keyset boundary on them would skip the events that have no value. @@ -432,7 +428,7 @@ it would mean renaming a resource silently abandoned its finding and started a s ### Fetch the queue -Filter on the type, ascending by `eventTime`, and page with [`after`](#paging): +Filter on the type, ascending by `eventTime`, and page with [`cursor`](#paging): @@ -582,14 +578,8 @@ narrows the phrase's hits: } ``` -:::note What changed -`filter` used to be accepted and silently ignored here, as it was on the resource and data set -searches. All four searches now apply it. `dataSetId` covers everything beneath the data sets you -name, exactly as it does on `/events/filter`. - -The description above is also a correction: this endpoint never was fuzzy, word-aware or -relevance-ranked, whatever the previous wording said. -::: +`dataSetId` in that block covers everything beneath the data sets you name, exactly as it does on +`/events/filter`. `query` must be 3 to 140 characters. There is no character restriction beyond that: punctuation, underscores and non-Latin scripts are all accepted, so an externalId or a Cyrillic asset name can @@ -776,8 +766,7 @@ api.events.delete(&vec![IdAndExtId::from_external_id("door_open")]).await?; ## What each client covers {#client-coverage} -The three clients cover the write and read paths, including the facet endpoints; the -administrative ones are HTTP-only so far. +The three clients cover every event endpoint, the facet endpoints included. | Operation | Java | Python | Rust | | --- | --- | --- | --- | @@ -797,6 +786,5 @@ All three carry the same four pairs: `list_types` / `search_types` and the same statuses and sources (`listTypes` / `searchTypes` … in Java). Take the paging value from the response envelope — `getNextCursor()` in Java, `page.next_cursor` -in Python, `page.next_cursor()` in Rust — and send it back unchanged. It is opaque; the -`_` value earlier versions had you assemble by hand no longer decodes, and an -undecodable cursor restarts the walk from the first page. +in Python, `page.next_cursor()` in Rust — and send it back unchanged. It is opaque; an +undecodable cursor is refused with a `400`. diff --git a/docs/reference/external-ids.md b/docs/reference/external-ids.md index 6e59b71..7746a7e 100644 --- a/docs/reference/external-ids.md +++ b/docs/reference/external-ids.md @@ -16,7 +16,7 @@ Same field name, different jobs. | On | What `externalId` is | Unique? | | --- | --- | --- | -| **Resources, data sets, time-series** | The **identity** of one thing — your key for it, and what every integration matches on. | Yes, per tenant, compared without case | +| **Resources, data sets, time series** | The **identity** of one thing — your key for it, and what every integration matches on. | Yes, per tenant, compared without case | | **Events** | A **correlation key** — the source system's key for the *subject* the event is about. | **No, and deliberately never** | An order that is created, amended and then shipped produces three events all carrying @@ -59,13 +59,12 @@ P1 rejected: shorter than 3 characters Byte-for-byte storage is what lets you join on identifiers you already maintain. The historian, the maintenance system and DataHub hold the same string, so a match is a string -comparison instead of a normalisation each integration has to reimplement identically, -forever. +comparison instead of a normalisation each integration has to reimplement identically. ### Layer 2 — the naming policy {#the-naming-policy} A convention an administrator configures on top of the floor. It applies to **every node type -whose external id is an identity**: resources and assets, data sets, time-series, functions and +whose external id is an identity**: resources and assets, data sets, time series, functions and policies. Enforcement sits on the one write path they share, so it cannot cover one create endpoint and miss another. @@ -93,8 +92,8 @@ than merging with it. :::caution The naming policy never applies to events Only the charset floor does. An event external id is not a name someone chose, it is the -source system's key for the subject, so the platform does not impose a convention on data you -do not own — and the policy's other rules would be meaningless there anyway, since events +source system's key for the subject. The platform does not impose a convention on data you do +not own, and the policy's other rules would be meaningless there anyway, since events deliberately share external ids. With a `snake_case` policy active and set to reject, `client.resources().create(...)` with @@ -104,7 +103,7 @@ correct behaviour, not a gap. ## Case: compared without it, stored with it -Uniqueness on resources, data sets and time-series ignores case. Creating `com-99-pt-1034` +Uniqueness on resources, data sets and time series ignores case. Creating `com-99-pt-1034` when `COM-99-PT-1034` already exists is a duplicate and comes back as **`409`**, with a message naming the id it collides with — it is the ordinary "this external id already exists" path, not a naming-policy rejection. @@ -242,7 +241,7 @@ Two things it is good for: Every warning and every rejection carries a `suggestion` where one can be derived — a conforming external id you can offer as a one-click fix. It is **offered, never applied**: -the platform stopped rewriting external ids, which is the whole point of this change. +the server never rewrites an external id. Two guarantees make it safe to wire straight into a form: @@ -349,34 +348,34 @@ does not change that. ### Paging the queue {#findings-paging} -Findings from a bulk import arrive in the thousands, so page with `after` rather than an -offset — `_`, from the last event you saw — and keep folding into the -same state as pages arrive: a `RESOLVED` on page 3 closes a finding whose `OPEN` came on page 1: +Findings from a bulk import arrive in the thousands, so page with the cursor rather than an +offset, and keep folding into the same state as pages arrive: a `RESOLVED` on page 3 closes a +finding whose `OPEN` came on page 1. Sort by `eventTime` ascending, which is the order a fold +needs, and echo each response's `nextCursor` back as `cursor` with the same `sort`: ```http POST /events/filter { "filter": { "type": "policy_finding" }, - "after": "1754476522104_0195f3a2-4c1b-7f9e-9c3a-1b2d4e6f8a90", + "sort": { "property": ["eventTime"], "order": "asc" }, + "cursor": "djE6ZXZlbnRUaW1lfGFzY3wxNzU0NDc2NTIyMTA0fDAxOTVmM2Ey", "limit": 200 } ``` -No `sort` here, and no `status` either. `after` fixes the order to `eventTime` then `id` -ascending on its own, which is exactly the order a fold needs — and narrowing to `OPEN` would -drop the `RESOLVED` events that close the findings you are folding. +No `status` in the filter: narrowing to `OPEN` would drop the `RESOLVED` events that close the +findings you are folding. -Both halves of `after` are required. Event times are not unique — an import lands thousands -of findings in the same millisecond — so paging on the timestamp alone would either skip that -group or repeat it forever. A short page is the last page. +The cursor is opaque and carries both the boundary event's time and its id, so an import that +lands thousands of findings in the same millisecond pages cleanly. `nextCursor` is absent on +the last page. [Fetching and folding the queue, with SDK examples →](./events#policy-findings) ## Practical advice -- **Send the identifier your source system already uses.** Pre-normalising to snake_case - still works and nothing that worked before has stopped, but it costs you the byte-for-byte - join the platform is built around. +- **Send the identifier your source system already uses.** Pre-normalising to snake_case is + accepted but loses the byte-for-byte match with the source system. - **Never change an external id after the fact.** It is a promise other systems have written down. The platform permits it; your integrations will not forgive it. - **Read `warnings` if you have a steward.** It is the difference between finding out now and diff --git a/docs/reference/files.md b/docs/reference/files.md index 41b854a..0ce15c6 100644 --- a/docs/reference/files.md +++ b/docs/reference/files.md @@ -41,8 +41,12 @@ let listing = api.files.list_directory_by_path("/reports/2026").await?; ## Upload -The Java client uploads raw `content` bytes to a destination `path`; the Python and Rust -clients upload a local file and a `destination_path`. +The server takes one path, `X-Datahub-Path`: the full path of the file including its name +(`/reports/2026/q2.csv`), from which it splits the name off the last `/`. The Java client +sends the `path` you give it as-is, so include the file name and a leading `/`. The Python +and Rust clients take a `destination_path` folder plus a `name` (defaulting to the local +file's name) and join the two into that same full path. The Java client uploads raw `content` +bytes; the Python and Rust clients read a local file. @@ -52,7 +56,7 @@ byte[] content = Files.readAllBytes(Path.of("report.csv")); DataWrapper uploaded = client.files().upload( FileUploadRequest.builder() - .path("reports/2026/q2.csv") + .path("/reports/2026/q2.csv") .content(content) .contentType("text/csv") // default: application/octet-stream .externalId("report_2026_q2") // optional @@ -96,9 +100,52 @@ let uploaded = api.files.upload_file(upload).await?; +The upload echo carries the stored node, id included. Read it back from the folder listing: + + + + +```java +client.files().list("/reports/2026").getItems() + .forEach(n -> System.out.println(n.getId() + " " + n.getName() + " " + n.getSize())); +``` + + + + +```python +for node in client.files.list_directory_by_path("/reports/2026"): + print(node.id, node.name, node.size) +``` + + + + +```rust +for node in api.files.list_directory_by_path("/reports/2026").await?.get_items() { + println!("{:?} {} {}", node.id, node.name, node.size); +} +``` + + + + +### When it fails {#errors} + +| Status | Means | +| --- | --- | +| `403` | You lack write access to the data set named in `dataSetId`, or to the parent folder's data set. | +| `404` | On download: no file with that id, or a file in a data set you may not read. The two are not distinguished, so a hidden file's existence is not leaked. | +| `409` | A file with that path, or that `externalId`, already exists. Uniqueness is tenant-wide, not per data set. | + +There is no size cap on `PUT /files`: the upload streams to disk and is exempt from the +[request-body limit](./limits#request-body-size). + ## Download {#download} -All three clients download a file's raw bytes by id. Python and Rust add a streaming variant +All three clients download a file's raw bytes by id. The id is numeric: Python and Rust take +it as an integer, Java as a string, because the endpoint also accepts an external id in that +position (`download("report_2026_q2")` works in Java). Python and Rust add a streaming variant that writes straight to a path, so a large file never has to sit in memory whole. diff --git a/docs/reference/limits.md b/docs/reference/limits.md index 0bb6750..4ea2fff 100644 --- a/docs/reference/limits.md +++ b/docs/reference/limits.md @@ -6,7 +6,7 @@ description: The size, rate and volume ceilings the API enforces, the status cod # Limits & quotas -The API enforces a handful of ceilings, and the status code says which one you hit and what to +The API enforces six kinds of ceiling, and the status code says which one you hit and what to do about it. That is the whole design: **what clears by waiting answers `429` and carries a `Retry-After`, and what does not answers something else.** Retry the first kind, fix the second. @@ -70,12 +70,9 @@ The tighter [`TEXT`/`MIXED`](./timeseries#value-types) cap is checked in the ser series' value type has been resolved, so it comes back naming the series type rather than the field. Split a text series into collections of 10 000 points or fewer. -:::caution The OpenAPI schema used to advertise caps nothing enforced -Some endpoints, `/events/delete` among them, carried a documented maximum in the schema that -the runtime never checked, so an oversized batch went through. Those caps are **enforced -now**. Code written against the advertised numbers is unaffected; code that quietly relied on -them not being real is not. -::: +The `items` cap is enforced wherever the handler validates the body. `/events/update` and +`/events/delete` do not, so an oversized update or delete batch is bounded only by the +[request body size](#request-body-size). ## Request body size {#request-body-size} @@ -120,7 +117,7 @@ be exported as one file at all. ## Rate limits {#rate-limits} -Counted per organisation and per user in a fixed one-minute window, with separate budgets for +Counted per organization and per user in a fixed one-minute window, with separate budgets for reads and writes. Which budget a request spends follows what it **does**, not which method it uses. A `GET` is a @@ -133,7 +130,7 @@ allowance, which is the larger of the two. | Scope | Writes / min | Reads / min | | --- | --- | --- | -| Organisation | 2 000 | 6 000 | +| Organization | 2 000 | 6 000 | | User | 600 | 1 200 | ```json @@ -157,7 +154,7 @@ ingest job share one allowance. ## Daily ingest quotas {#daily-ingest-quotas} -Per organisation, per UTC day, reset at 00:00 UTC. `Retry-After` points at that reset, so it +Per organization, per UTC day, reset at 00:00 UTC. `Retry-After` points at that reset, so it can be hours. | `metric` | Default per day | @@ -185,8 +182,8 @@ functions: they are one population, not five. ## Lifetime ceilings {#lifetime-ceilings} -Totals, not rates: how large an organisation may grow. Unlike everything above, these are -**off unless a deployment turns them on**, and the numbers size a free or trial organisation. +Totals, not rates: how large an organization may grow. Unlike everything above, these are +**off unless a deployment turns them on**, and the numbers size a free or trial organization. Handle the `403`, but do not plan your data model around these figures: ask whoever runs your deployment what applies to you. @@ -210,7 +207,7 @@ deployment what applies to you. There is deliberately **no `Retry-After`**: waiting does not clear a ceiling, and the status is `403` rather than `429` so no client mistakes it for one that does. The ceiling moves when -someone raises it, which is a conversation with IntelliStream, not a retry. +someone raises it, which is a conversation with whoever operates the deployment, not a retry. Whether deleting helps depends on the metric: @@ -226,7 +223,7 @@ Both endpoints, `/timeseries/datapoints/subscription/listen/**` and | Cap | Default | | --- | --- | -| Concurrent connections per organisation | 10 | +| Concurrent connections per organization | 10 | | Concurrent connections per user | 10 | | Subscriptions multiplexed over one socket | 10 | @@ -265,12 +262,10 @@ So rate limits and daily quotas take care of themselves: the client backs off an A `413` or a validation failure reaches your code, which is the right place for it, since neither is fixed by trying again. -A lifetime ceiling is the one `403` that does **not** spool. The others are worth spooling -because an expired token or a missing grant is fixed out of band and the data then flushes; a -ceiling never becomes acceptable by being replayed, so buffering it would fill the spool with -data the server refuses every time and bury the one message that says the limit is raised by -asking. The client tells them apart on the problem `type`, so the ceiling surfaces on the call -that hit it, in `errors()` on the [`IngestResult`](./timeseries#ingestresult). +A lifetime ceiling is the one `403` that does **not** spool: it surfaces on the call that hit +it, in `errors()` on the [`IngestResult`](./timeseries#ingestresult). Why the client treats it +differently from the other `403`s is under +[durable ingest buffering](./client#durable-ingest-buffering). Two things to check in your own configuration: diff --git a/docs/reference/resources.md b/docs/reference/resources.md index 294dd0d..7f960d8 100644 --- a/docs/reference/resources.md +++ b/docs/reference/resources.md @@ -29,13 +29,13 @@ it, and compared without case. Mirror the tag your operation already maintains | `dataSetId` | number | The data set the resource belongs to. | | `geoLocation` | GeoJSON geometry | `Point`, `Polygon`, … Validated on write; stored verbatim. Returned only on [assets](#typed-reads). | | `isRoot` | boolean | Whether the resource is a navigation root. Deletes are checked against reachability from a root — see [Delete](#delete). Returned only on resources and assets. | -| `relatedResources` | object[] | Read-only view of the graph: `{ id, externalId, relationshipType, direction }` per connected node. Populated where the graph is loaded, empty otherwise. | +| `relatedResources` | object[] | Read-only view of the graph: `{ id, externalId, relationshipType, direction, edgeId }` per connected node. Populated on the create echo and on `fetch-related` / `fetch-nearest`; empty on `/resources/{id}`, `byids`, `filter` and `search`. | | `createdTime`, `lastUpdatedTime` | epoch millis | Server-set. A create body may carry them, but they are ignored: the stored values are the server's. | Labels are how the platform types a node. The type-label (`ASSET`, `TIMESERIES`, `DATASET`, `POLICY`, `FUNCTION`) is what the create pipeline reads to decide which kind of entity to build, and free-form labels ride alongside it. That is also why one `/resources/create` call -can hold a mix of node types — a time-series next to an asset — rather than needing one +can hold a mix of node types — a time series next to an asset — rather than needing one endpoint per type. The same label types what a read returns: see [Reads come back typed](#typed-reads). @@ -57,12 +57,12 @@ holds for the ids on an [edge](./edges#body), `start` and `end` included. The read endpoints (`/resources/{id}`, `byids`, `filter`, `search`, `fetch-related`, `fetch-nearest`) return each node in the shape of its kind, and the type-label inside `labels` is the discriminator. There is deliberately no separate type property on the wire: -an element whose labels contain `TIMESERIES` *is* the time-series shape. +an element whose labels contain `TIMESERIES` *is* the time series shape. | Type-label present | Shape returned | | --- | --- | | `ASSET` | An asset: the body above, `geoLocation` included. | -| `TIMESERIES` | A [time-series](./timeseries): `unit`, `unitExternalId`, `valueType`. | +| `TIMESERIES` | A [time series](./timeseries): `unit`, `unitExternalId`, `valueType`. | | `DATASET` | A [data set](./datasets). | | `POLICY` | A policy: `type`, `value`, `deactivated`, `templateId`. | | `FUNCTION` | A function. | @@ -70,7 +70,7 @@ an element whose labels contain `TIMESERIES` *is* the time-series shape. Three rules govern which fields appear where: -- A time-series carries its **full label set**, not only `["TIMESERIES"]`. +- A time series carries its **full label set**, not only `["TIMESERIES"]`. - `isRoot` belongs to resources and assets; `geoLocation` belongs to assets. A flat resource body naming a `geoLocation` is a `400`: a plain resource has nowhere to store one, so it is refused rather than accepted and dropped. Send an `ASSET`-labelled body instead. @@ -95,8 +95,8 @@ for (NodeModel node : client.resources().filter(retriever).getItems()) { Each item is the same class the type's own endpoint returns, so `isinstance` works and a -time-series from `resources.filter()` behaves exactly like one from `timeseries.by_ids()`. -Two new classes join the set: `Asset` and `Policy`. +time series from `resources.filter()` behaves exactly like one from `timeseries.by_ids()`. +`Asset` and `Policy` are in the set. ```python from intellistream_datahub_sdk import TimeSeries @@ -234,17 +234,17 @@ Use [update](#update) to change an existing resource rather than re-creating it. decided before either check, so a caller who may not write the data set is told that (`403`) instead of being handed a `400` about an id they were never allowed to name. -:::note Edges into datasets and time-series are validated -Two endpoint rules apply to every edge, on create and on update (an update can retarget an -edge or change its type): +:::note Relationships into data sets and time series are validated +Two endpoint rules apply to every relationship, on create and on update (an update can +retarget a relationship or change its type): -- A relation **to a dataset** must use the `BELONGS_TO` relationship type — that is the - relation the dataset hierarchy and membership are built from, and anything else is +- A relationship **to a data set** must use the `BELONGS_TO` relationship type — that is the + relationship the data set hierarchy and membership are built from, and anything else is rejected with a `400`. -- A **dataset → time-series** edge is accepted only when the series has no dataset yet, or - already belongs to that very dataset (creating a series inside a dataset produces exactly - that membership edge). A series in a *different* dataset is rejected with a `400` — a - time-series has one dataset. +- A **data set → time series** relationship is accepted only when the series has no data set + yet, or already belongs to that very data set (creating a series inside a data set produces + exactly that membership relationship). A series in a *different* data set is rejected with a + `400` — a time series has one data set. ::: @@ -328,8 +328,8 @@ nodes and relations together, in one transaction. `POST /edges/create` sends the themselves, for when both ends already exist and repeating them would be noise — same fields, same rules, same edges back. -That endpoint, and the rest of the `/edges` surface (reading an edge back, deleting one -without touching its endpoints, the relationship-type catalog), has its own page. +That endpoint, and the rest of the `/edges` surface (reading a relationship back, deleting one +without touching its endpoints, the relationship-type catalogue), has its own page. [Edges →](./edges) To disconnect two resources without touching either of them, [delete the edge](./edges#delete). @@ -452,7 +452,7 @@ database scores and sorts every match before applying `limit`, so a very broad p than a narrow one. `limit` is capped at **1 000** here, lower than the 10 000 of `filter`, and `query` must be -3 to 140 characters. +3 to 140 characters. `limit` applies across all node types, and `POLICY` nodes are searched. ### Narrowing with `filter` {#search-filter} @@ -468,20 +468,6 @@ search query itself, everything else is applied to the hits afterwards. } ``` -:::note What changed -`filter` used to be accepted and silently ignored here, as it was on the data set and event -searches. All four searches now apply it. - -The phrase and the filter are now one query, so the database plans them together. They were briefly -two, with the phrase capped at a 10 000-row candidate set that the filter then narrowed, which -quietly dropped matches past that cap. - -Two other things moved with this. The search originally ran one query per node type and concatenated -the results, so `limit` applied per type (a request for 50 could return 250) and results came back -grouped by type. Policies were never searched at all, and now are, so a search with no `nodeType` -can return rows it did not before. -::: - @@ -561,8 +547,7 @@ Updatable node fields are `externalId`, `name`, `description`, `source`, `dataSe Sending both `set` and `setNull` for one field is a `400`: the request is contradictory, so it is refused rather than resolved by precedence. `setNull` against `name` or `externalId` is also a `400`, for the same reason a create cannot omit them: every resource has to have -both. Rename with `set` instead. (This used to return `200` and quietly change nothing, so -check the value rather than the status if you are working against an older deployment.) +both. Rename with `set` instead. Changing `externalId` runs it past the [naming policy](./external-ids#the-naming-policy), which reports violations per item in an RFC 9457 problem response. The whole batch is **all-or-nothing**. @@ -652,13 +637,12 @@ connected site an unbounded `depth` will hit 5 000 nodes long before it runs out and what you get back is a *neighbourhood*, not the component you asked for. Bound `depth` to 1–3 unless you know the graph is sparse. -Nodes from `fetchRelated` and `fetch-nearest` come back -[typed by label](#typed-reads) but sparsely populated: the graph holds a subset of each -node's columns, so a node from these endpoints is not the full record. Fetch by id when you -need everything. - -A `TIMESERIES` node from these endpoints carries `unit`, `unitExternalId` and `valueType`. -Every node carries its `metadata`. +Nodes from `fetchRelated` and `fetch-nearest` come back [typed by label](#typed-reads) and +carry the fields the graph mirror holds: `id`, `externalId`, `name`, `description`, `source`, +`dataSetId`, `labels`, `metadata`, `createdTime` and `lastUpdatedTime`, plus `relatedResources` +built from the edges of the network you fetched. By type: `isRoot` on resources and assets, +`geoLocation` on assets, `unit`, `unitExternalId` and `valueType` on time series, and +`isDeactivated` on policies. Fetch by id when you need a field outside that list. @@ -712,7 +696,7 @@ for node in net.nodes() { ### The nearest N of a kind {#fetch-nearest} `POST /resources/fetch-nearest` answers a question `fetchRelated` cannot: *the ten nearest -time-series to this pump*. It walks breadth-first and caps on the number of **matching +time series to this pump*. It walks breadth-first and caps on the number of **matching end-nodes**, not on hops or total nodes — so "the 10 nearest `TIMESERIES`" is exactly ten however many intermediate nodes lie between them. You get those nodes plus the sub-graph connecting them back to the start. @@ -898,7 +882,7 @@ several node types. | `GET /assets/{id}` | [look up](#look-up) | One asset, wrapped in `items` like every other read. | | `POST /assets/byids` | [look up](#look-up) | Ids that are missing, are not assets, or are not readable are omitted rather than failing the call. | | `POST /assets/filter` | [filter](#filter) | The same criteria, the same paging. A `nodeType` in the body is replaced, see below. | -| `POST /assets/search` | [search](#search) | Same replacement, and the `filter` block is still [accepted and ignored](#search). | +| `POST /assets/search` | [search](#search) | Same replacement, and the `filter` block is applied exactly as on [`/resources/search`](#search-filter). | | `POST /assets/update` | [update](#update) | Takes `nodes` and `relations` exactly as `/resources/update` does. | | `POST` or `DELETE /assets/delete` | [delete](#delete) | `204`, and the same [connectivity check](#delete). | @@ -937,9 +921,8 @@ resource. Its family is `POST /functions/create`, `GET /functions/list`, `GET /f `POST /functions/update` and `POST` or `DELETE /functions/delete`, on the same shared pipeline. `GET /functions/list` takes no filter: the inventory is expected to be small. -`GET /functions/{id}` is new, and completes the read surface: it returns the one function -wrapped in `items`, and reports a function you may not read as missing (`404`) rather than -forbidden, exactly as `GET /assets/{id}` does. +`GET /functions/{id}` returns the one function wrapped in `items`, and reports a function +you may not read as missing (`404`) rather than forbidden, exactly as `GET /assets/{id}` does. The Java client has no `assets()` or `functions()` service, so reach for the endpoints there. Creating an asset through `resources().create` with an `ASSET` label is the same pipeline and diff --git a/docs/reference/subscriptions.md b/docs/reference/subscriptions.md index ecd171f..8171487 100644 --- a/docs/reference/subscriptions.md +++ b/docs/reference/subscriptions.md @@ -7,7 +7,7 @@ import TabItem from '@theme/TabItem'; # Subscriptions -Durable, fan-out subscriptions over time-series, plus **live delivery over a WebSocket**. +Durable, fan-out subscriptions over time series, plus **live delivery over a WebSocket**. ## Manage subscriptions @@ -26,6 +26,10 @@ DataWrapper all = client.subscriptions().list(new SubscriptionRetr client.subscriptions().delete(List.of(IdCollection.createFromExternalId("engine_temps"))); ``` +`SubscriptionRetriever` takes a `filter` whose one criterion is `timeseries` (only +subscriptions bound to these series, by id or external id), a `limit` (default 100, at most +10 000), a `sort`, and `includeSystemManaged` (default `false`). + @@ -63,19 +67,21 @@ api.subscriptions.delete(&vec![IdAndExtId::from_external_id("engine_temps")]).aw -:::note Dataset access control -Creating a subscription requires **read access to every timeseries' dataset** it binds. If you +:::note Data set access control +Creating a subscription requires **read access to every bound series' data set**. If you lack read access to any of them, `create` fails with **HTTP 403** and nothing is persisted. Access is granted through Keycloak **organization groups**: `/datasets//read` for one data set (and everything beneath it), or the wildcard `/datasets/*/read` for all of them. -[Dataset access control →](./datasets#access-control) +[Data set access control →](./datasets#access-control) ::: ## Live delivery -`listen` opens an authenticated WebSocket over one or more subscriptions. Stream messages -to a handler or drive a loop, and **ack** the messages you've processed — anything left -unacked is redelivered on reconnect. +`listen` opens a WebSocket to `/timeseries/datapoints/subscription/listen//...`, +one path segment per subscription, and authenticates the upgrade request with the same +`Authorization: Bearer ` header as any REST call. Stream messages to a handler or drive +a loop, and **ack** the messages you've processed — anything left unacked is redelivered on +reconnect. @@ -148,22 +154,32 @@ Every listener also exposes `stream` for push delivery, `ack`/`nack`, `subscribe`/`unsubscribe`/`set_subscriptions` to change the live interest set at runtime, and `close`. -A delivered message carries the originating subscription's external id, an opaque -`messageId` you echo back to `ack`/`nack`, and a `payload` describing the fan-out event -(an action — create/update/delete — plus the affected datapoints). +A frame on the wire carries one subscription's messages: + +| Field | Type | | +| --- | --- | --- | +| `subscriptionExternalId` | string | The subscription the batch came from. | +| `messages[].messageId` | string | Opaque. Echo it back in an ack or nack. | +| `messages[].payload.eventAction` | `CREATE`, `UPDATE`, `DELETE` or `RENAME` | What happened. | +| `messages[].payload.eventObject` | `DATAPOINTS` | What it happened to. | +| `messages[].payload.items[]` | object[] | One entry per series: `id` (a JSON string), `externalId`, `valueType`, `datapoints[]`, and optionally `inclusiveBegin` and `exclusiveEnd`. | +| `messages[].payload.items[].datapoints[]` | `{ timestamp, value }` | `timestamp` is an ISO-8601 UTC string (`2026-08-30T22:00:00Z`); `value` is a string. | + +Ack and nack are `{"action": "ack", "messageIds": [...]}` and the same with `"nack"`. The +clients unpack each entry of `messages` into one message, whose `payload` is the object above. :::note Refused subscriptions surface as errors -Live delivery enforces the same dataset ACL: to attach a subscription you must be able to read -**all** of its bound timeseries. A subscription you can't read (`reason: "forbidden"`) or one that +Live delivery enforces the same data set ACL: to attach a subscription you must be able to read +**all** of its bound series. A subscription you can't read (`reason: "forbidden"`) or one that doesn't exist (`reason: "not-found"`) is refused per-subscription — the connection stays open for the subscriptions that did attach. The refusal is surfaced, not swallowed: a `SubscriptionError` via `pollError` in Java, an `Err(ListenError::Subscription { .. })` from `next().await` in Rust, and -an exception raised from the iterator in Python — so a refused subscription is visible instead of -looking like an indefinitely silent stream. +an exception raised from the iterator in Python. A refused subscription is therefore visible +instead of looking like an indefinitely silent stream. ::: :::note Sockets and subscriptions are capped -Ten concurrent connections per organisation, ten per user, and ten subscriptions multiplexed +Ten concurrent connections per organization, ten per user, and ten subscriptions multiplexed over one socket, by default. The two refusals behave differently, on purpose: | Over the cap on | The server | The socket | @@ -189,5 +205,3 @@ the server redelivers it — so make your handler idempotent. | List | `subscriptions().list` | `subscriptions.list` | `subscriptions.list` | | Delete | `subscriptions().delete` | `subscriptions.delete` | `subscriptions.delete` | | Live delivery | `subscriptions().listen` | `subscriptions.listen` | `subscriptions.listen` | - -Full parity — subscriptions are the one area where all three clients cover the same ground. diff --git a/docs/reference/timeseries.md b/docs/reference/timeseries.md index f3c78ca..748a77f 100644 --- a/docs/reference/timeseries.md +++ b/docs/reference/timeseries.md @@ -1,13 +1,13 @@ --- sidebar_position: 4 -title: Time-series +title: Time series --- import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; -# Time-series +# Time series -Time-series metadata, datapoint retrieval, and datapoint ingestion (single-request or +Time series metadata, datapoint retrieval, and datapoint ingestion (single-request or high-throughput). A series' `externalId` identifies it: unique per tenant, compared without case, and stored @@ -66,13 +66,18 @@ else: | --- | --- | | `float32` *(default)* | Sensor readings — 32-bit precision is plenty. | | `float` | Double-precision floating point. | -| `numeric` / `decimal32` | **Exact decimals** — money, lab values — stored without floating-point rounding. Pass the values as strings. | +| `numeric` / `decimal32` | **Exact decimals** — money, lab values — stored without floating-point rounding. Send the value's string form, see below. | | `bigint` | Whole numbers (counts, integer statuses). | | `text` | Non-numeric string values. | | `mixed` | Heterogeneous values in one series. | A float written to a `bigint` series is rejected, so pick the type that matches the data. +Every value crosses the wire as a string, whatever the type. For an exact decimal, send the +string form rather than a float: `Datapoint.of(ts, "12.34")` in Java, `DatapointString(ts, +"12.34")` with `insert_datapoints` in Python (`insert_from_lists` takes floats and converts +them), and `DatapointString` in Rust. + `text` and `mixed` also carry a tighter write cap than the numeric types: **10 000 datapoints per collection** rather than 100 000, and a lifetime ceiling of their own. The check runs once the series' value type is resolved, so it names the series type rather than a field. See @@ -143,12 +148,6 @@ Results come newest first unless you ask for another order — see read access to are silently omitted — the result is what your token may see, not an error. For free-text lookups use `POST /timeseries/search` instead. -:::note The `metadataKey` / `metadataValue` pair is gone -It existed only because `metadata` could not express "has this key, whatever its value". A null -value in the map says that now, and `{"health": "good", "tier": null}` asks for both conditions at -once. -::: - @@ -222,18 +221,8 @@ search query itself, everything else is applied to the hits afterwards. } ``` -:::warning `search.name` and `search.description` are gone -The phrase block is now a single `query`, the same shape the other three searches take. The two -alternatives it used to carry were removed rather than kept: `name` matched by **exact equality** -under an endpoint documented as full-text, and `description` ran a differently configured query -over one column. - -Both have a better replacement. `filter.name` matches names as a case-insensitive pattern list -(`["pump_*", "PMP-1"]`), which is more than `search.name` could do, and `query` already covers the -description column. - -Clients exposing these as separate calls (`search_by_name`, `search_by_description`) need updating -to match. +:::caution `search.name` and `search.description` are rejected +They are unknown fields, refused with a `400`; use `filter.name` and `query`. ::: ## Sorting and paging {#sorting-and-paging} @@ -270,7 +259,7 @@ A page that has a successor carries a `nextCursor`. Echo it back as `cursor` to The cursor is **opaque** — base64 of a versioned encoding carrying the sort, the boundary value and the id — so do not build or parse one. Send it with the **same** sort that produced it; a cursor is a position in one particular order, and continuing it under another is refused. One -that does not decode restarts the walk from the first page rather than failing. +that does not decode is refused with a `400` of `type: ".../errors/malformed-cursor"`. `nextCursor` is absent on a short page, so "keep going while it is present" is the whole loop. A full page may still be the last, so a complete walk ends with one empty request. @@ -305,7 +294,8 @@ while True: break ``` -`filter()` returns a `Page` — a list, so existing code is unaffected, carrying `.next_cursor`. +`filter()` returns a `Page`: iterable, indexable and sized like a list, carrying `.next_cursor` +and `.items` (a real `list`). It is not a `list` subclass, so `isinstance(page, list)` is `False`. @@ -462,7 +452,7 @@ client.timeseries.insert_from_lists( -`insert_datapoints` auto-batches large inputs (chunks above ~100k points): +`insert_datapoints` auto-batches large inputs (chunks at the 100 000-point collection cap): ```rust use intellistream_datahub_sdk::generic::{DataWrapper, DatapointsCollection, DatapointString}; @@ -481,7 +471,22 @@ api.time_series.insert_datapoints(&mut dw).await?; ## Retrieve datapoints -Identify a series (external id or id) and a time window. +`POST /timeseries/data/list` takes a list of `RetrieveFilter` items, one per series and +window: + +| Field | Meaning | +| --- | --- | +| `id` / `externalId` | The series. | +| `start`, `end` | ISO-8601 or epoch millis. At least one is required. | +| `limit` | Datapoints per page, default 100, at most 100 000. | +| `aggregates` | Any of `avg`, `sum`, `min`, `max`, lower-case. A name outside that set is dropped, not rejected. `avg` comes back as `average`. | +| `granularity` | A number and a unit: `s`, `m`, `h`, `d`, `w`, `mo`, `y`, or the words `sec`, `min`, `hour`, `day`, `week`, `month`, `year` and their plurals (`15m`, `1h`, `30 min`). Bare `m` is a minute; a month is `mo`. Required when `aggregates` is set. | +| `includeOutsidePoints`, `mergeDuplicates` | Booleans, default `false`. | +| `cursor` | The previous page's `nextCursor`, carried on each returned collection. | + +A datapoint comes back with an ISO-8601 UTC `timestamp` and a `value`. Python reads the +timestamp as a timezone-aware `datetime.datetime` and the value as `float | None`; Rust as +`chrono::DateTime` and `Option`; Java keeps both as strings on `DatapointString`. @@ -671,9 +676,8 @@ if (!result.isComplete()) { | Update | HTTP | `timeseries.update` | `time_series.update` | | Delete | `timeseries().delete` | `timeseries.delete` | `time_series.delete` | | Write datapoints | `insertDatapoints` / `ingest` | `insert_datapoints` / `insert_from_lists` | `insert_datapoint` / `insert_datapoints` | -| Read datapoints | `retrieve` / `retrieveAggregated` | `retrieve_datapoints` / `retrieve_latest_datapoints` | `retrieve_datapoints` / `retrieve_latest_datapoint` | +| Read datapoints (raw and [aggregated](#retrieve-datapoints)) | `retrieve` / `retrieveAggregated` | `retrieve_datapoints` / `retrieve_latest_datapoints` | `retrieve_datapoints` / `retrieve_latest_datapoint` | | Delete datapoints | `deleteDatapoints` | `timeseries.delete_datapoints` | `time_series.delete_datapoints` | Java is the one with `ingest`, the chunking, parallelising, retrying path described above. -It is missing `list` and `update`, so reach for the endpoint there. It gained `search` alongside -the resource, data set and event searches it already had. +It is missing `list` and `update`, so reach for the endpoint there. diff --git a/docs/reference/units.md b/docs/reference/units.md index ec1f711..947646b 100644 --- a/docs/reference/units.md +++ b/docs/reference/units.md @@ -7,7 +7,23 @@ import TabItem from '@theme/TabItem'; # Units -Units of measure (read-only reference data). +Units of measure (read-only reference data). The endpoints are `GET /units`, +`GET /units/{externalId}` and `POST /units/byids`. + +## The unit object {#body} + +| Field | Type | Notes | +| --- | --- | --- | +| `id` | number | Crosses the wire as a JSON string, like every other id. | +| `externalId` | string, 3–256 | The catalogue key, `_` in snake_case: `temperature_deg_c`, `pressure_bar`, `mass_flow_rate_kghr`. This is what a series' `unitExternalId` names. | +| `name` | string, 1–64 | Short code (`DEG_C`). | +| `longName` | string | `degree Celsius`. | +| `symbol` | string | `°C`. | +| `description` | string | Prose. | +| `aliasNames` | string[] | Other spellings (`C`, `degC`). | +| `quantity` | string | What it measures (`Temperature`). | +| `conversion` | `{ multiplier, offset }` | To the quantity's base unit. | +| `source`, `sourceReference` | string | Where the definition comes from (`qudt.org` and its URL). | ## List all units @@ -61,7 +77,7 @@ DataWrapper result = client.units().byIds(List.of(lookup)); ```python import intellistream_datahub_sdk -by_ext = client.units.by_external_ids("celsius") +by_ext = client.units.by_external_ids("temperature_deg_c") by_id = client.units.by_ids([intellistream_datahub_sdk.IdCollection(id=7)]) ``` @@ -71,7 +87,7 @@ by_id = client.units.by_ids([intellistream_datahub_sdk.IdCollection(id=7)]) ```rust use intellistream_datahub_sdk::generic::{DataWrapper, IdAndExtId}; -let by_ext = api.units.by_external_id("celsius").await?; +let by_ext = api.units.by_external_id("temperature_deg_c").await?; let by_id = api.units.by_ids(&DataWrapper::from(vec![IdAndExtId::from_id(7)])).await?; ``` diff --git a/docs/tutorial.mdx b/docs/tutorial.mdx index 22285f3..1acae0f 100644 --- a/docs/tutorial.mdx +++ b/docs/tutorial.mdx @@ -223,8 +223,8 @@ story. ## Step 2 — Ensure the time series exist -Before you can ingest, the target series have to exist. The create endpoint rejects -duplicates, so rather than blindly creating them the program **looks up first, then +Before you can ingest, the target series have to exist. The create endpoint rejects a +duplicate external id with `409`, so rather than blindly creating them the program **looks up first, then creates only the gap** — which makes it safe to re-run and safe to start on a machine where the series already exist. From 2a8f2b87574a953a4dc76ed2ab62e00ec14419a6 Mon Sep 17 00:00:00 2001 From: Olav Gjerde Date: Wed, 2 Sep 2026 14:48:13 +0200 Subject: [PATCH 7/7] docs(events): say once that the retired relation field names are refused The conflict resolution while rebasing this branch left the note saying the same thing twice. One sentence, with the link to the unknown-field contract. Co-Authored-By: Claude Fable 5.1 Signed-off-by: Olav Gjerde --- docs/reference/events.md | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-) diff --git a/docs/reference/events.md b/docs/reference/events.md index 7e6c449..78d3115 100644 --- a/docs/reference/events.md +++ b/docs/reference/events.md @@ -53,14 +53,11 @@ HTTP caller should expect the quotes. ::: :::note One list, not two parallel ones -A client still sending the retired `relatedResourceIds` / `relatedResourceExternalIds` names -is refused with a `400` naming them as [unknown fields](./client#unknown-fields). - Supply an `id`, an `externalId`, or both. The server resolves whichever side you left out and returns both, so a read always gives you the pair. Sending both when they name *different* resources is a `400` rather than a guess about which one you meant. The field names `relatedResourceIds` and `relatedResourceExternalIds` are unknown to the API and are refused -with a `400` (`type: ".../errors/unreadable-request-body"`), like any other unknown field. +with a `400`, like any other [unknown field](./client#unknown-fields). ::: ## Create {#create}