diff --git a/docs/advanced/asset-health-score.mdx b/docs/advanced/asset-health-score.mdx
index 4d00892..bad073a 100644
--- a/docs/advanced/asset-health-score.mdx
+++ b/docs/advanced/asset-health-score.mdx
@@ -10,33 +10,32 @@ import TabItem from '@theme/TabItem';
:::info At a glance
**Effort:** ~20–30 minutes · **You'll build:** a composite 0–100 health index from
several signals, with a healthy/watch/critical classification · **Stack:** the SDK plus
-a little arithmetic — no model training.
+a little arithmetic, no model training.
:::
A machine rarely fails on one signal. Vibration is creeping up, the bearing runs a
-little hot, oil pressure sags at load — each is fine alone, but together they tell a
+little hot, oil pressure sags at load, each is fine alone, but together they tell a
story. A **health score** rolls those signals into one comparable number so an operator
can rank a whole fleet at a glance and a dashboard can show green/amber/red. It's the
lightweight cousin of [predictive maintenance](/advanced/predictive-maintenance): no
model to train, just a transparent, tunable index.
-:::tip New to machine learning?
-No background needed. Skim the [gentle primer](/advanced/machine-learning-gently) for the
-ideas in plain language — model, feature, training, and the algorithm itself — and use
-[Generate sample data](/advanced/generate-sample-data) for a sandbox to run this against.
-:::
-
-:::tip Need data to run this?
+:::info Needs a sandbox
This reads several pump signals. [Generate a sandbox](/advanced/generate-sample-data)
-first — section A ingests `pump_07_bearing_temp_c`, `pump_07_oil_pressure_kpa` and a
+first, section A ingests `pump_07_bearing_temp_c`, `pump_07_oil_pressure_kpa` and a
stand-in `pump_07_vibration_anomaly` (or run [predictive maintenance](/advanced/predictive-maintenance)
to produce the real one).
:::
+:::tip New to machine learning?
+No background needed. Skim the [gentle primer](/advanced/machine-learning-gently) for the
+ideas in plain language, model, feature, training, and the algorithm itself.
+:::
+
## 1. Pull the latest value of each signal
-Take the most recent reading (or short average) of each contributing series for the
-asset.
+Take the most recent reading of each contributing series for the asset. The seed writes
+these hourly, so ask for the latest datapoint rather than a short window that may be empty.
@@ -47,13 +46,8 @@ import intellistream_datahub_sdk, pandas as pd
client = intellistream_datahub_sdk.DataHubClient.from_env()
def latest(external_id):
- rf = intellistream_datahub_sdk.RetrieveFilter(
- ts=external_id,
- start=pd.Timestamp.now(tz="UTC") - pd.Timedelta(minutes=15),
- end=pd.Timestamp.now(tz="UTC"),
- aggregates=["avg"], granularity="15m")
- pts = client.timeseries.retrieve_datapoints(rf)[0].get_datapoints()
- return float(pts[-1].average)
+ pts = client.timeseries.retrieve_latest_datapoints([external_id])[0].get_datapoints()
+ return float(pts[-1].value)
signals = {
"vibration": latest("pump_07_vibration_anomaly"), # 0..~1, from the anomaly model
@@ -66,13 +60,13 @@ signals = {
```java
-// The same retrieve, per signal; the scoring arithmetic below is shown in Python.
+// Java has no latest-datapoint call: read the last day and take the newest point.
+// The scoring arithmetic below is shown in Python.
var filter = new RetrieveFilter();
filter.setExternalId("pump_07_bearing_temp_c");
-filter.setStart(ZonedDateTime.now().minusMinutes(15));
+filter.setStart(ZonedDateTime.now().minusHours(24));
filter.setEnd(ZonedDateTime.now());
-filter.setAggregates(List.of("avg"));
-filter.setGranularity("15m");
+filter.setLimit(100);
var request = new DataRetriever();
request.setItems(List.of(filter));
@@ -84,21 +78,14 @@ double bearingTemp = Double.parseDouble(pts.get(pts.size() - 1).getValue());
```rust
-// The same retrieve, per signal; the scoring arithmetic below is shown in Python.
-use intellistream_datahub_sdk::generic::{DataWrapper, RetrieveFilter};
-use chrono::Utc;
-
-let filter = RetrieveFilter {
- external_id: Some("pump_07_bearing_temp_c".into()),
- start: Some(Utc::now() - chrono::Duration::minutes(15)),
- end: Some(Utc::now()),
- aggregates: Some(vec!["avg".into()]),
- granularity: Some("15m".into()),
- ..Default::default()
-};
-let s = api.time_series
- .retrieve_datapoints(&DataWrapper::from(vec![filter])).await?
+// The latest datapoint, per signal; the scoring arithmetic below is shown in Python.
+use intellistream_datahub_sdk::generic::{DataWrapper, IdAndExtId};
+
+let latest = api.time_series
+ .retrieve_latest_datapoint(&DataWrapper::from(vec![
+ IdAndExtId::from_external_id("pump_07_bearing_temp_c")])).await?
.get_items().remove(0);
+let bearing_temp = latest.datapoints.last().and_then(|p| p.value).unwrap_or(f64::NAN);
```
@@ -140,8 +127,8 @@ band = "healthy" if score >= 80 else "watch" if score >= 60 else "critical"
## 4. Publish the score and flag the bad ones
-Write the score back as its own series — now you can chart, rank and subscribe to asset
-health like any other signal — and raise an event when an asset drops to `critical`.
+Write the score back as its own series, now you can chart, rank and subscribe to asset
+health like any other signal, and raise an event when an asset drops to `critical`.
```python
client.timeseries.create([intellistream_datahub_sdk.TimeSeries(
@@ -156,27 +143,32 @@ if band == "critical":
event_time=pd.Timestamp.now(tz="UTC"),
metadata={"asset": "pump_07", "score": str(score),
"worst_signal": max(parts, key=parts.get)})])
+
+# read the score back
+stored = client.timeseries.retrieve_latest_datapoints(["pump_07_health_score"])[0].get_datapoints()
+print(f"health {stored[-1].value:.1f} ({band})")
+assert abs(stored[-1].value - score) < 0.01, "the score was not written"
```
-Run it across the fleet on a schedule and you have a single ranked health view —
+Run it across the fleet on a schedule and you have a single ranked health view,
`worst_signal` in the metadata tells maintenance *why* each asset is red.
## Where to take it further
- **Feed it the model.** Swap the hand-set vibration limits for the
[anomaly score](/advanced/predictive-maintenance) as a direct input.
-- **Trend the score.** A falling health score over days is itself a predictor — forecast
+- **Trend the score.** A falling health score over days is itself a predictor, forecast
it like any [other series](/advanced/demand-forecasting).
- **Roll up the graph.** Average child scores up a [site graph](/guides/model-assets-graph)
for a line- or plant-level health number.
## Further reading
-- **Normalising signals to a common scale** — [Feature scaling](https://en.wikipedia.org/wiki/Feature_scaling)
-- **The ideas in plain language** — [Machine learning, gently](/advanced/machine-learning-gently)
+- **Normalising signals to a common scale**: [Feature scaling](https://en.wikipedia.org/wiki/Feature_scaling)
+- **The ideas in plain language**: [Machine learning, gently](/advanced/machine-learning-gently)
## See also
-- [Query & aggregate](/guides/query-and-aggregate) — pulling the contributing signals.
-- [Predictive maintenance](/advanced/predictive-maintenance) — a learned input to the score.
-- [Turn readings into events](/guides/detect-events) — flagging critical assets.
+- [Query & aggregate](/guides/query-and-aggregate): pulling the contributing signals.
+- [Predictive maintenance](/advanced/predictive-maintenance): a learned input to the score.
+- [Turn readings into events](/guides/detect-events): flagging critical assets.
diff --git a/docs/advanced/data-cleaning-lineage.mdx b/docs/advanced/data-cleaning-lineage.mdx
index 03679b0..a32648c 100644
--- a/docs/advanced/data-cleaning-lineage.mdx
+++ b/docs/advanced/data-cleaning-lineage.mdx
@@ -14,19 +14,23 @@ linked in the graph, so the whole `raw → … → result` chain is traceable bo
**Stack:** the SDK for data and graph, plus `pandas`, `numpy`.
:::
-:::tip New to this?
-No data-science background needed — see the [gentle primer](/advanced/machine-learning-gently)
-for any unfamiliar term, and [generate a sandbox](/advanced/generate-sample-data) to run
-this against.
+:::info Needs a sandbox
+This reads two raw sensors that already exist. [Generate a sandbox](/advanced/generate-sample-data)
+first, section K ingests `engine_temperature_raw` and `engine_vibration_raw`, defects and all.
+:::
+
+:::tip New to machine learning?
+No data-science background needed. Skim the [gentle primer](/advanced/machine-learning-gently)
+for any unfamiliar term.
:::
**The idea in one paragraph.** Real data never goes straight from sensor to model. It's
-cleaned, resampled, turned into features, combined, scored — often a dozen steps, each
+cleaned, resampled, turned into features, combined, scored, often a dozen steps, each
producing a new series. After a few of these, nobody remembers what came from what. Two
-questions then become impossible to answer: *"this number looks wrong — where did it come
-from?"* and *"this sensor was faulty — what did it poison?"* **Data lineage** fixes both:
-record every step as nodes in the graph — `raw → cleaning function → cleaned → feature
-function → feature → …` — and the answers become a single graph walk.
+questions then become impossible to answer: *"this number looks wrong, where did it come
+from?"* and *"this sensor was faulty, what did it poison?"* **Data lineage** fixes both:
+record every step as nodes in the graph, `raw → cleaning function → cleaned → feature
+function → feature → …`, and the answers become a single graph walk.
## 1. Run the transformations, writing each result back
@@ -53,7 +57,7 @@ def store(ext_id, series, name=None, unit="value"):
client.timeseries.insert_from_lists(timestamps=s.index, values=s.to_numpy(), ts=ext_id)
```
-The pipeline — two raw sensors, cleaned, then several features extracted from each, then
+The pipeline, two raw sensors, cleaned, then several features extracted from each, then
assembled and scored:
```python
@@ -83,15 +87,16 @@ store("engine_feature_vector", feature_vector)
store("engine_health_score", (100 - feature_vector.abs()).clip(0, 100))
```
-Every box in the pipeline is now a stored series — but stored series alone don't tell you
+Every box in the pipeline is now a stored series, but stored series alone don't tell you
how they *relate*. That's step 2.
## 2. Record the lineage graph
Model each transformation as a **function node** and link it up: each input series is
`processed_by` the function, which `produces` an output series. Done across the pipeline,
-this builds a branching, ten-deep lineage DAG in one call. (`resources.create` is the
-same call in Java and Rust — see [model assets as a graph](/guides/model-assets-graph).)
+this builds a branching lineage DAG of 18 nodes (8 functions and 10 series) in one call.
+(`resources.create` is the same call in Java and Rust, see
+[model assets as a graph](/guides/model-assets-graph).)
```python
def fn(ext_id, name):
@@ -127,9 +132,12 @@ client.resources.create(functions, edges)
```
That graph branches (each cleaned signal feeds two features), converges (four features
-into one vector), and runs ten nodes deep from raw sensor to health score.
+into one vector), and runs nine nodes deep from raw sensor to health score:
+`engine_temperature_raw` → `clean_temp_fn` → `engine_temperature_clean` → `roll_mean_fn` →
+`engine_temp_roll_mean` → `assemble_fn` → `engine_feature_vector` → `score_fn` →
+`engine_health_score`.
-## 3. The payoff — trace it backward
+## 3. The payoff, trace it backward
Someone questions the health score. Walk the graph back from it and the entire recipe
appears: every function and every series between the score and the **raw sensors** it
@@ -137,15 +145,16 @@ ultimately came from. No guessing, no stale wiki page.
```python
back = client.resources.fetch_related(external_id="engine_health_score", depth=12)
-raw_sources = [n.external_id for n in back.nodes if n.external_id.endswith("_raw")]
+raw_sources = {n.external_id for n in back.nodes if n.external_id.endswith("_raw")}
print("this score derives from:", raw_sources) # engine_temperature_raw, engine_vibration_raw
+assert raw_sources == {"engine_temperature_raw", "engine_vibration_raw"}, raw_sources
```
-## 4. The payoff — trace it forward (impact analysis)
+## 4. The payoff, trace it forward (impact analysis)
Now the opposite, and the bigger win. The temperature sensor is found faulty. **What did
it poison?** Walk *forward* from the raw sensor and every downstream series it touched
-lights up — while the vibration-only features stay clean. That's the precise list to
+lights up, while the vibration-only features stay clean. That's the precise list to
recompute or quarantine.
```python
@@ -155,6 +164,7 @@ affected = [n.external_id for n in fwd.nodes
print("affected by the bad temperature sensor:", affected)
# engine_temperature_clean, engine_temp_roll_mean, engine_temp_roc,
# engine_feature_vector, engine_health_score — but NOT the vibration features
+assert "engine_health_score" in affected and "engine_vib_rms" not in affected, affected
```
This is the same [blast-radius traversal](/guides/correlate-alarms) used for asset
@@ -164,21 +174,21 @@ damage spreads.
## Where to take it further
- **Version the functions.** Bump `clean_temp_fn` to `clean_temp_fn_v2` and the old
- outputs still point at the version that made them — reproducibility for free.
+ outputs still point at the version that made them, reproducibility for free.
- **Quality events.** Emit a `data_quality` [event](/guides/detect-events) per run with
what each step changed, for an audit trail beside the lineage.
- **Feed the models.** Point the [forecasters](/advanced/lstm-forecasting) and
- [detectors](/advanced/predictive-maintenance) at the cleaned/feature series — and their
+ [detectors](/advanced/predictive-maintenance) at the cleaned/feature series, and their
outputs become new nodes on the same lineage graph.
## Further reading
-- **Data lineage** — [Wikipedia](https://en.wikipedia.org/wiki/Data_lineage)
-- **Directed acyclic graph (DAG)** — [Wikipedia](https://en.wikipedia.org/wiki/Directed_acyclic_graph)
-- **Outlier removal (Hampel / MAD)** — [Median absolute deviation](https://en.wikipedia.org/wiki/Median_absolute_deviation)
+- **Data lineage**: [Wikipedia](https://en.wikipedia.org/wiki/Data_lineage)
+- **Directed acyclic graph (DAG)**: [Wikipedia](https://en.wikipedia.org/wiki/Directed_acyclic_graph)
+- **Outlier removal (Hampel / MAD)**: [Median absolute deviation](https://en.wikipedia.org/wiki/Median_absolute_deviation)
## See also
-- [Model assets as a graph](/guides/model-assets-graph) — the nodes-and-relations basics.
-- [Correlate alarms with the graph](/guides/correlate-alarms) — the traversal behind both payoffs.
-- [Generate sample data](/advanced/generate-sample-data) — raw signals to run the pipeline on.
+- [Model assets as a graph](/guides/model-assets-graph): the nodes-and-relations basics.
+- [Correlate alarms with the graph](/guides/correlate-alarms): the traversal behind both payoffs.
+- [Generate sample data](/advanced/generate-sample-data): raw signals to run the pipeline on.
diff --git a/docs/advanced/demand-forecasting.mdx b/docs/advanced/demand-forecasting.mdx
index fa499c5..3ff85c8 100644
--- a/docs/advanced/demand-forecasting.mdx
+++ b/docs/advanced/demand-forecasting.mdx
@@ -15,19 +15,18 @@ boosted forecaster · **Stack:** the SDK for data in/out, plus `pandas`, `numpy`
Forecasting turns history into a plan: how much power a feeder will draw tomorrow, how
many units a store will sell next week, how much load a network will carry at the busy
-hour. The pattern is the same regardless of domain — past values plus calendar effects
-predict the next ones — and the forecast becomes a **new series** you can chart against
+hour. The pattern is the same regardless of domain, past values plus calendar effects
+predict the next ones, and the forecast becomes a **new series** you can chart against
actuals and alert on.
-:::tip New to machine learning?
-No background needed. Skim the [gentle primer](/advanced/machine-learning-gently) for the
-ideas in plain language — model, feature, training, and the algorithm itself — and use
-[Generate sample data](/advanced/generate-sample-data) for a sandbox to run this against.
+:::info Needs a sandbox
+This reads a load history that already exists. [Generate a sandbox](/advanced/generate-sample-data)
+first, section G ingests the `feeder_f12_load_mw` curve this build forecasts.
:::
-:::tip Need data to run this?
-This reads a load history that already exists. [Generate a sandbox](/advanced/generate-sample-data)
-first — section G ingests the `feeder_f12_load_mw` curve this build forecasts.
+:::tip New to machine learning?
+No background needed. Skim the [gentle primer](/advanced/machine-learning-gently) for the
+ideas in plain language, model, feature, training, and the algorithm itself.
:::
## 1. Load the history
@@ -50,7 +49,7 @@ rf = intellistream_datahub_sdk.RetrieveFilter(
points = client.timeseries.retrieve_datapoints(rf)[0].get_datapoints()
demand = pd.Series([float(p.average) for p in points],
- index=pd.to_datetime([p.timestamp for p in points])).sort_index()
+ index=pd.to_datetime([p.timestamp for p in points], utc=True)).sort_index()
demand = demand.asfreq("1h").interpolate() # regular hourly grid
```
@@ -139,7 +138,7 @@ print(f"holdout MAE: {mae:.2f} MW")
## 4. Forecast the horizon and write it back
-Future lags aren't known, so forecast **recursively** — predict one hour, feed it back
+Future lags aren't known, so forecast **recursively**, predict one hour, feed it back
in as the next hour's lag, and step forward. Then publish the forecast as its own
series alongside the actuals.
@@ -166,15 +165,23 @@ client.timeseries.create([intellistream_datahub_sdk.TimeSeries(
external_id="feeder_f12_load_mw_forecast", name="Feeder F12 load — 48h forecast", unit="mw", value_type="float")])
client.timeseries.insert_from_lists(
timestamps=forecast.index, values=forecast.to_numpy(), ts="feeder_f12_load_mw_forecast")
+
+# read the horizon back: one point per forecast hour
+stored = client.timeseries.retrieve_datapoints(intellistream_datahub_sdk.RetrieveFilter(
+ ts="feeder_f12_load_mw_forecast",
+ start=forecast.index[0], end=forecast.index[-1] + pd.Timedelta(minutes=1),
+ limit=1000))[0].get_datapoints()
+print(f"{len(stored)} forecast hours stored")
+assert len(stored) == HORIZON, f"expected {HORIZON}, got {len(stored)}"
```
With the forecast stored as a series, a [threshold rule](/guides/detect-events) on it
-becomes a *predictive* alert — raise a `capacity_risk` event when the forecast, not the
+becomes a *predictive* alert, raise a `capacity_risk` event when the forecast, not the
actual, is set to exceed a limit, hours before it happens.
## Where to take it further
-- **Exogenous drivers.** Add weather, price, or a promotions flag as features — usually
+- **Exogenous drivers.** Add weather, price, or a promotions flag as features, usually
the biggest accuracy win.
- **Prediction intervals.** Train quantile models (`loss="quantile"`) for a P10/P90 band
instead of a single line.
@@ -183,12 +190,12 @@ actual, is set to exceed a limit, hours before it happens.
## Further reading
-- **Gradient boosting** — [Wikipedia](https://en.wikipedia.org/wiki/Gradient_boosting) · [scikit-learn](https://scikit-learn.org/stable/modules/ensemble.html#histogram-based-gradient-boosting)
-- **Time-series forecasting** — [Wikipedia](https://en.wikipedia.org/wiki/Time_series)
-- **The ideas in plain language** — [Machine learning, gently](/advanced/machine-learning-gently)
+- **Gradient boosting**: [Wikipedia](https://en.wikipedia.org/wiki/Gradient_boosting) · [scikit-learn](https://scikit-learn.org/stable/modules/ensemble.html#histogram-based-gradient-boosting)
+- **Time series forecasting**: [Wikipedia](https://en.wikipedia.org/wiki/Time_series)
+- **The ideas in plain language**: [Machine learning, gently](/advanced/machine-learning-gently)
## See also
-- [Query & aggregate](/guides/query-and-aggregate) — building the hourly history.
-- [Turn readings into events](/guides/detect-events) — turning the forecast into a predictive alert.
-- [Renewable energy](/industries/energy-utilities/wind) · [Retail](/industries/transport-logistics/retail) — domains this fits.
+- [Query & aggregate](/guides/query-and-aggregate): building the hourly history.
+- [Turn readings into events](/guides/detect-events): turning the forecast into a predictive alert.
+- [Renewable energy](/industries/energy-utilities/wind) · [Retail](/industries/transport-logistics/retail), domains this fits.
diff --git a/docs/advanced/fraud-classification.mdx b/docs/advanced/fraud-classification.mdx
index 031d554..644914e 100644
--- a/docs/advanced/fraud-classification.mdx
+++ b/docs/advanced/fraud-classification.mdx
@@ -5,7 +5,7 @@ title: Fraud classification (graph + ML)
import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';
-# Fraud classification — graph features + machine learning
+# Fraud classification, graph features + machine learning
:::info At a glance
**Effort:** ~1–2 hours · **You'll build:** graph-derived features from network traversal,
@@ -13,32 +13,30 @@ combined with behavioural features, feeding a supervised classifier · **Stack:*
for data and traversal, plus `networkx`, `pandas`, `scikit-learn`.
:::
-This is the capstone: it fuses the two things the rest of the docs treat separately — the
-**knowledge graph** and **machine learning**. Money laundering doesn't look suspicious one
-payment at a time; it looks suspicious in the *shape* of the network — funds fanning out
-through mules and looping back — combined with behaviour like rapid pass-through. A rules
+Money laundering doesn't look suspicious one
+payment at a time; it looks suspicious in the *shape* of the network, funds fanning out
+through mules and looping back, combined with behaviour like rapid pass-through. A rules
engine flags thousands of alerts a day; a classifier that scores each one by its network
shape *and* its behaviour lets investigators work the riskiest first.
The technique generalises to any "score an entity by its connections plus its behaviour"
-problem — [wafer-lot risk](/industries/manufacturing-process/semiconductor), [insurance rings](/industries/financial-services/insurance-fraud),
+problem, [wafer-lot risk](/industries/manufacturing-process/semiconductor), [insurance rings](/industries/financial-services/insurance-fraud),
[telecom abuse](/industries/technology-operations/network).
-:::tip New to machine learning?
-No background needed. Skim the [gentle primer](/advanced/machine-learning-gently) for the
-ideas in plain language — model, feature, training, and the algorithm itself — and use
-[Generate sample data](/advanced/generate-sample-data) for a sandbox to run this against.
+:::info Needs a sandbox
+This walks a transfer graph that already exists. [Generate a sandbox](/advanced/generate-sample-data)
+first, section F creates the flagged ring around `account_77310`.
:::
-:::tip Need data to run this?
-This walks a transfer graph that already exists. [Generate a sandbox](/advanced/generate-sample-data)
-first — section F creates the flagged ring around `account_77310`.
+:::tip New to machine learning?
+No background needed. Skim the [gentle primer](/advanced/machine-learning-gently) for the
+ideas in plain language, model, feature, training, and the algorithm itself.
:::
## 1. Pull the account's network
For a given account, [walk the money-flow graph](/guides/correlate-alarms) to get its
-cluster — the sub-graph of accounts and the transfers between them.
+cluster, the sub-graph of accounts and the transfers between them.
@@ -86,7 +84,7 @@ let net = api.resources.fetch_related(
This is the heart of it. Load the returned nodes and edges into a directed graph and
compute the structural signals that distinguish a laundering cluster from a normal
-account's neighbourhood — ring size, pass-through "mules", and whether the money loops
+account's neighbourhood, ring size, pass-through "mules", and whether the money loops
back to where it started.
```python
@@ -119,7 +117,7 @@ def graph_features(account_external_id):
## 3. Add behavioural features from the transaction series
-Network shape alone produces false positives — a busy merchant looks connected too.
+Network shape alone produces false positives, a busy merchant looks connected too.
Combine it with how the account *behaves*: how much flows through, and whether what comes
in goes straight back out (the pass-through ratio that defines a mule).
@@ -185,7 +183,7 @@ print(classification_report(yte, (proba > 0.5).astype(int)))
Score a new alert, **write the risk back as a series** so it's trendable, and raise a
`sar_candidate` event for the high-risk ones. Crucially, include the features that drove
-the score — an investigator needs to know *why* it's flagged, not just that it is.
+the score, an investigator needs to know *why* it's flagged, not just that it is.
```python
alert = "account_77310"
@@ -210,10 +208,18 @@ if risk > 0.7:
"in_loop": str(feat["in_loop"]),
"pass_through": f"{feat['pass_through']:.2f}",
})])
+
+# read the score back
+pts = client.timeseries.retrieve_datapoints(intellistream_datahub_sdk.RetrieveFilter(
+ ts=f"{alert}_aml_risk",
+ start=pd.Timestamp.now(tz="UTC") - pd.Timedelta(minutes=5),
+ end=pd.Timestamp.now(tz="UTC")))[0].get_datapoints()
+print(f"{len(pts)} risk score(s) stored, latest {pts[-1].value:.2f}")
+assert len(pts) >= 1, "the risk score was not written"
```
The risk score is now a live series the investigations dashboard ranks on, and each
-`sar_candidate` event arrives with its network-shape evidence attached — the alert and
+`sar_candidate` event arrives with its network-shape evidence attached, the alert and
the reason for it, together.
## Where to take it further
@@ -221,18 +227,18 @@ the reason for it, together.
- **Real explainability.** Swap the hand-picked metadata for SHAP values so every score
comes with its true top contributors.
- **Richer graph features.** Add betweenness, community detection, or counterparty
- diversity — the structural signal is deep.
+ diversity, the structural signal is deep.
- **Stream it.** Drive scoring from the [flagged-payments subscription](/industries/financial-services/aml)
so alerts are scored the moment they're raised.
## Further reading
-- **Gradient boosting** — [Wikipedia](https://en.wikipedia.org/wiki/Gradient_boosting)
-- **Graph centrality** (the network features) — [Wikipedia](https://en.wikipedia.org/wiki/Centrality)
-- **Shapley values / SHAP** (explainability) — [Wikipedia](https://en.wikipedia.org/wiki/Shapley_value)
+- **Gradient boosting**: [Wikipedia](https://en.wikipedia.org/wiki/Gradient_boosting)
+- **Graph centrality** (the network features), [Wikipedia](https://en.wikipedia.org/wiki/Centrality)
+- **Shapley values / SHAP** (explainability), [Wikipedia](https://en.wikipedia.org/wiki/Shapley_value)
## See also
-- [Correlate alarms with the graph](/guides/correlate-alarms) — the traversal these features are built on.
-- [Banking — AML](/industries/financial-services/aml) · [Insurance — fraud rings](/industries/financial-services/insurance-fraud) — the quick versions.
-- [Predictive maintenance](/advanced/predictive-maintenance) — the other end-to-end model build.
+- [Correlate alarms with the graph](/guides/correlate-alarms): the traversal these features are built on.
+- [Banking, AML](/industries/financial-services/aml) · [Insurance, fraud rings](/industries/financial-services/insurance-fraud), the quick versions.
+- [Predictive maintenance](/advanced/predictive-maintenance): the other end-to-end model build.
diff --git a/docs/advanced/generate-sample-data.mdx b/docs/advanced/generate-sample-data.mdx
index ead0398..864e7a0 100644
--- a/docs/advanced/generate-sample-data.mdx
+++ b/docs/advanced/generate-sample-data.mdx
@@ -7,7 +7,7 @@ title: Generate sample data
:::info Start here
The advanced scenarios **read** series, events and graphs that already exist. To run any
-of them end-to-end, first populate a sandbox with realistic synthetic data — that's what
+of them end-to-end, first populate a sandbox with realistic synthetic data, that's what
this page does, using the same SDK ingestion calls you'd use for real data. Point each
scenario's `external_id`s at the series you create here.
:::
@@ -23,8 +23,12 @@ import intellistream_datahub_sdk, numpy as np, pandas as pd
client = intellistream_datahub_sdk.DataHubClient.from_env()
def ingest(external_id, index, values, unit=None, name=None):
- client.timeseries.create([intellistream_datahub_sdk.TimeSeries(
- external_id=external_id, name=name or external_id, unit=unit or "value", value_type="float")])
+ try:
+ client.timeseries.create([intellistream_datahub_sdk.TimeSeries(
+ external_id=external_id, name=name or external_id, unit=unit or "value", value_type="float")])
+ except intellistream_datahub_sdk.DataHubException as e:
+ if e.status_code != 409: # a second run finds the series already there
+ raise
client.timeseries.insert_from_lists(timestamps=index, values=np.asarray(values), ts=external_id)
print(f"ingested {len(values):,} points → {external_id}")
```
@@ -34,11 +38,13 @@ def ingest(external_id, index, values, unit=None, name=None):
For [predictive maintenance](/advanced/predictive-maintenance),
[LSTM anomaly detection](/advanced/lstm-anomaly-detection) and
[health scoring](/advanced/asset-health-score): a vibration signal that runs healthy,
-then develops a fault — rising amplitude and impulsiveness over the final stretch.
+then develops a fault, rising amplitude and impulsiveness over the final stretch. It is
+written at **one reading a second**, which is the rate the predictive-maintenance page's
+`SAMPLE_HZ` and window length assume.
```python
def degrading_vibration(hours=24, fs_per_hour=3600):
- n = hours * fs_per_hour
+ n = hours * fs_per_hour # one reading a second
idx = pd.date_range(end=pd.Timestamp.now(tz="UTC"), periods=n, freq="1s")
t = np.linspace(0, 1, n)
healthy = 0.6 * np.sin(2 * np.pi * 50 * np.arange(n) / fs_per_hour) # base tone
@@ -50,8 +56,8 @@ idx, vib = degrading_vibration()
ingest("pump_07_vibration_mm_s", idx, vib, unit="mm_s")
```
-The same pump's slower signals — bearing temperature, oil pressure, and a stand-in
-anomaly score — feed the [asset health score](/advanced/asset-health-score):
+The same pump's slower signals, bearing temperature, oil pressure, and a stand-in
+anomaly score, feed the [asset health score](/advanced/asset-health-score):
```python
hrs = pd.date_range(end=pd.Timestamp.now(tz="UTC"), periods=24*60, freq="1h")
@@ -65,7 +71,7 @@ ingest("pump_07_vibration_anomaly", hrs, np.clip(np.linspace(0, 0.9, n) + np.ran
## B. A production decline curve
For [LSTM forecasting](/advanced/lstm-forecasting): a hyperbolic decline with noise and a
-weekly wobble — the shape of a producing well's rate.
+weekly wobble, the shape of a producing well's rate.
```python
def decline_curve(days=540, qi=1200, b=0.8, d=0.006):
@@ -174,7 +180,7 @@ edges += [intellistream_datahub_sdk.RelForm.by_external_ids(ring[k], ring[(k+1)
client.resources.create(nodes, edges)
```
-Fraud classification turns the money *in* and *out* of each account into features — a
+Fraud classification turns the money *in* and *out* of each account into features, a
pass-through mule shows inflow ≈ outflow, an ordinary account doesn't. Seed both series
for every account in the graph:
@@ -194,7 +200,7 @@ for acct, (base, ratio) in profiles.items():
## G. An electrical feeder load curve
For [demand forecasting](/advanced/demand-forecasting): 120 days of hourly feeder load
-with the daily double-peak, a lighter weekend, and a slow seasonal drift — the shape the
+with the daily double-peak, a lighter weekend, and a slow seasonal drift, the shape the
forecaster learns.
```python
@@ -214,7 +220,7 @@ ingest("feeder_f12_load_mw", idx, load, unit="mw")
For the [oxygen-crash early warning](/advanced/oxygen-crash-early-warning): 30 days of
dissolved oxygen and water temperature at 15-minute resolution, with a few episodes where
-oxygen plunges — the crashes the model learns to see coming.
+oxygen plunges, the crashes the model learns to see coming.
```python
def pen_oxygen(days=30):
@@ -232,7 +238,7 @@ ingest("pen_h_07_dissolved_oxygen_mg_l", idx, do, unit="mg_l")
ingest("pen_h_07_water_temp_c", idx, temp, unit="celsius")
```
-## I. Operating channels — one unit's regimes and a peer fleet
+## I. Operating channels, one unit's regimes and a peer fleet
For [K-Means](/advanced/kmeans-clustering): four channels for `unit_3` as it swings
between idle, ramp, steady and overload (the *operating regimes* section clusters on all
@@ -266,7 +272,7 @@ for asset, scale, phase in [("pump_07", 0.9, 0.0), ("pump_08", 1.05, 0.4),
## J. Correlated drilling channels, with a kick
For the multivariate [LSTM autoencoder](/advanced/lstm-anomaly-detection): section C's
-exact shape with drilling names — mud flow-in, flow-out, pit volume and standpipe pressure
+exact shape with drilling names, mud flow-in, flow-out, pit volume and standpipe pressure
move together until a late **kick** (gas influx) pushes returns and pit volume up while
standpipe pressure sags, breaking the joint pattern the autoencoder learned.
@@ -291,9 +297,9 @@ for tag, values in channels.items():
## K. Messy raw sensors to clean
For the [data pipeline & lineage](/advanced/data-cleaning-lineage) build: two *raw* engine
-sensors carrying the real-world defects that pipeline exists to fix — spike outliers,
+sensors carrying the real-world defects that pipeline exists to fix, spike outliers,
scattered `NaN` dropouts, a longer gap and a stuck/flatline stretch. (Everything
-downstream — the `*_clean`, feature and score series — is written *by* that recipe, so
+downstream, the `*_clean`, feature and score series, is written *by* that recipe, so
only these raw inputs are seeded here.)
```python
@@ -345,5 +351,5 @@ print(f"{(bar > LIMIT_BAR).sum()} of {len(bar)} readings are above {LIMIT_BAR}")
With the sandbox populated, every advanced scenario will find the series, events and
graph it reads. As you swap in real data, the only thing that changes is where the
-numbers come from — the [ingestion guide](/guides/ingest-timeseries) covers doing it at
+numbers come from, the [ingestion guide](/guides/ingest-timeseries) covers doing it at
production volume.
diff --git a/docs/advanced/kmeans-clustering.mdx b/docs/advanced/kmeans-clustering.mdx
index ffdeb47..c86fdef 100644
--- a/docs/advanced/kmeans-clustering.mdx
+++ b/docs/advanced/kmeans-clustering.mdx
@@ -1,11 +1,11 @@
---
sidebar_position: 8
-title: K-Means — regimes, cohorts & communities
+title: K-Means, regimes, cohorts & communities
---
import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';
-# K-Means — operating regimes, asset cohorts & network communities
+# K-Means, operating regimes, asset cohorts & network communities
:::info At a glance
**Effort:** ~1–1.5 hours · **You'll build:** behavioural and graph feature vectors and
@@ -14,25 +14,24 @@ cluster them three ways · **Stack:** the SDK for data and traversal, plus `nump
:::
K-Means finds structure with no labels: it groups points so each sits near its cluster's
-centre. That one idea answers three different operational questions, all covered here —
+centre. That one idea answers three different operational questions, all covered here,
*what modes does this asset run in?* (operating regimes), *which assets behave alike?*
(cohorts), and *which parts of the network belong together?* (communities). The trick is
always the same: turn the thing you want to group into a **feature vector**.
-:::tip Need data to run this?
+:::info Needs a sandbox
These steps read series (and a graph for section 3) that already exist.
-[Generate a sandbox](/advanced/generate-sample-data) first.
+[Generate a sandbox](/advanced/generate-sample-data) first: sections I and F.
:::
:::tip New to machine learning?
No background needed. Skim the [gentle primer](/advanced/machine-learning-gently) for the
-ideas in plain language — model, feature, training, and the algorithm itself — and use
-[Generate sample data](/advanced/generate-sample-data) for a sandbox to run this against.
+ideas in plain language, model, feature, training, and the algorithm itself.
:::
-## 1. Asset cohorts — group assets that behave alike
+## 1. Asset cohorts, group assets that behave alike
-Summarise each asset by a behavioural signature — a few aggregates of its key series —
+Summarise each asset by a behavioural signature, a few aggregates of its key series,
then cluster. Assets land in peer groups you can benchmark within.
@@ -60,7 +59,8 @@ assets = ["pump_07", "pump_08", "pump_11", "pump_19"] # ...your fleet
X = pd.DataFrame([signature(a) for a in assets], index=assets)
Xs = StandardScaler().fit_transform(X)
-km = KMeans(n_clusters=4, n_init=10, random_state=0).fit(Xs)
+# k must be well below the number of assets, or every asset is its own cluster
+km = KMeans(n_clusters=2, n_init=10, random_state=0).fit(Xs)
X["cohort"] = km.labels_
```
@@ -107,7 +107,7 @@ let sig = api.time_series
### The payoff: the asset that doesn't fit its cohort
-An asset far from its own cluster's centre is behaving unlike its peers — a strong,
+An asset far from its own cluster's centre is behaving unlike its peers, a strong,
label-free anomaly signal. Flag it.
```python
@@ -119,12 +119,16 @@ for asset, d in zip(assets, dist):
type="peer_outlier", status="open",
event_time=pd.Timestamp.now(tz="UTC"),
metadata={"asset": asset, "cohort": str(int(km.labels_[assets.index(asset)]))})])
+
+outliers = list(client.events.filter(intellistream_datahub_sdk.EventFilter(
+ basic_filter=intellistream_datahub_sdk.BasicEventFilter(type="peer_outlier"), limit=100)))
+print(f"{len(outliers)} peer outlier(s) on record")
```
-## 2. Operating regimes — group an asset's *states*
+## 2. Operating regimes, group an asset's *states*
Now cluster *time* instead of assets. Each row is one moment described by the asset's
-sensor vector; the clusters are its operating modes — idle, ramp, steady, overload.
+sensor vector; the clusters are its operating modes, idle, ramp, steady, overload.
Labelling history this way makes "we only see this fault in mode 3" answerable.
```python
@@ -147,9 +151,9 @@ regimes = KMeans(n_clusters=4, n_init=10, random_state=0).fit_predict(
states["regime"] = regimes # now every minute is tagged with its operating mode
```
-## 3. Network communities — cluster the graph
+## 3. Network communities, cluster the graph
-K-Means needs vectors, and a raw graph isn't one — so featurise it. Either **enrich the
+K-Means needs vectors, and a raw graph isn't one, so featurise it. Either **enrich the
behavioural vector with graph-structural features** (so two assets are "alike" only if
they behave *and* sit similarly in the network), or cluster the structure directly with
**spectral clustering**, which is literally K-Means on the graph Laplacian's
@@ -180,29 +184,29 @@ communities = SpectralClustering(n_clusters=2, affinity="precomputed", # k ≤
assign_labels="kmeans").fit_predict(A)
```
-So yes — K-Means works **with** graphs: feed it structural features, or let spectral
+So yes, K-Means works **with** graphs: feed it structural features, or let spectral
clustering turn the network into vectors for it.
## Choosing k
-Don't guess the number of clusters — let the data say. The **silhouette score** peaks at
+Don't guess the number of clusters, let the data say. The **silhouette score** peaks at
the k that separates clusters best:
```python
from sklearn.metrics import silhouette_score
scores = {k: silhouette_score(Xs, KMeans(k, n_init=10, random_state=0).fit_predict(Xs))
- for k in range(2, 8)}
+ for k in range(2, len(Xs))} # the silhouette needs fewer clusters than points
best_k = max(scores, key=scores.get)
```
## Further reading
-- **k-means** — [Wikipedia](https://en.wikipedia.org/wiki/K-means_clustering)
-- **Spectral clustering** (clustering a graph) — [Wikipedia](https://en.wikipedia.org/wiki/Spectral_clustering)
-- **Silhouette score** (choosing k) — [Wikipedia](https://en.wikipedia.org/wiki/Silhouette_%28clustering%29)
+- **k-means**: [Wikipedia](https://en.wikipedia.org/wiki/K-means_clustering)
+- **Spectral clustering** (clustering a graph), [Wikipedia](https://en.wikipedia.org/wiki/Spectral_clustering)
+- **Silhouette score** (choosing k), [Wikipedia](https://en.wikipedia.org/wiki/Silhouette_%28clustering%29)
## See also
-- [Correlate alarms with the graph](/guides/correlate-alarms) — the traversal the graph features build on.
-- [Asset health scoring](/advanced/asset-health-score) — score within a cohort once you have one.
-- [Fraud classification](/advanced/fraud-classification) — graph features feeding a supervised model.
+- [Correlate alarms with the graph](/guides/correlate-alarms): the traversal the graph features build on.
+- [Asset health scoring](/advanced/asset-health-score): score within a cohort once you have one.
+- [Fraud classification](/advanced/fraud-classification): graph features feeding a supervised model.
diff --git a/docs/advanced/lstm-anomaly-detection.mdx b/docs/advanced/lstm-anomaly-detection.mdx
index 4446b40..6485ad4 100644
--- a/docs/advanced/lstm-anomaly-detection.mdx
+++ b/docs/advanced/lstm-anomaly-detection.mdx
@@ -18,17 +18,16 @@ drilling kick is the classic case: no single channel is alarming, but the joint
of mud flow-in, flow-out, pit volume and standpipe pressure over a few seconds is
unmistakable. An **LSTM autoencoder** learns to reconstruct that normal joint behaviour;
when a real event arrives, it can't reconstruct it well, and the reconstruction error
-spikes. It needs **no examples of failure** — only normal operation.
+spikes. It needs **no examples of failure**, only normal operation.
-:::tip Need data to run this?
+:::info Needs a sandbox
These steps read series that already exist. [Generate a sandbox](/advanced/generate-sample-data)
-first — section J ingests the correlated drilling channels (with an injected kick) this build watches.
+first, section J ingests the correlated drilling channels (with an injected kick) this build watches.
:::
:::tip New to machine learning?
No background needed. Skim the [gentle primer](/advanced/machine-learning-gently) for the
-ideas in plain language — model, feature, training, and the algorithm itself — and use
-[Generate sample data](/advanced/generate-sample-data) for a sandbox to run this against.
+ideas in plain language, model, feature, training, and the algorithm itself.
:::
## 1. Load the normal multivariate window
@@ -51,7 +50,7 @@ def load(external_id, start, end):
rf = intellistream_datahub_sdk.RetrieveFilter(ts=external_id, start=start, end=end, limit=100_000)
pts = client.timeseries.retrieve_datapoints(rf)[0].get_datapoints()
return pd.Series([float(p.value) for p in pts],
- index=pd.to_datetime([p.timestamp for p in pts])).sort_index()
+ index=pd.to_datetime([p.timestamp for p in pts], utc=True)).sort_index()
start = pd.Timestamp.now(tz="UTC") - pd.Timedelta(hours=1) # a stretch of normal running
end = pd.Timestamp.now(tz="UTC") - pd.Timedelta(minutes=12) # ends before the recent kick
@@ -103,7 +102,7 @@ let series = api.time_series
## 2. Build sequences and the autoencoder
Slice the multivariate signal into fixed windows of shape `(timesteps, channels)`. The
-autoencoder squeezes each window through a bottleneck and rebuilds it — learning only
+autoencoder squeezes each window through a bottleneck and rebuilds it, learning only
what normal sequences look like.
```python
@@ -141,7 +140,7 @@ THRESHOLD = np.percentile(train_error, 99.5)
## 4. Watch live and raise an event
Pull the most recent window, score it, write the anomaly score back as a series, and
-raise an event when it stays above the limit — catching the kick from the joint pattern,
+raise an event when it stays above the limit, catching the kick from the joint pattern,
seconds in.
```python
@@ -164,23 +163,30 @@ if error[-3:].mean() > THRESHOLD:
type="kick_detected", status="critical",
event_time=score_index[-1],
metadata={"rig": "rig_deepwater_1", "score": f"{error[-1]:.3f}", "model": "lstm_ae_v1"})])
+
+# read the score back: one point per scored second
+stored = client.timeseries.retrieve_datapoints(intellistream_datahub_sdk.RetrieveFilter(
+ ts="rig_dw1_anomaly_score", start=score_index[0], end=pd.Timestamp.now(tz="UTC"),
+ limit=10_000))[0].get_datapoints()
+print(f"{len(stored)} scores stored, latest {stored[-1].value:.3f} against a limit of {THRESHOLD:.3f}")
+assert len(stored) == len(error), f"expected {len(error)}, got {len(stored)}"
```
## Where to take it further
- **Attribute the anomaly.** Per-channel reconstruction error shows *which* signal broke
- the pattern — flow-out vs. pit volume points to different failure modes.
+ the pattern, flow-out vs. pit volume points to different failure modes.
- **Compare with the simpler detector.** The [Isolation Forest](/advanced/predictive-maintenance)
on engineered features is cheaper; reach for the LSTM when the *temporal* pattern matters.
## Further reading
-- **LSTM** — [Wikipedia](https://en.wikipedia.org/wiki/Long_short-term_memory)
-- **Autoencoder** — [Wikipedia](https://en.wikipedia.org/wiki/Autoencoder)
-- **Anomaly detection** — [Wikipedia](https://en.wikipedia.org/wiki/Anomaly_detection)
+- **LSTM**: [Wikipedia](https://en.wikipedia.org/wiki/Long_short-term_memory)
+- **Autoencoder**: [Wikipedia](https://en.wikipedia.org/wiki/Autoencoder)
+- **Anomaly detection**: [Wikipedia](https://en.wikipedia.org/wiki/Anomaly_detection)
## See also
-- [Oil & gas — drilling operations](/industries/oil-and-gas/drilling) — the reactive version this upgrades.
-- [Predictive maintenance](/advanced/predictive-maintenance) — feature-based anomaly detection.
-- [Turn readings into events](/guides/detect-events) — emitting the alarm.
+- [Oil & gas, drilling operations](/industries/oil-and-gas/drilling): the reactive version this upgrades.
+- [Predictive maintenance](/advanced/predictive-maintenance): feature-based anomaly detection.
+- [Turn readings into events](/guides/detect-events): emitting the alarm.
diff --git a/docs/advanced/lstm-forecasting.mdx b/docs/advanced/lstm-forecasting.mdx
index 3650d01..4cf5555 100644
--- a/docs/advanced/lstm-forecasting.mdx
+++ b/docs/advanced/lstm-forecasting.mdx
@@ -15,26 +15,25 @@ temporal dynamics directly · **Stack:** the SDK for data in/out, plus `numpy`,
The [gradient-boosted forecaster](/advanced/demand-forecasting) predicts from
hand-engineered lag features. An **LSTM** takes a different route: it reads the raw
-sequence and *learns* the temporal structure itself — useful when the dynamics are
+sequence and *learns* the temporal structure itself, useful when the dynamics are
non-linear and the right lags aren't obvious, like an oil well's production decline,
where rate, pressure and water-cut interact over time.
We'll forecast a well's oil rate, but the pattern is the same for any signal with memory.
-:::tip Need data to run this?
+:::info Needs a sandbox
These steps read a series that already exists. [Generate a sandbox](/advanced/generate-sample-data)
-first — section B ingests the decline curve this build forecasts.
+first, section B ingests the decline curve this build forecasts.
:::
:::tip New to machine learning?
No background needed. Skim the [gentle primer](/advanced/machine-learning-gently) for the
-ideas in plain language — model, feature, training, and the algorithm itself — and use
-[Generate sample data](/advanced/generate-sample-data) for a sandbox to run this against.
+ideas in plain language, model, feature, training, and the algorithm itself.
:::
## 1. Load the history
-Pull a long, regularly-sampled history and scale it — neural nets train best on values
+Pull a long, regularly-sampled history and scale it, neural nets train best on values
in a small range.
@@ -54,7 +53,7 @@ rf = intellistream_datahub_sdk.RetrieveFilter(
pts = client.timeseries.retrieve_datapoints(rf)[0].get_datapoints()
rate = pd.Series([float(p.average) for p in pts],
- index=pd.to_datetime([p.timestamp for p in pts])).sort_index().asfreq("1d").interpolate()
+ index=pd.to_datetime([p.timestamp for p in pts], utc=True)).sort_index().asfreq("1d").interpolate()
scaler = MinMaxScaler()
scaled = scaler.fit_transform(rate.to_numpy().reshape(-1, 1))
@@ -143,7 +142,7 @@ model.fit(Xtr, Ytr, validation_data=(Xte, Yte),
## 4. Forecast the horizon and write it back
-The model outputs the whole horizon in a single call — feed it the most recent
+The model outputs the whole horizon in a single call, feed it the most recent
`LOOKBACK` days, inverse-scale the result, and publish it as its own series.
```python
@@ -155,28 +154,35 @@ client.timeseries.create([intellistream_datahub_sdk.TimeSeries(
external_id="well_a12_oil_rate_forecast", name="Well A-12 oil-rate forecast", unit="bpd", value_type="float")])
client.timeseries.insert_from_lists(
timestamps=index, values=forecast, ts="well_a12_oil_rate_forecast")
+
+# read the horizon back: one point per forecast day
+stored = client.timeseries.retrieve_datapoints(intellistream_datahub_sdk.RetrieveFilter(
+ ts="well_a12_oil_rate_forecast",
+ start=index[0], end=index[-1] + pd.Timedelta(minutes=1), limit=1000))[0].get_datapoints()
+print(f"{len(stored)} forecast days stored")
+assert len(stored) == HORIZON, f"expected {HORIZON}, got {len(stored)}"
```
-The decline curve is now a stored series — chart it against actuals, and a
+The decline curve is now a stored series, chart it against actuals, and a
[threshold rule](/guides/detect-events) on it flags when the well is forecast to drop
below its economic limit.
## Where to take it further
- **Go multivariate.** Feed rate, tubing pressure and water-cut together (input shape
- `(N, 3)`) so the model uses their interaction — usually a big accuracy gain.
-- **Encoder–decoder.** For long horizons, a seq2seq LSTM decodes the horizon internally —
+ `(N, 3)`) so the model uses their interaction, usually a big accuracy gain.
+- **Encoder–decoder.** For long horizons, a seq2seq LSTM decodes the horizon internally,
more expressive than a single dense layer when the forecast shape is complex.
- **Quantiles.** Train with a pinball loss for a P10/P90 band, not just a point forecast.
## Further reading
-- **LSTM** — [Wikipedia](https://en.wikipedia.org/wiki/Long_short-term_memory) · [Keras](https://keras.io/api/layers/recurrent_layers/lstm/)
-- **Recurrent neural networks** — [Wikipedia](https://en.wikipedia.org/wiki/Recurrent_neural_network)
-- **Decline curve analysis** (the domain) — [Wikipedia](https://en.wikipedia.org/wiki/Decline_curve_analysis)
+- **LSTM**: [Wikipedia](https://en.wikipedia.org/wiki/Long_short-term_memory) · [Keras](https://keras.io/api/layers/recurrent_layers/lstm/)
+- **Recurrent neural networks**: [Wikipedia](https://en.wikipedia.org/wiki/Recurrent_neural_network)
+- **Decline curve analysis** (the domain), [Wikipedia](https://en.wikipedia.org/wiki/Decline_curve_analysis)
## See also
-- [Demand forecasting](/advanced/demand-forecasting) — the feature-based alternative.
-- [Oil & gas — production monitoring](/industries/oil-and-gas/production) — the domain this serves.
-- [Turn readings into events](/guides/detect-events) — alerting on the forecast.
+- [Demand forecasting](/advanced/demand-forecasting): the feature-based alternative.
+- [Oil & gas, production monitoring](/industries/oil-and-gas/production): the domain this serves.
+- [Turn readings into events](/guides/detect-events): alerting on the forecast.
diff --git a/docs/advanced/machine-learning-gently.mdx b/docs/advanced/machine-learning-gently.mdx
index 6eeee54..3c88de9 100644
--- a/docs/advanced/machine-learning-gently.mdx
+++ b/docs/advanced/machine-learning-gently.mdx
@@ -15,23 +15,23 @@ The advanced scenarios all follow the same simple shape:
> **get data out of the platform → learn a pattern from it → write a result back.**
-The SDK does the first and last parts (you already know those). The middle part — the
-"learn a pattern" bit — is what people call *machine learning*. Here's all you need to
+The SDK does the first and last parts (you already know those). The middle part, the
+"learn a pattern" bit, is what people call *machine learning*. Here's all you need to
know about it to follow along.
-## The five words that unlock everything
+## Five terms
-- **Model** — a thing that has *learned a pattern* from past data and can apply it to new
+- **Model**: a thing that has *learned a pattern* from past data and can apply it to new
data. Think of it as a function you didn't write by hand; you showed it examples and it
worked the rule out itself.
-- **Feature** — a single number that describes something useful about your data. "The
+- **Feature**: a single number that describes something useful about your data. "The
average vibration over the last minute" is a feature. Models work on lists of features,
not raw readings.
-- **Training** — showing the model lots of past examples so it can find the pattern. Done
+- **Training**: showing the model lots of past examples so it can find the pattern. Done
once, up front; afterwards the model is fast.
-- **Label** — the answer you want to predict, when you have it. "This pump failed" is a
+- **Label**: the answer you want to predict, when you have it. "This pump failed" is a
label. Some methods need labels (you teach by example); some don't.
-- **Score** — the model's output for a new example: a predicted number, a probability, or
+- **Score**: the model's output for a new example: a predicted number, a probability, or
a group it belongs to. The scenarios write this back to the platform as a new series or
an event.
@@ -52,56 +52,56 @@ asking:
## The algorithms, one line each
-Each scenario uses one of these. You don't need to understand the maths — just the
+Each scenario uses one of these. You don't need to understand the maths, just the
one-line idea and *when* you'd reach for it. The links go to plain explanations if you're
curious.
**For spotting the unusual (anomaly detection)**
-- **Isolation Forest** — learns what "normal" looks like and flags anything that stands
+- **Isolation Forest**: learns what "normal" looks like and flags anything that stands
apart, with no examples of failure needed.
[What it is](https://en.wikipedia.org/wiki/Isolation_forest) ·
used in [Predictive maintenance](/advanced/predictive-maintenance).
-- **Autoencoder** — a model that learns to *redraw* normal data; when it can't redraw
+- **Autoencoder**: a model that learns to *redraw* normal data; when it can't redraw
something well, that thing is abnormal.
[What it is](https://en.wikipedia.org/wiki/Autoencoder) ·
used in [LSTM anomaly detection](/advanced/lstm-anomaly-detection).
-- **PCA** — boils dozens of related sensors down to a few summary numbers, then notices
+- **PCA**: boils dozens of related sensors down to a few summary numbers, then notices
when they stop relating to each other the normal way.
[What it is](https://en.wikipedia.org/wiki/Principal_component_analysis) ·
used in [Process monitoring with PCA](/advanced/pca-process-monitoring).
**For predicting the future (forecasting)**
-- **LSTM** — a neural network with a memory, good at learning patterns that play out over
+- **LSTM**: a neural network with a memory, good at learning patterns that play out over
time (like a decline curve or a daily cycle).
[What it is](https://en.wikipedia.org/wiki/Long_short-term_memory) ·
used in [LSTM forecasting](/advanced/lstm-forecasting).
-- **Gradient boosting** — builds many tiny rules-of-thumb that together make accurate
+- **Gradient boosting**: builds many tiny rules-of-thumb that together make accurate
predictions from tabular features; the dependable workhorse.
[What it is](https://en.wikipedia.org/wiki/Gradient_boosting) ·
used in [Demand forecasting](/advanced/demand-forecasting).
**For predicting a label (classification)**
-- **XGBoost** — a fast, very popular version of gradient boosting, with a handy report of
+- **XGBoost**: a fast, very popular version of gradient boosting, with a handy report of
which features mattered most.
[What it is](https://en.wikipedia.org/wiki/XGBoost) ·
used in [Failure prediction](/advanced/xgboost-failure-prediction).
-- **Random Forest** — averages the opinions of many decision trees; robust and almost
+- **Random Forest**: averages the opinions of many decision trees; robust and almost
tuning-free. Works for labels *or* numbers.
[What it is](https://en.wikipedia.org/wiki/Random_forest) ·
used in [Soft sensor](/advanced/random-forest-soft-sensor).
**For grouping (clustering)**
-- **K-Means** — sorts things into a chosen number of groups so that members of a group are
+- **K-Means**: sorts things into a chosen number of groups so that members of a group are
alike. [What it is](https://en.wikipedia.org/wiki/K-means_clustering) ·
used in [K-Means clustering](/advanced/kmeans-clustering).
## What you actually need to run these
-The modelling code is **Python**, because that's where the data-science tools live —
+The modelling code is **Python**, because that's where the data-science tools live,
mainly [`scikit-learn`](https://scikit-learn.org/) (the standard ML toolbox),
[`pandas`](https://pandas.pydata.org/) (tables of data) and, for the neural networks,
[`tensorflow`/`keras`](https://keras.io/). Install with `pip`; each scenario lists what
@@ -109,16 +109,16 @@ it needs at the top.
The DataHub SDK's job is unchanged: it **gets the data out** (the same `retrieve` calls
you've seen) and **writes results back** (the same `ingest` and event calls). The Java and
-Rust clients do those data steps too — the [reference](/reference/client) has the
-equivalents — but the learning step in the middle is Python.
+Rust clients do those data steps too, the [reference](/reference/client) has the
+equivalents, but the learning step in the middle is Python.
## A safe way to start
1. **Populate a sandbox.** [Generate sample data](/advanced/generate-sample-data) creates
- realistic signals to practise on — nothing touches real systems.
+ realistic signals to practise on, nothing touches real systems.
2. **Pick the scenario that matches your question** from the four jobs above.
3. **Run it top to bottom.** Each one is written to be followed step by step, with the
*why* spelled out as you go.
You won't break anything by experimenting, and you don't have to understand the maths to
-get a useful result — that's rather the point.
+get a useful result, that's rather the point.
diff --git a/docs/advanced/oxygen-crash-early-warning.mdx b/docs/advanced/oxygen-crash-early-warning.mdx
index d654433..8c146f6 100644
--- a/docs/advanced/oxygen-crash-early-warning.mdx
+++ b/docs/advanced/oxygen-crash-early-warning.mdx
@@ -5,7 +5,7 @@ title: Early warning (predict before it happens)
import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';
-# Early warning — predicting a crash before it happens
+# Early warning, predicting a crash before it happens
:::info At a glance
**Effort:** ~1 hour · **You'll build:** a forward-looking labelled dataset and a
@@ -14,30 +14,29 @@ classifier that fires *ahead* of the event · **Stack:** the SDK for data in/out
:::
A threshold alarm tells you the bad thing is *already happening*. For some problems
-that's too late — by the time dissolved oxygen in a [salmon pen](/industries/agriculture-food/salmon-farming)
+that's too late, by the time dissolved oxygen in a [salmon pen](/industries/agriculture-food/salmon-farming)
hits the danger line, fish are already stressed. The goal here is to fire **before** the
crash: learn the early signature of a developing crash and predict it 45 minutes out,
while aeration can still prevent it.
-The technique — a classifier trained on a **forward-looking label** — generalises to any
+The technique, a classifier trained on a **forward-looking label**, generalises to any
"predict the incident ahead of time" problem: a transformer about to trip, a server
about to breach SLO, a line about to jam.
-:::tip New to machine learning?
-No background needed. Skim the [gentle primer](/advanced/machine-learning-gently) for the
-ideas in plain language — model, feature, training, and the algorithm itself — and use
-[Generate sample data](/advanced/generate-sample-data) for a sandbox to run this against.
-:::
-
-:::tip Need data to run this?
+:::info Needs a sandbox
This trains on pen-oxygen history. [Generate a sandbox](/advanced/generate-sample-data)
-first — section H ingests `pen_h_07_dissolved_oxygen_mg_l` and `pen_h_07_water_temp_c`
+first, section H ingests `pen_h_07_dissolved_oxygen_mg_l` and `pen_h_07_water_temp_c`
with the crash episodes the model learns.
:::
+:::tip New to machine learning?
+No background needed. Skim the [gentle primer](/advanced/machine-learning-gently) for the
+ideas in plain language, model, feature, training, and the algorithm itself.
+:::
+
## 1. Load the signals that precede a crash
-Pull the pen's history at a few-minute cadence — the oxygen itself plus the drivers that
+Pull the pen's history at a few-minute cadence, the oxygen itself plus the drivers that
move it (temperature, and tide if you have it).
```python
@@ -53,7 +52,7 @@ def load(external_id, days=90):
aggregates=["avg"], granularity="5m", limit=100_000)
pts = client.timeseries.retrieve_datapoints(rf)[0].get_datapoints()
return pd.Series([float(p.average) for p in pts],
- index=pd.to_datetime([p.timestamp for p in pts])).sort_index()
+ index=pd.to_datetime([p.timestamp for p in pts], utc=True)).sort_index()
do = load("pen_h_07_dissolved_oxygen_mg_l")
temp = load("pen_h_07_water_temp_c")
@@ -86,7 +85,7 @@ FEATURES = ["do", "do_slope", "do_std", "temp", "temp_slope", "hour"]
## 3. Train the classifier
-Crashes are rare, so the classes are imbalanced — weight them, and judge the model on
+Crashes are rare, so the classes are imbalanced, weight them, and judge the model on
precision/recall for the *crash* class, not raw accuracy.
```python
@@ -96,7 +95,7 @@ from sklearn.metrics import classification_report
split = int(len(frame) * 0.8)
train, test = frame.iloc[:split], frame.iloc[split:]
-# class_weight balances the rare positive class
+# sample_weight up-weights the rare positive class
sample_weight = np.where(train["will_crash"] == 1, 20.0, 1.0)
clf = HistGradientBoostingClassifier(max_iter=300, learning_rate=0.05)
clf.fit(train[FEATURES], train["will_crash"], sample_weight=sample_weight)
@@ -107,8 +106,8 @@ print(classification_report(test["will_crash"], clf.predict(test[FEATURES])))
## 4. Score live and warn ahead of time
Take the latest reading, predict the crash probability, and when it's high raise an
-`oxygen_crash_predicted` event — minutes before the [reactive low-oxygen
-alarm](/industries/agriculture-food/salmon-farming) would ever fire — so aeration starts in time.
+`oxygen_crash_predicted` event, minutes before the [reactive low-oxygen
+alarm](/industries/agriculture-food/salmon-farming) would ever fire, so aeration starts in time.
```python
latest = frame.iloc[[-1]]
@@ -126,14 +125,22 @@ if prob > 0.6:
type="oxygen_crash_predicted", status="warning",
event_time=latest.index[-1],
metadata={"pen": "pen_h_07", "risk": f"{prob:.2f}", "lead_minutes": "45"})])
+
+# read the risk back
+pts = client.timeseries.retrieve_datapoints(intellistream_datahub_sdk.RetrieveFilter(
+ ts="pen_h_07_crash_risk",
+ start=latest.index[-1] - pd.Timedelta(minutes=1),
+ end=pd.Timestamp.now(tz="UTC")))[0].get_datapoints()
+print(f"{len(pts)} risk value(s) stored, latest {pts[-1].value:.2f}")
+assert len(pts) >= 1, "the risk score was not written"
```
The risk score is now a live series, and the event gives the farm a 45-minute head
-start — the difference between a near-miss and a lost pen.
+start, the difference between a near-miss and a lost pen.
## Where to take it further
-- **Tune the lead time.** A longer `HORIZON` warns earlier but with more false alarms —
+- **Tune the lead time.** A longer `HORIZON` warns earlier but with more false alarms,
pick the trade-off the operation can act on.
- **Add neighbours.** Pens sharing a water current crash together; add neighbouring pens'
oxygen as features, or correlate via [the graph](/guides/correlate-alarms).
@@ -142,12 +149,12 @@ start — the difference between a near-miss and a lost pen.
## Further reading
-- **Classification** — [Wikipedia](https://en.wikipedia.org/wiki/Statistical_classification)
-- **Gradient boosting** — [Wikipedia](https://en.wikipedia.org/wiki/Gradient_boosting)
-- **Precision & recall** (judging a rare-event model) — [Wikipedia](https://en.wikipedia.org/wiki/Precision_and_recall)
+- **Classification**: [Wikipedia](https://en.wikipedia.org/wiki/Statistical_classification)
+- **Gradient boosting**: [Wikipedia](https://en.wikipedia.org/wiki/Gradient_boosting)
+- **Precision & recall** (judging a rare-event model), [Wikipedia](https://en.wikipedia.org/wiki/Precision_and_recall)
## See also
-- [Aquaculture — salmon farming](/industries/agriculture-food/salmon-farming) — the reactive version this upgrades.
-- [Turn readings into events](/guides/detect-events) — emitting the predictive warning.
-- [Query & aggregate](/guides/query-and-aggregate) — assembling the training history.
+- [Aquaculture, salmon farming](/industries/agriculture-food/salmon-farming): the reactive version this upgrades.
+- [Turn readings into events](/guides/detect-events): emitting the predictive warning.
+- [Query & aggregate](/guides/query-and-aggregate): assembling the training history.
diff --git a/docs/advanced/pca-process-monitoring.mdx b/docs/advanced/pca-process-monitoring.mdx
index 4863fe5..067744c 100644
--- a/docs/advanced/pca-process-monitoring.mdx
+++ b/docs/advanced/pca-process-monitoring.mdx
@@ -13,23 +13,22 @@ T² and Q monitoring statistics · **Stack:** the SDK for data in/out, plus `num
`pandas` and `scikit-learn`.
:::
-A process unit has dozens of sensors that all move together — when feed rises,
+A process unit has dozens of sensors that all move together, when feed rises,
temperatures, pressures and flows respond in a fixed, correlated dance. A fault breaks
that dance: the sensors stop relating to each other the way they should, often *before*
any single one crosses an alarm limit. **PCA** learns the normal correlation structure
-and reduces it to a few components; two statistics — **Hotelling's T²** and the **squared
-prediction error (Q / SPE)** — then flag when live data no longer fits, and a
+and reduces it to a few components; two statistics, **Hotelling's T²** and the **squared
+prediction error (Q / SPE)**, then flag when live data no longer fits, and a
contribution check points at the sensor responsible.
-:::tip Need data to run this?
+:::info Needs a sandbox
These steps read correlated process sensors that already exist.
-[Generate a sandbox](/advanced/generate-sample-data) first — section C ingests them, fault and all.
+[Generate a sandbox](/advanced/generate-sample-data) first, section C ingests them, fault and all.
:::
:::tip New to machine learning?
No background needed. Skim the [gentle primer](/advanced/machine-learning-gently) for the
-ideas in plain language — model, feature, training, and the algorithm itself — and use
-[Generate sample data](/advanced/generate-sample-data) for a sandbox to run this against.
+ideas in plain language, model, feature, training, and the algorithm itself.
:::
## 1. Learn normal operation
@@ -55,7 +54,7 @@ def series(tag, start, end):
aggregates=["avg"], granularity="1m", limit=100_000)
pts = client.timeseries.retrieve_datapoints(rf)[0].get_datapoints()
return pd.Series([float(p.average) for p in pts],
- index=pd.to_datetime([p.timestamp for p in pts])).sort_index()
+ index=pd.to_datetime([p.timestamp for p in pts], utc=True)).sort_index()
s = pd.Timestamp.now(tz="UTC") - pd.Timedelta(days=14) # a clean stretch of normal running
e = pd.Timestamp.now(tz="UTC") - pd.Timedelta(days=1) # ends before the recent fault window
@@ -112,7 +111,7 @@ let pts = api.time_series
## 2. Define the monitoring statistics and their limits
**T²** measures how far a point sits inside the model's space (an unusual but in-pattern
-state); **Q/SPE** measures how far it sits *outside* the model (a broken correlation —
+state); **Q/SPE** measures how far it sits *outside* the model (a broken correlation,
usually the real fault). Set each limit from the normal data.
```python
@@ -134,7 +133,7 @@ SPE_LIMIT = np.percentile(spe(Z), 99)
## 3. Monitor live and name the culprit sensor
Score the latest data; when either statistic exceeds its limit the process has moved off
-normal. The biggest term in the residual is the sensor most responsible — attach it to
+normal. The biggest term in the residual is the sensor most responsible, attach it to
the event so operators know *where* to look.
```python
@@ -153,27 +152,33 @@ if t2(znew)[-1] > T2_LIMIT or spe(znew)[-1] > SPE_LIMIT:
metadata={"unit": "crude_unit_1",
"t2": f"{t2(znew)[-1]:.1f}", "spe": f"{spe(znew)[-1]:.1f}",
"top_contributor": culprit})])
+
+# the seeded fault sits in the last ten hours, so the deviation must be on record
+found = list(client.events.filter(intellistream_datahub_sdk.EventFilter(
+ basic_filter=intellistream_datahub_sdk.BasicEventFilter(type="process_deviation"), limit=10)))
+assert found, "the seeded fault should have raised a process_deviation event"
+print(f"{len(found)} process deviation(s) on record, latest names {found[-1].metadata['top_contributor']}")
```
-You can also publish T² and SPE as their own series for a live "process health" chart —
+You can also publish T² and SPE as their own series for a live "process health" chart,
two lines that capture the state of dozens of sensors at once.
## Where to take it further
- **Dimensionality first.** PCA is also a feature-reduction step *before* clustering or a
- classifier — feed the components into [K-Means](/advanced/kmeans-clustering) instead of
+ classifier, feed the components into [K-Means](/advanced/kmeans-clustering) instead of
raw tags.
- **Dynamic PCA.** Include lagged sensor values so the model captures process dynamics,
not just instantaneous correlation.
## Further reading
-- **Principal component analysis** — [Wikipedia](https://en.wikipedia.org/wiki/Principal_component_analysis)
-- **Hotelling's T²** — [Wikipedia](https://en.wikipedia.org/wiki/Hotelling%27s_T-squared_distribution)
-- **Multivariate process monitoring** — [Statistical process control](https://en.wikipedia.org/wiki/Statistical_process_control)
+- **Principal component analysis**: [Wikipedia](https://en.wikipedia.org/wiki/Principal_component_analysis)
+- **Hotelling's T²**: [Wikipedia](https://en.wikipedia.org/wiki/Hotelling%27s_T-squared_distribution)
+- **Multivariate process monitoring**: [Statistical process control](https://en.wikipedia.org/wiki/Statistical_process_control)
## See also
-- [Oil & gas — refinery operations](/industries/oil-and-gas/refining) — the unit this watches.
-- [Predictive maintenance](/advanced/predictive-maintenance) · [LSTM anomaly detection](/advanced/lstm-anomaly-detection) — other ways to flag the abnormal.
-- [Turn readings into events](/guides/detect-events) — emitting the deviation.
+- [Oil & gas, refinery operations](/industries/oil-and-gas/refining): the unit this watches.
+- [Predictive maintenance](/advanced/predictive-maintenance) · [LSTM anomaly detection](/advanced/lstm-anomaly-detection), other ways to flag the abnormal.
+- [Turn readings into events](/guides/detect-events): emitting the deviation.
diff --git a/docs/advanced/predictive-maintenance.mdx b/docs/advanced/predictive-maintenance.mdx
index eb73299..288b594 100644
--- a/docs/advanced/predictive-maintenance.mdx
+++ b/docs/advanced/predictive-maintenance.mdx
@@ -5,37 +5,36 @@ title: Predictive maintenance (anomaly detection)
import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';
-# Predictive maintenance — vibration anomaly detection
+# Predictive maintenance, vibration anomaly detection
:::info At a glance
**Effort:** ~1–2 hours · **What you'll build:** a model that learns a machine's *healthy*
vibration and warns you when it starts to drift · **Stack:** the SDK for data in and out,
-plus three Python libraries — `numpy`, `scipy`, `scikit-learn`.
-:::
-
-:::tip New to this?
-No machine-learning background needed. If a word here is unfamiliar, the
-[gentle primer](/advanced/machine-learning-gently) explains it in one line. And to run
-this end-to-end, [generate a sandbox](/advanced/generate-sample-data) first (section A
-creates the vibration signal used below).
+plus three Python libraries, `numpy`, `scipy`, `scikit-learn`.
:::
**The idea in one paragraph.** A failing bearing vibrates differently long before it
-breaks — but not *louder*, so a simple "is it above X?" alarm misses it. The difference
+breaks, but not *louder*, so a simple "is it above X?" alarm misses it. The difference
is in the *texture* of the vibration: tiny repeated shocks, energy shifting into certain
frequencies. We'll let a model look at lots of examples of the machine running *healthy*,
learn what healthy texture looks like, and then raise a flag when new data stops looking
-like it. Nothing here needs labelled failures — only normal running.
+like it. Nothing here needs labelled failures, only normal running.
-The [quick examples](/industries/transport-logistics/aerospace) raise an alarm on a raw threshold; this is
-the grown-up version that *learns* normal first.
+The [quick examples](/industries/transport-logistics/aerospace) raise an alarm on a raw threshold; this
+version learns normal first.
-:::tip Need data to run this?
+:::info Needs a sandbox
These steps read a vibration series that already exists. [Generate a sandbox](/advanced/generate-sample-data)
-first — section A ingests the degrading `pump_07_vibration_mm_s` this build scores.
+first, section A ingests the degrading `pump_07_vibration_mm_s` this build scores, one reading
+a second.
+:::
+
+:::tip New to machine learning?
+No background needed. If a word here is unfamiliar, the
+[gentle primer](/advanced/machine-learning-gently) explains it in one line.
:::
-## Step 1 — Get a stretch of healthy data
+## Step 1: Get a stretch of healthy data
First we need examples of the machine running well. We pull a stretch of vibration readings
from a period the machine was known to be healthy, and load them into a plain array of
@@ -52,7 +51,7 @@ client = intellistream_datahub_sdk.DataHubClient.from_env()
def load_series(external_id, start, end):
rf = intellistream_datahub_sdk.RetrieveFilter(ts=external_id, start=start, end=end, limit=100_000)
points = client.timeseries.retrieve_datapoints(rf)[0].get_datapoints()
- idx = pd.to_datetime([p.timestamp for p in points])
+ idx = pd.to_datetime([p.timestamp for p in points], utc=True)
return pd.Series([float(p.value) for p in points], index=idx).sort_index()
healthy = load_series(
@@ -100,19 +99,19 @@ let series = api.time_series
-## Step 2 — Turn the raw wiggle into a few meaningful numbers
+## Step 2: Turn the raw wiggle into a few meaningful numbers
-A model can't learn much from one raw vibration value — it has no context. So we chop the
-signal into short **windows** (say, two seconds each) and, for each window, calculate a
-few numbers that capture its *texture*. These numbers are called **features**. We use
+A model can't learn much from one raw vibration value, it has no context. So we chop the
+signal into short **windows** (two minutes each, at the seed's one reading a second) and,
+for each window, calculate a few numbers that capture its *texture*. These numbers are called **features**. We use
four, and here's what each one means in plain terms:
-- **RMS** — roughly the overall "loudness" or energy of the window.
-- **Kurtosis** — how *spiky* it is. A healthy machine hums smoothly; a failing bearing
+- **RMS**: roughly the overall "loudness" or energy of the window.
+- **Kurtosis**: how *spiky* it is. A healthy machine hums smoothly; a failing bearing
adds sharp little shocks that push this number up.
-- **Crest factor** — the biggest peak compared to the average level; another way to catch
+- **Crest factor**: the biggest peak compared to the average level; another way to catch
those shocks.
-- **Band energy** — how much of the vibration sits in a particular frequency range, found
+- **Band energy**: how much of the vibration sits in a particular frequency range, found
with an [FFT](https://en.wikipedia.org/wiki/Fast_Fourier_transform) (a standard tool
that splits a wiggle into the pure tones it's made of). Bearing faults ring at
characteristic frequencies, so energy showing up there is a red flag.
@@ -121,8 +120,8 @@ four, and here's what each one means in plain terms:
from scipy.stats import kurtosis
from scipy.fft import rfft, rfftfreq
-SAMPLE_HZ = 1000 # the sensor reports 1000 readings per second
-WINDOW = SAMPLE_HZ * 2 # work in 2-second windows
+SAMPLE_HZ = 1 # the seed writes one reading per second
+WINDOW = SAMPLE_HZ * 120 # work in 2-minute windows
def features(window, fs=SAMPLE_HZ):
x = window - window.mean() # remove the average so we see the wobble
@@ -130,7 +129,7 @@ def features(window, fs=SAMPLE_HZ):
crest = np.max(np.abs(x)) / rms if rms else 0.0 # biggest peak vs. average
spectrum = np.abs(rfft(x)) # the FFT: how much of each frequency
freqs = rfftfreq(len(x), 1 / fs)
- band = spectrum[(freqs >= 120) & (freqs <= 180)] # the bearing's fault frequency range
+ band = spectrum[(freqs >= 0.1) & (freqs <= 0.4)] # the fault band; the healthy tone sits below it
band_energy = float(np.sum(band**2))
return [rms, kurtosis(x), crest, band_energy] # four numbers describe this window
@@ -142,16 +141,18 @@ def feature_matrix(series):
X_healthy = feature_matrix(healthy) # one row of four features per window
```
-So each two-second window is now just four numbers. A week of healthy data becomes a big
-table of "this is what normal looks like."
+At one reading a second the highest frequency a window can resolve is 0.5 Hz, so the fault
+band sits below that; a real 1 kHz sensor would put it at the bearing's fault frequencies
+instead. Each two-minute window is now just four numbers. Eighteen hours of healthy data
+become a table of 540 rows that says "this is what normal looks like."
-## Step 3 — Let the model learn "normal"
+## Step 3: Let the model learn "normal"
Now we hand that table to an **[Isolation Forest](https://en.wikipedia.org/wiki/Isolation_forest)**.
The one-line idea: it learns the shape of the normal cloud of points and can then tell, for
-any new point, how *far outside* that cloud it sits. It needs no examples of failure — a
+any new point, how *far outside* that cloud it sits. It needs no examples of failure, a
big deal, because you rarely have many. (We also scale the features first, so that one
-feature with big numbers doesn't drown out the others — a routine tidy-up step.)
+feature with big numbers doesn't drown out the others, a routine tidy-up step.)
```python
from sklearn.ensemble import IsolationForest
@@ -164,13 +165,16 @@ model.fit(scaler.transform(X_healthy)) # learn what heal
def anomaly_score(X):
# the model returns "how normal"; we flip the sign so higher = more abnormal
return -model.decision_function(scaler.transform(X))
+
+# the alarm level comes from the healthy data itself: the score 99% of normal windows stay under
+ALARM = float(np.percentile(anomaly_score(X_healthy), 99))
```
-## Step 4 — Score new data, and write the answer back
+## Step 4: Score new data, and write the answer back
Finally we pull the *latest* hour, turn it into the same four features, and ask the model
how abnormal each window is. We **write that score back to the platform as a new series**,
-so it's a normal, chartable signal — and if the score stays high (not just a one-off
+so it's a normal, chartable signal, and if the score stays high (not just a one-off
blip), we raise an event.
```python
@@ -181,21 +185,32 @@ X_recent = feature_matrix(recent)
scores = anomaly_score(X_recent)
score_index = recent.index[WINDOW::WINDOW][:len(scores)] # timestamp each window's score
-client.timeseries.create([intellistream_datahub_sdk.TimeSeries(
- external_id="pump_07_vibration_anomaly", name="Pump 07 vibration anomaly score", unit="score", value_type="float")])
+try:
+ client.timeseries.create([intellistream_datahub_sdk.TimeSeries(
+ external_id="pump_07_vibration_anomaly", name="Pump 07 vibration anomaly score", unit="score", value_type="float")])
+except intellistream_datahub_sdk.DataHubException as e:
+ if e.status_code != 409: # the seed's stand-in series may already be there
+ raise
client.timeseries.insert_from_lists(
timestamps=score_index, values=scores, ts="pump_07_vibration_anomaly")
-ALARM = 0.04 # healthy scores sit near/below 0; a sustained positive score is abnormal
-if scores[-5:].mean() > ALARM: # sustained, not a single spike
+if scores[-5:].mean() > ALARM: # sustained over the last ten minutes, not a single spike
client.events.create([intellistream_datahub_sdk.Event(
external_id=f"degradation_predicted_pump_07_{int(pd.Timestamp.now().timestamp())}",
type="degradation_predicted", status="open",
event_time=score_index[-1],
metadata={"asset": "pump_07", "score": f"{scores[-1]:.3f}", "model": "iforest_v1"})])
+
+# read the scores back: the seed's fault ramps in over the last few hours, so the latest hour is abnormal
+stored = client.timeseries.retrieve_datapoints(intellistream_datahub_sdk.RetrieveFilter(
+ ts="pump_07_vibration_anomaly", start=score_index[0], end=pd.Timestamp.now(tz="UTC"),
+ limit=1000))[0].get_datapoints()
+print(f"{len(stored)} scores stored, latest {stored[-1].value:.3f} against an alarm level of {ALARM:.3f}")
+assert len(stored) >= len(scores), f"expected {len(scores)} scores, found {len(stored)}"
+assert scores[-5:].mean() > ALARM, "the seeded fault should score as abnormal"
```
-That's the whole loop. The anomaly score is now a first-class series — chart it beside the
+That's the whole loop. The anomaly score is now a first-class series, chart it beside the
raw vibration, [fold it into a health score](/advanced/asset-health-score), or alert on
it like any other signal.
@@ -211,15 +226,15 @@ it like any other signal.
## Further reading
-- **Isolation Forest** — the anomaly detector used here:
+- **Isolation Forest**: the anomaly detector used here:
[Wikipedia](https://en.wikipedia.org/wiki/Isolation_forest) ·
[scikit-learn guide](https://scikit-learn.org/stable/modules/outlier_detection.html#isolation-forest)
-- **FFT (frequency analysis)** — [Wikipedia](https://en.wikipedia.org/wiki/Fast_Fourier_transform)
-- **Kurtosis (spikiness)** — [Wikipedia](https://en.wikipedia.org/wiki/Kurtosis)
-- **New to the ideas?** — [Machine learning, gently](/advanced/machine-learning-gently)
+- **FFT (frequency analysis)**: [Wikipedia](https://en.wikipedia.org/wiki/Fast_Fourier_transform)
+- **Kurtosis (spikiness)**: [Wikipedia](https://en.wikipedia.org/wiki/Kurtosis)
+- **New to the ideas?**: [Machine learning, gently](/advanced/machine-learning-gently)
## See also
-- [Query & aggregate](/guides/query-and-aggregate) — extracting the training windows.
-- [Asset health scoring](/advanced/asset-health-score) — fold this score into a composite.
-- [Failure prediction with XGBoost](/advanced/xgboost-failure-prediction) — the labelled, predict-the-type cousin.
+- [Query & aggregate](/guides/query-and-aggregate): extracting the training windows.
+- [Asset health scoring](/advanced/asset-health-score): fold this score into a composite.
+- [Failure prediction with XGBoost](/advanced/xgboost-failure-prediction): the labelled, predict-the-type cousin.
diff --git a/docs/advanced/random-forest-soft-sensor.mdx b/docs/advanced/random-forest-soft-sensor.mdx
index 6619751..4a8cb9c 100644
--- a/docs/advanced/random-forest-soft-sensor.mdx
+++ b/docs/advanced/random-forest-soft-sensor.mdx
@@ -14,21 +14,20 @@ quantity from cheap online sensors · **Stack:** the SDK for data in/out, plus `
:::
Some of the most important quantities are the hardest to measure. Product purity,
-produced-water oil content, melt viscosity — they come from a lab, hours apart, while
+produced-water oil content, melt viscosity, they come from a lab, hours apart, while
the process runs continuously. A **soft sensor** closes that gap: a model that infers
the lab value in real time from the cheap online sensors that *are* measured every
-second. Random Forest is a great fit — robust, little tuning, and an out-of-bag score
+second. Random Forest is a great fit, robust, little tuning, and an out-of-bag score
that estimates accuracy without a separate test set.
-:::tip Need data to run this?
+:::info Needs a sandbox
These steps read process sensors and sparse lab samples that already exist.
-[Generate a sandbox](/advanced/generate-sample-data) first — section D ingests both.
+[Generate a sandbox](/advanced/generate-sample-data) first, section D ingests both.
:::
:::tip New to machine learning?
No background needed. Skim the [gentle primer](/advanced/machine-learning-gently) for the
-ideas in plain language — model, feature, training, and the algorithm itself — and use
-[Generate sample data](/advanced/generate-sample-data) for a sandbox to run this against.
+ideas in plain language, model, feature, training, and the algorithm itself.
:::
## 1. Assemble lab targets against sensor features
@@ -49,7 +48,7 @@ def series(external_id, start, end, granularity="5m"):
aggregates=["avg"], granularity=granularity, limit=100_000)
pts = client.timeseries.retrieve_datapoints(rf)[0].get_datapoints()
return pd.Series([float(p.average) for p in pts],
- index=pd.to_datetime([p.timestamp for p in pts])).sort_index()
+ index=pd.to_datetime([p.timestamp for p in pts], utc=True)).sort_index()
start, end = pd.Timestamp.now(tz="UTC") - pd.Timedelta(days=14), pd.Timestamp.now(tz="UTC")
SENSORS = ["cdu_1_top_temp_c", "cdu_1_reflux_ratio", "cdu_1_feed_bpd", "cdu_1_pressure_kpa"]
@@ -106,7 +105,7 @@ let pts = api.time_series
## 2. Train the regressor
A Random Forest needs little tuning, and `oob_score` gives an honest accuracy estimate
-from the trees' out-of-bag samples — handy when labels are scarce.
+from the trees' out-of-bag samples, handy when labels are scarce.
```python
from sklearn.ensemble import RandomForestRegressor
@@ -125,7 +124,7 @@ for name, imp in sorted(zip(SENSORS, rf.feature_importances_),
## 3. Infer continuously and publish the virtual sensor
Run the model on the live sensor rows and write the prediction back as its own
-series — a continuous estimate of the lab value, updated every few minutes instead of
+series, a continuous estimate of the lab value, updated every few minutes instead of
once a shift. Now it can be charted, alerted on, and fed into control like any real tag.
```python
@@ -139,26 +138,34 @@ client.timeseries.create([intellistream_datahub_sdk.TimeSeries(
unit="ppm", value_type="float")])
client.timeseries.insert_from_lists(
timestamps=recent.index, values=virtual, ts="cdu_1_product_sulfur_ppm_soft")
+
+# read the virtual sensor back: one point per sensor row
+stored = client.timeseries.retrieve_datapoints(intellistream_datahub_sdk.RetrieveFilter(
+ ts="cdu_1_product_sulfur_ppm_soft",
+ start=recent.index[0], end=recent.index[-1] + pd.Timedelta(minutes=1),
+ limit=1000))[0].get_datapoints()
+print(f"{len(stored)} soft-sensor values stored, latest {stored[-1].value:.2f} ppm")
+assert len(stored) == len(virtual), f"expected {len(virtual)}, got {len(stored)}"
```
-When the next lab sample lands, append it to the training set and re-fit — the soft
+When the next lab sample lands, append it to the training set and re-fit, the soft
sensor stays calibrated as the process and feedstock change.
## Where to take it further
- **Quantify uncertainty.** The spread across the forest's trees gives a per-prediction
- confidence — widen alerts when the model is unsure.
+ confidence, widen alerts when the model is unsure.
- **Drift watch.** Compare each new lab sample to the soft sensor's prediction; a growing
gap is a [retrain trigger](/guides/detect-events).
## Further reading
-- **Random forest** — [Wikipedia](https://en.wikipedia.org/wiki/Random_forest)
-- **Soft sensor** — [Wikipedia](https://en.wikipedia.org/wiki/Soft_sensor)
-- **Out-of-bag error** — [Wikipedia](https://en.wikipedia.org/wiki/Out-of-bag_error)
+- **Random forest**: [Wikipedia](https://en.wikipedia.org/wiki/Random_forest)
+- **Soft sensor**: [Wikipedia](https://en.wikipedia.org/wiki/Soft_sensor)
+- **Out-of-bag error**: [Wikipedia](https://en.wikipedia.org/wiki/Out-of-bag_error)
## See also
-- [Oil & gas — refinery operations](/industries/oil-and-gas/refining) — where soft sensors earn their keep.
-- [XGBoost failure prediction](/advanced/xgboost-failure-prediction) — the boosted-tree cousin.
-- [Query & aggregate](/guides/query-and-aggregate) — assembling the feature history.
+- [Oil & gas, refinery operations](/industries/oil-and-gas/refining): where soft sensors earn their keep.
+- [XGBoost failure prediction](/advanced/xgboost-failure-prediction): the boosted-tree cousin.
+- [Query & aggregate](/guides/query-and-aggregate): assembling the feature history.
diff --git a/docs/advanced/sustained-alarm-window.mdx b/docs/advanced/sustained-alarm-window.mdx
index eb52ade..dc8f3dc 100644
--- a/docs/advanced/sustained-alarm-window.mdx
+++ b/docs/advanced/sustained-alarm-window.mdx
@@ -16,7 +16,7 @@ that is a question about a stretch of time. This page builds that rule end to en
signal, replay it to prove the rule behaves, then run the same code against a live
subscription.
-:::info Seed the data first
+:::info Needs a sandbox
Run generator **L** on [Generate sample data](/advanced/generate-sample-data#l-a-noisy-alarm-four-transients-and-one-real-excursion).
It writes `pump_p101_discharge_bar`: six hours at one reading a minute, four brief spikes
well above the limit, and one eighty-minute excursion just above it. Seeding is Python, as
@@ -519,14 +519,14 @@ readings never leave their limits and it is the shape that changed, that is wher
## Further reading
-- **Stream processing** — [Wikipedia](https://en.wikipedia.org/wiki/Stream_processing)
-- **Hysteresis** — [Wikipedia](https://en.wikipedia.org/wiki/Hysteresis)
-- **Idempotence** — [Wikipedia](https://en.wikipedia.org/wiki/Idempotence)
-- Stream processing, in the platform documentation — windows, event time against arrival time, and what a computation has to remember
+- **Stream processing**: [Wikipedia](https://en.wikipedia.org/wiki/Stream_processing)
+- **Hysteresis**: [Wikipedia](https://en.wikipedia.org/wiki/Hysteresis)
+- **Idempotence**: [Wikipedia](https://en.wikipedia.org/wiki/Idempotence)
+- Stream processing, in the platform documentation, windows, event time against arrival time, and what a computation has to remember
## See also
-- [Consume live data](/guides/realtime-subscriptions) — the delivery mechanics this builds on.
-- [Turn readings into events](/guides/detect-events) — the single-reading version of the same job.
-- [Generate sample data](/advanced/generate-sample-data#l-a-noisy-alarm-four-transients-and-one-real-excursion) — the signal this page runs against.
-- [LSTM anomaly detection](/advanced/lstm-anomaly-detection) — for the excursions no limit can describe.
+- [Consume live data](/guides/realtime-subscriptions): the delivery mechanics this builds on.
+- [Turn readings into events](/guides/detect-events): the single-reading version of the same job.
+- [Generate sample data](/advanced/generate-sample-data#l-a-noisy-alarm-four-transients-and-one-real-excursion): the signal this page runs against.
+- [LSTM anomaly detection](/advanced/lstm-anomaly-detection): for the excursions no limit can describe.
diff --git a/docs/advanced/xgboost-failure-prediction.mdx b/docs/advanced/xgboost-failure-prediction.mdx
index 05e185b..109e98a 100644
--- a/docs/advanced/xgboost-failure-prediction.mdx
+++ b/docs/advanced/xgboost-failure-prediction.mdx
@@ -14,26 +14,25 @@ plus `pandas`, `numpy` and `xgboost`.
:::
Some questions are best answered from **tabular features**, not raw sequences: *will
-this asset fail in the next week?* XGBoost is the workhorse here — it eats engineered
+this asset fail in the next week?* XGBoost is the workhorse here, it eats engineered
features, handles missing values natively, trains fast, and tells you which features
drive the prediction. We'll predict an electric submersible pump (ESP) failure seven
days out, but the recipe fits any "label the history, predict the window" problem.
-:::tip Need data to run this?
+:::info Needs a sandbox
These steps read series and failure events that already exist. [Generate a sandbox](/advanced/generate-sample-data)
-first — section E ingests the labelled failure history this build trains on.
+first, section E ingests the labelled failure history this build trains on.
:::
:::tip New to machine learning?
No background needed. Skim the [gentle primer](/advanced/machine-learning-gently) for the
-ideas in plain language — model, feature, training, and the algorithm itself — and use
-[Generate sample data](/advanced/generate-sample-data) for a sandbox to run this against.
+ideas in plain language, model, feature, training, and the algorithm itself.
:::
## 1. Engineer one feature row per asset per day
For each pump on each day, summarise the recent week of its sensors into a feature row.
-Gaps stay as `NaN` — XGBoost handles missing values without imputation.
+Gaps stay as `NaN`, XGBoost handles missing values without imputation.
@@ -131,7 +130,7 @@ X = pd.DataFrame([r for r, _ in rows])
y = np.array([lab for _, lab in rows])
```
-## 3. Train — with class imbalance and early stopping
+## 3. Train, with class imbalance and early stopping
Failures are rare, so weight the positive class and optimise for ranking (`aucpr`).
Early stopping on a validation split avoids over-fitting.
@@ -156,7 +155,7 @@ print(f"avg precision: {average_precision_score(yte, clf.predict_proba(Xte)[:, 1
## 4. Score live, with the reasons attached
Score today's row per pump, write the risk back as a series, and raise a
-`failure_predicted` event for high-risk pumps — including the **feature importances** so
+`failure_predicted` event for high-risk pumps, including the **feature importances** so
the planner sees *why* (rising vibration vs. falling intake pressure point to different
fixes).
@@ -179,12 +178,17 @@ if risk > 0.6:
event_time=pd.Timestamp.now(tz="UTC"),
metadata={"pump": "pump_esp_a12", "risk": f"{risk:.2f}",
"top_drivers": ", ".join(import_)})])
+
+# read the risk back
+stored = client.timeseries.retrieve_latest_datapoints(["pump_esp_a12_failure_risk"])[0].get_datapoints()
+print(f"7-day failure risk {stored[-1].value:.2f}")
+assert abs(stored[-1].value - risk) < 0.01, "the risk was not written"
```
## Where to take it further
- **Real explainability.** Use SHAP for per-prediction attributions, not just global
- importance — the planner sees what drove *this* pump's score.
+ importance, the planner sees what drove *this* pump's score.
- **Monotonic constraints.** Tell XGBoost that higher vibration only ever raises risk
(`monotone_constraints`) for a model that matches engineering intuition.
- **Compare.** Benchmark against [Random Forest](/advanced/random-forest-soft-sensor) and
@@ -192,12 +196,12 @@ if risk > 0.6:
## Further reading
-- **XGBoost** — [Wikipedia](https://en.wikipedia.org/wiki/XGBoost) · [docs](https://xgboost.readthedocs.io/)
-- **Gradient boosting** — [Wikipedia](https://en.wikipedia.org/wiki/Gradient_boosting)
-- **Feature importance** — [scikit-learn guide](https://scikit-learn.org/stable/modules/permutation_importance.html)
+- **XGBoost**: [Wikipedia](https://en.wikipedia.org/wiki/XGBoost) · [docs](https://xgboost.readthedocs.io/)
+- **Gradient boosting**: [Wikipedia](https://en.wikipedia.org/wiki/Gradient_boosting)
+- **Feature importance**: [scikit-learn guide](https://scikit-learn.org/stable/modules/permutation_importance.html)
## See also
-- [Oil & gas — production monitoring](/industries/oil-and-gas/production) — the ESPs this protects.
-- [Predictive maintenance](/advanced/predictive-maintenance) — the unsupervised counterpart.
-- [Turn readings into events](/guides/detect-events) — emitting the prediction.
+- [Oil & gas, production monitoring](/industries/oil-and-gas/production): the ESPs this protects.
+- [Predictive maintenance](/advanced/predictive-maintenance): the unsupervised counterpart.
+- [Turn readings into events](/guides/detect-events): emitting the prediction.
diff --git a/docs/guides/attach-files.mdx b/docs/guides/attach-files.mdx
index 03e4cb8..d6695cf 100644
--- a/docs/guides/attach-files.mdx
+++ b/docs/guides/attach-files.mdx
@@ -8,13 +8,15 @@ import TabItem from '@theme/TabItem';
# Attach files to assets
Not everything is a number. A pump has a calibration certificate, a plant has a
-piping diagram, a daily run produces a CSV export. Store these as **files** —
-organized in folders, tagged with metadata, and linked to the dataset they belong to.
+piping diagram, a daily run produces a CSV export. Store these as **files**,
+organized in folders, tagged with metadata, and linked to the data set they belong to.
## Upload a file
-Give the file a destination path and some metadata. The Java client uploads raw
-bytes; the Python and Rust clients upload a local file to a destination folder.
+Give the file a destination path and some metadata. The server stores one full path, name
+included; the Java client sends the `path` you give it as-is, and the Python and Rust
+clients join `destination_path` and `name` into the same thing. The Java client uploads raw
+bytes; the Python and Rust clients read a local file.
@@ -27,7 +29,7 @@ byte[] content = Files.readAllBytes(Path.of("calibration_a12.pdf"));
var uploaded = client.files().upload(
FileUploadRequest.builder()
- .path("certificates/2026/calibration_a12.pdf") // destination
+ .path("/certificates/2026/calibration_a12.pdf") // destination, name included
.content(content)
.contentType("application/pdf")
.externalId("calibration_pump_esp_a12")
@@ -72,10 +74,11 @@ let uploaded = api.files.upload_file(upload).await?;
-:::tip Link a file to its asset
-Set `dataSetId` (or the `external_id`/metadata) so a file is discoverable from the
-dataset and asset it documents — the calibration certificate next to the pump it
-certifies.
+:::tip Link a file to what it documents
+Set `dataSetId` so a file is discoverable from the data set it documents. To point it at the
+resource it documents, the calibration certificate at the pump it certifies, the Python and
+Rust clients take `related_resources`, a list of numeric resource ids; the Java client has no
+field for it.
:::
## List a directory
@@ -90,15 +93,24 @@ path and size.
var listing = client.files().list("/certificates/2026");
listing.getItems().forEach(node ->
System.out.println(node.getType() + " " + node.getName() + " " + node.getSize() + " bytes"));
+
+boolean present = listing.getItems().stream()
+ .anyMatch(node -> "calibration_a12.pdf".equals(node.getName()));
+if (!present) {
+ throw new AssertionError("upload not found in /certificates/2026");
+}
```
```python
-for node in client.files.list_directory_by_path("/certificates/2026"):
+listing = client.files.list_directory_by_path("/certificates/2026")
+for node in listing:
print(node.type, node.name, node.size)
+assert "calibration_a12.pdf" in [node.name for node in listing], "upload not found"
+
roots = client.files.list_root_directory() # top of the tree
```
@@ -110,6 +122,9 @@ let listing = api.files.list_directory_by_path("/certificates/2026").await?;
for node in listing.get_items() {
println!("{} {} {} bytes", node.r#type.as_deref().unwrap_or(""), node.name, node.size);
}
+
+assert!(listing.get_items().iter().any(|node| node.name == "calibration_a12.pdf"),
+ "upload not found in /certificates/2026");
```
@@ -117,13 +132,40 @@ for node in listing.get_items() {
## Download
-The Java client downloads a file's raw bytes by id:
+Download by the id the upload echoed. Python and Rust take it as an integer, Java as a
+string (the endpoint also accepts an external id in that position).
+
+
+
```java
-byte[] bytes = client.files().download("99");
+long id = uploaded.getItems().iterator().next().getId();
+byte[] bytes = client.files().download(String.valueOf(id));
Files.write(Path.of("calibration_a12.pdf"), bytes);
```
+
+
+
+```python
+from pathlib import Path
+
+download = client.files.download(uploaded[0].id)
+Path("calibration_a12.pdf").write_bytes(download.content)
+```
+
+
+
+
+```rust
+let id = uploaded.get_items()[0].id.expect("the upload echo carries the id");
+let download = api.files.download(id).await?;
+std::fs::write("calibration_a12.pdf", &download.bytes)?;
+```
+
+
+
+
## Delete
diff --git a/docs/guides/correlate-alarms.mdx b/docs/guides/correlate-alarms.mdx
index 03dced6..8fe1fde 100644
--- a/docs/guides/correlate-alarms.mdx
+++ b/docs/guides/correlate-alarms.mdx
@@ -8,13 +8,12 @@ import TabItem from '@theme/TabItem';
# Correlate alarms with the graph
Two alarms fire within seconds of each other. Are they one incident or two? A flat
-lookup can't tell you — but the **relationship graph** can. If both alarmed sensors
-sit under the same subsystem, it's almost certainly one root cause, not two
-coincidences.
+lookup can't tell you, but the **relationship graph** can. If both alarmed sensors
+sit under the same subsystem, the two alarms share a subsystem.
`fetchRelated` walks the graph outward from a node and returns the connected
sub-graph. Walk the neighbourhood of each alarm and intersect the two: a shared
-ancestor — say a `cooling_system` both sensors are `PART_OF` — is the common cause.
+ancestor, say a `cooling_system` both sensors are `PART_OF`, is the common cause.
:::info Needs a sandbox
This guide reads data it does not create. [Seed a sandbox](/guides/seed-a-sandbox) writes it
@@ -29,9 +28,10 @@ chain; a single-hop read stops at the first neighbour.
## 1. From alarms to resources
-An alarm is an [event](./detect-events.mdx) that references the resource it concerns
-(`relatedResources`). Start from the two resources the alarms point at —
-here `sensor_a` and `sensor_b`.
+An alarm is an [event](./detect-events.mdx) that names the resource it concerns. The two
+alarms the sandbox seeds carry the sensor in `metadata.resource`, so the ids are given here:
+`sensor_a` and `sensor_b`. In production an alarm carries the resource in `relatedResources`,
+and a read of the event gives you the id.
## 2. Walk each alarm's neighbourhood
@@ -88,7 +88,7 @@ let nb = api.resources.fetch_related(
## 3. Find the shared subsystem
Intersect the two node sets. Whatever remains (besides the sensors themselves) is a
-subsystem both alarms belong to — the likely common cause.
+subsystem both alarms belong to, the likely common cause.
@@ -104,8 +104,9 @@ Set shared = nb.nodes().stream()
shared.remove("sensor_a");
shared.remove("sensor_b");
-if (!shared.isEmpty()) {
- System.out.println("Both alarms are part of: " + shared); // e.g. [cooling_system]
+System.out.println("Both alarms are part of: " + shared); // [cooling_system, skid_1]
+if (!shared.contains("cooling_system")) {
+ throw new AssertionError("expected cooling_system, found " + shared);
}
```
@@ -117,8 +118,8 @@ a_nodes = {n.external_id for n in na.nodes}
shared = {n.external_id for n in nb.nodes} & a_nodes
shared -= {"sensor_a", "sensor_b"}
-if shared:
- print("Both alarms are part of:", shared) # e.g. {'cooling_system'}
+print("Both alarms are part of:", shared) # {'cooling_system', 'skid_1'}
+assert "cooling_system" in shared, f"expected cooling_system, found {shared}"
```
@@ -135,19 +136,18 @@ let mut shared: HashSet<&str> = nb.nodes().iter()
shared.remove("sensor_a");
shared.remove("sensor_b");
-if !shared.is_empty() {
- println!("Both alarms are part of: {:?}", shared); // e.g. {"cooling_system"}
-}
+println!("Both alarms are part of: {:?}", shared); // {"cooling_system", "skid_1"}
+assert!(shared.contains("cooling_system"), "expected cooling_system, found {:?}", shared);
```
:::tip Deeper networks, smarter correlation
-Raise `depth` to correlate across a deeper hierarchy — a whole plant rather than one
+Raise `depth` to correlate across a deeper hierarchy, a whole plant rather than one
skid. Drop the relationship-type filter to follow *any* connection (power feeds, data
flows), not just containment, when a fault can propagate sideways.
:::
-The same pattern answers "what else might this failure affect?" — walk out from a
+The same pattern answers "what else might this failure affect?", walk out from a
failing component and the returned `nodes` are its blast radius.
diff --git a/docs/guides/detect-events.mdx b/docs/guides/detect-events.mdx
index 0a8dd28..9ee29c9 100644
--- a/docs/guides/detect-events.mdx
+++ b/docs/guides/detect-events.mdx
@@ -9,7 +9,7 @@ import TabItem from '@theme/TabItem';
Raw datapoints answer "what is the value?"; **events** answer "what happened?". A
common pipeline reads recent readings, checks them against a rule, and records a
-discrete, queryable event when the rule fires — a threshold breach, a state change,
+discrete, queryable event when the rule fires, a threshold breach, a state change,
an alarm. Events carry a type, a time, and metadata, and can reference the resources
they concern.
@@ -109,7 +109,7 @@ if too_hot {
## Query the events later
-Events are first-class records — filter them by type, time or metadata for an audit
+Events are first-class records, filter them by type, time or metadata for an audit
trail or an incident timeline.
diff --git a/docs/guides/ingest-timeseries.mdx b/docs/guides/ingest-timeseries.mdx
index 385efde..0a18521 100644
--- a/docs/guides/ingest-timeseries.mdx
+++ b/docs/guides/ingest-timeseries.mdx
@@ -8,18 +8,20 @@ import TabItem from '@theme/TabItem';
# High-throughput ingestion
A common job: push a steady, high-volume stream of sensor readings in so a dashboard
-or alerting rule can act on them. The SDK does the heavy lifting — it **chunks the
-data into batches and sends them concurrently**, retrying transient failures.
-
-:::note How it works
-Datapoints are split into batches (default **10,000** per request — the store is
-optimised for large batches), sent concurrently up to a bounded in-flight limit, and
-transient failures (HTTP 429/5xx, network) are retried. The Java client returns an
-`IngestResult` summarising what landed and what didn't.
+or alerting rule can act on them. The SDK **chunks the data into batches and sends them
+concurrently**, retrying transient failures.
+
+:::info Needs a sandbox
+This guide writes into `engine_temperature`, which section **A** of
+[Seed a sandbox](/guides/seed-a-sandbox) creates.
:::
-:::caution Stay inside the ingest caps
-The defaults already fit, so this only matters if you have tuned something:
+:::note How it works, and the caps it fits
+Datapoints are split into batches of **10,000** per request (the store is optimised for
+large batches), sent concurrently up to a bounded in-flight limit, and transient failures
+(HTTP 429/5xx, network) are retried. The Java client returns an `IngestResult` summarising
+what landed and what didn't. The defaults already fit the caps, so the table only matters if
+you have tuned something:
| Cap | Value |
| --- | --- |
@@ -41,9 +43,19 @@ and both retried for you. See [Limits & quotas](/reference/limits).
`ingest` groups datapoints by series external id and returns an `IngestResult`.
```java
+import java.time.Instant;
+import java.util.*;
+
var client = DatahubClient.fromEnv();
-// readings: Map> grouped by time-series external id
+// a million readings, one per second from the start of 2026
+Instant start = Instant.parse("2026-01-01T00:00:00Z");
+List points = new ArrayList<>(1_000_000);
+for (int i = 0; i < 1_000_000; i++) {
+ points.add(Datapoint.of(start.plusSeconds(i), 90 + 5 * Math.random()));
+}
+Map> readings = Map.of("engine_temperature", points);
+
IngestResult result = client.timeseries().ingest(readings,
IngestOptions.builder()
.batchSize(10_000) // datapoints per request
@@ -52,6 +64,10 @@ IngestResult result = client.timeseries().ingest(readings,
.build());
System.out.printf("ingested %,d, failed %,d%n", result.succeeded(), result.failed());
+if (!result.isComplete()) {
+ result.errors().forEach(e -> System.err.println(e.statusCode() + ": " + e.message()));
+ throw new IllegalStateException("ingest incomplete");
+}
```
@@ -60,34 +76,67 @@ System.out.printf("ingested %,d, failed %,d%n", result.succeeded(), result.faile
`insert_from_lists` takes whole arrays (NumPy / pandas) and batches them for you.
```python
-import numpy as np, pandas as pd
+import intellistream_datahub_sdk, numpy as np, pandas as pd
-client = DataHubClient.from_env()
+client = intellistream_datahub_sdk.DataHubClient.from_env()
+# a million readings, one per second from the start of 2026
client.timeseries.insert_from_lists(
timestamps=pd.date_range("2026-01-01", periods=1_000_000, freq="s", tz="UTC"),
- values=np.random.rand(1_000_000),
+ values=90 + 5 * np.random.rand(1_000_000),
ts="engine_temperature")
+
+# read the first hour back: 3 600 readings
+rf = intellistream_datahub_sdk.RetrieveFilter(
+ ts="engine_temperature",
+ start=pd.Timestamp("2026-01-01 00:00", tz="UTC"),
+ end=pd.Timestamp("2026-01-01 01:00", tz="UTC"),
+ limit=10_000)
+points = client.timeseries.retrieve_datapoints(rf)[0].get_datapoints()
+print(f"{len(points)} readings in the first hour")
+assert len(points) >= 3600, len(points)
```
-`insert_datapoints` auto-batches large inputs (chunks above ~100k points).
+`insert_datapoints` auto-batches large inputs (chunks at the 100 000-point collection cap).
```rust
+use chrono::{Duration, TimeZone, Utc};
use intellistream_datahub_sdk::create_api_service;
-use intellistream_datahub_sdk::generic::{DataWrapper, DatapointsCollection};
+use intellistream_datahub_sdk::generic::{DataWrapper, DatapointString, DatapointsCollection, RetrieveFilter};
let api = create_api_service();
+// a million readings, one per second from the start of 2026
+let start = Utc.with_ymd_and_hms(2026, 1, 1, 0, 0, 0).unwrap();
+let readings: Vec = (0..1_000_000)
+ .map(|i| DatapointString::from_datetime(
+ start + Duration::seconds(i),
+ &format!("{:.2}", 90.0 + (i % 50) as f64 / 10.0)))
+ .collect();
+
let mut dw = DataWrapper::new();
dw.add_item(DatapointsCollection {
external_id: Some("engine_temperature".into()),
- datapoints: readings, // Vec
+ datapoints: readings,
..Default::default()
});
api.time_series.insert_datapoints(&mut dw).await?;
+
+// read the first hour back: 3 600 readings
+let first_hour = RetrieveFilter {
+ external_id: Some("engine_temperature".into()),
+ start: Some(start),
+ end: Some(start + Duration::hours(1)),
+ limit: Some(10_000),
+ ..Default::default()
+};
+let points = api.time_series.retrieve_datapoints(&DataWrapper::from(vec![first_hour])).await?;
+let n = points.get_items()[0].datapoints.len();
+println!("{n} readings in the first hour");
+assert!(n >= 3600, "expected 3600 readings, got {n}");
```
@@ -105,14 +154,14 @@ The `ingest` knobs let you trade throughput against load on the server:
| `failFast` | `false` | Abort on the first failed batch instead of collecting errors. |
When `failFast` is off, inspect `result.errors()` for the per-batch failures. See the
-[Time-series reference](/reference/timeseries#ingestresult) for the full result shape.
+[Time series reference](/reference/timeseries#ingestresult) for the full result shape.
## Then chart the trend
-Once the data is in, roll it up to hourly or daily buckets for a dashboard — see
-[Query & aggregate time-series](./query-and-aggregate.mdx).
+Once the data is in, roll it up to hourly or daily buckets for a dashboard, see
+[Query & aggregate time series](./query-and-aggregate.mdx).
:::warning Ordering
-Batches are sent in parallel, so there is **no cross-batch ordering guarantee** —
+Batches are sent in parallel, so there is **no cross-batch ordering guarantee**,
which is fine for time-stamped data, since each datapoint carries its own timestamp.
:::
diff --git a/docs/guides/model-assets-graph.mdx b/docs/guides/model-assets-graph.mdx
index b383f56..e5fb889 100644
--- a/docs/guides/model-assets-graph.mdx
+++ b/docs/guides/model-assets-graph.mdx
@@ -7,7 +7,7 @@ import TabItem from '@theme/TabItem';
# Model assets as a graph
-Most domains are a hierarchy of things that contain or feed each other — a plant
+Most domains are a hierarchy of things that contain or feed each other, a plant
contains lines, a line contains machines, a machine emits sensor readings. Model
that as **resources** (the nodes) and **relations** (the edges), and the SDK returns
the persisted graph in one call.
@@ -112,57 +112,114 @@ let graph = api.resources.create(
-## Attach time-series to a node
+## Attach a time series to a node
-A machine's sensors are time-series. Tie a series to the machine that produces it
-with a relation, exactly as above (`press_07` → `produces` → `press_07_oil_temp`),
-then [ingest its datapoints](./ingest-timeseries.mdx).
+A machine's sensors are time series. Create the series, tie it to the machine that produces
+it with a relationship (`press_07` → `PRODUCES` → `press_07_oil_temp`), then
+[ingest its datapoints](./ingest-timeseries.mdx).
+
+
+
+
+```java
+Timeseries oilTemp = new Timeseries()
+ .setExternalId("press_07_oil_temp")
+ .setName("Press 07 oil temperature");
+oilTemp.setUnit("celsius");
+oilTemp.setUnitExternalId("temperature_deg_c");
+client.timeseries().create(List.of(oilTemp));
+
+RelForm produces = new RelForm();
+produces.setRelationshipType("PRODUCES");
+produces.setFromExternalId("press_07");
+produces.setToExternalId("press_07_oil_temp");
+client.edges().create(List.of(produces));
+```
+
+
+
+
+```python
+client.timeseries.create([intellistream_datahub_sdk.TimeSeries(
+ external_id="press_07_oil_temp", name="Press 07 oil temperature",
+ unit="celsius", unit_external_id="temperature_deg_c")])
+
+client.edges.create([
+ intellistream_datahub_sdk.RelForm.by_external_ids("press_07", "press_07_oil_temp", "PRODUCES")])
+```
+
+
+
+
+```rust
+use intellistream_datahub_sdk::relations::RelForm;
+use intellistream_datahub_sdk::timeseries::TimeSeries;
+
+let mut oil_temp = TimeSeries::new("press_07_oil_temp", "Press 07 oil temperature");
+oil_temp.unit = Some("celsius".into());
+oil_temp.unit_external_id = Some("temperature_deg_c".into());
+api.time_series.create_one(&oil_temp).await?;
+
+api.edges.create(&vec![
+ RelForm::by_external_ids("press_07", "press_07_oil_temp", "PRODUCES"),
+]).await?;
+```
+
+
+
## Read the graph back
-Fetch nodes by external id; each resource carries its outgoing edges.
+Fetch a node's neighbourhood with `fetchRelated`; the returned network carries the nodes it
+found and the edges between them. (`byIds` returns the node alone: `relatedResources` is
+empty on that read.) One hop out from the press finds the two edges created above.
```java
-var some = client.resources().byIds(List.of(
- IdCollection.createFromExternalId("press_07")));
+ResourceNetwork net = client.resources().fetchRelated("press_07", 1);
+net.edges().forEach(edge ->
+ System.out.println(edge.getStart() + " -" + edge.getType() + "-> " + edge.getEnd()));
-Resource press = some.getItems().iterator().next();
-press.getRelations().forEach(edge ->
- System.out.println(edge.getType() + " -> " + edge.getEnd()));
+if (net.edges().size() != 2) {
+ throw new AssertionError("expected CONTAINS and PRODUCES, found " + net.edges().size());
+}
```
```python
-press = client.resources.by_ids(["press_07"])[0]
-for edge in press.relations:
- print(edge.relationship_type, "->", edge.end)
+net = client.resources.fetch_related(external_id="press_07", depth=1)
+for edge in net.edges:
+ print(edge.start, edge.relationship_type, edge.end)
+
+types = {edge.relationship_type for edge in net.edges}
+assert types == {"CONTAINS", "PRODUCES"}, types
```
```rust
-use intellistream_datahub_sdk::generic::IdAndExtId;
+use intellistream_datahub_sdk::resources::RelatedResourcesForm;
-let some = api.resources.by_ids(&vec![IdAndExtId::from_external_id("press_07")]).await?;
-if let Some(press) = some.nodes().first() {
- for edge in press.relations.iter().flatten() {
- println!("{} -> {}", edge.relationship_type, edge.end);
- }
+let net = api.resources.fetch_related(
+ &RelatedResourcesForm::from_external_id("press_07").with_depth(1)).await?;
+for edge in net.edges() {
+ println!("{:?} -{:?}-> {:?}", edge.start, edge.relationship_type, edge.end);
}
+
+assert_eq!(net.edges().len(), 2, "expected CONTAINS and PRODUCES");
```
-:::tip Group with datasets
-Relations model *structure*. To slice assets by ownership, environment or tenant —
-"everything in the Oslo plant" — also put them in a **dataset** (see the
-[Datasets reference](/reference/datasets)). A resource can sit in a graph and a
-dataset at once.
+:::tip Group with data sets
+Relationships model *structure*. To slice assets by ownership, environment or tenant,
+"everything in the Oslo plant", also put them in a **data set** (see the
+[Data sets reference](/reference/datasets)). A resource can sit in a graph and a
+data set at once.
:::
diff --git a/docs/guides/query-and-aggregate.mdx b/docs/guides/query-and-aggregate.mdx
index dea8cc0..78354d5 100644
--- a/docs/guides/query-and-aggregate.mdx
+++ b/docs/guides/query-and-aggregate.mdx
@@ -1,15 +1,15 @@
---
sidebar_position: 2
-title: Query & aggregate time-series
+title: Query & aggregate time series
---
import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';
-# Query & aggregate time-series
+# Query & aggregate time series
Read raw datapoints back for a window, then roll them up to fixed buckets
(hourly averages, daily maxima) for a chart or report. Aggregates and the bucket
-size are part of the retrieve filter — no separate endpoint.
+size are part of the retrieve filter, no separate endpoint.
:::info Needs a sandbox
This guide reads data it does not create. [Seed a sandbox](/guides/seed-a-sandbox) writes it
@@ -143,24 +143,28 @@ for series in result.get_items() {
-:::note Common aggregates
-`avg`, `min`, `max`, `sum`. Granularity is a number plus a unit —
-`30s`, `5m`, `1h`, `1d`. The Python and Rust datapoint types expose each requested
-aggregate as its own field — `min`/`max`/`sum` by name, and `avg` read back on
-`.average` (`dp.average`, `dp.max`, …).
+:::note The aggregates and granularities
+The aggregates are `avg`, `sum`, `min` and `max`, lower-case; a name outside that set is
+dropped, not rejected. Granularity is a number plus a unit: `s`, `m`, `h`, `d`, `w`, `mo`, `y`,
+or the words `sec`, `min`, `hour`, `day`, `week`, `month`, `year` and their plurals (`30s`,
+`5m`, `1h`, `1d`). Bare `m` is a minute; a month is `mo`. The Python and Rust datapoint types
+expose each requested aggregate as its own field: `min`/`max`/`sum` by name, and `avg` read
+back on `.average` (`dp.average`, `dp.max`, …).
+[Retrieve datapoints →](/reference/timeseries#retrieve-datapoints)
:::
:::warning Java aggregate reads
The Java client does **not** expose aggregate values yet: `getValue()` returns `null`
for aggregated datapoints (the server sends `avg`/`max`/… as named fields, but the Java
datapoint type carries only `timestamp`/`value`). For now use **Python or Rust** for
-aggregates — or in Java, fetch raw datapoints and aggregate them in your application.
+aggregates, or in Java, fetch raw datapoints and aggregate them in your application.
:::
## Paging large windows
-A wide window can exceed one response. When the result carries a cursor, pass it
-back to fetch the next page.
+A wide window can exceed one response. Each returned collection carries a `nextCursor` when
+there is more (`getNextCursor()` in Java, `next_cursor` in Python and Rust); pass it back as
+`cursor` to fetch the next page.
diff --git a/docs/guides/realtime-subscriptions.mdx b/docs/guides/realtime-subscriptions.mdx
index be5a22a..9ef0e5b 100644
--- a/docs/guides/realtime-subscriptions.mdx
+++ b/docs/guides/realtime-subscriptions.mdx
@@ -7,14 +7,16 @@ import TabItem from '@theme/TabItem';
# Consume live data
-Push-based monitoring: a **subscription** names a set of time-series, and a live
-connection delivers each new datapoint as it lands — no polling. A control-room
+Push-based monitoring: a **subscription** names a set of time series, and a live
+connection delivers each new datapoint as it lands, no polling. A control-room
dashboard, an alerting worker, or a downstream pipeline drives the loop and **acks**
-what it has handled. Anything unacked is redelivered, so a crash never loses data.
+what it has handled. Anything unacked is redelivered, so a crash is followed by redelivery.
:::info Needs a sandbox
This guide reads data it does not create. [Seed a sandbox](/guides/seed-a-sandbox) writes it
-in one go.
+in one go. Nothing arrives on the socket until something writes to the series, so once the
+listener below is running, re-run section **A** of the seed in a second shell and watch the
+messages arrive.
:::
## 1. Create a subscription
@@ -67,13 +69,13 @@ api.subscriptions.create(&sub).await?;
## 2. Listen and ack
Hand each message to a handler (or drive a loop yourself), and ack once you've durably
-handled it. Make the handler **idempotent** — a redelivery after a crash will replay
+handled it. Make the handler **idempotent**, a redelivery after a crash will replay
the last unacked messages.
-Register a handler with `stream` — a dedicated virtual thread delivers each message **as
+Register a handler with `stream`, a dedicated virtual thread delivers each message **as
it arrives** and acks it once your handler returns (or nacks it if the handler throws).
The returned handle stops delivery and closes the listener, so use try-with-resources.
@@ -86,7 +88,7 @@ try (var stream = client.subscriptions().listen(List.of("engine_room"))
}
```
-Prefer a loop, or want to ack on your own schedule? Drive `poll` yourself — it blocks up
+Prefer a loop, or want to ack on your own schedule? Drive `poll` yourself, it blocks up
to the timeout and returns `null` on a quiet interval (it is a blocking queue hand-off,
**not** network polling):
@@ -137,9 +139,56 @@ while let Some(result) = listener.next().await {
+## 3. Count what arrives
+
+Re-run section **A** of the seed in a second shell. Each batch it writes to
+`engine_temperature` or `engine_rpm` becomes a message here, so the count climbs.
+
+
+
+
+```java
+import java.util.concurrent.atomic.AtomicInteger;
+
+AtomicInteger seen = new AtomicInteger();
+try (var stream = client.subscriptions().listen(List.of("engine_room"))
+ .stream((SubscriptionMessage msg) ->
+ System.out.println(seen.incrementAndGet() + " messages so far"))) {
+ Thread.sleep(60_000); // long enough to re-run the seed in another shell
+}
+```
+
+
+
+
+```python
+seen = 0
+with client.subscriptions.listen(["engine_room"]) as listener:
+ for msg in listener:
+ listener.ack([msg.message_id])
+ seen += 1
+ print(f"{seen} messages so far")
+```
+
+
+
+
+```rust
+let mut seen = 0;
+let mut listener = api.subscriptions.listen(&["engine_room"]).await?;
+while let Some(Ok(msg)) = listener.next().await {
+ listener.ack(&[msg.message_id.as_str()]).await?;
+ seen += 1;
+ println!("{seen} messages so far");
+}
+```
+
+
+
+
## Change the interest set at runtime
-A long-lived listener doesn't need to reconnect to follow more (or fewer) series —
+A long-lived listener doesn't need to reconnect to follow more (or fewer) series,
`subscribe` / `unsubscribe` adjust it in place.
@@ -171,7 +220,7 @@ listener.unsubscribe(&["engine_rpm"]).await?;
:::tip nack to retry later
If a message can't be processed right now (a downstream system is down), `nack` it
-instead of acking — it will be redelivered rather than dropped. With `stream`, throwing
+instead of acking, it will be redelivered rather than dropped. With `stream`, throwing
from the handler nacks for you; to control acks explicitly use
`stream(handler, AckMode.MANUAL)` and call `ack`/`nack` yourself. With `poll`, just call
`nack` instead of `ack`.
@@ -179,7 +228,7 @@ from the handler nacks for you; to control acks explicitly use
## See also
-- [Sustained-condition alarms](/advanced/sustained-alarm-window) — a full worked example that keeps a sliding window over this stream, seeds its own data and then verifies what it found.
-- [Turn readings into events](/guides/detect-events) — recording what a rule decides.
-- [Seed a sandbox](/guides/seed-a-sandbox) — the series this guide subscribes to.
-- Stream processing, in the platform documentation — windows, event time against arrival time, and what has to be remembered.
+- [Sustained-condition alarms](/advanced/sustained-alarm-window): a full worked example that keeps a sliding window over this stream, seeds its own data and then verifies what it found.
+- [Turn readings into events](/guides/detect-events): recording what a rule decides.
+- [Seed a sandbox](/guides/seed-a-sandbox): the series this guide subscribes to.
+- Stream processing, in the platform documentation, windows, event time against arrival time, and what has to be remembered.
diff --git a/docs/guides/seed-a-sandbox.mdx b/docs/guides/seed-a-sandbox.mdx
index cb27c13..e8d04a5 100644
--- a/docs/guides/seed-a-sandbox.mdx
+++ b/docs/guides/seed-a-sandbox.mdx
@@ -12,13 +12,13 @@ against a sandbox rather than read.
:::info What needs seeding, and what does not
| Guide | Needs |
| --- | --- |
-| [High-throughput ingestion](/guides/ingest-timeseries) | Nothing. It writes its own data |
+| [High-throughput ingestion](/guides/ingest-timeseries) | **A** (the series must exist) |
| [Query & aggregate](/guides/query-and-aggregate) | **A** the engine series |
| [Model assets as a graph](/guides/model-assets-graph) | Nothing. It creates the graph it reads |
| [Consume live data](/guides/realtime-subscriptions) | **A** the engine series |
| [Turn readings into events](/guides/detect-events) | **A** the engine series, which include a hot stretch so the rule actually fires |
| [Attach files to assets](/guides/attach-files) | Nothing. It uploads what it then reads |
-| [Work with units](/guides/work-with-units) | Nothing. It creates the series it converts |
+| [Work with units](/guides/work-with-units) | Nothing. It creates the series it tags |
| [Correlate alarms](/guides/correlate-alarms) | **B** the cooling system, its two sensors and their alarms |
:::
@@ -32,14 +32,21 @@ import intellistream_datahub_sdk, numpy as np, pandas as pd
client = intellistream_datahub_sdk.DataHubClient.from_env()
def ingest(external_id, index, values, unit=None, name=None):
- client.timeseries.create([intellistream_datahub_sdk.TimeSeries(
- external_id=external_id, name=name or external_id,
- unit=unit or "value", value_type="float")])
+ try:
+ client.timeseries.create([intellistream_datahub_sdk.TimeSeries(
+ external_id=external_id, name=name or external_id,
+ unit=unit or "value", value_type="float")])
+ except intellistream_datahub_sdk.DataHubException as e:
+ if e.status_code != 409: # a second run finds the series already there
+ raise
client.timeseries.insert_from_lists(
timestamps=index, values=np.asarray(values), ts=external_id)
print(f"ingested {len(values):,} points → {external_id}")
```
+A second run of any block below answers `409` on the duplicate external id; the helper
+catches that and goes on to write the datapoints, which dedup on `(series, timestamp)`.
+
## A. An engine, with the last twenty minutes running hot
Three series for one engine: a day of readings at one a minute. The temperature ends in a
@@ -134,6 +141,6 @@ If both numbers are non-zero, every guide on this site has something to run agai
## See also
-- [Generate sample data](/advanced/generate-sample-data) — the equivalent for the advanced scenarios.
-- [Sustained-condition alarms](/advanced/sustained-alarm-window) — a full example that seeds, runs and then verifies itself.
-- [High-throughput ingestion](/guides/ingest-timeseries) — doing this at production volume.
+- [Generate sample data](/advanced/generate-sample-data): the equivalent for the advanced scenarios.
+- [Sustained-condition alarms](/advanced/sustained-alarm-window): a full example that seeds, runs and then verifies itself.
+- [High-throughput ingestion](/guides/ingest-timeseries): doing this at production volume.
diff --git a/docs/guides/work-with-units.mdx b/docs/guides/work-with-units.mdx
index 6a8ac07..d8cd983 100644
--- a/docs/guides/work-with-units.mdx
+++ b/docs/guides/work-with-units.mdx
@@ -7,9 +7,9 @@ import TabItem from '@theme/TabItem';
# Work with units of measure
-Units are shared reference data — `celsius`, `bar`, `m/s`, `usd`. Tagging a
-time-series with a unit makes its values self-describing: a chart can label its axis,
-and a consumer knows that `248.6` is bar, not psi. Browse the catalogue, then
+Units are shared reference data: `temperature_deg_c`, `pressure_bar`, `mass_flow_rate_kghr`.
+Tagging a time series with a unit makes its values self-describing: a chart can label its
+axis, and a consumer knows that `248.6` is bar, not psi. Browse the catalogue, then
reference a unit by its external id when you create a series.
## Browse the catalogue
@@ -44,8 +44,11 @@ for u in api.units.list().await?.get_items() {
## Tag a series with a unit
-The everyday use: pass a unit's external id when you create a time-series. The value
-type stays numeric; the unit just describes what the numbers mean.
+A series carries two unit fields. `unit` is required free text of at most 64 characters, the
+label a chart shows (`bar`, `°C`); it is never checked against the catalogue. `unitExternalId`
+is optional and is the catalogue key (`pressure_bar`); set it when a consumer should be able to
+resolve the unit's symbol, quantity and conversion. The value type stays numeric; the unit
+describes what the numbers mean.
@@ -53,6 +56,7 @@ type stays numeric; the unit just describes what the numbers mean.
```java
var series = Timeseries.of("wellhead_pressure_bar").name("Wellhead pressure");
series.setUnit("bar");
+series.setUnitExternalId("pressure_bar");
client.timeseries().create(series);
```
@@ -65,7 +69,8 @@ import intellistream_datahub_sdk
client.timeseries.create([intellistream_datahub_sdk.TimeSeries(
external_id="wellhead_pressure_bar",
name="Wellhead pressure",
- unit="bar")])
+ unit="bar",
+ unit_external_id="pressure_bar")])
```
@@ -76,6 +81,7 @@ use intellistream_datahub_sdk::timeseries::TimeSeries;
let mut ts = TimeSeries::new("wellhead_pressure_bar", "Wellhead pressure");
ts.unit = Some("bar".into());
+ts.unit_external_id = Some("pressure_bar".into());
api.time_series.create_one(&ts).await?;
```
@@ -84,7 +90,7 @@ api.time_series.create_one(&ts).await?;
## Look up a specific unit
-Resolve a unit you already know — by external id, or by numeric id.
+Resolve a unit you already know, by external id, or by numeric id.
@@ -120,11 +126,12 @@ let by_id = api.units.by_ids(&DataWrapper::from(vec![IdAndExtId::from_id(7)])).a
-:::note Unit external ids are snake_case
-The seeded unit catalogue uses snake_case throughout — `m_s` for metres per second, `deg_c`
-for degrees Celsius — so reference them that way and browse the catalogue to find the exact
-id. That is a property of the catalogue, not a rule about external ids in general: the ones
-*you* create are [stored exactly as you send them](/reference/external-ids).
+:::note Unit external ids are `_` in snake_case
+The seeded unit catalogue names each unit by its quantity and symbol: `temperature_deg_c` for
+degrees Celsius, `pressure_bar` for bar, `mass_flow_rate_kghr` for kilograms per hour. Browse
+the catalogue to find the exact id. That is a property of the catalogue, not a rule about
+external ids in general: the ones *you* create are
+[stored exactly as you send them](/reference/external-ids).
:::
See the [Units reference](/reference/units) for the full method list.
diff --git a/docs/intro.md b/docs/intro.md
index 79ba64d..ab2fbf5 100644
--- a/docs/intro.md
+++ b/docs/intro.md
@@ -6,7 +6,7 @@ title: DataHub SDK
# DataHub SDK
-A thin, fast client for the **DataHub Platform** — manage resources, time-series,
+A thin client for the **DataHub Platform**: manage resources, time series,
events, files, and stream live data from your own application.
The SDK is available for **Java**, **Python** and **Rust**. Pick your language once
@@ -20,15 +20,23 @@ datapoint, or jump to [high-throughput ingestion](/guides/ingest-timeseries) to
## What you can do
-- **Time-series** — create series, ingest datapoints in parallel, query raw values and aggregates.
-- **Resources** — model assets and their relationships as a graph.
-- **Events** — record and query operational events.
-- **Files** — attach documents and images to your assets.
-- **Subscriptions** — tail live data over a streaming connection.
+- **Time series**: create series, ingest datapoints in parallel, query raw values and aggregates.
+- **Resources**: model assets and their relationships as a graph.
+- **Events**: record and query operational events.
+- **Files**: attach documents and images to your assets.
+- **Subscriptions**: tail live data over a streaming connection.
+
+## Licence
+
+The three SDKs, and the Java wire-contract model the Java SDK depends on, are
+[Apache-2.0](https://www.apache.org/licenses/LICENSE-2.0). They were placed under that licence
+so that linking one into your application carries no copyleft obligation into your code. The
+platform they talk to is [AGPL-3.0](https://www.gnu.org/licenses/agpl-3.0.html); that licence
+covers the server, and using the server over its API is not what it restricts.
## Where to go next
-- **[Tutorial](/tutorial)** — build a small metrics agent end to end: create series, ingest on a schedule, and survive API outages.
-- **[Examples](/guides/ingest-timeseries)** — task recipes: ingestion, graph modeling, querying & aggregation, live consumption, and event detection.
-- **[Industry scenarios](/industries/oil-and-gas/production)** — end-to-end walkthroughs for oil & gas, energy grids, finance, and IT operations.
-- **[API reference](/reference/client)** — every service, method and option, in Java, Python and Rust.
+- **[Tutorial](/tutorial)**: build a small metrics agent end to end: create series, ingest on a schedule, and survive API outages.
+- **[Guides](/guides/ingest-timeseries)**: task recipes: ingestion, graph modeling, querying & aggregation, live consumption, and event detection.
+- **[Industry scenarios](/industries/overview)**: end-to-end walkthroughs across ten sectors, from oil & gas and energy to finance and healthcare.
+- **[API reference](/reference/client)**: every service, method and option, in Java, Python and Rust.
diff --git a/docs/mcp-server.mdx b/docs/mcp-server.mdx
index fb9fd3a..53bee61 100644
--- a/docs/mcp-server.mdx
+++ b/docs/mcp-server.mdx
@@ -11,7 +11,7 @@ against your data model directly instead of you hand-writing an integration laye
| Server | Endpoint | Exposes |
| --- | --- | --- |
-| API | `/mcp` (port 8081 locally) | **37 tools** across datasets, resources, relationships, timeseries, events, labels and units |
+| API | `/mcp` (port 8081 locally) | **37 tools** across data sets, resources, relationships, time series, events, labels and units |
| Analysis | the analysis service's `/mcp` (port 8082 locally) | **one tool**, [`analysis_related_series`](#the-analysis-server) |
The important part is what they *aren't*: each MCP endpoint is an ordinary Spring endpoint on
@@ -52,12 +52,11 @@ Three things are checked before any tool body runs:
`OrganizationValidator` used for REST, so an agent cannot reach another tenant's data
even by asking for an id it happens to know.
-Past those gates, per-dataset [access grants](/reference/datasets#access-control) apply
+Past those gates, per-data-set [access grants](/reference/datasets#access-control) apply
exactly as on REST: rows in data sets the identity cannot read are omitted, writes it lacks
a grant for are refused.
-The practical consequence, and the reason this design is worth the trouble: **an agent
-inherits the permissions of whoever it is acting for.** If you want an agent that can read
+The practical consequence: **an agent inherits the permissions of whoever it is acting for.** If you want an agent that can read
but not write, issue it a token that can read but not write. There is no second permission
model to keep in sync.
@@ -123,19 +122,19 @@ Calling a tool is the same request with a `tools/call` body:
Named `_` in snake_case, deliberately, so the listing reads the same to you
and to the model.
-### Datasets
+### Data sets
| Tool | What it does |
| --- | --- |
-| `dataset_list` | List datasets visible to the caller. The catalogue is usually small |
+| `dataset_list` | List data sets visible to the caller |
| `dataset_search` | Full-text search, returns ids needed by `timeseries_create` |
-| `dataset_create` | Create one dataset, returns the server-assigned id |
-| `dataset_update` | Update one dataset by id. Only provided fields change |
+| `dataset_create` | Create one data set, returns the server-assigned id |
+| `dataset_update` | Update one data set by id. Only provided fields change |
| `dataset_delete` | Delete by id or externalId. Does **not** cascade to members |
### Resources
-A resource is any node in the graph that is not a dataset or a timeseries: equipment, work
+A resource is any node in the graph that is not a data set or a time series: equipment, work
items, documents.
| Tool | What it does |
@@ -171,14 +170,14 @@ neighbour ("the time series measuring this pump").
| `edge_get` | Fetch a single edge by numeric id |
| `edge_delete` | Delete edges by id. Nodes on either side are untouched |
-### Timeseries
+### Time series
| Tool | What it does |
| --- | --- |
| `timeseries_list` | Browse when you do not yet have a search term |
| `timeseries_search` | Full-text search over name, externalId and description, ranked by relevance |
| `timeseries_get` | Look up by externalId or numeric id |
-| `timeseries_create` | Create one timeseries, returns the assigned id. Needs a `unit` or a `unitExternalId` (see `unit_list`) |
+| `timeseries_create` | Create one time series, returns the assigned id. Needs a `unit` or a `unitExternalId` (see `unit_list`) |
| `timeseries_update` | Update common fields by id |
| `timeseries_delete` | Delete the definition. Existing datapoints are not removed |
| `timeseries_get_latest` | Most recent datapoint, for "what is it reading now?" |
@@ -207,7 +206,7 @@ for when the agent only has words to go on.
| `label_create` | Create one label, coerced to `SCREAMING_SNAKE_CASE` |
| `label_update` | Update description, i18n code or colour by id |
| `unit_list` | Units of measure known to DataHub, e.g. Celsius |
-| `unit_get` | Look up one unit by externalId, e.g. `celsius` |
+| `unit_get` | Look up one unit by externalId, e.g. `temperature_deg_c` |
## The analysis server
@@ -314,18 +313,13 @@ and daily ingest quota as your own jobs, and hit the same field, batch and body-
agent that lists in a loop can rate-limit the pipeline it shares a tenant with. The refusal
comes back as a `429` carrying `Retry-After`, which is worth teaching the agent to honour.
-**Deletes do not cascade.** Deleting a dataset leaves its resources and timeseries in place;
-deleting a timeseries leaves its datapoints. This is deliberate, but it means an agent
+**Deletes do not cascade.** Deleting a data set leaves its resources and time series in place;
+deleting a time series leaves its datapoints. This is deliberate, but it means an agent
tidying up will need more than one call, and a half-finished tidy leaves orphans.
-**Name coercion is server-side.** Relationship types and labels are normalised to
-`SCREAMING_SNAKE_CASE`, so `derived from` becomes `DERIVED_FROM`. Agents that construct a
-name and then search for the string they constructed will miss.
-
-## Adding a tool
-
-Annotate a public method on a Spring bean in the API server's `mcp.tools` package with `@Tool`, reference
-`McpResultConverter` from it, and register the bean in the `ToolCallbackProvider`. If it is
-not registered there, the server does not advertise it. Spring AI builds the schema from the
-parameter types and the `@ToolParam` descriptions, so those descriptions are what the model
-reads: write them for the model, not for a colleague.
+**Name coercion is server-side, and not uniform.** `edge_create_type` and `label_create`
+normalise a name to `SCREAMING_SNAKE_CASE`, so `derived from` becomes `DERIVED_FROM`, and the
+labels given to `resource_create` are canonicalised the same way when stored. `edge_create`
+only upper-cases its `relationshipType`, so `derived from` becomes `DERIVED FROM`, a different
+type from `DERIVED_FROM`. Agents that construct a name and then search for the string they
+constructed will miss; have the agent write relationship types as `SCREAMING_SNAKE_CASE`.
diff --git a/docs/quickstart.mdx b/docs/quickstart.mdx
index 3a9b5c6..c22fefe 100644
--- a/docs/quickstart.mdx
+++ b/docs/quickstart.mdx
@@ -57,10 +57,9 @@ intellistream-datahub-sdk = "0.1"
:::tip The name is long on purpose, and you can shorten it
-`datahub-sdk` was already taken on PyPI by an unrelated project, and the bare `datahub`
-neighbourhood belongs to a much better-known metadata platform, so the package is qualified
-with the organisation and the Rust crate takes the same name. Alias it if you would rather
-not type it:
+`datahub-sdk` was already taken on PyPI by an unrelated project, and the `datahub` name is
+taken, so the package is qualified with the organization and the Rust crate takes the same
+name. Alias it if you would rather not type it:
```python
import intellistream_datahub_sdk as dh
@@ -173,16 +172,20 @@ for dp in client.timeseries.retrieve_datapoints(rf)[0].get_datapoints():
```rust
+use chrono::Utc;
use intellistream_datahub_sdk::generic::{DataWrapper, RetrieveFilter};
let filter = RetrieveFilter {
external_id: Some("engine_temperature".into()),
- limit: Some(100),
+ start: Some(Utc::now() - chrono::Duration::hours(1)),
+ end: Some(Utc::now()),
..Default::default()
};
let points = api.time_series.retrieve_datapoints(&DataWrapper::from(vec![filter])).await?;
for c in points.get_items() {
- println!("{} points", c.datapoints.len());
+ for p in &c.datapoints {
+ println!("{} = {}", p.timestamp, p.value);
+ }
}
```
@@ -190,7 +193,7 @@ for c in points.get_items() {
:::note Pick once, follow everywhere
-The tabs above all switch together — and your choice persists as you move through the
+The tabs above all switch together, and your choice persists as you move through the
docs. One selection for the whole site.
:::
diff --git a/docs/reference/client.md b/docs/reference/client.md
index 2c663a1..296c3a5 100644
--- a/docs/reference/client.md
+++ b/docs/reference/client.md
@@ -8,7 +8,7 @@ import TabItem from '@theme/TabItem';
# Client & configuration
The client is the entry point: it owns a shared HTTP connection and token handling and
-exposes one accessor per service. It is safe to share — **create one and reuse it** for
+exposes one accessor per service. It is safe to share: **create one and reuse it** for
the lifetime of your application.
@@ -42,7 +42,7 @@ client = DataHubClient.from_envfile("/path/to/.env")
client = DataHubClient(base_url="https://api.intellistream.ai", token="...")
```
-For `async`/`await`, use `AsyncDataHubClient` instead — same methods, awaited:
+For `async`/`await`, use `AsyncDataHubClient` instead, same methods, awaited:
```python
from intellistream_datahub_sdk import AsyncDataHubClient
@@ -63,7 +63,7 @@ Every method is `async`, so call them from an async runtime (e.g. `#[tokio::main
`.await` the result.
Don't want async? Enable the `blocking` cargo feature and use
-`intellistream_datahub_sdk::blocking` instead — the same services and methods without
+`intellistream_datahub_sdk::blocking` instead, the same services and methods without
`.await`, driven by the SDK's own runtime (the `reqwest` / `reqwest::blocking` split):
```rust
@@ -80,7 +80,7 @@ let api = blocking::create_api_service();
| Service | Java | Python | Rust |
| --- | --- | --- | --- |
| Resources | `client.resources()` | `client.resources` | `api.resources` |
-| Time-series | `client.timeseries()` | `client.timeseries` | `api.time_series` |
+| Time series | `client.timeseries()` | `client.timeseries` | `api.time_series` |
| Datasets | `client.datasets()` | `client.datasets` | `api.datasets` |
| Events | `client.events()` | `client.events` | `api.events` |
| Units | `client.units()` | `client.units` | `api.units` |
@@ -97,7 +97,6 @@ bearer token **or** OAuth2 client-credentials (the SDK fetches and refreshes the
| `BASE_URL` | API base URL (required) |
| `TOKEN` | Static bearer token |
| `CLIENT_ID` / `CLIENT_SECRET` / `TOKEN_URI` | OAuth2 client-credentials (all three) |
-| `PROJECT_NAME` | Optional project/tenant hint |
`fromEnv()` / `from_env()` / `create_api_service()` read these from the environment,
falling back to a `.env` file in the working directory (real environment variables win).
@@ -117,8 +116,6 @@ When the claim is present but names an organization this deployment holds no ten
(never onboarded, or since removed), every call fails **`403`** with an
`application/problem+json` body of `type: ".../errors/unknown-tenant"` naming the refused
`organizationId`. Retrying never helps: an administrator has to register the organization.
-This previously surfaced as a `500`, so retry logic that keys on 5xx should be told to give
-up on it.
| Variable | Java builder | Python kwarg | Rust setter | When you need it |
| --- | --- | --- | --- | --- |
@@ -135,8 +132,8 @@ up on it.
| The identity provider refused the token | Calls succeed, then start failing part-way through a run | Get a new token |
The second one catches long-running processes. The API checks your token locally (signature,
-expiry, issuer) and separately reads your dataset grants from the identity provider's UserInfo
-endpoint, so a token can pass the first check and still be refused by the second: it is unexpired,
+expiry, issuer) and separately reads your data set grants from the identity provider's UserInfo
+endpoint. A token can pass the first check and still be refused by the second: it is unexpired,
but the session behind it has ended, because an idle or maximum session lifetime elapsed or
somebody signed out. The response carries `WWW-Authenticate: Bearer error="invalid_token"` and a
problem+json body with `type: ".../errors/token-rejected"`.
@@ -168,7 +165,7 @@ Setting an assertion source switches the request at `TOKEN_URI` from client-cred
| Variable | Java builder | Python kwarg | Rust setter | Meaning |
| --- | --- | --- | --- | --- |
-| `ASSERTION` | `.assertion(...)` | `assertion=` | `set_assertion(...)` | A ready-made JWT. Never refreshed — prefer the credentials below. |
+| `ASSERTION` | `.assertion(...)` | `assertion=` | `set_assertion(...)` | A ready-made JWT. Never refreshed, prefer the credentials below. |
| `ASSERTION_CLIENT_ID` / `ASSERTION_CLIENT_SECRET` / `ASSERTION_TOKEN_URI` | `.assertionCredentials(...)` | `assertion_client_id=` / `assertion_client_secret=` / `assertion_token_url=` | `set_assertion_credentials(...)` | Fetch the assertion with client credentials from another provider (all three). |
| `ASSERTION_SCOPE` | `.assertionScope(...)` | `assertion_scope=` | `set_assertion_scope(...)` | `scope` for the assertion request. |
| `ASSERTION_AUDIENCE` | `.assertionAudience(...)` | `assertion_audience=` | `set_assertion_audience(...)` | `audience` for the assertion request. |
@@ -242,16 +239,17 @@ ASSERTION_SCOPE=api:///.default
```
The exchanged token is cached and refreshed exactly like a client-credentials one. The assertion
-itself is **never** cached — providers commonly reject a replayed assertion, so every exchange
+itself is **never** cached, providers commonly reject a replayed assertion, so every exchange
starts from a fresh request.
:::caution Server-side setup is required
The identity provider must be configured to trust the external issuer, and the external identity
must map to a real user on that side. For Keycloak that means an Identity Provider with **JWT
Authorization Grant** enabled (Keycloak 26.5+), a client with the matching capability, and a
-linked user carrying the roles and tenant claim. See `EntraID.md` in the platform repository for
-the full walkthrough, including the audience and assertion-lifetime settings that trip up a first
-attempt.
+linked user carrying the roles and tenant claim. See
+[`EntraID.md`](https://github.com/IntelliStream-DataHub/datahub-platform/blob/master/EntraID.md)
+in the platform repository for the full walkthrough, including the audience and
+assertion-lifetime settings that trip up a first attempt.
:::
### From HashiCorp Vault (Java)
@@ -269,14 +267,14 @@ DatahubConfig cfg = DatahubConfig.fromVaultAppRoleEnv("datahub/sdk"); /
## Durable ingest buffering
Optional and **off by default**. When enabled, datapoint and event ingestion that can't reach the
-API — or is rejected with an auth failure (HTTP 401/403, e.g. an expired or rotated token) — spools
-to disk and is flushed automatically on the next ingest call, so neither a transient outage nor a
+API, or is rejected with an auth failure (HTTP 401/403, e.g. an expired or rotated token), spools
+to disk and is flushed automatically on the next ingest call. Neither a transient outage nor a
credential hiccup loses data or raises. The buffer is a segmented, compressed log (gzip in Java,
zstd in Rust/Python) bounded on two axes, either of which may be left unset; an unset axis defaults
to **72 hours** / **5 GiB** once buffering is on:
-- **time** — datapoints/events older than the window are dropped.
-- **size** — when the on-disk spool exceeds the cap, the oldest segment is dropped.
+- **time**: datapoints/events older than the window are dropped.
+- **size**: when the on-disk spool exceeds the cap, the oldest segment is dropped.
It is memory-safe: the spool is drained in segments, so even a multi-gigabyte buffer never loads
into memory, and it is recovered from disk on the next start.
@@ -301,7 +299,7 @@ if (r.buffered() > 0) {
```
`fromEnv()` instead reads `BUFFER_RETENTION` (an ISO-8601 duration, e.g. `PT72H`),
-`BUFFER_MAX_BYTES` and `BUFFER_DIRECTORY` — setting either bound turns buffering on.
+`BUFFER_MAX_BYTES` and `BUFFER_DIRECTORY`. Setting either bound turns buffering on.
@@ -347,7 +345,7 @@ A [lifetime ceiling](./limits#lifetime-ceilings) answers `403` too, and that one
missing grant is fixed out of band and the data then flushes; a ceiling never becomes
acceptable by being replayed, so spooling it would fill the buffer with data the server
refuses every time. The client matches the problem `type`, so an ordinary permission `403`
-still buffers exactly as before.
+is buffered.
:::
:::note Retries are idempotent
@@ -364,7 +362,7 @@ response surfaces as an exception/error carrying the HTTP status and the raw bod
-Methods return `DataWrapper` — `getItems()` holds the results. Non-2xx throws
+Methods return `DataWrapper`: `getItems()` holds the results. Non-2xx throws
`DatahubApiException`:
```java
@@ -396,7 +394,7 @@ except DataHubException as e:
-Methods return `Result, ResponseError>` — `get_items()` holds the results,
+Methods return `Result, ResponseError>`: `get_items()` holds the results,
and `ResponseError` exposes `get_status()` and `get_message()` (its `Display` prints both):
```rust
@@ -438,12 +436,45 @@ the whole batch once you have fixed them.
Two responses are worth recognising by shape:
-- **`400` with `type: ".../errors/naming-policy"`** — one or more external ids broke the
+- **`400` with `type: ".../errors/naming-policy"`**: one or more external ids broke the
configured [naming policy](./external-ids#the-naming-policy). Nothing was created; the
`violations` array names each one and suggests a replacement.
-- **A `warnings` array beside `items` on a `2xx`** — the write succeeded, and the ids in it
- are in a data steward's queue. The field is absent when empty, so existing code is
- unaffected.
+- **A `warnings` array beside `items` on a `2xx`**: the write succeeded, and the ids in it
+ are in a data steward's queue. The field is absent when empty.
Both shapes, and the rules behind them, are in
[External ids & naming](./external-ids).
+
+### Unknown fields are refused {#unknown-fields}
+
+A request body naming a field the endpoint does not have is a `400`, not a silent success. A
+typo, or a field that has since been retired, would otherwise be dropped and answered `200`,
+telling you a change was applied when nothing happened. The body is an
+[RFC 9457](https://www.rfc-editor.org/rfc/rfc9457) problem document of
+`type: ".../errors/unreadable-request-body"`, with one `errors` entry per offender, each
+located by a JSON Pointer and listing the names accepted at that position:
+
+```json
+{
+ "type": "https://intellistream.ai/errors/unreadable-request-body",
+ "title": "Bad Request",
+ "status": 400,
+ "detail": "Unknown field: eventTime",
+ "errors": [
+ {
+ "detail": "Unknown field",
+ "pointer": "#/items/0/update/eventTime",
+ "allowedFields": ["dataSetId", "description", "externalId", "metadata",
+ "relatedResources", "source", "status", "subType", "type"]
+ }
+ ]
+}
+```
+
+Every offender in the body is reported at once, at whatever depth it sits, so several stale
+fields cost one round trip rather than one each. A body that cannot be parsed at all,
+malformed JSON or a value of the wrong shape, answers with the same `type` and a `detail`
+naming the problem, plus `line` and `column` where the parser can say.
+
+The clients only ever send fields they declare, so this reaches you when you build a body by
+hand, or keep an old field name in one.
diff --git a/docs/reference/datasets.md b/docs/reference/datasets.md
index 0bcd83e..c24911b 100644
--- a/docs/reference/datasets.md
+++ b/docs/reference/datasets.md
@@ -1,29 +1,29 @@
---
sidebar_position: 5
-title: Datasets
+title: Data sets
---
import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';
-# Datasets
+# Data sets
-Logical groupings of resources and time-series. Datasets can be nested (a dataset can
-belong to a parent dataset), and that hierarchy is live in queries: filtering time-series
-by a dataset also matches everything beneath it.
+Logical groupings of resources and time series. Data sets can be nested (a data set can
+belong to a parent data set), and that hierarchy is live in queries: filtering time series
+by a data set also matches everything beneath it.
[Filter series →](./timeseries#filter-series)
-The hierarchy is built from `BELONGS_TO` edges, and the server enforces that: a relation
-pointing at a dataset must be `BELONGS_TO`, and a dataset can only claim a time-series
-that isn't already in another dataset.
-[Edge rules →](./resources#create-resources-and-relations)
+The hierarchy is built from `BELONGS_TO` relationships, and the server enforces that: a
+relationship pointing at a data set must be `BELONGS_TO`, and a data set can only claim a
+time series that isn't already in another data set.
+[Relationship rules →](./resources#create-resources-and-relations)
:::note External ids are stored exactly as you send them
-The server does not rewrite a dataset external id: `Plant-A` stays `Plant-A`. Some clients
-*derive* one from the name as a convenience (the Rust `Dataset::new` below), and that
-derivation is snake_case — but it is a client-side default, not a server rule.
+The server does not rewrite a data set external id: `Plant-A` stays `Plant-A`. The Rust
+`Dataset::new` derives one from the name, in snake_case; that is a client-side default,
+not a server rule.
Uniqueness and lookup both ignore case, so `plant_a` collides with `PLANT_A` and either
-spelling finds the same dataset. Datasets are also subject to the
+spelling finds the same data set. Data sets are also subject to the
[naming policy](./external-ids#the-naming-policy) if an administrator has set one.
[External ids & naming →](./external-ids)
:::
@@ -101,7 +101,7 @@ api.datasets.delete(&vec![IdAndExtId::from_external_id("plant_a")]).await?;
## Filter {#filter}
`POST /datasets/filter` finds data sets by structured criteria, combined with AND. It is exactly
-the criteria every node type shares — a data set has no `dataSetId` of its own, being the thing
+the criteria every node type shares, a data set has no `dataSetId` of its own, being the thing
other nodes are scoped *by*:
| Criterion | Matching |
@@ -112,7 +112,7 @@ other nodes are scoped *by*:
| `createdTime`, `lastUpdatedTime` | `{ "min": …, "max": … }` bounds. |
Each field except `labels` and `metadata` takes **either a bare value or an array**, whose
-entries are combined with OR — which is why they are named in the singular. `limit` defaults to
+entries are combined with OR, which is why they are named in the singular. `limit` defaults to
1000 and is capped at 10000, and the page can be ordered and walked exactly as
[timeseries](./timeseries#sorting-and-paging) can.
@@ -156,9 +156,9 @@ let matches = api.datasets.filter(&DatasetFilter::from_filter(criteria)).await?;
-:::note There is no `writeProtected` or `deactivated`
-Both were removed server-side as inert. The api drops unknown keys silently, so a filter still
-carrying one looked like it was narrowing and was not.
+:::note There is no `writeProtected` or `deactivated` criterion
+A filter carrying either, or any other unknown field, is refused with a `400` as an
+[unknown field](./client#unknown-fields).
:::
## Search {#search}
@@ -185,12 +185,6 @@ ever removes matches: the phrase decides what the candidates are.
No match is an empty list, not a `404`.
-:::note What changed
-`filter` used to be accepted and silently ignored here, as it was on the resource and event
-searches. All four searches now apply it, as one query rather than a phrase pass followed by a
-narrowing pass.
-:::
-
## Access control {#access-control}
Access to a data set is administered in Keycloak (or the directory behind it), not in
@@ -213,19 +207,22 @@ Two consequences worth knowing when you code against this:
- **A missing grant is a `403`** with an `application/problem+json` body naming the
`dataSetId` and the `permission` (read or write) you lack. List, filter and search
endpoints never 403 on grants: rows in data sets you cannot read are silently
- omitted instead. Edge reads hide rather than refuse too: reading an edge needs read
- on both endpoints' data sets, and one you may not read is a `404` from
- `GET /edges/{id}` and omitted from `/edges/byids`, as if it did not exist.
+ omitted instead.
+- **Relationship reads hide rather than refuse too.** Reading a relationship needs read
+ on both endpoints' data sets. One you may not read is a `404` from `GET /edges/{id}`
+ and omitted from `/edges/byids`, as if it did not exist.
- **Managing a data set itself is stricter.** Creating, updating or deleting a data set
- (as opposed to the data in it) requires the `/datasets/*/write` grant or `DATAHUB_ADMIN`;
- grants on individual data sets are never enough, deliberately: a data set is the unit
+ (as opposed to the data in it) requires the `/datasets/*/write` grant or `DATAHUB_ADMIN`.
+ Grants on individual data sets are never enough, deliberately: a data set is the unit
access is granted on, so renaming or re-parenting one changes what existing grants
- cover. The rule follows the node, not the endpoint: a `DATASET`- or `POLICY`-labelled
+ cover.
+- **The rule follows the node, not the endpoint.** A `DATASET`- or `POLICY`-labelled
node reached through `/resources` answers the same way. The `403` detail spells this
out.
-The API reads grants from the identity provider's UserInfo endpoint, not from the token,
-so a changed grant takes effect within about a minute, without a new token.
+The API reads grants from the identity provider's UserInfo endpoint, not from the token, and
+caches them: the grant cache refreshes after 45 seconds, behind a 10-second in-process cache.
+A changed grant therefore takes effect within about a minute, without a new token.
## What each client covers {#client-coverage}
@@ -238,7 +235,7 @@ so a changed grant takes effect within about a minute, without a new token.
| Search | `datasets().search` | `datasets.search` | `datasets.search` |
| Update | `datasets().update` | `datasets.update` | `datasets.update` |
| Delete | `datasets().delete` | `datasets.delete` | `datasets.delete` |
-| Access policies | HTTP | `datasets.policies` | `datasets.policies` |
+| Policies (`GET /datasets/policies`) | HTTP | `datasets.policies` | `datasets.policies` |
-All three clients now cover the whole surface bar the access-policy read in Java, which still
-goes through the endpoint directly.
+Java has no policies call; use `GET /datasets/policies` directly. It returns every policy in
+the tenant that a data set can be associated with.
diff --git a/docs/reference/edges.md b/docs/reference/edges.md
index 48e27a0..6f27c8f 100644
--- a/docs/reference/edges.md
+++ b/docs/reference/edges.md
@@ -8,13 +8,13 @@ import TabItem from '@theme/TabItem';
# Edges
The relationships between resources, as objects in their own right. An edge is **directional**
-(`from` → `to`), **typed** by a relationship type, and **unique** per pair and type — two
+(`from` → `to`), **typed** by a relationship type, and **unique** per pair and type, two
resources can be connected many ways, but only once each way.
Most edges are born with their nodes: `POST /resources/create` takes `nodes` and `relations`
together and writes them in one transaction. The `/edges` endpoints are for everything after
-that — linking resources that already exist, reading an edge back, cutting one without
-touching its endpoints, and managing the relationship-type catalog.
+that, linking resources that already exist, reading an edge back, cutting one without
+touching its endpoints, and managing the relationship-type catalogue.
[Create resources and relations →](./resources#create-resources-and-relations)
## The edge object {#body}
@@ -25,7 +25,7 @@ touching its endpoints, and managing the relationship-type catalog.
| `start` | number | Id of the `from` node. |
| `end` | number | Id of the `to` node. |
| `type` | string | The relationship type name, upper-cased (`CONTAINS`, `FLOWS_TO`). |
-| `relationshipTypeId` | number | The type's id in the [catalog](#types). |
+| `relationshipTypeId` | number | The type's id in the [catalogue](#types). |
| `description` | string | Prose. |
| `metadata` | map<string, string> | Flat key/value. |
@@ -55,9 +55,9 @@ POST /edges/create
Name each end by external id (`fromExternalId`/`toExternalId`) or by numeric id
(`fromId`/`toId`), and the relation by `relationshipType` or `relationshipTypeId`. A type
name you haven't used before is created for you, so [pre-registering types](#types) is only
-for seeding the catalog or attaching a description.
+for seeding the catalogue or attaching a description.
-The batch is **all-or-nothing** — one relation the server won't take and none of them are
+The batch is **all-or-nothing**, one relation the server won't take and none of them are
written. Success is a `201` with the created edges under `items`.
@@ -95,7 +95,7 @@ print(created[0].id)
`RelForm.by_external_ids(from_external_id, to_external_id, relationship_type)` is the short
form when you only need the three; `RelForm.by_ids` is its numeric-id counterpart. The service
-unwraps for you — `create` hands back a plain `list[EdgeProxy]`, not a wrapper.
+unwraps for you, `create` hands back a plain `list[EdgeProxy]`, not a wrapper.
@@ -118,39 +118,37 @@ println!("{:?}", created.get_items()[0].id);
| Status | Means |
| --- | --- |
-| `400` | An end doesn't exist (the message names which), the relation has no type, or it breaks one of the dataset or time-series rules below. |
+| `400` | An end doesn't exist (the message names which), the relation has no type, or it breaks one of the data set or time series rules below. |
| `403` | You can't write one of the two resources. Both ends are checked, so linking something *into* a data set needs write access on that data set too. |
-| `409` | The two are already connected that way. `(start, end, type)` is unique — one relation per pair per type. |
+| `409` | The two are already connected that way. `(start, end, type)` is unique, one relation per pair per type. |
-:::note Edges into datasets and time-series are validated
+:::note Edges into data sets and time series are validated
The same two rules the [graph create](./resources#create-resources-and-relations) enforces
apply here:
-- A relation **to a dataset** must use the `BELONGS_TO` relationship type — that is the
- relation dataset membership is built from, and anything else is a `400`.
-- A **dataset → time-series** edge is accepted only when the series has no dataset yet, or
- already belongs to that very dataset. A series in a *different* dataset is a `400`: a
- time-series has one dataset.
+- A relation **to a data set** must use the `BELONGS_TO` relationship type, that is the
+ relation data set membership is built from, and anything else is a `400`.
+- A **data set → time series** edge is accepted only when the series has no data set yet, or
+ already belongs to that very data set. A series in a *different* data set is a `400`: a
+ time series has one data set.
:::
## Look up {#look-up}
-`GET /edges/{id}` returns a single edge; an id that doesn't exist is a `404`. Older backends
-answered `200` with an empty `items[]` here, so code that has to work against both should
-check the count rather than the status.
+`GET /edges/{id}` returns a single edge; an id that doesn't exist is a `404`.
`POST /edges/byids` takes several ids and answers with a **graph**: the edges under
`relations` and the resources at both ends under `nodes`, so you don't need a follow-up call
to resolve endpoints. Unlike the single lookup, ids that match nothing are **silently
-omitted** — compare what comes back against what you asked for.
+omitted**, compare what comes back against what you asked for.
-Both lookups are gated by dataset grants: reading an edge requires **read access to the data
+Both lookups are gated by data set grants: reading an edge requires **read access to the data
sets of both endpoints**, mirroring the write rule, because an edge reveals both ends. A denied
edge behaves exactly like a missing one: `byids` omits it just as it omits an unknown id, and
-the single lookup answers `404`, so an edge you may not read is indistinguishable from one that
-does not exist — don't infer that two resources are unlinked from a missing edge. The MCP
-`edge_get` tool follows the same rule.
-[Dataset access control →](./datasets#access-control)
+the single lookup answers `404`. An edge you may not read is therefore indistinguishable from
+one that does not exist, so don't infer that two resources are unlinked from a missing edge.
+The MCP `edge_get` tool follows the same rule.
+[Data set access control →](./datasets#access-control)
@@ -177,7 +175,7 @@ for endpoint in many.nodes:
print(endpoint.external_id)
```
-Edges have no external id, so `by_ids` and `delete` take numeric ids — or an `EdgeProxy` you
+Edges have no external id, so `by_ids` and `delete` take numeric ids, or an `EdgeProxy` you
already hold, which is accepted anywhere an id is.
@@ -200,7 +198,7 @@ for endpoint in many.nodes().unwrap_or_default() {
## Delete {#delete}
`POST /edges/delete` (or `DELETE`, the endpoint takes both) removes relationships by id and
-answers `204` with no body. The resources at each end are untouched — this is how you
+answers `204` with no body. The resources at each end are untouched, this is how you
disconnect two things without losing either. [Deleting a resource](./resources#delete) is the
heavier move: it takes every relation the resource had with it.
@@ -208,7 +206,7 @@ Deletion is **idempotent**: unknown ids are silently skipped, so a successful ca
evidence the edge existed.
It can still be refused. Cutting an edge is rejected with a `400` if it would leave a
-surviving resource unreachable from a root — the same connectivity rule
+surviving resource unreachable from a root, the same connectivity rule
[deleting a resource](./resources#delete) is checked against, and the response names the
resources that would be stranded. An edge that is the only path from a subtree to the root is
exactly the one you cannot cut: re-attach the subtree another way first, or delete it in the
@@ -242,9 +240,9 @@ api.edges.delete(&vec![IdAndExtId::from_id(341)]).await?;
## Relationship types {#types}
-Every edge carries a type, and the types are a per-tenant catalog: `GET /edges/types` lists
-them, `POST /edges/types/create` registers names up front. Registering is optional — a type
-is created the first time an edge uses its name — so reach for it when you want the catalog
+Every edge carries a type, and the types are a per-tenant catalogue: `GET /edges/types` lists
+them, `POST /edges/types/create` registers names up front. Registering is optional, a type
+is created the first time an edge uses its name, so reach for it when you want the catalogue
seeded before anyone writes, or a `description`/`i18nCode` attached to a type.
A type is `{ id, name, description, i18nCode }`.
@@ -297,12 +295,12 @@ create a type by accident.
`POST /edges/create` only **upper-cases** the `relationshipType` you give it. No underscores
are inserted, so an edge created with `"relationshipType": "Flows To"` gets the type
-`FLOWS TO` — a *different* type from `FLOWS_TO`, silently created on the spot. Lookup is on the
+`FLOWS TO`, a *different* type from `FLOWS_TO`, silently created on the spot. Lookup is on the
normalised name, so `flows_to` and `FLOWS_TO` are the same type either way.
The practical rule: write the type name the way you want it stored, `FLOWS_TO`, and the two
paths agree. In Java, `RelForm.setName("Flows To")` snake-upper-cases client-side and lines the
-create path up with the catalog; `setRelationshipType` passes the string through. Python and
+create path up with the catalogue; `setRelationshipType` passes the string through. Python and
Rust send what you give them.
A name with no letter or digit in it (blank, or symbols only) is a `400`, and names registered
@@ -311,14 +309,13 @@ through `types/create` are capped at 128 characters.
:::caution A duplicate type name takes the whole batch with it
`POST /edges/types/create` has no find-or-create: it saves a fresh type unconditionally, so a
name that already exists (matched case-insensitively) collides on the unique name hash and
-comes back as a **`409`** naming the conflict — not the "existing ones returned unchanged" the
-endpoint used to advertise. You cannot use it to look up the id of a type you did not just
-create; read `GET /edges/types` for that.
+comes back as a **`409`** naming the conflict. You cannot use it to look up the id of a type
+you did not just create; read `GET /edges/types` for that.
Every form in a batch is saved in one transaction, so a single duplicate rolls the valid new
types back alongside it. Treat the `409` as *"nothing in this batch was created"* rather than
*"one of these already existed"*. Creating an edge with an unknown type name does not have
-this problem — that path is a proper find-or-create.
+this problem, that path is a proper find-or-create.
:::
## What each client covers {#client-coverage}
diff --git a/docs/reference/events.md b/docs/reference/events.md
index c96f1b0..f71e383 100644
--- a/docs/reference/events.md
+++ b/docs/reference/events.md
@@ -11,8 +11,8 @@ Record and query operational events.
:::info An event's `externalId` is a correlation key, not an identity
This is the opposite of what it means on a resource, and it is deliberate. An event's
-external id is the **source system's key for the subject** the event is about — an order, a
-permit, a batch — so **many events share one**. "Everything that happened to `PO-4500171`"
+external id is the **source system's key for the subject** the event is about, an order, a
+permit, a batch, so **many events share one**. "Everything that happened to `PO-4500171`"
is one indexed lookup, and that is what makes the log an audit trail.
No uniqueness is enforced, and none ever will be. Per-event identity is the event `id`
@@ -26,9 +26,9 @@ event even when a `snake_case` policy is rejecting it on resources.
| Field | Type | Notes |
| --- | --- | --- |
-| `id` | UUID string | The event's identity. Time-ordered UUID v7 — see the note under [Create](#create). |
+| `id` | UUID string | The event's identity. Time-ordered UUID v7, see the note under [Create](#create). |
| `externalId` | string, 3–256 | **Required.** The subject's key in the source system. Shared across events on purpose. |
-| `eventTime` | epoch millis, or ISO-8601 on the way in | **Required.** When it happened at the source. Never defaulted — see [Create](#create). |
+| `eventTime` | epoch millis, or ISO-8601 on the way in | **Required.** When it happened at the source. Never defaulted, see [Create](#create). |
| `type` | string, 3–128 | Top-level categorization (`alarm`, `work_order`). |
| `subType` | string, 3–128 | Refinement of `type` (`overpressure`). |
| `status` | string, 3–128 | Free-form lifecycle marker (`OPEN`, `acknowledged`). No state machine is enforced. |
@@ -53,21 +53,16 @@ HTTP caller should expect the quotes.
:::
:::note One list, not two parallel ones
-`relatedResources` replaced a `relatedResourceIds` / `relatedResourceExternalIds` pair. The two
-were independent inputs and drifted: a mismatched pair was unioned into an event describing both
-resources, and a patch setting only the external ids left the stored ids stale. There are no
-aliases, and events ignore unknown properties, so a client still sending the old field names gets
-a `200` with its relations silently dropped. Java SDK users get a compile break on the removed
-setters instead.
-
Supply an `id`, an `externalId`, or both. The server resolves whichever side you left out and
returns both, so a read always gives you the pair. Sending both when they name *different*
-resources is a `400` rather than a guess about which one you meant.
+resources is a `400` rather than a guess about which one you meant. The field names
+`relatedResourceIds` and `relatedResourceExternalIds` are unknown to the API and are refused
+with a `400`, like any other [unknown field](./client#unknown-fields).
:::
## Create {#create}
-Every event must carry an **event time** — the moment it occurred at the source (sensor,
+Every event must carry an **event time**, the moment it occurred at the source (sensor,
PLC, upstream system). The SDK deliberately does *not* default it to "now": an event
without it is rejected rather than silently mis-timestamped.
@@ -116,22 +111,22 @@ api.events.create(&vec![event]).await?;
Creating is **all-or-nothing**: if one event in the batch fails validation, none are
written. Attaching the event to resources that do not exist is a `400`, as is a
-`dataSetId` naming no data set — so a typo surfaces at write time rather than as an event
+`dataSetId` naming no data set, so a typo surfaces at write time rather than as an event
that quietly relates to nothing.
Create, update and delete each take at most **10 000 events** per request, and one event
carries at most 10 000 characters of `description`, 256 metadata entries and 100
`relatedResources`. Past any of those is a `400`. [Limits & quotas](./limits) has the rest,
-including the daily and lifetime ceilings on how many events an organisation may hold.
+including the daily and lifetime ceilings on how many events an organization may hold.
:::note Event ids are time-ordered UUID v7
-The ingestion paths stamp every event that has no `id` with a **UUID v7** before sending —
+The ingestion paths stamp every event that has no `id` with a **UUID v7** before sending,
`create` in the Python and Rust clients, `ingest(...)` in Java (a plain Java `create` sends
events as-is and lets the server assign ids). The server honors a client-supplied id, which is
what makes retries idempotent: the events table is keyed by `id` and collapses rows that share
one, so re-sending the same event (for example after a
[buffered](./client#durable-ingest-buffering) outage) leaves one row instead of a duplicate.
-If you set the `id` yourself, use a time-ordered UUID v7 — a random v4 scatters writes across
+If you set the `id` yourself, use a time-ordered UUID v7, a random v4 scatters writes across
that key and hurts insert and query performance. The created event (with its id) is returned
from `create`.
:::
@@ -139,7 +134,7 @@ from `create`.
## Look up {#lookup}
Fetch a single event by its UUID, or a batch by any mix of `id` and `externalId`. Ids that
-match nothing are **silently omitted** — compare what came back against what you asked for
+match nothing are **silently omitted**, compare what came back against what you asked for
if a miss matters. A batch is capped at 10 000 ids.
Because an external id is a correlation key, looking one up returns **every** event filed
@@ -154,7 +149,7 @@ DataWrapper events = client.events().byIds(List.of(
```
`IdCollection` carries a numeric id, so the Java client can only look events up by external
-id — an event's id is a UUID. Call `POST /events/byids` directly to fetch by UUID.
+id, an event's id is a UUID. Call `POST /events/byids` directly to fetch by UUID.
@@ -178,7 +173,7 @@ let events = api.events
-`GET /events/{id}` fetches one event by UUID and returns `404` when there is none — the one
+`GET /events/{id}` fetches one event by UUID and returns `404` when there is none, the one
place a missing event is an error rather than an omission.
## Query
@@ -219,14 +214,14 @@ let events = api.events.filter(&filter).await?;
-`limit` defaults to **100** and is capped at **10 000**; a zero or negative value falls back
-to 100 rather than returning nothing. Whatever you ask for, the result is intersected with
-the data sets your token may read — a filter can never widen access, so an empty page can
+`limit` defaults to **1 000** and is capped at **10 000**; a zero or negative value falls back
+to the default rather than returning nothing. Whatever you ask for, the result is intersected with
+the data sets your token may read, a filter can never widen access, so an empty page can
mean "no matches" or "none you may see", and the two are not distinguished.
### Filtering {#filtering}
-Every field you supply is combined with **AND** — an event must match all of them.
+Every field you supply is combined with **AND**, an event must match all of them.
| Field | Matching |
| --- | --- |
@@ -235,7 +230,7 @@ Every field you supply is combined with **AND** — an event must match all of t
| `metadata` | Every key/value pair given must be present on the event. |
| `dataSetId` | Events belonging to these data sets. |
| `relatedResources` | Events attached to these resources. |
-| `eventTime`, `createdTime`, `lastUpdatedTime` | `{ "min": …, "max": … }` bounds — see the note below. |
+| `eventTime`, `createdTime`, `lastUpdatedTime` | `{ "min": …, "max": … }` bounds, see the note below. |
Each field above takes **either a bare value or an array**, and the entries of an array are
combined with **OR**. That is why they are named in the singular: asking for one thing is the
@@ -268,33 +263,31 @@ case-insensitively, so `"SHIFT_REPORT_1*"` is the case-insensitive way to ask th
:::
**A parent data set stands in for its children.** Naming one covers everything beneath it in the
-`BELONGS_TO` hierarchy, which is the same expansion access control applies to a grant, so the two
-now agree.
+`BELONGS_TO` hierarchy, the same expansion access control applies to a grant.
An `externalId` that names no data set contributes nothing. That can only ever narrow the
-result — a typo gives you too few events, never events you should not see.
+result, a typo gives you too few events, never events you should not see.
:::caution Omitting `dataSetId` and sending `[]` are opposites
Omit the field (or send `null`) for **no data set restriction**. An explicit empty list means
**narrow to no data sets**, which matches nothing.
-The distinction matters if you build the filter programmatically: code that collects data set
-references into a list and always sets the field will silently return zero events when that list
-comes back empty, rather than the unrestricted result the same code returns for every other
-filter field.
+The distinction matters if you build the filter programmatically. Code that collects data set
+references into a list and always sets the field silently returns zero events when that list
+comes back empty. For every other filter field the same code returns the unrestricted result.
:::
:::note `eventTime.max` is exclusive; the other maxima are inclusive
`eventTime` is matched as `min <= t < max`, while `createdTime` and `lastUpdatedTime` are
-matched as `min <= t <= max`. That makes back-to-back `eventTime` windows tile cleanly —
-`[Monday, Tuesday)` then `[Tuesday, Wednesday)` covers every event exactly once — where the
+matched as `min <= t <= max`. That makes back-to-back `eventTime` windows tile cleanly,
+`[Monday, Tuesday)` then `[Tuesday, Wednesday)` covers every event exactly once, where the
same pattern on `createdTime` double-counts the boundary millisecond.
:::
### Advanced filters {#advanced-filters}
`advancedFilter` sits alongside `filter` and builds a boolean expression when flat AND is not
-enough — "type is alarm **or** the source is SAP", or "everything except the `test_` prefix".
+enough, "type is alarm **or** the source is SAP", or "everything except the `test_` prefix".
Combine with `and`, `or` and `not`; the leaves take one of three operators:
| Operator | Meaning |
@@ -320,13 +313,13 @@ being ignored.
}
```
-Two things to know. `property` is a list, but only its **first** entry is read — there is no
+Two things to know. `property` is a list, but only its **first** entry is read, there is no
nested path into `metadata`. And every value is compared as a string, so `dataSetId` matches
`"43"`, not `43`.
### Ordering and paging {#paging}
-Events come back **`eventTime` ascending** unless you say otherwise — that is the order the
+Events come back **`eventTime` ascending** unless you say otherwise, that is the order the
cursor pages in, so paging does not change the order underneath you. Ask for another with
`sort`, over `eventTime`, `createdTime`, `lastUpdatedTime`, `externalId`, `type`, `subType`,
`status`, `source` or `dataSetId`:
@@ -337,10 +330,10 @@ cursor pages in, so paging does not change the order underneath you. Ask for ano
"limit": 200 }
```
-Only the **first** `property` is used, and `id` is appended behind it — a sort column alone is
+Only the **first** `property` is used, and `id` is appended behind it, a sort column alone is
not a position unless it is unique, and a page boundary inside a run of equal values repeats or
drops exactly those rows. A property that is not sortable is ignored rather than rejected, and
-any `order` that is not exactly `desc` sorts ascending — a malformed sort degrades to the
+any `order` that is not exactly `desc` sorts ascending, a malformed sort degrades to the
default instead of silently reversing your results. Null values sort last ascending, first
descending.
@@ -353,14 +346,14 @@ To walk past the first page, echo back the `nextCursor` the response carried:
"limit": 200 }
```
-The cursor is **opaque** — base64 of a versioned encoding carrying the sort, the boundary value
-and the id — so do not build or parse one. A cursor that does not decode restarts the walk from
-the first page rather than failing, which is obviously wrong to a caller, where guessing at half
-a position would silently skip or repeat the rows around the boundary.
+The cursor is **opaque**, base64 of a versioned encoding carrying the sort, the boundary value
+and the id, so do not build or parse one. A cursor that does not decode is refused with a
+`400` of `type: ".../errors/malformed-cursor"` rather than guessed at: half a position would
+silently skip or repeat the rows around the boundary.
Send it with the **same** `sort` that produced it: a cursor is a position in one particular
-order. Continuing it under another is refused, though today the refusal arrives as an empty
-response rather than a clean 400. Sorting by `subType` or `status` cannot be paged at all —
+order. Continuing it under another is refused with the same `400`, which names both sorts.
+Sorting by `subType` or `status` cannot be paged at all,
both columns are nullable, and a keyset boundary on them would skip the events that have no
value.
@@ -368,7 +361,7 @@ Prefer this to counting pages. Events are stored partitioned by event time, so r
a position lets whole partitions be skipped, where an offset re-reads everything ahead of it
and gets slower the further you page. Page 400 costs what page 1 costs. The trade is that
there is no random access: you walk forward from where you were and cannot jump to page 7.
-`nextCursor` is absent on a short page, so "keep going while it is present" is the whole loop —
+`nextCursor` is absent on a short page, so "keep going while it is present" is the whole loop,
and since a full page may still be the last, a complete walk ends with one empty request.
All three clients read the cursor off the response envelope rather than off the last event:
@@ -407,8 +400,8 @@ if let Some(cursor) = page.next_cursor() {
## Policy findings {#policy-findings}
-A **policy finding** — a naming-policy violation that was allowed through and recorded for a
-steward — is an ordinary event. There is no findings endpoint and no findings client: they are
+A **policy finding**, a naming-policy violation that was allowed through and recorded for a
+steward, is an ordinary event. There is no findings endpoint and no findings client: they are
written to the event store like anything else, so everything on this page already works on
them.
@@ -416,23 +409,23 @@ The encoding is a wire contract, so you can read findings without a policy-aware
| Field | Holds |
| --- | --- |
-| `type` | Always `policy_finding`. Matched exactly, never by prefix — this is the one filter separating findings from the tenant's real events. |
+| `type` | Always `policy_finding`. Matched exactly, never by prefix, this is the one filter separating findings from the tenant's real events. |
| `subType` | Which policy fired, by its external id. |
| `source` | `datahub_policy_`, truncated to 128 characters. |
-| `externalId` | `policy_finding__` — the correlation key every event in one finding's lifecycle shares. |
-| `status` | `OPEN` or `RESOLVED` — what *this event* asserts, not the finding's current state. |
+| `externalId` | `policy_finding__`, the correlation key every event in one finding's lifecycle shares. |
+| `status` | `OPEN` or `RESOLVED`, what *this event* asserts, not the finding's current state. |
| `description` | What is wrong, in words. |
| `relatedResources` | The entity the finding is about, by node id. |
| `dataSetId` | That entity's data set. |
| `metadata` | `offendingValue`, `suggestion` (when one could be derived), `raisedBy`. |
Note the external id is keyed on the entity's **node id**, not its external id. The external
-id is the offending value here — the thing a steward is most likely to change — and keying on
+id is the offending value here, the thing a steward is most likely to change, and keying on
it would mean renaming a resource silently abandoned its finding and started a second stream.
### Fetch the queue
-Filter on the type, ascending by `eventTime`, and page with [`after`](#paging):
+Filter on the type, ascending by `eventTime`, and page with [`cursor`](#paging):
@@ -484,7 +477,7 @@ let findings = api.events.filter(&filter).await?;
### Fold the stream
-A finding's current state is **not stored** — you derive it. Nothing is ever updated in place:
+A finding's current state is **not stored**, you derive it. Nothing is ever updated in place:
raising appends an `OPEN`, resolving appends a `RESOLVED` carrying the same `externalId`. Group
by external id, order by `eventTime` ascending, and the last event wins:
@@ -535,22 +528,22 @@ let open: Vec<_> = current.values().filter(|e| e.status.as_deref() == Some("OPEN
:::caution Do not filter on `status`
A stored `OPEN` event means *this was raised*, not *this is outstanding*. Filtering the query
-on `status: "OPEN"` returns the raise of every finding that has since been resolved — the
+on `status: "OPEN"` returns the raise of every finding that has since been resolved, the
resolve is a separate, later event, and narrowing the query hides it. Open-ness is a
conclusion drawn from the stream, not a fact the store holds, so fetch and fold.
Order ascending for the same reason: replay out of order and a stale `OPEN` overwrites the
-`RESOLVED` that followed it. Keep folding across pages too — a `RESOLVED` on page 3 closes a
+`RESOLVED` that followed it. Keep folding across pages too, a `RESOLVED` on page 3 closes a
finding whose `OPEN` arrived on page 1.
:::
:::caution A fold needs the *whole* stream, so a truncated page lies
Folding is only correct if every event sharing an external id is in front of you. Get one
page of a queue larger than your `limit` and an `OPEN` can arrive without the `RESOLVED` that
-closed it — the fold then reports a resolved finding as outstanding. It is a wrong answer,
+closed it, the fold then reports a resolved finding as outstanding. It is a wrong answer,
not an error, and nothing in the response marks it as partial.
-Page until short — a full page is a signal that there is more, never that you have it all.
+Page until short, a full page is a signal that there is more, never that you have it all.
Narrowing by `subType` and `dataSetId` keeps the walk cheap, but it is paging, not narrowing,
that makes the fold correct.
:::
@@ -558,10 +551,10 @@ that makes the fold correct.
Raising is idempotent: a raise event's id is derived from what it asserts, so re-evaluating an
entity whose external id has not changed collapses onto the raise already stored. An entity
written a thousand times contributes one `OPEN`, not a thousand. A raise for a *different*
-non-conforming value is a new fact and is appended — which is also all "reopening" is.
+non-conforming value is a new fact and is appended, which is also all "reopening" is.
-For the steward's side of this — how to resolve a finding, what resolving means, and why
-findings are raised for resources but never for events — see
+For the steward's side of this, how to resolve a finding, what resolving means, and why
+findings are raised for resources but never for events, see
[Findings](./external-ids#findings).
## Full-text search {#search}
@@ -582,14 +575,8 @@ narrows the phrase's hits:
}
```
-:::note What changed
-`filter` used to be accepted and silently ignored here, as it was on the resource and data set
-searches. All four searches now apply it. `dataSetId` covers everything beneath the data sets you
-name, exactly as it does on `/events/filter`.
-
-The description above is also a correction: this endpoint never was fuzzy, word-aware or
-relevance-ranked, whatever the previous wording said.
-:::
+`dataSetId` in that block covers everything beneath the data sets you name, exactly as it does on
+`/events/filter`.
`query` must be 3 to 140 characters. There is no character restriction beyond that: punctuation,
underscores and non-Latin scripts are all accepted, so an externalId or a Cyrillic asset name can
@@ -600,7 +587,7 @@ related resource). It is faster and its results are predictable.
## Distinct values {#distinct-values}
-Two families of endpoint answer "what values actually occur?" — the material for a filter
+Two families of endpoint answer "what values actually occur?", the material for a filter
drop-down or a type-ahead, without scanning events yourself. Both are restricted to the data
sets your token may read, so a UI built on them cannot offer a facet the user could not then
query.
@@ -627,13 +614,13 @@ GET /events/search/type?q=alarm&limit=20
## Count {#count}
`GET /events/count` returns `{ "count": 148392 }` for the tenant. It is a single cheap query,
-and it takes **no filters** — for a filtered count, run `POST /events/filter` with the `limit`
+and it takes **no filters**, for a filtered count, run `POST /events/filter` with the `limit`
you care about and measure the page.
## Update {#update}
`POST /events/update` changes fields on events that already exist. Identify each one by UUID
-`id` or by `externalId`, and name only the fields you want changed — anything you leave out
+`id` or by `externalId`, and name only the fields you want changed, anything you leave out
keeps its current value.
Each field is an object carrying a verb rather than a bare value, which is what lets "clear
@@ -642,7 +629,7 @@ this" be expressed distinctly from "leave it alone":
| Verb | Applies to | Effect |
| --- | --- | --- |
| `set` | every field | Replace the value. |
-| `setNull: true` | nullable fields only | Clear the value. `externalId`, `type` and `eventTime` are not nullable, so asking to clear any of them is a `400`. |
+| `setNull: true` | nullable fields only | Clear the value. `externalId` and `type` are not nullable, so asking to clear either is a `400`. |
| `add` | `metadata`, `relatedResources` | Merge entries in, keeping the rest. |
| `remove` | the same collections | Take entries out, keeping the rest. A `relatedResources` entry matches on either side, so you can remove by `id` or by `externalId` whichever you have. |
@@ -661,13 +648,19 @@ this" be expressed distinctly from "leave it alone":
```
Updatable fields are `externalId`, `description`, `type`, `subType`, `status`, `source`,
-`dataSetId`, `metadata`, `eventTime` and `relatedResources`. `eventTime` is set from an
-ISO-8601 string. Sending both `set` and `setNull` for one field is
-a `400` — the request is contradictory, so it is refused rather than resolved by precedence.
-
-`setNull` is refused on the three fields a create cannot omit: `externalId`, `type` and
-`eventTime`. Clearing `type` used to be accepted, and it left the event unreadable by any
-client that models `type` as required, so the read failed rather than the write. `dataSetId`
+`dataSetId`, `metadata` and `relatedResources`. Sending both `set` and `setNull` for one
+field is a `400`, the request is contradictory, so it is refused rather than resolved by
+precedence.
+
+`eventTime` is **fixed at creation** and is not an update field at all: an update naming it
+is a `400` that names the field, the same answer as for any field the form does not have. The
+store partitions events by their time, and a row cannot move between partitions. An event
+recorded against the wrong moment is deleted and written again, or corrected by a follow-up
+event, which the caution below recommends anyway.
+
+`setNull` is refused on `externalId` and `type`, the two fields a create cannot omit and an
+update can name. Clearing `type` would leave the event unreadable by any client that models
+`type` as required, so the write is refused rather than the read failing later. `dataSetId`
is the one field here that genuinely is nullable: `setNull` detaches the event from its data
set, and naming a `dataSetId` that no data set has is a `400` rather than a stored dangling
reference.
@@ -678,7 +671,7 @@ concurrent read of the same event can briefly return the pre-update version *or*
twice. Where the record matters for audit, write a new event that corrects the old one
instead: that is what an append-only log is for, and it keeps the correction itself visible.
-`status` is the honourable exception — acknowledging an alarm in place is what the field is
+`status` is the honourable exception, acknowledging an alarm in place is what the field is
there for.
:::
@@ -726,17 +719,17 @@ Deletes are **idempotent**: removing an event that is already gone returns `200`
nothing, so a retried delete needs no bookkeeping.
Remember that an external id names a *subject*, not an event. Deleting by external id removes
-**every event filed under it**, which is rarely what you want for a single mistaken record —
+**every event filed under it**, which is rarely what you want for a single mistaken record,
delete that one by its UUID.
:::caution A `200` means "accepted", not "gone"
The delete is published to the ingestion pipeline and marked in the backend without waiting
-for the removal to land — a background job does the actual work. Until it has run, the event
+for the removal to land, a background job does the actual work. Until it has run, the event
**can still come back from `filter` and `byids`**.
So a test that deletes an event and immediately asserts it is gone will flake, and so will a
UI that re-queries straight on the back of a delete. Poll until the event disappears rather
-than reading once, and treat its absence — not the `200` — as the signal. The same eventual
+than reading once, and treat its absence, not the `200`, as the signal. The same eventual
consistency applies in the other direction: an event is not necessarily queryable the instant
`create` returns.
@@ -770,8 +763,7 @@ api.events.delete(&vec![IdAndExtId::from_external_id("door_open")]).await?;
## What each client covers {#client-coverage}
-The three clients cover the write and read paths, including the facet endpoints; the
-administrative ones are HTTP-only so far.
+The three clients cover every event endpoint, the facet endpoints included.
| Operation | Java | Python | Rust |
| --- | --- | --- | --- |
@@ -779,8 +771,8 @@ administrative ones are HTTP-only so far.
| Get by id | `events().getById` | `events.get` | `events.get` |
| Look up by id / external id | `events().byIds` | `events.by_ids` | `events.by_ids` |
| Filter | `events().filter` | `events.filter` | `events.filter` |
-| — with `sort` | `EventRetreiver.sort` | `sort_by` / `sort_order` | `set_sort` |
-| — with paging | `EventRetreiver.cursor` | `cursor` | `set_cursor` |
+| Filter with `sort` | `EventRetreiver.sort` | `sort_by` / `sort_order` | `set_sort` |
+| Filter with paging | `EventRetreiver.cursor` | `cursor` | `set_cursor` |
| Update | `events().update` | `events.update` | `events.update` |
| Full-text search | `events().search` | `events.search` | `events.search` |
| Count | `events().count` | `events.count` | `events.count` |
@@ -790,7 +782,6 @@ administrative ones are HTTP-only so far.
All three carry the same four pairs: `list_types` / `search_types` and the same for sub-types,
statuses and sources (`listTypes` / `searchTypes` … in Java).
-Take the paging value from the response envelope — `getNextCursor()` in Java, `page.next_cursor`
-in Python, `page.next_cursor()` in Rust — and send it back unchanged. It is opaque; the
-`_` value earlier versions had you assemble by hand no longer decodes, and an
-undecodable cursor restarts the walk from the first page.
+Take the paging value from the response envelope, `getNextCursor()` in Java, `page.next_cursor`
+in Python, `page.next_cursor()` in Rust, and send it back unchanged. It is opaque; an
+undecodable cursor is refused with a `400`.
diff --git a/docs/reference/external-ids.md b/docs/reference/external-ids.md
index 6e59b71..236123f 100644
--- a/docs/reference/external-ids.md
+++ b/docs/reference/external-ids.md
@@ -16,17 +16,17 @@ Same field name, different jobs.
| On | What `externalId` is | Unique? |
| --- | --- | --- |
-| **Resources, data sets, time-series** | The **identity** of one thing — your key for it, and what every integration matches on. | Yes, per tenant, compared without case |
-| **Events** | A **correlation key** — the source system's key for the *subject* the event is about. | **No, and deliberately never** |
+| **Resources, data sets, time series** | The **identity** of one thing, your key for it, and what every integration matches on. | Yes, per tenant, compared without case |
+| **Events** | A **correlation key**, the source system's key for the *subject* the event is about. | **No, and deliberately never** |
An order that is created, amended and then shipped produces three events all carrying
`PO-4500171`. That is the point: the order's history is "every event with this external id,
in time order", and the log is an audit trail because of it. Per-event identity comes from
-the platform's own event `id` (a time-ordered UUID v7 — see [Events](./events)), never from
+the platform's own event `id` (a time-ordered UUID v7, see [Events](./events)), never from
the external id.
:::danger Do not synthesise per-event external ids
-Making event external ids unique — `PO-4500171-1`, `PO-4500171-2` — throws away the only
+Making event external ids unique, `PO-4500171-1`, `PO-4500171-2`, throws away the only
cheap way to ask for a subject's history, and pushes you toward updating events in place,
which destroys the append-only record. If you need to de-duplicate a redelivered snapshot,
set the event `id` yourself; retries then collapse to one row.
@@ -34,7 +34,7 @@ set the event `id` yourself; retries then collapse to one row.
## The two layers
-### Layer 1 — the charset floor {#the-charset-floor}
+### Layer 1: the charset floor {#the-charset-floor}
Platform-wide, not configurable, and it applies **everywhere an external id is accepted,
events included**.
@@ -59,13 +59,12 @@ P1 rejected: shorter than 3 characters
Byte-for-byte storage is what lets you join on identifiers you already maintain. The
historian, the maintenance system and DataHub hold the same string, so a match is a string
-comparison instead of a normalisation each integration has to reimplement identically,
-forever.
+comparison instead of a normalisation each integration has to reimplement identically.
-### Layer 2 — the naming policy {#the-naming-policy}
+### Layer 2: the naming policy {#the-naming-policy}
A convention an administrator configures on top of the floor. It applies to **every node type
-whose external id is an identity**: resources and assets, data sets, time-series, functions and
+whose external id is an identity**: resources and assets, data sets, time series, functions and
policies. Enforcement sits on the one write path they share, so it cannot cover one create
endpoint and miss another.
@@ -93,8 +92,8 @@ than merging with it.
:::caution The naming policy never applies to events
Only the charset floor does. An event external id is not a name someone chose, it is the
-source system's key for the subject, so the platform does not impose a convention on data you
-do not own — and the policy's other rules would be meaningless there anyway, since events
+source system's key for the subject. The platform does not impose a convention on data you do
+not own, and the policy's other rules would be meaningless there anyway, since events
deliberately share external ids.
With a `snake_case` policy active and set to reject, `client.resources().create(...)` with
@@ -104,9 +103,9 @@ correct behaviour, not a gap.
## Case: compared without it, stored with it
-Uniqueness on resources, data sets and time-series ignores case. Creating `com-99-pt-1034`
+Uniqueness on resources, data sets and time series ignores case. Creating `com-99-pt-1034`
when `COM-99-PT-1034` already exists is a duplicate and comes back as **`409`**, with a
-message naming the id it collides with — it is the ordinary "this external id already
+message naming the id it collides with, it is the ordinary "this external id already
exists" path, not a naming-policy rejection.
Lookups ignore case for the same reason, so uniqueness and lookup agree:
@@ -117,7 +116,7 @@ GET /resources/VAL-01 ─┐
GET /resources/val-01 ─┘
```
-Storage is still verbatim, so **what you read back is byte-identical to what you sent** — a
+Storage is still verbatim, so **what you read back is byte-identical to what you sent**, a
lookup by `val-01` returns an entity whose `externalId` is `VAL-01`. Compare external ids
case-insensitively in your own code if you compare them at all.
@@ -128,7 +127,7 @@ bad item in 500 creates nothing, and the error names every offending item rather
stopping at the first, so a rejected import is fixed in one pass.
Items in the same request are also compared against each other, not just against stored data
-— sending `PUMP-01` and `pump-01` together is rejected.
+sending `PUMP-01` and `pump-01` together is rejected.
## Warnings on the response
@@ -178,7 +177,7 @@ document:
}
```
-It is **`400`, not `403`** — malformed input, not an access decision. `detail` says outright
+It is **`400`, not `403`**, malformed input, not an access decision. `detail` says outright
that nothing was created, because that is the first thing you need to know before retrying.
The body reaches you through the ordinary error path in each client: `DatahubApiException`
@@ -215,7 +214,7 @@ POST /policies/naming/check
```
Takes candidate external ids and an optional data set id, and returns the same findings the
-write path would produce — **without writing anything**. It runs the same evaluator, so
+write path would produce, **without writing anything**. It runs the same evaluator, so
there is no second set of rules to keep in step.
```json
@@ -229,7 +228,7 @@ there is no second set of rules to keep in step.
`names` is optional and pairs with `externalIds` **by position**. Supply it when you have it:
suggestions are derived from the name first, so an entity called `Valve pressure sensors`
gets offered `valve_pressure_sensors`, where deriving from a broken id could only manage
-`vps`. Either omit `names` entirely or send exactly as many as there are external ids — a
+`vps`. Either omit `names` entirely or send exactly as many as there are external ids, a
partial list is rejected rather than paired up wrongly.
Two things it is good for:
@@ -240,35 +239,35 @@ Two things it is good for:
### Suggestions {#suggestions}
-Every warning and every rejection carries a `suggestion` where one can be derived — a
+Every warning and every rejection carries a `suggestion` where one can be derived, a
conforming external id you can offer as a one-click fix. It is **offered, never applied**:
-the platform stopped rewriting external ids, which is the whole point of this change.
+the server never rewrites an external id.
Two guarantees make it safe to wire straight into a form:
- **A suggestion always satisfies the policy it is offered for.** It is checked against the
charset floor, the length bounds and the active preset before being returned, so applying
one cannot bounce back with a second rejection.
-- **A suggestion is never an id that is already taken** — neither one that is stored nor one
+- **A suggestion is never an id that is already taken**: neither one that is stored nor one
claimed by an earlier item in the same batch.
When nothing can be derived that satisfies both, `suggestion` is absent. That is deliberate:
an honest omission beats a confident wrong answer. In particular a near-duplicate rejection
usually has no suggestion, because every variant of the same name folds to the same taken
-value — the `reason` names the existing id instead, since the likeliest fix is to use it.
+value, the `reason` names the existing id instead, since the likeliest fix is to use it.
## Findings {#findings}
Every warning is recorded, so warn means *allowed and in the steward's queue*, not *allowed
and forgotten*.
-**A finding is an event** — and more precisely, a *stream* of events. There is no findings
+**A finding is an event**, and more precisely, a *stream* of events. There is no findings
endpoint: findings are stored, filtered and resolved as ordinary events, so everything you
already use for events works on them.
Nothing is ever updated in place. Raising a finding appends an `OPEN` event; resolving it
appends a `RESOLVED` event carrying the **same `externalId`**. A finding's current state is not
-stored — you derive it: take every event sharing that external id, order by `eventTime`
+stored, you derive it: take every event sharing that external id, order by `eventTime`
ascending, and the last one wins.
Read the queue by filtering events on the finding type:
@@ -289,7 +288,7 @@ POST /events/filter
:::caution Do not filter on `status`
A stored `OPEN` event means *this was raised*, not *this is outstanding*. Filtering the query on
`status: "OPEN"` would return the raise of every finding that has since been resolved. Fetch the
-stream and fold it — that is what "the last event wins" means in practice.
+stream and fold it, that is what "the last event wins" means in practice.
Order ascending for the same reason: replaying out of order lets a stale `OPEN` overwrite the
`RESOLVED` that followed it.
@@ -299,14 +298,14 @@ Each finding event carries:
| Field | What it holds |
|---|---|
-| `externalId` | The finding this event belongs to — the correlation key you fold on |
+| `externalId` | The finding this event belongs to, the correlation key you fold on |
| `subType` | The policy that fired, by external id |
| `source` | `datahub_policy_` |
| `description` | What is wrong, in words |
| `relatedResources` | The entity the finding is about, by node id |
| `dataSetId` | That entity's data set |
| `eventTime` | When this happened |
-| `status` | `OPEN` or `RESOLVED` — what *this event* asserts |
+| `status` | `OPEN` or `RESOLVED`, what *this event* asserts |
| `metadata.offendingValue` | The external id that tripped the policy |
| `metadata.suggestion` | A conforming alternative, when one could be derived |
| `metadata.raisedBy` | Subject of whoever wrote the offending value |
@@ -334,7 +333,7 @@ misses the answer to it. Resolving therefore needs write access to that data set
Resolving is a judgement rather than a fix: the entity still breaks the policy, someone has
decided that is acceptable. Because it is appended rather than edited, it does not erase the
-raise it answers — the finding's history stays readable.
+raise it answers, the finding's history stays readable.
**Reopening needs no special rule.** If the external id later changes to another non-conforming
value, the policy appends a fresh `OPEN` after the `RESOLVED`, and the replay says open again.
@@ -349,34 +348,34 @@ does not change that.
### Paging the queue {#findings-paging}
-Findings from a bulk import arrive in the thousands, so page with `after` rather than an
-offset — `_`, from the last event you saw — and keep folding into the
-same state as pages arrive: a `RESOLVED` on page 3 closes a finding whose `OPEN` came on page 1:
+Findings from a bulk import arrive in the thousands, so page with the cursor rather than an
+offset, and keep folding into the same state as pages arrive: a `RESOLVED` on page 3 closes a
+finding whose `OPEN` came on page 1. Sort by `eventTime` ascending, which is the order a fold
+needs, and echo each response's `nextCursor` back as `cursor` with the same `sort`:
```http
POST /events/filter
{
"filter": { "type": "policy_finding" },
- "after": "1754476522104_0195f3a2-4c1b-7f9e-9c3a-1b2d4e6f8a90",
+ "sort": { "property": ["eventTime"], "order": "asc" },
+ "cursor": "djE6ZXZlbnRUaW1lfGFzY3wxNzU0NDc2NTIyMTA0fDAxOTVmM2Ey",
"limit": 200
}
```
-No `sort` here, and no `status` either. `after` fixes the order to `eventTime` then `id`
-ascending on its own, which is exactly the order a fold needs — and narrowing to `OPEN` would
-drop the `RESOLVED` events that close the findings you are folding.
+No `status` in the filter: narrowing to `OPEN` would drop the `RESOLVED` events that close the
+findings you are folding.
-Both halves of `after` are required. Event times are not unique — an import lands thousands
-of findings in the same millisecond — so paging on the timestamp alone would either skip that
-group or repeat it forever. A short page is the last page.
+The cursor is opaque and carries both the boundary event's time and its id, so an import that
+lands thousands of findings in the same millisecond pages cleanly. `nextCursor` is absent on
+the last page.
[Fetching and folding the queue, with SDK examples →](./events#policy-findings)
## Practical advice
-- **Send the identifier your source system already uses.** Pre-normalising to snake_case
- still works and nothing that worked before has stopped, but it costs you the byte-for-byte
- join the platform is built around.
+- **Send the identifier your source system already uses.** Pre-normalising to snake_case is
+ accepted but loses the byte-for-byte match with the source system.
- **Never change an external id after the fact.** It is a promise other systems have written
down. The platform permits it; your integrations will not forgive it.
- **Read `warnings` if you have a steward.** It is the difference between finding out now and
diff --git a/docs/reference/files.md b/docs/reference/files.md
index 41b854a..5d7e848 100644
--- a/docs/reference/files.md
+++ b/docs/reference/files.md
@@ -41,8 +41,12 @@ let listing = api.files.list_directory_by_path("/reports/2026").await?;
## Upload
-The Java client uploads raw `content` bytes to a destination `path`; the Python and Rust
-clients upload a local file and a `destination_path`.
+The server takes one path, `X-Datahub-Path`: the full path of the file including its name
+(`/reports/2026/q2.csv`), from which it splits the name off the last `/`. The Java client
+sends the `path` you give it as-is, so include the file name and a leading `/`. The Python
+and Rust clients take a `destination_path` folder plus a `name` (defaulting to the local
+file's name) and join the two into that same full path. The Java client uploads raw `content`
+bytes; the Python and Rust clients read a local file.
@@ -52,7 +56,7 @@ byte[] content = Files.readAllBytes(Path.of("report.csv"));
DataWrapper uploaded = client.files().upload(
FileUploadRequest.builder()
- .path("reports/2026/q2.csv")
+ .path("/reports/2026/q2.csv")
.content(content)
.contentType("text/csv") // default: application/octet-stream
.externalId("report_2026_q2") // optional
@@ -96,9 +100,52 @@ let uploaded = api.files.upload_file(upload).await?;
+The upload echo carries the stored node, id included. Read it back from the folder listing:
+
+
+
+
+```java
+client.files().list("/reports/2026").getItems()
+ .forEach(n -> System.out.println(n.getId() + " " + n.getName() + " " + n.getSize()));
+```
+
+
+
+
+```python
+for node in client.files.list_directory_by_path("/reports/2026"):
+ print(node.id, node.name, node.size)
+```
+
+
+
+
+```rust
+for node in api.files.list_directory_by_path("/reports/2026").await?.get_items() {
+ println!("{:?} {} {}", node.id, node.name, node.size);
+}
+```
+
+
+
+
+### When it fails {#errors}
+
+| Status | Means |
+| --- | --- |
+| `403` | You lack write access to the data set named in `dataSetId`, or to the parent folder's data set. |
+| `404` | On download: no file with that id, or a file in a data set you may not read. The two are not distinguished, so a hidden file's existence is not leaked. |
+| `409` | A file with that path, or that `externalId`, already exists. Uniqueness is tenant-wide, not per data set. |
+
+There is no size cap on `PUT /files`: the upload streams to disk and is exempt from the
+[request-body limit](./limits#request-body-size).
+
## Download {#download}
-All three clients download a file's raw bytes by id. Python and Rust add a streaming variant
+All three clients download a file's raw bytes by id. The id is numeric: Python and Rust take
+it as an integer, Java as a string, because the endpoint also accepts an external id in that
+position (`download("report_2026_q2")` works in Java). Python and Rust add a streaming variant
that writes straight to a path, so a large file never has to sit in memory whole.
@@ -174,4 +221,4 @@ api.files.delete(&DataWrapper::from(vec.
+file in memory, see [Download](#download).
diff --git a/docs/reference/limits.md b/docs/reference/limits.md
index ddbf590..4ea2fff 100644
--- a/docs/reference/limits.md
+++ b/docs/reference/limits.md
@@ -6,7 +6,7 @@ description: The size, rate and volume ceilings the API enforces, the status cod
# Limits & quotas
-The API enforces a handful of ceilings, and the status code says which one you hit and what to
+The API enforces six kinds of ceiling, and the status code says which one you hit and what to
do about it. That is the whole design: **what clears by waiting answers `429` and carries a
`Retry-After`, and what does not answers something else.** Retry the first kind, fix the
second.
@@ -19,6 +19,7 @@ second.
| [Rate limit](#rate-limits) | `429` + `Retry-After` | `.../errors/rate-limit-exceeded` | Wait the seconds it names |
| [Daily ingest quota](#daily-ingest-quotas) | `429` + `Retry-After` | `.../errors/ingest-quota-exceeded` | Wait until 00:00 UTC |
| [Lifetime ceiling](#lifetime-ceilings) | `403`, no `Retry-After` | `.../errors/tenant-limit-reached` | Ask for it to be raised |
+| [Graph transfer](#graph-transfer) | `400` on export, `413` on import | an `error` body, not a problem document | It does not: the component is too large to move as one file |
Every `type` above is prefixed `https://intellistream.ai/errors/`, and the `429` and `413`
bodies are [RFC 9457](https://www.rfc-editor.org/rfc/rfc9457) problem documents served as
@@ -69,12 +70,9 @@ The tighter [`TEXT`/`MIXED`](./timeseries#value-types) cap is checked in the ser
series' value type has been resolved, so it comes back naming the series type rather than the
field. Split a text series into collections of 10 000 points or fewer.
-:::caution The OpenAPI schema used to advertise caps nothing enforced
-Some endpoints, `/events/delete` among them, carried a documented maximum in the schema that
-the runtime never checked, so an oversized batch went through. Those caps are **enforced
-now**. Code written against the advertised numbers is unaffected; code that quietly relied on
-them not being real is not.
-:::
+The `items` cap is enforced wherever the handler validates the body. `/events/update` and
+`/events/delete` do not, so an oversized update or delete batch is bounded only by the
+[request body size](#request-body-size).
## Request body size {#request-body-size}
@@ -97,9 +95,29 @@ them not being real is not.
A `413` is **terminal**. The same request will never become acceptable by being sent again,
so split the batch instead of retrying it.
+`POST /resources/import` is under the 4 MiB cap like everything else, even though the
+[graph file format](./resources#graph-transfer) itself allows up to 512 MB. A deployment that
+imports larger graphs raises `datahub.limits.max-body-bytes`, which is deployment-wide, and
+the ceiling on any reverse proxy in front of it.
+
+## Graph transfer {#graph-transfer}
+
+[Export and import](./resources#graph-transfer) of a graph component have ceilings of their
+own, fixed in the file format rather than set by the deployment:
+
+| Limit | Cap | Answered with |
+| --- | --- | --- |
+| Nodes in one component or file | 2 000 000 | `400` on export, `413` on import |
+| Relationships in one component or file | 2 000 000 | `400` on export, `413` on import |
+| Compressed file size on import | 512 MB | `413` |
+
+These answer with the `{ "error": { "code", "message" } }` body the resource endpoints use,
+not a problem document, and none of them clears by waiting: a component over the cap cannot
+be exported as one file at all.
+
## Rate limits {#rate-limits}
-Counted per organisation and per user in a fixed one-minute window, with separate budgets for
+Counted per organization and per user in a fixed one-minute window, with separate budgets for
reads and writes.
Which budget a request spends follows what it **does**, not which method it uses. A `GET` is a
@@ -112,7 +130,7 @@ allowance, which is the larger of the two.
| Scope | Writes / min | Reads / min |
| --- | --- | --- |
-| Organisation | 2 000 | 6 000 |
+| Organization | 2 000 | 6 000 |
| User | 600 | 1 200 |
```json
@@ -136,7 +154,7 @@ ingest job share one allowance.
## Daily ingest quotas {#daily-ingest-quotas}
-Per organisation, per UTC day, reset at 00:00 UTC. `Retry-After` points at that reset, so it
+Per organization, per UTC day, reset at 00:00 UTC. `Retry-After` points at that reset, so it
can be hours.
| `metric` | Default per day |
@@ -164,8 +182,8 @@ functions: they are one population, not five.
## Lifetime ceilings {#lifetime-ceilings}
-Totals, not rates: how large an organisation may grow. Unlike everything above, these are
-**off unless a deployment turns them on**, and the numbers size a free or trial organisation.
+Totals, not rates: how large an organization may grow. Unlike everything above, these are
+**off unless a deployment turns them on**, and the numbers size a free or trial organization.
Handle the `403`, but do not plan your data model around these figures: ask whoever runs your
deployment what applies to you.
@@ -189,7 +207,7 @@ deployment what applies to you.
There is deliberately **no `Retry-After`**: waiting does not clear a ceiling, and the status
is `403` rather than `429` so no client mistakes it for one that does. The ceiling moves when
-someone raises it, which is a conversation with IntelliStream, not a retry.
+someone raises it, which is a conversation with whoever operates the deployment, not a retry.
Whether deleting helps depends on the metric:
@@ -205,7 +223,7 @@ Both endpoints, `/timeseries/datapoints/subscription/listen/**` and
| Cap | Default |
| --- | --- |
-| Concurrent connections per organisation | 10 |
+| Concurrent connections per organization | 10 |
| Concurrent connections per user | 10 |
| Subscriptions multiplexed over one socket | 10 |
@@ -244,12 +262,10 @@ So rate limits and daily quotas take care of themselves: the client backs off an
A `413` or a validation failure reaches your code, which is the right place for it, since
neither is fixed by trying again.
-A lifetime ceiling is the one `403` that does **not** spool. The others are worth spooling
-because an expired token or a missing grant is fixed out of band and the data then flushes; a
-ceiling never becomes acceptable by being replayed, so buffering it would fill the spool with
-data the server refuses every time and bury the one message that says the limit is raised by
-asking. The client tells them apart on the problem `type`, so the ceiling surfaces on the call
-that hit it, in `errors()` on the [`IngestResult`](./timeseries#ingestresult).
+A lifetime ceiling is the one `403` that does **not** spool: it surfaces on the call that hit
+it, in `errors()` on the [`IngestResult`](./timeseries#ingestresult). Why the client treats it
+differently from the other `403`s is under
+[durable ingest buffering](./client#durable-ingest-buffering).
Two things to check in your own configuration:
diff --git a/docs/reference/resources.md b/docs/reference/resources.md
index ee20fbc..b20ca1f 100644
--- a/docs/reference/resources.md
+++ b/docs/reference/resources.md
@@ -11,7 +11,7 @@ Hierarchical, asset-like entities and the relationships between them. Create res
and the edges between them in one call; the server returns the persisted graph.
A resource's `externalId` is its **identity**: unique per tenant, stored exactly as you send
-it, and compared without case. Mirror the tag your operation already maintains —
+it, and compared without case. Mirror the tag your operation already maintains,
`COM-99-PT-1034` is stored as `COM-99-PT-1034`, not rewritten.
[External ids & naming →](./external-ids)
@@ -19,7 +19,7 @@ it, and compared without case. Mirror the tag your operation already maintains
| Field | Type | Notes |
| --- | --- | --- |
-| `id` | number | Server-assigned. Crosses the wire as a JSON string — see the note below. |
+| `id` | number | Server-assigned. Crosses the wire as a JSON string, see the note below. |
| `externalId` | string, 3–256 | **Required.** Unique per tenant, stored verbatim, matched case-insensitively. |
| `name` | string, 3–512 | **Required.** What a human calls it. This is the field [search](#search) reads. |
| `labels` | string[] | **Required, at least one.** The type tags (`Pump`, `Plant`). Upper-cased by the server. |
@@ -28,14 +28,14 @@ it, and compared without case. Mirror the tag your operation already maintains
| `source` | string, 2–128 | The upstream system of record this came from (`SAP`, a historian, a file drop). |
| `dataSetId` | number | The data set the resource belongs to. |
| `geoLocation` | GeoJSON geometry | `Point`, `Polygon`, … Validated on write; stored verbatim. Returned only on [assets](#typed-reads). |
-| `isRoot` | boolean | Whether the resource is a navigation root. Deletes are checked against reachability from a root — see [Delete](#delete). Returned only on resources and assets. |
-| `relatedResources` | object[] | Read-only view of the graph: `{ id, externalId, relationshipType, direction }` per connected node. Populated where the graph is loaded, empty otherwise. |
+| `isRoot` | boolean | Whether the resource is a navigation root. Deletes are checked against reachability from a root, see [Delete](#delete). Returned only on resources and assets. |
+| `relatedResources` | object[] | Read-only view of the graph: `{ id, externalId, relationshipType, direction, edgeId }` per connected node. Populated on the create echo and on `fetch-related` / `fetch-nearest`; empty on `/resources/{id}`, `byids`, `filter` and `search`. |
| `createdTime`, `lastUpdatedTime` | epoch millis | Server-set. A create body may carry them, but they are ignored: the stored values are the server's. |
Labels are how the platform types a node. The type-label (`ASSET`, `TIMESERIES`, `DATASET`,
`POLICY`, `FUNCTION`) is what the create pipeline reads to decide which kind of entity to
build, and free-form labels ride alongside it. That is also why one `/resources/create` call
-can hold a mix of node types — a time-series next to an asset — rather than needing one
+can hold a mix of node types, a time series next to an asset, rather than needing one
endpoint per type. The same label types what a read returns: see
[Reads come back typed](#typed-reads).
@@ -47,7 +47,7 @@ through. Without it the call is a `403`, even when you can write the data set na
edges onto a data set node. See [Access control](./datasets#access-control).
:::note Numeric ids cross the wire as JSON strings
-`id` and `dataSetId` serialize as `"5677892"`, not `5677892` — ids can exceed the 53-bit
+`id` and `dataSetId` serialize as `"5677892"`, not `5677892`, ids can exceed the 53-bit
integer a JSON number is safe for in JavaScript. The clients parse them back for you. The same
holds for the ids on an [edge](./edges#body), `start` and `end` included.
:::
@@ -57,12 +57,12 @@ holds for the ids on an [edge](./edges#body), `start` and `end` included.
The read endpoints (`/resources/{id}`, `byids`, `filter`, `search`, `fetch-related`,
`fetch-nearest`) return each node in the shape of its kind, and the type-label inside
`labels` is the discriminator. There is deliberately no separate type property on the wire:
-an element whose labels contain `TIMESERIES` *is* the time-series shape.
+an element whose labels contain `TIMESERIES` *is* the time series shape.
| Type-label present | Shape returned |
| --- | --- |
| `ASSET` | An asset: the body above, `geoLocation` included. |
-| `TIMESERIES` | A [time-series](./timeseries): `unit`, `unitExternalId`, `valueType`. |
+| `TIMESERIES` | A [time series](./timeseries): `unit`, `unitExternalId`, `valueType`. |
| `DATASET` | A [data set](./datasets). |
| `POLICY` | A policy: `type`, `value`, `deactivated`, `templateId`. |
| `FUNCTION` | A function. |
@@ -70,7 +70,7 @@ an element whose labels contain `TIMESERIES` *is* the time-series shape.
Three rules govern which fields appear where:
-- A time-series carries its **full label set**, not only `["TIMESERIES"]`.
+- A time series carries its **full label set**, not only `["TIMESERIES"]`.
- `isRoot` belongs to resources and assets; `geoLocation` belongs to assets. A flat resource
body naming a `geoLocation` is a `400`: a plain resource has nowhere to store one, so it is
refused rather than accepted and dropped. Send an `ASSET`-labelled body instead.
@@ -95,8 +95,8 @@ for (NodeModel node : client.resources().filter(retriever).getItems()) {
Each item is the same class the type's own endpoint returns, so `isinstance` works and a
-time-series from `resources.filter()` behaves exactly like one from `timeseries.by_ids()`.
-Two new classes join the set: `Asset` and `Policy`.
+time series from `resources.filter()` behaves exactly like one from `timeseries.by_ids()`.
+`Asset` and `Policy` are in the set.
```python
from intellistream_datahub_sdk import TimeSeries
@@ -190,8 +190,8 @@ let resources = api.resources.by_ids(&vec to change an existing resource rather than re-creating it.
decided before either check, so a caller who may not write the data set is told that (`403`)
instead of being handed a `400` about an id they were never allowed to name.
-:::note Edges into datasets and time-series are validated
-Two endpoint rules apply to every edge, on create and on update (an update can retarget an
-edge or change its type):
+:::note Relationships into data sets and time series are validated
+Two endpoint rules apply to every relationship, on create and on update (an update can
+retarget a relationship or change its type):
-- A relation **to a dataset** must use the `BELONGS_TO` relationship type — that is the
- relation the dataset hierarchy and membership are built from, and anything else is
+- A relationship **to a data set** must use the `BELONGS_TO` relationship type, that is the
+ relationship the data set hierarchy and membership are built from, and anything else is
rejected with a `400`.
-- A **dataset → time-series** edge is accepted only when the series has no dataset yet, or
- already belongs to that very dataset (creating a series inside a dataset produces exactly
- that membership edge). A series in a *different* dataset is rejected with a `400` — a
- time-series has one dataset.
+- A **data set → time series** relationship is accepted only when the series has no data set
+ yet, or already belongs to that very data set (creating a series inside a data set produces
+ exactly that membership relationship). A series in a *different* data set is rejected with a
+ `400`, a time series has one data set.
:::
@@ -312,7 +312,7 @@ let created = api.resources.create(vec![plant, pump], vec![contains]).await?;
-An edge comes back as a `Relation` — `{ id, start, end, type, description, metadata }`,
+An edge comes back as a `Relation`, `{ id, start, end, type, description, metadata }`,
where `start` and `end` are the ids of the two nodes (as JSON strings, like every other id).
That is why you send `fromExternalId`/`toExternalId` but read `start`/`end`: the write side
speaks in your identifiers, the read side in the graph's.
@@ -325,11 +325,11 @@ describe different graphs.
There are two ways to create a relation and they produce the same edge. The call above sends
nodes and relations together, in one transaction. `POST /edges/create` sends the relations by
-themselves, for when both ends already exist and repeating them would be noise — same fields,
+themselves, for when both ends already exist and repeating them would be noise, same fields,
same rules, same edges back.
-That endpoint, and the rest of the `/edges` surface (reading an edge back, deleting one
-without touching its endpoints, the relationship-type catalog), has its own page.
+That endpoint, and the rest of the `/edges` surface (reading a relationship back, deleting one
+without touching its endpoints, the relationship-type catalogue), has its own page.
[Edges →](./edges)
To disconnect two resources without touching either of them, [delete the edge](./edges#delete).
@@ -374,7 +374,7 @@ plural names because adding an entry there narrows the result where adding a `na
`limit` defaults to **1 000** and is capped at **10 000**; a zero, negative or null value
falls back to the default rather than returning nothing. Results come newest created first
-unless ordered otherwise, and page with a cursor — the same contract as
+unless ordered otherwise, and page with a cursor, the same contract as
[timeseries](./timeseries#sorting-and-paging), over the same sortable properties.
:::caution A pattern-less value matches exactly, not as a substring
@@ -452,7 +452,7 @@ database scores and sorts every match before applying `limit`, so a very broad p
than a narrow one.
`limit` is capped at **1 000** here, lower than the 10 000 of `filter`, and `query` must be
-3 to 140 characters.
+3 to 140 characters. `limit` applies across all node types, and `POLICY` nodes are searched.
### Narrowing with `filter` {#search-filter}
@@ -468,20 +468,6 @@ search query itself, everything else is applied to the hits afterwards.
}
```
-:::note What changed
-`filter` used to be accepted and silently ignored here, as it was on the data set and event
-searches. All four searches now apply it.
-
-The phrase and the filter are now one query, so the database plans them together. They were briefly
-two, with the phrase capped at a 10 000-row candidate set that the filter then narrowed, which
-quietly dropped matches past that cap.
-
-Two other things moved with this. The search originally ran one query per node type and concatenated
-the results, so `limit` applied per type (a request for 50 could return 250) and results came back
-grouped by type. Policies were never searched at all, and now are, so a search with no `nodeType`
-can return rows it did not before.
-:::
-
@@ -517,14 +503,14 @@ let matches = api.resources.search(&form).await?;
-Reach for `filter` instead whenever the question is structured — an exact external id, a
+Reach for `filter` instead whenever the question is structured, an exact external id, a
metadata value, a data set, a time range. It is faster and its results are predictable.
## Update {#update}
`POST /resources/update` changes fields on resources and relations that already exist.
Identify each node by `id` or `externalId`, each relation by `id`, and name only what you
-want changed — anything you leave out keeps its current value.
+want changed, anything you leave out keeps its current value.
Each field is an object carrying a verb rather than a bare value, which is what lets "clear
this" be said distinctly from "leave it alone":
@@ -555,14 +541,13 @@ this" be said distinctly from "leave it alone":
Updatable node fields are `externalId`, `name`, `description`, `source`, `dataSetId`,
`metadata`, `labels` and `geoLocation`. On a relation they are `start`, `end`,
`fromExternalId`, `toExternalId`, `relationship`, `relationshipId`, `description` and
-`metadata` — so an edge can be retargeted or retyped in place, subject to the same
+`metadata`, so an edge can be retargeted or retyped in place, subject to the same
[endpoint rules](#create-resources-and-relations) as a create.
Sending both `set` and `setNull` for one field is a `400`: the request is contradictory, so
it is refused rather than resolved by precedence. `setNull` against `name` or `externalId`
is also a `400`, for the same reason a create cannot omit them: every resource has to have
-both. Rename with `set` instead. (This used to return `200` and quietly change nothing, so
-check the value rather than the status if you are working against an older deployment.)
+both. Rename with `set` instead.
Changing `externalId` runs it past the
[naming policy](./external-ids#the-naming-policy), which reports violations per item in an
RFC 9457 problem response. The whole batch is **all-or-nothing**.
@@ -570,7 +555,7 @@ RFC 9457 problem response. The whole batch is **all-or-nothing**.
:::caution A `409` means someone else got there first
Updates are guarded by optimistic locking. If another request changed or deleted the
resource while yours was in flight, you get a `409` with `"cause": "concurrency"` and
-**nothing was written** — no partial application to unpick. Re-read the resource with
+**nothing was written**, no partial application to unpick. Re-read the resource with
`byIds` and retry the update against fresh state.
This is worth designing for rather than retrying blindly: two writers doing
@@ -585,7 +570,7 @@ per-entry update forms above.
## Delete {#delete}
Delete by id or external id; unknown identifiers are silently skipped. A successful delete
-returns `204` with no body, and deleting something already gone is a no-op — so a retried
+returns `204` with no body, and deleting something already gone is a no-op, so a retried
delete needs no bookkeeping.
Deleting a resource takes **all** of its relationships with it, inbound and outbound. That is
@@ -593,7 +578,7 @@ where the one real constraint comes from:
:::caution The graph must stay connected
A delete is rejected with `400` if it would leave any surviving resource unreachable from a
-root resource — that is, if it would strand part of the graph. The response names the
+root resource, that is, if it would strand part of the graph. The response names the
resources that would be stranded, so the fix is either to include them in the same delete or
to re-attach them through another path first.
@@ -602,7 +587,7 @@ holds twenty pumps takes the edges to those pumps with it, stranding all twenty.
is what stops a routine cleanup from quietly orphaning half a site.
:::
-A single safety-check failure rolls the whole batch back — nothing is deleted unless
+A single safety-check failure rolls the whole batch back, nothing is deleted unless
everything can be. As with update, a concurrent modification surfaces as a `409` with
nothing removed.
@@ -633,18 +618,18 @@ api.resources.delete(&vec![IdAndExtId::from_external_id("pump_1")]).await?;
## Traverse the graph
`fetchRelated` walks the graph outward from a starting resource and returns the
-connected sub-graph — a `ResourceNetwork` of `nodes`, the `edges` between them, and
+connected sub-graph, a `ResourceNetwork` of `nodes`, the `edges` between them, and
their `labels`. Traversal is **undirected** and bounded by `depth` (`-1` = the whole
connected component), optionally filtered to specific relationship types. Use it for
-relationship reasoning — root-cause correlation, blast radius — that a flat lookup
+relationship reasoning, root-cause correlation, blast radius, that a flat lookup
can't do. See [Correlate alarms with the graph](/guides/correlate-alarms).
| Field | Default | Meaning |
| --- | --- | --- |
-| `id` / `externalId` | — | Where to start. Supply exactly one. |
+| `id` / `externalId` | none | Where to start. Supply exactly one. |
| `depth` | `-1` | Hops to follow. `-1` loads the entire connected component. |
| `relationshipTypes` | all | Which edge types the walk may follow. |
-| `excludedLabels` | none | Labels the walk neither passes through nor returns — e.g. `["POLICY"]` to keep governance nodes out of an asset view. |
+| `excludedLabels` | none | Labels the walk neither passes through nor returns, e.g. `["POLICY"]` to keep governance nodes out of an asset view. |
| `limit` | `5000` | Safety cap on nodes loaded. When the component is bigger, the nearest `limit` nodes come back. |
That `limit` is the one to watch: it is a silent truncation, not an error. On a densely
@@ -652,13 +637,12 @@ connected site an unbounded `depth` will hit 5 000 nodes long before it runs out
and what you get back is a *neighbourhood*, not the component you asked for. Bound `depth`
to 1–3 unless you know the graph is sparse.
-Nodes from `fetchRelated` and `fetch-nearest` come back
-[typed by label](#typed-reads) but sparsely populated: the graph holds a subset of each
-node's columns, so a node from these endpoints is not the full record. Fetch by id when you
-need everything.
-
-A `TIMESERIES` node from these endpoints carries `unit`, `unitExternalId` and `valueType`.
-Every node carries its `metadata`.
+Nodes from `fetchRelated` and `fetch-nearest` come back [typed by label](#typed-reads) and
+carry the fields the graph mirror holds: `id`, `externalId`, `name`, `description`, `source`,
+`dataSetId`, `labels`, `metadata`, `createdTime` and `lastUpdatedTime`, plus `relatedResources`
+built from the edges of the network you fetched. By type: `isRoot` on resources and assets,
+`geoLocation` on assets, `unit`, `unitExternalId` and `valueType` on time series, and
+`isDeactivated` on policies. Fetch by id when you need a field outside that list.
@@ -712,15 +696,15 @@ for node in net.nodes() {
### The nearest N of a kind {#fetch-nearest}
`POST /resources/fetch-nearest` answers a question `fetchRelated` cannot: *the ten nearest
-time-series to this pump*. It walks breadth-first and caps on the number of **matching
-end-nodes**, not on hops or total nodes — so "the 10 nearest `TIMESERIES`" is exactly ten
+time series to this pump*. It walks breadth-first and caps on the number of **matching
+end-nodes**, not on hops or total nodes, so "the 10 nearest `TIMESERIES`" is exactly ten
however many intermediate nodes lie between them. You get those nodes plus the sub-graph
connecting them back to the start.
| Field | Default | Meaning |
| --- | --- | --- |
-| `id` | — | Where to start. **Numeric id only** — see below. |
-| `endLabels` | — | Labels that qualify as a match, e.g. `["TIMESERIES"]`. The walk continues past them. |
+| `id` / `externalId` | none | Where to start. Supply exactly one, as for `fetchRelated`. |
+| `endLabels` | none | Labels that qualify as a match, e.g. `["TIMESERIES"]`. The walk continues past them. |
| `limit` | `10` | How many matching end-nodes to return. |
| `relationshipTypes` | all | Which edge types the walk may follow. |
| `excludedLabels` | none | Labels never traversed or returned. |
@@ -729,18 +713,16 @@ That is the difference worth internalising: with `fetchRelated` you pick a radiu
out what is inside it, which on an unfamiliar graph is a guess. With `fetch-nearest` you name
what you are looking for and how many you want, and the radius follows.
-:::caution `externalId` is accepted but not read
-The request form carries an `externalId` field, but this endpoint starts from `id` only —
-sending an external id alone gets you a `404`. Resolve it to a numeric id with `byIds` first.
-`fetchRelated` takes either.
-:::
+The endpoint resolves an `externalId` for you, and so does the Java form. The Python and Rust
+clients build the request from a numeric `id` only, so there resolve an external id with
+`by_ids` first when that is all you have.
```java
FetchNearestResourcesForm form = new FetchNearestResourcesForm();
-form.setId(5677892L); // numeric id, not external id
+form.setExternalId("pump_1"); // or form.setId(5677892L)
form.setEndLabels(List.of("TIMESERIES"));
form.setLimit(10);
form.setExcludedLabels(List.of("POLICY"));
@@ -753,7 +735,7 @@ ResourceNetwork nearest = client.resources().fetchNearest(form);
```python
nearest = client.resources.fetch_nearest(
- 5677892, # numeric id, not external id
+ 5677892, # the Python client takes the numeric id
end_labels=["TIMESERIES"],
limit=10,
excluded_labels=["POLICY"])
@@ -766,7 +748,7 @@ nearest = client.resources.fetch_nearest(
use intellistream_datahub_sdk::resources::FetchNearestResourcesForm;
let nearest = api.resources.fetch_nearest(
- &FetchNearestResourcesForm::from_id(5677892) // numeric id, not external id
+ &FetchNearestResourcesForm::from_id(5677892) // the Rust client takes the numeric id
.with_end_labels(vec!["TIMESERIES".into()])
.with_limit(10)
.with_excluded_labels(vec!["POLICY".into()])).await?;
@@ -775,6 +757,116 @@ let nearest = api.resources.fetch_nearest(
+## Export and import a graph {#graph-transfer}
+
+Two endpoints move a whole connected component between tenants or environments as one file.
+Export starts from one resource, walks outward with no depth limit, and writes every reachable
+node and every relationship between them; import recreates them somewhere else. The file
+references everything by `externalId` and never by numeric id, which is what makes it
+portable: numeric ids are database identities and do not survive the transfer.
+
+| Endpoint | Body | Returns |
+| --- | --- | --- |
+| `GET /resources/export/{id}` | none; `id` is the numeric id of the resource to start from | The file, `application/octet-stream`, as an attachment named `.dhgraph` |
+| `POST /resources/import` | the file, verbatim, as `application/octet-stream` | A JSON summary of what was created and what was skipped |
+
+The file is a gzip-compressed binary, streamed on the way out and decoded incrementally on the
+way in, so neither side holds it whole in memory. Treat it as opaque: the format is versioned
+by the server and is not part of the API contract.
+
+**What the file carries.** Per node: `externalId`, `name`, `description`, `source`, `isRoot`,
+`labels`, `metadata`, the data set it belongs to (by that data set's `externalId`) and, for an
+`ASSET`, a point `geoLocation`; other geometries are not carried. Per relationship: both
+endpoints by `externalId`, the type, `description`, `metadata` and the data set. A data set
+node inside the component is exported as a node like any other, ahead of everything that
+references it.
+
+**What it does not carry.** A time-series's `unit` and `valueType`, so a time-series node in
+the file cannot be created on import and is reported instead; datapoints; events; files.
+
+### Export {#graph-export}
+
+Export needs read access to the data set of the starting resource, and nothing more: the walk
+is [gated on the starting node only](./datasets#access-control), so the file holds every node
+the component reaches. A component of more than **2 000 000 nodes** or **2 000 000
+relationships** is refused with a `400` naming the limit, and nothing is exported partially.
+
+| Status | Meaning |
+| --- | --- |
+| `200` | The file. |
+| `404` | No such resource, or the caller may not read it. |
+| `400` | The component is over the export limit. |
+
+### Import {#graph-import}
+
+Import replays the file through the same pipeline as [create](#create-resources-and-relations),
+so everything a create does, an import does: the [naming policy](./external-ids#the-naming-policy)
+is applied, the data set ACLs are checked, and each committed segment is published to the
+message bus and mirrored into the graph. The caller needs write access to every data set a
+node or a relationship lands in, and the all-data-sets grant for a node that arrives with no
+data set and for any `DATASET` node in the file. A denial is a `403`.
+
+**Skipped, not refused.** A node whose `externalId` already exists in the tenant is left as it
+is, and so is a relationship already present between the same two endpoints with the same
+type. Importing a file back into the tenant it came from is therefore a no-op, and
+re-uploading after a failure is safe. Time-series nodes are skipped and listed by
+`externalId`, and the relationships touching them are skipped with them; to keep those, create
+the series through [`/timeseries`](./timeseries#create-a-series) first and import the same file
+again. A data set reference is resolved by `externalId` against the data sets in the file and
+those already in the tenant; one that resolves nowhere is dropped, and the node is created
+without a data set.
+
+**Segments.** The upload is committed as it streams in, one transaction per 50 000 objects,
+nodes first and relationships after, so memory stays flat however large the file. Each segment
+is atomic on its own: a failure keeps the segments already committed and rejects the rest.
+Because import skips what already exists, re-upload the same file once the cause is fixed and
+it fast-forwards through the committed segments and resumes where it stopped. The response
+counts the segments committed.
+
+```json
+{
+ "nodesCreated": 4210,
+ "relationsCreated": 4209,
+ "nodesSkippedExisting": 3,
+ "nodesSkippedTimeseries": ["pump_1_vibration", "pump_1_temperature"],
+ "relationsSkipped": 2,
+ "dataSetReferencesDropped": 0,
+ "segments": 1,
+ "warnings": []
+}
+```
+
+`warnings` carries [naming-policy warnings](./external-ids#the-naming-policy) exactly as a
+create does, and a naming-policy refusal is the same `400` problem body a create returns.
+
+| Status | Meaning |
+| --- | --- |
+| `200` | The summary above, also when everything was skipped. |
+| `400` | Not a readable graph file, a naming-policy refusal, or a value that failed validation. |
+| `403` | A data set the caller may not write to. |
+| `413` | Over a transfer limit: more than 2 000 000 nodes or relationships in the file, or a file larger than 512 MB. Nothing is imported. |
+
+:::caution The general request-body cap applies first
+`POST /resources/import` is not exempt from the [request body size](./limits#request-body-size)
+cap, which is 4 MiB unless the deployment raises `datahub.limits.max-body-bytes`. A larger
+file is refused with that cap's own `413` (`.../errors/request-too-large`) before the import
+reads it, so the 512 MB figure above is the format's ceiling, not what a default deployment
+accepts. The upload's size also counts against the tenant's daily ingest byte quota.
+:::
+
+No client wraps the pair. Call them over HTTP with the bearer token the client already holds:
+
+```bash
+curl -fsS -H "Authorization: Bearer $TOKEN" \
+ -o plant_oslo.dhgraph "$API/resources/export/5677892"
+
+curl -fsS -H "Authorization: Bearer $TOKEN" \
+ -H "Content-Type: application/octet-stream" \
+ --data-binary @plant_oslo.dhgraph "$API/resources/import"
+```
+
+Against the [rate-limit](./limits#rate-limits) budget, export is a read and import a write.
+
## The `/assets` endpoints {#assets}
An **asset** is the node type that can be a navigation root and the only one that carries a
@@ -790,7 +882,7 @@ several node types.
| `GET /assets/{id}` | [look up](#look-up) | One asset, wrapped in `items` like every other read. |
| `POST /assets/byids` | [look up](#look-up) | Ids that are missing, are not assets, or are not readable are omitted rather than failing the call. |
| `POST /assets/filter` | [filter](#filter) | The same criteria, the same paging. A `nodeType` in the body is replaced, see below. |
-| `POST /assets/search` | [search](#search) | Same replacement, and the `filter` block is still [accepted and ignored](#search). |
+| `POST /assets/search` | [search](#search) | Same replacement, and the `filter` block is applied exactly as on [`/resources/search`](#search-filter). |
| `POST /assets/update` | [update](#update) | Takes `nodes` and `relations` exactly as `/resources/update` does. |
| `POST` or `DELETE /assets/delete` | [delete](#delete) | `204`, and the same [connectivity check](#delete). |
@@ -829,9 +921,8 @@ resource. Its family is `POST /functions/create`, `GET /functions/list`, `GET /f
`POST /functions/update` and `POST` or `DELETE /functions/delete`, on the same shared pipeline.
`GET /functions/list` takes no filter: the inventory is expected to be small.
-`GET /functions/{id}` is new, and completes the read surface: it returns the one function
-wrapped in `items`, and reports a function you may not read as missing (`404`) rather than
-forbidden, exactly as `GET /assets/{id}` does.
+`GET /functions/{id}` returns the one function wrapped in `items`, and reports a function
+you may not read as missing (`404`) rather than forbidden, exactly as `GET /assets/{id}` does.
The Java client has no `assets()` or `functions()` service, so reach for the endpoints there.
Creating an asset through `resources().create` with an `ASSET` label is the same pipeline and
@@ -850,6 +941,7 @@ gives you the same asset back.
| Filter | `resources().filter` | `resources.filter` | `resources.filter` |
| Traverse (`fetch-related`) | `resources().fetchRelated` | `resources.fetch_related` | `resources.fetch_related` |
| Nearest N (`fetch-nearest`) | `resources().fetchNearest` | `resources.fetch_nearest` | `resources.fetch_nearest` |
+| [Export / import a graph](#graph-transfer) | HTTP only | HTTP only | HTTP only |
-Relations have their own client surface in all three clients — `edges()` in Java, `edges` in
+Relations have their own client surface in all three clients, `edges()` in Java, `edges` in
Python and Rust. [Edges → client coverage](./edges#client-coverage)
diff --git a/docs/reference/subscriptions.md b/docs/reference/subscriptions.md
index ecd171f..10c1971 100644
--- a/docs/reference/subscriptions.md
+++ b/docs/reference/subscriptions.md
@@ -7,7 +7,7 @@ import TabItem from '@theme/TabItem';
# Subscriptions
-Durable, fan-out subscriptions over time-series, plus **live delivery over a WebSocket**.
+Durable, fan-out subscriptions over time series, plus **live delivery over a WebSocket**.
## Manage subscriptions
@@ -26,6 +26,10 @@ DataWrapper all = client.subscriptions().list(new SubscriptionRetr
client.subscriptions().delete(List.of(IdCollection.createFromExternalId("engine_temps")));
```
+`SubscriptionRetriever` takes a `filter` whose one criterion is `timeseries` (only
+subscriptions bound to these series, by id or external id), a `limit` (default 100, at most
+10 000), a `sort`, and `includeSystemManaged` (default `false`).
+
@@ -63,19 +67,21 @@ api.subscriptions.delete(&vec![IdAndExtId::from_external_id("engine_temps")]).aw
-:::note Dataset access control
-Creating a subscription requires **read access to every timeseries' dataset** it binds. If you
+:::note Data set access control
+Creating a subscription requires **read access to every bound series' data set**. If you
lack read access to any of them, `create` fails with **HTTP 403** and nothing is persisted.
Access is granted through Keycloak **organization groups**: `/datasets//read` for one
data set (and everything beneath it), or the wildcard `/datasets/*/read` for all of them.
-[Dataset access control →](./datasets#access-control)
+[Data set access control →](./datasets#access-control)
:::
## Live delivery
-`listen` opens an authenticated WebSocket over one or more subscriptions. Stream messages
-to a handler or drive a loop, and **ack** the messages you've processed — anything left
-unacked is redelivered on reconnect.
+`listen` opens a WebSocket to `/timeseries/datapoints/subscription/listen//...`,
+one path segment per subscription, and authenticates the upgrade request with the same
+`Authorization: Bearer ` header as any REST call. Stream messages to a handler or drive
+a loop, and **ack** the messages you've processed, anything left unacked is redelivered on
+reconnect.
@@ -92,7 +98,7 @@ try (var stream = client.subscriptions().listen(List.of("engine_temps"))
}
```
-Or drive `poll` yourself — a blocking queue hand-off (not network polling) that returns
+Or drive `poll` yourself, a blocking queue hand-off (not network polling) that returns
`null` on timeout. Reach for `poll`, or `stream(handler, AckMode.MANUAL)`, when you need
to ack on your own schedule:
@@ -148,22 +154,32 @@ Every listener also exposes `stream` for push delivery, `ack`/`nack`,
`subscribe`/`unsubscribe`/`set_subscriptions` to change the live interest set at runtime,
and `close`.
-A delivered message carries the originating subscription's external id, an opaque
-`messageId` you echo back to `ack`/`nack`, and a `payload` describing the fan-out event
-(an action — create/update/delete — plus the affected datapoints).
+A frame on the wire carries one subscription's messages:
+
+| Field | Type | |
+| --- | --- | --- |
+| `subscriptionExternalId` | string | The subscription the batch came from. |
+| `messages[].messageId` | string | Opaque. Echo it back in an ack or nack. |
+| `messages[].payload.eventAction` | `CREATE`, `UPDATE`, `DELETE` or `RENAME` | What happened. |
+| `messages[].payload.eventObject` | `DATAPOINTS` | What it happened to. |
+| `messages[].payload.items[]` | object[] | One entry per series: `id` (a JSON string), `externalId`, `valueType`, `datapoints[]`, and optionally `inclusiveBegin` and `exclusiveEnd`. |
+| `messages[].payload.items[].datapoints[]` | `{ timestamp, value }` | `timestamp` is an ISO-8601 UTC string (`2026-08-30T22:00:00Z`); `value` is a string. |
+
+Ack and nack are `{"action": "ack", "messageIds": [...]}` and the same with `"nack"`. The
+clients unpack each entry of `messages` into one message, whose `payload` is the object above.
:::note Refused subscriptions surface as errors
-Live delivery enforces the same dataset ACL: to attach a subscription you must be able to read
-**all** of its bound timeseries. A subscription you can't read (`reason: "forbidden"`) or one that
-doesn't exist (`reason: "not-found"`) is refused per-subscription — the connection stays open for
+Live delivery enforces the same data set ACL: to attach a subscription you must be able to read
+**all** of its bound series. A subscription you can't read (`reason: "forbidden"`) or one that
+doesn't exist (`reason: "not-found"`) is refused per-subscription, the connection stays open for
the subscriptions that did attach. The refusal is surfaced, not swallowed: a `SubscriptionError`
via `pollError` in Java, an `Err(ListenError::Subscription { .. })` from `next().await` in Rust, and
-an exception raised from the iterator in Python — so a refused subscription is visible instead of
-looking like an indefinitely silent stream.
+an exception raised from the iterator in Python. A refused subscription is therefore visible
+instead of looking like an indefinitely silent stream.
:::
:::note Sockets and subscriptions are capped
-Ten concurrent connections per organisation, ten per user, and ten subscriptions multiplexed
+Ten concurrent connections per organization, ten per user, and ten subscriptions multiplexed
over one socket, by default. The two refusals behave differently, on purpose:
| Over the cap on | The server | The socket |
@@ -178,7 +194,7 @@ are in [Limits & quotas](./limits#websockets).
:::tip Acking is at-least-once
Ack a message only after you've durably handled it. If your process dies before the ack,
-the server redelivers it — so make your handler idempotent.
+the server redelivers it, so make your handler idempotent.
:::
## What each client covers {#client-coverage}
@@ -189,5 +205,3 @@ the server redelivers it — so make your handler idempotent.
| List | `subscriptions().list` | `subscriptions.list` | `subscriptions.list` |
| Delete | `subscriptions().delete` | `subscriptions.delete` | `subscriptions.delete` |
| Live delivery | `subscriptions().listen` | `subscriptions.listen` | `subscriptions.listen` |
-
-Full parity — subscriptions are the one area where all three clients cover the same ground.
diff --git a/docs/reference/timeseries.md b/docs/reference/timeseries.md
index f3c78ca..3e9d658 100644
--- a/docs/reference/timeseries.md
+++ b/docs/reference/timeseries.md
@@ -1,13 +1,13 @@
---
sidebar_position: 4
-title: Time-series
+title: Time series
---
import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';
-# Time-series
+# Time series
-Time-series metadata, datapoint retrieval, and datapoint ingestion (single-request or
+Time series metadata, datapoint retrieval, and datapoint ingestion (single-request or
high-throughput).
A series' `externalId` identifies it: unique per tenant, compared without case, and stored
@@ -58,21 +58,26 @@ api.time_series.create_one(&ts).await?;
## Value types
Every series has a **value type** that decides how its datapoints are stored. Leave it
-unset and the series is floating-point (`float32`) — right for most sensor readings, so
+unset and the series is floating-point (`float32`), right for most sensor readings, so
the create above accepts decimal values as-is. Set it explicitly when you need something
else:
| Value type | Use it for |
| --- | --- |
-| `float32` *(default)* | Sensor readings — 32-bit precision is plenty. |
+| `float32` *(default)* | Sensor readings, 32-bit precision is plenty. |
| `float` | Double-precision floating point. |
-| `numeric` / `decimal32` | **Exact decimals** — money, lab values — stored without floating-point rounding. Pass the values as strings. |
+| `numeric` / `decimal32` | **Exact decimals**, money, lab values, stored without floating-point rounding. Send the value's string form, see below. |
| `bigint` | Whole numbers (counts, integer statuses). |
| `text` | Non-numeric string values. |
| `mixed` | Heterogeneous values in one series. |
A float written to a `bigint` series is rejected, so pick the type that matches the data.
+Every value crosses the wire as a string, whatever the type. For an exact decimal, send the
+string form rather than a float: `Datapoint.of(ts, "12.34")` in Java, `DatapointString(ts,
+"12.34")` with `insert_datapoints` in Python (`insert_from_lists` takes floats and converts
+them), and `DatapointString` in Rust.
+
`text` and `mixed` also carry a tighter write cap than the numeric types: **10 000 datapoints
per collection** rather than 100 000, and a lifetime ceiling of their own. The check runs once
the series' value type is resolved, so it names the series type rather than a field. See
@@ -118,7 +123,7 @@ api.time_series.create_one(&price).await?;
## Filter series
`POST /timeseries/filter` finds series by structured criteria. Everything you supply is
-combined with AND — a series must match every criterion to be included.
+combined with AND, a series must match every criterion to be included.
| Criterion | Matching |
| --- | --- |
@@ -137,18 +142,12 @@ entries of an array are combined with **OR**. That is why they are named in the
`labels` and `metadata` require **all** entries to match and keep their plural names for that
reason.
-Results come newest first unless you ask for another order — see
-[sorting and paging](#sorting-and-paging) — capped by `limit` (default 1000, max 10000; a value
+Results come newest first unless you ask for another order, see
+[sorting and paging](#sorting-and-paging): capped by `limit` (default 1000, max 10000; a value
`<= 0` falls back to the default, and above the ceiling is a 400). Series in data sets you lack
-read access to are silently omitted — the result is what your token may see, not an error. For
+read access to are silently omitted, the result is what your token may see, not an error. For
free-text lookups use `POST /timeseries/search` instead.
-:::note The `metadataKey` / `metadataValue` pair is gone
-It existed only because `metadata` could not express "has this key, whatever its value". A null
-value in the map says that now, and `{"health": "good", "tier": null}` asks for both conditions at
-once.
-:::
-
@@ -222,28 +221,18 @@ search query itself, everything else is applied to the hits afterwards.
}
```
-:::warning `search.name` and `search.description` are gone
-The phrase block is now a single `query`, the same shape the other three searches take. The two
-alternatives it used to carry were removed rather than kept: `name` matched by **exact equality**
-under an endpoint documented as full-text, and `description` ran a differently configured query
-over one column.
-
-Both have a better replacement. `filter.name` matches names as a case-insensitive pattern list
-(`["pump_*", "PMP-1"]`), which is more than `search.name` could do, and `query` already covers the
-description column.
-
-Clients exposing these as separate calls (`search_by_name`, `search_by_description`) need updating
-to match.
+:::caution `search.name` and `search.description` are rejected
+They are unknown fields, refused with a `400`; use `filter.name` and `query`.
:::
## Sorting and paging {#sorting-and-paging}
-The three node filters — `/timeseries/filter`, `/resources/filter` and `/datasets/filter` —
+The three node filters, `/timeseries/filter`, `/resources/filter` and `/datasets/filter`,
share this contract. (`/events/filter` works the same way over its own columns; see
[events](./events#paging).)
Order a page with `sort`, over `id`, `externalId`, `name`, `source`, `description`,
-`createdTime`, `lastUpdatedTime` or `dataSetId`. The default is `createdTime` descending —
+`createdTime`, `lastUpdatedTime` or `dataSetId`. The default is `createdTime` descending,
newest created first.
```json
@@ -256,7 +245,7 @@ Only the **first** `property` is used, and `id` is appended behind it: a sort co
a position unless it is unique, and a page boundary inside a run of equal values repeats or drops
exactly those rows. An unrecognised property falls back to the default rather than being
rejected, and any `order` that is not exactly `desc` sorts ascending. Nulls sort last ascending,
-first descending — most of these columns are nullable, since every node type shares one table.
+first descending, most of these columns are nullable, since every node type shares one table.
A page that has a successor carries a `nextCursor`. Echo it back as `cursor` to continue:
@@ -267,10 +256,10 @@ A page that has a successor carries a `nextCursor`. Echo it back as `cursor` to
"limit": 100 }
```
-The cursor is **opaque** — base64 of a versioned encoding carrying the sort, the boundary value
-and the id — so do not build or parse one. Send it with the **same** sort that produced it; a
+The cursor is **opaque**, base64 of a versioned encoding carrying the sort, the boundary value
+and the id, so do not build or parse one. Send it with the **same** sort that produced it; a
cursor is a position in one particular order, and continuing it under another is refused. One
-that does not decode restarts the walk from the first page rather than failing.
+that does not decode is refused with a `400` of `type: ".../errors/malformed-cursor"`.
`nextCursor` is absent on a short page, so "keep going while it is present" is the whole loop. A
full page may still be the last, so a complete walk ends with one empty request.
@@ -305,7 +294,8 @@ while True:
break
```
-`filter()` returns a `Page` — a list, so existing code is unaffected, carrying `.next_cursor`.
+`filter()` returns a `Page`: iterable, indexable and sized like a list, carrying `.next_cursor`
+and `.items` (a real `list`). It is not a `list` subclass, so `isinstance(page, list)` is `False`.
@@ -462,7 +452,7 @@ client.timeseries.insert_from_lists(
-`insert_datapoints` auto-batches large inputs (chunks above ~100k points):
+`insert_datapoints` auto-batches large inputs (chunks at the 100 000-point collection cap):
```rust
use intellistream_datahub_sdk::generic::{DataWrapper, DatapointsCollection, DatapointString};
@@ -481,7 +471,22 @@ api.time_series.insert_datapoints(&mut dw).await?;
## Retrieve datapoints
-Identify a series (external id or id) and a time window.
+`POST /timeseries/data/list` takes a list of `RetrieveFilter` items, one per series and
+window:
+
+| Field | Meaning |
+| --- | --- |
+| `id` / `externalId` | The series. |
+| `start`, `end` | ISO-8601 or epoch millis. At least one is required. |
+| `limit` | Datapoints per page, default 100, at most 100 000. |
+| `aggregates` | Any of `avg`, `sum`, `min`, `max`, lower-case. A name outside that set is dropped, not rejected. `avg` comes back as `average`. |
+| `granularity` | A number and a unit: `s`, `m`, `h`, `d`, `w`, `mo`, `y`, or the words `sec`, `min`, `hour`, `day`, `week`, `month`, `year` and their plurals (`15m`, `1h`, `30 min`). Bare `m` is a minute; a month is `mo`. Required when `aggregates` is set. |
+| `includeOutsidePoints`, `mergeDuplicates` | Booleans, default `false`. |
+| `cursor` | The previous page's `nextCursor`, carried on each returned collection. |
+
+A datapoint comes back with an ISO-8601 UTC `timestamp` and a `value`. Python reads the
+timestamp as a timezone-aware `datetime.datetime` and the value as `float | None`; Rust as
+`chrono::DateTime` and `Option`; Java keeps both as strings on `DatapointString`.
@@ -671,9 +676,8 @@ if (!result.isComplete()) {
| Update | HTTP | `timeseries.update` | `time_series.update` |
| Delete | `timeseries().delete` | `timeseries.delete` | `time_series.delete` |
| Write datapoints | `insertDatapoints` / `ingest` | `insert_datapoints` / `insert_from_lists` | `insert_datapoint` / `insert_datapoints` |
-| Read datapoints | `retrieve` / `retrieveAggregated` | `retrieve_datapoints` / `retrieve_latest_datapoints` | `retrieve_datapoints` / `retrieve_latest_datapoint` |
+| Read datapoints (raw and [aggregated](#retrieve-datapoints)) | `retrieve` / `retrieveAggregated` | `retrieve_datapoints` / `retrieve_latest_datapoints` | `retrieve_datapoints` / `retrieve_latest_datapoint` |
| Delete datapoints | `deleteDatapoints` | `timeseries.delete_datapoints` | `time_series.delete_datapoints` |
Java is the one with `ingest`, the chunking, parallelising, retrying path described above.
-It is missing `list` and `update`, so reach for the endpoint there. It gained `search` alongside
-the resource, data set and event searches it already had.
+It is missing `list` and `update`, so reach for the endpoint there.
diff --git a/docs/reference/units.md b/docs/reference/units.md
index ec1f711..947646b 100644
--- a/docs/reference/units.md
+++ b/docs/reference/units.md
@@ -7,7 +7,23 @@ import TabItem from '@theme/TabItem';
# Units
-Units of measure (read-only reference data).
+Units of measure (read-only reference data). The endpoints are `GET /units`,
+`GET /units/{externalId}` and `POST /units/byids`.
+
+## The unit object {#body}
+
+| Field | Type | Notes |
+| --- | --- | --- |
+| `id` | number | Crosses the wire as a JSON string, like every other id. |
+| `externalId` | string, 3–256 | The catalogue key, `_` in snake_case: `temperature_deg_c`, `pressure_bar`, `mass_flow_rate_kghr`. This is what a series' `unitExternalId` names. |
+| `name` | string, 1–64 | Short code (`DEG_C`). |
+| `longName` | string | `degree Celsius`. |
+| `symbol` | string | `°C`. |
+| `description` | string | Prose. |
+| `aliasNames` | string[] | Other spellings (`C`, `degC`). |
+| `quantity` | string | What it measures (`Temperature`). |
+| `conversion` | `{ multiplier, offset }` | To the quantity's base unit. |
+| `source`, `sourceReference` | string | Where the definition comes from (`qudt.org` and its URL). |
## List all units
@@ -61,7 +77,7 @@ DataWrapper result = client.units().byIds(List.of(lookup));
```python
import intellistream_datahub_sdk
-by_ext = client.units.by_external_ids("celsius")
+by_ext = client.units.by_external_ids("temperature_deg_c")
by_id = client.units.by_ids([intellistream_datahub_sdk.IdCollection(id=7)])
```
@@ -71,7 +87,7 @@ by_id = client.units.by_ids([intellistream_datahub_sdk.IdCollection(id=7)])
```rust
use intellistream_datahub_sdk::generic::{DataWrapper, IdAndExtId};
-let by_ext = api.units.by_external_id("celsius").await?;
+let by_ext = api.units.by_external_id("temperature_deg_c").await?;
let by_id = api.units.by_ids(&DataWrapper::from(vec![IdAndExtId::from_id(7)])).await?;
```
diff --git a/docs/tutorial.mdx b/docs/tutorial.mdx
index 22285f3..1119461 100644
--- a/docs/tutorial.mdx
+++ b/docs/tutorial.mdx
@@ -9,27 +9,27 @@ import TabItem from '@theme/TabItem';
# Tutorial: stream system metrics
Build a small **metrics agent**, end to end: a program that samples this machine's
-memory every three seconds and streams it into DataHub as time-series data — and keeps
+memory every three seconds and streams it into DataHub as time-series data, and keeps
working when the API doesn't. It runs on Linux, macOS, and Windows. The same program is
shown in Java, Python and Rust; pick your language once and every code block on the
page follows.
The whole flow is five steps, and each is a section below:
-1. **Build the client** — authenticate with a bearer token and turn on a durable,
+1. **Build the client**: authenticate with a bearer token and turn on a durable,
spool-to-disk ingest buffer.
-2. **Ensure the time series exist** — look up three series by external id and create
+2. **Ensure the time series exist**: look up three series by external id and create
only the ones that are missing, so the program is safe to re-run.
-3. **Sample the data** — read memory usage via a small OS-info library (this is just
- the tutorial's data source — swap in your own).
-4. **Ingest datapoints** — every three seconds, send one datapoint per series.
-5. **Survive outages** — when the API is unreachable, datapoints spool to disk and
+3. **Sample the data**: read memory usage via a small OS-info library (this is just
+ the tutorial's data source, swap in your own).
+4. **Ingest datapoints**: every three seconds, send one datapoint per series.
+5. **Survive outages**: when the API is unreachable, datapoints spool to disk and
flush automatically once it recovers.
## Before you start
You need a reachable DataHub API and credentials. This tutorial uses **OAuth2
-client-credentials** — the SDK fetches the bearer token and refreshes it when it
+client-credentials**, the SDK fetches the bearer token and refreshes it when it
expires, so a long-running agent never works with a stale token. The client reads the
configuration from the environment (or a `.env` file in the working directory):
@@ -123,17 +123,17 @@ hostname = "0.4"
```
The code below lives in `src/main.rs`. The `blocking` feature enables the SDK's
-synchronous client (`intellistream_datahub_sdk::blocking`) — same services and methods as
+synchronous client (`intellistream_datahub_sdk::blocking`), same services and methods as
the async API, but no async runtime and no `async`/`.await` in your code.
-## Step 1 — Build the client
+## Step 1: Build the client
Everything starts from one client object. Two things happen here: authentication
-(client-credentials from the environment — the SDK exchanges them for a bearer token
-and refreshes it as needed) and — the part that makes this program resilient — turning
+(client-credentials from the environment, the SDK exchanges them for a bearer token
+and refreshes it as needed) and, the part that makes this program resilient, turning
on the **durable ingest buffer**. With it enabled, an ingest that can't reach the API
is written to compressed segments on disk and replayed automatically later. Without
it, ingestion is best-effort and an outage loses data.
@@ -221,11 +221,11 @@ Buffering is optional and off by default, bounded by a time window and a size ca
story.
:::
-## Step 2 — Ensure the time series exist
+## Step 2: Ensure the time series exist
-Before you can ingest, the target series have to exist. The create endpoint rejects
-duplicates, so rather than blindly creating them the program **looks up first, then
-creates only the gap** — which makes it safe to re-run and safe to start on a machine
+Before you can ingest, the target series have to exist. The create endpoint rejects a
+duplicate external id with `409`, so rather than blindly creating them the program **looks up first, then
+creates only the gap**, which makes it safe to re-run and safe to start on a machine
where the series already exist.
Each external id is **namespaced by hostname** (`system_memory_used_web01`, …), so the
@@ -241,9 +241,9 @@ overwriting one another. The three series, all integer-valued in `bytes`:
:::note Numbers aren't identical across languages
`used`/`available` come from a different OS API per language, and those APIs don't all
mean quite the same thing. Java's `available` is raw OS free memory; Python's and
-Rust's `available` is closer to Linux's cache-aware `MemAvailable` estimate — so on the
+Rust's `available` is closer to Linux's cache-aware `MemAvailable` estimate, so on the
same machine, Java tends to read a lower `available` (and higher `used`) than Python or
-Rust. `used + available == total` holds for Java but generally not for Python/Rust —
+Rust. `used + available == total` holds for Java but generally not for Python/Rust,
that's expected, not a bug.
:::
@@ -343,12 +343,12 @@ for (base, name) in METRICS {
The lookup is **best effort**: if the API is down at startup, the program just retries
-on the next tick — and thanks to the buffer from step 1, datapoints ingested in the
+on the next tick, and thanks to the buffer from step 1, datapoints ingested in the
meantime aren't lost.
-## Step 3 — Sample the data
+## Step 3: Sample the data
-This step has nothing to do with the SDK — it's just where the numbers come from. Each
+This step has nothing to do with the SDK. It is just where the numbers come from. Each
language uses a small OS-info library to read memory and swap in one call, in bytes, so
the same code works on Linux, macOS and Windows. Replace this function with any data
source of your own and the DataHub steps (1, 2, 4, 5) don't change.
@@ -356,7 +356,7 @@ source of your own and the DataHub steps (1, 2, 4, 5) don't change.
-`com.sun.management.OperatingSystemMXBean` is built into the JDK — no dependency
+`com.sun.management.OperatingSystemMXBean` is built into the JDK, no dependency
needed. There's no direct "used" getter, so it's derived as total minus free:
```java
@@ -392,7 +392,7 @@ def sample():
-The sampler is plain synchronous code — identical in both Rust flavors. `sysinfo`
+The sampler is plain synchronous code, identical in both Rust flavors. `sysinfo`
exposes `used_memory()`/`available_memory()`/`used_swap()` directly:
```rust
@@ -410,7 +410,7 @@ fn sample() -> HashMap<&'static str, i64> {
-The sampler is plain synchronous code — identical in both Rust flavors. `sysinfo`
+The sampler is plain synchronous code, identical in both Rust flavors. `sysinfo`
exposes `used_memory()`/`available_memory()`/`used_swap()` directly:
```rust
@@ -428,10 +428,10 @@ fn sample() -> HashMap<&'static str, i64> {
-## Step 4 — Ingest on a tick
+## Step 4: Ingest on a tick
A timer fires every three seconds. Each tick samples memory and sends one datapoint per
-series — three datapoints, stamped with the same timestamp.
+series, three datapoints, stamped with the same timestamp.
@@ -453,7 +453,7 @@ IngestResult result = client.timeseries().ingest(byExternalId);
-One `insert_from_lists` call per series — here each carries a single datapoint, but the
+One `insert_from_lists` call per series, here each carries a single datapoint, but the
same call takes whole arrays when you have them. Any timezone-aware timestamps work
(plain `datetime` here; pandas if you already have it):
@@ -497,23 +497,23 @@ for (base, _) in METRICS {
-Whatever a single tick does — a sampling error, a rejected request — it must never kill
+Whatever a single tick does, a sampling error, a rejected request, it must never kill
the loop: catch, log, and let the next tick try again. The complete programs below wrap
each tick that way.
-## Step 5 — Survive an outage
+## Step 5: Survive an outage
This is what the buffer from step 1 buys you. Stop the DataHub API while the program is
running: instead of raising, each ingest spools its datapoints to compressed segments
under `datahub-spool/`. Start the API again and the backlog flushes automatically on
-the next call — no data lost within the retention window. The same happens when the
+the next call, no data lost within the retention window. The same happens when the
token is rejected (HTTP 401/403, e.g. expired): datapoints keep spooling until you
restore a valid token.
-The Java client reports buffering explicitly — `IngestResult.buffered()` is the number
+The Java client reports buffering explicitly, `IngestResult.buffered()` is the number
of datapoints that were spooled instead of sent:
```java
@@ -533,7 +533,7 @@ if (result.buffered() > 0) {
-With buffering enabled the calls simply don't raise on an outage — the datapoints are
+With buffering enabled the calls simply don't raise on an outage, the datapoints are
spooled and the program keeps ticking. Watch the spool itself to see it happen:
```bash
@@ -546,7 +546,7 @@ the tick wrapped in `try`/`except`.
-With buffering enabled the calls return `Ok` on an outage — the datapoints are spooled
+With buffering enabled the calls return `Ok` on an outage, the datapoints are spooled
and the program keeps ticking. Watch the spool itself to see it happen:
```bash
@@ -559,7 +559,7 @@ handling the `Result`.
-With buffering enabled the calls return `Ok` on an outage — the datapoints are spooled
+With buffering enabled the calls return `Ok` on an outage, the datapoints are spooled
and the program keeps ticking. Watch the spool itself to see it happen:
```bash
@@ -1077,10 +1077,10 @@ fn hostname() -> String {
## Try this next
- **Add a metric.** Add one entry to the metric table (e.g. total memory) and extend
- the sampler — series creation and ingestion both pick it up with no other changes.
+ the sampler, series creation and ingestion both pick it up with no other changes.
- **Namespace by host.** Run the program on two machines (or set `HOST_ID` to two
different values) and watch each get its own set of series.
-- **Swap the data source.** Replace the sampler with your own readings — the DataHub
+- **Swap the data source.** Replace the sampler with your own readings, the DataHub
steps don't change.
- **Read it back.** Chart the trend with
[Query & aggregate time-series](/guides/query-and-aggregate), or tail it live with