Skip to content

Commit 1ebfa27

Browse files
committed
Update README and docs to relfect the revamp to the federation and sync system
1 parent 7d522a3 commit 1ebfa27

3 files changed

Lines changed: 141 additions & 631 deletions

File tree

README.md

Lines changed: 48 additions & 180 deletions
Original file line numberDiff line numberDiff line change
@@ -936,162 +936,88 @@ In this scenario, we demonstrate a hybrid data architecture where the goal is to
936936
> Query remote data **as if it were local**, while controlling
937937
> what stays remote, what gets cached locally, and what stays in sync.
938938
939-
This is where LinkedQL moves beyond “client” and becomes a **data architecture layer**.
940-
941-
At a high level, this system works like this:
942-
943-
- You define **where data comes from** (local vs remote)
944-
- You define **how it is stored** (none, cached, or synced)
945-
- LinkedQL ensures everything behaves like a single database
946-
947-
The idea starts with the local database – this time, instantiated with a hook to the remote database.
939+
The idea is straight-forward in FlashQL: you simply create a view (a database view) in your local database that mirrors the remote database.
948940
949941
```js
950-
const db = new FlashQL({
951-
// The hook to remote
952-
async onCreateForeignClient() {
953-
return new EdgeClient({ url: '/api/db', type: 'http' });
954-
}
955-
});
956-
957-
await db.connect();
958-
959-
// The queries
960-
// which can span local and remote tables
961942
await db.query(`
962-
SELECT * FROM remote.users
943+
CREATE VIEW public.users AS
944+
SELECT * FROM public.users
945+
WITH (replication_origin = '/api/db')
963946
`);
964947
```
965948
966-
This query may:
949+
Notice the `WITH (replication_origin = ...)` specifier. That's the magic.
967950

968-
- hit a remote database,
969-
- use a local replica,
970-
- or combine both
951+
Rows in this view (`public.users`) will mirror `public.users` in the upstream database.
971952

972-
But it always behaves like a single SQL query.
953+
But just one more thing is required for this to work:
973954

974-
The system is introduced step by step below. Much of that is mere configuration.
955+
> a way to connect the local FlashQL instance to the upstream database.
975956

976-
---
977-
978-
#### Step 1: Connect a Local Engine to a Remote Origin
979-
980-
We start with FlashQL as our local database.
981-
982-
Then we teach it how to reach a remote database when needed.
957+
For this, the `EdgeClient` interface introduced above comes to play:
983958

984959
```js
985960
import { FlashQL } from '@linked-db/linked-ql/flashql';
986961
import { EdgeClient } from '@linked-db/linked-ql/edge';
987962
988-
// The local database
989963
const db = new FlashQL({
990-
991-
// Called whenever a query references a foreign origin and needs a client
992-
async onCreateForeignClient(origin) {
993-
994-
if (origin === 'primary') {
995-
// This client will be used to reach the remote database we've designated as 'primary'
996-
return new EdgeClient({
997-
url: '/api/db',
998-
type: 'http',
999-
});
1000-
}
1001-
1002-
throw new Error(`Unknown origin: ${origin}`);
964+
// The hook to remote
965+
async onCreateForeignClient(originUrl) {
966+
return new EdgeClient({ url: originUrl, type: 'http' });
1003967
}
1004968
});
1005969
1006970
await db.connect();
1007971
```
1008972

1009-
Next, we create a schema – more specifically called a "namespace" in FlashQL – with the foreign origin behaviour.
973+
This is now a local FlashQL instance that can talk to an upstream database ondemand.
1010974

1011-
```js
1012-
await db.query(`
1013-
CREATE SCHEMA remote
1014-
WITH (replication_origin = 'primary')
1015-
`);
1016-
```
1017-
1018-
This namespace will be able to incorporate data from a foreign origin tagged here as "primary". The result of `onCreateForeignClient()`
1019-
will be the foreign client interface.
1020-
1021-
---
1022-
1023-
##### Decoding the above
1024-
1025-
* FlashQL is your **local database**
1026-
* `remote` schema is a **regular schema (namespace)** that can have tables and views (`VIEWS`)
1027-
* `replication_origin` is a setting that extends the behaviour of the namespace to include objects from a foreign origin
1028-
* the value: `"primary"` is a custom identifier for this **remote database**. (FlashQL lets the origin details be anything: an identifier, a URL, a database connection string, etc.)
1029-
* `EdgeClient` – the result of `onCreateForeignClient()` – is how FlashQL will talk to that remote system
1030-
1031-
At this point:
1032-
1033-
* nothing is mirrored yet
1034-
* no data is fetched
1035-
* we’ve only defined **how to reach the remote** from the local
975+
Above, the `onCreateForeignClient()` hook will recieve `'/api/db'` – the value of the `replication_origin` config.
1036976

1037-
But the local namespace by itself is ready for use as normal. You can create tables as necessary:
977+
Now, querying `public.users` on the local database will query `public.users` on the upstream database:
1038978

1039979
```js
1040-
// FlashQL accepts multiple statements in a single call
1041980
await db.query(`
1042-
-- Define tables explicitly
1043-
CREATE TABLE remote.users (
1044-
id INT PRIMARY KEY,
1045-
name TEXT
1046-
);
1047-
1048-
-- Seed local data
1049-
INSERT INTO remote.users (id, name)
1050-
VALUES (1, 'Ada'), (2, 'Linus');
981+
SELECT * FROM public.users;
1051982
`);
1052983
```
1053984

1054-
But the containers of foreign data will not be tables. They will be views (VIEWS).
1055-
1056-
Think of views as table-like objects whose contents come from other tables – whether local tables or remote tables.
1057-
1058-
---
1059-
1060-
#### Step 2: Let’s Mirror Foreign Tables Locally
1061-
1062-
A view created inside a namespace with `replication_origin`, like the above, automatically resolves its data from the foreign origin.
985+
Being a regular table, it can be used just like one – e.g. in joins:
1063986

1064987
```js
1065988
await db.query(`
1066-
CREATE VIEW remote.users AS
1067-
SELECT * FROM public.users
989+
SELECT * FROM public.posts
990+
LEFT JOIN public.users ON posts.user_id = users.id;
1068991
`);
1069992
```
1070993

1071-
Rows in this view (`remote.users`) will mirror `public.users` in foreign origin. The persistence of this data is configurable.
994+
The query executes as one relational graph – but composed of both local and remote data.
1072995

1073-
FlashQL supports **three persistence modes**. These modes determine how mirroring works; i.e. whether data stays remote, or is cached locally, or stays in sync.
996+
Given this as the base, FlashQL further lets you create other types of views with different replication modes.
997+
998+
These modes determine how mirroring works; i.e. whether data stays remote, or is cached locally, or stays in sync.
1074999

10751000
| Mode | Behavior |
10761001
| :------------------------------------------------ | :-------------------------------------- |
1077-
| `persistence="none"`(the default) | Views set to this mode don't copy the remote data locally; they simply act as local references to remote data. These are called "non-persistent views" |
1078-
| `persistence="materialized"` | Views set to this mode copy the origin data locally and behave as local tables from that moment on; cached data is refreshed manually. These are called "materialized views" |
1079-
| `persistence="realtime"` | Views set to this mode copy the remote data locally and behave as local tables from that moment on; **but most notably, local data is kept in sync with origin data**. These are called "realtime views" |
1002+
| Basic views (the default) | This is the default idea of a view: a table that has no actual rows but just a query that executes at query-time. |
1003+
| Materialized views | These views go ahead to copy the origin data for local use and behave as local tables from that moment on. |
1004+
| Realtime views | These views are materialized views that not just copy origin data, but also stay in sync with origin data. |
10801005

1081-
* use `none` when freshness matters more than offline access
1082-
* use `materialized` when you want a local cache you can refresh deliberately
1083-
* use `realtime` when the local copy should stay warm automatically after initial sync
1006+
* use basic views when you just want to federate remote data and don't need offline access
1007+
* use `materialized` views when you want a local copy for offline access
1008+
* use `realtime` views when the local copy should stay in sync origin data
10841009
10851010
Each mode is demonstrated below.
10861011
10871012
---
10881013
1089-
##### Mode 1: Non-Persistent Views (Pure Federation)
1014+
##### Mode 1: Basic Views (Pure Federation)
10901015
10911016
```js
10921017
await db.query(`
1093-
CREATE VIEW remote.users AS
1018+
CREATE VIEW public.users AS
10941019
SELECT * FROM public.users
1020+
WITH (replication_origin = '/api/db')
10951021
`);
10961022
```
10971023
@@ -1115,8 +1041,9 @@ This is the lightest-weight mode. It gives you unification without local storage
11151041
11161042
```js
11171043
await db.query(`
1118-
CREATE MATERIALIZED VIEW remote.orders AS
1044+
CREATE MATERIALIZED VIEW public.orders AS
11191045
SELECT * FROM public.orders
1046+
WITH (replication_origin = '/api/db')
11201047
`);
11211048
```
11221049
@@ -1130,13 +1057,15 @@ await db.query(`
11301057
11311058
This is **materialization**:
11321059
1133-
> Keeping a local snapshot of remote data for performance or offline use
1060+
This is the mode to reach for when the data should remain queryable while offline.
11341061
1135-
This is the mode to reach for when:
1062+
These views can be refreshed explicitly:
11361063
1137-
* the dataset is expensive to fetch repeatedly
1138-
* it should remain queryable while offline
1139-
* and "fresh on demand" is good enough
1064+
```js
1065+
await db.query(`
1066+
REFRESH MATERIALIZED VIEW public.orders
1067+
`);
1068+
```
11401069
11411070
---
11421071
@@ -1145,9 +1074,8 @@ This is the mode to reach for when:
11451074
```js
11461075
await db.query(`
11471076
CREATE REALTIME VIEW remote.posts AS
1148-
SELECT *
1149-
FROM public.posts
1150-
WHERE post_type = 'NEWS'
1077+
SELECT * FROM public.posts
1078+
WITH (replication_origin = '/api/db')
11511079
`);
11521080
```
11531081
@@ -1161,76 +1089,28 @@ await db.query(`
11611089
11621090
This is **realtime mirroring**:
11631091
1164-
> A local table that tracks and syncs with the remote table over time
1165-
1166-
This is the richest mode:
1167-
11681092
* it starts with local state
11691093
* keeps that state queryable even when the app is temporarily disconnected
11701094
* and then catches up again when connectivity returns
11711095
1172-
---
1173-
1174-
#### Step 3: Running Sync
1175-
1176-
On having defined the views, you activate the synchronization via:
1177-
1178-
```js
1179-
await db.sync.sync();
1180-
```
1181-
1182-
---
1183-
1184-
##### What `sync()` does
1185-
1186-
`sync()` is the coordination engine for "materialized" and "realtime" views.
1187-
It is what turns definitions (VIEWS) into state (local data + subscriptions).
1188-
1189-
It:
1190-
1191-
* fetches data for all `materialized` views
1192-
* does the same for `realtime` views and starts syncing right away – with backpressure and replay support:
1193-
* performs catch-up if the app was offline
1194-
* ensures local state matches expected remote state
1195-
1196-
---
1197-
1198-
##### `sync()` is:
1199-
1200-
* **Idempotent** → safe to call multiple times
1201-
* **Resumable** → knows hot to continue from last known state
1202-
* **Network-aware** → designed for reconnect flows
1203-
1204-
##### Typical usage:
1205-
1206-
First: **the initial call after defining views**:
1207-
1208-
```js
1209-
await db.sync.sync();
1210-
```
1211-
1212-
Second: **the optional wiring to the app-level network signal:**
1096+
Realtime views are designed to be resilient to network disconnects. All you need to do in
1097+
web app, for example, is call FlashQL's `sync.sync()` API to resume work on network reconnection:
12131098

12141099
```js
12151100
window.addEventListener('online', () => {
12161101
db.sync.sync(); // re-sync on reconnect
12171102
});
12181103
```
12191104

1220-
At that point, your local database is no longer just "configured".
1221-
It is now hydrated, subscribed where necessary, and ready to behave like a unified relational graph.
1222-
1223-
> Note that the initial `db.sync.sync()` can be automatically-handled by FlashQL. Simply pass `autoSync: true` in constructor parameters:
1224-
>
1225-
> `new FlashQL({ autoSync: true });`
1105+
**`sync()`** knows how to continue from last known state.
12261106

12271107
---
12281108

12291109
#### Step 5: Querying the Unified Graph
12301110

12311111
At query time, LinkedQL builds a composed execution plan:
12321112

1233-
* non-persistent views are resolved on demand
1113+
* basic views are resolved on demand
12341114
* `materialized` and `realtime` views are resolved locally
12351115
* results are merged into a single relational execution
12361116

@@ -1254,19 +1134,7 @@ const result = await db.query(`
12541134

12551135
---
12561136

1257-
#### Resolution summary
1258-
1259-
* `remote.users` → fetched on demand from the remote DB
1260-
* `remote.orders` → served from the local cache created by materialization
1261-
* `remote.posts` → served locally, then kept hot by realtime sync
1262-
* `public.test` → an ordinary local table with no remote involvement
1263-
* The planner treats all of them as one relational graph even though they come from different storage modes
1264-
1265-
This is the key architectural promise of LinkedQL:
1266-
1267-
> You choose data placement and sync policy per relation, but you still query the result as one database.
1268-
1269-
For the FlashQL side of this model, see [Federation & Sync ↗](https://linked-ql.netlify.app/flashql/foreign-io) and [FlashQL Sync ↗](https://linked-ql.netlify.app/flashql/sync).
1137+
For full details, see [Federation & Sync ↗](https://linked-ql.netlify.app/flashql/foreign-io) and [FlashQL Sync ↗](https://linked-ql.netlify.app/flashql/sync).
12701138

12711139
---
12721140

0 commit comments

Comments
 (0)