feat(backend): add TypeORM migrations and disable synchronize outside development

backend/docs/DATABASE_MIGRATIONS.md

@@ -0,0 +1,130 @@
+# Database Migrations
+
+The schema is owned by TypeORM migrations. `synchronize` is enabled **only**
+when `NODE_ENV=development`; every other environment (including an unset
+`NODE_ENV`) gets its schema exclusively from the migration files in
+`src/migrations/`.
+
+## Files
+
+| File | Purpose |
+| --- | --- |
+| `src/config/database.config.ts` | Single source of truth for the connection options, shared by the Nest app and the CLI. |
+| `src/data-source.ts` | `DataSource` instance the TypeORM CLI loads (`-d src/data-source.ts`). |
+| `src/database/run-migrations.ts` | Run / revert / baseline, each under a PostgreSQL advisory lock. |
+| `src/migrate.ts` | Standalone deploy entry point (`node dist/migrate.js <command>`). |
+| `src/migrations/` | Generated migration files, applied in timestamp order. |
+
+`src/data-source.ts` loads `.env` itself, so CLI commands pick up the same
+`DB_HOST` / `DB_PORT` / `DB_USERNAME` / `DB_PASSWORD` / `DB_DATABASE` values the
+application uses.
+
+## Commands
+
+Run these from `backend/`:
+
+```bash
+# Apply all pending migrations
+npm run migration:run
+
+# Revert the most recently applied migration
+npm run migration:revert
+
+# List applied ([X]) and pending ([ ]) migrations
+npm run migration:show
+
+# Generate a migration from the diff between entities and the live database
+npm run migration:generate -- src/migrations/DescriptiveName
+
+# Create an empty migration to hand-write (e.g. a data backfill)
+npm run migration:create -- src/migrations/DescriptiveName
+```
+
+`migration:generate` diffs your **entities** against a **live database**, so
+point it at a database that is already up to date with the committed
+migrations, then commit the file it writes.
+
+## Baselining a database built by `synchronize`
+
+Any database created before this workflow existed already has the tables, but
+no row in the `migrations` table. Running `InitialSchema` against it would fail
+on `CREATE TABLE "verifications"` — the table is already there.
+
+Such a database must be **baselined once**: record the initial migration as
+applied without replaying its DDL.
+
+```bash
+# 1. Confirm the live schema really does match the initial migration.
+#    Silence here ("No changes in database schema were found") means they agree.
+npm run migration:generate -- src/migrations/BaselineCheck
+
+# 2. If — and only if — step 1 reported no changes, mark migrations as applied
+#    without executing them.
+npm run migration:baseline:prod
+```
+
+If step 1 *does* emit a migration, the live schema has drifted from the
+entities. Delete the generated file, reconcile the difference deliberately, and
+only then baseline. Rehearse the whole procedure against a restored copy of the
+production database before touching production.
+
+Fresh databases need none of this — `migration:run` handles them.
+
+## Production
+
+**Every schema change against a shared database must go through a `:prod`
+command or the application's own startup path.** Those are the only routes that
+take the advisory lock. The plain `migration:run` / `migration:revert` /
+`migration:baseline` scripts drive the TypeORM CLI directly, take no lock, and
+are meant for a developer's local database.
+
+| Command | Effect |
+| --- | --- |
+| `npm run migration:run:prod` | `node dist/migrate.js run` — apply pending migrations |
+| `npm run migration:revert:prod` | `node dist/migrate.js revert` — revert the last migration |
+| `npm run migration:baseline:prod` | `node dist/migrate.js baseline` — record migrations as applied without running them |
+| `npm run migration:show:prod` | List applied / pending migrations (read-only, no lock) |
+
+None of these need `ts-node`, so they work on a host installed with
+`npm install --production`.
+
+Two paths apply pending migrations on a deploy, and both take the same lock, so
+either order is safe:
+
+1. **On startup.** Whenever `NODE_ENV` is not `development`, `main.ts` calls
+   `runPendingMigrations` before `app.listen()`, so the app never serves
+   traffic against a stale schema.
+2. **From the deploy script.** `scripts/deploy-backend.sh` runs
+   `npm run migration:run:prod` after the build.
+
+### Why the advisory lock
+
+TypeORM decides which migrations are pending *before* recording them and takes
+no cross-process lock of its own. During a rolling deploy, with more than one
+replica, or when an operator runs a revert while an instance is booting, two
+processes can therefore each conclude the same migration is pending and run its
+non-idempotent DDL twice. `withMigrationLock` wraps every schema operation in a
+PostgreSQL session-level advisory lock (`MIGRATION_ADVISORY_LOCK_KEY`): the
+second process blocks until the first commits and releases, then re-reads the
+migrations table and finds the work already done.
+
+The lock is held on its own connection rather than the one TypeORM uses for the
+DDL. An advisory lock is a mutex between *sessions* and only has to be held for
+the duration of the work it guards; holding it on a dedicated session also
+keeps it alive across the several connections TypeORM opens and closes while
+migrating.
+
+This relies on a session-pooled connection. Behind a transaction-pooling proxy
+(PgBouncer in `transaction` mode), session-level advisory locks are not held
+across statements — point migrations at a direct connection there.
+
+`migrationsRun` is deliberately left `false` everywhere, because TypeORM's own
+bootstrap hook runs migrations inside `DataSource.initialize()`, before this
+lock can be taken.
+
+## Adding an entity
+
+1. Write the entity as usual — it is picked up by the `*.entity.{ts,js}` glob.
+2. Run `npm run migration:generate -- src/migrations/AddYourEntity` against a
+   database that is current.
+3. Review the generated SQL, then commit the migration alongside the entity.

backend/package.json

@@ -12,6 +12,18 @@
     "start:dev": "nest start --watch",
     "start:debug": "nest start --debug --watch",
     "start:prod": "node dist/main",
+    "typeorm": "typeorm-ts-node-commonjs -d src/data-source.ts",
+    "migration:generate": "npm run typeorm -- migration:generate",
+    "migration:create": "typeorm-ts-node-commonjs migration:create",
+    "migration:run": "npm run typeorm -- migration:run",
+    "migration:revert": "npm run typeorm -- migration:revert",
+    "migration:show": "npm run typeorm -- migration:show",
+    "migration:baseline": "npm run typeorm -- migration:run --fake",
+    "typeorm:prod": "typeorm -d dist/data-source.js",
+    "migration:run:prod": "node dist/migrate.js run",
+    "migration:revert:prod": "node dist/migrate.js revert",
+    "migration:baseline:prod": "node dist/migrate.js baseline",
+    "migration:show:prod": "npm run typeorm:prod -- migration:show",
     "lint": "eslint \"{src,apps,libs,test}/**/*.ts\" --fix",
     "test": "jest",
     "test:watch": "jest --watch",
@@ -45,6 +57,7 @@
     "cache-manager-redis-store": "^3.0.1",
     "class-transformer": "^0.5.1",
     "class-validator": "^0.14.0",
+    "dotenv": "^16.6.1",
     "groq-sdk": "^0.3.0",
     "ioredis": "^5.3.2",
     "ipfs-http-client": "^60.0.0",

backend/src/app.module.ts

@@ -6,6 +6,7 @@ import { ThrottlerModule } from '@nestjs/throttler';
 import { APP_GUARD } from '@nestjs/core';
 import { ThrottlerGuard } from '@nestjs/throttler';
 import { redisStore } from 'cache-manager-redis-store';
+import { buildDatabaseOptions } from './config/database.config';
 import { IdentityModule } from './identity/identity.module';
 import { VerificationModule } from './verification/verification.module';
 import { AccessControlModule } from './access-control/access-control.module';
@@ -37,17 +38,8 @@ import { RequestContextMiddleware } from './common/logger/logger.middleware';
     LoggerModule,
     TypeOrmModule.forRootAsync({
       imports: [ConfigModule],
-      useFactory: (configService: ConfigService) => ({
-        type: 'postgres',
-        host: configService.get('DB_HOST'),
-        port: configService.get('DB_PORT'),
-        username: configService.get('DB_USERNAME'),
-        password: configService.get('DB_PASSWORD'),
-        database: configService.get('DB_DATABASE'),
-        entities: [__dirname + '/**/*.entity{.ts,.js}'],
-        synchronize: configService.get('NODE_ENV') === 'development',
-        logging: configService.get('NODE_ENV') === 'development',
-      }),
+      useFactory: (configService: ConfigService) =>
+        buildDatabaseOptions((key) => configService.get<string>(key)),
       inject: [ConfigService],
     }),
     ThrottlerModule.forRootAsync({

backend/src/config/database.config.spec.ts

@@ -0,0 +1,69 @@
+import { buildDatabaseOptions, isDevelopment } from './database.config';
+
+const reader = (env: Record<string, string | undefined>) => (key: string) =>
+  env[key];
+
+describe('buildDatabaseOptions', () => {
+  it('enables synchronize only when NODE_ENV is development', () => {
+    const options = buildDatabaseOptions(reader({ NODE_ENV: 'development' }));
+
+    expect(options.synchronize).toBe(true);
+  });
+
+  it.each(['production', 'staging', 'test', undefined])(
+    'disables synchronize when NODE_ENV is %s',
+    (nodeEnv) => {
+      const options = buildDatabaseOptions(reader({ NODE_ENV: nodeEnv }));
+
+      expect(options.synchronize).toBe(false);
+    },
+  );
+
+  it.each(['development', 'production', undefined])(
+    'never lets TypeORM auto-run migrations when NODE_ENV is %s',
+    (nodeEnv) => {
+      // Migrations are applied explicitly by `runPendingMigrations`, which
+      // holds an advisory lock that `migrationsRun` cannot take.
+      const options = buildDatabaseOptions(reader({ NODE_ENV: nodeEnv }));
+
+      expect(options.migrationsRun).toBe(false);
+    },
+  );
+
+  it('reads connection settings from the environment', () => {
+    const options = buildDatabaseOptions(
+      reader({
+        NODE_ENV: 'production',
+        DB_HOST: 'db.internal',
+        DB_PORT: '6543',
+        DB_USERNAME: 'validfi',
+        DB_PASSWORD: 's3cret',
+        DB_DATABASE: 'validfi',
+      }),
+    );
+
+    expect(options).toMatchObject({
+      type: 'postgres',
+      host: 'db.internal',
+      port: 6543,
+      username: 'validfi',
+      password: 's3cret',
+      database: 'validfi',
+    });
+  });
+
+  it('points at the migrations directory and tracking table', () => {
+    const options = buildDatabaseOptions(reader({ NODE_ENV: 'production' }));
+
+    expect(options.migrationsTableName).toBe('migrations');
+    expect(options.migrations).toEqual([
+      expect.stringContaining('migrations'),
+    ]);
+  });
+});
+
+describe('isDevelopment', () => {
+  it('is false when NODE_ENV is unset', () => {
+    expect(isDevelopment(reader({}))).toBe(false);
+  });
+});

backend/src/config/database.config.ts

@@ -0,0 +1,45 @@
+import { join } from 'path';
+import { DataSourceOptions } from 'typeorm';
+
+/**
+ * Reads a configuration value. Lets the same options be built from either
+ * Nest's ConfigService or raw `process.env` (TypeORM CLI).
+ */
+export type EnvReader = (key: string) => string | undefined;
+
+/**
+ * Strict check: anything other than an explicit `development` is treated as a
+ * non-development environment, so an unset NODE_ENV never enables auto sync.
+ */
+export const isDevelopment = (get: EnvReader): boolean =>
+  get('NODE_ENV') === 'development';
+
+/**
+ * Single source of truth for the database connection, shared by the Nest
+ * application and the TypeORM CLI data source so both always agree on where
+ * entities and migrations live.
+ */
+export const buildDatabaseOptions = (get: EnvReader): DataSourceOptions => {
+  const development = isDevelopment(get);
+
+  return {
+    type: 'postgres',
+    host: get('DB_HOST') ?? 'localhost',
+    port: parseInt(get('DB_PORT') ?? '5432', 10),
+    username: get('DB_USERNAME') ?? 'postgres',
+    password: get('DB_PASSWORD') ?? 'postgres',
+    database: get('DB_DATABASE') ?? 'securedata',
+    entities: [join(__dirname, '..', '**', '*.entity{.ts,.js}')],
+    migrations: [join(__dirname, '..', 'migrations', '*{.ts,.js}')],
+    migrationsTableName: 'migrations',
+    // Auto schema sync is a development-only convenience; every other
+    // environment gets its schema exclusively from migrations.
+    synchronize: development,
+    // Migrations are never run by TypeORM's own bootstrap hook: outside
+    // development the app applies them explicitly in `main.ts` via
+    // `runPendingMigrations`, which serialises concurrent instances behind a
+    // PostgreSQL advisory lock that `migrationsRun` cannot take.
+    migrationsRun: false,
+    logging: development,
+  };
+};

backend/src/data-source.ts

@@ -0,0 +1,22 @@
+import { config as loadEnv } from 'dotenv';
+import { DataSource, DataSourceOptions } from 'typeorm';
+import { buildDatabaseOptions } from './config/database.config';
+
+// The CLI runs outside Nest, so the .env file has to be loaded by hand.
+loadEnv();
+
+/**
+ * Options used by the TypeORM CLI. Schema changes must go through migrations,
+ * so the CLI never synchronizes or auto-runs migrations as a side effect.
+ */
+export const dataSourceOptions: DataSourceOptions = {
+  ...buildDatabaseOptions((key) => process.env[key]),
+  synchronize: false,
+  migrationsRun: false,
+};
+
+/**
+ * Data source for `npm run typeorm -- <command>`, e.g.
+ * `npm run migration:generate -- src/migrations/AddSomething`.
+ */
+export const AppDataSource = new DataSource(dataSourceOptions);