Migrations from TM and docs

.gitignore

@@ -170,3 +170,4 @@ docs/tasking-mvp/tasking-mvp.postman_environment.json
 docs/tasking-mvp/_enrich_postman.py
 docs/tasking-mvp/feature-coverage.md
 .idea/
+/.claude

README.md

@@ -6,6 +6,77 @@ This is a combination API backend for workspaces, providing /workspaces* methods
 the OSM API ("openstreetmap website") plus OSM CGI-map (the C-accelerated methods) that enforces 
 authorization and authentication based on a TDEI/Keycloak JWT token (see main.py for this proxy logic). 
 
+## Deployment architecture
+
+The deployed system is defined by [`docker-compose.az.yml`](docker-compose.az.yml). It runs the
+**application tier** as four containers; the **data tier** (Postgres/PostGIS) is external, managed
+Azure Database for PostgreSQL, not part of this compose file.
+
+When deployed by `workspaces-stack` this model also holds. 
+
+```
+                    client  (TDEI/Keycloak JWT)
+                       │
+                       ▼ :8000
+        ┌──────────────────────────────────────┐
+        │          workspaces-backend           │  this repo — FastAPI front door.
+        │   authn/authz + OSM reverse proxy     │  Serves /api/v1/*, proxies the rest.
+        └───┬──────────────────────────────┬────┘
+     WS_OSM_HOST                   TASK_DATABASE_URL
+   (→ osm-rails)                   OSM_DATABASE_URL
+            │                              │
+            ▼                              │
+     ┌──────────────┐                      │
+     │  osm-rails   │  OSM website (Rails); the single OSM entry point.
+     │  :3000       │  Serves the API/UI and fronts cgimap for the
+     └──────┬───────┘  performance-critical /api/0.6 calls.
+            │ (internal)                   │
+            ▼                              │
+     ┌──────────────┐                      │
+     │  osm-cgimap  │  C-accelerated /api/0.6 (map, changeset bulk)
+     │  :8000       │                      │
+     └──────────────┘                      │
+                                           │
+     osm-rails-worker  (rake jobs:work)    │  background jobs
+            │                              │
+            │  backend, rails, cgimap, worker all connect to ▼
+   ┌─────────────────────────────────────────────────────────────────┐
+   │  data tier — Azure Postgres (external, PostGIS)                   │
+   │  opensidewalks-${ENV}.postgres.database.azure.com:5432            │
+   │    • workspaces-tasks-${ENV}   TASK db  (alembic_task; backend)   │
+   │    • workspaces-osm-${ENV}     OSM db   (alembic_osm; all four)   │
+   └─────────────────────────────────────────────────────────────────┘
+```
+
+### Services
+
+| Service | Image | Role |
+|---|---|---|
+| `workspaces-backend` | `workspaces-backend-v2:${ENV}` | This repo. The only host-exposed service (`8000:8000`). Validates the TDEI/Keycloak JWT, enforces workspace authorization, serves `/api/v1/*`, and proxies everything else to the OSM tier. Connects to **both** databases. |
+| `osm-rails` | `workspaces-osm-rails-v2:${ENV}` | The OpenStreetMap website (Rails) — the **single OSM entry point** the backend proxies to (`WS_OSM_HOST`). Serves the OSM API/UI and fronts cgimap for the heavy `/api/0.6` calls. Connects to the OSM db. |
+| `osm-cgimap` | `workspaces-osm-cgimap-v2:${ENV}` | C++ reimplementation of the performance-critical OSM `0.6` calls (map queries, changeset upload/download), sitting behind `osm-rails`. Tuned here for large imports (`CGIMAP_MAX_*`). Connects to the OSM db. |
+| `osm-rails-worker` | `workspaces-osm-rails-v2:${ENV}` | Background job runner (`rake jobs:work`) for the Rails app. Connects to the OSM db. |
+
+### Two databases
+
+The backend holds two connections, and the two alembic trees target them independently (see
+`CLAUDE.md` and `api/utils/migrations.py`):
+
+* **TASK db** (`TASK_DATABASE_URL` → `workspaces-tasks-${ENV}`) — the workspaces + tasking-manager
+  schema, built by the `alembic_task` tree. Only the backend connects here.
+* **OSM db** (`OSM_DATABASE_URL` → `workspaces-osm-${ENV}`) — OSM data plus `users` and the
+  `tasking_*` tables, built by the `alembic_osm` tree. The backend, cgimap, rails, and the worker
+  all connect here.
+
+On startup (outside of pytest) the backend runs `alembic -n task upgrade head` and
+`alembic -n osm upgrade head`, applying each tree to its database.
+
+### Environment templating
+
+Every image tag, database name/user, and server host is parameterized by `${ENV}`
+(`dev` / `stage` / `prod`), and secrets are injected from the shell environment
+(`${WS_TASKS_DB_PASS}`, `${WS_OSM_DB_PASS}`, `${WS_OSM_SECRET_KEY_BASE}`). Branches map to these
+environments — see the Branch Index below.
 ## What the proxy must provide for osm-rails / osm-web
 
 This backend is the **only entry point** to the OSM tier (osm-rails + cgimap,

alembic_task/env.py

@@ -1,4 +1,3 @@
-import asyncio
 import importlib
 import os
 import sys
@@ -8,14 +7,21 @@
 # Add the project root directory to the Python path
 sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), "..")))
 
+import geoalchemy2.alembic_helpers  # noqa: F401
 from alembic import context
-from sqlalchemy import pool
+from sqlalchemy import engine_from_config, pool
 from sqlalchemy.engine import Connection
-from sqlalchemy.ext.asyncio import async_engine_from_config
 
 from api.core.config import settings
 from api.core.database import Base
 
+# Importing geoalchemy2.alembic_helpers (above) registers the geospatial
+# operations (`op.create_geospatial_table` / `create_geospatial_index`) that the
+# tasking-manager migration `cbc419d1740c` uses to create the `workspaces`
+# table. A bare `from geoalchemy2 import Geometry` does NOT register them, so
+# without this import that migration fails at upgrade time.
+
+
 # Automatically import all models
 src_path = Path(__file__).parent.parent / "api" / "src"
 for path in src_path.rglob("*.py"):
@@ -35,8 +41,16 @@
 if config.config_file_name is not None:
     fileConfig(config.config_file_name)
 
-# Set sqlalchemy.url from settings
-config.set_main_option("sqlalchemy.url", settings.TASK_DATABASE_URL)
+# Run migrations with a SYNCHRONOUS psycopg2 driver, even though the app uses
+# asyncpg at runtime. Many of the imported tasking-manager migrations were
+# authored for psycopg2 and use patterns asyncpg rejects — most notably
+# multi-statement `op.execute` (asyncpg runs every statement as a prepared
+# statement and raises "cannot insert multiple commands into a prepared
+# statement", e.g. the full-text-search trigger in 451f6bd05a19). Only the
+# driver in the URL is swapped; host/database/credentials are untouched.
+config.set_main_option(
+    "sqlalchemy.url", settings.TASK_DATABASE_URL.replace("+asyncpg", "+psycopg2")
+)
 
 # Add your model's MetaData object here for 'autogenerate' support
 target_metadata = Base.metadata
@@ -63,26 +77,19 @@ def do_run_migrations(connection: Connection) -> None:
         context.run_migrations()
 
 
-async def run_async_migrations() -> None:
-    """In this scenario we need to create an Engine
-    and associate a connection with the context."""
+def run_migrations_online() -> None:
+    """Run migrations in 'online' mode with a synchronous engine."""
 
-    connectable = async_engine_from_config(
+    connectable = engine_from_config(
         config.get_section(config.config_ini_section, {}),
         prefix="sqlalchemy.",
         poolclass=pool.NullPool,
     )
 
-    async with connectable.connect() as connection:
-        await connection.run_sync(do_run_migrations)
-
-    await connectable.dispose()
-
-
-def run_migrations_online() -> None:
-    """Run migrations in 'online' mode."""
+    with connectable.connect() as connection:
+        do_run_migrations(connection)
 
-    asyncio.run(run_async_migrations())
+    connectable.dispose()
 
 
 if context.is_offline_mode():

alembic_task/versions/05e1ecf9953a_.py

@@ -0,0 +1,54 @@
+"""empty message
+
+Revision ID: 05e1ecf9953a
+Revises: 22e7d7e0fa02
+Create Date: 2018-12-04 19:53:41.477085
+
+"""
+
+import sqlalchemy as sa
+from alembic import op
+
+# revision identifiers, used by Alembic.
+revision = "05e1ecf9953a"
+down_revision = "22e7d7e0fa02"
+branch_labels = None
+depends_on = None
+
+
+def upgrade():
+    op.create_table(
+        "task_annotations",
+        sa.Column("id", sa.Integer(), nullable=False),
+        sa.Column("project_id", sa.Integer(), nullable=True),
+        sa.Column("task_id", sa.Integer(), nullable=False),
+        sa.Column("annotation_type", sa.String(), nullable=False),
+        sa.Column("annotation_source", sa.String(), nullable=True),
+        sa.Column("updated_timestamp", sa.DateTime(), nullable=False),
+        sa.Column("properties", sa.JSON(), nullable=False),
+        sa.ForeignKeyConstraint(["project_id"], ["projects.id"]),
+        sa.ForeignKeyConstraint(
+            ["task_id", "project_id"],
+            ["tasks.id", "tasks.project_id"],
+            name="fk_task_annotations",
+        ),
+        sa.PrimaryKeyConstraint("id"),
+    )
+    op.create_index(
+        "idx_task_annotations_composite",
+        "task_annotations",
+        ["task_id", "project_id"],
+        unique=False,
+    )
+    op.create_index(
+        op.f("ix_task_annotations_project_id"),
+        "task_annotations",
+        ["project_id"],
+        unique=False,
+    )
+
+
+def downgrade():
+    op.drop_index(op.f("ix_task_annotations_project_id"), table_name="task_annotations")
+    op.drop_index("idx_task_annotations_composite", table_name="task_annotations")
+    op.drop_table("task_annotations")

alembic_task/versions/05f1b650ddbc_.py

@@ -0,0 +1,34 @@
+"""empty message
+
+Revision ID: 05f1b650ddbc
+Revises: 3b8b0956b217
+Create Date: 2022-08-17 15:58:37.118728
+
+"""
+
+import sqlalchemy as sa
+from alembic import op
+
+# revision identifiers, used by Alembic.
+revision = "05f1b650ddbc"
+down_revision = "3b8b0956b217"
+branch_labels = None
+depends_on = None
+
+
+def upgrade():
+    # ### commands auto generated by Alembic - please adjust! ###
+    op.create_table(
+        "banner",
+        sa.Column("id", sa.Integer(), nullable=False),
+        sa.Column("message", sa.String(), nullable=False),
+        sa.Column("visible", sa.Boolean(), nullable=False),
+        sa.PrimaryKeyConstraint("id"),
+    )
+    # ### end Alembic commands ###
+
+
+def downgrade():
+    # ### commands auto generated by Alembic - please adjust! ###
+    op.drop_table("banner")
+    # ### end Alembic commands ###

alembic_task/versions/068674f06b0f_.py

@@ -0,0 +1,42 @@
+"""empty message
+
+Revision ID: 068674f06b0f
+Revises: 0eee8c1abd3a
+Create Date: 2019-10-02 08:45:00.553060
+
+"""
+
+import sqlalchemy as sa
+from alembic import op
+
+# revision identifiers, used by Alembic.
+revision = "068674f06b0f"
+down_revision = "0eee8c1abd3a"
+branch_labels = None
+depends_on = None
+
+
+def upgrade():
+    # ### commands auto generated by Alembic - please adjust! ###
+    op.add_column("users", sa.Column("picture_url", sa.String(), nullable=True))
+    op.alter_column(
+        "users",
+        "expert_mode",
+        existing_type=sa.BOOLEAN(),
+        nullable=True,
+        existing_server_default=sa.text("false"),
+    )
+    # ### end Alembic commands ###
+
+
+def downgrade():
+    # ### commands auto generated by Alembic - please adjust! ###
+    op.alter_column(
+        "users",
+        "expert_mode",
+        existing_type=sa.BOOLEAN(),
+        nullable=False,
+        existing_server_default=sa.text("false"),
+    )
+    op.drop_column("users", "picture_url")
+    # ### end Alembic commands ###

alembic_task/versions/073a96d114ab_.py

@@ -0,0 +1,50 @@
+"""create workspaces_imagery table and drop imageryList column from workspaces
+
+Revision ID: 073a96d114ab
+Revises: 279f8d753529
+Create Date: 2025-09-09 07:58:35.746897
+
+"""
+
+import sqlalchemy as sa
+from alembic import op
+
+# revision identifiers, used by Alembic.
+revision = "073a96d114ab"
+down_revision = "279f8d753529"
+branch_labels = None
+depends_on = None
+
+
+def upgrade():
+    # ### commands auto generated by Alembic - please adjust! ###
+    op.create_table(
+        "workspaces_imagery",
+        sa.Column("workspace_id", sa.Integer(), nullable=False),
+        sa.Column("definition", sa.Unicode(), nullable=False),
+        sa.Column("modifiedAt", sa.DateTime(), nullable=False),
+        sa.Column("modifiedBy", sa.UUID(), nullable=False),
+        sa.Column("modifiedByName", sa.Unicode(), nullable=False),
+        sa.ForeignKeyConstraint(
+            ["workspace_id"],
+            ["workspaces.id"],
+        ),
+        sa.PrimaryKeyConstraint("workspace_id"),
+    )
+
+    with op.batch_alter_table("workspaces", schema=None) as batch_op:
+        batch_op.drop_column("imageryList")
+
+    # ### end Alembic commands ###
+
+
+def downgrade():
+    # ### commands auto generated by Alembic - please adjust! ###
+
+    with op.batch_alter_table("workspaces", schema=None) as batch_op:
+        batch_op.add_column(
+            sa.Column("imageryList", sa.VARCHAR(), autoincrement=False, nullable=True)
+        )
+
+    op.drop_table("workspaces_imagery")
+    # ### end Alembic commands ###