Skip to content

feat: user-defined sql scripts#339

Open
jason-lynch wants to merge 9 commits intomainfrom
feat/PLAT-543/user-defined-scripts
Open

feat: user-defined sql scripts#339
jason-lynch wants to merge 9 commits intomainfrom
feat/PLAT-543/user-defined-scripts

Conversation

@jason-lynch
Copy link
Copy Markdown
Member

@jason-lynch jason-lynch commented Apr 13, 2026

Summary

Adds the ability to run arbitrary SQL at two different points in the database creation process:

  • post_init: This script runs immediately after the database is created, before any users are created.
  • post_database_create: This script runs immediately after Spock is initialized, before any subscriptions have been created.

This feature can be used to perform first-time setup for roles and similar operations.

Changes

  • Added a scripts field to the database spec:
{
  // ...
  "scripts": {
    "post_init": [
      "CREATE ROLE read_only NOLOGIN",
      "...",
      "..."
    ],
    "post_database_create": [
      "ALTER DEFAULT PRIVILEGES FOR ROLE admin GRANT USAGE ON SCHEMAS TO read_only",
      "...",
      "..."
    ]
  }
}

Testing

Basic usage:

cp1-req create-database <<EOF | cp-follow-task
{
  "id": "storefront",
  "spec": {
    "database_name": "storefront",
    "database_users": [
      {
        "username": "admin",
        "password": "password",
        "db_owner": true,
        "attributes": ["SUPERUSER", "LOGIN"]
      },
      {
        "username": "app",
        "password": "password",
        "roles": ["foo_role"]
      }
    ],
    "port": 0,
    "patroni_port": 0,
    "nodes": [
      { "name": "n1", "host_ids": ["host-1"] },
      { "name": "n2", "host_ids": ["host-2"] }
    ],
    "scripts": {
      "post_init": [
        "CREATE ROLE foo_role NOLOGIN"
      ],
      "post_database_create": [
        "CREATE TABLE foo (id int primary key, val text)",
        "INSERT INTO foo (id, val) VALUES (1, 'foo')"
      ]
    }
  }
}
EOF

# validate that admin has foo_role role
cp-psql -U admin -i storefront-n1-689qacsi -- -c "SELECT oid, rolname FROM pg_roles WHERE pg_has_role('admin', oid, 'member') and rolname = 'foo_role'"
cp-psql -U admin -i storefront-n2-9ptayhma -- -c "SELECT oid, rolname FROM pg_roles WHERE pg_has_role('admin', oid, 'member') and rolname = 'foo_role'"

# validate that foo table is populated
cp-psql -U admin -i storefront-n1-689qacsi -- -c "SELECT val FROM foo WHERE id = 1"
cp-psql -U admin -i storefront-n2-9ptayhma -- -c "SELECT val FROM foo WHERE id = 1"

# validate that replication works for table foo
cp-psql -U admin -i storefront-n1-689qacsi -- -c "INSERT INTO foo (id, val) VALUES (2, 'bar')"
cp-psql -U admin -i storefront-n2-9ptayhma -- -c "INSERT INTO foo (id, val) VALUES (3, 'baz')."
cp-psql -U admin -i storefront-n2-9ptayhma -- -c "SELECT val FROM foo WHERE id = 2"
cp-psql -U admin -i storefront-n1-689qacsi -- -c "SELECT val FROM foo WHERE id = 3"

After that, you can try changing your statements and validating that the changed statements do not run.

You can also try specifying an invalid statement in post_init for a new database, like CREATE DATABASE foo, and observe that the database creation process fails with an error about how CREATE DATABASE is not allowed within a transaction. Then remove or replace the bad statement, submit an update-database request, and verify that the creation process resumes where it left off and executes your updated statement.

You can also look at the included examples in the API reference, docs, and tests for other things to try.

Notes for Reviewers

Roles added in the pre_init field will not carry over to new nodes. This will be addressed in a subsequent PR.

PLAT-543

Adds a `Variables` field to the `resource.Context` type.  Variables are
useful for dynamic variables that do not fit neatly within a single
resource attribute, and that may have effects that across multiple
resources.

A subsequent commit will use this field for a variable that indicates
whether the database has already been created.

PLAT-543
Adds a `NotCreated` field to track when a database has not yet been
created. This field is set to `true` when the `database.Database` record
is first created. It is changed to `false` after it becomes available
for the first time. This means that if a `create-database` operation
fails, and a subsequent `update-database` succeeds, this flag is only
set to `false` after the `update-database` completes.

Note that this field is implemented in the negative, as `NotCreated`
instead of `Created`, so that it will default to `false` for existing
databases. This lets us avoid a migration.

PLAT-543
Plumbs the database `NotCreated` field into all of the workflows via
`resource.Variables` in the `resource.Context`.

PLAT-543
Adds a `scripts` field to the database spec in the API and in the
`database` package. The contents of this field are validated to be
syntactically correct SQL.

PLAT-543
@coderabbitai
Copy link
Copy Markdown

coderabbitai bot commented Apr 13, 2026

Warning

Rate limit exceeded

@jason-lynch has exceeded the limit for the number of commits that can be reviewed per hour. Please wait 10 minutes and 42 seconds before requesting another review.

Your organization is not enrolled in usage-based pricing. Contact your admin to enable usage-based pricing to continue reviews beyond the rate limit, or try again in 10 minutes and 42 seconds.

⌛ How to resolve this issue?

After the wait time has elapsed, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

We recommend that you space out your commits to avoid hitting the rate limit.

🚦 How do rate limits work?

CodeRabbit enforces hourly rate limits for each developer per organization.

Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout.

Please see our FAQ for further information.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 6397cfab-bb5f-4964-b66c-6139fdf42f22

📥 Commits

Reviewing files that changed from the base of the PR and between 2b652a3 and f3805f2.

📒 Files selected for processing (20)
  • changes/unreleased/Added-20260412-202528.yaml
  • docs/using/create-db.md
  • e2e/scripts_test.go
  • server/internal/database/instance_resource.go
  • server/internal/database/operations/common.go
  • server/internal/database/operations/restore_database.go
  • server/internal/database/orchestrator.go
  • server/internal/database/postgres_database.go
  • server/internal/database/script.go
  • server/internal/database/script_result_store.go
  • server/internal/database/service.go
  • server/internal/database/store.go
  • server/internal/orchestrator/swarm/orchestrator.go
  • server/internal/orchestrator/systemd/orchestrator.go
  • server/internal/workflows/activities/activities.go
  • server/internal/workflows/activities/get_instance_resources.go
  • server/internal/workflows/activities/get_restore_resources.go
  • server/internal/workflows/activities/get_script_results.go
  • server/internal/workflows/common.go
  • server/internal/workflows/plan_restore.go
📝 Walkthrough

Walkthrough

This pull request adds support for user-defined SQL scripts in database creation. It introduces new API types for defining scripts (post_init and post_database_create), script execution logic with state tracking, and workflow integration to manage script execution across database instances. Supporting infrastructure updates include validation, storage, and resource context variables.

Changes

Cohort / File(s) Summary
API Design & DSL
api/apiv1/design/database.go, changes/unreleased/Added-*.yaml
Added SQLScript and DatabaseScripts types defining two script hooks (post_init, post_database_create). Extended DatabaseSpec with optional scripts field. Updated CreateDatabaseRequest example to demonstrate script usage.
API Validation & Conversion
api/apiv1/validate.go, api/apiv1/validate_test.go, api/apiv1/convert.go
Added strict SQL parsing validation via postgresparser for script statements with per-statement error reporting. Implemented conversion functions to map scripts between API and internal representations.
Handler Integration
api/apiv1/post_init_handlers.go
Updated database lifecycle handlers to pass persisted Database objects (instead of specs) to workflow service methods.
Core Database Models
server/internal/database/database.go, server/internal/database/spec.go, server/internal/database/database_store.go
Added NotCreated flag to track initial database creation state. Introduced ScriptStatements type with PostInit and PostDatabaseCreate statement arrays. Extended Database and storage layer with script and NotCreated fields. Added Variables() method exposing database_not_created variable.
Script Execution Framework
server/internal/database/script.go, server/internal/database/script_result_store.go
Introduced Script, ScriptName, and ScriptResult types. Implemented ExecuteScript for SQL execution with precondition checking, transaction management, and result persistence. Added script store for etcd-backed script result storage.
Database Instance & Resource Models
server/internal/database/instance_resource.go, server/internal/database/postgres_database.go, server/internal/database/operations/common.go, server/internal/database/operations/restore_database.go
Added PostInit and PostDatabaseCreate script fields to instance and database resources. Updated resource generation to include DatabaseID and scripts. Integrated script execution into resource initialization and creation workflows.
Orchestration Interface
server/internal/database/orchestrator.go, server/internal/orchestrator/swarm/orchestrator.go, server/internal/orchestrator/systemd/orchestrator.go
Updated orchestrator interface GenerateInstanceResources to accept scripts parameter. Implemented script injection into generated instance resources across swarm and systemd backends.
Database Service Layer
server/internal/database/service.go, server/internal/database/store.go
Added GetScriptResult and UpdateScriptResult methods. Integrated script result store initialization. Enhanced database creation with NotCreated tracking and cleanup on state transitions.
Workflow Core & Variables
server/internal/resource/resource.go, server/internal/resource/resource_test.go, server/internal/workflows/common.go
Introduced Variables context type and VariableFromContext function for resource-scoped variable access. Threaded variables through workflow orchestration. Added getScripts helper to load and evaluate script results.
Workflow Activities
server/internal/workflows/activities/get_script_results.go, server/internal/workflows/activities/apply_event.go, server/internal/workflows/activities/get_instance_resources.go, server/internal/workflows/activities/get_restore_resources.go, server/internal/workflows/activities/executor.go, server/internal/workflows/activities/activities.go
Implemented new GetScriptResults activity for fetching script execution outcomes. Extended ApplyEventInput and resource generation inputs with scripts/variables parameters. Simplified executor by removing redundant context construction.
Workflow Orchestration
server/internal/workflows/service.go, server/internal/workflows/update_database.go, server/internal/workflows/delete_database.go, server/internal/workflows/pgbackrest_restore.go, server/internal/workflows/refresh_current_state.go, server/internal/workflows/plan_restore.go
Changed service entrypoints to accept Database objects instead of specs, enabling variable capture via db.Variables(). Added variables threading throughout workflow inputs (UpdateDatabaseInput, DeleteDatabaseInput, PgBackRestRestoreInput, RefreshCurrentStateInput). Enhanced resource restoration to include DatabaseID.
Testing & Documentation
e2e/scripts_test.go, docs/using/create-db.md
Added end-to-end test demonstrating script execution during initial creation and immutability on subsequent updates. Documented new scripts field with execution timing, limits (256 statements, 1024 chars per statement), and curl examples.
Dependencies & Notices
go.mod, NOTICE.txt
Updated Go version to 1.25.6. Added direct dependency github.com/valkdb/postgresparser (v1.1.9) with indirect dependencies github.com/antlr4-go/antlr/v4 and golang.org/x/exp. Updated license attributions.

Poem

🐰 Hops of joy for scripts so grand,
SQL statements, perfectly planned,
Post-init and post-create run true,
Databases born with code that's new!
Variables dance through workflows' way,
Making databases work today! 🌱

🚥 Pre-merge checks | ✅ 2 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 6.67% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (2 passed)
Check name Status Explanation
Title check ✅ Passed The title 'feat: user-defined sql scripts' directly describes the main feature added in this PR, following conventional commit format and clearly summarizing the primary change.
Description check ✅ Passed The pull request description follows the template structure with comprehensive summary, detailed changes, testing instructions, and reviewer notes.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch feat/PLAT-543/user-defined-scripts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@jason-lynch
Copy link
Copy Markdown
Member Author

@coderabbitai review

@coderabbitai
Copy link
Copy Markdown

coderabbitai bot commented Apr 13, 2026

✅ Actions performed

Review triggered.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@codacy-production
Copy link
Copy Markdown

codacy-production bot commented Apr 13, 2026

Up to standards ✅

🟢 Issues 2 medium

Results:
2 new issues

Category Results
Complexity 2 medium

View in Codacy

🟢 Metrics 43 complexity · -7 duplication

Metric Results
Complexity 43
Duplication -7

View in Codacy

TIP This summary will be updated as you push new changes. Give us feedback

Copy link
Copy Markdown

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 3

🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@server/internal/database/script.go`:
- Around line 134-136: The result struct's Error field isn't cleared on success,
so previous failure text can persist; in the function where you set
result.CompletedAt and result.Succeeded (look for the block assigning
result.CompletedAt = time.Now() and result.Succeeded = len(errs) == 0), update
it to reset result.Error to an empty string when result.Succeeded is true (and
otherwise populate it from errs as appropriate). Ensure you reference and mutate
the same result variable (e.g., result.Error = "" when len(errs) == 0) so a
successful rerun clears stale error text.
- Around line 98-101: Before starting the SQL transaction, gate execution by
inspecting the persisted result from svc.GetScriptResult: check the returned
result's Succeeded flag (from the GetScriptResult call using script.DatabaseID,
script.Name, script.NodeName) and if it is true, return early (no-op) so the
non-idempotent SQL is not re-run; only proceed to BeginTx/transaction and
subsequent store-update if the persisted result indicates not succeeded. Ensure
you use the fetched result (not the possibly stale script.Succeeded) for this
check and handle any nil/absent result appropriately.
- Around line 121-123: The code currently embeds the raw SQL variable statement
into the wrapped error and persists it to result.Error (via fmt.Errorf("failed
to execute statement '%s': %w", statement, err) and result.Error = err.Error()),
which can leak secrets; change the wrapping so it does not include the raw SQL
(e.g. use fmt.Errorf("failed to execute statement: %w", execErr) or a redacted
message) and ensure result.Error stores a non-sensitive, generic or sanitized
message (or a fingerprint/hash of the statement) instead of the full statement;
update places that append to errs so they append the sanitized/wrapped error
(referencing variables statement, execErr/err, result.Error, and errs).
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 3ce6e556-735b-4e3c-b8b7-1d4ccd394ab6

📥 Commits

Reviewing files that changed from the base of the PR and between b79ea7b and 2b652a3.

⛔ Files ignored due to path filters (10)
  • api/apiv1/gen/control_plane/service.go is excluded by !**/gen/**
  • api/apiv1/gen/http/control_plane/client/encode_decode.go is excluded by !**/gen/**
  • api/apiv1/gen/http/control_plane/client/types.go is excluded by !**/gen/**
  • api/apiv1/gen/http/control_plane/server/encode_decode.go is excluded by !**/gen/**
  • api/apiv1/gen/http/control_plane/server/types.go is excluded by !**/gen/**
  • api/apiv1/gen/http/openapi.json is excluded by !**/gen/**
  • api/apiv1/gen/http/openapi.yaml is excluded by !**/gen/**
  • api/apiv1/gen/http/openapi3.json is excluded by !**/gen/**
  • api/apiv1/gen/http/openapi3.yaml is excluded by !**/gen/**
  • go.sum is excluded by !**/*.sum
📒 Files selected for processing (39)
  • NOTICE.txt
  • api/apiv1/design/database.go
  • changes/unreleased/Added-20260412-202528.yaml
  • docs/using/create-db.md
  • e2e/scripts_test.go
  • go.mod
  • server/internal/api/apiv1/convert.go
  • server/internal/api/apiv1/post_init_handlers.go
  • server/internal/api/apiv1/validate.go
  • server/internal/api/apiv1/validate_test.go
  • server/internal/database/database.go
  • server/internal/database/database_store.go
  • server/internal/database/instance_resource.go
  • server/internal/database/operations/common.go
  • server/internal/database/operations/restore_database.go
  • server/internal/database/orchestrator.go
  • server/internal/database/postgres_database.go
  • server/internal/database/script.go
  • server/internal/database/script_result_store.go
  • server/internal/database/service.go
  • server/internal/database/spec.go
  • server/internal/database/store.go
  • server/internal/orchestrator/swarm/orchestrator.go
  • server/internal/orchestrator/systemd/orchestrator.go
  • server/internal/resource/resource.go
  • server/internal/resource/resource_test.go
  • server/internal/workflows/activities/activities.go
  • server/internal/workflows/activities/apply_event.go
  • server/internal/workflows/activities/executor.go
  • server/internal/workflows/activities/get_instance_resources.go
  • server/internal/workflows/activities/get_restore_resources.go
  • server/internal/workflows/activities/get_script_results.go
  • server/internal/workflows/common.go
  • server/internal/workflows/delete_database.go
  • server/internal/workflows/pgbackrest_restore.go
  • server/internal/workflows/plan_restore.go
  • server/internal/workflows/refresh_current_state.go
  • server/internal/workflows/service.go
  • server/internal/workflows/update_database.go

Adds script entities to the `database` package, including utilities and
types that can be used by resources to execute scripts at different
points in the database creation process, as well as storage for script
results.

PLAT-543
Adds scripts to two resources in the `database` package:

- `InstanceResource.PostInit`: executes immediately after connecting to
  the database for the first time, before any users are created.
- `PostgresDatabaseResource.PostDatabaseCreate`: executes immediately
  after Spock is initialized.

PLAT-543
Adds an E2E test to exercise the new `scripts` field.

PLAT-543
@jason-lynch jason-lynch force-pushed the feat/PLAT-543/user-defined-scripts branch from 2b652a3 to 71093cc Compare April 13, 2026 12:32
@jason-lynch jason-lynch marked this pull request as ready for review April 13, 2026 16:16
@jason-lynch jason-lynch force-pushed the feat/PLAT-543/user-defined-scripts branch from 71093cc to f3805f2 Compare April 13, 2026 17:00
return err
}
if !ok {
return nil
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thoughts on explicitly logging when scripts are NOT executed? I think this can only happen during update-database, which might be valuable to see in the logs. I'm not sure.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a specific use case that you're thinking of when those logs would be helpful?

As it is now, these are executed only once per node/database during the database's lifetime. They can execute during update-database if you're using update-database to retry a create-database that failed before the scripts executed successfully. If we were to log on non-executions, you'd see "not executing scripts" on every update or restore operation after that point.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If there are too many then the message wouldn't be useful. I was just thinking that people who don't read the docs might be confused why the script isn't running.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants