2.0.4 wip

Author: jeffreyaven

.github/workflows/release.yml

@@ -257,7 +257,7 @@ jobs:
 
             If you are currently using the Python package on PyPI, please migrate to this
             release. The Python package is now deprecated and will no longer receive updates:
-            https://pypi.org/project/stackql-deploy/
+            https://crates.io/crates/stackql-deploy
 
             The CLI interface is fully compatible — existing `stackql_manifest.yml` files and
             project layouts work without modification.

.gitignore

@@ -6,7 +6,9 @@ stackql*.zip
 stackql*.pkg
 stackql_history.txt
 stackql.log
+stackql-zip
 .env
 nohup.out
 contributors.csv
-.claude/
+.claude/
+nohup.out

CHANGELOG.md

@@ -1,5 +1,74 @@
 # Changelog
 
+## 2.0.4 (2026-03-18)
+
+### Identifier capture from `exists` queries
+
+The `exists` query can now return a named field (e.g. `vpc_id`) instead of `count`. The returned value is automatically captured as a resource-scoped variable (`{{ this.<field> }}`) and made available to all subsequent queries (`statecheck`, `exports`, `delete`) for that resource. This enables a two-step workflow where `exists` discovers the resource identifier and `statecheck` verifies its properties.
+
+- When `exists` returns `null` or empty for the captured field, the resource is treated as non-existent
+- Multiple rows from an `exists` (identifier pattern) or `exports` query is now a fatal error
+- After a `create`, the `exists` query is automatically re-run to capture the identifier for use in post-deploy `statecheck` and `exports` queries
+
+### `RETURNING *` identifier capture
+
+When a `create` statement includes `RETURNING *` and the response contains an `Identifier` field, it is automatically injected as `this.identifier` — skipping the post-create `exists` re-run and saving an API call per resource.
+
+### `return_vals` manifest field
+
+New optional `return_vals` field on resources to explicitly map fields from `RETURNING *` responses to resource-scoped variables:
+
+```yaml
+return_vals:
+  create:
+    - Identifier: identifier   # rename pattern
+    - ErrorCode                 # direct capture
+```
+
+If `return_vals` is specified but the field is missing from the response, the build fails.
+
+### `to_aws_tag_filters` template filter
+
+New AWS-specific Tera filter that converts `global_tags` (list of `Key`/`Value` pairs) to the AWS Resource Groups Tagging API `TagFilters` format:
+
+```sql
+AND TagFilters = '{{ global_tags | to_aws_tag_filters }}'
+```
+
+### YAML type preservation fix
+
+Fixed an issue where YAML string values that look like numbers (e.g. `IpProtocol: "-1"`) were being coerced to integers during JSON serialization. String types declared in YAML are now preserved through to the rendered query.
+
+### Teardown improvements
+
+- Teardown no longer retries exports queries that return empty results — missing exports are set to `<unknown>` and teardown continues best-effort
+- Post-delete existence checks accept the first empty response instead of retrying, reducing teardown time significantly
+
+### AWS starter template updated
+
+The `stackql-deploy init --provider aws` starter template now uses:
+- `awscc` (Cloud Control) provider instead of `aws`
+- CTE + INNER JOIN exists pattern with `to_aws_tag_filters`
+- `AWS_POLICY_EQUAL` for statecheck tag comparison
+- `this.<field>` identifier capture pattern
+- `RETURNING *` on create statements
+- `stackql:stack-name` / `stackql:stack-env` / `stackql:resource-name` tag taxonomy
+
+### AWS VPC Web Server example
+
+Complete rewrite of the `examples/aws/aws-vpc-webserver` stack (renamed from `aws-stack`) using the `awscc` provider exclusively. Includes 10 resources demonstrating all query patterns: tag-based discovery, identifier capture, property-level statechecks, PatchDocument updates, and the `to_aws_tag_filters` filter.
+
+### Patch Document Test example
+
+New `examples/aws/patch-doc-test` example demonstrating the Cloud Control API `UPDATE` workflow with `PatchDocument` — deploy an S3 bucket, modify its versioning config in the manifest, and re-deploy to apply the update.
+
+### Other changes
+
+- Fixed `init` command missing `--env` argument (defaulting to `dev`)
+- Added `debug` log import to build command
+- Debug logging now shows full `RETURNING *` payloads
+- Documentation updates: `resource-query-files.md`, `template-filters.md`, `manifest-file.md`, and AWS template library
+
 ## 2.0.0 (2026-03-14)
 
 ### Initial Rust Release

Cargo.lock

@@ -1,6 +1,6 @@
 [package]
 name = "stackql-deploy"
-version = "2.0.3"
+version = "2.0.4"
 edition = "2021"
 rust-version = "1.75"
 description = "Infrastructure-as-code framework for declarative cloud resource management using StackQL"

Cargo.toml

@@ -0,0 +1,225 @@
+# Resource Processing Flows
+
+This document describes every code path in the `build`, `test`, and `teardown` commands based on which anchors are present in a resource's `.iql` file.
+
+## Anchor Reference
+
+| Anchor | Purpose |
+|--------|---------|
+| `exists` | Check if resource exists (returns `count` or a named field) |
+| `statecheck` | Verify resource properties match desired state |
+| `create` | Create the resource |
+| `update` | Update the resource (patch document) |
+| `createorupdate` | Always execute (skip exists/statecheck) |
+| `exports` | Extract values for downstream resources |
+| `delete` | Remove the resource |
+
+## Exists Query Variants
+
+The `exists` query has two modes based on the returned column name:
+
+**Count mode** — returns `count`, existence is `count > 0`:
+```sql
+SELECT count(*) as count FROM awscc.s3.buckets WHERE Identifier = '{{ bucket_name }}' AND region = '{{ region }}';
+```
+
+**Field capture mode** — returns a named field (e.g. `vpc_id`), captured as `this.<field>` for downstream queries:
+```sql
+SELECT split_part(ResourceARN, '/', 2) as vpc_id FROM awscc.tagging.tagged_resources WHERE ...;
+```
+
+When `null` or empty is returned in field capture mode, the resource is treated as non-existent.
+
+---
+
+## Build Flows
+
+### Flow A: `createorupdate` + `exports`
+
+Used for resources that should always be applied (e.g. routes, gateway attachments).
+
+```mermaid
+graph LR
+    A[createorupdate] --> B[exports proxy?]
+    B -->|rows returned| C[done ✓]
+    B -->|no rows| D[FAIL ✗]
+```
+
+**Anchors:** `createorupdate`, optionally `exports`
+**Examples:** `example_inet_route`, `example_inet_gw_attachment` (old pattern)
+
+---
+
+### Flow B: `exists`(count) + `statecheck` + `create`/`update` + `exports`
+
+Classic pattern for resources identified by a known name/identifier.
+
+```mermaid
+graph LR
+    A[exists count] -->|0| B[create]
+    A -->|>0| C[statecheck]
+    C -->|pass| D[exports]
+    C -->|fail| E[update]
+    B --> F[post-create exists]
+    F --> G[statecheck]
+    G --> D
+    E --> H[post-update exists]
+    H --> I[statecheck]
+    I --> D
+    D --> J[done ✓]
+```
+
+**Anchors:** `exists`(count), `statecheck`, `create`, optionally `update`, optionally `exports`
+**Examples:** `databricks_account_credentials`, `aws_s3_workspace_bucket_policy`
+
+---
+
+### Flow C: `exists`(field) + `statecheck` + `create` + `exports`
+
+Pattern for awscc resources using tagging for identification. The exists query returns a resource-specific field (e.g. `vpc_id`) which is captured as `this.<field>` and used in statecheck and exports.
+
+```mermaid
+graph LR
+    A[exists field] -->|null| B[create]
+    A -->|value| C["this.field = value"]
+    C --> D["statecheck(this.field)"]
+    D -->|pass| E["exports(this.field)"]
+    D -->|fail| F["update(this.field)"]
+    B --> G[post-create exists]
+    G --> H["this.field = value"]
+    H --> I["statecheck(this.field)"]
+    I --> E
+    F --> J[post-update exists]
+    J --> K["statecheck(this.field)"]
+    K --> E
+    E --> L[done ✓]
+```
+
+**Anchors:** `exists`(field), `statecheck`, `create`, optionally `update`, `exports`
+**Examples:** `example_vpc`, `example_subnet`, `example_inet_gateway`, `example_route_table`, `example_security_group`, `example_web_server`
+
+---
+
+### Flow D: `exists`(count) + `exports`-as-proxy (no statecheck)
+
+The exports query doubles as a statecheck — if it returns rows, the resource is in the desired state.
+
+```mermaid
+graph LR
+    A[exists count] -->|0| B[create]
+    A -->|>0| C[exports proxy]
+    C -->|rows returned| D[done ✓]
+    C -->|no rows| E[update]
+    B --> F[post-create exists]
+    F --> G[exports proxy]
+    G --> D
+    E --> H[post-update exists]
+    H --> I[exports proxy]
+    I --> D
+```
+
+**Anchors:** `exists`(count), `create`, optionally `update`, `exports`
+**Examples:** `aws_s3_workspace_bucket`, `databricks_storage_configuration`
+
+---
+
+### Flow E: `exists`(field) + `exports` (no statecheck)
+
+Field capture from exists, exports uses `this.<field>`. Exports acts as statecheck proxy.
+
+```mermaid
+graph LR
+    A[exists field] -->|null| B[create]
+    A -->|value| C["this.field = value"]
+    C --> D["exports proxy(this.field)"]
+    D -->|rows returned| E[done ✓]
+    D -->|no rows| F[update]
+    B --> G[post-create exists]
+    G --> H["this.field = value"]
+    H --> I["exports proxy(this.field)"]
+    I --> E
+```
+
+**Anchors:** `exists`(field), `create`, `exports`
+**Examples:** `example_subnet_rt_assn`
+
+---
+
+### Flow F: `exists`(field) only (no statecheck, no exports)
+
+Minimal pattern — exists confirms the resource, the captured field is the only output.
+
+```mermaid
+graph LR
+    A[exists field] -->|null| B[create]
+    A -->|value| C[done ✓]
+    B --> D[post-create exists]
+    D -->|value| C
+```
+
+**Anchors:** `exists`(field), `create`
+**Examples:** Simple resources where existence is sufficient
+
+---
+
+## Test Flows
+
+Test runs the same exists → statecheck/exports-proxy sequence as build, but never creates or updates. If a resource doesn't exist or fails statecheck, the test fails.
+
+```mermaid
+graph LR
+    A[exists] -->|not found| B[FAIL ✗]
+    A -->|found| C[statecheck or exports proxy]
+    C -->|pass| D[exports]
+    C -->|fail| B
+    D --> E[done ✓]
+```
+
+---
+
+## Teardown Flows
+
+Teardown processes resources in reverse manifest order. First collects all exports (running exists to capture `this.*` fields), then deletes.
+
+```mermaid
+graph LR
+    A[collect exports phase] --> B[exists per resource]
+    B --> C[exports per resource]
+    C --> D[reverse order delete phase]
+    D --> E[exists check]
+    E -->|found| F[delete]
+    E -->|not found| G[skip]
+    F --> H[post-delete exists]
+    H -->|gone| I[done ✓]
+    H -->|still there| I
+```
+
+During teardown export collection, missing exports are set to `<unknown>` rather than failing — the stack may be partially deployed.
+
+---
+
+## `this.*` Variable Lifecycle
+
+When an exists query returns a named field (not `count`), the value is captured as a resource-scoped variable:
+
+| Phase | Variable | Available to |
+|-------|----------|-------------|
+| exists returns `vpc_id` | `this.vpc_id` → `example_vpc.vpc_id` | statecheck, exports, delete for this resource |
+| exports returns `vpc_id` | `vpc_id` and `example_vpc.vpc_id` | all subsequent resources |
+
+The `this.*` prefix is syntactic sugar — `{{ this.vpc_id }}` in `example_vpc`'s queries is preprocessed to `{{ example_vpc.vpc_id }}`.
+
+---
+
+## RETURNING * Optimization
+
+When a `create` query includes `RETURNING *`, the Cloud Control API returns the operation metadata immediately. If `return_vals` is configured in the manifest, specified fields are captured as `this.*` variables, eliminating the need for a post-create exists re-run.
+
+```yaml
+resources:
+  - name: example_vpc
+    return_vals:
+      create:
+        - Identifier: vpc_id    # rename Identifier → this.vpc_id
+        - ErrorCode             # capture as this.ErrorCode
+```