feat(gcp): composer + submodules refactor, hardening fixes, cross-cloud contract, serverless egress & CMEK

README.md

@@ -62,16 +62,19 @@ The folder `examples` contains the following Terraform implementation examples :
 | AWS   | [aws-databricks-uc-bootstrap](examples/aws-databricks-uc-bootstrap/)               | AWS UC                                                                                                                                                                                               |
 | AWS   | [aws-remote-backend-infra](examples/aws-remote-backend-infra/)                     | Simple example on remote backend                                                                                                                                                                     |
 | AWS   | [aws-workspace-config](examples/aws-workspace-config/)                             | Configure workspace objects                                                                                 |
-| GCP   | [gcp-sa-provisionning](examples/gcp-sa-provisionning/)                                                                         |    Provisionning of the identity with the permissions required to deploy on GCP.                                                                                                                                 |
-| GCP   | [gcp-basic](examples/gcp-basic/)                                                                         |    Workspace Deployment with managed vpc                                                                                                                               |
-| GCP   | [gcp-byovpc](examples/gcp-byovpc/)                                                                         |    Workspace Deployment with customer-managed vpc                                                                                                                              |
+| GCP   | [gcp-sa-provisioning](examples/gcp-sa-provisioning/)                                                     | Provisioning the identity (service account) with permissions required to deploy on GCP                                                                                                          |
+| GCP   | [gcp-basic](examples/gcp-basic/)                                                                         | Workspace deployment with Databricks-managed VPC                                                                                                                               |
+| GCP   | [gcp-byovpc](examples/gcp-byovpc/)                                                                       | Workspace deployment with customer-managed VPC (Terraform creates the VPC)                                                                                                                      |
+| GCP   | [gcp-existing-vpc](examples/gcp-existing-vpc/)                                                           | Workspace deployment into a pre-existing VPC                                                                                                                                                    |
+| GCP   | [gcp-with-psc-exfiltration-protection](examples/gcp-with-psc-exfiltration-protection/)                   | Workspace with PrivateLink (PSC), private DNS, and restricted egress (hub-and-spoke topology)                                                                                                  |
 ### Modules
 
 The folder `modules` contains the following Terraform modules :
 
 | Cloud | Module                                                                                                    | Description                                                                                                                                                                               |
 | ----- |-----------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
 | All   | [databricks-department-clusters](modules/databricks-department-clusters/)                                 | Terraform module that creates Databricks resources for a team                                                                                                                             |
+| All   | [databricks/security-settings](modules/databricks/security-settings/)                                     | Cloud-neutral workspace security settings: Compliance Security Profile, Enhanced Security Monitoring, automatic cluster update, and IP access lists                                      |
 | Azure | [adb-lakehouse](modules/adb-lakehouse/)                                                                   | Lakehouse terraform blueprints                                                                                                                                                            |
 | Azure | [adb-lakehouse-uc](modules/adb-lakehouse-uc/)                                                             | Provisioning Unity Catalog resources and accounts principals                                                                                                                              |
 | Azure | [adb-with-private-link-standard](modules/adb-with-private-link-standard/)                                 | Provisioning Databricks on Azure with Private Link - Standard deployment                                                                                                                  |
@@ -89,9 +92,13 @@ The folder `modules` contains the following Terraform modules :
 | AWS   | [aws-workspace-with-firewall](modules/aws-workspace-with-firewall/)                                       | Provisioning AWS Databricks E2 with an AWS Firewall                                                                                                                                       |
 | AWS   | [aws-exfiltration-protection](modules/aws-exfiltration-protection/)                                       | An implementation of [Data Exfiltration Protection on AWS](https://www.databricks.com/blog/2021/02/02/data-exfiltration-protection-with-databricks-on-aws.html)                           |
 | AWS   | aws-workspace-with-private-link                                                                           | Coming soon                                                                                                                                                                               |
-| GCP   | [gcp-sa-provisionning](modules/gcp-sa-provisionning/)                                                                                                |                  Provisions the identity (SA) with the correct permissions  |
-| GCP   | [gcp-workspace-basic](modules/gcp-workspace-basic/)                                                                                                |                  Provisions a workspace with managed VPC                                                                                                                                                          |
-| GCP   | [gcp-workspace-byovpc](modules/gcp-workspace-byovpc/)                                                                                                |                  Workspace with customer-managed VPC.                                                                                                                                                             |
+| GCP   | [gcp/databricks-workspace](modules/gcp/databricks-workspace/)                                            | Composer that orchestrates network, PSC, workspace, and DNS submodules based on scenario flags                                                                                                  |
+| GCP   | [gcp/network](modules/gcp/network/)                                                                      | VPC, subnet, router, NAT, peering, and shared-VPC binding (create or data-source lookup)                                                                                                       |
+| GCP   | [gcp/private-connectivity](modules/gcp/private-connectivity/)                                            | PSC endpoints (frontend, backend, hub-transit) and restricted-egress firewall rules                                                                                                            |
+| GCP   | [gcp/workspace](modules/gcp/workspace/)                                                                  | All databricks_mws_* resources: networks, workspaces, vpc_endpoint, private_access_settings                                                                                                    |
+| GCP   | [gcp/dns](modules/gcp/dns/)                                                                              | Private DNS zones (gcp.databricks.com, gcr.io, googleapis.com, pkg.dev) for restricted-egress workspaces                                                                                       |
+| GCP   | [gcp/service-account](modules/gcp/service-account/)                                                      | Service account with the IAM permissions required to provision Databricks workspaces                                                                                                          |
+| GCP   | [gcp/unity-catalog](modules/gcp/unity-catalog/)                                                          | Metastore, GCS bucket, storage credential, external location, and default catalog                                                                                                             |
 ### CI/CD pipelines
 
 The `cicd-pipelines` folder contains the following implementation examples of pipeline:

docs/cross-cloud-module-contract.md

@@ -0,0 +1,86 @@
+# Cross-Cloud Module Contract
+
+Normative conventions for `modules/<cloud>/` in this repository. The GCP tree
+is the reference implementation (PR #233). Azure and AWS refactors are
+reviewed against this document.
+
+## 1. Module slots
+
+Every cloud provides these modules, defined by responsibility:
+
+| Slot | Responsibility | GCP | Azure (dialect) | AWS (dialect) |
+|---|---|---|---|---|
+| `network` | Data-plane network: create or look up VPC/VNet, subnets, NAT, hub-spoke peering, shared-network bindings | `modules/gcp/network` | VNet + subnets + NSGs | VPC + subnets + NAT |
+| `private-connectivity` | Private endpoints to the Databricks control plane + egress firewall stack | PSC endpoints | Private Endpoints | VPC endpoints (PrivateLink) |
+| `workspace` | Everything registering the workspace with the Databricks control plane | `databricks_mws_*` | `azurerm_databricks_workspace` + access connector | `databricks_mws_*` |
+| `dns` | Private DNS zones/records for the private endpoints | Cloud DNS private + peering zones | Private DNS zones | Route 53 private hosted zones |
+| `databricks-workspace` | Composer: takes scenario flags, conditionally instantiates the slots (`network → private-connectivity → workspace → dns`) | done | future | future |
+| `serverless-connectivity` (reserved) | NCC: serverless private connectivity to customer resources | pending GCP release (near Public Preview, Jul 2026) | `databricks_mws_network_connectivity_config` | same |
+
+Cloud-neutral modules (identical resources on every cloud) live under
+`modules/databricks/` (first resident: `security-settings`).
+
+## 2. Frozen interface (identical names on every cloud)
+
+Composer variables: `prefix`, `workspace_name`, `databricks_account_id`,
+`tags`; `private_link_frontend`, `private_link_backend`,
+`private_access_only`, `restricted_egress`; `serverless_egress_mode`
+(`unmanaged|full|restricted`), `serverless_allowed_internet_destinations`,
+`serverless_allowed_storage_destinations`, `serverless_egress_enforcement`
+(`enforced|dry_run`); `cmek_managed_services_key_id`, `cmek_storage_key_id`
+(the key-reference format is per-cloud).
+
+The network-source variable uses the cloud's noun (`vpc_source` /
+`vnet_source`) and is an object with frozen keys and values: `{ spoke =
+databricks_managed | create | existing, hub = create | existing (optional;
+only settable when spoke is customer-managed) }` — `hub` is consumed only by
+egress-restricted topologies. The boolean `enable_hub_spoke_peering` (default
+true) is frozen alongside it.
+
+Composer outputs: `workspace_id`, `workspace_url`, `suffix`,
+`serverless_network_policy_id`, plus per-cloud network outputs following the
+noun table.
+
+Cloud dialect constraints are allowed where the platform demands them and
+must be enforced as preconditions with an explanatory error message (GCP:
+`private_link_frontend == private_link_backend`, because
+`mws_networks.vpc_endpoints` requires both endpoint references).
+
+## 3. Two-tier naming
+
+Tier 1 (frozen): everything in section 2, plus the vocabulary `hub`/`spoke`
+and `frontend`/`backend`.
+
+Tier 2 (native nouns): each cloud names cloud objects with its own noun.
+
+| Concept | GCP | Azure | AWS |
+|---|---|---|---|
+| Network | `*_vpc_*` | `*_vnet_*` | `*_vpc_*` |
+| Private endpoint mechanism | `psc` | `private_endpoint` | `vpc_endpoint` |
+| Region | `google_region` | `azure_region` | `aws_region` |
+| Resource container | `google_project` | `azure_resource_group` | (account-level, none) |
+| Endpoint subnet | `psc_subnet_cidr` | `private_endpoint_subnet_cidr` | `endpoint_subnet_cidrs` |
+| Endpoint IP outputs | `frontend_psc_ip_<spoke\|hub>` | `frontend_endpoint_ip_<spoke\|hub>` | n/a (ENI-based) |
+
+## 4. File shape
+
+Modules: one concern per `.tf` file; `versions.tf` declares
+`required_version` + `required_providers` floors only (`google >= 6.0`,
+`databricks >= 1.81.1`, `random >= 3.0`); no `provider {}` blocks.
+Examples: `versions.tf` (pessimistic pins `~> 6.17` / `~> 1.81`) +
+`providers.tf` + `main.tf` + `variables.tf` + `outputs.tf` +
+`terraform.tfvars` + `README.md` + `Makefile`.
+
+Every variable and output has a `description`. Module READMEs carry a
+`## Usage` HCL block above the terraform-docs markers.
+
+## 5. Validation & testing standard
+
+- Cross-variable rules live in the composer's `preconditions.tf` on a
+  `terraform_data` resource; the PR description documents the rule table.
+- `count`/`for_each` must never depend on apply-time values.
+- Each composer scenario has a positive fixture under `tests/<scenario>/`
+  that completes `terraform plan` offline; preconditions are covered by
+  `tests/negative-*` fixtures that fail plan with the expected message (at
+  minimum, every rule that guards a scenario flag combination).
+- Submodules carry per-scenario plan fixtures.

examples/gcp-basic/README.md

@@ -1,30 +1,36 @@
-# Provisioning Databricks workspace on GCP with managed VPC
-=========================
+# examples/gcp-basic — Databricks-managed VPC
 
-In this template, we show how to deploy a workspace with managed VPC.
+Calls `modules/gcp/databricks-workspace` with `vpc_source = { spoke = "databricks_managed" }`.
+The Databricks platform provisions the workspace VPC; you provide only the GCP
+project, region, and prefix.
 
+## Prerequisites
 
-## Requirements
-
-- You need to have run gcp-sa-provisionning and have a service account to fill in the variables.
-- If you want to deploy to a new project, you will need to grant the custom role generated in that template to the service acount in the new project.
-- The Service Account needs to be added as Databricks Admin in the account console
-
-## Run as an SA 
+- A GCP project with the Databricks platform onboarded
+- A service account with workspace-creator role (see `examples/gcp-sa-provisioning`)
+- Databricks account ID
 
-You can do the same thing by provisionning a service account that will have the same permissions - and associate the key associated to it.
+## Apply
 
+```bash
+terraform init
+terraform apply
+```
 
-## Run the tempalte
+## Migrating from the old example
 
-- You need to fill in the `variables.tf`
-- run `terraform init`
-- run `teraform apply`
+This example previously called `modules/gcp-workspace-basic`. State from the
+old apply does **not** migrate cleanly to the new composer because the
+`databricks_mws_workspaces` resource address differs. Re-apply on clean state.
 
 <!-- BEGIN_TF_DOCS -->
 ## Requirements
 
-No requirements.
+| Name | Version |
+|------|---------|
+| <a name="requirement_terraform"></a> [terraform](#requirement\_terraform) | >= 1.5 |
+| <a name="requirement_databricks"></a> [databricks](#requirement\_databricks) | ~> 1.81 |
+| <a name="requirement_google"></a> [google](#requirement\_google) | ~> 6.17 |
 
 ## Providers
 
@@ -34,7 +40,7 @@ No providers.
 
 | Name | Source | Version |
 |------|--------|---------|
-| <a name="module_gcp-basic"></a> [gcp-basic](#module\_gcp-basic) | github.com/databricks/terraform-databricks-examples/modules/gcp-workspace-basic | n/a |
+| <a name="module_workspace"></a> [workspace](#module\_workspace) | ../../modules/gcp/databricks-workspace | n/a |
 
 ## Resources
 
@@ -45,18 +51,17 @@ No resources.
 | Name | Description | Type | Default | Required |
 |------|-------------|------|---------|:--------:|
 | <a name="input_databricks_account_id"></a> [databricks\_account\_id](#input\_databricks\_account\_id) | Databricks Account ID | `string` | n/a | yes |
-| <a name="input_databricks_google_service_account"></a> [databricks\_google\_service\_account](#input\_databricks\_google\_service\_account) | Email of the service account used for deployment | `string` | n/a | yes |
-| <a name="input_delegate_from"></a> [delegate\_from](#input\_delegate\_from) | Identities to allow to impersonate created service account (in form of user:user.name@example.com, group:deployers@example.com or serviceAccount:sa1@project.iam.gserviceaccount.com) | `list(string)` | n/a | yes |
-| <a name="input_google_project"></a> [google\_project](#input\_google\_project) | Google project for VCP/workspace deployment | `string` | n/a | yes |
-| <a name="input_google_region"></a> [google\_region](#input\_google\_region) | Google region for VCP/workspace deployment | `string` | n/a | yes |
-| <a name="input_google_zone"></a> [google\_zone](#input\_google\_zone) | Zone in GCP region | `string` | n/a | yes |
-| <a name="input_prefix"></a> [prefix](#input\_prefix) | Prefix to use in generated VPC name | `string` | n/a | yes |
-| <a name="input_workspace_name"></a> [workspace\_name](#input\_workspace\_name) | Name of the workspace to create | `string` | n/a | yes |
+| <a name="input_databricks_google_service_account"></a> [databricks\_google\_service\_account](#input\_databricks\_google\_service\_account) | Service account email used for Databricks provider authentication | `string` | n/a | yes |
+| <a name="input_google_project"></a> [google\_project](#input\_google\_project) | GCP project where the workspace will be created | `string` | n/a | yes |
+| <a name="input_google_region"></a> [google\_region](#input\_google\_region) | GCP region for workspace deployment | `string` | n/a | yes |
+| <a name="input_google_zone"></a> [google\_zone](#input\_google\_zone) | GCP zone (used by the google provider) | `string` | n/a | yes |
+| <a name="input_prefix"></a> [prefix](#input\_prefix) | Prefix used to name generated resources | `string` | n/a | yes |
+| <a name="input_workspace_name"></a> [workspace\_name](#input\_workspace\_name) | Workspace name | `string` | n/a | yes |
 
 ## Outputs
 
 | Name | Description |
 |------|-------------|
-| <a name="output_databricks_host"></a> [databricks\_host](#output\_databricks\_host) | n/a |
-| <a name="output_databricks_token"></a> [databricks\_token](#output\_databricks\_token) | n/a |
+| <a name="output_workspace_id"></a> [workspace\_id](#output\_workspace\_id) | Databricks workspace ID |
+| <a name="output_workspace_url"></a> [workspace\_url](#output\_workspace\_url) | Databricks workspace URL |
 <!-- END_TF_DOCS -->