Skip to content

Add AKS Automatic custom virtual network quickstart samples - #490

Open
Allen Sudbring (asudbring) wants to merge 6 commits into
Azure:masterfrom
asudbring:asudbring/101-aks-automatic-custom-network
Open

Add AKS Automatic custom virtual network quickstart samples#490
Allen Sudbring (asudbring) wants to merge 6 commits into
Azure:masterfrom
asudbring:asudbring/101-aks-automatic-custom-network

Conversation

@asudbring

Copy link
Copy Markdown
Contributor

Summary

Adds four Terraform quickstart samples that deploy AKS Automatic clusters into a
custom (bring-your-own) virtual network, covering both a public and a private
API server, in both the AzureRM and AzAPI providers.

Sample Provider Cluster resource API server
quickstart/101-aks-automatic-custom-network azurerm ~>5.0 azurerm_kubernetes_automatic_cluster Public
quickstart/101-aks-automatic-custom-network-azapi azapi ~>2.0 azapi_resource Public
quickstart/101-aks-automatic-private-custom-network azurerm ~>5.0 azurerm_kubernetes_automatic_cluster Private
quickstart/101-aks-automatic-private-custom-network-azapi azapi ~>2.0 azapi_resource Private

Both providers are covered per the review feedback on #487. The AzureRM samples
use the native azurerm_kubernetes_automatic_cluster resource; the AzAPI samples
are for cases where you need direct control over the managed cluster API payload.

Each sample creates a virtual network with a subnet delegated to the cluster API
server, a user node subnet, and a managed system node subnet, plus a
user-assigned managed identity granted Network Contributor on the virtual
network. The readmes cross-link the four samples.

Testing

All four samples were deployed and torn down against a live Azure subscription
before submission.

Step AzureRM public AzAPI public AzureRM private AzAPI private
terraform fmt -check -diff no changes no changes no changes no changes
terraform validate success success success success
terraform apply 10 added, 13m19s 10 added, 18m04s 10 added, 11m47s 10 added, 13m54s
Verification sku.name = Automatic, identity = UserAssigned, API server / node / system node subnet IDs matched, one Network Contributor assignment same same same
Private cluster false false true true
terraform plan -detailed-exitcode (post-apply) no changes no changes no changes no changes
terraform destroy 10 destroyed 10 destroyed 10 destroyed 10 destroyed
Teardown check resource group and node resource group confirmed deleted same same same

Region: westus2. Note for anyone reproducing — eastus currently returns
AKSCapacityHeavyUsage for API Server VNet Integration, so testing moved to
westus2. The samples still default to eastus.

The private AzAPI sample sets AzAPI create/delete timeouts of 2h because a
default-timeout create returned before Azure finished provisioning, leaving a
successful deployment untracked in state.

TestRecord.md is intentionally omitted for all four — the CI pipeline generates it.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@asudbring
Allen Sudbring (asudbring) marked this pull request as draft August 5, 2026 16:59
@asudbring
Allen Sudbring (asudbring) marked this pull request as ready for review August 5, 2026 17:32
@asudbring

Copy link
Copy Markdown
Contributor Author

Steven Ma (@stevenjma)
Can you review and merge?

Thanks!

stemaMSFT

This comment was marked as duplicate.

stemaMSFT

This comment was marked as duplicate.

@stemaMSFT Steven (stemaMSFT) left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The four-sample matrix is useful and the live apply/no-op-plan/destroy coverage is strong. I found several issues to address before merge:

  • All four samples default to eastus, but every successful deployment used westus2 because eastus returned AKSCapacityHeavyUsage for API Server VNet Integration. The literal copy/paste path therefore uses the known failing configuration. Please use the validated region as the default, or require location and put an explicit tested-region override beside the commands in every README.
  • Both AzAPI variants accept cluster_sku_name = "Base", allowing these AKS Automatic quickstarts to create a different product mode. Please hard-code Automatic and remove the variable/README rows.
  • The private-cluster text says the API server is reachable only from inside the VNet. More precisely, it uses a private IP and requires private network connectivity and DNS; peering, VPN, or ExpressRoute can provide access from outside that VNet. Please also explain that enabling a public FQDN changes DNS naming, not endpoint routability.

Please also document subnet containment/non-overlap/sizing constraints, the blast radius of VNet-scoped Network Contributor when copied to a shared VNet, and that ignore_changes = [delegation] hides delegation drift. These are guardrails for a BYO-network sample and do not require redesigning the standalone examples.

Default to the validated westus2 region, hard-code the Automatic SKU in the AzAPI variants, clarify private cluster reachability, and document BYO-network guardrails.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@asudbring

Copy link
Copy Markdown
Contributor Author

Thanks for the review Steven (@stemaMSFT) — all points are addressed in d24368bf.

1. Samples defaulted to eastus but were validated in westus2

All four samples now default to resource_group_location = "westus2", the region they were actually deployed and destroyed in. Each README carries a note beside the commands:

The default location is westus2 because that's the region these samples were validated in. At the time of testing, eastus returned AKSCapacityHeavyUsage for API Server VNet Integration. Set resource_group_location to deploy elsewhere.

The literal copy/paste path now uses the working configuration.

2. AzAPI variants accepted cluster_sku_name = "Base"

Hard-coded in both AzAPI samples. sku.name is the literal "Automatic", and the cluster_sku_name variable and README rows are removed. Same change applied to #487.

3. Private-cluster reachability wording

Rewritten in both private samples, in the README prose and the inline main.tf comments:

Because the cluster is private, the API server endpoint is assigned a private IP address in the virtual network instead of a public one. Reaching it requires private network connectivity to that virtual network and DNS resolution of the private FQDN, so access isn't limited to clients inside the virtual network itself: virtual network peering, a VPN gateway, or Azure ExpressRoute can all provide a path from outside it. Enabling a public FQDN changes only how the cluster is named in DNS. It doesn't make the private endpoint routable from the internet.

4. BYO-network guardrails

Added a Network guardrails section to all four READMEs, covering the four points you raised:

  • Subnet containment and non-overlap — every subnet prefix must sit inside the VNet address space and must not overlap another subnet; the defaults carve 172.19.0.0/28, 172.19.0.64/26, and 172.19.1.0/24 out of 172.19.0.0/16.
  • Subnet sizing — API server subnet at least a /28, AKS reserves at least nine addresses in it, and node subnets should be sized for expected pod and node scale.
  • Role assignment blast radius — the identity gets Network Contributor on the whole VNet for Node Autoprovisioning; retargeting at a shared VNet extends that grant to everything in it, so scope it tighter or keep the cluster in a dedicated network.
  • Delegation driftignore_changes = [delegation] keeps plans clean because AKS adds its own delegation post-create, but it also means Terraform won't report later out-of-band changes to that delegation.

Kept as documentation, per your note that this doesn't require redesigning the four standalone examples.

Validation after the changes

terraform fmt -check -diff, init, validate, and plan pass for all four samples: 10 to add each, location westus2, no diffs.

The corresponding article PR (azure-aks-docs-pr#3646) is updated so its embedded code still matches these samples byte-for-byte, and it carries the same private-cluster correction.

@stevenjma Steven Ma (stevenjma) left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

a few small changes here and then we should be good to go



> [!NOTE]
> The default location is `westus2` because that's the region these samples were validated in. At the time of testing, `eastus` returned `AKSCapacityHeavyUsage` for API Server VNet Integration. Set `resource_group_location` to deploy elsewhere.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This can be removed, it's just noise for the reader

required_providers {
azurerm = {
source = "hashicorp/azurerm"
version = "~>5.0"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This sample uses 5.0 but the other providers.tf file used 4.0 as the pin. I'd recommend having both set to 5.0



> [!NOTE]
> The default location is `westus2` because that's the region these samples were validated in. At the time of testing, `eastus` returned `AKSCapacityHeavyUsage` for API Server VNet Integration. Set `resource_group_location` to deploy elsewhere.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same as other file, remove for noise

}
azurerm = {
source = "hashicorp/azurerm"
version = "~>4.0"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same as above, pin to a consistent version for these samples


The virtual network contains a subnet delegated to the cluster API server, a subnet for the user node pools, and a subnet for the managed system node pool. The cluster uses a user-assigned managed identity that's granted the Network Contributor role on the virtual network, which is required when you bring your own network. Because the cluster is private, the API server endpoint is assigned a private IP address in the virtual network instead of a public one. Reaching it requires private network connectivity to that virtual network and DNS resolution of the private FQDN, so access isn't limited to clients inside the virtual network itself: virtual network peering, a VPN gateway, or Azure ExpressRoute can all provide a path from outside it. Enabling a public FQDN changes only how the cluster is named in DNS. It doesn't make the private endpoint routable from the internet.

The cluster is created with the AzureRM provider's [`azurerm_kubernetes_automatic_cluster`](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/kubernetes_automatic_cluster) resource, which is the recommended way to declare an AKS Automatic cluster. This resource requires AzureRM provider `5.0` or later.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
The cluster is created with the AzureRM provider's [`azurerm_kubernetes_automatic_cluster`](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/kubernetes_automatic_cluster) resource, which is the recommended way to declare an AKS Automatic cluster. This resource requires AzureRM provider `5.0` or later.
The cluster is created with the AzureRM provider's [`azurerm_kubernetes_automatic_cluster`](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/kubernetes_automatic_cluster) resource, which is the recommended way to declare an AKS Automatic cluster. This resource requires AzureRM provider `v4.81` or later.

Set the azurerm pin to ~>5.0 in the AzAPI samples to match the provider-native ones, correct the minimum provider version to v4.81, and remove the tested-region notes.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@asudbring

Copy link
Copy Markdown
Contributor Author

Thanks Steven Ma (@stevenjma) — all four inline comments addressed in 011a1690.

Tested-region notes (custom-network-azapi/readme.md, custom-network/readme.md)

Removed from all four READMEs. The westus2 default stays, since that's the validated region, but the explanation is gone.

Provider pin consistency (custom-network/providers.tf, private-custom-network-azapi/providers.tf)

All four samples now pin azurerm = "~>5.0". The two AzAPI samples were on ~>4.0; they're aligned with the provider-native ones. azapi stays at ~>2.0 and random at ~>3.0.

private-custom-network/readme.md — v4.81 suggestion

Applied, and to the other provider-native README as well. I verified before propagating: website/docs/r/kubernetes_automatic_cluster.html.markdown exists at provider tag v4.81.0 (July 14, 2026) but 404s at v4.80.0, and the 5.0.0 changelog only adds azurerm_kubernetes_automatic_cluster_datasource. So v4.81 is right and my original 5.0 was wrong.

Validation

After terraform init -upgrade, all four resolve azurerm 5.0.1 (plus azapi 2.11.0 where applicable) and pass fmt -check, validate, and plan — 10 to add each, no diffs.

The article PR (azure-aks-docs-pr#3646) is updated to match: same version correction, notes removed, and the embedded code blocks resynced byte-for-byte against these samples.

@stemaMSFT

Steven (stemaMSFT) commented Aug 7, 2026

Copy link
Copy Markdown
Contributor

Allen Sudbring (@asudbring) my test account doesn't have permissions to live deploy this scenario and verify it works. can you paste proof that the apply and destroy is successful? then i will approve.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants