Azure Landing Zone

Problem

Governance applied after workloads exist is negotiation. Governance applied to a management group before the first subscription lands there is just the environment. The distinction decides whether a policy is a guardrail or a ticket.

But a governed foundation is only half the story. A paved road is only real if a workload can actually land on it and inherit the controls without hand-wiring them, so this project is both: the base landing zone, and a prod workload (private AKS plus a managed database) that deploys onto it as the reference paved road.

Approach: the base

• A management group tree under the tenant root splitting Platform, Workloads, and Sandbox, with dev, test, and prod beneath Workloads and the subscription moved into Workloads so every assignment applies automatically.
• Deny-effect Azure Policy at the hierarchy: no public IPs, allowed locations (a stricter prod assignment is wired but binds nothing until prod has its own subscription), and required owner, cost-center, environment, and data-classification tags, plus the built-in CIS Azure Foundations initiative and HITRUST/HIPAA scoring in Defender for Cloud.
• data_classification is limited to public, internal, confidential, or phi, and a resource group tagged phi cannot hold a Key Vault, storage account, SQL server, or PostgreSQL server with public network access, so the classification enforces the control rather than describing it.
• A hub VNet at 10.0.0.0/16 with Azure Firewall (threat intelligence in Deny mode) and Bastion as the only admin path, flag-gated as the hourly layer, and dev, test, and sandbox spokes peered through a reusable module, each with a default-deny NSG. One action group alerts on Deny policy events, Key Vault 403s, and firewall deny spikes.

Approach: the workload paved road

• A separate Terraform root vends a prod VNet (10.3.0.0/16) peered to the hub and a private AKS cluster with no public API, OIDC workload identity, and outbound_type=userDefinedRouting so all egress leaves only through the hub Azure Firewall.
• etcd secrets are envelope-encrypted with a customer-managed key reached over a Key Vault private endpoint; the vault stays default-Deny and AKS API Server VNet Integration projects the control plane into a delegated subnet so it can reach the key privately. Node OS disks use a separate CMK via a disk encryption set.
• The data tier is a zone-redundant HA PostgreSQL Flexible Server, VNet-injected into a delegated subnet, CMK-encrypted, Entra-auth enabled, with a geo-redundant Data Protection backup vault protecting it.
• ACR sits behind a private endpoint with a cache rule; External Secrets gets a federated workload identity; the AKS control-plane identity carries a least-privilege custom role scoped to just the Key Vault private-endpoint approval actions it needs.

Architecture: the load-bearing decisions

Reserved subnets are the cheap detail that matters most. Azure Firewall, Gateway, and Bastion each require an exactly-named subnet at a minimum prefix, so those are carved and named up front; activating any later is a resource addition rather than a re-addressing exercise across every peered network.

A private Key Vault for etcd KMS forces a chain of non-obvious requirements the platform enforces rather than documents. Private KMS is rejected outright unless API Server VNet Integration is enabled, and once enabled, AKS stands up its own managed private endpoint to the vault, which fails unless the cluster identity can approve that connection, a permission no built-in Crypto or Network role grants. The answer is a custom role with exactly the private-endpoint proxy and approval actions, not weakening the vault to public.

Zone-redundant Postgres HA replicates between a primary and a standby inside one delegated subnet, so a data NSG that denies everything but the app and AKS tiers silently blocks the standby from reaching the primary on 5432. The fix is an explicit intra-subnet allow, and the region itself was moved to centralus because Postgres Flexible Server is capacity-restricted in the original region.

A second workload: the member portal

• Front Door Premium with WAF (Default Rule Set 2.1, Bot Manager, a per-IP rate limit) in front of Container Apps in two regions. The apps are internal with public access disabled and reached only over Private Link, so the base deny-public-IP policy holds with no exception.
• Azure SQL in a failover group with private endpoints and Entra-only auth, in a resource group tagged phi so the base policy denies it public network access. Both regions connect to the failover group listener, so the app never needs to know which database is primary.
• API Management for partners, Entra External ID in its own tenant for member sign-in, a Logic App integration, and Application Insights with an availability test and a 99.9% SLO fast-burn alert, all landing in the base Log Analytics workspace.
• Failover runs on two clocks: Front Door shifts stateless traffic on its health probes, and the SQL failover group moves the data on its own. Neither depends on the other.

Measured, not claimed

A smoke test passed 7 of 7 through Front Door, including a SQL injection probe the WAF blocked with a 403 and a partner call refused without a key. With the primary region's ingress disabled, Front Door served every request from the second region 83.9 seconds later, a number that includes the disable command's own run time. A planned SQL failover, writing every second throughout, had a longest gap of 7.3 seconds and lost zero acknowledged writes, and the failback measured 7.4 seconds with 0 of 36 lost. Planned failovers are lossless by design, which the write-up says plainly.

The first deploy taught as much as the drills. The landing zone's own allowed-locations policy denied the portal's second region until a reviewed one-line change added it. Container Apps had no capacity in centralus and Azure SQL was restricted in three eastern regions, so regions are now chosen per tier. The app created its table before listening and the liveness probe killed it, so schema setup moved to a background thread. One honest gap remains: the portal VNets do not yet route egress through the hub firewall.

Outcome

Deployed against a real Azure tenant across two Terraform roots and verified through the control plane, not the plan file: the management group hierarchy and policy assignments, private AKS nodes Ready with etcd KMS active over the Key Vault private endpoint, PostgreSQL zone-redundant HA reporting Healthy, and the Postgres backup instance ProtectionConfigured.

The demo workloads were removed, with the External ID tenant emptied first. A backup vault remains during its protected recovery window, alongside soft-deleted Key Vaults. Deploy-demo-destroy discipline keeps the whole reference reproducible for roughly the price of a couple of hours of runtime rather than a standing bill.

Compute baseline proven live

Locally implemented controls restrict standalone VMs to the landing-zone Compute Gallery, approved sizes, and encryption at host, with guest configuration and periodic update assessment. AKS retains its managed Ubuntu node image with SecurityPatch and weekly maintenance settings validated statically.

Packer publishes a gallery image using the shared cis_baseline Ansible role, with narrow, expiring build exemptions. On 2026-10-05, the image passed 28 hardening checks after reboot and a new private management VM passed the same checks through Azure Run Command. All 16 guardrail checks passed, including isolated denials for an unapproved image, forbidden size, and missing host encryption. The first VM exposed an Apport boot override, which the corrected role removes. A retained backup vault remains during its recovery window; teardown is not a zero-invoice guarantee.

Parity gap-close proven live

On 2026-10-06 the zone moved to azurerm 4.x and gained VNet flow logs, an observability root of change alerts, and a secrets root, then ran all of it against real Azure and was destroyed. Flow logs passed 9 of 9, with hub and prod logs written to a CMK-encrypted, default-Deny storage account through the trusted-service bypass and 190 Traffic Analytics rows returned. The observability alerts passed 4 of 4.

The secrets root passed 11 of 11: a metadata-only Key Vault Reader identity whose effective access shows readMetadata allowed and getSecret not allowed, a real secret value read that returns 403, a seeded no-expiry secret that meets the finding criterion, and a NearExpiry audit event that reached the workspace and fired its alert. The 403 also fired the forbidden-read alert, and that email reached a phone, which proves the detection path end to end. The Postgres backup instance moved to azapi after the azurerm resource returned 406 on create and delete.

AKS and ACR on azurerm 4.x were proven the same day, but not where the zone usually runs. centralus refused every new AKS cluster with AKSCapacityHeavyUsage, private or public, Free or Standard tier, with or without API Server VNet Integration. Private KMS needs VNet integration, so the config stayed and the region moved: one-node probes passed in six other US regions, and the base and workload redeployed in eastus2. Checked through the control plane: private cluster with both nodes Ready, VNet integration on its delegated subnet, etcd KMS over the private Key Vault path, ACR behind an approved private endpoint with CMK and its cache rule, and a post-apply plan with no changes on either root, which is the rename proof.

Not proven: warm standby as an LZ-wide capability. The portal's two-region setup was drilled with planned failovers only, and the prod tier has in-region zone HA plus a geo-redundant backup vault.

Gated CI on federated identities

Until 2026-10-26 every Azure apply ran from an operator laptop, because the shared pipeline only knew how to sign in to AWS. It now signs in to Azure through GitHub OIDC with two user-assigned identities, and no client secret exists anywhere. The plan identity trusts pull requests and main and holds Reader on the subscription. The apply identity trusts only the prod-apply and destroy environments, so a run has to clear a required reviewer before it can get a token, and it holds Contributor on one resource group and nothing else. It cannot create a role assignment, and both identities live in the operator-applied bootstrap root, so CI can never rewrite its own trust.

State got the same treatment. Both identities can read every root's state, which plans and the TTL guard need, but an ABAC condition on the storage role limits writes, deletes, and leases to the one state blob CI owns. A pull request that plans hostile Terraform cannot overwrite another root's state.

The first run failed at sign-in with AADSTS700213. GitHub issues this repo's tokens with the immutable subject, which carries the owner and repo IDs, and Entra matches a federated credential's subject exactly. It is the same mismatch the AWS pipeline found, repeated because the Azure credentials were written from the old name-based pattern. With the subjects corrected, a gated apply built four resources after approval, a gated destroy removed all four and found no hourly resources left, and the TTL guard read all four roots green.

Scope, stated plainly: CI applies only a small proof root in that resource group. Governance, the workload, and compute stay operator-applied. Moving the workload behind CI needs runner access through its Key Vault firewall and a constrained way to create role assignments, and neither is built.

The same root now has drift detection. A daily refresh-only plan runs as the read-only identity and opens one issue naming the changed resource and attributes, never their values, and the next clean run closes it. Its first live run found a bug in the shared check rather than in Azure: on azurerm, Terraform's refresh-only exit code says changes exist when nothing drifted, so every clean run opened an issue listing zero resources. The shared workflow now decides from the drift list itself. On 2026-10-29 a clean run stayed quiet, a tag changed from the CLI opened an issue naming the VNet, and reverting it closed the issue.

SRE on the paved road

Two flags turn the prod workload into something an on-call engineer would recognise. Flux reconciles a .NET 10 service, its load generator, Grafana, and a Postgres StatefulSet on CMK disks from Git, with every image pinned by digest. Managed Prometheus writes to an Azure Monitor workspace, where recording rules compute the error ratio over four windows and multiwindow burn-rate rules page through the landing zone's action group. Secrets come from Key Vault through the CSI driver, one workload identity per consumer, and none of them enter Git.

Run live on 2026-10-29 in eastus2. A hand scale from 3 replicas to 1 was reverted by Flux in 29 seconds, and a commit changing the load generator's rate was live in 63 seconds with no kubectl. A 50% fault fired the fast-burn alert 5 minutes 28 seconds later, the page email arrived, and the alert resolved after the fault was removed. Then the Postgres namespace was deleted outright and restored from AKS Backup: the restore job took 219 seconds and writes resumed 405 seconds after the delete. The last durable write before the loss was 3301 and the restore came back at 3007 with no gaps, so 294 writes were lost, counted from the data rather than the backup's timestamp, which turned out to be 2.5 minutes later than the data it held.

The session found twelve things a plan never shows, and the most useful came at the end. Terraform deleted the hub peerings and the Key Vault private endpoint before the cluster, because nothing said they were related. The nodes lost their route out, went NotReady, and the GitOps and backup deletes hung for 30 minutes. The cluster now depends on its whole network path, and a check over Terraform's dependency graph, run in CI with no credentials, fails if any of 52 ordering rules goes missing. The firewall logs also showed a broad AzureCloud rule carrying 2,874 connections ahead of every domain rule, so it was cut to the one Azure Monitor rule the docs require.

A second, shorter session that evening proved both fixes. The workload's 98 resources were destroyed in 23 minutes in the designed order, the GitOps and backup deletes taking one and two minutes instead of hanging, and the narrowed Azure Monitor rule carried 2,025 connections with every add-on and an on-demand backup healthy. It also found two more: three nodes still left the backup agent short of CPU, so the default is now four, and the backup instance checks its role once at creation and never retries, so it now waits five minutes for the role to propagate. The session costs are estimates, about $7 to $9.50 and $6 to $7.50, until Cost Management posts.