# Panfactum Changelog — edge.26-04-24

> Adds Temporal workflow orchestration, per-schema Vault roles for PostgreSQL clusters, AWS Service Quotas Automatic Management, and a new cluster reset command, alongside extensive reliability improvements to cluster and SSO installation flows.

## Highlights

- New `kube_temporal` module — deploy Temporal workflow orchestration with full Panfactum operational guarantees
- New `pf cluster reset` command — safely reset EKS clusters by removing default AWS resources
- `aws_account` now enables AWS Service Quotas Automatic Management for proactive quota monitoring and increase requests
- `kube_pg_cluster` gains per-schema Vault roles and automatic schema initialization — re-apply to enable new capabilities
- `authentik_vault_sso` now filters regex redirect URIs Vault cannot accept — re-apply `authentik_vault_sso` and `vault_auth_oidc` modules
- `kube_cilium` operator now runs 2 replicas on all clusters regardless of SLA target — re-apply `kube_cilium` to apply
- `kube_aws_ebs_csi` PodDisruptionBudget switched to server-side apply to fix Helm race condition — re-apply `kube_aws_ebs_csi`
- Authentik default login session extended from 8 hours to 30 days — matches industry norms while MFA remains enforced
- `pf cluster add` and `pf sso add` receive major reliability improvements including bootstrap anti-affinity bypass, idempotent re-runs, and pre-flight checks

## Breaking Changes

- `aws_organization` upgrades to AWS provider v6, enables `SERVICE_CONTROL_POLICY`, `TAG_POLICY`, and greatly expands trusted service access principals by default.
  - Run `terraform init -upgrade` in all management environment module directories to update provider lock files to v6.
  - Review the AWS Provider v6 Upgrade Guide for breaking changes that may affect other modules in your stack.
  - Run `terraform plan` on `aws_organization` and review the diff — expect new policy types (`SERVICE_CONTROL_POLICY`, `TAG_POLICY`) and an `aws_notifications_organizations_access` resource to be created.
  - If any newly-enabled service access principals or policy types are unwanted, set `disabled_aws_service_access_principals` or `disabled_enabled_policy_types` before applying.
  - Impacts: iac-module `aws_organization` — Pins `hashicorp/aws` to v6.38.0; enables `SERVICE_CONTROL_POLICY` and `TAG_POLICY` by default; adds `aws_notifications_organizations_access` resource; greatly expands default trusted service access principals; adds opt-out variables for all new defaults
  - Reference (internal-commit): [feat(aws_organization): upgrade AWS provider to v6, expand org service access](https://github.com/Panfactum/stack/commit/b5cab6457e1c5ba504e14e063839827285694d9d)
  - Reference (external-docs): [AWS Provider v6 Upgrade Guide](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/guides/version-6-upgrade)
  - Reference (external-docs): [hashicorp/terraform-provider-aws v6.38.0 release notes](https://github.com/hashicorp/terraform-provider-aws/releases/tag/v6.38.0)
  - Reference (external-docs): [AWS services that support trusted access with Organizations](https://docs.aws.amazon.com/organizations/latest/userguide/orgs_integrate_services_list.html)
  - Reference (internal-docs): [`aws_organization` module reference](https://panfactum.com/docs/main/modules/aws_organization/reference)

## Additions

- Added `kube_temporal` module for deploying Temporal workflow orchestration with full Panfactum operational guarantees
  - Impacts: iac-module `kube_temporal` — New module providing a self-hosted Temporal workflow orchestration server on Kubernetes
  - Impacts: iac-module `kube_pg_cluster` — Added `extra_schemas` variable to support initializing multiple PostgreSQL schemas at cluster creation time
  - Reference (internal-commit): [feat(kube_temporal): add Temporal workflow orchestration module](https://github.com/Panfactum/stack/commit/95d846afa1014c48bdbe6a031b1048bb3b49608b)
  - Reference (internal-docs): [`kube_temporal` module reference documentation](https://panfactum.com/docs/main/modules/kube_temporal/reference)
  - Reference (external-docs): [Temporal — durable execution platform official website](https://temporal.io)
  - Reference (external-docs): [Temporal server architecture — Frontend, History, Matching, and Worker services](https://docs.temporal.io/server)
  - Reference (external-docs): [temporalio/temporal — Temporal server GitHub repository](https://github.com/temporalio/temporal)
- `cluster add` now validates EC2 vCPU quota headroom before deployment and auto-submits increase requests when insufficient.
  - Impacts: cli `cluster add` — Now checks EC2 On-Demand and Spot Standard vCPU quota headroom before deployment and auto-submits quota increase requests when headroom is insufficient
  - Reference (internal-commit): [feat(cluster/add): add EC2 vCPU quota headroom check before cluster deploy](https://github.com/Panfactum/stack/commit/a44aa31c65415ef3c6085a129b8689ba8edd9666)
  - Reference (external-docs): [Amazon EC2 service quotas — AWS documentation](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-resource-limits.html)
- `pf cluster add` gains a CloudFront account verification pre-flight check, preventing 30+ minutes of wasted deployment on unverified AWS accounts.
  - Impacts: cli `cluster add` — New pre-flight check verifies the AWS account is permitted to create CloudFront distributions before any infrastructure is deployed, providing actionable guidance when the account is unverified or IAM permissions are insufficient
  - Reference (internal-commit): [feat(cluster): add CloudFront account verification pre-flight check](https://github.com/Panfactum/stack/commit/895d099dcf643dce2fbd9ef5dbb4c58c4a5a8546)
  - Reference (issue-report): [AWS CloudFront account verification requirement preventing resource creation](https://repost.aws/questions/QUHAzHD_-nSjiaAxMw7toQ3A/unable-to-create-cloudfront-distribution-account-must-be-verified)
- New `pf cluster reset` command safely resets EKS clusters; `--force` flag skips protection of already-deployed Panfactum-managed workloads.
  - Impacts: cli `cluster reset` — New command removes default AWS EKS addons and resources in preparation for Panfactum's hardened replacements; skips resources labeled `panfactum.com/workload` by default to protect already-deployed Panfactum workloads during resumed installations
  - Reference (internal-commit): [feat(cluster): add `pf cluster reset` command with --force flag](https://github.com/Panfactum/stack/commit/cfdc6a1d55b023f8f057fd0135fa8876ba028974)
  - Reference (internal-docs): [Kubernetes Cluster bootstrapping guide — Reset EKS Cluster section](https://panfactum.com/docs/main/guides/bootstrapping/kubernetes-cluster)
- Added automated AWS SES production access request during `pf sso add` command to streamline email functionality setup
  - Impacts: cli `sso add` — Now automatically requests AWS SES production access and provides status updates on the review process
  - Reference (internal-commit): [feat(sso): automate SES sandbox-to-production promotion](https://github.com/Panfactum/stack/commit/668538efd8dcb78d88b5317f3c31a5d71a936ee2)
  - Reference (external-docs): [AWS SES production access request documentation](https://docs.aws.amazon.com/ses/latest/dg/request-production-access.html)
- Added configurable pre-commit hooks (`tf-fmt` and `hcl-fmt`) to `mkDevShell` with automatic merge support for custom user hooks
  - Impacts: devshell `enter-shell-local` — Added configurable pre-commit hooks for infrastructure formatting with automatic merge of custom user configurations
  - Impacts: cli `precommit install` — Added `precommit install` command to merge Panfactum and user-defined pre-commit configurations
  - Reference (internal-commit): [feat(devshell): add tf-fmt and hcl-fmt pre-commit hooks to mkDevShell](https://github.com/Panfactum/stack/commit/6cf4816914e8cd69e2bfb01749e6402d3c21fd91)
  - Reference (external-docs): [prek - Faster pre-commit alternative built in Rust](https://github.com/j178/prek)
  - Reference (internal-docs): [Development Shell Customization Guide](https://panfactum.com/docs/main/guides/development-shell/customizing)
- Added AWS Service Quotas Automatic Management to `aws_account` module for proactive quota monitoring and automatic increase requests
  - Review new module inputs: `quota_auto_management_opt_in_type`, `quota_auto_management_regions`, and `quota_auto_management_exclusion_list`
  - Configure `operations_contact` if you want quota threshold notifications sent to your operations team
  - Optionally customize the `quota_auto_management_regions` list to target specific AWS regions
  - Impacts: iac-module `aws_account` — Added Service Quotas Automatic Management with configurable regions, exclusion lists, and optional notification pipeline
  - Reference (internal-commit): [feat(aws_account): enable Service Quotas Automatic Management](https://github.com/Panfactum/stack/commit/cef868ff0510bf116b0bac5c15c504b89cc39d06)
  - Reference (internal-docs): [aws_account module reference](https://panfactum.com/modules/aws_account/reference)
  - Reference (external-docs): [AWS Service Quotas Automatic Management Documentation](https://docs.aws.amazon.com/servicequotas/latest/userguide/automatic-management.html)
  - Reference (external-commit): [AWS Provider v6.40.0 release with aws_servicequotas_auto_management resource](https://github.com/hashicorp/terraform-provider-aws/releases/tag/v6.40.0)

## Version Updates

- Updated Panfactum container image tag in `kube_constants` to `e61269a`, refreshing `panfactum`, `vault`, `bastion`, and `pvc-autoresizer` images.
  - Impacts: iac-module `kube_constants` — Updated pinned image tag to `e61269a` for all Panfactum-built container images (panfactum devShell, vault sidecar, bastion, and pvc-autoresizer)
  - Reference (internal-commit): [chore(kube_constants): bump kube image tag to e61269a](https://github.com/Panfactum/stack/commit/eaf701cf3b45b28fe4e35bb3e1df1f7aca679e98)
  - Reference (internal-docs): [kube_constants module documentation](https://panfactum.com/docs/main/reference/infrastructure-modules/direct/kubernetes/kube_constants)
- Updated AWS Terraform provider from `v6.38.0` to `v6.40.0` across all infrastructure modules
  - Impacts: iac-provider `aws` — Updated from `v6.38.0` to `v6.40.0` across all infrastructure modules
  - Reference (internal-commit): [feat(aws_account): enable Service Quotas Automatic Management](https://github.com/Panfactum/stack/commit/cef868ff0510bf116b0bac5c15c504b89cc39d06)
  - Reference (external-docs): [AWS Terraform Provider v6.40.0 Release Notes](https://github.com/hashicorp/terraform-provider-aws/releases/tag/v6.40.0)
  - Reference (issue-report): [feat: add service quotas automatic management](https://github.com/hashicorp/terraform-provider-aws/pull/45968)
  - Reference (internal-docs): [`aws_account` module documentation](https://panfactum.com/docs/main/modules/aws_account/overview)

## Improvements

- Extended default authentication session duration from 8 hours to 30 days in `authentik_core_resources`, keeping recovery flow sessions at 8 hours.
  - Impacts: iac-module `authentik_core_resources` — Extended default `session_duration` to 30 days and added `recovery_session_duration` variable for independent control of recovery flow session length
  - Reference (internal-commit): [feat(authentik): extend auth session to 30 days, keep recovery at 8h](https://github.com/Panfactum/stack/commit/5c9b2c00bde6bd370ccb1c1b7544199a6b9620a2)
  - Reference (internal-docs): [authentik_core_resources module documentation](https://panfactum.com/docs/main/modules/authentik_core_resources/overview)
  - Reference (external-docs): [Authentik User Login stage — session duration configuration](https://docs.goauthentik.io/add-secure-apps/flows-stages/stages/user_login/)
- Replaced `bun` workspace architecture with per-package lockfiles to eliminate Docker build failures and simplify dependency management.
  - Impacts: devshell `pf` — CLI Nix build simplified by removing `sed+jq` lockfile transformation hack; devshell now runs `bun install` once per package directory instead of a single root-level install
  - Reference (internal-commit): [refactor(bun): replace workspace with per-package lockfiles](https://github.com/Panfactum/stack/commit/6127695e8ea9a5135419804986b9d6359a1b762b)
  - Reference (external-docs): [`bun install --frozen-lockfile --filter` fails in Docker when `bun.lock` was generated from the full workspace](https://github.com/oven-sh/bun/issues/28402)
- Enhanced `cluster add` to detect and gracefully handle NAT ASG capacity failures with clear error messages
  - Impacts: iac-module `aws_vpc` — Sets `wait_for_capacity_timeout = "0"` on NAT ASGs and exposes `nat_config` output for CLI polling
  - Impacts: cli `cluster add` — Adds "Verify NAT Gateways" task that polls ASG instances and provides clear capacity failure messages
  - Reference (internal-commit): [feat(vpc): detect NAT ASG capacity failures during cluster add](https://github.com/Panfactum/stack/commit/18037313b748444c9a8e28ae40307bcdc0807e21)
  - Reference (issue-report): [Terraform provider bug: wait_for_capacity_timeout ignored on InsufficientInstanceCapacity](https://github.com/hashicorp/terraform-provider-aws/issues/29753)
  - Reference (internal-docs): [aws_vpc module reference](https://panfactum.com/docs/main/modules/aws_vpc/overview)
- Optimized git commit verification for the Panfactum stack repository by using the GitHub REST API instead of `git fetch`.
  - Impacts: cli `util get-commit-hash` — Commit SHA verification for the Panfactum stack repository is now significantly faster via the GitHub REST API
  - Reference (internal-commit): [perf(cli/git): use GitHub API to verify Panfactum stack commits](https://github.com/Panfactum/stack/commit/cddda88240b583463fc8b0005d6ec662f3247623)
  - Reference (external-docs): [GitHub REST API endpoints for Git commits](https://docs.github.com/en/rest/git/commits)
  - Reference (issue-report): [Bug: git fetch failures with .netrc parser error when verifying Panfactum stack commits](https://github.com/Panfactum/stack/issues/287)
- Replaced static `.pre-commit-config.yaml` with Nix-generated configuration for reproducible development environments.
  - Impacts: devshell `git` — Pre-commit configuration is now Nix-generated from `lint.nix` with exact tool paths for reproducible linting and formatting; also adds `deadnix` and `statix` Nix quality hooks and renames dev scripts with a `ds-` prefix to avoid PATH collisions
  - Reference (internal-commit): [build(devshell): generate pre-commit config from Nix for reproducibility](https://github.com/Panfactum/stack/commit/c3fbf871e8db75a60787e81c0cf225381125b46d)
  - Reference (internal-docs): [Development shell customization guide](https://panfactum.com/docs/main/guides/development-shell/customizing)
- Removed overly prescriptive Vault recovery key storage guidance to allow teams to follow their own security policies
  - Impacts: cli `cluster add` — Success message no longer prohibits storing Vault recovery keys in company password vaults
  - Reference (internal-commit): [fix(cluster/add): remove overly prescriptive Vault key storage guidance](https://github.com/Panfactum/stack/commit/91f0bc927c03b1a37e3b9e873e43b66425115d5d)
  - Reference (internal-docs): [Vault bootstrapping guide covering recovery key setup and storage decisions](https://panfactum.com/docs/main/guides/bootstrapping/vault)
- `sso add` now automates Authentik user setup entirely via API, eliminating the two-browser-tab workflow and enabling fully headless deployment.
  - Impacts: cli `sso add` — Authentik user setup is now fully automated via API — no browser tabs, no manual token paste, and no incompatibility with headless or CI environments.
  - Reference (internal-commit): [feat(authentik): automate user setup without browser interaction](https://github.com/Panfactum/stack/commit/a4626526d92176e71713c13835045616c0efeb59)
  - Reference (external-docs): [Authentik API - Set password for user (coreUsersSetPasswordCreate)](https://docs.goauthentik.io/docs/developer-docs/api/reference/core-users-set-password-create)
  - Reference (external-docs): [Authentik API - Token management (coreTokensCreate / coreTokensViewKeyRetrieve)](https://docs.goauthentik.io/docs/developer-docs/api/reference/core-tokens-create)
  - Reference (internal-docs): [kube_authentik module reference](https://panfactum.com/docs/main/modules/kube_authentik)
- Streamlined SSO setup by removing unnecessary bootstrap admin email prompt and hardcoding it based on SSO domain.
  - Impacts: cli `sso add` — Removes `akadmin_email` prompt and hardcodes the bootstrap admin email based on SSO domain configuration
  - Reference (internal-commit): [refactor(sso): hardcode akadmin email, remove unnecessary prompt](https://github.com/Panfactum/stack/commit/900fd7f3dc3bd611be5d4967a1a2ed81d92fed28)
  - Reference (internal-docs): [Identity provider bootstrapping guide — akadmin_email and akadmin user setup](https://panfactum.com/docs/main/guides/bootstrapping/identity-provider)
  - Reference (internal-docs): [kube_authentik module documentation](https://panfactum.com/docs/main/modules/kube_authentik/overview)
- Enhanced `sso add` command with inline validation for confirmation prompts and automatic IAM Identity Center verification
  - Impacts: cli `sso add` — Enhanced confirmation prompts with inline validation and added automatic IAM Identity Center verification
  - Reference (internal-commit): [feat(sso): add inline confirm validation to SSO federated auth setup](https://github.com/Panfactum/stack/commit/7ddf20ded72f5807243954fd43b9e4046aad84c9)
  - Reference (external-docs): [AWS IAM Identity Center ListInstances API Documentation](https://docs.aws.amazon.com/singlesignon/latest/APIReference/API_ListInstances.html)
  - Reference (external-docs): [Enable IAM Identity Center - AWS Documentation](https://docs.aws.amazon.com/singlesignon/latest/userguide/enable-identity-center.html)
- Automated SAML metadata download in `sso add` command to eliminate manual Authentik UI navigation steps
  - Impacts: cli `sso add` — No longer requires manual metadata download from Authentik UI
  - Impacts: iac-module `authentik_aws_sso` — SAML metadata output now automatically consumed during SSO setup
  - Reference (internal-commit): [feat(sso): automate SAML metadata download in SSO setup](https://github.com/Panfactum/stack/commit/d686a88e72714378a164dfcaf24b06102bf048a4)
  - Reference (internal-docs): [authentik_aws_sso module documentation](https://panfactum.com/docs/main/reference/infrastructure-modules/authentik_aws_sso)
- Enhanced CLI build configuration with structured options to support smaller Docker image builds
  - Impacts: devshell `pf` — Now supports configurable CLI build options including smaller binary compilation mode
  - Impacts: configuration `flake.nix` — Updated to use structured `cli` configuration instead of boolean parameter
  - Reference (internal-commit): [refactor(cli): replace withPFCLI bool with structured cli config](https://github.com/Panfactum/stack/commit/2c27bbd2eb8071b414ce97c8ae432b455604fb42)
  - Reference (external-docs): [Bun --smol compilation mode documentation](https://bun.sh/docs/bundler/executables)
  - Reference (internal-docs): [mkDevShell customization guide](https://panfactum.com/docs/main/guides/development-shell/customizing)
- Enhanced Terragrunt backend bootstrapping to be explicit and compatible with Terragrunt >= 0.87.0
  - Impacts: cli `env add` — Backend bootstrapping dependency is now explicit and self-documenting
  - Impacts: installer `install.sh` — Environment setup now explicitly handles Terragrunt backend initialization
  - Impacts: iac-module `tf_bootstrap_resources` — Module initialization now uses explicit backend bootstrap to resolve circular dependency
  - Reference (internal-commit): [feat(terragrunt): make backend bootstrap explicit, require pre-existing](https://github.com/Panfactum/stack/commit/98330d5cea30ddcbd554e32b947184ff50bd2810)
  - Reference (external-docs): [Terragrunt v0.87.0 release notes - Breaking change: Automatic Backend Provisioning Removed](https://github.com/gruntwork-io/terragrunt/releases/tag/v0.87.0)
  - Reference (internal-docs): [tf_bootstrap_resources module documentation](https://panfactum.com/docs/main/modules/tf_bootstrap_resources/overview)
- Increased minimum PostgreSQL memory allocation in Authentik deployment from 500MB to 1GB for better performance and stability
  - Impacts: iac-module `kube_authentik` — PostgreSQL minimum memory increased from 500MB to 1GB
  - Reference (internal-commit): [perf(kube_authentik): increase Authentik postgres min memory to 1GB](https://github.com/Panfactum/stack/commit/d243b6130426aa2b62bf65b8f40f175f8a7be5fd)
  - Reference (internal-docs): [kube_authentik module documentation](https://panfactum.com/docs/main/modules/kube_authentik)
  - Reference (issue-report): [Significant PostgreSQL DB load on Authentik 2025.10.x causing issues](https://github.com/goauthentik/authentik/issues/19302)
  - Reference (issue-report): [2026.2 worker memory usage doubled vs 2025.12 (~500Mi → ~1020Mi)](https://github.com/goauthentik/authentik/issues/20537)
- Removed PostHog analytics telemetry from the CLI to eliminate user tracking and reduce dependency surface area
  - Impacts: devshell `pf` — Removed all PostHog analytics tracking calls and dependencies
  - Reference (internal-commit): [feat(cli): remove PostHog analytics telemetry from CLI](https://github.com/Panfactum/stack/commit/951c8a1f552d7bc61f01a895ad3779a155a816c9)
  - Reference (external-docs): [PostHog analytics platform website](https://posthog.com/)
- Enhanced `kube_pg_cluster` module with per-schema Vault roles, automatic schema initialization, and PgBouncer `search_path` support
  - Impacts: iac-module `kube_pg_cluster` — Added per-schema Vault roles with pre-configured `search_path`, schema initialization Job for existing clusters, and `pgbouncer_ignore_startup_parameters` variable for JDBC compatibility
  - Reference (internal-commit): [feat(kube_pg_cluster): add per-schema roles, schema init job, PgBouncer fix](https://github.com/Panfactum/stack/commit/9053ab836b67d438d2d1587c7624f63c53adf45b)
  - Reference (external-docs): [CloudNativePG PostgreSQL Operator](https://github.com/cloudnative-pg/cloudnative-pg)
  - Reference (external-docs): [PgBouncer Configuration Documentation - ignore_startup_parameters](https://github.com/pgbouncer/pgbouncer/blob/master/doc/config.md)
  - Reference (external-docs): [Vault PostgreSQL Database Secrets Engine](https://developer.hashicorp.com/vault/docs/secrets/databases/postgresql)
  - Reference (internal-docs): [kube_pg_cluster Module Documentation](https://panfactum.com/docs/main/modules/kube_pg_cluster/overview)

## Fixes

- Fixed `wf_dockerfile_build` `scale-buildkit` steps crashing with "Devshell configuration file does not exist" inside CI containers without a `panfactum.yaml`.
  - Impacts: cli `buildkit clear-cache` — Migrated to `PanfactumLightCommand`; no longer has unsafe access to `devshellConfig`
  - Impacts: cli `buildkit get-address` — Migrated to `PanfactumLightCommand`; `workingDirectory` now passed explicitly instead of reading from `devshellConfig`
  - Impacts: cli `buildkit record-build` — Migrated to `PanfactumLightCommand`; `workingDirectory` now passed explicitly instead of reading from `devshellConfig`
  - Impacts: cli `buildkit scale up` — Migrated to `PanfactumLightCommand`; `workingDirectory` now passed explicitly instead of reading from `devshellConfig`
  - Impacts: cli `buildkit suspend` — Migrated to `PanfactumLightCommand`; `workingDirectory` now passed explicitly instead of reading from `devshellConfig`
  - Impacts: cli `iac update-module-status` — Migrated to `PanfactumLightCommand`; no longer has unsafe access to `devshellConfig`
  - Impacts: cli `kube disable-disruptions` — Migrated to `PanfactumLightCommand`; no longer has unsafe access to `devshellConfig`
  - Impacts: cli `kube enable-disruptions` — Migrated to `PanfactumLightCommand`; no longer has unsafe access to `devshellConfig`
  - Impacts: cli `k8s velero snapshot-gc` — Migrated to `PanfactumLightCommand`; no longer has unsafe access to `devshellConfig`
  - Impacts: cli `util get-commit-hash` — Migrated to `PanfactumLightCommand`; `workingDirectory` now passed explicitly instead of reading from `devshellConfig`
  - Impacts: cli `util get-module-hash` — Migrated to `PanfactumLightCommand`; `workingDirectory` now passed explicitly instead of reading from `devshellConfig`
  - Impacts: cli `wf git-checkout` — Migrated to `PanfactumLightCommand`; `workingDirectory` now passed explicitly instead of reading from `devshellConfig`
  - Impacts: cli `wf sops-set-profile` — Migrated to `PanfactumLightCommand`; no longer has unsafe access to `devshellConfig`
  - Reference (internal-commit): [refactor(cli): encode devshell requirement in command type hierarchy](https://github.com/Panfactum/stack/commit/e61269a70c071f8c41f16647f735c90bd372d6de)
  - Reference (internal-docs): [`wf_dockerfile_build` module documentation](https://panfactum.com/docs/main/modules/wf_dockerfile_build/overview)
- Fixed `kube_argo` Workflows executor containers hitting memory limits by increasing resource allocations
  - Impacts: iac-module `kube_argo` — Increased executor container memory request from `50Mi` to `75Mi` and limit from `70Mi` to `100Mi` to prevent OOM failures
  - Reference (internal-commit): [fix(kube_argo): increase argo exec container memory limits](https://github.com/Panfactum/stack/commit/6821b621005e6a52e5272c7c164ec934ba70ec44)
  - Reference (internal-docs): [kube_argo module documentation](https://panfactum.com/docs/main/reference/infrastructure-modules/kubernetes/kube_argo)
- Fixed `authentik_vault_sso` module passing regex redirect URIs to `vault_auth_oidc`, causing silent misconfigurations since Vault does not support regex patterns.
  - Re-apply the `authentik_vault_sso` module to update redirect URI filtering
  - Re-apply the `vault_auth_oidc` module to get the built-in localhost callback URI
  - If you pass `authentik_vault_sso`'s `oidc_redirect_uris` output to any non-Vault OIDC provider (e.g. ArgoCD, Grafana), switch to `oidc_redirect_uris_including_regexes` to continue receiving the full URI set including regex patterns
  - Impacts: iac-module `authentik_vault_sso` — `oidc_redirect_uris` output now returns only strict-mode URIs; new `oidc_redirect_uris_including_regexes` output provides the full set including regex patterns for providers that support them
  - Impacts: iac-module `vault_auth_oidc` — Hardcodes `http://localhost:8250/oidc/callback` internally so callers no longer need to supply it
  - Reference (internal-commit): [fix(oidc): prevent regex redirect URIs from reaching Vault OIDC](https://github.com/Panfactum/stack/commit/df00050879ea60803aca824155bde37fc8d19e2c)
  - Reference (external-docs): [RFC 8252 - OAuth 2.0 for Native Apps: Loopback interface redirection](https://tools.ietf.org/html/rfc8252#section-7.3)
  - Reference (external-docs): [Vault JWT/OIDC Auth Method — Redirect URIs and allowed_redirect_uris configuration](https://developer.hashicorp.com/vault/docs/auth/jwt#redirect-uris)
  - Reference (internal-docs): [authentik_vault_sso module reference](https://panfactum.com/docs/main/modules/authentik_vault_sso)
  - Reference (internal-docs): [vault_auth_oidc module reference](https://panfactum.com/docs/main/modules/vault_auth_oidc)
- Fixed Cilium operator only running 1 replica on SLA-1 clusters, causing networking failures during pod disruptions
  - Re-apply the `kube_cilium` module to update the operator deployment configuration
  - Impacts: iac-module `kube_cilium` — Operator replica count hardcoded to 2 for all clusters; SLA-1 clusters now run a highly available operator instead of a single replica
  - Reference (internal-commit): [fix: ensure cilium operator HA and fix website TS types](https://github.com/Panfactum/stack/commit/c64d99d027fb6c53b366a22cdfd6e10420f14076)
  - Reference (external-docs): [Cilium operator proper rolling update support (enables safe HA upgrades)](https://github.com/cilium/cilium/pull/23589)
  - Reference (external-docs): [Cilium operator HA mode internals documentation](https://github.com/cilium/cilium/blob/master/Documentation/internals/cilium_operator.rst)
  - Reference (internal-docs): [kube_cilium module reference](https://panfactum.com/modules/kube_cilium)
- Fixed `cluster add` command failing to detect partially deployed cluster regions, making automated recovery impossible
  - Impacts: cli `cluster add` — Enhanced cluster deployment detection to prevent skipping partially-installed regions
  - Reference (internal-commit): [fix(cluster): detect partial setupClusterExtensions deployments](https://github.com/Panfactum/stack/commit/55a4bca301fe41b8851b7a4c784af0e530ae8439)
- Fixed `cluster add` command Vault address becoming permanently stuck on localhost after DNS timeout during ingress setup
  - Impacts: cli `cluster add` — Fixed Vault address remaining on localhost after DNS propagation timeout during setup
  - Reference (internal-commit): [fix(cluster/add): prevent vault_addr stuck on local after DNS timeout](https://github.com/Panfactum/stack/commit/db90bc63ddedd76053a16a8df49fa2f51418f258)
  - Reference (internal-docs): [Bootstrapping guide: Inbound Networking](https://panfactum.com/docs/main/guides/bootstrapping/inbound-networking)
- Fixed `pf` CLI Nix build failing with `EPERM` errors when `bun` hardlinks read-only files from the Nix store.
  - Impacts: devshell `pf` — Fixed `EPERM` errors during Nix build caused by read-only `bun` cache files — both the symlink-dereference issue and the `bun2nix` hook copy issue are resolved
  - Impacts: devshell `parallel` — Fixed `nixpkgs` overlay hash for `parallel` after GNU re-released the tarball with different content
  - Reference (internal-commit): [fix(build): fix bun EPERM and parallel hash in nix builds](https://github.com/Panfactum/stack/commit/7b4fe36705dbfdadc48b159772815758e2a450f9)
  - Reference (internal-commit): [fix(cli): chmod bun cache after copy from read-only nix store](https://github.com/Panfactum/stack/commit/a689ba3e921f95c0deb7af061a7864fd3e9c5524)
  - Reference (issue-report): [bun2nix copies read-only files causing EPERM on hardlink](https://github.com/nix-community/bun2nix/issues/73)
  - Reference (external-docs): [bun2nix — Nix build tool for bun packages](https://github.com/nix-community/bun2nix)
- Fixed Vault OIDC authentication failing when CLI uses dynamic ports by switching Authentik redirect URI from strict to regex matching.
  - Impacts: iac-module `authentik_vault_sso` — Changed localhost redirect URI from strict to regex matching for dynamic port support
  - Reference (internal-commit): [fix(authentik_vault_sso): allow dynamic ports in Vault SSO OIDC callback URI](https://github.com/Panfactum/stack/commit/9fc490ccb99c5a9a56580187e83b47bd628f08b4)
  - Reference (external-docs): [OAuth 2.0 provider redirect URI configuration with regex matching](https://docs.goauthentik.io/docs/add-secure-apps/providers/oauth2)
  - Reference (issue-report): [Authentik upstream: wildcard regex in port part of redirect URI causes ValueError](https://github.com/goauthentik/authentik/issues/13023)
  - Reference (internal-docs): [authentik_vault_sso module reference](https://panfactum.com/modules/authentik_vault_sso)
- Fixed `pf wf git-checkout` command failing on private repositories due to incorrect authentication handling
  - Impacts: cli `wf git-checkout` — Now properly handles private repository authentication during git reference resolution
  - Reference (internal-commit): [fix(wf/git-checkout): fix getCommitHash failing on private repos](https://github.com/Panfactum/stack/commit/2725a5c0419aa9e917fd623fddd10f3105996d00)
  - Reference (internal-docs): [Checking out Git Repositories guide](https://panfactum.com/docs/main/guides/cicd/checking-out-code)
- Fixed concurrent Vault OIDC authentication failures when multiple processes attempt login simultaneously
  - Impacts: cli `vault get-token` — Now supports concurrent OIDC authentication sessions without port binding conflicts
  - Reference (internal-commit): [fix(vault): use dynamic port for OIDC login callback listener](https://github.com/Panfactum/stack/commit/5b0a4cc9a59c841bd558a42f016727b1d74bfa50)
  - Reference (issue-report): [Vault OIDC authentication fails when used simultaneously by multiple users on shared machine (port 8250 conflict)](https://github.com/hashicorp/vault/issues/15421)
  - Reference (external-docs): [Vault JWT/OIDC auth method — OIDC login CLI port parameter documentation](https://developer.hashicorp.com/vault/docs/auth/jwt)
- Fixed `authentik_github_sso` `sso_post_url` output returning the wrong SAML binding URL, breaking GitHub SSO configuration.
  - Impacts: iac-module `authentik_github_sso` — The `sso_post_url` output now returns the correct SAML HTTP-POST binding URL instead of the HTTP-Redirect URL, enabling successful GitHub SAML SSO configuration
  - Reference (internal-commit): [fix(authentik_github_sso): had the wrong output which caused confusion during setup](https://github.com/Panfactum/stack/commit/bfc83fb51c7a1eac473533c10120ba0cee78cbc9)
  - Reference (internal-docs): [authentik_github_sso module documentation](https://panfactum.com/docs/main/modules/authentik_github_sso/overview)
  - Reference (external-docs): [Authentik — Integrate with GitHub Organization (SAML POST binding requirement)](https://docs.goauthentik.io/integrations/services/github-organization/)
- Fixed `pf sso add` crashing on first-time AWS SSO setup when the `authentik_aws_sso` module has not yet been deployed
  - Impacts: cli `sso add` — No longer crashes before showing prompts when `authentik_aws_sso` has not yet been deployed
  - Reference (internal-commit): [fix(sso): handle missing org module YAML on first SSO setup](https://github.com/Panfactum/stack/commit/a53de6458d3d8502334cdf90a47d78444ce03306)
  - Reference (internal-docs): [authentik_aws_sso module documentation](https://panfactum.com/docs/main/modules/authentik_aws_sso/overview)
- Fixed Authentik restart process in `sso add` command to wait for deployment rollout completion before proceeding to user setup.
  - Impacts: cli `sso add` — Authentik restart process now blocks until `kubectl rollout status` confirms the deployment is ready, preventing flaky failures when subsequent setup steps call the Authentik API against pods that are still cycling.
  - Reference (internal-commit): [feat(sso/add): wait for Authentik rollout readiness after restart](https://github.com/Panfactum/stack/commit/09becda2183b2dce0e5091db9774cdd77d4f69a3)
- Fixed `cluster add` extension setup to prevent Terragrunt provider cache corruption during concurrent module initialization.
  - Impacts: cli `cluster add` — Extension module setup now runs a pre-initialization phase to prevent provider cache corruption during concurrent `terragrunt apply` operations
  - Impacts: devshell `terragrunt` — `terragruntInitAll` now accepts an optional `modules` parameter to scope initialization to specific module directories using `--queue-include-dir` and `--queue-strict-include` flags
  - Reference (internal-commit): [feat(cluster): pre-init extension modules to prevent cache corruption](https://github.com/Panfactum/stack/commit/734cabbde022dba65ca5b255406fcfd3aa7b30e1)
  - Reference (issue-report): [Allow multiple Terraform instances to write to plugin_cache_dir concurrently](https://github.com/hashicorp/terraform/issues/31964)
  - Reference (issue-report): [Terragrunt run-all init concurrency/parallelism issues with shared plugin cache](https://github.com/gruntwork-io/terragrunt/issues/2542)
- Fixed EC2 vCPU quota check to be aware of existing deployments and avoid false failures on `cluster add` re-runs
  - Impacts: cli `cluster add` — EC2 vCPU quota check now accounts for existing `aws_vpc`, `aws_eks`, and `kube_karpenter` deployment status to prevent unnecessary quota increase requests on re-runs
  - Reference (internal-commit): [fix(cluster/add): make vCPU quota check aware of existing deployments](https://github.com/Panfactum/stack/commit/b3b91a48ce6eff18e40c5ab64d7d8e75ce17d991)
  - Reference (external-docs): [Amazon EC2 service quotas — vCPU-based On-Demand Instance limits](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-resource-limits.html)
- Fixed Vault setup to be idempotent on re-runs by checking initialization status and reading tokens from proper file locations
  - Impacts: cli `cluster add` — Vault setup phase now properly handles re-runs without failing on already-initialized Vault instances
  - Impacts: iac-module `kube_vault` — Step completion detection updated to treat non-local `vault_addr` as a valid completion signal for `vault_core_resources` re-deploys
  - Reference (internal-commit): [fix(cluster/vault): make vault setup idempotent on re-runs](https://github.com/Panfactum/stack/commit/8084b20fbe5a6174adf836a019ec8f69e12f7dd7)
  - Reference (internal-docs): [kube_vault module reference](https://panfactum.com/docs/main/reference/infrastructure-modules/kubernetes/kube_vault)
  - Reference (internal-docs): [Bootstrapping guide: Vault setup](https://panfactum.com/docs/main/guides/bootstrapping/vault)
- Restored the `pf env add` command that was accidentally unregistered from the CLI
  - Impacts: cli `env add` — Command was unreachable after a class rename dropped its CLI registration; it is now restored and fully functional
  - Reference (internal-commit): [fix(cli): reregister env add command that was accidentally removed](https://github.com/Panfactum/stack/commit/dc09c52849fa03d5a420f0e9193f3f83b1318c95)
- Fixes Terragrunt CLI flag compatibility with `v0.85.0`+ by replacing removed `--terragrunt-*` prefixed flags with their modern equivalents.
  - Impacts: devshell `terragrunt` — Invocations now use `--non-interactive`, `--no-color`, and `--provider-cache` flags, and `providers lock` now uses `run -- providers lock`; compatible with `v0.85.0`+
  - Reference (internal-commit): [fix(terragrunt): update flags to new Terragrunt CLI syntax](https://github.com/Panfactum/stack/commit/b01be825db8880f2a5d46a0e01b1357a024ec149)
  - Reference (external-docs): [Terragrunt v0.85.0 release notes — removal of legacy `--terragrunt-*` flags](https://github.com/gruntwork-io/terragrunt/releases/tag/v0.85.0)
  - Reference (external-docs): [Terragrunt CLI redesign migration guide](https://docs.terragrunt.com/migrate/cli-redesign/)
- Fixed Vault OIDC browser login silently suppressed and `--silent` not blocking interactive prompts in `pf vault get-token`
  - Impacts: cli `vault get-token` — OIDC authentication now properly respects `--silent` mode and the browser-based login flow works correctly in interactive terminal sessions
  - Impacts: devshell `pf` — Terragrunt config loading via `getPanfactumConfig` no longer suppresses interactive Vault OIDC login when running in a developer shell
  - Reference (internal-commit): [fix(vault): fix interactive OIDC login and silent mode behavior](https://github.com/Panfactum/stack/commit/68eb0373a2032ee75a41aa01daa8df140cc9d676)
- Fixed `kube get-token` command to properly require the devshell environment
  - Impacts: cli `kube get-token` — Now correctly requires the devshell and exits with a clear error message when run outside it
  - Reference (internal-commit): [fix(cli): require devshell for kube get-token command](https://github.com/Panfactum/stack/commit/6111f6c0ceb4b99881f5a1de54736779c765bb54)
- Fixed `cluster add` and `sso add` commands crashing when resuming a partially completed installation.
  - Impacts: cli `cluster add` — No longer crashes when resuming installation at a step whose `module.yaml` has not yet been written
  - Impacts: cli `sso add` — No longer crashes when resuming installation at a step whose `module.yaml` has not yet been written
  - Reference (internal-commit): [fix(cli/cluster): prevent crash resuming install at certificates step](https://github.com/Panfactum/stack/commit/fd2a9c8f19cd183bb9ebbcc22bb9b22fb99646dd)
  - Reference (internal-commit): [fix(cli/cluster): tolerate missing YAML files when resuming install](https://github.com/Panfactum/stack/commit/faaac6c484140a28353016b4295eecebb20ae039)
  - Reference (internal-commit): [fix(cluster/add): allow missing module.yaml during cluster extension setup](https://github.com/Panfactum/stack/commit/90a154bdf13bc1bec435c1620235840146b59caa)
- Fixed `default_file_strict` regex in `aws_s3_public_website` to correctly recognize paths with non-extension dots (e.g., date-suffixed paths like `edge.25-04-03`) as directories.
  - Impacts: iac-module `aws_s3_public_website` — `default_file_strict` mode now correctly appends `default_file` to directory-like paths containing non-extension dots, not just dot-free paths
  - Reference (internal-commit): [fix(s3_public_website): fix strict mode regex to handle non-extension dots](https://github.com/Panfactum/stack/commit/445862a20bccb8e9b1ef06dd985486aa4d5d389f)
  - Reference (internal-docs): [aws_s3_public_website module documentation](https://panfactum.com/docs/main/modules/aws_s3_public_website/overview)
- Fixed Vault pods failing to start during cluster bootstrap due to instance-type anti-affinity constraints.
  - Impacts: iac-module `kube_vault` — Adds `bootstrap_mode_enabled` parameter to temporarily bypass instance-type anti-affinity during cluster setup
  - Impacts: cli `cluster add` — Automatically sets `bootstrap_mode_enabled = true` during `setupVault` and re-applies with `false` after Karpenter node pools are available
  - Reference (internal-commit): [fix(kube_vault): add bootstrap mode to bypass instance-type anti-affinity](https://github.com/Panfactum/stack/commit/4b2379c6d54f0f0d0b8d4c764427eaea7d7278eb)
  - Reference (internal-docs): [kube_vault module documentation](https://panfactum.com/modules/kube_vault)
- Fixed cert-manager webhook pods failing to start during SLA-3 cluster bootstrap due to instance-type anti-affinity constraints
  - Impacts: iac-module `kube_cert_manager` — Adds `bootstrap_mode_enabled` parameter to temporarily bypass webhook instance-type anti-affinity during cluster setup
  - Impacts: iac-module `kube_certificates` — Adds `bootstrap_mode_enabled` parameter to temporarily bypass webhook instance-type anti-affinity during cluster setup
  - Impacts: cli `cluster add` — Automatically manages cert-manager anti-affinity settings during bootstrap and post-Karpenter phases
  - Reference (internal-commit): [fix(cert-manager): add bootstrap_mode_enabled to fix SLA-3 spinup](https://github.com/Panfactum/stack/commit/e65792746f79608e289128bda697e18b64b52239)
  - Reference (internal-docs): [kube_cert_manager module documentation](https://panfactum.com/docs/main/reference/infrastructure-modules/kubernetes/kube_cert_manager)
  - Reference (internal-docs): [kube_certificates module documentation](https://panfactum.com/docs/main/reference/infrastructure-modules/kubernetes/kube_certificates)
- Fixed Linkerd pods failing to start during SLA-3 cluster bootstrap due to instance-type anti-affinity constraints
  - Impacts: iac-module `kube_linkerd` — Adds `bootstrap_mode_enabled` parameter to temporarily bypass instance-type anti-affinity during cluster setup
  - Impacts: cli `cluster add` — Automatically manages Linkerd anti-affinity settings during bootstrap and post-Karpenter phases
  - Reference (internal-commit): [fix(kube_linkerd): disable anti-affinity constraints during cluster bootstrap](https://github.com/Panfactum/stack/commit/619581f64077dbfbf6e98c1b658266d547781134)
  - Reference (internal-docs): [Service mesh bootstrapping guide](https://panfactum.com/docs/main/guides/bootstrapping/service-mesh)
- Fixed `prek` hook installation and autofix detection in multi-worktree Git setups and switched Nix formatter to `nixfmt`
  - Impacts: devshell `git` — `prek` hooks now work correctly across multiple Git worktrees and reliably distinguish autofix runs from real errors
  - Impacts: devshell `enter-shell-local` — Drops `-c` from `prek install` and unsets `core.hooksPath` so each worktree uses its own `.pre-commit-config.yaml`
  - Reference (internal-commit): [fix(dev-tooling): fix prek hooks and worktree install behavior](https://github.com/Panfactum/stack/commit/7d52bf149064f1f8348420a2db8ae77331d48ac4)
  - Reference (external-docs): [prek — faster pre-commit alternative used for Panfactum's git hooks](https://github.com/j178/prek)
- Fixed `kube_ingress_nginx` pods failing to start during cluster bootstrap due to instance-type anti-affinity constraints
  - Impacts: iac-module `kube_ingress_nginx` — Adds `bootstrap_mode_enabled` input to temporarily bypass instance-type anti-affinity during cluster bootstrap
  - Impacts: cli `cluster add` — Automatically sets `bootstrap_mode_enabled` on `kube_ingress_nginx` during bootstrap and clears it once the cluster is fully operational
  - Reference (internal-commit): [feat(kube_ingress_nginx): add bootstrap mode to skip instance anti-affinity](https://github.com/Panfactum/stack/commit/e6966a7265c57c351bf4bff88fa8b56cd6e1ce53)
  - Reference (issue-report): [Pod Anti-Affinity prevents scale up, requires manual pod deletion](https://github.com/kubernetes/autoscaler/issues/5741)
  - Reference (internal-docs): [kube_ingress_nginx module documentation](https://panfactum.com/docs/main/modules/kube_ingress_nginx/overview)
- Fixed `kube_ingress_nginx` replica count calculation during bootstrap mode to prevent pod scheduling failures on clusters with SLA target >= 2.
  - Impacts: iac-module `kube_ingress_nginx` — `bootstrap_mode_enabled` now caps `replicaCount` at 3, preventing scheduling failures on SLA >= 2 clusters during initial cluster deployment
  - Reference (internal-commit): [fix(kube_ingress_nginx): use 3 replicas in bootstrap mode regardless of SLA](https://github.com/Panfactum/stack/commit/4f3f9e9c5c64e824747fea6f05d55efc1b93d716)
  - Reference (internal-docs): [kube_ingress_nginx module reference](https://panfactum.com/docs/main/modules/kube_ingress_nginx/overview)
- Increased default `memory_mb` for `wf_dockerfile_build` workflow pods from 100MB to 150MB to prevent out-of-memory failures during Docker builds.
  - Impacts: iac-module `wf_dockerfile_build` — Default `memory_mb` for workflow pods increased from 100MB to 150MB to handle typical Docker build memory requirements
  - Reference (internal-commit): [fix: bump memory on steps in wf_dockerfile_build](https://github.com/Panfactum/stack/commit/5665081a522208fccd77f18c21337d027a270c3b)
  - Reference (internal-docs): [wf_dockerfile_build module overview](https://panfactum.com/docs/main/modules/wf_dockerfile_build)
- Fixed CLI template generation for `kube_linkerd` and removed obsolete certificate manager templates.
  - Impacts: cli `cluster add` — `kube_linkerd` template generation now uses the correct `kube_certificates` dependency name and obsolete `kube_cert_manager` and `kube_cert_issuers` templates are removed.
  - Reference (internal-commit): [fix(templates): fix linkerd cert deps and add cert module templates](https://github.com/Panfactum/stack/commit/e25a952b7916898c077dcb704db6604cc07079c7)
  - Reference (internal-docs): [kube_linkerd module documentation](https://panfactum.com/docs/main/modules/kube_linkerd)
  - Reference (internal-docs): [kube_certificates module documentation](https://panfactum.com/docs/main/modules/kube_certificates)
- Fixed `kube_aws_lb_controller` webhooks to prevent an unrecoverable bootstrap deadlock by restricting `serviceMutatorWebhook` and `ingressValidationWebhook` scope to `loadbalancer/enabled` namespaces only.
  - Impacts: iac-module `kube_aws_lb_controller` — `serviceMutatorWebhook` and `ingressValidationWebhook` are now scoped to `loadbalancer/enabled` namespaces only, preventing `kube-system` service creation deadlocks during cluster bootstrap
  - Reference (internal-commit): [fix(aws-lb-controller): restrict webhooks to lb-enabled namespaces only](https://github.com/Panfactum/stack/commit/14c87e0f28a12809fe103b1a91ad2655fd293833)
  - Reference (issue-report): [AWS Load Balancer Controller webhook deadlock preventing resource creation](https://github.com/kubernetes-sigs/aws-load-balancer-controller/issues/4140)
  - Reference (issue-report): [fix: Allowing namespace selectors for mservice webhook (upstream PR #4646)](https://github.com/kubernetes-sigs/aws-load-balancer-controller/pull/4646)
  - Reference (internal-docs): [kube_aws_lb_controller module documentation](https://panfactum.com/modules/kube_aws_lb_controller/overview)
- Fixed Cilium pods failing to start on nodes during Karpenter disruption by tolerating both `karpenter.sh/disruption` and `karpenter.sh/disrupted` taints.
  - Impacts: iac-module `kube_cilium` — Cilium pods now tolerate the `karpenter.sh/disrupted` taint to prevent networking outages during active node draining
  - Reference (internal-commit): [fix(kube_cilium): tolerate karpenter.sh/disrupted taint on cilium pods](https://github.com/Panfactum/stack/commit/f083d4346bd5baf413f140b2b57d177cc4225e11)
  - Reference (issue-report): [Deadlock: disrupted taint blocks DaemonSets when do-not-disrupt prevents node disruption](https://github.com/kubernetes-sigs/karpenter/issues/2704)
  - Reference (external-docs): [Karpenter disruption lifecycle documentation](https://karpenter.sh/docs/concepts/disruption/)
  - Reference (internal-docs): [kube_cilium module documentation](https://panfactum.com/modules/kube_cilium/overview)
- Fixed `authentik_core_resources` deployment failing due to missing `AUTHENTIK_TOKEN` environment variable during `sso add` setup.
  - Impacts: cli `sso add` — `authentik_core_resources` deployment step now properly forwards the authentication token, preventing silent Terraform provider failures during SSO setup
  - Reference (internal-commit): [fix(sso): forward AUTHENTIK_TOKEN to core resources deployment](https://github.com/Panfactum/stack/commit/2adceae0c0722bfc3973d2a33fed5e84fed4a239)
  - Reference (internal-docs): [authentik_core_resources module reference](https://panfactum.com/docs/main/modules/authentik_core_resources/overview)
- Fixed fragile skip condition for disabling default Authentik resources in `sso add` command
  - Impacts: cli `sso add` — Default Authentik resources disabling step is now properly idempotent and won't be silently skipped inappropriately
  - Reference (internal-commit): [fix(sso): fix fragile skip for disabling default Authentik resources](https://github.com/Panfactum/stack/commit/84d74a048c1a183fbf1648297c441fa77b266435)
  - Reference (external-docs): [Authentik Identity Provider Documentation](https://docs.goauthentik.io/)
  - Reference (internal-docs): [kube_authentik module documentation](https://panfactum.com/docs/main/modules/kube_authentik/overview)
- Fixed `sso add` command to throw actionable error messages when Authentik domain is missing from module configuration
  - Impacts: cli `sso add` — Now provides clear error messages and remediation guidance when `kube_authentik` domain configuration is missing or malformed
  - Reference (internal-commit): [fix(sso): throw actionable error when authentik domain is missing](https://github.com/Panfactum/stack/commit/7901a721ce816bd99f2f5eadb488a8d8e9f6a826)
  - Reference (internal-docs): [kube_authentik module reference documentation](https://panfactum.com/modules/kube_authentik/reference)
- Fixed Docker Hub PAT validation in `cluster enable` ECR setup to use proper registry token endpoint
  - Impacts: cli `cluster enable` — No longer incorrectly rejects valid Docker Hub PATs during ECR pull-through cache setup
  - Reference (internal-commit): [fix(cli): fix Docker Hub PAT validation and tsc in pre-commit hook](https://github.com/Panfactum/stack/commit/eeed85350ada15d10ca67eb32e91a6317e65cdf0)
  - Reference (external-docs): [Docker registry authentication flow documentation](https://docs.docker.com/reference/api/registry/auth/)
  - Reference (external-docs): [Docker Hub personal access tokens guide](https://docs.docker.com/security/access-tokens/)
  - Reference (internal-docs): [Kubernetes cluster bootstrapping guide covering Docker Hub credentials](https://panfactum.com/docs/main/guides/bootstrapping/kubernetes-cluster)
- Fixed `kube_policies` ECR pull-through cache to route both `docker.io` and `index.docker.io` registry hostnames to the Docker Hub cache
  - Impacts: iac-module `kube_policies` — Added index.docker.io → docker-hub/ mapping to registry_replacements
  - Reference (internal-commit): [fix(kube_policies): route index.docker.io through ECR pull-through cache](https://github.com/Panfactum/stack/commit/8ee8165a8db8ed21096e14df04cfecde0312898d)
  - Reference (external-docs): [Docker Hub Registry Hostnames and API Endpoints](https://stackoverflow.com/questions/34198392/docker-official-registry-docker-hub-url)
  - Reference (external-docs): [Kyverno Policy Examples for Registry Rewriting](https://blog.oponomarov.com/posts/rewriting-docker-image-registries-with-kyverno/)
- Fixed EBS CSI controller `PodDisruptionBudget` failing to update during Helm upgrades due to race condition with other controllers
  - Re-apply the `kube_aws_ebs_csi` module to update the PDB management configuration
  - Impacts: iac-module `kube_aws_ebs_csi` — Disabled Helm-managed `PodDisruptionBudget` and replaced with `kubectl_manifest` using server-side apply to prevent upgrade conflicts
  - Reference (internal-commit): [fix(kube_aws_ebs_csi): use SSA for EBS CSI controller PDB to avoid 409s](https://github.com/Panfactum/stack/commit/be0b6ea7c643b3efd786a98c78a9eafac2e26aa3)
  - Reference (external-docs): [Kubernetes Server-Side Apply documentation](https://kubernetes.io/docs/reference/using-api/server-side-apply/)
  - Reference (issue-report): [AWS EBS CSI driver race condition issue in volume creation](https://github.com/kubernetes-sigs/aws-ebs-csi-driver/issues/1951)
- Fixed `pf buildkit suspend` command crashing when running inside its Kubernetes pod due to git repository lookup failures
  - Impacts: cli `buildkit suspend` — Fixed crash when running in containerized environments, enabling proper cost optimization scaling
  - Reference (internal-commit): [fix(buildkit/suspend): prevent git lookup crash when running in-cluster](https://github.com/Panfactum/stack/commit/ce7399b51922537b02f7951a5d5052724c3a5dae)
  - Reference (internal-docs): [BuildKit concepts and architecture](https://panfactum.com/docs/main/concepts/buildkit)

## Upgrade Instructions

import MarkdownAlert from "@/components/markdown/MarkdownAlert.astro"


## Upgrade `aws_organization` to AWS Provider v6

The [`aws_organization`](https://panfactum.com/docs/edge/reference/infrastructure-modules/direct/aws/aws_organization) module now pins the
`hashicorp/aws` provider to v6 (from 5.x). This is a major provider version bump. In addition to the provider upgrade,
`SERVICE_CONTROL_POLICY` and `TAG_POLICY` are now enabled by default, and trusted access is enabled for a
significantly broader set of AWS governance services.

<MarkdownAlert severity="warning">
  Run `terraform init -upgrade` **before** running `terraform plan` or `terraform apply` in your management environment.
  Applying without reinitializing will fail because the lock file still references the old 5.x provider version.
</MarkdownAlert>

### 1. Review the AWS Provider v6 Upgrade Guide

Before proceeding, read the
[AWS Provider v6 Upgrade Guide](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/guides/version-6-upgrade)
to understand all breaking changes from 5.x. Key removals include OpsWorks, SimpleDB, and WorkLink resources, along
with changes to nullable boolean validation. Confirm your management environment does not use any removed resources or
arguments.

### 2. Update Provider Lock Files

In each management environment module directory that uses `aws_organization` (and any other module that transitively
pins `hashicorp/aws`), run:

```bash
terraform init -upgrade
```

This regenerates the `.terraform.lock.hcl` file with the new v6 provider hash. Commit the updated lock files.

### 3. Review the Plan for `aws_organization`

Run a plan on the `aws_organization` module and carefully review the diff:

```bash
cd <management-environment>/aws_organization
terragrunt plan
```

Expect the following new resources and changes:

- **`SERVICE_CONTROL_POLICY`** and **`TAG_POLICY`** added to `enabled_policy_types` on `aws_organizations_organization`
- **`aws_notifications_organizations_access`** resource created (enables AWS User Notifications trusted access)
- A significantly expanded list of `aws_service_access_principals` (GuardDuty, Security Hub, AWS Backup, Inspector,
  CloudTrail, Access Analyzer, Audit Manager, IPAM, License Manager, Network Manager, and others)

These changes are expected. If any of them are unwanted for your organization, proceed to the next step before applying.

### 4. Opt Out of Unwanted Defaults (If Needed)

If any of the newly-enabled policy types or service access principals are not desired, use the new opt-out variables
**before** running `terraform apply`:

```hcl
inputs = {
  # Disable specific policy types if not needed
  disabled_enabled_policy_types = ["SERVICE_CONTROL_POLICY", "TAG_POLICY"]

  # Disable specific service access principals if not needed
  disabled_aws_service_access_principals = [
    "guardduty.amazonaws.com",
    "securityhub.amazonaws.com",
    # ... add others as needed
  ]

  # Disable AWS User Notifications trusted access if not desired
  enable_notifications_access = false

  # ... other existing inputs
}
```

See the [module reference](https://panfactum.com/docs/edge/reference/infrastructure-modules/direct/aws/aws_organization) for the full
list of opt-out variables.

### 5. Apply `aws_organization`

Once satisfied with the plan, apply the module:

```bash
terragrunt apply
```

## Related Resources

- [JSON Data](https://panfactum.com/docs/changelog/edge.26-04-24.json): Machine-readable data
- [Channel Release List](https://panfactum.com/docs/changelog/edge.json): All releases in this channel