# Panfactum Changelog — edge.26-04-05 > Launches the new `pf` CLI with guided wizards for environment, cluster, domain, and SSO provisioning, upgrades Kubernetes to 1.33 and AWS provider to 6.x, migrates legacy devshell scripts to TypeScript, and consolidates several IaC modules. ## Highlights - New `pf` CLI tool with guided installers — `pf env add`, `pf cluster add`, `pf domain add`, and `pf sso add` automate end-to-end infrastructure provisioning - `kube_cert_manager` and `kube_cert_issuers` consolidated into `kube_certificates` — state migration required - Kubernetes default upgraded to 1.33 — review the [K8s 1.33 changelog](https://kubernetes.io/blog/2025/04/23/kubernetes-v1-33-release/) for deprecated APIs - Node image cache modules (`kube_node_image_cache`, `kube_node_image_cache_controller`) removed — destroy existing deployments before upgrading - Legacy bash devshell scripts migrated to `pf` subcommands — IaC modules now call `pf buildkit`, `pf wf`, and `pf kube` commands - OpenTofu upgraded to 1.9.1 and AWS provider to 6.x — re-apply all modules after upgrading - KEDA added to base cluster — deploy [`kube_keda`](https://panfactum.com/docs/main/reference/infrastructure-modules/direct/kubernetes/kube_keda) before applying other modules ## Breaking Changes - `kube_domain` is now a required `region.yaml` field for every cluster, specifying a dedicated DNS zone for control-plane utilities. - Add a `kube_domain` field to `region.yaml` for every region that contains a Kubernetes cluster. The value must be a subdomain of a domain already available in the environment (e.g., `kube.example.com` if `example.com` is managed). - Deploy the corresponding DNS zone via `aws_dns_zones` before applying cluster modules. - Impacts: configuration `region.yaml` — New required `kube_domain` field specifying the DNS zone for cluster control-plane utilities - Impacts: iac-module `kube_certificates` — Reads `kube_domain` from `region.yaml` to register certificate issuer endpoints under the cluster DNS zone - Impacts: iac-module `kube_cert_issuers` — Reads `kube_domain` from `region.yaml` to include the cluster DNS zone in wildcard certificate coverage - Reference (internal-docs): [Terragrunt variables reference — region.yaml configuration fields](https://panfactum.com/docs/main/reference/configuration/terragrunt-variables) - Reference (internal-docs): [aws_dns_zones module — create and manage Route 53 DNS zones](https://panfactum.com/docs/main/modules/aws_dns_zones) - `kube_authentik` now requires `organization_name` and creates the Authentik email template; update `authentik_core_resources` to consume this output. - Add the `organization_name` variable to your `kube_authentik` module configuration. - Wire the `organization_name` output from `kube_authentik` into the `authentik_core_resources` module as an input via a `dependency` block. - Impacts: iac-module `kube_authentik` — New required `organization_name` input; now creates email template `ConfigMap` and exposes `organization_name` output - Impacts: iac-module `authentik_core_resources` — Must receive `organization_name` wired from `kube_authentik` output via a `dependency` block - Reference (internal-docs): [kube_authentik module reference](https://panfactum.com/docs/main/modules/kube_authentik) - Reference (internal-docs): [authentik_core_resources module reference](https://panfactum.com/docs/main/modules/authentik_core_resources) - `kube_cert_manager` and `kube_cert_issuers` have been consolidated into a single `kube_certificates` module to address various race conditions on cluster installation. - Create a new `kube_certificates` module directory and deploy it using the configuration shown in the upgrade instructions. - Run the state migration script from the upgrade instructions to merge Terraform state from both `kube_cert_manager` and `kube_cert_issuers` into `kube_certificates`. - Remove the `kube_cert_issuers` and `kube_cert_manager` directories after confirming the migration succeeds. - Replace any references to `kube_cert_issuers` or `kube_cert_manager` dependency paths in your `terragrunt.hcl` files with `kube_certificates`. - Impacts: iac-module `kube_cert_manager` — Deprecated and consolidated into `kube_certificates`; directory should be removed after state migration - Impacts: iac-module `kube_cert_issuers` — Deprecated and consolidated into `kube_certificates`; directory should be removed after state migration - Impacts: iac-module `kube_certificates` — New module replacing both `kube_cert_manager` and `kube_cert_issuers`; deploys cert-manager and cluster issuers atomically - Reference (internal-commit): [Merge certificates modules (#353)](https://github.com/Panfactum/stack/commit/48d67a2721e3df5e482db5e975bc76722451dd2a) - Reference (internal-docs): [kube_certificates module reference](https://panfactum.com/docs/main/modules/kube_certificates) - Reference (internal-docs): [Certificate management bootstrapping guide](https://panfactum.com/docs/main/guides/bootstrapping/certificate-management) - `kube_keda` is now required as part of the base Panfactum cluster installation, and other modules assume it is present. - Deploy the `kube_keda` module in every cluster following the [installation guide](https://panfactum.com/docs/main/guides/bootstrapping/autoscaling#deploy-keda). This must be done before applying other modules that now depend on KEDA being present. - Impacts: iac-module `kube_keda` — Now a required part of base cluster installation; must be deployed before other modules that depend on KEDA - Reference (internal-commit): [Add kube_keda module and integrate KEDA into base cluster setup](https://github.com/Panfactum/stack/commit/09bdf9841b6f58d66cb3768cc81a331eb6ce8631) - Reference (external-docs): [KEDA — Kubernetes Event-driven Autoscaling official site](https://keda.sh/) - Reference (internal-docs): [`kube_keda` module reference documentation](https://panfactum.com/docs/main/modules/kube_keda/overview) - Reference (internal-docs): [Bootstrapping guide — Deploy KEDA section](https://panfactum.com/docs/main/guides/bootstrapping/autoscaling) - `burstable_nodes_enabled` now defaults to `true` for all workload modules, restoring the intended behavior that was broken in `edge.25-03-26`. - If you explicitly set `burstable_nodes_enabled = false` in any module, no action is needed. - If you rely on the previous `false` default and your workloads are not compatible with burstable (T-family) instances, explicitly set `burstable_nodes_enabled = false` in the affected module configurations before upgrading. - Review workloads with sustained CPU utilization above ~30% — burstable instances run in unlimited mode and may cost more than standard `M`-type instances for these workloads. - Impacts: iac-module `kube_workload_utility` — `burstable_nodes_enabled` default changed from `false` to `true` - Impacts: iac-module `kube_pod` — Inherits new `burstable_nodes_enabled` default from `kube_workload_utility` - Impacts: iac-module `kube_deployment` — Inherits new `burstable_nodes_enabled` default from `kube_workload_utility` - Impacts: iac-module `kube_stateful_set` — Inherits new `burstable_nodes_enabled` default from `kube_workload_utility` - Impacts: iac-module `kube_cron_job` — Inherits new `burstable_nodes_enabled` default from `kube_workload_utility` - Impacts: iac-module `kube_daemon_set` — Inherits new `burstable_nodes_enabled` default from `kube_workload_utility` - Impacts: iac-module `kube_job` — Inherits new `burstable_nodes_enabled` default from `kube_workload_utility` - Reference (internal-docs): [Deploying workloads guide — node scheduling options](https://panfactum.com/docs/main/guides/deploying-workloads/basics) - Reference (internal-docs): [`kube_workload_utility` module reference](https://panfactum.com/docs/main/modules/kube_workload_utility/overview) - Reference (external-docs): [AWS burstable performance instances documentation](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/burstable-performance-instances.html) - `aws_account` and `aws_registered_domains` contact information variables have been consolidated from individual fields into single object variables per contact type. - In `aws_account`, replace the individual `contact_*` variables (`contact_full_name`, `contact_phone_number`, `contact_address_line_1`, etc.) with a single `primary_contact` object. Similarly, replace `security_*`, `operations_*`, and `billing_*` variables with `security_contact`, `operations_contact`, and `billing_contact` objects respectively. - In `aws_registered_domains`, replace the individual `admin_*` variables (`admin_first_name`, `admin_last_name`, `admin_email_address`, etc.) with a single `admin_contact` object. Similarly, replace `registrant_*` and `tech_*` variables with `registrant_contact` and `tech_contact` objects respectively. - Run `terragrunt apply` for both modules after updating the variable definitions to reconcile the state. - Impacts: iac-module `aws_account` — Individual `contact_*`, `security_*`, `operations_*`, and `billing_*` variables replaced by `primary_contact`, `security_contact`, `operations_contact`, and `billing_contact` typed objects - Impacts: iac-module `aws_registered_domains` — Individual `admin_*`, `registrant_*`, and `tech_*` variables replaced by `admin_contact`, `registrant_contact`, and `tech_contact` typed objects - Reference (internal-commit): [Consolidate `aws_account` contact variables into typed objects](https://github.com/Panfactum/stack/commit/265d0a3fa27c15364fd814fd44d66aac4396bb88) - Reference (internal-commit): [Consolidate `aws_registered_domains` contact variables into typed objects](https://github.com/Panfactum/stack/commit/b9257537a66620e8a531a64fc5f136739e3230d6) - Reference (internal-docs): [`aws_account` module reference](https://panfactum.com/docs/main/reference/infrastructure-modules/direct/aws/aws_account) - Reference (internal-docs): [`aws_registered_domains` module reference](https://panfactum.com/docs/main/reference/infrastructure-modules/direct/aws/aws_registered_domains) - `aws_dns_zones` inputs have been consolidated to a single `domains` object for better per-domain granular configuration. - Migrate your `aws_dns_zones` inputs from separate domain list variables to the new `domains` map. Each key in the map is a domain name, and the value is an object containing per-domain configuration (e.g., `dnssec_enabled`). - Review the updated module reference for the full `domains` input schema. - Impacts: iac-module `aws_dns_zones` — All inputs replaced by a single `domains` map; existing configurations must be migrated - Reference (internal-docs): [aws_dns_zones module reference](https://panfactum.com/docs/main/reference/infrastructure-modules/direct/aws/aws_dns_zones) - `tf_bootstrap_resources` backup vault name now includes a unique suffix. Manually delete the existing `terraform-` vault before applying. - Delete the existing AWS Backup vault named `terraform-` in each environment via the AWS console or `aws backup delete-backup-vault --backup-vault-name terraform-`. - Re-apply the `tf_bootstrap_resources` module to create the replacement vault with the new uniquely-suffixed name. - Impacts: iac-module `tf_bootstrap_resources` — Backup vault name now includes a random unique suffix; existing vault must be manually deleted before applying - Reference (internal-commit): [Add unique suffix to backup vault name in `tf_bootstrap_resources`](https://github.com/Panfactum/stack/commit/d07955d5d1f1de54783cd8baf5c8777f5c9588ed) - Reference (internal-docs): [`tf_bootstrap_resources` module reference](https://panfactum.com/docs/main/reference/infrastructure-modules/direct/aws/tf_bootstrap_resources) - Adds the `pf` CLI tool; run `pf devshell sync` and `terragrunt run-all apply` after upgrading. - Run `pf devshell sync` to synchronize local kubeconfigs, SSH settings, AWS profiles, and standard configuration files with live infrastructure. - Run `terragrunt run-all apply` across all modules to propagate the configuration changes introduced by the sync. - Impacts: cli `devshell sync` — New command that synchronizes local devshell configuration with live infrastructure state - Impacts: devshell `pf` — New unified CLI tool replacing ad-hoc shell scripts with a structured command hierarchy - Reference (internal-commit): [Initial implementation of the `pf` CLI tool](https://github.com/Panfactum/stack/commit/aa49614d10a189ac5e6744b0949f8cf994e27009) - Reference (internal-commit): [Adds `pf devshell sync` tasks for infrastructure synchronization](https://github.com/Panfactum/stack/commit/7b1f505992598d12b16864c9ec33865c97232d9b) - The `PF_SKIP_REPO_CHECK` environment variable used to bypass DevShell repository setup validation has been renamed to `PF_SKIP_CHECK_REPO_SETUP`. - Update any environment configurations, CI pipelines, and scripts that set `PF_SKIP_REPO_CHECK=1` to use `PF_SKIP_CHECK_REPO_SETUP=1` instead. - Impacts: devshell `enter-shell-local` — Renamed `PF_SKIP_REPO_CHECK` to `PF_SKIP_CHECK_REPO_SETUP` - Reference (internal-commit): [Rename `PF_SKIP_REPO_CHECK` to `PF_SKIP_CHECK_REPO_SETUP` in `enter-shell-local.sh`](https://github.com/Panfactum/stack/commit/a9b39c8b808f69cf147c83342fecf49404bad8f3) - `pf install-cluster` has been renamed to `pf cluster add`. Update all scripts referencing the old command. - Replace all invocations of `pf install-cluster` with `pf cluster add` in your scripts, CI pipelines, and runbooks. - Impacts: cli `cluster add` — Renamed from `pf install-cluster`; update scripts and documentation accordingly - Reference (internal-commit): [refactor: rename install-cluster to cluster install](https://github.com/Panfactum/stack/commit/8c03b01043302de2bced3649ebf6e7cc112be3b1) - Reference (internal-commit): [chore: rename 'cluster install' to 'cluster add'](https://github.com/Panfactum/stack/commit/30521a81a5640d770e5e6e1811c76d2eb5de715b) - `pull_through_cache_enabled` now defaults to `true` for several workload and data modules. - If ECR pull-through caching is not configured in your environment, set `pull_through_cache_enabled = false` on `kube_nats`, `kube_pg_cluster`, `kube_redis_sentinel`, `kube_stateful_set`, and `kube_deployment` modules. - Alternatively, deploy the `aws_ecr_pull_through_cache` module to enable pull-through caching before re-applying these modules. - Impacts: iac-module `kube_nats` — `pull_through_cache_enabled` now defaults to `true` - Impacts: iac-module `kube_pg_cluster` — `pull_through_cache_enabled` now defaults to `true` - Impacts: iac-module `kube_redis_sentinel` — `pull_through_cache_enabled` now defaults to `true` - Impacts: iac-module `kube_stateful_set` — `pull_through_cache_enabled` now defaults to `true` - Impacts: iac-module `kube_deployment` — `pull_through_cache_enabled` now defaults to `true` - Reference (internal-commit): [Change `pull_through_cache_enabled` default from `false` to `true` across workload modules](https://github.com/Panfactum/stack/commit/dafb4d5782ba675eb6a0c4eff407ea59ab6f815e) - Reference (external-docs): [AWS ECR pull-through cache documentation](https://docs.aws.amazon.com/AmazonECR/latest/userguide/pull-through-cache-creating-rule.html) - The `update_type` variable has been removed from the `kube_cron_job` module. - Remove any `update_type` input from your `kube_cron_job` module configurations. - Run `terragrunt apply` on affected modules to confirm the configuration is valid after removal. - Impacts: iac-module `kube_cron_job` — Removed the `update_type` input variable - Reference (internal-commit): [Remove `update_type` variable from `kube_cron_job`](https://github.com/Panfactum/stack/commit/dafb4d5782ba675eb6a0c4eff407ea59ab6f815e) - Reference (internal-docs): [`kube_cron_job` module reference](https://panfactum.com/docs/main/reference/infrastructure-modules/submodule/kubernetes/kube_cron_job) - The `aws.secondary` provider alias and cross-region DynamoDB replica have been removed from `tf_bootstrap_resources`. - Remove the `aws.secondary` provider alias from your `tf_bootstrap_resources` Terragrunt configuration if present. - Run `terragrunt apply` on `tf_bootstrap_resources` to remove the cross-region DynamoDB replica from your state lock table. - Impacts: iac-module `tf_bootstrap_resources` — Removed the `aws.secondary` provider `configuration_aliases` entry, the `aws_region.secondary` data source, the `pf_aws_tags.seondary_tags` data source, and the DynamoDB `replica` block for cross-region replication. - Reference (internal-commit): [feat: opensearch alpha state](https://github.com/Panfactum/stack/commit/265d0a3fa27c15364fd814fd44d66aac4396bb88) - Reference (internal-docs): [`tf_bootstrap_resources` module reference](https://panfactum.com/docs/main/reference/infrastructure-modules/direct/aws/tf_bootstrap_resources) - The `kube_node_image_cache` and `kube_node_image_cache_controller` modules have been removed along with all related image caching input variables across workload modules. - Destroy any active `kube_node_image_cache` module deployments before upgrading. - Destroy any active `kube_node_image_cache_controller` module deployments before upgrading. - Remove the module directories for `kube_node_image_cache` and `kube_node_image_cache_controller` from your Terragrunt configuration. - Remove `node_image_cached_enabled` from any module configurations that set it (affected modules include `kube_airbyte`, `kube_alloy`, `kube_argo_event_bus`, `kube_authentik`, `kube_aws_ebs_csi`, `kube_cloudnative_pg`, `kube_gha_runners`, `kube_ingress_nginx`, `kube_linkerd`, `kube_monitoring`, `kube_nats`, `kube_opensearch`, `kube_pg_cluster`, `kube_redis_sentinel`, `kube_vault`). - Remove `node_image_cache_enabled` from any module configurations that set it. - Remove `image_prepull_enabled` and `image_pin_enabled` from all container spec blocks in `kube_pod`, `kube_deployment`, `kube_daemon_set`, `kube_stateful_set`, `kube_cron_job`, and `kube_job` module configurations. - Remove `panfactum_node_image_cache_enabled` from any `kube_policies` module configuration. - Impacts: iac-module `kube_node_image_cache` — Module removed entirely - Impacts: iac-module `kube_node_image_cache_controller` — Module removed entirely - Impacts: iac-module `kube_airbyte` — `node_image_cached_enabled` variable removed - Impacts: iac-module `kube_alloy` — `node_image_cached_enabled` variable removed - Impacts: iac-module `kube_argo_event_bus` — `node_image_cached_enabled` variable removed - Impacts: iac-module `kube_authentik` — `node_image_cached_enabled` variable removed - Impacts: iac-module `kube_aws_ebs_csi` — `node_image_cached_enabled` variable removed - Impacts: iac-module `kube_cloudnative_pg` — `node_image_cached_enabled` variable removed - Impacts: iac-module `kube_gha_runners` — `node_image_cached_enabled` variable removed - Impacts: iac-module `kube_ingress_nginx` — `node_image_cached_enabled` variable removed - Impacts: iac-module `kube_linkerd` — `node_image_cached_enabled` variable removed - Impacts: iac-module `kube_monitoring` — `node_image_cached_enabled` variable removed - Impacts: iac-module `kube_nats` — `node_image_cached_enabled` variable removed - Impacts: iac-module `kube_opensearch` — `node_image_cached_enabled` variable removed - Impacts: iac-module `kube_pg_cluster` — `node_image_cached_enabled` variable removed - Impacts: iac-module `kube_redis_sentinel` — `node_image_cached_enabled` variable removed - Impacts: iac-module `kube_vault` — `node_image_cached_enabled` variable removed - Impacts: iac-module `kube_pod` — `image_prepull_enabled` and `image_pin_enabled` fields removed from container spec inputs - Impacts: iac-module `kube_deployment` — `image_prepull_enabled` and `image_pin_enabled` fields removed from container spec inputs - Impacts: iac-module `kube_daemon_set` — `image_prepull_enabled` and `image_pin_enabled` fields removed from container spec inputs - Impacts: iac-module `kube_stateful_set` — `image_prepull_enabled` and `image_pin_enabled` fields removed from container spec inputs - Impacts: iac-module `kube_cron_job` — `image_prepull_enabled` and `image_pin_enabled` fields removed from container spec inputs - Impacts: iac-module `kube_job` — `image_prepull_enabled` and `image_pin_enabled` fields removed from container spec inputs - Impacts: iac-module `kube_policies` — `panfactum_node_image_cache_enabled` variable removed - Reference (internal-commit): [feat(infrastructure): remove node image cache system](https://github.com/Panfactum/stack/commit/525cdf8407c293a329a91221c91cc4eb73e2007b) - Reference (issue-report): [Kyverno high memory usage when using generate policies](https://github.com/kyverno/kyverno/issues/7278) - The default Kubernetes version in `aws_eks` has been upgraded from 1.30 to 1.33 with updated addon charts. - Review the Kubernetes 1.33 release announcement for any deprecated APIs or behavior changes that affect your workloads. - If you pin `kube_version` explicitly in `aws_eks`, update it to `1.33`. Otherwise, re-apply `aws_eks` to trigger the upgrade. - Re-apply `kube_aws_ebs_csi`, `kube_descheduler`, and `kube_ingress_nginx` to pick up the bundled compatibility fixes. - Impacts: iac-module `aws_eks` — Default Kubernetes version upgraded from 1.30 to 1.33; Bottlerocket AMI and addon Helm charts updated to compatible releases - Impacts: iac-module `kube_aws_ebs_csi` — EBS CSI driver Helm chart pinned to v2.45.1 -- v2.46.0 is incompatible with K8s 1.33 - Impacts: iac-module `kube_descheduler` — `DefaultEvictor` config migrated from flat flags to `podProtections.defaultDisabled` array for descheduler v0.35.x - Impacts: iac-module `kube_ingress_nginx` — `annotations-risk-level` explicitly set to `Critical` to restore snippet annotation support after ingress-nginx 4.12.x - Reference (internal-commit): [feat(infrastructure): upgrade EKS default version to 1.33 with addon updates](https://github.com/Panfactum/stack/commit/b5455f09eb40dd2fac7a7fafc40fc6bbfbac5121) - Reference (external-docs): [Kubernetes v1.33: Octarine release announcement](https://kubernetes.io/blog/2025/04/23/kubernetes-v1-33-release/) - Reference (external-docs): [Amazon EKS Kubernetes 1.33 release notes](https://docs.aws.amazon.com/eks/latest/userguide/kubernetes-versions-standard.html) - Reference (issue-report): [Helm chart v2.46.0 cannot be installed on a default K8s 1.33 cluster](https://github.com/kubernetes-sigs/aws-ebs-csi-driver/issues/2574) - Reference (external-docs): [Descheduler v0.35.0 release -- adds PodProtections for DefaultEvictorArgs](https://github.com/kubernetes-sigs/descheduler/releases/tag/v0.35.0) - Reference (issue-report): [After upgrade to 4.12 version there is strange error appeared: annotation group StreamSnippet contains risky annotation](https://github.com/kubernetes/ingress-nginx/issues/12656) - Reference (internal-docs): [`aws_eks` module reference](https://panfactum.com/docs/main/reference/infrastructure-modules/direct/aws/aws_eks) - Reference (internal-docs): [`kube_aws_ebs_csi` module reference](https://panfactum.com/docs/main/reference/infrastructure-modules/direct/kubernetes/kube_aws_ebs_csi) - Reference (internal-docs): [`kube_descheduler` module reference](https://panfactum.com/docs/main/reference/infrastructure-modules/direct/kubernetes/kube_descheduler) - Reference (internal-docs): [`kube_ingress_nginx` module reference](https://panfactum.com/docs/main/reference/infrastructure-modules/direct/kubernetes/kube_ingress_nginx) - `vpa_enabled` now defaults to `false` in several ingress and infrastructure modules. - If you rely on VPA for autoscaling in `kube_aws_lb_controller`, `kube_external_dns`, `kube_ingress_nginx`, or `vault_core_resources`, add `vpa_enabled = true` to each module's configuration. - Impacts: iac-module `kube_aws_lb_controller` — `vpa_enabled` now defaults to `false` - Impacts: iac-module `kube_external_dns` — `vpa_enabled` now defaults to `false` - Impacts: iac-module `kube_ingress_nginx` — `vpa_enabled` now defaults to `false` - Impacts: iac-module `vault_core_resources` — `vpa_enabled` now defaults to `false` - Reference (internal-commit): [fix: disable vpa by default in ingress modules](https://github.com/Panfactum/stack/commit/cc1da7cb297e608219710dc6534d9280821f4719) - `aws_organization` now requires a new `alias` input to manage the IAM account alias for the management AWS account. - Add an `alias` input to your `aws_organization` module configuration with a human-readable string for the management account (e.g., `my-org-management`). - Run `terragrunt apply` on your `aws_organization` deployment after adding the new input. - Impacts: iac-module `aws_organization` — New required `alias` string input — management account alias is now managed via `aws_iam_account_alias` - Impacts: cli `env add` — Account alias collected during setup is now forwarded to the `aws_organization` module via `defineInputUpdate` - Reference (internal-commit): [fix: set account alias in aws_organization module](https://github.com/Panfactum/stack/commit/fd8d7b26d3cffcebc0b3d37491cb92a76dc7d016) - Reference (internal-docs): [`aws_organization` module documentation](https://panfactum.com/docs/main/reference/infrastructure-modules/aws/aws_organization) - The Authentik API token stored during `pf sso add` has been moved from `authentik_core_resources/secrets.yaml` to `region.secrets.yaml` under the key `authentik_token`. - Copy your existing Authentik token from `/authentik_core_resources/secrets.yaml` (key `authentikUserToken`) to `/region.secrets.yaml` (key `authentik_token`) using SOPS: `sops --set '["authentik_token"] "your-token-here"' region.secrets.yaml`. - Verify that `authentik_token` is present in `region.secrets.yaml` (SOPS-encrypted) before running any Terragrunt modules that use the Authentik provider, as the token is now loaded automatically via `panfactum.hcl`. - Impacts: cli `sso add` — Stores the Authentik API token as `authentik_token` in `region.secrets.yaml` instead of `authentikUserToken` in `authentik_core_resources/secrets.yaml` - Impacts: configuration `region.secrets.yaml` — New `authentik_token` key is required for the Authentik Terraform provider to authenticate - Impacts: configuration `panfactum.hcl` — Reads `authentik_token` from Terragrunt variables and passes it directly to the Authentik provider, removing the need for manual environment variable injection - Reference (internal-commit): [Federated auth (#360)](https://github.com/Panfactum/stack/commit/daaa3b2f592d21ef0990170d7efd92bcc13c7d96) - Reference (internal-commit): [Switch to `config.authentik_token` instead of module output](https://github.com/Panfactum/stack/commit/27cdbcf16525e9989430f33ef211b5a3f9e17560) - Reference (internal-docs): [Terragrunt variables configuration reference](https://panfactum.com/docs/main/reference/configuration/terragrunt-variables) - Removes `namespace` and `service_account` inputs from `kube_certificates` to fix a namespace race condition. - Remove any `namespace` or `service_account` input variable assignments from your `kube_certificates` module configuration. These variables no longer exist and will cause a Terraform error if present. - Impacts: iac-module `kube_certificates` — All Kubernetes resources now reference `local.namespace` instead of `var.namespace`; the now-unused `namespace` and `service_account` input variables have been removed - Reference (internal-commit): [fix: namespace not existing race condition (#362)](https://github.com/Panfactum/stack/commit/3903a9470f20e0be50e630af78cb0235304ff0ae) - Reference (internal-docs): [`kube_certificates` module documentation](https://panfactum.com/docs/main/modules/kube_certificates) - `pf sso add` now deploys Authentik at `sso.` instead of `authentik.`. - Update the `authentik_url` value in `global.yaml` from `https://authentik.` to `https://sso.`. - Update DNS records to point `sso.` to the Authentik ingress. Remove the old `authentik.` record after verifying the new one works. - Re-apply `kube_authentik` by running `terragrunt apply` in its module directory so it picks up the new domain. - Impacts: cli `sso add` — Deploys Authentik at `sso.` instead of `authentik.` - Impacts: configuration `global.yaml` — `authentik_url` must be updated to use the new `sso.` subdomain - Impacts: iac-module `kube_authentik` — The `domain` input now receives `sso.` instead of `authentik.` - Reference (internal-commit): [refactor: standardize SSO subdomain configuration and update domain references](https://github.com/Panfactum/stack/commit/07d42f09a43b367e79433147d0e9c46a083e95b9) - Reference (external-docs): [Authentik identity provider documentation](https://docs.goauthentik.io/) - Reference (internal-docs): [`kube_authentik` module reference](https://panfactum.com/docs/main/reference/infrastructure-modules/kubernetes/kube_authentik) - `kube_opensearch` now enforces a minimum of 1000 MB for `minimum_memory_mb` and raises the default from 25 MB to 4000 MB. - If you previously set `minimum_memory_mb` to a value below 1000, update it to at least 1000 or remove the override to accept the new default of 4000. - If you relied on the previous default of 25 MB, be aware that the new default of 4000 MB will significantly increase the memory request for each OpenSearch pod. Review your node capacity and adjust `minimum_memory_mb` explicitly if needed. - Run `terragrunt apply` on your `kube_opensearch` deployment to apply the new memory configuration. - Impacts: iac-module `kube_opensearch` — `minimum_memory_mb` is now wired into the container spec (previously ignored, hardcoded at 1000 MB), the validation minimum is raised from 25 MB to 1000 MB, and the default is raised from 25 MB to 4000 MB - Reference (internal-commit): [feat: improve OpenSearch memory configuration](https://github.com/Panfactum/stack/commit/d044ca03b10987eb7c93d42fc77f561641201a85) - Reference (internal-docs): [`kube_opensearch` module reference](https://panfactum.com/docs/main/reference/infrastructure-modules/submodule/kubernetes/kube_opensearch) ## Additions - Adds the `kube_job` submodule for running one-off Kubernetes Jobs as part of module deployment processes (e.g., database migrations). - Impacts: iac-module `kube_job` — New submodule providing a production-hardened Kubernetes Job with pod security, VPA, PDB, bin-packing scheduling, and failure policy support - Reference (internal-commit): [Initial implementation of `kube_job` submodule](https://github.com/Panfactum/stack/commit/dafb4d5782ba675eb6a0c4eff407ea59ab6f815e) - Reference (internal-docs): [`kube_job` module documentation](https://panfactum.com/docs/main/modules/kube_job) - Reference (external-docs): [Kubernetes Job documentation](https://kubernetes.io/docs/concepts/workloads/controllers/job/) - Adds the `kube_keda` module for deploying KEDA (Kubernetes Event-Driven Autoscaling) as part of the base cluster installation. - Deploy the `kube_keda` module as part of your cluster bootstrap process. See the related breaking change entry for installation order details. - Impacts: iac-module `kube_keda` — New module deploying KEDA via its official Helm chart with cert-manager TLS, VPA, PDBs, topology spread constraints, and SLA-aware replica configuration - Reference (internal-commit): [Add `kube_keda` module and integrate KEDA into the base cluster setup](https://github.com/Panfactum/stack/commit/09bdf9841b6f58d66cb3768cc81a331eb6ce8631) - Reference (external-docs): [KEDA official website](https://keda.sh/) - Reference (external-docs): [KEDA GitHub repository](https://github.com/kedacore/keda) - Reference (internal-docs): [`kube_keda` module documentation](https://panfactum.com/docs/main/modules/kube_keda/overview) - Reference (internal-docs): [Bootstrapping guide — Deploy KEDA section](https://panfactum.com/docs/main/guides/bootstrapping/autoscaling) - Adds `sub_paths` option to `config_map_mounts` and `secret_mounts` inputs in all workload modules for mounting individual files instead of entire volumes. - Impacts: iac-module `kube_pod` — Added `sub_paths` option to `secret_mounts` and `config_map_mounts` inputs - Impacts: iac-module `kube_deployment` — Added `sub_paths` option to `secret_mounts` and `config_map_mounts` inputs - Impacts: iac-module `kube_stateful_set` — Added `sub_paths` option to `secret_mounts` and `config_map_mounts` inputs - Impacts: iac-module `kube_cron_job` — Added `sub_paths` option to `secret_mounts` and `config_map_mounts` inputs - Impacts: iac-module `kube_job` — Added `sub_paths` option to `secret_mounts` and `config_map_mounts` inputs - Impacts: iac-module `wf_spec` — Added `sub_paths` option to `secret_mounts` and `config_map_mounts` inputs - Reference (internal-commit): [Add `sub_paths` to ConfigMap and Secret mount inputs](https://github.com/Panfactum/stack/commit/dafb4d5782ba675eb6a0c4eff407ea59ab6f815e) - Reference (external-docs): [Kubernetes documentation — using `subPath` to mount specific ConfigMap keys](https://kubernetes.io/docs/tasks/configure-pod-container/configure-pod-configmap/) - Reference (internal-docs): [`kube_pod` module reference](https://panfactum.com/docs/main/modules/kube_pod) - New `install.sh` script automates full Panfactum repository and DevShell setup from scratch. - Impacts: installer `install.sh` — New bootstrapper script that automates prerequisite validation, `flake.nix` generation, DevShell build via `pf devshell sync`, and `direnv` activation - Impacts: devshell `enter-shell-local` — Now supports `PF_SKIP_CHECK_REPO_SETUP=1` to bypass repo setup validation on shell entry - Reference (internal-commit): [chore: creates initial panfactum installer](https://github.com/Panfactum/stack/commit/74bc98bf931abfee89143015332bd3261cefcb59) - Reference (internal-commit): [feat(installer): complete end-to-end install flow with flake.nix, pf-update, and direnv allow](https://github.com/Panfactum/stack/commit/e7ade5824f74f7dc6a778514a23c7da891c3a4fb) - Reference (internal-commit): [fix: installer user input via pipe](https://github.com/Panfactum/stack/commit/dd6a76203cec41092a81ea963b513732159f5a70) - Reference (internal-commit): [fix: installer user input via pipe - take 2](https://github.com/Panfactum/stack/commit/ab63aee9dfda4b84b936583c229ff68d3d14a204) - Reference (internal-commit): [fix: installer git checks and ignore nix warnings](https://github.com/Panfactum/stack/commit/9aa5de525f498086e3f0a9f95f24cf292de8056e) - Reference (internal-commit): [fix: source nix in installer](https://github.com/Panfactum/stack/commit/0b66035e3ed76459ad2e1faa5150a52e509d467d) - Reference (internal-commit): [fix: repo url input for installer](https://github.com/Panfactum/stack/commit/6ff1dbd263c3082b52743eba81be1edd9b42aed7) - Reference (internal-commit): [fix: default repo name for installer](https://github.com/Panfactum/stack/commit/7be27a540ce6a3c7ca1fd52f65d4962ade472045) - Reference (internal-commit): [fix: direnv installer](https://github.com/Panfactum/stack/commit/fd8126dddc92d163f8080babaf07558568606fee) - Reference (internal-commit): [fix: direnv hook check](https://github.com/Panfactum/stack/commit/7b1091c71ab8443e201aa32e95bd7a609ad29e43) - Reference (internal-commit): [fix: auto-append direnv hook to shell config in install.sh](https://github.com/Panfactum/stack/commit/eb60e19e23fe6b4013d109d35121ac875c472618) - Reference (internal-commit): [fix: direnv hook installer](https://github.com/Panfactum/stack/commit/486c6c56ce10edc26253d6a52f789859230978ab) - Reference (internal-commit): [fix: update stage 1 installer to use pf commands](https://github.com/Panfactum/stack/commit/485bb8f898b65caacfb8d9180dec0a111f95c88c) - Reference (internal-commit): [fix: better install instructions + missing region](https://github.com/Panfactum/stack/commit/21d70937a4329fabf4872039ce14b63bb039dc91) - Reference (internal-commit): [fix: reduce the required git version](https://github.com/Panfactum/stack/commit/4b3b81c075f4092137104004181723c904f7fc24) - Reference (internal-commit): [fix: add automatic direnv sourcing](https://github.com/Panfactum/stack/commit/b41077c7f5b6eac6865506c1f27c5f247cfaed04) - Reference (internal-commit): [fix:installer: direnv shell reload warning](https://github.com/Panfactum/stack/commit/eef70afeaf4814a695db8360d8489a246d5bb9c7) - Reference (external-docs): [direnv hook installation documentation](https://direnv.net/docs/hook.html) - Reference (external-docs): [Determinate Systems Nix installer](https://determinate.systems/nix-installer/) - Reference (internal-docs): [Install tooling prerequisites guide](https://panfactum.com/docs/main/guides/getting-started/install-tooling) - Reference (internal-docs): [Boot developer environment guide](https://panfactum.com/docs/main/guides/getting-started/boot-developer-environment) - Adds the `pf config get` CLI command, which merges all applicable Panfactum configuration files and outputs the result as JSON. - Impacts: cli `config get` — New command to retrieve and display the merged Panfactum configuration for a directory - Reference (internal-commit): [chore: cli refactor](https://github.com/Panfactum/stack/commit/6a380d2664c6020bf7fe5e60d7df289fb1b3486a) - Reference (internal-docs): [Terragrunt variables and configuration file hierarchy](https://panfactum.com/docs/main/reference/configuration/terragrunt-variables) - Adds `pf aws vpc-network-test` CLI command for validating VPC network connectivity after deploying `aws_vpc`. - Impacts: cli `aws vpc-network-test` — New first-class `pf` CLI subcommand for testing VPC egress network connectivity - Reference (internal-commit): [feat: cli refactor](https://github.com/Panfactum/stack/commit/31fd6940342ea29532080d6ff2a3a835bbb55995) - Reference (internal-docs): [`aws_vpc` module reference](https://panfactum.com/docs/main/reference/infrastructure-modules/aws/aws_vpc) - Reference (internal-docs): [AWS networking bootstrapping guide](https://panfactum.com/docs/main/guides/bootstrapping/aws-networking) - Adds the `kube_opensearch` module and `opensearch-cli` DevShell tool for deploying and managing OpenSearch clusters on Kubernetes. - Impacts: iac-module `kube_opensearch` — New module deploying a secured OpenSearch cluster with mTLS authentication, RBAC, S3 remote storage, segment replication, and optional Dashboards UI - Impacts: devshell `opensearch-cli` — New CLI tool for interacting with OpenSearch clusters from the DevShell - Reference (internal-commit): [feat: alpha opensearch module](https://github.com/Panfactum/stack/commit/dafb4d5782ba675eb6a0c4eff407ea59ab6f815e) - Reference (external-docs): [OpenSearch official documentation](https://opensearch.org/docs) - Reference (internal-docs): [`kube_opensearch` module overview](https://panfactum.com/docs/main/modules/kube_opensearch/overview) - Adds the `aws_organization` IaC module and `pf env add` CLI command for automating new Panfactum environment provisioning. - Impacts: iac-module `aws_organization` — New module for managing AWS Organization structure and member accounts - Impacts: cli `env add` — New guided wizard for installing a new Panfactum environment end-to-end - Reference (internal-commit): [Add `aws_organization` module and `pf env add` CLI command](https://github.com/Panfactum/stack/commit/265d0a3fa27c15364fd814fd44d66aac4396bb88) - Reference (internal-docs): [`aws_organization` module documentation](https://panfactum.com/docs/main/reference/infrastructure-modules/direct/aws/aws_organization) - Adds `pf domain add`, a guided wizard for adding a domain to a Panfactum environment. - Impacts: cli `domain add` — New guided wizard that automates domain registration, DNS zone deployment, and cross-environment subdomain delegation - Reference (internal-commit): [Add `pf domain add` CLI command for guided domain onboarding](https://github.com/Panfactum/stack/commit/7b1f505992598d12b16864c9ec33865c97232d9b) - Reference (internal-docs): [DNS bootstrapping guide](https://panfactum.com/docs/main/guides/bootstrapping/dns) - Reference (internal-docs): [Subdomain delegation concept](https://panfactum.com/docs/main/concepts/networking/subdomain-delegation) - Adds the `pf welcome` CLI command with onboarding content, auto-launch on DevShell startup, and a persisted `installation_id` in `panfactum.yaml`. - Impacts: cli `welcome` — New command; displays Environments/Regions/Modules layout, Getting Started steps, and Getting Help links; generates a UUID `installation_id` saved to `panfactum.yaml` on first run - Impacts: devshell `enter-shell-local` — Now calls `pf welcome` on shell launch to display the onboarding screen - Impacts: configuration `panfactum.yaml` — New optional `installation_id` field (UUID) written automatically by `pf welcome` on first run - Reference (internal-commit): [fix: undefined import](https://github.com/Panfactum/stack/commit/0d1327c722d578fbe23280fc0ff98efaa0f0c10e) - Reference (internal-commit): [fix: nameservers output in cli](https://github.com/Panfactum/stack/commit/23a2d48fabd479261dbbf07c87101cf0322dc3c8) - Reference (internal-commit): [feat: adds welcome screen](https://github.com/Panfactum/stack/commit/11c41aa1fc01f6067bc76d2822199bc39bbfa40d) - Reference (internal-commit): [feat: adds welcome screen to devshell launch](https://github.com/Panfactum/stack/commit/d8d40050f39848b34c8c9a684211850db03ffe2f) - Reference (internal-commit): [feat: adds installation_id](https://github.com/Panfactum/stack/commit/bc4b30db4802ea40f7f9096dc17c1a929103cbc8) - Reference (internal-docs): [Installing the DevShell](https://panfactum.com/docs/main/guides/bootstrapping/installing-devshell) - Reference (internal-docs): [Customizing the Development Shell](https://panfactum.com/docs/main/guides/development-shell/customizing) - `aws_account` now automatically requests an EC2 on-demand vCPU quota increase to 32. - Impacts: iac-module `aws_account` — Adds a new `aws_servicequotas_service_quota` resource that requests the EC2 on-demand standard vCPU quota be raised to 32 on apply. - Reference (internal-commit): [feat: automatically request increase in vcpu quota in aws_account module](https://github.com/Panfactum/stack/commit/6496caab324c847f53e59b82ecd948e799f7df13) - Reference (external-docs): [Amazon EC2 instance type quotas documentation](https://docs.aws.amazon.com/ec2/latest/instancetypes/ec2-instance-quotas.html) - Reference (external-docs): [How to request an EC2 On-Demand Instance vCPU quota increase](https://repost.aws/knowledge-center/ec2-on-demand-instance-vcpu-increase) - `pf env add` displays a success message after bootstrapping the management environment. - Impacts: cli `env add` — Displays a success message after the management environment bootstrap explaining its purpose, IaC file location, and next steps. - Reference (internal-commit): [fix:cli: aws client fixes + welcome screen + org create success section](https://github.com/Panfactum/stack/commit/572dddd507d8318b9ebd48dedc47f4351f73626d) - `pf config get` now reads SOPS-encrypted `*.secrets.yaml` files, corrects config load order, and handles missing `secrets.yaml` in `pf cluster add`. - Impacts: cli `config get` — Load order corrected (global, environment, region, module); SOPS-encrypted `*.secrets.yaml` files now decrypted and merged in the same precedence order - Impacts: cli `cluster add` — Steps that read the Vault root token now handle a missing `secrets.yaml` gracefully instead of crashing with a destructuring error - Impacts: configuration `global.secrets.yaml` — New SOPS-encrypted config file read by `pf config get` at the global level - Impacts: configuration `environment.secrets.yaml` — New SOPS-encrypted config file read by `pf config get` at the environment level - Impacts: configuration `region.secrets.yaml` — New SOPS-encrypted config file read by `pf config get` at the region level - Impacts: configuration `module.secrets.yaml` — New SOPS-encrypted config file read by `pf config get` at the module level - Reference (internal-commit): [fix:cli: config file precedence and secret config files](https://github.com/Panfactum/stack/commit/474c4b6ddd5d82f1bcacf43fe564cff67c9cd890) - Reference (internal-docs): [Terragrunt variables and config file hierarchy](https://panfactum.com/docs/main/reference/configuration/terragrunt-variables) - Reference (external-docs): [SOPS — Secrets OPerationS encryption tool](https://getsops.io/docs/) - Adds the `pf sso add` CLI command for guided Authentik and AWS federated SSO installation. - Impacts: cli `sso add` — New command that automates Authentik deployment and AWS federated SSO configuration - Impacts: iac-module `kube_authentik` — Adds `organization_name` input and moves email template management into the module - Impacts: iac-module `authentik_core_resources` — Removes email template resources (moved to `kube_authentik`) and adjusts branding - Reference (internal-commit): [Authentik SSO in CLI installer (#359)](https://github.com/Panfactum/stack/commit/a3febb0f7ea71f7eb58968aac0223e6360264465) - Reference (external-docs): [Authentik identity provider documentation](https://docs.goauthentik.io/) - Adds `pf iac update-module-status` to track module `init_status` and `deploy_status` in `.pf.yaml`, enabling step resumability in `pf cluster add`. - Impacts: cli `iac update-module-status` — New command that writes `init_status` and `deploy_status` to a module's `.pf.yaml` file - Impacts: configuration `panfactum.hcl` — Terragrunt hooks replaced inline `bash` status writes with `pf iac update-module-status` invocations for `init` and `apply` phases - Impacts: cli `cluster add` — Module resume logic now reads `deploy_status` from `.pf.yaml` instead of the previous `status` field - Reference (internal-commit): [feat: enhances module status architecture](https://github.com/Panfactum/stack/commit/00029f4e98aec9c1119bba2ff2a7fcb2d228620b) - Reference (internal-commit): [feat: misc cli improvements](https://github.com/Panfactum/stack/commit/e19f5da99390eed58ae797f556578af89d98532f) - Reference (internal-commit): [fix: module status update](https://github.com/Panfactum/stack/commit/eaa53405e53cf5eb0926a18daaa6561ea729030c) - Reference (internal-commit): [fix: module status update](https://github.com/Panfactum/stack/commit/0ac254981c3829dc40dc6453e0bd355cd187747d) - `pf sso add` now automates federated auth with IAM Identity Center and adds `authentik_token` to the Panfactum config schema. - Impacts: cli `sso add` — Federated auth (AWS SSO SAML + SCIM) is now automated as a resumable step within `pf sso add` - Impacts: cli `config get` — `authentik_token` is now a recognized field in the Panfactum config schema - Impacts: configuration `panfactum.hcl` — The `authentik_provider` generated block now passes `authentik_token` from the config to the Authentik provider - Reference (internal-commit): [Federated auth (#360)](https://github.com/Panfactum/stack/commit/daaa3b2f592d21ef0990170d7efd92bcc13c7d96) - Reference (external-commit): [Federated auth PR #360](https://github.com/Panfactum/stack/pull/360) - Reference (internal-docs): [Federated auth bootstrapping guide](https://panfactum.com/docs/main/guides/bootstrapping/federated-auth) - Reference (internal-docs): [Identity provider bootstrapping guide](https://panfactum.com/docs/main/guides/bootstrapping/identity-provider) - `pf cluster add` now checks the EC2 vCPU quota and warns if it is below 16. - Impacts: cli `cluster add` — Validates EC2 vCPU quota is at least 16 before starting cluster installation - Reference (internal-commit): [Add vCPU quota check to `pf cluster add`](https://github.com/Panfactum/stack/commit/5cd779112382e803f6b4e20ce35f7f76788ebff0) - Reference (internal-commit): [Use factory function for `ServiceQuotasClient` instantiation](https://github.com/Panfactum/stack/commit/1dc45f74fb324cf0a8cd35be642b2ef3f1a244b5) - Reference (issue-report): [AWS SDK JS v3: `profile` option does not resolve credentials from file](https://github.com/aws/aws-sdk-js-v3/issues/6872) - `pf cluster add` auto-deploys Vault Federated SSO when an Authentik instance is detected. - Impacts: cli `cluster add` — After the cluster extensions phase, searches all environments for a deployed `kube_authentik` module and auto-deploys Vault SSO if found - Impacts: iac-module `authentik_vault_sso` — Deployed automatically by `pf cluster add` when Authentik is present - Impacts: iac-module `vault_auth_oidc` — Deployed automatically by `pf cluster add` when Authentik is present - Reference (internal-commit): [fix: standardize AWS region parameter handling and improve SSO flexibility](https://github.com/Panfactum/stack/commit/67143b00110420fe0253a47e18988b072350f8a0) - Reference (internal-docs): [`vault_auth_oidc` module overview](https://panfactum.com/docs/main/modules/vault_auth_oidc/overview) - Adds `pf cluster enable` command for activating optional cluster features and a `--cwd` flag to all `pf` CLI commands. - Impacts: cli `cluster enable` — New command providing a guided wizard for activating opt-in cluster features such as `ecr-pull-through-cache` - Impacts: cli `cluster add` — Now respects `--cwd` for repository root resolution - Impacts: cli `config get` — Now respects `--cwd` for repository root resolution - Impacts: iac-module `aws_ecr_pull_through_cache` — Deployed automatically by `pf cluster enable` when activating the `ecr-pull-through-cache` feature - Impacts: iac-module `kube_policies` — Re-deployed with `pull_through_cache_enabled` set to `true` after ECR pull-through cache setup - Reference (internal-commit): [feat: implement cluster enable command with ECR pull-through cache support](https://github.com/Panfactum/stack/commit/654a4ad80408b2347cdd87aed1623d28bb5c3725) - Reference (internal-docs): [`aws_ecr_pull_through_cache` module overview](https://panfactum.com/docs/main/modules/aws_ecr_pull_through_cache/overview) - Reference (internal-docs): [Kubernetes cluster bootstrapping guide covering ECR pull-through cache setup](https://panfactum.com/docs/main/guides/bootstrapping/kubernetes-cluster) - Adds anonymous CLI usage telemetry via PostHog and a new `panfactum.user.yaml` per-user config file. - Impacts: cli `welcome` — Generates a UUID `user_id` on first run, stores it in `panfactum.user.yaml`, and fires a `cli-welcome` PostHog event - Impacts: configuration `panfactum.user.yaml` — New optional per-user config file whose values merge on top of `panfactum.yaml` and are gitignored automatically - Impacts: cli `devshell sync` — `*.user.yaml` files are now added to the root and environments `.gitignore` entries - Reference (internal-commit): [feat: add anonymous usage tracing](https://github.com/Panfactum/stack/commit/5d6c7ee80a7966988687ca691f5080b28599c772) - Reference (external-docs): [PostHog product analytics platform](https://posthog.com/) - Reference (internal-docs): [Panfactum repo variables reference](https://panfactum.com/docs/main/reference/configuration/repo-variables) - Adds TypeScript `pf` CLI subcommands replacing legacy bash devshell scripts and updates IaC modules to use the new CLI. - Impacts: cli `buildkit build` — New CLI command replacing the `pf-buildkit-build` devshell script - Impacts: cli `buildkit clear-cache` — New CLI command replacing the `pf-buildkit-clear-cache` devshell script - Impacts: cli `buildkit get-address` — New CLI command replacing the `pf-buildkit-get-address` devshell script - Impacts: cli `buildkit record-build` — New CLI command replacing the `pf-buildkit-record-build` devshell script - Impacts: cli `buildkit suspend` — New CLI command replacing the `pf-buildkit-scale-down` devshell script - Impacts: cli `buildkit tunnel` — New CLI command replacing the `pf-buildkit-tunnel` devshell script - Impacts: cli `db get-creds` — New CLI command replacing the `pf-get-db-creds` devshell script - Impacts: cli `db tunnel` — New CLI command replacing the `pf-db-tunnel` devshell script - Impacts: cli `docker credential-helper` — New CLI command replacing the `docker-credential-panfactum` devshell script - Impacts: cli `kube disable-disruptions` — New CLI command replacing the `pf-voluntary-disruptions-disable` devshell script - Impacts: cli `kube enable-disruptions` — New CLI command replacing the `pf-voluntary-disruptions-enable` devshell script - Impacts: cli `kube cluster-resume` — New CLI command replacing the `pf-eks-resume` devshell script - Impacts: cli `kube cluster-suspend` — New CLI command replacing the `pf-eks-suspend` devshell script - Impacts: cli `kube get-token` — New CLI command replacing the `pf-get-kube-token` devshell script - Impacts: cli `kube profile-for-context` — New CLI command replacing the `pf-get-aws-profile-for-kube-context` devshell script - Impacts: cli `k8s velero snapshot-gc` — New CLI command replacing the `pf-velero-snapshot-gc` devshell script - Impacts: cli `tunnel` — New CLI command replacing the `pf-tunnel` devshell script - Impacts: cli `util get-commit-hash` — New CLI command replacing the `pf-get-commit-hash` devshell script - Impacts: cli `util get-module-hash` — New CLI command replacing the `pf-get-local-module-hash` devshell script - Impacts: cli `vault get-token` — New CLI command replacing the `pf-get-vault-token` devshell script - Impacts: cli `wf git-checkout` — New CLI command replacing the `pf-wf-git-checkout` devshell script - Impacts: cli `wf sops-set-profile` — New CLI command replacing the `pf-sops-set-profile` devshell script - Impacts: cli `aws ecr wait-on-image` — New CLI command for polling ECR until a specific image tag becomes available - Impacts: cli `iac delete-locks` — New CLI command replacing the `pf-tf-delete-locks` devshell script - Impacts: iac-module `kube_buildkit` — Scale-to-zero and cache-clear cron jobs now run `pf buildkit suspend` and `pf buildkit clear-cache` - Impacts: iac-module `kube_disruption_window_controller` — Enabler and disabler cron jobs now run `pf kube enable-disruptions` and `pf kube disable-disruptions` - Impacts: iac-module `kube_velero` — Snapshot garbage-collection cron job now runs `pf kube velero-snapshot-gc` - Impacts: iac-module `wf_dockerfile_build` — Build, clone, scale-buildkit, and setup scripts now call `pf buildkit` and `pf wf` subcommands - Impacts: iac-module `wf_tf_deploy` — Deploy and force-unlock scripts now call `pf wf git-checkout`, `pf wf sops-set-profile`, `pf config get`, and `pf iac delete-locks` - Reference (internal-commit): [refactor: migrate bash scripts to TypeScript CLI commands](https://github.com/Panfactum/stack/commit/dcfa7211e0afa76c8656bb706b61cac0a5dea76f) - Reference (internal-docs): [`kube_buildkit` module documentation](https://panfactum.com/docs/main/reference/infrastructure-modules/kubernetes/kube_buildkit) - Reference (internal-docs): [`wf_dockerfile_build` module documentation](https://panfactum.com/docs/main/reference/infrastructure-modules/workflow/wf_dockerfile_build) - Reference (internal-docs): [`wf_tf_deploy` module documentation](https://panfactum.com/docs/main/reference/infrastructure-modules/workflow/wf_tf_deploy) - `kube_karpenter_node_pools` now creates dedicated workflow-only Karpenter node pools for each existing pool type. - Impacts: iac-module `kube_karpenter_node_pools` — Six new workflow-only `NodePool` resources added (one per existing pool type); each carries a `workflow=true:NoSchedule` taint, a `panfactum.com/workflow-only=true` label, and a new `default_termination_grace_period` variable for on-demand pools - Reference (internal-commit): [feat: adds workflow-only node pools](https://github.com/Panfactum/stack/commit/2702b0c70a3f8aed03ea6606ba291b6e2b380885) - Reference (internal-docs): [`kube_karpenter_node_pools` module documentation](https://panfactum.com/docs/main/reference/infrastructure-modules/direct/kubernetes/kube_karpenter_node_pools) - Reference (external-docs): [Karpenter NodePool documentation](https://karpenter.sh/docs/concepts/nodepools/) - Adds a `workflow_nodes_required` input to workflow and workload modules to schedule pods exclusively on dedicated workflow-only nodes. - Impacts: iac-module `wf_spec` — New `workflow_nodes_required` input (default `false`) constrains workflow pods to workflow-only nodes - Impacts: iac-module `kube_workload_utility` — New `workflow_nodes_required` input adds a `panfactum.com/workflow-only` node selector and `workflow=true:NoSchedule` taint toleration - Reference (internal-commit): [feat: adds workflow-only node support to wf_spec](https://github.com/Panfactum/stack/commit/30f40bd6c4414685d5923bfe6b0029557acab1d3) - Reference (internal-docs): [`wf_spec` module reference](https://panfactum.com/docs/main/modules/wf_spec) - Reference (internal-docs): [`kube_workload_utility` module reference](https://panfactum.com/docs/main/modules/kube_workload_utility) - Reference (external-docs): [Karpenter NodePools documentation](https://karpenter.sh/docs/concepts/nodepools) - `kube_opensearch` now supports single-node deployments via `replica_count=1`. - Impacts: iac-module `kube_opensearch` — `replica_count` minimum of 3 removed; setting `replica_count=1` now configures `discovery.type=single-node` and omits multi-node seed hosts configuration - Reference (internal-commit): [feat: adds the ability to run opensearch in single-node mode](https://github.com/Panfactum/stack/commit/14cb9e692705dfa5b13f113d9699abb6df58b97c) - Reference (external-docs): [OpenSearch discovery and cluster formation settings](https://docs.opensearch.org/latest/tuning-your-cluster/discovery-cluster-formation/settings/) - `kube_pg_cluster` adds `pg_recovery_target_immediate` input for `targetImmediate` CNPG recovery mode. - Impacts: iac-module `kube_pg_cluster` — New optional `pg_recovery_target_immediate` input enables `targetImmediate` CNPG recovery mode as an alternative to `pg_recovery_target_time` - Reference (internal-commit): [feat(kube_pg_cluster): support targetImmediate recovery (#375)](https://github.com/Panfactum/stack/commit/aaa63ac02ce6adee3e990bc8c45c49a1ca2089c0) - Reference (external-commit): [PR #375: feat(kube_pg_cluster): support targetImmediate recovery](https://github.com/Panfactum/stack/pull/375) - Reference (external-docs): [CloudNativePG recovery documentation — recovery targets including `targetImmediate`](https://cloudnative-pg.io/docs/1.29/recovery/) - Reference (internal-docs): [`kube_pg_cluster` module reference documentation](https://panfactum.com/docs/main/reference/infrastructure-modules/submodule/kubernetes/kube_pg_cluster) - `kube_pg_cluster` adds a `pg_custom_image` input to support custom PostgreSQL container images with extensions. - Impacts: iac-module `kube_pg_cluster` — New optional `pg_custom_image` input overrides `pg_version` for image selection when set - Reference (internal-commit): [feat(kube_pg_cluster): support custom_image (#376)](https://github.com/Panfactum/stack/commit/5809f908864efcc93847361fabefc7abfa764d61) - Reference (external-docs): [CloudNativePG blog: building custom PostgreSQL images with Docker Bake](https://cloudnative-pg.io/blog/building-images-bake/) - Reference (internal-docs): [`kube_pg_cluster` module documentation](https://panfactum.com/docs/main/reference/infrastructure-modules/submodule/kubernetes/kube_pg_cluster) - `aws_ecr_repos` now accepts a `lifecycle_policy_json` input to override the default ECR lifecycle policy per repository. - Impacts: iac-module `aws_ecr_repos` — New optional `lifecycle_policy_json` field on each repository object; overrides built-in expiration rules when set - Reference (internal-commit): [feat(aws_ecr_repos): support custom lifecycle policies (#379)](https://github.com/Panfactum/stack/commit/292b996bfdc0aef6c102493da0dbe4221548d0e3) - Reference (issue-report): [Support custom ECR lifecycle policies in aws_ecr_repos](https://github.com/Panfactum/stack/issues/381) - Reference (internal-docs): [`aws_ecr_repos` module documentation](https://panfactum.com/docs/main/reference/infrastructure-modules/direct/aws/aws_ecr_repos) - Reference (external-docs): [AWS ECR lifecycle policy documentation](https://docs.aws.amazon.com/AmazonECR/latest/userguide/LifecyclePolicies.html) - Adds optional `seccomp_profile_type` field to container specs in all `kube_*` workload modules, enabling per-container seccomp profile overrides. - Impacts: iac-module `kube_pod` — New optional `seccomp_profile_type` field (default `null`) on each container object; when set, applies a container-level `seccompProfile` override in the pod security context - Impacts: iac-module `kube_deployment` — Exposes new optional `seccomp_profile_type` field on each container object, propagated to the underlying `kube_pod` - Impacts: iac-module `kube_daemon_set` — Exposes new optional `seccomp_profile_type` field on each container object, propagated to the underlying `kube_pod` - Impacts: iac-module `kube_stateful_set` — Exposes new optional `seccomp_profile_type` field on each container object, propagated to the underlying `kube_pod` - Impacts: iac-module `kube_cron_job` — Exposes new optional `seccomp_profile_type` field on each container object, propagated to the underlying `kube_pod` - Impacts: iac-module `kube_job` — Exposes new optional `seccomp_profile_type` field on each container object, propagated to the underlying `kube_pod` - Impacts: iac-module `kube_buildkit` — Sets `seccomp_profile_type` to `Unconfined` to satisfy BuildKit's elevated syscall requirements - Reference (internal-commit): [feat(infrastructure): add seccomp_profile_type to container specs](https://github.com/Panfactum/stack/commit/c6e79ec355572f2d89ada7ceaac8b1aab481cc23) - Reference (external-docs): [Kubernetes seccomp profiles documentation](https://kubernetes.io/docs/tutorials/security/seccomp/) - Reference (internal-docs): [`kube_pod` module documentation](https://panfactum.com/docs/main/modules/kube_pod) - Reference (internal-docs): [`kube_buildkit` module documentation](https://panfactum.com/docs/main/modules/kube_buildkit) ## Version Updates - Updates `kube_logging` to Loki 6.29.0 with tuned caching and query parallelism, and drops high-cardinality labels in `kube_alloy`. - Impacts: iac-module `kube_logging` — Loki chart updated from 6.6.2 to 6.29.0; default `log_level` changed to `info`; Redis cache TTLs extended to 12 h; query parallelism and compaction interval tuned - Impacts: iac-module `kube_node_settings` — `max-locked-memory` OCI resource limit set to `unlimited` for OpenSearch and other `mlock` workloads - Impacts: iac-module `kube_alloy` — High-cardinality labels (`filename`, `job`, `pod`) dropped from log streams before forwarding to Loki - Reference (internal-commit): [feat: opensearch alpha state](https://github.com/Panfactum/stack/commit/265d0a3fa27c15364fd814fd44d66aac4396bb88) - Reference (external-docs): [Loki Helm chart 6.29.0 on Artifact Hub](https://artifacthub.io/packages/helm/grafana/loki/6.29.0) - Reference (external-docs): [Grafana Loki GitHub repository](https://github.com/grafana/loki) - Reference (internal-docs): [`kube_logging` module reference](https://panfactum.com/docs/main/reference/infrastructure-modules/direct/kubernetes/kube_logging) - Reference (internal-docs): [`kube_alloy` module reference](https://panfactum.com/docs/main/reference/infrastructure-modules/direct/kubernetes/kube_alloy) - `kube_authentik` default Helm chart version bumped from 2024.8.4 to 2024.10.5, picking up two minor releases of upstream improvements. - Impacts: iac-module `kube_authentik` — Default Helm chart version updated from 2024.8.4 to 2024.10.5 - Reference (internal-commit): [chore(kube_authentik): bump default Authentik helm chart to 2024.10.5](https://github.com/Panfactum/stack/commit/1ab7f7d7f223bd1cfd73d4c8fcd09d5d22b3dde0) - Reference (external-docs): [Authentik 2024.10 release notes](https://docs.goauthentik.io/releases/2024.10/) - Reference (external-docs): [Authentik 2024.8 release notes](https://docs.goauthentik.io/releases/2024.8/) - Reference (external-docs): [Announcing Authentik 2024.10](https://goauthentik.io/blog/2024-10-31-announcing-release-2024-10/) - `kube_buildkit` upgrades the default BuildKit version from `v0.18.1` to `v0.28.1`. - Impacts: iac-module `kube_buildkit` — Default BuildKit image version updated from `v0.18.1-rootless` to `v0.28.1-rootless` - Reference (internal-commit): [feat(buildkit): upgrade BuildKit from v0.18.1 to v0.28.1](https://github.com/Panfactum/stack/commit/7f7db1fa18e34e7751bfbd47b60f9ad8330385e2) - Reference (external-docs): [BuildKit GitHub repository](https://github.com/moby/buildkit) - Reference (external-docs): [BuildKit v0.28.1 release notes](https://github.com/moby/buildkit/releases/tag/v0.28.1) - Reference (internal-docs): [`kube_buildkit` module documentation](https://panfactum.com/docs/main/modules/kube_buildkit/overview) - Upgrades DevShell to OpenTofu 1.9.1 and Terragrunt 0.78.2 via a refreshed `nixos-25.05` nixpkgs pin. - Impacts: devshell `tofu` — Upgraded from 1.8.x to 1.9.1; adds provider `for_each` and `-exclude` flag - Impacts: devshell `terragrunt` — Upgraded to 0.78.2 via `nixos-25.05` nixpkgs pin - Reference (internal-commit): [build(tooling): upgrade OpenTofu 1.8→1.9 via nixos-25.05 nixpkgs pin](https://github.com/Panfactum/stack/commit/e9738e4241e1e2925e18e419125b4d56fe5086c3) - Reference (external-docs): [OpenTofu 1.9.0 release announcement — provider `for_each`](https://opentofu.org/blog/opentofu-1-9-0/) - Reference (external-docs): [OpenTofu 1.9 — what's new (provider iteration and `-exclude` flag)](https://opentofu.org/docs/v1.9/intro/whats-new) - Reference (external-docs): [Terragrunt v0.78.0 release](https://github.com/gruntwork-io/terragrunt/releases/tag/v0.78.0) - Upgrades the AWS Load Balancer Controller to `v2.12.0` (Helm chart `1.12.0`) in `kube_aws_lb_controller`. - Impacts: iac-module `kube_aws_lb_controller` — Default Helm chart version updated from `1.11.0` to `1.12.0` (app version `v2.12.0`) - Reference (internal-commit): [Upgrade `aws-load-balancer-controller` Helm chart from `1.11.0` to `1.12.0`](https://github.com/Panfactum/stack/commit/b949ef07943277aebe73c457d477c2235480251d) - Reference (external-docs): [AWS Load Balancer Controller `v2.12.0` release notes](https://github.com/kubernetes-sigs/aws-load-balancer-controller/releases/tag/v2.12.0) - Reference (external-docs): [Helm chart `1.12.0` on Artifact Hub](https://artifacthub.io/packages/helm/aws/aws-load-balancer-controller/1.12.0) - Reference (external-docs): [AWS Load Balancer Controller GitHub repository](https://github.com/kubernetes-sigs/aws-load-balancer-controller) - Reference (internal-docs): [`kube_aws_lb_controller` module reference](https://panfactum.com/docs/main/reference/infrastructure-modules/direct/kubernetes/kube_aws_lb_controller) - The Panfactum DevShell now ships Node.js v25, upgraded from v22. - Impacts: devshell `enter-shell-local` — Node.js runtime upgraded from `nodejs_22` to `nodejs_25` - Reference (internal-commit): [Upgrade Node.js from v22 to v25 and refresh nixpkgs input](https://github.com/Panfactum/stack/commit/eced1ab337f51c01a7a045c34a372cf1aae95321) - Reference (external-docs): [Node.js v25.0.0 release notes](https://nodejs.org/en/blog/release/v25.0.0) - Reference (external-docs): [Node.js v25 changelog](https://github.com/nodejs/node/blob/main/doc/changelogs/CHANGELOG_V25.md) - All `authentik_*` modules updated to `goauthentik/authentik` provider 2024.10.2 with `invalidation_flow` support for SAML and OAuth2 providers. - Run `pf-tf-init` in each affected module directory to pick up the new provider version. - Re-apply all `authentik_*` modules and `mongodb_atlas_identity_provider` with `terragrunt apply`. - Impacts: iac-module `authentik_aws_sso` — `goauthentik/authentik` provider bumped from 2024.8.4 to 2024.10.2; SAML provider now sets `invalidation_flow` to `default-provider-invalidation-flow` as required by Authentik 2024.10 - Impacts: iac-module `authentik_core_resources` — `goauthentik/authentik` provider bumped from 2024.8.4 to 2024.10.2 - Impacts: iac-module `authentik_github_sso` — `goauthentik/authentik` provider bumped from 2024.8.4 to 2024.10.2; SAML provider now sets `invalidation_flow` to `default-provider-invalidation-flow` - Impacts: iac-module `authentik_mongodb_atlas_sso` — `goauthentik/authentik` provider bumped from 2024.8.4 to 2024.10.2; SAML provider now sets `invalidation_flow` to `default-provider-invalidation-flow` - Impacts: iac-module `authentik_vault_sso` — `goauthentik/authentik` provider bumped from 2024.8.4 to 2024.10.2; OAuth2 provider now sets `invalidation_flow` and uses `allowed_redirect_uris` instead of `redirect_uris` - Impacts: iac-module `authentik_zoho_sso` — `goauthentik/authentik` provider bumped from 2024.8.4 to 2024.10.2; SAML provider now sets `invalidation_flow` to `default-provider-invalidation-flow` - Impacts: iac-module `mongodb_atlas_identity_provider` — `goauthentik/authentik` provider bumped from 2024.8.4 to 2024.10.2 - Reference (internal-commit): [refactor(hooks): move formatters and linters to Stop hook for batch runs](https://github.com/Panfactum/stack/commit/f2dd861ecf213bfef61cabebe1a75e6ea9843abf) - Reference (external-docs): [Authentik 2024.10 release notes](https://docs.goauthentik.io/releases/2024.10/) - Reference (external-docs): [`goauthentik/terraform-provider-authentik` v2024.10.2 release](https://github.com/goauthentik/terraform-provider-authentik/releases/tag/v2024.10.2) - Reference (external-docs): [Authentik official website](https://goauthentik.io/) - Upgrades all OpenTofu providers, including major bumps for AWS (5.x to 6.x) and Helm (2.x to 3.x). - Impacts: iac-provider `aws` — Upgraded from 5.80.0 to 6.38.0 (major); boolean-string literals in `aws_eks` launch template corrected for 6.x strict type requirements - Impacts: iac-provider `aws_global` — Upgraded from 5.80.0 to 6.38.0 (major) - Impacts: iac-provider `aws_secondary` — Upgraded from 5.80.0 to 6.38.0 (major) - Impacts: iac-provider `helm` — Upgraded from 2.12.1 to 3.1.1 (major); `set_string` removed, automatic state migration from SDKv2 to Plugin Framework - Impacts: iac-provider `vault` — Upgraded from 4.5.0 to 4.8.0; held at 4.x because 5.x drops support for Vault 1.14.x - Impacts: iac-provider `kubernetes` — Upgraded from 2.34.0 to 2.35.0; held at 2.x because 3.x renames every resource type with a `_v1` suffix - Impacts: iac-provider `tls` — Upgraded from 4.0.6 to 4.2.1 - Impacts: iac-provider `random` — Upgraded from 3.6.3 to 3.8.1 - Impacts: iac-provider `time` — Upgraded from 0.10.0 to 0.13.1 - Reference (internal-commit): [chore(infrastructure): upgrade OpenTofu provider versions across all modules](https://github.com/Panfactum/stack/commit/18d7371cab0ca2d4f0543e21c6484b6980120ce2) - Reference (internal-commit): [chore(terraform): bump AWS provider 5.x to 6.x across all environments](https://github.com/Panfactum/stack/commit/43989c48de224cccb1665bbf2eced22fd4166499) - Reference (external-docs): [AWS Provider v6.0.0 release notes](https://github.com/hashicorp/terraform-provider-aws/releases/tag/v6.0.0) - Reference (external-docs): [AWS Provider v6 upgrade guide](https://registry.terraform.io/providers/-/aws/latest/docs/guides/version-6-upgrade) - Reference (external-docs): [Helm Provider v3.0.0 upgrade guide](https://registry.terraform.io/providers/hashicorp/helm/latest/docs/guides/v3-upgrade-guide) - Reference (external-docs): [Helm Provider v3.0.0 release notes](https://github.com/hashicorp/terraform-provider-helm/releases/tag/v3.0.0) - The bastion host image is upgraded from `debian:bookworm` to `debian:trixie` for a newer OpenSSH version with up-to-date security patches. - Impacts: iac-module `kube_bastion` — Base container image upgraded from `debian:bookworm` to `debian:trixie`, providing a newer OpenSSH release with current security patches. The `Protocol 2` sshd directive is removed as it is no longer recognized by modern OpenSSH versions. - Reference (internal-commit): [feat(bastion): upgrade to debian trixie and harden sshd config](https://github.com/Panfactum/stack/commit/1f80a81aa5c1be1437e4ec6bb00761500a0f4c8e) - Reference (external-docs): [Debian Trixie (Debian 13) release information](https://www.debian.org/releases/trixie/) - Reference (internal-docs): [`kube_bastion` module documentation](https://panfactum.com/docs/main/reference/infrastructure-modules/kubernetes/kube_bastion) ## Improvements - `tf_bootstrap_resources` now includes the `backup:TagResource` IAM permission required by AWS Backup to tag recovery points. - Re-apply `tf_bootstrap_resources` to receive the updated IAM policy. - Impacts: iac-module `tf_bootstrap_resources` — Added `backup:TagResource` permission to the backup IAM policy - Reference (internal-commit): [Add `backup:TagResource` to backup-terraform role](https://github.com/Panfactum/stack/commit/bf3eebd3148ad48ef5d443d511195888027d62fb) - Reference (issue-report): [AWS notification about missing `backup:TagResource` permission](https://github.com/Panfactum/stack/issues/291) - Reference (external-docs): [AWS Backup IAM actions and resource types reference](https://docs.aws.amazon.com/service-authorization/latest/reference/list_awsbackup.html) - Stable releases are now open to all users with a new `stable..` format and six-month support window. - Reference (internal-commit): [Open stable releases to all users and update release format, cadence, and support policy](https://github.com/Panfactum/stack/commit/70491731293c2383488a1ca9bd68f93f6bcc8c27) - Reference (internal-docs): [Versioning and releases documentation](https://panfactum.com/docs/main/guides/versioning/releases) - Reference (internal-docs): [Contributing release process documentation](https://panfactum.com/docs/main/guides/contributing/releasing) - `pf cluster add` now fully automates end-to-end cluster installation with fine-grained resumable steps. - Impacts: cli `cluster add` — Automates Vault, certificate management, Linkerd, autoscaling, inbound networking, maintenance controllers, and CloudNativePG with fine-grained checkpointing for step-level resumability - Impacts: iac-module `kube_vault` — Deployed and initialized automatically including `vault operator init` and unsealing - Impacts: iac-module `vault_core_resources` — Deployed automatically after Vault initialization via port-forward proxy - Impacts: iac-module `kube_cert_manager` — Deployed automatically as part of the certificate management phase - Impacts: iac-module `kube_cert_issuers` — Deployed automatically with delegated zone configuration and alert email prompts - Impacts: iac-module `kube_linkerd` — Deployed automatically with post-deployment control plane health verification - Impacts: iac-module `kube_metrics_server` — Deployed automatically as the first autoscaling sub-step - Impacts: iac-module `kube_vpa` — Deployed automatically; triggers resource reconciliation for all prior deployments - Impacts: iac-module `kube_karpenter` — Deployed automatically to enable cluster autoscaling - Impacts: iac-module `kube_karpenter_node_pools` — Deployed automatically to configure Karpenter provisioning rules - Impacts: iac-module `kube_scheduler` — Deployed automatically as part of autoscaling setup - Impacts: iac-module `kube_keda` — Deployed automatically to enable event-driven autoscaling - Impacts: iac-module `kube_external_dns` — Deployed automatically as the first inbound networking sub-step - Impacts: iac-module `kube_aws_lb_controller` — Deployed automatically to manage AWS load balancer resources - Impacts: iac-module `kube_ingress_nginx` — Deployed automatically to handle cluster ingress traffic - Impacts: iac-module `kube_bastion` — Deployed automatically to enable secure SSH access to the cluster - Impacts: iac-module `kube_reloader` — Deployed automatically to restart workloads on config/secret changes - Impacts: iac-module `kube_node_image_cache` — Deployed automatically to pre-cache container images on nodes - Impacts: iac-module `kube_pvc_autoresizer` — Deployed automatically to resize persistent volumes as needed - Impacts: iac-module `kube_descheduler` — Deployed automatically to rebalance pod scheduling across nodes - Impacts: iac-module `kube_external_snapshotter` — Deployed automatically to enable volume snapshot support - Impacts: iac-module `kube_velero` — Deployed automatically to enable cluster backup and disaster recovery - Impacts: iac-module `kube_cloudnative_pg` — Deployed automatically as the final installation step - Reference (internal-commit): [CLI improvements (#330)](https://github.com/Panfactum/stack/commit/733c3c8cf4b9c33e17adc99f6527108c58713144) - Reference (internal-commit): [Vault deployment and initialization automation](https://github.com/Panfactum/stack/commit/6519c292ef148081c10a2aa089a295d63827af27) - Reference (internal-commit): [Vault core resources automation](https://github.com/Panfactum/stack/commit/403036045129cbd3a0fe203d9c0d8d4a6d8abdc5) - Reference (internal-commit): [Certificate management automation](https://github.com/Panfactum/stack/commit/ad33083d8f32e1a143531a845d845b5543dcc5a1) - Reference (internal-commit): [Certificate issuers automation](https://github.com/Panfactum/stack/commit/2ca29499f6d742ee64fe75d3671674e441291bb1) - Reference (internal-commit): [Linkerd deployment and health verification](https://github.com/Panfactum/stack/commit/b04b36dad98ca97a3d416543ff95e42811665c13) - Reference (internal-commit): [Autoscaling components automation](https://github.com/Panfactum/stack/commit/76adbbf33773a0e28308e5bda34960e567ea8a3b) - Reference (internal-commit): [Inbound networking automation](https://github.com/Panfactum/stack/commit/6bc9823b0a576fb0d56a63478ae39872682cf9fc) - Reference (internal-commit): [Maintenance controllers and CloudNativePG automation](https://github.com/Panfactum/stack/commit/91fb52517c3c24210dee2bd96ee8e89d28293f01) - Reference (internal-commit): [AWS profile handling and VPC/EKS setup refinements (#339)](https://github.com/Panfactum/stack/commit/d966e41f762d8e82e3b088a66bddd8cabb29ff93) - Reference (external-commit): [CLI improvements: modular setup and checkpointing (PR #330)](https://github.com/Panfactum/stack/pull/330) - Reference (external-commit): [CLI updates: AWS profile handling and setup refinements (PR #339)](https://github.com/Panfactum/stack/pull/339) - Refactors `pf devshell sync` into reusable Listr task builders and fixes `pf env add` EC2 Spot role import. - Impacts: cli `devshell sync` — Internal sync logic refactored into composable Listr task builders with AWS Identity Center sync added - Impacts: cli `env add` — Conditionally skips `AWSServiceRoleForEC2Spot` import when the role does not exist in the account - Reference (internal-commit): [Refactor devshell update to use Listr task builders and add AWS Identity Center sync](https://github.com/Panfactum/stack/commit/7b1f505992598d12b16864c9ec33865c97232d9b) - Reference (external-docs): [Listr2 — terminal task list library used for CLI progress rendering](https://github.com/listr2/listr2) - `pf config get` now returns `environment_dir`, `region_dir`, and `module_dir` derived fields. - Impacts: cli `config get` — Now returns `environment_dir`, `region_dir`, and `module_dir` derived fields - Reference (internal-commit): [Add `environment_dir`, `region_dir`, and `module_dir` derived fields to `pf config get`](https://github.com/Panfactum/stack/commit/7b1f505992598d12b16864c9ec33865c97232d9b) - `aws_account` now automatically requests CloudFront service quota increases for origin request, response header, and cache policies. - Impacts: iac-module `aws_account` — Adds `aws_servicequotas_service_quota` resources to raise CloudFront origin request policy, response header policy, and cache policy quotas to 100 - Reference (internal-commit): [Add CloudFront service quota increase resources to `aws_account`](https://github.com/Panfactum/stack/commit/7b1f505992598d12b16864c9ec33865c97232d9b) - Reference (internal-docs): [`aws_account` module reference documentation](https://panfactum.com/docs/main/reference/infrastructure-modules/direct/aws/aws_account) - Reference (external-docs): [Amazon CloudFront quotas](https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/cloudfront-limits.html) - `pf cluster add` now streams live `linkerd check` output in a rolling five-line window during the Linkerd control plane check. - Impacts: cli `cluster add` — The Linkerd control plane check step now streams live `linkerd check` output to the terminal in a rolling five-line window instead of showing no output until the check completes - Reference (internal-commit): [fix(cluster install): stream linkerd check output line by line](https://github.com/Panfactum/stack/commit/989fcf567fb8e3868435b678dd76609ee40b4fff) - `pf cluster add` now streams real-time `terragrunt run-all apply` output in a rolling five-line bar during autoscaling setup. - Impacts: cli `cluster add` — Displays real-time `terragrunt run-all apply` output in a rolling five-line bar during autoscaling setup - Reference (internal-commit): [feat: log lines out of run-all apply](https://github.com/Panfactum/stack/commit/96f0de01ef79e15ffb70c5473e0c45070190e1eb) - `pf env add` UX improvements: pre-filled contact info, alias explainer, finish message, and clearer setup instructions. - Impacts: cli `env add` — Contact info pre-filled from AWS account; alias prompt explains global uniqueness; management bootstrap deploys `aws_organization`; finish message shown on completion; step-by-step AWS setup instructions with interactive confirmations; `OrganizationsClient` and `STSClient` now explicitly set region to `us-east-1` - Reference (internal-commit): [fix: env add: misc cli improvements](https://github.com/Panfactum/stack/commit/e0bedc9006e289a7756917a08a9c34d0400ecae3) - Reference (internal-commit): [feat: adds finish message to 'pf env add'](https://github.com/Panfactum/stack/commit/3747d0982bbff1e6ac4d2684a417dbd9c1f16d1e) - Reference (internal-commit): [fix: better install instructions + missing region](https://github.com/Panfactum/stack/commit/21d70937a4329fabf4872039ce14b63bb039dc91) - Reference (external-docs): [AWS IAM access keys documentation](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html) - Reference (external-docs): [AWS Organizations documentation](https://docs.aws.amazon.com/organizations/latest/userguide/orgs_introduction.html) - `pf domain add` now shows the live AWS registration status alongside the poll attempt counter. - Impacts: cli `domain add` — Registration polling task title now includes the live AWS status string - Reference (internal-commit): [feat: add domain registration status](https://github.com/Panfactum/stack/commit/3200adaba5619879fd3da810d5e69a46e0fbec39) - The `pf` CLI now wraps long output lines at 100 characters instead of 128, improving readability on standard 120-column terminals. - Impacts: cli `welcome` — Output text now wraps at 100 characters instead of 128 - Reference (internal-commit): [feat: adds welcome screen](https://github.com/Panfactum/stack/commit/11c41aa1fc01f6067bc76d2822199bc39bbfa40d) - Panfactum terragrunt hooks now write `.pf.yaml` module status; `pf cluster add` uses it for accurate resumability. - Impacts: cli `cluster add` — Checkpointing uses `.pf.yaml` deploy status; cert management and cert issuers steps consolidated into a single `Certificates` step - Impacts: cli `iac update-module-status` — New CLI command invoked by terragrunt hooks to write `initStatus` and `deployStatus` to each module's `.pf.yaml` - Impacts: configuration `panfactum.hcl` — Adds before/after/error hooks for `init` and `apply` that call `pf iac update-module-status` to track deployment status - Reference (internal-commit): [feat: add Panfactum terragrunt hooks](https://github.com/Panfactum/stack/commit/a316a5dc7cc6eb0a4acc93f1a333d8e40a50036a) - Reference (internal-commit): [feat: better resumability (#351)](https://github.com/Panfactum/stack/commit/70421c38fa9eb6f3238b5f1e7dfa9101f8a84f7e) - Reference (internal-commit): [feat: enhances module status architecture](https://github.com/Panfactum/stack/commit/00029f4e98aec9c1119bba2ff2a7fcb2d228620b) - `pf env add` now supports resuming interrupted installations and fixes phone number formatting. - Impacts: cli `env add` — Detects partially-deployed environments, skips completed provisioning steps, restores previously-selected regions, offers to copy region config from another environment, and creates AWS accounts directly via the SDK with email-retry support; also fixes phone number dash formatting - Reference (internal-commit): [feat:cli: env add resumability](https://github.com/Panfactum/stack/commit/e463425f6558d3105371d21787b3a1884414109e) - Reference (internal-docs): [Bootstrapping guide for preparing AWS accounts](https://panfactum.com/docs/main/guides/bootstrapping/preparing-aws) - `pf cluster add` post-installation success message is streamlined and reformatted with blank lines between numbered steps for better readability. - Impacts: cli `cluster add` — Post-installation `k9s` verification steps reformatted with blank lines and reduced from seven to five - Reference (internal-commit): [fix:cli:cluster add: success message format](https://github.com/Panfactum/stack/commit/4f0f981677d36b1cb2269a9b4e81287fdbab5217) - Reference (internal-docs): [Kubernetes cluster bootstrapping guide](https://panfactum.com/docs/main/guides/bootstrapping/kubernetes-cluster) - `pf env add` now suggests an environment-specific IAM username instead of the hardcoded `pf-bootstrap-user` name. - Impacts: cli `env add` — Suggests `-superuser` as the IAM username and uses it in the access key creation URL - Reference (internal-commit): [Derive bootstrap IAM username from environment name](https://github.com/Panfactum/stack/commit/0886f756c32dabcfb9ee647b991a3cb4e247d225) - Reference (internal-docs): [Bootstrapping guide documenting the `-superuser` AWS profile convention](https://panfactum.com/docs/main/guides/bootstrapping/preparing-aws) - `panfactum.hcl` auto-injects `route53_zones` and `kube_domain` with safe defaults, and `kube_certificates` makes `vault_internal_url` optional. - Impacts: iac-module `kube_certificates` — `vault_internal_url` now defaults to `http://vault-active.vault.svc.cluster.local:8200`, making the input optional for standard Panfactum cluster layouts. - Impacts: configuration `panfactum.hcl` — Injects `route53_zones` and `kube_domain` as global default module inputs using `lookup()` with safe defaults (`{}` and `null` respectively). - Reference (internal-commit): [fix: kube_certificates migration guide](https://github.com/Panfactum/stack/commit/8dddf6bf63532b62b4cf55e7b598421a645b1643) - Reference (internal-commit): [fix: kube_domain default input](https://github.com/Panfactum/stack/commit/d14a548a5cd98b7e2bfee632bbe118459be04b6a) - Reference (internal-docs): [`kube_certificates` module reference](https://panfactum.com/docs/main/reference/infrastructure-modules/kubernetes/kube_certificates) - `kube_opensearch` adds opt-in Dashboards, re-enables S3 remote storage, switches to static IAM credentials, and sizes JVM heap dynamically. - Impacts: iac-module `kube_opensearch` — Adds opt-in OpenSearch Dashboards via `dashboard_enabled`, re-enables S3 segment and translog remote storage, switches S3 plugin authentication to static IAM credentials via `kube_aws_creds`, and sizes JVM heap dynamically to 50% of the container memory request. - Reference (internal-commit): [feat: adds opensearch dashboards, remote storage, and v3](https://github.com/Panfactum/stack/commit/55a2cd5eea66664a7994f5841abda062d982fe1e) - Reference (issue-report): [OpenSearch issue #15902 (segment replication remote store)](https://github.com/opensearch-project/OpenSearch/issues/15902) - Reference (issue-report): [OpenSearch issue #16523 (IRSA web identity token not usable with S3 plugin)](https://github.com/opensearch-project/OpenSearch/issues/16523) - Reference (internal-docs): [`kube_opensearch` module documentation](https://panfactum.com/docs/main/modules/kube_opensearch) - Reference (external-docs): [OpenSearch official documentation](https://docs.opensearch.org/) - `pf cluster add` and `pf sso add` now skip `terragrunt init` for already-initialized modules, reducing redundant work during installation. - Impacts: cli `cluster add` — Module deploy tasks skip `terragrunt init` when the module's `init_status` is already `success` or `running`, speeding up resumed installs - Impacts: cli `sso add` — Module deploy tasks skip `terragrunt init` when the module's `init_status` is already `success` or `running`, speeding up resumed installs - Reference (internal-commit): [feat: intelligently skip init in deployModuleTask when possible](https://github.com/Panfactum/stack/commit/084e5910a3bad7d214090c78cca30f8caf650036) - `pf cluster add` now reads the Vault root token from `region.secrets.yaml` via `getPanfactumConfig()` instead of directly SOPS-decrypting `kube_vault/secrets.yaml`. - Impacts: cli `cluster add` — Vault root token is now stored as `vault_token` in `region.secrets.yaml` and read through the merged `getPanfactumConfig()` result instead of per-step `sopsDecrypt` calls. - Impacts: cli `config get` — `vault_token` is now a recognized field in the Panfactum config schema and appears in `pf config get` output. - Reference (internal-commit): [moved `vault_token` to `region.secrets.yaml`](https://github.com/Panfactum/stack/commit/829ad02cd9296a82c8918338fdefbdafd5ca2fae) - Reference (internal-docs): [Bootstrapping guide for Vault setup](https://panfactum.com/docs/main/guides/bootstrapping/vault) - `pf cluster add` now sources the Vault token automatically via `panfactum.hcl` instead of manually threading it through each setup step. - Impacts: cli `cluster add` — The `config` field and `refreshConfig` helper have been removed from `InstallClusterStepOptions`. The Vault root token is no longer injected as a `VAULT_TOKEN` environment variable; Terragrunt reads it automatically from `region.secrets.yaml` via `panfactum.hcl`. - Impacts: configuration `panfactum.hcl` — Now reads `vault_token` from `local.vars` (sourced from `region.secrets.yaml`) and supplies it to the Vault provider, eliminating the need for explicit `VAULT_TOKEN` environment variable injection. - Reference (internal-commit): [Moved `vault_token` to autoload in `panfactum.hcl`](https://github.com/Panfactum/stack/commit/ed1a146dd4c4d3d92866ac369bb653ce9d9f0d19) - Reference (internal-docs): [Terragrunt variables reference documenting `panfactum.hcl` configuration](https://panfactum.com/docs/main/reference/configuration/terragrunt-variables) - `pf sso add` now fully automates Vault SSO setup with resumable steps and root token revocation. - Impacts: cli `sso add` — Deploys `authentik_vault_sso` and `vault_auth_oidc` automatically; `vault_name` prefixed correctly; `akadmin_email` fixed; Identity Center auto-synced; root token revoked and removed; EKS federated auth step checkpointed; Authentik AWS SSO step resumable via `.pf.yaml` - Impacts: iac-module `authentik_vault_sso` — Now deployed automatically by `pf sso add` with correct `vault_name` derived from `kube_config_context` - Impacts: iac-module `vault_auth_oidc` — Now deployed automatically by `pf sso add` with OIDC outputs wired from `authentik_vault_sso` - Reference (internal-commit): [Vault SSO (#361)](https://github.com/Panfactum/stack/commit/cf2ef5abf90ba6c01e78276cdae63fd22bd54288) - Reference (internal-commit): [fix: various SSO fixes](https://github.com/Panfactum/stack/commit/f66cfb1a19fb8e5898b85a7c1a0993b265c3b2e4) - Reference (internal-commit): [feat: better resumability & revoke vault root token](https://github.com/Panfactum/stack/commit/4f3e553b1425807c7dd84a7bc65d6b5e2c8d1e94) - Reference (internal-commit): [chore: disable vault credential removal task](https://github.com/Panfactum/stack/commit/c365437e5673edcb6c052cf78e94003c90a90407) - Reference (internal-commit): [added skip and vault name](https://github.com/Panfactum/stack/commit/97cc08f087b13cfd822827ae645929ccc7fab738) - Reference (internal-docs): [`authentik_vault_sso` module reference](https://panfactum.com/docs/main/reference/infrastructure-modules/direct/authentik/authentik_vault_sso) - Reference (internal-docs): [`vault_auth_oidc` module reference](https://panfactum.com/docs/main/reference/infrastructure-modules/direct/vault/vault_auth_oidc) - `pf sso add` environment selection prompt uses clearer wording and an explainer describing access levels. - Impacts: cli `sso add` — Checkbox prompt for selecting security-hardened environments now includes an explainer and clearer label - Impacts: iac-provider `authentik` — Token lookup falls back to the `AUTHENTIK_TOKEN` environment variable when not set in config - Reference (internal-commit): [Improve federated auth input instructions and add authentik provider fallback](https://github.com/Panfactum/stack/commit/0e5d41b543b20f222ae7afa692e48d283216be37) - Reference (internal-docs): [Federated auth bootstrapping guide](https://panfactum.com/docs/main/guides/bootstrapping/federated-auth) - `kube_policies` now injects `NODE_OPTIONS=--max-old-space-size` into all containers and init containers proportional to the container's memory request. - Impacts: iac-module `kube_policies` — `NODE_OPTIONS=--max-old-space-size` injected into all containers and init containers proportional to memory request; init-container Kyverno rule corrected to iterate `spec.initContainers` instead of `spec.containers` - Reference (internal-commit): [feat: adds opensearch dashboards, remote storage, and v3](https://github.com/Panfactum/stack/commit/55a2cd5eea66664a7994f5841abda062d982fe1e) - Reference (internal-commit): [fix init container policies](https://github.com/Panfactum/stack/commit/c7178d3ab8a857594c8c79bce7a4411176ca2244) - Reference (internal-docs): [`kube_policies` module overview](https://panfactum.com/docs/main/modules/kube_policies/overview) - `pf cluster add` no longer deploys ECR pull-through cache; setup moved to `pf cluster enable`. - Impacts: cli `cluster add` — ECR pull-through cache deployment and credential collection removed from VPC setup step; `pull_through_cache_enabled` now defaults to `false` in the generated config - Impacts: cli `cluster enable` — New command scaffolded for opt-in ECR pull-through cache setup (stub only, not yet functional) - Reference (internal-commit): [removed ECR setup as part of VPC setup](https://github.com/Panfactum/stack/commit/acd5bfc8864fd1277504137996a536b8c72494eb) - Reference (internal-docs): [`aws_ecr_pull_through_cache` module overview](https://panfactum.com/docs/main/modules/aws_ecr_pull_through_cache/overview) - Reference (internal-docs): [Kubernetes cluster bootstrapping guide](https://panfactum.com/docs/main/guides/bootstrapping/kubernetes-cluster) - Reference (external-docs): [AWS ECR pull through cache documentation](https://docs.aws.amazon.com/AmazonECR/latest/userguide/pull-through-cache.html) - `pf cluster add` now interactively prompts for environment and region, removing the need to run from the correct directory. - Impacts: cli `cluster add` — Prompts for environment and region at startup; no longer requires running from the region directory - Reference (internal-commit): [Refactor `pf cluster add` to use interactive environment and region selection](https://github.com/Panfactum/stack/commit/4825a917123a4894df14c2fb41548c9d9227a79c) - Reference (internal-docs): [Kubernetes cluster bootstrapping guide](https://panfactum.com/docs/main/guides/bootstrapping/kubernetes-cluster) - Improves cluster-deployed detection accuracy and filters undeployed environments from `pf cluster add` and `pf sso add` prompts. - Impacts: cli `cluster add` — Cluster-deployed detection now uses `kube_reloader` `.pf.yaml` status; selection prompts filter to deployed environments only - Impacts: cli `sso add` — Environment and region selection now filters to deployed environments only - Reference (internal-commit): [using kubeApiServer to mark clusterDeployed](https://github.com/Panfactum/stack/commit/b4439e1bf5c8e329a58c8889fa8c29c58f4f25e7) - Reference (internal-commit): [refactor: improve cluster deployment detection to use actual deployment status](https://github.com/Panfactum/stack/commit/d34274c7b1d70d75c666734a4ca6cfb5cb9e7f0e) - `pf env add` improves existing-org IAM setup, adds post-bootstrap billing guidance, and pre-fills smart environment name defaults. - Impacts: cli `env add` — Existing-org flow now guides IAM user creation; post-bootstrap message includes optional billing console instructions; environment name prompt pre-fills with smart contextual default - Impacts: cli `logging` — Default answer values now displayed in italic gray for visual distinction - Reference (internal-commit): [feat: provide instructions for creating a user if importing existing AWS org](https://github.com/Panfactum/stack/commit/d4da65931ff9bd8e12707f179430d5d107c31215) - Reference (internal-commit): [feat: adds instructions for enabling billing when organization setup is complete](https://github.com/Panfactum/stack/commit/d09ed80f4a3c9b5d7ba8a9fc46912f00f0bf8f39) - Reference (internal-commit): [fix: default answer styles and adds 'pf env add' default env name](https://github.com/Panfactum/stack/commit/c30b909eac165687992550d41c9987cbdfc95207) - Reference (external-docs): [AWS IAM User Guide — Grant access to the billing console](https://docs.aws.amazon.com/IAM/latest/UserGuide/getting-started-account-iam.html) - `pf cluster add` now configures the bastion domain dynamically as `bastion.` instead of leaving `bastion_domains` empty. - Impacts: cli `cluster add` — Bastion domain is now automatically set to `bastion.` during cluster setup - Impacts: iac-module `kube_bastion` — Receives `bastion_domains` input populated from the cluster's `kube_domain` config value - Reference (internal-commit): [feat: configure bastion domain dynamically from kube_domain](https://github.com/Panfactum/stack/commit/cefc9900cb364e5056c1e5c73a4624e30d17f3bf) - Reference (internal-docs): [`kube_bastion` module reference](https://panfactum.com/docs/main/modules/kube_bastion) - `pf cluster add` EKS deployment step now displays a 15-minute ETA warning in the task title during the apply phase. - Impacts: cli `cluster add` — EKS deploy step now displays a time estimate warning in the task title while planning changes - Reference (internal-commit): [Add ETA warning message to EKS deployment task title](https://github.com/Panfactum/stack/commit/5e4d6524bb7f8d283e86f3775fdd4067f0b1d7ac) - `pf cluster enable` ECR setup now runs `terragrunt run-all apply` after enabling pull-through cache. - Impacts: cli `cluster enable` — ECR setup step replaces the single `kube_policies` re-deploy with a region-wide `terragrunt run-all apply` - Reference (internal-commit): [Replace targeted `kube_policies` re-deploy with full `terragrunt run-all apply` after enabling ECR pull-through cache](https://github.com/Panfactum/stack/commit/34ee420cad0fd8bc0c5fd72674bf734b02771ca7) - Reference (internal-docs): [`aws_ecr_pull_through_cache` module reference](https://panfactum.com/docs/main/modules/aws_ecr_pull_through_cache) - `pf sso add` now automatically opens the browser to the Authentik token settings page after creating the API token. - Impacts: cli `sso add` — Browser automatically opens to the Authentik token settings page after the API token is created, eliminating the manual navigation step - Reference (internal-commit): [feat: add automatic browser opening to Authentik token page](https://github.com/Panfactum/stack/commit/ce9783a125f31347aa835d2885816d0abae57807) - Reference (internal-docs): [Identity provider bootstrapping guide covering the `pf sso add` workflow](https://panfactum.com/docs/main/guides/bootstrapping/identity-provider) - `pf sso add` root email defaults to `authentik-root@sso.` and admin email prompt warns against reusing AWS root email. - Impacts: cli `sso add` — Root email prompt pre-filled with `authentik-root@sso.`; admin email explainer clarifies uniqueness requirement - Reference (internal-commit): [feat: improve Authentik setup with default root email and better instructions](https://github.com/Panfactum/stack/commit/f9903b9bdafe6dd44f8aeea43a75ca9b0a0a1419) - Reference (internal-docs): [Identity Provider bootstrapping guide](https://panfactum.com/docs/main/guides/bootstrapping/identity-provider) - Updates release scripts to use the structured YAML changelog format instead of the legacy MDX-based changelog. - Impacts: devshell `make-new-edge-release` — Now copies `changelog/main/log.yaml` to a versioned `edge//log.yaml` directory on release and resets `main/log.yaml` to a clean template - Impacts: devshell `make-new-stable-release-channel` — Now creates a `stable//log.yaml` entry directory instead of writing to the legacy `docs/changelog` MDX path - Reference (internal-commit): [feat(changelog): add structured YAML schema and validation tooling](https://github.com/Panfactum/stack/commit/8d4b44e31d14a8c5ce25d04a0c0b4b535d2c0afc) - `ds-generate-changelog-schemas` no longer generates `metadata.schema.json`; that responsibility has moved to `ds-validate-iac-metadata`. - Reference (internal-commit): [refactor(changelog): remove metadata schema gen; add contributor docs](https://github.com/Panfactum/stack/commit/d0166798440a5eea576ef2e986ea7c875e82b519) - The Nix image builder Argo workflow now skips redundant builds when the target image already exists in ECR. - Impacts: iac-module `kube_constants` — Internal image tag updated for the Nix image builder with idempotent ECR check - Reference (internal-commit): [feat(nix-image-builder): skip redundant builds if image already exists in ECR](https://github.com/Panfactum/stack/commit/884e7e5056f7b72b8d07b94d8a7a3ca16cbc9efc) - `pf wf sops-set-profile` now skips YAML parsing for non-SOPS files via a regex pre-check, reducing overhead in large directories. - Impacts: cli `wf sops-set-profile` — Faster execution in repositories with many non-encrypted YAML files due to a regex pre-check that skips full YAML parsing - Reference (internal-commit): [perf(sops-set-profile): skip YAML parsing for non-SOPS files early](https://github.com/Panfactum/stack/commit/81808b61f0e73ac032b9f27700f2facbc18a9061) ## Fixes - `aws_eks` access entries now grant correct permissions and RBAC policies include new CRDs. - Re-apply `aws_eks` to receive the corrected access entry permissions and updated RBAC policies. - Impacts: iac-module `aws_eks` — Fixed access entry permissions and updated RBAC policies for new CRDs - Reference (issue-report): [EKS access entries preventing proper RBAC permission grants](https://github.com/Panfactum/stack/issues/311) - Reference (external-docs): [AWS documentation on granting IAM users access via EKS access entries](https://docs.aws.amazon.com/eks/latest/userguide/access-entries.html) - Reference (internal-docs): [`aws_eks` module reference documentation](https://panfactum.com/docs/main/reference/infrastructure-modules/direct/aws/aws_eks) - AWS SSO sessions now sync across all profiles, eliminating redundant login prompts. - Run `pf devshell sync` to regenerate your AWS config files with the corrected SSO session handling. - Impacts: cli `devshell sync` — Generates AWS config with a shared `sso-session` block across all profiles - Reference (issue-report): [AWS SSO sessions not syncing across profiles](https://github.com/Panfactum/stack/issues/221) - Fixes the default value for `min_node_cpu` in `kube_karpenter_node_pools` so that Karpenter no longer excludes valid small instance types. - Re-apply `kube_karpenter_node_pools` to pick up the corrected default. - If you previously set `min_node_cpu` explicitly to work around this issue, you can remove the override. - Impacts: iac-module `kube_karpenter_node_pools` — Changed `min_node_cpu` default from `0.5` to `0` - Reference (internal-commit): [Fix `min_node_cpu` default for `kube_karpenter_node_pools`](https://github.com/Panfactum/stack/commit/40f95e1a37803879a090c1bc1abb73994173ed07) - Reference (internal-docs): [`kube_karpenter_node_pools` module documentation](https://panfactum.com/docs/main/reference/infrastructure-modules/direct/kubernetes/kube_karpenter_node_pools) - Reference (external-docs): [Karpenter NodePools concept documentation](https://karpenter.sh/docs/concepts/nodepools/) - Increased kubelet `registry-qps` and `registry-burst` settings in `kube_node_settings` to eliminate self-imposed rate-limiting that slowed node bootstrap image pulls. - Impacts: iac-module `kube_node_settings` — Added `registry-qps` (50) and `registry-burst` (100) to Bottlerocket user-data - Impacts: iac-module `kube_karpenter_node_pools` — Nodes provisioned by Karpenter inherit the faster image pull settings - Reference (internal-commit): [Bump registry QPS and burst settings in node user-data](https://github.com/Panfactum/stack/commit/27c0d4149e0d14d7f643b90194a5deee30b25e3e) - Reference (issue-report): [Bottlerocket issue requesting `registry-burst` and `registry-qps` kubelet arguments](https://github.com/bottlerocket-os/bottlerocket/issues/1495) - Reference (external-docs): [Bottlerocket Kubernetes settings documentation](https://bottlerocket.dev/en/os/1.54.x/api/settings/kubernetes/) - Fixes `wf_dockerfile_build` failing to authenticate when cloning private Git repositories. - Re-apply any modules that use `wf_dockerfile_build` with private repositories to pick up the corrected setup script. - Impacts: iac-module `wf_dockerfile_build` — Fixed setup script to inject `git_username` and `git_password` into the HTTPS clone URL so private repository workflows authenticate correctly - Reference (internal-commit): [Inject `git_username`/`git_password` into repo URL for private repo auth](https://github.com/Panfactum/stack/commit/36018714e101bc216537a8eb6dfc1854c60cfe01) - Reference (internal-commit): [Fix incorrect variable names (`USERNAME`/`PASSWORD` to `GIT_USERNAME`/`GIT_PASSWORD`)](https://github.com/Panfactum/stack/commit/8fe2e40a5f3174679470209208133a32698a2f66) - Reference (internal-docs): [`wf_dockerfile_build` module documentation](https://panfactum.com/docs/main/reference/infrastructure-modules/submodule/workflow/wf_dockerfile_build) - `kube_stateful_set` now exposes the `lifetime_evictions_enabled` input that was missing from its interface. - Impacts: iac-module `kube_stateful_set` — Added `lifetime_evictions_enabled` input, allowing control over Descheduler pod lifetime evictions - Reference (internal-docs): [`kube_stateful_set` module documentation](https://panfactum.com/docs/main/reference/infrastructure-modules/submodule/kubernetes/kube_stateful_set) - `aws_cdn` now creates AAAA DNS records so that IPv6 clients can resolve CloudFront distribution domains. - Re-apply any `aws_cdn` deployments with `terragrunt apply` to create the new AAAA records. - Impacts: iac-module `aws_cdn` — Added AAAA alias record for each domain to enable IPv6 DNS resolution - Reference (internal-commit): [Add AAAA Route 53 alias record for each CloudFront domain](https://github.com/Panfactum/stack/commit/5a39ffe25bc2abac97a76b802195f4935acd1126) - Reference (external-docs): [AWS documentation on enabling IPv6 for CloudFront distributions](https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/cloudfront-enable-ipv6.html) - Reference (external-docs): [Routing traffic to a CloudFront distribution using Route 53 alias records](https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/routing-to-cloudfront-distribution.html) - `aws_dnssec` KMS key policy was missing a required `resources` field in the `CreateGrant` statement, causing policy validation failures. - Re-apply `aws_dnssec` to update the KMS key policy with the corrected `CreateGrant` statement. - Impacts: iac-module `aws_dnssec` — Added missing `resources` field to the `CreateGrant` policy statement for DNSSEC KMS keys - Reference (internal-commit): [fix: dnssec kms key policy document](https://github.com/Panfactum/stack/commit/460bafdf28bfb9dc5f31c0de2b00d084309e0c15) - Reference (external-docs): [AWS documentation on customer managed keys for DNSSEC](https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/dns-configuring-dnssec-cmk-requirements.html) - Reference (internal-docs): [`aws_dnssec` module reference](https://panfactum.com/docs/main/reference/infrastructure-modules/submodule/aws/aws_dnssec) - Fixes `pf cluster add` where a missing return statement in `setSLA` caused the selected SLA target to be silently discarded. - Impacts: cli `cluster add` — Selected SLA target was silently discarded; cluster installed with default value instead - Reference (internal-commit): [Fix missing `return` in `setSLA` step](https://github.com/Panfactum/stack/commit/5bf5678581290b8c1b786f8756db00b664c62c7f) - Reference (internal-docs): [`sla_target` Terragrunt variable reference](https://panfactum.com/docs/main/reference/configuration/terragrunt-variables) - Fixes a compilation error in `pf devshell sync` caused by references to removed internal utility functions. - Impacts: cli `devshell sync` — Compilation error resolved; command runs correctly again - Reference (internal-commit): [Comment out removed utility references to fix compilation](https://github.com/Panfactum/stack/commit/738389f5a03f39078b5cfcce211254397cd83465) - Fixes `pf cluster add` VPC setup step where the AWS CLI could hang waiting for user input from a pager. - Impacts: cli `cluster add` — VPC name uniqueness check no longer hangs on the AWS CLI pager - Reference (internal-commit): [Add `--no-cli-pager` flag to VPC name validation and add debug logging](https://github.com/Panfactum/stack/commit/9b8ead4095a2fb15c4adc4f55fd21ba2e8de8d9d) - Reference (external-docs): [AWS CLI pagination options and `--no-cli-pager` flag](https://docs.aws.amazon.com/cli/latest/userguide/cli-usage-pagination.html) - Fixes multiple `pf` CLI bugs affecting checkpoint resumption, SOPS directory creation, spinner cleanup, and subprocess hangs. - Impacts: cli `cluster add` — VPC setup checkpoint now uses the correct ID (`setupVPCIaC`); vault secrets step auto-creates missing SOPS directories - Impacts: cli `aws vpc-network-test` — Progress spinners now close correctly on error; SSM command polling retries increased from 20 to 60 - Reference (internal-commit): [fix: cli updates](https://github.com/Panfactum/stack/commit/8d47487d857f75f57f6b1265beb51266e41c56f8) - Reference (internal-docs): [VPC networking bootstrapping guide](https://panfactum.com/docs/main/guides/bootstrapping/aws-networking) - Fixes `pf cluster add` SLA target prompt repeating on resume by persisting the selection via the checkpointer. - Impacts: cli `cluster add` — SLA target prompt shown only once per session; selection persisted via checkpointer - Reference (internal-commit): [fix: only ask for SLA confirmation once](https://github.com/Panfactum/stack/commit/eb2069598796850a9a9eb985614e69c6124d5525) - Fixes S3 snapshot connectivity in `kube_opensearch` by setting `s3.client.default.endpoint` to the regional endpoint, preventing 307 redirects and auth failures. - Impacts: iac-module `kube_opensearch` — Adds `s3.client.default.endpoint` to the OpenSearch configuration, pointing to the regional S3 endpoint so that snapshot operations authenticate and route correctly. - Reference (internal-commit): [fix: set `s3.client.default.endpoint` for `kube_opensearch` S3 repository plugin](https://github.com/Panfactum/stack/commit/997834fa45e43fdb304c4597dd15824660695c97) - Reference (issue-report): [OpenSearch S3 snapshot repository fails to select correct AWS region](https://github.com/opensearch-project/OpenSearch/issues/9265) - Reference (internal-docs): [`kube_opensearch` module reference](https://panfactum.com/docs/main/reference/infrastructure-modules/submodule/kubernetes/kube_opensearch) - Fixes multiple `pf cluster add` bugs — Vault secret paths, pod readiness scoping, EKS API server URL, and EC2 termination. - Impacts: cli `cluster add` — Corrects Vault `secrets.yaml` path for ingress and Linkerd steps, scopes Vault pod readiness check to the `vault` namespace with 60 retries, writes `kube_api_server` from module output, fixes EC2 instance termination to split on all whitespace, and kills background processes on step failure - Reference (internal-commit): [fix(cli): cluster install fixes and improvements](https://github.com/Panfactum/stack/commit/5de4d4dd0d508552d06d46cf763af56302d80721) - Fixes `pf cluster add` Vault domain lookup, hardcoded Terragrunt template values, and snapshotter/Velero deploy ordering. - Impacts: cli `cluster add` — Vault domain now read from `extra_inputs.vault_domain`; hardcoded `subnets`, `ingress_domains`, and `sla_level` removed from LB controller and nginx templates; `kube_external_snapshotter` and `kube_velero` deployed before `kube_descheduler` - Reference (internal-commit): [fix(cli): inbound networking and support services fixes](https://github.com/Panfactum/stack/commit/5440599fd7a771ef230de32ee6557731d7a14338) - Fixes `pf env add` account alias bugs — wrong source field, inverted task condition, and static credentials not written correctly. - Impacts: cli `env add` — Account alias now correctly uses the account name; the alias setup task runs when needed; static credential profiles are written and updated reliably - Reference (internal-commit): [fix(env): use accountName instead of accountId for alias input update](https://github.com/Panfactum/stack/commit/79b510609863bade8668dad6357fda3280eab11d) - Reference (internal-commit): [fix(env): correct enabled condition and expand error output for bootstrapEnvironment](https://github.com/Panfactum/stack/commit/322d777893c39e78d00eb8b40e3b872a50e21d7b) - Reference (internal-commit): [fix: aws static creds update cli fx](https://github.com/Panfactum/stack/commit/c70f947891cc2dc96c6f8c330b8c70fb985f0c7c) - Fixes `pf domain add` where the domain registration request was silently skipped because the submission task had `skip: true` hardcoded. - Impacts: cli `domain add` — Domain registration request is now actually submitted to AWS - Reference (internal-commit): [Remove `skip: true` from domain registration submission task](https://github.com/Panfactum/stack/commit/9d9d9b14061f6be41d14551a7d3278d45bd52f17) - Fixes the `zones` output of `aws_registered_domains` to expose `zone_id` per domain, unblocking `pf domain add` registration. - Impacts: iac-module `aws_registered_domains` — `zones` output now exposes `zone_id` per domain in the correct structure - Impacts: cli `domain add` — Can now correctly read zone metadata after `aws_registered_domains` deployment - Reference (internal-commit): [fix: aws_registered_domains output](https://github.com/Panfactum/stack/commit/4c7728c99d5ec35bbf3a663ac9681b43d5666cc1) - Reference (internal-docs): [`aws_registered_domains` module reference](https://panfactum.com/docs/main/reference/infrastructure-modules/direct/aws/aws_registered_domains) - Fixes `pf cluster add` `kubectl` commands that were missing the `--context` flag, causing them to operate against the wrong cluster. - Impacts: cli `cluster add` — All `kubectl` commands now include the `--context` flag to target the correct cluster - Reference (internal-commit): [fix: pass kube context to kubectl commands](https://github.com/Panfactum/stack/commit/3637fac4ac001ddacf2bd48cc9897ae5543cd173) - Fixes `kube_certificates` to issue TLS certificates for subdomains of the cluster domain by adding `kube_domain` to the certificate `dnsNames`. - Impacts: iac-module `kube_cert_issuers` — Adds the `kube_domain` input variable and appends `kube_domain` and `*.kube_domain` to the ingress TLS certificate `dnsNames`, enabling certificate issuance for cluster utility subdomains. - Impacts: iac-module `kube_certificates` — Inherits the subdomain certificate fix from `kube_cert_issuers`, which was consolidated into this module. - Reference (internal-commit): [fix: provider cert-issuers the ability to create certs for subdomains of the cluster domain](https://github.com/Panfactum/stack/commit/8aa22456510df9d9cc7a85730a810079e75ed424) - Reference (internal-docs): [`kube_cert_issuers` module documentation](https://panfactum.com/docs/main/reference/infrastructure-modules/kubernetes/kube_cert_issuers) - Reference (internal-docs): [`kube_certificates` module documentation](https://panfactum.com/docs/main/reference/infrastructure-modules/kubernetes/kube_certificates) - Fixes `panfactum.hcl` for new `pf` CLI interface and removes stale `--terragrunt-global-cache` from `pf config get`. - Impacts: configuration `panfactum.hcl` — Updated CLI invocations and removed `--terragrunt-global-cache` from `pf config get` - Impacts: cli `config get` — No longer called with `--terragrunt-global-cache`, preventing stale cached results during `run-all` sessions - Reference (internal-commit): [fix: panfactum.hcl for new cli](https://github.com/Panfactum/stack/commit/778dac2c2e3e7cbb9141666e0eac0afefedabea1) - Reference (internal-commit): [fix: misc cli installer bugfixes](https://github.com/Panfactum/stack/commit/c71b2b32a1f58286448afdf1419443e0f7c5f9f9) - Reference (internal-docs): [Terragrunt variables configuration reference](https://panfactum.com/docs/main/reference/configuration/terragrunt-variables) - Fixes six installer and configuration bugs affecting CLI commands, `panfactum.hcl`, and the `aws_organization` module. - Impacts: cli `env add` — Contact info now written with the correct `primary_contact` key and matching field names; optional prompts for Organization Name and Street Address 2 accept empty input; IAM Identity Center step skipped when the `aws_iam_identity_center_permissions` module is not deployed - Impacts: cli `devshell sync` — HCL files now overwritten on sync; `.envrc` update subtask correctly registered and executed instead of being silently dropped - Impacts: configuration `panfactum.hcl` — `authentik_url` is now read with `lookup()` and a sentinel fallback so the file no longer fails when the variable is absent - Impacts: iac-module `aws_organization` — `aws_organizations_features` now depends on `aws_organizations_organization` to prevent a race condition during initial organization creation - Reference (internal-commit): [fix: misc cli installer fixes](https://github.com/Panfactum/stack/commit/7be6e52c65c27ddf9af94a0367a56b5d37b93035) - Fixes `pf env add` credential validation loop, misplaced `AdministratorAccess` error, and silent exit on unknown errors. - Impacts: cli `env add` — Credential validation loop restructured into nested loops; `AdministratorAccess` error shown in the correct branch; unknown errors re-prompt instead of exiting; root account credentials rejected with a clear error; suggested IAM username now defaults to `-superuser` - Reference (internal-commit): [fix: AdministratorAccess credential validation loop](https://github.com/Panfactum/stack/commit/1c7d2dd454ef5fa216abe3fabb12aa1bcdf76bfb) - Reference (internal-commit): [fix: env add default IAM username](https://github.com/Panfactum/stack/commit/0886f756c32dabcfb9ee647b991a3cb4e247d225) - Fixes `pf domain add` to correctly skip environment subzone setup when no eligible environments exist. - Impacts: cli `domain add` — Subzone setup now correctly exits early when no eligible environments are present - Reference (internal-commit): [fix: properly skip subzone setup if applicable](https://github.com/Panfactum/stack/commit/6f47d8e6ee2cd4f2d409ccaf189293f919cdb027) - Fixes `pf cluster add` `setSLA` step to persist the selected SLA target in the correct numeric format to `region.yaml`. - Impacts: cli `cluster add` — The `setSLA` step now persists the selected SLA target as a numeric value in `region.yaml`, preventing formatting mismatches consumed by downstream IaC modules. - Reference (internal-commit): [fix: set sla formatting for cluster install](https://github.com/Panfactum/stack/commit/d8976a0552a5a9f6e4d81b58269c96b219dcb392) - Reference (internal-docs): [`sla_target` Terragrunt variable reference](https://panfactum.com/docs/main/reference/configuration/terragrunt-variables) - Fixes `pf env add` to import an existing AWS organization instead of failing to create a duplicate. - Impacts: cli `env add` — Correctly imports an existing AWS organization instead of failing when one already exists - Impacts: iac-module `aws_organization` — Adds `shouldImport` logic to detect and import an existing AWS Organization resource - Reference (internal-commit): [fix: import existing org in env add command](https://github.com/Panfactum/stack/commit/0a0e3ca4e78569312fc7bb4763c120a92674bb5d) - Reference (internal-docs): [`aws_organization` module reference](https://panfactum.com/docs/main/reference/infrastructure-modules/direct/aws/aws_organization) - Fixes install step nesting in `pf cluster add` by removing redundant parent tasks that caused incorrect progress display. - Impacts: cli `cluster add` — Flattened task nesting in the install workflow to render steps at the correct depth - Reference (internal-commit): [fix: cluster install step nesting](https://github.com/Panfactum/stack/commit/7c80f0a2d2fe0346cb7a1943baa97984d09ba851) - Reference (internal-docs): [Kubernetes cluster bootstrapping guide](https://panfactum.com/docs/main/guides/bootstrapping/kubernetes-cluster) - Fixes image pull failures in `vault_core_resources` by upgrading the vault-secrets-operator Helm chart to `1.3.0`. - Impacts: iac-module `vault_core_resources` — vault-secrets-operator Helm chart upgraded from `0.8.1` to `1.3.0` to resolve `gcr.io/kubebuilder/kube-rbac-proxy` image pull failure - Reference (internal-commit): [fix(deps): bump vault-secrets-operator chart to 1.3.0](https://github.com/Panfactum/stack/commit/051d0b1213f4b8db3d2bf3e36b30a8e9c88f28ee) - Reference (external-docs): [Vault Secrets Operator v1.3.0 release notes](https://github.com/hashicorp/vault-secrets-operator/releases/tag/v1.3.0) - Reference (external-docs): [HashiCorp Vault Secrets Operator GitHub repository](https://github.com/hashicorp/vault-secrets-operator) - `aws_account` now skips service quota increases when the current value already meets the target, preventing spurious `apply` failures. - Impacts: iac-module `aws_account` — Quota increase resources for CloudFront and EC2 are now gated behind a count check using data source lookups, so requests are only submitted when the current value is below the target. - Reference (internal-commit): [fix(aws_account): skip quota increases already at target value](https://github.com/Panfactum/stack/commit/838357f4a1cca6a118c1f67b71f7920e54170c79) - Reference (issue-report): [Upstream bug: aws_servicequotas_service_quota submits increase even when quota is above requested amount](https://github.com/hashicorp/terraform-provider-aws/issues/27797) - Reference (internal-docs): [`aws_account` module reference](https://panfactum.com/docs/main/reference/infrastructure-modules/direct/aws/aws_account) - Fixes three module deployment failures caused by a stale AMI filter, an outdated default version, and a missing image-manifest flag. - Impacts: iac-module `aws_vpc` — NAT gateway AMI `creation-date` filter updated from `2024-01-25` to `2026-01-26` - Impacts: iac-module `kube_opensearch` — Default `opensearch_version` bumped from `3.0.0` to `3.2.0` - Impacts: iac-module `kube_nats` — Added `allowInsecureImages` flag to unblock pod scheduling with new Bitnami NATS image manifest format - Reference (internal-commit): [fix(production): restore production deployments after module errors](https://github.com/Panfactum/stack/commit/8bcb5e0be6c69d0be9a699df62e0f0461b8cccc6) - Reference (external-docs): [fck-nat — open-source AWS NAT instance AMI](https://github.com/AndrewGuenther/fck-nat) - Reference (internal-docs): [`aws_vpc` module documentation](https://panfactum.com/docs/main/reference/infrastructure-modules/direct/aws/aws_vpc) - Reference (internal-docs): [`kube_opensearch` module documentation](https://panfactum.com/docs/main/reference/infrastructure-modules/submodule/kubernetes/kube_opensearch) - Reference (internal-docs): [`kube_nats` module documentation](https://panfactum.com/docs/main/reference/infrastructure-modules/submodule/kubernetes/kube_nats) - Image pulls for several Bitnami-based modules are restored after the `docker.io/bitnami/*` registry was deprecated. - Impacts: iac-module `kube_nats` — Helm image override added to pull `bitnamilegacy/nats` instead of `bitnami/nats` - Impacts: iac-module `kube_monitoring` — Helm image override added to pull `bitnamilegacy/thanos` instead of `bitnami/thanos` - Impacts: iac-module `kube_redis_sentinel` — Helm image overrides added for `bitnamilegacy/redis`, `bitnamilegacy/redis-sentinel`, `bitnamilegacy/redis-exporter`, and `bitnamilegacy/kubectl` - Impacts: iac-module `kube_linkerd` — Hardcoded `kubectl` sidecar image updated from `bitnami/kubectl` to `bitnamilegacy/kubectl` - Reference (internal-commit): [fix(infrastructure): migrate bitnami images to bitnamilegacy registry](https://github.com/Panfactum/stack/commit/f3f6bd1ecffd9012a0af3fc0b6d58671a1314cd4) - Reference (internal-commit): [chore: upgrade kubernetes to 1.32 and moves module directory](https://github.com/Panfactum/stack/commit/e714cfc07835ddfaa414efb73cab508ad22b490f) - Reference (external-docs): [Bitnami deprecated their public docker.io/bitnami/* catalog — bitnamilegacy fallback info](https://github.com/bitnami/bitnami-docker-archive) - Reference (external-docs): [Broadcom advisory on preparing for the Bitnami deprecation](https://community.broadcom.com/tanzu/blogs/beltran-rueda-borrego/2025/08/18/how-to-prepare-for-the-bitnami-changes-coming-soon/) - Reference (internal-docs): [`kube_nats` module reference](https://panfactum.com/docs/main/reference/infrastructure-modules/submodule/kubernetes/kube_nats) - Reference (internal-docs): [`kube_monitoring` module reference](https://panfactum.com/docs/main/reference/infrastructure-modules/direct/kubernetes/kube_monitoring) - Reference (internal-docs): [`kube_redis_sentinel` module reference](https://panfactum.com/docs/main/reference/infrastructure-modules/submodule/kubernetes/kube_redis_sentinel) - Reference (internal-docs): [`kube_linkerd` module reference](https://panfactum.com/docs/main/reference/infrastructure-modules/direct/kubernetes/kube_linkerd) - The Panfactum DevShell now unsets the `CI` environment variable instead of setting it to `"false"`. - Impacts: devshell `enter-shell-local` — `CI` is now unset on shell entry instead of being set to `"false"` - Reference (internal-commit): [fix(devshell): unset CI instead of setting it to "false"](https://github.com/Panfactum/stack/commit/9bdb249fe7b0b7d439d1601737fe8336f785fa58) - Reference (internal-docs): [Customizing the Panfactum development shell](https://panfactum.com/docs/main/guides/development-shell/customizing) - Fixes `pf cluster add` EKS setup where the superuser IAM principal ARN was not configured during cluster deployment. - Impacts: cli `cluster add` — Captures the caller's IAM ARN during the "Verify access" step and passes it as the initial superuser principal when deploying the EKS cluster - Impacts: iac-module `aws_eks` — The `extra_superuser_principal_arns` input is now automatically populated with the deploying user's ARN during `pf cluster add` - Reference (internal-commit): [fix: initial cluster permissions](https://github.com/Panfactum/stack/commit/d127b6f43fc5e715bed6b714ae08e112cbf4a1ca) - Reference (internal-docs): [`aws_eks` module reference — EKS cluster configuration](https://panfactum.com/docs/main/modules/aws_eks) - Fixes `pf cluster add` Vault unseal step where `kubectl exec` failed to pass arguments correctly. - Impacts: cli `cluster add` — Added missing `--` separator to `kubectl exec` in the Vault unseal step so arguments are passed correctly - Reference (internal-commit): [fix: kube exec in vault setup](https://github.com/Panfactum/stack/commit/1b4bb5bf5d6b1c3f644f1bb955f1dbc1014b37d4) - Reference (internal-docs): [Vault bootstrapping guide](https://panfactum.com/docs/main/guides/bootstrapping/vault) - Fixes broken alert email prompt in the `pf cluster add` cert issuers step caused by a missing `task` parameter. - Impacts: cli `cluster add` — Alert email prompt in the cert issuers step now receives the required `task` parameter and renders correctly - Reference (internal-commit): [Add missing `task` parameter to cert issuer email prompt](https://github.com/Panfactum/stack/commit/1983c232d7dd2507df93fc28b8743d1b2afb401d) - Reference (internal-docs): [Certificate management bootstrapping guide](https://panfactum.com/docs/main/guides/bootstrapping/certificate-management) - Fixes `pf-domain-add` contact prompts with incorrect `required` settings that rejected valid input. - Impacts: cli `domain add` — Corrected `required` flags on contact information prompts during domain registration - Reference (internal-commit): [Fix contact info prompt `required` flags in `pf domain add`](https://github.com/Panfactum/stack/commit/1983c232d7dd2507df93fc28b8743d1b2afb401d) - Fixes `pf domain add` to display nameservers one per line during manual DNS zone setup, improving copy-paste usability. - Impacts: cli `domain add` — Nameservers are now listed one per line during manual DNS zone setup - Reference (internal-commit): [fix: nameserver listing](https://github.com/Panfactum/stack/commit/0d65d646b488645ac3f8ad05d0852c5eedbcd3b1) - Reference (internal-commit): [fix: nameservers output in cli](https://github.com/Panfactum/stack/commit/23a2d48fabd479261dbbf07c87101cf0322dc3c8) - `pf aws vpc-network-test` inbound blocking check now uses TCP instead of ICMP ping for reliable verification. - Impacts: cli `aws vpc-network-test` — Inbound blocking check now uses a TCP socket connection on port 80 instead of ICMP ping - Reference (internal-commit): [Replace ICMP ping with TCP connection check for inbound blocking test](https://github.com/Panfactum/stack/commit/9bb03e5270ef399bf42d603795e2c8698394e588) - Reference (internal-docs): [AWS networking bootstrapping guide covering `pf aws vpc-network-test` usage](https://panfactum.com/docs/main/guides/bootstrapping/aws-networking) - Fixes multiple `pf cluster add` Inbound Networking bugs including phase ordering, missing Vault env vars, SLA bootstrapping, and deployment concurrency. - Impacts: cli `cluster add` — Inbound Networking phase now runs before Autoscaling, modules deploy sequentially instead of concurrently, and all Vault environment variables are correctly passed to each module deployment. - Impacts: iac-module `kube_ingress_nginx` — Bootstrapped with `sla_level: 1` during cluster install and post-deploy input update Zod schema corrected from `z.undefined()` to `z.number().optional()`. - Impacts: iac-module `kube_external_dns` — Now receives `VAULT_ADDR` environment variable during `pf cluster add` deployment. - Impacts: iac-module `kube_aws_lb_controller` — No longer deployed concurrently with `kube_external_dns` to avoid race conditions. - Reference (internal-commit): [fix: move inbound networking before autoscaling in cluster auto install](https://github.com/Panfactum/stack/commit/92da33ac2a718d02d99cd0471e4c653d2426f9ef) - Reference (internal-commit): [fix: nginx post deploy input update](https://github.com/Panfactum/stack/commit/63497c19c9b6cc1faf01c137f7be019fa77ee583) - Reference (internal-commit): [fix: remove concurrency](https://github.com/Panfactum/stack/commit/4f42f9ba484e2b9275588c3522858327b1b17c33) - `pf cluster add` now passes `VAULT_TOKEN` to autoscaling and inbound networking deployments, fixing Vault authentication failures. - Impacts: cli `cluster add` — Vault root token now passed as `VAULT_TOKEN` to Metrics Server, VPA, Karpenter, and `vault_core_resources` deployments during autoscaling and inbound networking setup - Reference (internal-commit): [fix: pass vault token](https://github.com/Panfactum/stack/commit/b949ef07943277aebe73c457d477c2235480251d) - Improves `pf cluster add` Vault ingress verification with real-time polling status, a DNS propagation warning, and a longer timeout. - Impacts: cli `cluster add` — Vault ingress verification now shows real-time polling status, displays a DNS propagation warning, and allows up to 30 minutes for the health check to succeed. - Reference (internal-commit): [fix: better ux for vault ingress](https://github.com/Panfactum/stack/commit/35dfbc34d67e5844d6bfb627e35f94e05e14452b) - Reference (internal-docs): [Bootstrapping guide for inbound networking and Vault ingress setup](https://panfactum.com/docs/main/guides/bootstrapping/inbound-networking) - Fixes a bug in the `pf` CLI where Terragrunt resource imports failed during module deployment when `resourceId` resolved to `undefined`. - Impacts: cli `cluster add` — Import tasks in `deployModuleTask` now skip gracefully when `resourceId` resolves to `undefined`. - Impacts: cli `env add` — Import tasks in `deployModuleTask` now skip gracefully when `resourceId` resolves to `undefined`. - Impacts: cli `domain add` — Import tasks in `deployModuleTask` now skip gracefully when `resourceId` resolves to `undefined`. - Reference (internal-commit): [fix: undefined import](https://github.com/Panfactum/stack/commit/0d1327c722d578fbe23280fc0ff98efaa0f0c10e) - Fixes and hardens the Nix version check in the DevShell `.envrc` template. - Impacts: cli `devshell sync` — `.envrc` template updated with a POSIX-compatible Nix version check and a missing-Nix installation prompt - Impacts: installer `install.sh` — DevShell build time estimate updated from 30 to 15 minutes - Reference (internal-commit): [fix: nix version check in envrc](https://github.com/Panfactum/stack/commit/ac01f734d88d02ebfbf26aeaefaf434f8be25df4) - Reference (internal-commit): [fix: refactor nix version check in .envrc to match install.sh](https://github.com/Panfactum/stack/commit/3e06e130f943907d5409980df2333fbdc813a5bf) - Reference (internal-docs): [Guide for installing and booting the Panfactum DevShell](https://panfactum.com/docs/main/guides/bootstrapping/installing-devshell) - Fixes `pf env add` credential loading failure for the management account profile during environment provisioning. - Impacts: cli `env add` — `OrganizationsClient` and `STSClient` now read credentials directly from disk instead of relying on the SDK provider cache. - Reference (internal-commit): [Fix AWS SDK credentials bug in `pf env add`](https://github.com/Panfactum/stack/commit/db4b7e515ee1a16e6fdbd392a2adcbe00024d669) - Reference (issue-report): [AWS SDK credential provider cache does not pick up newly written credentials](https://github.com/aws/aws-sdk-js-v3/issues/5829) - `pf domain add` now uses a WHOIS fallback to fix false-negative domain registration detection for newly purchased domains. - Impacts: cli `domain add` — Domain registration detection now uses a two-step check: first querying DNS nameservers via `dig`, then falling back to a WHOIS database lookup when no nameservers are found. - Impacts: devshell `whois` — Added `whois` to the DevShell to support the improved domain registration check. - Reference (internal-commit): [Improve domain `isRegistered` check with WHOIS fallback](https://github.com/Panfactum/stack/commit/4156dcf072d3e93735c7f16361fcbcbe61c8b7cd) - `pf env add` phone number prompt now includes a format example, clarifying that a country-code prefix is required. - Impacts: cli `env add` — Phone number prompt now shows an example format to guide user input - Reference (internal-commit): [fix: phone number format in question](https://github.com/Panfactum/stack/commit/b186300f9298fe13d12b7f05abdeb090f4756dd1) - Fixes `pf` CLI `applyColors` incorrectly dedenting text after applying ANSI color codes, causing misaligned styled output. - Impacts: cli `logging` — Fixes operation order in `applyColors` so `dedent` runs on the raw string before color styling is applied, preventing ANSI escape sequences from interfering with indentation analysis. - Reference (internal-commit): [fix: dedent order in applyColors](https://github.com/Panfactum/stack/commit/b695023a1701ff34709eeee49c64d2c9aaf79050) - Fixes `pf domain add` to require a non-management environment and corrects stale `pf env install` references to `pf env add`. - Impacts: cli `domain add` — Now validates at least one non-management environment exists; error messages updated from `pf env install` to `pf env add` - Reference (internal-commit): [fix:domains: at least one env check + 'env install' -> 'env add'](https://github.com/Panfactum/stack/commit/a9bb99d813ead7fb63050b375087628e07eea487) - Fixes `pf-cluster-add` cert issuers step to restart cert-manager after nginx ingress deployment, preventing certificates from staying not-ready. - Impacts: cli `cluster add` — Cert issuers step now runs a concurrent retry loop that restarts the cert-manager deployment until the certificate reaches a ready state - Reference (internal-commit): [fix: reset cert manager on nginx ingress deploy (#350)](https://github.com/Panfactum/stack/commit/74894a454a64f7cd6285257ef3bebda2ba19e601) - DevShell boot script now unsets `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` to prevent host credentials from overriding profile-based configuration. - Impacts: devshell `enter-shell-local` — `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` are now unset on shell entry to prevent host credentials from leaking into the DevShell. - Reference (internal-commit): [fix: unload AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY env vars on devshell boot](https://github.com/Panfactum/stack/commit/4ed4265aa34302f474654217144017cca60903a8) - Reference (internal-docs): [Installing the Panfactum DevShell](https://panfactum.com/docs/main/guides/bootstrapping/installing-devshell) - Fixes Street Address 2 minimum length validation in `pf env add` and `pf domain add` contact prompts. - Impacts: cli `env add` — Street Address 2 minimum length corrected to 2 characters - Impacts: cli `domain add` — Street Address 2 minimum length corrected to 2 characters in domain registration flow - Reference (internal-commit): [fix: address line 2 min length](https://github.com/Panfactum/stack/commit/03663dd54d9623b344f2bd070617645856ff6feb) - Reference (internal-commit): [fix: address line 2 min length](https://github.com/Panfactum/stack/commit/1cfe02357781201fcc0ee25d825572790480709c) - Fixes `kube_ingress_nginx` to allow multiple nginx replicas to be scheduled on the same host when `sla_target` is set to 1. - Impacts: iac-module `kube_ingress_nginx` — `host_anti_affinity_required` is now set to `false` when `sla_target` is 1, allowing multiple replicas on the same node - Reference (internal-commit): [fix: allow multiple nginx instances on one host for sla 1](https://github.com/Panfactum/stack/commit/de4a60f36ef43cf341131d3fbd80765c66d95693) - Reference (external-docs): [Ingress NGINX Controller documentation](https://kubernetes.github.io/ingress-nginx/) - Reference (internal-docs): [`kube_ingress_nginx` module reference](https://panfactum.com/docs/main/reference/infrastructure-modules/direct/kubernetes/kube_ingress_nginx) - Fixes `pf cluster add` Vault key storage to split root token and recovery keys into separate SOPS-encrypted files. - Impacts: cli `cluster add` — Recovery keys written to `kube_vault/recovery.yaml` and root token to `kube_vault/secrets.yaml` immediately after `vault operator init`; read paths updated to match the new split layout - Reference (internal-commit): [feat: better resumability (#351)](https://github.com/Panfactum/stack/commit/70421c38fa9eb6f3238b5f1e7dfa9101f8a84f7e) - Reference (internal-commit): [fix: split vault recovery keys and root token](https://github.com/Panfactum/stack/commit/62031a6aa8fe0352dd3ed1abd3062d9ff134f6b0) - Reference (internal-commit): [fix: reading from recovery.yaml](https://github.com/Panfactum/stack/commit/3d7608ca62c6a10f0216f6bc4c9c2eb514fcf414) - Reference (internal-docs): [Vault bootstrapping guide](https://panfactum.com/guides/bootstrapping/vault) - Lowers Bun DNS cache TTL during `pf cluster add` Vault ingress verification to detect DNS propagation faster. - Impacts: cli `cluster add` — Vault ingress verification detects DNS propagation faster by using a 5-second DNS cache TTL - Reference (internal-commit): [chore: lower bun dns cache ttl](https://github.com/Panfactum/stack/commit/c62f63d9eefb4549882e254f15041b7d50c2a3d2) - Reference (external-docs): [Bun DNS cache TTL configuration](https://bun.com/docs/runtime/networking/dns) - Fixes `pf cluster add` autoscaling task ordering and `bootstrap_mode_enabled` schema path in the cluster extensions step. - Impacts: cli `cluster add` — Autoscaling step task hierarchy flattened; `bootstrap_mode_enabled` now read from correct `extra_inputs` schema path - Reference (internal-commit): [fix: unneeded nesting & validation schema](https://github.com/Panfactum/stack/commit/67f55056af01643193d3d24ddd70976932112642) - Reference (internal-docs): [Bootstrapping autoscaling guide](https://panfactum.com/docs/main/guides/bootstrapping/autoscaling) - Fixes `pf cluster add` Vault setup where `.pf.yaml` error status writes failed silently if the file already existed. - Impacts: cli `cluster add` — Vault init and unseal error status now correctly written to `.pf.yaml` on retry - Reference (internal-commit): [Merge certificates modules (#353)](https://github.com/Panfactum/stack/commit/48d67a2721e3df5e482db5e975bc76722451dd2a) - Reference (external-commit): [Merge certificates modules (PR #353)](https://github.com/Panfactum/stack/pull/353) - Fixes domain name interpolation bugs in `pf domain add` that displayed raw variable names instead of the actual domain. - Impacts: cli `domain add` — Domain name now correctly interpolated in NS record instructions and already-registered message - Reference (internal-commit): [fix:cli: domain interpolation issues](https://github.com/Panfactum/stack/commit/b0bfa5f7433a9abeaf680a361a8092fd257eea46) - Fixes a circular dependency and apply-order race condition in `kube_certificates` by using direct resource attribute references in `aws_permissions`. - Impacts: iac-module `kube_certificates` — `aws_permissions` module now reads `service_account` and `service_account_namespace` directly from the `kubernetes_service_account.cert_manager` resource instead of using input variables, creating an implicit dependency that eliminates the circular dependency and the need for an explicit `depends_on`. - Reference (internal-commit): [fix: permissions after service account is created](https://github.com/Panfactum/stack/commit/463dcb5a14dd40263c0ee4ae0ec090382ddcc19d) - Reference (internal-commit): [fix: correct the circular dependency](https://github.com/Panfactum/stack/commit/cb8ec840f552abf60423394febdaaba48312714c) - Reference (internal-docs): [`kube_certificates` module documentation](https://panfactum.com/docs/main/modules/kube_certificates) - Fixes AWS SDK credential-loading failures in `pf env add` and `pf domain add`, and corrects `pf welcome` installer display. - Impacts: cli `env add` — AWS SDK clients now use a centralized `createAWSClient` factory that reads credentials from disk on each call, bypassing the provider cache - Impacts: cli `domain add` — `Route53DomainsClient` and `AccountClient` now use the centralized credentials-from-file factory, resolving authentication failures during domain registration - Impacts: cli `welcome` — Welcome screen is suppressed when `.envrc` is absent, preventing onboarding content from displaying during initial `install.sh` execution - Reference (internal-commit): [fix:cli: aws client fixes + welcome screen + org create success section](https://github.com/Panfactum/stack/commit/572dddd507d8318b9ebd48dedc47f4351f73626d) - Reference (internal-commit): [fix: welcome screen on install](https://github.com/Panfactum/stack/commit/d5736a893be8a7c8245d73f075a8e4620d018a87) - Reference (issue-report): [AWS SDK JS v3 credential providers do not reload credentials from disk within the same process](https://github.com/aws/aws-sdk-js-v3/issues/6872) - Fixes `pf cluster add` EKS step to exclude SSO ARNs from `extra_superuser_principal_arns` and Vault pod readiness check to pass `--context`. - Impacts: cli `cluster add` — EKS setup excludes transient SSO assumed-role ARNs from `extra_superuser_principal_arns`; Vault pod readiness check now passes `--context` to `kubectl` - Reference (internal-commit): [Fix sso cluster add (#357)](https://github.com/Panfactum/stack/commit/ebbc5fa198626b320fcc6ef02d5f18a6d72932a5) - Reference (external-commit): [Pull request: Fix sso cluster add (#357)](https://github.com/Panfactum/stack/pull/357) - Fixes `pf domain add` to show context-appropriate DNS setup instructions for apex domains versus subdomains. - Impacts: cli `domain add` — Manual DNS zone setup step now shows registrar-specific nameserver instructions for apex domains and DNS-host NS record instructions for subdomains - Reference (internal-commit): [Adjust message for apex vs non-apex manual NS setup](https://github.com/Panfactum/stack/commit/2bc2ae52e6ab29ef1def3f01474e11242058e09f) - Reference (internal-docs): [DNS bootstrapping guide](https://panfactum.com/docs/main/guides/bootstrapping/dns) - Reference (internal-docs): [Subdomain delegation networking concept](https://panfactum.com/docs/main/concepts/networking/subdomain-delegation) - Fixes `pf devshell sync` incorrectly requiring `.pf.yaml` in the environments `.gitignore`. - Impacts: cli `devshell sync` — Removes `.pf.yaml` from the expected environments `.gitignore` entries - Reference (internal-commit): [feat: misc cli improvements](https://github.com/Panfactum/stack/commit/e19f5da99390eed58ae797f556578af89d98532f) - Fixes `pf env add` failing on resume because the state bucket name was missing from the task context. - Impacts: cli `env add` — State bucket and lock table names are now populated in context even when resuming an interrupted setup - Reference (internal-commit): [fix: bootstrap env resume](https://github.com/Panfactum/stack/commit/6a17572cde08a016348a43d09b3e9c1ba014aabc) - Fixes CLI commands failing to update existing SOPS-encrypted files when merging new secret values. - Impacts: cli `cluster add` — SOPS secret upserts during Vault and inbound networking setup no longer fail on existing files - Impacts: cli `cluster enable` — SOPS secret upserts during ECR setup no longer fail on existing files - Impacts: cli `sso add` — SOPS secret upserts during federated auth and Authentik setup no longer fail on existing files - Reference (internal-commit): [fix: set overwrite to true for `sopsUpsert`](https://github.com/Panfactum/stack/commit/f33f0f128c2b05ca31e10fecf2d0be3dd05d1ac2) - Fixes `pf cluster add` Vault unseal skip check to query live seal status instead of a stale configuration flag. - Impacts: cli `cluster add` — Vault unseal skip check now calls `vault status -format=json` directly instead of reading a stale `module.yaml` flag, and validates the response with a Zod schema - Reference (internal-commit): [fix: using vault status to check if unseal step should be skipped](https://github.com/Panfactum/stack/commit/a30fefc697ec9ecd01f72accdd2f83f6d3b7c413) - Reference (internal-commit): [lint: added schema validation on vault status](https://github.com/Panfactum/stack/commit/0c1e3abe3ebb11f8985a37394060017c83e812ad) - Reference (external-docs): [Vault seal-status API reference](https://developer.hashicorp.com/vault/api-docs/system/seal-status) - Reference (internal-docs): [Bootstrapping guide for Vault](https://panfactum.com/docs/main/guides/bootstrapping/vault) - Fixes `pf domain add` poll title showing `undefined` and `kube_vault` KMS unseal key race condition. - Impacts: cli `domain add` — Poll title no longer shows `undefined` before the first status response is received - Impacts: iac-module `kube_vault` — Vault Helm release now explicitly depends on `module.unseal_key` to prevent a deploy race condition. - Reference (internal-commit): [fix:cli: initial domain register poll title](https://github.com/Panfactum/stack/commit/ccecb3270b0384feeaeaf62cf47124942071b121) - Reference (internal-commit): [fix: kube_vault depends_on unseal_key to prevent deploy race condition](https://github.com/Panfactum/stack/commit/ccecb3270b0384feeaeaf62cf47124942071b121) - Reference (internal-docs): [`kube_vault` module documentation](https://panfactum.com/docs/main/reference/infrastructure-modules/kubernetes/kube_vault) - Fixes Vault token noise, `vault_token` crash, and Authentik provider fallback in `panfactum.hcl`. - Impacts: configuration `panfactum.hcl` — Silences `pf-get-vault-token` when a local token is already configured, wraps `vault_token` lookup in `try(...)` to avoid crashes, and falls back to the `AUTHENTIK_TOKEN` env var for the Authentik provider - Impacts: cli `sso add` — Removes explicit `VAULT_TOKEN` injection and reads `authentik_token` via `getPanfactumConfig()` - Impacts: iac-provider `authentik` — Falls back to the `AUTHENTIK_TOKEN` environment variable when `authentik_token` is absent from config - Reference (internal-commit): [Silence Vault token output when token is already configured](https://github.com/Panfactum/stack/commit/1ac42430c0dd1fc59d2f3b0aab851416fbe099dc) - Reference (internal-commit): [Read `authentik_token` from config instead of module output](https://github.com/Panfactum/stack/commit/27cdbcf16525e9989430f33ef211b5a3f9e17560) - Reference (internal-commit): [Add proper fallback for Authentik provider token](https://github.com/Panfactum/stack/commit/0e5d41b543b20f222ae7afa692e48d283216be37) - Reference (internal-docs): [Terragrunt variables configuration reference](https://panfactum.com/docs/main/reference/configuration/terragrunt-variables) - Fixes `pf sso add` skip logic bugs that caused SCIM, Vault SSO, and federated auth steps to re-run after completion. - Impacts: cli `sso add` — Corrects skip conditions for SCIM setup, user sync, Vault SSO OIDC output parsing, and federated auth detection so completed steps no longer re-run on subsequent invocations. - Reference (internal-commit): [fix: skip logic and input schema](https://github.com/Panfactum/stack/commit/1a8e4add783f1ba32bc55664b3ec154a22a51e7a) - Reference (internal-commit): [fix: skip logic for federated auth](https://github.com/Panfactum/stack/commit/c9bac8f242beff2b5d28239a364665d4a43a2773) - Reference (internal-docs): [Federated auth bootstrapping guide](https://panfactum.com/docs/main/guides/bootstrapping/federated-auth) - Fixes `pf sso add` Vault root token revocation failing silently inside `kubectl exec`. - Impacts: cli `sso add` — Vault root token revocation now wraps the command in `sh -c` so the `VAULT_TOKEN` environment variable is properly expanded inside the container. - Reference (internal-commit): [fix: vault token revocation](https://github.com/Panfactum/stack/commit/33160da3a8492a06323de484697f4e50a31210c6) - Reference (internal-commit): [providing token to destroy and using -self](https://github.com/Panfactum/stack/commit/1c45d231c3b1f8b8d1ddac4fba751a0490dcffe9) - Reference (external-docs): [Vault `token revoke` CLI reference](https://developer.hashicorp.com/vault/docs/commands/token/revoke) - Reference (internal-docs): [Bootstrapping federated auth guide covering Vault SSO setup](https://panfactum.com/docs/main/guides/bootstrapping/federated-auth) - Fixes primary region identification in the CLI to exclude the `global` region from being marked as primary. - Impacts: cli `cluster add` — No longer incorrectly identifies the `global` region as primary when adding a new cluster - Impacts: cli `cluster enable` — No longer incorrectly identifies the `global` region as primary when enabling cluster features - Impacts: cli `env add` — No longer incorrectly identifies the `global` region as primary when bootstrapping an environment - Impacts: cli `sso add` — No longer incorrectly identifies the `global` region as primary when configuring SSO - Reference (internal-commit): [fix:cli: primary region identification algo](https://github.com/Panfactum/stack/commit/56d4c8c464f92fb4aa821bcedc0cb20e6568453e) - Reference (internal-commit): [fix: attempt 2 at fix primary region](https://github.com/Panfactum/stack/commit/edf631ddf4a14bff92aaea31b2a6ca08a55a7833) - Reference (internal-docs): [Terragrunt variables reference documenting `tf_state_region`](https://panfactum.com/docs/main/reference/configuration/terragrunt-variables) - Fixes `pf env add` state bucket generation failing on new AWS accounts where S3 is not yet active. - Impacts: cli `env add` — Adds an S3 activation check with retry logic before state bucket name generation on new AWS accounts - Reference (internal-commit): [fix:cli: check if s3 service is active before generating state bucket name](https://github.com/Panfactum/stack/commit/389632ed207eb8bbb26e48dac85f4661b7e119c8) - Fixes `pf sso add` to display the Authentik password reset link directly in the confirmation prompt text. - Impacts: cli `sso add` — Password reset link is now embedded in the explainer text of the browser-open confirmation prompt - Reference (internal-commit): [Add Authentik recovery link to browser-open prompt output](https://github.com/Panfactum/stack/commit/a2141bfbfaebae9280252968bc89b084f40f5b3c) - Fixes `pf devshell sync` generating the `.aws` directory gitignore at the wrong path (`.aws/.aws` instead of `.aws/.gitignore`). - Impacts: cli `devshell sync` — The `.aws` directory `.gitignore` file is now written to the correct path - Reference (internal-commit): [fix: `.aws` gitignore generation](https://github.com/Panfactum/stack/commit/e5edff11ae9c8e7251764f58b1e4357ba8ee3dc6) - Fixes `pf-tunnel` and `pf-db-tunnel` failing at startup due to references to the removed `pf-check-ssh` script and undefined SSH path variables. - Impacts: devshell `pf-db-tunnel` — Removed obsolete `pf-check-ssh` pre-flight call that caused startup failure - Impacts: devshell `pf-tunnel` — Removed obsolete `pf-check-ssh` source call and defined missing `CONNECTION_INFO_FILE`, `CONFIG_FILE`, and `KNOWN_HOSTS_FILE` variables inline - Reference (internal-commit): [fix: remove obsolete `pf-check-ssh` call from `pf-db-tunnel`](https://github.com/Panfactum/stack/commit/da2c2c0b6d592fb74487da17f8f9df5db839e1a2) - Reference (internal-commit): [fix: remove obsolete `pf-check-ssh` source from `pf-tunnel`](https://github.com/Panfactum/stack/commit/9349ff71dee07eaf056b1d15111e8676838e89e7) - Reference (internal-commit): [fix: define missing SSH configuration variables in `pf-tunnel`](https://github.com/Panfactum/stack/commit/6f6211d9d7a9774022c215afe4d5092e69a7f65f) - Reference (internal-docs): [SSH configuration reference](https://panfactum.com/docs/main/reference/configuration/ssh) - `pf cluster add` now passes the correct `aws_region` code to all AWS API calls instead of the directory slug. - Impacts: cli `cluster add` — VPC lookup, EKS addon management, EC2 instance termination, and Service Quotas calls now receive `aws_region` instead of the logical `region` directory slug - Impacts: cli `sso add` — `setupVaultSSO` now accepts an explicit `regionPath` parameter instead of using `process.cwd()` - Reference (internal-commit): [fix: standardize AWS region parameter handling and improve SSO flexibility](https://github.com/Panfactum/stack/commit/67143b00110420fe0253a47e18988b072350f8a0) - Fixes `pf cluster add` EKS setup where the default cluster name was not pre-formatted, allowing invalid characters. - Impacts: cli `cluster add` — Default cluster name is now passed through `clusterNameFormatter` before display, ensuring it contains only lowercase letters, digits, and hyphens - Reference (internal-commit): [fix: improve cluster name handling in EKS setup](https://github.com/Panfactum/stack/commit/1aed9737fed298a33b7dbfae6cb4370a8f2b472d) - Reference (internal-docs): [`aws_eks` module reference](https://panfactum.com/docs/main/modules/aws_eks) - Fixes `pf` CLI usage telemetry to fire after command parsing so that command path and flags are captured. - Reference (internal-commit): [fix: move usage tracking from context to command execution](https://github.com/Panfactum/stack/commit/d7e596c735284934c4000156cdce6ede26dedf5f) - Fixes `kube_opensearch` dashboard superuser password containing special characters that caused authentication failures and security update job ignoring the `vpa_enabled` input. - Impacts: iac-module `kube_opensearch` — Dashboard superuser password restricted to alphanumeric characters via `special = false`; security update job now correctly inherits `vpa_enabled` from the parent module - Reference (internal-commit): [fix: restrict opensearch dashboard password to alphanumeric characters only](https://github.com/Panfactum/stack/commit/919e46249012e6623547cdbbb7da12b014314c8d) - Reference (internal-commit): [fix: pass `vpa_enabled` to `kube_opensearch` security update job](https://github.com/Panfactum/stack/commit/6001eed85f76a8aaf53597846df50d66992e53d7) - Reference (issue-report): [Special characters not well supported in OpenSearch admin password](https://github.com/opensearch-project/opensearch-k8s-operator/issues/955) - Reference (internal-docs): [`kube_opensearch` module reference](https://panfactum.com/docs/main/reference/infrastructure-modules/submodule/kubernetes/kube_opensearch) - Fixes `kube_policies` read-only ClusterRoles incorrectly granting full write access and restricts `pf:admins` from writing to security-sensitive Kubernetes resources. - Re-apply `kube_policies` via `terragrunt apply` to receive the corrected RBAC permissions. - Impacts: iac-module `kube_policies` — `read_verbs` corrected to `get`/`list`/`watch`; `pf:admins` restricted from writing RBAC resources, webhooks, CRDs, and Kyverno policies - Reference (internal-commit): [feat: upgrades kubernetes to 1.31](https://github.com/Panfactum/stack/commit/33d887c36fae56bc624511e72a8ac7f939db47f3) - Reference (internal-docs): [`kube_policies` module documentation](https://panfactum.com/docs/main/modules/kube_policies) - Fixes `pf domain add` path resolution and CLI config file loading to handle missing files gracefully. - Impacts: cli `domain add` — `getDomains` was using `environment_dir` (a short directory name) instead of the full absolute filesystem path, causing downstream file resolution to fail when resolving environment files. - Impacts: cli `config get` — `getConfigValuesFromFile` now passes `throwOnMissing: false` and `throwOnEmpty: false` to the YAML reader and SOPS decryptor, returning `null` instead of throwing when optional config files are absent. - Reference (internal-commit): [Fix `getDomains` environment path and `getConfigValuesFromFile` missing-file handling](https://github.com/Panfactum/stack/commit/b7b790dd661af18039021d3352666f3ba3fc41c9) - Fixes CLI subprocess output capture where merged stdout/stderr was silently dropped, causing missing error diagnostics. - Impacts: cli `logging` — Merged subprocess output stream now correctly captures both stdout and stderr instead of returning empty content. - Reference (internal-commit): [feat: adds unit tests to the CLI](https://github.com/Panfactum/stack/commit/dd36035b968faeb3c4f64406aea69e47353f968a) - `kube_aws_creds` no longer conditionally omits the `aws_iam_user_policy` resource when `iam_policy_json` is null, preventing potential Terraform plan errors. - Impacts: iac-module `kube_aws_creds` — `aws_iam_user_policy` is now always created using `coalesce(var.iam_policy_json, "{}")` instead of being conditionally omitted when `iam_policy_json` is null - Reference (internal-commit): [Fix `kube_aws_creds` to always create IAM user policy with coalesce fallback](https://github.com/Panfactum/stack/commit/14cb9e692705dfa5b13f113d9699abb6df58b97c) - Reference (internal-docs): [`kube_aws_creds` module documentation](https://panfactum.com/docs/main/modules/kube_aws_creds/overview) - Fixes `kube_pg_cluster` cross-account recovery — missing IAM permissions on remote bucket and `recoveryTarget` schema validation error. - Impacts: iac-module `kube_pg_cluster` — IAM policy now grants `s3:*` on the `pg_recovery_bucket` when set; `recoveryTarget` block uses `merge()` with all fields explicitly declared; `backendID` corrected to `backupID` to match the CNPG CRD field name. - Reference (internal-commit): [fix: kube pg cluster recover from remote backups (#377)](https://github.com/Panfactum/stack/commit/b1168e61fb424a8e66a542c5c232721195300cfc) - Reference (issue-report): [Cross-account PG cluster recovery — missing IAM permissions on remote S3 bucket](https://github.com/Panfactum/stack/issues/378) - Reference (external-docs): [CloudNativePG backup and recovery documentation](https://cloudnative-pg.io/documentation/1.25/recovery/) - Reference (internal-docs): [`kube_pg_cluster` module documentation](https://panfactum.com/docs/main/reference/infrastructure-modules/submodule/kubernetes/kube_pg_cluster) - Updates several modules to use Terragrunt's current `TG_*` env var prefix and `--all` flag, replacing deprecated `TERRAGRUNT_*` and `run-all` syntax. - Re-apply `wf_tf_deploy` to pick up the updated `deploy.sh` script. - Re-run `pf-tf-init` to regenerate provider lock files using the new Terragrunt syntax. - Impacts: devshell `enter-shell-local` — `TERRAGRUNT_DOWNLOAD`, `TERRAGRUNT_FETCH_DEPENDENCY_OUTPUT_FROM_STATE`, and `TERRAGRUNT_FORWARD_TF_STDOUT` renamed to `TG_DOWNLOAD_DIR`, `TG_DEPENDENCY_FETCH_OUTPUT_FROM_STATE`, and `TG_TF_FORWARD_STDOUT` respectively - Impacts: devshell `pf-tf-init` — `terragrunt run-all init` replaced with `terragrunt init --all --queue-exclude-external`; providers lock migrated to `terragrunt run --all -- providers lock` - Impacts: iac-module `wf_tf_deploy` — `deploy.sh` now uses `terragrunt apply --all --queue-exclude-external` instead of the deprecated `terragrunt run-all apply --terragrunt-ignore-external-dependencies` - Reference (internal-commit): [refactor(hooks): move formatters and linters to Stop hook for batch runs](https://github.com/Panfactum/stack/commit/f2dd861ecf213bfef61cabebe1a75e6ea9843abf) - Reference (external-docs): [Terragrunt CLI Redesign migration guide](https://terragrunt.gruntwork.io/docs/migrate/cli-redesign) - Reference (external-docs): [Terragrunt PR #4233: Deprecating run-all command](https://github.com/gruntwork-io/terragrunt/pull/4233) - Reference (external-docs): [Terragrunt PR #3964: Strict control for old deprecated env vars](https://github.com/gruntwork-io/terragrunt/pull/3964) - Reference (internal-docs): [`wf_tf_deploy` module documentation](https://panfactum.com/docs/main/modules/wf_tf_deploy/overview) - Panfactum Docker image now includes the CLI tools and `/usr/bin/env`, fixing broken script execution inside the container. - Impacts: configuration `flake.nix` — `panfactumPackages` call corrected to `panfactumPackages true` and `usrBinEnv` added to image contents so `/usr/bin/env` is available inside the container - Reference (internal-commit): [build(flake): update Docker image root env path linking](https://github.com/Panfactum/stack/commit/606d656bd5c1742eee0cb9ba9c4b940240328ddd) - Fixes ten `pf` CLI commands that incorrectly required DevShell initialization, blocking execution in CI/CD and automation contexts. - Impacts: cli `wf git-checkout` — No longer requires DevShell; runs correctly in CI/CD containers before a repository is checked out - Impacts: cli `wf sops-set-profile` — No longer requires DevShell; runs correctly in CI/CD containers for SOPS profile configuration - Impacts: cli `util get-commit-hash` — No longer requires DevShell; resolves git refs to commit SHAs in CI and pre-DevShell contexts - Impacts: cli `buildkit scale up` — No longer requires DevShell; scales up BuildKit instances from CI pipelines and automation scripts - Impacts: cli `iac update-module-status` — No longer requires DevShell; tracks module deployment status from CI pipelines - Impacts: cli `kube disable-disruptions` — No longer requires DevShell; disables voluntary pod disruptions from automation workflows - Impacts: cli `kube enable-disruptions` — No longer requires DevShell; enables maintenance-window disruptions from automation workflows - Impacts: cli `kube get-token` — No longer requires DevShell; `kubectl` credential plugins now work in CI environments - Impacts: cli `k8s velero snapshot-gc` — No longer requires DevShell; snapshot garbage collection runs in non-DevShell containers - Impacts: cli `util get-module-hash` — No longer requires DevShell; computes module hashes in CI change-detection scripts - Reference (internal-commit): [fix(cli): allow wf commands to run without a devshell](https://github.com/Panfactum/stack/commit/e45d7351a4736a7142100f63a8a23e19cc71d1c6) - Reference (internal-commit): [chore(cicd): set memory limits on image builders and fix devshell flag](https://github.com/Panfactum/stack/commit/c44313b973c668fc4c0a526ca9cdb62e9b84a320) - Reference (internal-commit): [fix(buildkit): allow scale-up command to run without devshell](https://github.com/Panfactum/stack/commit/02becf232c45cdeae884c24a05d59c6e009c4fc9) - Reference (internal-commit): [fix(cli): mark several commands as devshell-independent](https://github.com/Panfactum/stack/commit/7b44c3a891a3c733bc752f74b13955af8d74b2c8) - Fixes `pf` CLI error exit code and updates CI workflow scripts to use renamed CLI subcommands. - Impacts: devshell `pf` — Top-level error handler now sets `process.exitCode = 1` so CLI failures propagate a non-zero exit code - Impacts: iac-module `wf_dockerfile_build` — `scale-buildkit.sh` now calls `pf buildkit scale up` instead of the removed `pf buildkit resume` - Impacts: iac-module `wf_tf_deploy` — `deploy.sh` `sops-set-profile` call updated to positional argument syntax (removed `--directory` and `--profile` flags) - Reference (internal-commit): [fix: correct error handling and CI workflow gaps](https://github.com/Panfactum/stack/commit/ddd7657e87f776df75cb102ec19447a732fff553) - Reference (internal-docs): [`wf_dockerfile_build` module documentation](https://panfactum.com/docs/main/reference/infrastructure-modules/submodule/workflow/wf_dockerfile_build) - Reference (internal-docs): [`wf_tf_deploy` module documentation](https://panfactum.com/docs/main/reference/infrastructure-modules/submodule/workflow/wf_tf_deploy) - Fixes `wf_tf_deploy` Terragrunt flags and `pf wf sops-set-profile` non-KMS SOPS schema crash. - Re-apply the `wf_tf_deploy` module to deploy the updated `deploy.sh` script with the correct Terragrunt flag format. - Impacts: iac-module `wf_tf_deploy` — `deploy.sh` now uses short-form Terragrunt flags (`--download-dir`, `--non-interactive`, etc.) instead of deprecated `--terragrunt-*` equivalents - Impacts: cli `wf sops-set-profile` — Inner `sops` Zod schema now uses `.passthrough()` to tolerate unknown fields from non-KMS SOPS configurations - Reference (internal-commit): [fix(wf): migrate terragrunt flags to new CLI format](https://github.com/Panfactum/stack/commit/cfb4c739cf687095795a2e2a8a0439ef408858fc) - Reference (issue-report): [Removal of legacy Terragrunt flags (all flags that start with `terragrunt-`)](https://github.com/gruntwork-io/terragrunt/issues/4597) - Reference (internal-docs): [`wf_tf_deploy` module reference](https://panfactum.com/docs/main/reference/infrastructure-modules/submodule/workflow/wf_tf_deploy) - The DevShell `terragrunt` wrapper no longer runs the `GIT_PASSWORD` redaction pipeline in CI when `GIT_PASSWORD` is unset. - Impacts: devshell `terragrunt` — Skips the `sed`-based `GIT_PASSWORD` redaction when `GIT_PASSWORD` is unset in CI - Reference (internal-commit): [fix(terragrunt): skip secret redaction when GIT_PASSWORD is unset](https://github.com/Panfactum/stack/commit/9e8754ede8059150ab93bd2f9515e9df2dc61f12) ## Upgrade Instructions import MarkdownAlert from "@/components/markdown/MarkdownAlert.astro" ## Add `alias` to `aws_organization` The [`aws_organization`](https://panfactum.com/docs/edge/reference/infrastructure-modules/direct/aws/aws_organization) module now manages the IAM account alias for the management AWS account via a new required `alias` variable. Add an `alias` input to your `aws_organization` module configuration before the next `terragrunt apply`: ```hcl inputs = { alias = "my-org-management" # A human-readable string for the management account # ... other existing inputs } ``` ## Consolidate Contact Information Variables The contact information variables on [`aws_account`](https://panfactum.com/docs/edge/reference/infrastructure-modules/direct/aws/aws_account) and [`aws_registered_domains`](https://panfactum.com/docs/edge/reference/infrastructure-modules/direct/aws/aws_registered_domains) have been consolidated to single objects for each contact. The inputs to [`aws_dns_zones`](https://panfactum.com/docs/edge/reference/infrastructure-modules/direct/aws/aws_dns_zones) have been consolidated to a single `domains` object for better per-domain configuration. Replace the individual contact field variables (e.g., `contact_first_name`, `contact_last_name`, etc.) with the new consolidated contact objects in your `aws_account` and `aws_registered_domains` module configurations. Refer to the updated [module reference docs](https://panfactum.com/docs/edge/reference/infrastructure-modules/direct/aws/aws_account) for the exact object structure. ## Migrate `aws_dns_zones` Inputs to `domains` Object The inputs to [`aws_dns_zones`](https://panfactum.com/docs/edge/reference/infrastructure-modules/direct/aws/aws_dns_zones) have been consolidated from separate domain list variables into a single `domains` object that allows per-domain granular configuration. Migrate your `aws_dns_zones` inputs so that each key in the `domains` object is a domain name and the value contains per-domain configuration. See the [module reference](https://panfactum.com/docs/edge/reference/infrastructure-modules/direct/aws/aws_dns_zones) for the new input schema. ## Rename Dedicated Cluster DNS Zone Every cluster now has a dedicated DNS zone for hosting control-plane utilities. `kube_domain` is now a required configuration value and should be set in the `region.yaml` for every region that houses a Kubernetes cluster. The value must be a subdomain of a domain available to the environment. 1. Add a `kube_domain` field to `region.yaml` for every region that contains a Kubernetes cluster. The value must be a subdomain of a domain already available in the environment (e.g., `kube.example.com` if `example.com` is managed). 2. Deploy the corresponding DNS zone via `aws_dns_zones` before applying cluster modules. ## Remove Secondary Provider from `tf_bootstrap_resources` The `aws.secondary` provider alias and cross-region DynamoDB replica have been removed from [`tf_bootstrap_resources`](https://panfactum.com/docs/edge/reference/infrastructure-modules/direct/aws/tf_bootstrap_resources). 1. Remove the `aws.secondary` provider alias from your `tf_bootstrap_resources` Terragrunt configuration if present. 2. Re-apply `tf_bootstrap_resources` to remove the cross-region DynamoDB replica from your state lock table. ## Rename Backup Vault in `tf_bootstrap_resources` Complete the secondary provider removal step above before proceeding. Both changes affect `tf_bootstrap_resources` and should be applied together. The backup vault name in [`tf_bootstrap_resources`](https://panfactum.com/docs/edge/reference/infrastructure-modules/direct/aws/tf_bootstrap_resources) now has a unique suffix to prevent conflicts. **Before applying the updated module, manually delete the existing backup vault** named `terraform-`. Delete all recovery points first, then the vault. ## Migrate Authentik Token to `region.secrets.yaml` The `pf sso add` command previously stored the Authentik API token as `authentikUserToken` inside `authentik_core_resources/secrets.yaml`. The token has been relocated to `region.secrets.yaml` under the standardized key `authentik_token`. If you have already run `pf sso add` and have an existing token stored in `/authentik_core_resources/secrets.yaml` under the key `authentikUserToken`, migrate it to the region-level file: ```bash sops --set '["authentik_token"] "your-token-here"' /region.secrets.yaml ``` Ensure `authentik_token` is set in `region.secrets.yaml` (SOPS-encrypted) before re-applying any Authentik modules. ## Update Authentik URL to `sso.` Complete the Authentik token migration step above before re-applying `kube_authentik`. The Authentik subdomain has been standardized from `authentik.` to `sso.`. If you previously deployed Authentik at `authentik.`, you must: 1. Update the `authentik_url` value in `global.yaml` from `authentik.` to `sso.`. 2. Create a DNS record for `sso.` pointing to the Authentik ingress. 3. Re-apply `kube_authentik` so the module picks up the new domain. 4. After verifying the new domain works, remove the old `authentik.` DNS record. ## Configure Authentik Organization Name We now create the Authentik email template in `kube_authentik`. The organization name is now a required input. Use the new `organization_name` output from the `kube_authentik` module as an input to the `authentik_core_resources` module. 1. Add the `organization_name` variable to your `kube_authentik` module configuration. 2. Wire the `organization_name` output from `kube_authentik` into the `authentik_core_resources` module as an input via a `dependency` block: ```hcl dependency "authentik" { config_path = "../kube_authentik" } inputs = { organization_name = dependency.authentik.outputs.organization_name # ... other inputs } ``` ## Install KEDA We now include KEDA in our base Panfactum clusters and our modules assume that you have it installed. See the instructions [here](https://panfactum.com/docs/edge/guides/bootstrapping/autoscaling/#deploy-keda). ## Remove Node Image Cache Modules This step must be completed **before** applying the Kubernetes 1.33 upgrade below. The Kyverno-based image cache system causes resource exhaustion issues in production clusters. Destroying it first prevents instability during the upgrade. The `kube_node_image_cache` and `kube_node_image_cache_controller` modules have been removed entirely. 1. Destroy any active `kube_node_image_cache` module deployments. 2. Destroy any active `kube_node_image_cache_controller` module deployments. 3. Remove the module directories for `kube_node_image_cache` and `kube_node_image_cache_controller` from your Terragrunt configuration. 4. Remove the following input variables from any module configurations that set them: * `node_image_cached_enabled` — from `kube_airbyte`, `kube_alloy`, `kube_argo_event_bus`, `kube_authentik`, `kube_aws_ebs_csi`, `kube_cloudnative_pg`, `kube_gha_runners`, `kube_ingress_nginx`, `kube_linkerd`, `kube_monitoring`, `kube_nats`, `kube_opensearch`, `kube_pg_cluster`, `kube_redis_sentinel`, `kube_vault` * `node_image_cache_enabled` — from any module that sets it * `image_prepull_enabled` and `image_pin_enabled` — from container spec blocks in `kube_pod`, `kube_deployment`, `kube_daemon_set`, `kube_stateful_set`, `kube_cron_job`, and `kube_job` * `panfactum_node_image_cache_enabled` — from `kube_policies` ## Upgrade Kubernetes to 1.33 The default Kubernetes version has been upgraded to 1.33 (from 1.30, incrementing through 1.31 and 1.32 in this release). 1. Review the [Kubernetes 1.33 changelog](https://kubernetes.io/blog/2025/04/23/kubernetes-v1-33-release/) for any deprecated APIs or behavior changes that affect your workloads. 2. If you pin `kube_version` explicitly in `aws_eks`, update it to `1.33`. Otherwise, re-apply `aws_eks` to trigger the upgrade. No additional action is required for the bundled compatibility fixes (EBS CSI driver pinning, Descheduler config migration, and Ingress-nginx annotation risk level). ## Migrate `kube_cert_manager` and `kube_cert_issuers` to `kube_certificates` `kube_cert_manager` and `kube_cert_issuers` have been consolidated into a single `kube_certificates` module. 1. Create a `kube_certificates` directory as a sibling to the `kube_cert_manager` and `kube_cert_issuers` directories. 2. Create a `terragrunt.hcl` file in the `kube_certificates` directory with the following contents: ```hcl include "panfactum" { path = find_in_parent_folders("panfactum.hcl") expose = true } terraform { source = include.panfactum.locals.pf_stack_source } dependency "vault_core" { config_path = "../vault_core_resources" skip_outputs = true } inputs = { alert_email = "..." # Copy from kube_cert_issuers } ``` 3. From the **region** directory, run the following: ```bash (cd kube_cert_issuers; terragrunt state pull > state.json); (cd kube_cert_manager; terragrunt state pull > state.json); jq -s ' .[0] as $f1 | .[1] as $f2 | $f1 | .outputs = ($f1.outputs + $f2.outputs) | .resources = ($f1.resources + $f2.resources) ' kube_cert_issuers/state.json kube_cert_manager/state.json > kube_certificates/state.json; jq 'del( .resources[] | select( .type == "pf_kube_labels" and (has("module") | not) ) )' kube_certificates/state.json > tmp && mv tmp kube_certificates/state.json; rm kube_cert_issuers/state.json; rm kube_cert_manager/state.json; ``` 4. Update the version of the Panfactum framework you are using. 5. Navigate to the `kube_certificates` directory. 1. Run `terragrunt init`. 2. Run `terragrunt state push state.json && rm state.json`. 3. Run `terragrunt apply` and review the changes. There should be only a few resources that will be replaced. 6. Remove the `kube_cert_issuers` and `kube_cert_manager` directories. 7. Replace any references to `kube_cert_issuers` and `kube_cert_manager` with `kube_certificates` in your code. ## Review `pull_through_cache_enabled` Default `pull_through_cache_enabled` now defaults to `true` for `kube_nats`, `kube_pg_cluster`, `kube_redis_sentinel`, `kube_stateful_set`, and `kube_deployment`. If ECR pull-through caching is not configured in your environment, image pulls for these modules will fail. Choose one of the following options: * **Option A**: Deploy the [`aws_ecr_pull_through_cache`](https://panfactum.com/docs/edge/reference/infrastructure-modules/direct/aws/aws_ecr_pull_through_cache) module to enable pull-through caching before re-applying these modules. * **Option B**: Explicitly set `pull_through_cache_enabled = false` on each of `kube_nats`, `kube_pg_cluster`, `kube_redis_sentinel`, `kube_stateful_set`, and `kube_deployment` in your module configurations. ## Remove `update_type` from `kube_cron_job` The `update_type` variable has been removed from [`kube_cron_job`](https://panfactum.com/docs/edge/reference/infrastructure-modules/submodule/kubernetes/kube_cron_job). Remove any `update_type` input from your `kube_cron_job` module configurations to avoid a Terraform error. ## Review `burstable_nodes_enabled` Default `burstable_nodes_enabled` now defaults to `true`. If you explicitly set it to `false`, no action is needed. Otherwise, confirm your workloads are compatible with burstable (T-family) instances. ## Update `PF_SKIP_REPO_CHECK` Environment Variable The `PF_SKIP_REPO_CHECK` environment variable has been renamed to `PF_SKIP_CHECK_REPO_SETUP`. If you previously set `PF_SKIP_REPO_CHECK=1` to skip the repo setup check, update your environment, CI pipelines, and scripts to use `PF_SKIP_CHECK_REPO_SETUP=1` instead. ## Update `pf install-cluster` References The `pf install-cluster` CLI command has been renamed to `pf cluster add`. Replace all invocations of `pf install-cluster` with `pf cluster add` in your scripts, CI pipelines, and runbooks. ## Initialize the `pf` CLI This step must be completed **after** all migration steps above. This release adds the new `pf` CLI tool. To begin using it: 1. Complete all migration steps for the breaking changes above. 2. Run `pf devshell sync`. Ensure this completes successfully before proceeding. 3. Run `terragrunt apply` on **all** modules (or `terragrunt run-all apply`). ## Related Resources - [JSON Data](https://panfactum.com/docs/changelog/edge.26-04-05.json): Machine-readable data - [Channel Release List](https://panfactum.com/docs/changelog/edge.json): All releases in this channel