# Panfactum Changelog — Upcoming

> This entry contains unreleased changes targeting the next edge release.

> Introduces dedicated DNS zones for cluster management, consolidates certificate modules, adds KEDA for autoscaling, launches the new Panfactum CLI tool, and restructures several AWS modules for better configuration management.

## Highlights

- New `pf` CLI tool — run `pf devshell sync` then `terragrunt run-all apply` to initialize
- `kube_cert_manager` and `kube_cert_issuers` consolidated into new `kube_certificates` module — state migration required
- KEDA added to base cluster — install via [`kube_keda`](https://docs.panfactum.com/docs/edge/reference/infrastructure-modules/direct/kubernetes/kube_keda)
- Kubernetes default version updated to 1.31
- `burstable_nodes_enabled` now defaults to `true` for all modules

## Breaking Changes

- Every cluster now requires a dedicated DNS zone for hosting control-plane utilities. `kube_domain` is a new required configuration value in `region.yaml` for every region housing a Kubernetes cluster. It must be a subdomain of a domain available to the environment.
  - Add a `kube_domain` field to `region.yaml` for every region that contains a Kubernetes cluster. The value must be a subdomain of a domain already available in the environment (e.g., `kube.example.com` if `example.com` is managed).
  - Deploy the corresponding DNS zone via `aws_dns_zones` before applying cluster modules.
  - Impacts: configuration `region.yaml` — New required kube_domain field for cluster DNS zones
- We now create the Authentik email template in `kube_authentik`. The organization name is now a required input. Use the new `organization_name` output from `kube_authentik` as an input to `authentik_core_resources`.
  - Add the `organization_name` variable to your `kube_authentik` module configuration.
  - Wire the `organization_name` output from `kube_authentik` into the `authentik_core_resources` module as an input via a `dependency` block.
  - Impacts: iac-module `kube_authentik` — Organization name now required input; produces email template
  - Impacts: iac-module `authentik_core_resources` — Must now consume organization_name output from kube_authentik
- `kube_cert_manager` and `kube_cert_issuers` have been consolidated into a single `kube_certificates` module to address various race conditions on cluster installation. See upgrade instructions for the state migration steps.
  - Create a new `kube_certificates` module deployment.
  - Migrate the Terraform state from both `kube_cert_manager` and `kube_cert_issuers` into the new module.
  - Remove the old module directories after confirming the migration succeeds.
  - Follow the detailed state migration steps in the upgrade instructions.
  - Impacts: iac-module `kube_cert_manager` — Deprecated and consolidated into kube_certificates
  - Impacts: iac-module `kube_cert_issuers` — Deprecated and consolidated into kube_certificates
  - Impacts: iac-module `kube_certificates` — New module replacing kube_cert_manager and kube_cert_issuers
- KEDA is now included in the base Panfactum cluster and modules assume it is installed. See the installation instructions [here](https://docs.panfactum.com/docs/edge/guides/bootstrapping/autoscaling/#deploy-keda).
  - Deploy the `kube_keda` module in every cluster following the [installation guide](https://docs.panfactum.com/docs/edge/guides/bootstrapping/autoscaling/#deploy-keda). This must be done before applying other modules that now depend on KEDA being present.
  - Impacts: iac-module `kube_keda` — Now required as part of base cluster installation
- `burstable_nodes_enabled` now defaults to `true` for all modules. This fixes the broken default introduced in `edge.25-03-26`.
- Contact information variables on [`aws_account`](https://docs.panfactum.com/docs/edge/reference/infrastructure-modules/direct/aws/aws_account) and [`aws_registered_domains`](https://docs.panfactum.com/docs/edge/reference/infrastructure-modules/direct/aws/aws_registered_domains) have been consolidated to single objects for each contact.
  - Replace the individual contact field variables (e.g., `contact_first_name`, `contact_last_name`, etc.) with the new consolidated contact objects in your `aws_account` and `aws_registered_domains` module configurations.
  - Refer to the updated module reference docs for the exact object structure.
  - Impacts: iac-module `aws_account` — Contact variables consolidated into single objects per contact
  - Impacts: iac-module `aws_registered_domains` — Contact variables consolidated into single objects per contact
- Inputs to [`aws_dns_zones`](https://docs.panfactum.com/docs/edge/reference/infrastructure-modules/direct/aws/aws_dns_zones) have been consolidated to a single `domains` object for better per-domain granular configuration.
  - Migrate your `aws_dns_zones` inputs from separate domain list variables to the new `domains` object. Each key in the object is a domain name, and the value contains per-domain configuration.
  - See the [module reference](https://docs.panfactum.com/docs/edge/reference/infrastructure-modules/direct/aws/aws_dns_zones) for the new input schema.
  - Impacts: iac-module `aws_dns_zones` — Inputs consolidated into single domains object
- The backup vault name in [`tf_bootstrap_resources`](https://docs.panfactum.com/docs/edge/reference/infrastructure-modules/direct/aws/tf_bootstrap_resources) now has a unique suffix. **Manually delete the existing `terraform-<env_name>` backup vault before applying the updated module.**
  - In the AWS console (or via CLI), manually delete the existing AWS Backup vault named `terraform-<env_name>` in each environment.
  - Re-apply the `tf_bootstrap_resources` module, which will create the replacement vault with the new uniquely-suffixed name.
  - Impacts: iac-module `tf_bootstrap_resources` — Backup vault name now includes unique suffix; manual deletion required
- This release adds the new `pf` CLI tool. After completing all other migration steps, run `pf devshell sync` and then `terragrunt apply` on all modules.
  - Run `pf devshell sync` to initialize the new CLI tool's local configuration.
  - Run `terragrunt run-all apply` across all modules to propagate the changes introduced by the sync.
  - Impacts: cli `devshell sync` — New command to sync devshell configuration
  - Impacts: devshell `pf` — New CLI tool added to the devshell
- The default Kubernetes version has been updated to 1.31 from 1.30.
  - Review the [Kubernetes 1.31 changelog](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.31.md) for any deprecated APIs or behavior changes that affect your workloads.
  - If you pin `kube_version` explicitly, update it to `1.31`. Otherwise, re-apply `aws_eks` to trigger the upgrade.
  - Impacts: iac-module `aws_eks` — Default Kubernetes version updated to 1.31

## Additions

- Adds the [`kube_job`](https://docs.panfactum.com/docs/edge/reference/infrastructure-modules/submodule/kubernetes/kube_job) submodule for running one-off Jobs as part of module deployment processes (e.g., database migrations).
  - Impacts: iac-module `kube_job` — New submodule for running one-off Kubernetes Jobs
- Adds [KEDA](https://keda.sh/) to the base cluster installation via the [`kube_keda`](https://docs.panfactum.com/docs/edge/reference/infrastructure-modules/direct/kubernetes/kube_keda) module.
  - Deploy the `kube_keda` module as part of your cluster bootstrap process. See the related breaking change entry for installation order details.
  - Impacts: iac-module `kube_keda` — New module for deploying KEDA autoscaler
- Adds `sub_paths` to `config_map_mounts` and `secret_mounts` inputs in all applicable Panfactum submodules for more granular control over mounted files.
- Adds `backup:TagResource` to [`tf_bootstrap_resources`](https://docs.panfactum.com/docs/edge/reference/infrastructure-modules/direct/aws/tf_bootstrap_resources) to address [AWS Notification](https://github.com/Panfactum/stack/issues/291).
  - Re-apply `tf_bootstrap_resources` to receive the updated IAM policy.
  - Impacts: iac-module `tf_bootstrap_resources` — Added backup:TagResource permission to IAM policy

## Fixes

- Fixed Kubernetes RBAC issues: addressed problems with EKS access entries preventing proper permission grants and updated RBAC policies to incorporate new CRDs added since the last stable release.
  - Re-apply `aws_eks` to receive the corrected access entry permissions and updated RBAC policies.
  - Impacts: iac-module `aws_eks` — Fixed access entry permissions and updated RBAC for new CRDs
  - Reference (issue-report): [EKS access entries preventing proper RBAC permission grants](https://github.com/Panfactum/stack/issues/311)
- Fixed issue where AWS SSO sessions would not automatically sync across profiles, resulting in unnecessary SSO login prompts. Now only one SSO login is required for all AWS profiles.
  - Run `pf devshell sync` to regenerate your AWS config files with the corrected SSO session handling.
  - Reference (issue-report): [AWS SSO sessions not syncing across profiles](https://github.com/Panfactum/stack/issues/221)
- Fixes the default value for `min_node_cpu` of [`kube_karpenter_node_pools`](https://docs.panfactum.com/docs/edge/reference/infrastructure-modules/direct/kubernetes/kube_karpenter_node_pools).
  - Re-apply `kube_karpenter_node_pools` to pick up the corrected default.
  - If you previously set `min_node_cpu` explicitly to work around this issue, you can remove the override.
  - Impacts: iac-module `kube_karpenter_node_pools` — Fixed incorrect default value for min_node_cpu
- Addressed an issue that slowed down node bootstrapping due to self-imposed rate-limits on image pulling.
- Fixes private repo authentication in [`wf_dockerfile_build`](https://docs.panfactum.com/docs/edge/reference/infrastructure-modules/submodule/workflow/wf_dockerfile_build).
  - Re-apply any modules that use `wf_dockerfile_build` with private repositories to receive the fix.
  - Impacts: iac-module `wf_dockerfile_build` — Fixed private repository authentication
- Adds missing `lifetime_evictions_enabled` input to [`kube_stateful_set`](https://docs.panfactum.com/docs/edge/reference/infrastructure-modules/submodule/kubernetes/kube_stateful_set).
  - Impacts: iac-module `kube_stateful_set` — Added missing lifetime_evictions_enabled input
- Adds additional AAAA record for IPv6 support for [`aws_cdn`](https://docs.panfactum.com/docs/edge/reference/infrastructure-modules/submodule/aws/aws_cdn).
  - Re-apply any modules that use `aws_cdn` to create the AAAA record.
  - Impacts: iac-module `aws_cdn` — Added AAAA record for IPv6 support
- Addresses invalid policy document format for the DNSSEC KMS keys in [`aws_dnssec`](https://docs.panfactum.com/docs/edge/reference/infrastructure-modules/submodule/aws/aws_dnssec).
  - Re-apply any modules that use `aws_dnssec` to receive the corrected KMS key policy.
  - Impacts: iac-module `aws_dnssec` — Fixed invalid DNSSEC KMS key policy document format

## Upgrade Instructions

## Install KEDA

We now include KEDA in our base Panfactum clusters and our modules assume that you have it installed.
See the instructions [here](https://docs.panfactum.com/docs/edge/guides/bootstrapping/autoscaling/#deploy-keda).

## Migrate `kube_cert_manager` and `kube_cert_issuers` to `kube_certificates`

`kube_cert_manager` and `kube_cert_issuers` have been consolidated into a single `kube_certificates` module.

1. Create a `kube_certificates` directory as a sibling to the `kube_cert_manager` and `kube_cert_issuers` directories.

2. Create a `terragrunt.hcl` file in the `kube_certificates` directory with the following contents:

   ```hcl
   include "panfactum" {
     path   = find_in_parent_folders("panfactum.hcl")
     expose = true
   }

   terraform {
     source = include.panfactum.locals.pf_stack_source
   }

   dependency "vault_core" {
     config_path  = "../vault_core_resources"
     skip_outputs = true
   }

   inputs = {
     alert_email = "..." # Copy from kube_cert_issuers
   }
   ```

3. From the **region** directory, run the following:

   ```bash
   (cd kube_cert_issuers; terragrunt state pull > state.json);
   (cd kube_cert_manager; terragrunt state pull > state.json);
   jq -s '
     .[0] as $f1
     | .[1] as $f2
     | $f1
       | .outputs = ($f1.outputs + $f2.outputs)
       | .resources = ($f1.resources + $f2.resources)
   ' kube_cert_issuers/state.json kube_cert_manager/state.json > kube_certificates/state.json;
   jq 'del(
     .resources[]
     | select(
       .type == "pf_kube_labels"
       and (has("module") | not)
     )
   )' kube_certificates/state.json > tmp && mv tmp kube_certificates/state.json;
   rm kube_cert_issuers/state.json;
   rm kube_cert_manager/state.json;
   ```

4. Update the version of the Panfactum framework you are using.

5. Navigate to the `kube_certificates` directory.

   1. Run `terragrunt init`.
   2. Run `terragrunt state push state.json && rm state.json`.
   3. Run `terragrunt apply` and review the changes. There should be only a few resources that will be replaced.

6. Remove the `kube_cert_issuers` and `kube_cert_manager` directories.

7. Replace any references to `kube_cert_issuers` and `kube_cert_manager` with `kube_certificates` in your code.

## Rename Dedicated Cluster DNS Zone

Every cluster now has a dedicated DNS zone for hosting control-plane utilities. `kube_domain` is now a required
configuration value and should be set in the `region.yaml` for every region that houses a Kubernetes cluster.
The value must be a subdomain of a domain available to the environment.

## Configure Authentik Organization Name

We now create the Authentik email template in `kube_authentik`. The organization name is now a required input.
Use the new `organization_name` output from the `kube_authentik` module as an input to the `authentik_core_resources` module.

## Rename Backup Vault in `tf_bootstrap_resources`

The backup vault name in [`tf_bootstrap_resources`](https://docs.panfactum.com/docs/edge/reference/infrastructure-modules/direct/aws/tf_bootstrap_resources)
now has a unique suffix to prevent conflicts. **Before applying the updated module, manually delete the existing
backup vault** named `terraform-<env_name>`. Delete all recovery points first, then the vault.

## Initialize the `pf` CLI

This release adds the new `pf` CLI tool. To begin using it:

1. Complete all migration steps for the breaking changes above.
2. Run `pf devshell sync`. Ensure this completes successfully before proceeding.
3. Run `terragrunt apply` on **all** modules (or `terragrunt run-all apply`).

## Consolidate Contact Information Variables

The contact information variables on
[`aws_account`](https://docs.panfactum.com/docs/edge/reference/infrastructure-modules/direct/aws/aws_account) and
[`aws_registered_domains`](https://docs.panfactum.com/docs/edge/reference/infrastructure-modules/direct/aws/aws_registered_domains) have been
consolidated to single objects for each contact.

The inputs to [`aws_dns_zones`](https://docs.panfactum.com/docs/edge/reference/infrastructure-modules/direct/aws/aws_dns_zones) have been
consolidated to a single `domains` object for better per-domain configuration.

## Related Resources

- [JSON Data](https://docs.panfactum.com/docs/changelog/main.json): Machine-readable data