{ "id": "edge.26-04-05", "name": "edge.26-04-05", "summary": "Launches the new `pf` CLI with guided wizards for environment, cluster, domain, and SSO provisioning, upgrades Kubernetes to 1.33 and AWS provider to 6.x, migrates legacy devshell scripts to TypeScript, and consolidates several IaC modules.", "skip": false, "highlights": [ "New `pf` CLI tool with guided installers — `pf env add`, `pf cluster add`, `pf domain add`, and `pf sso add` automate end-to-end infrastructure provisioning", "`kube_cert_manager` and `kube_cert_issuers` consolidated into `kube_certificates` — state migration required", "Kubernetes default upgraded to 1.33 — review the [K8s 1.33 changelog](https://kubernetes.io/blog/2025/04/23/kubernetes-v1-33-release/) for deprecated APIs", "Node image cache modules (`kube_node_image_cache`, `kube_node_image_cache_controller`) removed — destroy existing deployments before upgrading", "Legacy bash devshell scripts migrated to `pf` subcommands — IaC modules now call `pf buildkit`, `pf wf`, and `pf kube` commands", "OpenTofu upgraded to 1.9.1 and AWS provider to 6.x — re-apply all modules after upgrading", "KEDA added to base cluster — deploy [`kube_keda`](/docs/main/reference/infrastructure-modules/direct/kubernetes/kube_keda) before applying other modules" ], "changes": [ { "id": "06c32c67-a423-459a-aa8d-f68f44397a51", "type": "breaking_change", "summary": "`kube_domain` is now a required `region.yaml` field for every cluster, specifying a dedicated DNS zone for control-plane utilities.", "description": "Panfactum clusters now require a dedicated DNS zone to host internal control-plane utilities (e.g., certificate issuers and internal service endpoints). This zone is configured via the new `kube_domain` field in `region.yaml` and must be a subdomain of a domain already managed within the environment. Separating cluster utilities onto their own DNS zone improves isolation and avoids polluting the primary domain with infrastructure-level records.", "action_items": [ "Add a `kube_domain` field to `region.yaml` for every region that contains a Kubernetes cluster. The value must be a subdomain of a domain already available in the environment (e.g., `kube.example.com` if `example.com` is managed).", "Deploy the corresponding DNS zone via `aws_dns_zones` before applying cluster modules." ], "references": [ { "type": "internal-docs", "summary": "Terragrunt variables reference — region.yaml configuration fields", "link": "/docs/main/reference/configuration/terragrunt-variables" }, { "type": "internal-docs", "summary": "aws_dns_zones module — create and manage Route 53 DNS zones", "link": "/docs/main/modules/aws_dns_zones" } ], "impacts": [ { "type": "configuration", "component": "region.yaml", "summary": "New required `kube_domain` field specifying the DNS zone for cluster control-plane utilities" }, { "type": "iac-module", "component": "kube_certificates", "summary": "Reads `kube_domain` from `region.yaml` to register certificate issuer endpoints under the cluster DNS zone" }, { "type": "iac-module", "component": "kube_cert_issuers", "summary": "Reads `kube_domain` from `region.yaml` to include the cluster DNS zone in wildcard certificate coverage" } ] }, { "id": "aa942c95-ee99-4da9-af63-2251eda0cbd3", "type": "breaking_change", "summary": "`kube_authentik` now requires `organization_name` and creates the Authentik email template; update `authentik_core_resources` to consume this output.", "description": "Email templates rendered by Authentik now embed the organization name for branding, so the `organization_name` input must be provided to `kube_authentik` at deploy time. The module creates the email template `ConfigMap` and exposes `organization_name` as an output so that `authentik_core_resources` can consume it via a Terragrunt `dependency` block. This establishes a clear ownership boundary: `kube_authentik` owns Authentik infrastructure and branding assets, while `authentik_core_resources` owns application-level configuration built on top of it.", "action_items": [ "Add the `organization_name` variable to your `kube_authentik` module configuration.", "Wire the `organization_name` output from `kube_authentik` into the `authentik_core_resources` module as an input via a `dependency` block." ], "references": [ { "type": "internal-docs", "summary": "kube_authentik module reference", "link": "/docs/main/modules/kube_authentik" }, { "type": "internal-docs", "summary": "authentik_core_resources module reference", "link": "/docs/main/modules/authentik_core_resources" } ], "impacts": [ { "type": "iac-module", "component": "kube_authentik", "summary": "New required `organization_name` input; now creates email template `ConfigMap` and exposes `organization_name` output" }, { "type": "iac-module", "component": "authentik_core_resources", "summary": "Must receive `organization_name` wired from `kube_authentik` output via a `dependency` block" } ] }, { "id": "83b916af-9fd7-4378-8f6f-90ca50c0b090", "type": "breaking_change", "summary": "`kube_cert_manager` and `kube_cert_issuers` have been consolidated into a single `kube_certificates` module to address various race conditions on cluster installation.", "description": "The separate `kube_cert_manager` and `kube_cert_issuers` modules created a race condition during cluster bootstrap where cert-manager would be ready before its issuers were configured, causing dependent modules to fail. Merging both into a single `kube_certificates` module eliminates this timing issue by ensuring cert-manager and its cluster issuers are deployed atomically in one apply. This also reduces the number of module directories users must manage and simplifies dependency wiring in downstream modules.", "action_items": [ "Create a new `kube_certificates` module directory and deploy it using the configuration shown in the upgrade instructions.", "Run the state migration script from the upgrade instructions to merge Terraform state from both `kube_cert_manager` and `kube_cert_issuers` into `kube_certificates`.", "Remove the `kube_cert_issuers` and `kube_cert_manager` directories after confirming the migration succeeds.", "Replace any references to `kube_cert_issuers` or `kube_cert_manager` dependency paths in your `terragrunt.hcl` files with `kube_certificates`." ], "references": [ { "type": "internal-commit", "summary": "Merge certificates modules (#353)", "link": "https://github.com/Panfactum/stack/commit/48d67a2721e3df5e482db5e975bc76722451dd2a" }, { "type": "internal-docs", "summary": "kube_certificates module reference", "link": "/docs/main/modules/kube_certificates" }, { "type": "internal-docs", "summary": "Certificate management bootstrapping guide", "link": "/docs/main/guides/bootstrapping/certificate-management" } ], "impacts": [ { "type": "iac-module", "component": "kube_cert_manager", "summary": "Deprecated and consolidated into `kube_certificates`; directory should be removed after state migration" }, { "type": "iac-module", "component": "kube_cert_issuers", "summary": "Deprecated and consolidated into `kube_certificates`; directory should be removed after state migration" }, { "type": "iac-module", "component": "kube_certificates", "summary": "New module replacing both `kube_cert_manager` and `kube_cert_issuers`; deploys cert-manager and cluster issuers atomically" } ] }, { "id": "ff442e61-ff63-44c6-a53a-faf5c05895a3", "type": "breaking_change", "summary": "`kube_keda` is now required as part of the base Panfactum cluster installation, and other modules assume it is present.", "description": "KEDA (Kubernetes Event-driven Autoscaling) extends the built-in Kubernetes Horizontal Pod Autoscaler with capabilities the HPA cannot provide: scaling to zero, combining horizontal and vertical autoscaling simultaneously, and reacting to hundreds of external event sources beyond CPU and memory. Making `kube_keda` a required base component unlocks these patterns for all Panfactum workloads and allows other platform modules to depend on KEDA CRDs being available. Clusters that do not have `kube_keda` deployed will fail to apply any module that now assumes its presence.", "action_items": [ "Deploy the `kube_keda` module in every cluster following the [installation guide](/docs/main/guides/bootstrapping/autoscaling#deploy-keda). This must be done before applying other modules that now depend on KEDA being present." ], "references": [ { "type": "internal-commit", "summary": "Add kube_keda module and integrate KEDA into base cluster setup", "link": "https://github.com/Panfactum/stack/commit/09bdf9841b6f58d66cb3768cc81a331eb6ce8631" }, { "type": "external-docs", "summary": "KEDA — Kubernetes Event-driven Autoscaling official site", "link": "https://keda.sh/" }, { "type": "internal-docs", "summary": "`kube_keda` module reference documentation", "link": "/docs/main/modules/kube_keda/overview" }, { "type": "internal-docs", "summary": "Bootstrapping guide — Deploy KEDA section", "link": "/docs/main/guides/bootstrapping/autoscaling" } ], "impacts": [ { "type": "iac-module", "component": "kube_keda", "summary": "Now a required part of base cluster installation; must be deployed before other modules that depend on KEDA" } ] }, { "id": "a1c98be8-0d45-4a96-bc57-955149a655ad", "type": "breaking_change", "summary": "`burstable_nodes_enabled` now defaults to `true` for all workload modules, restoring the intended behavior that was broken in `edge.25-03-26`.", "description": "In `edge.25-03-26`, the separation of `burstable_nodes_enabled` and `spot_nodes_enabled` inadvertently changed the default for `burstable_nodes_enabled` to `false`, preventing workloads from scheduling on cost-effective T-family instances unless explicitly opted in. This release restores the default to `true` so that burstable instances are available for scheduling by default, which better reflects the intended cost-optimization posture of the framework.", "action_items": [ "If you explicitly set `burstable_nodes_enabled = false` in any module, no action is needed.", "If you rely on the previous `false` default and your workloads are not compatible with burstable (T-family) instances, explicitly set `burstable_nodes_enabled = false` in the affected module configurations before upgrading.", "Review workloads with sustained CPU utilization above ~30% — burstable instances run in unlimited mode and may cost more than standard `M`-type instances for these workloads." ], "references": [ { "type": "internal-docs", "summary": "Deploying workloads guide — node scheduling options", "link": "/docs/main/guides/deploying-workloads/basics" }, { "type": "internal-docs", "summary": "`kube_workload_utility` module reference", "link": "/docs/main/modules/kube_workload_utility/overview" }, { "type": "external-docs", "summary": "AWS burstable performance instances documentation", "link": "https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/burstable-performance-instances.html" } ], "impacts": [ { "type": "iac-module", "component": "kube_workload_utility", "summary": "`burstable_nodes_enabled` default changed from `false` to `true`" }, { "type": "iac-module", "component": "kube_pod", "summary": "Inherits new `burstable_nodes_enabled` default from `kube_workload_utility`" }, { "type": "iac-module", "component": "kube_deployment", "summary": "Inherits new `burstable_nodes_enabled` default from `kube_workload_utility`" }, { "type": "iac-module", "component": "kube_stateful_set", "summary": "Inherits new `burstable_nodes_enabled` default from `kube_workload_utility`" }, { "type": "iac-module", "component": "kube_cron_job", "summary": "Inherits new `burstable_nodes_enabled` default from `kube_workload_utility`" }, { "type": "iac-module", "component": "kube_daemon_set", "summary": "Inherits new `burstable_nodes_enabled` default from `kube_workload_utility`" }, { "type": "iac-module", "component": "kube_job", "summary": "Inherits new `burstable_nodes_enabled` default from `kube_workload_utility`" } ] }, { "id": "2be688f2-375c-451b-909a-71b2831f3bd2", "type": "breaking_change", "summary": "`aws_account` and `aws_registered_domains` contact information variables have been consolidated from individual fields into single object variables per contact type.", "description": "Previously, each contact field (name, phone, address, etc.) was a separate top-level variable, making module calls verbose and error-prone. Both modules now accept a single typed object per contact role, grouping all related fields together. This reduces the number of variables, improves readability, and enables better validation at the object level. For `aws_account`, the individual `contact_*`, `security_*`, `operations_*`, and `billing_*` variables are replaced by `primary_contact`, `security_contact`, `operations_contact`, and `billing_contact` objects. For `aws_registered_domains`, the individual `admin_*`, `registrant_*`, and `tech_*` variables are replaced by `admin_contact`, `registrant_contact`, and `tech_contact` objects.", "action_items": [ "In `aws_account`, replace the individual `contact_*` variables (`contact_full_name`, `contact_phone_number`, `contact_address_line_1`, etc.) with a single `primary_contact` object. Similarly, replace `security_*`, `operations_*`, and `billing_*` variables with `security_contact`, `operations_contact`, and `billing_contact` objects respectively.", "In `aws_registered_domains`, replace the individual `admin_*` variables (`admin_first_name`, `admin_last_name`, `admin_email_address`, etc.) with a single `admin_contact` object. Similarly, replace `registrant_*` and `tech_*` variables with `registrant_contact` and `tech_contact` objects respectively.", "Run `terragrunt apply` for both modules after updating the variable definitions to reconcile the state." ], "references": [ { "type": "internal-commit", "summary": "Consolidate `aws_account` contact variables into typed objects", "link": "https://github.com/Panfactum/stack/commit/265d0a3fa27c15364fd814fd44d66aac4396bb88" }, { "type": "internal-commit", "summary": "Consolidate `aws_registered_domains` contact variables into typed objects", "link": "https://github.com/Panfactum/stack/commit/b9257537a66620e8a531a64fc5f136739e3230d6" }, { "type": "internal-docs", "summary": "`aws_account` module reference", "link": "/docs/main/reference/infrastructure-modules/direct/aws/aws_account" }, { "type": "internal-docs", "summary": "`aws_registered_domains` module reference", "link": "/docs/main/reference/infrastructure-modules/direct/aws/aws_registered_domains" } ], "impacts": [ { "type": "iac-module", "component": "aws_account", "summary": "Individual `contact_*`, `security_*`, `operations_*`, and `billing_*` variables replaced by `primary_contact`, `security_contact`, `operations_contact`, and `billing_contact` typed objects" }, { "type": "iac-module", "component": "aws_registered_domains", "summary": "Individual `admin_*`, `registrant_*`, and `tech_*` variables replaced by `admin_contact`, `registrant_contact`, and `tech_contact` typed objects" } ] }, { "id": "2e4d9f33-5244-4e43-a80a-958942c69735", "type": "breaking_change", "summary": "`aws_dns_zones` inputs have been consolidated to a single `domains` object for better per-domain granular configuration.", "description": "Previously, `aws_dns_zones` accepted separate list variables for domain names and their settings. These have been replaced with a single `domains` map where each key is a domain name and the value holds per-domain configuration such as `dnssec_enabled`. This makes it easier to configure individual domains independently without needing to keep parallel lists in sync.", "action_items": [ "Migrate your `aws_dns_zones` inputs from separate domain list variables to the new `domains` map. Each key in the map is a domain name, and the value is an object containing per-domain configuration (e.g., `dnssec_enabled`).", "Review the updated module reference for the full `domains` input schema." ], "references": [ { "type": "internal-docs", "summary": "aws_dns_zones module reference", "link": "/docs/main/reference/infrastructure-modules/direct/aws/aws_dns_zones" } ], "impacts": [ { "type": "iac-module", "component": "aws_dns_zones", "summary": "All inputs replaced by a single `domains` map; existing configurations must be migrated" } ] }, { "id": "7c95855b-139a-4ed7-a5d8-c5a941403255", "type": "breaking_change", "summary": "`tf_bootstrap_resources` backup vault name now includes a unique suffix. Manually delete the existing `terraform-` vault before applying.", "description": "The backup vault created by `tf_bootstrap_resources` previously used a predictable name (`terraform-`), which could not be re-created after deletion without manual intervention due to AWS Backup vault naming constraints. The vault name now includes a random hex suffix generated by a `random_id` resource, ensuring uniqueness and enabling clean re-creation. Because AWS Backup does not support in-place renames, the existing vault must be manually deleted before applying the updated module.", "action_items": [ "Delete the existing AWS Backup vault named `terraform-` in each environment via the AWS console or `aws backup delete-backup-vault --backup-vault-name terraform-`.", "Re-apply the `tf_bootstrap_resources` module to create the replacement vault with the new uniquely-suffixed name." ], "references": [ { "type": "internal-commit", "summary": "Add unique suffix to backup vault name in `tf_bootstrap_resources`", "link": "https://github.com/Panfactum/stack/commit/d07955d5d1f1de54783cd8baf5c8777f5c9588ed" }, { "type": "internal-docs", "summary": "`tf_bootstrap_resources` module reference", "link": "/docs/main/reference/infrastructure-modules/direct/aws/tf_bootstrap_resources" } ], "impacts": [ { "type": "iac-module", "component": "tf_bootstrap_resources", "summary": "Backup vault name now includes a random unique suffix; existing vault must be manually deleted before applying" } ] }, { "id": "cb38cb96-5cf7-4eb9-9e30-f8879f4fb3e5", "type": "breaking_change", "summary": "Adds the `pf` CLI tool; run `pf devshell sync` and `terragrunt run-all apply` after upgrading.", "description": "The `pf` CLI centralizes Panfactum's operational commands into a single, structured tool built on Clipanion. It replaces the collection of ad-hoc shell scripts previously scattered across the devshell with a unified command hierarchy (`pf devshell sync`, `pf db tunnel`, `pf vault get-token`, etc.). The `pf devshell sync` command synchronizes local configuration files, kubeconfigs, SSH settings, and AWS Identity Center profiles with live infrastructure state, ensuring local development environments stay consistent with deployed resources. Running `terragrunt run-all apply` afterward propagates any configuration changes introduced by the sync into your infrastructure modules.", "action_items": [ "Run `pf devshell sync` to synchronize local kubeconfigs, SSH settings, AWS profiles, and standard configuration files with live infrastructure.", "Run `terragrunt run-all apply` across all modules to propagate the configuration changes introduced by the sync." ], "references": [ { "type": "internal-commit", "summary": "Initial implementation of the `pf` CLI tool", "link": "https://github.com/Panfactum/stack/commit/aa49614d10a189ac5e6744b0949f8cf994e27009" }, { "type": "internal-commit", "summary": "Adds `pf devshell sync` tasks for infrastructure synchronization", "link": "https://github.com/Panfactum/stack/commit/7b1f505992598d12b16864c9ec33865c97232d9b" } ], "impacts": [ { "type": "cli", "component": "devshell sync", "summary": "New command that synchronizes local devshell configuration with live infrastructure state" }, { "type": "devshell", "component": "pf", "summary": "New unified CLI tool replacing ad-hoc shell scripts with a structured command hierarchy" } ] }, { "id": "291aeaa5-62df-456c-ae43-20a074e4c34e", "type": "addition", "summary": "Adds the `kube_job` submodule for running one-off Kubernetes Jobs as part of module deployment processes (e.g., database migrations).", "description": "Many deployment workflows require one-off tasks such as database migrations, data seeding, or cluster bootstrapping steps that must complete before the main workload starts. The `kube_job` submodule provides a production-hardened Kubernetes Job abstraction with the same security hardening, scheduling, autoscaling, and observability integrations as the other Panfactum workload modules. Use it in your custom IaC modules whenever you need a run-to-completion task as part of a deployment.", "references": [ { "type": "internal-commit", "summary": "Initial implementation of `kube_job` submodule", "link": "https://github.com/Panfactum/stack/commit/dafb4d5782ba675eb6a0c4eff407ea59ab6f815e" }, { "type": "internal-docs", "summary": "`kube_job` module documentation", "link": "/docs/main/modules/kube_job" }, { "type": "external-docs", "summary": "Kubernetes Job documentation", "link": "https://kubernetes.io/docs/concepts/workloads/controllers/job/" } ], "impacts": [ { "type": "iac-module", "component": "kube_job", "summary": "New submodule providing a production-hardened Kubernetes Job with pod security, VPA, PDB, bin-packing scheduling, and failure policy support" } ] }, { "id": "64d65505-8929-404e-8c16-1083e6f46ed8", "type": "addition", "summary": "Adds the `kube_keda` module for deploying KEDA (Kubernetes Event-Driven Autoscaling) as part of the base cluster installation.", "description": "KEDA extends the built-in Kubernetes Horizontal Pod Autoscaler with capabilities the HPA cannot provide on its own: scaling to zero, combining horizontal and vertical autoscaling simultaneously, and reacting to hundreds of external event sources beyond CPU and memory. The new `kube_keda` module deploys KEDA via its official Helm chart with production-hardened defaults including cert-manager integration, VPA resources, pod disruption budgets, and SLA-aware replica counts. Adding KEDA as a base platform component enables all Panfactum workloads to leverage event-driven autoscaling without additional setup.", "action_items": [ "Deploy the `kube_keda` module as part of your cluster bootstrap process. See the related breaking change entry for installation order details." ], "references": [ { "type": "internal-commit", "summary": "Add `kube_keda` module and integrate KEDA into the base cluster setup", "link": "https://github.com/Panfactum/stack/commit/09bdf9841b6f58d66cb3768cc81a331eb6ce8631" }, { "type": "external-docs", "summary": "KEDA official website", "link": "https://keda.sh/" }, { "type": "external-docs", "summary": "KEDA GitHub repository", "link": "https://github.com/kedacore/keda" }, { "type": "internal-docs", "summary": "`kube_keda` module documentation", "link": "/docs/main/modules/kube_keda/overview" }, { "type": "internal-docs", "summary": "Bootstrapping guide — Deploy KEDA section", "link": "/docs/main/guides/bootstrapping/autoscaling" } ], "impacts": [ { "type": "iac-module", "component": "kube_keda", "summary": "New module deploying KEDA via its official Helm chart with cert-manager TLS, VPA, PDBs, topology spread constraints, and SLA-aware replica configuration" } ] }, { "id": "769d03d6-ee6f-4401-8229-81a88ae024d5", "type": "addition", "summary": "Adds `sub_paths` option to `config_map_mounts` and `secret_mounts` inputs in all workload modules for mounting individual files instead of entire volumes.", "description": "Previously, mounting a ConfigMap or Secret in a Panfactum workload module would always mount the entire volume at the specified path, replacing any existing files in that directory. This made it impossible to inject a single configuration file into a directory that already contained other files. The new `sub_paths` option uses the Kubernetes `subPath` mechanism to mount only the specified keys, each as an individual file at `/`, leaving the rest of the directory intact.", "references": [ { "type": "internal-commit", "summary": "Add `sub_paths` to ConfigMap and Secret mount inputs", "link": "https://github.com/Panfactum/stack/commit/dafb4d5782ba675eb6a0c4eff407ea59ab6f815e" }, { "type": "external-docs", "summary": "Kubernetes documentation — using `subPath` to mount specific ConfigMap keys", "link": "https://kubernetes.io/docs/tasks/configure-pod-container/configure-pod-configmap/" }, { "type": "internal-docs", "summary": "`kube_pod` module reference", "link": "/docs/main/modules/kube_pod" } ], "impacts": [ { "type": "iac-module", "component": "kube_pod", "summary": "Added `sub_paths` option to `secret_mounts` and `config_map_mounts` inputs" }, { "type": "iac-module", "component": "kube_deployment", "summary": "Added `sub_paths` option to `secret_mounts` and `config_map_mounts` inputs" }, { "type": "iac-module", "component": "kube_stateful_set", "summary": "Added `sub_paths` option to `secret_mounts` and `config_map_mounts` inputs" }, { "type": "iac-module", "component": "kube_cron_job", "summary": "Added `sub_paths` option to `secret_mounts` and `config_map_mounts` inputs" }, { "type": "iac-module", "component": "kube_job", "summary": "Added `sub_paths` option to `secret_mounts` and `config_map_mounts` inputs" }, { "type": "iac-module", "component": "wf_spec", "summary": "Added `sub_paths` option to `secret_mounts` and `config_map_mounts` inputs" } ] }, { "id": "9427e24b-80be-40a0-b915-fc37f0739ee4", "type": "improvement", "summary": "`tf_bootstrap_resources` now includes the `backup:TagResource` IAM permission required by AWS Backup to tag recovery points.", "description": "AWS began enforcing the `backup:TagResource` permission for backup service roles that tag recovery points. Without this permission, AWS Backup jobs that apply tags to recovery points fail with an access-denied error. This change adds the missing permission to the backup IAM policy managed by `tf_bootstrap_resources`.", "action_items": [ "Re-apply `tf_bootstrap_resources` to receive the updated IAM policy." ], "references": [ { "type": "internal-commit", "summary": "Add `backup:TagResource` to backup-terraform role", "link": "https://github.com/Panfactum/stack/commit/bf3eebd3148ad48ef5d443d511195888027d62fb" }, { "type": "issue-report", "summary": "AWS notification about missing `backup:TagResource` permission", "link": "https://github.com/Panfactum/stack/issues/291" }, { "type": "external-docs", "summary": "AWS Backup IAM actions and resource types reference", "link": "https://docs.aws.amazon.com/service-authorization/latest/reference/list_awsbackup.html" } ], "impacts": [ { "type": "iac-module", "component": "tf_bootstrap_resources", "summary": "Added `backup:TagResource` permission to the backup IAM policy" } ] }, { "id": "4a30e149-bbb1-48d6-97dc-96096df6e4f0", "type": "fix", "summary": "`aws_eks` access entries now grant correct permissions and RBAC policies include new CRDs.", "description": "EKS access entries were misconfigured, preventing IAM principals from receiving the intended Kubernetes permissions. Additionally, RBAC policies had not been updated to cover CRDs introduced since the last stable release, causing permission errors when interacting with newer custom resources. Both issues are resolved in this fix.", "action_items": [ "Re-apply `aws_eks` to receive the corrected access entry permissions and updated RBAC policies." ], "references": [ { "type": "issue-report", "summary": "EKS access entries preventing proper RBAC permission grants", "link": "https://github.com/Panfactum/stack/issues/311" }, { "type": "external-docs", "summary": "AWS documentation on granting IAM users access via EKS access entries", "link": "https://docs.aws.amazon.com/eks/latest/userguide/access-entries.html" }, { "type": "internal-docs", "summary": "`aws_eks` module reference documentation", "link": "/docs/main/reference/infrastructure-modules/direct/aws/aws_eks" } ], "impacts": [ { "type": "iac-module", "component": "aws_eks", "summary": "Fixed access entry permissions and updated RBAC policies for new CRDs" } ] }, { "id": "5cae977d-c96b-4581-833c-ee6b028c3dda", "type": "fix", "summary": "AWS SSO sessions now sync across all profiles, eliminating redundant login prompts.", "description": "Previously, each AWS profile maintained its own independent SSO session, so users were prompted to re-authenticate when switching between profiles even though they had already completed SSO login. The generated AWS config now uses a shared `sso-session` block, ensuring a single SSO login is valid across all configured profiles.", "action_items": [ "Run `pf devshell sync` to regenerate your AWS config files with the corrected SSO session handling." ], "references": [ { "type": "issue-report", "summary": "AWS SSO sessions not syncing across profiles", "link": "https://github.com/Panfactum/stack/issues/221" } ], "impacts": [ { "type": "cli", "component": "devshell sync", "summary": "Generates AWS config with a shared `sso-session` block across all profiles" } ] }, { "id": "43ce5fc7-4f72-4b27-ad8a-cea0c8693091", "type": "fix", "summary": "Fixes the default value for `min_node_cpu` in `kube_karpenter_node_pools` so that Karpenter no longer excludes valid small instance types.", "description": "The `min_node_cpu` variable was originally introduced with a default of `0.5`. Because `kube_karpenter_node_pools` uses a strict greater-than (`Gt`) requirement on `karpenter.k8s.aws/instance-cpu`, this effectively set a floor just below 1 vCPU. While 1-vCPU instances still qualified, the non-integer default was semantically incorrect and could cause confusion. The default is now `0`, which imposes no minimum CPU constraint and lets Karpenter consider the full range of instance types allowed by other NodePool requirements.", "action_items": [ "Re-apply `kube_karpenter_node_pools` to pick up the corrected default.", "If you previously set `min_node_cpu` explicitly to work around this issue, you can remove the override." ], "references": [ { "type": "internal-commit", "summary": "Fix `min_node_cpu` default for `kube_karpenter_node_pools`", "link": "https://github.com/Panfactum/stack/commit/40f95e1a37803879a090c1bc1abb73994173ed07" }, { "type": "internal-docs", "summary": "`kube_karpenter_node_pools` module documentation", "link": "/docs/main/reference/infrastructure-modules/direct/kubernetes/kube_karpenter_node_pools" }, { "type": "external-docs", "summary": "Karpenter NodePools concept documentation", "link": "https://karpenter.sh/docs/concepts/nodepools/" } ], "impacts": [ { "type": "iac-module", "component": "kube_karpenter_node_pools", "summary": "Changed `min_node_cpu` default from `0.5` to `0`" } ] }, { "id": "64f4d36f-9324-4308-9c48-f9fae50e773e", "type": "fix", "summary": "Increased kubelet `registry-qps` and `registry-burst` settings in `kube_node_settings` to eliminate self-imposed rate-limiting that slowed node bootstrap image pulls.", "description": "The default Kubernetes kubelet settings for `registry-qps` (5) and `registry-burst` (10) throttle how quickly a node can pull container images from registries. On new nodes with many pods scheduled simultaneously, this caused a bottleneck during bootstrapping as images queued behind the rate limiter. This fix raises the limits to 50 QPS and 100 burst in the Bottlerocket user-data configuration, allowing nodes to pull images in parallel without artificial throttling. No action required beyond re-applying the `kube_karpenter_node_pools` module.", "references": [ { "type": "internal-commit", "summary": "Bump registry QPS and burst settings in node user-data", "link": "https://github.com/Panfactum/stack/commit/27c0d4149e0d14d7f643b90194a5deee30b25e3e" }, { "type": "issue-report", "summary": "Bottlerocket issue requesting `registry-burst` and `registry-qps` kubelet arguments", "link": "https://github.com/bottlerocket-os/bottlerocket/issues/1495" }, { "type": "external-docs", "summary": "Bottlerocket Kubernetes settings documentation", "link": "https://bottlerocket.dev/en/os/1.54.x/api/settings/kubernetes/" } ], "impacts": [ { "type": "iac-module", "component": "kube_node_settings", "summary": "Added `registry-qps` (50) and `registry-burst` (100) to Bottlerocket user-data" }, { "type": "iac-module", "component": "kube_karpenter_node_pools", "summary": "Nodes provisioned by Karpenter inherit the faster image pull settings" } ] }, { "id": "0efd6aa3-cb9e-4b3f-8ad8-c27f1c1933d4", "type": "fix", "summary": "Fixes `wf_dockerfile_build` failing to authenticate when cloning private Git repositories.", "description": "The `setup.sh` script in `wf_dockerfile_build` was not incorporating the `git_username` and `git_password` variables into the HTTPS URL used by `pf-get-commit-hash` to resolve the commit reference. As a result, workflows targeting private Git repositories would fail at the setup step with an authentication error. The fix constructs the authenticated URL (`https://:@`) when `GIT_USERNAME` is provided, and also corrects a follow-up bug where the wrong shell variable names (`USERNAME`/`PASSWORD` instead of `GIT_USERNAME`/`GIT_PASSWORD`) were used.", "action_items": [ "Re-apply any modules that use `wf_dockerfile_build` with private repositories to pick up the corrected setup script." ], "references": [ { "type": "internal-commit", "summary": "Inject `git_username`/`git_password` into repo URL for private repo auth", "link": "https://github.com/Panfactum/stack/commit/36018714e101bc216537a8eb6dfc1854c60cfe01" }, { "type": "internal-commit", "summary": "Fix incorrect variable names (`USERNAME`/`PASSWORD` to `GIT_USERNAME`/`GIT_PASSWORD`)", "link": "https://github.com/Panfactum/stack/commit/8fe2e40a5f3174679470209208133a32698a2f66" }, { "type": "internal-docs", "summary": "`wf_dockerfile_build` module documentation", "link": "/docs/main/reference/infrastructure-modules/submodule/workflow/wf_dockerfile_build" } ], "impacts": [ { "type": "iac-module", "component": "wf_dockerfile_build", "summary": "Fixed setup script to inject `git_username` and `git_password` into the HTTPS clone URL so private repository workflows authenticate correctly" } ] }, { "id": "a018e043-8497-43a2-9242-3f5e8bc75e82", "type": "fix", "summary": "`kube_stateful_set` now exposes the `lifetime_evictions_enabled` input that was missing from its interface.", "description": "The `kube_stateful_set` module delegates to `kube_pod`, which accepts a `lifetime_evictions_enabled` input to control whether the Descheduler can evict pods after exceeding a configured maximum age. This input was not surfaced through `kube_stateful_set`, so users had no way to toggle lifetime eviction behavior for StatefulSet workloads. The input is now exposed with a default of `false`, matching `kube_pod` behavior.", "references": [ { "type": "internal-docs", "summary": "`kube_stateful_set` module documentation", "link": "/docs/main/reference/infrastructure-modules/submodule/kubernetes/kube_stateful_set" } ], "impacts": [ { "type": "iac-module", "component": "kube_stateful_set", "summary": "Added `lifetime_evictions_enabled` input, allowing control over Descheduler pod lifetime evictions" } ] }, { "id": "4ef9ab0a-6ac6-45b6-a624-7e9a090075c6", "type": "fix", "summary": "`aws_cdn` now creates AAAA DNS records so that IPv6 clients can resolve CloudFront distribution domains.", "description": "CloudFront distributions have IPv6 enabled by default, but the `aws_cdn` module only created A (IPv4) alias records in Route 53. IPv6-only clients could not resolve the CDN domain names. This fix adds a corresponding AAAA alias record for every configured domain, ensuring full dual-stack DNS resolution.", "action_items": [ "Re-apply any `aws_cdn` deployments with `terragrunt apply` to create the new AAAA records." ], "references": [ { "type": "internal-commit", "summary": "Add AAAA Route 53 alias record for each CloudFront domain", "link": "https://github.com/Panfactum/stack/commit/5a39ffe25bc2abac97a76b802195f4935acd1126" }, { "type": "external-docs", "summary": "AWS documentation on enabling IPv6 for CloudFront distributions", "link": "https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/cloudfront-enable-ipv6.html" }, { "type": "external-docs", "summary": "Routing traffic to a CloudFront distribution using Route 53 alias records", "link": "https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/routing-to-cloudfront-distribution.html" } ], "impacts": [ { "type": "iac-module", "component": "aws_cdn", "summary": "Added AAAA alias record for each domain to enable IPv6 DNS resolution" } ] }, { "id": "81c4abd3-af9e-4028-a6f2-c78cf5fa9388", "type": "fix", "summary": "`aws_dnssec` KMS key policy was missing a required `resources` field in the `CreateGrant` statement, causing policy validation failures.", "description": "The IAM policy document for the DNSSEC KMS key was missing a `resources = [\"*\"]` block in the `CreateGrant` statement. While the other two statements in the policy included the required `resources` field, this omission could cause Terraform to produce an invalid policy document, preventing key creation or updates.", "action_items": [ "Re-apply `aws_dnssec` to update the KMS key policy with the corrected `CreateGrant` statement." ], "references": [ { "type": "internal-commit", "summary": "fix: dnssec kms key policy document", "link": "https://github.com/Panfactum/stack/commit/460bafdf28bfb9dc5f31c0de2b00d084309e0c15" }, { "type": "external-docs", "summary": "AWS documentation on customer managed keys for DNSSEC", "link": "https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/dns-configuring-dnssec-cmk-requirements.html" }, { "type": "internal-docs", "summary": "`aws_dnssec` module reference", "link": "/docs/main/reference/infrastructure-modules/submodule/aws/aws_dnssec" } ], "impacts": [ { "type": "iac-module", "component": "aws_dnssec", "summary": "Added missing `resources` field to the `CreateGrant` policy statement for DNSSEC KMS keys" } ] }, { "id": "7ce75fe6-f7d1-45df-9f12-2ef29dfa1b96", "type": "improvement", "summary": "Stable releases are now open to all users with a new `stable..` format and six-month support window.", "description": "Previously, stable releases were only available to stack subscribers. Stable releases are now publicly available to all users of the Panfactum framework. The release naming convention has changed from `.` to `stable..` (e.g., `stable.25-04.0`). The support window has been reduced from twelve months to six months from the initial release date. New stable release channels will be created roughly three times per year, aligned with new Kubernetes releases rather than the previous twice-per-year cadence.", "references": [ { "type": "internal-commit", "summary": "Open stable releases to all users and update release format, cadence, and support policy", "link": "https://github.com/Panfactum/stack/commit/70491731293c2383488a1ca9bd68f93f6bcc8c27" }, { "type": "internal-docs", "summary": "Versioning and releases documentation", "link": "/docs/main/guides/versioning/releases" }, { "type": "internal-docs", "summary": "Contributing release process documentation", "link": "/docs/main/guides/contributing/releasing" } ] }, { "id": "1ccbab75-947d-48e7-99ce-1b8bff00b478", "type": "addition", "summary": "New `install.sh` script automates full Panfactum repository and DevShell setup from scratch.", "description": "Getting started with Panfactum previously required manually installing prerequisites, cloning a repository, creating a `flake.nix`, and configuring `direnv` -- a multi-step process that was error-prone for new users. The new `install.sh` bootstrapper handles all of this in a single command. It validates that `git` >= 2.39, `nix` >= 2.23, and `direnv` >= 2.32 are present (installing `nix` and `direnv` automatically if missing), prompts for repository URL, name, and branch, generates `flake.nix` and `panfactum.yaml`, builds the DevShell via `pf devshell sync`, and activates `direnv`. The script reads user input from `/dev/tty` so it works when piped via `curl`, detects the user's shell (`bash` or `zsh`) to append the `direnv` hook to the appropriate config file, and converts SSH remote URLs to HTTPS automatically.", "references": [ { "type": "internal-commit", "summary": "chore: creates initial panfactum installer", "link": "https://github.com/Panfactum/stack/commit/74bc98bf931abfee89143015332bd3261cefcb59" }, { "type": "internal-commit", "summary": "feat(installer): complete end-to-end install flow with flake.nix, pf-update, and direnv allow", "link": "https://github.com/Panfactum/stack/commit/e7ade5824f74f7dc6a778514a23c7da891c3a4fb" }, { "type": "internal-commit", "summary": "fix: installer user input via pipe", "link": "https://github.com/Panfactum/stack/commit/dd6a76203cec41092a81ea963b513732159f5a70" }, { "type": "internal-commit", "summary": "fix: installer user input via pipe - take 2", "link": "https://github.com/Panfactum/stack/commit/ab63aee9dfda4b84b936583c229ff68d3d14a204" }, { "type": "internal-commit", "summary": "fix: installer git checks and ignore nix warnings", "link": "https://github.com/Panfactum/stack/commit/9aa5de525f498086e3f0a9f95f24cf292de8056e" }, { "type": "internal-commit", "summary": "fix: source nix in installer", "link": "https://github.com/Panfactum/stack/commit/0b66035e3ed76459ad2e1faa5150a52e509d467d" }, { "type": "internal-commit", "summary": "fix: repo url input for installer", "link": "https://github.com/Panfactum/stack/commit/6ff1dbd263c3082b52743eba81be1edd9b42aed7" }, { "type": "internal-commit", "summary": "fix: default repo name for installer", "link": "https://github.com/Panfactum/stack/commit/7be27a540ce6a3c7ca1fd52f65d4962ade472045" }, { "type": "internal-commit", "summary": "fix: direnv installer", "link": "https://github.com/Panfactum/stack/commit/fd8126dddc92d163f8080babaf07558568606fee" }, { "type": "internal-commit", "summary": "fix: direnv hook check", "link": "https://github.com/Panfactum/stack/commit/7b1091c71ab8443e201aa32e95bd7a609ad29e43" }, { "type": "internal-commit", "summary": "fix: auto-append direnv hook to shell config in install.sh", "link": "https://github.com/Panfactum/stack/commit/eb60e19e23fe6b4013d109d35121ac875c472618" }, { "type": "internal-commit", "summary": "fix: direnv hook installer", "link": "https://github.com/Panfactum/stack/commit/486c6c56ce10edc26253d6a52f789859230978ab" }, { "type": "internal-commit", "summary": "fix: update stage 1 installer to use pf commands", "link": "https://github.com/Panfactum/stack/commit/485bb8f898b65caacfb8d9180dec0a111f95c88c" }, { "type": "internal-commit", "summary": "fix: better install instructions + missing region", "link": "https://github.com/Panfactum/stack/commit/21d70937a4329fabf4872039ce14b63bb039dc91" }, { "type": "internal-commit", "summary": "fix: reduce the required git version", "link": "https://github.com/Panfactum/stack/commit/4b3b81c075f4092137104004181723c904f7fc24" }, { "type": "internal-commit", "summary": "fix: add automatic direnv sourcing", "link": "https://github.com/Panfactum/stack/commit/b41077c7f5b6eac6865506c1f27c5f247cfaed04" }, { "type": "internal-commit", "summary": "fix:installer: direnv shell reload warning", "link": "https://github.com/Panfactum/stack/commit/eef70afeaf4814a695db8360d8489a246d5bb9c7" }, { "type": "external-docs", "summary": "direnv hook installation documentation", "link": "https://direnv.net/docs/hook.html" }, { "type": "external-docs", "summary": "Determinate Systems Nix installer", "link": "https://determinate.systems/nix-installer/" }, { "type": "internal-docs", "summary": "Install tooling prerequisites guide", "link": "/docs/main/guides/getting-started/install-tooling" }, { "type": "internal-docs", "summary": "Boot developer environment guide", "link": "/docs/main/guides/getting-started/boot-developer-environment" } ], "impacts": [ { "type": "installer", "component": "install.sh", "summary": "New bootstrapper script that automates prerequisite validation, `flake.nix` generation, DevShell build via `pf devshell sync`, and `direnv` activation" }, { "type": "devshell", "component": "enter-shell-local", "summary": "Now supports `PF_SKIP_CHECK_REPO_SETUP=1` to bypass repo setup validation on shell entry" } ] }, { "id": "f2af2200-44c4-4c0b-859c-150e07f82592", "type": "breaking_change", "summary": "The `PF_SKIP_REPO_CHECK` environment variable used to bypass DevShell repository setup validation has been renamed to `PF_SKIP_CHECK_REPO_SETUP`.", "description": "The environment variable was renamed from `PF_SKIP_REPO_CHECK` to `PF_SKIP_CHECK_REPO_SETUP` to align with the `PF_SKIP_CHECK_*` naming convention used by other DevShell validation bypass flags. This improves discoverability and consistency across the configuration surface.", "action_items": [ "Update any environment configurations, CI pipelines, and scripts that set `PF_SKIP_REPO_CHECK=1` to use `PF_SKIP_CHECK_REPO_SETUP=1` instead." ], "references": [ { "type": "internal-commit", "summary": "Rename `PF_SKIP_REPO_CHECK` to `PF_SKIP_CHECK_REPO_SETUP` in `enter-shell-local.sh`", "link": "https://github.com/Panfactum/stack/commit/a9b39c8b808f69cf147c83342fecf49404bad8f3" } ], "impacts": [ { "type": "devshell", "component": "enter-shell-local", "summary": "Renamed `PF_SKIP_REPO_CHECK` to `PF_SKIP_CHECK_REPO_SETUP`" } ] }, { "id": "ccdda923-a0f8-4d56-b9ce-d0830ba8c06f", "type": "improvement", "summary": "`pf cluster add` now fully automates end-to-end cluster installation with fine-grained resumable steps.", "description": "The `pf cluster add` command now automates every phase of cluster installation. Each phase and its\nsub-steps are individually checkpointed, so interrupted installs resume from the exact failing step\nrather than restarting the entire phase.\n\nNew automated phases:\n\n1. **Vault** -- deploys `kube_vault`, runs `vault operator init`, unseals with recovery keys, and deploys `vault_core_resources` via a dynamically-selected port-forward proxy.\n2. **Certificate management** -- deploys `kube_cert_manager` and `kube_cert_issuers` (now `kube_certificates`) with resumable sub-steps for each; prompts for alert email.\n3. **Linkerd** -- deploys `kube_linkerd` and verifies control plane health.\n4. **Autoscaling** -- deploys metrics server, VPA, Karpenter, node pools, scheduler, and KEDA as independently-resumable sub-steps.\n5. **Inbound Networking** -- deploys ExternalDNS, AWS LB Controller, nginx ingress, Vault ingress, and bastion.\n6. **Maintenance controllers** -- deploys Reloader, node image caches, PVC Autoresizer, Descheduler, External Snapshotter, and Velero.\n7. **CloudNativePG** -- final installation step.\n\nAdditional improvements: TLS DH parameters now use `-dsaparam` (seconds instead of minutes), VPC name uniqueness is validated before provisioning, EKS cluster names are auto-formatted to lowercase with hyphens, and AWS sessions are validated before `terragrunt output` calls.\n", "references": [ { "type": "internal-commit", "summary": "CLI improvements (#330)", "link": "https://github.com/Panfactum/stack/commit/733c3c8cf4b9c33e17adc99f6527108c58713144" }, { "type": "internal-commit", "summary": "Vault deployment and initialization automation", "link": "https://github.com/Panfactum/stack/commit/6519c292ef148081c10a2aa089a295d63827af27" }, { "type": "internal-commit", "summary": "Vault core resources automation", "link": "https://github.com/Panfactum/stack/commit/403036045129cbd3a0fe203d9c0d8d4a6d8abdc5" }, { "type": "internal-commit", "summary": "Certificate management automation", "link": "https://github.com/Panfactum/stack/commit/ad33083d8f32e1a143531a845d845b5543dcc5a1" }, { "type": "internal-commit", "summary": "Certificate issuers automation", "link": "https://github.com/Panfactum/stack/commit/2ca29499f6d742ee64fe75d3671674e441291bb1" }, { "type": "internal-commit", "summary": "Linkerd deployment and health verification", "link": "https://github.com/Panfactum/stack/commit/b04b36dad98ca97a3d416543ff95e42811665c13" }, { "type": "internal-commit", "summary": "Autoscaling components automation", "link": "https://github.com/Panfactum/stack/commit/76adbbf33773a0e28308e5bda34960e567ea8a3b" }, { "type": "internal-commit", "summary": "Inbound networking automation", "link": "https://github.com/Panfactum/stack/commit/6bc9823b0a576fb0d56a63478ae39872682cf9fc" }, { "type": "internal-commit", "summary": "Maintenance controllers and CloudNativePG automation", "link": "https://github.com/Panfactum/stack/commit/91fb52517c3c24210dee2bd96ee8e89d28293f01" }, { "type": "internal-commit", "summary": "AWS profile handling and VPC/EKS setup refinements (#339)", "link": "https://github.com/Panfactum/stack/commit/d966e41f762d8e82e3b088a66bddd8cabb29ff93" }, { "type": "external-commit", "summary": "CLI improvements: modular setup and checkpointing (PR #330)", "link": "https://github.com/Panfactum/stack/pull/330" }, { "type": "external-commit", "summary": "CLI updates: AWS profile handling and setup refinements (PR #339)", "link": "https://github.com/Panfactum/stack/pull/339" } ], "impacts": [ { "type": "cli", "component": "cluster add", "summary": "Automates Vault, certificate management, Linkerd, autoscaling, inbound networking, maintenance controllers, and CloudNativePG with fine-grained checkpointing for step-level resumability" }, { "type": "iac-module", "component": "kube_vault", "summary": "Deployed and initialized automatically including `vault operator init` and unsealing" }, { "type": "iac-module", "component": "vault_core_resources", "summary": "Deployed automatically after Vault initialization via port-forward proxy" }, { "type": "iac-module", "component": "kube_cert_manager", "summary": "Deployed automatically as part of the certificate management phase" }, { "type": "iac-module", "component": "kube_cert_issuers", "summary": "Deployed automatically with delegated zone configuration and alert email prompts" }, { "type": "iac-module", "component": "kube_linkerd", "summary": "Deployed automatically with post-deployment control plane health verification" }, { "type": "iac-module", "component": "kube_metrics_server", "summary": "Deployed automatically as the first autoscaling sub-step" }, { "type": "iac-module", "component": "kube_vpa", "summary": "Deployed automatically; triggers resource reconciliation for all prior deployments" }, { "type": "iac-module", "component": "kube_karpenter", "summary": "Deployed automatically to enable cluster autoscaling" }, { "type": "iac-module", "component": "kube_karpenter_node_pools", "summary": "Deployed automatically to configure Karpenter provisioning rules" }, { "type": "iac-module", "component": "kube_scheduler", "summary": "Deployed automatically as part of autoscaling setup" }, { "type": "iac-module", "component": "kube_keda", "summary": "Deployed automatically to enable event-driven autoscaling" }, { "type": "iac-module", "component": "kube_external_dns", "summary": "Deployed automatically as the first inbound networking sub-step" }, { "type": "iac-module", "component": "kube_aws_lb_controller", "summary": "Deployed automatically to manage AWS load balancer resources" }, { "type": "iac-module", "component": "kube_ingress_nginx", "summary": "Deployed automatically to handle cluster ingress traffic" }, { "type": "iac-module", "component": "kube_bastion", "summary": "Deployed automatically to enable secure SSH access to the cluster" }, { "type": "iac-module", "component": "kube_reloader", "summary": "Deployed automatically to restart workloads on config/secret changes" }, { "type": "iac-module", "component": "kube_node_image_cache", "summary": "Deployed automatically to pre-cache container images on nodes" }, { "type": "iac-module", "component": "kube_pvc_autoresizer", "summary": "Deployed automatically to resize persistent volumes as needed" }, { "type": "iac-module", "component": "kube_descheduler", "summary": "Deployed automatically to rebalance pod scheduling across nodes" }, { "type": "iac-module", "component": "kube_external_snapshotter", "summary": "Deployed automatically to enable volume snapshot support" }, { "type": "iac-module", "component": "kube_velero", "summary": "Deployed automatically to enable cluster backup and disaster recovery" }, { "type": "iac-module", "component": "kube_cloudnative_pg", "summary": "Deployed automatically as the final installation step" } ] }, { "id": "2f4c36bb-ce00-406b-b92d-ec1d78f3bc0a", "type": "breaking_change", "summary": "`pf install-cluster` has been renamed to `pf cluster add`. Update all scripts referencing the old command.", "description": "The CLI is moving to a hierarchical subcommand structure (`pf `) for consistency and discoverability. This rename aligns the cluster installation command with the new pattern used by `pf env add`, `pf domain add`, and other recently added commands.", "action_items": [ "Replace all invocations of `pf install-cluster` with `pf cluster add` in your scripts, CI pipelines, and runbooks." ], "references": [ { "type": "internal-commit", "summary": "refactor: rename install-cluster to cluster install", "link": "https://github.com/Panfactum/stack/commit/8c03b01043302de2bced3649ebf6e7cc112be3b1" }, { "type": "internal-commit", "summary": "chore: rename 'cluster install' to 'cluster add'", "link": "https://github.com/Panfactum/stack/commit/30521a81a5640d770e5e6e1811c76d2eb5de715b" } ], "impacts": [ { "type": "cli", "component": "cluster add", "summary": "Renamed from `pf install-cluster`; update scripts and documentation accordingly" } ] }, { "id": "adf75342-3656-40d4-b6f9-c525e4a270de", "type": "addition", "summary": "Adds the `pf config get` CLI command, which merges all applicable Panfactum configuration files and outputs the result as JSON.", "description": "This command is useful for debugging configuration issues: it shows exactly which settings are active by merging all configuration sources in precedence order. It also includes computed directory values such as `environment_dir`, `region_dir`, and `module_dir` that are inferred from the directory structure. A `--directory` flag lets you inspect the configuration for any path in the repository without changing your working directory.", "references": [ { "type": "internal-commit", "summary": "chore: cli refactor", "link": "https://github.com/Panfactum/stack/commit/6a380d2664c6020bf7fe5e60d7df289fb1b3486a" }, { "type": "internal-docs", "summary": "Terragrunt variables and configuration file hierarchy", "link": "/docs/main/reference/configuration/terragrunt-variables" } ], "impacts": [ { "type": "cli", "component": "config get", "summary": "New command to retrieve and display the merged Panfactum configuration for a directory" } ] }, { "id": "3e7445b7-8e6c-462e-a28c-a311e5c044fe", "type": "addition", "summary": "Adds `pf aws vpc-network-test` CLI command for validating VPC network connectivity after deploying `aws_vpc`.", "description": "The `pf aws vpc-network-test` command is now a first-class `pf` CLI subcommand, making it easier for operators to validate VPC egress network configuration after deploying or modifying an `aws_vpc` module. Previously this functionality was only available as the standalone `pf-vpc-network-test` script.\n\nThis change also introduces a structured `Logger` class and `PanfactumCommand` base class to the CLI internals, producing consistently formatted output across all commands. Error output now includes the failed command, working directory, and full subprocess logs, making it significantly easier to diagnose failures.", "references": [ { "type": "internal-commit", "summary": "feat: cli refactor", "link": "https://github.com/Panfactum/stack/commit/31fd6940342ea29532080d6ff2a3a835bbb55995" }, { "type": "internal-docs", "summary": "`aws_vpc` module reference", "link": "/docs/main/reference/infrastructure-modules/aws/aws_vpc" }, { "type": "internal-docs", "summary": "AWS networking bootstrapping guide", "link": "/docs/main/guides/bootstrapping/aws-networking" } ], "impacts": [ { "type": "cli", "component": "aws vpc-network-test", "summary": "New first-class `pf` CLI subcommand for testing VPC egress network connectivity" } ] }, { "id": "fb3e130d-05a7-4242-9c1b-f7e891eaff6d", "type": "fix", "summary": "Fixes `pf cluster add` where a missing return statement in `setSLA` caused the selected SLA target to be silently discarded.", "description": "The `setSLA` function in `pf cluster add` was missing a `return` statement, so the user's selected SLA target was silently dropped. The cluster installation proceeded with the default SLA value instead of the one the user chose. This also required fixing the function's return type annotation to `Promise>` so TypeScript could catch similar issues in the future.", "references": [ { "type": "internal-commit", "summary": "Fix missing `return` in `setSLA` step", "link": "https://github.com/Panfactum/stack/commit/5bf5678581290b8c1b786f8756db00b664c62c7f" }, { "type": "internal-docs", "summary": "`sla_target` Terragrunt variable reference", "link": "/docs/main/reference/configuration/terragrunt-variables" } ], "impacts": [ { "type": "cli", "component": "cluster add", "summary": "Selected SLA target was silently discarded; cluster installed with default value instead" } ] }, { "id": "5b4df196-32a5-4b78-8e8c-39dd823dd8c0", "type": "fix", "summary": "Fixes a compilation error in `pf devshell sync` caused by references to removed internal utility functions.", "description": "The `buildKubeConfig` and `buildSSHConfig` modules inside `pf devshell sync` imported two internal utilities (`safeFileExists` and `createNullWriter`) that had been removed from the codebase. This caused the CLI to fail at compile time, preventing the command from running entirely. The fix comments out the call sites that depended on these utilities so the command compiles and runs correctly again.", "references": [ { "type": "internal-commit", "summary": "Comment out removed utility references to fix compilation", "link": "https://github.com/Panfactum/stack/commit/738389f5a03f39078b5cfcce211254397cd83465" } ], "impacts": [ { "type": "cli", "component": "devshell sync", "summary": "Compilation error resolved; command runs correctly again" } ] }, { "id": "a38b174c-70f0-465f-a8d1-489b5af859c3", "type": "fix", "summary": "Fixes `pf cluster add` VPC setup step where the AWS CLI could hang waiting for user input from a pager.", "description": "During the VPC setup step of `pf cluster add`, the `aws ec2 describe-vpcs` call used to validate VPC name uniqueness could open an interactive pager (e.g., `less`) when the output exceeded the terminal height. In non-interactive or automated contexts this caused the command to hang indefinitely, blocking the entire cluster installation flow. The `--no-cli-pager` flag is now passed to the AWS CLI call, ensuring output is always written directly to stdout.", "references": [ { "type": "internal-commit", "summary": "Add `--no-cli-pager` flag to VPC name validation and add debug logging", "link": "https://github.com/Panfactum/stack/commit/9b8ead4095a2fb15c4adc4f55fd21ba2e8de8d9d" }, { "type": "external-docs", "summary": "AWS CLI pagination options and `--no-cli-pager` flag", "link": "https://docs.aws.amazon.com/cli/latest/userguide/cli-usage-pagination.html" } ], "impacts": [ { "type": "cli", "component": "cluster add", "summary": "VPC name uniqueness check no longer hangs on the AWS CLI pager" } ] }, { "id": "802ae499-4800-41f5-b251-b2cb609fb70d", "type": "addition", "summary": "Adds the `kube_opensearch` module and `opensearch-cli` DevShell tool for deploying and managing OpenSearch clusters on Kubernetes.", "description": "Two additions bring first-class OpenSearch support to the Panfactum Stack:\n\n1. The `kube_opensearch` module provisions a production-grade OpenSearch cluster with mutual TLS via Vault-backed certificate infrastructure, role-based access control (superuser, admin, reader), automated S3 snapshot backups, segment replication with remote store, and Vault-managed dynamic credentials that rotate automatically.\n2. The `opensearch-cli` tool is added to the Panfactum DevShell for interacting with OpenSearch clusters from the command line.\n\nThe module also supports an optional OpenSearch Dashboards deployment with ingress, VPA autoscaling, and PDB-managed disruptions.\n", "references": [ { "type": "internal-commit", "summary": "feat: alpha opensearch module", "link": "https://github.com/Panfactum/stack/commit/dafb4d5782ba675eb6a0c4eff407ea59ab6f815e" }, { "type": "external-docs", "summary": "OpenSearch official documentation", "link": "https://opensearch.org/docs" }, { "type": "internal-docs", "summary": "`kube_opensearch` module overview", "link": "/docs/main/modules/kube_opensearch/overview" } ], "impacts": [ { "type": "iac-module", "component": "kube_opensearch", "summary": "New module deploying a secured OpenSearch cluster with mTLS authentication, RBAC, S3 remote storage, segment replication, and optional Dashboards UI" }, { "type": "devshell", "component": "opensearch-cli", "summary": "New CLI tool for interacting with OpenSearch clusters from the DevShell" } ] }, { "id": "7e185f6c-060c-4c31-aec6-59e33f4a8c91", "type": "breaking_change", "summary": "`pull_through_cache_enabled` now defaults to `true` for several workload and data modules.", "description": "ECR pull-through caching reduces image pull latency and avoids upstream rate limits by routing container image pulls through a regional ECR cache. Enabling it by default ensures new deployments benefit automatically, but environments without an ECR pull-through cache configured will experience image pull failures on these modules.", "action_items": [ "If ECR pull-through caching is not configured in your environment, set `pull_through_cache_enabled = false` on `kube_nats`, `kube_pg_cluster`, `kube_redis_sentinel`, `kube_stateful_set`, and `kube_deployment` modules.", "Alternatively, deploy the `aws_ecr_pull_through_cache` module to enable pull-through caching before re-applying these modules." ], "references": [ { "type": "internal-commit", "summary": "Change `pull_through_cache_enabled` default from `false` to `true` across workload modules", "link": "https://github.com/Panfactum/stack/commit/dafb4d5782ba675eb6a0c4eff407ea59ab6f815e" }, { "type": "external-docs", "summary": "AWS ECR pull-through cache documentation", "link": "https://docs.aws.amazon.com/AmazonECR/latest/userguide/pull-through-cache-creating-rule.html" } ], "impacts": [ { "type": "iac-module", "component": "kube_nats", "summary": "`pull_through_cache_enabled` now defaults to `true`" }, { "type": "iac-module", "component": "kube_pg_cluster", "summary": "`pull_through_cache_enabled` now defaults to `true`" }, { "type": "iac-module", "component": "kube_redis_sentinel", "summary": "`pull_through_cache_enabled` now defaults to `true`" }, { "type": "iac-module", "component": "kube_stateful_set", "summary": "`pull_through_cache_enabled` now defaults to `true`" }, { "type": "iac-module", "component": "kube_deployment", "summary": "`pull_through_cache_enabled` now defaults to `true`" } ] }, { "id": "41f23dbe-3077-458e-883d-c44b1d31ddfb", "type": "breaking_change", "summary": "The `update_type` variable has been removed from the `kube_cron_job` module.", "description": "CronJobs do not use rolling update strategies, so the `update_type` variable was unnecessary and has been removed. Any Terraform configuration that explicitly sets this variable will now produce a validation error.", "action_items": [ "Remove any `update_type` input from your `kube_cron_job` module configurations.", "Run `terragrunt apply` on affected modules to confirm the configuration is valid after removal." ], "references": [ { "type": "internal-commit", "summary": "Remove `update_type` variable from `kube_cron_job`", "link": "https://github.com/Panfactum/stack/commit/dafb4d5782ba675eb6a0c4eff407ea59ab6f815e" }, { "type": "internal-docs", "summary": "`kube_cron_job` module reference", "link": "/docs/main/reference/infrastructure-modules/submodule/kubernetes/kube_cron_job" } ], "impacts": [ { "type": "iac-module", "component": "kube_cron_job", "summary": "Removed the `update_type` input variable" } ] }, { "id": "3e6859b1-5a32-42f6-889b-97e1166ffb94", "type": "fix", "summary": "Fixes multiple `pf` CLI bugs affecting checkpoint resumption, SOPS directory creation, spinner cleanup, and subprocess hangs.", "description": "Five reliability fixes for the `pf` CLI, primarily affecting `pf cluster add` and `pf aws vpc-network-test`. These issues caused unnecessary re-work on resumed installations, hard failures on first-time setups, and intermittent hangs during subprocess execution.\n\n1. The `pf cluster add` VPC setup checkpoint was tracked under the wrong ID (`kyvernoIaCSetup` instead of `setupVPCIaC`), causing resumed installs to re-run the VPC deployment unnecessarily.\n2. The vault secrets step in `pf cluster add` failed if the target SOPS directory did not exist; the CLI now auto-creates the full directory path before writing.\n3. `pf aws vpc-network-test` progress spinners were left open on error, cluttering terminal output. They now close correctly in all code paths.\n4. SSM command polling retries in `pf aws vpc-network-test` increased from 20 to 60 to reduce spurious timeouts on slow instances.\n5. CLI subprocess execution could hang because `stdout`/`stderr` streams were consumed sequentially and line-callback promises were not awaited. Streams are now read concurrently via `Promise.all`.\n", "references": [ { "type": "internal-commit", "summary": "fix: cli updates", "link": "https://github.com/Panfactum/stack/commit/8d47487d857f75f57f6b1265beb51266e41c56f8" }, { "type": "internal-docs", "summary": "VPC networking bootstrapping guide", "link": "/docs/main/guides/bootstrapping/aws-networking" } ], "impacts": [ { "type": "cli", "component": "cluster add", "summary": "VPC setup checkpoint now uses the correct ID (`setupVPCIaC`); vault secrets step auto-creates missing SOPS directories" }, { "type": "cli", "component": "aws vpc-network-test", "summary": "Progress spinners now close correctly on error; SSM command polling retries increased from 20 to 60" } ] }, { "id": "256ec569-15b5-43bd-8d03-ed1bf8bdf6a8", "type": "fix", "summary": "Fixes `pf cluster add` SLA target prompt repeating on resume by persisting the selection via the checkpointer.", "description": "Previously, resuming an interrupted `pf cluster add` session would re-display the SLA target selection prompt even though the user had already chosen a value. This happened because the SLA confirmation ran before the checkpointer was initialized, so the selection was never saved. This fix moves SLA confirmation after checkpointer initialization and adds `slaTarget` to the checkpointer's saved-inputs schema. On resume, the previously selected SLA target is loaded from the checkpoint file and the prompt is skipped entirely.", "references": [ { "type": "internal-commit", "summary": "fix: only ask for SLA confirmation once", "link": "https://github.com/Panfactum/stack/commit/eb2069598796850a9a9eb985614e69c6124d5525" } ], "impacts": [ { "type": "cli", "component": "cluster add", "summary": "SLA target prompt shown only once per session; selection persisted via checkpointer" } ] }, { "id": "4dcea900-4664-449b-b1ee-7a14b8259c1a", "type": "addition", "summary": "Adds the `aws_organization` IaC module and `pf env add` CLI command for automating new Panfactum environment provisioning.", "description": "Two related additions that together automate environment provisioning:\n\n1. The `aws_organization` module manages your entire AWS Organization as infrastructure as code — creating member accounts, configuring IAM Organizations features (root credentials management, root sessions), and setting per-account contact information.\n2. The `pf env add` CLI command provides a guided wizard that detects whether an AWS Organization exists, provisions a new AWS account, bootstraps the Terraform state bucket and lock table, and optionally connects the environment to AWS SSO, significantly reducing manual steps required to add a new environment.\n", "references": [ { "type": "internal-commit", "summary": "Add `aws_organization` module and `pf env add` CLI command", "link": "https://github.com/Panfactum/stack/commit/265d0a3fa27c15364fd814fd44d66aac4396bb88" }, { "type": "internal-docs", "summary": "`aws_organization` module documentation", "link": "/docs/main/reference/infrastructure-modules/direct/aws/aws_organization" } ], "impacts": [ { "type": "iac-module", "component": "aws_organization", "summary": "New module for managing AWS Organization structure and member accounts" }, { "type": "cli", "component": "env add", "summary": "New guided wizard for installing a new Panfactum environment end-to-end" } ] }, { "id": "921dbe39-4f2b-4838-9380-c5e85a0e496f", "type": "breaking_change", "summary": "The `aws.secondary` provider alias and cross-region DynamoDB replica have been removed from `tf_bootstrap_resources`.", "description": "The `tf_bootstrap_resources` module previously required an `aws.secondary` provider alias and used it to create a cross-region DynamoDB Global Table replica of the Terraform state lock table. This cross-region replica added operational complexity without providing meaningful benefit, since the state lock table is only accessed from a single region during normal Terraform operations.\n\nThis change simplifies the module by:\n\n1. Removing the `configuration_aliases` entry for `aws.secondary` from the provider block\n2. Removing the `aws_region.secondary` and `pf_aws_tags.seondary_tags` data sources\n3. Removing the DynamoDB `replica` block that created the cross-region table copy\n\nUsers who have the secondary provider alias configured in their Terragrunt `generate` or `provider` blocks must remove it before applying, or Terraform will fail with an unknown provider error.\n", "action_items": [ "Remove the `aws.secondary` provider alias from your `tf_bootstrap_resources` Terragrunt configuration if present.", "Run `terragrunt apply` on `tf_bootstrap_resources` to remove the cross-region DynamoDB replica from your state lock table." ], "references": [ { "type": "internal-commit", "summary": "feat: opensearch alpha state", "link": "https://github.com/Panfactum/stack/commit/265d0a3fa27c15364fd814fd44d66aac4396bb88" }, { "type": "internal-docs", "summary": "`tf_bootstrap_resources` module reference", "link": "/docs/main/reference/infrastructure-modules/direct/aws/tf_bootstrap_resources" } ], "impacts": [ { "type": "iac-module", "component": "tf_bootstrap_resources", "summary": "Removed the `aws.secondary` provider `configuration_aliases` entry, the `aws_region.secondary` data source, the `pf_aws_tags.seondary_tags` data source, and the DynamoDB `replica` block for cross-region replication." } ] }, { "id": "b66747b4-6e58-44ed-b722-f2b278b79420", "type": "update", "summary": "Updates `kube_logging` to Loki 6.29.0 with tuned caching and query parallelism, and drops high-cardinality labels in `kube_alloy`.", "description": "Three observability and infrastructure improvements bundled in the same commit:\n\n1. **`kube_logging`** -- Loki Helm chart upgraded from 6.6.2 to 6.29.0. Default log level changed from `warn` to `info`. Redis cache TTLs extended from 1 h to 12 h across all cache layers (chunks, results, index stats, volume, instant metrics, series, labels, and index queries). Query parallelism capped at 4, ingestion rate raised to 50 MB/s with 100 MB burst, compaction interval widened from 20 m to 75 m, and TSDB shipper resync interval set to 60 m. Redis connection pool sizes increased to 100 with explicit timeouts and connection age limits.\n2. **`kube_node_settings`** -- `max-locked-memory` set to `unlimited` in node OCI defaults, required for OpenSearch and other memory-intensive workloads that use `mlock` to prevent critical data from being swapped to disk.\n3. **`kube_alloy`** -- Alloy log pipeline now drops the `filename`, `job`, and `pod` labels from log streams before forwarding to Loki. These labels generated excessive cardinality, inflating storage costs and slowing queries without providing proportional value.\n", "references": [ { "type": "internal-commit", "summary": "feat: opensearch alpha state", "link": "https://github.com/Panfactum/stack/commit/265d0a3fa27c15364fd814fd44d66aac4396bb88" }, { "type": "external-docs", "summary": "Loki Helm chart 6.29.0 on Artifact Hub", "link": "https://artifacthub.io/packages/helm/grafana/loki/6.29.0" }, { "type": "external-docs", "summary": "Grafana Loki GitHub repository", "link": "https://github.com/grafana/loki" }, { "type": "internal-docs", "summary": "`kube_logging` module reference", "link": "/docs/main/reference/infrastructure-modules/direct/kubernetes/kube_logging" }, { "type": "internal-docs", "summary": "`kube_alloy` module reference", "link": "/docs/main/reference/infrastructure-modules/direct/kubernetes/kube_alloy" } ], "impacts": [ { "type": "iac-module", "component": "kube_logging", "summary": "Loki chart updated from 6.6.2 to 6.29.0; default `log_level` changed to `info`; Redis cache TTLs extended to 12 h; query parallelism and compaction interval tuned" }, { "type": "iac-module", "component": "kube_node_settings", "summary": "`max-locked-memory` OCI resource limit set to `unlimited` for OpenSearch and other `mlock` workloads" }, { "type": "iac-module", "component": "kube_alloy", "summary": "High-cardinality labels (`filename`, `job`, `pod`) dropped from log streams before forwarding to Loki" } ] }, { "id": "aa50121c-eb86-42fb-9847-10f4a6bc3764", "type": "fix", "summary": "Fixes S3 snapshot connectivity in `kube_opensearch` by setting `s3.client.default.endpoint` to the regional endpoint, preventing 307 redirects and auth failures.", "description": "Without an explicit `s3.client.default.endpoint`, the OpenSearch S3 repository plugin defaults to the global `s3.amazonaws.com` endpoint, which can return HTTP 307 redirects or authorization-header mismatches for buckets outside `us-east-1`. This caused snapshot registration and restore operations to fail silently under IRSA-based authentication. Setting the endpoint to `s3..amazonaws.com` ensures requests are routed directly to the correct regional endpoint, avoiding both redirect loops and credential-scope errors.", "references": [ { "type": "internal-commit", "summary": "fix: set `s3.client.default.endpoint` for `kube_opensearch` S3 repository plugin", "link": "https://github.com/Panfactum/stack/commit/997834fa45e43fdb304c4597dd15824660695c97" }, { "type": "issue-report", "summary": "OpenSearch S3 snapshot repository fails to select correct AWS region", "link": "https://github.com/opensearch-project/OpenSearch/issues/9265" }, { "type": "internal-docs", "summary": "`kube_opensearch` module reference", "link": "/docs/main/reference/infrastructure-modules/submodule/kubernetes/kube_opensearch" } ], "impacts": [ { "type": "iac-module", "component": "kube_opensearch", "summary": "Adds `s3.client.default.endpoint` to the OpenSearch configuration, pointing to the regional S3 endpoint so that snapshot operations authenticate and route correctly." } ] }, { "id": "20552955-a5ee-4d9c-be5a-a6ac82c277a7", "type": "improvement", "summary": "Refactors `pf devshell sync` into reusable Listr task builders and fixes `pf env add` EC2 Spot role import.", "description": "Two related improvements from the same commit:\n\n1. `pf devshell sync` internal sync logic was refactored into composable Listr task builders (`buildSyncKubeClustersTask`, `buildSyncSSHTask`, `buildSyncAWSIdentityCenterTask`) shared across `pf devshell sync`, `pf env add`, and future CLI commands. This improves reliability and gives users real-time progress visibility via Listr's interactive task renderer.\n2. `pf env add` now conditionally skips importing the `AWSServiceRoleForEC2Spot` IAM service-linked role when it does not already exist in the account, fixing failures on accounts where spot instances have never been used.\n", "references": [ { "type": "internal-commit", "summary": "Refactor devshell update to use Listr task builders and add AWS Identity Center sync", "link": "https://github.com/Panfactum/stack/commit/7b1f505992598d12b16864c9ec33865c97232d9b" }, { "type": "external-docs", "summary": "Listr2 — terminal task list library used for CLI progress rendering", "link": "https://github.com/listr2/listr2" } ], "impacts": [ { "type": "cli", "component": "devshell sync", "summary": "Internal sync logic refactored into composable Listr task builders with AWS Identity Center sync added" }, { "type": "cli", "component": "env add", "summary": "Conditionally skips `AWSServiceRoleForEC2Spot` import when the role does not exist in the account" } ] }, { "id": "54efecd9-8b68-41d8-876e-c6c1a00d175a", "type": "addition", "summary": "Adds `pf domain add`, a guided wizard for adding a domain to a Panfactum environment.", "description": "The new `pf domain add` CLI command automates the entire domain onboarding workflow that was previously a multi-step manual process. The wizard handles:\n\n1. Purchasing new domains through AWS Route 53\n2. Importing already-owned domains from other registrars\n3. Creating environment-specific subdomains with automatic DNS delegation\n4. Deploying and configuring Route 53 hosted zones\n5. Linking subdomains across environments for security isolation\n\nDomains in Panfactum are scoped to environments so that workloads in one environment cannot alter DNS records belonging to another. The wizard detects whether a domain is already registered, whether it is an apex domain or a subdomain, and whether any ancestor domains are already configured, then guides the user through the appropriate setup path.\n", "references": [ { "type": "internal-commit", "summary": "Add `pf domain add` CLI command for guided domain onboarding", "link": "https://github.com/Panfactum/stack/commit/7b1f505992598d12b16864c9ec33865c97232d9b" }, { "type": "internal-docs", "summary": "DNS bootstrapping guide", "link": "/docs/main/guides/bootstrapping/dns" }, { "type": "internal-docs", "summary": "Subdomain delegation concept", "link": "/docs/main/concepts/networking/subdomain-delegation" } ], "impacts": [ { "type": "cli", "component": "domain add", "summary": "New guided wizard that automates domain registration, DNS zone deployment, and cross-environment subdomain delegation" } ] }, { "id": "978d75b9-1eb2-470c-b391-c5b158ddfa65", "type": "improvement", "summary": "`pf config get` now returns `environment_dir`, `region_dir`, and `module_dir` derived fields.", "description": "Directory names in the Panfactum environments tree can differ from the logical environment, region, and module names set in configuration files. Previously, `pf config get` only exposed the logical names, which made it difficult for scripts and CLI commands to resolve the actual filesystem paths. The three new derived fields — `environment_dir`, `region_dir`, and `module_dir` — provide the real directory names extracted from the current working directory's position in the environments tree, enabling reliable path construction without manual parsing.", "references": [ { "type": "internal-commit", "summary": "Add `environment_dir`, `region_dir`, and `module_dir` derived fields to `pf config get`", "link": "https://github.com/Panfactum/stack/commit/7b1f505992598d12b16864c9ec33865c97232d9b" } ], "impacts": [ { "type": "cli", "component": "config get", "summary": "Now returns `environment_dir`, `region_dir`, and `module_dir` derived fields" } ] }, { "id": "ff6dfb1b-d766-475d-94cc-5a7cf4594529", "type": "improvement", "summary": "`aws_account` now automatically requests CloudFront service quota increases for origin request, response header, and cache policies.", "description": "Previously, CloudFront quota increase resources existed as commented-out code in `aws_organization` using `aws_servicequotas_template`. This change moves them into `aws_account` as active `aws_servicequotas_service_quota` resources, each raising the limit to 100. This prevents quota exhaustion errors when managing many CloudFront distributions across an account.", "references": [ { "type": "internal-commit", "summary": "Add CloudFront service quota increase resources to `aws_account`", "link": "https://github.com/Panfactum/stack/commit/7b1f505992598d12b16864c9ec33865c97232d9b" }, { "type": "internal-docs", "summary": "`aws_account` module reference documentation", "link": "/docs/main/reference/infrastructure-modules/direct/aws/aws_account" }, { "type": "external-docs", "summary": "Amazon CloudFront quotas", "link": "https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/cloudfront-limits.html" } ], "impacts": [ { "type": "iac-module", "component": "aws_account", "summary": "Adds `aws_servicequotas_service_quota` resources to raise CloudFront origin request policy, response header policy, and cache policy quotas to 100" } ] }, { "id": "809a061f-5602-45f2-8ccc-a143b0d2ca60", "type": "fix", "summary": "Fixes multiple `pf cluster add` bugs — Vault secret paths, pod readiness scoping, EKS API server URL, and EC2 termination.", "description": "Five fixes discovered during end-to-end cluster installation testing:\n\n1. The inbound networking and Linkerd setup steps were reading Vault `secrets.yaml` from their own module directories instead of the `kube_vault` module directory, causing resumability failures when the secret file did not exist at the expected path.\n2. The Vault pod readiness check was querying pods across all namespaces instead of scoping to the `vault` namespace, and the retry limit has been increased from 20 to 60 to allow more time for pods to become ready.\n3. The EKS setup step was writing `kube_api_server` as an empty string instead of reading the cluster URL from `terragrunt output`, breaking downstream steps that depend on the API server address.\n4. The cluster reset step split EC2 instance IDs only on newlines and checked the wrong variable length, silently skipping instance termination. The split now uses a whitespace regex and checks the raw string length.\n5. `pf cluster add` now calls `killAllBackgroundProcesses` when any setup step throws an error, ensuring long-lived processes like the Vault proxy are cleaned up on failure.\n", "references": [ { "type": "internal-commit", "summary": "fix(cli): cluster install fixes and improvements", "link": "https://github.com/Panfactum/stack/commit/5de4d4dd0d508552d06d46cf763af56302d80721" } ], "impacts": [ { "type": "cli", "component": "cluster add", "summary": "Corrects Vault `secrets.yaml` path for ingress and Linkerd steps, scopes Vault pod readiness check to the `vault` namespace with 60 retries, writes `kube_api_server` from module output, fixes EC2 instance termination to split on all whitespace, and kills background processes on step failure" } ] }, { "id": "c25f4353-e3c7-488e-8d62-b3b15e1f42f5", "type": "improvement", "summary": "`pf cluster add` now streams live `linkerd check` output in a rolling five-line window during the Linkerd control plane check.", "description": "Previously, subprocess output was only surfaced to the Listr task renderer once after the subprocess finished. This made long-running checks like `linkerd check` appear to stall with no feedback. The `execute` utility now accepts an optional `task` parameter and updates `task.output` on each line received from stdout/stderr, giving every caller the ability to stream live output. The Linkerd check step uses `rendererOptions.outputBar: 5` to display the last five lines in a scrolling window.", "references": [ { "type": "internal-commit", "summary": "fix(cluster install): stream linkerd check output line by line", "link": "https://github.com/Panfactum/stack/commit/989fcf567fb8e3868435b678dd76609ee40b4fff" } ], "impacts": [ { "type": "cli", "component": "cluster add", "summary": "The Linkerd control plane check step now streams live `linkerd check` output to the terminal in a rolling five-line window instead of showing no output until the check completes" } ] }, { "id": "3ec606b8-32d6-4c70-b92d-8f0d75940c64", "type": "fix", "summary": "Fixes `pf cluster add` Vault domain lookup, hardcoded Terragrunt template values, and snapshotter/Velero deploy ordering.", "description": "Three `pf cluster add` fixes:\n\n1. The inbound networking step was reading the Vault domain from `vault_addr` at the YAML top level, but it is stored under `extra_inputs.vault_domain`, causing DNS propagation checks to use an undefined value.\n2. The `kube_aws_lb_controller` Terragrunt template hardcoded `subnets = [\"PUBLIC_A\", \"PUBLIC_B\"]` and the `kube_ingress_nginx` template hardcoded `ingress_domains` and `sla_level`, causing failures on real clusters. These values are now omitted or set programmatically.\n3. `kube_external_snapshotter` and `kube_velero` were deployed after `kube_descheduler`. They are now deployed together as a concurrent sub-group earlier in the maintenance controllers sequence.\n", "references": [ { "type": "internal-commit", "summary": "fix(cli): inbound networking and support services fixes", "link": "https://github.com/Panfactum/stack/commit/5440599fd7a771ef230de32ee6557731d7a14338" } ], "impacts": [ { "type": "cli", "component": "cluster add", "summary": "Vault domain now read from `extra_inputs.vault_domain`; hardcoded `subnets`, `ingress_domains`, and `sla_level` removed from LB controller and nginx templates; `kube_external_snapshotter` and `kube_velero` deployed before `kube_descheduler`" } ] }, { "id": "b48bbcc7-5f30-4aa8-9d46-7d4ce8bb9a5d", "type": "fix", "summary": "Fixes `pf env add` account alias bugs — wrong source field, inverted task condition, and static credentials not written correctly.", "description": "Three related `pf env add` fixes:\n\n1. The `alias` input in the `aws_account` module deployment was reading from `ctx.accountId` instead of `ctx.accountName`, causing the AWS account alias to be set to the numeric account ID rather than the human-readable account name.\n2. The \"Set AWS account alias\" task's `enabled` condition was inverted — it ran when the account name was already known and was skipped when it was actually needed. Additionally, `collapseErrors: false` was added to the Listr renderer options so that full error details are visible on task failure.\n3. `addAWSProfileFromStaticCreds` checked whether the config file existed but assumed the credentials file also existed inside the same branch. The config and credentials files are now checked independently, and the function correctly merges new profile entries into existing files or creates new ones on first run.\n", "references": [ { "type": "internal-commit", "summary": "fix(env): use accountName instead of accountId for alias input update", "link": "https://github.com/Panfactum/stack/commit/79b510609863bade8668dad6357fda3280eab11d" }, { "type": "internal-commit", "summary": "fix(env): correct enabled condition and expand error output for bootstrapEnvironment", "link": "https://github.com/Panfactum/stack/commit/322d777893c39e78d00eb8b40e3b872a50e21d7b" }, { "type": "internal-commit", "summary": "fix: aws static creds update cli fx", "link": "https://github.com/Panfactum/stack/commit/c70f947891cc2dc96c6f8c330b8c70fb985f0c7c" } ], "impacts": [ { "type": "cli", "component": "env add", "summary": "Account alias now correctly uses the account name; the alias setup task runs when needed; static credential profiles are written and updated reliably" } ] }, { "id": "9fcd8bee-ad2f-49d4-a1e6-1ce7e7cff2a3", "type": "fix", "summary": "Fixes `pf domain add` where the domain registration request was silently skipped because the submission task had `skip: true` hardcoded.", "description": "The \"Submitting registration request to AWS\" task in `registerDomain` had `skip: true` hardcoded, causing the wizard to complete successfully without ever calling the Route 53 `RegisterDomain` API. Users who ran `pf domain add` would see the wizard finish with no errors, but the domain was never registered. Removing the `skip` flag restores the intended behavior so the registration request is sent to AWS.", "references": [ { "type": "internal-commit", "summary": "Remove `skip: true` from domain registration submission task", "link": "https://github.com/Panfactum/stack/commit/9d9d9b14061f6be41d14551a7d3278d45bd52f17" } ], "impacts": [ { "type": "cli", "component": "domain add", "summary": "Domain registration request is now actually submitted to AWS" } ] }, { "id": "ac7d0142-b7a5-4dd0-9d79-640782b1319c", "type": "fix", "summary": "Fixes the `zones` output of `aws_registered_domains` to expose `zone_id` per domain, unblocking `pf domain add` registration.", "description": "The `zones` output of `aws_registered_domains` was not structured correctly, which prevented `pf domain add` from reading zone metadata after deploying the module. This fix restructures the output to expose `zone_id` per domain, allowing the CLI wizard to proceed through domain registration without errors.", "references": [ { "type": "internal-commit", "summary": "fix: aws_registered_domains output", "link": "https://github.com/Panfactum/stack/commit/4c7728c99d5ec35bbf3a663ac9681b43d5666cc1" }, { "type": "internal-docs", "summary": "`aws_registered_domains` module reference", "link": "/docs/main/reference/infrastructure-modules/direct/aws/aws_registered_domains" } ], "impacts": [ { "type": "iac-module", "component": "aws_registered_domains", "summary": "`zones` output now exposes `zone_id` per domain in the correct structure" }, { "type": "cli", "component": "domain add", "summary": "Can now correctly read zone metadata after `aws_registered_domains` deployment" } ] }, { "id": "cb255949-d8c2-48bb-b1f9-f58d0b139164", "type": "fix", "summary": "Fixes `pf cluster add` `kubectl` commands that were missing the `--context` flag, causing them to operate against the wrong cluster.", "description": "When multiple Kubernetes contexts are present in the `kubeconfig`, `kubectl` commands without an explicit `--context` flag default to whichever context is currently active. During `pf cluster add`, the Vault operator init and unseal steps used bare `kubectl exec` calls, which could silently target the wrong cluster. The `kube_config_context` value is now read from the Panfactum config and passed as `--context` to all `kubectl` invocations during cluster installation.", "references": [ { "type": "internal-commit", "summary": "fix: pass kube context to kubectl commands", "link": "https://github.com/Panfactum/stack/commit/3637fac4ac001ddacf2bd48cc9897ae5543cd173" } ], "impacts": [ { "type": "cli", "component": "cluster add", "summary": "All `kubectl` commands now include the `--context` flag to target the correct cluster" } ] }, { "id": "8ba600f8-89f4-4765-951b-770fbb499c3c", "type": "fix", "summary": "Fixes `kube_certificates` to issue TLS certificates for subdomains of the cluster domain by adding `kube_domain` to the certificate `dnsNames`.", "description": "The ingress TLS certificate in `kube_cert_issuers` only included domains from the `route53_zones` input, omitting the cluster's own `kube_domain`. Internal services running on subdomains of that domain (e.g., Vault, bastion) could not obtain valid TLS certificates because the cert-manager `Certificate` resource did not list `kube_domain` or `*.kube_domain` in its `dnsNames`. This fix adds a new `kube_domain` variable and appends both the bare domain and its wildcard to the certificate's `dnsNames` list, ensuring all cluster utility subdomains receive valid certificates.", "references": [ { "type": "internal-commit", "summary": "fix: provider cert-issuers the ability to create certs for subdomains of the cluster domain", "link": "https://github.com/Panfactum/stack/commit/8aa22456510df9d9cc7a85730a810079e75ed424" }, { "type": "internal-docs", "summary": "`kube_cert_issuers` module documentation", "link": "/docs/main/reference/infrastructure-modules/kubernetes/kube_cert_issuers" }, { "type": "internal-docs", "summary": "`kube_certificates` module documentation", "link": "/docs/main/reference/infrastructure-modules/kubernetes/kube_certificates" } ], "impacts": [ { "type": "iac-module", "component": "kube_cert_issuers", "summary": "Adds the `kube_domain` input variable and appends `kube_domain` and `*.kube_domain` to the ingress TLS certificate `dnsNames`, enabling certificate issuance for cluster utility subdomains." }, { "type": "iac-module", "component": "kube_certificates", "summary": "Inherits the subdomain certificate fix from `kube_cert_issuers`, which was consolidated into this module." } ] }, { "id": "feb40709-17f8-4e6e-a591-76cd9b08a9b9", "type": "improvement", "summary": "`pf cluster add` now streams real-time `terragrunt run-all apply` output in a rolling five-line bar during autoscaling setup.", "description": "Previously, `terragruntApplyAll` forwarded subprocess output directly to the Listr task renderer via a `task` parameter tightly coupled into the `execute` utility. This coupling has been replaced with an `onLogLine` callback pattern so that callers control how log lines are surfaced. The autoscaling step uses this callback to display rolling output, giving users visibility into the apply progress without having to wait for the full command to finish.", "references": [ { "type": "internal-commit", "summary": "feat: log lines out of run-all apply", "link": "https://github.com/Panfactum/stack/commit/96f0de01ef79e15ffb70c5473e0c45070190e1eb" } ], "impacts": [ { "type": "cli", "component": "cluster add", "summary": "Displays real-time `terragrunt run-all apply` output in a rolling five-line bar during autoscaling setup" } ] }, { "id": "f28262fc-f2db-41f7-9b42-c0a7391bdedf", "type": "fix", "summary": "Fixes `panfactum.hcl` for new `pf` CLI interface and removes stale `--terragrunt-global-cache` from `pf config get`.", "description": "Three fixes related to the `panfactum.hcl` Terragrunt configuration and the `pf` CLI:\n\n1. Updated `panfactum.hcl` to invoke `pf` CLI commands with the flags and arguments expected by the new CLI interface, replacing the old `pf-get-repo-variables` shell script with `pf config get`.\n2. Removed `--terragrunt-global-cache` from the `pf config get` call. The global cache caused Terragrunt to reuse the first result for all modules in a `run-all` session, returning stale per-module configuration.\n3. Fixed `CLIError` cause chaining so the original root cause is correctly propagated through multi-level error wrapping.\n", "references": [ { "type": "internal-commit", "summary": "fix: panfactum.hcl for new cli", "link": "https://github.com/Panfactum/stack/commit/778dac2c2e3e7cbb9141666e0eac0afefedabea1" }, { "type": "internal-commit", "summary": "fix: misc cli installer bugfixes", "link": "https://github.com/Panfactum/stack/commit/c71b2b32a1f58286448afdf1419443e0f7c5f9f9" }, { "type": "internal-docs", "summary": "Terragrunt variables configuration reference", "link": "/docs/main/reference/configuration/terragrunt-variables" } ], "impacts": [ { "type": "configuration", "component": "panfactum.hcl", "summary": "Updated CLI invocations and removed `--terragrunt-global-cache` from `pf config get`" }, { "type": "cli", "component": "config get", "summary": "No longer called with `--terragrunt-global-cache`, preventing stale cached results during `run-all` sessions" } ] }, { "id": "8fd16915-1e38-4c66-b974-73427354c243", "type": "fix", "summary": "Fixes six installer and configuration bugs affecting CLI commands, `panfactum.hcl`, and the `aws_organization` module.", "description": "Six fixes from the same commit:\n\n1. `pf env add` contact info was written with the wrong key (`account_access_configuration` instead of `primary_contact`) and mismatched field names, so it was silently discarded.\n2. Optional contact info prompts for Organization Name and Street Address 2 incorrectly required non-empty input.\n3. The IAM Identity Center update step ran unconditionally even when the permissions module was not deployed; it now checks for the module directory first.\n4. `pf devshell sync` did not overwrite existing HCL files, and the `.envrc` update subtask was silently dropped due to a syntax error in the task registration.\n5. `panfactum.hcl` failed when `authentik_url` was absent; the value is now read with `lookup()` and a sentinel fallback.\n6. `aws_organization` had a race condition where `aws_organizations_features` could be enabled before the organization was fully created; an explicit `depends_on` is now in place.\n", "references": [ { "type": "internal-commit", "summary": "fix: misc cli installer fixes", "link": "https://github.com/Panfactum/stack/commit/7be6e52c65c27ddf9af94a0367a56b5d37b93035" } ], "impacts": [ { "type": "cli", "component": "env add", "summary": "Contact info now written with the correct `primary_contact` key and matching field names; optional prompts for Organization Name and Street Address 2 accept empty input; IAM Identity Center step skipped when the `aws_iam_identity_center_permissions` module is not deployed" }, { "type": "cli", "component": "devshell sync", "summary": "HCL files now overwritten on sync; `.envrc` update subtask correctly registered and executed instead of being silently dropped" }, { "type": "configuration", "component": "panfactum.hcl", "summary": "`authentik_url` is now read with `lookup()` and a sentinel fallback so the file no longer fails when the variable is absent" }, { "type": "iac-module", "component": "aws_organization", "summary": "`aws_organizations_features` now depends on `aws_organizations_organization` to prevent a race condition during initial organization creation" } ] }, { "id": "4d244b70-9769-4eb7-8d2a-68af97c54de4", "type": "improvement", "summary": "`pf env add` UX improvements: pre-filled contact info, alias explainer, finish message, and clearer setup instructions.", "description": "Several `pf env add` UX improvements:\n\n1. Contact information prompts pre-fill with existing AWS account data; the new account email prompt auto-suggests a plus-suffixed address.\n2. The account alias prompt includes an explainer that the name must be globally unique.\n3. The management bootstrap now deploys `aws_organization` directly.\n4. A finish message is displayed on successful completion with recommended next steps (`pf domain add` and `pf cluster add`).\n5. Setup instructions now include direct links, interactive confirmation prompts for account/IAM user creation, and guidance for generating access keys.\n6. `OrganizationsClient` and `STSClient` now explicitly use `us-east-1` as required for AWS Organizations/STS global endpoints.\n", "references": [ { "type": "internal-commit", "summary": "fix: env add: misc cli improvements", "link": "https://github.com/Panfactum/stack/commit/e0bedc9006e289a7756917a08a9c34d0400ecae3" }, { "type": "internal-commit", "summary": "feat: adds finish message to 'pf env add'", "link": "https://github.com/Panfactum/stack/commit/3747d0982bbff1e6ac4d2684a417dbd9c1f16d1e" }, { "type": "internal-commit", "summary": "fix: better install instructions + missing region", "link": "https://github.com/Panfactum/stack/commit/21d70937a4329fabf4872039ce14b63bb039dc91" }, { "type": "external-docs", "summary": "AWS IAM access keys documentation", "link": "https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html" }, { "type": "external-docs", "summary": "AWS Organizations documentation", "link": "https://docs.aws.amazon.com/organizations/latest/userguide/orgs_introduction.html" } ], "impacts": [ { "type": "cli", "component": "env add", "summary": "Contact info pre-filled from AWS account; alias prompt explains global uniqueness; management bootstrap deploys `aws_organization`; finish message shown on completion; step-by-step AWS setup instructions with interactive confirmations; `OrganizationsClient` and `STSClient` now explicitly set region to `us-east-1`" } ] }, { "id": "746c467d-2217-4724-80ea-fb6c26086261", "type": "fix", "summary": "Fixes `pf env add` credential validation loop, misplaced `AdministratorAccess` error, and silent exit on unknown errors.", "description": "Several fixes to `pf env add` credential validation:\n\n1. The retry prompt looped without allowing the user to enter different credentials when the IAM user lacked `AdministratorAccess`. The validation was restructured into nested loops supporting both retrying with the same credentials and entering new ones.\n2. Root-user detection was added so that AWS account root credentials are rejected with a clear error directing the user to create a non-root IAM user.\n3. The `AdministratorAccess` error message was emitted after the `catch` block instead of inside the `if (!hasAdminAccess)` branch, causing it to display at the wrong time. It is now shown immediately when the policy check fails.\n4. A `break` on unknown errors caused the outer credentials loop to exit without re-prompting. Unknown errors now display a diagnostic message and loop back to the retry prompt.\n5. The suggested IAM username during account setup now defaults to `-superuser` (e.g., `production-superuser`) instead of the hardcoded `pf-bootstrap-user`.\n", "references": [ { "type": "internal-commit", "summary": "fix: AdministratorAccess credential validation loop", "link": "https://github.com/Panfactum/stack/commit/1c7d2dd454ef5fa216abe3fabb12aa1bcdf76bfb" }, { "type": "internal-commit", "summary": "fix: env add default IAM username", "link": "https://github.com/Panfactum/stack/commit/0886f756c32dabcfb9ee647b991a3cb4e247d225" } ], "impacts": [ { "type": "cli", "component": "env add", "summary": "Credential validation loop restructured into nested loops; `AdministratorAccess` error shown in the correct branch; unknown errors re-prompt instead of exiting; root account credentials rejected with a clear error; suggested IAM username now defaults to `-superuser`" } ] }, { "id": "9c2f8cad-d3d3-45da-9648-59e23efc0bb1", "type": "fix", "summary": "Fixes `pf domain add` to correctly skip environment subzone setup when no eligible environments exist.", "description": "A missing `return` statement in `createEnvironmentSubzones` caused execution to fall through to the subzone configuration prompts even when no candidate environments were available. This could lead to incorrect DNS delegations or confusing prompts. The fix adds the missing early return so the function exits cleanly after logging the skip message.", "references": [ { "type": "internal-commit", "summary": "fix: properly skip subzone setup if applicable", "link": "https://github.com/Panfactum/stack/commit/6f47d8e6ee2cd4f2d409ccaf189293f919cdb027" } ], "impacts": [ { "type": "cli", "component": "domain add", "summary": "Subzone setup now correctly exits early when no eligible environments are present" } ] }, { "id": "5aabfcd4-fe8d-4741-b16e-8be114e96db4", "type": "fix", "summary": "Fixes `pf cluster add` `setSLA` step to persist the selected SLA target in the correct numeric format to `region.yaml`.", "description": "The `setSLA` step was migrated from raw `@inquirer/prompts` calls to the unified `context.logger` prompt API, and the domain selection flow was refactored to separate ancestor domain selection from subdomain input. During this migration the `sla_target` value written to `region.yaml` could be formatted incorrectly, causing downstream modules that depend on a numeric `sla_target` (1, 2, or 3) to receive unexpected input. This fix ensures the confirmed SLA level is persisted as a proper numeric value.", "references": [ { "type": "internal-commit", "summary": "fix: set sla formatting for cluster install", "link": "https://github.com/Panfactum/stack/commit/d8976a0552a5a9f6e4d81b58269c96b219dcb392" }, { "type": "internal-docs", "summary": "`sla_target` Terragrunt variable reference", "link": "/docs/main/reference/configuration/terragrunt-variables" } ], "impacts": [ { "type": "cli", "component": "cluster add", "summary": "The `setSLA` step now persists the selected SLA target as a numeric value in `region.yaml`, preventing formatting mismatches consumed by downstream IaC modules." } ] }, { "id": "e61e18ad-b43f-4555-8229-63d3af536ee2", "type": "fix", "summary": "Fixes `pf env add` to import an existing AWS organization instead of failing to create a duplicate.", "description": "When running `pf env add` against an AWS account that already had an AWS Organization, the `aws_organization` module deployment would attempt to create a new organization and fail because one already existed. The fix adds a `shouldImport` check that calls `DescribeOrganization` via the AWS SDK. If an organization is found, its ID is used to import the existing resource into Terraform state instead of creating a duplicate, allowing management environment setup to complete successfully.", "references": [ { "type": "internal-commit", "summary": "fix: import existing org in env add command", "link": "https://github.com/Panfactum/stack/commit/0a0e3ca4e78569312fc7bb4763c120a92674bb5d" }, { "type": "internal-docs", "summary": "`aws_organization` module reference", "link": "/docs/main/reference/infrastructure-modules/direct/aws/aws_organization" } ], "impacts": [ { "type": "cli", "component": "env add", "summary": "Correctly imports an existing AWS organization instead of failing when one already exists" }, { "type": "iac-module", "component": "aws_organization", "summary": "Adds `shouldImport` logic to detect and import an existing AWS Organization resource" } ] }, { "id": "3258d6e2-3425-443f-89ef-e61a185dbe41", "type": "fix", "summary": "Fixes install step nesting in `pf cluster add` by removing redundant parent tasks that caused incorrect progress display.", "description": "Each setup phase (e.g., Autoscaling, Inbound Networking) was wrapped in an extra parent task that carried its own `skip` check and created an unnecessary nesting level in the Listr task runner. This caused subtasks to render at the wrong indentation depth and could lead to confusing progress output during cluster installation. The fix removes the redundant `completed` parameter from every setup function and flattens each phase's subtasks so they attach directly to the main task list.", "references": [ { "type": "internal-commit", "summary": "fix: cluster install step nesting", "link": "https://github.com/Panfactum/stack/commit/7c80f0a2d2fe0346cb7a1943baa97984d09ba851" }, { "type": "internal-docs", "summary": "Kubernetes cluster bootstrapping guide", "link": "/docs/main/guides/bootstrapping/kubernetes-cluster" } ], "impacts": [ { "type": "cli", "component": "cluster add", "summary": "Flattened task nesting in the install workflow to render steps at the correct depth" } ] }, { "id": "4fdc87b2-6202-401e-b5e0-2248bab7760f", "type": "fix", "summary": "Fixes image pull failures in `vault_core_resources` by upgrading the vault-secrets-operator Helm chart to `1.3.0`.", "description": "The `gcr.io/kubebuilder/kube-rbac-proxy:v0.15.0` image bundled with vault-secrets-operator chart `0.8.1` was removed from GCR as part of a broader deprecation of the `gcr.io/kubebuilder/kube-rbac-proxy` registry. Chart `0.9.0` and later switched to `quay.io/brancz/kube-rbac-proxy:v0.18.1`, resolving the pull failure without requiring manual image overrides.", "references": [ { "type": "internal-commit", "summary": "fix(deps): bump vault-secrets-operator chart to 1.3.0", "link": "https://github.com/Panfactum/stack/commit/051d0b1213f4b8db3d2bf3e36b30a8e9c88f28ee" }, { "type": "external-docs", "summary": "Vault Secrets Operator v1.3.0 release notes", "link": "https://github.com/hashicorp/vault-secrets-operator/releases/tag/v1.3.0" }, { "type": "external-docs", "summary": "HashiCorp Vault Secrets Operator GitHub repository", "link": "https://github.com/hashicorp/vault-secrets-operator" } ], "impacts": [ { "type": "iac-module", "component": "vault_core_resources", "summary": "vault-secrets-operator Helm chart upgraded from `0.8.1` to `1.3.0` to resolve `gcr.io/kubebuilder/kube-rbac-proxy` image pull failure" } ] }, { "id": "8b11685e-04b3-4744-b86e-d92cadac01c4", "type": "fix", "summary": "`aws_account` now skips service quota increases when the current value already meets the target, preventing spurious `apply` failures.", "description": "Previously, `aws_account` unconditionally submitted AWS service quota increase requests for CloudFront and EC2 quotas. If the current quota already met or exceeded the requested value, AWS would reject the request with an error, causing `terraform apply` to fail. This was especially problematic on accounts where quotas had been raised manually or by a previous apply. The fix reads each quota's current value via data sources and gates the increase resource behind a count check so requests are only submitted when the existing quota is actually below the desired threshold.", "references": [ { "type": "internal-commit", "summary": "fix(aws_account): skip quota increases already at target value", "link": "https://github.com/Panfactum/stack/commit/838357f4a1cca6a118c1f67b71f7920e54170c79" }, { "type": "issue-report", "summary": "Upstream bug: aws_servicequotas_service_quota submits increase even when quota is above requested amount", "link": "https://github.com/hashicorp/terraform-provider-aws/issues/27797" }, { "type": "internal-docs", "summary": "`aws_account` module reference", "link": "/docs/main/reference/infrastructure-modules/direct/aws/aws_account" } ], "impacts": [ { "type": "iac-module", "component": "aws_account", "summary": "Quota increase resources for CloudFront and EC2 are now gated behind a count check using data source lookups, so requests are only submitted when the current value is below the target." } ] }, { "id": "dd462930-1253-4c6e-828c-6133fd0b50b7", "type": "fix", "summary": "Fixes three module deployment failures caused by a stale AMI filter, an outdated default version, and a missing image-manifest flag.", "description": "Three independent issues were blocking production deployments:\n\n1. `aws_vpc` NAT gateway AMI lookup was failing because the `creation-date` filter was stale (`2024-01-25`); updated to `2026-01-26` to match the current fck-nat AMI.\n2. `kube_opensearch` default `opensearch_version` bumped from `3.0.0` to `3.2.0` to resolve a deploy error with the previous default.\n3. `kube_nats` pod scheduling was failing due to image manifest validation errors; the `allowInsecureImages` flag required by the new Bitnami NATS image manifest format has been added.\n", "references": [ { "type": "internal-commit", "summary": "fix(production): restore production deployments after module errors", "link": "https://github.com/Panfactum/stack/commit/8bcb5e0be6c69d0be9a699df62e0f0461b8cccc6" }, { "type": "external-docs", "summary": "fck-nat — open-source AWS NAT instance AMI", "link": "https://github.com/AndrewGuenther/fck-nat" }, { "type": "internal-docs", "summary": "`aws_vpc` module documentation", "link": "/docs/main/reference/infrastructure-modules/direct/aws/aws_vpc" }, { "type": "internal-docs", "summary": "`kube_opensearch` module documentation", "link": "/docs/main/reference/infrastructure-modules/submodule/kubernetes/kube_opensearch" }, { "type": "internal-docs", "summary": "`kube_nats` module documentation", "link": "/docs/main/reference/infrastructure-modules/submodule/kubernetes/kube_nats" } ], "impacts": [ { "type": "iac-module", "component": "aws_vpc", "summary": "NAT gateway AMI `creation-date` filter updated from `2024-01-25` to `2026-01-26`" }, { "type": "iac-module", "component": "kube_opensearch", "summary": "Default `opensearch_version` bumped from `3.0.0` to `3.2.0`" }, { "type": "iac-module", "component": "kube_nats", "summary": "Added `allowInsecureImages` flag to unblock pod scheduling with new Bitnami NATS image manifest format" } ] }, { "id": "a48ee49b-823c-494f-980d-de0bf75bba7d", "type": "breaking_change", "summary": "The `kube_node_image_cache` and `kube_node_image_cache_controller` modules have been removed along with all related image caching input variables across workload modules.", "description": "The node image cache system used Kyverno generate policies to pre-pull container images onto nodes, aiming to reduce pod startup latency. In practice, the generated policy resources caused Kyverno's background controller to consume excessive memory and CPU in production clusters, producing real stability issues that outweighed the latency benefit. Rather than investing in workarounds for an upstream Kyverno scalability limitation, the feature has been removed entirely. All image caching modules, variables, and container spec fields have been deleted.", "action_items": [ "Destroy any active `kube_node_image_cache` module deployments before upgrading.", "Destroy any active `kube_node_image_cache_controller` module deployments before upgrading.", "Remove the module directories for `kube_node_image_cache` and `kube_node_image_cache_controller` from your Terragrunt configuration.", "Remove `node_image_cached_enabled` from any module configurations that set it (affected modules include `kube_airbyte`, `kube_alloy`, `kube_argo_event_bus`, `kube_authentik`, `kube_aws_ebs_csi`, `kube_cloudnative_pg`, `kube_gha_runners`, `kube_ingress_nginx`, `kube_linkerd`, `kube_monitoring`, `kube_nats`, `kube_opensearch`, `kube_pg_cluster`, `kube_redis_sentinel`, `kube_vault`).", "Remove `node_image_cache_enabled` from any module configurations that set it.", "Remove `image_prepull_enabled` and `image_pin_enabled` from all container spec blocks in `kube_pod`, `kube_deployment`, `kube_daemon_set`, `kube_stateful_set`, `kube_cron_job`, and `kube_job` module configurations.", "Remove `panfactum_node_image_cache_enabled` from any `kube_policies` module configuration." ], "references": [ { "type": "internal-commit", "summary": "feat(infrastructure): remove node image cache system", "link": "https://github.com/Panfactum/stack/commit/525cdf8407c293a329a91221c91cc4eb73e2007b" }, { "type": "issue-report", "summary": "Kyverno high memory usage when using generate policies", "link": "https://github.com/kyverno/kyverno/issues/7278" } ], "impacts": [ { "type": "iac-module", "component": "kube_node_image_cache", "summary": "Module removed entirely" }, { "type": "iac-module", "component": "kube_node_image_cache_controller", "summary": "Module removed entirely" }, { "type": "iac-module", "component": "kube_airbyte", "summary": "`node_image_cached_enabled` variable removed" }, { "type": "iac-module", "component": "kube_alloy", "summary": "`node_image_cached_enabled` variable removed" }, { "type": "iac-module", "component": "kube_argo_event_bus", "summary": "`node_image_cached_enabled` variable removed" }, { "type": "iac-module", "component": "kube_authentik", "summary": "`node_image_cached_enabled` variable removed" }, { "type": "iac-module", "component": "kube_aws_ebs_csi", "summary": "`node_image_cached_enabled` variable removed" }, { "type": "iac-module", "component": "kube_cloudnative_pg", "summary": "`node_image_cached_enabled` variable removed" }, { "type": "iac-module", "component": "kube_gha_runners", "summary": "`node_image_cached_enabled` variable removed" }, { "type": "iac-module", "component": "kube_ingress_nginx", "summary": "`node_image_cached_enabled` variable removed" }, { "type": "iac-module", "component": "kube_linkerd", "summary": "`node_image_cached_enabled` variable removed" }, { "type": "iac-module", "component": "kube_monitoring", "summary": "`node_image_cached_enabled` variable removed" }, { "type": "iac-module", "component": "kube_nats", "summary": "`node_image_cached_enabled` variable removed" }, { "type": "iac-module", "component": "kube_opensearch", "summary": "`node_image_cached_enabled` variable removed" }, { "type": "iac-module", "component": "kube_pg_cluster", "summary": "`node_image_cached_enabled` variable removed" }, { "type": "iac-module", "component": "kube_redis_sentinel", "summary": "`node_image_cached_enabled` variable removed" }, { "type": "iac-module", "component": "kube_vault", "summary": "`node_image_cached_enabled` variable removed" }, { "type": "iac-module", "component": "kube_pod", "summary": "`image_prepull_enabled` and `image_pin_enabled` fields removed from container spec inputs" }, { "type": "iac-module", "component": "kube_deployment", "summary": "`image_prepull_enabled` and `image_pin_enabled` fields removed from container spec inputs" }, { "type": "iac-module", "component": "kube_daemon_set", "summary": "`image_prepull_enabled` and `image_pin_enabled` fields removed from container spec inputs" }, { "type": "iac-module", "component": "kube_stateful_set", "summary": "`image_prepull_enabled` and `image_pin_enabled` fields removed from container spec inputs" }, { "type": "iac-module", "component": "kube_cron_job", "summary": "`image_prepull_enabled` and `image_pin_enabled` fields removed from container spec inputs" }, { "type": "iac-module", "component": "kube_job", "summary": "`image_prepull_enabled` and `image_pin_enabled` fields removed from container spec inputs" }, { "type": "iac-module", "component": "kube_policies", "summary": "`panfactum_node_image_cache_enabled` variable removed" } ] }, { "id": "73631989-fb56-4b6c-b33c-ba1f17c3b011", "type": "fix", "summary": "Image pulls for several Bitnami-based modules are restored after the `docker.io/bitnami/*` registry was deprecated.", "description": "Bitnami deprecated their public `docker.io/bitnami/*` image catalog on August 28, 2025, causing image pull failures for any module that referenced those images. All affected Helm chart image overrides and hardcoded image references have been migrated to the `docker.io/bitnamilegacy/*` read-only fallback registry. This is a temporary mitigation; future releases will migrate these modules to non-Bitnami base images.", "references": [ { "type": "internal-commit", "summary": "fix(infrastructure): migrate bitnami images to bitnamilegacy registry", "link": "https://github.com/Panfactum/stack/commit/f3f6bd1ecffd9012a0af3fc0b6d58671a1314cd4" }, { "type": "internal-commit", "summary": "chore: upgrade kubernetes to 1.32 and moves module directory", "link": "https://github.com/Panfactum/stack/commit/e714cfc07835ddfaa414efb73cab508ad22b490f" }, { "type": "external-docs", "summary": "Bitnami deprecated their public docker.io/bitnami/* catalog — bitnamilegacy fallback info", "link": "https://github.com/bitnami/bitnami-docker-archive" }, { "type": "external-docs", "summary": "Broadcom advisory on preparing for the Bitnami deprecation", "link": "https://community.broadcom.com/tanzu/blogs/beltran-rueda-borrego/2025/08/18/how-to-prepare-for-the-bitnami-changes-coming-soon/" }, { "type": "internal-docs", "summary": "`kube_nats` module reference", "link": "/docs/main/reference/infrastructure-modules/submodule/kubernetes/kube_nats" }, { "type": "internal-docs", "summary": "`kube_monitoring` module reference", "link": "/docs/main/reference/infrastructure-modules/direct/kubernetes/kube_monitoring" }, { "type": "internal-docs", "summary": "`kube_redis_sentinel` module reference", "link": "/docs/main/reference/infrastructure-modules/submodule/kubernetes/kube_redis_sentinel" }, { "type": "internal-docs", "summary": "`kube_linkerd` module reference", "link": "/docs/main/reference/infrastructure-modules/direct/kubernetes/kube_linkerd" } ], "impacts": [ { "type": "iac-module", "component": "kube_nats", "summary": "Helm image override added to pull `bitnamilegacy/nats` instead of `bitnami/nats`" }, { "type": "iac-module", "component": "kube_monitoring", "summary": "Helm image override added to pull `bitnamilegacy/thanos` instead of `bitnami/thanos`" }, { "type": "iac-module", "component": "kube_redis_sentinel", "summary": "Helm image overrides added for `bitnamilegacy/redis`, `bitnamilegacy/redis-sentinel`, `bitnamilegacy/redis-exporter`, and `bitnamilegacy/kubectl`" }, { "type": "iac-module", "component": "kube_linkerd", "summary": "Hardcoded `kubectl` sidecar image updated from `bitnami/kubectl` to `bitnamilegacy/kubectl`" } ] }, { "id": "74cb8a25-0018-4325-8390-0fcf62e31f0a", "type": "breaking_change", "summary": "The default Kubernetes version in `aws_eks` has been upgraded from 1.30 to 1.33 with updated addon charts.", "description": "The default Kubernetes version has been upgraded to 1.33 (from 1.30, incrementing through 1.31 and 1.32 in this release). Bottlerocket AMI, all addon Helm chart versions, and the `kubectl` Terraform provider have been updated to compatible releases. Three compatibility fixes are bundled:\n\n1. `aws-ebs-csi-driver` v2.46.0 cannot be installed on K8s 1.33; the chart is pinned to v2.45.1.\n2. Descheduler v0.35.x changed the `DefaultEvictor` API -- flat boolean flags were consolidated into a `podProtections.defaultDisabled` array.\n3. Ingress-nginx v4.12.x lowered the default `annotations-risk-level` from `Critical` to `High`, blocking `configuration-snippet` annotations; the setting is now explicitly pinned to `Critical`.\n\nNo user action is required for the compatibility fixes.\n", "action_items": [ "Review the Kubernetes 1.33 release announcement for any deprecated APIs or behavior changes that affect your workloads.", "If you pin `kube_version` explicitly in `aws_eks`, update it to `1.33`. Otherwise, re-apply `aws_eks` to trigger the upgrade.", "Re-apply `kube_aws_ebs_csi`, `kube_descheduler`, and `kube_ingress_nginx` to pick up the bundled compatibility fixes." ], "references": [ { "type": "internal-commit", "summary": "feat(infrastructure): upgrade EKS default version to 1.33 with addon updates", "link": "https://github.com/Panfactum/stack/commit/b5455f09eb40dd2fac7a7fafc40fc6bbfbac5121" }, { "type": "external-docs", "summary": "Kubernetes v1.33: Octarine release announcement", "link": "https://kubernetes.io/blog/2025/04/23/kubernetes-v1-33-release/" }, { "type": "external-docs", "summary": "Amazon EKS Kubernetes 1.33 release notes", "link": "https://docs.aws.amazon.com/eks/latest/userguide/kubernetes-versions-standard.html" }, { "type": "issue-report", "summary": "Helm chart v2.46.0 cannot be installed on a default K8s 1.33 cluster", "link": "https://github.com/kubernetes-sigs/aws-ebs-csi-driver/issues/2574" }, { "type": "external-docs", "summary": "Descheduler v0.35.0 release -- adds PodProtections for DefaultEvictorArgs", "link": "https://github.com/kubernetes-sigs/descheduler/releases/tag/v0.35.0" }, { "type": "issue-report", "summary": "After upgrade to 4.12 version there is strange error appeared: annotation group StreamSnippet contains risky annotation", "link": "https://github.com/kubernetes/ingress-nginx/issues/12656" }, { "type": "internal-docs", "summary": "`aws_eks` module reference", "link": "/docs/main/reference/infrastructure-modules/direct/aws/aws_eks" }, { "type": "internal-docs", "summary": "`kube_aws_ebs_csi` module reference", "link": "/docs/main/reference/infrastructure-modules/direct/kubernetes/kube_aws_ebs_csi" }, { "type": "internal-docs", "summary": "`kube_descheduler` module reference", "link": "/docs/main/reference/infrastructure-modules/direct/kubernetes/kube_descheduler" }, { "type": "internal-docs", "summary": "`kube_ingress_nginx` module reference", "link": "/docs/main/reference/infrastructure-modules/direct/kubernetes/kube_ingress_nginx" } ], "impacts": [ { "type": "iac-module", "component": "aws_eks", "summary": "Default Kubernetes version upgraded from 1.30 to 1.33; Bottlerocket AMI and addon Helm charts updated to compatible releases" }, { "type": "iac-module", "component": "kube_aws_ebs_csi", "summary": "EBS CSI driver Helm chart pinned to v2.45.1 -- v2.46.0 is incompatible with K8s 1.33" }, { "type": "iac-module", "component": "kube_descheduler", "summary": "`DefaultEvictor` config migrated from flat flags to `podProtections.defaultDisabled` array for descheduler v0.35.x" }, { "type": "iac-module", "component": "kube_ingress_nginx", "summary": "`annotations-risk-level` explicitly set to `Critical` to restore snippet annotation support after ingress-nginx 4.12.x" } ] }, { "id": "a7b0967f-a5ad-4d72-8420-9f8b3b7aad43", "type": "update", "summary": "`kube_authentik` default Helm chart version bumped from 2024.8.4 to 2024.10.5, picking up two minor releases of upstream improvements.", "description": "This upgrade advances `kube_authentik` across two Authentik minor releases (2024.8 to 2024.10), incorporating upstream bug fixes, security patches, and new features such as Chrome Device Trust and FIPS/FAL3 compliance support. Keeping Authentik current reduces drift from upstream and ensures the Panfactum SSO stack benefits from the latest identity provider improvements.", "references": [ { "type": "internal-commit", "summary": "chore(kube_authentik): bump default Authentik helm chart to 2024.10.5", "link": "https://github.com/Panfactum/stack/commit/1ab7f7d7f223bd1cfd73d4c8fcd09d5d22b3dde0" }, { "type": "external-docs", "summary": "Authentik 2024.10 release notes", "link": "https://docs.goauthentik.io/releases/2024.10/" }, { "type": "external-docs", "summary": "Authentik 2024.8 release notes", "link": "https://docs.goauthentik.io/releases/2024.8/" }, { "type": "external-docs", "summary": "Announcing Authentik 2024.10", "link": "https://goauthentik.io/blog/2024-10-31-announcing-release-2024-10/" } ], "impacts": [ { "type": "iac-module", "component": "kube_authentik", "summary": "Default Helm chart version updated from 2024.8.4 to 2024.10.5" } ] }, { "id": "dd36f105-484b-4ec2-b0fc-fc63c235dcd7", "type": "fix", "summary": "The Panfactum DevShell now unsets the `CI` environment variable instead of setting it to `\"false\"`.", "description": "Many tools (e.g., `npm`, Jest, CI platforms) check for the *presence* of the `CI` environment variable, not its value. Setting `CI=\"false\"` still caused those tools to treat the DevShell as a CI environment, disabling interactivity and altering output formatting. Unsetting `CI` entirely ensures the DevShell is correctly recognized as a local, non-CI environment by all tooling.", "references": [ { "type": "internal-commit", "summary": "fix(devshell): unset CI instead of setting it to \"false\"", "link": "https://github.com/Panfactum/stack/commit/9bdb249fe7b0b7d439d1601737fe8336f785fa58" }, { "type": "internal-docs", "summary": "Customizing the Panfactum development shell", "link": "/docs/main/guides/development-shell/customizing" } ], "impacts": [ { "type": "devshell", "component": "enter-shell-local", "summary": "`CI` is now unset on shell entry instead of being set to `\"false\"`" } ] }, { "id": "6847f0f9-c571-44b7-b068-c84388e677f8", "type": "update", "summary": "`kube_buildkit` upgrades the default BuildKit version from `v0.18.1` to `v0.28.1`.", "description": "This upgrade spans 10 minor releases of BuildKit, accumulating substantial improvements to build performance and reliability. Key areas of improvement include more efficient layer caching, smarter garbage collection for build artifacts, and expanded Dockerfile frontend capabilities. The nixpkgs source pin for the `buildkitPkgsSrc` flake input has also been updated to a newer revision that carries the `v0.28.1` package.", "references": [ { "type": "internal-commit", "summary": "feat(buildkit): upgrade BuildKit from v0.18.1 to v0.28.1", "link": "https://github.com/Panfactum/stack/commit/7f7db1fa18e34e7751bfbd47b60f9ad8330385e2" }, { "type": "external-docs", "summary": "BuildKit GitHub repository", "link": "https://github.com/moby/buildkit" }, { "type": "external-docs", "summary": "BuildKit v0.28.1 release notes", "link": "https://github.com/moby/buildkit/releases/tag/v0.28.1" }, { "type": "internal-docs", "summary": "`kube_buildkit` module documentation", "link": "/docs/main/modules/kube_buildkit/overview" } ], "impacts": [ { "type": "iac-module", "component": "kube_buildkit", "summary": "Default BuildKit image version updated from `v0.18.1-rootless` to `v0.28.1-rootless`" } ] }, { "id": "1208e7bc-37f7-4e37-8bb1-3d22f8e70dec", "type": "fix", "summary": "Fixes `pf cluster add` EKS setup where the superuser IAM principal ARN was not configured during cluster deployment.", "description": "During initial cluster bootstrapping via `pf cluster add`, the `aws_eks` module was deployed without setting the caller's IAM ARN as a superuser principal. This caused subsequent install steps to fail because the deploying identity lacked the required EKS cluster access. The fix captures the caller's ARN from `sts:GetCallerIdentity` during the \"Verify access\" step and passes it to the `extra_superuser_principal_arns` input of the `aws_eks` module, ensuring the deployer has full cluster access for the remainder of the installation.", "references": [ { "type": "internal-commit", "summary": "fix: initial cluster permissions", "link": "https://github.com/Panfactum/stack/commit/d127b6f43fc5e715bed6b714ae08e112cbf4a1ca" }, { "type": "internal-docs", "summary": "`aws_eks` module reference — EKS cluster configuration", "link": "/docs/main/modules/aws_eks" } ], "impacts": [ { "type": "cli", "component": "cluster add", "summary": "Captures the caller's IAM ARN during the \"Verify access\" step and passes it as the initial superuser principal when deploying the EKS cluster" }, { "type": "iac-module", "component": "aws_eks", "summary": "The `extra_superuser_principal_arns` input is now automatically populated with the deploying user's ARN during `pf cluster add`" } ] }, { "id": "dcb8989f-e283-49a6-b924-c9b927c57d3d", "type": "update", "summary": "Upgrades DevShell to OpenTofu 1.9.1 and Terragrunt 0.78.2 via a refreshed `nixos-25.05` nixpkgs pin.", "description": "OpenTofu 1.9 introduces two headline features:\n\n1. **Provider `for_each`** — allows a single provider block to be instantiated multiple times (e.g., one AWS provider per account) without repetitive configuration.\n2. **The `-exclude` flag** — lets you skip specific resources during `plan` or `apply` without modifying your configuration files.\n\nThe nixpkgs pin was previously from December 2024; moving to `nixos-25.05` reduces the accumulated upgrade cost and keeps DevShell tooling closer to current upstream releases. The `terraform_version_constraint` in `panfactum.hcl` has been updated from `~> 1.8` to `~> 1.9` to match.\n", "references": [ { "type": "internal-commit", "summary": "build(tooling): upgrade OpenTofu 1.8→1.9 via nixos-25.05 nixpkgs pin", "link": "https://github.com/Panfactum/stack/commit/e9738e4241e1e2925e18e419125b4d56fe5086c3" }, { "type": "external-docs", "summary": "OpenTofu 1.9.0 release announcement — provider `for_each`", "link": "https://opentofu.org/blog/opentofu-1-9-0/" }, { "type": "external-docs", "summary": "OpenTofu 1.9 — what's new (provider iteration and `-exclude` flag)", "link": "https://opentofu.org/docs/v1.9/intro/whats-new" }, { "type": "external-docs", "summary": "Terragrunt v0.78.0 release", "link": "https://github.com/gruntwork-io/terragrunt/releases/tag/v0.78.0" } ], "impacts": [ { "type": "devshell", "component": "tofu", "summary": "Upgraded from 1.8.x to 1.9.1; adds provider `for_each` and `-exclude` flag" }, { "type": "devshell", "component": "terragrunt", "summary": "Upgraded to 0.78.2 via `nixos-25.05` nixpkgs pin" } ] }, { "id": "352b1d0e-23be-49aa-bf45-8030faf27498", "type": "improvement", "summary": "`pf domain add` now shows the live AWS registration status alongside the poll attempt counter.", "description": "Domain registration via AWS Route 53 can take several minutes, and the previous polling loop only showed an attempt counter with no indication of what AWS was actually doing. The task title now includes the live operation status string returned by the `GetOperationDetail` API, giving users immediate visibility into whether registration is still in progress, awaiting verification, or has completed.", "references": [ { "type": "internal-commit", "summary": "feat: add domain registration status", "link": "https://github.com/Panfactum/stack/commit/3200adaba5619879fd3da810d5e69a46e0fbec39" } ], "impacts": [ { "type": "cli", "component": "domain add", "summary": "Registration polling task title now includes the live AWS status string" } ] }, { "id": "54696ecb-99ba-430d-bff1-11004f92c222", "type": "fix", "summary": "Fixes `pf cluster add` Vault unseal step where `kubectl exec` failed to pass arguments correctly.", "description": "The `vault operator unseal` command executed via `kubectl exec` in the Vault Operator Initialization step was missing the `--` argument separator between `kubectl` flags and the Vault command. Without this separator, `kubectl` could misinterpret Vault arguments (such as `-format=json`) as its own flags, causing the unseal operation to fail. This made it impossible to complete the Vault initialization during cluster bootstrapping.", "references": [ { "type": "internal-commit", "summary": "fix: kube exec in vault setup", "link": "https://github.com/Panfactum/stack/commit/1b4bb5bf5d6b1c3f644f1bb955f1dbc1014b37d4" }, { "type": "internal-docs", "summary": "Vault bootstrapping guide", "link": "/docs/main/guides/bootstrapping/vault" } ], "impacts": [ { "type": "cli", "component": "cluster add", "summary": "Added missing `--` separator to `kubectl exec` in the Vault unseal step so arguments are passed correctly" } ] }, { "id": "a6965793-76b8-45ef-a51a-ad289152ea52", "type": "fix", "summary": "Fixes broken alert email prompt in the `pf cluster add` cert issuers step caused by a missing `task` parameter.", "description": "The `context.logger.input()` call for the alert email prompt in `setupCertificateIssuers` was not receiving the `task` parameter required by Listr's task context. Without it, the prompt could not render correctly within the task runner, causing the cert issuers setup step to fail before users could provide their certificate renewal notification email address.", "references": [ { "type": "internal-commit", "summary": "Add missing `task` parameter to cert issuer email prompt", "link": "https://github.com/Panfactum/stack/commit/1983c232d7dd2507df93fc28b8743d1b2afb401d" }, { "type": "internal-docs", "summary": "Certificate management bootstrapping guide", "link": "/docs/main/guides/bootstrapping/certificate-management" } ], "impacts": [ { "type": "cli", "component": "cluster add", "summary": "Alert email prompt in the cert issuers step now receives the required `task` parameter and renders correctly" } ] }, { "id": "b4bd1035-bfe0-45e5-a1e3-fee8123d6b0b", "type": "fix", "summary": "Fixes `pf-domain-add` contact prompts with incorrect `required` settings that rejected valid input.", "description": "Multiple contact information prompts in the `pf domain add` registration flow had incorrect `required` settings:\n\n1. The First Name, Last Name, Email, Phone, Street Address 1, City, State/Region, and Postal Code prompts incorrectly set `required: true`, which conflicted with their custom validators and caused valid input to be rejected in some situations.\n2. The Organization Name and Street Address 2 prompts were missing an explicit `required: false`, preventing them from being properly skipped as optional fields.\n", "references": [ { "type": "internal-commit", "summary": "Fix contact info prompt `required` flags in `pf domain add`", "link": "https://github.com/Panfactum/stack/commit/1983c232d7dd2507df93fc28b8743d1b2afb401d" } ], "impacts": [ { "type": "cli", "component": "domain add", "summary": "Corrected `required` flags on contact information prompts during domain registration" } ] }, { "id": "32e90af6-82a5-44bd-8689-5cc4bd221f01", "type": "fix", "summary": "Fixes `pf domain add` to display nameservers one per line during manual DNS zone setup, improving copy-paste usability.", "description": "Previously, nameservers were printed as a dense block of text via the logger's warning output, making it difficult to identify and copy individual nameserver addresses into a domain registrar. This change reformats the output so each nameserver appears on its own line with a bullet prefix, improving readability during the manual DNS delegation step.", "references": [ { "type": "internal-commit", "summary": "fix: nameserver listing", "link": "https://github.com/Panfactum/stack/commit/0d65d646b488645ac3f8ad05d0852c5eedbcd3b1" }, { "type": "internal-commit", "summary": "fix: nameservers output in cli", "link": "https://github.com/Panfactum/stack/commit/23a2d48fabd479261dbbf07c87101cf0322dc3c8" } ], "impacts": [ { "type": "cli", "component": "domain add", "summary": "Nameservers are now listed one per line during manual DNS zone setup" } ] }, { "id": "15b34ceb-1658-4b63-9766-3e7f9eb727ea", "type": "fix", "summary": "`pf aws vpc-network-test` inbound blocking check now uses TCP instead of ICMP ping for reliable verification.", "description": "The previous inbound blocking check used `ping` (ICMP echo) to verify that NAT IPs were not reachable. However, ICMP traffic can be blocked by security groups or NACLs independently of TCP, making the check unreliable — a NAT IP could accept TCP connections while still blocking pings, causing the test to pass incorrectly. The new implementation uses a TCP socket connection attempt on port 80 via a dedicated `checkConnection` utility, which directly tests the protocol that matters for real-world inbound traffic.", "references": [ { "type": "internal-commit", "summary": "Replace ICMP ping with TCP connection check for inbound blocking test", "link": "https://github.com/Panfactum/stack/commit/9bb03e5270ef399bf42d603795e2c8698394e588" }, { "type": "internal-docs", "summary": "AWS networking bootstrapping guide covering `pf aws vpc-network-test` usage", "link": "/docs/main/guides/bootstrapping/aws-networking" } ], "impacts": [ { "type": "cli", "component": "aws vpc-network-test", "summary": "Inbound blocking check now uses a TCP socket connection on port 80 instead of ICMP ping" } ] }, { "id": "aa44b0b2-6c6d-4a05-8dd0-def391b9a452", "type": "fix", "summary": "Fixes multiple `pf cluster add` Inbound Networking bugs including phase ordering, missing Vault env vars, SLA bootstrapping, and deployment concurrency.", "description": "The Inbound Networking phase of `pf cluster add` had five interrelated bugs that could cause cluster installation to fail:\n\n1. The Inbound Networking phase was executing after Autoscaling, requiring each autoscaling module to receive Vault credentials before Vault had an ingress. The phase ordering has been corrected so Inbound Networking runs before Autoscaling as intended.\n2. `kube_external_dns` was not receiving the `VAULT_ADDR` environment variable, causing it to fail Vault authentication. It is now passed alongside `VAULT_TOKEN`.\n3. `kube_ingress_nginx` was deployed with the full final `sla_level` before Vault ingress was available, causing failures. It is now bootstrapped with `sla_level: 1` and cleaned up via a post-deploy update.\n4. The post-deploy cleanup Zod schema for `sla_level` used `z.undefined()` instead of `z.number().optional()`, causing the cleanup step to fail validation. The schema has been corrected.\n5. The `kube_aws_lb_controller` and `kube_external_dns` modules were deployed concurrently, causing race conditions. They are now deployed sequentially.\n", "references": [ { "type": "internal-commit", "summary": "fix: move inbound networking before autoscaling in cluster auto install", "link": "https://github.com/Panfactum/stack/commit/92da33ac2a718d02d99cd0471e4c653d2426f9ef" }, { "type": "internal-commit", "summary": "fix: nginx post deploy input update", "link": "https://github.com/Panfactum/stack/commit/63497c19c9b6cc1faf01c137f7be019fa77ee583" }, { "type": "internal-commit", "summary": "fix: remove concurrency", "link": "https://github.com/Panfactum/stack/commit/4f42f9ba484e2b9275588c3522858327b1b17c33" } ], "impacts": [ { "type": "cli", "component": "cluster add", "summary": "Inbound Networking phase now runs before Autoscaling, modules deploy sequentially instead of concurrently, and all Vault environment variables are correctly passed to each module deployment." }, { "type": "iac-module", "component": "kube_ingress_nginx", "summary": "Bootstrapped with `sla_level: 1` during cluster install and post-deploy input update Zod schema corrected from `z.undefined()` to `z.number().optional()`." }, { "type": "iac-module", "component": "kube_external_dns", "summary": "Now receives `VAULT_ADDR` environment variable during `pf cluster add` deployment." }, { "type": "iac-module", "component": "kube_aws_lb_controller", "summary": "No longer deployed concurrently with `kube_external_dns` to avoid race conditions." } ] }, { "id": "83531ff4-6c7a-4c3d-b0f2-f4b28822c2d8", "type": "fix", "summary": "`pf cluster add` now passes `VAULT_TOKEN` to autoscaling and inbound networking deployments, fixing Vault authentication failures.", "description": "The `VAULT_TOKEN` environment variable was not being passed when deploying Metrics Server, VPA, Karpenter, and `vault_core_resources` during the autoscaling and inbound networking steps of `pf cluster add`. This caused Terragrunt deployments to fail Vault authentication. The fix decrypts the Vault root token from the `kube_vault` secrets and injects it into the environment for all affected module deployments.", "references": [ { "type": "internal-commit", "summary": "fix: pass vault token", "link": "https://github.com/Panfactum/stack/commit/b949ef07943277aebe73c457d477c2235480251d" } ], "impacts": [ { "type": "cli", "component": "cluster add", "summary": "Vault root token now passed as `VAULT_TOKEN` to Metrics Server, VPA, Karpenter, and `vault_core_resources` deployments during autoscaling and inbound networking setup" } ] }, { "id": "d54adc55-ed6e-403b-94e7-89cfd903e7c7", "type": "update", "summary": "Upgrades the AWS Load Balancer Controller to `v2.12.0` (Helm chart `1.12.0`) in `kube_aws_lb_controller`.", "description": "This update brings several improvements from the upstream AWS Load Balancer Controller `v2.12.0` release:\n\n- AWS VPC IPAM support for bring-your-own-IP (BYOIP) on ALBs\n- mTLS enhancement to advertise trusted CA subject names\n- NLB ICMP-based path MTU discovery via service annotations\n- Cross-account target group registration\n- Refactored listener rule modifications for zero-downtime routing updates\n- Changed the default webhook failure policy from `Fail` to `Ignore` for improved disaster recovery\n", "references": [ { "type": "internal-commit", "summary": "Upgrade `aws-load-balancer-controller` Helm chart from `1.11.0` to `1.12.0`", "link": "https://github.com/Panfactum/stack/commit/b949ef07943277aebe73c457d477c2235480251d" }, { "type": "external-docs", "summary": "AWS Load Balancer Controller `v2.12.0` release notes", "link": "https://github.com/kubernetes-sigs/aws-load-balancer-controller/releases/tag/v2.12.0" }, { "type": "external-docs", "summary": "Helm chart `1.12.0` on Artifact Hub", "link": "https://artifacthub.io/packages/helm/aws/aws-load-balancer-controller/1.12.0" }, { "type": "external-docs", "summary": "AWS Load Balancer Controller GitHub repository", "link": "https://github.com/kubernetes-sigs/aws-load-balancer-controller" }, { "type": "internal-docs", "summary": "`kube_aws_lb_controller` module reference", "link": "/docs/main/reference/infrastructure-modules/direct/kubernetes/kube_aws_lb_controller" } ], "impacts": [ { "type": "iac-module", "component": "kube_aws_lb_controller", "summary": "Default Helm chart version updated from `1.11.0` to `1.12.0` (app version `v2.12.0`)" } ] }, { "id": "e081fc69-c1f8-4c43-b2d4-e5bcd41c8c35", "type": "breaking_change", "summary": "`vpa_enabled` now defaults to `false` in several ingress and infrastructure modules.", "description": "VPA was previously enabled by default in these modules, which could cause issues when `kube_vpa` had not yet been deployed in the cluster. Disabling VPA by default ensures these modules work correctly out of the box without requiring a VPA installation, while still allowing users to opt in by setting `vpa_enabled = true` when VPA is available.", "action_items": [ "If you rely on VPA for autoscaling in `kube_aws_lb_controller`, `kube_external_dns`, `kube_ingress_nginx`, or `vault_core_resources`, add `vpa_enabled = true` to each module's configuration." ], "references": [ { "type": "internal-commit", "summary": "fix: disable vpa by default in ingress modules", "link": "https://github.com/Panfactum/stack/commit/cc1da7cb297e608219710dc6534d9280821f4719" } ], "impacts": [ { "type": "iac-module", "component": "kube_aws_lb_controller", "summary": "`vpa_enabled` now defaults to `false`" }, { "type": "iac-module", "component": "kube_external_dns", "summary": "`vpa_enabled` now defaults to `false`" }, { "type": "iac-module", "component": "kube_ingress_nginx", "summary": "`vpa_enabled` now defaults to `false`" }, { "type": "iac-module", "component": "vault_core_resources", "summary": "`vpa_enabled` now defaults to `false`" } ] }, { "id": "1a75a888-209c-487d-8683-8c62f4fcdb2c", "type": "fix", "summary": "Improves `pf cluster add` Vault ingress verification with real-time polling status, a DNS propagation warning, and a longer timeout.", "description": "The Vault ingress health check during `pf cluster add` previously provided unclear feedback while waiting for\nDNS propagation, making it difficult for users to understand what was happening. This change addresses three\nissues:\n\n1. The task title now updates in real time to show the current polling attempt count and whether it is waiting\n for DNS propagation or for Vault readiness.\n2. A warning message is displayed indicating that DNS propagation may take 10-30 minutes.\n3. The polling timeout has been increased from 10 minutes (60 attempts) to 30 minutes (180 attempts) to\n accommodate slower DNS propagation without false failures.\n", "references": [ { "type": "internal-commit", "summary": "fix: better ux for vault ingress", "link": "https://github.com/Panfactum/stack/commit/35dfbc34d67e5844d6bfb627e35f94e05e14452b" }, { "type": "internal-docs", "summary": "Bootstrapping guide for inbound networking and Vault ingress setup", "link": "/docs/main/guides/bootstrapping/inbound-networking" } ], "impacts": [ { "type": "cli", "component": "cluster add", "summary": "Vault ingress verification now shows real-time polling status, displays a DNS propagation warning, and allows up to 30 minutes for the health check to succeed." } ] }, { "id": "8f690f60-8010-4b00-b7a9-78309e1b143e", "type": "addition", "summary": "Adds the `pf welcome` CLI command with onboarding content, auto-launch on DevShell startup, and a persisted `installation_id` in `panfactum.yaml`.", "description": "Adds the `pf welcome` CLI command, which displays an introductory welcome screen orienting new users to the `pf` CLI, IaC modules, and environment structure. The welcome screen covers:\n\n1. A DevShell overview explaining the version-pinned CLI tools available in the terminal environment\n2. A Getting Started section listing the key setup commands (`pf env add`, `pf domain add`, `pf cluster add`, `pf idp add`)\n3. A Concepts section introducing Infrastructure-as-Code, the Environments/Regions/Modules directory layout, and Panfactum Kubernetes clusters\n4. A Getting Help section with links to the Panfactum Discord and GitHub issue tracker\n\nThe DevShell now automatically runs `pf welcome` on shell launch via `enter-shell-local.sh`. On first run, the command generates a UUID `installation_id` and a `user_id`, saves them to `panfactum.yaml`, and sends an anonymous telemetry event. Subsequent launches detect the existing `installation_id` and suppress the welcome screen.\n", "references": [ { "type": "internal-commit", "summary": "fix: undefined import", "link": "https://github.com/Panfactum/stack/commit/0d1327c722d578fbe23280fc0ff98efaa0f0c10e" }, { "type": "internal-commit", "summary": "fix: nameservers output in cli", "link": "https://github.com/Panfactum/stack/commit/23a2d48fabd479261dbbf07c87101cf0322dc3c8" }, { "type": "internal-commit", "summary": "feat: adds welcome screen", "link": "https://github.com/Panfactum/stack/commit/11c41aa1fc01f6067bc76d2822199bc39bbfa40d" }, { "type": "internal-commit", "summary": "feat: adds welcome screen to devshell launch", "link": "https://github.com/Panfactum/stack/commit/d8d40050f39848b34c8c9a684211850db03ffe2f" }, { "type": "internal-commit", "summary": "feat: adds installation_id", "link": "https://github.com/Panfactum/stack/commit/bc4b30db4802ea40f7f9096dc17c1a929103cbc8" }, { "type": "internal-docs", "summary": "Installing the DevShell", "link": "/docs/main/guides/bootstrapping/installing-devshell" }, { "type": "internal-docs", "summary": "Customizing the Development Shell", "link": "/docs/main/guides/development-shell/customizing" } ], "impacts": [ { "type": "cli", "component": "welcome", "summary": "New command; displays Environments/Regions/Modules layout, Getting Started steps, and Getting Help links; generates a UUID `installation_id` saved to `panfactum.yaml` on first run" }, { "type": "devshell", "component": "enter-shell-local", "summary": "Now calls `pf welcome` on shell launch to display the onboarding screen" }, { "type": "configuration", "component": "panfactum.yaml", "summary": "New optional `installation_id` field (UUID) written automatically by `pf welcome` on first run" } ] }, { "id": "acb80b21-ace8-44f3-88d0-1655a89b9f96", "type": "fix", "summary": "Fixes a bug in the `pf` CLI where Terragrunt resource imports failed during module deployment when `resourceId` resolved to `undefined`.", "description": "The `deployModuleTask` utility accepted a `resourceId` that could be a static string or a function returning a string. When a function returned `undefined` (e.g., because the upstream resource had not been created yet), the import task would attempt to import a literal `undefined` value, causing the deployment to error out. The fix changes the `resourceId` type signature to explicitly allow `undefined` and adds an early guard that skips the import task gracefully when the resolved ID is absent.", "references": [ { "type": "internal-commit", "summary": "fix: undefined import", "link": "https://github.com/Panfactum/stack/commit/0d1327c722d578fbe23280fc0ff98efaa0f0c10e" } ], "impacts": [ { "type": "cli", "component": "cluster add", "summary": "Import tasks in `deployModuleTask` now skip gracefully when `resourceId` resolves to `undefined`." }, { "type": "cli", "component": "env add", "summary": "Import tasks in `deployModuleTask` now skip gracefully when `resourceId` resolves to `undefined`." }, { "type": "cli", "component": "domain add", "summary": "Import tasks in `deployModuleTask` now skip gracefully when `resourceId` resolves to `undefined`." } ] }, { "id": "cb271a09-f90b-430d-8534-b2d5fbe3c1d9", "type": "improvement", "summary": "The `pf` CLI now wraps long output lines at 100 characters instead of 128, improving readability on standard 120-column terminals.", "description": "The previous `MAX_WIDTH` of 128 characters caused text to overflow or wrap awkwardly on the most common terminal width of 120 columns. Reducing the wrap threshold to 100 characters ensures that CLI output, including the welcome screen and all logger messages, fits comfortably within a standard terminal without horizontal scrolling or mid-word breaks.", "references": [ { "type": "internal-commit", "summary": "feat: adds welcome screen", "link": "https://github.com/Panfactum/stack/commit/11c41aa1fc01f6067bc76d2822199bc39bbfa40d" } ], "impacts": [ { "type": "cli", "component": "welcome", "summary": "Output text now wraps at 100 characters instead of 128" } ] }, { "id": "3efa85f8-1dd7-4401-ae18-657f6919ae56", "type": "fix", "summary": "Fixes and hardens the Nix version check in the DevShell `.envrc` template.", "description": "Two iterative fixes to the Nix version check in the `.envrc` template:\n\n1. The version field parse was corrected from `$3` to `$NF` to handle `nix --version` output with more than three whitespace-separated tokens.\n2. The entire version comparison was refactored to use a POSIX-compatible `sort -V` approach instead of a bash-specific array function, improving portability across shells.\n3. A missing-Nix prompt was added for when Nix is not on `PATH` at all, guiding the user to install it rather than failing silently.\n\nAdditionally, the `install.sh` DevShell build time estimate was updated from 30 minutes to 15 minutes to reflect current build performance.\n", "references": [ { "type": "internal-commit", "summary": "fix: nix version check in envrc", "link": "https://github.com/Panfactum/stack/commit/ac01f734d88d02ebfbf26aeaefaf434f8be25df4" }, { "type": "internal-commit", "summary": "fix: refactor nix version check in .envrc to match install.sh", "link": "https://github.com/Panfactum/stack/commit/3e06e130f943907d5409980df2333fbdc813a5bf" }, { "type": "internal-docs", "summary": "Guide for installing and booting the Panfactum DevShell", "link": "/docs/main/guides/bootstrapping/installing-devshell" } ], "impacts": [ { "type": "cli", "component": "devshell sync", "summary": "`.envrc` template updated with a POSIX-compatible Nix version check and a missing-Nix installation prompt" }, { "type": "installer", "component": "install.sh", "summary": "DevShell build time estimate updated from 30 to 15 minutes" } ] }, { "id": "bfc3b7c4-ba45-4623-96ff-0a185ea08536", "type": "fix", "summary": "Fixes `pf env add` credential loading failure for the management account profile during environment provisioning.", "description": "The AWS Node SDK caches credential-provider results in memory, so credentials written to `~/.aws/credentials` during the same process run are invisible to subsequent SDK clients that use the default provider chain. This caused `pf env add` to fail when provisioning new AWS accounts because the `OrganizationsClient` and `STSClient` could not find the management account credentials that an earlier step had just saved to disk. The fix explicitly reads credentials from the credentials file via `getCredsFromFile` and passes them directly to each client constructor, bypassing the SDK's provider cache entirely.", "references": [ { "type": "internal-commit", "summary": "Fix AWS SDK credentials bug in `pf env add`", "link": "https://github.com/Panfactum/stack/commit/db4b7e515ee1a16e6fdbd392a2adcbe00024d669" }, { "type": "issue-report", "summary": "AWS SDK credential provider cache does not pick up newly written credentials", "link": "https://github.com/aws/aws-sdk-js-v3/issues/5829" } ], "impacts": [ { "type": "cli", "component": "env add", "summary": "`OrganizationsClient` and `STSClient` now read credentials directly from disk instead of relying on the SDK provider cache." } ] }, { "id": "ef6a3975-91f7-4fdd-82cf-e03f3bf402b9", "type": "fix", "summary": "`pf domain add` now uses a WHOIS fallback to fix false-negative domain registration detection for newly purchased domains.", "description": "Previously, `pf domain add` determined whether a domain was registered solely by checking if it had DNS nameservers configured. A recently purchased domain with no nameservers would be incorrectly reported as unregistered, forcing users to work around the false negative. The registration check now uses a two-step approach: it first queries nameservers via `dig` and, if none are found, falls back to a WHOIS database lookup. Although the WHOIS database may be 24-72 hours out-of-date, it provides a reliable secondary signal for newly registered domains. The `whois` CLI tool has been added to the Panfactum DevShell as a dependency for this check.", "references": [ { "type": "internal-commit", "summary": "Improve domain `isRegistered` check with WHOIS fallback", "link": "https://github.com/Panfactum/stack/commit/4156dcf072d3e93735c7f16361fcbcbe61c8b7cd" } ], "impacts": [ { "type": "cli", "component": "domain add", "summary": "Domain registration detection now uses a two-step check: first querying DNS nameservers via `dig`, then falling back to a WHOIS database lookup when no nameservers are found." }, { "type": "devshell", "component": "whois", "summary": "Added `whois` to the DevShell to support the improved domain registration check." } ] }, { "id": "0af75a20-e01a-4785-afe2-2a64f746b3c5", "type": "fix", "summary": "`pf env add` phone number prompt now includes a format example, clarifying that a country-code prefix is required.", "description": "The phone number prompt in `pf env add` previously displayed only the label `Phone #:` with no indication of the expected format. Users had to guess whether to include a country code, use dashes, or follow E.164 conventions. The prompt now reads `Phone # (+1 555-555-5555):`, giving an inline example that clarifies the expected structure and reduces input errors during environment setup.", "references": [ { "type": "internal-commit", "summary": "fix: phone number format in question", "link": "https://github.com/Panfactum/stack/commit/b186300f9298fe13d12b7f05abdeb090f4756dd1" } ], "impacts": [ { "type": "cli", "component": "env add", "summary": "Phone number prompt now shows an example format to guide user input" } ] }, { "id": "cd9082c0-3a6c-48db-8311-40c7b7ff41c8", "type": "fix", "summary": "Fixes `pf` CLI `applyColors` incorrectly dedenting text after applying ANSI color codes, causing misaligned styled output.", "description": "The `applyColors` method in the `pf` CLI logger accepted a `dedent: true` option, but previously applied dedentation after ANSI color codes had already been injected into the string. Because ANSI escape sequences contain non-whitespace characters, the dedent algorithm could not correctly identify common leading whitespace, resulting in misaligned or un-dedented output. The fix reorders the operations so that `dedent` runs on the raw string before any color styling is applied.", "references": [ { "type": "internal-commit", "summary": "fix: dedent order in applyColors", "link": "https://github.com/Panfactum/stack/commit/b695023a1701ff34709eeee49c64d2c9aaf79050" } ], "impacts": [ { "type": "cli", "component": "logging", "summary": "Fixes operation order in `applyColors` so `dedent` runs on the raw string before color styling is applied, preventing ANSI escape sequences from interfering with indentation analysis." } ] }, { "id": "7ed789c6-7d04-4f58-b4c6-47f9f1519dad", "type": "fix", "summary": "Fixes `pf domain add` to require a non-management environment and corrects stale `pf env install` references to `pf env add`.", "description": "Two fixes from the same commit:\n\n1. `pf domain add` now validates that at least one non-management environment exists before proceeding, exiting early with a clear message directing users to run `pf env add` first instead of failing with a cryptic error later.\n2. Error messages that incorrectly referenced the old `pf env install` command name have been updated to use the current `pf env add` name.\n", "references": [ { "type": "internal-commit", "summary": "fix:domains: at least one env check + 'env install' -> 'env add'", "link": "https://github.com/Panfactum/stack/commit/a9bb99d813ead7fb63050b375087628e07eea487" } ], "impacts": [ { "type": "cli", "component": "domain add", "summary": "Now validates at least one non-management environment exists; error messages updated from `pf env install` to `pf env add`" } ] }, { "id": "70cfc13d-0a61-4b45-94a4-695fa89b244e", "type": "addition", "summary": "`aws_account` now automatically requests an EC2 on-demand vCPU quota increase to 32.", "description": "New AWS accounts default to a low EC2 on-demand vCPU limit that is insufficient for typical Panfactum workloads. Previously, users had to manually request a quota increase through the AWS console before deploying clusters. The `aws_account` module now automatically submits a Service Quotas request to raise the on-demand standard instance family vCPU limit to 32, eliminating this manual step.", "references": [ { "type": "internal-commit", "summary": "feat: automatically request increase in vcpu quota in aws_account module", "link": "https://github.com/Panfactum/stack/commit/6496caab324c847f53e59b82ecd948e799f7df13" }, { "type": "external-docs", "summary": "Amazon EC2 instance type quotas documentation", "link": "https://docs.aws.amazon.com/ec2/latest/instancetypes/ec2-instance-quotas.html" }, { "type": "external-docs", "summary": "How to request an EC2 On-Demand Instance vCPU quota increase", "link": "https://repost.aws/knowledge-center/ec2-on-demand-instance-vcpu-increase" } ], "impacts": [ { "type": "iac-module", "component": "aws_account", "summary": "Adds a new `aws_servicequotas_service_quota` resource that requests the EC2 on-demand standard vCPU quota be raised to 32 on apply." } ] }, { "id": "545fed40-165b-49fb-bb05-815fa72c0065", "type": "fix", "summary": "Fixes `pf-cluster-add` cert issuers step to restart cert-manager after nginx ingress deployment, preventing certificates from staying not-ready.", "description": "After deploying the nginx ingress controller, cert-manager sometimes fails to issue certificates because it\nis unaware of the new ingress class. Previously, the cert issuers deployment and the first certificate deployment\nran sequentially without any retry logic, so a stale cert-manager pod could block cluster bootstrapping indefinitely.\n\nThe fix runs the cert issuers deployment and a new cert-manager restart loop concurrently:\n\n1. The restart loop checks every 90 seconds whether the certificate labeled `panfactum.com/root-module=kube_cert_issuers` has a `Ready` condition.\n2. If the certificate is not ready, it runs `kubectl rollout restart deployment -n cert-manager` to force cert-manager to pick up the new ingress configuration.\n3. After up to 10 attempts, the loop either succeeds or throws a `CLIError`.\n\nOnce the certificate is confirmed ready, the \"Deploy The First Certificate\" step proceeds to disable self-generated certs on `kube_cert_manager`.\n", "references": [ { "type": "internal-commit", "summary": "fix: reset cert manager on nginx ingress deploy (#350)", "link": "https://github.com/Panfactum/stack/commit/74894a454a64f7cd6285257ef3bebda2ba19e601" } ], "impacts": [ { "type": "cli", "component": "cluster add", "summary": "Cert issuers step now runs a concurrent retry loop that restarts the cert-manager deployment until the certificate reaches a ready state" } ] }, { "id": "a4a37bfa-ad7b-4662-8ea1-17d072139c36", "type": "fix", "summary": "DevShell boot script now unsets `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` to prevent host credentials from overriding profile-based configuration.", "description": "When users had `AWS_ACCESS_KEY_ID` or `AWS_SECRET_ACCESS_KEY` set in their host environment, those static credentials would leak into the Panfactum DevShell and take precedence over the DevShell's own profile-based credential configuration via `AWS_CONFIG_FILE` and `AWS_SHARED_CREDENTIALS_FILE`. The boot script (`enter-shell-local.sh`) now explicitly unsets both variables on shell entry, ensuring the DevShell always uses its managed credential workflow.", "references": [ { "type": "internal-commit", "summary": "fix: unload AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY env vars on devshell boot", "link": "https://github.com/Panfactum/stack/commit/4ed4265aa34302f474654217144017cca60903a8" }, { "type": "internal-docs", "summary": "Installing the Panfactum DevShell", "link": "/docs/main/guides/bootstrapping/installing-devshell" } ], "impacts": [ { "type": "devshell", "component": "enter-shell-local", "summary": "`AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` are now unset on shell entry to prevent host credentials from leaking into the DevShell." } ] }, { "id": "ce37fe88-a513-4383-8d60-2afa8c21c9ea", "type": "fix", "summary": "Fixes Street Address 2 minimum length validation in `pf env add` and `pf domain add` contact prompts.", "description": "The Street Address 2 input validation in both `pf env add` and `pf domain add` contact information prompts incorrectly shared the same 5-character minimum as Street Address 1, rejecting valid short values like `Apt 2` or `# 5`. The minimum for Street Address 2 is now 2 characters, matching other optional address fields.", "references": [ { "type": "internal-commit", "summary": "fix: address line 2 min length", "link": "https://github.com/Panfactum/stack/commit/03663dd54d9623b344f2bd070617645856ff6feb" }, { "type": "internal-commit", "summary": "fix: address line 2 min length", "link": "https://github.com/Panfactum/stack/commit/1cfe02357781201fcc0ee25d825572790480709c" } ], "impacts": [ { "type": "cli", "component": "env add", "summary": "Street Address 2 minimum length corrected to 2 characters" }, { "type": "cli", "component": "domain add", "summary": "Street Address 2 minimum length corrected to 2 characters in domain registration flow" } ] }, { "id": "7b3cc166-af27-4203-b9ef-627fe7ddd329", "type": "fix", "summary": "Fixes `kube_ingress_nginx` to allow multiple nginx replicas to be scheduled on the same host when `sla_target` is set to 1.", "description": "Previously, `host_anti_affinity_required` was unconditionally set to `true` in the `kube_ingress_nginx` module, which prevented the Kubernetes scheduler from placing more than one nginx pod per node. For `sla_target` 1 deployments -- which do not guarantee high availability -- this made it impossible to run multiple replicas on a single-node or small cluster. The fix conditionally sets `host_anti_affinity_required` to `false` when `sla_target` is 1, allowing replicas to colocate on the same host while preserving the anti-affinity requirement for higher SLA tiers.", "references": [ { "type": "internal-commit", "summary": "fix: allow multiple nginx instances on one host for sla 1", "link": "https://github.com/Panfactum/stack/commit/de4a60f36ef43cf341131d3fbd80765c66d95693" }, { "type": "external-docs", "summary": "Ingress NGINX Controller documentation", "link": "https://kubernetes.github.io/ingress-nginx/" }, { "type": "internal-docs", "summary": "`kube_ingress_nginx` module reference", "link": "/docs/main/reference/infrastructure-modules/direct/kubernetes/kube_ingress_nginx" } ], "impacts": [ { "type": "iac-module", "component": "kube_ingress_nginx", "summary": "`host_anti_affinity_required` is now set to `false` when `sla_target` is 1, allowing multiple replicas on the same node" } ] }, { "id": "79122405-3b24-4696-8d75-75edcc780808", "type": "improvement", "summary": "Panfactum terragrunt hooks now write `.pf.yaml` module status; `pf cluster add` uses it for accurate resumability.", "description": "Two related improvements:\n\n1. `panfactum.hcl` now calls `pf iac update-module-status` in before/after/error hooks for both `init` and `apply` phases, writing `initStatus` and `deployStatus` to each module's `.pf.yaml` file.\n2. The `pf cluster add` checkpointing mechanism now reads `deployStatus: success` from each module's `.pf.yaml` instead of testing for directory presence, so a partially-completed or failed apply is no longer treated as complete.\n\nAdditionally, the `Certificate Management` and `Certificate Issuers` steps are consolidated into a single `Certificates` step with independently-skippable sub-tasks.\n", "references": [ { "type": "internal-commit", "summary": "feat: add Panfactum terragrunt hooks", "link": "https://github.com/Panfactum/stack/commit/a316a5dc7cc6eb0a4acc93f1a333d8e40a50036a" }, { "type": "internal-commit", "summary": "feat: better resumability (#351)", "link": "https://github.com/Panfactum/stack/commit/70421c38fa9eb6f3238b5f1e7dfa9101f8a84f7e" }, { "type": "internal-commit", "summary": "feat: enhances module status architecture", "link": "https://github.com/Panfactum/stack/commit/00029f4e98aec9c1119bba2ff2a7fcb2d228620b" } ], "impacts": [ { "type": "cli", "component": "cluster add", "summary": "Checkpointing uses `.pf.yaml` deploy status; cert management and cert issuers steps consolidated into a single `Certificates` step" }, { "type": "cli", "component": "iac update-module-status", "summary": "New CLI command invoked by terragrunt hooks to write `initStatus` and `deployStatus` to each module's `.pf.yaml`" }, { "type": "configuration", "component": "panfactum.hcl", "summary": "Adds before/after/error hooks for `init` and `apply` that call `pf iac update-module-status` to track deployment status" } ] }, { "id": "87680f8c-db77-4069-afb2-bebd0d5a3efe", "type": "fix", "summary": "Fixes `pf cluster add` Vault key storage to split root token and recovery keys into separate SOPS-encrypted files.", "description": "Three Vault-related `pf cluster add` fixes:\n\n1. Recovery keys and the root token are now saved immediately after `vault operator init` completes, before the unseal step, so a CLI crash cannot lose the recovery material. The root token is written to `kube_vault/secrets.yaml` and recovery keys to `kube_vault/recovery.yaml`. A follow-on fix updated the read-side code to match — previously the setup step still read both from a single `kube_vault/secrets.yaml` file, failing on any cluster where the split had already occurred.\n2. The Vault `kubectl port-forward` proxy now passes `--context` to target the correct cluster when multiple Kubernetes contexts are present.\n3. The `Deploy Vault Core Resources` step now runs at the start of the autoscaling phase, ensuring the Vault ingress is healthy before the step executes.\n", "references": [ { "type": "internal-commit", "summary": "feat: better resumability (#351)", "link": "https://github.com/Panfactum/stack/commit/70421c38fa9eb6f3238b5f1e7dfa9101f8a84f7e" }, { "type": "internal-commit", "summary": "fix: split vault recovery keys and root token", "link": "https://github.com/Panfactum/stack/commit/62031a6aa8fe0352dd3ed1abd3062d9ff134f6b0" }, { "type": "internal-commit", "summary": "fix: reading from recovery.yaml", "link": "https://github.com/Panfactum/stack/commit/3d7608ca62c6a10f0216f6bc4c9c2eb514fcf414" }, { "type": "internal-docs", "summary": "Vault bootstrapping guide", "link": "/guides/bootstrapping/vault" } ], "impacts": [ { "type": "cli", "component": "cluster add", "summary": "Recovery keys written to `kube_vault/recovery.yaml` and root token to `kube_vault/secrets.yaml` immediately after `vault operator init`; read paths updated to match the new split layout" } ] }, { "id": "4dbc4497-4dc0-4dbe-a981-d0a1633f6bff", "type": "fix", "summary": "Lowers Bun DNS cache TTL during `pf cluster add` Vault ingress verification to detect DNS propagation faster.", "description": "During `pf cluster add`, the Vault ingress verification step polls the Vault health endpoint while waiting for DNS\nto propagate. Previously, Bun's default DNS cache TTL caused the polling loop to re-use stale DNS responses for an\nextended period, adding unnecessary wait time even after DNS had already propagated.\n\nThis fix sets `BUN_CONFIG_DNS_TIME_TO_LIVE_SECONDS` to `5` for the duration of the health check, ensuring the\npolling loop picks up fresh DNS records within seconds of propagation completing rather than waiting for the\ndefault cache TTL to expire.\n", "references": [ { "type": "internal-commit", "summary": "chore: lower bun dns cache ttl", "link": "https://github.com/Panfactum/stack/commit/c62f63d9eefb4549882e254f15041b7d50c2a3d2" }, { "type": "external-docs", "summary": "Bun DNS cache TTL configuration", "link": "https://bun.com/docs/runtime/networking/dns" } ], "impacts": [ { "type": "cli", "component": "cluster add", "summary": "Vault ingress verification detects DNS propagation faster by using a 5-second DNS cache TTL" } ] }, { "id": "95cc8349-fa69-46c9-a126-7f08f47dadd0", "type": "fix", "summary": "Fixes `pf cluster add` autoscaling task ordering and `bootstrap_mode_enabled` schema path in the cluster extensions step.", "description": "Two `pf cluster add` fixes from the same commit:\n\n1. The Metrics Server, VPA, Karpenter, Karpenter Node Pools, and Scheduler module deployments in the autoscaling step were wrapped in a redundant intermediate `newListr` group, which could cause steps to be hidden or run out of order. They are now direct items in the top-level task list.\n2. The cluster extensions step was reading the `bootstrap_mode_enabled` flag from the top level of `aws_eks/module.yaml` instead of the correct `extra_inputs` nested object. The incorrect schema caused the node pool adjustment skip-check to always fail to find the flag, potentially re-running EKS node pool adjustment on every install resume.\n", "references": [ { "type": "internal-commit", "summary": "fix: unneeded nesting & validation schema", "link": "https://github.com/Panfactum/stack/commit/67f55056af01643193d3d24ddd70976932112642" }, { "type": "internal-docs", "summary": "Bootstrapping autoscaling guide", "link": "/docs/main/guides/bootstrapping/autoscaling" } ], "impacts": [ { "type": "cli", "component": "cluster add", "summary": "Autoscaling step task hierarchy flattened; `bootstrap_mode_enabled` now read from correct `extra_inputs` schema path" } ] }, { "id": "b74c37ff-aeb3-4ab8-92b3-b9ee99255fb5", "type": "fix", "summary": "Fixes `pf cluster add` Vault setup where `.pf.yaml` error status writes failed silently if the file already existed.", "description": "During `pf cluster add`, the Vault operator init and Vault unseal tasks write an error status to `.pf.yaml` when\nthey fail so the CLI can skip already-completed steps on retry. However, `writeYAMLFile` was called without\n`overwrite: true`, so if `.pf.yaml` already existed from a previous run the write would fail silently. This left\nstale status data in place and could cause the CLI to incorrectly skip or re-run steps on subsequent attempts.\n\nThe `overwrite: true` flag is now passed in all three `writeYAMLFile` error-handling paths in `setupVault.ts`,\nensuring the error status is always recorded correctly when retrying after a failure.\n", "references": [ { "type": "internal-commit", "summary": "Merge certificates modules (#353)", "link": "https://github.com/Panfactum/stack/commit/48d67a2721e3df5e482db5e975bc76722451dd2a" }, { "type": "external-commit", "summary": "Merge certificates modules (PR #353)", "link": "https://github.com/Panfactum/stack/pull/353" } ], "impacts": [ { "type": "cli", "component": "cluster add", "summary": "Vault init and unseal error status now correctly written to `.pf.yaml` on retry" } ] }, { "id": "82b741d8-a599-41d0-b336-bbcb2fe04408", "type": "fix", "summary": "Fixes domain name interpolation bugs in `pf domain add` that displayed raw variable names instead of the actual domain.", "description": "Two interpolation bugs in `pf domain add` caused raw variable references to appear instead of the resolved domain name:\n\n1. In the \"already registered\" flow within `command.ts`, a plain string used `${domain}` (referencing the raw CLI option) instead of `${newDomain}` (the resolved domain). Users saw the unresolved option variable in the subdomain suggestion message.\n2. In `manualZoneSetup.ts`, the NS record instruction block used a regular string instead of a template literal, causing the literal text `${domain}` to be displayed rather than the actual domain name.\n\nThe fix also adds labeled fields (Record Type, Record Name, and Record Values) to the NS record output, making the registrar update instructions clearer.\n", "references": [ { "type": "internal-commit", "summary": "fix:cli: domain interpolation issues", "link": "https://github.com/Panfactum/stack/commit/b0bfa5f7433a9abeaf680a361a8092fd257eea46" } ], "impacts": [ { "type": "cli", "component": "domain add", "summary": "Domain name now correctly interpolated in NS record instructions and already-registered message" } ] }, { "id": "d3961452-f3da-4aca-888f-576d2f22a3f8", "type": "fix", "summary": "Fixes a circular dependency and apply-order race condition in `kube_certificates` by using direct resource attribute references in `aws_permissions`.", "description": "The `aws_permissions` submodule in `kube_certificates` previously received the cert-manager service account name and namespace through input variables (`var.service_account` and `var.namespace`). Because these values did not create a Terraform dependency on the actual `kubernetes_service_account.cert_manager` resource, the module could be applied before the service account existed, causing a race condition. An intermediate fix added an explicit `depends_on`, but this introduced a circular dependency. The final fix replaces both the input variable references and the `depends_on` with direct attribute references to `kubernetes_service_account.cert_manager.metadata[0].name` and `local.namespace`, which creates an implicit dependency graph that Terraform resolves correctly.", "references": [ { "type": "internal-commit", "summary": "fix: permissions after service account is created", "link": "https://github.com/Panfactum/stack/commit/463dcb5a14dd40263c0ee4ae0ec090382ddcc19d" }, { "type": "internal-commit", "summary": "fix: correct the circular dependency", "link": "https://github.com/Panfactum/stack/commit/cb8ec840f552abf60423394febdaaba48312714c" }, { "type": "internal-docs", "summary": "`kube_certificates` module documentation", "link": "/docs/main/modules/kube_certificates" } ], "impacts": [ { "type": "iac-module", "component": "kube_certificates", "summary": "`aws_permissions` module now reads `service_account` and `service_account_namespace` directly from the `kubernetes_service_account.cert_manager` resource instead of using input variables, creating an implicit dependency that eliminates the circular dependency and the need for an explicit `depends_on`." } ] }, { "id": "162dffd8-1294-483f-9da5-9ba26eab52d5", "type": "fix", "summary": "Fixes AWS SDK credential-loading failures in `pf env add` and `pf domain add`, and corrects `pf welcome` installer display.", "description": "Two fixes from the same commit:\n\n1. The AWS SDK v3 has a known bug where credential providers do not reload credentials written to the credentials file during the current process lifetime. Client construction is now centralized into dedicated factory functions (`getAccountClient`, `getIAMClient`, `getS3Client`, `getRoute53DomainsClient`, `getOrganizationsClient`, `getSTSClient`) that each read credentials from disk when available, bypassing the provider cache and resolving credential failures across `pf env add` and `pf domain add` workflows.\n2. `pf welcome` no longer displays onboarding content when invoked from `install.sh` before an `.envrc` file is present. Previously the command only checked for a missing `installation_id`; the screen is now suppressed when `.envrc` does not yet exist.\n", "references": [ { "type": "internal-commit", "summary": "fix:cli: aws client fixes + welcome screen + org create success section", "link": "https://github.com/Panfactum/stack/commit/572dddd507d8318b9ebd48dedc47f4351f73626d" }, { "type": "internal-commit", "summary": "fix: welcome screen on install", "link": "https://github.com/Panfactum/stack/commit/d5736a893be8a7c8245d73f075a8e4620d018a87" }, { "type": "issue-report", "summary": "AWS SDK JS v3 credential providers do not reload credentials from disk within the same process", "link": "https://github.com/aws/aws-sdk-js-v3/issues/6872" } ], "impacts": [ { "type": "cli", "component": "env add", "summary": "AWS SDK clients now use a centralized `createAWSClient` factory that reads credentials from disk on each call, bypassing the provider cache" }, { "type": "cli", "component": "domain add", "summary": "`Route53DomainsClient` and `AccountClient` now use the centralized credentials-from-file factory, resolving authentication failures during domain registration" }, { "type": "cli", "component": "welcome", "summary": "Welcome screen is suppressed when `.envrc` is absent, preventing onboarding content from displaying during initial `install.sh` execution" } ] }, { "id": "cc5f5d14-0d27-40d6-9072-f5e778f87ace", "type": "addition", "summary": "`pf env add` displays a success message after bootstrapping the management environment.", "description": "After the AWS Organization management environment is bootstrapped, `pf env add` now presents a\nsuccess message that orients the user before continuing. The message explains:\n\n1. That the AWS Organization has been configured\n2. Where the management environment IaC files are stored (the `environments/management` directory)\n3. That the management environment is a special environment for global settings\n4. That the installer will now proceed to add a normal environment\n\nThis reduces confusion during first-time setup by clearly distinguishing the management\nenvironment bootstrap from the subsequent normal environment creation.\n", "references": [ { "type": "internal-commit", "summary": "fix:cli: aws client fixes + welcome screen + org create success section", "link": "https://github.com/Panfactum/stack/commit/572dddd507d8318b9ebd48dedc47f4351f73626d" } ], "impacts": [ { "type": "cli", "component": "env add", "summary": "Displays a success message after the management environment bootstrap explaining its purpose, IaC file location, and next steps." } ] }, { "id": "4c58208b-e9a1-48e0-8da4-f0f1d312fe27", "type": "fix", "summary": "Fixes `pf cluster add` EKS step to exclude SSO ARNs from `extra_superuser_principal_arns` and Vault pod readiness check to pass `--context`.", "description": "Two fixes for SSO-authenticated `pf cluster add` runs:\n\n1. The EKS setup step was including the caller's SSO assumed-role ARN in `extra_superuser_principal_arns`. These ARNs contain `AWSReservedSSO` and are transient, so they cannot be used as persistent EKS access entry principals — they are now excluded.\n2. The `kubectl get pods` command used to check Vault pod readiness was missing the `--context` flag, causing it to target the wrong cluster. The step now validates that `kubeConfigContext` is present and passes it explicitly.\n", "references": [ { "type": "internal-commit", "summary": "Fix sso cluster add (#357)", "link": "https://github.com/Panfactum/stack/commit/ebbc5fa198626b320fcc6ef02d5f18a6d72932a5" }, { "type": "external-commit", "summary": "Pull request: Fix sso cluster add (#357)", "link": "https://github.com/Panfactum/stack/pull/357" } ], "impacts": [ { "type": "cli", "component": "cluster add", "summary": "EKS setup excludes transient SSO assumed-role ARNs from `extra_superuser_principal_arns`; Vault pod readiness check now passes `--context` to `kubectl`" } ] }, { "id": "e214742c-32e8-4a9a-a08f-5124a525ba8a", "type": "addition", "summary": "`pf config get` now reads SOPS-encrypted `*.secrets.yaml` files, corrects config load order, and handles missing `secrets.yaml` in `pf cluster add`.", "description": "Two related `pf config get` changes from the same commit:\n\n1. The config file load order was inverted — module-level files were loaded before global-level files, causing global values to incorrectly override module-level values. The order is now correct: `global.yaml`, `environment.yaml`, `region.yaml`, `module.yaml` (with `*.user.yaml` and `*.secrets.yaml` variants at each level). SOPS-encrypted `*.secrets.yaml` files are now decrypted on-demand via `sopsDecrypt` and merged in the same precedence order.\n\n2. Multiple `pf cluster add` steps (Autoscaling, Certificates, Cluster Extensions, Inbound Networking, Linkerd, VPC/ECR, Vault) would crash with a destructuring error when `secrets.yaml` did not yet exist. The `sopsDecrypt` utility now returns `null` instead of throwing for missing files, and each step handles `null` gracefully.\n", "references": [ { "type": "internal-commit", "summary": "fix:cli: config file precedence and secret config files", "link": "https://github.com/Panfactum/stack/commit/474c4b6ddd5d82f1bcacf43fe564cff67c9cd890" }, { "type": "internal-docs", "summary": "Terragrunt variables and config file hierarchy", "link": "/docs/main/reference/configuration/terragrunt-variables" }, { "type": "external-docs", "summary": "SOPS — Secrets OPerationS encryption tool", "link": "https://getsops.io/docs/" } ], "impacts": [ { "type": "cli", "component": "config get", "summary": "Load order corrected (global, environment, region, module); SOPS-encrypted `*.secrets.yaml` files now decrypted and merged in the same precedence order" }, { "type": "cli", "component": "cluster add", "summary": "Steps that read the Vault root token now handle a missing `secrets.yaml` gracefully instead of crashing with a destructuring error" }, { "type": "configuration", "component": "global.secrets.yaml", "summary": "New SOPS-encrypted config file read by `pf config get` at the global level" }, { "type": "configuration", "component": "environment.secrets.yaml", "summary": "New SOPS-encrypted config file read by `pf config get` at the environment level" }, { "type": "configuration", "component": "region.secrets.yaml", "summary": "New SOPS-encrypted config file read by `pf config get` at the region level" }, { "type": "configuration", "component": "module.secrets.yaml", "summary": "New SOPS-encrypted config file read by `pf config get` at the module level" } ] }, { "id": "71c55c5e-6cbf-4adb-a20b-590d07af0595", "type": "fix", "summary": "Fixes `pf domain add` to show context-appropriate DNS setup instructions for apex domains versus subdomains.", "description": "Previously, `pf domain add` showed the same registrar-targeted nameserver instructions for all domains regardless of whether they were apex or subdomain zones. This was misleading for subdomain users who needed to add NS records at their existing DNS host rather than change nameservers at their registrar.\n\nThe fix passes an `isApex` flag through to the `manualZoneSetup` function so that the interactive prompt now branches:\n\n- **Apex domains**: directs users to update nameservers at their domain registrar, with an example link to Namecheap's nameserver configuration guide\n- **Non-apex subdomains**: directs users to add NS records at their DNS host, with an example link to Namecheap's host record setup guide\n", "references": [ { "type": "internal-commit", "summary": "Adjust message for apex vs non-apex manual NS setup", "link": "https://github.com/Panfactum/stack/commit/2bc2ae52e6ab29ef1def3f01474e11242058e09f" }, { "type": "internal-docs", "summary": "DNS bootstrapping guide", "link": "/docs/main/guides/bootstrapping/dns" }, { "type": "internal-docs", "summary": "Subdomain delegation networking concept", "link": "/docs/main/concepts/networking/subdomain-delegation" } ], "impacts": [ { "type": "cli", "component": "domain add", "summary": "Manual DNS zone setup step now shows registrar-specific nameserver instructions for apex domains and DNS-host NS record instructions for subdomains" } ] }, { "id": "e2ecc5ef-e611-46d7-a9c5-561afc0acbbd", "type": "addition", "summary": "Adds the `pf sso add` CLI command for guided Authentik and AWS federated SSO installation.", "description": "Setting up Authentik and AWS IAM Identity Center SSO previously required manually deploying\nand configuring multiple infrastructure modules in the correct order. The new `pf sso add`\ncommand automates this entire workflow with a guided wizard that handles:\n\n1. Deploying `aws_ses_domain` for email delivery\n2. Deploying `kube_authentik` with organization branding and email templates\n3. Deploying `authentik_core_resources` with user groups, admin accounts, and recovery flows\n4. Verifying the Authentik ingress is reachable\n5. Configuring AWS IAM Identity Center federation with `authentik_aws_sso`\n6. Setting up `aws_iam_identity_center_permissions` with role-based access per environment\n\nThis reduces the SSO bootstrapping process from a multi-step manual procedure to a single\ninteractive command with built-in resumability.\n", "references": [ { "type": "internal-commit", "summary": "Authentik SSO in CLI installer (#359)", "link": "https://github.com/Panfactum/stack/commit/a3febb0f7ea71f7eb58968aac0223e6360264465" }, { "type": "external-docs", "summary": "Authentik identity provider documentation", "link": "https://docs.goauthentik.io/" } ], "impacts": [ { "type": "cli", "component": "sso add", "summary": "New command that automates Authentik deployment and AWS federated SSO configuration" }, { "type": "iac-module", "component": "kube_authentik", "summary": "Adds `organization_name` input and moves email template management into the module" }, { "type": "iac-module", "component": "authentik_core_resources", "summary": "Removes email template resources (moved to `kube_authentik`) and adjusts branding" } ] }, { "id": "1e6bdd79-0818-4c38-a586-c716a8643e31", "type": "addition", "summary": "Adds `pf iac update-module-status` to track module `init_status` and `deploy_status` in `.pf.yaml`, enabling step resumability in `pf cluster add`.", "description": "Replaces the inline bash `echo` approach in `panfactum.hcl` hooks with a dedicated CLI command that records both in-progress and final status for `init` and `apply` phases. The `panfactum.hcl` Terragrunt hooks now invoke `pf iac update-module-status` at the start, completion, and failure of each phase, writing structured YAML status to `.pf.yaml`. This enables `pf cluster add` to accurately detect which modules have already been deployed and skip them on resume.\n\nThree iterative fixes were applied after the initial implementation:\n\n1. `panfactum.hcl` hooks were calling `pf update-module-status` instead of `pf iac update-module-status` -- all six hook invocations were corrected to use the proper subcommand path.\n2. The command wrote camelCase keys (`initStatus`, `deployStatus`) instead of the required snake_case (`init_status`, `deploy_status`).\n3. A broken object spread nested existing fields as a key instead of merging them. The status-merge logic was rewritten, and a `terragrunt.hcl` presence check was added to prevent accidental writes to non-module directories.\n", "references": [ { "type": "internal-commit", "summary": "feat: enhances module status architecture", "link": "https://github.com/Panfactum/stack/commit/00029f4e98aec9c1119bba2ff2a7fcb2d228620b" }, { "type": "internal-commit", "summary": "feat: misc cli improvements", "link": "https://github.com/Panfactum/stack/commit/e19f5da99390eed58ae797f556578af89d98532f" }, { "type": "internal-commit", "summary": "fix: module status update", "link": "https://github.com/Panfactum/stack/commit/eaa53405e53cf5eb0926a18daaa6561ea729030c" }, { "type": "internal-commit", "summary": "fix: module status update", "link": "https://github.com/Panfactum/stack/commit/0ac254981c3829dc40dc6453e0bd355cd187747d" } ], "impacts": [ { "type": "cli", "component": "iac update-module-status", "summary": "New command that writes `init_status` and `deploy_status` to a module's `.pf.yaml` file" }, { "type": "configuration", "component": "panfactum.hcl", "summary": "Terragrunt hooks replaced inline `bash` status writes with `pf iac update-module-status` invocations for `init` and `apply` phases" }, { "type": "cli", "component": "cluster add", "summary": "Module resume logic now reads `deploy_status` from `.pf.yaml` instead of the previous `status` field" } ] }, { "id": "3888baee-3716-43e0-8688-7cadb552f367", "type": "fix", "summary": "Fixes `pf devshell sync` incorrectly requiring `.pf.yaml` in the environments `.gitignore`.", "description": "The `EXPECTED_GITIGNORE_CONTENTS` for the environments directory previously included `MODULE_STATUS_FILE` (which resolved to `.pf.yaml`). Since module status tracking has been moved out of the environments directory, this entry was no longer valid and caused `pf devshell sync` to flag the environments `.gitignore` as out of date even when it was correctly configured.", "references": [ { "type": "internal-commit", "summary": "feat: misc cli improvements", "link": "https://github.com/Panfactum/stack/commit/e19f5da99390eed58ae797f556578af89d98532f" } ], "impacts": [ { "type": "cli", "component": "devshell sync", "summary": "Removes `.pf.yaml` from the expected environments `.gitignore` entries" } ] }, { "id": "132df601-d683-459d-803e-ea45e5498a39", "type": "fix", "summary": "Fixes `pf env add` failing on resume because the state bucket name was missing from the task context.", "description": "The `Generate unique state bucket name` task previously used a `skip` condition to bypass execution entirely when `tf_state_bucket` was already set in the Panfactum config. While this avoided re-generating a new bucket name on resume, it also meant that `ctx.bucketName` and `ctx.locktableName` were never assigned, causing downstream tasks (such as deploying `tf_bootstrap_resources`) to receive `undefined` resource IDs and fail.\n\nThe fix removes the `skip` condition and instead populates the context values from the existing config at the start of the task body, so the bucket name flows correctly to all subsequent steps whether this is a fresh run or a resumed one.\n", "references": [ { "type": "internal-commit", "summary": "fix: bootstrap env resume", "link": "https://github.com/Panfactum/stack/commit/6a17572cde08a016348a43d09b3e9c1ba014aabc" } ], "impacts": [ { "type": "cli", "component": "env add", "summary": "State bucket and lock table names are now populated in context even when resuming an interrupted setup" } ] }, { "id": "06b8e872-fa6c-4e9b-b571-416f37bc5e0f", "type": "breaking_change", "summary": "`aws_organization` now requires a new `alias` input to manage the IAM account alias for the management AWS account.", "description": "Previously, the IAM account alias was only set through `aws_account` for non-management environments. The management account had no equivalent mechanism, leaving it without a human-readable alias. This change brings the management account into alignment by:\n\n1. Adding a required `alias` variable to the `aws_organization` module that creates an `aws_iam_account_alias` resource.\n2. Updating `pf env add` to forward the alias collected during environment setup into the `aws_organization` deployment step for the management environment.\n", "action_items": [ "Add an `alias` input to your `aws_organization` module configuration with a human-readable string for the management account (e.g., `my-org-management`).", "Run `terragrunt apply` on your `aws_organization` deployment after adding the new input." ], "references": [ { "type": "internal-commit", "summary": "fix: set account alias in aws_organization module", "link": "https://github.com/Panfactum/stack/commit/fd8d7b26d3cffcebc0b3d37491cb92a76dc7d016" }, { "type": "internal-docs", "summary": "`aws_organization` module documentation", "link": "/docs/main/reference/infrastructure-modules/aws/aws_organization" } ], "impacts": [ { "type": "iac-module", "component": "aws_organization", "summary": "New required `alias` string input — management account alias is now managed via `aws_iam_account_alias`" }, { "type": "cli", "component": "env add", "summary": "Account alias collected during setup is now forwarded to the `aws_organization` module via `defineInputUpdate`" } ] }, { "id": "680b6e2a-958c-4379-b2b2-6a3b120178ca", "type": "improvement", "summary": "`pf env add` now supports resuming interrupted installations and fixes phone number formatting.", "description": "Two improvements to the `pf env add` command:\n\n1. **Resumability** -- `pf env add` can now resume an interrupted installation without repeating completed steps. The command detects whether an AWS account was already provisioned, restores previously-selected primary and secondary regions from config, and offers to copy region configuration from another existing environment. AWS account creation is now performed directly via the AWS SDK before the IaC module is applied, enabling retry with a different email on duplicate-email errors from AWS Organizations.\n2. **Phone number formatting** -- The phone number pre-fill fetched from the AWS Account API is now formatted with dashes (e.g., `555-555-5555`) instead of dots, matching the expected input format.\n", "references": [ { "type": "internal-commit", "summary": "feat:cli: env add resumability", "link": "https://github.com/Panfactum/stack/commit/e463425f6558d3105371d21787b3a1884414109e" }, { "type": "internal-docs", "summary": "Bootstrapping guide for preparing AWS accounts", "link": "/docs/main/guides/bootstrapping/preparing-aws" } ], "impacts": [ { "type": "cli", "component": "env add", "summary": "Detects partially-deployed environments, skips completed provisioning steps, restores previously-selected regions, offers to copy region config from another environment, and creates AWS accounts directly via the SDK with email-retry support; also fixes phone number dash formatting" } ] }, { "id": "b40deacb-6f87-4522-8f9f-03fc413c7346", "type": "improvement", "summary": "`pf cluster add` post-installation success message is streamlined and reformatted with blank lines between numbered steps for better readability.", "description": "The success message displayed after `pf cluster add` completes was difficult to scan because the numbered `k9s` verification steps ran together without visual separation. This change adds blank lines between each step and removes two unnecessary steps (checking for running pods and contacting Discord), reducing the list from seven steps to five. The result is a cleaner, more focused set of instructions that guides users through verifying their new cluster with `k9s`.", "references": [ { "type": "internal-commit", "summary": "fix:cli:cluster add: success message format", "link": "https://github.com/Panfactum/stack/commit/4f0f981677d36b1cb2269a9b4e81287fdbab5217" }, { "type": "internal-docs", "summary": "Kubernetes cluster bootstrapping guide", "link": "/docs/main/guides/bootstrapping/kubernetes-cluster" } ], "impacts": [ { "type": "cli", "component": "cluster add", "summary": "Post-installation `k9s` verification steps reformatted with blank lines and reduced from seven to five" } ] }, { "id": "9b7fcf1c-efab-4188-b5c1-b15ef7cc3f5f", "type": "improvement", "summary": "`pf env add` now suggests an environment-specific IAM username instead of the hardcoded `pf-bootstrap-user` name.", "description": "Previously, `pf env add` always suggested `pf-bootstrap-user` as the IAM username when walking users through creating a new AWS account. This made it difficult to identify which account a given IAM user belonged to when viewing the IAM console.\n\nThe suggested username is now derived from the environment name using the `-superuser` pattern (e.g., `management-superuser`). The access key creation URL is also updated to reference the derived username, so the entire guided setup flow stays consistent. This aligns the installer with Panfactum's existing AWS profile naming convention documented in the bootstrapping guide.\n", "references": [ { "type": "internal-commit", "summary": "Derive bootstrap IAM username from environment name", "link": "https://github.com/Panfactum/stack/commit/0886f756c32dabcfb9ee647b991a3cb4e247d225" }, { "type": "internal-docs", "summary": "Bootstrapping guide documenting the `-superuser` AWS profile convention", "link": "/docs/main/guides/bootstrapping/preparing-aws" } ], "impacts": [ { "type": "cli", "component": "env add", "summary": "Suggests `-superuser` as the IAM username and uses it in the access key creation URL" } ] }, { "id": "38bd713e-571d-4337-a656-a632de3b993d", "type": "fix", "summary": "Fixes CLI commands failing to update existing SOPS-encrypted files when merging new secret values.", "description": "The internal `sopsUpsert` utility merges new key/value pairs into an existing SOPS-encrypted file by decrypting the current contents, spreading in the new values, and re-encrypting the result. However, the final `sopsWrite` call was missing `overwrite: true`, causing it to throw an error whenever the target file already existed on disk.\n\nThis meant that any CLI command relying on `sopsUpsert` to update secrets -- including `pf cluster add`, `pf cluster enable`, and `pf sso add` -- would fail on the second run or whenever the secrets file had been previously created. The fix passes `overwrite: true` in the existing-file code path, ensuring the merged result is written back correctly on every invocation.\n", "references": [ { "type": "internal-commit", "summary": "fix: set overwrite to true for `sopsUpsert`", "link": "https://github.com/Panfactum/stack/commit/f33f0f128c2b05ca31e10fecf2d0be3dd05d1ac2" } ], "impacts": [ { "type": "cli", "component": "cluster add", "summary": "SOPS secret upserts during Vault and inbound networking setup no longer fail on existing files" }, { "type": "cli", "component": "cluster enable", "summary": "SOPS secret upserts during ECR setup no longer fail on existing files" }, { "type": "cli", "component": "sso add", "summary": "SOPS secret upserts during federated auth and Authentik setup no longer fail on existing files" } ] }, { "id": "815fbc87-b244-4c09-ad52-67c1009a0aa2", "type": "improvement", "summary": "`panfactum.hcl` auto-injects `route53_zones` and `kube_domain` with safe defaults, and `kube_certificates` makes `vault_internal_url` optional.", "description": "Two improvements that simplify migrating to `kube_certificates`:\n\n1. `panfactum.hcl` now injects `route53_zones` (from `local.vars.domains`) and `kube_domain` (from `local.vars.kube_domain`) as global default module inputs using `lookup()` with safe defaults (`{}` and `null` respectively). Modules receive these values automatically without manual wiring, and the template works in environments where these keys are absent from the configuration.\n2. `vault_internal_url` in `kube_certificates` defaults to `http://vault-active.vault.svc.cluster.local:8200`, making it optional for standard Panfactum layouts. Users no longer need to explicitly provide the Vault URL when deploying this module in a cluster that follows the default Panfactum conventions.\n", "references": [ { "type": "internal-commit", "summary": "fix: kube_certificates migration guide", "link": "https://github.com/Panfactum/stack/commit/8dddf6bf63532b62b4cf55e7b598421a645b1643" }, { "type": "internal-commit", "summary": "fix: kube_domain default input", "link": "https://github.com/Panfactum/stack/commit/d14a548a5cd98b7e2bfee632bbe118459be04b6a" }, { "type": "internal-docs", "summary": "`kube_certificates` module reference", "link": "/docs/main/reference/infrastructure-modules/kubernetes/kube_certificates" } ], "impacts": [ { "type": "iac-module", "component": "kube_certificates", "summary": "`vault_internal_url` now defaults to `http://vault-active.vault.svc.cluster.local:8200`, making the input optional for standard Panfactum cluster layouts." }, { "type": "configuration", "component": "panfactum.hcl", "summary": "Injects `route53_zones` and `kube_domain` as global default module inputs using `lookup()` with safe defaults (`{}` and `null` respectively)." } ] }, { "id": "4285e484-36fe-4a55-acb7-454bcd6b45db", "type": "improvement", "summary": "`kube_opensearch` adds opt-in Dashboards, re-enables S3 remote storage, switches to static IAM credentials, and sizes JVM heap dynamically.", "description": "Four improvements to the `kube_opensearch` module:\n\n1. **OpenSearch Dashboards** can now be deployed alongside the cluster. Enable via `dashboard_enabled` (default `false`); when enabled, `dashboard_domain` configures the ingress hostname. New inputs `log_level`, `slow_request_log_thresholds`, and `extra_cluster_settings` provide additional cluster tuning.\n2. **S3 remote storage re-enabled.** `node.attr.remote_store.segment.repository` and `translog.repository` were previously disabled due to upstream bug [OpenSearch#15902](https://github.com/opensearch-project/OpenSearch/issues/15902). This is resolved in OpenSearch 3.x, so both are now re-enabled for S3-backed segment replication and recovery.\n3. **Static IAM credentials for the S3 plugin.** The S3 repository plugin cannot use the web identity token required by IRSA ([OpenSearch#16523](https://github.com/opensearch-project/OpenSearch/issues/16523)). The module now uses `kube_aws_creds` to provision short-lived static IAM access keys (rotated every 14 days) injected via the OpenSearch keystore.\n4. **Dynamic JVM heap sizing.** The JVM heap was previously hardcoded to `-Xmx512M -Xms512M`. The entrypoint script now reads `CONTAINER_MEMORY_REQUEST` and sets the heap to 50% of the request, keeping JVM memory proportional to the VPA-adjusted pod allocation.\n", "references": [ { "type": "internal-commit", "summary": "feat: adds opensearch dashboards, remote storage, and v3", "link": "https://github.com/Panfactum/stack/commit/55a2cd5eea66664a7994f5841abda062d982fe1e" }, { "type": "issue-report", "summary": "OpenSearch issue #15902 (segment replication remote store)", "link": "https://github.com/opensearch-project/OpenSearch/issues/15902" }, { "type": "issue-report", "summary": "OpenSearch issue #16523 (IRSA web identity token not usable with S3 plugin)", "link": "https://github.com/opensearch-project/OpenSearch/issues/16523" }, { "type": "internal-docs", "summary": "`kube_opensearch` module documentation", "link": "/docs/main/modules/kube_opensearch" }, { "type": "external-docs", "summary": "OpenSearch official documentation", "link": "https://docs.opensearch.org/" } ], "impacts": [ { "type": "iac-module", "component": "kube_opensearch", "summary": "Adds opt-in OpenSearch Dashboards via `dashboard_enabled`, re-enables S3 segment and translog remote storage, switches S3 plugin authentication to static IAM credentials via `kube_aws_creds`, and sizes JVM heap dynamically to 50% of the container memory request." } ] }, { "id": "07ae1274-c0e5-484f-be87-1b8ea0f4995b", "type": "improvement", "summary": "`pf cluster add` and `pf sso add` now skip `terragrunt init` for already-initialized modules, reducing redundant work during installation.", "description": "Previously, every module deployment task in `pf cluster add` and `pf sso add` ran `terragrunt init` unconditionally before applying, even on resumed installs where the module had already been initialized. This added unnecessary network round-trips and slowed down cluster setup, especially when resuming a partially completed installation.\n\nThe `buildDeployModuleTask` utility now checks the module's `.pf.yaml` `init_status` field (written by the Panfactum terragrunt hooks) before deciding whether to run init. If `init_status` is `success` or `running`, the init step is skipped entirely. A new `forceInitModule` option can be set to override this behavior when a re-init is explicitly needed.\n\nThe old `initModule` boolean parameter has been removed from the `buildDeployModuleTask` interface and replaced with the automatic `init_status` check, so callers no longer need to specify whether init should run.\n", "references": [ { "type": "internal-commit", "summary": "feat: intelligently skip init in deployModuleTask when possible", "link": "https://github.com/Panfactum/stack/commit/084e5910a3bad7d214090c78cca30f8caf650036" } ], "impacts": [ { "type": "cli", "component": "cluster add", "summary": "Module deploy tasks skip `terragrunt init` when the module's `init_status` is already `success` or `running`, speeding up resumed installs" }, { "type": "cli", "component": "sso add", "summary": "Module deploy tasks skip `terragrunt init` when the module's `init_status` is already `success` or `running`, speeding up resumed installs" } ] }, { "id": "ec8f3272-b5cf-4657-a4e3-ff2a7082e712", "type": "fix", "summary": "Fixes `pf cluster add` Vault unseal skip check to query live seal status instead of a stale configuration flag.", "description": "The `pf cluster add` Vault setup step previously decided whether to skip unsealing by reading the\n`extra_inputs.wait` flag from `kube_vault/module.yaml`, an indirect heuristic that could become stale\nif a prior run failed partway through. This caused the unseal step to be incorrectly skipped, leaving\nVault sealed and blocking subsequent bootstrapping tasks.\n\nThe fix replaces that heuristic with a direct query:\n\n1. The skip check now runs `vault status -format=json` via `kubectl exec` against the `vault-0` pod.\n2. Unsealing is skipped only when Vault reports `sealed: false`.\n3. The JSON response is parsed through a `parseJson` utility backed by a Zod schema, so malformed\n output produces a clear validation error instead of silently yielding `undefined`.\n", "references": [ { "type": "internal-commit", "summary": "fix: using vault status to check if unseal step should be skipped", "link": "https://github.com/Panfactum/stack/commit/a30fefc697ec9ecd01f72accdd2f83f6d3b7c413" }, { "type": "internal-commit", "summary": "lint: added schema validation on vault status", "link": "https://github.com/Panfactum/stack/commit/0c1e3abe3ebb11f8985a37394060017c83e812ad" }, { "type": "external-docs", "summary": "Vault seal-status API reference", "link": "https://developer.hashicorp.com/vault/api-docs/system/seal-status" }, { "type": "internal-docs", "summary": "Bootstrapping guide for Vault", "link": "/docs/main/guides/bootstrapping/vault" } ], "impacts": [ { "type": "cli", "component": "cluster add", "summary": "Vault unseal skip check now calls `vault status -format=json` directly instead of reading a stale `module.yaml` flag, and validates the response with a Zod schema" } ] }, { "id": "4456efe7-31d2-4f59-bc75-604e61248dfc", "type": "fix", "summary": "Fixes `pf domain add` poll title showing `undefined` and `kube_vault` KMS unseal key race condition.", "description": "Two fixes from the same commit:\n\n1. The `pf domain add` registration poll title was displaying `undefined` as the initial status label before the first status response was received. The `lastStatus` variable was initialized as `undefined`, but the old check (`lastStatus !== \"\"`) evaluated to `true` for `undefined`, causing it to be interpolated into the title string. The fix changes the check to a truthy guard so the status is omitted until an actual value is available.\n2. The `helm_release.vault` resource only listed `module.aws_permissions` in its `depends_on`, meaning Terraform could attempt to deploy Vault before the KMS unseal key (`module.unseal_key`) was ready. Adding `module.unseal_key` to `depends_on` ensures the key exists before Vault starts, preventing initialization failures caused by a missing unseal key.\n", "references": [ { "type": "internal-commit", "summary": "fix:cli: initial domain register poll title", "link": "https://github.com/Panfactum/stack/commit/ccecb3270b0384feeaeaf62cf47124942071b121" }, { "type": "internal-commit", "summary": "fix: kube_vault depends_on unseal_key to prevent deploy race condition", "link": "https://github.com/Panfactum/stack/commit/ccecb3270b0384feeaeaf62cf47124942071b121" }, { "type": "internal-docs", "summary": "`kube_vault` module documentation", "link": "/docs/main/reference/infrastructure-modules/kubernetes/kube_vault" } ], "impacts": [ { "type": "cli", "component": "domain add", "summary": "Poll title no longer shows `undefined` before the first status response is received" }, { "type": "iac-module", "component": "kube_vault", "summary": "Vault Helm release now explicitly depends on `module.unseal_key` to prevent a deploy race condition." } ] }, { "id": "775eb8af-ed0a-4cc8-bb5d-64cfc7477989", "type": "improvement", "summary": "`pf cluster add` now reads the Vault root token from `region.secrets.yaml` via `getPanfactumConfig()` instead of directly SOPS-decrypting `kube_vault/secrets.yaml`.", "description": "Previously, each `pf cluster add` setup step (autoscaling, certificates, cluster extensions, inbound networking, Linkerd, Vault) independently called `sopsDecrypt` on `kube_vault/secrets.yaml` to obtain the Vault root token. This was fragile because every step had to know the exact file path and field name (`root_token`), creating tight coupling between unrelated setup stages.\n\nThe token is now stored as `vault_token` in `region.secrets.yaml` and surfaced through the standard `getPanfactumConfig()` merge, so all steps read it from the shared `config` object. Key changes:\n\n1. The Vault initialization step writes `vault_token` to `region.secrets.yaml` instead of `kube_vault/secrets.yaml`.\n2. A `refreshConfig()` helper re-hydrates the config object after Vault initialization writes the token, ensuring downstream steps see the new value without a full restart.\n3. All six setup steps that previously called `sopsDecrypt` now read `config.vault_token` directly.\n4. The `config` object is passed through `InstallClusterStepOptions`, eliminating redundant file reads.\n", "references": [ { "type": "internal-commit", "summary": "moved `vault_token` to `region.secrets.yaml`", "link": "https://github.com/Panfactum/stack/commit/829ad02cd9296a82c8918338fdefbdafd5ca2fae" }, { "type": "internal-docs", "summary": "Bootstrapping guide for Vault setup", "link": "/docs/main/guides/bootstrapping/vault" } ], "impacts": [ { "type": "cli", "component": "cluster add", "summary": "Vault root token is now stored as `vault_token` in `region.secrets.yaml` and read through the merged `getPanfactumConfig()` result instead of per-step `sopsDecrypt` calls." }, { "type": "cli", "component": "config get", "summary": "`vault_token` is now a recognized field in the Panfactum config schema and appears in `pf config get` output." } ] }, { "id": "73830a77-9505-4a9b-b0d6-eb852516a8e2", "type": "addition", "summary": "`pf sso add` now automates federated auth with IAM Identity Center and adds `authentik_token` to the Panfactum config schema.", "description": "Two related additions from the federated auth commit:\n\n1. `pf sso add` now includes a \"Setup Federated Auth\" step that automates the full federated auth workflow with AWS IAM Identity Center — deploying `authentik_aws_sso`, collecting SCIM credentials, syncing users, deploying `aws_iam_identity_center_permissions`, and updating EKS. Each sub-step is resumable via `.pf.yaml` checkpoints.\n2. `authentik_token` is now part of `PANFACTUM_CONFIG_SCHEMA`. The bundled `authentik.tftpl` provider template passes both `url` and `token` to the Authentik provider automatically when `authentik_token` is present in the merged config.\n", "references": [ { "type": "internal-commit", "summary": "Federated auth (#360)", "link": "https://github.com/Panfactum/stack/commit/daaa3b2f592d21ef0990170d7efd92bcc13c7d96" }, { "type": "external-commit", "summary": "Federated auth PR #360", "link": "https://github.com/Panfactum/stack/pull/360" }, { "type": "internal-docs", "summary": "Federated auth bootstrapping guide", "link": "/docs/main/guides/bootstrapping/federated-auth" }, { "type": "internal-docs", "summary": "Identity provider bootstrapping guide", "link": "/docs/main/guides/bootstrapping/identity-provider" } ], "impacts": [ { "type": "cli", "component": "sso add", "summary": "Federated auth (AWS SSO SAML + SCIM) is now automated as a resumable step within `pf sso add`" }, { "type": "cli", "component": "config get", "summary": "`authentik_token` is now a recognized field in the Panfactum config schema" }, { "type": "configuration", "component": "panfactum.hcl", "summary": "The `authentik_provider` generated block now passes `authentik_token` from the config to the Authentik provider" } ] }, { "id": "197c376f-a5b6-4d0e-b109-8d87ecc9065d", "type": "breaking_change", "summary": "The Authentik API token stored during `pf sso add` has been moved from `authentik_core_resources/secrets.yaml` to `region.secrets.yaml` under the key `authentik_token`.", "description": "The `pf sso add` command previously stored the Authentik API token as `authentikUserToken` inside the module-level `authentik_core_resources/secrets.yaml` file. This change relocates the token to the region-level `region.secrets.yaml` under the standardized key `authentik_token`, aligning it with other region-scoped secrets such as `vault_token`.\n\nAs part of this change:\n\n1. The `authentik_token` field has been added to `PANFACTUM_CONFIG_SCHEMA` in `schemas.ts`.\n2. The `panfactum.hcl` Terragrunt configuration now reads `authentik_token` from local variables and passes it directly to the Authentik Terraform provider via `authentik.tftpl`.\n3. Modules that use the Authentik provider authenticate automatically without manual environment variable configuration.\n\nThis consolidation simplifies secret management by keeping all provider tokens at the region level, making it easier to share credentials across modules within the same region.\n", "action_items": [ "Copy your existing Authentik token from `/authentik_core_resources/secrets.yaml` (key `authentikUserToken`) to `/region.secrets.yaml` (key `authentik_token`) using SOPS: `sops --set '[\"authentik_token\"] \"your-token-here\"' region.secrets.yaml`.", "Verify that `authentik_token` is present in `region.secrets.yaml` (SOPS-encrypted) before running any Terragrunt modules that use the Authentik provider, as the token is now loaded automatically via `panfactum.hcl`." ], "references": [ { "type": "internal-commit", "summary": "Federated auth (#360)", "link": "https://github.com/Panfactum/stack/commit/daaa3b2f592d21ef0990170d7efd92bcc13c7d96" }, { "type": "internal-commit", "summary": "Switch to `config.authentik_token` instead of module output", "link": "https://github.com/Panfactum/stack/commit/27cdbcf16525e9989430f33ef211b5a3f9e17560" }, { "type": "internal-docs", "summary": "Terragrunt variables configuration reference", "link": "/docs/main/reference/configuration/terragrunt-variables" } ], "impacts": [ { "type": "cli", "component": "sso add", "summary": "Stores the Authentik API token as `authentik_token` in `region.secrets.yaml` instead of `authentikUserToken` in `authentik_core_resources/secrets.yaml`" }, { "type": "configuration", "component": "region.secrets.yaml", "summary": "New `authentik_token` key is required for the Authentik Terraform provider to authenticate" }, { "type": "configuration", "component": "panfactum.hcl", "summary": "Reads `authentik_token` from Terragrunt variables and passes it directly to the Authentik provider, removing the need for manual environment variable injection" } ] }, { "id": "7eb61140-4687-4f1a-a213-28fccda34cc0", "type": "improvement", "summary": "`pf cluster add` now sources the Vault token automatically via `panfactum.hcl` instead of manually threading it through each setup step.", "description": "Previously, `pf cluster add` read `vault_token` from an in-memory config object and injected it as a `VAULT_TOKEN` environment variable for every Terragrunt module deployment across all setup steps (autoscaling, certificates, cluster extensions, inbound networking, Linkerd, and Vault core resources). This required the CLI to:\n\n1. Carry a mutable `config` reference on `InstallClusterStepOptions`\n2. Call `refreshConfig` after Vault initialization so later steps could access the freshly-written token\n3. Explicitly set `VAULT_TOKEN` in the `env` block of every `buildDeployModuleTask` call\n\nThis approach was fragile and created tight coupling between the CLI orchestration logic and Vault credential management. The `refreshConfig` helper and the `config` field on `InstallClusterStepOptions` have been removed. Instead, `panfactum.hcl` now reads `local.vars.vault_token` directly, so once the token is persisted to `region.secrets.yaml` it is picked up automatically by all subsequent Terragrunt invocations without any explicit environment variable injection.\n", "references": [ { "type": "internal-commit", "summary": "Moved `vault_token` to autoload in `panfactum.hcl`", "link": "https://github.com/Panfactum/stack/commit/ed1a146dd4c4d3d92866ac369bb653ce9d9f0d19" }, { "type": "internal-docs", "summary": "Terragrunt variables reference documenting `panfactum.hcl` configuration", "link": "/docs/main/reference/configuration/terragrunt-variables" } ], "impacts": [ { "type": "cli", "component": "cluster add", "summary": "The `config` field and `refreshConfig` helper have been removed from `InstallClusterStepOptions`. The Vault root token is no longer injected as a `VAULT_TOKEN` environment variable; Terragrunt reads it automatically from `region.secrets.yaml` via `panfactum.hcl`." }, { "type": "configuration", "component": "panfactum.hcl", "summary": "Now reads `vault_token` from `local.vars` (sourced from `region.secrets.yaml`) and supplies it to the Vault provider, eliminating the need for explicit `VAULT_TOKEN` environment variable injection." } ] }, { "id": "91352f22-fd00-4c98-988d-3a2388b9ac4a", "type": "improvement", "summary": "`pf sso add` now fully automates Vault SSO setup with resumable steps and root token revocation.", "description": "Previously, setting up Vault SSO required manually deploying and configuring the `authentik_vault_sso` and `vault_auth_oidc` modules. The `pf sso add` command now handles this entire phase automatically as a set of resumable, checkpoint-guarded steps. This reduces the SSO bootstrapping process from a multi-step manual procedure to a single command invocation.\n\nKey changes:\n\n1. Deploys `authentik_vault_sso` and `vault_auth_oidc` end-to-end -- reads `kube_config_context` and `vault_addr` from region config, stores the OIDC client secret in SOPS, wires the discovery URL, redirect URIs, and issuer into `vault_auth_oidc`, and removes the static `vault_addr` from config.\n2. Corrects `vault_name` to use the `kube_config_context` value directly, matching the expected Authentik application name.\n3. Fixes `akadmin_email` to use `ctx.authentikRootEmail` instead of the incorrect `ctx.authentikAdminEmail`.\n4. Appends an Identity Center sync step after Vault SSO setup completes.\n5. Revokes and removes the Vault root token from `region.secrets.yaml` after SSO is operational.\n6. Checkpoints the EKS federated auth step by writing `federated_auth_enabled: true` to `aws_eks/.pf.yaml`, allowing safe re-runs.\n7. Makes the Authentik AWS SSO deployment step resumable by tracking `first_applied` in `.pf.yaml`.\n", "references": [ { "type": "internal-commit", "summary": "Vault SSO (#361)", "link": "https://github.com/Panfactum/stack/commit/cf2ef5abf90ba6c01e78276cdae63fd22bd54288" }, { "type": "internal-commit", "summary": "fix: various SSO fixes", "link": "https://github.com/Panfactum/stack/commit/f66cfb1a19fb8e5898b85a7c1a0993b265c3b2e4" }, { "type": "internal-commit", "summary": "feat: better resumability & revoke vault root token", "link": "https://github.com/Panfactum/stack/commit/4f3e553b1425807c7dd84a7bc65d6b5e2c8d1e94" }, { "type": "internal-commit", "summary": "chore: disable vault credential removal task", "link": "https://github.com/Panfactum/stack/commit/c365437e5673edcb6c052cf78e94003c90a90407" }, { "type": "internal-commit", "summary": "added skip and vault name", "link": "https://github.com/Panfactum/stack/commit/97cc08f087b13cfd822827ae645929ccc7fab738" }, { "type": "internal-docs", "summary": "`authentik_vault_sso` module reference", "link": "/docs/main/reference/infrastructure-modules/direct/authentik/authentik_vault_sso" }, { "type": "internal-docs", "summary": "`vault_auth_oidc` module reference", "link": "/docs/main/reference/infrastructure-modules/direct/vault/vault_auth_oidc" } ], "impacts": [ { "type": "cli", "component": "sso add", "summary": "Deploys `authentik_vault_sso` and `vault_auth_oidc` automatically; `vault_name` prefixed correctly; `akadmin_email` fixed; Identity Center auto-synced; root token revoked and removed; EKS federated auth step checkpointed; Authentik AWS SSO step resumable via `.pf.yaml`" }, { "type": "iac-module", "component": "authentik_vault_sso", "summary": "Now deployed automatically by `pf sso add` with correct `vault_name` derived from `kube_config_context`" }, { "type": "iac-module", "component": "vault_auth_oidc", "summary": "Now deployed automatically by `pf sso add` with OIDC outputs wired from `authentik_vault_sso`" } ] }, { "id": "b25a29bb-a11e-4bc7-89ab-fe6aa0835a27", "type": "fix", "summary": "Fixes Vault token noise, `vault_token` crash, and Authentik provider fallback in `panfactum.hcl`.", "description": "Three fixes to `panfactum.hcl` that resolve token-handling edge cases during Terragrunt runs:\n\n1. `pf-get-vault-token` ran and produced noisy output even when `vault_token` was already set in config. It is now invoked with `--noop` when a local token is present.\n2. The `use_local_vault_token` local threw a Terragrunt error when `vault_token` was absent from config. It is now wrapped in `try(...)` to fall back to `false`.\n3. When `authentik_token` was absent from config, the Authentik Terraform provider received `@@TERRAGRUNT_INVALID@@` instead of a usable value. It now falls back to the `AUTHENTIK_TOKEN` environment variable.\n\nAdditionally, `pf sso add` no longer injects `VAULT_TOKEN` explicitly and reads `authentik_token` via `getPanfactumConfig()` instead of decrypting a Vault secrets file.\n", "references": [ { "type": "internal-commit", "summary": "Silence Vault token output when token is already configured", "link": "https://github.com/Panfactum/stack/commit/1ac42430c0dd1fc59d2f3b0aab851416fbe099dc" }, { "type": "internal-commit", "summary": "Read `authentik_token` from config instead of module output", "link": "https://github.com/Panfactum/stack/commit/27cdbcf16525e9989430f33ef211b5a3f9e17560" }, { "type": "internal-commit", "summary": "Add proper fallback for Authentik provider token", "link": "https://github.com/Panfactum/stack/commit/0e5d41b543b20f222ae7afa692e48d283216be37" }, { "type": "internal-docs", "summary": "Terragrunt variables configuration reference", "link": "/docs/main/reference/configuration/terragrunt-variables" } ], "impacts": [ { "type": "configuration", "component": "panfactum.hcl", "summary": "Silences `pf-get-vault-token` when a local token is already configured, wraps `vault_token` lookup in `try(...)` to avoid crashes, and falls back to the `AUTHENTIK_TOKEN` env var for the Authentik provider" }, { "type": "cli", "component": "sso add", "summary": "Removes explicit `VAULT_TOKEN` injection and reads `authentik_token` via `getPanfactumConfig()`" }, { "type": "iac-provider", "component": "authentik", "summary": "Falls back to the `AUTHENTIK_TOKEN` environment variable when `authentik_token` is absent from config" } ] }, { "id": "986baaa9-1a69-47da-8373-83f979281f8d", "type": "improvement", "summary": "`pf sso add` environment selection prompt uses clearer wording and an explainer describing access levels.", "description": "During `pf sso add` federated auth setup, the checkbox prompt that asks which environments to security-harden was unclear about the consequences of the selection. This commit makes two improvements:\n\n1. Relabels the prompt from `Select your production environment(s)` to `Select the environment(s) you want to security harden:` so the intent is explicit.\n2. Adds an explainer below the prompt: \"All other environments will grant users full access. This can be changed later.\"\n\nTogether, these changes make it immediately clear that non-selected environments receive full access and that the choice is not permanent, reducing confusion for first-time users.\n\nAdditionally, the Authentik provider in `panfactum.hcl` now falls back to the `AUTHENTIK_TOKEN` environment variable when the token is not present in the Terragrunt variable configuration. This prevents provider initialization failures during automated runs where the token is injected via the environment.\n", "references": [ { "type": "internal-commit", "summary": "Improve federated auth input instructions and add authentik provider fallback", "link": "https://github.com/Panfactum/stack/commit/0e5d41b543b20f222ae7afa692e48d283216be37" }, { "type": "internal-docs", "summary": "Federated auth bootstrapping guide", "link": "/docs/main/guides/bootstrapping/federated-auth" } ], "impacts": [ { "type": "cli", "component": "sso add", "summary": "Checkbox prompt for selecting security-hardened environments now includes an explainer and clearer label" }, { "type": "iac-provider", "component": "authentik", "summary": "Token lookup falls back to the `AUTHENTIK_TOKEN` environment variable when not set in config" } ] }, { "id": "57ec061f-0037-426e-9e06-6f04c4e714b4", "type": "fix", "summary": "Fixes `pf sso add` skip logic bugs that caused SCIM, Vault SSO, and federated auth steps to re-run after completion.", "description": "Three categories of skip-logic bugs in `pf sso add` caused already-completed steps to re-run on every invocation:\n\n1. Two `skip` callbacks in `setupFederatedAuth` had inverted boolean conditions. \"Get SCIM User Configuration\" returned `!token || !url` instead of `!!token && !!url`, and \"Sync Users and Groups\" returned `!userSyncComplete` instead of `!!userSyncComplete`, causing both steps to always re-execute after successful completion.\n2. The \"Get Vault OIDC configuration\" task called `terragruntOutput` against the wrong module (`vault_auth_oidc` instead of the `authentik_vault_sso` deployment) and used a flat `z.string()` Zod schema instead of the correct `z.object({ value: z.string() })` shape, causing output parsing failures.\n3. The \"Setup AWS Federated SSO\" skip callback read `federated_auth_enabled` (snake_case) from the `.pf.yaml` file instead of `federatedAuthEnabled` (camelCase), so the step never detected that it had already completed.\n", "references": [ { "type": "internal-commit", "summary": "fix: skip logic and input schema", "link": "https://github.com/Panfactum/stack/commit/1a8e4add783f1ba32bc55664b3ec154a22a51e7a" }, { "type": "internal-commit", "summary": "fix: skip logic for federated auth", "link": "https://github.com/Panfactum/stack/commit/c9bac8f242beff2b5d28239a364665d4a43a2773" }, { "type": "internal-docs", "summary": "Federated auth bootstrapping guide", "link": "/docs/main/guides/bootstrapping/federated-auth" } ], "impacts": [ { "type": "cli", "component": "sso add", "summary": "Corrects skip conditions for SCIM setup, user sync, Vault SSO OIDC output parsing, and federated auth detection so completed steps no longer re-run on subsequent invocations." } ] }, { "id": "f968f24a-dee2-4baf-a8c4-5a665a4310de", "type": "improvement", "summary": "`kube_policies` now injects `NODE_OPTIONS=--max-old-space-size` into all containers and init containers proportional to the container's memory request.", "description": "`kube_policies` now automatically sets `NODE_OPTIONS=--max-old-space-size=` on every container and init container, configuring the Node.js V8 heap limit to match the container's memory request. This prevents OOM crashes that occur when Node.js defaults to a 512 MB heap regardless of the container's actual memory allocation.\n\nTwo changes were made:\n\n1. A new `patchStrategicMerge` rule injects the `NODE_OPTIONS` environment variable into both containers and init containers, with the value derived from the container's memory request. The `patchStrategicMerge` approach ensures the variable is correctly re-applied after VPA mutations adjust resource requests.\n2. A follow-on fix corrected the `add-environment-variables-init-containers` Kyverno rule, which was incorrectly iterating `spec.containers` instead of `spec.initContainers`, causing init containers to never receive the injected environment variables.\n", "references": [ { "type": "internal-commit", "summary": "feat: adds opensearch dashboards, remote storage, and v3", "link": "https://github.com/Panfactum/stack/commit/55a2cd5eea66664a7994f5841abda062d982fe1e" }, { "type": "internal-commit", "summary": "fix init container policies", "link": "https://github.com/Panfactum/stack/commit/c7178d3ab8a857594c8c79bce7a4411176ca2244" }, { "type": "internal-docs", "summary": "`kube_policies` module overview", "link": "/docs/main/modules/kube_policies/overview" } ], "impacts": [ { "type": "iac-module", "component": "kube_policies", "summary": "`NODE_OPTIONS=--max-old-space-size` injected into all containers and init containers proportional to memory request; init-container Kyverno rule corrected to iterate `spec.initContainers` instead of `spec.containers`" } ] }, { "id": "457419da-6771-4cfc-9f2d-bb5fc1a77d36", "type": "improvement", "summary": "`pf cluster add` no longer deploys ECR pull-through cache; setup moved to `pf cluster enable`.", "description": "Previously, `pf cluster add` bundled the `aws_ecr_pull_through_cache` deployment and DockerHub/GitHub credential collection into the initial VPC setup step. This required users to have those credentials ready at bootstrap time, adding friction to the first-run experience.\n\nECR pull-through cache is now an optional post-install feature accessible via `pf cluster enable`, which simplifies the initial cluster bootstrapping flow by:\n\n1. Removing the `aws_ecr_pull_through_cache` module deployment from the VPC setup step\n2. Eliminating the DockerHub and GitHub credential prompts during bootstrap\n3. Renaming `setupVPCandECR` to `setupVPC` so the step label accurately reflects its scope\n\nThe `pull_through_cache_enabled` config value is explicitly set to `false` during VPC setup to prevent image pull failures until the feature is intentionally enabled. Existing clusters with ECR pull-through cache already deployed are unaffected.\n", "references": [ { "type": "internal-commit", "summary": "removed ECR setup as part of VPC setup", "link": "https://github.com/Panfactum/stack/commit/acd5bfc8864fd1277504137996a536b8c72494eb" }, { "type": "internal-docs", "summary": "`aws_ecr_pull_through_cache` module overview", "link": "/docs/main/modules/aws_ecr_pull_through_cache/overview" }, { "type": "internal-docs", "summary": "Kubernetes cluster bootstrapping guide", "link": "/docs/main/guides/bootstrapping/kubernetes-cluster" }, { "type": "external-docs", "summary": "AWS ECR pull through cache documentation", "link": "https://docs.aws.amazon.com/AmazonECR/latest/userguide/pull-through-cache.html" } ], "impacts": [ { "type": "cli", "component": "cluster add", "summary": "ECR pull-through cache deployment and credential collection removed from VPC setup step; `pull_through_cache_enabled` now defaults to `false` in the generated config" }, { "type": "cli", "component": "cluster enable", "summary": "New command scaffolded for opt-in ECR pull-through cache setup (stub only, not yet functional)" } ] }, { "id": "4a33cb69-8730-46d2-8df9-91ea05653389", "type": "addition", "summary": "`pf cluster add` now checks the EC2 vCPU quota and warns if it is below 16.", "description": "Cluster installation requires a minimum of 16 on-demand EC2 vCPUs, but new AWS accounts often start with a lower default quota. Previously, `pf cluster add` would fail mid-install with an opaque capacity error if the quota was insufficient.\n\nThe command now queries the AWS Service Quotas API for the running on-demand standard instance quota (`L-1216C47A`) before proceeding. If the quota is below 16, it emits a clear warning showing the current value. If the environment was created with `pf env add`, the quota increase was already requested automatically.\n\nA follow-on fix resolved credential failures in the quota check: the `ServiceQuotasClient` was being instantiated with a `profile` option, which triggers a known AWS SDK JS v3 bug ([aws-sdk-js-v3#6872](https://github.com/aws/aws-sdk-js-v3/issues/6872)) that prevents credentials from being loaded from file. The client now uses a factory function that explicitly reads credentials from disk, bypassing the SDK bug.\n", "references": [ { "type": "internal-commit", "summary": "Add vCPU quota check to `pf cluster add`", "link": "https://github.com/Panfactum/stack/commit/5cd779112382e803f6b4e20ce35f7f76788ebff0" }, { "type": "internal-commit", "summary": "Use factory function for `ServiceQuotasClient` instantiation", "link": "https://github.com/Panfactum/stack/commit/1dc45f74fb324cf0a8cd35be642b2ef3f1a244b5" }, { "type": "issue-report", "summary": "AWS SDK JS v3: `profile` option does not resolve credentials from file", "link": "https://github.com/aws/aws-sdk-js-v3/issues/6872" } ], "impacts": [ { "type": "cli", "component": "cluster add", "summary": "Validates EC2 vCPU quota is at least 16 before starting cluster installation" } ] }, { "id": "260c75a6-6052-4f91-bb0e-23f869282f2d", "type": "breaking_change", "summary": "Removes `namespace` and `service_account` inputs from `kube_certificates` to fix a namespace race condition.", "description": "Multiple resources in `kube_certificates` (including `kubernetes_secret`, `kubernetes_service_account`,\n`kubernetes_role`, `kubernetes_role_binding`, `kubernetes_config_map`, and `kubectl_manifest`) were referencing\n`var.namespace` directly. Because Terraform evaluates variable references without creating an implicit dependency\non the resource that creates the namespace, these resources could be scheduled before the namespace existed, causing\nintermittent apply failures.\n\nSwitching to `local.namespace` creates the correct implicit dependency chain. The now-unused `namespace` and\n`service_account` input variables (which both defaulted to `cert-manager`) have been removed from the module\ninterface, making this a breaking change for any deployment that was explicitly setting those inputs.\n", "action_items": [ "Remove any `namespace` or `service_account` input variable assignments from your `kube_certificates` module configuration. These variables no longer exist and will cause a Terraform error if present." ], "references": [ { "type": "internal-commit", "summary": "fix: namespace not existing race condition (#362)", "link": "https://github.com/Panfactum/stack/commit/3903a9470f20e0be50e630af78cb0235304ff0ae" }, { "type": "internal-docs", "summary": "`kube_certificates` module documentation", "link": "/docs/main/modules/kube_certificates" } ], "impacts": [ { "type": "iac-module", "component": "kube_certificates", "summary": "All Kubernetes resources now reference `local.namespace` instead of `var.namespace`; the now-unused `namespace` and `service_account` input variables have been removed" } ] }, { "id": "214c7792-4629-4ddb-b895-0053a8674e13", "type": "fix", "summary": "Fixes `pf sso add` Vault root token revocation failing silently inside `kubectl exec`.", "description": "Two iterative fixes to the Vault root-token revocation step during SSO setup:\n\n1. The token was changed from a positional argument (`vault token revoke `) to the correct self-revocation approach using the `VAULT_TOKEN` environment variable and the `-self` flag.\n2. Because `kubectl exec` does not invoke a shell and therefore cannot evaluate environment variable assignments, the command was wrapped in `sh -c \"VAULT_TOKEN=... vault token revoke -self\"` so the shell properly expands the variable before invoking `vault`.\n\nTogether these ensure the Vault root token is reliably revoked after SSO setup completes.\n", "references": [ { "type": "internal-commit", "summary": "fix: vault token revocation", "link": "https://github.com/Panfactum/stack/commit/33160da3a8492a06323de484697f4e50a31210c6" }, { "type": "internal-commit", "summary": "providing token to destroy and using -self", "link": "https://github.com/Panfactum/stack/commit/1c45d231c3b1f8b8d1ddac4fba751a0490dcffe9" }, { "type": "external-docs", "summary": "Vault `token revoke` CLI reference", "link": "https://developer.hashicorp.com/vault/docs/commands/token/revoke" }, { "type": "internal-docs", "summary": "Bootstrapping federated auth guide covering Vault SSO setup", "link": "/docs/main/guides/bootstrapping/federated-auth" } ], "impacts": [ { "type": "cli", "component": "sso add", "summary": "Vault root token revocation now wraps the command in `sh -c` so the `VAULT_TOKEN` environment variable is properly expanded inside the container." } ] }, { "id": "3438a013-cd47-41cb-b13b-80070ca0bcee", "type": "fix", "summary": "Fixes primary region identification in the CLI to exclude the `global` region from being marked as primary.", "description": "The `getRegions` utility determines which region hosts the Terraform state bucket by matching each region's `aws_region` against the environment-level `tf_state_region` value. In environments that include a `global` pseudo-region directory, this comparison could incorrectly flag `global` as the primary region when its `aws_region` happened to match `tf_state_region`.\n\nThe root cause involved two issues:\n\n1. The initial exclusion check compared the raw `region` config key against `GLOBAL_REGION`, but `region` can be `undefined` when `region.yaml` does not explicitly set a `region` key. An `undefined` value bypasses the exclusion entirely.\n2. The fix resolves this by computing the resolved region name (`region ?? basename(regionPath)`) first, then using that resolved name in the `GLOBAL_REGION` exclusion check. This ensures the `global` directory is always excluded regardless of whether its `region.yaml` explicitly sets the `region` key.\n\nAll CLI commands that depend on primary region detection now correctly identify the primary region in environments containing a `global` region directory.\n", "references": [ { "type": "internal-commit", "summary": "fix:cli: primary region identification algo", "link": "https://github.com/Panfactum/stack/commit/56d4c8c464f92fb4aa821bcedc0cb20e6568453e" }, { "type": "internal-commit", "summary": "fix: attempt 2 at fix primary region", "link": "https://github.com/Panfactum/stack/commit/edf631ddf4a14bff92aaea31b2a6ca08a55a7833" }, { "type": "internal-docs", "summary": "Terragrunt variables reference documenting `tf_state_region`", "link": "/docs/main/reference/configuration/terragrunt-variables" } ], "impacts": [ { "type": "cli", "component": "cluster add", "summary": "No longer incorrectly identifies the `global` region as primary when adding a new cluster" }, { "type": "cli", "component": "cluster enable", "summary": "No longer incorrectly identifies the `global` region as primary when enabling cluster features" }, { "type": "cli", "component": "env add", "summary": "No longer incorrectly identifies the `global` region as primary when bootstrapping an environment" }, { "type": "cli", "component": "sso add", "summary": "No longer incorrectly identifies the `global` region as primary when configuring SSO" } ] }, { "id": "fb4cb81f-02f6-4d5f-a64f-be12dacfeda1", "type": "fix", "summary": "Fixes `pf env add` state bucket generation failing on new AWS accounts where S3 is not yet active.", "description": "On freshly created AWS accounts, the S3 service may not be immediately active, causing the `HeadBucketCommand` call during state bucket name generation to fail with a `not signed up` error. Two resilience measures were added:\n\n1. A new `Activate S3 service` step now runs before bucket name generation. It attempts to create and delete a dummy bucket, retrying with exponential backoff up to 30 times until S3 becomes available.\n2. The `Generate unique state bucket name` step now includes a retry loop (up to 3 attempts) around the `HeadBucketCommand` call to handle transient connectivity issues that are common immediately after account creation.\n", "references": [ { "type": "internal-commit", "summary": "fix:cli: check if s3 service is active before generating state bucket name", "link": "https://github.com/Panfactum/stack/commit/389632ed207eb8bbb26e48dac85f4661b7e119c8" } ], "impacts": [ { "type": "cli", "component": "env add", "summary": "Adds an S3 activation check with retry logic before state bucket name generation on new AWS accounts" } ] }, { "id": "87a8a9dc-863e-4c99-8690-659fd2d79695", "type": "fix", "summary": "Fixes `pf sso add` to display the Authentik password reset link directly in the confirmation prompt text.", "description": "Previously, the Authentik password reset link was only accessible by automatically opening the browser. If the user declined the browser prompt or the browser failed to open, there was no visible copy of the link to use manually. The link is now included directly in the explainer text of the confirmation prompt, so users can copy and paste it regardless of whether the browser opens successfully.", "references": [ { "type": "internal-commit", "summary": "Add Authentik recovery link to browser-open prompt output", "link": "https://github.com/Panfactum/stack/commit/a2141bfbfaebae9280252968bc89b084f40f5b3c" } ], "impacts": [ { "type": "cli", "component": "sso add", "summary": "Password reset link is now embedded in the explainer text of the browser-open confirmation prompt" } ] }, { "id": "a53dfdad-2e0b-404a-8b7d-81ad78dac036", "type": "fix", "summary": "Fixes `pf devshell sync` generating the `.aws` directory gitignore at the wrong path (`.aws/.aws` instead of `.aws/.gitignore`).", "description": "The `syncStandardFiles` task in `pf devshell sync` was passing `.aws` as the filename argument to `upsertGitIgnore` instead of `.gitignore`. This caused the gitignore file for the `.aws` directory to be written to a file literally named `.aws` inside the `.aws` directory, rather than to the expected `.gitignore` file. Without a valid `.gitignore`, AWS credentials and config files in the `.aws` directory were not protected from being accidentally committed to version control.", "references": [ { "type": "internal-commit", "summary": "fix: `.aws` gitignore generation", "link": "https://github.com/Panfactum/stack/commit/e5edff11ae9c8e7251764f58b1e4357ba8ee3dc6" } ], "impacts": [ { "type": "cli", "component": "devshell sync", "summary": "The `.aws` directory `.gitignore` file is now written to the correct path" } ] }, { "id": "1953d9f8-29ef-46c8-b0f6-5e7031fb447f", "type": "improvement", "summary": "`pf cluster add` now interactively prompts for environment and region, removing the need to run from the correct directory.", "description": "Previously, `pf cluster add` read configuration from the current working directory, requiring users to manually `cd` into the correct region folder before running the command. This was a common source of error during cluster installation.\n\nThe command now presents interactive prompts at startup to:\n\n1. Select the target environment (excluding the management environment)\n2. Select the target region (excluding global and regions that already have a cluster deployed)\n\nAll file paths are derived from the selection, making the cluster installation workflow location-independent. Users can run `pf cluster add` from any directory in their repository.\n", "references": [ { "type": "internal-commit", "summary": "Refactor `pf cluster add` to use interactive environment and region selection", "link": "https://github.com/Panfactum/stack/commit/4825a917123a4894df14c2fb41548c9d9227a79c" }, { "type": "internal-docs", "summary": "Kubernetes cluster bootstrapping guide", "link": "/docs/main/guides/bootstrapping/kubernetes-cluster" } ], "impacts": [ { "type": "cli", "component": "cluster add", "summary": "Prompts for environment and region at startup; no longer requires running from the region directory" } ] }, { "id": "7e84edc5-4af2-41b9-969a-bd40fa4f91ef", "type": "improvement", "summary": "Improves cluster-deployed detection accuracy and filters undeployed environments from `pf cluster add` and `pf sso add` prompts.", "description": "Two related improvements to cluster deployment detection:\n\n1. The `getRegions` utility previously read `kube_domain` / `kube_api_server` from `region.yaml` to determine `clusterDeployed`. Both values can be set before the cluster is actually running, making the check unreliable. Detection now uses `isClusterDeployed`, which calls `getModuleStatus` to inspect the `.pf.yaml` deploy status of `kube_reloader` — a module that only reaches `success` state after the full cluster stack has been applied.\n2. `pf cluster add` and `pf sso add` selection prompts now pre-filter to only fully deployed environments, removing the confusing \"(not deployed)\" labels and preventing accidental selection of undeployed environments.\n", "references": [ { "type": "internal-commit", "summary": "using kubeApiServer to mark clusterDeployed", "link": "https://github.com/Panfactum/stack/commit/b4439e1bf5c8e329a58c8889fa8c29c58f4f25e7" }, { "type": "internal-commit", "summary": "refactor: improve cluster deployment detection to use actual deployment status", "link": "https://github.com/Panfactum/stack/commit/d34274c7b1d70d75c666734a4ca6cfb5cb9e7f0e" } ], "impacts": [ { "type": "cli", "component": "cluster add", "summary": "Cluster-deployed detection now uses `kube_reloader` `.pf.yaml` status; selection prompts filter to deployed environments only" }, { "type": "cli", "component": "sso add", "summary": "Environment and region selection now filters to deployed environments only" } ] }, { "id": "0c4a85be-4664-4785-91ac-cba41b8752e5", "type": "improvement", "summary": "`pf env add` improves existing-org IAM setup, adds post-bootstrap billing guidance, and pre-fills smart environment name defaults.", "description": "Three UX improvements to `pf env add`:\n\n1. When importing an existing AWS organization, the flow now guides users through creating a `management-superuser` IAM user with `AdministratorAccess`, automatically opening the AWS console to the relevant pages.\n2. After bootstrapping the management environment, the success message surfaces the optional `Grant access to the billing console` step and labels both the billing setup and account-invitation steps as optional.\n3. The environment name prompt now pre-fills a smart contextual default: the partially-deployed environment name (for resume), `production` if none exists, `development` if no dev environment exists, or blank otherwise.\n\nCLI default answer values are also now displayed in italic gray for visual clarity.\n", "references": [ { "type": "internal-commit", "summary": "feat: provide instructions for creating a user if importing existing AWS org", "link": "https://github.com/Panfactum/stack/commit/d4da65931ff9bd8e12707f179430d5d107c31215" }, { "type": "internal-commit", "summary": "feat: adds instructions for enabling billing when organization setup is complete", "link": "https://github.com/Panfactum/stack/commit/d09ed80f4a3c9b5d7ba8a9fc46912f00f0bf8f39" }, { "type": "internal-commit", "summary": "fix: default answer styles and adds 'pf env add' default env name", "link": "https://github.com/Panfactum/stack/commit/c30b909eac165687992550d41c9987cbdfc95207" }, { "type": "external-docs", "summary": "AWS IAM User Guide — Grant access to the billing console", "link": "https://docs.aws.amazon.com/IAM/latest/UserGuide/getting-started-account-iam.html" } ], "impacts": [ { "type": "cli", "component": "env add", "summary": "Existing-org flow now guides IAM user creation; post-bootstrap message includes optional billing console instructions; environment name prompt pre-fills with smart contextual default" }, { "type": "cli", "component": "logging", "summary": "Default answer values now displayed in italic gray for visual distinction" } ] }, { "id": "2d32917d-0fac-4b7b-ada2-fae810431ee9", "type": "breaking_change", "summary": "`pf sso add` now deploys Authentik at `sso.` instead of `authentik.`.", "description": "The Authentik subdomain has been standardized from `authentik.` to `sso.` across the entire SSO setup workflow. This change unifies four previously inconsistent domain references:\n\n1. The `domain` input passed to the `kube_authentik` module\n2. The Authentik API base path used during setup\n3. The `authentik_url` written to `global.yaml`\n4. The user-facing prompts shown during `pf sso add`\n\nA shared `SSO_SUBDOMAIN` constant now governs all domain construction, preventing drift between these references. The shorter `sso` prefix also better communicates the service's purpose to end users visiting the login page.\n", "action_items": [ "Update the `authentik_url` value in `global.yaml` from `https://authentik.` to `https://sso.`.", "Update DNS records to point `sso.` to the Authentik ingress. Remove the old `authentik.` record after verifying the new one works.", "Re-apply `kube_authentik` by running `terragrunt apply` in its module directory so it picks up the new domain." ], "references": [ { "type": "internal-commit", "summary": "refactor: standardize SSO subdomain configuration and update domain references", "link": "https://github.com/Panfactum/stack/commit/07d42f09a43b367e79433147d0e9c46a083e95b9" }, { "type": "external-docs", "summary": "Authentik identity provider documentation", "link": "https://docs.goauthentik.io/" }, { "type": "internal-docs", "summary": "`kube_authentik` module reference", "link": "/docs/main/reference/infrastructure-modules/kubernetes/kube_authentik" } ], "impacts": [ { "type": "cli", "component": "sso add", "summary": "Deploys Authentik at `sso.` instead of `authentik.`" }, { "type": "configuration", "component": "global.yaml", "summary": "`authentik_url` must be updated to use the new `sso.` subdomain" }, { "type": "iac-module", "component": "kube_authentik", "summary": "The `domain` input now receives `sso.` instead of `authentik.`" } ] }, { "id": "64f226f5-9ae3-42fd-915e-35d37a824fe9", "type": "improvement", "summary": "`pf cluster add` now configures the bastion domain dynamically as `bastion.` instead of leaving `bastion_domains` empty.", "description": "Previously, the `kube_bastion` Terragrunt template generated by `pf cluster add` hardcoded `bastion_domains = []`, leaving the bastion with no configured domain. The installer now reads `kube_domain` from the Panfactum config and passes `bastion.` as the `bastion_domains` input when deploying the bastion module.\n\nThis mirrors the pattern established for the SSO subdomain by introducing a shared `BASTION_SUBDOMAIN` constant and removes a manual configuration step that previously had to be completed after cluster bootstrap.\n", "references": [ { "type": "internal-commit", "summary": "feat: configure bastion domain dynamically from kube_domain", "link": "https://github.com/Panfactum/stack/commit/cefc9900cb364e5056c1e5c73a4624e30d17f3bf" }, { "type": "internal-docs", "summary": "`kube_bastion` module reference", "link": "/docs/main/modules/kube_bastion" } ], "impacts": [ { "type": "cli", "component": "cluster add", "summary": "Bastion domain is now automatically set to `bastion.` during cluster setup" }, { "type": "iac-module", "component": "kube_bastion", "summary": "Receives `bastion_domains` input populated from the cluster's `kube_domain` config value" } ] }, { "id": "3b7684da-36ba-4f66-b315-6a1fabe8e6cd", "type": "fix", "summary": "Fixes `pf-tunnel` and `pf-db-tunnel` failing at startup due to references to the removed `pf-check-ssh` script and undefined SSH path variables.", "description": "After `pf-check-ssh` was removed from the devshell, both `pf-tunnel` and `pf-db-tunnel` would fail immediately on launch because they still tried to call or source the now-nonexistent script. Additionally, `pf-tunnel` relied on `CONNECTION_INFO_FILE`, `CONFIG_FILE`, and `KNOWN_HOSTS_FILE` variables that `pf-check-ssh` previously defined, so even removing the calls alone was not sufficient. This fix removes the obsolete `pf-check-ssh` invocations and defines the three missing variables inline, restoring both commands to working order.", "references": [ { "type": "internal-commit", "summary": "fix: remove obsolete `pf-check-ssh` call from `pf-db-tunnel`", "link": "https://github.com/Panfactum/stack/commit/da2c2c0b6d592fb74487da17f8f9df5db839e1a2" }, { "type": "internal-commit", "summary": "fix: remove obsolete `pf-check-ssh` source from `pf-tunnel`", "link": "https://github.com/Panfactum/stack/commit/9349ff71dee07eaf056b1d15111e8676838e89e7" }, { "type": "internal-commit", "summary": "fix: define missing SSH configuration variables in `pf-tunnel`", "link": "https://github.com/Panfactum/stack/commit/6f6211d9d7a9774022c215afe4d5092e69a7f65f" }, { "type": "internal-docs", "summary": "SSH configuration reference", "link": "/docs/main/reference/configuration/ssh" } ], "impacts": [ { "type": "devshell", "component": "pf-db-tunnel", "summary": "Removed obsolete `pf-check-ssh` pre-flight call that caused startup failure" }, { "type": "devshell", "component": "pf-tunnel", "summary": "Removed obsolete `pf-check-ssh` source call and defined missing `CONNECTION_INFO_FILE`, `CONFIG_FILE`, and `KNOWN_HOSTS_FILE` variables inline" } ] }, { "id": "b0f99908-394d-446c-86fc-dceeef9b4eb6", "type": "breaking_change", "summary": "`kube_opensearch` now enforces a minimum of 1000 MB for `minimum_memory_mb` and raises the default from 25 MB to 4000 MB.", "description": "The `minimum_memory_mb` variable in `kube_opensearch` was not previously connected to the container spec -- the container always requested a hardcoded 1000 MB regardless of the variable's value. This change wires `minimum_memory_mb` through to the `kube_stateful_set` container spec so the setting actually takes effect.\n\nIn addition, the default and validation constraints are updated:\n\n1. The validation minimum is raised from 25 MB to 1000 MB, reflecting the minimum memory OpenSearch needs to start reliably.\n2. The default is raised from 25 MB to 4000 MB, matching the minimum viable memory for stable OpenSearch operation under typical workloads.\n3. The variable description is corrected to reference OpenSearch instead of Redis (a copy-paste artifact from the original module).\n", "action_items": [ "If you previously set `minimum_memory_mb` to a value below 1000, update it to at least 1000 or remove the override to accept the new default of 4000.", "If you relied on the previous default of 25 MB, be aware that the new default of 4000 MB will significantly increase the memory request for each OpenSearch pod. Review your node capacity and adjust `minimum_memory_mb` explicitly if needed.", "Run `terragrunt apply` on your `kube_opensearch` deployment to apply the new memory configuration." ], "references": [ { "type": "internal-commit", "summary": "feat: improve OpenSearch memory configuration", "link": "https://github.com/Panfactum/stack/commit/d044ca03b10987eb7c93d42fc77f561641201a85" }, { "type": "internal-docs", "summary": "`kube_opensearch` module reference", "link": "/docs/main/reference/infrastructure-modules/submodule/kubernetes/kube_opensearch" } ], "impacts": [ { "type": "iac-module", "component": "kube_opensearch", "summary": "`minimum_memory_mb` is now wired into the container spec (previously ignored, hardcoded at 1000 MB), the validation minimum is raised from 25 MB to 1000 MB, and the default is raised from 25 MB to 4000 MB" } ] }, { "id": "b162cbb7-0677-4181-a1bf-3d7b17cfd9bf", "type": "fix", "summary": "`pf cluster add` now passes the correct `aws_region` code to all AWS API calls instead of the directory slug.", "description": "Several AWS CLI and SDK calls in `pf cluster add` were receiving the Panfactum logical `region` directory slug\ninstead of the actual `aws_region` configuration value. Affected operations:\n\n1. VPC name uniqueness check via `aws ec2 describe-vpcs`\n2. EKS addon listing and deletion via `aws eks list-addons` / `delete-addon`\n3. EC2 node termination via `aws ec2 describe-instances` / `terminate-instances`\n4. Service Quotas vCPU check via `GetServiceQuotaCommand`\n\nAll four now read `aws_region` from the Panfactum config and pass it to the AWS client. In addition,\n`setupVaultSSO` was calling `getPanfactumConfig` with `process.cwd()` instead of the selected region path,\nwhich could load the wrong configuration when the working directory did not match the target region. It now\naccepts an explicit `regionPath` parameter.\n", "references": [ { "type": "internal-commit", "summary": "fix: standardize AWS region parameter handling and improve SSO flexibility", "link": "https://github.com/Panfactum/stack/commit/67143b00110420fe0253a47e18988b072350f8a0" } ], "impacts": [ { "type": "cli", "component": "cluster add", "summary": "VPC lookup, EKS addon management, EC2 instance termination, and Service Quotas calls now receive `aws_region` instead of the logical `region` directory slug" }, { "type": "cli", "component": "sso add", "summary": "`setupVaultSSO` now accepts an explicit `regionPath` parameter instead of using `process.cwd()`" } ] }, { "id": "4342e9eb-99fe-4ab1-9a61-805ec89ff37b", "type": "addition", "summary": "`pf cluster add` auto-deploys Vault Federated SSO when an Authentik instance is detected.", "description": "After the cluster extensions phase completes, `pf cluster add` now searches all environments for a deployed\n`kube_authentik` module. If one is found, it automatically deploys `authentik_vault_sso` and `vault_auth_oidc`,\nthen revokes the static Vault root token. If no Authentik deployment is detected, the step is silently skipped.\n\nThis removes the need to run `pf sso add` separately for Vault SSO when Authentik is already available,\nstreamlining the cluster bootstrap process.\n", "references": [ { "type": "internal-commit", "summary": "fix: standardize AWS region parameter handling and improve SSO flexibility", "link": "https://github.com/Panfactum/stack/commit/67143b00110420fe0253a47e18988b072350f8a0" }, { "type": "internal-docs", "summary": "`vault_auth_oidc` module overview", "link": "/docs/main/modules/vault_auth_oidc/overview" } ], "impacts": [ { "type": "cli", "component": "cluster add", "summary": "After the cluster extensions phase, searches all environments for a deployed `kube_authentik` module and auto-deploys Vault SSO if found" }, { "type": "iac-module", "component": "authentik_vault_sso", "summary": "Deployed automatically by `pf cluster add` when Authentik is present" }, { "type": "iac-module", "component": "vault_auth_oidc", "summary": "Deployed automatically by `pf cluster add` when Authentik is present" } ] }, { "id": "4571ca06-9c7f-4f47-b1cc-c508f22d7ed8", "type": "addition", "summary": "Adds `pf cluster enable` command for activating optional cluster features and a `--cwd` flag to all `pf` CLI commands.", "description": "Two additions from the same commit:\n\n1. `pf cluster enable` introduces a guided wizard for activating opt-in cluster features. The first supported feature is `ecr-pull-through-cache`, which prompts for Docker Hub and GitHub credentials, stores them via SOPS, deploys the `aws_ecr_pull_through_cache` module, and re-deploys `kube_policies` with `pull_through_cache_enabled` set to `true`. This automates a multi-step manual process that previously required users to scaffold the module directory, write a `terragrunt.hcl`, encrypt secrets, and run multiple `terragrunt apply` commands by hand.\n2. All `pf` CLI commands now accept a `--cwd` flag (also readable from the `CWD` environment variable) to override the working directory used for repository root detection. This enables running `pf` from CI pipelines, editor integrations, and other contexts outside the repository tree.\n", "references": [ { "type": "internal-commit", "summary": "feat: implement cluster enable command with ECR pull-through cache support", "link": "https://github.com/Panfactum/stack/commit/654a4ad80408b2347cdd87aed1623d28bb5c3725" }, { "type": "internal-docs", "summary": "`aws_ecr_pull_through_cache` module overview", "link": "/docs/main/modules/aws_ecr_pull_through_cache/overview" }, { "type": "internal-docs", "summary": "Kubernetes cluster bootstrapping guide covering ECR pull-through cache setup", "link": "/docs/main/guides/bootstrapping/kubernetes-cluster" } ], "impacts": [ { "type": "cli", "component": "cluster enable", "summary": "New command providing a guided wizard for activating opt-in cluster features such as `ecr-pull-through-cache`" }, { "type": "cli", "component": "cluster add", "summary": "Now respects `--cwd` for repository root resolution" }, { "type": "cli", "component": "config get", "summary": "Now respects `--cwd` for repository root resolution" }, { "type": "iac-module", "component": "aws_ecr_pull_through_cache", "summary": "Deployed automatically by `pf cluster enable` when activating the `ecr-pull-through-cache` feature" }, { "type": "iac-module", "component": "kube_policies", "summary": "Re-deployed with `pull_through_cache_enabled` set to `true` after ECR pull-through cache setup" } ] }, { "id": "bd4ca5e9-2d13-4a87-a6d7-dfe48ba14a24", "type": "improvement", "summary": "`pf cluster add` EKS deployment step now displays a 15-minute ETA warning in the task title during the apply phase.", "description": "The EKS cluster provisioning step in `pf cluster add` can take up to 15 minutes, which previously left users staring at an `Applying - Planning changes` task title with no indication of how long the wait would be. The deploy task now appends a time estimate warning to the task title so users know the delay is expected. The underlying `buildDeployModuleTask` utility accepts a new optional `etaWarningMessage` parameter, making this pattern reusable for other long-running deploy steps.", "references": [ { "type": "internal-commit", "summary": "Add ETA warning message to EKS deployment task title", "link": "https://github.com/Panfactum/stack/commit/5e4d6524bb7f8d283e86f3775fdd4067f0b1d7ac" } ], "impacts": [ { "type": "cli", "component": "cluster add", "summary": "EKS deploy step now displays a time estimate warning in the task title while planning changes" } ] }, { "id": "4b30f388-7287-4c39-ac21-d50ab21fd0c8", "type": "improvement", "summary": "`pf cluster enable` ECR setup now runs `terragrunt run-all apply` after enabling pull-through cache.", "description": "Previously the ECR enable step in `pf cluster enable` only re-deployed `kube_policies` after setting `pull_through_cache_enabled: true` in the region config. This was insufficient because many other modules reference that flag and would remain out of date until the next manual apply.\n\nThe step now runs `terragrunt run-all apply` across the entire region directory, ensuring every module that depends on pull-through cache settings is updated in a single operation. This eliminates a class of subtle post-enable failures where workloads would still attempt to pull images directly instead of through the ECR cache.\n", "references": [ { "type": "internal-commit", "summary": "Replace targeted `kube_policies` re-deploy with full `terragrunt run-all apply` after enabling ECR pull-through cache", "link": "https://github.com/Panfactum/stack/commit/34ee420cad0fd8bc0c5fd72674bf734b02771ca7" }, { "type": "internal-docs", "summary": "`aws_ecr_pull_through_cache` module reference", "link": "/docs/main/modules/aws_ecr_pull_through_cache" } ], "impacts": [ { "type": "cli", "component": "cluster enable", "summary": "ECR setup step replaces the single `kube_policies` re-deploy with a region-wide `terragrunt run-all apply`" } ] }, { "id": "6ff7eec5-54f9-4fdc-9138-d43ae085ec39", "type": "improvement", "summary": "`pf sso add` now automatically opens the browser to the Authentik token settings page after creating the API token.", "description": "Previously, the `pf sso add` Authentik setup step instructed users to manually navigate to the Authentik user token settings page (`/if/user/#/settings;{\"page\":\"page-tokens\"}`) to copy the newly created API token. The browser now opens automatically to that URL immediately after the token is created, reducing the number of manual steps and making the token retrieval flow consistent with the earlier password-reset browser-open step.", "references": [ { "type": "internal-commit", "summary": "feat: add automatic browser opening to Authentik token page", "link": "https://github.com/Panfactum/stack/commit/ce9783a125f31347aa835d2885816d0abae57807" }, { "type": "internal-docs", "summary": "Identity provider bootstrapping guide covering the `pf sso add` workflow", "link": "/docs/main/guides/bootstrapping/identity-provider" } ], "impacts": [ { "type": "cli", "component": "sso add", "summary": "Browser automatically opens to the Authentik token settings page after the API token is created, eliminating the manual navigation step" } ] }, { "id": "748b9ad4-52f4-432c-8db2-f794b8177964", "type": "improvement", "summary": "`pf sso add` root email defaults to `authentik-root@sso.` and admin email prompt warns against reusing AWS root email.", "description": "Previously, the Authentik root user email prompt had no default value, requiring users to type a full email address. This change makes two improvements:\n\n1. The root email prompt now pre-populates with `authentik-root@sso.` (using the selected SSO subdomain) so users can press Enter to accept a sensible default.\n2. The admin user email explainer now clarifies that the email must be unique and must not be the AWS root account email address, preventing a common setup error where users accidentally reuse the account root email.\n", "references": [ { "type": "internal-commit", "summary": "feat: improve Authentik setup with default root email and better instructions", "link": "https://github.com/Panfactum/stack/commit/f9903b9bdafe6dd44f8aeea43a75ca9b0a0a1419" }, { "type": "internal-docs", "summary": "Identity Provider bootstrapping guide", "link": "/docs/main/guides/bootstrapping/identity-provider" } ], "impacts": [ { "type": "cli", "component": "sso add", "summary": "Root email prompt pre-filled with `authentik-root@sso.`; admin email explainer clarifies uniqueness requirement" } ] }, { "id": "66a9facf-1f8d-4fb8-be04-5f39dfc86ae9", "type": "fix", "summary": "Fixes `pf cluster add` EKS setup where the default cluster name was not pre-formatted, allowing invalid characters.", "description": "The EKS cluster name prompt already applied `clusterNameFormatter` as a real-time transformer while the user typed, but the `default` value passed to the prompt was the raw `${environment}-${region}` string. If a user accepted the default without modification, the formatter was never invoked, so the saved cluster name could contain uppercase letters or underscores that violate the EKS naming constraint of lowercase alphanumeric characters and hyphens only. The default is now passed through `clusterNameFormatter` before being shown, ensuring the pre-filled value is always a valid, formatted cluster name.", "references": [ { "type": "internal-commit", "summary": "fix: improve cluster name handling in EKS setup", "link": "https://github.com/Panfactum/stack/commit/1aed9737fed298a33b7dbfae6cb4370a8f2b472d" }, { "type": "internal-docs", "summary": "`aws_eks` module reference", "link": "/docs/main/modules/aws_eks" } ], "impacts": [ { "type": "cli", "component": "cluster add", "summary": "Default cluster name is now passed through `clusterNameFormatter` before display, ensuring it contains only lowercase letters, digits, and hyphens" } ] }, { "id": "32294577-c16f-49c5-90c1-0b2fb09f9145", "type": "addition", "summary": "Adds anonymous CLI usage telemetry via PostHog and a new `panfactum.user.yaml` per-user config file.", "description": "Two related additions for user-specific configuration and telemetry:\n\n1. The `pf` CLI now fires a `cli-start` PostHog telemetry event on each invocation when a `user_id` is present in the Panfactum repo variables. The `user_id` is a UUID generated by `pf welcome` on first DevShell setup. All telemetry is anonymous; no personally identifiable information is collected. This gives the Panfactum team visibility into adoption and usage patterns so they can prioritize improvements.\n2. `panfactum.user.yaml` is a new optional per-user configuration file alongside `panfactum.yaml`. Its values are merged on top of `panfactum.yaml` at read time, enabling user-specific settings (such as `user_id`) without modifying the shared repo config. The file is automatically gitignored by `pf devshell sync`.\n", "references": [ { "type": "internal-commit", "summary": "feat: add anonymous usage tracing", "link": "https://github.com/Panfactum/stack/commit/5d6c7ee80a7966988687ca691f5080b28599c772" }, { "type": "external-docs", "summary": "PostHog product analytics platform", "link": "https://posthog.com/" }, { "type": "internal-docs", "summary": "Panfactum repo variables reference", "link": "/docs/main/reference/configuration/repo-variables" } ], "impacts": [ { "type": "cli", "component": "welcome", "summary": "Generates a UUID `user_id` on first run, stores it in `panfactum.user.yaml`, and fires a `cli-welcome` PostHog event" }, { "type": "configuration", "component": "panfactum.user.yaml", "summary": "New optional per-user config file whose values merge on top of `panfactum.yaml` and are gitignored automatically" }, { "type": "cli", "component": "devshell sync", "summary": "`*.user.yaml` files are now added to the root and environments `.gitignore` entries" } ] }, { "id": "d3e867c6-a220-4d8a-be29-789893991087", "type": "fix", "summary": "Fixes `pf` CLI usage telemetry to fire after command parsing so that command path and flags are captured.", "description": "The `cli-start` PostHog event was previously fired inside `createPanfactumContext`, before the command had been fully parsed. At that point, `proc.path`, `proc.help`, and other command-level properties were not yet populated, so every telemetry event was sent without useful context. The tracking call now runs in `index.ts` after `cli.process()` completes, and the event records the command path, help flag, debug flag, and working directory alongside the `user_id`. This gives the team accurate per-command usage data to guide development priorities.", "references": [ { "type": "internal-commit", "summary": "fix: move usage tracking from context to command execution", "link": "https://github.com/Panfactum/stack/commit/d7e596c735284934c4000156cdce6ede26dedf5f" } ] }, { "id": "c71df44a-91c3-4459-8077-9671352133b8", "type": "fix", "summary": "Fixes `kube_opensearch` dashboard superuser password containing special characters that caused authentication failures and security update job ignoring the `vpa_enabled` input.", "description": "Two `kube_opensearch` fixes:\n\n1. The dashboard superuser password generated by `random_password` previously included special characters. Certain special characters cause authentication failures in the OpenSearch security plugin when used in internal user credentials. The password now sets `special = false` to restrict generation to alphanumeric characters only, preventing these failures.\n2. The security update job (a `kube_job` submodule invocation that runs `securityadmin.sh` to apply RBAC configuration) did not pass the `vpa_enabled` variable through to its underlying `kube_job` module. This meant VPA resource recommendations were always applied to the job pod regardless of the module-level setting. The variable is now forwarded correctly.\n", "references": [ { "type": "internal-commit", "summary": "fix: restrict opensearch dashboard password to alphanumeric characters only", "link": "https://github.com/Panfactum/stack/commit/919e46249012e6623547cdbbb7da12b014314c8d" }, { "type": "internal-commit", "summary": "fix: pass `vpa_enabled` to `kube_opensearch` security update job", "link": "https://github.com/Panfactum/stack/commit/6001eed85f76a8aaf53597846df50d66992e53d7" }, { "type": "issue-report", "summary": "Special characters not well supported in OpenSearch admin password", "link": "https://github.com/opensearch-project/opensearch-k8s-operator/issues/955" }, { "type": "internal-docs", "summary": "`kube_opensearch` module reference", "link": "/docs/main/reference/infrastructure-modules/submodule/kubernetes/kube_opensearch" } ], "impacts": [ { "type": "iac-module", "component": "kube_opensearch", "summary": "Dashboard superuser password restricted to alphanumeric characters via `special = false`; security update job now correctly inherits `vpa_enabled` from the parent module" } ] }, { "id": "d48d0fb0-0dfb-4a58-85e4-71825b0784b8", "type": "improvement", "summary": "Updates release scripts to use the structured YAML changelog format instead of the legacy MDX-based changelog.", "description": "The release scripts have been updated to work with the new directory-based structured changelog system. Previously,\nboth scripts wrote flat MDX files under `docs/changelog`. Now `make-new-edge-release` copies\n`changelog/main/log.yaml` to a versioned `edge//log.yaml` directory and resets `main/log.yaml` to a blank\ntemplate, while `make-new-stable-release-channel` creates a `stable//log.yaml` entry directory. This\naligns the release workflow with the validated YAML schema and enables richer changelog rendering on the website.\n", "references": [ { "type": "internal-commit", "summary": "feat(changelog): add structured YAML schema and validation tooling", "link": "https://github.com/Panfactum/stack/commit/8d4b44e31d14a8c5ce25d04a0c0b4b535d2c0afc" } ], "impacts": [ { "type": "devshell", "component": "make-new-edge-release", "summary": "Now copies `changelog/main/log.yaml` to a versioned `edge//log.yaml` directory on release and resets `main/log.yaml` to a clean template" }, { "type": "devshell", "component": "make-new-stable-release-channel", "summary": "Now creates a `stable//log.yaml` entry directory instead of writing to the legacy `docs/changelog` MDX path" } ] }, { "id": "369b598b-9974-4693-b2e1-5a7f730c8452", "type": "fix", "summary": "Fixes `kube_policies` read-only ClusterRoles incorrectly granting full write access and restricts `pf:admins` from writing to security-sensitive Kubernetes resources.", "description": "Two `kube_policies` RBAC security fixes from the same commit:\n\n1. The `read_verbs` local variable was accidentally set to the same value as `all_verbs`, giving the `reader` and `restricted_reader` ClusterRoles full write access to non-secret resources. `read_verbs` is now corrected to `[\"get\", \"list\", \"watch\"]`.\n\n2. The `pf:admins` ClusterRole previously granted full write access to all non-secret resources, including RBAC roles/bindings, admission webhooks, CRDs, and Kyverno policies. The role now defines a `sensitive_resources` list (`clusterroles`, `clusterrolebindings`, `roles`, `rolebindings`, `apiservices`, `validatingadmissionpolicies`, `mutatingwebhookconfigurations`, `validatingadmissionpolicybindings`, `validatingwebhookconfigurations`, `flowschemas`, `prioritylevelconfigurations`, `customresourcedefinitions`, `clustercleanuppolicies`, `clusterpolicies`, `policyexceptions`) and grants only `get`/`list`/`watch` access to them.\n\nThese changes harden the default RBAC posture by ensuring read-only roles cannot modify resources and admin roles cannot escalate privileges or disrupt cluster-critical components.\n", "action_items": [ "Re-apply `kube_policies` via `terragrunt apply` to receive the corrected RBAC permissions." ], "references": [ { "type": "internal-commit", "summary": "feat: upgrades kubernetes to 1.31", "link": "https://github.com/Panfactum/stack/commit/33d887c36fae56bc624511e72a8ac7f939db47f3" }, { "type": "internal-docs", "summary": "`kube_policies` module documentation", "link": "/docs/main/modules/kube_policies" } ], "impacts": [ { "type": "iac-module", "component": "kube_policies", "summary": "`read_verbs` corrected to `get`/`list`/`watch`; `pf:admins` restricted from writing RBAC resources, webhooks, CRDs, and Kyverno policies" } ] }, { "id": "7bba77bf-9cf9-499b-a541-ac8effaed8f1", "type": "addition", "summary": "Adds TypeScript `pf` CLI subcommands replacing legacy bash devshell scripts and updates IaC modules to use the new CLI.", "description": "The legacy bash devshell scripts have been rewritten as TypeScript `pf` CLI subcommands, built on the Clipanion framework with a shared `PanfactumCommand` base class. This migration improves:\n\n- **Maintainability** -- TypeScript provides stronger type safety and centralized error handling via `CLIError`\n- **Discoverability** -- all commands are organized under a unified `pf` namespace with built-in help\n- **Consistency** -- shared utilities for AWS, Vault, BuildKit, and database operations reduce code duplication\n\nThe old bash scripts remain available as pass-through wrappers, so existing workflows continue to function without changes. Infrastructure modules (`kube_buildkit`, `kube_disruption_window_controller`, `kube_velero`, `wf_dockerfile_build`, `wf_tf_deploy`) have been updated to invoke the new `pf` subcommands in their cron jobs and workflow scripts. No user-visible behavior changes are expected.\n", "references": [ { "type": "internal-commit", "summary": "refactor: migrate bash scripts to TypeScript CLI commands", "link": "https://github.com/Panfactum/stack/commit/dcfa7211e0afa76c8656bb706b61cac0a5dea76f" }, { "type": "internal-docs", "summary": "`kube_buildkit` module documentation", "link": "/docs/main/reference/infrastructure-modules/kubernetes/kube_buildkit" }, { "type": "internal-docs", "summary": "`wf_dockerfile_build` module documentation", "link": "/docs/main/reference/infrastructure-modules/workflow/wf_dockerfile_build" }, { "type": "internal-docs", "summary": "`wf_tf_deploy` module documentation", "link": "/docs/main/reference/infrastructure-modules/workflow/wf_tf_deploy" } ], "impacts": [ { "type": "cli", "component": "buildkit build", "summary": "New CLI command replacing the `pf-buildkit-build` devshell script" }, { "type": "cli", "component": "buildkit clear-cache", "summary": "New CLI command replacing the `pf-buildkit-clear-cache` devshell script" }, { "type": "cli", "component": "buildkit get-address", "summary": "New CLI command replacing the `pf-buildkit-get-address` devshell script" }, { "type": "cli", "component": "buildkit record-build", "summary": "New CLI command replacing the `pf-buildkit-record-build` devshell script" }, { "type": "cli", "component": "buildkit suspend", "summary": "New CLI command replacing the `pf-buildkit-scale-down` devshell script" }, { "type": "cli", "component": "buildkit tunnel", "summary": "New CLI command replacing the `pf-buildkit-tunnel` devshell script" }, { "type": "cli", "component": "db get-creds", "summary": "New CLI command replacing the `pf-get-db-creds` devshell script" }, { "type": "cli", "component": "db tunnel", "summary": "New CLI command replacing the `pf-db-tunnel` devshell script" }, { "type": "cli", "component": "docker credential-helper", "summary": "New CLI command replacing the `docker-credential-panfactum` devshell script" }, { "type": "cli", "component": "kube disable-disruptions", "summary": "New CLI command replacing the `pf-voluntary-disruptions-disable` devshell script" }, { "type": "cli", "component": "kube enable-disruptions", "summary": "New CLI command replacing the `pf-voluntary-disruptions-enable` devshell script" }, { "type": "cli", "component": "kube cluster-resume", "summary": "New CLI command replacing the `pf-eks-resume` devshell script" }, { "type": "cli", "component": "kube cluster-suspend", "summary": "New CLI command replacing the `pf-eks-suspend` devshell script" }, { "type": "cli", "component": "kube get-token", "summary": "New CLI command replacing the `pf-get-kube-token` devshell script" }, { "type": "cli", "component": "kube profile-for-context", "summary": "New CLI command replacing the `pf-get-aws-profile-for-kube-context` devshell script" }, { "type": "cli", "component": "k8s velero snapshot-gc", "summary": "New CLI command replacing the `pf-velero-snapshot-gc` devshell script" }, { "type": "cli", "component": "tunnel", "summary": "New CLI command replacing the `pf-tunnel` devshell script" }, { "type": "cli", "component": "util get-commit-hash", "summary": "New CLI command replacing the `pf-get-commit-hash` devshell script" }, { "type": "cli", "component": "util get-module-hash", "summary": "New CLI command replacing the `pf-get-local-module-hash` devshell script" }, { "type": "cli", "component": "vault get-token", "summary": "New CLI command replacing the `pf-get-vault-token` devshell script" }, { "type": "cli", "component": "wf git-checkout", "summary": "New CLI command replacing the `pf-wf-git-checkout` devshell script" }, { "type": "cli", "component": "wf sops-set-profile", "summary": "New CLI command replacing the `pf-sops-set-profile` devshell script" }, { "type": "cli", "component": "aws ecr wait-on-image", "summary": "New CLI command for polling ECR until a specific image tag becomes available" }, { "type": "cli", "component": "iac delete-locks", "summary": "New CLI command replacing the `pf-tf-delete-locks` devshell script" }, { "type": "iac-module", "component": "kube_buildkit", "summary": "Scale-to-zero and cache-clear cron jobs now run `pf buildkit suspend` and `pf buildkit clear-cache`" }, { "type": "iac-module", "component": "kube_disruption_window_controller", "summary": "Enabler and disabler cron jobs now run `pf kube enable-disruptions` and `pf kube disable-disruptions`" }, { "type": "iac-module", "component": "kube_velero", "summary": "Snapshot garbage-collection cron job now runs `pf kube velero-snapshot-gc`" }, { "type": "iac-module", "component": "wf_dockerfile_build", "summary": "Build, clone, scale-buildkit, and setup scripts now call `pf buildkit` and `pf wf` subcommands" }, { "type": "iac-module", "component": "wf_tf_deploy", "summary": "Deploy and force-unlock scripts now call `pf wf git-checkout`, `pf wf sops-set-profile`, `pf config get`, and `pf iac delete-locks`" } ] }, { "id": "e240c03b-1971-40bc-b9c8-49a3a9a85332", "type": "fix", "summary": "Fixes `pf domain add` path resolution and CLI config file loading to handle missing files gracefully.", "description": "Two CLI infrastructure bugs were fixed:\n\n1. `getDomains` was assigning the `env.path` field from `environment_dir` (a short directory basename like `production`) instead of the full absolute filesystem path (e.g., `/repo/environments/production`). Downstream code that joined this path with filenames produced invalid paths, breaking `pf domain add` and any other command that relies on `getDomains`.\n2. `getConfigValuesFromFile` was calling `readYAMLFile` and `sopsDecrypt` without `throwOnMissing: false`, causing them to throw when an optional config file (e.g., `module.yaml`, `environment.secret.yaml`) did not exist. The function now returns `null` for missing files, matching its documented contract.\n", "references": [ { "type": "internal-commit", "summary": "Fix `getDomains` environment path and `getConfigValuesFromFile` missing-file handling", "link": "https://github.com/Panfactum/stack/commit/b7b790dd661af18039021d3352666f3ba3fc41c9" } ], "impacts": [ { "type": "cli", "component": "domain add", "summary": "`getDomains` was using `environment_dir` (a short directory name) instead of the full absolute filesystem path, causing downstream file resolution to fail when resolving environment files." }, { "type": "cli", "component": "config get", "summary": "`getConfigValuesFromFile` now passes `throwOnMissing: false` and `throwOnEmpty: false` to the YAML reader and SOPS decryptor, returning `null` instead of throwing when optional config files are absent." } ] }, { "id": "884600c0-ec49-400e-8b07-3a7372c379cc", "type": "fix", "summary": "Fixes CLI subprocess output capture where merged stdout/stderr was silently dropped, causing missing error diagnostics.", "description": "The `execute` subprocess utility called `concatStreams` with a raw array instead of the expected `{ streams: [...] }` object signature. This caused the merged output stream to be empty on every subprocess invocation, affecting two areas:\n\n1. **Logging** -- CLI commands that depend on capturing combined stdout/stderr output (such as error messages from `terragrunt` or `kubectl`) would silently lose that diagnostic output.\n2. **Retry diagnostics** -- When a subprocess failed and the CLI attempted to log the combined output for troubleshooting, the output was always empty, making it difficult to diagnose transient failures.\n\nThe call site in `execute.ts` has been corrected to pass `{ streams: [stdout, stderr] }` instead of a bare array, matching the `concatStreams` API signature.\n", "references": [ { "type": "internal-commit", "summary": "feat: adds unit tests to the CLI", "link": "https://github.com/Panfactum/stack/commit/dd36035b968faeb3c4f64406aea69e47353f968a" } ], "impacts": [ { "type": "cli", "component": "logging", "summary": "Merged subprocess output stream now correctly captures both stdout and stderr instead of returning empty content." } ] }, { "id": "f09ebec3-500f-4b45-8988-96daa670be82", "type": "addition", "summary": "`kube_karpenter_node_pools` now creates dedicated workflow-only Karpenter node pools for each existing pool type.", "description": "Each existing Karpenter node pool now has a parallel workflow-only counterpart (prefixed with `wf-`). The workflow pools carry a `workflow: \"true\"` taint with `NoSchedule` effect and label nodes with `panfactum.com/workflow-only: \"true\"`. This isolates Argo Workflow pods (which tolerate the `workflow` taint) onto their own nodes, preventing long-running workflows from displacing regular workloads.\n\nThe six new workflow-only pools mirror the existing pool types:\n\n- `wf-burstable` (amd64)\n- `wf-burstable-arm` (amd64/arm64)\n- `wf-spot` (amd64)\n- `wf-spot-arm` (amd64/arm64)\n- `wf-on-demand` (amd64)\n- `wf-on-demand-arm` (amd64/arm64)\n\nA new `default_termination_grace_period` variable (default `1h0m0s`) has also been introduced to control the termination grace period for on-demand node pools, replacing the previously hardcoded value.\n", "references": [ { "type": "internal-commit", "summary": "feat: adds workflow-only node pools", "link": "https://github.com/Panfactum/stack/commit/2702b0c70a3f8aed03ea6606ba291b6e2b380885" }, { "type": "internal-docs", "summary": "`kube_karpenter_node_pools` module documentation", "link": "/docs/main/reference/infrastructure-modules/direct/kubernetes/kube_karpenter_node_pools" }, { "type": "external-docs", "summary": "Karpenter NodePool documentation", "link": "https://karpenter.sh/docs/concepts/nodepools/" } ], "impacts": [ { "type": "iac-module", "component": "kube_karpenter_node_pools", "summary": "Six new workflow-only `NodePool` resources added (one per existing pool type); each carries a `workflow=true:NoSchedule` taint, a `panfactum.com/workflow-only=true` label, and a new `default_termination_grace_period` variable for on-demand pools" } ] }, { "id": "4c93824e-83eb-469d-86ed-2e99252b05a9", "type": "addition", "summary": "Adds a `workflow_nodes_required` input to workflow and workload modules to schedule pods exclusively on dedicated workflow-only nodes.", "description": "When `workflow_nodes_required` is set to `true`, pods are constrained to nodes labeled\n`panfactum.com/workflow-only=true` (the workflow-only Karpenter pools introduced alongside\nthis change) and are granted the `workflow=true:NoSchedule` taint toleration.\n\nThis allows long-running, non-resumable Argo Workflows to run on dedicated nodes that regular\nworkloads cannot be scheduled onto, preventing workflow disruption caused by bin-packing\npressure from other services.\n", "references": [ { "type": "internal-commit", "summary": "feat: adds workflow-only node support to wf_spec", "link": "https://github.com/Panfactum/stack/commit/30f40bd6c4414685d5923bfe6b0029557acab1d3" }, { "type": "internal-docs", "summary": "`wf_spec` module reference", "link": "/docs/main/modules/wf_spec" }, { "type": "internal-docs", "summary": "`kube_workload_utility` module reference", "link": "/docs/main/modules/kube_workload_utility" }, { "type": "external-docs", "summary": "Karpenter NodePools documentation", "link": "https://karpenter.sh/docs/concepts/nodepools" } ], "impacts": [ { "type": "iac-module", "component": "wf_spec", "summary": "New `workflow_nodes_required` input (default `false`) constrains workflow pods to workflow-only nodes" }, { "type": "iac-module", "component": "kube_workload_utility", "summary": "New `workflow_nodes_required` input adds a `panfactum.com/workflow-only` node selector and `workflow=true:NoSchedule` taint toleration" } ] }, { "id": "9bb6797e-2a5f-4cbc-9bcf-82320833fa67", "type": "addition", "summary": "`kube_opensearch` now supports single-node deployments via `replica_count=1`.", "description": "Previously, `kube_opensearch` required at least three replicas and would fail validation for smaller deployments. This change allows single-node deployments by:\n\n1. Removing the `replica_count >= 3` validation constraint\n2. Automatically setting `discovery.type` to `single-node` when `replica_count` is 1\n3. Omitting the `discovery.seed_hosts` and `cluster.initial_cluster_manager_nodes` settings that are only relevant for multi-node clusters\n\nThis is useful for development environments and smaller workloads where high availability is not required.\n", "references": [ { "type": "internal-commit", "summary": "feat: adds the ability to run opensearch in single-node mode", "link": "https://github.com/Panfactum/stack/commit/14cb9e692705dfa5b13f113d9699abb6df58b97c" }, { "type": "external-docs", "summary": "OpenSearch discovery and cluster formation settings", "link": "https://docs.opensearch.org/latest/tuning-your-cluster/discovery-cluster-formation/settings/" } ], "impacts": [ { "type": "iac-module", "component": "kube_opensearch", "summary": "`replica_count` minimum of 3 removed; setting `replica_count=1` now configures `discovery.type=single-node` and omits multi-node seed hosts configuration" } ] }, { "id": "7f0e874c-d9de-4b0a-81b0-a3119383d7a5", "type": "fix", "summary": "`kube_aws_creds` no longer conditionally omits the `aws_iam_user_policy` resource when `iam_policy_json` is null, preventing potential Terraform plan errors.", "description": "Previously, the `aws_iam_user_policy` resource in `kube_aws_creds` used a `count` conditional\nto skip creation when `iam_policy_json` was null. This could cause issues in certain\nconfigurations where downstream resources referenced the policy. The fix replaces the\nconditional `count` with `coalesce(var.iam_policy_json, \"{}\")`, ensuring the resource is\nalways created with a valid (possibly empty) policy document. Existing deployments will see\nTerraform create the `aws_iam_user_policy` resource on the next apply if it was previously\nomitted.\n", "references": [ { "type": "internal-commit", "summary": "Fix `kube_aws_creds` to always create IAM user policy with coalesce fallback", "link": "https://github.com/Panfactum/stack/commit/14cb9e692705dfa5b13f113d9699abb6df58b97c" }, { "type": "internal-docs", "summary": "`kube_aws_creds` module documentation", "link": "/docs/main/modules/kube_aws_creds/overview" } ], "impacts": [ { "type": "iac-module", "component": "kube_aws_creds", "summary": "`aws_iam_user_policy` is now always created using `coalesce(var.iam_policy_json, \"{}\")` instead of being conditionally omitted when `iam_policy_json` is null" } ] }, { "id": "3513552e-192b-4c06-8584-6ce87c9a8df2", "type": "addition", "summary": "`kube_pg_cluster` adds `pg_recovery_target_immediate` input for `targetImmediate` CNPG recovery mode.", "description": "The new `pg_recovery_target_immediate` input (default `null`) configures the CNPG cluster bootstrap recovery to use `targetImmediate: true`, stopping recovery at the earliest consistent point after the base backup completed. When set, the input value is used as the `backupID` in the CNPG `recoveryTarget` block.\n\nThis provides a faster and simpler alternative to point-in-time recovery via `pg_recovery_target_time` when you only need to restore to the exact state at the end of a specific backup rather than replaying WAL to a precise timestamp. The two options are mutually exclusive.\n\nThe input accepts a backup ID in `YYYYMMDDTHHmmss` format and requires `pg_recovery_mode_enabled` and `pg_recovery_directory` to be set. Existing deployments are unaffected since the input defaults to `null`.\n", "references": [ { "type": "internal-commit", "summary": "feat(kube_pg_cluster): support targetImmediate recovery (#375)", "link": "https://github.com/Panfactum/stack/commit/aaa63ac02ce6adee3e990bc8c45c49a1ca2089c0" }, { "type": "external-commit", "summary": "PR #375: feat(kube_pg_cluster): support targetImmediate recovery", "link": "https://github.com/Panfactum/stack/pull/375" }, { "type": "external-docs", "summary": "CloudNativePG recovery documentation — recovery targets including `targetImmediate`", "link": "https://cloudnative-pg.io/docs/1.29/recovery/" }, { "type": "internal-docs", "summary": "`kube_pg_cluster` module reference documentation", "link": "/docs/main/reference/infrastructure-modules/submodule/kubernetes/kube_pg_cluster" } ], "impacts": [ { "type": "iac-module", "component": "kube_pg_cluster", "summary": "New optional `pg_recovery_target_immediate` input enables `targetImmediate` CNPG recovery mode as an alternative to `pg_recovery_target_time`" } ] }, { "id": "672b8c19-8304-489d-bd14-12ac39527c6a", "type": "addition", "summary": "`kube_pg_cluster` adds a `pg_custom_image` input to support custom PostgreSQL container images with extensions.", "description": "The new `pg_custom_image` input (default `null`) allows deploying a custom PostgreSQL container image instead of the default CloudNativePG image selected by `pg_version`. This is useful when you need PostgreSQL extensions that are not included in the standard CNPG images.\n\nSupported use cases include:\n\n1. Pre-built CNPG extension images such as `ghcr.io/cloudnative-pg/postgis:17` for PostGIS support\n2. Custom-built images with arbitrary Debian-packaged extensions (e.g., `pgvector`, `pg_cron`, `TimescaleDB`) assembled via Docker Bake\n\nWhen `pg_custom_image` is set, `pg_version` is ignored for image selection. Note that custom images bypass the ECR pull-through cache, so the cluster must have direct registry access and any required image pull secrets configured.\n", "references": [ { "type": "internal-commit", "summary": "feat(kube_pg_cluster): support custom_image (#376)", "link": "https://github.com/Panfactum/stack/commit/5809f908864efcc93847361fabefc7abfa764d61" }, { "type": "external-docs", "summary": "CloudNativePG blog: building custom PostgreSQL images with Docker Bake", "link": "https://cloudnative-pg.io/blog/building-images-bake/" }, { "type": "internal-docs", "summary": "`kube_pg_cluster` module documentation", "link": "/docs/main/reference/infrastructure-modules/submodule/kubernetes/kube_pg_cluster" } ], "impacts": [ { "type": "iac-module", "component": "kube_pg_cluster", "summary": "New optional `pg_custom_image` input overrides `pg_version` for image selection when set" } ] }, { "id": "782bf2e9-c7d9-4dd5-a6b6-2ea1c03ae1f4", "type": "fix", "summary": "Fixes `kube_pg_cluster` cross-account recovery — missing IAM permissions on remote bucket and `recoveryTarget` schema validation error.", "description": "Two recovery fixes from the same commit:\n\n1. When using `pg_recovery_bucket` to restore from a remote S3 bucket, the cluster's IAM role lacked `s3:*` permissions on that bucket, causing authentication failures. The IAM policy now includes the remote bucket and its objects when `pg_recovery_bucket` is set.\n\n2. The `recoveryTarget` block conditionally set only some fields, triggering a `Manifest configuration incompatible with resource schema` planning error because the Terraform `kubernetes` provider requires all nested object fields to be explicitly declared (even as `null`). The fix uses `merge()` to provide all `recoveryTarget` fields and corrects `backendID` to `backupID` to match the actual CNPG CRD field name.\n", "references": [ { "type": "internal-commit", "summary": "fix: kube pg cluster recover from remote backups (#377)", "link": "https://github.com/Panfactum/stack/commit/b1168e61fb424a8e66a542c5c232721195300cfc" }, { "type": "issue-report", "summary": "Cross-account PG cluster recovery — missing IAM permissions on remote S3 bucket", "link": "https://github.com/Panfactum/stack/issues/378" }, { "type": "external-docs", "summary": "CloudNativePG backup and recovery documentation", "link": "https://cloudnative-pg.io/documentation/1.25/recovery/" }, { "type": "internal-docs", "summary": "`kube_pg_cluster` module documentation", "link": "/docs/main/reference/infrastructure-modules/submodule/kubernetes/kube_pg_cluster" } ], "impacts": [ { "type": "iac-module", "component": "kube_pg_cluster", "summary": "IAM policy now grants `s3:*` on the `pg_recovery_bucket` when set; `recoveryTarget` block uses `merge()` with all fields explicitly declared; `backendID` corrected to `backupID` to match the CNPG CRD field name." } ] }, { "id": "35fd07ac-6319-4f91-9c30-f72bb76e3aee", "type": "addition", "summary": "`aws_ecr_repos` now accepts a `lifecycle_policy_json` input to override the default ECR lifecycle policy per repository.", "description": "The built-in expiration rules in `aws_ecr_repos` (`expire_all_images` and `expiration_rules`) cover common cases but are too rigid for teams that need fine-grained control — for example, keeping a minimum number of tagged images regardless of age, or applying complex multi-rule policies. The new `lifecycle_policy_json` field accepts a raw JSON string (matching the AWS ECR lifecycle policy format) and, when set, replaces the module-generated policy entirely. Existing repositories that do not set this field continue to behave exactly as before.", "references": [ { "type": "internal-commit", "summary": "feat(aws_ecr_repos): support custom lifecycle policies (#379)", "link": "https://github.com/Panfactum/stack/commit/292b996bfdc0aef6c102493da0dbe4221548d0e3" }, { "type": "issue-report", "summary": "Support custom ECR lifecycle policies in aws_ecr_repos", "link": "https://github.com/Panfactum/stack/issues/381" }, { "type": "internal-docs", "summary": "`aws_ecr_repos` module documentation", "link": "/docs/main/reference/infrastructure-modules/direct/aws/aws_ecr_repos" }, { "type": "external-docs", "summary": "AWS ECR lifecycle policy documentation", "link": "https://docs.aws.amazon.com/AmazonECR/latest/userguide/LifecyclePolicies.html" } ], "impacts": [ { "type": "iac-module", "component": "aws_ecr_repos", "summary": "New optional `lifecycle_policy_json` field on each repository object; overrides built-in expiration rules when set" } ] }, { "id": "efd2a6c8-3a5c-49a1-b01c-34b30870a394", "type": "update", "summary": "The Panfactum DevShell now ships Node.js v25, upgraded from v22.", "description": "Node.js v25 includes V8 14.1 with significant `JSON.stringify` performance improvements, built-in `Uint8Array` base64/hex conversion, and other runtime enhancements. The upgrade also refreshes the primary `nixpkgs` input to a newer revision, ensuring compatibility with the `nodejs_25` package.", "references": [ { "type": "internal-commit", "summary": "Upgrade Node.js from v22 to v25 and refresh nixpkgs input", "link": "https://github.com/Panfactum/stack/commit/eced1ab337f51c01a7a045c34a372cf1aae95321" }, { "type": "external-docs", "summary": "Node.js v25.0.0 release notes", "link": "https://nodejs.org/en/blog/release/v25.0.0" }, { "type": "external-docs", "summary": "Node.js v25 changelog", "link": "https://github.com/nodejs/node/blob/main/doc/changelogs/CHANGELOG_V25.md" } ], "impacts": [ { "type": "devshell", "component": "enter-shell-local", "summary": "Node.js runtime upgraded from `nodejs_22` to `nodejs_25`" } ] }, { "id": "57e371a1-6ebe-4e18-9699-d4e15240958a", "type": "fix", "summary": "Updates several modules to use Terragrunt's current `TG_*` env var prefix and `--all` flag, replacing deprecated `TERRAGRUNT_*` and `run-all` syntax.", "description": "Terragrunt deprecated both the `TERRAGRUNT_*` environment variable prefix (replaced by `TG_*`) and the `run-all` subcommand (replaced by `--all` flags on native commands). This change updates all affected components to stay compatible with current and future Terragrunt releases:\n\n1. The DevShell entry-point and `loadEnv.sh` now export `TG_DOWNLOAD_DIR`, `TG_DEPENDENCY_FETCH_OUTPUT_FROM_STATE`, and `TG_TF_FORWARD_STDOUT` instead of their `TERRAGRUNT_*` predecessors.\n2. `pf-tf-init` calls `terragrunt init --all --queue-exclude-external` and `terragrunt run --all -- providers lock` instead of the legacy `run-all` variants.\n3. `wf_tf_deploy` `deploy.sh` calls `terragrunt apply --all --queue-exclude-external` instead of `terragrunt run-all apply --terragrunt-ignore-external-dependencies`.\n", "action_items": [ "Re-apply `wf_tf_deploy` to pick up the updated `deploy.sh` script.", "Re-run `pf-tf-init` to regenerate provider lock files using the new Terragrunt syntax." ], "references": [ { "type": "internal-commit", "summary": "refactor(hooks): move formatters and linters to Stop hook for batch runs", "link": "https://github.com/Panfactum/stack/commit/f2dd861ecf213bfef61cabebe1a75e6ea9843abf" }, { "type": "external-docs", "summary": "Terragrunt CLI Redesign migration guide", "link": "https://terragrunt.gruntwork.io/docs/migrate/cli-redesign" }, { "type": "external-docs", "summary": "Terragrunt PR #4233: Deprecating run-all command", "link": "https://github.com/gruntwork-io/terragrunt/pull/4233" }, { "type": "external-docs", "summary": "Terragrunt PR #3964: Strict control for old deprecated env vars", "link": "https://github.com/gruntwork-io/terragrunt/pull/3964" }, { "type": "internal-docs", "summary": "`wf_tf_deploy` module documentation", "link": "/docs/main/modules/wf_tf_deploy/overview" } ], "impacts": [ { "type": "devshell", "component": "enter-shell-local", "summary": "`TERRAGRUNT_DOWNLOAD`, `TERRAGRUNT_FETCH_DEPENDENCY_OUTPUT_FROM_STATE`, and `TERRAGRUNT_FORWARD_TF_STDOUT` renamed to `TG_DOWNLOAD_DIR`, `TG_DEPENDENCY_FETCH_OUTPUT_FROM_STATE`, and `TG_TF_FORWARD_STDOUT` respectively" }, { "type": "devshell", "component": "pf-tf-init", "summary": "`terragrunt run-all init` replaced with `terragrunt init --all --queue-exclude-external`; providers lock migrated to `terragrunt run --all -- providers lock`" }, { "type": "iac-module", "component": "wf_tf_deploy", "summary": "`deploy.sh` now uses `terragrunt apply --all --queue-exclude-external` instead of the deprecated `terragrunt run-all apply --terragrunt-ignore-external-dependencies`" } ] }, { "id": "545764fb-149e-408b-8f30-d7230f1d9d88", "type": "update", "summary": "All `authentik_*` modules updated to `goauthentik/authentik` provider 2024.10.2 with `invalidation_flow` support for SAML and OAuth2 providers.", "description": "All Authentik IaC modules now require `goauthentik/authentik` provider 2024.10.2 (up from 2024.8.4) to align with the Authentik 2024.10 Helm chart. Authentik 2024.10 introduced a required `invalidation_flow` field on all providers for proper session invalidation and logout handling. This update adds the `default-provider-invalidation-flow` to every SAML and OAuth2 provider across all SSO modules.\n\nAdditionally, `authentik_vault_sso` adapts to a breaking change in the provider API where `redirect_uris` was replaced by `allowed_redirect_uris`, and the output for `oidc_redirect_uris` is updated to extract the `.url` field from the new structured URI format.\n", "action_items": [ "Run `pf-tf-init` in each affected module directory to pick up the new provider version.", "Re-apply all `authentik_*` modules and `mongodb_atlas_identity_provider` with `terragrunt apply`." ], "references": [ { "type": "internal-commit", "summary": "refactor(hooks): move formatters and linters to Stop hook for batch runs", "link": "https://github.com/Panfactum/stack/commit/f2dd861ecf213bfef61cabebe1a75e6ea9843abf" }, { "type": "external-docs", "summary": "Authentik 2024.10 release notes", "link": "https://docs.goauthentik.io/releases/2024.10/" }, { "type": "external-docs", "summary": "`goauthentik/terraform-provider-authentik` v2024.10.2 release", "link": "https://github.com/goauthentik/terraform-provider-authentik/releases/tag/v2024.10.2" }, { "type": "external-docs", "summary": "Authentik official website", "link": "https://goauthentik.io/" } ], "impacts": [ { "type": "iac-module", "component": "authentik_aws_sso", "summary": "`goauthentik/authentik` provider bumped from 2024.8.4 to 2024.10.2; SAML provider now sets `invalidation_flow` to `default-provider-invalidation-flow` as required by Authentik 2024.10" }, { "type": "iac-module", "component": "authentik_core_resources", "summary": "`goauthentik/authentik` provider bumped from 2024.8.4 to 2024.10.2" }, { "type": "iac-module", "component": "authentik_github_sso", "summary": "`goauthentik/authentik` provider bumped from 2024.8.4 to 2024.10.2; SAML provider now sets `invalidation_flow` to `default-provider-invalidation-flow`" }, { "type": "iac-module", "component": "authentik_mongodb_atlas_sso", "summary": "`goauthentik/authentik` provider bumped from 2024.8.4 to 2024.10.2; SAML provider now sets `invalidation_flow` to `default-provider-invalidation-flow`" }, { "type": "iac-module", "component": "authentik_vault_sso", "summary": "`goauthentik/authentik` provider bumped from 2024.8.4 to 2024.10.2; OAuth2 provider now sets `invalidation_flow` and uses `allowed_redirect_uris` instead of `redirect_uris`" }, { "type": "iac-module", "component": "authentik_zoho_sso", "summary": "`goauthentik/authentik` provider bumped from 2024.8.4 to 2024.10.2; SAML provider now sets `invalidation_flow` to `default-provider-invalidation-flow`" }, { "type": "iac-module", "component": "mongodb_atlas_identity_provider", "summary": "`goauthentik/authentik` provider bumped from 2024.8.4 to 2024.10.2" } ] }, { "id": "3294ce58-1fbd-49d0-be3c-7fca2a3c5d15", "type": "improvement", "summary": "`ds-generate-changelog-schemas` no longer generates `metadata.schema.json`; that responsibility has moved to `ds-validate-iac-metadata`.", "description": "Previously `ds-generate-changelog-schemas` generated both `log.schema.json` and `metadata.schema.json`. Unlike `log.schema.json`, whose component enums must be introspected from infrastructure module directories and CLI commands, `metadata.schema.json` is entirely static — it has no dynamic enum values derived from the filesystem. Generating it programmatically added complexity with no benefit. The metadata schema is now owned by the separate `ds-validate-iac-metadata` tool, keeping each script focused on a single responsibility.", "references": [ { "type": "internal-commit", "summary": "refactor(changelog): remove metadata schema gen; add contributor docs", "link": "https://github.com/Panfactum/stack/commit/d0166798440a5eea576ef2e986ea7c875e82b519" } ] }, { "id": "df8a13fd-ae3f-4493-8bef-440e4f2c72be", "type": "update", "summary": "Upgrades all OpenTofu providers, including major bumps for AWS (5.x to 6.x) and Helm (2.x to 3.x).", "description": "Provider versions across all Panfactum infrastructure modules were stale, accumulating security and compatibility debt. This upgrade brings every module to current provider releases in a single coordinated change across approximately 105 modules.\n\nKey upgrades:\n- **AWS** (`hashicorp/aws`): 5.80.0 to 6.38.0 (major). The 6.x release drops deprecated resources and attributes and enforces stricter typing. Boolean-string literals in the `aws_eks` launch template (`delete_on_termination`, `encrypted`) were corrected from quoted strings to native booleans to satisfy the new requirements. The `data.aws_region` output attribute was also renamed from `.name` to `.region`, and the Helm provider template syntax changed from block to assignment (`kubernetes = {`).\n- **Helm** (`hashicorp/helm`): 2.12.1 to 3.1.1 (major). The `set_string` attribute is removed in 3.x, and the provider includes an automatic state migration from SDKv2 to the Terraform Plugin Framework.\n- **Vault** (`hashicorp/vault`): 4.5.0 to 4.8.0. Held at 4.x because 5.x drops support for Vault 1.14.x, which is the last open-source release currently deployed.\n- **Kubernetes** (`hashicorp/kubernetes`): 2.34.0 to 2.35.0. Held at 2.x because 3.x renames every resource type with a `_v1` suffix and has no declarative state migration path.\n- **Minor/patch**: `tls` 4.0.6 to 4.2.1, `random` 3.6.3 to 3.8.1, `time` 0.10.0 to 0.13.1, `archive` 2.6.0 to 2.7.1.\n\nNo user action is required. Provider pins are managed internally by Panfactum and are picked up automatically on the next `terragrunt init`.\n", "references": [ { "type": "internal-commit", "summary": "chore(infrastructure): upgrade OpenTofu provider versions across all modules", "link": "https://github.com/Panfactum/stack/commit/18d7371cab0ca2d4f0543e21c6484b6980120ce2" }, { "type": "internal-commit", "summary": "chore(terraform): bump AWS provider 5.x to 6.x across all environments", "link": "https://github.com/Panfactum/stack/commit/43989c48de224cccb1665bbf2eced22fd4166499" }, { "type": "external-docs", "summary": "AWS Provider v6.0.0 release notes", "link": "https://github.com/hashicorp/terraform-provider-aws/releases/tag/v6.0.0" }, { "type": "external-docs", "summary": "AWS Provider v6 upgrade guide", "link": "https://registry.terraform.io/providers/-/aws/latest/docs/guides/version-6-upgrade" }, { "type": "external-docs", "summary": "Helm Provider v3.0.0 upgrade guide", "link": "https://registry.terraform.io/providers/hashicorp/helm/latest/docs/guides/v3-upgrade-guide" }, { "type": "external-docs", "summary": "Helm Provider v3.0.0 release notes", "link": "https://github.com/hashicorp/terraform-provider-helm/releases/tag/v3.0.0" } ], "impacts": [ { "type": "iac-provider", "component": "aws", "summary": "Upgraded from 5.80.0 to 6.38.0 (major); boolean-string literals in `aws_eks` launch template corrected for 6.x strict type requirements" }, { "type": "iac-provider", "component": "aws_global", "summary": "Upgraded from 5.80.0 to 6.38.0 (major)" }, { "type": "iac-provider", "component": "aws_secondary", "summary": "Upgraded from 5.80.0 to 6.38.0 (major)" }, { "type": "iac-provider", "component": "helm", "summary": "Upgraded from 2.12.1 to 3.1.1 (major); `set_string` removed, automatic state migration from SDKv2 to Plugin Framework" }, { "type": "iac-provider", "component": "vault", "summary": "Upgraded from 4.5.0 to 4.8.0; held at 4.x because 5.x drops support for Vault 1.14.x" }, { "type": "iac-provider", "component": "kubernetes", "summary": "Upgraded from 2.34.0 to 2.35.0; held at 2.x because 3.x renames every resource type with a `_v1` suffix" }, { "type": "iac-provider", "component": "tls", "summary": "Upgraded from 4.0.6 to 4.2.1" }, { "type": "iac-provider", "component": "random", "summary": "Upgraded from 3.6.3 to 3.8.1" }, { "type": "iac-provider", "component": "time", "summary": "Upgraded from 0.10.0 to 0.13.1" } ] }, { "id": "a6765460-0030-4d71-8213-68380f9f42ec", "type": "addition", "summary": "Adds optional `seccomp_profile_type` field to container specs in all `kube_*` workload modules, enabling per-container seccomp profile overrides.", "description": "Some workloads require a non-default seccomp profile to function correctly, but previously there was no way\nto override the cluster-level seccomp policy on a per-container basis. For example, BuildKit uses `rootlesskit`\nwith `SYS_ADMIN` capabilities whose syscall surface exceeds what `RuntimeDefault` allows, causing silent\nfailures or permission-denied errors at runtime.\n\nThe new `seccomp_profile_type` field defaults to `null` (no explicit profile override), allowing callers to\nspecify any valid Kubernetes seccomp profile type string such as `RuntimeDefault`, `Unconfined`, or\n`Localhost`. `kube_buildkit` uses this to set `Unconfined`, matching the existing AppArmor `unconfined`\nannotation already applied to its pod. No changes are required for existing modules that do not need a\ncustom seccomp profile.\n", "references": [ { "type": "internal-commit", "summary": "feat(infrastructure): add seccomp_profile_type to container specs", "link": "https://github.com/Panfactum/stack/commit/c6e79ec355572f2d89ada7ceaac8b1aab481cc23" }, { "type": "external-docs", "summary": "Kubernetes seccomp profiles documentation", "link": "https://kubernetes.io/docs/tutorials/security/seccomp/" }, { "type": "internal-docs", "summary": "`kube_pod` module documentation", "link": "/docs/main/modules/kube_pod" }, { "type": "internal-docs", "summary": "`kube_buildkit` module documentation", "link": "/docs/main/modules/kube_buildkit" } ], "impacts": [ { "type": "iac-module", "component": "kube_pod", "summary": "New optional `seccomp_profile_type` field (default `null`) on each container object; when set, applies a container-level `seccompProfile` override in the pod security context" }, { "type": "iac-module", "component": "kube_deployment", "summary": "Exposes new optional `seccomp_profile_type` field on each container object, propagated to the underlying `kube_pod`" }, { "type": "iac-module", "component": "kube_daemon_set", "summary": "Exposes new optional `seccomp_profile_type` field on each container object, propagated to the underlying `kube_pod`" }, { "type": "iac-module", "component": "kube_stateful_set", "summary": "Exposes new optional `seccomp_profile_type` field on each container object, propagated to the underlying `kube_pod`" }, { "type": "iac-module", "component": "kube_cron_job", "summary": "Exposes new optional `seccomp_profile_type` field on each container object, propagated to the underlying `kube_pod`" }, { "type": "iac-module", "component": "kube_job", "summary": "Exposes new optional `seccomp_profile_type` field on each container object, propagated to the underlying `kube_pod`" }, { "type": "iac-module", "component": "kube_buildkit", "summary": "Sets `seccomp_profile_type` to `Unconfined` to satisfy BuildKit's elevated syscall requirements" } ] }, { "id": "7006e65f-f90a-4988-82fb-e98ca68b1671", "type": "fix", "summary": "Panfactum Docker image now includes the CLI tools and `/usr/bin/env`, fixing broken script execution inside the container.", "description": "The `flake.nix` Docker image definition called `panfactumPackages` without the required `true` argument, which\nmeant the Panfactum CLI tools were omitted from the built image. Additionally, `usrBinEnv` was not included in the\nimage contents, so scripts using `#!/usr/bin/env` shebangs could not locate their interpreters.\n\nThis fix passes `panfactumPackages true` to include the full CLI toolset and adds `usrBinEnv` to provide the\n`/usr/bin/env` symlink, restoring correct binary resolution inside the container.\n", "references": [ { "type": "internal-commit", "summary": "build(flake): update Docker image root env path linking", "link": "https://github.com/Panfactum/stack/commit/606d656bd5c1742eee0cb9ba9c4b940240328ddd" } ], "impacts": [ { "type": "configuration", "component": "flake.nix", "summary": "`panfactumPackages` call corrected to `panfactumPackages true` and `usrBinEnv` added to image contents so `/usr/bin/env` is available inside the container" } ] }, { "id": "e8a8d9e7-b78c-4e7a-b26c-ffefbb40bb3e", "type": "update", "summary": "The bastion host image is upgraded from `debian:bookworm` to `debian:trixie` for a newer OpenSSH version with up-to-date security patches.", "description": "Debian Trixie (Debian 13) ships a significantly newer OpenSSH release compared to Bookworm. Upgrading\nthe bastion base image ensures the SSH daemon includes the latest security fixes and cryptographic\nimprovements without requiring users to rebuild or customize the image themselves.\n\nThe `Protocol 2` directive has also been removed from `sshd_config` because modern OpenSSH versions\nno longer support SSH Protocol 1, making the directive unnecessary.\n", "references": [ { "type": "internal-commit", "summary": "feat(bastion): upgrade to debian trixie and harden sshd config", "link": "https://github.com/Panfactum/stack/commit/1f80a81aa5c1be1437e4ec6bb00761500a0f4c8e" }, { "type": "external-docs", "summary": "Debian Trixie (Debian 13) release information", "link": "https://www.debian.org/releases/trixie/" }, { "type": "internal-docs", "summary": "`kube_bastion` module documentation", "link": "/docs/main/reference/infrastructure-modules/kubernetes/kube_bastion" } ], "impacts": [ { "type": "iac-module", "component": "kube_bastion", "summary": "Base container image upgraded from `debian:bookworm` to `debian:trixie`, providing a newer OpenSSH release with current security patches. The `Protocol 2` sshd directive is removed as it is no longer recognized by modern OpenSSH versions." } ] }, { "id": "fa55d1ce-c6cb-471d-858c-30a7fb9cb3ea", "type": "fix", "summary": "Fixes ten `pf` CLI commands that incorrectly required DevShell initialization, blocking execution in CI/CD and automation contexts.", "description": "Several `pf` CLI commands intended for use in CI/CD pipelines and automation containers were\nincorrectly requiring a full DevShell environment. When these commands ran outside the DevShell\n(e.g., inside Argo workflow containers or CI runners), the CLI called `git rev-parse --show-toplevel`\nto load DevShell configuration and failed immediately because no repository or DevShell context existed.\n\nThe fix introduces a lighter initialization path that skips DevShell configuration loading for commands\nthat do not need it. Each affected command now opts out of the full initialization, allowing it to run\nwith only the base context (logger, analytics, process manager). The following commands are now\nDevShell-independent:\n\n- `pf wf git-checkout`\n- `pf wf sops-set-profile`\n- `pf util get-commit-hash`\n- `pf buildkit scale up`\n- `pf iac update-module-status`\n- `pf kube disable-disruptions`\n- `pf kube enable-disruptions`\n- `pf kube get-token`\n- `pf k8s velero snapshot-gc`\n- `pf util get-module-hash`\n", "references": [ { "type": "internal-commit", "summary": "fix(cli): allow wf commands to run without a devshell", "link": "https://github.com/Panfactum/stack/commit/e45d7351a4736a7142100f63a8a23e19cc71d1c6" }, { "type": "internal-commit", "summary": "chore(cicd): set memory limits on image builders and fix devshell flag", "link": "https://github.com/Panfactum/stack/commit/c44313b973c668fc4c0a526ca9cdb62e9b84a320" }, { "type": "internal-commit", "summary": "fix(buildkit): allow scale-up command to run without devshell", "link": "https://github.com/Panfactum/stack/commit/02becf232c45cdeae884c24a05d59c6e009c4fc9" }, { "type": "internal-commit", "summary": "fix(cli): mark several commands as devshell-independent", "link": "https://github.com/Panfactum/stack/commit/7b44c3a891a3c733bc752f74b13955af8d74b2c8" } ], "impacts": [ { "type": "cli", "component": "wf git-checkout", "summary": "No longer requires DevShell; runs correctly in CI/CD containers before a repository is checked out" }, { "type": "cli", "component": "wf sops-set-profile", "summary": "No longer requires DevShell; runs correctly in CI/CD containers for SOPS profile configuration" }, { "type": "cli", "component": "util get-commit-hash", "summary": "No longer requires DevShell; resolves git refs to commit SHAs in CI and pre-DevShell contexts" }, { "type": "cli", "component": "buildkit scale up", "summary": "No longer requires DevShell; scales up BuildKit instances from CI pipelines and automation scripts" }, { "type": "cli", "component": "iac update-module-status", "summary": "No longer requires DevShell; tracks module deployment status from CI pipelines" }, { "type": "cli", "component": "kube disable-disruptions", "summary": "No longer requires DevShell; disables voluntary pod disruptions from automation workflows" }, { "type": "cli", "component": "kube enable-disruptions", "summary": "No longer requires DevShell; enables maintenance-window disruptions from automation workflows" }, { "type": "cli", "component": "kube get-token", "summary": "No longer requires DevShell; `kubectl` credential plugins now work in CI environments" }, { "type": "cli", "component": "k8s velero snapshot-gc", "summary": "No longer requires DevShell; snapshot garbage collection runs in non-DevShell containers" }, { "type": "cli", "component": "util get-module-hash", "summary": "No longer requires DevShell; computes module hashes in CI change-detection scripts" } ] }, { "id": "87822b32-4951-44b0-ab9a-bdcd211fe844", "type": "improvement", "summary": "The Nix image builder Argo workflow now skips redundant builds when the target image already exists in ECR.", "description": "A new `check-image` pre-flight DAG step resolves the target Git ref to a commit SHA and queries ECR for existing per-architecture images and the multi-arch manifest before any build work begins.\n\n- The `build-and-push`, `build-and-push-arm`, and `merge-and-copy` tasks are now conditionally executed based on output flags from `check-image`.\n- Re-triggering a workflow for a SHA that was already built results in a no-op instead of re-building identical layers.\n\nThis saves compute time and avoids unnecessary ECR pushes when workflows are re-triggered for commits that have already been built.\n", "references": [ { "type": "internal-commit", "summary": "feat(nix-image-builder): skip redundant builds if image already exists in ECR", "link": "https://github.com/Panfactum/stack/commit/884e7e5056f7b72b8d07b94d8a7a3ca16cbc9efc" } ], "impacts": [ { "type": "iac-module", "component": "kube_constants", "summary": "Internal image tag updated for the Nix image builder with idempotent ECR check" } ] }, { "id": "de994766-1a79-4a61-b9cf-7a614666eef3", "type": "fix", "summary": "Fixes `pf` CLI error exit code and updates CI workflow scripts to use renamed CLI subcommands.", "description": "Three fixes from the same commit:\n\n1. The `pf` CLI now sets `process.exitCode = 1` in the top-level error handler so that callers and CI pipelines can detect failures reliably. Previously, unhandled errors were logged but the process exited with code 0, masking failures in automation.\n2. `wf_dockerfile_build` `scale-buildkit.sh` now calls `pf buildkit scale up` instead of the removed `pf buildkit resume` subcommand, which was renamed in an earlier release.\n3. `wf_tf_deploy` `deploy.sh` now calls `pf wf sops-set-profile` with positional arguments instead of the removed `--directory` and `--profile` flags, aligning with the updated CLI interface.\n", "references": [ { "type": "internal-commit", "summary": "fix: correct error handling and CI workflow gaps", "link": "https://github.com/Panfactum/stack/commit/ddd7657e87f776df75cb102ec19447a732fff553" }, { "type": "internal-docs", "summary": "`wf_dockerfile_build` module documentation", "link": "/docs/main/reference/infrastructure-modules/submodule/workflow/wf_dockerfile_build" }, { "type": "internal-docs", "summary": "`wf_tf_deploy` module documentation", "link": "/docs/main/reference/infrastructure-modules/submodule/workflow/wf_tf_deploy" } ], "impacts": [ { "type": "devshell", "component": "pf", "summary": "Top-level error handler now sets `process.exitCode = 1` so CLI failures propagate a non-zero exit code" }, { "type": "iac-module", "component": "wf_dockerfile_build", "summary": "`scale-buildkit.sh` now calls `pf buildkit scale up` instead of the removed `pf buildkit resume`" }, { "type": "iac-module", "component": "wf_tf_deploy", "summary": "`deploy.sh` `sops-set-profile` call updated to positional argument syntax (removed `--directory` and `--profile` flags)" } ] }, { "id": "3ddac59b-8f07-4b08-80c3-3d1366d1e6b9", "type": "improvement", "summary": "`pf wf sops-set-profile` now skips YAML parsing for non-SOPS files via a regex pre-check, reducing overhead in large directories.", "description": "When scanning directories, most files are not SOPS-encrypted. Previously, every YAML file was fully parsed to detect the `sops:` key before determining whether to skip it. A `fileContains` regex check now runs first and bails out immediately when no `sops:` line is found, avoiding the expensive YAML parse for the majority of files. This significantly reduces execution time for `pf wf sops-set-profile` in repositories with many non-encrypted YAML files.", "references": [ { "type": "internal-commit", "summary": "perf(sops-set-profile): skip YAML parsing for non-SOPS files early", "link": "https://github.com/Panfactum/stack/commit/81808b61f0e73ac032b9f27700f2facbc18a9061" } ], "impacts": [ { "type": "cli", "component": "wf sops-set-profile", "summary": "Faster execution in repositories with many non-encrypted YAML files due to a regex pre-check that skips full YAML parsing" } ] }, { "id": "d25833a1-b552-4ae6-97bc-4c0332a59596", "type": "fix", "summary": "Fixes `wf_tf_deploy` Terragrunt flags and `pf wf sops-set-profile` non-KMS SOPS schema crash.", "description": "Two fixes from the same commit:\n\n1. `wf_tf_deploy` `deploy.sh` was using deprecated `--terragrunt-*` flags that newer Terragrunt versions no longer accept. All flags have been updated to their short-form equivalents (`--download-dir`, `--non-interactive`, `--dependency-fetch-output-from-state`, `--provider-cache`, `--provider-cache-dir`, `--parallelism`).\n2. `pf wf sops-set-profile` crashed on SOPS files encrypted with non-KMS backends (e.g., `age`) because the Zod schema for the inner `sops` object was strict and rejected extra fields. Adding `.passthrough()` tolerates unknown fields so non-KMS configurations no longer cause validation errors.\n", "action_items": [ "Re-apply the `wf_tf_deploy` module to deploy the updated `deploy.sh` script with the correct Terragrunt flag format." ], "references": [ { "type": "internal-commit", "summary": "fix(wf): migrate terragrunt flags to new CLI format", "link": "https://github.com/Panfactum/stack/commit/cfb4c739cf687095795a2e2a8a0439ef408858fc" }, { "type": "issue-report", "summary": "Removal of legacy Terragrunt flags (all flags that start with `terragrunt-`)", "link": "https://github.com/gruntwork-io/terragrunt/issues/4597" }, { "type": "internal-docs", "summary": "`wf_tf_deploy` module reference", "link": "/docs/main/reference/infrastructure-modules/submodule/workflow/wf_tf_deploy" } ], "impacts": [ { "type": "iac-module", "component": "wf_tf_deploy", "summary": "`deploy.sh` now uses short-form Terragrunt flags (`--download-dir`, `--non-interactive`, etc.) instead of deprecated `--terragrunt-*` equivalents" }, { "type": "cli", "component": "wf sops-set-profile", "summary": "Inner `sops` Zod schema now uses `.passthrough()` to tolerate unknown fields from non-KMS SOPS configurations" } ] }, { "id": "fb8ff96f-fe3f-43d3-b667-4246f073341b", "type": "fix", "summary": "The DevShell `terragrunt` wrapper no longer runs the `GIT_PASSWORD` redaction pipeline in CI when `GIT_PASSWORD` is unset.", "description": "Previously the custom `terragrunt` wrapper unconditionally piped all CI output through a `sed` substitution to redact `GIT_PASSWORD`. When `GIT_PASSWORD` was empty, this produced an empty `sed` pattern, which is a no-op at best and can cause unexpected pipe behavior at worst. The wrapper now checks that `GIT_PASSWORD` is non-empty before activating the redaction pipeline, and falls through to a plain `exec` path in CI when no password is present.", "references": [ { "type": "internal-commit", "summary": "fix(terragrunt): skip secret redaction when GIT_PASSWORD is unset", "link": "https://github.com/Panfactum/stack/commit/9e8754ede8059150ab93bd2f9515e9df2dc61f12" } ], "impacts": [ { "type": "devshell", "component": "terragrunt", "summary": "Skips the `sed`-based `GIT_PASSWORD` redaction when `GIT_PASSWORD` is unset in CI" } ] } ], "on_upgrade_path": true, "list_url": "/docs/changelog/edge.json", "llm_txt_url": "/docs/changelog/edge.26-04-05/llm.txt", "next": "/docs/changelog/edge.26-04-24.json", "prev": "/docs/changelog/edge.25-04-03.json" }