# Panfactum Changelog — edge.24-11-13

> Introduces Kyverno policy engine as a core component, replaces Fledged and Reflector with Kyverno-based alternatives, improves PostgreSQL autoscaling capabilities, and enhances pod scheduling with automatic ARM64 and spot instance tolerations.

## Highlights

- Kyverno policy engine added as a core Panfactum component — **must install before other modules work**
- `kube_fledged` and `kube_reflector` deprecated — must remove before upgrading to next version
- `pg_memory_mb` and `pg_cpu_millicores` replaced with min/max VPA inputs in `kube_pg_cluster`
- All pods now automatically tolerate arm64 and spot node taints cluster-wide

## Breaking Changes

- The [Kyverno](https://kyverno.io/docs/introduction/) policy engine has been added as a core part of the Panfactum Stack. **You must install Kyverno** by
  - Impacts: iac-module `kube_kyverno` — New core module that must be installed before other modules
- `kube_fledged` has been replaced by a new node-local image caching mechanism built on top of Kyverno. `kube_fledged` must be removed
  - Impacts: iac-module `kube_kyverno` — Now provides node-local image caching replacing kube_fledged
- `kube_reflector` has been replaced with Kyverno-based syncing. Must be removed before the next version. Use [`kube_sync_config_map`](https://panfactum.com/docs/edge/reference/infrastructure-modules/submodule/kubernetes/kube_sync_config_map) and [`kube_sync_secret`](https://panfactum.com/docs/edge/reference/infrastructure-modules/submodule/kubernetes/kube_sync_secret) instead.
  - Impacts: iac-module `kube_sync_config_map` — New Kyverno-based replacement for kube_reflector ConfigMap syncing
  - Impacts: iac-module `kube_sync_secret` — New Kyverno-based replacement for kube_reflector Secret syncing
  - Impacts: iac-module `kube_kyverno` — Now handles resource syncing replacing kube_reflector
- Vertical pod autoscaling now works for PostgreSQL clusters in [`kube_pg_cluster`](https://panfactum.com/docs/edge/reference/infrastructure-modules/submodule/kubernetes/kube_pg_cluster). `pg_memory_mb` and `pg_cpu_millicores` have been removed and replaced with min/max
  - Impacts: iac-module `kube_pg_cluster` — Replaced pg_memory_mb and pg_cpu_millicores with VPA min/max inputs
- All pods in Panfactum clusters now automatically tolerate `arm64` and `spot` node taints. To disable for a specific pod, add
  - Impacts: iac-module `kube_kyverno` — Adds cluster-wide arm64 and spot toleration policies

## Additions

- Adds a new submodule, [`kube_daemon_set`](https://panfactum.com/docs/edge/reference/infrastructure-modules/submodule/kubernetes/kube_daemon_set), for creating Kubernetes DaemonSets.
  - Impacts: iac-module `kube_daemon_set` — New submodule for creating Kubernetes DaemonSets

## Improvements

- Upgraded the CNPG operator to 1.24, adding additional stability improvements during failover events. The old `kube_pg_cluster` submodule is no longer
  - Impacts: iac-module `kube_cloudnative_pg` — Upgraded CNPG operator to 1.24 with failover stability fixes
  - Impacts: iac-module `kube_pg_cluster` — Updated for compatibility with CNPG operator 1.24
- Upgraded the default PostgreSQL version to 16.4.
  - Impacts: iac-module `kube_pg_cluster` — Default PostgreSQL version upgraded to 16.4

## Fixes

- Added Kyverno rule that forces Linkerd sidecars to terminate prior to `terminationGracePeriodSeconds`.
  - Impacts: iac-module `kube_kyverno` — Added policy for Linkerd sidecar termination ordering
  - Impacts: iac-module `kube_linkerd` — Sidecars now terminate before terminationGracePeriodSeconds
  - Reference (issue-report): [Linkerd sidecar not terminating before grace period](https://github.com/Panfactum/stack/issues/164)
  - Reference (issue-report): [Pod termination blocked by Linkerd sidecar](https://github.com/Panfactum/stack/issues/148)
- Adjusted Cilium deployment to address edge cases where Cilium would not successfully launch new nodes after a complete zonal or
  - Impacts: iac-module `kube_cilium` — Fixed node launch failures after zonal or cluster outages

## Upgrade Instructions

import MarkdownAlert from "@/components/markdown/MarkdownAlert.astro"

<MarkdownAlert severity="warning">
  This release introduces [Kyverno](https://kyverno.io/docs/introduction/). We discovered several issues
  with the initial architecture that could cause degenerate cluster behavior eventually resulting in a full cluster shutdown.

  Generally, this takes days to occur, so it is safe to upgrade to this release so long as you immediately continue to
  upgrade to subsequent releases where the issues are resolved. All issues were fully resolved in the `edge.25-01-04` release.
</MarkdownAlert>

## Install Kyverno

**You must install Kyverno by following the [new bootstrapping guide section.](https://panfactum.com/docs/edge/guides/bootstrapping/policy-controller)**
Many modules now depend on Kyverno and will not function without it.

## Remove `kube_fledged` and `kube_reflector`

Both `kube_fledged` and `kube_reflector` have been deprecated in favor of Kyverno-based alternatives.

**Important:** These modules must be removed from your clusters **before upgrading to the next version**. Do not
remove them until Kyverno is installed.

* **Replacement for `kube_fledged`:** Follow [this guide](https://panfactum.com/docs/edge/guides/bootstrapping/maintenance-controllers#node-image-caches)
  and use the new [`kube_node_image_cache`](https://panfactum.com/docs/edge/reference/infrastructure-modules/submodule/kubernetes/kube_node_image_cache) module.
* **Replacement for `kube_reflector`:**
  * To sync ConfigMaps, use [`kube_sync_config_map`](https://panfactum.com/docs/edge/reference/infrastructure-modules/submodule/kubernetes/kube_sync_config_map).
  * To sync Secrets, use [`kube_sync_secret`](https://panfactum.com/docs/edge/reference/infrastructure-modules/submodule/kubernetes/kube_sync_secret).

## Update `kube_pg_cluster` VPA Inputs

Vertical pod autoscaling now works for PostgreSQL clusters. The following variables have been removed:

* `pg_memory_mb`
* `pg_cpu_millicores`

And the following variables have been added:

* `pg_minimum_memory_mb`
* `pg_maximum_memory_mb`
* `pg_minimum_cpu_millicores`
* `pg_maximum_cpu_millicores`
* `pgbouncer_minimum_memory_mb`
* `pgbouncer_maximum_memory_mb`
* `pgbouncer_minimum_cpu_millicores`
* `pgbouncer_maximum_cpu_millicores`

## Automatic ARM64 and Spot Tolerations

All pods in Panfactum clusters will now automatically tolerate `arm64` and `spot` node taints regardless of whether
they were created by Panfactum modules. To disable these tolerations for a specific pod, add the labels:

* `panfactum.com/arm64-enabled = "false"` to disable arm64 tolerance
* `panfactum.com/spot-enabled = "false"` to disable spot tolerance

## Related Resources

- [JSON Data](https://panfactum.com/docs/changelog/edge.24-11-13.json): Machine-readable data
- [Channel Release List](https://panfactum.com/docs/changelog/edge.json): All releases in this channel