Internal Cluster Networking

Objective

Install the basic kubernetes cluster networking primitives via the kube_cilium module.

Background

In the Panfactum stack, we use Cilium to handle all of the L3/L4 networking in our Kubernetes cluster.

In this guide, we won't go into detail about the underlying design decisions and network concepts, so we recommend reviewing the concept documentation for more information.

Deploy Cilium

Deploy the Infrastructure Module

1. Create a new directory adjacent to your aws_eks module called kube_cilium.

2. Add a terragrunt.hcl to that directory that looks like this.

3. For now, set vpa_enabled to false. We will enable it when we install the autoscalers.

4. Add a module.yaml that enables the aws, kubernetes, and helm providers.

5. Run terragrunt apply.

6. If the deployment succeeds, you should see the various cilium pods running:

   

Remove the Old Networking Stack

Unfortunately, EKS automatically installs a networking stack with no way to disable it via infrastructure-as-code. We need to remove those components as they will conflict with cilium's operations.

1. Run kubectl -n kube-system delete ds aws-node. This remove the AWS VPC CNI (cilium already contains similar functionality).

2. Run kubectl -n kube-system delete ds kube-proxy. This removes kube-proxy (cilium contains a replacement).

3. You must terminate every node in the cluster to ensure each node's networking configuration can be rebuilt using only cilium. You can manually terminate all nodes simultaneously in the AWS web console. 1

Run Network Tests

Cilium comes with a companion CLI tool that is bundled with the Panfactum devenv. We will use that to test that cilium is working as intended:

1. Run cilium connectivity test.

2. Wait about 20-30 minutes for the test to complete.

3. If everything completes successfully, you should receive a message like this:

   ✅ All 46 tests (472 actions) successful, 18 tests skipped, 0 scenarios skipped.
   

4. Unfortunately, the test does not clean up after itself. You should run kubectl delete ns cilium-test to remove the test resources.

Next Steps

Now that basic networking is working within the cluster, we will configure your storage drivers.