Internal Cluster Networking

Objective

Install the basic Kubernetes cluster networking primitives via the kube_core_dns and kube_cilium modules.

Background

In the Panfactum stack, we use CoreDNS to handle cluster DNS resolution and Cilium to handle all the L3/L4 networking in our Kubernetes cluster.

In this guide, we won't go into detail about the underlying design decisions and network concepts, so we recommend reviewing the concept documentation for more information.

Deploy Cilium

Cilium provides workloads in your clusters with network interfaces that allow them to connect with each other and the wider internet. Without this controller, your pods would not be able to communicate. We provide a module for deploying Cilium: kube_cilium.

Let's deploy it now.

Deploy the Cilium Module

1. Create a new directory adjacent to your aws_eks module called kube_cilium.

2. Add a terragrunt.hcl to that directory that looks like this.

3. Run pf-tf-init to enable the required providers.

4. Run terragrunt apply.

5. If the deployment succeeds, you should see the various cilium pods running:

   

   Additionally, all the nodes should now be in the Ready state:

   

Deploy CoreDNS

Kubernetes provides human-readable DNS names for pods and services running inside the cluster (e.g., my-service.namespace.svc.cluster.local); however, it does not come with its own DNS servers. The standard way to provide this functionality is via CoreDNS. We provide a module to deploy CoreDNS called kube_core_dns.

Let's deploy it now.

Deploy the CoreDNS Module

1. Create a new directory adjacent to your aws_eks module called kube_core_dns.

2. Add a terragrunt.hcl to that directory that looks like this.

3. If you used our recommendation of 172.20.0.0/16 for the service_cidr in the cluster setup docs, you should use a service_ip of 172.20.0.10 as this is the well-known DNS IP in Kubernetes.

4. Run pf-tf-init to enable the required providers.

5. Run terragrunt apply.

6. If the deployment succeeds, you should see a core-dns deployment with 2/2 pods running:

   

Run Network Tests

Cilium comes with a companion CLI tool that is bundled with the Panfactum devShell. We will use that to test that cilium is working as intended:

1. Run cilium connectivity test.

2. Wait about 20-30 minutes for the test to complete.

3. If everything completes successfully, you should receive a message like this:

   ✅ All 46 tests (472 actions) successful, 18 tests skipped, 0 scenarios skipped.
   

4. Unfortunately, the test does not clean up after itself. You should run kubectl delete ns cilium-test to remove the test resources.

Next Steps

Now that basic networking is working within the cluster, we will configure your storage drivers.