AWS Networking

Objective

Deploy the core AWS networking primitives including your Virtual Private Cloud (VPC) and Route 53 Hosted Zone.

A Quick Note

Up to now, we have focused on setting up all environments in each section. Moving forward, this guide will focus on setting up infrastructure for a single environment at a time. We recommend starting with your production environment and then returning here for additional environments as needed.

Additionally, unless otherwise specified, the following sections of this guide only apply to environments that will be hosting Kubernetes clusters. Those that do not host live infrastructure (e.g., management) do not need much further setup.

Background

If you are new to cloud networking, we recommend that your review the concept documentation before proceeding.

Create your VPC

We will use the aws_vpc module to deploy the VPC. This includes not only the VPC but also configuration of subnets, routing tables, flow logs, etc. See the module reference docs for a more comprehensive overview.

Choose your CIDR Blocks and Subnets

One of the most important decision that you make throughout this entire guide is deciding your CIDR blocks for subnet setup. This is very difficult to change later and will usually require redeploying your entire VPC and all the resources it contains.

We strongly recommend choosing the largest possible CIDR block for your VPC: 10.0.0/16. You want to ensure that you have at least 100 IPs available in each public subnet and 1,000 IPs available in each private / isolated subnet. Choosing a large VPC CIDR gives you the most flexibility. 1

We also strongly recommend that you set up at least three of each subnet type (nine total). This allows you to have one subnet type in at least three availability zones which is required to achieve a highly available setup.

More specifically, we recommend you use the following subnet configuration): 2 3

If you are choosing a different network layout, we recommend this site for helping to divide your network.

Understanding NAT

If you are unfamiliar with NAT, you should stop now to review the NAT concept documentation.

NAT is the one component of the VPC configuration that we have enhanced beyond the typical AWS-recommended setup.

Specifically, we do NOT use AWS NAT Gateways by default. They are far too expensive for behavior that should ultimately be available for free (and is in other cloud providers). For many organizations, NAT Gateway costs alone can produce 10-50% of total AWS spend.

Instead, we deploy a Panfactum-enhanced version of the fck-nat project. Using this pattern, our module launches self-hosted NAT nodes in EC2 autoscaling groups and reduces the costs of NAT by over 90%.

See the aws_vpc documentation for more details and to decide if it is right for you (it probably is).

Deploy the AWS VPC Module

We will now deploy the aws_vpc module via terragrunt:

1. Set up a new aws_vpc directory in the primary region for your environment (not global).

2. Add a terragrunt.hcl that looks like this. Replace the subnet configuration with your chosen settings.

3. Enable the aws provider in the module.yaml.

4. Run terragrunt apply.

5. Ensure that your VPC looks similar to the following:

   

6. Ensure that your NAT nodes are running and healthy:

   

   Note that each node should have a Public IPv4 address which should match its Elastic IP. All traffic from your cluster will appear to originate from one of these IP addresses, and they will remain the same for the lifetime of your VPC.

Test Connectivity

We will now verify a few features of the network:

1. Attempt to ping one of the public IP addresses of your NAT nodes. The ping should NOT go through as our NAT nodes only allow outbound connections.

2. We will launch a test EC2 node in a private subnet to verify that it has outbound network connectivity.

   1. Navigate to the AWS console.

   2. Launch a new EC2 node under EC2 > Instances > Launch Instances (orange button, top-right).

   3. Configure the new instance.

      1. Use Amazon Linux.

      2. Select the smallest instance size.

      3. Proceed without a keypair.

      4. Select the VPC we just deployed (not the default).

      5. Select a private subnet.

      6. Disable public IP assignment.

      7. Select "Create a new security group."

      8. Leave the rest as-is.

      9. Launch the instance.

   4. Select the instance.

   5. Notice that there is no public IP address and it received a private IP address in the subnet CIDR block.

   6. Connect to the instance.

      1. Select "Connect" (top right).

      2. Select "Session Manager."

      3. Select "Open Systems Manager Quick Setup."

      4. Leave everything as the default except manually target only the new test instance you just launched.

      5. Wait for this to successfully deploy. It may take a few minutes.

      6. Reboot the instance to register the node with AWS Session Manager.

      7. Return to Connect > Session Manager. Click "Connect." This will open a terminal in your browser.

   7. Run curl ifconfig.me.

      1. Notice that this works, verifying your private instance is able to connect to the public internet!

      2. Notice that the returned IP address is that of your NAT node in the same AZ as this instance. This verifies that NAT is working correctly.

   8. Close the connection, and terminate the instance.

Next Steps

Now that networking is set up in AWS, we can deploy your Kubernetes cluster.