Bootstrapping Infrastructure-as-Code

Objective

Deploy your first modules using Terragrunt and OpenTofu (Terraform).

Set Up State Backends

To begin deploying infrastructure, you will first need a state backend... which is itself infrastructure. In fact, we provide an infrastructure module for setting up the backend: tf_bootstrap_resources.

This creates a circular dependency that we will resolve as follows:

1. Use terragrunt's built-in S3 backend generation to build the state backend.

2. Import those autogenerated resources into a deployment of tf_bootstrap_resources.

3. Apply the tf_bootstrap_resources module to deploy the Panfactum-provided enhancements.

Create the State Backend for Management Environment

Terragrunt will automatically generate the resources required to bootstrap an S3 backend if they do not already exist. This greatly simplifies the bootstrapping process.

To take advantage of this, we will create our first terragrunt deployment:

1. Generate a deployment directory for tf_bootstrap_resources: environments/management/global/tf_bootstrap_resources

2. Add a terragrunt.hcl to that directory that looks like this.

3. Add a module.yaml to that directory that enables the aws provider:

   providers:
     - aws
   

4. Open your terminal to that directory and run terragrunt init. If your system is set up correctly, you should see a prompt like the following:

   Remote state S3 bucket panfactum-tf-state-management does not exist or you don't have permissions to access it. Would you like Terragrunt to create it? (y/n)
   

5. Type y and press enter. If everything completes successfully you should see a message such as

   Initializing the backend...
   
   Successfully configured the backend "s3"! OpenTofu will automatically
   use this backend unless the backend configuration changes.
   
   Initializing provider plugins...
   - Finding hashicorp/aws versions matching "~> 4.13"...
   - Installing hashicorp/aws v4.67.0...
   - Installed hashicorp/aws v4.67.0 (signed by HashiCorp)
   
   OpenTofu has created a lock file .terraform.lock.hcl to record the provider
   selections it made above. Include this file in your version control repository
   so that OpenTofu can guarantee to make the same selections by default when
   you run "tofu init" in the future.
   
   OpenTofu has been successfully initialized!
   
   You may now begin working with OpenTofu. Try running "tofu plan" to see
   any changes that are required for your infrastructure. All OpenTofu commands
   should now work.
   
   If you ever set or change modules or backend configuration for OpenTofu,
   rerun this command to reinitialize your working directory. If you forget, other
   commands will detect it and remind you to do so if necessary.
   

6. Navigate to S3 in the AWS console. You should see an empty S3 bucket that terragrunt just instantiated.

   

Import the Autogenerated Resources

While terragrunt autogenerated the minimum necessary resources, they are not yet being managed by your tf_bootstrap_resources module. 1 We will need to import them into your state.

1. Run the following series of commands:

   terragrunt import aws_s3_bucket.state <tf_state_bucket>
   terragrunt import aws_dynamodb_table.lock <tf_state_lock_table>
   

   Replace <tf_state_bucket> and <tf_state_lock_table> with your state bucket name and lock table name respectively.

2. If your import is successful, you should have your first state file.

   

3. Run terragrunt apply to synchronize the resources with the tf_bootstrap_resources module. This module adds several enhancements above and beyond the basic autogenerated resources created by terragrunt.

   This may result in an error Error: Error releasing the state lock. This is because we update the DynamoDB lock table to enabled multi-region replication. This is a benign error, and no action is needed.

4. Run terragrunt plan. You should see the following message: No changes. Your infrastructure matches the configuration.. Congratulations! You have successfully deployed your first infrastructure module in the Panfactum stack. 🎉 🥳 🎉

Bootstrap other Environments

Now you will boostrap the state backends for your other environments by repeating the process above for every environment. You can simply copy the tf_boostrap_resources directory into the equivalent location in every other environment to get started.

Set Up sops

Terragrunt uses sops to store secret configuration values. This allows you to commit secrets in encrypted form directly to version control which comes with a host of benefits:

• All of your settings for all infrastructure can now be found in a single location.

• You do not need a separate change management or CI/CD process for secrets and non-secret values.

• You will implicitly have an audit log of all changes.

• You can utilize sops for other git ops activities in addition to deploying infrastructure modules (e.g., performing automatic rotations).

We will configure sops to use AWS KMS for the encryption keys. This provides several benefits:

• sops uses transit encryption. As a result, the encryption keys never leave the key store. By using KMS, noone in your organization will ever have access to the encryption keys which means you do not need to rotate these keys (and thus all secrets encrypted with them) every time you offboard a member of your organization.

• KMS allows you to replicate the encryption keys across multiple regions, ensuring you will never lose access.

• KMS will provide an audit log for every time a secret value is accessed which will augment the git commit history that records every time a secret is changed.

• Access to each KMS key will inherit our AWS role-based access control paradigm. As keys are scoped to each environment, access to secrets will automatically align with environment permissions. In other words, users with access to development will have access to development secrets but not necessary secrets in other environments.

Let's set this up.

1. In every environment, create a deployment of the aws_kms_encrypt_key module.

   1. Add a new directory called sops to the global region.

   2. Add a terragrunt.hcl that looks like this.

   3. Create a module.yaml that enables the aws provider and uses the aws_kms_encrypt_key module:

      providers:
        - aws
      module: "aws_kms_encrypt_key"
      

   4. Run terragrunt apply. Take note of both the arn and arn2 outputs as we will need in a following step.

2. In the root of your repo, we will create a .sops.yaml configuration file. This will enable you to easily assign KMS keys to the secrets you create based on the secret's file name.

   Under the creation_rules key, you will add an array element for every environment that is an object that has the form:

   path_regex: .*/<environment>/.*
   aws_profile: <environment>-superuser
   kms: '<arn>,<arn2>'
   

   Replace <environment> with the name of the environment. Replace <arn> and <arn2> with the ARN outputs from the deployment step.

   Ultimately, you should construct a file that looks like this.

3. Let's perform a quick test to ensure sops is working properly.

   1. Create a new file at environments/management/test.yaml:

      foo: bar
      

   2. Run sops -e -i test.yaml ((e)ncrypt (i)n-place). The file contents should be transformed into something like the following:

      foo: ENC[AES256_GCM,data:JJWK,iv:iF2zywZM3DWObiJCPsaPzETwnlQ1q2lh+zgHfCmk/PM=,tag:m2Ii1IlH05fT/TE4SStWIA==,type:str]
      sops:
          kms:
            - arn: arn:aws:kms:us-east-2:143003111016:key/mrk-955687aaf5124a07837ae4e2a442f8ec
              created_at: "2024-03-08T17:27:00Z"
              enc: AQICAHgMz35tnCYOcZgsSkZfKep5SPbKOCK5kzijAQLnZXO3TAHQ6ctemzPMzRenMG2LWQjAAAAAfjB8BgkqhkiG9w0BBwagbzBtAgEAMGgGCSqGSIb3DQEHATAeBglghkgBZQMEAS4wEQQM4JXy9Gd99htGiu7aAgEQgDuxh107pk18bFU5Q8vzeu1rI+u6+/7s7Xao5DUSE/86Uyvo7USMny58KqJnqUdIvGmj3xYqV5dVMGJxAQ==
              aws_profile: management-superuser
            - arn: arn:aws:kms:us-west-2:143003111016:key/mrk-955687aaf5124a07837ae4e2a442f8ec
              created_at: "2024-03-08T17:27:00Z"
              enc: AQICAHgMz35tnCYOcZgsSkZfKep5SPbKOCK5kzijAQLnZXO3TAFwqRvjtwaFuBNp9ppYc58OAAAAfjB8BgkqhkiG9w0BBwagbzBtAgEAMGgGCSqGSIb3DQEHATAeBglghkgBZQMEAS4wEQQMagf0c2ADsKLKXBFQAgEQgDsP5rqepGI3MvwhtRVxmS6/hyWWDyqEbOIxYS1WybNinBggkfKNRTw0Z8vob2tuSg5ZUeRpABry/C84PA==
              aws_profile: management-superuser
          gcp_kms: []
          azure_kv: []
          hc_vault: []
          age: []
          lastmodified: "2024-03-08T17:27:01Z"
          mac: ENC[AES256_GCM,data:JwzOMNKLn5ETSQY6QiGmxkxLqzX6buvs5OccPZ/BVOdfvgUW8vsgv34l6UiOidHq71GeGo5G9sPHeNiFGWPgqBMNU/qUugt9qldHs5k/oo6e4wFIzOxkIT2NZ1CmLw/9vxw8ZWskWX43XPC8tQlu9NmxE65G7NSDTaHBIFVVG44=,iv:L1pBiv+2y066RNr9qppBjuCjgLkaui6yOGJ6iPSCg8w=,tag:kmN5KBJ7gkhvcnkV67/xjw==,type:str]
          pgp: []
          unencrypted_suffix: _unencrypted
          version: 3.8.1
      

      This file is now safe to commit to version control.

   3. Run sops -d test.yaml ((d)ecrypt). Notice the original file contents should have been output to your terminal.

   4. Delete the file.

Deploy the AWS Account Module

Now that you have both the state backends and encryption keys established, we will put them both to use in deploying the aws_account module. This module adds some critical metadata to your account including setting up its alias and contact information.

Let's set it up.

1. In every environment's global region, create an aws_account directory.

2. Add a terragrunt.hcl to the directory that looks like this.

   Note the addition of this line secrets = yamldecode(sops_decrypt_file("secrets.yaml")) which demonstrates how terragrunt and sops integrate with one another.

   Replace the values as needed for your organization. Feel free to also change what values are secret or not.

3. Add a module.yaml with the aws provider enabled.

4. Add a secrets.yaml to the directory:

   address_line_1: 1234 Platform Engineering Way
   city: Developer Junction
   district_or_county: Marion
   state_or_region: CA
   postal_code: "12345"
   phone_number: "+15555555555"
   email_address: spam@panfactum.com
   

   Adjust the values as necessary.

5. Run sops -d -i secrets.yaml to encrypt the file.

6. Run terragrunt apply.

Next Steps

Now that you are set up to deploy infrastructure-as-code, we are ready to setup DNS.