Getting Started with CI / CD

Objective

Set up a CI / CD system that deploys IaC when committing changes to your stack repository.

Prerequisites

This guide requires that you have installed the Workflow Engine Stack Addon.
Setting up your CI / CD integrations will require writing new IaC modules. Ensure you understand the basic concepts for this by reviewing our first-party IaC guide.
You will need admin access to your organization's git repositories as you will need to adjust sensitive settings (e.g., webhooks).

Core Concepts

Our recommended architecture uses Argo Workflows and Events as the basis for the CI / CD system. That said, we do offer alternative modules for self-hosted GitHub Actions.

We will be deploying our recommended CI / CD architecture. Please ensure you have reviewed the architectural diagram in the linked concept documentation so that you have a concrete understanding of the end state for this guide.

This guide will focus on setting up CI / CD in a single cluster; however, you will likely want to repeat these steps for every region / environment.

We provide a complete example the IaC of a fully functional CI / CD system here.

Deploy an Event Bus

You trigger workflows from an external system, you will first need to create an Event Bus. Let's do this now:

Create a new IaC module for your CI/CD configuration in your infrastructure repository.
Add a new Kubernetes namespace using our kube_namespace submodule.
Follow our guide for creating a new Event Bus.
1. Set up the relevant source for your specific git hosting provider:
  - GitHub
  - GitLab
  - BitBucket
  - Webhook (for generic git hosting provider)
  When providing an API token for your hosting provider, ensure you limit its scope to only the necessary permissions for the webhook configuration.
  
  Additionally, ensure that you set a secret to protect the generated webhook endpoint. Use the random_password resource to generate the secret, and expose it as a sensitive module output.
  
  If you do not set deleteHookOnFinish on the EventSource, your webhook will not be deleted from your git hosting provider when the EventSource is updated / deleted. This will a conflict issues if you ever recreate the webhook in the future. As a result, we recommend setting this to true.
2. In the Sensor, create a dependency with no filters (yet). Additionally, create a Log trigger so that you can examine the webhook payload.
Create a public Ingress for your webhook using our kube_ingress submodule.
Deploy your new IaC module to your desired Kubernetes cluster.

To verify that everything was set up correctly:

Check that the EventBus, EventSource, and Sensor pods are all running in a healthy state.
Verify that the Webhooks were successfully installed in the repositories in your git hosting provider. Below is an example of success from GitHub.
Most git hosting providers will send test / ping webhook events on first install.

If your provider does send these checks, verify that your system received the event by examining the pod logs. You should either see printouts of the webhook payload (since you installed the Log trigger) and/or messages such as not interested in dependency xxxxxxxxxxxx (indicating that your Sensor dependency filters are working correctly).

Create IaC Deployment WorkflowTemplate

While the Event Bus will handle receiving and filtering events from your git repository, we still need to define what actions to take when events are received. This is done via Argo Workflows.

We provide several prebuilt Workflows that you can use for common CI / CD tasks.

Right now, let's set up the IaC deployment workflow, wf_tf_deploy, so that we can automatically deploy your infrastructure changes.

Add an instance of the wf_tf_deploy submodule to the CI / CD module you created above and re-apply it:

module "tf_deploy" {
  source = "github.com/Panfactum/stack.git//packages/infrastructure/wf_tf_deploy?ref=edge.24-09-12" #pf-update

  name = "tf-deploy"
  namespace = local.namespace
  eks_cluster_name          = var.eks_cluster_name

  # This is the repository url for that contains your terragrunt configuration files
  repo = "github.com/panfactum/stack"

  # All modules in this directory of the repository will be deployed
  tf_apply_dir = "packages/reference/environments/production/us-east-2"

  # These secrets will be exposed as environment variables during the
  # `terragrunt apply`. This can be used to supply credentials to
  # the OpenTofu providers.
  secrets = {
    AUTHENTIK_TOKEN = var.authentik_token
  }

  # pf-generate: pass_vars
  pf_stack_version = var.pf_stack_version
  pf_stack_commit  = var.pf_stack_commit
  environment      = var.environment
  region           = var.region
  pf_root_module   = var.pf_root_module
  is_local         = var.is_local
  extra_tags       = var.extra_tags
  # end-generate
}

A couple important notes:

You can override tf_apply_dir when you create new Workflows from this WorkflowTemplate. As a result, you can re-use this one WorkflowTemplate to deploy any particular subset of your IaC.
All the credentialing for AWS, Kubernetes, and Vault are done for you. However, each Workflow should deploy at most a single environment as credentials are environment-scoped.

After deploying this update, you should now be able to see the WorkflowTemplate in the Argo UI:

WorkflowTemplate created and registered with Argo

If all is working, you can now manually submit a new Workflow from the WorkflowTemplate and deploy your IaC. Test this now, but be aware that this will deploy the committed configuration from your remote repository and not from your local machine.

Triggering Workflows from Events

The final step is to automate the Workflow creation by connecting the IaC deployment WorkflowTemplate to the EventBus via an Argo Workflow Trigger.

Let's do this now:

Adjust the dependencies array of your Event Bus Sensor to match the specific webhook events that you are interested in. In the below example, we match GitHub webhooks for push events to the main branch. Adjust this as necessary for your desired configuration.

module "sensor" {
  source = "github.com/Panfactum/stack.git//packages/infrastructure/kube_argo_sensor?ref=edge.24-09-12" #pf-update  ...
  ...
  dependencies = [
    {
      name = "push-to-main"

      # These depend on how you configured your EventSource
      eventSourceName = "cicd"
      eventName = "github"

      # These filter the webhook payloads for only matching events
      filters = {
        data = [
          {
            path = "body.X-GitHub-Event"
            type = "string"
            value = ["push"]
          },
          {
            path = "body.ref"
            type = "string"
            value = ["refs/heads/main"]
          }
        ]
      }
    }
  ]
  ...
}

Add a new trigger to your Event Bus Sensor that will create a workflow based on your generated dependencies. The below example, adds a trigger that runs the IaC deployment on each new push to the main branch of any connected repository.

module "sensor" {
  source = "github.com/Panfactum/stack.git//packages/infrastructure/kube_argo_sensor?ref=edge.24-09-12" #pf-update  ...
  ...
  dependencies = [
    ...
  ]
  ...
  triggers = [
    {
      template = {
        name = module.tf_deploy_prod_us_east_2.name # Arbitrary string, but we generally like to name it the same as the WorkflowTemplate

        conditions = "push-to-main" # Which dependencies from the dependencies array will activate this trigger

        # This exemplifies how to create a Workflow from a WorkflowTemplate
        argoWorkflow = {

          # "submit" is the operating for creating a new Workflow
          operation = "submit"

          # This defines the workflow spec. Most of the time the Workflow spec should
          # be sourced from a WorkflowTemplate via `workflowTemplateRef`
          source = {
            resource = {
              apiVersion = "argoproj.io/v1alpha1"
              kind = "Workflow"
              metadata = {
                generateName = "${module.tf_deploy_prod_us_east_2.name}-"
                namespace = local.namespace
              }
              spec = {
                workflowTemplateRef = {
                  name = module.tf_deploy_prod_us_east_2.name
                }

               # Even though arguments is set in the WorkflowTemplate,
               # arguments must be set here as well in order to pass in values from the event
               # payload using parameters
               arguments = module.tf_deploy_prod_us_east_2.arguments
              }
            }
          }

          # This enables you to pass values from the event into the Workflows spec
          parameters = [
            {
              # `src` defines how to extract values from the event
              src = {
                dependencyName = "push-to-main"
                dataKey = "body.after" # The git commit hash after the push
              }

              # `dest` defines where in the Workflow spec to set the value (via JSONPath)
              dest = "spec.arguments.parameters.0.value"
            }
          ]
        }
      }
    }
  ]
  ...
}

Re-apply your CI / CD module with the new updates. To test that it is working correctly, execute the action that would generate the required webhook event to activate the trigger. In the above examples, this would be creating and pushing a new commit to the main branch of a repository in your organization.

Next Steps

Congratulations! You have the foundations of CI / CD up and running.

You can begin adding additional GitOps automation by creating and connecting your own Workflows.

You can examine some of the additional guides in this section for some common patterns you might want to add to your system. Additionally, don't forget to take advantage of our prebuilt workflows.