Installing the Workflow Engine

Objective

Install the management controllers and Web dashboard for Argo Workflows and Argo Events.

Background

The Argo Project is a popular collection of Kubernetes controllers. Each controller operates independently, but they do have integrations with one another.

Two of these controllers work together to enable sophisticated workflow logic: Argo Workflows and Argo Events.

Argo Workflows

Argo Workflows provide a mechanism for defining DAGs that orchestrate the execution of pods and containers inside your Kubernetes clusters.

Workflows have many use cases:

• a part of your core business logic (e.g., kicking off an async task when receiving a user request)
• the driver of your CI / CD platform (e.g., orchestrating testing and deployments of new code)
• an ETL tool (e.g., periodically moving data from one system to another)
• and much, much more.

We strongly prefer Argo Workflows over other workflow alternatives for the following reasons:

• Language-agnostic; as long as your code can be containerized, it can be run in a workflow.
• Manged via declarative IaC (through Kubernetes custom resources for Argo).
• Provides an integrated web dashboard that (a) provides granular execution and debugging information, (b) the ability for non-technical users to create and manage workflows, and (c) integrations into our SSO framework.
• Includes a companion CLI that can be used to easily script workflow execution.
• Comprehensive options for managing workflow concurrency, step dependencies, retries, rate limits, artifacts, etc. (much more flexibility and control than we have found in any other workflow engine).
• Straightforward mechanisms for templating and parameterization.
• Deep integrations with Kubernetes, allowing it to benefit from all the other utilities installed in the Panfactum Stack.
• A long history and stable future as a CNCF-maintained project.

Argo Events

While workflows can be triggered manually or via a cron schedule, Argo Events allows them to be triggered via events from external systems. 1

Events can be inbound webhooks, messages in a message queue, or any other EventSource that Argo Events supports.

Because you will often want these types of external triggers, we install both Argo Workflows and Argo Events in this guide even though they are separate projects.

Installing

We provide a module, kube_argo, that deploys the controllers for both Argo Workflows and Argo Events.

Let's deploy it now:

1. In the region with the cluster that you wish to deploy the workflow engine to, add a new directory called kube_argo.

2. Add a terragrunt.hcl that looks like this.

   1. The argo_domain will be the domain where the Argo web UI will be exposed.

   2. Set test_workflow_enabled to false as it cannot be deployed until the controllers are running.

3. Run pf-tf-init to enable the required providers.

4. Run terragrunt apply.

5. Change test_workflow_enabled to true and re-run terragrunt apply.

Testing the Workflow Engine

In the kube_argo module, we create a WorkflowTemplate called test in the argo namespace. 2 We will use this to create a Workflow and test if Argo Workflows is functioning correctly.

Web UI

Let's first approach this task from the integrated web UI:

1. In a web browser, navigate to the domain you supplied for argo_domain. You should be met with a screen that looks like this:

   

2. Click "Login" and you will be asked to authenticate with the Vault instance running the cluster. After supplying your credentials, you will be redirected to a landing page showing recently run Workflows (yours will likely be empty):

   

3. Select "Workflow Templates", the second menu item on the left-hand navigation. On this page, you will see all the WorkflowTemplates that can be used to create new Workflows:

   

   You should see the test template. If you do not see any templates, ensure you have selected argo in the namespace filter.

4. Select the test template and click "+ Submit" to open the submission drawer:

   

5. The Workflow that gets generated from the test template will simply run a container that prints whatever string you supply as the message parameter. Adjust the message parameter as desired and click "+ Submit".

6. You will be redirected to a live view of the Workflow execution graph:

   

7. Click "Logs" to verify the generated pods printed your message:

   

CLI

Let's now attempt to create a Workflow from the test template using the argo CLI that comes bundled in the Panfactum devShell:

1. Use kubectx to switch your local Kubernetes context to the cluster where you just deployed Argo Workflows.

2. Run argo submit -n argo --from=WorkflowTemplate/test --parameter message='Hi!' --wait. You will receive a message

   like this:

   Name:                test-w9vtr
   Namespace:           argo
   ServiceAccount:      unset
   Status:              Pending
   Created:             Thu Jul 25 16:09:30 -0400 (now)
   Progress:
   Parameters:
       message:           Hi!
   

3. Wait 10-30 seconds for the workflow to complete.

4. Run argo logs -n argo <workflow-name> where <workflow-name> is the generated name from step 2. Verify that your message was successfully printed:

   test-w9vtr-test-1726256991: {"argo":true,"level":"info","msg":"capturing logs","time":"2024-07-25T20:09:37.490Z"}
   test-w9vtr-test-1726256991: You entered: Hi!
   test-w9vtr-test-1726256991: Workflow completed successfully.
   test-w9vtr-test-1726256991: {"argo":true,"error":null,"level":"info","msg":"sub-process exited","time":"2024-07-25T20:09:38.496Z"}
   

Next Steps

Congratulations! You have successfully deployed and tested the Workflow Engine addon.

However, we have just barely scratched the surface of what is possible with Argo Workflows and Events. To dive deeper, check out the following resources:

• For a brief overview of all the capabilities of Argo Workflows, we strongly recommend working through these Argo Workflow tutorials.

• To start creating your own custom Workflows, take a look at our documentation.

• Deploy some of our pre-built Workflows for common tasks in the Panfactum Stack

• Learn how to use Workflows for your CI / CD system here.

• Learn more about how to leverage Argo Events here.