Installing an Event Bus

Objective

Deploy an instance of the Argo Events architecture.

Prerequisites

Deploying the components of the Event Bus requires that the Argo Events controller is running. This is done via deploying the Workflow Engine Stack Addon.
Deploying an Event Bus requires writing your own IaC. Be sure you have reviewed our guide on how to do this in the Panfactum stack.

Background

When we refer to "Event Bus" in the Panfactum Stack, we are actually referring to an entire set of Argo Events components, of which the EventBus is just one. However, none of them are useful in isolation, so we refer to the collection as a unit: the Event Bus.

More specifically, Argo Events has three independent components in its architecture:

One or more instances of an EventSource: Every EventSource creates a Kubernetes Deployment that either receives or polls for events from external systems and deposits them on the EventBus.
One or more instances of a Sensor: Every Sensor creates a Kubernetes Deployment that consumes events from the EventBus and triggers some predefined action, called a Trigger.
An EventBus: The EventBus stores events that have been created by EventSources until they are consumed by Sensors. Under the hood, it is a NATS JetStream cluster.

We offer three submodules for deploying these components:

This particular addon has a few peculiarities:

You can have many instances of this architecture running independently in your cluster at once.
Due to its flexible nature, we provide submodules for the components rather than modules to deploy directly from terragrunt.
All the above resources are namespace-scoped. In other words, an EventSource can only publish events to an EventBus running in the same namespace, and a Sensor can only read events from an EventBus in the same namespace.

Installing

To install the Event Bus, you must define each of the event bus submodules in your own IaC module and then deploy it. Below is a code snippet that exemplifies the deployment pattern:

module "event_bus" {
  source =   "${var.pf_module_source}kube_argo_event_bus${var.pf_module_ref}"

  namespace = local.namespace
}

module "event_source" {
  source = "${var.pf_module_source}kube_argo_event_source${var.pf_module_ref}"

  name        = "example"
  namespace   = local.namespace
  vpa_enabled = var.vpa_enabled

  # For Active-Active EventSources, we recommend setting the replica count to 2.
  # For Active-Passive EventSources, we recommend setting the replica count to 1.
  # See https://argoproj.github.io/argo-events/eventsources/ha/
  replicas = 2

  event_source_spec = {

    # Note that this is just an example of one of the many types of EventSource specifications.
    # In this case, we are creating a webhook.
    # For the full list, see your these docs: https://argoproj.github.io/argo-events/eventsources/setup/amqp/
    #
    # Note that you can add multiple sources to a single EventSource object which
    # we recommend over creating multiple discrete EventSource objects as it reduces overall resource utilization.
    # See https://argoproj.github.io/argo-events/eventsources/multiple-events/
    webhook = {
      example = {
        active = true
        port = "12000"
        endpoint = "/example"
        method = "POST"
      }
      example2 = {
        active = true
        port = "12000"
        endpoint = "/example2"
        method = "POST"
      }
    }

    # For sources that receive messages over the network (e.g., webhooks), the EventSource will create
    # the necessary Kubernetes Service for you. You can re-use this single Service across multiple sources
    # (just ensure that you set different endpoints).
    #
    # That said, you will need to create your own Ingress. You can use our kube_ingress submodule for this.
    service = {
      ports = [
        {
          name = "default"
          port = 12000
          targetPort = 12000
        }
      ]
    }
  }
}

module "sensor" {
  source = "${var.pf_module_source}kube_argo_sensor${var.pf_module_ref}"

  name = "example"
  namespace = local.namespace
  vpa_enabled = var.vpa_enabled

  # The dependencies array lists all the events that the sensor will subscribe to.
  # It can contain multiple different filters for selecting only certain events:
  # https://argoproj.github.io/argo-events/sensors/filters/intro/
  #
  # This particular example assumes the events are GitHub webhook payloads and filters
  # for events generated when someone pushes to the main branch of a repository.
  dependencies = [
    {
      name = "push-to-main"
      eventSourceName = "example"
      eventName = "example" # This is the name of the source in the EventSource spec (i.e., the name of the webhook in this particular example)
      filters = {
        data = [
          {
            path = "body.X-GitHub-Event"
            type = "string"
            value = ["push"]
          },
          {
            path = "body.ref"
            type = "string"
            value = ["refs/heads/main"]
          }
        ]
      }
    }
  ]

  # Triggers are the actual actions that will get fired when an event is matched.
  # You can have many triggers on the same sensor, but this sensor just has one, a Log trigger (prints the event to stdout)
  # There are many different types of triggers available:
  # https://argoproj.github.io/argo-events/sensors/triggers/log/
  triggers = [
    {
      template = {

        # The conditions field controls which events will activate the trigger based on the filters provided
        # in the indicated dependencies. This supports boolean logic. See more here:
        # https://argoproj.github.io/argo-events/sensors/trigger-conditions/
        conditions = "push-to-main"

        name = "log"
        log = {
          intervalSeconds = 1
        }
      }
    }
  ]
}

For more advanced patterns, see the documentation for each of the modules:

Sensor Utilities

When setting up the Event Bus, the bulk of the logic tends to fall in defining which events should trigger which actions (i.e., the Sensor spec).

We provide a submodule that contains a collection of utilities that make writing this logic easier: TODO.

Best Practices

Limit the number of Event Buses in your cluster

Every deployment of the Event Bus addon has a non-trivial amount of fixed overhead: at 6 pods running across 2 Deployments and 1 StatefulSet.

Instead of creating an Event Bus for every namespace, try to consolidate them whenever possible. For example, you might choose to have an Event Bus that handles inbound webhooks for all services across your entire organization.

Additionally, you should never need more than one Event Bus deployment per namespace. Deploying more than one not only wastes resource but also will cause conflicts in event delivery with the default module settings.

DLQ Trigger

If a Sensor experiences an error or does not fire its triggers within the appropriate time (a couple of days by default) for any given event, that event will be lost as the EventBus is not a long-term storage system.

If preventing data loss is important to your event pipeline, you should set up a dead letter queue (DLQ) trigger.

Performance Management

We have not yet tested the Event Bus architecture for a high volume of events (> 10 / second). While we don't expect any issues, you will want to do adequate performance testing before deploying the Event Bus in high-load scenarios.