Redis with Sentinel

This module deploys a highly-available set of Redis nodes.

This is deployed in a single master, many replica configuration. Failover is handled by Redis Sentinel which is also deployed by this module.

Usage

Credentials

For in-cluster applications, credentials can be sourced from the following Kubernetes Secrets named in the module’s outputs:

• superuser_creds_secret: Complete access to the database
• admin_creds_secret: Read and write access to the database (does not include the ability to preform sensitive operations like schema or permission manipulation)
• reader_creds_secret: Read-only access to the database

Each of the above named Secrets contains the following values:

• username: The username to use for authentication
• password: The password to use for authentication

The credentials in each Secret are managed by Vault and rotated automatically before they expire. In the Panfactum Stack, credential rotation will automatically trigger a pod restart for pods that reference the credentials.

The credential lifetime is configured by the vault_credential_lifetime_hours input (defaults to 16 hours). Credentials are rotated 50% of the way through their lifetime. Thus, in the worst-case, credentials that a pod receives are valid for vault_credential_lifetime_hours / 2.

Connecting

The below example show how to connect to the Redis master using dynamically rotated admin credentials by setting various environment variables in our kube_deployment module.

module "redis" {
  source = "${var.pf_module_source}kube_redis_sentinel${var.pf_module_ref}"
  ...
}

module "deployment" {
  source = "${var.pf_module_source}kube_deployment${var.pf_module_ref}"
  ...

  common_env_from_secrets = {
    REDIS_USERNAME = {
      secret_name = module.redis.admin_creds_secret
      key = "username"
    }
    REDIS_PASSWORD = {
      secret_name = module.redis.admin_creds_secret
      key = "password"
    }
  }
  common_env = {
    REDIS_HOST = module.redis.redis_master_host
    REDIS_PORT = module.redis.redis_port
  }
}

Persistence

Redis provides two mechanisms for persistence: AOF and RDB. This module uses RDB by default (tuned via redis_save).

Using AOF (whether independently or concurrently with RDB) negates the ability to do partial resynchronizations after restarts and failovers. Instead, a copy of the database must be transferred from the current master to restarted or new replicas. This greatly increases the time-to-recover as well as incurs a high network cost. In fact, there is arguably no benefit to AOF-based persistence at all with our replicated architecture as new Redis nodes will always pull their data from the running master, not from their local AOF. The only benefit would be if all Redis nodes simultaneously failed with a non-graceful shutdown (an incredibly unlikely scenario).

Persistence is always enabled in this module for similar reasons. Without persistence, an entire copy of the database would have to be transferred from the master to each replica on every Redis node restart. The cost of storing data on disk is far less than the network costs associated with this transfer. Moreover, persistence should never impact performance as writes are completed asynchronously unless configured otherwise.

Once the Redis cluster is running, the PVC autoresizer (provided by kube_pvc_autoresizer) will automatically expand the EBS volumes once the free space drops below persistence_storage_increase_threshold_percent of the current EBS volume size. The size of the EBS volume will grow by persistence_storage_increase_gb on every scaling event until a maximum of persistence_storage_limit_gb.

Disruptions

By default, failovers of Redis pods in this module can be initiated at any time. This enables the cluster to automatically perform maintenance operations such as instance resizing, AZ re-balancing, version upgrades, etc. However, every time a Redis pod is disrupted, a short period of downtime might occur if the disrupted pod is the master instance.

While this can generally be mitigated when using a Sentinel-aware client, you may want to provide more control over when these failovers can occur, so we provide the following options:

Disruption Windows

Disruption windows provide the ability to confine disruptions to specific time intervals (e.g., periods of low load) if this is needed to meet your stability goals. You can enable this feature by setting voluntary_disruption_window_enabled to true.

The disruption windows are scheduled via voluntary_disruption_window_cron_schedule and the length of time of each window via voluntary_disruption_window_seconds.

If you use this feature, we strongly recommend that you allow disruptions at least once per day, and ideally more frequently.

For more information on how this works, see the kube_disruption_window_controller submodule.

Custom PDBs

Rather than time-based disruption windows, you may want more granular control of when disruptions are allowed and disallowed.

You can do this by managing your own PodDisruptionBudgets. This module provides outputs that will allow you to match certain subsets of Redis pods.

For example:

module "redis" {
  source = "${var.pf_module_source}kube_redis_sentinel${var.pf_module_ref}"
  ...
}

resource "kubectl_manifest" "pdb" {
  yaml_body = yamlencode({
    apiVersion = "policy/v1"
    kind       = "PodDisruptionBudget"
    metadata = {
      name      = "custom-pdb"
      namespace = module.redis.namespace
    }
    spec = {
      unhealthyPodEvictionPolicy = "AlwaysAllow"
      selector = {
        matchLabels = module.redis.match_labels_master # Selects only the Redis master (writable) pod
      }
      maxUnavailable = 0 # Prevents any disruptions
    }
  })
  force_conflicts   = true
  server_side_apply = true
}

While this example is constructed via IaC, you can also create / destroy these PDBs directly in your application logic via YAML manifests and the Kubernetes API. This would allow you to create a PDB prior to initiating a long-running operation that you do not want disrupted and then delete it upon completion.

Completely Disabling Voluntary Disruptions

Allowing the cluster to periodically initiate failovers of Redis is critical to maintaining system health. However, there are rare cases where you want to override the safe behavior and disable voluntary disruptions altogether. Setting the voluntary_disruptions_enabled to false will set up PDBs that disallow any voluntary disruption of any Redis pod in this module.

This is strongly discouraged. If limiting any and all potential disruptions is of primary importance you should instead:

• Create a one-hour weekly disruption window to allow some opportunity for automatic maintenance operations
• Ensure that spot_instances_enabled and burstable_instances_enabled are both set to false

Note that the above configuration will significantly increase the costs of running the Redis cluster (2.5-5x) versus more flexible settings. In the vast majority of cases, this is entirely unnecessary, so this should only be used as a last resort.

Extra Redis Configuration

You can add extra Redis configuration flags via the redis_flags module variable.

These flags are passed as commandline arguments to the redis servers. This ensures they will be of the highest precedence.

For more information about passing flags through the commandline and available options, see this documentation.