Performance

Overview

The performance pillar focuses on the ability of your systems to deliver a responsive experience to end-users.

Optimizing performance in modern platform engineering requires addressing two distinct problems:

• Autoscaling: Autoscaling (automatic scaling) ensures that workloads receive the compute resources necessary to execute successfully. Compute resources can include CPU shares, memory allocations, storage drives, and peripheral devices such as GPUs.

  Assuming that workloads make valid resource requests, 1 there are three types of autoscaling:

  • Vertical Autoscaling: Adjusting the resource requests of individual workloads based on historical resource usage. For example, increasing the amount of memory allocated to an individual server.
  • Horizontal Autoscaling: Adjusting the number of workloads running based on overall system load. For example, provisioning additional webservers to handle load-balanced HTTP traffic.
  • Infrastructure Autoscaling: When workloads are provisioned on discrete hosts, infrastructure autoscaling adjusts the number and type of hosts to ensure that all workload resource requests can be met. 2

  Regardless of your system design, effective autoscaling depends on a few key assumptions:

  • Workloads define the resources they need before they begin executing.
  • Scaled workloads are homogenous in nature. 3
  • Workload scaling (both up and down) does not cause service disruptions. 4
  • Adjustments to workload scale can occur before workloads become resource constrained.

• Unconstrained Latency: Latency measures time a single action takes to execute. Depending on the action, latency might include compute time, network transit time, external system response times, workload startup time, etc.

  Unconstrained latency measures latency when a workload has received all its requested compute resources. 5

  We care specifically about unconstrained latency because autoscaling will generally ensure workloads receive their requested resources. However, autoscaling can only take you so far. Even when a workload receives its requested resources, there are many additional factors to consider:

  • Data can only travel so quickly between different systems.
  • Storage drives only read and write data at certain speeds.
  • No cloud provider can provision infinitely large instances.
  • Some systems may not be able to be scaled, such as client's web browser or external vendor APIs.
  • Autoscaling itself can only occur so rapidly.
  • etc.

  While less latency is always preferred, you must also be aware of hard limits on the maximum latency for certain actions. For example, your underlying infrastructure system may require you to be able to process inbound HTTP requests within a certain time window in order to prevent requests from being dropped. 6

Motivation

This pillar provides organization value for the following reasons:

• Small differences in performance (< 100ms) can dramatically conversion rates (ref).

• Perceived performance can dramatically impact user satisfaction (ref).

• Performance impacts SEO and discoverability (ref).

• Poor performance can lead to serious system issues:

  • Actions exceeding certain latency thresholds might be terminated prior to completion (e.g., dropped HTTP requests).
  • Improperly configured autoscaling can lead to resource contention that causes component crashes (e.g., process OOMs).
  • Improperly configured autoscaling can lead to load-shedding (e.g., incoming actions such as HTTP requests cannot be processed at all).

• Performant design decisions reduces the amount of future rework necessary as usage increases over time. 7

Benchmark

Metrics

These are common measurements that can help an organization assess their performance in this pillar. These are intended to be assessed within the context of performance on the key platform metrics, not used in a standalone manner.

Organization Goals

These are common goals to help organizations improve their performance on the key platform metrics. While each goal represents a best practice, the level of impact and optimal approach will depend on your organization's unique context.