Resiliency

Overview

Resiliency focuses on preventing unintended system problems and reducing the impact when system problems do occur. A system problem might be as small as a UI bug or as large as a complete system outage.

Resilient platforms excel in four distinct areas:

• High Availability (HA): The ability of a system to remain functional despite arbitrary problems in any failure domain. Some examples of failure domains include a single replica of a multi-replica service (smallest), a single VM in a cluster (small), a single datacenter in a region (medium), or even an entire cloud provider such as AWS (large).

  Every organization will have to choose what level of HA it needs. Resiliency against outages in larger failure domains results in higher costs, not just in infrastructure resources but also engineering complexity. For attaining 99.9% uptime SLAs in cloud computing environments, organizations will typically design for single datacenter / availability zone failures. Beyond this, organizations must carefully weigh the costs and benefits as costs increase dramatically.

• Automated Testing: The ability of an organization to tests its systems with the goal of uncovering problems as early in the software development lifecycle as possible. There are generally five categories of automated testing that an organization can develop:

  • Unit Tests: Tests written against individual components of codebase in isolation. Typically, these target individual functions / methods. 1

  • Static Analysis: Tests written that analyze the source code directly without invoking it. Static analysis can be extremely effective as it requires no additional work other than producing the source code itself. This usually includes using a typed language and various linters. 2

  • Integration Tests: Tests written against the public interface of the codebase. For example, an HTTP service might have integration tests that submit HTTP requests and validate various system properties such as database records. As another example, a programming library might have tests written against its public API to validate the expected behavior.

  • End-to-End (E2E) Tests: Tests written against the overall system as a whole. Usually these are written with the goal of emulating the end user experience. For example, a SaaS company with a web UI might use a tool like Playwright to create simulated user actions directly in a web browser.

    E2E tests can be used to gate deployments to production, for example by running the test suite in a production-like staging environment. Additionally, the tests can be even be run against production itself in order to create better downtime visibility.

  • Simulated Failure Tests: Also known as chaos engineering, these test simulate real-world failure cases that are often not related to the specific business logic of the code itself. Simulated failures might include dropped network packets / requests, a sudden service shutdown caused by an OOM error, or even the unexpected loss of an entire datacenter.

• Automated Remediation: The ability of a system to automatically fix itself when issues are detected. Most commonly, this practices involves incorporating automatic rollbacks to an organization's deployment pipelines where new code and / or configuration changes are applied to an environment.

  While conceptually simple, this requires strong foundations in the observability pillar in order to quickly detect when issues occurs. Additionally, automatic rollbacks can introduce complexity in organizations using GitOps where it will cause differences between what is deployed and what is running in the system.

• Disaster Recovery: The ability to recover from system changes that cannot be easily rolled back. A catastrophic system change could come from both internal or external sources. An internal example would be an accidental database deletion, and an external example would be a long-term outage from a foundational service provider.

  We break this into two focus areas:

  • Data Durability: How your organization would recover its data in the event the primary storage location becomes unavailable or corrupted (e.g., service provider locks your accounts, invalid schema migration, ransomware attack, etc.).

  • System Restoration: How your organization would redeploy all of its systems in the event the primary system becomes unavailable or compromised (e.g., regional outage, lost system access, etc.). Highly related to the recovery time objective (RTO).

Motivation

This pillar provides organizational value for the following reasons:

• Prevents risks to business continuity:

  • irrecoverable data loss
  • irrecoverable or extended system outages

• Prevents damage to public reputation and customer trust caused by bugs and outages

• Improves development velocity by

  • reducing the amount of rework
  • reducing time spent debugging production issues
  • reducing the amount of manual scrutiny necessary during development by relying on resilient CI / CD pipelines

• Improves development predictability by minimizing unplanned work

Benchmark

Metrics

These are common measurements that can help an organization assess their performance in this pillar. These are intended to be assessed within the context of performance on the key platform metrics, not used in a standalone manner.

Organization Goals

These are common goals to help organizations improve their performance on the key platform metrics. While each goal represents a best practice, the level of impact and optimal approach will depend on your organization's unique context.