Observability

Overview

Observability focuses on the active collection, parsing, indexing of live system data for the purposes of providing system insights, alerting, and auditability. Six types of data are typically collected:

• Metrics: Metrics provide point-in-time measures of various system properties that are aggregated into time-series data stores. The vast majority of services will expose metrics in the industry-standard Prometheus or OpenMetrics data format.

• Traces: Also known as "distributed traces," traces records the paths taken by data as it propagates through a system. Traces are composed of "spans" which track how long various system component spent processing the data. Spans record a location identifier, a start time, an end time, and other optional metadata. Tracing provides two key capabilities:

  • the ability quickly identify issues that occur at the granularity of specific actions such as individual HTTP requests

  • aggregated span metrics that can expose problems with specific system components such as performance issues or elevated error rates

• Logs: Logs are the human-readable information output from a system component. Logs can either be in plaintext or a structured format such as JSON for easier parsing. Most system components output logs to stdout / stderr or a specific log file. An observability platform will collect logs from each component, parse them (add additional structure and metadata), and store them in a data store optimized for text searching.

  Of the three types of data, logs allow for the most verbosity and flexibility; however, they are also the most expensive to manage and store. Often logs have an associated "log level" (e.g., info, warn, error, etc.) that can help automated systems identify which are the most important. In distributed systems, they can also contain trace and / or span IDs to correlate verbose logs with specific traces.

• Errors: Application code can often generate unhandled errors. 1 Error tracking involves deploying a global wrapper that captures all unhandled errors in order to send them to a central monitoring platform. 2

  While an "error" has language-specific meaning, the vast majority of languages have the concept of an Error / Exception object that contains a stack trace. This stack trace can be used to pinpoint the exact line of code that caused the issue, often with the help of uploaded source maps. 3 Often the error tracking can also capture other helpful data about the local state of the system when the error was generated.

• Application Profiles: Many languages execute on a runtime (e.g., JVM, V8, etc.) that can provide performance data from the perspective of individual functions or methods. This includes granular information on memory and CPU statistics as well as language-specific information such as runtime locks. This can be deployed when trying to optimize performance-critical code paths often for the purposes of cost-savings. 4

• Synthetic Tests: Synthetic tests are automated programs run against live systems for the purpose of collecting data in a controlled manner. These can be as simple as uptime tests such as repeatedly issuing an HTTP request against an endpoint to ensure it is online. These can also be complex end-to-end tests run against production a part of a continuous testing paradigm. This is explained further in our downtime visibility guide.

• Real User Monitoring (RUM): For systems that have a client-side UI, a common practice is to embed monitoring scripts into the distributed binaries / bundles in order to track various aspects of the user experience. These tools can capture not only basic data such as load times, errors, and logs, but also more advanced information such as the entire user session. This can be helpful not just for debugging but also to inform product development.

A good observability platform will provide a unified mechanism for collecting, searching, analyzing, and monitoring each of the above data types. Once the data is collected, the most critical capabilities include:

• Querying: All data should be easily accessible using a standard, well-documented query language. Retrieval of indexed data should happen nearly instantly.

• Correlations: Data of one type should automatically link to data of another type. For example, traces should link to logs generated during that trace.

• Alerting: The system should provide robust capabilities for defining automated monitors that trigger alerts when certain criteria are met (e.g., error rates spike). The system should be able to send alerts to arbitrary destinations as well as an integrated on-call platform.

• Dashboards: Most users will not be masters of navigating the observability data. The platform should provide the ability to generate predefined dashboards to aid in top-level status reporting and user-driven debugging.

• Archiving: Some types of observability data will need to be shipped to long-term storage for audit and compliance purposes; the system should easily support this archival functionality.

• Cost Efficiency: Observability platforms tend to be one of the largest organizational costs and pricing can differ dramatically between vendors. At the same time, poorly implemented platforms can cost even more in second-order costs such as lost engineering time or unidentified user experience issues. Ensure you choose a platform that minimizes the sum of both the platform and second-order costs.

• Access Controls: Often your observability platform will not only contain sensitive information about your users and your system at large, but it will also be used to provide an audit log and active alerting for security issues. The platform should contain a robust framework for assigning granular capabilities to its users.

Motivation

This pillar provides organization value for the following reasons:

• Improves the ability to address production issues by

  • reducing the latency to issue identification (MTTD)
  • reducing the latency to issue resolution (MTTR)
  • providing necessary context that would otherwise make remediation impossible

• Improves development velocity by

  • reducing the time spent on debugging
  • improving engineer confidence in system resiliency; coupled with automated rollbacks, observability tooling allows engineers to build and deploy features more rapidly

• Informs product development by

  • allowing for a data-driven approach to issue prioritization
  • providing critical insights into user behavior during application use

• Reduces operational costs by

  • highlighting inefficient code that bloats infrastructure costs
  • reducing the load placed on customer support teams
  • reducing the necessity for stringent QA test suites

• Improves the user experience by

  • Reducing the frequency of systems problems and providing faster resolution for those that do occur
  • Highlighting UX degradations (e.g., application slowness) that would not otherwise be caught or reported
  • Building trust by providing visibility into the operational health of systems

• Provides the necessary infrastructure and processes to meet certain compliance standards such as audit logging

Benchmark

Metrics

These are common measurements that can help an organization assess their performance in this pillar. These are intended to be assessed within the context of performance on the key platform metrics, not used in a standalone manner.

Organization Goals

These are common goals to help organizations improve their performance on the key platform metrics. While each goal represents a best practice, the level of impact and optimal approach will depend on your organization's unique context.