Coordinated Omission

Coordinated Omission is a specific type of measurement error that occurs in performance testing and monitoring when the measurement system (typically a client-side load generator) fails to record or account for the latency of requests that are delayed or dropped due to system overload. Instead of measuring the true end-to-end latency from the moment a request should have been sent until its response is received, it often only measures the latency of requests that successfully completed within a certain window or before the next request was scheduled. This omission of delayed requests, especially under high load, leads to an underestimation of actual system latency and an overestimation of throughput.

Background and Evolution

The concept gained significant prominence through the work of Gil Tene, co-founder of Azul Systems and creator of HdrHistogram. Tene extensively highlighted how traditional latency measurement techniques, particularly those used in many load testing tools, inherently suffer from this flaw. His presentations and writings brought widespread awareness to the issue, demonstrating how common practices could mask severe tail latency problems. The development of HdrHistogram was a direct response to the need for accurate latency measurement that correctly accounts for Coordinated Omission.

Purpose and Importance

The primary purpose of understanding and addressing Coordinated Omission is to obtain an accurate and truthful representation of a system's performance characteristics, especially its latency profile under various load conditions. Without accounting for it, performance engineers risk making critical decisions based on flawed data, leading to systems that perform poorly in production despite "passing" performance tests.

• Accurate Latency Metrics: It ensures that reported average, median, and especially tail latencies (P99, P99.9, P99.99) reflect the actual experience of users or downstream systems. Ignoring it can make P99 latency appear acceptable when, in reality, a significant percentage of requests are experiencing severe delays.
• Effective Optimization: Misleading metrics can lead to misdirected optimization efforts. Engineers might focus on optimizing components that appear slow based on flawed data, while the true bottlenecks causing significant delays remain unaddressed.
• Reliable Capacity Planning: Incorrect latency data can lead to under-provisioning resources, as the system's true capacity and breaking points are obscured. This can result in production outages or degraded service quality.
• Realistic Benchmarking: For comparing different systems or configurations, benchmarks must be free from Coordinated Omission to provide a fair and accurate comparison.

Relationship to the Knowledge Graph

Coordinated Omission is deeply intertwined with fundamental performance engineering concepts. It directly impacts the validity of Latency and Response Time measurement, particularly for Tail Latency analysis, which is crucial for understanding user experience. It's a fundamental consideration in Performance Engineering and Benchmarking methodologies. Concepts like Throughput and Resource Utilization are also affected, as an overloaded system might appear to maintain throughput while silently accumulating massive latency due to omitted requests. It highlights the importance of accurate Workload Characterization and understanding the difference between open-loop and closed-loop systems, and how Little's Law can be misapplied without accurate latency data.

Coordinated Omission typically arises in client-side load generation scenarios where the client attempts to maintain a fixed request rate (e.g., 100 requests per second). The "coordination" refers to the client's internal scheduling of requests, which can inadvertently lead to the omission of delayed responses from latency measurements.

The Mechanism of Omission

Consider a client configured to send requests at a fixed rate of 10 requests per second (RPS), meaning a new request is scheduled every 100 milliseconds (ms).

Time (ms) | Event
----------|-----------------------------------------------------------------
0         | Client sends Request A
100       | Client sends Request B
200       | Client sends Request C
...

Now, imagine Request A encounters a delay and takes 150ms to complete. If the client's measurement system only records the time from when a request is sent until its response is received, and it doesn't account for the fact that Request A's delay might have impacted the client's ability to process or even send subsequent requests efficiently, the reported latency for Request A would be 150ms. However, if the client's internal logic then moves on to send Request B at 100ms and Request C at 200ms, and it drops or misattributes the long-running Request A from its overall latency statistics, then Coordinated Omission occurs.

The core issue is that the client's fixed-rate generation can mask the true impact of slow responses. If a request takes longer than the inter-request interval, the client might still send the next request on schedule, but the measurement of the previous slow request might be incomplete or simply dropped from the sample if it exceeds a timeout or if the client moves on. This leads to an underrepresentation of the longest latencies.

Impact on Performance Metrics

• Average Latency: Artificially lowered because the longest-running requests (which are the most impactful on user experience) are underrepresented or completely excluded from the average.
• Percentile Latency (Tail Latency): Severely distorted. P99 or P99.9 values will appear much better than reality, as the requests that would define these high percentiles are precisely the ones being omitted.
• Throughput: Can appear stable or even high, masking the fact that many requests are experiencing unacceptable delays or are being dropped.

Addressing Coordinated Omission

To accurately measure latency and avoid Coordinated Omission, the measurement system must:

• Record Intended Start Time: Timestamp each request at the moment it is scheduled to be sent or initiated, not just when it leaves the client.
• Measure End-to-End: Calculate latency from this intended start time until the full response is received.
• Account for All Requests: Ensure that all requests, including those that are severely delayed or time out, are included in the latency distribution. No request should be dropped from the sample simply because it took too long.
• Use Appropriate Tools: Employ specialized libraries and tools (like HdrHistogram) that are designed to correctly capture and report latency distributions, even with extreme outliers.

Benefits of Addressing Coordinated Omission

• Accurate Performance Insights: Provides a true picture of system behavior under load, revealing actual user experience and system bottlenecks.
• Improved User Experience: By identifying and mitigating true tail latencies, systems can be optimized to provide a more consistent and responsive experience for all users.
• Effective Resource Allocation: Enables more precise capacity planning and resource provisioning, preventing both under-provisioning (leading to outages) and over-provisioning (leading to unnecessary costs).
• Reliable Benchmarking: Ensures that performance comparisons between different architectures, configurations, or software versions are fair and based on valid data.
• Faster Troubleshooting: Accurate latency data helps pinpoint the exact components or stages causing delays, accelerating root cause analysis during incidents.

Limitations of Ignoring Coordinated Omission

• Misleading Performance Metrics: Generates artificially low average and percentile latencies, creating a false sense of security about system performance.
• Ineffective Optimization: Leads to wasted effort on optimizing non-critical paths while the real performance inhibitors remain unaddressed.
• Production Incidents: Systems that appear performant in testing due to omitted delays can fail catastrophically or degrade severely in production under real-world load.
• Poor User Experience: Users experience much higher latencies than reported, leading to frustration, abandonment, and reputational damage.
• Flawed Capacity Planning: Results in incorrect resource estimates, potentially leading to costly over-provisioning or dangerous under-provisioning.

Common Mistakes

• Using Simple Client-Side Timers: Relying solely on time.Now() at the start and end of a request handler without accounting for the time the request spent waiting to be handled or waiting to be sent by the client.
• Fixed-Rate Load Generators with Naive Measurement: Many traditional load testing tools, when configured for a fixed request rate, can suffer from Coordinated Omission if they don't explicitly track the full lifecycle of each request.
• Ignoring Tail Latency: Focusing only on average response times, which are easily skewed by Coordinated Omission, rather than critical percentiles like P99 or P99.9.
• Assuming Server-Side Metrics are Sufficient: While valuable, server-side metrics don't capture network latency or client-side queueing, which are crucial for end-to-end user experience.
• Not Using Tools Designed for Accurate Latency: Failing to employ specialized libraries or tools like HdrHistogram that are built to correctly measure latency distributions.

Real-world Examples

• Microservices Architectures: A client making requests to a service mesh might experience delays due to overloaded proxies or slow downstream services. If the client's load generator doesn't account for these delays, the reported latency for the service mesh will be optimistic.
• Database Performance: A load test against a database might show good average query times, but if the database is struggling and queueing requests, the client might be omitting the long-tail queries, making the database appear faster than it is.
• Message Queues: Producers sending messages at a fixed rate might report low latency for message publishing, but if the queue is backed up, consumers might experience significant delays that are not captured by the producer's metrics.
• Web Servers Under Load: A web server experiencing high CPU utilization or I/O contention will introduce delays. A client-side test that doesn't correctly account for these delays will underreport the actual user-perceived latency.

Best Practices

• Measure from the Client's Perspective (End-to-End): Always measure the total time from the moment a request is intended to be sent until the full response is received by the client.
• Use HdrHistogram or Similar Libraries: Integrate libraries specifically designed for accurate latency measurement into your load generators and monitoring tools.
• Timestamp Requests at Inception: Record a timestamp when a request is scheduled or initiated, not just when it's sent over the wire.
• Track All Outstanding Requests: Ensure your load generator continues to track and measure the latency of all requests, even those that are severely delayed or time out. Do not drop them from your sample.
• Understand Open-Loop vs. Closed-Loop: For simulating real-world user behavior, open-loop load generation (fixed request rate) is often necessary, but it must be combined with Coordinated Omission-aware measurement.
• Combine Client-Side and Server-Side Metrics: Use client-side metrics for end-to-end user experience and server-side metrics (CPU, memory, I/O, queue lengths, internal service latencies) to diagnose the root cause of delays.
• Monitor Queueing: Pay close attention to queue lengths at various layers of your system (load balancers, application servers, databases, message queues) as these are often indicators of impending Coordinated Omission issues.