Tech Beetle briefing UNITED STATES OF AMERICA AI

Peloton's engineering team makes the case for test in production

Essential brief

In 2024, Peloton's engineering team eliminated its traditional performance environment and shifted to testing in production to handle peak traffic exceeding 450,000 requests per second while reduci

Key topics

peloton engineering team case test production Turkey Burn Traditionally However

Key facts

Peloton eliminated its traditional performance environment to reduce AWS costs and improve performance.

Testing in production enabled Peloton to handle over 450,000 requests per second without major incidents.

The company saved an estimated 30% to 40% on infrastructure spending by adopting this approach.

Engineering teams incorporated cost as a key metric alongside latency and uptime to balance performance and expenses.

Highlights

Peloton faced peak traffic exceeding 450,000 requests per second.

In 2024, Peloton removed its performance environment and shifted to test in production.

This change resulted in a 30% to 40% reduction in AWS infrastructure costs.

No major incidents occurred during high-stakes events like the annual Turkey Burn workout.

Peloton redesigned systems for failure and used a mix of custom and commercial tools to support testing in production.

Why it matters

Peloton's shift to testing in production highlights a significant change in how large-scale tech companies manage performance and costs. By eliminating traditional performance environments, companies can reduce infrastructure expenses while maintaining reliability during peak traffic. This approach may influence other organizations to reconsider their testing strategies in favor of more cost-effective and resilient solutions.

Peloton faced a significant challenge in managing peak traffic events that exceeded 450,000 requests per second while controlling the costs associated with its AWS infrastructure. Traditionally, the company relied on a dedicated performance environment to simulate real-world workloads and prevent outages or slowdowns. However, in 2024, Peloton's engineering team decided to eliminate this performance environment entirely and instead adopt a test-in-production strategy.

The decision was driven by an evaluation of the return on investment for maintaining a separate performance environment, which was found to be insufficient. By testing directly in production, Peloton aimed to reduce infrastructure costs and improve system performance. This shift led to an estimated 30% to 40% reduction in AWS spending.

To support this new approach, Peloton's engineers redesigned their systems with failure in mind, ensuring resilience and reliability even when issues occur. They also introduced cost as a key engineering metric alongside traditional measures such as latency and uptime. This holistic view helped balance performance and expenses effectively.

Peloton utilized a combination of custom-built and off-the-shelf management tools to monitor and maintain the member experience during testing in production. This approach has proven successful, with no major incidents reported, including during critical high-traffic events like the annual Turkey Burn workout.

The move away from a dedicated performance environment reflects a broader industry trend toward more agile and cost-effective testing methodologies. Peloton's experience demonstrates that, with proper design and monitoring, testing in production can be a viable strategy for managing large-scale, high-demand systems.

Key topics in this update include peloton, engineering team, and case.