In the early days, we configured a simple Pod Disruption Budget (PDB) across a majority of our service deployments. However, as we’ve scaled and refined our infrastructure, we encountered a few inefficiencies that required a more nuanced solution.
Our original PDB setup was straightforward:
This worked well for many scenarios, but we quickly discovered a flaw in the way minAvailable
was calculated, particularly for services with fewer than five pods.
According to the Kubernetes documentation, the minAvailable
calculation always rounds up, meaning that service deployments running fewer than five pods were effectively locked from any disruptions (minAvailable == total number of pods).
While this ensures high availability, it also introduced several long-term challenges:
Resource waste: During traffic slowdowns, hosts would end up stuck running a single pod that couldn’t be removed. This resulted in wasted resources during periods of low traffic, leading to higher costs and less efficient scaling.
Blocked updates: Automated rolling updates to node pools were often blocked, slowing down our ability to deploy improvements or patches. This became especially frustrating during time-sensitive updates, such as security patches, when certain nodes were left running outdated configurations.
To resolve these issues, we revisited our approach to Pod Disruption Budgets. Instead of using minAvailable
, we switched to maxUnavailable
. This change addressed our problem in a simple yet effective way.
Just as minAvailable
rounds up its calculations, so does maxUnavailable
. By defining maxUnavailable
, we were able to guarantee that at least one pod could always be disrupted, even in deployments with fewer than five pods.
This small change had a big impact, allowing us to handle disruptions in a more controlled and flexible manner while preventing resource waste and blocking issues.
Here’s what our updated PDB configuration looks like:
Now, with this updated configuration, our service deployments can gracefully handle disruptions, even during low-traffic periods. By ensuring that at least one pod can be interrupted, we’ve mitigated the long-term challenges and improved both our resource management and deployment velocity.
For teams managing Kubernetes service deployments, especially those with variable pod counts, swapping to maxUnavailable
can be a small but impactful change.
By allowing one pod to be disrupted at all times, we prevent unnecessary resource waste and ensure that rolling updates can happen without unnecessary friction.
Understand the difference between one-tailed and two-tailed tests. This guide will help you choose between using a one-tailed or two-tailed hypothesis! Read More ⇾
This guide explains why the allocation point may differ from the exposure point, how it happens, and what you to do about it. Read More ⇾
From continuous integration and deployment to a scrappy, results-driven mindset, learn how we prioritize speed and precision to deliver results quickly and safely Read More ⇾
The Statsig <> Azure AI Integration is a powerful solution for configuring, measuring, and optimizing AI applications. Read More ⇾
Take an inside look at how we built Statsig, and why we handle assignment the way we do. Read More ⇾
Learn the takeaways from Ron Kohavi's presentation at Significance Summit wherein he discussed the challenges of experimentation and how to overcome them. Read More ⇾