In the early days, we configured a simple Pod Disruption Budget (PDB) across a majority of our service deployments. However, as we’ve scaled and refined our infrastructure, we encountered a few inefficiencies that required a more nuanced solution.
Our original PDB setup was straightforward:
This worked well for many scenarios, but we quickly discovered a flaw in the way minAvailable
was calculated, particularly for services with fewer than five pods.
According to the Kubernetes documentation, the minAvailable
calculation always rounds up, meaning that service deployments running fewer than five pods were effectively locked from any disruptions (minAvailable == total number of pods).
While this ensures high availability, it also introduced several long-term challenges:
Resource waste: During traffic slowdowns, hosts would end up stuck running a single pod that couldn’t be removed. This resulted in wasted resources during periods of low traffic, leading to higher costs and less efficient scaling.
Blocked updates: Automated rolling updates to node pools were often blocked, slowing down our ability to deploy improvements or patches. This became especially frustrating during time-sensitive updates, such as security patches, when certain nodes were left running outdated configurations.
To resolve these issues, we revisited our approach to Pod Disruption Budgets. Instead of using minAvailable
, we switched to maxUnavailable
. This change addressed our problem in a simple yet effective way.
Just as minAvailable
rounds up its calculations, so does maxUnavailable
. By defining maxUnavailable
, we were able to guarantee that at least one pod could always be disrupted, even in deployments with fewer than five pods.
This small change had a big impact, allowing us to handle disruptions in a more controlled and flexible manner while preventing resource waste and blocking issues.
Here’s what our updated PDB configuration looks like:
Now, with this updated configuration, our service deployments can gracefully handle disruptions, even during low-traffic periods. By ensuring that at least one pod can be interrupted, we’ve mitigated the long-term challenges and improved both our resource management and deployment velocity.
For teams managing Kubernetes service deployments, especially those with variable pod counts, swapping to maxUnavailable
can be a small but impactful change.
By allowing one pod to be disrupted at all times, we prevent unnecessary resource waste and ensure that rolling updates can happen without unnecessary friction.
A short list of reasons why a great experimentation tool is a horrible idea.
Statsig's AI Prompt Experiments allow you to run experiments for AI-powered products and gain real-time insights into what's working and what's not.
Master data-driven product development with Statsig. Simplify experimentation, make informed decisions, and accelerate your product's growth—all without complex coding.
Debunk the myth that you can never accept the null hypothesis and learn when you should by exploring the key differences between Fisher’s and Neyman-Pearson’s frameworks.
Use our customizable, detailed cost comparison tool and flexible pricing assumptions to find out which platform reigns supreme.
Our recent optimizations to target apps significantly reduce config propagation latency, ensuring performance and stability for large-scale environments using Statsig.