What is a 1% level of significance? When to use it

Have you ever wondered why some statistical tests seem more stringent than others? Or why we sometimes demand stronger evidence before accepting a new finding? Significance levels are at the heart of these decisions, playing a crucial role in how we interpret data.

In this blog, we'll dive into the basics of significance levels, zoom in on the 1% significance level, and explore when and why you might choose this stricter threshold. Whether you're running experiments with Statsig or just curious about statistical testing, understanding significance levels will help you make better decisions based on data.

The basics of significance levels in statistics

Significance levels are simply thresholds we set to decide when to reject the null hypothesis in statistical tests. They represent the maximum acceptable risk we're willing to take of making a Type I error—rejecting a true null hypothesis. You can read more about this concept here.

In practice, we compare the p-value from our test to the chosen significance level. If the p-value is less than or equal to the significance level, we consider the result statistically significant. This means there's enough evidence to suggest that the observed effect isn't just due to random chance. Learn more about p-values and significance levels here.

The most common significance levels are 5% (0.05) and 1% (0.01). When we choose a 5% level, we're accepting a 5% chance of incorrectly rejecting the null hypothesis. Opting for a 1% level means we require stronger evidence before we're willing to reject it. Read more about these levels here.

So, how do we decide which level to use? It depends on the potential consequences of making errors. In fields like medicine, where false positives can have serious implications, a lower significance level like 1% is often preferred to minimize the risk of incorrect conclusions. For more details, check out this resource.

Understanding the 1% significance level

Using a 1% significance level sets a stricter standard for rejecting the null hypothesis compared to the more common 5% level. By lowering the significance level to 0.01, we reduce the probability of making a Type I error—rejecting a true null hypothesis. But this increased strictness can also raise the chance of making a Type II error—failing to reject a false null hypothesis. (Learn more about this trade-off here.)

To see how this plays out, imagine you're running an A/B test on your website with Statsig. With a 5% significance level, a p-value of 0.03 would lead you to reject the null hypothesis and conclude that one design performs better. However, at a 1% level, that same p-value isn't sufficient, so you'd conclude there's no significant difference.

Choosing between a 5% or 1% significance level depends on the context and the consequences of errors in your specific situation. In areas like medicine, where false positives could have serious effects, a 1% level might be more appropriate to minimize risks. In contrast, fields like marketing or product development might accept a 5% level to avoid missing out on potentially beneficial changes.

When deciding on the significance level for your tests, it's important to balance reducing false positives with detecting true effects. Some argue that a 1% level is superior due to its lower risk of Type I errors (see discussion here). Others contend that the choice should be context-dependent, considering factors like sample size, effect size, and the costs associated with errors.

When to use a 1% level of significance

Sometimes, minimizing the risk of false positives is essential. This is when a 1% significance level is most appropriate (as discussed here). Fields like medicine, where Type I errors can have severe consequences, often require this stricter threshold.

For example, incorrectly concluding that a drug is effective when it's not could lead to harmful treatments. In these cases, a 1% level of significance provides greater confidence in the results. However, using a lower significance level comes with trade-offs.

To maintain the same statistical power at a 1% level, you'll need larger sample sizes. This can increase the cost and duration of experiments. Researchers must carefully consider the balance between minimizing false positives and the feasibility of their studies. More on significance levels can be found here.

Applying the 1% significance level in practice

When designing studies with a 1% significance level, careful planning is essential to ensure robust results. A lower significance level requires stronger evidence to reject the null hypothesis, reducing the risk of Type I errors. However, this also increases the chance of missing true effects—Type II errors.

To achieve adequate statistical power at a 1% significance level, larger sample sizes are necessary. This ensures your study has a high probability of detecting a true effect if one exists. Tools like Statsig can help you calculate the required sample size based on your desired power, effect size, and significance level. You can learn more about sample size calculation here.

When interpreting results using a 1% significance threshold, focus on the practical significance alongside statistical significance. A p-value below 0.01 indicates strong evidence against the null hypothesis, but you should also consider the effect size and confidence intervals. This helps determine whether the observed effect is meaningful and actionable in the context of the study. For more on this topic, check out this blog post.

Replication studies and meta-analyses can further validate findings at a 1% significance level. By conducting multiple studies and combining their results, researchers can increase confidence in their conclusions. This is particularly important when making critical decisions based on the results, such as in medical or high-stakes business contexts.

Closing thoughts

Understanding when and how to use different significance levels is crucial for making informed decisions based on data. A 1% significance level offers a stricter threshold, reducing the risk of false positives but potentially increasing the chance of missing true effects. The choice between a 1% and 5% significance level should be guided by the context of your study, the consequences of errors, and the resources available.

Whether you're working in medicine, business, or any other field that relies on statistical testing, carefully considering your significance level is key. Tools like Statsig can help you design, run, and analyze experiments with the appropriate level of rigor.

If you're interested in learning more about significance levels and statistical testing, check out resources like Statsig's blog and other articles linked throughout this post.

Hope you found this helpful!