Cutting Test Runtime by 60% with Selective Parallelism in Go 

At Mattermost, we recently set out to enable code coverage across our backend server test suite, which is a significant step toward improving product quality and test visibility.  

For a while, we’d been aware that our CI test jobs were — to put it mildly — a bit slow, often averaging around 40 minutes to complete. When we looked into adding code coverage, we hit a roadblock: calculating it, especially with the -covermode=atomic flag, introduced a noticeable slowdown due to synchronization overhead. 

We saw an opportunity to tackle both issues at once. We decided to implement selective parallelism, a strategy focused on speeding up the tests from our largest and slowest packages. This targeted approach allowed us to address the root causes of our lengthy test times while seamlessly integrating the code coverage support we needed. 

The Problem 

Our backend includes nearly 16,000 unit and integration tests, but not all tests contribute equally to runtime. Large packages like app and api4 contain numerous tests that rely on timeouts and sleeps, which, when executed serially, disproportionately increase the overall test duration. 

We used gotestsum, an invaluable tool for aggregating and analyzing test output, to quickly identify the slowest packages and individual tests. Rather than attempting to parallelize the entire suite, we focused on the areas where parallelization would have the most significant impact: the largest, slowest-running packages. 

What We Did 

While Go runs tests from different packages in parallel by default, tests within a package are executed serially, unless explicitly marked with t.Parallel(). Unlocking intra-package parallelism meant carefully reviewing our tests to ensure they were concurrency-safe and properly isolated. 

This was mostly a manual process, going file by file and test by test to identify safe candidates for parallel execution. Along the way, we encountered several common pitfalls, such as shared global state, improper use of subtests, and tests modifying environment variables. We’ll cover these in more detail in the sections on guidelines and gotchas. 

Another critical piece was optimizing database usage. Running parallel tests interacting with the database could have introduced excessive overhead if each required a freshly provisioned instance. To avoid that, we introduced a simple pooling mechanism: a fixed-size pool of isolated databases (matching the parallelism level) that can be reset and reused between test runs. This gave each test safe, exclusive access to a database without the cost of full reinitialization. 

So far, we’ve applied these improvements to a dedicated CI job for code coverage. We’re monitoring its behavior and considering a phased rollout of parallelism across additional test jobs. 

Results 

• ✅ Coverage job execution time dropped from ~40 minutes to ~15 minutes 

• ✅ Focused changes limited flakiness and made debugging manageable 

• ✅ Infrastructure and best practices are now in place to support broader adoption 

While these improvements aren’t yet rolled out to every job, the early results are promising. 

Common Pitfalls and Gotchas 

Parallelism in Go tests is powerful but tricky. Here are some of the most common issues we encountered while converting tests to run in parallel: 

• Modifying global state: Tests writing to or reading from shared variables caused race conditions. 

• Changing directories: os.Chdir() affects the entire test process and will break other parallel tests. 

• Setting environment variables: Using os.Setenv() introduces changes in global state and race conditions. 

• Improper test teardown: Tests that rely on side effects, shared caches, or specific execution order often become flaky or fail outright. 

When to Use Parallel Tests 

As part of this effort, we refined our testing guidelines to help contributors safely write parallel tests. A few key rules we follow: 

• ✅ Generally safe: Tests with fully isolated setup/teardown logic and no reliance on shared state. 

• ⚠️ Subtests: Only safe if each subtest has its own setup logic. If one subtest depends on the state set by another, running them in parallel can cause failures. 

• ❌ Unsafe: Tests that modify global state, set environment variables, or perform process-wide actions (like changing the working directory). 

We’ve encoded these principles into our documentation and code review practices to help catch issues early and prevent subtle concurrency bugs. 

What’s Next 

We’re continuing to monitor the parallel coverage job for signs of flakiness and plan to expand parallelism to more CI jobs in the near future. The initial investment of reviewing tests, refactoring legacy patterns, and building supporting infrastructure was considerable. Still, the payoff in execution time and developer velocity could be significant. 

If you’re working on a large Go codebase and struggling with slow tests, consider a selective approach: Start with the slowest packages, isolate shared state, and gradually introduce parallelism.  

For more details, check out our testing guidelines or join the Mattermost open source community.