Mattermost not affected by Codecov breach

Recently the software industry has been shaken by multiple high-profile supply chain security incidents. The latest such case, rooted at Codecov, also touched Mattermost as a Codecov user. And while Mattermost products, infrastructure, or customers were not in the end impacted in any way by the Codecov breach, it did lead to a swift internal investigation of Codecov use cases and patterns.

In the spirit of open source, transparency and knowledge sharing, we’re covering some of the details of the investigation here.

What is Codecov?

As the name suggests, Codecov is a SaaS product for measuring test coverage in code. Mattermost uses Codecov for a number of open source projects, such as officially supported Mattermost plugins and mmctl. The tool neatly integrates with CircleCI as well as several other CI/CD platforms and provides feedback to developers directly on GitHub pull requests. A pretty useful thing, then.

Because Codecov is so useful, and on top of that, free for open source projects and education, it’s also used very widely across the software industry. But wide adoption and easy integration with build pipelines comes with a downside; when a product like Codecov suffers a security breach, a lot of people are going to be affected.

On April 15, Codecov announced that exactly that had happened. Attackers had managed to replace the code of the Bash uploader — used by Codecov to upload coverage information from build environments to their cloud — with a malicious credential grabber. Integrations like the official CircleCI Orb would automatically download the malicious code, extract any available credentials from the environment it’s running in, and send them to the attackers.

CI/CD pipelines often contain secret keys and tokens for things like signing code, uploading build artifacts, and even deploying websites and apps to production environments. The attackers that successfully compromised Codecov could potentially access all that. And while it seems they didn’t, the attackers could also have tampered with builds, injecting malware directly into the artifacts produced.

How is Codecov used at Mattermost?

CI/CD at Mattermost is split across multiple platforms: public CircleCI infrastructure, gitlab.com, and a self-hosted GitLab instance. Codecov is used exclusively on CircleCI, where it scans incoming changes from pull requests and posts results on GitHub as feedback for developers. In most cases the scan is an independent Job that doesn’t do anything else and doesn’t have access to Contexts. This means that no credentials are exposed to the Codecov script and that the script cannot tamper with any other part of the pipeline.

The pipelines, as they stand, are an evolution of earlier CircleCI and Jenkins pipelines that went through major changes during 2020 in order to both pre-emptively tackle issues similar to the Codecov incident and to support the launch of Mattermost Cloud.

Response to the Codecov incident at Mattermost

On April 15, when Codecov first started publicly communicating about the security incident, the first one at Mattermost to take notice was one of our Site Reliability Engineers. He, in turn, made the Security Team aware, which kicked off an investigation.

Within 20 minutes of being notified, the Security Team at Mattermost, already familiar with CI pipeline internals and previous efforts to secure them, had an initial assessment: any of the main source repositories and pipelines, such as mattermost-server or mattermost-webapp, were not affected, and impact to any others was highly unlikely.

On April 16, a thorough review of the CI pipeline configurations was completed. The Security Team’s initial assumptions were confirmed: no impact to any system invoking Codecov was possible.

Designing secure CI/CD pipelines

A handful of high-level guiding principles were in the end more than enough to secure Mattermost against the Codecov incident:

• The principle of least privilege
• Separation of privileges
• Defense in depth

These are principles that every Security Engineer is already familiar with. In our next post, we elaborate on how we applied them to CI specifically.