Coordinated disclosure of XML round-trip vulnerabilities in Go’s standard library

December 14, 2020

Engineering

This blog post is a part of Mattermost’s public disclosure of three serious vulnerabilities in Go’s encoding/xml related to tokenization round-trips. The public disclosure comes as a result of several months of work, including collaborating with the Go security team since August 2020 and with affected downstream project maintainers since earlier this month. There are several potential security problems created by these vulnerabilities, one of which is a complete bypass of SAML authentication.

These issues do not impact Mattermost customers. See “No impact to Mattermost customers” below for more info.

Despite significant efforts by the Go security team, it has not been possible to patch the vulnerabilities discussed in this blog post. Only one of the three is on the roadmap to be patched in the foreseeable future (see “Update 12/13/2020” below). The mitigation for the remaining issues will be adding a new mode in Go to make explicit that these vulnerabilities exist. Downstream projects that assume certain integrity guarantees from Go’s standard library are vulnerable when those guarantees do not hold. The implications of these vulnerabilities in key use cases we have identified present a material security concern. If you maintain a Go-based project that relies on XML integrity, we urge you to read this post carefully.

We don’t take this type of public disclosure lightly, and it has not been an easy decision to make. There are three reasons why we are doing this now:

The Go security team has determined that the root causes of the vulnerabilities cannot be reliably addressed.
Go is introducing publicly-visible API changes related to these issues in an upcoming major release, which risks making the vulnerabilities public without explicit public disclosure.
- The Go security team has agreed to Mattermost coordinating disclosure prior to making their changes public in order to minimize impact to parts of the ecosystem that can’t benefit from the new API. Refer to the “Upcoming changes in Go” section for details.
Careful steps have been taken to privately disclose the issues to as many affected parties as possible prior to proceeding with public disclosure. All parties involved in the disclosure have collaborated to ensure mitigations are in place and sufficient.

The vulnerabilities

As part of our security work, we have identified three independently exploitable vulnerabilities in Go’s encoding/xml. The advisories are linked below:

As evident from the titles, the vulnerabilities are closely related. The core issue is the same in all three: maliciously crafted XML markup mutates during round-trips through Go’s decoder and encoder implementations.

What does that mean in practice? If your application processes XML and, while processing it, parses markup that’s the output of at least one preceding round of parsing and serialization, you can no longer assume the output of that parsing matches the output from the preceding round. In other words, passing XML through Go’s decoder and encoder doesn’t preserve its semantics.

This may sound like a non-issue or a source of functional bugs at worst, but the reality of it is much more serious. There are applications that absolutely require semantic integrity in XML in order to be secure — a major example being SAML.

SAML, or the Security Assertion Markup Language, is an XML-based standard approach to single sign-on on the web. It conveys user identity in XML documents whose integrity and authenticity are guaranteed using cryptographic signatures, a scheme called XML-DSig. Because of these vulnerabilities, Go-based SAML implementations are in many cases open to tampering by an attacker: by injecting malicious markup to a correctly signed SAML message, it’s possible to make it still appear correctly signed, but change its semantics to convey a different identity than the original document.

The actual impact of these XML round-trip vulnerabilities of course varies by use case, but in SAML SSO it’s easy to understand: if your SAML messages can be altered to say you’re someone you’re not, the result is arbitrary privilege escalation within the scope of the SAML Service Provider, or in some cases even complete authentication bypass.

As mentioned before, despite significant efforts by the Go security team, there is no Go patch that would reasonably address these vulnerabilities. That doesn’t mean all hope is lost: Along with the public disclosure of these vulnerabilities, we’re open-sourcing an XML validation library that can be used as a workaround. The library as well as basic instructions for its use can be found at github.com/mattermost/xml-roundtrip-validator.

No impact to Mattermost customers

At the time this public disclosure is taking place, no supported versions of Mattermost products are affected by any of the vulnerabilities. Any impact to Mattermost software has been addressed independently prior to this disclosure and communicated to customers according to Mattermost’s standard disclosure policy. No action is required from Mattermost customers at this time.

Mattermost staff originally detected the first one of these vulnerabilities during an internal security review on the 28th of August, 2020. As an immediate response the only affected feature of Mattermost server, a new SAML implementation still in beta at the time, was forcefully disabled. The update was rolled out to customers in a dot release on the 3rd of September. Mattermost Cloud was not generally available at that time, and so was never affected.

The affected SAML implementation was available as a non-default configuration in Mattermost server versions from 5.20.0 up to and including 5.26.1. Mattermost deployments were only vulnerable if all of the following were simultaneously true:

The server was running a version between 5.20.0 and 5.26.1, released before September 2020. This excludes Extended Support Releases 5.25.5 and above.
The server was configured to use SAML SSO authentication.
The setting “Use Improved SAML Library (Beta)” had been explicitly enabled by an administrator.

Details of the security release addressing these vulnerabilities can be found on the Mattermost Security Updates page under the identifier MMSA-2020-0030.

Since the September dot release, Mattermost products have not been vulnerable. We have no evidence of this issue being ever exploited.

Impact to the Go ecosystem

We have identified three major open source SAML implementations affected by the Go XML round-trip vulnerabilities: Dex SAML Connector, github.com/crewjam/saml, and github.com/russellhaering/gosaml2. The maintainers of all three projects were included in private embargoed disclosure prior to publishing any details. Maintainers were also given advance access to the github.com/mattermost/xml-roundtrip-validator repository.

Patches are available in the following releases:

We have also privately reached out to maintainers of significant applications and products that depend on these SAML implementations and given them an advance warning of today’s disclosure without revealing technical details.

Having an idea of the scale and range of affected products may help in understanding why we’ve gone to such lengths in coordinating this. We reached out to a total of 20 companies and projects in addition to the maintainers of the SAML implementations named above. The affected products include:

A variety of cloud orchestration tools
Security products from DFIR to secure access and endpoint management
A commercial IAM solution from a major cloud hosting company
Multiple web software frameworks
Hardware root of trust APIs
Well-known SaaS products

If you find yourself among the organizations we contacted, we urge you to start patching immediately.

It’s unlikely that we found all affected use cases, libraries, and products, and we may have missed even major ones. Even if you didn’t hear from us but suspect your particular use case of encoding/xml may be affected, please consider reviewing your codebase for vulnerable patterns.

Upcoming changes in Go

The Go security team is planning changes to encoding/xml that address round-trip vulnerabilities by deprecating existing behaviors from a security perspective. A new API is expected to land in Go 1.16 that will allow disabling namespace prefix parsing entirely. The current functionality will remain available, but it will no longer be considered secure for use cases that require round-trip stability; Go will stop accepting vulnerability submissions related to its behavior.

The Go security team has chosen this path after determining that the encoding/xml package and its data structures were not designed to make safe round-trips possible. The new API allows them to lower developers’ expectations to a more manageable level, as the support for round-trips in the most problematic cases can be explicitly dropped.

By Mattermost’s estimates this new API will not be a reasonable solution for most use cases currently affected by the vulnerabilities. Parsing and resolving namespaces is an essential requirement for correctly implementing SAML, and even considering only a limited set of real-world SAML messages without strict namespacing requirements would be unlikely to allow for a secure implementation.

Because of this reason, we recommend that affected developers continue using the github.com/mattermost/xml-roundtrip-validator module even after Go 1.16 becomes available. As a long-term solution, it may make sense to refactor code in a way that avoids encoding round-trips altogether, but it’s unclear if such refactoring is possible in all cases.

Update 12/13/2020

We just heard from the Go team that there are possibly additional changes coming in a future version. It’s not clear to us whether these changes fully fix the XML validation issues in Go, but we’re hopeful that these issues will get fully patched in Go in the near future. We have been informed that a public discussion with the Go team will be opened in this issue at https://golang.org/issue/43168.

Disclosure timeline

The first of the XML round-trip vulnerabilities was originally discovered on the 28th of August, 2020, and by the 31st of August Mattermost had traced the root cause to encoding/xml and reached out to the Go security team. The graphic below shows the full timeline from initial discovery to public disclosure. Blue circles represent actions taken by Mattermost, red circles are responses from the Go security team.

Mattermost would like to thank the Go security team and downstream library maintainers for their many efforts and their collaboration in responding to these important issues.

Security contacts at Mattermost

If you suspect you’ve found a vulnerability in a Mattermost product or open source project such as the github.com/mattermost/xml-roundtrip-validator module, please report your findings to [email protected]. We ask you not to use public GitHub issues for reporting security issues.

If you have general questions about application security at Mattermost or the vulnerabilities included in this disclosure, don’t hesitate to reach out to us at [email protected].