Go fixes its 7th code execution bug in the same feature

If there’s one Go programming language feature that just doesn’t seem to catch a break when it comes to security, it’s the CFLAGS and LDFLAGS handling in cgo. This is a feature that lets parts of Go source code control the compiler and linker flags that are used to build that same code. It’s a highly useful and necessary feature, enabling things like specifying statically linked libraries and platform-specific options without having to rely on external makefiles or manually typed compiler invocations. But it’s also incredibly difficult to implement correctly, in a way that doesn’t expose you to build-time attacks.

So far, this single feature has been affected by seven different vulnerabilities that all lead to arbitrary code execution: CVE-2018-6574, CVE-2020-28366, CVE-2020-28367, CVE-2023-29404, CVE-2023-29405, CVE-2023-39323, and the latest addition fixed in Go 1.21.10 and 1.22.3, CVE-2024-24787. Four out of these seven, including the latest one, were discovered as part of research done at Mattermost.

Mattermost products or infrastructure have not been affected by these vulnerabilities at any point. Security research at Mattermost is often done preemptively and exploratorily. When we discover vulnerabilities before adopting the affected product or feature, we can work around these security issues or delay the adoption and wait for patches as needed.

A common root cause

At a high level, all of the seven vulnerabilities share a similar root cause. The oldest vulnerability, CVE-2018-6574, is also the easiest to understand. Prior to Go 1.8.7 and 1.9.4, the following Go code was allowed by the compiler toolchain:

// #cgo CFLAGS: -fplugin=attack.so

This simple line comment is a cgo directive. When building a program with cgo enabled, this line tells the toolchain to pass the -fplugin=attack.so flag to the C compiler. The flag, in turn, tells the C compiler to load a plugin from the file attack.so. The plugin code is then executed at compile time, even if the program being compiled is never run.

This original vulnerability was fixed by implementing an allowlist for the compiler and linker flags that would be accepted via cgo directives. Any exceptions to that allowlist would have to be specified via environment variables, which are not attacker-controlled even in scenarios where the source code is.

The six CVEs that have since followed this original one have been various ways of bypassing the allowlist, sneaking disallowed flags past it, or smuggling them inside other flags.

Complexity as the enemy of security

Due to the nature of their core function, compiler toolchains are unavoidably complex. This complexity also extends to features that, on the surface, seem relatively simple, like the compiler and linker flag handling Go. Indicative of the complexity of the flag handling is the length of the allowlist intended to secure it: already 41 entries long when originally introduced, it has since grown to 172 entries in Go 1.22.3. Not only is that a large number on its own, most of the entries are regular expressions that match much more than just a single static string. A total of 25 commits have touched the allowlist since its introduction, adding new entries or adjusting existing ones. Go 1.22.3 also introduced an entirely new denylist – so far only one entry long.

Another hint is in the name: it’s called the toolchain, not the tool. When you invoke go build, you’re not just invoking a Go compiler that produces a single binary. When cgo is in use, go build first invokes go tool cgo to produce a transformed representation of your Go and C (and C++ and Fortran!) source files. Then it invokes the compilers: either gc or gccgo for Go, and whichever compilers you have configured for the other three languages: GCC, LLVM Clang, Apple Clang, MinGW, GFortran, Flang, and so forth. Some compilers might invoke a separate preprocessor tool before compiling, or the preprocessor might be built in. Then the linkers, again depending on your configuration and platform: go tool link, GNU ld, LLD, ld64… And all of these steps – preprocessing, compiling, and linking – are run separately for each Go package.

There are also multiple ways of invoking different parts of the toolchain and passing the flags needed: differences between gc and gccgo, situations where command-line flags are used directly versus through response files, variants on response file syntax, tunneling linker flags through the compiler command-line, and of course differences between how command-line invocations are handled on Windows versus Unix-like platforms.

All of the CVEs that came after CVE-2018-6574 can be considered direct results of this complexity. The latest issue, for example, stems from the fact that Apple’s ld64 linker supports a dangerous flag named -lto_library, whereas on other platforms all linker flags with a -l prefix are safe.

More bugs likely to come

After seven CVEs in the same feature, it’s not unreasonable to think that you’d eventually run out of places where bugs can hide. And with many kinds of features, that’s probably what would happen: each new CVE found means more unit tests added, and each new researcher involved means more eyes on the code, after all.

Unfortunately, with these bugs, it’s unlikely we’ve seen the end yet.

Because of the inherent complexity, covering every edge case is difficult. And because of the external dependencies—all the compiler and linker command-line interfaces and their platform-specific variants—there’s no single developer organization that has control over the entire toolchain. Even if the Go development team does their utmost to secure their end, as they have done, a relatively minor change in any other part of the chain could easily introduce an issue that only manifests when invoked through Go.

To put it bluntly, the compiler toolchain is just incredibly difficult to secure.

No need to panic

Although arbitrary code execution sounds bad, and it certainly can be, most Go developers or organizations shipping programs written in Go probably needn’t worry. In the cases discussed in this blog post, you’re only vulnerable if you build code from untrusted sources without intending to ever execute it in the same context, and that’s not entirely typical. A developer running go install example.com/some/package will be building a package downloaded from the Internet, but chances are they also intend to run it – so if the package is malicious, it makes no difference whether it exploits a build-time execution bug or waits for willing execution. An automated build pipeline, on the other hand, might build a package without intending to execute it, but the source is usually trusted.

Developers should pay attention to a rare scenario where they are accepting untrusted code into their automated build pipeline. An example of this could be a PR build on GitHub that then gets uploaded to some external object storage for outside consumption without ever executing the built program. In that scenario, an attacker could leverage build-time code execution to compromise the build pipeline and exfiltrate secrets or poison the caches used for later builds.

If you do have infrastructure that might be at risk of these types of attacks, we have a few tips: Starting with the obvious, keep your Go versions up to date. But also, prepare for zero-days – try to design your systems in a way that minimizes the impact of these types of attacks. Compartmentalize, isolate, sandbox. And finally, consider disabling cgo if you’re not using it.