A year of security fixes in Go

Over the past couple of months, we’ve analyzed the security fixes published for the Go programming language this year. Some of the results of that analysis you may have already read about in our previous blog post on the topic. But while our previous blog post focused on particular findings related to a single CVE, this post is about the wider project that spawned those discoveries.

12 months, 6 security releases, 14 CVEs

Starting sometime in October 2024, we took it upon ourselves to go over, review, and analyze as many recent Go security releases as we could—and at that time, the three latest ones seemed like a reasonable target to start with. Our goal was simple: learn as much as possible, gain confidence in the quality of the fixes, and contribute back to the Go community if the opportunity arises. Largely thanks to the high quality of the security releases, the target soon grew from three releases to everything released in 2024, and here we are!

During October and November, we plowed through 6 Go security releases, each addressing 2 release branches, and meticulously reproduced 14 vulnerabilities, assessed their impact, read and analyzed the fixes (in both release branches as well as the master branch), their correctness and completeness, and scoured the codebase for additional instances of similar bugs. The results are interesting, to say the least. Some quick statistics for those too impatient to read through the full data below:

• Our analysis covers 6 (x2) security releases between the 5th of March and the 5th of September 2024
• The release versions range from Go 1.21.8 and 1.22.1 in the March release to Go 1.22.7 and 1.23.1 in September
• Out of the 14 vulnerabilities fixed in those releases, 3 were originally discovered at Mattermost
• All 14 vulnerabilities were easily reproducible thanks to accurate descriptions, readable code, and good test cases
• The real-world impact of all 14 vulnerabilities, with one exception, matched the descriptions published by the Go team
• 1 fix out of 14 was partial and bypassable
• 1 new vulnerability was found that hadn’t landed in a release yet and we were able to stop from ever doing so!

The results, month by month

We began our analysis with the latest release and worked our way backward. The reasoning was that recent releases are more likely to contain undiscovered bugs, as they’ve spent less time in public and production use. However, one could argue the opposite: software evolves over time, and older fixes have undergone more changes, creating more opportunities for new issues to emerge.

Either way, it makes more sense to present the results by starting from the beginning—from the release that came out first.

March, Go 1.21.8 and 1.22.1

The first security release of the year came out in March. It wasn’t the first overall release of the year: the Go 1.22.0 major release had come out a month before, and the minor releases 1.21.7 and 1.20.14 a day before that.

The March release was, however, the most substantial security release of the year, and that’s explained at least partially by the timing of the previous non-security releases. Bundling security fixes with major releases is something you understandably want to avoid, and including security fixes in the minor releases right before the major release was obviously not an option either. Together with all the holidays near the turn of the year, that meant the March release covered security issues reported as far as four months earlier.

Two of the five vulnerabilities covered in the March release were originally reported by Mattermost: CVE-2023-45289, an HTTP cookie and authentication header scoping issue that we’ve blogged about before, and CVE-2024-24784, a PUBLIC track parsing bug in net/mail that could lead to parser alignment issues. The fix for the former, also detailed in the earlier blog post, was still holding up well at the time we revisited it. The same goes for the latter, where a major part of the fix was to properly handle the return value of the net/mail.addrParser.skipCFWS method.

The third vulnerability was CVE-2023-45290, a memory exhaustion bug in net/http.Request.ParseMultipartForm, where individual header line lengths could exceed the limit specified in the argument. Line-specific limits-checking was added to complement the higher-level limits already in place. The fix was still solid when we reviewed it several versions later; only one commit had since touched the file where the fix was implemented.

Fourth was CVE-2024-24783, a denial-of-service vulnerability affecting TLS clients and some servers. The root cause of the vulnerability was a type assertion against a nil value; the fix was a simple nil check.

Finally, CVE-2024-24785 was a cross-site scripting vulnerability in very specific uses of the html/template package: a custom JSON marshaler could include user input in the errors it returned, and when that marshaler was passed to a template action, the error message was not correctly escaped. The fix involved scanning the error message for known bad substrings and replacing them with their escaped counterparts. This type of blocklisting approach is widely recognized as error-prone, and, in fact, our analysis quickly uncovered a loophole in the fix: the search-and-replace logic was case-sensitive and could be trivially bypassed using upper-case or mixed-case inputs. We reported this to the Go team and the bypass is now being fixed as a PUBLIC track vulnerability.

April, Go 1.21.9 and 1.22.2

With the April security release came one fix: CVE-2023-45288. You may recognize the CVE number from multiple news articles, blog posts, and technical writeups as well as the CERT-CC vulnerability note VU#421644 published on the same date as the Go release. It is, of course, the HTTP/2 CONTINUATION flood DoS vulnerability.

Bartek Nowotarski, the researcher who originally discovered the vulnerability, wrote a detailed account of the technical aspects that also covers its specific manifestation in Go. The post is definitely worth a read. We found no issues with the fix during our analysis.

May, Go 1.21.10 and 1.22.3

In May one fix for the net package was published. The vulnerability in question, CVE-2024-24788, was another PUBLIC track issue resulting from an unhandled error return value in Go’s DNS client: a bad DNS response could send the client into an infinite loop. The fix was a simple matter of adding the missing error check, and that fix was still holding when we reviewed it.

Another fix published in May was for the cgo compiler toolchain: CVE-2024-24787. This was another instance of a bug originally discovered at Mattermost and one which we’ve also blogged about. In short, attempting to build a program that included malicious LDFLAGS or CFLAGS values in cgo directives could result in code execution, contrary to Go’s explicit promise of not executing code at build time. This was the 7th instance of a code execution bug in the same feature – that we know of – and in our blog post in May we predicted that it would not be the last.

When reviewing the latest code relevant to CVE-2024-24787, we came across some new changes: support for the -Wl,--push-state and -Wl,--pop-state LDFLAGS values had been recently added. The handling of these new flags had a bug in it: the regular expression matching used for detecting the flags was not properly anchored to the beginning of a string. This resulted in another code execution bug, which we reported to the Go team. Luckily the new changes had not been included in a release yet, and the vulnerability never made it to a release. A fix was applied to the master branch and no security release was necessary.

June, Go 1.21.11 and 1.22.4

The release in June contained two fixes: CVE-2024-24789, a PUBLIC track parsing issue in archive/zip, and CVE-2024-24790, where incorrect values were returned by some methods in net/netip.

The ZIP parsing fix addressed an issue where errors in the End of Central Directory Record (EOCDR) were ignored, resulting in the parser potentially seeing content in ZIP files that other, correctly behaving parsers would not see. This was an example of a parser alignment vulnerability: if, for example, a malware scanner used one ZIP implementation to scan the contents of a file and then a separate program implemented in Go extracted the contents, the extracted contents might not be what was scanned. We’ve dealt with parser alignment issues before; their impact is difficult to estimate, but can sometimes be serious, like in the case of the XML parsing bugs we blogged about in 2020.

The fix for the ZIP issue involved adding proper error handling and error propagation, and refusing to parse malformed files, which seems to work well.

The bug in net/netip was a rather interesting one: methods such as IsLoopback, IsLinkLocal, and IsMulticast were incorrectly returning false for IPv4-mapped IPv6 addresses. For example netip.IsLoopback would return false for the address ::ffff:7f00:1, which is the IPv4-mapped IPv6 address corresponding to the IPv4 loopback address 127.0.0.1. This could result in, for example, IP address filtering bypasses.

The fix for the IP address issue was to explicitly unmap those addresses in the affected methods before doing any comparisons: instead of attempting to check whether ::ffff:7f00:1 is a loopback address, it would be converted to 127.0.0.1 first. After the fix, the IsUnspecified method is still missing this unwrapping, and the behaviors between net.IP and net/netip.Addr differ. This one remaining method working somewhat unintuitively doesn’t seem to have any security implications, however.

July, Go 1.21.12 and 1.22.5

In July came CVE-2024-24791, a bug in HTTP 100-continue handling whose impact the Go team described as denial of service. Our analysis was different, and in November we blogged in length about how the actual impact is HTTP desynchronization, which could be far worse than DoS in some circumstances.

As we noted in the November blog post, the fix was still correct, and we found no other instances of similar issues.

September, Go 1.22.7 and 1.23.1

The last security release of the year came already in September, with three new CVEs: CVE-2024-34155, CVE-2024-34156, and CVE-2024-34158. All three were denial of service bugs caused by stack exhaustion. In CVE-2024-34158, build constraint parsing was missing size tracking, in CVE-2024-34155 similar tracking was missing from code parsing nested literals in Go source code, and in CVE-2024-34156 the depth tracking in encoding/gob was reset to zero in some recursive code paths.

All three vulnerabilities were fixed by improving constraint enforcement and rejecting deeply nested inputs. In our analysis the fixes for these specific issues were sufficient, but stack exhaustion in inherently recursive code is a difficult problem, particularly in Go, from where it cannot be recovered. We’ve seen numerous instances of these types of vulnerabilities before and will likely continue to see them in the future.

Takeaways

The project of analyzing an entire year’s worth of CVEs was an interesting undertaking: a nontrivial effort, but easily worth it, since new vulnerabilities were found and vulnerable code prevented from reaching a release. Taking existing vulnerabilities and making an effort to understand them thoroughly also turns out to be a solid approach to making exploratory security research somewhat more systematic.

Although we did find new vulnerabilities and issues in some of the vulnerability fixes, the project also highlights the quality of work the Go team does: the vast majority of vulnerabilities were fixed correctly and comprehensively, and described in enough detail to allow for easy reproduction and verification.

We hope that this blog post also has value to the Go community in increasing technical understanding and security awareness, showcasing one approach to security research, and encouraging security contributions both in the Go standard library and tooling and elsewhere in the ecosystem, like the Mattermost bug bounty program.