How our remote team ships like clockwork
On the 16th of every month, we release an update to the Mattermost server. The release happens on the same day, every month, without fail. It’s a cadence that our customers have come to rely on, and it helps us deliver new features and updates with drumbeat regularity. Hitting this hard deadline every month while ensuring high-quality releases requires clear processes and organizational discipline. This is a challenge for any team. But when all members are remote, it takes even tighter collaboration to stay on track.
At Mattermost, we’ve developed a release process and toolbox that is robust, reliable, and remote-friendly. Because our engineering organization is spread across the globe, we’ve learned how to keep everyone focused and in lockstep with a common schedule.
Releasing a self-hosted product is different from a cloud product
The nature of the cloud is such that releases can happen at any time. Teams can push small updates multiple times per day or push lightly-tested updates that can be immediately patched if necessary. In addition, data from SaaS or PaaS products is available right away, so teams can immediately analyze load times, usage patterns, and other metrics.
With self-hosted software like Mattermost servers and clients, releases need careful planning. Each product update requires users to download, install, and sometimes test the release in their own environment, which for many customers is not always possible. This means that it takes longer to gather performance data and leverage it in future updates.
A “tick tock” cadence supports our commitment to quality
To help address the challenge of self-hosted software releases, we’ve established a “tick tock” release cycle at Mattermost. One month we do a “feature release” that includes new or updated features, and the next month is a “quality release” with fixes or quality improvements to existing code. This gives our development teams a two-month timeframe to work on features or fixes, do the appropriate testing, and merge code with confidence. It also helps us build trust with our customers by setting expectations and focusing on quality.
Timing is everything (so is prioritization)
To hit our monthly release day, the whole process absolutely must run like clockwork. There’s no room for error. Code needs to be ready by the deadline, testing needs to happen quickly and efficiently, and tough decisions need to be made along the way. If a feature is not ready by “judgement day,” then it gets cut. Period. This is not as harsh as it may sound. We tend to include as many features as possible in our release plan at the beginning and hope to get as much done as possible, yet we expect that roughly 70–80% will actually make it into the release. Every effort is made to prevent urgent issues arising at the last minute and to preserve the standards of quality and customer trust that we apply to every release.
Prioritization and timing go hand in hand. Starting with the release plan, we evaluate which features have the highest impact for our customers and which are feasible within the timeframe. Later, prioritization also becomes critical during testing and “go/no go” decisions are made.
When the clock is ticking, communication happens 24/7
As time and the team march toward the 16th, good communication helps that march go smoothly. There are multiple parties involved in every release—developers, QA engineers, product managers, and the release manager—who must collaborate constantly. One of the benefits of a remote team is that there’s usually someone online at any time of day, whether it be dev or QA, to answer questions or help solve problems. Business teams, such as marketing, sales, and customer success managers, also need to stay up to date on releases. We also provide status updates externally to our customers and community, because as an open source company, we strongly believe that transparency goes a long way in building trust.
The Mattermost team maintains a toolbox of services that support our collaboration. Mattermost itself, of course, is at the heart of it all. Our Release Discussion channel is the primary hub of all release communications. It’s open to the entire Mattermost community, but is most heavily used by those directly involved in getting a release out. We also use Jira to track new features and bug fixes, GoogleDocs for documentation, and Zoom for team meetings, such as R&D standups or issue triage.
Inside the Mattermost release process
So how does our remote release process happen in practice? To give you an idea, here’s a look at the major milestones of a server feature release.
Feature Complete (T-minus 20 working days)
All major features must be complete and ready to merge. Prior to the day, the release manager makes sure everyone is aware of the impending deadline by posting reminders in the Release Discussion channel and messaging team leads. If any pull requests are not merged by this date, then the release manager changes release milestones in Jira after checking with team leads that those features can be pushed to a future release.
Judgement Day (T-minus 15 working days)
The release team reviews the status of all features and issues in a Zoom meeting. If anything needs to be cut, the Release Manager informs internal stakeholders in the Release Discussion channel and asks the feature owners to revert the code. As things change rapidly at this stage, the channel is critical for status communications.
Code Complete (T-minus 14 working days)
All major bugs have been fixed, code is locked, and the release is ready for testing.
Release Candidate Cut (T-minus 9 working days)
The first release candidate is cut and ready to test. Prior to this, the Release Manager has been posting lists of bugs and issues to the Release Discussion channel for dev teams to work through. When the candidate is cut, there should be no major issues. Mattermost’s Jenkins integration enables an automatic notification in the channel that includes the release version number and RC1 ID.
Release Candidate Testing (T-minus 8 working days)
Testing of the release candidate happens over an intense two days. Test automation tools, such as Rainforest, Cypress, and Selenium, are activated and do much of the heavy lifting. Manual testing supplements the process, and the QA team delivers test assignments to product managers, developers, and QA engineers via the Release Discussion channel. They use a spreadsheet to document all tests and record results. Bugs and issues get triaged during release team meetings on Zoom. Ad hoc conversations happen between testers and developers in the Release Discussion and other channels.
Code Freeze (T-minus 5 working days)
At this point, all bug fixes are complete except for any showstoppers. No further code changes will be accepted, preventing last minute regressions, fixes, or testing needed. The release team continues to triage issues and may push minor bugs to the next (quality) release. Everyone is kept informed on the Release Discussion channel
Release Build Cut (T-minus 2 working days)
This is the final build, ready to ship. The QA team has finished all testing, and barring any major issues, they post their approval of the release in the Release Discussion channel. The release is pushed to production and a notification is posted in the channel.
Release Day (T-minus 0 working days)
Marketing communications begin to roll out, and the release is communicated on the blog and other marketing channels. The Release Manager schedules a retrospective Zoom meeting within the next five days. Using the channel, she notifies the release team and asks them to add topics for discussion to a Google Doc. Afterward, she posts a summary of the retro in the channel for information sharing and archiving purposes.
Check out the Mattermost documentation for a full deep dive into our release processes. Note that our mobile apps follow the same release process and cycle as our server, but our desktop apps differ slightly.
What have we learned along the way?
Thanks to our monthly releases, our entire engineering organization has become a well-oiled, code-shipping machine. Everyone knows the drill, and critically, everyone knows the importance of our shared process. We’ve learned a lot over the years, but three best practices in particular can help other remote release teams:
- Deadline commitments—make sure everyone involved in a release is fully on board with the release plan, schedule, and deadlines. Be prepared to make changes along the way. If a feature is not ready, push it out rather than risk delaying the release. Communication and reminders are vital to helping everyone keep their commitments.
- Automation—try to automate repeatable steps wherever possible in order to speed up processes and free up the team to focus on larger issues.
- Thoughtful iteration—after every release, gather as much feedback as possible from the team and customers, identify learnings, and make needed changes to the process for the next release cycle.
On our release process roadmap: full QA test automation
Although our release process is working well for us, we’re constantly looking for ways to improve. One of our highest priorities is test automation. We’re looking to incorporate unit tests for new features and write more end-to-end tests using the Cypress testing framework.
More automation would help make the process move faster and make those deadlines less painful. It would also free up developers and product managers from doing manual testing, so they can better focus on building great new features for the next release.
Get help adopting remote work
If your team is considering adopting Mattermost to collaborate remotely, our team can help. We’re offering free 30-min Zoom sessions on using Mattermost for remote work – contact us for more info.
