Top 7 missteps of the Mattermost open source project
Today marks five years since the start of the Mattermost open source project!
We reflected on this exciting journey and reminisced about our missteps along the way. We’ve had many laughs and wanted to share them with our community:
1. Flipping the table: Our user model
Before Mattermost v3.0, things looked very different for our users.
For example, users were owned by the teams they were members of. If you wanted to be a member of multiple teams, you needed multiple accounts.
We did have some minimal UI if you had multiple accounts on the same server. Each team you were a member of would show in the menu (no sidebar). But it would only work if you used the same email address on both accounts. What a drag.
We were thinking like a SaaS service and sharding by teams. With Mattermost v3.0, we put aside our SaaS ideas and made a product designed for self-hosting. Each server has a set of users and users were members of teams rather than teams owning users.
As you might imagine, this was quite the refactor. It came to be known as “the user model flip” and still has ramifications in our code today. This is visible to users in the way certain elements are scoped. For example, custom emoji are server scoped but accessed through a team route. It is also the reason that slash commands and other integrations don’t work across teams.
2. textToJsx
Today, Mattermost has a complex Markdown rendering pipeline that handles all sorts of different formatting. In the past, it was…less pretty.
Let me introduce you to an amazing function called textToJsx:
export function textToJsx(textin, options) {
var text = textin;
if (options && options.singleline) {
var repRegex = new RegExp('\n', 'g'); //eslint-disable-line no-control-regex
text = text.replace(repRegex, ' ');
}
var searchTerm = '';
if (options && options.searchTerm) {
searchTerm = options.searchTerm.toLowerCase();
}
var mentionClass = 'mention-highlight';
if (options && options.noMentionHighlight) {
mentionClass = '';
}
var inner = [];
// Function specific regex
var hashRegex = /^href="#[^']+"|(^#[A-Za-z]+[A-Za-z0-9_\-]*[A-Za-z0-9])$/g;
var implicitKeywords = UserStore.getCurrentMentionKeys();
var lines = text.split('\n');
for (let i = 0; i < lines.length; i++) {
var line = lines[i];
var words = line.split(' ');
var highlightSearchClass = '';
for (let z = 0; z < words.length; z++) {
var word = words[z];
var trimWord = word.replace(puncStartRegex, '').replace(puncEndRegex, '').trim();
var mentionRegex = /^(?:@)([a-z0-9_]+)$/gi; // looks loop invariant but a weird JS bug needs it to be redefined here
var explicitMention = mentionRegex.exec(trimWord);
if (searchTerm !== '') {
let searchWords = searchTerm.split(' ');
for (let idx in searchWords) {
if ({}.hasOwnProperty.call(searchWords, idx)) {
let searchWord = searchWords[idx];
if (searchWord === word.toLowerCase() || searchWord === trimWord.toLowerCase()) {
highlightSearchClass = ' search-highlight';
break;
} else if (searchWord.charAt(searchWord.length - 1) === '*') {
let searchWordPrefix = searchWord.slice(0, -1);
if (trimWord.toLowerCase().indexOf(searchWordPrefix) > -1 || word.toLowerCase().indexOf(searchWordPrefix) > -1) {
highlightSearchClass = ' search-highlight';
break;
}
}
}
}
}
if (explicitMention &&
(UserStore.getProfileByUsername(explicitMention[1]) ||
Constants.SPECIAL_MENTIONS.indexOf(explicitMention[1]) !== -1)) {
let name = explicitMention[1];
// do both a non-case sensitive and case sensitive check
let mClass = '';
if (implicitKeywords.indexOf('@' + name.toLowerCase()) !== -1 || implicitKeywords.indexOf('@' + name) !== -1) {
mClass = mentionClass;
}
let suffix = word.match(puncEndRegex);
let prefix = word.match(puncStartRegex);
if (searchTerm === name) {
highlightSearchClass = ' search-highlight';
}
inner.push(
<span key={name + i + z + '_span'}>
{prefix}
<a
className={mClass + highlightSearchClass + ' mention-link'}
key={name + i + z + '_link'}
href='#'
onClick={() => searchForTerm(name)} //eslint-disable-line no-loop-func
>
@{name}
</a>
{suffix}
{' '}
</span>
);
} else if (testUrlMatch(word).length) {
let match = testUrlMatch(word)[0];
let link = match.link;
let prefix = word.substring(0, word.indexOf(match.text));
let suffix = word.substring(word.indexOf(match.text) + match.text.length);
inner.push(
<span key={word + i + z + '_span'}>
{prefix}
<a
key={word + i + z + '_link'}
className={'theme' + highlightSearchClass}
target='_blank'
href={link}
>
{match.text}
</a>
{suffix}
{' '}
</span>
);
} else if (trimWord.match(hashRegex)) {
let suffix = word.match(puncEndRegex);
let prefix = word.match(puncStartRegex);
let mClass = '';
if (implicitKeywords.indexOf(trimWord) !== -1 || implicitKeywords.indexOf(trimWord.toLowerCase()) !== -1) {
mClass = mentionClass;
}
if (searchTerm === trimWord.substring(1).toLowerCase() || searchTerm === trimWord.toLowerCase()) {
highlightSearchClass = ' search-highlight';
}
inner.push(
<span key={word + i + z + '_span'}>
{prefix}
<a
key={word + i + z + '_hash'}
className={'theme ' + mClass + highlightSearchClass}
href='#'
onClick={searchForTerm.bind(this, trimWord)} //eslint-disable-line no-loop-func
>
{trimWord}
</a>
{suffix}
{' '}
</span>
);
} else if (implicitKeywords.indexOf(trimWord) !== -1 || implicitKeywords.indexOf(trimWord.toLowerCase()) !== -1) {
let suffix = word.match(puncEndRegex);
let prefix = word.match(puncStartRegex);
if (trimWord.charAt(0) === '@') {
if (searchTerm === trimWord.substring(1).toLowerCase()) {
highlightSearchClass = ' search-highlight';
}
inner.push(
<span key={word + i + z + '_span'}>
{prefix}
<a
className={mentionClass + highlightSearchClass}
key={name + i + z + '_link'}
href='#'
>
{trimWord}
</a>
{suffix}
{' '}
</span>
);
} else {
inner.push(
<span key={word + i + z + '_span'}>
{prefix}
<span className={mentionClass + highlightSearchClass}>
{replaceHtmlEntities(trimWord)}
</span>
{suffix}
{' '}
</span>
);
}
} else if (word === '') {
// if word is empty dont include a span
} else {
inner.push(
<span key={word + i + z + '_span'}>
<span className={highlightSearchClass}>
{replaceHtmlEntities(word)}
</span>
{' '}
</span>
);
}
highlightSearchClass = '';
}
if (i !== lines.length - 1) {
inner.push(
<br key={'br_' + i}/>
);
}
}
return inner;
}
Full of special cases and hard-to-understand code, textToJsx was a thorn in our sides for a while. Its output slowed the webapp to a crawl with more complicated posts. Really, its output was crazy, with deeply nested span tags serving as its signature feature. There were so many DOM elements that the browser couldn’t handle it.
Thankfully, its reign was short-lived, and it has since been replaced by a better Markdown rendering pipeline. Long-time Mattermost developers still remember it vividly, and we immortalized its output on a shirt. Here’s the output of a simple plain text message:
3. OAuth2 backwards
Back in the early days of Mattermost—before our support of third-party authentication existed—we were preparing Mattermost for inclusion in the GitLab Omnibus.
As part of this work, we implemented support for Postgres. But GitLab also asked us to implement OAuth2. We didn’t know a whole lot about OAuth at the time, but we wanted to be included in the Omnibus so we dutifully implemented OAuth2…as a service provider.
For those unfamiliar with OAuth2, the service provider acts as the authentication source, providing third-party “consumers” with authorized access to user information. This can allow a user to login with that service, such as logging in with GitHub. But in that example, it wouldn’t allow users to use another service provider to login to GitHub.
Of course, GitLab didn’t want Mattermost acting as a service provider. They wanted Mattermost to act as a consumer so that users would be able to log in to Mattermost with their GitLab accounts.
With egg on our faces, we then had to put all that code aside and reimplement as a consumer. The service provider code was not wasted, however. It’s still available today for integrations to use if the developer prefers them over user access tokens. This functionality may have not been implemented if we hadn’t misunderstood GitLab’s request in the first place!
4. Various #Mattermug incidents
When someone contributes to the Mattermost open source project for the first time, we send them a mug in appreciation:
But things don’t always go smoothly. Some people get the wrong mug:
Others are surprised we know where they live (we send mugs to the mailing address submitted in the Mattermost CLA):
And some others have accidents:
5. A lack of localization support
When Mattermost v1.0 was released, there was no localization support, which limited the utility of Mattermost in many places.
We knew this, and had been discussing implementing a localization framework for a long time. Time and complexity kept pushing it back, even past v1.0.
Little did we know that, during this time, Elias Nahum had been translating Mattermost into Spanish for his own use. He eventually got tired of keeping up with the translations himself and dropped a ~40,000 line pull request on us.
Needless to say, we were surprised. We worked with him to split his pull request into smaller pieces so we could reasonably review each and get it merged. Suddenly, through the power of open source contributors, Mattermost was localized in Spanish, and soon other languages followed.
We offered Elias a job, he accepted it, and he still works at Mattermost today.
6. Prioritizing features
Sometimes, you really have to ask customers what they want.
In the early days of Mattermost, we were debating the priority of some features. We had just added emoji support and one of the candidates was our “custom emoji” feature.
Some of our early customers had asked for the feature, but we hadn’t considered it a priority. Surely our large business customers would be more concerned about compliance features or our plans to improve the sidebar, right?
That’s what we thought until one of our largest customers at the time said this:
Custom emojis are integral to our workflow.
They put it second on the list of features they wanted us to implement—second only to LDAP synchronization. Based on this and other customer feedback, we prioritized the feature. Once custom emoji was released, customers came back saying Now we can be productive!
7. Issues with our public Jira instance
Being an open company, our development process is open for all to see. This includes our ticketing system Jira. We think this is a great way to make sure the community and customers can follow along and provide feedback as we develop Mattermost.
Having an open ticketing system isn’t without its issues (actually, that’s the whole point, right?). The first issue is the constant emails from well-meaning security researchers letting us know that our Jira instance is public. Yes, we’re aware!
But the more interesting issue is that sometimes customers will watch these issues. Throughout the normal development process, tickets are changed and closed. Sometimes, we would close tickets as “Won’t Do” that customers were watching. Understandably, they would be confused by this.
Being aware of all of the outside audiences for your ticketing system is not something inherent to company structures. Most assume this kind of internal communication is private. Over time, we became more disciplined about our use of Jira, such as sharing why we weren’t going to implement a feature, so that anyone watching wouldn’t be confused by our decisions.
Final thoughts
So there you have it: a snapshot of some of the bigger missteps we’ve made over the last five years. We’ve done our best to learn from these miscues and prevent similar mishaps from occurring.
With software, it’s all about iteration and continuing to improve. If you’re interested in making an impact in open source, learn more about contributing to Mattermost.
To all of our users and community members, thanks so much for being part of our journey over the last five years. We look forward to seeing what happens over the next five and beyond!
