Flowdock is a modern web-based team messenger. All software developers should be using it instead of their Campfires, Skype Chats, IRCs, etc. because it better supports their actual workflow.

Last weekend we completed a transition from Flowdock’s database of choice, Cassandra, to another NoSQL alternative, MongoDB. Since our technology stack has always generated some interest, I’ll now try to justify our decision in public.

Some of our users might remember this:

Twitter screenshot: having some database problems

At some point we started to have some stability issues with Cassandra. All nodes would go into an infinite loop, running GC and trying to compact the data files – occasionally falling off the cluster. We were unable to solve the problem, except that restarting and then compacting a node usually settled it down for a while. Other people had reported similar problems. Last couple of weeks our Cassandra nodes always ate all the resources they were given, slowing down Flowdock.

This was not the first time we had run into problems because of our bleeding edge database choice. When upgrading from 0.4 to 0.5, we had to shut down the cluster, only to find out that it hadn’t flushed everything to the disk (even though we explicitly flushed it, as instructed). Thus we ended up having a couple of minutes of discussions lost, and our custom-built indices were miserably out of date and needed to be rebuilt. I think it was 4 AM when we finally got to leave the office.

The NoSQL scene has evolved since we made our original decision to go with Cassandra. MongoDB is changing rapidly, and the latest addition of auto-sharding and replica sets made it a compelling alternative to Cassandra. So I decided to give it a try.

It took me a day to write a conversion script for our data. Within a week or so we were able to run Flowdock purely on MongoDB. It was tested internally for a couple of weeks before it was deployed to production.

Now that we have done the change, I’m happy to see that we got some benefits (very well known in most databases) in addition to the performance and reliability characteristics:

  1. Smart (multikey) indices. Manually maintaining indices was always tedious, and MongoDB can index everything we need out-of-the-box. For example, our messages have tags, implying a document format like this:
    { content: "Write a blog post about #mongodb.",
      workspace: 'myflow',
      tags: ["mongodb", "todo", "@Otto"] }
    
    Now when looking for my own tasks, Flowdock backend only needs to do this query:
    db.messages.find({
      workspace: 'myflow',
      tags: { $all: ["todo", "@Otto"] }
    })
    
  2. Queries. No matter how simple your data model is, every once in a while you need to perform a query that you didn’t plan in advance. MongoDB lets you construct complex queries directly from the console, pretty much like an SQL database. It will then perform a sequential scan, which is still much faster and more convenient than processing millions of rows manually, on the client-side.
  3. Map-Reduce. It’s great for stuff like analytics. MongoDB’s Map-Reduce support is not perfect, but at least it’s easy to use.
  4. GridFS makes storing our files very easy. The storage capabilities expand together with the rest of our MongoDB cluster.

We have faced only some minor limitations:

  1. We found a bug in JSON parsing that got fixed in 10 minutes.
  2. Dots are not allowed in BSON document keys. Typically it might not be a problem, but we had to work around it in our data migration.
  3. Document size is limited to 4 megabytes. It’s not a problem with our data model, but since MongoDB supports fantastic atomic in-place updates, you have to be careful not to grow your documents above this limit.
  4. Adding new nodes is not as easy as it is with Cassandra. However, Cassandra has its own problems load-balancing them.

So far it’s been a very smooth ride. Development and database administration just got a whole lot easier.

» Now give the new and better Flowdock a try!