Ionrock Dot Org

My Weblog

Why Choose MongoDB

A co-worker will occasionally send along recent blog posts that place into question our decision to use MongoDB. These are small reminders that technical decisions are never black and white and that no matter how careful you are, there are going to be trade offs. Usually the complaints I hear about MongoDB have to do with durability. Specifically, people have a problem with 10gen (the folks who make MongoDB) suggesting that you run at least two MongoDB instances to provide some level of reliability in the face of one system going down.

The fact is we have been bitten by a bug that required a good deal of downtime. This kind of thing totally stinks. I’d say it is unacceptable, but the reality is that at some point you have to trust that a system will do its job. In other words, it is not enough to prevent problems, you must also have a means of recovering from problems. In our case, that recovery plan was the problem. Shame on us.

This has since been remedied and there is probably more to do. I’m sure other organizations might have considered the hiccup a worthy reason to abandon MongoDB for another more durable system. The problem is that while MongoDB’s durability did prove to be a major problem, that isn’t the whole story. There are other reasons to choose MongoDB, but durability is not one of them.

In our situation, the biggest benefit we get from MongoDB is the ability to query. This seems like such a basic tenant of databases that you’d be hard pressed to find someone who didn’t believe running queries on your data isn’t the killer feature of a database. With that in mind, I’d point out that many NoSQL systems do not provide the same level of support for queries. CouchDB (which I’m personally a fan of) is in fact really bad at ad-hoc queries. Once you save a view in CouchDB, you’re in pretty good shape, but randomly querying the DB is considered bad form. Likewise, generating a new view on a massive dataset is time consuming as well. That is the trade off CouchDB made. In return you get excellent durability, a fantastic HTTP based interface and the ability to utilize pure JSON as a document format. Totally reasonable trade offs if you ask me. The only problem is, the ability to query the dataset in real time is an order of magnitude more important to us than the other features. We can run a slave (for reads even), deal with BSON, and use the socket based Python driver that MongoDB provides, yet if we couldn’t do realtime queries in a reasonable about of time, then what is the point?

By the way, I don’t mean to pick on CouchDB or any other system. The fact of the matter the discussion is not over what system to use as much as weighting what features are important. For us, the ability to query very quickly and under load trumps the other features such as single machine durability. We’ve paid the price of that trade off once, so we’re aiming not to do so again. Thanks to my cautious and objective thinking co-worker, we also consistently consider other data stores. That said, I think we made a good choice to go with MongoDB. It is not perfect, but nothing is. MongoDB has been meeting needs very successfully and I have no problem recommending it.

Concurrency and Focus

Last night as we were driving back towards the great state of Texas, we listened to Coast to Coast. If you’ve never listened, it is a radio show that traditionally focuses on non-mainstream topics that can often stretch well into the weird. It is a fun show to listen to for freak factor as well as the interesting perspectives. The guest in this case was the author of the book The Shallows, Nicholas Carr. The book argues that the Internet has changed the way people think and robbed people of focus, and personally, I have to agree.

One interesting thing he talked about was the brain’s concept of memory. He described the basic ideas of RAM vs. CPU Cache vs. the hard drive. The idea was that we’ve gotten faster RAM thanks to the web, but in doing so, we’ve lost the ability to save ideas and knowledge to our long term memory storage. One argument he disagreed with was that the web offers you a place to store information so you can think about other things. I partially disagreed with this because as a programmer, much of my job is finding interfaces and layers to help manage complexity, effectively relieving my mind of the burdens associated with lower level problems.

This quest to hide complexity has been a major theme in computing. Back when computers were actual people, the idea was that a scientist could let the computer do the low level work while they continued to hone their larger level algorithms and models. The ability to abstract and classify is critical to moving beyond what you can do by yourself. There are so many details in the world that without tools to manage them, life would be unbearable.

Concurrency is a very similar situation. I read this article discussing how trends in concurrency often do not fairly characterize the problem. This got me thinking about the issues with focus. If we consider concurrency in terms of workers that all equally do things, you miss the obvious abstractions you get from more specific tools. Like learning to focus, the goal is not to store information as much it is to manage the transitions and interruptions. It is often simpler to have a master process that handles specifically who does what than relying on a generic solution to magically do everything correctly. Just like the scientists in the early days of computing, we should think of concurrency in terms of how we can create abstractions while freeing up resources in a way that doesn’t force us to pass everything through to a intermediary. There were probably plenty of computations that early programmers did themselves. But at some point, there was a decision made on a set of parameters that suggested the solution be sent to a computer to do the work. It is this kind of mindset that will make concurrency in the future relevant in the future and reasonable to implement.

Band Relationships

We just got back from playing Denver for The UMS. Our first show was pretty crappy and honestly we didn’t play very well. It ended up being OK as people showed up, but for whatever reasons, the stage manager and people running it were non-existent. Not a huge deal, but it made for a rough start after a long trip from Oklahoma and getting in a small fender bender in our hotel parking lot. The second show was much better. Tons of people came out, it was in an actual club (the first was in a stage built into a truck) and we sold a bunch of merch. All in all, not a bad trip.

We also spent the weekend with our manager and generally tried to hash out our plans. After being in the studio for a time we have some songs recorded but the mix is pretty terrible, which is frustrating. The short and skinny is that the folks working on the record could have done a better job. Our expectations might very well have been higher this time around, but at the same time, we still expected more engagement and effort in terms of doing things right. It felt like an up hill battle, which is unfortunate, since we did spend a lot of time and worked really hard. The result though is that we’ll need to get our tracks to get remixed.

This brings me to the actual title of this blog. Our friend Dave constantly tells us to just ask our fans for money and use tools like Kickstarter to raise funds for things like recording. I always agreed with him that it would work for some bands, but for whatever reason I didn’t think it would work for us. Lately, I think I’ve realized why. It all comes down the relationship bands have with their fans.

In my mind, I see our music a certain way. It is probably an ideal vision of super cool people pushing the boundaries of tone and ripping faces off live. The reality is though, I think we’re actually more of a rock band. The concepts of cool that I always try to instantiate in my visions are probably not really there. This isn’t to say we are “cool”, but rather we’re not the leather jacket, cigarette smoking, sun glasses wearing, modern Velvet Underground. Live we try to put on a show and there is intensity, but the more I think about who we are and what we really sound like, it feels like I’ve misunderstood something.

This disconnect between my vision of our music, our aesthetic reality, is what has always dissuaded me from wanting to ask fans for money and be more communicative. In my vision, we’re an enigma and a mystery. The reality everyone says we’re really nice people. We shred on stage and off stage you can come up to us and we’ll smile and talk to you. In short, asking for money from fans and providing a more open relationship with them seems totally natural in reality.

The reason this has begun to feel natural is because fans understand music is entertainment. I don’t believe anyone thinks we are simply acting on stage, but rather they realize that in addition to getting on stage and playing music, we also drive around in our van, eat cheap food and generally struggle like anyone else. The 40 minutes on stage is the show and the rest is real life. Up until this point, I don’t think we’ve connected with our fans in real life and the result is that we haven’t made clear our needs, and more importantly, given them a chance to help.

Moving forward, I’m not sure what will change today, but something definitely should. There are some obvious things we can do to shed a little light on what our day to day looks like. There will always be some mystery, but we don’t really have much to hide. One thing that has become exceptionally clear is that we can’t try to be something we’re not and now that I think I have a more realistic view of what we really are, avoiding the things we’re not should be much easier.