Ionrock Dot Org

My Weblog

Not Writing

My writing has dropped off dramatically lately. This blog is not exactly the brainchild of an extremely prolific writer, but it also isn’t a stagnant pool of text bit rotting on the web. So what has changed?

The first big change has been some of my responsibilities at work. The project that was lead by my previous partner in crime has become more my responsibility. This isn’t really a promotion, but rather a shifting of guidance and responsibility. Even though it has not been a huge change, it is still change, which is rarely easy. Things have begun to pick up as we’ve gotten a bit of momentum, but it is still an adjustment for me.

The other big change has been less stark. There is some new music coming down the pipe, but it is not without its share of blood, sweat and tears. If anyone says recording a record is easy, they have never done it. And if by chance they have done it, then they weren’t successful at it. If they were successful at making a record and still claim it is easy then I don’t know what to tell you other than they are really lucky. This has been some of the toughest music to write and record. We’ve been playing for a really long time and while in sense our chemistry is pretty well solidified, it also feels somewhat too familiar at times. That feeling isn’t too much of a detractor, except, when after you finish an exceptionally painful recording process the only feed back suggests you have a lot more work to do and no money to do it with. The details aren’t terribly important, but it has been a really tough couple months.

The good side is that we have some tours coming up and, as I said before, we’ve gotten a bit of momentum at work. The goal is that some of that momentum can be directed towards writing. Whether or not it shows up on my blog is another question all together, but I’m sure that there will be some more reflections and discoveries I’d like to share sooner rather than later.

Sending Email with Wanderlust

At work we have a bunch of email addresses that get sent to a large set of users. Since I started, I've always gotten my mail forwarded to my personal email account, which works really well except for these mailing lists that complain that the mail being sent is not coming from the real email address. Seeing as I use Wanderlust and Emacs for my email, I knew there had to be a way to configure things to use my work smtp server for work emails and gmail for the rest. The key was looking at the the 'wl-draft-config-alist' setting. This is a simple list of key/values that get applied when a header or part of the body get matched. In my case, I wanted to match on the From field and send it via the correct smtp server.

Along side the 'wl-draft-config-alist' is the 'wl-templates-alist', which helps to setup a draft with appropriate headers. I should mention that I tried a ton of different settings to get this to work. I'm sure I could have put all this in my .wl file and that would have been enough, but alas that wasn't the case.

The first step is to go ahead and make a couple templates for writing mail. I created a default and an work template that maily just set the 'From' header to the appropriate value. After that I added some code to my 'wl-draft-config-alist' for each account. The the templates I added via customize in Emacs, while the 'wl-draft-config-alist' I was able to add to my .wl file. The latter seemed to fail when adding variables that needed to be changed. Here is an example of what I added:

(setq wl-draft-config-alist '( ("^From: .*eric@work.com" (wl-smtp-authenticate-type . "login") (wl-smtp-posting-user . "eric") (wl-smtp-posting-server . "webmail.work.com") (wl-local-domain . "work.com") (wl-from . "Eric Larson ") (wl-reply-to . "Eric Larson ") ) ("^From: .*eric@personal.org" (wl-smtp-authenticate-type . "plain") (wl-smtp-posting-user . "eric@personal.org") (wl-smtp-posting-server . "smtp.gmail.com") (wl-local-domain . "gmail.com") (wl-from . "Eric Larson ") (wl-reply-to . "Eric Larson ") ) ))

Band Relationships

We just got back from playing Denver for The UMS. Our first show was pretty crappy and honestly we didn’t play very well. It ended up being OK as people showed up, but for whatever reasons, the stage manager and people running it were non-existent. Not a huge deal, but it made for a rough start after a long trip from Oklahoma and getting in a small fender bender in our hotel parking lot. The second show was much better. Tons of people came out, it was in an actual club (the first was in a stage built into a truck) and we sold a bunch of merch. All in all, not a bad trip.

We also spent the weekend with our manager and generally tried to hash out our plans. After being in the studio for a time we have some songs recorded but the mix is pretty terrible, which is frustrating. The short and skinny is that the folks working on the record could have done a better job. Our expectations might very well have been higher this time around, but at the same time, we still expected more engagement and effort in terms of doing things right. It felt like an up hill battle, which is unfortunate, since we did spend a lot of time and worked really hard. The result though is that we’ll need to get our tracks to get remixed.

This brings me to the actual title of this blog. Our friend Dave constantly tells us to just ask our fans for money and use tools like Kickstarter to raise funds for things like recording. I always agreed with him that it would work for some bands, but for whatever reason I didn’t think it would work for us. Lately, I think I’ve realized why. It all comes down the relationship bands have with their fans.

In my mind, I see our music a certain way. It is probably an ideal vision of super cool people pushing the boundaries of tone and ripping faces off live. The reality is though, I think we’re actually more of a rock band. The concepts of cool that I always try to instantiate in my visions are probably not really there. This isn’t to say we are “cool”, but rather we’re not the leather jacket, cigarette smoking, sun glasses wearing, modern Velvet Underground. Live we try to put on a show and there is intensity, but the more I think about who we are and what we really sound like, it feels like I’ve misunderstood something.

This disconnect between my vision of our music, our aesthetic reality, is what has always dissuaded me from wanting to ask fans for money and be more communicative. In my vision, we’re an enigma and a mystery. The reality everyone says we’re really nice people. We shred on stage and off stage you can come up to us and we’ll smile and talk to you. In short, asking for money from fans and providing a more open relationship with them seems totally natural in reality.

The reason this has begun to feel natural is because fans understand music is entertainment. I don’t believe anyone thinks we are simply acting on stage, but rather they realize that in addition to getting on stage and playing music, we also drive around in our van, eat cheap food and generally struggle like anyone else. The 40 minutes on stage is the show and the rest is real life. Up until this point, I don’t think we’ve connected with our fans in real life and the result is that we haven’t made clear our needs, and more importantly, given them a chance to help.

Moving forward, I’m not sure what will change today, but something definitely should. There are some obvious things we can do to shed a little light on what our day to day looks like. There will always be some mystery, but we don’t really have much to hide. One thing that has become exceptionally clear is that we can’t try to be something we’re not and now that I think I have a more realistic view of what we really are, avoiding the things we’re not should be much easier.

Concurrency and Focus

Last night as we were driving back towards the great state of Texas, we listened to Coast to Coast. If you’ve never listened, it is a radio show that traditionally focuses on non-mainstream topics that can often stretch well into the weird. It is a fun show to listen to for freak factor as well as the interesting perspectives. The guest in this case was the author of the book The Shallows, Nicholas Carr. The book argues that the Internet has changed the way people think and robbed people of focus, and personally, I have to agree.

One interesting thing he talked about was the brain’s concept of memory. He described the basic ideas of RAM vs. CPU Cache vs. the hard drive. The idea was that we’ve gotten faster RAM thanks to the web, but in doing so, we’ve lost the ability to save ideas and knowledge to our long term memory storage. One argument he disagreed with was that the web offers you a place to store information so you can think about other things. I partially disagreed with this because as a programmer, much of my job is finding interfaces and layers to help manage complexity, effectively relieving my mind of the burdens associated with lower level problems.

This quest to hide complexity has been a major theme in computing. Back when computers were actual people, the idea was that a scientist could let the computer do the low level work while they continued to hone their larger level algorithms and models. The ability to abstract and classify is critical to moving beyond what you can do by yourself. There are so many details in the world that without tools to manage them, life would be unbearable.

Concurrency is a very similar situation. I read this article discussing how trends in concurrency often do not fairly characterize the problem. This got me thinking about the issues with focus. If we consider concurrency in terms of workers that all equally do things, you miss the obvious abstractions you get from more specific tools. Like learning to focus, the goal is not to store information as much it is to manage the transitions and interruptions. It is often simpler to have a master process that handles specifically who does what than relying on a generic solution to magically do everything correctly. Just like the scientists in the early days of computing, we should think of concurrency in terms of how we can create abstractions while freeing up resources in a way that doesn’t force us to pass everything through to a intermediary. There were probably plenty of computations that early programmers did themselves. But at some point, there was a decision made on a set of parameters that suggested the solution be sent to a computer to do the work. It is this kind of mindset that will make concurrency in the future relevant in the future and reasonable to implement.

Why Choose MongoDB

A co-worker will occasionally send along recent blog posts that place into question our decision to use MongoDB. These are small reminders that technical decisions are never black and white and that no matter how careful you are, there are going to be trade offs. Usually the complaints I hear about MongoDB have to do with durability. Specifically, people have a problem with 10gen (the folks who make MongoDB) suggesting that you run at least two MongoDB instances to provide some level of reliability in the face of one system going down.

The fact is we have been bitten by a bug that required a good deal of downtime. This kind of thing totally stinks. I’d say it is unacceptable, but the reality is that at some point you have to trust that a system will do its job. In other words, it is not enough to prevent problems, you must also have a means of recovering from problems. In our case, that recovery plan was the problem. Shame on us.

This has since been remedied and there is probably more to do. I’m sure other organizations might have considered the hiccup a worthy reason to abandon MongoDB for another more durable system. The problem is that while MongoDB’s durability did prove to be a major problem, that isn’t the whole story. There are other reasons to choose MongoDB, but durability is not one of them.

In our situation, the biggest benefit we get from MongoDB is the ability to query. This seems like such a basic tenant of databases that you’d be hard pressed to find someone who didn’t believe running queries on your data isn’t the killer feature of a database. With that in mind, I’d point out that many NoSQL systems do not provide the same level of support for queries. CouchDB (which I’m personally a fan of) is in fact really bad at ad-hoc queries. Once you save a view in CouchDB, you’re in pretty good shape, but randomly querying the DB is considered bad form. Likewise, generating a new view on a massive dataset is time consuming as well. That is the trade off CouchDB made. In return you get excellent durability, a fantastic HTTP based interface and the ability to utilize pure JSON as a document format. Totally reasonable trade offs if you ask me. The only problem is, the ability to query the dataset in real time is an order of magnitude more important to us than the other features. We can run a slave (for reads even), deal with BSON, and use the socket based Python driver that MongoDB provides, yet if we couldn’t do realtime queries in a reasonable about of time, then what is the point?

By the way, I don’t mean to pick on CouchDB or any other system. The fact of the matter the discussion is not over what system to use as much as weighting what features are important. For us, the ability to query very quickly and under load trumps the other features such as single machine durability. We’ve paid the price of that trade off once, so we’re aiming not to do so again. Thanks to my cautious and objective thinking co-worker, we also consistently consider other data stores. That said, I think we made a good choice to go with MongoDB. It is not perfect, but nothing is. MongoDB has been meeting needs very successfully and I have no problem recommending it.

Mobile is the Future

I read a great article the other day by someone who used to work at Nokia on how the US is so radically different than the rest of the world when it comes to cellular technology. It was really interesting to see how Nokia as a company was driven by research in its own local market, which happened to be a great predictor of how the rest of the world uses mobile phones. What was also really interesting was the impact of SMS on the mobile industry, and more specifically, for carriers in terms of profits. These profits (along with a wildly different geography) also seemed to help with providing better service to carrier customers. For example, no one wanted antenna on their phones because they got in the way of putting it in your pocket. In the US though, an antenna meant better reception and fewer dropped calls. That never happened, so users just assumed the phone always worked.

Articles like this are really interesting as programmer because reinforces the importance gathering data. Nokia’s reel claim to fame in the story is its research. It payed closed attention to its users and acted accordingly. As a result, Nokia sell millions of phones at a really high price all over the world and have customer loyalty. While it is something I’m not very good at, it is a good reminder that being able to gather data can make the difference between tweaking a setting and rewriting an application. A wrong hypothesis can be expensive, and data helps to make better guesses as to what the problem is. It is something I personally want to get better at.

The other theme of the article is the importance of mobile in terms of technology. The author described impact of SMS on the industry, which was enormous. Any company that has a technology focus needs to see the importance of mobile phones as a platform. Also, the platform is not the iPhone. The author made it clear that the iPhone application business model doesn’t work, which is something I have felt for a while now. The numbers just aren’t there. You sell the app for a couple bucks to a million people and you might have made a couple million dollars. But that’s it. There is nothing else to really be done except releasing updates and hope people keep buying it. The problem is that if it gets extremely popular, there is a really good chance Apple will just release its own version and put you out of business. The other tact is to release system services for other companies to use, but again this seems like a bad idea. You’re selling some service to the guy that only make a few million dollars off some application. Some percentage of those sales is pretty weak in terms of revenue and you still have the same problem of Apple simply destroying your business by releasing their own version. Apps just don’t make sense in the long term and doesn’t reflect the rest of the world.

The reason for all this is that defaults rule. SMS changed everything because it was a built-in feature of every phone. There are tons of phones now with data plans, but the reality is that everyone texts. This is not because texts are better than Safari or Twitter, but because interoperability is taken care of. The web flourished because browsers felt the same. The mobile industry gets the same feeling from SMS. The article states that every day users won’t buy apps and after thinking about it a bit, I have to agree. The one thing that could change that would be the iPad. The iPad has the opportunity to change the way people think of computers and it is built completely on the idea of apps. Before you had a filesystem and there was a disconnect between files and applications. The iPad is destroying that assumption and people seem to appreciate the simplicity. If the new assumption is that applications are the baseline and it is a pattern that naturally works on a phone (which it arguably has), it seems possible that the idea of the app store could become widespread. A person would have their iPad as their computer and their iPhone as their laptop with both synced via the cloud and apps. I don’t know that it will work like that but it could.

The point is that in either case, mobile computing is only getting more important. One interesting side effect is that mobile platforms are only now considering how to handle multiprocessing. This is not even how to mimic threading, but rather how to do two things at once. It is getting solved quickly, but it is interesting because developers have been focused on how to handle the problem of writing apps for more than one processor. It appears it doesn’t matter because while desktops and laptops are getting more cores, people might just stop using them in favor of a different kind of machine. It still means the web and servers are still very important, but we can probably keep punting needing things like functional paradigms in order to handle the multiple processors all our users supposedly will be utilizing in the future.

MongoUI – Barely Better Than a Terminal

At work we use MongoDB for storing most of our critical data. We started with customized key value store we called Faststore. It was built in Python and used BsdDB. It was a pretty successful system in that it was reliable and performed very well. The thing that killed Faststore was that there wasn’t a good way to replicate the data. We tried considering a different key/value database in the core such as Tokyo Cabinet, but it quickly became clear the speed wasn’t there for our critical operations. After some research, we settled on MongoDB for our storage needs. So, far the decision has been a rather successful one. We did hit a rather nasty bug that required some significant downtime, but beyond that, MongoDB has been excellent. This is especially true when you consider the amount of data.

As we’ve used MongoDB one theme that I’ve hit is that is can be a little hard to view the data. Things like CSVs and tables in terminals can be a little easier to read than a pretty printed dictionary. In order to help making the perusing of data a little bit easier when developing with MongoDB, I made a little tool called MongoUI. MongoUI is really simple. It is only a little better than a terminal. That said, it makes basic browsing a little easier.

Pretty much all you have to do to use it is download it (hg clone http://bitbucket.org/elarson/mongoui), install it (cd mongoui && python setup install) and then run it (mongoui). Once it is running, point your browser to localhost, port 9000 and you’ll see a simple list of your databases. Clicking on a DB will show you the collections and clicking on the collections gives you a couple fields to query. The first field is for excluding/including fields. The second is for providing parameters ({‘foo’: ‘bar’}). You can include a count by clicking the count checkbox and change the include to an exclude by clicking the exclude checkbox.

The output is still pretty much just a pretty printed dictionary, but the browser is a slightly easier place to scroll up and down than a terminal. There is no writing or editing of documents, and I seriously doubt I’d add that anytime soon. It also is not meant for huge databases. For example, if you have 20k collections in a DB, it will show a really big list. Same goes for large collections. I’m not paging or anything like that. It really is meant make basic queries on local tests databases a little bit easier.

If it seems helpful I’ll try to develop the viewing of the data bit more as well as possibly including some editing bits. That said, it is no where on my todo list. The other thing I’d like to try and incorporate is a good RESTful API. My biggest complaint about MongoDB is that the protocol to the DB is not via HTTP. It was a basic design decision, but it would be nice at times to have a basic HTTP enabled caching proxy in front of a MongoDB for things like saving queries as views. Likewise things like replication and load balancing might also be interesting uses. That said, I don’t really have a need for it at work, so I don’t know that I would get very far.

Feedback is always welcome!

MongoDB Schema

Unfortunately, I can’t remember where I read it, but there was a blog post that mentioned the importance of schema in a schemaless database. It hit me as somewhat close minded at the time. After all, they don’t call them “schemaless” for nothing. Slowly there has been a change in how I view the issue.

Contracts have been a theme recently in my programming life. Tests essentially create contracts. The environment you run in can be rather limited and yet more robust thanks to contracts with the underlying system. Contracts are what let you sleep at night because without some guarantees you’d never get anything done. The key with contracts is that they happen at the right time and place. Abstraction (and much of OO) basically is a way to take some level of complexity and hide it behind a contract. Before when you had to clean up a file name, verify it wasn’t written to the data store and replicate it to three other systems. Now, you call the “save” method and forget about the details. The benefit most people site is flexibility, but the reality is that the contract behind the “save” method lets you abstract the more complicated process down to one thing, letting you free of some mental space for new problems.

The idea then behind a schema for something like MongoDB is that you get to stop thinking in terms of JSON or Python dictionaries and begin to use real objects. The reason is the same as the save method example above. We need a contract. A schemaless database is advantageous because it moves the verification and contracts for the data to the application level where it belongs. The RDBMS can help by saying something is a string or int, but past that you’re working with really basic types. You’re going to need some larger concepts to keep everything in place. ORMs are one example. In the case of MongoDB, you still need a schema because you need the contract. The only difference is that there is a slightly more automatic translation between a strings, ints, lists, and dicts along side the fact MongoDB doesn’t require setting everything up front.

It is always interesting to find how some buzzword or concept really ends up being a slightly different perspective on system design. Most people (myself included) hear “schemaless” and think, “I can ditch my ORM! Declare nothing! Everything is just a dict!”. The reality is that in the case of MongoDB you have a database that can understand JSON. That is pretty much it. There are interesting outcomes to that reality, but no where does it say you never have to verify your data. That is what a schema does.

The Social Web is Boring

Has anyone else noticed that most discussions on social media have gotten a bit stale? Everyone is a social networking/media guru. There are folks that actually do understand social media but it’s not because it is social media they are understanding. The folks who understand the impact of branding and quality content are the ones who are continuing to be successful utilizing the social web. This is not really surprising as it parallels the SEO concept as well. I remember when people felt it revolutionary that instead of focusing on search terms you simply try to be a good writer and gather an audience. It shouldn’t be surprising that then that social media has taken a similar turn.

With the marketing angle and business model of social media exposed, what else could be interesting about it? Well we’ve seen the fall of MySpace and everyone thinks Facebook is up next. It is not a matter of why, but simply when. Social networks have an Achilles heel call size. The reason is because social networks have to provide the interface to people and make it evolve along side relationships. That is really, really hard. People change all the time and so does technology. The big shift with Twitter and Facebook was that the expectations of the user switched from being browser based to mobile enabled. It moved the social network out of the browser and into real life by allowing updates from mobile devices and in doing so, made MySpace, which really requires a browser, irrelevant. The problem is as each social network gets larger and larger, the technology changes and as such, the expectations of the users.

With that in mind, it still seems like social networking and social media are at least technically interesting. The question is how do you keep the massive user base while continuing to evolve the service alongside technology and user expectations? It is really simple, you don’t. In technology there is always a point where a system becomes too large for a host of reasons. What you do is refactor and reorganize the code to reflect different abstractions. Eventually you start breaking the system up into more manageable parts. That is why we now have things like Foursquare, even though Twitter seems to already do geo locations. People end up wanting one tool to do one thing really well. Where have I heard that before… Unix!

The fact is we’ve learned to scale and realized the potential in relationships. It is reset time. Computing started simple and boomed when every day people got involved. The original Windows made it possible for you to slap some clip art on a layout and send your grandparents a homemade card. It was communication and relationships evolving through a better user interface. The time has come once again for this kind of user interface shift and it has already happened.

Yup, I’m talking about the iPad. I was a skeptic, but I now see the light. It is going to change the world and that is that. Do I have one? Nope, not yet, but I do want one. The thing that tipped me off that I was wrong about the iPad before I even got one was some example music software. We are recording our latest music with a guy that doesn’t want to use computers. The result is more time consuming and in all honesty rather frustrating. But, I realize that with the iPad, a person gets to have a customized interface without the cruft of the operating system. The iPad has taken the single user iPhone interface, made it big and provided the tools for others to build interesting interfaces with it.

The timing is impeccable because as I’ve been saying only a few paragraphs earlier, social media and social networking is boring. The iPad is the new form factor to change everything. No more browsing the Internet at home, finding a link and sending it to your friend. Now, you hang out with your friend, remember you saw something cool, pull out your iPad and show them. There are still all the same network enabled capabilities but they are services no one really cares about. The real deal excitement is getting out from behind the desktop and experiencing relationships by utilizing a screen. Social media is boring because you are forced to have an awkward relationship through a computer screen.

Getting Started with JVM Languages is Hard

I watched this video on integrating Scala. Now, I’m not a Java developer. I used it in college and had to mess with small bits of it throughout the years, but generally, it is something I’ve avoided. Scala is a language built on the JVM that is (as I naively understand) well suited for concurrency thanks to it support of Actors and is relatively friendly to the functional style of programming. It also is statically typed, which is what initially got me interested in looking at it.

Seeing as I’m a web developer, my first search on a new language is always a quick review of the web frameworks. Web frameworks usually do a good job getting you up and running quickly, which is really beneficial for a non-Java developer like myself. Unfortunately, this is also usually where I end up losing steam in JVM based languages. The problem is Java and the JVM make it difficult to get started.

Part of this evaluation is pretty unfair. I have no doubt Scala and other JVM languages (Clojure is another I had a limited interest in) can be really excellent. The problem is you have to understand the JVM. Am I whining? Probably. Does it matter? Nope.

When learning a language friction is a killer. Understanding the programming constructs and ideals of the languages is usually least of your worries. It is when you have to find dependencies and simply get started you have a problem. The Java classpath is a great example of pure stopping power when learning a language. The concept makes sense. You create a path for searching for libraries and necessary files. It is just like the operating system’s PATH only for code. Got it. Unfortunately, it just doesn’t work that way. There is almost always a stack trace that reflects javac not being able to find some aspect of the program to make things work. When I start hitting these kinds of problems, my eyes glaze over, I start closing buffers in Emacs and start doing something else. It is a pain in the neck and it doesn’t make sense why it is so hard.

This pain also goes for things like Maven, Ant and all the surrounding Java ecosystem tools. Let’s take a look at getting started with Lift:

mvn archetype:generate -U \
  -DarchetypeGroupId=net.liftweb \
  -DarchetypeArtifactId=lift-archetype-blank \
  -DarchetypeVersion=1.0 \
  -DremoteRepositories=http://scala-tools.org/repo-releases \
  -DgroupId=demo.helloworld \
  -DartifactId=helloworld \
  -Dversion=1.0-SNAPSHOT

Now to a Java developer this might not be a big deal. This might even be elegant. To me, it is a huge blinking light that says I’ll be doing a ton of configuration, dealing with obtuse XML and generally wasting my time on the unimportant details. If that is how much I have to write in order to get a Hello World application running, then the serious complexity of a real world application is going to be a nightmare. Even if it is not, I’ve lost my will to find out. Game over.

The reason I mention it at all is that there seems to be tons of cool stuff happening on the JVM. Scala is one language I’d like to try out. Clojure is another I was really excited about getting my hands dirty with. Even Jython seems like an interesting tool to have in one’s programming tool belt. Yet, even though the concept of an interesting language on top of the production proven JVM is really promising, the reality is the interface is a nightmare. That fact is too bad because it means that most JVM based languages that don’t make an effort to hide the JVM-ism are somewhat limited in scope to Java friendly developers.

The one JVM based language that I’ve used without having much trouble getting started was Rhino. It was relatively trivial to get up and running and I never even thought about the classpath. At a minimum, that is what I’d hope to find from other JVM based languages, especially if their features are based more around the language than integrating with existing Java applications. For example, I understand that Scala is a language that seems pride itself on how easily it integrates in a Java application. My point is that it would be really helpful to be able to use JVM based languages without having to know I’m using a JVM based language.