Ionrock Dot Org

My Weblog

Slowing Succumbing to TDD

You always hear things about great athletes just doing the right thing. Golfers couldn’t really tell you how they manage to sync up their entire body in order to slap a ball down the fairway. They’ve just practiced like crazy and let their body do the execution. This kind of unknown benefit is something I’ve seen others bring up regarding TDD. At this point, I’m apt to believe them.

Writing tests make you feel good. Tests help improve your confidence because there is something to point to that says, “see, this works”. What is interesting though is that the more you test, the more you depend on your tests. It no longer a side story for the code, it is what makes the code valid. I made this transition recently and if I hadn’t noticed it on a personal level and acknowledged it, I’d continue to have a slim argument for TDD.

One facet that is key to TDD is that you write tests first. Up until this point, my assumption was that it was due to the benefits of taking the time. You preload your app with a client library that you need to make work. You have to think about the design from the user’s perspective, which is a Good Thing. The thing is, once you get started writing more tests and you’ve reached that point where the tests are what validate the code, it becomes apparent it is easier to change the tests before changing the code.

Changing the tests means that you not only have something to fix instead of something to write. This is a subtle constraint that helps to improve focus. It also keeps you designing when you may very well be doing maintenance. Ironically, it is also a little faster. TDD must be really slow is a really popular argument, but I’m inclined to suggest it is wrong. It actually should make you much faster.

Writing code quickly requires complete ideas. If you know exactly what the library does and what you want to do with it along with a rough idea of the algorithm, code can really fly into the editor. Writing tests first almost forces complete ideas. Since it forces design as well, you often don’t need to rewrite or refactor as much code. With practice, the speed only increases.

I’m really glad that I’ve taken the time to find an interest in testing. It is a hassle, but like exercise, you start to need it. Also, like exercise, it really improves the entire experience. It is a lot of fun to write code really fast. You feel confident and a feeling of accomplishment. Sooner or later you’ll find yourself churning out tons of great code and people might wonder how you got to be such a great coder. When they ask you can have a real answer beyond some buzz words.

Getting Ready for SXSW

Once again SXSW is coming up an we’re getting ready. We’re in full on practice mode focusing on getting our songs extremely solid. In addition to playing, we managed to get ourselves roped into helping out with a day show. This was a first. The first step was getting sponsors. That managed to work itself out at the eleventh hour. Our next task is to put together a backline for the bands as many are from out of town and won’t be touring down.

Outside of the obvious logistics and preparation, we’re getting ready mentally. It is funny because last year we really had high expectations. We didn’t really think that we’d be walking out with mountains of record deals or massive hype, but it did feel as though some opportunities might come to fruition. This year, we had hoped to have solidified our plans for immediately after southby (as we say in the ATX). There was an assumption that we’d know what we were doing next in terms of a full length, which meant that SXSW was not a critical event for us. Life is never what you expect and things didn’t happen as we expected. The result is southby is once again an important event for us.

There is one thing in life that seem to be true. When you’re planning an expecting great things to happen, they probably won’t. The best example I have of this is how I met my wife. I was in high school and gotten sick of wanting to be in a heavy relationship. The truth is I hadn’t been some sort of serious relationship or anything close, but in my own childish high school way, my mind was made up to stop trying to find a soul mate and just get to know people. Not a week or so after making this internal decision do I meet my wife and the rest is history! I’ve been happily married for 10 years, have had a blast and learned more about life than I could ever imagine.

I think the lesson to take away when preparing for something important like SXSW is that it is really just a few days and it doesn’t define what or who you are. Believe it or not great opportunities happen all the time. It is not realistic to be maxed out all the time in terms of readiness, so be realistic. This is the cliche “it’s a marathon, not a sprint” idea. The thing about cliches is that they often are painfully true, even though you don’t really want them to be. This year for SXSW we’re expecting to play some shows and have a good time. Things have already been more exciting, so simply trying to be ready is the mindset going forward. I have a feeling it will all work out in the end.

Responsible Musicians

Last night I had an epiphany. I was reading an article on Courtney Love in the latest Spin and it occurred to me that there is the not the same sort of “trash the hotel room” mystique bands used to have. I’m sure plenty of bands are still living hard and doing their share of trashing rooms, but I’d say generally bands are more responsible, at least in terms of their image.

What changed was the web. The web opened the door to real time updates for everything and that includes the day to day lives of musicians. Back in the day (whenever that actually was), a band could make a stink and it would last for a year or more. It became a oral legend passed around between friends and most likely blown slightly out of proportion. Now, when an artist trashes a hotel room, there was probably a cleaning person at the hotel that got pictures on their cell phone and can tweet that they really just left out their uneaten dinner on the table.

In some ways it is too bad we’ve lost the mystique. The silent tortured artist is gone forever. Fans know it is all an act and they don’t really care. The fact the fans have changed is what interests me.

When I was a kid there was always a concern about selling out. As I’ve gotten older and had more responsibility, it became clear that selling out was extremely relative. Unless we bring in money, we can’t make as much music. Something has to be sold. As a kid, if I heard a band on a commercial or even on the radio, that band had sold out, period. Now, no one cares. Part of me is glad since we could use the money and most of the time licensing music is a great way to bring in income. At the same time, I wonder why kids don’t care anymore that the musicians they love are so easily selling their music to commercial entities.

My theory is that fans are smarter now. In addition to having constant up to the minute updates on their favorite artists, they also have a constant stream of music industry content. It is difficult not to see articles and blogs discussing the state of the music industry and what is happening. When an artist says to their fans they don’t make any money, fans believe them and give them money. They may still pirate the music, but then they make sure to buy a t-shirt and donate on Kickstarter. Others have pointed out the importance of super fans, but I think it is quickly becoming the norm. Fans see the real struggles of musicians such that accepting money is not selling out, but simply part of the job. A band can be respected for touring like crazy, putting out a ton of music, and creating relationships with fans. Selling out is no longer a function money, but of time.

If you can’t tell, this is really exciting. The prospects of doing everything yourself never looked so good. Fans on a large scale are realizing they need to support artists and that doesn’t mean buying a record. Support is a different from the economics of selling records. It costs more for all the parties involve, but it pays more dividends. In the DIY punk scene bands used to tour based on letters they received from kids who said they could put a show on. Now, instead of the 50 kids putting on shows, you have a website with a forum, twitter and a blog to talk to thousands of people and create similar relationships.

None of this is easy, but it really never was. I’m really excited because it is apparent that there is a new generation approaching that will appreciate integrity. Not only that, integrity won’t be defined by a lack of money, but relationships. Ian MacKaye always did this for me. I saw him play in The Evens and bought a record afterward to support him. He said thank you in the most honest of ways and shook my hand, staring me straight in the eye and smiled. That guy has and probably always will define punk for me. It is about community and relationships. It is about working hard and having control over your life. There is a new era of punk as an ideal on the horizon.

Someone Please Get Rid of Paper

Printing has never been a large part of my life with computers. I remember when I was Novell and we needed to print something in color, there was puzzling amount of communication regarding how to get it done that usually ended up in asking a mac user to print it. Sometimes eating your own dog food is painful.

Recently at work I’ve had to turn in some hard copy materials. The first step was to try and scan them. No dice there. Our printer / scanner wasn’t scanning (with OS X no less!), so I had to find another way. My next option was sending a fax. Yet another troublesome practice that forces me to go to Kinkos and spend money on a silly expense. Did I mention that these hard copy materials are partially an expense report? One might suggest that I simply “expense” the fax. The irony here is that I’d have to send another fax with the fax receipt, but hold for a second. That would be another fax!!! A truly vicious and far from worth it cycle.

I hope that someday we won’t need so much paper. I’m totally fine dealing with real deal hard copy for things like contracts or other documents that are valuable outside of the digital realm. But for things like receipts and expense reports it just seems silly. Credit card companies should let you pick a set of charges and sent a link to a certified statement. They would be vouching that the expenses are real and the SSL cert makes it verified. Expense reports could be a thing of the past if there was a good way to verify and agree on digital documents.

There has always been the idea of the digital signature floating around. It never seems to catch on though. It might be a chicken/egg kind of problem where there aren’t enough users to validate a service, yet no one wants to jump a service that is not really trusted. Some might argue the government should step in but they shouldn’t. Paper will continue to stick around until we can find a way to put our mark on digital documents that we can all agree on.

I think the real reason there hasn’t been a digital signature success story is no one knows how to do it the right way. If I knew what the right way was, rest assured I wouldn’t be blogging about it right now. No, I would be looking for funding and start my search for the perfect island to buy. My guess is that the real answer is going to be a simple fiat system built around existing web services. Something like a seal that folks like PayPal, eBay, Amazon and the credit industry agree on for no other reason than to set the standard. On a technical level, there won’t be much past some libraries that do whatever conversation with web entities. OAuth is probably good example of transferring trust from one entity to another.

My hope is that we do start looking for ways to phase out our use of paper. Eliminating it is not a good idea because bits are too easy to flip. But if we can migrate some everyday requirements to something that digital it would be helpful. It is pretty shameful that we still use so much paper and that it is made from trees. I had a teacher as a kid mention that part of the reason marijuana is illegal is because the logging industry sees a major threat in paper produced via marijuana plants. This argument seems fishy since we have hemp, but in either case if there is a renewable process to get paper, it seems like a good idea to go ahead and use it instead of using trees.

No matter what happens with our digital signatures and paper usage, I’ll most likely hear I should be a doctor thanks to messiness of my signature (digital or otherwise).

Music vs. Programming

Last night I realized the biggest difference between programming and writing music. It is pretty obvious when you think about it. The difference is the audience. Both instances you are trying to communicate something. It is never easy and rarely does it work perfectly the first time. There is a ton of practice involved alongside a whole set of tools. Both have artistic aspects that always seem to be based on some mathematical concepts. But biggest difference is really the audience.

On the programming side of things you have an emotionless set of components that epitomizes stupidity. Sure, you can make a computer seem smart, but really that hunk of silicon and connections is about as dumb as it gets. It is either on or off. As a programmer, your goal is to make that big hunk of simplicity do something interesting and/or useful. It is a huge challenge because you have to use language that is going to be read by other programmers. This means there is a secondary audience (think other folks in the band) who need to understand what the heck is going on. The biggest problem on all fronts is communication thanks to the horrible medium of 1s and 0s.

Music on the other hand is another extreme when it comes to audiences. The listener is the person you write for. Your goal is not to tell them what to do, but rather to relate to their emotions. In a way, you’re communicating how to actually feel! Likewise, you have a set of musicians or a band that you need to communicate how to communicate to the listener. If you’ve ever been in a studio where they are trying to get sounds you’ll quickly see the deconstruction of genre devolve into hand gestures accompanied by dancing and random vocal noises. Again, the problem is the medium of emotion that makes music such a horrible communication method.

Programming and music are at two edges of the spectrum of communication. When you are writing a song what you really are doing is communicating. Sometimes that communication is focused on those around you and other times it is meant for the masses. Likewise, when you program you write for millions of x86 processors as well as the other developers on your team. In both of these cases the language you are given provides a huge challenge.

Working in a team can be challenging enough, but it becomes even more difficult when the output is a really hard communication medium. What’s is interesting is that you do see similar arguments even though the medium of music and code are so different. One would assume that in the coding world data and facts would reign supreme, but programmers can often become emotional about implementation details. Likewise, in a band there are tons of times where emotions run high, but the vast majority of the time it’s simple repetition and counting parts or measures.

The thing to take away though is that at the core is communication. If you are not communicating something to the audience, whether that is your laptop or a few hundred fans, you are not really doing anything. Effective communication is what spawns action. In fact action is really one of the best ways to communicate and is what leading by example really means. When you consider what you do in terms of communication I think it helps to gain a valuable perspective on what is really important. The focus shifts to others and that is always a good.

Code Virtues

I read this article about 7 virtues of good code. A part of me agrees, especially because the suggestions seem very similar to the zen of python. At the same time, the virtues are a bit unclear in terms of what the real goals are.

The first two feel like no brainers to me. The first is that code should work. This is true on both a practical and meta counts. On a practical note, broken code breaks builds. So, yes, working code is important. The second is unique as opposed to duplicated. Avoiding copy/paste is another good thing. Writing small functions to avoid side effects and the like is another +1 in my book.

The third is simplicity. I think this is generally a good thing, but one must be conscious of the fact that constantly moving code out can have some side effects as well. In languages like Java or C# you get an editor that lets you go to the class or method definition. In this case you have almost nothing constraining you from finding a piece of code. In Python though, the ability to immediately know where code is coming from drops considerably. This is especially true when you are talking about passing around functions/callables or objects that implement core types. Duck typing can really bite you in the end, so while moving code around to keep things simple is often a good idea, you really need to be aware that context is important. Moving code can take it out of context, which has dangers of its own.

The fourth is “Clear, as opposed to puzzling”. I’m going to call bs on this one. It is definitely true good code is clear. How you communicate what code is doing is a whole other story. The article points out naming variables and classes is the main aspect of clarity. Wrong! Just because you name some class effectively or have a relatively easy to read bit of code, you’re not done. Here is an example:

class GettextParer(object):
    def __init__(self...):

    def parse(self, ...):

# and later

gt = GettextParser()
catalogs = gt.parse(some_gettext_file)

organized_messages = dict([
    (catalog, [msg for msg in msgs if m != ''])
    for catalog, msgs in catalogs
])

The thing that is fishy about the code is the why and how. First off how do you know what sort of data structure you’re going to get back from the GettextParser. How do you even know it returns something? Secondly, why are you stripping out the empties in the msgs list? Hopefully some context would help provide the answers, but the fact is the naming doesn’t tell the whole story. The intersection of code, helpful comments and context all need to work together to gain the clarity that I’d describe as a virtue.

And as for comments being unnecessary, if you’ve never gotten the lyric wrong to a song you enjoy without ever looking, then I’d say you’re a prime candidate for not using comments.

The fifth virtue deals with how easy it is to update some bit of code. Personally, I believe this is one of the most important metrics for how well code has been written. This is what makes writing code really hard. The sixth virtue deals with code being developed and I think that ends up being a technique used for achieving ease. The better your internal APIs and models are, the better adding and removing code is going to be.

The last virtue is being brief. I’m all for brevity, but it really is a side effect. If your code is communicative, there is helpful context that gets supplemented by meaning comments, then chances are the code will be a reasonable size. Huge functions definitely smell, but at the same time helpful comments might make the context and meaning clear. A long but clear list is better than a black magic and poorly organized sets of methods.

Overall I think the virtues in the article are more or less agreeable, but the simplicity bothers me. I think most coders who deal with large code bases have their personal gripes, but they are probably just that, personal viewpoints on why something is wrong. The real issue is not so much the quality of the code. I would argue that should be measured by how effectively it succeeds solving the user’s problem. The real issue is communication. You need to communicate what the code does if you want good code. At the same time, if the code works perfectly or close to it, then it doesn’t really matter because no one needs to read it.

Adding “like” to Dingus

In making serious efforts to become a better tester I’ve tried to get better acquainted with testing tools. Mocking specifically is one technique that seems extremely helpful, especially when you have an application built upon many other systems that you’d want to test without having a huge test setup stage. The library I settled on was Dingus thanks to Gary’s screencasts and seeing first how how flexible it is in terms its ability to almost magically replace everything except that which is tested.

At the moment the Dingus docs are rather sparse, but the basic idea is you use a Dingus object and then assert different things happened to the object. This primarily done by using a method called “calls”. Here is an example:

from dingus import Dingus

d = Dingus()
d.foo('bar')

assert d.calls('foo', ('bar'))

Pretty easy so far.

Dingus can also be used to replace all the objects in a test using a DingusTestCase. So, for example, you can do something like this:

class TestSomeModel(DingusTestCase(MyModel)):

    def test_getting_a_thing(self):
        m = MyModel()
        m.get('foo')

        assert m.db.calls('query', (), {'name': 'foo'})

This makes sure the “db” object in the MyModel class calls its “query” function with the query argument “name” equal to “foo”. Again this is pretty simple and is really helpful because you verify that you are using an API correctly without having to spin up some external service.

This all works really well when you’re looking at mocking something one level deep, but often times it would be beneficial to go a little deeper. In my specific situation, I’m using a MongoDB connection, so I want to verify that specific things happened when I created the object and when I query something. To do this, I have to go a bit deeper with the calls. Here is an example:

from dragoman.testing import DingusTestCase
from dragoman import storage

class TestLanguageList(DingusTestCase(storage.LanguageList)):

    def test_get(self):
        result = storage.LanguageList.get('foo')
        db = storage.get_db.calls('()').one().return_value
        collection = db.calls('__getitem__', ('config.languages')).one().return_value
        assert collection.calls('find_one', ({'abbrev': 'foo'}))

My project is a gettext-like service and the test is for testing getting a language. You can see in the above code I have to go into the calls list and pull out the return value of the “get_db” function and continue to traverse the graph of calls to make sure the query was correct.

What I’d like to do is to describe what it should have looked like. Something like this:

def test_get(self):
    result = storage.LanguageList.get('foo')
    expected = Dingus()
    expected()['config.languages'].find({'abbrev': 'foo'}).count()
    assert expected.like(storage.get_db)

Instead of manually traversing I can just utilize a Dingus and compare it with the one that was called previously. This is really helpful for defining protocols of sorts. Often times applications have a set of business rules that go along with certain actions and it can be difficult to test that sort of thing. The idea here is that those sort of protocols can be defined clearly in the tests and there is a simple way to test them.

Here is another example. Say you have a User object that when saved needs to create or update a datastore. Here is what the test might look like:

class TestUser(DingusTestCase(User)):
   def test_create(self):
       u = User('eric')
       mock_db = Dingus()
       mock_db.find('user')
       mock_db.create('user_%s' % eric)
       mock_db.commit({'username': 'eric', 'type': 'user'})
       assert mock_db.like(u.db)

Based on the tests we verify that the create method will be using a “user_” prefix in its create method. If we assume the database is a document store like CouchDB, then we can glean from the tests that there is a “username” field that is needed for documents of that type. Likewise, it is clear that the protocol for adding a user involves checking if the user there, creating the user then committing the transaction.

Here is the implementation along with a few tests:

from dingus import Dingus

class LikeDingus(dingus.Dingus):
    def _flatten_calls(self, d):
        calls = d.calls()
        def flattener(cl):
            calls = []
            for call in cl:
                if call.return_value:
                    calls.append((call.name,
                                  call.args,
                                  call.kwargs,
                                  flattener(call.return_value.calls())))
                else:
                    calls.append((call.name, call.args, call.kwarg))
            return calls
        return flattener(calls)

    def like(self, other_dingus):
        return self._flatten_calls(self) == self._flatten_calls(other_dingus)

## tests

class TestDingusLikeStmt(object):

    def test_singular(self):
        tested = LikeDingus()
        tested('foo')['bar'].find({'name': 'baz'})

        expected = Dingus()
        expected('foo')['bar'].find({'name': 'baz'})

        assert tested.like(expected)

    def test_multiple(self):
        tested = LikeDingus()
        tested('foo')
        tested('bar')

        expected = Dingus()
        expected('foo')
        expected('bar')
        assert tested.like(expected)

    def test_the_order(self):
        tested = LikeDingus()
        tested('foo')
        tested('bar')

        expected = Dingus()
        expected('bar')
        expected('foo')

        assert not tested.like(expected)

Seeing as I’m still learning to be a more effective tester, I can’t say whether these kinds of assertions are extremely helpful or not. I do think that defining the expected assertions in terms of some other Dingus seems helpful in that you get to keep on mental model. Also I think the definition feels more natural than focusing entirely on the calls method of a Dingus.

What do you think?

SXSW 2010

This afternoon we played our last show of SXSW. It was quite a week. My arms are sore from lugging around gear and my feet are sore from roaming the sidewalks to venues and far off parking. Yet, it has easily been one of the best south by experiences yet.

This year I made a point to not sweat SXSW. It might be an underlying feeling of confidence or timing, but the results were very good. This is the first year with our brand spankin new manager in tow and that was really nice. He didn’t help load, but he did help make some good connections and I think get us on the right foot to start really establishing the business side of the band.

The shows were also much better this year. Almost every show felt like people actually came to see us. I say that with a bit of reservation but it really is the truth. There is an insane amount of competition. It is like a mini-internet in real life, full of ads and content all vying for your attention. As a band when you play shows, you usually hope that the band before or after you is someone people want to see because it is doubtful you’ll actually get many people out to see you unless you are really popular. No one sees the 11th result and the same is true at SXSW in terms of bands. This is why each show felt so amazing and why we felt very grateful. There were always people there, with most being reasonably crowded. It made us feel like we might be doing something right and that there were opportunities that haven’t shown up just yet. The future looks bright and I’m really happy about that.

Beyond playing a huge amount of time was spent lugging around equipment and parking the van. So much time in fact that eating became something of a luxury. A lack of real food along side a wealth of free beer and a desire to have something cool to drink meant feeling light headed a few times. After playing it is something of a rush, so it is easy to grab a beer and drink it down pretty fast and quickly look for another one. I’m not one to intentionally drink too much, but it sure can be easy to feel a bit tipsy when your last meal was a taco the night before and you realize it is 5 in the evening. I’m pretty sure I lost a pound or two from the mixture of physical work and avoiding nourishment. That said, I’m on a mission to score a sausage dog sometime tonight ;)

The best part of SXSW though is just hanging out. You see old friends and make new ones. You see famous people walking around and it doesn’t even matter because that is just part of the experience. Even though some may consider SXSW nothing but corporate slime making an effort to sway taste makers in their corner, the reality is people make the festival something worth going to. I’m already looking forward to next year!

A Future for Magazines

The other day while reading Spin, it occurred to me Pitchfork should have a magazine. It is not a matter money but rather a matter of brand and more importantly scaling out the perception of being a taste maker. Personally, I don’t read Pitchfork as bloggers suit me better, but that is the entire point. Different mediums provide different contexts for communication, which in turn speak to different audiences. When you are running a business that is based on advertising, eyeballs are important. Scaling out that business then is finding ways to get a larger audience.

I’ve always felt that social software often times simply makes an effort to fill different types of communication. Sometimes you need to describe things in great detail while other times a few words is enough. There are also photos, videos and sound that have their place among social communication. The social web is simply the translation of communication subtleties to real mechanisms on line.

Now, how this translates to Pitchfork is that unless they are able to transfer their branding to new markets via different kinds of communication mediums, they risk losing their status as the music taste maker. I say this because recently they did a redesign. Personally I liked it, but what became clear was that their web prescience had become more authoritative and respectable. The previous design was a little sloppier and edgy. It portrayed a message of something abrasive and different from the status quo. Now their brand looks like a catalog or newspaper. They provide the facts instead of the future.

Changing the design marks the beginning of the end for Pitchfork unless it continues to expand. Its initial medium of the web may simply have become more receptive to the newspaper like message. But, there are still those people who want the edgy and future based message. Pitchfork should aim to keep these information seekers close if they want to stay as taste makers. Likewise, if Pitchfork does want to expand into the more casual music listener’s world, something like a magazine might very well be a great way to expand that market and brand.

I have no idea whether it is even feasible to start a new music magazine in this day and age, but I think it is worth thinking about in terms of audiences. There are still people who find music through music magazines just like there are people who listen to the radio. There is innovation on the way that will help migrate those people to other mediums, which is why helping to push traffic to something like pitchfork.com via a magazine is a good move. The goal is not to have a successful magazine but to speak to those people stuck in the old medium in a way to help them move to the new.

Bootstrapping Deployments

Recently at work I’ve started working towards deploying Dragoman, my gettext service. The whole process has been rather frustrating because much of the low level bootstrapping needed to happen. It doesn’t bother me that a blank box needed configuration, but it was frustrating that there wasn’t a repeatable pattern to get things to a generally workable state.

What is necessary is some baseline level of functionality that fits within our organization’s system management tools. This is different from something application specific such as the specific runtime libraries. While there should be a baseline in terms of an environment, that should be minimal and standardized.

In our case we use Python, so for bootstrapping there are a few common items I’d personally like to see. These are in no particular order.

• Python 2.5 (this is just our standard version)
• Easy Install/Setuptools
• Egg Monster

Eggmonster is our deployment tool. It is a pull based system where nodes phone home to a central server that tells each node what package to run as well as how to satisfy its dependencies. It is all built around setuptools/distutils and works really well.

In terms of base utilities the biggest that I know of off hand is daemontools. This was a hassle to install because the distribution support was not there. I’m on the fence regarding what is best in terms of whether to build from scratch or use the distros package management tools, but in either case a decision needs to be made and stuck to. Building your own packages also seems like a decent option as you could then utilize your own server to provide tools reliably.

All this seems pretty simple, but time consuming. The packages are just the beginnings. There is also the file system and where tools should be used. There is also the question of what sort of security should be utilized so that developers can login and debug services. I don’t think the organizational aspects on the file system need to be anything more than decisions. The same goes with adding single sign on facilities. The biggest problem is just getting it done reliably such that you can spin up new machines easily as well as upgrade machines as the operating system improves. Hopefully I’ll get a minute to give it a try.

Threading Really Does Suck

For a large portion of my programming career, I’ve made an effort to avoid threads. The constant barrage of negative press and issues associated with the concepts behind threads made me nervous about their use and more importantly, my ability to effectively. As I’ve grown as a programmer my perspective on threads has changed. They seemed less magical and more practical. Discipline could make them viable options and in fact a really helpful tool.

While I believe threads have their value, I’m back to thinking they sort of suck. Recently trying to profile some code I realized that a whole host of functionality was never getting touched. This was because there was only one thread and one process. I’ve yet to actually get very far in writing a more extensive test to utilize more threads/processes (I need to stress the connections to the database), but I’m not looking forward to it.

The big issue is not so much idea of threads or processes, but just the lack of polish when it comes to working with them. Part of the problem is your truly. I’m always forgetting to do some bit of preparation or have screwed up some thread safety issue. But there are other times where what I’m trying really feels like it should work, but it doesn’t. All these issues are really my own responsibility, but I do wish someone could come along and make a really simple threading/process package that goes beyond the constructs and doesn’t do anything odd. I’m thinking of things like the different actor implementations or possibly something like Kamaelia without all the networking oddness. I’m sure something will happen in this space since it is so critical with the move to multiple cores and the need to scale horizontally. In the mean time, I’m going to keep trying to get better at threads even though they sort of suck.

Things to do When You’re Sick: Mess with Multiprocessing in Python

This morning I got my wisdom teeth out. It was full on surgery. I had my first IV and it was the first time to go under anesthesia. I was nervous at first, but honestly it wasn’t a big deal. In some senses, it was kind of fun to disappear for a minute and be awake with gause in your mouth the next.

Since I was essentially supposed to be out of commission the rest of the day, I took a sick day at work. I’m glad I did because from what my wife says, I should not have been working on production code! That said, I haven’t been able to sleep today so I went ahead and thought about doing a bit of coding.

In my recent mini-rant on threading, I alluded to creating simpler constructs for separating processing. I always have in my mind a system where specific actions can be run in another process or thread. More specifically, the only difference from this idea and normal threading is that you utilize the thread or process via a service like interface. It is just basic message passing, so not rocket science.

A quick glance at the multiprocessing module in Python revealed pipes. I had heard about them before, but had never used it. My task then was to create a simple web application that instead of handling the request inline, it would send it to a worker process to do it using a Pipe. So, here is the code:

import cherrypy

from multiprocessing import Process, Pipe

class BaseHandler(Process):
    def __init__(self, conn):
        Process.__init__(self)
        self.conn = conn

class HelloHandler(BaseHandler):
    def run(self):
        msg = self.conn.recv()
        while msg != 'close':
            self.conn.send('hello world from another process')

class HelloPage(object):
    def __init__(self):
        self.conn, child_conn = Pipe()
        self.hello_processor = HelloHandler(child_conn)
        self.hello_processor.start()

    def index(self, *args, **kw):
        self.conn.send(args)
        return self.conn.recv()
    index.exposed = True

def run():
    cherrypy.tree.mount(HelloPage(), '/')
    cherrypy.quickstart()

if __name__ == '__main__':
    run()

Of course immediately after I do this I read in the docs that it is easy to corrupt the data passing through the pipe if the same endpoint is being used by more than one thread or process. My conclusion then is that, while this seems pretty slick, a Pipe is probably better used for a series of callbacks. This would be analogous to iteration through recursion where the final step of the function calls itself with the remaining items. In this case a process might create another child to return and send the rest of the data to process things that way.

It should be noted that I’m taking pain killers, so this might be a little off.

What I also found was that a Queue was going to be safe to use with multiple threads/processes. This makes sense since a queue sort of includes the idea of single sequence since it can’t pop an item off twice. My next step in this experiment is going to be trying to make a threadsafe and process safe wrapper around a traditionally non-thread/process safe store such as bsddb, buzhug or sqlite. The queue will let me handle things in an order and I think I’ll be able to push a pipe on the queue in order to return the value to the correct parent.

I’m also going to reserve the right to delete this post later if I look back an realize that I was more messed up than I felt!

Why Emacs (or Vim)

I just read this article on why use Emacs. Of course, it got me thinking about why I use Emacs.

The whole Emacs vs. Vim debate is really a difference in preference. Part of why I use Emacs is because you never need to leave it. But this is really a preference. Vimmers probably like vim because it doesn’t get in the way of their shell. Again, a totally valid opinion. In both cases though there is a consistent line of thinking that deals with how you solve problems. Both Emacs and Vim are hammers and when you have a hammer everything looks like a nail.

Traditionally this idea of forcing problems into one paradigm is considered a bad idea. The problem is that for programmers you edit code. That is the most important thing to consider. You have to write/edit the code, which is in many languages, organize the files and compile and/or run it. Both Emacs and Vim are tools for the most common case.

This is why I like Emacs. It lets me cram things like email, irc, using a shell and a host of other activities into the paradigm of editing. The result is that I get better at editing text while I browsing the web, looking for files or chatting on irc. Usually this kind of problem solving is a bad idea, but in terms of Emacs, it always pushes you to learn more and makes you a better programmer. I’d argue Vim’s ability to keep you on the shell is a similar situation. You get really good at working in your shell every time you edit a file.

The power that Emacs (and Vim) affords is what makes it such a great editor. In almost all cases, taking a single tool to solve problems is a bad idea, but for Emacs, it makes it an exercise to become a better programmer. That is why I use Emacs.