Ionrock Dot Org

My Weblog

The Quest for One Language

It seems that one goal of frameworks is to allow a programmer to abstract things away such that one language can be used everywhere. ORMs are the best example of this in that they translate SQL to language constructs so a programmer doesn't need to make the paradigm shift to SQL's declarative model. Templating languages are another example where often times the selling point is to use your language of choice in the template. Generally, I've thought that these sort of single language abstractions are good, but have the potential to be dangerous. Google Web Toolkit is a good example of a system that seems to go too far abstracting away an entire programming environment (javascript). 

Still, the idea that one language can be used everywhere has its benefits. For example, if there is an expensive loop for the server, pushing it to the client could be trivial in the case of a single language. Also, the simple differences in basic looping techniques can take its cognitive toll. When you're accustomed to doing list comprehensions or blocks and then need to use something like each() (in jquery), it can take a little thought. I'm not going to act as though this transition is very difficult, but simply that the changes add up over time. This is especially true when you think about something as expansive as the Python standard library vs. jQuery, Prototype and Dojo. They meet different needs for different domains, but the fact remains there is a ton of code on both sides of the fence that might be very similar in intent. 

The only pragmatic decision then in finding a single language is Javascript. Lets face it, the browsers are not going to change anytime soon. Even ActionScript is effectively a Javascript with a more traditional object oriented model. This makes things like the recently released Joyent Smart Platform appealing. The now defunt AppJet was another example of a Javascript based server side framework that aimed to make similar strides in the cloud. They even released their platform so it could be run locally. 

These platforms seem pretty compelling when you consider the current technologies available to web developers. While the browser is still a painful platform at times, JSON is everywhere. If you choose Flash or Silverlight for your frontend, you can use Javascript (more or less). Likewise, there is an obvious relationship between client and server side Javascript. Also, configuring a server wouldn't be any more difficult, as using something like nginx as a proxy is simple and convenient. The biggest gotchas I see are persistence in traditional databases, even though I'm sure the runtime a server side Javascript run on top of provides more than enough well tested persistence capabilities. 

When it gets down to it, my bet is that the hesitence to use Javascript on the server side is a marketing issue. People still don't like Javascript. I know I'm sick of it. But, when I really consider what I'm sick of, it isn't Javascript as a language, it is the browser implementation. There is a certain amount of uncertainty that is always present for me when writing Javascript. Will this attribute be there when I use another browser? Will the DOM be written to at the same time when I test this in IE? If this goes slowly, am I going to find a way to speed it up reliably? My curiousity is that when I take these issues away, does Javascript become a more enjoyable platform? I'd imagine it might. 

Hopefully I can find some time to play around with it. At the moment, I'm really happy with Python and don't see many reasons to try other options. And yes, I've used Ruby and I prefer Python thank you very much. That in mind, I find most of my day ends up in Javascript one way or another, so I bet trying it server side might be a beneficial experience.

A Quick Javascript Server Side Framework

Recently, I've been thinking a lot about Javascript on the server. Last night I took a minute to see what a simple framework might look like. After going through the different frameworks available, it was clear there were a wide array of reasons people created server side Javascript. Very few are thinking in terms of traditional web frameworks at this point, so the available options are slim and rather high level. Fortunately, I found Jack, the WSGI/Rack clone for server side Javascript. After reading up a bit, I figured I'd see how fast I could put together a simple framework for writing basic apps. Here is what I came up with:

var Jack = require("jack"); var http_status = function (code, msg, body) {     var template = {         head : '<html><head><title>{MSG}</title><head>',         body : '<body><h3>{BODY}</h3></body></html>'     };     var redirects = {         300 : 'MultipleChoices',         301 : 'Moved Permanently',         302 : 'Found',         303 : 'See Other',         304 : 'Not Modified',         305 : 'Use Proxy',         // 306 : 'Unused (not implemented, obviously)',         307 : 'Temporary Redirect'     };     var messages = {         400 : 'Bad Request',         401 : 'Unauthorized',         402 : 'Payment Required',         403 : 'Forbidden',         404 : 'Not Found',         405 : 'Method Not Allowed',         406 : 'Not Acceptable',         407 : 'Proxy Authentication Required',         408 : 'Request Timeout',         409 : 'Confict',         410 : 'Gone',         411 : 'Length Required',         412 : 'Precondition Failed',         413 : 'Request Entity Too Large',         414 : 'Request URI Too Long',         415 : 'Unsupported Media Type',         416 : 'Request Range Not Satisfiable',         417 : 'Expectation Failed',         500 : 'Internal Server Error',         501 : 'Not Implemented',         502 : 'Bad Gateway',         503 : 'Service Unavailable',         504 : 'Gateway Timeout',         505 : 'Version Not Supported'     };     var res = new Jack.Response();     if (redirects[code]) {         res.redirect(msg, code);         return res;     }     var msg = msg ? msg : messages[code];     msg = msg ? msg : code;     body = body ? body : msg;     res.status = code;     res.write(template.head.replace('{MSG}', msg));     res.write(template.body.replace('{BODY}', body));     return res.finish(); }; var pp = function (o) {     for (k in o) {         print (k.toString() + ': ' + o[k]);     } }; var app = function (routes) {     return function (env) {         var req = new Jack.Request(env);         var meth = req.requestMethod();         var path_info = req.pathInfo();         for each(route in routes) {             var m = path_info.match(route.regex);             if ( m ) {                 if (route.methods[meth]) {                     env['app.matches'] = [p for each(p in m.slice(1)) if (p)];                     return route.methods[meth](env);                 }                 return http_status(405);             }         }         return http_status(404);     }; }; var route = function (regex, methods) {     return {         regex : regex,         methods : methods     }; }; exports.app = app([     route(/^\/(\d+)\/(\w+)(\/)?$/, {         GET: function (env) {             var url = env['app.matches'];             var res = new Jack.Response();             res.write('The id is ' + url[0] + ' and the name is ' + url[1]);             return res.finish();         }     }),     route(/\//, {         GET: function (env) {             var res = new Jack.Response();             res.write('hello world!');             return res.finish();         }     }) ]);

There is no templating or anything, but having the basic means of setting up some handlers based on the request URL and method seems like a good start. Hopefully someone else might find it interesting and/or helpful.

Groking Mercurial Queues

I've mentioned plenty of times that we use Mercurial at work. Well the other day Dowski mentioned that he started playing with Mercurial Queues (mq) and found it to be pretty helpful. As I'm always interested to see where these more advanced DVCS concepts lead, I gave it a try.

My basic use case is that I'm working on a feature and I have to change gears to work on something else. This usually involves pulling from the main repo, doing whatever new work I need to do and pushing it back before going back to where I left off. When using mq, the idea is that you are working on a set of patches locally that, when finished, you'll push or send to someone. The way these patches are organized is in a stack. Each patch gets a name and they can be applied in a specific order, with the age of the patch being the default.

So, the smart way to start would be to create a new patch for a new feature. In my case, this is rarely the case, so I have to figure out another way. I usually have some commits and a bunch of stuff I didn't commit just yet, so here's what I do:

> hg ci -m 'Commit my outstanding change before pulling' > hg log -l 5 # check for the latest revision I want to effectively "rebase" to > hg export -r $REV > working.patch > hg strip $REV # this reverts the entire repo back to the point of the revision > hg pull -U # get the latest

At this point my repo is all up to date and my only record of my previous work is my patch file I exported. Not very cool, so lets fix that first by storing my work in the queue.

> hg qinit -c # this versions the patch queue so you can save patches even after their done > hg qnew -m 'Add my working patch for later' working-feature > patch -i working.patch -p1 > hg qrefresh

While it make not be clear what happens is the patch gets saved on the queue in a patch called "working-feature". The qnew creates the patch, the changes are made and the qrefresh updates the current patch with the changes. It should also be noted that this updates the patch's changeset in the repo. These patches are actual changesets so you can use the other hg commands. The bisect command is usually given as a good example, but anything should work such as hg diff.

Now that I have patch in the queue, I can see what has happened.

> hg qapplied

This will tell me what patches have been applied. There are effectively two stacks involved. There is the applied stack and the series stack. The applied stack is considered to be applied to the repo, meaning the source files contain the changes in the patch. The series stack lists all the patches in the queue whether they have been applied or not.

In this example, I want to take my curent work, the working-feature patch, and not have it applied since the changes are unfinished. I then want to make changes for some other feature, commit them, push and finally start back where I left off.

> hg qpop # takes my currently applied patch and un-applies it, yet keeps it in available patches in the queue > hg qnew -m 'A hotfix for production' fix-for-bug # create a new patch for my fix # fix the bug! > hg qrefresh # update my bug fix patch > hg qnew -m 'Update the tests for the bug' fix-for-bug-tests # create patch updating the tests # update the tests > hg qrefresh # update my second patch # ready to commit! > hg qfinish fix-for-bug && hg qfinish fix-for-bug-tests # another option > hg qfinish -a # turn all the applied changesets into real commited changesets > hg push > hg qpush working-feature # back to where we left off

Things can obviously get more complicated than this, but off hand it makes a good deal of sense once you give it a try. I should also mention that there might be better ways to deal with external patches like I did initially. There is a qimport command (for example) that seems promising assuming you're working with patches people send.

The biggest benefit of using mq was that it was relatively painless to go back and make a new patch. This may seem like a small issue, but for me knowing that I can quickly save my work for later without branching or cloning is a huge win. My biggest complaint always ends up being that I've gone and written code, committed (b/c that is the point of being distributed) and found that I'll have to jump through hoops to only push a certain set of changes that are finished. MQ has always been presented as the way to handle my use cases, but experimenting never yielded the confidence to commit to using it. At this point, I feel much better about using mq consistently and integrating it into my workflow. Also, thanks to the folks in #mercurial on freenode for all the explanations and helping me wrap my head around what is happening.

Music Industry Contracts

One of the biggest problems I've personally noted with the music industry is the desire for exclusivity. The music industry typically tries to manipulate the economic equation by making music more scarce, so they can in turn charge more for it. Labels want to own the masters. Pubslishing companies singularly own the publishing rights (song writing). They all want to be the single source for providing a particular artists song. Likewise, managers provide a single interface for working with a band. Often times they want a contract setting the length of time they represent the band. This is obviously to protect their initial time investment where they can begin to make a lot of money if things with the band work out. This meant getting a huge payoff when a major gives the band advances as well as getting money from tours and licensing. The problem is often times a manager will have outlasted the bands needs and the payments are for things the manager has no relation with. 

The root of this idea always goes back to a contract. The one system in the music industry that I appreciate is a lawyer's agreement. They effectively send you a description of what they'll do for you and that is that. If they don't do it, you find another lawyer. Pretty simple. There is no contract either! It is a simple agreement. I have no idea why everything doesn't work this way. Everyone is selling something in one way or another, so why not simply pay a commission? If a manager finds you a huge gig that pays well, the manager gets the money. If the booking agent doesn't book the show, nothing for them. If a label pays for your t-shirts to be made, then they get their share, otherwise, it is yours. If a publishing company finds you a placement, they get paid, if not, tough luck. 

The current situation is that everyone wants a contract. When you are starting out, these contracts seem fishy and gross. You can tell that they are written in such a way that someone will do a minimum of work now and even less later. The opportunity usually has little chance of pushing forward your career, and in exchange, you provide the entity with money once your hard paid off! This is where the exclusivity is such a terrible problem. Someone wants to be the exclusive resource for some artist, even when it is better to move to a better resource. This is typically not entirely present in other industries. If I own a shop and another distributor approaches me to sell my goods, I can switch. If a sales person at a store doesn't make any sales, they don't get paid very much and can be fired. Contrast this to the music industry which effectively is saying a sales person will try to make a sale for three months and in turn gets 15% of the profits for three years off everything in the store. 

My biggest frustration is that people have little incentive to work hard. You often have a very limited time frame to make things happen. If things don't work out, the contract protects the person who didn't make it happen from profiting from the artist, who might eventually find a way to become profitable through hard work and persistence. In our situation, we have worked very hard and put a lot of money into the music. We need to be repaid before others have because we've done all the work. Are we open to finding a manager or label or anyone that can help us? Of course we are! Do we want to give them a 3 year contract giving them 15% of everything we make? No way! If you want to profit from my band, you're more than welcome to do so under the simple terms that you get paid for what you do. If that is not enough money to make it worthwhile, then don't do it. 

The RIAA, Really?

I recently read this blog by Joel Tenebaum who is getting sued by the RIAA. From what I gather from music blog headlines, this is a pretty important case in so far as it seems to take up a good portion of attention on a few music related blogs.

Without following what has been going on, it is clear from Joel's blog that this case is ridiculous. The fact the RIAA can sue people for such extravagant amounts is more a testament to how the legal system is flawed then simply they are a terrible organization. That said, the RIAA are a terrible organization! I find it very difficult to see how file sharing is really costing the industry so much money. If I owned a video cassette factory, do I get to sue DVD manufacturers for killing my business? The US auto market seems get help from the government even though they made terrible cars. It might have been a better tactic for GM to sue Toyota for profits lost due to unfair competitive practices, aka making decent cars.

On a side note, if Toyota or Honda wants to make a 15 passenger van (a la doge sprinter), sign me up. Our 96 Toyota Corrolla has over 185k miles and is going strong. Our 02 Dodge Ram van has been is heading back to the shop after getting a new engine at 67k miles. The only Dodge I'll ever consider is their Sprinter van and here is a hint why, Mercedez Benz diesal engine.

Car frustrations aside, the real problem with the RIAA and the lawsuits they promote has to do with responsibility. Back in the 90s, the music industry was making massive amounts of money. Relatively small bands were going platinum with CDs that cost $17. I'm not saying this was a problem, but it obviously tainted the music industry's view of what they are owed. It seems that now when profits are dropping, the answer is not something like, "how do we make a better product?" or "is there a way we can provide more value to our customers?". Instead they form a legal team to go find people who are sharing music and use the legal system as means of taking justice into their own hands. The lawyers effectively act as the music industry's mercenaries and aim to extract their due vigilante style. Honestly, it is pretty screwed up.

Now, I'm not saying it is totally fine to pirate music. As a musician, I'll be extremely frank, it is expensive, time consuming and difficult to make music that is effectively seen as worthless. That said, that is where we are. I'm ok with it. I don't make music to make a million dollars. What is frustrating is that in order to make music anywhere close to the quality I'd like my music to be at, we are talking a hefty chunk of change, that I cannot afford. While many people say take it on the road and sell t-shirts, I'd bring up the above points about a van being in the shop with a new engine. Van issues aside, it is and order of magnitude cheaper to get 1000 CDs pressed than 1000 t-shirts. We have to buy around 75 at a time because that is all we can afford. And on that note, I'll stop whining because this past year of playing music has been more fun and exciting than I can describe, I'll keep paying down my credit card debt and shut up.

At this point, I don't think it is OK to pirate music in so far as if you really like an artist and want to support them, getting their music without paying seems like a poor way to do it. I also don't think you should have to fork out $17 to get a CD that is so-so at best. It is also ridiculous that our friend Joel is having to deal with a million dollar lawsuit. Since the RIAA is suing, it seems someone else should do the same. Why not fight fire with fire. If there are any smart lawyers out there that want to make a bunch of money on a class action law suit and put a nail in the major label coffin, why not strike up a pricing fixing class action lawsuit. Does it seem at all odd that CDs seem to always be roughly the same price no matter the label? If the RIAA gets to sue for lost revenue because people used an existing technology, then why not sue for raising the price of the CDs to a ridiculous level?

Ok, I know what you're thinking. The $17 CD was a function of supply and demand. People were willing to pay such a high amount, therefore, it was simply the happy intersection where supply and demand met, had coffee and talked about old times. And based on my economics courses, this is entirely correct. But guess what, with the introductions of mp3s, guess what happened to scarcity. Ah yes, another economic term that defines the amount of available resources, aka supply. Does it seem fair then that because the supply of music became saturated the RIAA gets to swoop down and sue the pirates for all the lost revenue? I'm saying no.

The one positive thing the RIAA have done is pose enough of a threat to make people prefer purchasing music over pirating it. Honestly, paying $50 for 4 or 5 records is a much better deal than pirating them and being asked to pay $3k. If I were the music industry, I'd consider it a win and get back to business. I'd probably keep the RIAA around to keep making an effort to get folks to settle. It is still pretty slimy, but honestly, it is stealing and simply because the government can't (and really shouldn't) spend the time and money to monitor file sharing, I'm OK with the RIAA making its effort, just as long as the goal is simply to make it clear that it is cheaper to buy music than pirate it most of the time. Unfortunately, this is not the goal of the RIAA. They are trying to get lost revenue and that screams the "I don't like my customers and I'm lazy" mentality of the major labels.

I think in the end the best way to combat the RIAA is simply to avoid pirating music. If you want some free music there is traditionally a wealth of ways to try it legally. More and more bands are giving away their music in hopes of getting fans. Honestly, if someone reaches out saying they have enjoyed what they've heard, it is pretty easy for me to say thanks and give them a download code. If you really want to get music for free, then start a relationship with an artist. Let them know you're listening to them and telling your friends. Talk about their shows, videos, blog posts, reviews, and twitter about the band. If you champion a band, they will pay you back. We give away stuff all the time to people who support us. It really is the least we can do.

Of course, if you want to drop off a van at my door step, I'm cool with that too.

Javascript DB Revelation

I finally realized why having a database for client side Javascript is such a great idea. It took a while for me see a use case that really exposed why it is such a helpful concept. I always thought to myself that if you have an application pushing enough data to the client where you need to do SQL queries, then something just isn't designed correctly. I'm all for pushing work to the client when it helps speed up the response, but pushing large batches of data to the client can only hurt. It has never made sense then why toolkits like Gears provide a database feature.

Like so much in software, there is always a consideration of state. What does the world look like at a particular point in time. One of the more difficult aspects of Javascript is that the state can be difficult to determine because there are times where taking a look at some value might be misleading. A great example is if you check the status of an input. That input might change by some other event. The easy answer is to avoid randomly attaching events. The problem is things can get complicated when events start bubbling and they can be fired multiple times. I'm sure smarter folks than me have developed excellent skills working with the Javascript event models (and I hope they all contribute to jQuery), but for me, it is still very much a challenge to keep straight why I might see an event a few times when it seems like once would be enough (with bubbling considered).

Having a client side database takes much of this confusion out of the picture because you have a place to keep your current state that can be locked. Sessions are a great example where multiple requests can be considered in their own scope even though different threads may handle them. One common means for storing sessions is via a database. The general idea is that you have a single canonical resource for making a decision. Even though the browser might be in the middle of making changes, you have a means of saving and retriveing critical details about the state of the applicaiton without trusting the asynchronous nature of Javascript events.

I might still be misunderstanding the thinking behind a Javascript database, but assuming it gives me a place outside the constant changing of the browser DOM for keeping important state information, then I think it makes a lot more sense.