Ionrock Dot Org

My Weblog

Testing if an Element Has a Class in Selenium

In Selenium tests tend to get brittle very quickly. This is a function of testing something like the DOM that is effectively like machine code for the browser. What happens is changing the DOM to support some new feature written in Javascript almost always radically changes the tests. These changes typically have two aspects.

1. Changing the locators needed to find different elements
2. Changing the verify/assert statements to test a new attribute, style, value or element

Changing locators is almost a given and while you can minimize the changes by selecting as close to the necessary element as possible and through using unique identifiers (either IDs or classes have worked for me), it is always something of a struggle. Likewise, changing the tests to effectively reflect what changed can be difficult as well, since the change might be as simple as adding or removing a new class.

All that in mind, I always wished there was a simple way to test whether an element as a class. For example, if you had a div that needed an image as the background, testing for something like a 'button-bg' class or something similar seems really helpful. It seemed simple enough to implement, so I went about creating a 'assertElementHasClass' function.

First off, Selenium supports adding new functionality via user-extensions.js. This is just a file that will be included and in turn made available in your tests. My first cut at getting the required functionality was to have a helper function and eval the result. This was easy enough, but somewhat undesireable since it effectively masked what was being tested. That said it was a really easy way to get started.

The "correct" way to add your own functionality is to add a prototype to the Selenium object. There are a few guidelines listed in an example user-extensions.js, so I'd suggest taking a close look at that. It can also be somewhat helpful to check out the selenium-api.js which is where all the typical selenese commands are.

In the end here is what I came up with:

Selenium.prototype.assertElementHasClass = function (locator, cls) {     var element = this.page().findElement(locator);     var cls_name = element.className;     var clses = cls_name.split(' ');     var actual = false;     for (var i=0; i < clses.length; i++) {         if (clses[i] == cls) {             actual = true;         }     }     var expected = true;     return actual == expected; };

One thing to note is that the assert functions want you to return either true or false. Other examples I looked at used an Assert.match function for comparisons. I'm not really sure what the difference is at this point, but as I couldnt' get the Assert.match to work, it seems safe to avoid it.

Hope this helps someone!

Pushing Code

I'm quickly becoming a true fan of using a DVCS. I've always liked the ideas, but in practice it has been somewhat difficult to establish a workflow that seems an order of magnatude better than something like SVN. One aspect that can get a little confusing is when to push code. Using something like SVN, when you commit you effectively have added your change to the repository not only for yourself, but for everyone who uses the code base. This obviously makes a commit a serious thing because it is public and has an immediate effect.

When you use a DVCS there is a separation between making a commit and making it public. You should be committing quite often in a DVCS because that is partly the point. What comes into question then is how you make the decision to push. In one sense it seems obvious tha you'd take the same precautions pushing in a DVCS as you would commiting in something like SVN. When the feature is finished, then push. As simple as that souinds, you have to consider that you might need to pull changes before you push. This may or may not conflict with your work. How you handle merging conflicts doesn't necessarily mean simply making sure there are no conflicts. There is also the log and history to consider. If you rollback to a certain revision, have you also rolled back someone elses mid-feature commits?

The point here is not necessarily answering the question of when to push. I think you should take the same precautions as before. The code should work and you break the build (even if that build is starting a python web app). The question is how to manage your local repo in a way that makes helpful breakpoints possible while still keeping the public repository an effective record of where the code has been. I always believed DVCS' would make that local repository management obvious, which is entirely untrue. But, now that I've managed to wrap my brain ever so slightly around how to keep a decent local repo, my ideal DVCS world seems to be working out.

Templating Needs

As someone who started programming with PHP, you would think that templating would be a pretty essential part of my web development life. The fact is nothing is farther from the truth. While I was a big XSLT user for a time, that was more a function of my day job and wanting to learn more than necessarily having a passion for declaratively programming the elements that would render a page. Even now, as I find myself more and more involved in writing Javascript than actual server side code, templates seem more a hindrance than help.

That is probably a bit strong. A good template language is very helpful as a gateway between programming language data structures and the intended output. The problem with template languages is not that they lack features or have odd syntax. The real issue is that it is difficult to understand the best practices or design patterns that make the best use of the templates.

The best example I have of this is the question of whether to include or inherit. On the one hand, you might have a template that includes different templates, that in turn might include other templates. The initial page might be very simple, although through including other templates, a complex feature rich page is created. Some people feel this is too unstructured since there is really no rhyme or reason to how things are included. Another option is to allow templates to inherit from larger, more generalized templates. In this way you have a more programmatic means of building pages. It might be slightly more front loaded in terms of design since you usually have to modularize the templates sooner, but that is simply part of the trade off.

Honestly, I think both methods are valid in terms of template language features. You can establish a layout and design in a way that is flexible and allows making sweeping changes or small updates. At least that is the impression I get. Like I said, the issue I find is not necessarily in the template features (or lack there of), but the actual layout patterns.

Hopefully it is clear that a simple header, footer, body layout with some columns is pretty trivial. Heading back to the land of Javascript and building modular widgets or components or whatever else you might call them (jQuery plugins), there is a question of how to properly introduce not only modularlized templates, but an efficient and simple way of also including their respective Javascript. Oh yeah, we'll throw things like CSS, Flash and any extras like custom meta/link tags in the HTML head section as well.

Obviously folks have been dealing with these concepts for as long as there has been web development. My question then is why have we not seen more frameworks or tools that help to provide an answer for these questions? Providing an answer myself, it is because there are no real best practices people want to commit to. This might be because it is hard enough to get a HTML interface working reliably in a cross browser environment, so why waste any more time. The real answer I think lies in the lack of communication regarding what is really needed in template languages and Javascript.

With that in mind, here is my rough and probably wildly wrong idea for a few tools that a template language could provide that might help improve the state of the art.

Template Parts

This entirely generic term is meant to express the idea of nesting templates in a way that is more than simply inclusion or inheriting. The problem with the current pattern is you have to include one template where you might really want to include a sequence of the same template providing different arguments. This is pretty much a "duh" kind of request, but I've yet to see something like:

template.header.push('head/javascript.tmpl', {'src': '/path/to/widget.js'})

The idea is that you could define the "header" attribute to organize your list of templates however you'd like. This might also do something as simple as appending to a single Javascript file served with a highly cacheable minted URI as well. Who knows.

Efficient Modules

Again, another super generic term that alludes to a subtle lack of understanding on my part. One problem with templates is that they often rely on external pieces to be correctly used. For example, most of the time web frameworks and the like default to using a "text/html" content type. What happens is when they return "text/plain" or "application/json" is that the method/object/etc. that handles the response must set the correct content type. XSLT does things correctly here by allowing the output tag that define the media type. It might seem a little nit picky, but the impact of concept is farther reaching then one might expect, especially when you are talking about making a template more efficient.

Again, a method that returns a response will often times returned that response in chunks or in one piece. It would be great if you could do similar things with templates. Some templates need to understand an entire template while others work as a streaming filter. Most template languages don't necessarily allow ultra fine grained control of this sort of thing. What's more, they most definitely don't allow the template to define this kind of attribute. I don't blame template language authors since this would probably end up making a mess of things in terms of complexity. That said, it would be nice to define within a template whether it was "streamable" or not. Likewise, it would be also be nice to declare within the template the content type if necessary. In these kinds of situations you might have a streamable template that outputs a CSV content type. Another example is a single Javascript file that must be in the correct order. There are other tools that help with different aspects of this kind of problem, but letting the template define these bits might help to move some details into the template where it arguably belongs.

Normalized Data Environment

This is just a really terrible name for being able to easily provide a template with some JSON as the variable environment and return the result. Pretty simple stuff. It has probably been done a million times in one way or another but it seems helpful. One gotcha is that often times people will pass in things like database connections to a template. This is totally fine as long as the objects allow specific operations. The best example of this is iteration and in fact I can't really even think of another off hand, so I'll stop there. This last bit is probably a little outside the scope of a template language need and would probably fall into the realm of a tool or framework. In any case, the idea is to provide a reliable means of encapsulating templates, most obviously for things like Ajax.

Like I said, these ideas are probably somewhat incorrect and it wouldn't surprise me template language authors simply laugh at these suggestions. But if that is the case, then I'd say it might very well be up framework writers to establish the best practices and potentially integration with specific template languages. It would be really nice to have a "MakoNiceSite - The Effective Mako Toolkit" or some other collection of tools that make common Mako patterns more obvious. Also, it seems reasonable to have a few other dependencies such as a CSS framework. I've been using the YUI CSS tools and while I'm sure there are some caveats to be aware of, the constraints have been helpful and faster to develop with.

In the end it would be really great if there was a HTML on Rails kind of system that makes things like modular templating simple. The pieces all seem to be present, it is just a matter of putting them together that seems so difficult.

Hybrid Widgets

It has been really interesting to think about how jquery ui fits within the design of our applications at work and the concept of a "widget". On the one hand, a widget is very much a javascript/jquery plugin in that it effectively must do all its communication over with ajax and is in fact loaded via ajax. The converse is that the HTML generated often comes directly from the server. There is always a struggle in my mind where this division really should be and while "it depends" definitely holds true, the scenario that helps define best practices seems to allude finding a set of general rule of thumb ideals.

When I worked at Novell in college it was when they were venturing into the webmail space with Hula. The idea for Hula was a webmail app that rivaled desktop applications while still providing an open platform for both email and calendaring. I got to see demos of the UI and was always impressed with what they were doing. When I looked at the source there was a huge gap when trying to find templates that defined the look and feel. In talking to the developers, the entire UI basically was constructed via javascript and only passed data back and forth to the server. At the time it seemed like a perfect idea.

Well, time seems to be the best tester of software as Hula was eventually dropped. I remember seeing cool demos here and there, but it was obvious it was a huge burden building the UI in this way. The process was slow and difficult to debug. Like any major javascript application, testing across supported browsers is most definitely an issue. In the end, it seemed that while the design seemed solid, the actual practicalities of building a UI completely in javascript was too difficult to overcome. I'm not saying that this is what sunk the project, but it was clear it was an issue.

What is interesting then is that jQuery UI uses a similar pattern of forcing the widget to write the HTML. This really isn't a problem in this respect because the widgets are limited in scope and must be decoupled from any specific application. In my use case at work, a widget also encompasses a server component, which makes using jQuery UI widgets more difficult. There are things the server markup must provide that the widget knows about and our requirement for decoupling is much less.

The goal then is to find a way to create a widget that has a specific server component while still writing generic widgets that can be decoupled in terms of each other, while still keeping the ease of integration required. The design I'm coming up with then is effectively a hybrid approach that aims to allow the server side templates define the majority of the widget and rely on specific cues to communicate the coupling between the javascript and the widget markup.

A good example would be a list of inputs. If you select one input, the values of the others must be cleared. So, the markup for the inputs should all be given semantic class names to help make it clear what the set of inputs are. This is usually unique according to the question's unique ID. Then any sort of grouping element would also be used to delineate the border or limits within the DOM the widget should not effect. Finally, the markup should utilize an internal data structure for keeping track of state that is verified through DOM elements. In other words, a two pronged attack at making sure the widget does the right thing when different events occur.

My goal is to keep the template changes to a minimum past changing the semantic classes and rely on the javascript be flexible to handle small changes in markup as needed. The way this all will need to come together is through configuration. The widget initializer will need to include the border or root element from which the widget works as well as the selectors for any nodes that need to be used as reference points. In terms of actual code then, the basic design will be a widget that does the generic functionality and then a set of initializers that will handle adding the context specific aspects.

Hopefully this small change will help make the difference for implementing widgets in a way that provides the flexibility we're beginning to need. Also, my hope is that in the future we can write the actual widget javascript to be more modular from the beginning instead of having to special case things so much afterwards. While the code often seems pretty simple, it becomes clear there is too much coupling between the server templates and the client side javascript. The result of this coupling is that small DOM changes wreak havoc on the functionality, when arguably, it should be relatively benign changing the look and feel or style. Likewise, no one wants to be tied down dealing with massive abstractions and libraries, so simplicity is a must.

The design is still very much in the air, but the ideas seem to be settling. I don't imagine that when it is all said and done things will be radically different. I also think that limiting the changes is probably a good thing and will help provide a healthy constraint for simplicity.

Make a Great Product

We recently bought a new van. Ok, it's not really "new", but it is new to us. We needed some thing bigger that had more space. On the last Ume tour our van broke down. The computer blew after getting a new engine. The whole experience made it exceptionally clear how important quality really is. I'm never going to buy another Dodge.

The new van is a Ford, which I still have some reservations about. That said, it is a diesel, which means there are some essential aspects of the engine that require quality, so it seems like a good decision.

One of the things that hit me was that everyone wants to take shortcuts. I was concerned the engine shop was trying to take some short cuts to fixing things and everything I read about dealerships enforced the idea of getting something for nothing. This whole time it seemed clear that the lack focus on great products makes the entire system faulty. While supply and demand are technically meeting in the middle, the real problem is that of quality.

If dealerships and auto makers put customers first and tried to make great cars, the idea of a pushy sales person and dread at the dealership would most likely slip away. I truly think this mentality is throughout our entire economy. People are trying to make a buck and they are willing to sacrifice quality to make more money. The result is poorly made products that don't meet customer's needs and ends up making things difficult for everyone.

I'm convinced that if a business focuses on making a great product they can be successful. I know that is who I'd buy from. It doesn't take much to convince a person that it is a huge waste of time and money buying crappy products. There are definitely ways to save money, but rarely is it through buying a product that is less than ideal. Likewise, if you want to make money, selling off cheap products can only be a short term solution.

I'm probably being somewhat idealistic, but I still think the idea of a great product is where the future is. Too many industries seem to be collapsing from the weight of bigger gains while reducing quality. I just want to be treated with respect and pay for products that are woth the money I'm paying. Pretty simple.

Streaming is For Old Folks

I just read this article on a new streaming music service for Austrailia. It isn't going to work. The problem with streaming is that it doesn't consider the audience. Folks who have jobs and more than likely work on a computer all day love the idea of opening some streaming service app and listening to great music all day. A select few might be interested in DJing their dinner party or get together with friends using the catalog of a great streaming service. I'm not saying that this is a bad thing, but if you ask the kids, my bet is they'll just use YouTube.

This lack of acknowledgement for the audience of these services doesn't provide much hope for the music industry. Kids are different from young adults and folks halfway through life are radically different from both. This can't be surprising. People have phones, text messaging, instant messaging, email, twitter, facebook, posterous, tumblr, etc. that all basically do the same thing. We're talking communication and the possibilities are endless. It makes no sense that the music industry think streaming is the next album. Technology has blown the doors open, so why sit there and try and squeeze the populace into one medium. Everyone has different contexts when listening to music, so the future is providing those contexts online and offline.

I think the music industry needs to consider its place within youth culture. When I was a kid, the radio seemed to have integrity and cool. I could listen to edgy bands even though my sister was pushing Tiffany. While I wasn't on the cusp of underground music, I ended up looking and finding bands like Fugazi. The music defined who I was. Now, kids are still defined by music, but the difference is the concept of an underground is gone. It is too easy to be found, which means that kids don't necessarily care about obscurity. They just want to find music that helps give them definition. Here is where streaming services fall short. The focus is providing music instead of expressing youth culture. The sttreaming service can't define a genre or present a style. The streaming service isn't the local record store staff picks. While it opens up possibilities in terms of access, it doesn't help youth find bands that signify who they are. That is why things like blogs, youtube, facebook, twitter and the wealth of tools aiding communication are where the future is. These tools let people find the artists that help to define their persona.

Personally, I'm not really a fan of streaming services. Yet, I do understand the appeal. I used to run a server from my house so I could stream songs on demand at work. It was kind of slow, but I really enjoyed having the access. That said, discovering new bands was still all about google and blogs. My guess is that there will never be another "album" or medium that corners the music market. It is a waste of time at this point. The album artwork is effectively worthless compared to a website with a blog and forum. When you throw flickr and youtube in the mix, the idea of selling a CD with 10 songs on it makes no sense at all. Nostalga is definitely worth something, but in the end, self expression is where music has always been and should stay. In other words, no one defines themselves by a pipe of on demand music.

Emacs Regexp Search and Replace

This is mainly just a reminder on how to do a search and replace with emacs using regular expressions.

First step is to find what you want to find using the regexp builder (M-x regexp-builder). I'm going to consider this crucial since emacs regex seems to be a little different from other regex engines (ie Python).

Second, once you have your regex, you define a section to capture for later use. Emacs uses parens as literals in regex, so grouping is done via escaping them:

\(http[|s]\)://\(.*\)

This provides two groups, the "http" and the rest after the "://".

Lastly, you do the actual replace-regex and provide the necessary details. To use something you caputured (put in a group) in your replacement,  you either use "\&" when you have no groups, or "\1" where 1 is the group index (like in other regex engines). Here is some code I used it on:

class SomeTest(SeleniumTest):     def __init__(self):         self.t.open('/path/to/test', '')         self.t.pause('500', '')         # ... other commands         self.t.pause('1100', '')         #.... more commands         self.t.pause('1500, '')

What I wanted to do was turn all those pause calls to "self.pause(${time})". Here is what I did in emacs:

M-x replace-regexp RET self.t.pause('\([[:digit:]]+\)', '') RET self.pause('\1') RET

Not very difficult, but when you don't practice that sort of thing it becomes easy to forget.

Free Time Coding

The other day a miracle happened and I had a little time to code on whatever I wanted. Lately I've been so busy with music and work that there has been almost no opportunities to hack on something fun. Honestly, even though work as been tough, I've actually been having a lot of fun with Javascript, so it is not as though coding is a chore. The bigger issue is finding something creative to code.

The last time a little free time came up I took a look at Javascript on the server. I wrote a tiny framework and started writing a simple template language. Eventually I got stuck trying to figure out to compile/eval some Javascript code within the context of a set of variables, which is something I had done in Python. Since the tactic wasn't going to work and it seemed as though time was no on my side, I figured I'd move along.

This most recent bit of free time got me thinking about Erlang. Recently _why disappeared and as a result Hacker News became a posting board for theories and eulogies. To combat this theme a bunch of users started posting Erlang articles. It got me interested so I started taking a look at the Erlang web frameworks and tools out there. It didn't take long for me to lose interest in Erlang. It seems like a great language in terms having functional qualities, but past that it doesn't seem like that much fun. I'm sure I'll take a stab some other time at really digging in, but for now, I'm ok letting Erlang gestate a bit more in my head.

So, the big question is what's next? I'm coming to the conclusion that my desire to start something new is rapidly falling by the wayside. I'm not in the mood to learn an entirely new framework or language at this point. This attitude could change, but at the moment it seems like it would be better to really dig in on an idea or existing project. The hardest thing about figuring out what that project or idea might be is simply an itch that needs to be scratched.

I know Bob has quite a few projects that could use help. Dejavu seems incredibly interesting as an ORM, but I'm really not much of a database guy. There is always making an effort to tackle a ticket on CherryPy, but again, that code is darn solid with quite a bit of history, so bugs seem like they would need a good deal of knowledge. This is totally fine, but it doesn't necessarily fit my free time coding ideal of doing something that lets me code quickly. That said, I also don't want to be lazy either so biting the bullet and working with something like Dejavu or CherryPy would be worthwhile.

Another area that is of interest to me is creating some tools for Ume. We are terribly unorganized when it comes to our merch. We're hitting the road soon and having some simple tools to help keep track of how we're doing financially seems like it might be nice. The problem there would be getting users. I know my wife pretty well and suggesting that we use some hacked together web page to input how many shirts we sold is going to be a challenge when you have a mini-rush at the merch table. Also, it is doubtful a well designed spreadsheet wouldn't be more than adequate.

It looks like my best bet is to take another look at Dejavu and see if there might be something I could do. At the very least it would be nice to get back in to the SQL world a bit and possibly brush up on things that I've long forgotten. I can use a simple accounting or merch tracking app as an example and see what I figure out. Making it a CherryPy app instead of WSGI might also help me learn something as well and it might be possible to help folks out on the mailing list a bit more.

Honestly, I can't imagine I'm going to get very far, but having a plan seems like part of the battle. Wish me luck as I'm sure I'll need it!

A Kind of Programmer

I got into programming with PHP. It was an easy way to get started and proved to be a valuable tool for a very long time. In school, it was a no brainer to knock out projects with a few bits of PHP and gladly take the A. Although, for whatever reason, I ended up getting more interested in other languages. My interest in programming partly was fueled by an interest in open source communities such as Gnome and Linux. This got me interested in C# and Mono, and to some extent, Python. As I moved forward in learning these other languages, my opinion of PHP fell radically. It started to become ugly and archaic. Eventually, PHP seemed like a dirty and uninteresting tool.

Now that I'm a bit older, I realize that my opinions were immaturity. There is nothing wrong with PHP and there are many great examples of how it is used effectively and elegantly. Yet, even though I recognize this silly reluctance to enjoy a practical language, there is still a part of me that glamorizes more complicated systems. At times I idolize those developers that have to wrangle memory into place and construct interesting data structures to improve I/O and caching. In my mind these are the people that create the basic environment that all other programmers must work under, which must mean they are an extremely intelligent and elite group of people.

I'm not for a second going to diminish what these excellent programmers do. But, I am going to stop believing it is something out of the ordinary. When I was young, my view of PHP was partly attributed to my desire to be one of these influential hackers who seemed to write the code that other coders would use. People who wrote databases, kernels, and servers seemed like their influence was spread far and wide among people who had to get things done. In a way, my attitude was that writing code for people was not as cool or interesting as writing code that enabled more code. Obviously, this is one of the lamest ideas ever.

While I still have dreams of being associated with the elite group of people who manage to write amazing databases and highly distributed server systems, the reality is I'm just not that kind of programmer. I've been doing a lot of work with Javascript and it has made me realize that it is a problem space that fits me rather well. Up until this point, I don't know that I would have wanted to admit that, but the reality is reality. I wouldn't say I'm a "front end" coder as I would associate that more with having an eye for design. But at this point in my programming career I'm happy to admit I've finally begun to really appreciate writing applications for end users.

I'm sure there are still tons of interesting lower level applications that I'll dream of writing, but for the time being I'm going to stick with thinking about end users. I missed an opportunity when I first started programming to appreciate people using my code because I wanted those people using it to be other programmers. That shouldn't happen again. I'm hoping this kind of attitude will not only help me improve my code, also help improve the lives of those using my code.