TDD: The door analogy

Recently, I was explaining what test driven development was to my wife and used a description involving the creation of a door, and I realised this may be a very good way to explain what TDD is, how it's meant to function and why it produces superior results. I've thought a bit more about the analogy and fleshed it out some, so here goes:

Peter has asked you to create a door so you go away and start writing some tests for what the door should do based on his statement of what he wants. You start initially with:
1) The door should have a handle
2) If you turn the handle and push then the door should open

So you go away and create a door that fulfills these tests. You present this to Peter, he opens the door and it does this by falling over. So you go back to the tests and add some tests you missed:
3) When the door is open, you should be able to pull on the handle and it will close
4) The door should stay upright when both open and closed

You then create this door (realising with these extra tests that you needed hinges) and present it again. Peter is happier with this new door, but then notices that if he pushes the door without turning the handle it still opens. This is another missed test so you add it to your tests:
5) If you push on the door without turning the handle the door should stay closed

You then create a door, adding a latch to the door that retracts when you turn the handle and test again. When running the tests you notice that half the time, test's 3 and 5 are failing and you realise that it's because of the construction of the latch. If the door opens in one direction then the latch won't retract automatically when pulling the door closed. You go back to Peter and say you need to clarify what he wants to happen and present him with the following alternatives:
1) The door can only open in one direction so you need to push the door from one side and pull the door from the other
or
2) In order to close the door you must turn the handle in order to manually retract the latch and close the door fully

Peter considers this and says he wants option one. This then causes a rewrite of the test cases to the following:
1) The door should have a handle
2) The door should only open in one direction
3) To open the door twist the handle and either push or pull. Only one of these should work depending on which side of the door you are on
4) When the door is open, you can close the door by performing the opposite action to the one used to open it
5) The door should stay upright when both open and closed
6) If you attempt to open the door without turning the handle the door should stay closed

You rewrite the tests and then run the previously constructed door through these tests to see where the problems are. This time, tests 2, 3 and 6 fail. Looking at the first of these you see that the door opens in both directions, which is now a violation of the tests so you add a ridge to the door frame that stops it from opening in one direction and re-run the tests. This time no tests fail, you present your door to Peter one last time and he is happy with it and installs the door all over his house.

So with this process, you have several iterations and each one improves the door. More importantly, each iteration adds more tests in which show that the door is improved. Now, most people would think 'but a door is obvious, the final version is how I'd have created it initially' but consider... what if Peter had wanted a door that opened in both directions and required you to turn the handle to shut it? You would have created a door that didn't do that and wouldn't have identified the point where it was required. You would have just given Peter his door and he would have gone away less satisfied.

Also, this is a high level example. Consider what you'd do if you didn't know what a door was? You'd look at some doors and create something that looked like it, with no way of knowing it was correct or not. It may work initially, but after some improvements it suddenly starts falling over. Now you are stuck in the position of having no clue about why it's falling over and what it was meant to do initially. So you go 'Right, it shouldn't fall over' and just prop it up so it won't fall over... but then the door doesn't open and you have an annoyed customer. If you had your tests there you would be able to point to what it was meant to do (e.g. test 5, 'stay upright'), prop it up and then when retesting spot that other tests are failing. So you go away and look at the problem some more, coming up with the solution that you need a third hinge on the door to reinforce it and stop it falling over after some use. You also add to your test cases:
test 7 - The door should be able to be opened and shut multiple times without falling over
and present the new door to your customer who is now delighted that you've solved the problem properly.

The analogy is probably a bit strained now, but the principle still holds... the tests are there for more than just 'testing' the system. They are there as a verification, they are there as a safety net, they are there as your specifications and they are there as your guide in an area you may not know much about. If you have something that isn't working correctly (due to a lack of understanding for example), you should identify which test(s) are testing things incorrectly and then modify them to test for correct behaviour. You then re-run these tests *without* changing any code (even if you 'know' what's wrong) so that you verify first that the new tests are failing. If you modify your tests and they still pass then your tests are still wrong (as the program has incorrect behaviour still), but if you modify your tests and your code and your new tests pass you don't know if it was because your new code works perfectly or the modified tests are incorrect.

Now, I know this is all standard to people who are avid TDD people, but I'm still getting up to speed on this methodology and the reasons behind why I've avoided it are:
1) Difficulty of testing - Big things are hard to test, but the little things are trivial and seem like they don't need testing... test them anyway, you never know if you will find a bug there and by testing the little things you are then able to break your big things down into smaller tests that *you've already written* and only test the small bit of new behaviour.
2) Benefits - Until you really think about the process and break down where the tests come in, TDD seems like a silly reversal. Why would it have advantages? Of course, with the above description the advantages are that you more quickly identify problems (in the third iteration, you immediately spot the test failures when adding the latch and ask for what the customer wants done when you realise it's due to mutually exclusive test cases). If you didn't have tests, or your tests were incidental things written after you finished creating your door how you thought it should work then you lose this benefit. There are other benefits that I'm getting clearer on, but they are left as an exercise for the reader ;)
3) 'But some things can't be tested' - This is a common concern, and it is false. Some people see the UI as untestable, but there are a lot of tools nowadays that allow you to test the UI in it's entirety. And before you get to testing the full UI you have a lot of components building it up. These CAN be tested. You can test that they change state as expected when called with fake input. You can get it to draw to a bitmap and check that against a pixel perfect bitmap that is how it *should* look. So you can verify every step of the way and build up bigger tests from well tested components, making this exactly the same as reason 1.

Those are the big reasons for me, and they are very much false reasons. I'm starting to get on board the TDD bandwagon and in the future I intend to have much better tests and try to write them before writing my code :) Of course, if I sometimes fail then it's not the end of the world, but I'll know what to blame when things start going wrong.

Reflections on Python, testing, Continuous Integration and other things

It's been a busy few weeks, both in work and out (for out of work, see Shared Illusions). In work, and of a technical nature, I've been focusing on testing and Continuous Integration and I feel it's time to look back and see what has come out of it.

Firstly, what I've been using for this. The list is:

• Python
• Py.test - My first test package. Very nice and easy to use but abandoned for reasons explained later
• Nose - A nice package available for Python for running tests with a variety of options including profiling and code coverage (using coverage.py)
  
• Hudson - A well featured build server package with a plugin architecture and several useful Python related plugins

It should be fairly obvious from my previous posts that I'm using mainly Python for work code at the moment. I've gone on at length about various Python packages and my 'fun' with transforming a complex XML structure into an equally complex and completely disjoint object structure :)

So, onto Py.test. This was my first pick for Python unit testing and I'd written a fair few tests with it. I've now moved away from it, and this is mainly because it was a bit too simple for what I needed for CI. Basically, Py.test is an absolute gem for quickly writing and running unit tests. Simply prefix tests with 'test_' and run py.test in the project's root directory and your tests will be run and results printed out. The approach is great - it's simple, it's transparent and it's fairly stress free. It just doesn't provide some key features from Nose, namely the ability to output the test report in an XUnit XML file and code coverage integration. For a lot of people, these aren't necesarry, but to get the most out of a CI build server, they are very much desirable as then the build server can generate nice graphics over time for test failures and give much more accurate reports on build status. Therefore py.test was abandoned in favour of a more complete tool.

That tool was Nose. This is a more featured test runner for Python and it is pretty easy to transfer from using Py.test to using Nose (and also from using PyUnit to using Nose, if that's what you prefer). For Py.test to nose, you don't need to modify your tests at all, unless you have function/method level fixtures. If you have these then you need to decorate the methods that use them with @setupmethod(setup_func, teardown_func)... hardly a difficult change. Nose will then do the same thing as Py.test and discover all your test functions following a configurable nameing convention (which defaults to 'test_'). For this minimal change, you are now able to run your unit tests with a wider range of report generators (including the aforementiond xunit), profile your code and run code-coverage tools all with Nose plugins and simple command line switches. Well worth it if Py.test just doesn't quite provide the test reports you need.

Finally, I needed an automated build server. While python code isn't built/compiled, the other features of a build server were very much desirable, such as its source code repository monitoring, automated running of unit tests and a nice web interface to view test results and previous builds through. I've previously encountered CruiseControl and was less than impressed with it (not particularly easy to configure, lacking in some key areas, and the interface is a bit clunky). I went looking and found Hudson. This is an 'evil Java app' that runs through tomcat, but is pretty damn good at what it does :) It provides a much cleaner web interface than Cruise Control, it comes with the ability to create new projects through the web interface (I previously had to create my own project creation for Cruise Control) and has the ability to create build scripts within the web interface using Ant targets, Maven targets, shell commands and allows new options to be added with plugins. For all of that, I can stand the minor contamination of java testing my Python code, especially as I don't have to write any java code ;)

So, what was the purpose of all of these tools? Basically, they were to get my projects up to speed with automated unit testing. This is an area I've never managed to get fully on board with (it seemed a waste at uni and I didn't manage to get the build server set up for unit tests at EMCC before they went under) but it's benefits are starting to show themselves. While I don't think the current projects I'm working with will get full unit test suites (not enough time to do so, as they are primarily research rather than production projects), I now have the necesarry legwork out the way to provide much more comprehensive testing in the future and for the testing to be run easily and hassle free every time I commit a change.

Refs:
Hudson: https://hudson.dev.java.net/
Py.test: http://codespeak.net/py/dist/test/test.html
Nose: http://code.google.com/p/python-nose/
Python: http://www.python.org/

XML, Objects and Python

I briefly mentioned that I had created two object related schemas for XML in my previous post and then elaborated on XOMS, the mapping schema. This time I'll elaborate on XOS - XML Object Schema. This is a schema I created to allow the specification of classes in XML (although I've called them objects for the XML syntax). I consider this less interesting than the mapping, although the implementation of the XOS processor in python led to some more interesting problems to overcome.

First though, a quick overview of the XML schema. This is simpler than the XOMS schema. Here is a schema for the same data model as seen previously:

<xos>
<xos:object name="Person">
<xos:attribute name="name" type="xos:string" />
<xos:attribute name="address" type="Address" />
</xos:object>
<xos:object name="Address">
<xos:attribute name="address" />
</xos:object>

Again, this is fairly self explanatory. Given this schema, the XOS processor would produce two classes, one called Person, one called Address, with the attributes specified. In most languages, the XOS processor would really be more of a pre-processor - taking an object model and outputting a set of files that define the classes required. In Python though, I could go a step further and process a model at run-time, creating the classes dynamically and registering them for use the same as any other class.

This is where the more interesting stuff came in - dynamically creating classes in python. It turns out that doing this is remarkably easy, with the following code:


def create_object(object_name):
try:
getattr(sys.modules['__main__'], object_name)
except AttributeError:
class BaseObject(object):
pass
BaseObject.__name__ = object_name
BaseObject.__module__ = '__main__'
setattr(sys.modules['__main__'], object_name, BaseObject)

(apologies for the lack of indentation. It should be clear where the indentation should be though)

This surprisingly simple bit of code creates a 'template' object derived from object, assigns it a name and a module and then registers it in the '__main__' module with the same name. It doesn't create attributes on the object, but with python this isn't required as setattr() can add arbitrary attributes to an object or class. I have plans to add in some ability for this in the future though, as I could then provide chunks of python code to do things such as create SQLAlchemy database objects on the fly.

The next step is then allowing these to be defined in arbitrary modules. For this, I needed another function:

def create_module(module_name):
if sys.modules.has_key(module_name):
return
previous_module = sys.modules['__main__']
full_mod_name = ""
for mod_name in module_name.split('.'):
full_mod_name = ".".join([full_mod_name, mod_name])
try:
previous_module = getattr(previous_module, mod_name)
except AttributeError:
mod = type(previous_module)(name=full_mod_name)
sys.modules[full_mod_name] = mod
setattr(previous_module, mod_name, mod)
previous_mod = mod


This is a fairly interesting function too. It first checks that the module being asked for hasn't already been loaded (the first line of the function). If it hasn't been loaded then the function loops through the module name split down by '.' and builds up the fully qualified name through the loop. For each loop iteration, it first checks that the previous module doesn't already have a module with the expected name. If it doesn't it creates a module with the fully qualified name, registers it in the sys.modules dictionary with that name as well, then uses setattr() on the previous module to set it with the individual name. It then sets the new module as the previous module and iterates.

The next step for XOS would be to have an inheritance mechanism modelled in the XML. Some preliminary experiments in python have shown that I'll need to use a metaclass to correctly set the base class for the objects created but I haven't finished this yet.

In the end, what this experiment has shown is that it's possible to re-implement the python object and module creation using Python and your own syntax. More surprisingly, what has been shown is that the actual creation is easy! Less than 20 lines of code to create objects and modules, something that wouldn't even be possible in something like C++ and would require huge amounts of reflection code in C# or Java.

XML and objects - my approach

So recently, I've been working on a project where I have the two extremes... a very complex tree structure in XML format and a very complex object structure in Python (which is incidentally mapped to an equally complex 56 table database using Elixir). The object structure would make no sense as a tree and XML doesn't work well as objects so I was left to consider alternatives.

My solution was to create a way of specifying the relationship between the XML nodes and the objects and created an XML schema language of sorts to do this (well, I created two, but the second one was more for completeness. I may mention it in a later blog post). I've unimaginatively called this schema the 'XML Object Mapping Schema' (XOMS for short) and a quick sample is as follows:

<xoms>
<person xoms_to="Person" name="Person.name">
<address xoms_to="Address" xoms_link="Person.address" xoms_content="Address.address">
</person>
</xoms>

It should be fairly obvious what the above should do. It says that a 'person' element should be mapped to a 'Person' object and it's 'name' attribute should be shoved into the 'Person.name' property. The 'address' node should similarly be mapped to an 'Address' object and the content of an 'address' element should be shoved into the 'Address.address' property. The xoms_link says that this Address object should then be linked to the Person.address property. So with this XML:

<person name="Joe Bloggs">
<address>
1 Somewhere drive, Someplace, Earth
</address>
</person>

would produce the following objects:

[<person name="Joe Bloggs" address="<Address address="...">" >, <Address address="...">]

with a Python implementation.

I've got a fair few more features than this now implemented in a Python library that allows automated object construction from XML to a fairly arbitrary object structure, including the ability to call functions on the objects with parameters take from subnodes (for when I can't manage what I need just with XOMS). It's not perfect but it's doing well enough to populate the above mentioned data models and it's a lot leaner than my original approach. With about 500 lines of code and about 100 lines of XML I've replaced the same amount of XML processing, except that my new approach is complete where my original code was only mapping about 10 objects and not all their attributes. If I'd gone down that road I'd have gotten a couple of thousand lines of code that would have been messing, horrible to maintain and impossible to see where everything was going. This way I have 500 lines of code that performs some magic and all of the interesting stuff is kept in the XOMS file where I can easily see what's happening and change it.

So I've now become guilty of that heinous crime - solving problems with XML by throwing more XML at it... this time it seems to have worked though!

Elixir and SQLAlchemy

After my initial foray into SQLAlchemy I started the process of mapping my current object model onto database tables and came across issues with my custom, hand-rolled relationship classes. While I expect I could have shoehorned these into SQLAlchemy mappers I decided instead that I'd be better served with an object model that was slightly more integrated with the database structure.

Cue Elixir. Elixir is a thin, ActiveRecord style wrapper on top of SQLAlchemy. It provides some useful features (such as simple to use relationship objects, and polymorphic associations) and allows you to drop down into SQLAlchemy for those features that they haven't wrapped, such as AssociationProxy (for doing a relationship through another relationship). Elixir is the missing link in the stack I was building up, giving me a powerful ORM and database layer with a lovely declarative syntax for building up all the objects I needed.

It took me about a week to go from my old model, through SQLAlchemy and into Elixir but in doing so, I have avoided a whole mess of spaghetti code involved in writing my objects out to a database, obtained database independence, and forced myself to refactor out a lot of code that was proving less than suitable for my requirements. On top of that, I should be able to integrate my system into a pylons or django web application much more easily now than I could have a week ago. Now if only I could get a similar system for XML parsing and allow me to concentrate on the important tasks I still have to do :)

SQLAlchemy

Continuing on with my sort of 'series' on various libraries and languages, I've recently started to use SQLAlchemy as an ORM layer in my current project.

The reason for using an ORM library came from the immortal problem of suddenly realising that I had a complicated object model defined and an equally complicated database schema and needing to get information between the two of them. I had started to hack together a script that would dump the data in using SQL, but I realised after the first few inserts that it would be a long hard slog that would be inflexible and complicated spaggeti code.

I didn't want to have to rewrite my object model significantly though, which meant I had to find an ORM library that would allow me to set up the mapping myself. I also needed it to work in Python, which then led me to SQLAlchemy. This nifty library comes in several libraries and is very flexible in the right ways to make it suitable for my needs (at least so far).

I'm still in the process of setting up the internal data structures. I'm needing to redo some parts of my internal object system but not significantly so, although I'm worried about it becoming a bit of a mess.

I'll comment on a bit more of the technical side of things in a few days when I've gotten more up to speed with the library.

SAX Parsing

After some time playing with minidom in Python, I finally looked at alternatives and discovered SAX. For my usual requirements (pulling data out of XML files into various internal data structures) this is so much of an improvement over minidom that I wish I'd come to it first, rather than deal with the mess that is minidom for reading XML. I'm still learning the ins and outs of SAX, but as a brief summary for those who know it even less than me I'll explain the basics.

SAX is an event-based XML parser. What this means is that it will read in XML and generate events based on what it has read in. Events would be something like the start of an element, the end of an element, etc. Your application then receives these events and can respond to them appropriately. This can entail a small amount of extra book-keeping in your application to know where you are in the XML tree (assuming that's important for your app), but it is so much easier to deal with than the multiple levels of looping through child nodes that is required to pull data out of an entire DOM tree.

My first problem with SAX (at least from what I know of it so far anyway) is that with a complicated XML file you can end up with an extremely large and non-cohesive handler for your file. To get around this I created a SAX content handler that maintains a 'processor stack' that allows you to split down your processing into more flexible, cohesive units. It isn't perfect (it has some ugliness when it comes to transferring data between processors currently, and adding a new processor is something that is hard-coded into each processor) but it makes for a nice system to me as I can split up my processing to any level I like, the processor stack acts as that extra book keeping I mentioned and it does so transparently. It allows for more flexibility in changing the structure and reuse of components.

All in all, I quite like SAX and I'll probably like it even more if I spend some time on my content processing to make it a more generic solution :)