Reflections on Python, testing, Continuous Integration and other things

It's been a busy few weeks, both in work and out (for out of work, see Shared Illusions). In work, and of a technical nature, I've been focusing on testing and Continuous Integration and I feel it's time to look back and see what has come out of it.

Firstly, what I've been using for this. The list is:

• Python
• Py.test - My first test package. Very nice and easy to use but abandoned for reasons explained later
• Nose - A nice package available for Python for running tests with a variety of options including profiling and code coverage (using coverage.py)
  
• Hudson - A well featured build server package with a plugin architecture and several useful Python related plugins

It should be fairly obvious from my previous posts that I'm using mainly Python for work code at the moment. I've gone on at length about various Python packages and my 'fun' with transforming a complex XML structure into an equally complex and completely disjoint object structure :)

So, onto Py.test. This was my first pick for Python unit testing and I'd written a fair few tests with it. I've now moved away from it, and this is mainly because it was a bit too simple for what I needed for CI. Basically, Py.test is an absolute gem for quickly writing and running unit tests. Simply prefix tests with 'test_' and run py.test in the project's root directory and your tests will be run and results printed out. The approach is great - it's simple, it's transparent and it's fairly stress free. It just doesn't provide some key features from Nose, namely the ability to output the test report in an XUnit XML file and code coverage integration. For a lot of people, these aren't necesarry, but to get the most out of a CI build server, they are very much desirable as then the build server can generate nice graphics over time for test failures and give much more accurate reports on build status. Therefore py.test was abandoned in favour of a more complete tool.

That tool was Nose. This is a more featured test runner for Python and it is pretty easy to transfer from using Py.test to using Nose (and also from using PyUnit to using Nose, if that's what you prefer). For Py.test to nose, you don't need to modify your tests at all, unless you have function/method level fixtures. If you have these then you need to decorate the methods that use them with @setupmethod(setup_func, teardown_func)... hardly a difficult change. Nose will then do the same thing as Py.test and discover all your test functions following a configurable nameing convention (which defaults to 'test_'). For this minimal change, you are now able to run your unit tests with a wider range of report generators (including the aforementiond xunit), profile your code and run code-coverage tools all with Nose plugins and simple command line switches. Well worth it if Py.test just doesn't quite provide the test reports you need.

Finally, I needed an automated build server. While python code isn't built/compiled, the other features of a build server were very much desirable, such as its source code repository monitoring, automated running of unit tests and a nice web interface to view test results and previous builds through. I've previously encountered CruiseControl and was less than impressed with it (not particularly easy to configure, lacking in some key areas, and the interface is a bit clunky). I went looking and found Hudson. This is an 'evil Java app' that runs through tomcat, but is pretty damn good at what it does :) It provides a much cleaner web interface than Cruise Control, it comes with the ability to create new projects through the web interface (I previously had to create my own project creation for Cruise Control) and has the ability to create build scripts within the web interface using Ant targets, Maven targets, shell commands and allows new options to be added with plugins. For all of that, I can stand the minor contamination of java testing my Python code, especially as I don't have to write any java code ;)

So, what was the purpose of all of these tools? Basically, they were to get my projects up to speed with automated unit testing. This is an area I've never managed to get fully on board with (it seemed a waste at uni and I didn't manage to get the build server set up for unit tests at EMCC before they went under) but it's benefits are starting to show themselves. While I don't think the current projects I'm working with will get full unit test suites (not enough time to do so, as they are primarily research rather than production projects), I now have the necesarry legwork out the way to provide much more comprehensive testing in the future and for the testing to be run easily and hassle free every time I commit a change.

Refs:
Hudson: https://hudson.dev.java.net/
Py.test: http://codespeak.net/py/dist/test/test.html
Nose: http://code.google.com/p/python-nose/
Python: http://www.python.org/