Sunday, April 12, 2009

Distributed testing - the easy way.

Distributed testing is easy.

Include your tests in your package, and then whoever can install your package can install your tests.

At pygame we have had much luck with including the tests with pygame.

Then people can just do "import pygame.tests;pygame.tests.run()"

It makes distributed testing easy, since now any of your users can run all the tests without having to have anything else installed(like source code, and a development environment).

Then you need one machine to package your build for windows, and then it can be tested on lots of other windows machines. (same with mac). The builds are uploaded to the testing machines(that don't need a dev environment set up), and then tested there.

Cherrypy and numpy(numpy.test()) also include tests with their downloads, and including tests is in the original unittest white paper.



Testing bots.

Then creating a network of test bots is trivial. Upload your package to them, or have them download it using http, and then get them to run the tests - then upload the test results to a central server.

Rather than them have to install a development environment - they can just download a pre-packaged version and run the tests.

Users are now equipped to be testers.

Now all of your users can also be testers. However they install your package they can run your tests. They don't require the source code to run the tests.

If they install with a apt-get install, an easy_install, a windows installer, or a mac installer, they can run your tests.

Someone posts a bug, you can make a bug fix, then ask them to download your latest pre-release and get them to run "yourpackage.test()". Saves you and the bug reporter time.

So rather than having 1-20 testing machines you can now have 1000's of machines testing your code.

Put your tests in setup.py install
Have your tests run at the end of your:
make install

python setup.py install


Make them fail if they fail your tests. Every time someone installs, your tests are run :)

Then have your setup ask the user to upload to your test result collection web page.

5 comments:

Michael Foord said...

Test on install is fine if you don't mind your users finding your bugs!

Actually it sounds like good practise - but refusing to allow an install because of a spurious bug sounds like it might cause frustration.

Michael Foord said...

I missed the 'Test Bots' part of your entry. This is similar to the way we do distributed testing at Resolver Systems - we have a farm of several build machines and push the whole application *and* test framework to each machine to run the tests.

We do central administration though, so that each machine pulls which tests to run from a database. That way each individual run completes quicker as the tests are distributed across all the machines.

Titus Brown said...

See cmake/ctest/DART -- VTK and ITK have been doing this for a while! It's a great idea and one I hope to enable with pony-build.

Jack Diederich said...

CPAN modules ship with tests and run them with 'make test' as a standard. As I recall you really have to run the tests before installing to make sure you have all the right dependencies.

Of course my perl expertise is about a decade out of date. Still, I'd be happy if python modules shipped with tests by default and 'setup.py test' ran them as a convention.

illume said...

@Michael: Yeah, failing to install on test is a bit brutal. However then at least the person knows it's all working ok. Perhaps only with beta/pre-release builds is a good alternative.

@Michael: another good thing for resolver would be if your customers make a bug report, you can maybe get them to write a test for you :) Or ship them a test/patch combination to make sure it works on their platform.

@Titus: yeah it's an old idea - so old it's in the original unittest white paper from 1989 - "I recommend that all classes respond to the message "testSuite", returning a suite containing the unit tests." || 'Simple Smalltalk Testing: With Patterns' || http://www.xprogramming.com/testfram.htm

cu,