Distributed testing - the easy way.

-
Distributed testing is easy.

Include your tests in your package, and then whoever can install your package can install your tests.

At pygame we have had much luck with including the tests with pygame.

Then people can just do "import pygame.tests;pygame.tests.run()"

It makes distributed testing easy, since now any of your users can run all the tests without having to have anything else installed(like source code, and a development environment).

Then you need one machine to package your build for windows, and then it can be tested on lots of other windows machines. (same with mac). The builds are uploaded to the testing machines(that don't need a dev environment set up), and then tested there.

Cherrypy and numpy(numpy.test()) also include tests with their downloads, and including tests is in the original unittest white paper.



Testing bots.

Then creating a network of test bots is trivial. Upload your package to them, or have them download it using http, and then get them to run the tests - then upload the test results to a central server.

Rather than them have to install a development environment - they can just download a pre-packaged version and run the tests.

Users are now equipped to be testers.

Now all of your users can also be testers. However they install your package they can run your tests. They don't require the source code to run the tests.

If they install with a apt-get install, an easy_install, a windows installer, or a mac installer, they can run your tests.

Someone posts a bug, you can make a bug fix, then ask them to download your latest pre-release and get them to run "yourpackage.test()". Saves you and the bug reporter time.

So rather than having 1-20 testing machines you can now have 1000's of machines testing your code.

Put your tests in setup.py install
Have your tests run at the end of your:
make install

python setup.py install


Make them fail if they fail your tests. Every time someone installs, your tests are run :)

Then have your setup ask the user to upload to your test result collection web page.

Comments

Michael Foord said…
Test on install is fine if you don't mind your users finding your bugs!

Actually it sounds like good practise - but refusing to allow an install because of a spurious bug sounds like it might cause frustration.
3:51 PM
Michael Foord said…
I missed the 'Test Bots' part of your entry. This is similar to the way we do distributed testing at Resolver Systems - we have a farm of several build machines and push the whole application *and* test framework to each machine to run the tests.

We do central administration though, so that each machine pulls which tests to run from a database. That way each individual run completes quicker as the tests are distributed across all the machines.
3:55 PM
Titus Brown said…
See cmake/ctest/DART -- VTK and ITK have been doing this for a while! It's a great idea and one I hope to enable with pony-build.
4:36 PM
Jack Diederich said…
CPAN modules ship with tests and run them with 'make test' as a standard. As I recall you really have to run the tests before installing to make sure you have all the right dependencies.

Of course my perl expertise is about a decade out of date. Still, I'd be happy if python modules shipped with tests by default and 'setup.py test' ran them as a convention.
3:31 AM
René Dudfield said…
@Michael: Yeah, failing to install on test is a bit brutal. However then at least the person knows it's all working ok. Perhaps only with beta/pre-release builds is a good alternative.

@Michael: another good thing for resolver would be if your customers make a bug report, you can maybe get them to write a test for you :) Or ship them a test/patch combination to make sure it works on their platform.

@Titus: yeah it's an old idea - so old it's in the original unittest white paper from 1989 - "I recommend that all classes respond to the message "testSuite", returning a suite containing the unit tests." || 'Simple Smalltalk Testing: With Patterns' || http://www.xprogramming.com/testfram.htm

cu,
6:57 AM
Post a Comment

Popular posts from this blog

Draft 3 of, ^Let's write a unit test!^

-
Image
So, I started writing this for people who want to 'contribute' to Community projects, and also Free Libre or Open source projects. Maybe you'd like to get involved, but are unsure of where to begin? Follow along with this tutorial, and peek at the end in the "what is a git for?" section for explanations of what some of the words mean. Draft 1, 2018/07/18 - initial draft. Draft 2, 2019/11/04 - two full unit test examples, assertions, making a pull request, use python 3 unittest substring search, "good first issue" is a thing now. Started "What is a git for? Jargon" section. Draft 3, 2020/12/12 - default branch name is main. What's first? A test is first. A unit test is a piece of code which tests one thing works well in isolation from other parts of software. In this guide, I'm going to explain how to write one using the standard python unittest module, for the pygame game library. You can apply this advice to most python pr...
Read more

Is PostgreSQL good enough?

-
Image
tldr; you can do jobs, queues, real time change feeds, time series, object store, document store, full text search with PostgreSQL. How to, pros/cons, rough performance and complexity levels are all discussed. Many sources and relevant documentation is linked to. Your database is first. But can PostgreSQL be second? Web/app projects these days often have many distributed parts. It's not uncommon for groups to use the right tool for the job. The right tools are often something like the choice below. Redis for queuing, and caching. Elastic Search for searching, and log stash. Influxdb or RRD for timeseries. S3 for an object store. PostgreSQL for relational data with constraints, and validation via schemas. Celery for job queues. Kafka for a buffer of queues or stream processing. Exception logging with PostgreSQL (perhaps using Sentry) KDB for low latency analytics on your column oriented data. Mongo/ZODB for storing documents JSON (or mangodb for /dev/null replacemen...
Read more

post modern C tooling - draft 6

-
Image
Contemporary C tooling for making higher quality C, faster or more safely. DRAFT 0 - 10/11/18,  DRAFT 1 - 9/16/19, 7:19 PM, I'm still working on this, but it's already useful and I'd like some feedback - so I decided to share it early. DRAFT 2 - 10/1/19, mostly additions to static analysis tools. DRAFT 3 - 10/4/19, updates on build systems, package management, and complexity analysis.  DRAFT 4 - 10/6/19, run time dynamic verification and instrumentation, sanitizers (asan/ubsan/etc), performance tools, static analyzers. DRAFT 5 - C interpreter(s).  DRAFT 6 - 11/6/19 , mention TermDebug vim,  more windows debugging tools, C drawing for intro. In 2001 or so people started using the phrase "Modern C++". So now that it's 2019, I guess we're in the post modern era? Anyway, this isn't a post about C++ code, but some of this information applies there too. No logo, but it's used everywhere. Welcome to the post modern era. S...
Read more