Wednesday, September 02, 2009

python build bots down... maybe they need a spectacularly adequate build page instead?

Seems the python build bot pages are down. Maybe something simpler is needed instead of build bots? Something that requires less maintenance.


update from svn
compile
run tests if compile completes
upload results to a simple page
(configure(stdout, stderr),
build(stdout, stderr),
test results(stdout, stderr),
binaries).


The beauty of this is that it can be easily de-centralized.

Could even use pypi infrastructure for this now. Each buildbot has a new project setup, which then updates the pypi project each time it builds. Have a special pypi tag(category) for python build bots, so people can easily search for them. To reduce the spam on pypi, just make it mark the releases as hidden... so they don't show up. The results are added to the pypi listing.


Probably only needs one cmd added to the python setup script to do the upload(based on existing pypi code).

No central authority, and very simple. Anyone who wants to run a 'buildbot' can without authority.


Can probably make it all work so the buildbot runner just does a special 'python setup.py buildbot' added to their cron, assuming they are already registered with pypi. So they don't need to setup a special buildbot, just use the existing python source code to do it.


work required:
  • write a script to add a buildbot command. To be added to python source code when finished.
    • uploads the results of the build/test into a .zip file to pypi. The zip contains stderr and stdout of each config, build, test stages. Also contains the resulting binaries. eg, .msi .exe .tar.gz etc.
    • Only update binaries if all tests pass.
    • Updates from svn, and adds the svn revision to the python version.
  • LATER - add pypi category for python build bot results.
  • LATER - write code to parse results from any uploads into that category into one main page with all results listed.
Instead of modifying the python source, this command could be made separately, and then added to python source after it has been proven. Adding it to python source however will mean that anyone can just do a 'python setup.py buildbot'. It turns all of the python developers... and anyone else who feels like helping into a buildbot.

As a bonus people could even use pypi to keep track of python, and not just python modules. It also turns python development into a series of continuous releases, stabalized when the tree decides to pass all the tests.


Something like this pseudo code would work:
 ./configure > config_stdout.txt \
2> config_stderr.txt && \
make > make_stdout.txt 2> make_stderr.txt && \
run_tests.py test_stdout.txt 2> test_stderr.txt && \
upload_build_results.py my_python_buildbot_pypi_page


Anyone else want to finish the script? To add building, and installing into a local directory, and the actual 'upload_build_results.py' to pypi etc.

7 comments:

Jesse said...

Illume; there's a few things wrong here. The first of which is the fact that we still need someone to maintain something, even *if* it is homegrown like the system you describe. Additionally, overloading pypi to act as a buildserver is not a good idea. Let pypi be pypi.

Also, we *need* some amount of authority in this system. We need "official" build slaves representative of certain platforms, and we need some amount of dedicated time on those slaves.

Something lighter-weight than say, buildbot wouldn't be a bad thing - for example titus brown's pony-build system (http://github.com/ctb/pony-build/commits/master) aims to be a simpler, lighter weight CI server.

But we *still* need dedicated boxes for platforms, and ultimately someone to be on the hook to maintain the infrastructure. Offloading it to infrastructure that wasn't designed for it isn't the answer.

what you describe however

illume said...

Hi Jesse,

it looks like the end of your comment was cut off.

If not pypi, then a google apps webhost, or something else.

I think you are probably right about not putting load on pypi. However lots of python developers have accounts there, so maybe using those somehow would be good.

However, even with no authentication the build/test results could be useful.

If you have 30 core developers making changes every week, if each of them does 'python setup.py buildbot' at the end of their checkin, then you have 30 maintained buildbots.

Not to mention X other people who decide to submit their test results. X amount of people could easily be 1000s of people who decide to update their test results.

Anyway... this is the approach we working towards with pygame. We already have something like it for the last couple of years. Except the machines are maintained by 1-2 very nice people over a few different projects.

The step with this next pygame release is to distribute the testing/building more so - to take authentication out of the loop.

Titus Brown said...

Rene, I can't find your e-mail -- could you drop me a note at t@idyll.org?

thanks,
--titus

Jean-Paul said...

I doubt that writing more software is the answer here. Lots of people run BuildBot (or Hudson, or one of the many other existing CIS systems). The issue seems to be that either no one cares, or the people who care are not being involved in the process. That's the problem to fix. Not the technology.

illume said...

@Jean-Paul: that's exactly the problem I'm describing to fix. To make it a single command for the not-care people to be able to contribute build/test results. If every time a core python contributor builds/tests the results are uploaded, then the problem is solved. If it's zero effort for people to do, then lots more people will do it.

After talking with Titus, I think his system will be able to solve just that problem.

cu,

Jean-Paul said...

Hey Illume,

I'm not sure about this. I'm considering the way things broken in this particular instance because it seems typical of how these things go. In this case, the entire system hosting the BuildBot build master was upgraded (I suppose because some hardware died and so the need to replace that was taken as an opportunity to change some other things - never a good idea, but there you go). As a result of that upgrade, the build master's configuration stopped working (I don't have the details of this, though - I just know that some people said the configuration needs to be updated before it will work again). Then, no one cared.

Does Titus's system solve the problem that humans can change systems in a way that causes them to stop working? If so, then I guess he has solved this problem. Otherwise, it seems not.

illume said...

Hello again,

if the build results are coming from working developer machines, then yes. It doesn't matter if half the developers machines stop working - at least some will still be able to upload results. The server part of it should be simplified so any web server that can serve up files will work.

The old buildbot system python used required separate machines to be configured to do the building and testing. The proposed system makes use of existing machines for building and testing.

Anyone who can compile python should be able to post build results.

Anyone able to run a build of python should be able to run the test suite - and submit the test results.