Saturday, December 08, 2007

Bleten. My game for the ludumdare 48h 2002

This is a game made to the theme of sheep. This game is about sheep poo, and grass.

You play the part of a sheep, against a friend sitting at the same keyboard as you.

You must eat the grass on the opponents side of the screen, whilst pooing on your grass so it grows quicker. It's a resource game really. You need to make the trade of between blocking the other sheep, eating the other sheeps grass, and pooing on your grass.

My Evil sheep picture for the intro. I think the sheep is possessed by satan. Maybe. Or maybe it's just angry all it's lambs have been taken to be eaten. Not sure why it's angry really.

As you can see I didn't have time to draw any good looking sheep - so there's just stick figure sheep. Stick sheep that poo.

You can download the source for python+pygame here.

Ludumdare is where people make games from scratch in 48 hours. A single person does all the artwork, code, sound and everything else in the game to a theme. It's what inspired pyweek except you can use what ever programming language you like, there are no teams - one person does everything - and it only takes a weekend, not a week.

Saturday, October 13, 2007

A Book "Beginning Game Development with Python and Pygame"

I haven't read this book about making games with python and pygame yet, but I think it seems like a good one for beginners. The free chapter is fun to read. Which is the essence of making games I think. It should be entertaining to make a game, and to read about making games.

This book looks like it covers most basics of programming a game - with python and pygame. Like it's title suggests it is for beginners, which is great.

PyGPU - hardware graphics programming using the GPU and python.

Have a look at PyGPU

With it you can code the GPU using python. It translates code from python into GLSL (the opengl shading language).

It's amazing what type of algorithms you can quickly write using the GPU these days.

There are papers describing it's implementation and other information here: It takes a similar approach to psyco or pypy... in a way. But is different from both of these.

Very cool.

Saturday, October 06, 2007

Girl start project teaching girls programming with pygame.

I saw this on the pygame mailing list the other day about ...

"... Project IT Girl, a program that teaches 60 high school girls in Austin, Texas, about how they can change the world through the use of technology. This year, Project IT Girls learn basic programming through designing and developing their own unique educational games written with Python and **Pygame**! ..."

Looks like a cool project.

Ya! A Galcon T shirt.

After the Galcon T shirt design competition, Phil has decided to send me one of the winning designed shirts for helping out with testing.

You can see some of the winning designs here: Galcon t shirts.

Galcon is a multiplayer, and single player game kind of like risk, but with space ships. It's made with python + pygame by Phil Hassey.

Thursday, September 06, 2007

-- Ninja out.

Saturday, September 01, 2007

python 3000 breaks hello world

$ ./python 
Python 3.0a1 (py3k, Sep 1 2007, 14:48:21)
[GCC 4.1.2 20061115 (prerelease) (Debian 4.1.1-21)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> print "hello world"
File "", line 1
print "hello world"
SyntaxError: invalid syntax

Weird. I've been trying to learn this python programming language, but the first command in my lecture notes, the three textbooks I have, and all the online tutorials seem to be wrong.

Anyone know how to get hello world working?

Update: oh it seems that print is a function. ---> ;)

Maybe it should show this in the shell, kind of like what typing help does:

>>> help
Type help() for interactive help, or help(object) for help about object.

>>>print 'hello world'
Type print('hello world') to print a string, or print(object) to print an object.

Has any other major programming language ever broken a program as simple as hello world before?

Thursday, August 30, 2007

PyWeek #5 theme voting on now! 84 entries so far.

84 entries so far for the pyweek game development competition. Some entries are teams, and some are solo efforts.

The themes for this pyweek game development competition are:

  • Twinkle
  • Turmoil
  • Ticking
  • Twisted
  • Tyger, Tyger

    So join up, and vote for a theme. The Pyweek game competition starts soon.

    Friday 2007/08/03 Registration open
    Sunday 2007/08/26 Theme voting underway
    Sunday 2007/09/02 Challenge start
    Sunday 2007/09/09 Challenge end, judging begins
    Sunday 2007/09/23 Judging closes, winners announced
  • Friday, August 24, 2007

    Delight at my new-old recycled laptop

    I don't like to contribute too much to all the pollution that getting new hardware contributes too. There's lots of perfectly good old hardware being given away or sold at low prices in second hand stores and online auction sites.

    A couple of months ago I got a Dell latitude C610 laptop from a place down the road from me that restores and sells old laptops. With only 20 gigs of HD space, and 256 megs of ram it's not anything close to what you'd buy new. However it seems to do ok.

    I also like to use older hardware for developing software - because it's like 'eating my own dogfood' in a way. If I see the performance problems that the slow computers experience - then I can fix them up. All too often I see websites that perform slowly on old machines, or don't fit important information on their small resolution screens.

    My franken box

    The laptop is so nice, that I've mostly retired my old duron 850mhz desktop machine... which I've had since 1997 or so. Well, some parts I have had since then - it's changed power supplies, motherboards, cpus, hard drives, network cards, sound cards, memory and video cards. Over time I've been upgrading parts of it as I found new parts. When people throw away their old graphics card, I replaced my old graphics card with a better one. The same with all the other parts. I don't think any of the parts in it are from the original 1997 machine any more, but I still think of it as the same franken-machine.

    This is the first time since 1997 that I have stopped using that hardware as my main machine. I still have my original home directories though :) I'm also using the mouse from that machine, and a few usb parts... so I guess the franken-box continues! My franken monster of a machine lives on :) I wonder if 20 years from now I'll still be using parts of that original machine configuration - with files from that original DebIan 1.3 install?

    Ubuntu failed to install for some reason - so I stuck with Debian. With Debian installed, everything on the laptop is working. I can even suspend to disk, or ram. The on board sound isn't the best, but that's ok. Even the function keys work for adjusting the volume, and changing the LCD brightness/contrast. It doesn't have built in wireless, but came with a usb wireless adapter. I have a few old pmcia wireless adapters lying around too that I could use (and did when I was in europe last). I never use modems these days, so I'm not sure if the on board modem works. The video card works quite well, with even basic opengl hardware acceleration seeming to work fine. All without the hassle of using a binary driver - the open source video driver works fine.

    Using my old desktop machine over the network - from my slow laptop.

    Firefox is a memory hog, sometimes using up more memory than is even installed in my laptop(256 MiB). So I run firefox now from my old desktop machine over the network.

    Just running firefox from another machine has made this laptop *heaps* more usable.

    btw, it's easy to run X programs from another machine. Just ssh -X host, the -X enables X11 forwarding. It's a tiny bit more laggy over the network - but definitely better than using all that swap.

    I only started doing this a couple of days ago. The downside is that I need to sync a couple of directories on my old desktop with my laptop. The .mozilla directory, and my file download directory. So when I save files with the browser running off my laptop I can access them easily from my laptop. A couple of scripts accomplish this easily. A next step is to automate this in the background. So I don't have to manually run the scripts. I think a watch type script which runs on my old desktop would work fine - or a file event using cron. Although I think just mounting my laptop home directory from the desktop box could be optimal - with maybe the firefox cache dir being mounted on my desktop box(or I could just make the disk cache at 0, and increase the firefox memory cache).

    Another problem with running firefox over the network is sound. Sound doesn't seem to automatically work from firefox when running it over the network. I haven't figured this one out yet - but probably some sort of sound network daemon is needed. I don't think firefox supports network sound by itself. I'll have to investigate network sound...

    Another downside is that I need my desktop machine on whilst I use my laptop. This obviously uses up more power :( Maybe even being worse for the environment than just buying a newer, more efficient laptop. However I only use it during my work day - when my requirements for the computer are higher. So it goes to sleep at night, or when ever I leave the studio. It also has the HD going to sleep, and it often runs in low power mode.

    Network boot would be nice... so I could somehow just get that machine to boot from my laptop... as long as I can get a connection to the network. I haven't found anything that'll allow me to do this from a wireless connection yet. Maybe if linux was running on my router I could do it. Get my router to network boot my desktop box. In the mean time I can just get off my bum and walk over to it.

    I still use my old desktop box for backing up stuff, and will probably use it for any programming that requires lots of disk space. It's also handy for testing out load balancing for web apps, and other programming tasks.

    I plan to upgrade the ram on this laptop at some point... but at the moment 512MB costs around $200AUD. A tiny bit less than what I bought the laptop for. Probably worth it though.

    linux 2.6.23 is looking to be a big improvement for desktop use.

    I'd like to try the swap prefetch linux mods... as well as the memory compression linux mod. I think they'd speed this machine up. It's looking like linux 2.6.23 will have a few desktop fixes that help a lot - the CFS scheduler and swap prefetch.

    However I think the compressed cache work has stalled since the last bit of work done on it for the google summer of code - There's some interesting research, including benchmarks there for the compressed cache. It makes perfect sense a compressed cache would be faster since hitting the HD is really slow, and compression/decompression is really quick on modern CPUs. There's only a patch for 2.6.21 so I haven't tried it out.

    Here's some benchmarks someone made which measures responsiveness by window moving for the new linux kernel...

    terminal window visible: real time was 59 seconds.
    terminal window covered: real time was 32 seconds.
    terminal window shaded: real time was 135 seconds.
    terminal window iconified: real time was 160 seconds. with latest CFS patch applied:
    terminal window visible: real time was 27 seconds.
    terminal window covered: real time was 13 seconds.
    terminal window shaded: real time was 21 seconds.
    terminal window iconified: real time was 21 seconds.

    compare to CK's -
    terminal window visible: real time was 25 seconds.
    terminal window covered: real time was 13 seconds.
    terminal window shaded: real time was 22 seconds.
    terminal window iconified: real time was 22 seconds.

    As you can see the 2.22.5 with latest CFS patch seems to be quite a lot more responsive than the vanilla kernel. I've got a feeling it's going to be as big a performance jump as the move from linux 2.2 to 2.4. Well I hope so anyway :)

    Wednesday, August 22, 2007

    plone 3 released

    Plone 3 has been released!!!

    Plone is not beta like those other toy frameworks (django, pylons, paste, turbogears). I guess there are more python web frame works out there after all than some people would like you to believe ;)

    kidding, kidding... don't eat me.

    Should be fun to play with... I haven't used it since the 2.x series - so I'm looking forward to seeing all the improvements. Congratulations to the plone team.

    Pygame weekly mini sprint 2007/08/22

    This week we found and fixed a long time problem with the SRCALPHA flag. Which is the flag used for per pixel alpha (transparent images).

    In fixing that it turned up a few other issues with Surface. There were problems with error checking, and keyword arguments didn't work.

    So we got the error checking working, as well as keyword arguments.

    The major piece of work that went in was the PixelArray from Marcus. This will be what we use to replace surfarray. A PixelArray is like a basic numeric/numpy array, and like what you'd expect returned from a surf.tobuffer() call. With it we will be able to support Numeric, and Numpy by loading them dynamically.

    I still need to write the which will replace the compiled surfarray. There will be a surfarray_numeric, and surfarray_numpy. So it will be backwards compatible, and you'll be able to use the array type you choose. We will include a frombuffer() method for Numeric arrays - which won't require Numeric to be installed to compile. Numpy already has a frombuffer method - so that shouldn't be too hard.

    PixelArray will also support basic slicing functionality like numeric - so you can do things like array[x:y, a:b:z] = ... etc. This is what Marcus is working on next. So there will be basic effects you'll be able to do even without Numpy, or Numeric. As well as being able to send data to PIL, opengl, wx etc without using Numeric. So you won't need the extra dependency for these common use cases.

    In other things I started to play a little bit with the upcoming SDL 1.3. This is the release with opengl and direct3d hardware acceleration, multiple windows, recording sound - and many other goodies. It's not finished yet, but is already fun to play with. Pygame will probably start to support it only when the final version is released - as some of the new APIs could still change. There is a SDL 1.2 compatible API - you just can't access any of the new features with the old API (obviously :)

    pygame 1.8 will take advantages of all the improvements going on in the underlying SDL libraries. All the different users or different languages - C++, ruby, python all feed back into the C SDL. Lots of different engines use it, and heaps of game developers. So it gets *heaps* of users and testing - as well as optimization. Lots of linux distributions test it, as well as people releasing games on different platforms. Pygame 1.8 on the mac will have a whole bunch of altivec optimizations. On windows, and unix there have been a bunch of mmx optimizations too since the last pygame release(1.7.1). The image, sound and font modules have all gotten fixes, speed ups, and bug fixes - improving all of their quality. So not only will pygame get the pygame specific improvements - but it gets the improvements from SDL too. Read the SDL release notes for more details.

    Over on the pyweek site someone has made a python wrapper for the 2d physics library chipmonk. Check out the youtube videos of the physics... Domino pyramid Domino smash. It might be a good thing to add to pygame in the next release after pygame 1.8.

    The other week the SDL_gfx author mentioned that he was interested in collaborating with the pygame people some more to get more of SDL_gfx into pygame, and to move the pygame improvements back into SDL_gfx. The new improvements to SDL_gfx include bezier curves, more antialised shapes, and a textured polygon drawer.

    Sunday, August 19, 2007

    javascript for templates - it's happening.

    It seems like javascript for template languages might be the way forward.

  • designer friendly. lots of web designers know a little javascript or actionscript.
  • can run in a browser.
  • javascript can be sandboxed.
  • most web developers know a little js too. Web developers that don't know javascript will most likely at know at least a little C/php/perl/java to be able to do a lot of things with javascript.

    Tenjin allows javascript templates. It also allows many other scripting languages to be used in templates... but that's not really what I'm talking about.

    Fast, small(200k), opensource and widely deployed javascript/emca script interpreters exist. Tamarin is a emca JIT optimized virtual machine from flash 9, that mozilla is using in upcoming versions of firefox. There is also spidermonkey - the current firefox javascript implementation. Finally there is haxe, which is a emca script like language - that can output .js files, flash swf files, and also neko which runs as a virtual machine in apache. So it would be possible to even use your templates in flash, as well as in normal html browsers.

    However you don't need to use javascript in your templates in order to convert the templates into javascript. Genshi2js can convert genshi templates into javascript, so you can run your templates in a browser, or server side. This shows that you don't need to use the javascript language in order to gain the benefit of using your templates in the browser.

    Language agnostic template languages are important - because a wider selection of people can edit your templates. Javascript usage in the template allows you to get that wider selection of people being able to edit your templates. The genshi2js solution is quite nice, and it proves that you can still compile other templates languages to be executed in the browser - however it is not known by as many people as a javascript.

    An exciting possibility is for applications which serve dynamic pages via haxe->mod_neko with compiled tenjin templates. This would seem like a very fast way to generate dynamic pages from json data sources. The json could quite easily be generated in any language - including python, static files, or any webservice through mod_proxy.

    I'm sticking with python, and php on the server side for the time being - not switching to emca script server side. Python because I like coding python, and php because it is widely installed (and there seems to be more work available). However using javascript for the templates might just be an option I can live with. Especially if I can compile the javascript using templates into php, and python.

    It all depends on the project of course. But I'm definitely considering genshi (with genshi2js), and tenjin for future projects.
  • Friday, August 17, 2007

    Don't trust database input either.

    The database is just another form of input into your program which you should not trust.

    You should validate the data coming from your database as much as you do validating the data going into the database.

    How do you know the database has not been corrupted, or compromised? Or some script on your website is not validating data properly when it updates the database. Or an DB admin decides to edit the database directly and puts in some invalid data.

    What if someone a year from now hooks up another program to the database, which doesn't use the same data validation that you do? Then your program not validating input data from the database will start to fail.

    There are many ways data from the database might not be what you are expecting. Not including people putting data in there maliciously. Like if they somehow get your database password. Or if they find an SQL injection.

    Executing code from the database surely sounds a bit crazy... right? People execute code when they use python pickle to store data. If for any reason above someone can change your database, then you are allowing them to execute code. So if you are storing pickles - please stop being a mental patient, and change to using a safer form of serialisation.

    If you have validation functions, it's fairly easy to reuse that code for validating data from the database. Especially if you are using ORM, MVC, or have a central get() type function.

    Unfortunately validating data from the database seems to be a bit too cpu intensive for some people. That, and it's not as big a risk as many other inputs.

    Thursday, August 16, 2007

    collections in python - using less memory

    Each object in python takes up far more memory than you might think. An int object for example does not take up 4 bytes.

    So creating python objects for each element of a collection of data can use up far more memory than is needed.

    A simple pattern for avoiding this wasted memory is to store the data in a array.array() then construct an object from the part of the data as you need it.

    Using python classes to store an int.
    Virtual 23992
    Resident 21176

    Constructing python classes to store an int dynamically.
    Virtual 6572
    Resident 3992

    As you can see this method can save a *lot* of memory.

    Here's some basic code demonstrating this technique... this isn't necessarily the API to use, but just demonstrates the memory savings. You can make a nicer to use API on top of that... or use your existing api with get magic properties.


    # using python objects...
    python 100000 -object & sleep 2 ; ps aux | grep python
    # using a collection of python objects, which constructs a class from the raw data.
    python 100000 & sleep 2 ; ps aux | grep python

    Many uses of python could use this technique to save lots of memory. Anything that operates on a large number of python objects.

  • Python database drivers are one area which could use this technique.

  • Sprite classes (like pygame sprites) could use an array of underlying data to store all the attributes.
  • Wednesday, August 15, 2007

    Pygame weekly mini sprint 2007/08/15

    There's been a few new things going into pygame recently.

    Today two things came off the 'todo before release list'.

    The first was the pygame.mask module was finished - the remaining from_surface function was implemented. It is 128x faster than the version written in python. It could still be optimized more, but I think it should be fast enough.

    The second was the new sprite code from DR0ID. This has been a long time in development, and allows some pretty useful functionality for the pygame.sprite module. Like support for layers, blend modes, as well as automatic detection of what is faster - full screen update, or dirty rect update.

    Now the 'todo before release list' is a lot shorter:

  • windows+mingw compilation instructions
  • remove current C based surfarray which uses Numeric. Replace it with a PixelArray C type. Then implement Numeric and Numpy support in python.
  • Mac OSX scrap fixes - using the new scrap api for clip board support.

    Marcus has the basics of the PixelArray code done, and we have figured out a way to get it all working with numeric, and numpy. It's kind of like implementing a Surface.tobuffer() function - but more general purpose. We are aiming for a situation where people can do quick transfers of data to libraries like PIL, opengl etc without requiring Numeric, like is currently required. We also want to support Numeric, and Numpy without requiring them at C compilation time.

    The mingw compilation instructions are already on the wiki... but are not 100% complete. We want to make it so people can basically do python install. Right now there are 100 or so steps to do before that will work. We want to be able to download mingw, download all the source dependencies, and download all patches required. Then build and install all the separate pieces. Each step will be optional, so for people who have mingw installed, or SDL installed will be able to skip those steps.

    Hopefully then we can get more people on windows able to modify pygame if they need too. Without spending a week trying to get it to compile.

    On the documentation front I'm thinking of a few things to improve. The first is
    a pygame glossary. Where terms that are used in pygame, graphics, and game programming are explained. At the moment there's only one term described... 'Dirty Rects'... but hopefully more will be added over time.

    The next one is being able to type in module, function, and classes after the url. Like for example: -> . It won't do any searching (for now), but will just be a little easier to find the docs online.

    The other documentation thing will be something like the pyopengl api guide. For each function and constant it has a link to the use of it in the Demo programs that come with pyopengl. This is really useful when trying to see how a function is used.

    Here's the top of the WHATSNEW file from recent checkins.

    Aug 15, 2007
  • The sprite module has had some big changes from DR0ID. It now has a LayeredUpdates, and LayeredDirty groups. For using layers when rendering the sprites. LayeredDirty is an alternative to RenderUpdates that automatically finds the best display method (either full screen updates, or dirty rect updates). It's faster if you have sprites that don't move. Thanks DR0ID!
  • Added pygame.mask.from_surface which can make a Mask object from a surface. It's 128x faster than the python version!
  • bug fix. Thanks Lenard Lindstrom!

    Jun 25, 2007
  • Removed QNX support from scrap module. Thanks Marcus!
  • Added smoothscale(with MMX!) function from Richard Goedeken

    Jun 27, 2007
  • Fixes from Marcus for ref counting bugs.
  • Also using METH_NOARGS for functions with no arguments. Which should make some functions slightly faster. Thanks Marcus, and thanks Campbell Barton for spotting them.

    It turns out there's some SSE instructions in the mmx scaling code - so they need to be fixed too. I think Lendard Lindstrom has those fixed though - but it still needs testing.
  • Friday, August 10, 2007

    timing and unittests - graphing speed regressions/improvements

    Are there any python tools which allow you to run timing tests inside of unittests, and then see a useful report?

    I'd like to see a report of how long each timing test took, and also see differences between runs. When comparing two runs I'd like to see visually which is faster.

    Timing information is a very important thing to time for gui applications like games, and websites. As so many machines are very different, it's useful to be able to time things as they run on different machines. A game or website can quite easily run 10-30 times slower even on machines with the same CPU. Many other factors like OS, hard drive speed, available memory, installed drivers, directx/opengl/X/frame buffer, different browser speed, different installed versions of libraries or plugins like flash. Testing all these things manually is almost impossible, testing them manually every time something changes is definitely impossible.

    So I want to use this tool pygame unittests specifically, and also for websites. If there's another testing framework which can do timing very well then we could change testing frame works away from unittest.

    So I'd like to be able to save the timing data to a file or a database, and select the runs I'd want compared against.

    Extra points for being able to do that automatically for different revisions of subversion. So I could tell it to checkout different revisions/branches, run the, run the tests, then show me a report. But that's something I could script later I guess.

    It would be nice if the tests could be separate processes, so I could do something like:

    def some_tests():

    flags = ['1', '2', '3', '4', '5', '6', '7']
    for f in flags:
    my_f = lambda x: os.system("python -doit=%s" % f)
    do_test(name = "a test for:%s" % f, my_f)

    So it would just time how long each one takes to run as many times as it needs.

    It'd be nice if I could read what the process prints, and provide a parsing function which can extract timing information from the output. So then when something prints "33FPS" or "blit_blend_ADD 10202094.0093" I could tell the timing framework what those mean.

    Where script would be timed automatically, and I could see results for tests like "a test for 1", "a test for 2" etc.

    That way I could more easily reuse existing timing/benchmark scripts, as well as time code which is not python - like any executable.

    It would be nice to be able to keep the timing tests with the other tests, but be able to disable the timing tests when needed. because, it's likely some timing tests will run for a while, and do things like open many windows (which takes time).

    Website collection would get bonus points. So it could collect the timing information from any machine, so I can compare them all in one place. Since I want to run these timing tests on different machines, and allow random people on the internet to submit their timing tests. Since I don't have 1000's of videocard/OS/cpu/hard disk combinations - it would be useful if anyone could submit their test results. A http collection method would be useful for storing timing information from other languages, like from flash, javascript, or html too.

    Looking at average, mean, fastest run, and slowest run would be good too - rather than just average.

    Merging multiple runs together so you can see the slowest results from each run, or the fastest from each run would be good. Then you do things like combine results for 1000 different machines, then find out the machine configuration for the slowest runs. If there common slow downs then you can try and see if there are any similarities for system configurations. Thus allowing you to direct your optimization work better.

    Being able to run tests with a profiler active, then store the profiling data would be nice. This would be for different profilers like gprof, or a python profiler. So if you can see a profile report from a slow run, and find out which functions you might need to optimize at an even lower level.

    Allowing different system configuration detection programs to run would be good too. So you can see what HD the slow machine was using, how much free ram it had, the load on the machine at the time of running, etc,etc.

    All this tied in with respect for peoples privacy, so they know what data they are supplying, and how long it will be archived for.

    Support for series of time would be nice too. Like giving it a list of how long each frame took to render. So then I could see a graph of that, then compare it over multiple runs. Then you could look at data from many machines, and compare how they are performing, as well as compare different algorithms on different machines.

    Hopefully something already exists which can do some of these things?

    What I've written above is more a wish list, but even a portion of that functionality would allow people to more easily test for performance regressions.

    Wednesday, August 08, 2007

    Pyweek 5 - make a game in a week

    pyweek registration is open. For the biannual game jam.

    Which means you can join, and put your self in a team, or join up as a solo entrant.

    Spend a week(part time) finishing a game using python. Sunday 2nd September to Sunday 9th of September.

    It is inspired by the ludumdare 48h comps, but people only use python, it is a week long, and there can be teams. Over 100 entrants joined in on the fun previous competitions.

    Enter to have a chance to prototype your next game, or see if working in a team will work on a small project. Or just have a break, to get your creative juices going, and to feel all of the energy of 100+ people simultaneously feeding off each others creations.

    It's also a great way to learn, and have fun with python. It's possibly the best way to improve your programming skills, and game making skills there is.

    Thursday, July 26, 2007

    Git problems... when moving.

    There is a good explanation of the problems people have because of gits lack of move support. This is Linus explaining this problem he has with git.

    The problem is that git gets confused when you move code from one file to another, and change it at the same time. Since it uses code similarity to find moved code - changing the code at the same time that you move it confuses it's move detection code.

    So the solution is a convention of making almost no code changes when moving a file. Delete the file, and create your new file with no changes, and merge - then make your code changes in the new file.

    See the problem with git and move for more details.

    Of course systems which don't use move detection by code similarity completely fail when people delete a file, and create a new file(eg. bzr).

    So both git, and bzr fail when you forget to follow conventions. An ultimate system would do both - detect code moves automatically, and allow you to do explicit moves.

    Update: Jay Parlar comments that you can do explicit moves with git. By using the git-mv tool.

    Saturday, July 21, 2007

    Why urlencoding is a good 'format' for ajax.

    urlencoding works for most languages. eg, javascript, flash, python, php. So you can use it in a limited sense to encode stuff for transport.

    eg. a script could return this:

    Streaming is the cool thing you can do that you can't really do with json, or xml. Well you can, but it's a tad harder. Decode/encode is really quick for urlencoding, and can be slightly better than json/xml.

    This is an old trick that's been used in the flash world forever, but you can do it with js too(but no one seems to). Since not many people seem to be doing it in js, I thought I'd share the technique. Json is probably a better encoding to use most of the time, but this method has it's advantages.

    With the partial data you download you can try to urldecode it. If you put markers in the data, then you can check up to that point.


    at data == "a=3&r=hello+th"
    You can tell that a=3 is correct, but not what r equals. You also no you aren't at the end, because the end=1 part has not been reached.

    So your code can do this:

    data = get_any_data_available()
    decoded = urldecode(data)
    while decoded['end'] != '1':
    # do stuff here.
    data = get_any_data_available()
    decoded = urldecode(data)

    There's other tricks like continuously reading from an open connection. Your server side script can spit out new stuff as it wants - and your client side can just keep trying to decode the new data looking for marker points like the 'end=1' in the example above. An example is a chat script - which just prints out changes from the db as they happen - when a trigger fires it prints out the changes, then goes back to sleep.

    Filling up tables with data is useful using streaming. Say you have 200KB of data to make a table with, creating the table as the data comes in allows you to show the data quicker. Because you don't need to wait for the entire 200KB to download before you start constructing the table in html.

    In a similar way to the table filling example, building other parts of html is quicker with streaming. Because you don't need to wait for it all to download.

    You can encode whatever you want in the url encoded variables... including json, or xml - or just plain strings.

    For doing a sequence of things I usually use numbered variables: eg:

    Which could be translated into: data = {'a':[2,22], 'b':[3,33]}

    As you can see urlencoding can also encode data with less wasted space than json, or xml.

    Yes it's hacky, wacky and weird - but works quite well for some things.

    Revision tracking of functions is more important than file names.

    With git, and bazaar two differences are that one tracks code moves fairly well and one tracks file moves well. One was originally written mostly by someone managing a code base, and the other is originally written by a company doing an operating system. So you can see their priorities - one has lots of code it cares more about, and the other has a lot of files it cares more about. (There are quite a few other priorities and features that each have).

    I guess tracking the code itself is much more important than tracking file renames - for me. Much more useful anyway.

    However *both* are important parts of revision control.

    Many times during development I might cut a function/class from one file and move it into another. I can't get reports of this information in a meaningful way with bazaar (or can I?).

    Also I might accidentally move a file with the command line tools, change it, then add it in again to the revision control system. This is because I forget that I need to use the revision controls system of moves.

    A system that can automatically track *code* moving around, then can see that a file has been moved/renamed. It shows you that the code has moved, which is often more important, than showing you the file names have moved.

    However the system that can track file renames *when you tell it* can not automatically see that I moved a function from one file to another. Or see that I split a big file of code into separate pieces - that is I moved one file into five.

    Well it could, and I think that's important. I often care about what has changed at a module level, but more often care about what has changed at a class level, or a at the function level. I even care about change sets at the module, package, and application levels.

    Full text search could help dramatically here, however having the tools built into the revision control system is much handier - and probably quicker.

    So I think you should be able to see that the 'bazaar is lossless' argument is not true... or is it? I think bazaar is still lossless, it just doesn't let you get at all that information yet - just the data.

    In the future it would be possible given a bazaar tree to find where a function has lived, no matter which file it has lived in.

    Caring about what is in the files, is more important than caring about the files themselves. So these tools that work with revision control systems should be able to say things like

    'svn log my_superfunction'

    and see a log of changes to my_superfunction no matter where it lives. Perhaps giving it a hint of which file to look in, or even provide the full text of the current function.

    However would this mean that the revision control system would need to know more about the content it is storing? I think a lot of it could be generalised, like 'this is a block of code' could be defined as a C++ class, a C function, or a python function. These tools could probably even be built outside of revision control systems. There's probably some tool that can be used for this already?

    What about the case where code moves from one repository to another? This is a common case in a lot of projects. It'd be nice to be able to tell your RCS that you copied this function from project X with the repository XX. Then have your same commands for looking at history automatically work. I think bazaar - with launch pad - is interested in doing this type of stuff, and probably GIT too. I think with bazaar you can already do this - tell it you're merging from a separate svn repository, and it'll copy all of the history. Well, for files anyway, not sure about separate functions. Mostly when you move code from one project to the other, the file names will not be the same.

    It's all very interesting reading about these different systems, and the thought that goes into them. Mostly though, I'll probably still just use add del update and commit - with hopefully a bit more of push, and merge. I'm using bazaar more and more now, along with svn too.

    Friday, July 20, 2007

    My issues with python ORMs

    If a python ORM you know of addresses these issues, please let me know.

    Python ORMs break with multiple processes.

    Multiple processes are not assumed. Python ORMs all seem to use heavy local caching, which fails when the database is modified by another process. This is unfortunate as for me I like to use different tools for different jobs. Or there might be different people I work with that write tools in different processes. Or even the common case that each web request is run in a different process - or on a different machine.

    It is not commonly known that just because something outside of a python app changes a database that the python app will break. Most applications that use databases do not break if the database changes from outside of the application.

    Using memcache or something like it seems to be a solution to some of this problem. Also optionally allowing the ORM to not cache certain queries - or even stopping all caching. Caching using python dicts is bad anyway, because of pythons memory wastage.

    Constructors are limited to creation.

    Insert seems to be assumed to be the most common operation. So python ORMs seem to only allow you to do inserts with the constructors.

    Having constructors which get a row based on a primary key would be nice. Since that is one of the most common things I do.
    eg. 'p = Person(1)' would get the person with primary key 1.

    A row is not the only use of mapping.

    Even being able to do no select, insert or update in the constructor is useful.
    eg. 'p = Person()' would create a person instance which is so far empty. I use this to shortcut things I want to do with that table - not on a row of that table.
    So I can do things like this:
    eg. 'rows = p.get_all("active=3")' It's just shorter for me to type this stuff, easier to remember how to do things, and is easier to read(I think).
    eg2.{'id':1, "name":"george"})

    By using methods on a class which says 'I am acting on this table' you can shorten a lot of code. The two examples above show how much shorter, and simpler select and save are. The methods of the instance don't necessarily act on the row that instance might represent - but on the table that instance represents.

    Having to pass each attribute by name.

    I like to do code like this:

    Not this:
    t = Table(a=2, b = "asdfsadf", g="123123")

    Where vars is a dict that has all of the things I want to save. If vars is a dict from a webpage where you use POST variables, then you should be able to recognize time savings. Also if you update the fields in a table you don't need to update the code here.

    That way it can either do an insert or update depending if it already exists or not(determined by a primary key often).

    The niceness of this is that I don't need to recode the save part depending on which variables the table takes. Also if there are extra variables passed in they are ignored. I don't need to write each individual attribute for each table.

    A workaround would be to override the constructors to do this. But this behavior is not built in by default - and most python ORMs use the python constructor to pass in arguments.

    Using python constructors you get a TypeError if you pass it unexpected variables. So for me that is a major problem for using the python constructor for this, and not having a separate save, insert or update methods. The python class creation semantics are not exactly what you want at all times.

    Issues can be worked around.

    These last issues are caused because I want a python relational mapper, not really an object relational mapper. Python doesn't force you to use objects for everything - neither should a relational mapper for python. Well, maybe it should - if the authors want that.

    I work around these issues myself, so I can build my favourite API on top of existing python ORMs. But I'm wondering if I missed some solutions to these issues that are already in the python ORMs? Or why people don't see these as problems?

    Wednesday, July 11, 2007

    europython 2007 - batching apis as applied to webpages


    Here's my paper and code for combining multiple images - and other things.

    Monday, July 09, 2007

    europython2007 - Taking advantage of multiple CPUs for games - simply

    Here's my Multiple CPU paper I am presenting tomorrow morning at europython. The pdf on the europython website is a bit old.

    Hope to see you there! ... if you're not too hung over.


    Taking advantage of multiple CPUs for games --- simply, is the topic of this paper. Using a simple interface that many are already familiar with --- Pythons 'map' function. It talks about how to avoid the Global Interpreter Lock (GIL) limitation in Pythons threading libraries. As well as how to choose which parts of a game to thread. Finally it shows how easy it can be, by converting a full pygame to make use of multiple CPUs --- and benchmarking the game.

    snakes on a phone - europython 2007

    This talk was about Python on Nokia s60 phones.

    These are my notes...

    Discusses implementation details - like what they had to change to get it working nicely. He also discussed what you can do with the phone - which turns out to be most things. Like accessing the camera, and sending/receiving sms.

    It's based on python 2.2, but has had some things back ported. Like the pymalloc. He said at some point they might update the python to a newer version. The schedule is 'The future - a closer future than before.'. Then of course before that he said 'the future is now' - probably those two statements aren't related, and were taken out of context.

    Including not using writable static data, and the modifying interpreter for the security model on the phone. The security model includes using capabilities, and signing executables. There are different levels of signing certificates. The basic ones are free, but others require contracts and money.

    By default as python can run scripts the default capabilities. So the capabilities for the python are limited. You can sign it yourself if you want to though.

    Even then it won't let you do somethings to your own phone. You'll need a manufacturer certificate to do some things.

    It's interesting to compare the pocketpc python which you can run on windows mobile phones, and the nokia one. The pocketpc python seems to be a newer version - but the nokia python is supported by nokia employees. I'm not really sure which are integrated better with the phone.

    He demonstrated using the phone via blue tooth. So he could have the interpreter running on his laptop, and running the code on the phone.

    import camera

    Guess what that code does? That's pretty easy!

    import messaging
    messaging.sms_send('+3112342134234', u'a message')

    import telephone
    telephone.say(u'hello europython')

    It all seems pretty cool, and makes me want to pick up a second hand s60 phone. I guess the next step will be for someone to make portable code so you can run the same python code on different mobile phones (pocketpc, s60, linux, etc).

    He talked about trying to get their changes into the main python. Including things like reducing memory usage. Reducing memory usage would also be great for servers too - the less memory you use. So hopefully some of those changes can go back into the main python.

    Friday, July 06, 2007

    europython 2007 - hello from Vilnius


    I'm sitting at my hostel in Vilnius preparing for my presentations.

    I've managed to get wireless internet working from windows here, but for some reason my linux doesn't like this particular access point.

    After a marathon 38ish hour journey I arrived at almost midnight, then fell asleep. I spent half the day today wandering around the old town looking at things.

    I'm looking forward to when the conference starts. Should be good to meet everyone, and see what everyone is up to.

    Here's some pictures from my adventures in old town today...

    Sunday, July 01, 2007


    I have been working on a virtual lounge room for - an online rug shop. The owner (David), also has a real shop a few blocks from me, and is a young guy who's got good ideas about websites.

    This is a flashy website where you can select a rug, some furniture, and some paintings by a local artist. So you can kind of see how the rug might look in a room of your own. I guess kind of like an Ikea catalogue where you can change the items in the photo.

    You can also change the colour of the walls, the shadows of the room, and the type of floor under the rug.

    The front part was made with flash - with a lot of action script.

    One of the hard parts was doing a 3D transform of the photo of the rug. So it sat in the room with the correct perspective. All of the photos were taken over head, so the transform needed to be done to make the rugs look ok in the scene. Since all of the photos were already taken, retaking the photos from a different angle wasn't practical.

    I had to also code some functions to get rid of white parts around the edges of some images. I first coded it in php, which worked ok after caching the results - even if a little slow. Unfortunately I couldn't get the texture mapping code to use the alpha transparency part of the image. So I had to recode it in action script - which also was initially slow. So I had to use a built in function which was much faster (probably written in C) - but with not quite as good quality. You have to make trade offs sometimes right?

    The bitmap functions in flash 8 are pretty good. Before flash 8 they were very limited. You can do many things which are too slow for actionscript by itself - by using the built in functions. However sometimes the documentation is lacking. I don't know if adobe-flash actively seeks feedback from people using their APIs. The quality of the documentation is quite good - and consistent. However common problems people had with the APIs are not addressed. It's like a release and forget style of documentation. Or maybe they're just busy! Since many of their APIs do get better with every release.

    A better method for documentation is to look at problems people are having, and then either fix the API, or improve the documentation until people stop asking those questions. It's a good method we try and use with the pygame mailing list, and website documentation. If there are common problems - we try and document them, or fix the api so people don't have those problems or questions. It's better to fix it so people don't have to look at the documentation in the first place - but some times that's not possible. There's still some recurring questions/problems people have with pygame of course! There's also over 100 doc comments which need to be addressed, and folded back into the documentation.

    Adobe do have user comments on their website though. However not many people use them, since people use the inbuilt flash documentation - which doesn't download the user comments from the website. Also you can't make user comments from inside flash. So it's no where near as helpful as the php documentation, or the pygame documentation doc comments. For the amount of people using flash, there should be FAR more doc comments than there is. So I think it's a user interface and work flow issue. Also maybe adobe not embracing contributions from people using their software - as best they could.

    Not that pygame, or php allow you to make doc comments from within php, or pygame. But mostly people look at the documentation on the web. Maybe we could build a doc comment function into pygame itself? Since people use help(pygame.something) a lot, or see the doc strings in their editor. Maybe a pygame.add_doc_comment(pygame.somefunction) function, which uploads comments to the website ;)

    Doc comments on websites are like bug reports - or patches. They are a very valuable source of information when trying to improve something.

    Oops... I started ranting about documentation. Back to the virtual lounge room...

    The flash front end integrates with a database which is shared with the shop part of the website. So the same rug descriptions, and photos are all used - and controlled from a management interface. David (the owner of rugs online) can change which couches, floors, and paintings are displayed. I made the database for the rest of the shop first, but it wasn't hard getting flash to use the same database. Flash has a number of different ways it can request information from websites - so that part wasn't very hard. Flash has been doing AJAXy style communication with websites since flash 4. I used php to communicate with flash, but it's really easy to integrate with other languages too. You can talk to flash a number of ways, but the simplest methods are to use urlencoding, or xml encoding. Almost every language used for making websites can do either urlencoding or xml encoding - often both. You can also use json, xmlrpc etc - or even sockets.

    It would have been possible to do this virtual lounge room with javascript using the Canvas tag using modern browsers, but that's not what was chosen.

    Have a play - it’s kinda fun making the images zoom across the screen: virtual lounge room.

    Melbourne Web Developer Written by a Melbourne freelance web developer. Available for your projects - php, mysql, e commerce, javascript, CMS, css, flash, actionscript, python, games, postgresql, xml.

    Monday, June 25, 2007

    php 5.x is four years old. php 4 is still used. Pygame with py3k?

    A cautionary tale for py3k planners from the php situation with php 4, and php 5.

    php4 is still so popular and widely used, even after four years.

    py3k should really study the php situation in order to avoid repeating those mistakes.

    I don't think python has that much legacy code out there compared to php, but I still think there is enough that python2.4, 2.5, and 2.6 will be around with us for ages.

    This page lists the reasons for php4 still hanging around. php is often installed by the web hosts, not by individuals. This is the main benefit for php versus python. Php is already installed. However as it is hard to run both php4 and php5 at the same time guess which one gets installed? Php 4 gets installed, because of all the legacy applications which require php 4. Since there are 20-1000 web pages on a single host, if only 5% require php4 then that's what will be installed. Some major webhosts have just upgraded to php4 this year!!! (Media temple) Many open source projects still support php4 since that is what most people are running. Most of them have finally been upgraded to support php 5, but still work best with php4.

    Being able to run both php4 and php5 at the same time would have helped this situation a lot. This can be done with fcgi, but most hosts use mod_php. mod_php, like mod_python can only run one version of python at a time.

    php 5.x has cleaned up a *lot* of things in php 4.x so I think the pain was worth it in the end, for some. For others it has created an extra four years of work, with still more work coming. I think py3k will have a bunch of things cleaned up to make it worthwhile too.

    Many games written with pygame have little to no unittests. Even the best programmers who certainly are not 'dumb asses' do not write unittests for their games. It will be annoying to break pretty much all of the games on py3k, but py2.x will still be around to run those. Unlike with webhosts, most python game people can choose which python they use.

    Pygame has a limited test suite. However we do have many programs to test against, with a large set of people who report bugs. The unittests for pygame are also slowly improving, so hopefully by the time of py3k, there will be plenty of unittests to help with the transition.

    Will it be possible to support both 2.2, 2.3, 2.4, 2.5, 2.6 and py3k at the same time? At the moment we support >= 2.2 with python2.6 not having been even put in alpha yet, and py3k a long way away. We could probably drop support for python 2.2, and python 2.3 if needed. I don't think debian will do another stable release before py3k comes out, so python 2.4 will still have to be supported then.

    Does anyone know how the C API will be for py3k? Is it going to be massively different? Every python release so far has broken the C api compatibility in some small way, so that's to be expected with py3k. I'm guessing it won't be all that much different though.

    Or maybe py3k is the time to clean up the C API too? I haven't read anything about the C api, in the py3k plans, so I would like to know.

    Friday, June 22, 2007

    qhtml, qurl, qsql, qjs

    These are some easily remembered functions for quoting used in website programming.

    Using the idea of consistency allows you to remember how to quote things. Just use the function with a q in front of it.

    url quoting, html quoting, sql quoting, and javascript quoting are things that web developers do almost every day in some frameworks.

    Please consider using these short cut functions in your web frame work. Maybe if quoting was easier to use then people would use it more often.

    It's probably the next best step compared to quoting by default, and not needing to quote at all.

    They should be top level functions as well so they are easy to use.

    * Note, I also use qhtm as a shortcut for qhtml. Just like how htm and html are used for webservers. qjavascript would probably be another good alias - but not one I use. I don't use a qxml, since when writing xml I use more verbose forms for constructing xml - but maybe it would make sense too?

    Wednesday, June 20, 2007

    Abstract Base Class - a poor name. Role is better.

    For someone new to programming, or maybe from a non-maths background 'Abstract Base Class' sounds foreign, and weird. Whereas Role rolls off the tongue - giving me language elegance goose bumps.

    How many people can guess what an Abstract Base Class is from the name? Even given it's context of 'relating to objects'. I think more people could guess from the name Role.

    Reading about Abstract Base Classes over time always gave me troubles. Even reading the PEP now I find it hard to answer the question 'What is an Abstract Base Class'. A simple question don't you think?

    Maybe the name is too abstract.

    When discussing objects to people I could talk about classes, then about instances, inheritance, and interfaces. When I begin to talk about Abstract Base Classes then blank looks jump out. Maybe it's that the name is so long. The abbreviation into the acronym ABC makes even less sense - as it is using an existing term and changing its meaning. Not that the world needs another acronym being used.

    Is Role a good name for what Abstract Base Classes are? Or is there a better name for them?

    I think this question needs to be answered:
    'An Abstract Base Class is [...]'.

    This is the part of the PEP 3119 which describes what Abstract Base Classes are.

    "This PEP proposes a particular strategy for organizing these tests known as Abstract Base Classes, or ABC. ABCs are simply Python classes that are added into an object's inheritance tree to signal certain features of that object to an external inspector. Tests are done using isinstance(), and the presence of a particular ABC means that the test has passed.

    In addition, the ABCs define a minimal set of methods that establish the characteristic behavior of the type. Code that discriminates objects based on their ABC type can trust that those methods will always be present. Each of these methods are accompanied by an generalized abstract semantic definition that is described in the documentation for the ABC. These standard semantic definitions are not enforced, but are strongly recommended."

    From a dictionary, the meaning of Role:
    1. a part or character played by an actor or actress.
    2. proper or customary function: the teacher's role in society.
    3. Sociology. the rights, obligations, and expected behavior patterns associated with a particular social status.

    I think the word Role should be used instead of Abstract Base Classes. It's shorter, and makes much more sense.

    If not Role, then anything but Abstract Base Classes.

    PEP 3133 is another PEP which uses the term Role - that has been rejected.

    Tuesday, June 19, 2007

    Pygame weekly mini sprint 2007/06/20

    I spent some time looking at the FastRenderGroups code by DR0ID.

    I sent a first review to the mailing list about FastRenderGroup. So we can discuss some of the things we need to do to it before getting into pygame. It has a number of features missing from the current pygame sprite code. Including being adaptable when updating the whole screen, or just parts of the screen is quicker, as well as layers and support for the new pygame blending modes (like additive blending).

    ideasman_42 came along and noted a few reference counting bugs in pygame. He also plans to send in a patch with a few speed ups - by avoiding PyArg_ParseTuple when not needed, and using METH_NOARGS when appropriate.

    I started on a mask.from_surface() function, but got caught up finding a bug when printing a surface. I think that was caused by the last set of changes to surface. I wrote a unittest for it, and submitted the bug to the mailing list.

    Richard Goedeken submitted a patch for a smooth scaling function for pygame.
    You can read about it here. It comes with an mmx function. However pygame doesn't have the infrastructure set up yet to support mmx or other asm optimizations. So we plan to use the cpu detection code that SDL uses.

    Monday, June 18, 2007

    Webserver DOS, with linux move file - and broken file move semantics with webservers.

    When a file is moved or removed on linux any processes with that file open still see the old file. So this means if you move a new 2 gig file over the top of an old 2 gig file, and some processes still have that file open there will be about 4 gigs of space used up - until the old file is closed.

    Some webservers keep a file open for as long as the client is downloading it. Apache is one web server that does this. Some other webservers do not do this - like lighttpd.

    The problem with reopening a file for smaller parts of a file as it is served to a web client - is that it breaks unix move semantics. The webclient will get a combination of both files, not one file or the other. This can be a problem in many cases. Consider a client downloading a html file that changes mid file. html tags won't balance up, and the client will download a syntactically invalid file.

    So here is how a DOS can happen...

    Say you have a big file mirror or something with lots of files that change fairly regularly. Perhaps a debian mirror, or a shareware mirror. If a DOS client wants to fill up a drive, and possibly cause corruption, or an incomplete mirror - all they need to do is start slowly downloading many files close to the time when the files are supposed to be updated. This requires very little resources on the client side to cause massive resource use on the server side. One client could take up less than 5 MiB of memory to make 2000+ connections and cause eg 2000 * 2 gig disk use - leading to disk empty situations.

    However constantly reopening a file will help stop this type of DOS attack, it might corrupt downloaded files.

    The changed file move semantics are something to watch out for with some webservers. Servers like lighttpd stop this form of DOS attack, but break file move semantics in the process. So you can not rely on them in your applications.

    Thursday, May 31, 2007

    Making Pygames more easily runnable.

    There are a few things which would allow pygames to be run more easily. I had a little chat with Phil the other day about where pygame could be heading. So these are some of the things we talked about. In case anyone else is interested.

    First though, the why. Why should pygames be easier to run? So all of the people making games can have their friends and family play them. So they can get more feedback about them. People like feedback - feedback is an engine which drives people to work on something for fun. They create things in a way, and want to show the world. Or sometimes people make things to just show their friends and family. Or their peers. However do you want opinionated, uneducated game players giving you feedback? Isn't a nice little community of game developers a worthwhile thing? I don't want to lose our nice little pygame community of artistic folks. People interested in making small games, rather than working on engines - and 3D MMORPG web browser plugins (as their first game). Do we want people to install pygame, so they can modify the games themselves? Tweaking games is the most fun part of games for some creative people. So making pygames more easily run-able should be for those who want non pygame programmers to play their games - however we should still try and let people modify games. Once people modify games, then they can create new games themselves - or continue to modify existing games.

    So here is the list of things which could make pygames more easily runnable. Most of these are _not_ new ideas - these ideas just need to be done well.

    * installer maker scripts which work. So people can go python make_executables and it would generate executables for each platform it supports. Failing that it could make it for just the platform it is running on. It should work out of the box on at least MacOSX, Windows, and Linux. There's been installer scripts around since 2001 - however they are just not polished enough. Recently Phil of Galcon fame has worked out a lot of the issues for macosx, windows, and linux binary generators. I think these will be a great start - however they'll need a bunch of tweaking to support other popular difficult libraries - like pyopengl, pyode, and pyogre. A common game skeleton would help too.

    * a common game skeleton(or using convention). Richard plans to make a skeleton for the next pyweek. So each game would have common directories. Eg. a data/images data/sound data/music directories. It would be run with a script, or something similar. It would have screen shots in the screenshots/ directory with file names: screenshot001.png screenshot002.png screenshot00x.png It might have a .ini file with meta data like game instructions, name of game, authors, etc. Stuff like that. People have been proposing a common skeleton since 2000. However it needs to designed quite well, and we need the pygame, and pyweek site to document it well, and encourage people to use it. So we first need to design a skeleton which will work for everyone - and then update all the tutorials, and documentation to use it.

    * a portable pygame runner. Something which could take a .zip file or a directory, and run the game. The runner should be able to run on at least macosx, windows, linux, freebsd etc. There have been a few attempts at this over the years - and it's been suggested about 100 times. However it's quite hard work getting this done. Phils work with the installer scripts will help out a lot here. I can't remember the other efforts to get this working. Something which used OS level protections would be great too. Things such as jail, chroot, ulimit, capabilities etc. There is work on safer python which works quite well. However OS level protection would be great too. Also a way for people to inspect games before they can be played on the runner - giving quality control and trust to the games available.

    * Converting to flash swf, AND/OR javascript. Pypy is a great little python which has backends for javascript. Haxe is a javascript/actionscript compiler that can generate .swf files, or run javascript. So write the pygame api in flash/javascript(with the canvas tag) then have pypy pipe its output into haxe - which would generate the swf file. Phil has done a manual conversion from a pygame to actionscript - using basically the same api. So theoretically it should work quite well.

    * More games getting into distributions. There's heaps of pygames these days. Good quality pygames. If more of them get out into the world, then there's going to be more pygame installs around. There's only 3 pygames in debian for example!!! If more pygames get out there - then people are more likely to install pygame. So we need to make it easier for people to publicise their games. Help them to port their games. Help people to announce them on freshmeat, and big game portals.

    * Getting pygame installed by default in OSX. Python is installed on OSX by default- so why not pygame too? OSX is supposed to be about letting people do things easily - making games easily would be a good fit for them. Even getting SDL installed on OSX by default would be great - since then it's much easier to get pygame working. Especially with Alex's great work on pygame-ctypes (if ctypes is to be on OSX too). It'd be great to start an effort to get at least SDL installed by default. Does anyone have connections within Apple? Who are the Apple python people? Who are the Apple game people?

    * The OLPC project will increase pygame usage a lot. However I'm quite scared about millions of kids using pygame - how will this change the community? It would be great if the OLPC people could come to a pygame mini sprint - we really need to start working on the issues. I'm not sure who the OLPC people are who are working on pygame? So if anyone knows - please let me know?

    Melbourne Web Developer Written by a Melbourne web developer. Available for your projects - php, mysql, e commerce, javascript, CMS, css, flash, actionscript, python, games, postgresql, xml.

    Wednesday, May 30, 2007

    Validating XML with a DTD and python.

    Below is a way to validate an XML file using python and an external DTD.

    Are there any other ways to do XML DTD validation with python?

    A DTD is one way of specifying how an XML should be formed - a schema for XML documents.

    from xml.parsers.xmlproc import xmlproc
    from xml.parsers.xmlproc import xmlval
    from xml.parsers.xmlproc import xmldtd

    def validate_xml(xml_filename, dtd_filename):
    """Validate a given XML file with a given external DTD.
    If the XML file is not valid, an exception will be
    printed with an error message.
    dtd = xmldtd.load_dtd(dtd_filename)
    parser = xmlproc.XMLProcessor()
    parser.set_application(xmlval.ValidatingApp(dtd, parser))
    parser.dtd = dtd
    parser.ent = dtd

    if __name__ == "__main__":
    import sys
    xml_filename, dtd_filename = sys.argv[1], sys.argv[2]
    validate_xml(xml_filename, dtd_filename)

    Melbourne Web Developer Written by a Melbourne web developer. Available for your projects - php, mysql, e commerce, javascript, CMS, css, flash, actionscript, python, games, postgresql, xml.

    Pygame weekly mini sprint 2007/05/30

    These are my notes from this weeks pygame mini sprint.

    Fixed some documentation. mask, scrap, font modules.

    Fixed the mask.set_at get_at functions to raise a nice exception. Thanks piman!

    surface.fill() now takes the same BLEND_ADD BLEND_SUB etc flags that
    surface.blit() takes. Which makes fade to white, and fade to black
    type operations simple and fast. Thanks Marcus!!

    Added the GL_SWAP_CONTROL constant from SDL. Thanks Eyal Lotem!

    Added the new blitters from Marcus. These speed up the blend functions
    and the alpha blit function.

    Added a -warnings flag to for extra warnings with gcc.

    A fix from Marcus for the scrap module in X11.

    DR0ID has been working on a new sprite module which has a number of features. Including support for layers, and blend modes. It's supposed to be quite quick, and automatically detect if dirty rectangles will be faster, or a full screen update. DR0ID wants to make it compatible with the pygame.sprite.RenderUpdates so that people can drop it into existing games.
    Here is his work so far.

    I made a couple of changes to the pygame website. So that comments for releases are listed newest first, and also so that nick names are shown. This is useful for people in irc who would like to know who the person on the website is in irc. eg. if I make a comment it would show Rene Dudfield nickname:(illume)

    Sunday, May 20, 2007

    Pretty print xml with python - indenting xml.

    Here's a fairly simple way to pretty print xml with python. By pretty printing XML, I mean indenting XML with python nicely.

    from xml.dom.ext import PrettyPrint
    from xml.dom.ext.reader.Sax import FromXmlFile
    import sys
    doc = FromXmlFile(sys.argv[1])
    PrettyPrint(doc, sys.stdout)

    Are there any other ways to pretty print xml with python?

    How do you pretty print xml with your favourite xml API? Pretty printing xml with ElementTree anyone?

    UPDATE: there's a few other ways listed in the comments.

    Melbourne Web Developer Written by a Melbourne web developer. Available for your projects - php, mysql, e commerce, javascript, CMS, css, flash, actionscript, python, games, postgresql, xml.

    Wednesday, May 16, 2007

    Pygame weekly mini sprint 2007/05/16

    This week there were a number of patches, and bug reports to the mailing list.

    Marcus got to most of them - fixing up some 64bit issues, as well as some crashes with hardware surfaces inside the transform module. Thanks to John Myers, John Poppleweel, and Christopher Stone.

    Marcus also added some more warnings for gcc. Which you can enable with the command:
    python build -bigw

    Marcus then went through and got rid of a bunch of problems that the warnings showed.

    I committed sprite changes that John Krukoff proposed in an email to the mailing list last week. They allowed the sprite module to be extended with regards to collision detection. We fixed up a slight performance problem with it for the existing type of collision detection it does.

    I started to look at adding the bitmask collision stuff into the sprite module - and then noticed we weren't using a lot of its functionality. Then we got onto discussing collision response... and I wrote another big email to the mailing list about that. The subject of the email thread is:
    "[pygame] [Announce] Python bindings for the bitmask collision detection library"
    It hasn't appeared to have made it to the archives.

    Marcus got into working on the alpha compositing patch. After some tests he found that it was slightly faster for small surfaces, but up to two times slower for large surfaces. So he plans on looking at tweaking some loops, and I plan to tighten up the graphics.

    Jacob- wanted to know what could be done by someone who doesn't know C that well. Pygame needs unittests. Testing what happens with different inputs to functions is one area that needs testing. Like what happens if I pass -1 into a set clip function? Also the example programs could be extended or changed. Especially the sprite module needs unittests for the collision functions - since we plan on refactoring them soon. So Jacob- said he would make some unittests for the sprite module. the test/ is where they would go.

    JKrukoff is writing another proposal for sprite collision - but this time using a more OO api. Possibly with mixins. To see how that API would turn out. He's also working on a proposal to add a Vector class to pygame - for the collision response stuff. But sshhhhh - don't tell anyone.

    Melbourne Web Developer Written by a Melbourne web developer. Available for your projects - php, mysql, e commerce, javascript, CMS, css, flash, actionscript, python, games, postgresql, xml.

    Wednesday, May 09, 2007

    Pygame weekly mini sprint 2007/05/09

    In this weeks pygame mini sprint a few things got done.

    Marcus has been working on the scrap module some more this week. He got image pasting working with the gimp. As well as got images working on windows. In the process of trying to get it working on MacOSX a mac got broken! oh noes!! Hopefully Bob can step up and do his MacOSX magic once more.

    I worked on getting the bitmask library from Ulf Ekstrom integrated into pygame.

    Pixel perfect collision detection is what it is good for. Since it is in C it runs very quickly too. The demo looks really cool, with things bouncing off each other.

    It's in subversion now, but there's still a few things I need to do.

    - make constructor which takes a surface.
    - a 'make_from_surface' method in C. should take a threshold for alpha.
    - complete the documentation (now in src/mask.doc)
    - complete the unittests (test/test/
    - make set_at and get_at raise IndexError when wrong index is given.
    - double check the reference counting.
    - integrate Mask into examples/

    You can try it out by doing running:
        cd examples
        python data/alien1.gif

    JKrukoff made a proposal in IRC about how to modify the current sprite class, so that it is extendible for different types of collision detection. So for example sphere based, AABB, or Rect, bitmask collision detection could be used. Or your own custom collision detection function. He's going to make a proposal, and implementation to the pygame mailing list - so people there can comment on it.

    Melbourne Web Developer Written by a Melbourne web developer. Available for your projects - php, mysql, e commerce, javascript, CMS, css, flash, actionscript, python, games, postgresql, xml.

    Saturday, May 05, 2007

    Consistent variable names across different languages - for Webs.

    A dynamic website is often made up of at least three parts:
    - html
    - a server side programming language (eg python, php)
    - a database

    (* note, in the current web world, there's more than three. You probably have CSS, javascript, json, xmlprc, rss, and possibly many other encodings of a variable name.)

    Why do people still use different variable names for the same data? The html form variable name, the programming variable name, and the database variable name.

    Say you have a form which asks people to type their name in and their favourite colour.

    First name <input type="text" name="first_name">
    Favourite colour <input type="text" name="colour">
    <input type="submit" name="submit" value="What is your favourite Colour?">

    First name
    Favourite colour

    Now in the server side programming language they convert that to 'firstName' for some weird reason. That's one change that has to be done manually. Since the html variable name is 'first_name'.

    $firstName = $_GET['first_name'];

    Now in the database they have:

    CREATE TABLE products (
    `fname` varchar(255) default NULL,
    `fav_color` varchar(255) default NULL,

    That's a third different variable name 'fname', for the same data!

    Ok, now here's the fun part. Let's add an 'ORM' which is a Object Relation Mum^H^H^H Manager. For some reason the ORM ads one more variable name change into the mix.

    So rather than one variable name, you now have four. Four times as many variable names? No. Because the different names could be anything, fname, f_name, fName, first_name, firstname, firstn, Firstname, etc etc. You have to ask yourself, how was the variable name in the database written? How was it written in the HTML form? Rather than ask yourself these questions you should apply consistency, and then you will know how it is spelt.

    Even if you are consistent amongst each different language, you still need to make that mental leap each time - which variable naming scheme am I using now?

    Consistent variable names in one programming language is mostly a common practice. However keeping consistent variable names across all of the languages used seems to be lacking. I think the reason for this is that each language has its ways of variable naming. So people forget to apply this consistency across all of them. Preferring to keep consistent in each individual language.

    Keep variable names consistent across different languages, and you will have to remember less. Which leaves room in your head for other things.

    Melbourne Web Developer Written by a Melbourne web developer. Available for your projects - php, mysql, e commerce, javascript, CMS, css, flash, actionscript, python, games, postgresql, xml.

    Wednesday, May 02, 2007

    Pygame weekly mini sprint 2007/05/02

    Today I worked through the list of patches, and bug reports.

    [BUG] fromstring,tostring work for alpha. Thanks Brian Fisher.
    [BUG] Surf.set_clip(None) works correctly. Thanks Diego Essaya.
    Scrap changes from Marcus so windows/osx compile.
    Added scancode attribute to keyboard events. Thanks Quentin Smith.
    [BUG] movie_set_display hangs on movie file-like object. Thanks Martin.

    So there's a few new unittests in there for these bugs/patches too.

    I think I'll finish my way through the remaining bugs, and patches next week. After that I want to resurrect movieext. movieext is a half finished module that uses ffmpeg to display movies. Also there are heaps of doc comments to look through. About 300 spam ones I reckon - but there's good stuff in there too. I also need to set my windows development environment up again. I'm hoping to fully document it this time, so that other people can get a windows pygame dev environment set up easily.

    Marcus is trying to contact some Gimp people to solve an issue we are having with pasting image data into the Gimp. Pasting from the Gimp works ok - but mysteriously pasting into the Gimp does not.

    We discovered that gcc isn't reporting uninitialised values of structs. It should be an extra warning flag we could use. If not, we might change to using pointers so we can get those warnings. The Surface.set_clip(None) bug brought this up.

    We also had some chats about improving the display of pygame projects on the website. Browsing by date, rankings, and tags would be cool. Also the concept of a favourite playlist came up. Where people can list their favourite games in a play list. So you can share what games you like with others.

    More details about the pygame weekly mini sprint can be found here: Anyone who wants to join in next week is most welcome!

    Melbourne Web Developer Written by a Melbourne web developer. Available for your projects - php, mysql, e commerce, javascript, CMS, css, flash, actionscript, python, games, postgresql, xml.

    Tuesday, May 01, 2007

    One Speaker Per Child (OSPC) project.

    100 speakers. One for each person at a party.

    Here's six mini usb sound cards attached to some usb hubs and extension cables.

    The plan is to hook up 100 speakers to 50 sound cards attached to a bunch of linux computers connected via the network. Then using the OSC protocol to get them to act as one massive speaker system.

    The cool thing is that there'll also be 50 microphone inputs.

    I'm not sure what kind of limits I'll run into yet. It'll be interesting to see how many sound cards I can attach to each computer. Hopefully all 50 sound cards on one computer!

    Melbourne Web Developer Written by a Melbourne web developer. Available for your projects - php, mysql, e commerce, javascript, CMS, css, flash, actionscript, python, games, postgresql, xml.

    Friday, April 27, 2007

    pygame - weekly mini development sprint.

    Marcus and I started up a weekly pygame sprint for a few hours on Wednesdays/Tuesdays.

    Using this link you can find out what time it starts in other places:

    Germany: Tuesday 10pm
    UK: Tuesday 9pm
    US: Tuesday 4pm
    AU: Wednesday 6am

    What happened in this weeks mini sprint

    I collected all of the bugs and patches together onto a todo page. This took a while, so I didn't get that much work done on code things.

    Marcus got the font metrics patch integrated. This allows pygame access to font metrics information. To understand what font metrics are read this. Pango seems to be the preferred way forward for fonts. There are plans for pygame support of Pango too.

    The scrap clipboard module also got some love from Marcus. It's much more complete than the last version that's been in svn for a few months now. You can get all different types of clipboard data. I even managed to paste an image from The Gimp into my pygame program. The latest changes still need to be tested on windows, and the macosx code needs to be updated too.

    Luke on the pygame mailing list has been talking about his work on WII controller code to be used with pygame. It's up to the stage where it is usable, but still needs more work. There's been a discussion on the mailing list about how to integrate it with pygame. This almost makes me want to get a WII - just for the controllers.

    There's also been a few clean-ups with the pygame code. Marcus removed a bunch of warnings and made some other clean ups, including indentation fixes and other niceties. I removed a few more warnings on my debian setup too.

    Recently on the mailing list there's been talk of including pygame with the One Laptop Per Child (OLPC) project. This would be the preferred method for people making games. It would be great if those OLPC people could join us next week for our sprints. Pygame gets used a lot by lecturers and teachers helping people learn programming via getting them to make games. So I think pygame will be a nice fit for the OLPC project.

    Over on the SDL mailing list it was announced that work on a GTK backend for SDL has been started. GTK is the standard tool kit for use with the OLPC computer - which is why this has been made. This is a very interesting development - that should be useful for others too.

    More information about the sprint

    The weekly pygame mini sprint only goes for 3-4 hours. We're working on getting pygame released, as well as making plans, fixing bugs, writing docs, writing tests, testing, adding features and having a bit of a fun old pygame coding time.

    More people who would like to join in the merrier! If you want to try and get a bug fixed, talk about adding some feature, or learn about coding in C with python/SDL/pygame then it would be good to join in. Hopefully the regular sprints will add some more enthusiasm about pygame development.

    irc channel is the normal pygame one: #pygame

    Here is our development todo list. There's a few interesting pieces of code that should be added to pygame at the next sprint.

    Melbourne Web Developer Written by a Melbourne web developer. Available for your projects - php, mysql, e commerce, javascript, CMS, css, flash, actionscript, python, games, postgresql, xml.

    Backup option for apps. - web apps need it too.

    All of your web applications should have an automated backup - and restore procedure.

    For web applications, allow your users to back up their accounts.

    Luckily many web hosts have backup software enabled within their control panels (cpanel etc). So it's easy to set up automated backups from these control panels.

    End users should also have the option of backing up their data. Just so they can have an extra piece of mind. If they stuff something up, they can just restore from a backup. Pretty simple.

    An sql dump might not be the best backup format... although it's pretty good. Maybe you didn't separate your data for different users when you designed your database. So just doing a database dump might not be available. If you give them a database dump they might get other clients data. You might need to write a custom backup procedure - one which knows how your database is designed.

    I guess this is why most web applications fail to give end users backups - All of the user information is not separated within their database clearly. Because they forgot to design for backups in the beginning, and now it is too hard to implement on top.

    So the moral of this web design fairy tale - 'design for backups in the beginning'.

    Melbourne Web Developer Written by a Melbourne web developer. Available for your projects - php, mysql, e commerce, javascript, CMS, css, flash, actionscript, python, games, postgresql, xml.

    Tuesday, March 27, 2007

    Green PCs in open source, free software - the next challenge.

    Linux is seeing massive install bases around the world. Millions of PC computers - not including the millions of embedded systems like routers, and phones.

    OSS, and free software already does a great job in reducing the amount of waste generated by computer use. Being able to make use of old hardware, and the ability efficiently virtualise computers resulting in less hardware being used. Web hosts can run thousands of websites off one computer - now other servers are taking virtualisation of computers for other tasks too.

    However there is much more that could be done for saving energy with OSS, free software such as Linux, Debian, and Ubuntu.

    Distributions need to focus on turning on the power saving features. If Debian, and Ubuntu make this easy, then lots of power can be saved in the world. Debian should put something about power consumption, and reducing resource use into its social contract.

    Is concentrating on power savings worth while? If a 10% difference could be made in power saving, then over one million machines it would save a massive amount of energy - and reduce green house gasses.

    Monitors need to be put in suspend mode when not being used. Monitors are the most power hungry parts of a computer system. The more common LCD monitors do not use as much power as CRT ones, so this is an advancement. I still find monitors that stay on when not being used. For an X server, check out the manual of xset for dpms power saving features.

    Hard drives need to be set so that they go to sleep when they're not being used. The 'hdparm' program can be used to put hard drives to sleep. Not only does this save power, but makes hard drives last longer. The longer a hard drive lasts, the less it needs to be replaced - saving production energy costs. There's also less noise because your hard drive will be turned off.

    Suspend to disk needs to work. This is so people can easily put their computers to 'sleep'. So that people can have their system power up quickly with all of their applications where they left them. Mostly this works, but it needs more testing, and fixes from some poorly behaving drivers. If people can put their computers to sleep easily, then they will turn them off when not in use.

    Daily tasks should run at off peak power times. Debian uses 6am as a time to run daily tasks. Many tasks that can be put into off peak times should be put there.

    Power saving measures should all be turned on by default.

    Using 'thin' client machines without hard drives uses less power over all. It also means that less hardware is used - resulting in less production energy used. Also older machines can be used for the terminals, resulting in reuse of computer parts. The linux terminal server project makes setting up thin clients with linux easy. Operating systems could do more to make this kind of usage easier still. Rather than concentrating on making their installers easier to use for single machine use.

    More thought needs to be put into saving power with changes to operating systems. During the design, and implementation of features.

    The design of software should take into consideration the power use of it. If your software can with one hours work reduce it's power usage by 10%, then you've possibly saved millions of dollars a year in power costs. If you can reduce the resource usage of your program with a little work, then you may only use one server instead of two. Over a couple of years that can add up significantly. Does your web application need a dedicated server at the moment?

    Reviews of Operating systems need to take 'Green' factors into account. Are power saving features used? How well are they used? Is there good support for older machines? The media has a part to play in supporting those that aim to reduce energy use, and reduce the use of computer parts. A "ten things Ubuntu could do to reduce energy use" article could be quite popular. What about an article on the bad ram linux patch(which allows linux to use defective ram)?

    Good reviews on power saving features can allow people to make more informed choices about how much power their purchases will use. How much energy has been used in producing this equipment?

    Just as free software, and OSS people shout from the hills about reliability, security, and performance - we need to shout about power saving benefits. Reduction of two servers down to one, can save on air conditioning bills, office space, and a power bill reduction.

    A mailing list should be joined by groups in each distribution aiming to implement power saving changes to their operating systems. If there is already a group out there, it needs to be more heavily publicised. It should be run by a paid member, or members of some of the biggest distributions. Power saving should be a much higher priority than it currently is. Start a discussion amongst your groups on how to reduce power usage.

    Other widely used open source projects need to talk about how their projects are affecting power, and resource usage too. For example take the python programming language. Recently the python project has managed to reduce memory usage. This will result in being able to run more python programs on one machine. However further steps could be taken to make running web programs on the main cheap virtual hosts less resource hungry. This is starting to happen, but more could be done. However many web applications seem to need a dedicated server to run acceptably(plone). Pygame(game software for python) has been modified so that instead of using a 'busy loop' by default, it uses a less accurate, but less cpu using operating system sleep.

    If you make your program use less power, then those with laptops are more likely to use your programs - over the competitors program that uses more power. Make your lower power usage a feature.

    I think people using Open Source Software, and free software should be interested in power saving, reduction in hardware use, and reuse of old computer parts. We just need to talk about power saving more, and make sure we factor it in with our designs.

    Some links to readings about saving power.
    - Ecology HOWTO - power saving techniques with linux.
    - Linux terminal server project.
    - Gentu power management guide

    If you have any other relevant links on power saving, please leave them in the comments section of this page. Cheers!

    Melbourne Web Developer Written by a Melbourne web developer. Available for your projects - php, mysql, e commerce, javascript, CMS, css, flash, actionscript, python, games, postgresql, xml.