making apps, making webs.

I've done some quick benchmarks for the unreleased python3.2 on the unreleased cherrypy webserver for python 3. Also with the unreleased python2.7. Here are the results... python3.1 Client Thread Report (1000 requests, 14 byte response body, 10 server threads): threads | Completed | Failed | req/sec | msec/req | KB/sec | 25 | 1000.0 | 0.0 | 533.32 | 1.875 | 93.75 | 50 | 1000.0 | 0.0 | 525.86 | 1.902 | 92.69 | 100 | 1000.0 | 0.0 | 522.96 | 1.912 | 92.1 | 200 | 1000.0 | 0.0 | 523.83 | 1.909 | 92.25 | 400 | 1000.0 | 0.0 | 506.92 | 1.973 | 89.27 | Average | 1000.0 | 0.0 | 522.578 | 1.9142 | 92.012 | python3.2 Client Thread Report (1000 requests, 14 byte response body, 10 server threads): threads | Completed | Failed | req/sec | msec/req | KB/sec | 25 | 1000.0 | 0.0 | 555.72 | 1.799 | 97.78 | 50 | 1000.0 | 0.0 | 558.86 | 1.789 | 98.52 | 100 | 1000.0 | 0.0 | 552.87 | 1.809...

Invent with python 'Teach yourself how to program by making computer games!' is a free book now in its second edition. It is released under the creative commons licence Attribution-Noncommercial-Share Alike 3.0 United States - and the author just asks for a donation if you liked reading it. The book takes the fun approach of teaching programming through making games... the way many of us learnt(and are still learning) to program. As a bonus it uses python 3 - which pygame has supported for a while. Congratulations to Albert Sweigart for finishing his second edition of the book.

Integrated modules are nice, but so are modular packages. How can we have both? That is, keep all things for a module(or sub package) in one directory, but also have a nice integrated system built on top of that? Often for one package I will have a file layout like so: /setup.py /[package_name] /[package_name]/[package_module].py /[package_name]/tests/[package_module]_test.py /[package_name]/examples/[package_module].py /[package_name]/docs/[package_module].py Then each module has its tests, docs, and examples all mixed in the one directory. This is nice if you want to have all of the tests together, and all of the docs, and examples together. However then all of the modules are mixed in together. Meaning it is harder to separate them, and keep one module in its own directory. Having everything for one thing together is nicer for developers I think. Using namespace packages through setuptools distribute is one way. There is a draft pep around to put namespace packages into python...

Using a hash with a salt is one way of validating urls. With it you can tell that it is quite likely the url was generated by someone with a secret. That is, generated by someone authorised to make that url. This post describes using this technique with the hmac python module applied to urls(and object method calls). Generating thumbnail urls is a good example. If you have a url which can generate a thumbnail of an image at a specified size that is pretty cool. Except then you can have someone generating images at all sorts of other weird sizes by exploiting the width, and height parameters (eg making 100,000 pixel wide thumbnails). Or perhaps you want to limit it to a certain set of images. You could always code in your thumb nail generating routine which variables are valid... but this has problems. First it makes your routine less flexible. Separating authorisation is always a nice idea. Another way (with HMAC like using pywebsite.signed_url) is to generate the urls and add ...

In Switzerland. Had fun visiting an old friend and his new girl friend. Lots of snow came whilst we're here which was fun. However, the planes didn't like it, and they got cancelled. So going back to London today instead(if the planes work).

The unladen swallow project comments on the global interpreter lock and suggests long term dropping reference counting in favour of GC. http://code.google.com/p/unladen-swallow/wiki/ProjectPlan#Global_Interpreter_Lock . They are also now favouring an incremental GIL removal from python. However, confusing getting rid of the GIL, with requiring using garbage collection is wrong. This mini-essay will explain why reference counting is good for python, and why you do not need to get rid of it for multiple CPU systems. Then there will be descriptions of other improvements to python for multiple CPU systems that can be made. Reference counting has a number of advantages over various GC schemes see http://en.wikipedia.org/wiki/Reference_counting#Advantages_and_disadvantages . As a python programmer, optimizing for the programmer is what is most important... and reference counting makes it easier for the programmer. It makes programs much more deterministic, meaning that reference counti...

Writing for the latest python version is just simply... fun. Even the name 'python 3000' or py3k is amusing. It's been a joy over the last week plugging away at a few personal website and game projects using the latest python version. Over the last year I've been involved in upgrading existing code to the latest versions of python... but not really in writing it JUST for the latest versions. No need to consider backwards compatibility problems... older pythons be damned when programming for fun! I'm able to take advantage of the latest python features in deep architectural ways. To fiddle around with the new deep magic python internals. But the new deep magic parts aren't the best bits though... it's all the elegant little improvements that make a difference. It is of great delight finding python 3 compatible modules. It turns out that the modules available for python 3 are often of quite good quality - and well maintained. There's not all that ...

The sqlite and pickle modules that come with python are quite useful. So lets mix them together and see what comes out. SQLITE Vs PICKLE pywebsite.sqlitepickle is a little module I just made which combines sqlite and pickle for persistence. Useful since it works with python3, and both pickle and sqlite are included with pythons (including pypy). Import the module. >>> from pywebsite import sqlitepickle An in memory db. >>> db = sqlitepickle.SQLPickle() >>> db.save('key', 'value') >>> db.get('key') 'value' Can also save to a file. So we first get a temp file name. >>> import tempfile >>> f = tempfile.NamedTemporaryFile() >>> fname = f.name >>> db = sqlitepickle.SQLPickle(fname) >>> db.save('key', 'value') >>> db.get('key') 'value' >>> db.close() The issues with this are that sqlite does not like sharing conn...

Play cards. They're like business cards, but take 30 minutes to make five. Also, they are for handing out to people you want to play with, rather than to people you want to do business with. Well it could be used for both... but I think the normal style business card is a bit boring to give to friends. Made some of these the other day. Based on an earlier version I designed a few years ago. They're about the size of a finger, but unfold to show other details (eg, email, website, etc). --

Here is an example buildout project that uses numpy: https://launchpad.net/numpybuildout/ If you have bazaar, you can check it out like so: cd /tmp/ bzr branch lp:numpybuildout cd numpybuildout/trunk/ To use distribute instead of setuptools use the -d flag of bootstrap.py. python bootstrap.py -d ./bin/buildout ./bin/py >>> import numpy >>> numpy.__file__ '/tmp/numpybuildout/trunk/eggs/numpy-1.4.0-py2.6-linux-i686.egg/numpy/__init__.pyc' There you have it, a simple project that can download and build numpy off of the python package index and install it in it's own private directory for use. update: It was broken for python2.6 and numpy 1.3. However it works again with numpy 1.4

gvim in ubuntu karmic koala is really annoying. It prints out a bunch of gtk warnings each time you call it. However thanks to someone making a patch, you can clean it up. It seems that the fix isn't going to come out until the next ubuntu. ** (gvim:13354): CRITICAL **: gtk_form_set_static_gravity: assertion `static_gravity_supported' failed ** (gvim:13354): CRITICAL **: gtk_form_set_static_gravity: assertion `static_gravity_supported' failed ** (gvim:13354): CRITICAL **: gtk_form_set_static_gravity: assertion `static_gravity_supported' failed ** (gvim:13354): CRITICAL **: gtk_form_set_static_gravity: assertion `static_gravity_supported' failed ** (gvim:13354): CRITICAL **: gtk_form_set_static_gravity: assertion `static_gravity_supported' failed Very annoying to see that every time you edit a file. https://bugs.launchpad.net/ubuntu/+source/vim/+bug/402188 So instead of switching to arch linux, or gentoo linux... or one of the other developer centric distros t...

Arrived back in London a few days ago, after a month living in Berlin. I really enjoyed Berlin, and thought about staying there for longer... but in the end decided against it for now. Would really like to visit in the summer, when it's apparently quite a different place! Some pictures from Berlin Finding the bustle and life on the streets of London quite refreshing. I think I'll be based in London for the next six months or so at this point(probably longer). Going to the london python dojo again on the 10th. This is an interesting format for a get together. People take turns pair programming with a projector, whilst the rest of the people in the room offer comments. This time there's going to be a few short interactive presentations too. It was nice meeting python people from around London last time. Fry-it are the hosts, who offer their office, pizza(+salad) and beer(+wine) for everyone at each dojo.

Pythons distutils has a race condition where it starts to copy files into the python path whilst installing. This is a race condition , since python programs can be importing the package whilst the package is being installed. It would be good for distutils to install things in an atomic manner. Where things can be installed, or not installed. Like, on unix by moving the files in from a temporary directory. This would also help reduce breakages. Since if a package breaks half way installing a package then the broken version will not over write the existing version. It's not a very serious problem, since most people don't install things on live important systems. Also some packaging tools fix the issues with the source installs.

"2to3c: an implementation of Python's 2to3 for C code" See http://dmalcolm.livejournal.com/3935.html for a description of this tool. It works on C modules. So it should be easier for people to do ports to py3k for their C modules. The 2to3c tool uses Coccinelle for transformations on the C code. This program has been used for linux, and other software... for updating code when their APIs change. The perfect fit for the python C API changes!

rakarrack is a decent effects rack for linux. It's not packaged for ubuntu(of course), but is fairly easy to compile. I've been using the cvs version (yes, people are still quite happily using cvs it seems). Once you get it set up you have access to 80 presets of realtime guitar effects(2-40ms depending on your computer sound card setup). It works with the jack audio system, so you can route audio into it, and route its audio out to other programs easily enough. You can also control it with midi(alsa, or jack). Meaning you can hook it up to a midi controller of some sort. I've been using it with an maudio axiom 25 controller, and with python scripts via pygame.midi. It even has a very nice help manual integrated into the program... I wish more programs had a good help section. It details the 17 different effects that come with the program. One thing which is a bit funny is that it doesn't use LADSPA or lv2 plugins. You can of course still use these other plugi...

Berlin seems like a fun and laid back place. Arrived here the other day... haven't had much time to look around so far.

This is for the beta release, and most(I think all) of these bugs have been reported in the bug tracker. (Note the bug tracker is currently reporting errors... so I can not link to bugs). I do like a lot of things about the release... these are mainly things I don't like. So please don't take this as a bad view of Ubuntu 9.10 overall... just some criticism. Hibernation seems to not work correctly all the time(for me)... when I close the lid, and open it later it does not seem restore properly. It shows the screen, but does not respond to input on the keyboard/trackpad. Then after a bit the screen goes black. Then I need to press the power button, and then the login box comes back up - and I can log in again. This has worked for the last two ubuntu releases, so it's annoying. Booting seems slower... and is actually slower(timed it) for me. Perhaps I need to install from scratch for it to be faster or something. Pete Shinners has some benchmarks on his blog of ubuntu ...

More robots are reading websites than humans - so should we be designing the websites for robots? Thanks to the search engine wars, most visitors to most websites are by robots. But has anyone asked what these robots enjoy? There's often a divide in artists between introverted, and extroverted artists. "Fuck you - I make art for myself and not for others." OR "I make art for people to enjoy, I'm not so selfish and self indulgent that I just make art for myself". So I decided to interview a few robots to find out what some of their favourite websites are. But where to find a robot to ask questions of? I had to look no further than an internet enabled fridge. So with my notebook in hand I pull a chair up to the fridge and ask it some questions. I open with a flurry of questions, trying to provoke a response. "Do you have any favourite websites?" I ask. No answer. Obviously this robot does not know english. I give it another ten minutes of que...

Tonk Tanks is a really small, and fun game. All game play is on one screen, and the controls are simple. You have arrow keys to move your tank around, and a button to shoot the other tanks. It's like one of those old atari 2600 tank games, but a more modern version. Once you die, you teleport to a different respawn positions around the map. The game is quite playable single player - but the author is working on a networked multiplayer version. Another nice addition would be multiplayer on one machine - especially with joysticks or mice support (since keyboards are evil ). The tank AI seems different and varied enough, that I have not worked it out from playing it ten times or so. Usually my games only last about five minutes before I move onto something else. It's one of those games I can play for little while when I want a short break. Tonk Tanks works well on linux, windows and Mac (and probably other platforms supported by python+pygame). There's a windows .exe...

Assume you have a user content site - or you're using software that can somehow get spam links inserted into it. How do you find out if your website has spam put on it? It seems a common enough problem these days... people putting spam links on websites. Surely there must be a service or piece of software to detect such a thing? I can think of a few ways to go about writing one fairly easily (using existing spam detection tools... but applying them to a spiders crawl of your website). It would be much nicer if there's already a tool which does such a thing though.

If you don't have a midi output on linux(cause your laptop has crappy audio hardware) you can use timidity or fluidsynth to emulate it. timidity -iA -B2,8 -Os -EFreverb=0 Well, this piece of html has a bunch of incantations for using timidity on linux... and also gives insight into how to use alsa midi tools. Like listing midi ports, and connection midi ports with these two commands: $ pmidi -l Port Client name Port name 14:0 Midi Through Midi Through Port-0 20:0 USB Axiom 25 USB Axiom 25 MIDI 1 128:0 TiMidity TiMidity port 0 128:1 TiMidity TiMidity port 1 128:2 TiMidity TiMidity port 2 128:3 TiMidity TiMidity port 3 To connect the midi input from my usb Axiom 25 keyboard to the timidity synth the aconnect is the command to use. aconnect 20:0 128:0 The AlsaMidiOverview has more informatio...

GNU screen is a good little tool for server administration, or running things on your own remote machines. It's even good for running things locally. I hope this is useful for people who want to run scripts every time they login, or reboot... and who need interactive access to those scripts. Or useful for those people who are already using screen, but would like to make their setup a bit better: scripting sessions, rather than doing them manually at each login or reboot, finding your screen sessions more easily. restarting scripts at reboot, monitoring, logging, resource control Running things as daemons is cool... but if you'd also like interactive control occasionally, running things with screen is useful. Most servers have screen, watch and crontab(osx is lacking watch though) - including most linux distros, *bsd, osx, windows(with cygwin). Most OSes also have their own style init scripts(scripts to run things at boot or logon). So this screen, watch, crontab combinat...

No I'm not talking about the free software song sung by Richard Stallman (very funny, but in a low quality .au format). Or the pronouciation of Linus and linux . To start on this long-journey-of-a-rambling-diatribe-of-words, there's two good audio patches in the SDL bug tracker for the upcoming SDL 1.2.14 release. One patch is for the pulse audio driver, and the other is for the alsa backend. These solve some of the high latency or scratchy sound issues some have. That's right a new SDL release very soon ... it's over a year since the last 1.2.13 release, and it seems like forever since the SDL 1.3 series begun. Most new development has been happening on the SDL 1.3 tree in the last year... so the 1.2 releases have slowed to an almost stop. There's a good article on a x-platform atomic operation API for SDL http://www.thegrumpyprogrammer.com/node/16 . That's one of the features that's been evolving over a few years, and is being implemented in svn. In py...

Anyone know where the 'new' module went in python 3? 2to3 can't seem to find 'new', and I can't find anywhere with my favourite search engine either... filed bug at: issue6964 . A complete 2to3 tool should know about all modules that are missing at least. It needs to actually know what to do with those modules, but should be able to at least tell you which modules are missing. I'm not sure how to get a complete top level module list sanely... I guess by scanning the libs directory of python. Or maybe there is a module to find all python modules? Each platform would be slightly different of course... and there'd be differences based on configure. Also some modules have probably stopped importing or compiling at all these days. Then you could just find the intersection and differences with the lovely set module :) # find the difference between the modules. top_level_modules_not_in_3 = set(top_level_modules3series) - set(top_level_modules1_2series) Wel...

Python3 was released more than a year ago so far, and the release candidates and beta releases much before then. How successful has the transition been to python3 so far? One way to measure that is to look at how many python packages have been ported to py3k. One way to measure what packages are released is to look at the python package index(aka cheeseshop, aka pypi) . Not all packages ported to python3 are listed on pypi, and not all packages are listed on pypi. Also there are many packages which haven't been updated in a long time. However it is still a pretty good way to get a vague idea of how things are going. 73 packages are listed in the python3 section of pypi, and 7568 packages in total. That's 0.96% of python packages having been ported to python3. Another large project index for python is the pygame.org website. Where there are currently over 2000 projects which use pygame. I think there are 2 projects ported to python3 on there(but I can't find them at t...

It seems LLVM has gained another user, in the parrot multi language VM project. They plan to ditch their current JIT implementation and start using LLVM. Full details are on their jit wiki planning page . There is more of a discussion on the parrot developer Andrew Whitworths blog here and here . Parrot aligns very nicely with the LLVM project which itself is attempting to be used by many language projects. Along with the unladen swallow project (python using LLVM for JIT), this brings other dynamic languages in contact with LLVM. This can only mean good things for dynamically typed languages on top of LLVM. Mac ruby is another project switching to LLVM - they have been working on it since march. Rubinius seems to be another ruby implementation mostly written in ruby, and the rest in C++ with LLVM. It even supports C API modules written for the main ruby implementation. 'Rubinius is already faster than MRI on micro-benchmarks but is often slower than MRI running applicat...

The new linux kernel has been released. Here are the human readable changes . Here's the cool stuff (the links in the original article were broken, so I've fixed the links here): USB 3 support CUSE (character devices in userspace) and OSS Proxy Improve desktop interactivity under memory pressure ATI Radeon Kernel Mode Setting support Performance Counters IEEE 802.15.4 Low-Rate Wireless Personal Area Networks support Gcov support Kmemcheck Kmemleak Fsnotify Preliminary NFS 4.1 client support Context Readahead algorithm and mmap readhead improvements For me the performance counters will be the most useful thing. Also being able to use and write user space character devices is cool(especially for audio). USB3 support is awesome, but not useful right now... since there isn't even much hardware out yet! More info on what that low power wireless support is, can be found on the wikipedia: IEEE_802.15.4-2006 .