Monday, March 15, 2010

better search engine

Friday, March 12, 2010

Memory usage of processes from python?

Is there a way to find the memory usage of python processes?

Trying to find some portable way of doing this. However, so far I think a new module might be needed...

I've got linux mostly covered, but maybe you know how with freebsd, OSX, windows(9x-7)?

So is there something built into python already? Is there a X-platform third party module already? Or a module just for one platform available?

update: here's the linux code I found and cleaned up a bit if anyone is interested. bytes_resident = memory_usage.resident(). It reads /proc/PID/status... eg, like "$ cat /proc/PID/status | grep VmRSS" would.

pympler: 'Pympler is a development tool to measure, monitor and analyze the memory behavior of Python objects in a running Python application.'

psutil: 'psutil is a module providing an interface for retrieving information on running processes and system utilization (CPU, memory) in a portable way by using Python, implementing many functionalities offered by tools like ps, top and Windows task manager.'

dowser: 'Dowser is a CherryPy application that displays sparklines of Python object counts, and allows you to trace their referents. This helps you track memory usage and leaks in any Python program, but especially CherryPy sites.'

syrupy: 'Syrupy is a Python script that regularly takes snapshots of the memory and CPU load of one or more running processes, so as to dynamically build up a profile of their usage of system resources.'

Some non-pythony memory tools: valgrind memcheck, massif and cachegrind (linux), MallocDebug (osx)

Wednesday, March 10, 2010

how I recovered a friends MacOSX drive

MacOSX can sometimes corrupt a drive (like most OSen).

I've seen it happen a few times, in a couple of different ways. One way is if you interrupt some file transfers. Like by pressing 'stop' on a big transfer - this can trash your partition table. You'll likely see vfs errors in your log at this point. Anyway... linux, ubuntu and 'testdisk' came to the rescue. testdisk found the partition for me, and wrote it back... luckily it worked and my friend and I did a happy dance.

The HFS+ partition was saved.

Sunday, March 07, 2010

gnome multimedia keys via dbus-python

Ever wanted to get your multi media keys to do something other that play multi media? You could get these key events any number of ways. One way is through dbus. Here is an example:

Here is a version which does not block your mainloop. So it's useful for integrating with other libraries (like pygame :) I made a dbus with pygame example too.

Saturday, March 06, 2010

Ideas for Super Surfaces in pygame.

pygame already has a sub-surface, which is part of a larger surface. Sub-surfaces refer to the same pixels as the surface they come from, and share the same Interface as a Surface. It's good for doing sprite sheets, where you save one file with many smaller images - but then being able to manipulate them as if you loaded the images as smaller separate surfaces.

However, sometimes we would like to operate on a whole bunch of smaller surfaces stuck together. This is what I'd like to call a Super Surface - a collection of smaller surfaces which can act as one big surface. It's a complementary idea to the sub-surface, and intuitively it should be there... but it's not yet.

Unfortunately it's a lot harder to code a Super Surface compared to a sub-surface. Since a Super Surface would need to have all the surface affecting routines changed to work with it.

For example, everything in the draw modules would need to be redone. So would all of the surface methods. So when you draw a line on a super surface, it should affect all of the sub-surfaces.

How can Super Surfaces be implemented.

Still trying to think of a simple/clever way of implementing it efficiently, without having to recode all of the drawing/blitting routines. However, just like how sub-surfaces in pygame can simplify code immensely - so too will Super Surfaces.

One method, might be to do a rect translation, and then broadcast the translated method calls over the sub surfaces of the Super Surface. So we translate the rects into the sub-surface coordinate space from the Super Surfaces coordinate space.

I'm not so sure if we can get a function pointer to a method from within the function pointer itself with the python C API. This would simplify it, because the translate-and-broadcast functionality could be implemented in one place and reused within all of the various surface affecting routines automatically. Something to research...

Common methods for implementing tiling engines.

Super Surfaces share a lot of the same properties a tiling engine. So how are tiling engines implemented?

A common way for tiling engines to work is to have the images being placed in a grid, and for each of the sub-surfaces to be the same size. So the image at [0][0] in the array is at those coordinates. This simplifies the implementation, but is not very flexible. A slightly more complex way to implement it is to have each sub surface have its position stored relative to the super surface. This complicates the clipping a little, and for speedy use it really requires a spatial hash... like a quadtree.

So I'm thinking of doing the more free form implementation... probably just a naive implementation to start with... then perhaps a quatree version as an improvement. It would be better to use a spatial hash more efficient at moving elements - but that could be a third improvement.

Use cases for Super Surfaces

Use cases include viewing massive images. Much graphics hardware has texture size limits, so it is common to combine many smaller textures when drawing larger images.

Another common use case is for 'tile engines'. Where tile engines show a larger image made up of many smaller images.

To allow using different image formats for different parts of an image is another use case. For example, a piece of the image which is just black only needs a few bytes to represent the image... it does not need 32bits per pixel. Whereas the part of the image which includes a persons face with 50% transparency will require 32bits per pixel for sure. This will save memory, and increase speed. It increases speed, because it is possible less memory bandwidth means faster operations. Also, less complicated blending - or more optimized blitting can happen (eg, fast 8 bit Run Length Encoding blitters). This should be possible with no extra effort from the programmer too.

Finally, by using smaller sub-surfaces it in effect becomes a tiling engine automatically. Tiling in graphics is used to split operations up easily. This gives memory, and speed advantages. By processing a tile at a time - you only need to have those tiles in memory at that time. It also gives easy parallelism, since the separate tiles are a form of data parallelism (which is the easiest type). For some hardware, it is possible to use parallelism when the surfaces are kept separate.

I'm interested in any comments, especially if you've made a tiling engine like this before?

Wednesday, March 03, 2010

Why bzr and launchpad? launchpad is open source

Why bzr and launchpad? Bzr AND Launchpad are open source.

launchpad: open source
github: closed source
sourceforge: closed source (was open source in the past)
bitbucket: closed source
googlecode: closed source

You can submit changes to launchpad at: As well as (submit launchpad bugs) and feature requests against it if you don't want to make the patch yourself.

When there is a good open source alternative, I always choose the good open source option. I initially had problems with bzr a couple of years ago... but it has been quite good to me over the last couple of months. So I'm moving over all of my projects from other version control hosting services to launchpad.

Of course bzr and launchpad are also written in python (with selected optional C optimizations), so that makes for happy hacking :)

update: reported a bug here about the 'can not find source code easily on launchpad' issue. Any other issues with launchpad?

Monday, March 01, 2010

London code dojo - 4th March '10 18:30 – 21:30 (ish).

Details, and booking here:

"""What is a coding dojo? This is a coding dojo.

Last time we attempted to refactor the various adventure game solutions from the January dojo. Whilst interesting, perhaps refactoring isn’t that exciting an activity for a dojo :-). Nevertheless, people seemed to be having fun and we did achieve our goal of a “one true” adventure game code base that allows us to define and navigate around a game world. The code can be found here:

After discussion at the end of the February dojo (and later in the pub) we decided that this time round we’re going to try another small-groups based exercise with a “show and tell” at the end as we continue to build the world’s greatest adventure game. Problems we might want to tackle include: a command parser, keeping track of game state/score/objects, NPCs/AI, authoring tools, turning it into a MUD (and so the list goes on…)

Photos from all the dojos so far can be found here:

Free pizza and beer will be provided.

Participants get the chance to win a cool book (thanks O’Reilly)."""