making apps, making webs.

CherryPy - I love it because it's pyhon.

2011-12-19T06:24:00.003Z

CherryPy moved to bitbucket a while ago, and also got a website refresh.

I love cherrypy, since I can use it with python 3 - not legacy python.

I love cherrypy, because I can just use python objects.

I love cherrypy because it has a clever name.

I love cherrypy, because it just works.

I love cherrypy, it is minimal.

I love cherrypy.

Thoughts on Python 3

2011-12-07T15:11:00.001Z

It's awesome.

Go get some!

PyGameZine issue0

2011-11-20T13:11:00.001Z

Today we are very excited to announce the launch of PyGameZine!

There is more information about it on the http://www.pygame.org/ website.

Vintage in Paris

2011-11-11T07:43:00.000Z

Our Vintage in Paris app has been published on the Nokia app store!

The process was pretty simple, since it was a very technically basic but content heavy app.

The main difficulty in this app was walking around the streets of Paris taking pictures, and researching really good places to find Vintage.

Why a Vintage in Paris app?

Finding cool second hand and retro clothes is a hobby of mine, and my partners. It's always fun finding interesting old styles, rummaging through clothes racks. Especially in other countries, and especially when you find a bargain.

There's something special about wandering down some back alley, and finding an old shop stuffed to the rafters with old stuff. It's like taking a journey back in time to some fantastical version of Paris you may have in your head. If you like Art Deco stuff, then Paris is the place to get it.

What was the signup to the Nokia store like?

I have been signed up as a Nokia developer for quite a while already, so there hasn't been too much paper work, or forms to fill out. I can't remember what I needed to do originally, so there might be more bureaucracy than I remember. I'm sure there will be more forms for me to fill out in the future too.

Technical parts.

The app is available as a J2ME Java app, a Meego app, and a WRT app. Nokia has so many different platforms now! They are adding a couple more platforms too. Luckily HTMLish apps "work" on all their platforms.

Our Content Management System(CMS) is written in Python/JavaScript/C/SQLite and uses pygame for image processing. The CMS part is still in development, but is slowly getting there.

We use a form based system to define schemas. I wrote about this in a "Validation through html forms" blog post, and mainly in "Using a html form as the model". Basically this allows us to create a html form, and then that's it for the content type. No need to mess around with schemas, or python models, just create a html form, and that is it. Since I usually follow a design led approach, the html forms are usually done before anything else.

The data is stored in a mini database system I built which I detailed in the blog post "sqlitepickle SQLite Vs pickle". It lets you use a dict like interface to save python objects. It's basically a very simple key value store built on python stdlib modules.

We're using JavaScript on the server side as I detailed in a blog post "server side jQuery" to generate the html. The nice thing is I can use jQuery plugins on either the client side or server side depending on the need. Including form validation, and other goodies. I see it as a replacement for a templating language.

This app as it stands doesn't need much of a CMS, but future apps we have in development require a beast of a system in order to process the vast amount of data - and squeeze it down onto various different formats ready for distribution on multiple channels.

Well it's out there now. So hopefully it'll help some fellow vintage treasure hunters find some good stuff in Paris.

This is part of a series of blog posts on "my journey getting published on various app stores".

Halloween pumpkin juggling published to intel appup

2011-10-28T09:04:00.000+01:00

So it looks like the little Halloween game we made has been published on the intel appup store.

It is listed here:
http://www.appup.com/applications/applications-Halloween+Pumpkin+Juggling

Getting into the Samsung store

2011-10-21T10:13:00.000+01:00

So, I registered for the Samsung app store 'thingy' the other day.

Samsung have a number of platforms themselves, not just one platform. Including Bada, Android, Tizen, and windows mobile 7. They have tablets, TVs, laptops, mobile phones... and probably other things too (maybe internet fridges?).

It seems most of the platforms support web apps of some sort. Either through wrapping with phonegap (Bada, Android, win7) or just accepting an archive of the files.

Once joined you can distribute free apps through there. Which is quite lovely really. They have their own advertising program, and I'm not sure if you're allowed to do In App purchases with them.

It seems for TV, you can only distribute to the US. Not to other places around the world. No idea when or if that is going to be possible. Yes, their SDK does include support for 3D content. After speaking briefly with someone(I didn't get their name) at a JavaScript meetup I found out you can try out apps on your own TV/DVD player without having to release to the store.

In order to sell applications, you have to submit a bunch of company paperwork - and sign over your first born child*. They don't currently charge you join though, which is nice compared to some other programs.

_{* may be a joke.}

Android developer store, problem. No problem.

2011-10-20T13:34:00.000+01:00

So, I've got my app working on Android. It's not even all that slow. Ok, it is a little slow. However the new webkit upgrade in Android 4.0 should get it going fast. Note that Android 4.0 has approximately 0% of the android users on it at the moment - since they haven't really released it yet.

I had to get touch events working with gamejs, which wasn't all that hard. I had a little chat on the mailing list, about it, then implemented it not long after. I will send a pull request soon to that project soon.

I'd like to do a phonegap implementation of sound for shitsound too, but haven't got around to that. This should get around the html5 Audio issues on the android browsers. But the game is pretty much ready.

So now that it's ready, I went to sign up to the Android market place so that I could try and release the application. I went through the signup process, and paid my $25USD. I even got sent a receipt. But then something weird happened. I followed the link, to try and login... but it just sent me back to the signup page again.

The help suggested I try all sorts of things, including just signing up again with another credit card... and if all that fails finally to send them a support request. So now I'm waiting on the famous Google Support to get back to me to fix up my account... or so I thought.

I tried one more time, but this time with another browser. Somehow it had gotten my personal account mixed up with my business account. So I cancelled the registration that got mixed up, and then reapplied again after cleaning cache and clearing cookies.

So anyway... long story short, my Android Market account seems to work now.

I linked up my google checkout account to it - the same one I used for the google chrome web store. Then it told me I can now sell apps on the android store.

So now I have to actually try and release this game, and get it through the validation process. Luckily I heard the android validation process is really quick.

notes on chome, and android.

2011-10-17T15:43:00.000+01:00

One thing I forgot to mention was that my flash sound for chrome did not work after uploading to the chrome web store. I made the change at the last minute to use flash for sound, because chrome sound was bad for me. Of course, a change made at the last minute will be a bug! Luckily the bug was reported to me, and I fixed it quickly. This made me glad that the chrome webstore allowed very fast updating. I can deploy updates there as quickly as uploading to a website. It also made me remember the importance of testing from the users perspective. Always check you changes on the live version that people use, rather than just on development servers or simulators. You need to get into the users shoes and experience it the same way they do.

I did a quick port to android, and tested it on my Advent Vega tablet with android 2.2. Surprisingly the game ran fairly well. There were a couple of modern javascript features I was using that did not work, but I quickly found those bugs (html5 data attributes was the main one).

I also needed to add touch support to the game, since so far it only works with mouse and keyboard. You can play it with the mouse emulation that the browsers support by tapping on the screen to move the player around. However it would be much nicer if it worked by dragging your finger around.

I have an android environment setup from some previous projects, so I'll be doing that port next. I'm only going to release it for very modern androids that have a fast enough browser in it. Canvas and javascript have changed so much in the last year, and last years android just isn't good enough (neither is todays really... we wait for android 4 for decent graphics and audio support). The other point in favour of releasing on android next is that publishing to the android market is a very easy thing to do.

I'm not sure about doing an iphone port yet, since Halloween is 13 days away - and I am doubtful I could get it through their store acceptance process in time. But I'll make the decision on which store goes next after the android port is done.

I decided to up the $ amount for the Ludumdare October challenge, since the game has made over $1 now. So my next personal challenge amount is now $1000 in October.

beta testing intel appup apps

2011-10-17T09:39:00.000+01:00

So, I've been going around in circles the last few days with the validation our app on the intel appup store.

The support people escalated my request, and it looks like they have fixed a problem with the validation program. It looks like their automated validation program had a bug.

In the mean time I've gone through the beta testing process. This is where you can send your app to some beta testers and ask them to test for you. If they already have the intel appup installed on their computer, your app appears in their "My Apps" section. Then you can click on it, to try and download+install.

Just like the Mac OSX Application stores, if the download fails halfway through the system fails. This is 2011 mind you, so you'd hope that people had figured out that internet downloads could possibly fail, and might need resuming. Especially on mobile netbooks which are using unreliable 3g wireless internet connections.

As far as I can tell, there is no automated feedback collecting system in place. Which means you will need to manage that feedback yourself. Either by emailing your beta testers or using your own in game beta feedback forms. If you're going to be releasing on multiple platforms, you'll want to create your own beta testing process anyway. Since not all platforms make feedback easy, or possible at all.

It took me quite a while to actually download anything on their store. Even small programs took a while. I guess they are having connectivity issues with their download servers. A 0.42MB app took over 10 minutes and multiple tries to download successfully.

The apps the html5 encapsulator by intel creates all include their own web browser. Which means that every app is at least 10MB big, not even including your app data. This is compared to other platforms, you mostly only need to include your actual data, or have a thin wrapper - like phonegap - which doesn't take up too much space.

Anyway... fingers crossed it passes their validation process soon.

'Halloween Juggling Pumpkin' published to the chrome web store... I think.

2011-10-14T10:54:00.003+01:00

Well, I clicked the publish button, and it seems the 'Halloween Juggling Pumpkin' app has been published to the chrome web store. I'm quite happy that it is up!

Here is a screen shot of the menu screen.

Well, that wasn't too hard to do after all.

Here is a screenshot of some game play.

Here is the lovely store URL it gave me for the store:https://chrome.google.com/webstore/detail/ndnpdjgfifincamlnbidkagoljbiagcb

This was way less work compared to the intel appup store, which I'm still having issues publishing it. To be fair, their store is still in beta for html5 apps.

encapsulator app 'binary validation failed' no other details.

2011-10-14T08:37:00.000+01:00

Just got a rejection notice for my intel appup app, with 'binary validation failed'. There were no other details, which seems to be typical for this program.

I've sent them some messages for more information, so I'm expecting a few days of waiting before anything happens.

chrome audio sucks

2011-10-14T07:37:00.002+01:00

Chrome Audio is worse than firefox, safari, and internet explorer Audio.

There are cracks, pops, and sounds cut short. It seems there's also not many sounds that can be played at once.

Luckily I can use jquery.shitsound and soundmanager2 to prefer using flash audio over a javascript bridge. This has a slight delay compared to firefox/safari html5 Audio, but is better than the built in chrome Audio.

export scripts for app

2011-10-13T14:44:00.001+01:00

So today I've been writing a couple of export scripts for my October challenge apps.

I set the config in the main html file using html5 data attributes. Then read that config on app startup. This seems like a fairly easy way to customise the app depending on platform. My export script selects the correct html for each platform on export.

I have to create the correct data sets for each platform. The right sound formats (ogg, mp3, wav), and the right sized images (From 1080 HD to 320x200 screen). As well as select the correct fonts. I can't just include all the assets for each platform, since there are size restrictions. As well the source, full sized raw data can be over 5GB for one app (for my content heavy app).

Then I have to package them up depending on platform, with manifest files and other strangeness. Or I need to zip them up in a zip archive.

I'm not completely automating the process yet. I'll wait to see if the apps get any traction before investing time in complete automation. Completely automating releases is essential for supporting lots of platforms. Without it, your whole time can be spent releasing, rather than making the apps better.

Signed my app with certificate, and submitted for validation again.

2011-10-12T10:00:00.000+01:00

So, I managed to sign my app and submit it for validation on the appup store.
It took at bit of messing around, converting certificates, and installing lots of microsoft software, but eventually got my msi signed, and submitted to the store.
That made me happy. Now I’m off to drink wine!

Intel appup progress. Got certificate, looking forward to webkit upgrade.

2011-10-12T06:30:00.000+01:00

Well, some good news on the intel appup front.
I finally managed to get my certificate, so I should be able to now sign my apps to upload. Unfortunately I used firefox to download the certificate, which means that I need to convert files in order to get into the microsoft fomat needed for code signing. So I’d recommend anyone else use IE8 to save that hassle.
As well, I’ve been reporting my problems with the webkit they use in their forums. They just gotten back to me, and say they hope to use the latest webkit (which should include a year of webkit bug fixes and feature additions).
So now, I’m going back to trying to submit my app again. I hope that it gets through!

chrome web store

2011-10-11T02:30:00.000+01:00

We’ve started to look into distributing our app via the chrome web store.
I’ve signed up for a developer account, and started reading the documentation on how to publish apps. So far it has been easy. All of their forms have worked, and so far the documentation is easy. This time I’m trying to do as much of the administration up front. Things like validating my account and setting up payments. Only after that works will I actually attempt to publish my app.
Let’s hope it’s easier than the intel appup program, which I’m still not able to publish my app through.

Applied for certificate, thinking about what to do next?

2011-10-10T09:30:00.000+01:00

I got an email back from Comodo about the code signing certificate, and now their form works. Then I had to submit my company certificate, and then have to wait for up to 2 days to get the certificate.
I probably should have realised I needed this certificate before, and applied earlier. Oh well. For some reason I didn’t remember this was needed.
Next I’ll begin packaging it up for the google web chrome store, so maybe we can sell some on there too. I don’t think that will take all that long to do.
Finishing the iOS port might be hard since I’ve noticed the performance of JavaScript and canvas on old iOS devices is terrible. I could just make it available for iOS 5, and the latest iphone 4 phones though… they might be fast enough. I’ve been exploring possible optimizations for it too. So I’ll see how that goes.
I’m in the same situation with the android port. The current web browsers on there are fairly garbage for canvas performance, so I’d need to do some performance work before it would work on there. Making it available on the latest versions of Android is also one possibility. I know an android application is likely to be accepted faster than an iOS one, so that might sway me to release there first.
A Mac application is also possible I guess. For that I’d need to figure out if there is a UIWebKit wrapper that is easy to use, and quick to integrate.
The other option is to finish the other intel appup app I have in development. But porting this game to other platforms seems easier I think. Once the whole process is done I’ll have an idea of how long the other app would take to do for the appup platform.

Battling Intel Appup submission

2011-10-08T08:30:00.000+01:00

Well, I managed to prepare my app for submission to the intel appup store.
I first had to figure out bugs: Sound doesn’t work. Also fonts don’t work. This was very disappointing considering that I’ve been working with a musician on sound effects and music. So we’re not able to use any of that work we did until they fix sound. I even tried to use a flash based sound player, and use the embed tag. None of it worked.
Also the lovely font we selected does not work. Trying to use a font-face crashed the browser completely. So instead I had to use ‘Arial’. arg.
I’m not sure what sort of html5 games and apps they expect if fonts and sounds aren’t working?
It seems they are using a 1 year old webkit implementation, so I hope they will upgrade their encapsulator soon. One year in web browser years is about 70 human years.
Anyway, we decided to submit the app anyway. The font makes it look a bit ugly, and without sounds it looses a lot – but at least the game play works.
Then I filled out a few pages of forms, prepared images, wrote copy, and did all sorts of guff so that I could upload the app. For some reason they required silverlight or google gears to upload the file (weird). Not sure why they couldn’t use a normal html upload form, or even a flash upload form.
Anyway… all that got done until I had to apply for a security certificate to sign the files. I fill out another form of my personal details, press submit and I’m greeted with a great error message: “An error has occurred. -1″
Oh well. Sent off an email for support, and now I wait.
Finally blocked after a couple of long days and nights – stopped by the error “-1″.

EULA we can use? (End User License Agreement)

2011-10-07T06:30:00.000+01:00

Hi,
got my app up to the packaging stage now…
Does anyone know of a decent (End User License Agreement) EULA we can use for our games? If you know of ones we can use please leave a comment.
cheers,

I'm in

2011-10-06T13:21:00.000+01:00

October came around, and I started working on a game.
It’s javascript/canvas based, instead of my usual pygame. But, I’m using gamejs which uses the pygame API, so I feel pretty much at home.
We got most of our new graphics and sound into the game today, and all of the game play is done.
It’s not running fast enough on my ipod touch 2g, but I think with some work it could be playable. So we’re aiming to release on the various web stores first. Then perhaps improve things for the mobile platforms.
I had to work on improving a sound engine I call jquery.shitsound, or just shitsound for short. Which uses various web browser sound APIs to work. I’ll have to get it using the phonegap framework to play audio too, since html5 Audio on mobile platforms is simply broken.
We’re in, I should say. Since I’m doing a collaboration with Spencer Sternberg ( http://www.spencersternberg.com/ ) video game audio and music productions.
The second app I’m working on next after this one is released I’m doing as a Powered By Bees production ( http://www.poweredbybees.com ). It’s less game related, and more content related, but I still think it will count for this October Challenge.
Thanks Ludumdare! This October competition has really given me a big kick up the backside.

Server Side jQuery - ssjquery

2011-10-02T10:22:00.000+01:00

I've got a new server side jQuery project up. It's an extension on some ideas and projects over the past couple of years. It uses the node.js javascript interpreter, and node libraries like jsdom, and htmlparser to do all the work. It can be used within various projects that are not javascript via a command line process or a web service.

This is what the pipeline looks like on the server side.

json + html + server side jquery => rendered html

Instead you can run it on the client side so you don't need a server for development.

json + html + ssjquery run on client first before other scripts => rendered html

A webservice creates the json, and your server side jQuery scripts populate the html. Think of it as an industrial strength templating language that millions of front end developers know how to use.

Here are my server side jQuery slides I gave at the reject.js conference in Berlin last week. It wasn't a great talk, but I think it's helped me understand a little more on how to explain the concept to people. I purposely did not put any code in that talk, so I'm providing it here. Instead I concentrated on why you'd want to do it. I might try and write up my talk notes as a separate blog post.

Here is the project page: https://bitbucket.org/illume/ssjquery

There is a python buildout script which downloads node.js, and all required libraries into a directory ready to run (with the npm package manager). In there it creates a bin directory for the bin/ssjquery script for the command line version and bin/ssjqueryServer for the server version.

As a bonus, I also install the uglifyjs binaries, and coffee script binaries - but they are not required.

Here is how you run the python buildout...

$ hg clone https://bitbucket.org/illume/ssjquery
$ cd ssjquery/pythonbuildout
$ python bootstrap -d
$ ./bin/buildout
$ cd ..

Here is how the command line script ssjquery is called to output rendered html.

$ cd js/example/

$ ../../pythonbuildout/bin/ssjquery ./ server.json index.html jquery.js yourServerSide.js yourServerSide2.js > renderedHtml.html

Here is the ssjqueryServer script which listens as a webservice to render html.

$ cd js/example/
$ ../../pythonbuildout/bin/ssjqueryServer ./ server.json index.html jquery.js yourServerSide.js yourServerSide2.js

Server running at http://127.0.0.1:8124/try url: http://localhost:8124/?basePath=./&serverJson=server.json&serverJs=jquery.js,,,yourServerSide.js,,,yourServerSide2.js&templateUrl=index.html

That runs a web server which renders the files for you. It's a bit quicker than calling a process each time you want to do things. There is a possibility of embedding the javascript interpreter inside python (or php/java/etc) too.

Structure of html

The structure of your html is something like this:

include jquery
include code which will be used on server side and client side.
include server side jquery, and run it if have not processed it server side.
include client side jquery

There is a minimal index.html example here if you want to see what it looks like.

This lets you use the same html in either server side processing mode, or client side processing mode and the results should be the same.

pygame on raspberrypi, pyconuk was awesome

2011-09-27T11:22:00.000+01:00

Thanks to @ntoll I had a chance to play around with the raspberrypi alpha board at the pyconuk conference (which was a great little community conference with a big heart).

That is it on the pink bubble wrap. It's hooked up to a monitor via the digital HDMI video port, with a keyboard and mouse connected up via USB.

We only had a few hours to play around with it, but managed to try a few things out. Whilst we were there playing around with it, a whole bunch of people popped there heads in to have a look and ask questions. Lots of enthusiasm for the project, and also the machine, but also a bit of cautiousness about the project.

In another room, there was a talk going on about how the BBC was going to help bring a programming education project into the schools of the UK again - like they did with the bbc micro. Tweets started to fly around on the tweet projection screen about it, but then at the end of the talk we found out it was a hoax. Haha. It was more a "wouldn't it be awesome if..." kind of talk.

So, what did we find out in our exploration of the raspberrypi? Firstly, the rasberrypi had a debian linux distribution running on it, and many packages were available to install. Including an old version of pygame. Sound wasn't working for us, there were none of the various sound systems running. It turns out the analog output port is currently really terrible. With a 1bit DAC, I guess it will sound something like a PC speaker or maybe slightly better. However, digital audio is going to come from the HDMI port along with the video. So that should sound really good, assuming it is hooked up to a TV with HDMI. There's also the option of using a $3 USB sound card - which the project reccommends if you want audio input (eg, a microphone).

One interesting thing is they have not put a real time clock on there. So when you reboot the system it has the wrong date all the time, which really messes with things like the packaging system, and the file system. However, once connected to the network it can grab the time from there. You may be noticing a theme... they've cut back on components to keep the cost low. I think that it's a great approach in order to get something useful at the price point they are aiming for.

The alpha board had no WIFI hardware, but we found an ethernet cable and plugged it into my laptop, so we were able to share the laptops WIFI internet connection through it. @ntoll had the pygame package already installed, but it was an old version that didn't include the example programs.

@ntoll has previously blogged about his raspberrypi

So we set the debian package manager the task of installing the gcc compiler, and various other things needed to compile pygame from source(SDL, ffmpeg, portmidi, etc). It was downloading packages over the network connection at about 1MB/s which is pretty fast for a 10/100 ethernet port (which seems to be attached via USB), and then out over my laptops wireless connection. It did take a while to install the packages on the SD card though. Perhaps a faster SD card could be used, or maybe the SD card driver could do with some speed improvements. However, it wasn't too slow, definitely usable.

All the dependencies installed with no issues... winning! Then we started up the compiler, and we got pygame to compile with gcc after a fair wait. This was quite impressive, compiling stuff on the raspberrypi itself, rather than having to use a cross compiler.

So X was running, but so far there was no hardware acceleration of graphics. I guess in time, someone will be working on the video drivers. This little machine is designed around a GPU, so it is definitely capable of some fast video. So web browsing with iceweasle (rebranded firefox) was a bit slow. It was pretty much unusable whilst pygame was compiling, but usable when the machine wasn't doing much else.

@ntoll got his xmas card pygame program he wrote for the london dojo last year running on the raspberrypi. It was going a bit slow, but after a few tweaks it ran at an acceptable frame rate (if not exactly smooth). So this is good news, because even though the graphics drivers are not hardware accelerated yet, the little machine can run simple programs at an acceptable speed.

I also tried running a very CPU and graphics demanding pygame game zanthor. But it wasn't able to run above 1fps. The zanthor game has trouble getting 30FPS on a 5 year old laptop with good graphics drivers, so this wasn't unexpected on an alpha board that doesn't even have its hardware acceleration available yet.

We also ran the pygame test suite, and most everything passed except for the sound related tests (and things like the camera, and joystick tests, since it doesn't have a camera or joystick attached). Apparently there are pins that could be used to attach a camera, since this is a mobile phone chip. There's a lot of work that could be done to pygame to make it run better on the raspberrypi, but I'm happy that it's mostly working already.

I also tried running an OpenGL demo, but it ran really slowly. So I'm guessing that hardware accelerated OpenGL was also not working on the distribution that it came with. But they have shown it working in other demos, so it is only time before it is working in X. Also, I think the board supports OpenGL ES 2.0 which there is not currently a python wrapper for (please leave a comment if you know of a project for this).

There seemed to be about 189MB of ram on this board according to 'top'. I guess it was a 256MB ram model that had 60ish MB of ram taken up by the GPU.

@tjguk was there too hacking on his quiz program. It was interesting listening to him talk about his program, and what he was doing with it. But I won't go into it here, since he might want to blog about it himself.

Also at the conference was Garry Bulmer who was talking about the ARM board that he has made. Previously he has been using the arduino micro controller platform as a robotics teaching platform for kids and adults. I'd like to follow his project, and look forward to seeing it progress. He talked about wanting more processing power to do things like process cd quality audio, which the arduino can not really do. His arm cpu board was aimed at also teaching hardware related things, so his board has more usable GPIO inputs and output pins for controlling electronics. Whereas the raspberrypi board is not currently designed to work well for arduino uses. It is good to see competition in the cheap educational ARM board space, but I think the two projects have slightly different use cases.

One concern some people had about the raspberrypi project is there seems to be a link between the broadcom employee and the chips they use on the raspberrypi board. This could be seen as a conflict of interest. However, maybe it doesn't matter. Perhaps it is even a good thing! Since then there might be better support from the company because an employee might have better access to docs, and such things. At least there will be someone on the project who has access to the tools needed to fix bugs. There were also concerns about the lack of source code for the chips 18MB binary blob firmware. What does that code do really? I think the project wants to open that source code, but so far it is not available because the company does not want to open it. As a pragmatic option, I still think the raspberrypi is ok. It's not completely open hardware, but it's open enough to be useful for an educational project for creating software. The projects goals are to provide a cheap PC to children, so they can use it themselves to learn - it is not about creating an open hardware platform.

I'd love to get a development board, but the raspberrypi foundation only made 50 and they are all distributed to developers so I'll have to wait like everyone else until they are up for general sale. I think they are aiming to make them available in a few months.

There were plenty of other interesting things going on at pyconuk this year too. Including talks about testing(mocks, extending unittest and such things), a realtime web socket game (kwizzle) that uses tornado and mongodb, fluiddb (kinda wikipedia for databases), a visual programming environment (The Larch Environment), a django work shop, and the always good 5 minute lightning talks - where random people get up and talk about something for 5 minutes. Amongst plenty of other things going on. The volunteers put on a great conference again, with very little resources. I wish I could have talked with more people at more length, but there's not really enough time to do so. There's always a creative buzz at these things. So many creative, passionate people talking about their projects in one spot is quite inspirational.

jquery.shitsound.js - a shit sound player for web browsers

2011-09-21T12:55:00.001+01:00

Yesterday, I made the first release of shitsound.

jquery.shitsound.js is a shit sound player for web browsers.

Listen (demo): http://rene.f0o.com/~rene/shitsound
Source code: https://www.bitbucket.org/illume/shitsound

Used like this:

$.snd.init({}, function () {
        // we are ready to play sounds!  Brilliant.
        $.snd.play('hello');
});

shitsound only does a few things with sound::: play, stop, stopAll
jquery.awesound.js might do:: changing volume, pitch, playing with a different tempo, playing in stereo, 3d sound, etc.

jquery.shitsound can not do any of this fancy stuff. It can just play sounds, and stop playing them.
shitsound detects which implementation to use at init time.

html5, using the html5 audio tag.
embed, for using the embed tag.
soundmanager1, for old flash 7.
soundmanager2, for modern flash (8,9,10+).
empty, pretend implementation that does nothing.

It currently requires jquery, since I use jquery. However it could quite easily drop the jquery dependency with some work. Other dependencies are soundmanager 1 and soundmanager 2, as well as (optionally) swfobject. These are all bundled in ready to use though.
Each implementation is kept separate in the code base, and it is fairly easy to extend with other implementation. The html5 implementation is separate from the soundmanager1 implementation for example. I plan on implementing a phonegap implementation in the future.
If you'd like to see another JavaScript audio engine supported, please let me know.

Yes, it does support netscape navigator 3+. As well as old versions of internet explorer that don't have flash installed, or have old versions of flash installed.

preparing your audio

Because sound support on the web is shit, you need to prepare your audio in various different formats for it to work cross platform.
You need a .wav file, a .mp3 file, and an .ogg file to support all platforms.
It might not be a good idea to include big .wav files for large amounts of audio like songs though. Because they can be 10 times larger than a lossily compressed mp3 or ogg file. But if you want full browser support, then .wav files are required.
The ffmpeg is quite a common powerhouse of a multimedia tool. It is available on OSX through brew, or macports. It can even convert audio for you.

ffmpeg -i file.wav file.mp3
ffmpeg -i file.wav file.ogg

preparing your webserver

If you use nginx you might need to modify your /etc/nginx/mime.types file for oggs::: audio/ogg ogg;

Other web servers may also need mime types being added.

future

Maybe these things will be done in the future.

volume control. This is possible with all backends.
Better browser detection, and blacklisting of bad html5 audio engines.
wider testing and debugging.

pygame sprint at pyconuk, and virtually. Friday 23rd to Sunday 25th of September

2011-09-14T18:51:00.000+01:00

There is going to be a pygame sprint at this years pyconuk.We're going to tackle whatever the participants who turn up want to hack on. That might be porting to a new platform like the $25 rasberrypi computer(there's going to be a couple at pyconuk), or improving the pygame android support. Or it might be polishing off one of the new modules like the new freetype font module, or perhaps implementing a brand new module. Or maybe it's jazzing up the documentation, or working on a new import pygame.examples.I'm getting there on Friday, so I hope to do some hacking then, and we might be able to get some sprint space for the Friday(conference propper starts Saturday, but some people are getting there early) - otherwise it will have to be in a cafe or something.

Also going on during the conference is the python core sprint and CKAN sprint. It should be fun to join in on those sprints too. Of course there will be talks, work shops, an unconference room, lightning talks (5 minute talks) and coffee fueled hallway conversations. There's even going to be a Code Dojo - london style, and of course - most importantly: lunch/dinner.

pygame recently migrated to mercurial and bitbucket: https://bitbucket.org/pygame/pygame We'll be able to commit changes to there or work off branches and submit pull requests later. There is a pygame hacking guide which we will make notes on any questions people have with working on the pygame code base.

It will also run as a virtual sprint in the irc chat room (freenode #pygame), so if anyone wants to join in remotely they can. If you only write python code, there are plenty of parts of pygame that can be worked on. For anyone who turns up I can show them the basics of CPython modules if they already know C.

We'll try and tweet sprint goings on via the @pygame_org twitter account. Hope to see you there.

Let's make a shit JavaScript interpreter! Part Three.

2011-08-24T01:51:00.012+01:00

Let's put the Research into R&D. I guess it should be called Shit Research to go along with the name of this series (p1) of articles (p2) I started one year ago. It takes a lot of time to read over hundreds of thousands of lines of undocumented C++ and java code, so part three took much longer than expected.

Let's explore existing JavaScript implementations.

In this part we are going to take a small digression to look over other JavaScript implementations. I'll provide a short description of how each JavaScript implementation is made. We can use the Architectural knowledge from each implementation to inspire our own implementation.

Another benefit of researching each implementation is that we can find all the different tools they use. For example different test suites.

Make sure to read the URL's for each implementation to find out more information about each one.

There is a list of ECMAScript implementations at wikipedia. We will not cover all of the ones listed there. If like me, you may spend a few hours or even days reading through the links from there.

narcissus - js in js.

Narcissus is a javascript implementation written in javascript (with some SpiderMonkey language extensions). It is written by the same author as the original JavaScript implementation back in 2005-2007 (Brendan Eich). In mid 2010 the Narcissus code has been taken up again by the Mozilla project to make researching JavaScript changes easier.

It is a good implementation to study, since it is fairly small, and is meant to be easy enough to read.

It uses a hand written parser, not a generated one.

The original narcissus source repository.

The new repository of narcissus, and two articles about it.

There is a port to python of the narcissus parser called pynarcissus. The authors of pynarcissus found it difficult to port the rest of the interpreter since it relies on JavaScript features itself.

It weighs in at about 7000 lines of code counted with wc -l. I had to count lines in this way since my SLOC counter does not seem to count javascript.

Spider monkey.

Spider Monkey is the original JavaScript implementation used by Netscape, and the Mozilla project.

SpiderMonkey is a production grade, high quality JavaScript implementation.

It also has excellent documentation. Especially the js/src/README.html file which includes a design walk through.

"""The compiler consists of a recursive-descent parser and a random-logic rather than table-driven lexical scanner. Semantic and lexical feedback are used to disambiguate hard cases such as missing semicolons, assignable expressions ("lvalues" in C parlance), etc. The parser generates bytecode as it parses, using fixup lists for downward branches and code buffering and rewriting for exceptional cases such as for loops. It attempts no error recovery. The interpreter executes the bytecode of top-level scripts, and calls itself indirectly to interpret function bodies (which are also scripts). All state associated with an interpreter instance is passed through formal parameters to the interpreter entry point; most implicit state is collected in a type named JSContext. Therefore, all API and almost all other functions in JSRef take a JSContext pointer as their first argument.

The decompiler translates postfix bytecode into infix source by consulting a separate byte-sized code, called source notes, to disambiguate bytecodes that result from more than one grammatical production."""

Google V8.

Written in C++, javascript and assembler. Uses hidden classes, and generates machine code at run time.

The parser is hand written parser in C++. It's not generated.

'Preparses', which creates tokens. Then creates an AST by parsing. Finally compiling the AST. parser.css is where all the parsing happens.

The projects documentation is quite limited - so reading the source is the best way to get in there. There are some videos which describe the architecture, and engineering decisions behind the choices taken.

http://code.google.com/p/v8/

V8 weighs in at a mighty 605,962 SLOC. Making it the largest, biggest, badest V8 JavaScript engine around.

JSLint.

JSLint is written in JavaScript and uses a TDOP approach to the parser like we are using. I won't discuss this, since it is documented well elsewhere.

Rhino

Rhino can run as either an interpreter or a java byte code generator. It's hosted by the mozilla project (who make firefox). It complements their C++ implementation (spidermonkey) and their JavaScript implementation (narcissus).

There are a few things which show it is a very mature as well as modern implementation. The project was started in 1999, and has been developed ever since. It has partial support for JavaScript 1.8, and ECMAScript 5 standards. Weighing in at 50,551 source lines of Java code (SLOC), it can be considered a very large code base. Another modern feature is that it supports the CommonJS javascript module standard. Despite still being hosted in cvs, it's still being maintained, and features are being added regularly.

The Rhino JavaScript team maintains a library of tests and other documentation for JavaScript. Tests are being shared with the other JavaScript implementations within the mozilla spidermonkey project. There have also been parts which have been ported from spidermonkey - such as the hand written parser.

A handwritten scanner they call TokenStream creates tokens, and then parses those into an AST. Despite being mostly hand written, some parts are generated. Specifically the stringToKeyword method, which detects keywords is generated somehow.

The documentation of the architecture of the project is limited. There is however some API documentation. With a couple of modifications to some ant build files I was able to build it, as well as even make a few small modifications.

The wikipedia Rhino page has some great information on the rhino javascript engine.

KJS

KJS is the KDE JavaScript implementation for the konqueror browser. It was the parent of the JavaScript implementations done by Apple Computer, inc. I won't go into any detail on this one, since I'll cover JavaScriptCore instead. KJS is written in C++, for the QT library.

JavaScriptCore, Squirrelfish, Nitro

You can browse the source here: http://trac.webkit.org/browser/trunk/Source/JavaScriptCore

This uses a hand written lexer(tokeniser), and a hand written parser. The code structure of the parser and lexer looked eerily familiar. The code base is mostly written in C++ and is quite massive. 140,837 SLOC

There is lots of platform specific code, but it also has a jit, uses byte code, and an interpreter. There is also lots of development code in there for things like debuggers, and profilers.

Closed source JavaScript implementations

There are a few JavaScript implementations that are closed source. The two main ones in widespread use are the ones from Microsoft, and the ones from Opera.

They have however published papers and blog posts about their implementations. I won't cover them any more, because not as much can be learned without the source code.

pypy javascript

The pypy project started a javascript interpreter now too.
https://bitbucket.org/pypy/lang-js/src/de89ec32a7dc/js/javascript-interpreter.txt

The description of the project mentions it's currently using the spider monkey parser, but it appears to generate one using a parser generator provided by pypy. Using a EBNF grammar file. It also creates an AST.

It works for some simple javascript programs that don't use the javascript standard library. I'm not sure of the future of the project, since it appears it was a GSOC project which has now finished, so there might not be any full time developers left on it.

It's written in RPython (a restricted subset of python) and python. Running on top of pypy, it should theoretically be able to take advantage of that platforms jit and garbage collector.

This project makes use of some JavaScript tests and benchmarks from other projects. Specifically some benchmarks from v8, and the language shootout website. It also includes the "ECMA 262 Edition 1" tests.

It weighs in at 5452 Source Lines of Code (SLOC). Which is much smaller, but the implementation is also not complete, so that is to be expected.

So what have we learned then?

We see that most of the implementations use a hand written parser. We also see that the implementations in js and python are much smaller. So despite them being incomplete, I think it proves that it should be feasible to make our shit interpreter in python. We don't need half a million lines of C++ to do our project.

We have also learned that there are test suites available, which should help us out a lot. In fact, many of the implementations share the test suites. Having a test suite already available makes it way easier to write an implementation of something yourself. It acts as a guide to development, and also reduces the time for testing since a lot of it can be automated.

Exercise for next time

Choose One(1) of the implementations, build it, run it, and modify it slightly to do something different. Try and run the tests that come with it.

bugzilla bugs to bitbucket issues

2011-08-21T14:56:00.004+01:00

As part of the pygame move to bitbucket, we are converting a bug database stored in bugzilla.

Work is progressing on the issues migration:
https://bitbucket.org/illume/bugzilla_bitbucket

Bugzilla is more comprehensive compared to bitbucket. So not all things can be migrated from the bugzilla bugs to the bitbucket issues.

Also, the bitbucket API is limited in what it can do. You can't add user comments or add attachments.

To get around the attachments part, I made the migration script create a directory of attachments, and then link to them in the bug text. Then I've extracted the attachments from bugzilla and put them online.

To hack around the lack of comment adding in the bitbucket API, I'm just combining all of the comments into the content text.

Other features bitbucket issues API does not have are priorities, and platforms. The issues features _does_ support priorities, but it doesn't look like the API is set up yet. You can't add platforms (like 'Windows XP SP2') at all it seems. You also can not add free form tags of any sort. Another couple of basic things you can not set with bitbucket issues are the date, and the bug id.

So there you have it. This code worked ok for us.

This code might be a useful start for anyone else wanting to start on a migration. Feel free to go fork it yourself: https://bitbucket.org/illume/bugzilla_bitbucket

bye bye launchpad

2011-07-17T10:27:00.004+01:00

I'm starting to move most of my projects off launchpad onto bitbucket.

launchpad just hasn't gone in the direction that is useful for me, or my projects.

It still has some nice features, such as packaging farms (PPAs) and of course bzr integration. However, github and bitbucket have just been moving more towards making it easy for developers compared to launchpad which is more geared towards for distributors who work with an upstream. Last year I hoped launchpad would improve after it went open source, and it has improved. Just not really in the direction useful for my projects.

Many people these days are able to use hg, or git... but still very many people have never even heard of bzr. I feel bzr+launchpad was holding one project back a little from gaining collaborators because of this. By that logic, I should be using svn for it... but I don't want to do that :)

The documentation to convert repositories to mecurial is on the hg wiki. The conversion seems to mostly have worked ok for me on my python newsletter project. Except it did not put the correct email address attached to each of the commits. Oh well.

shadertoy.py opengl shading language toy shader

2011-07-04T20:50:00.006+01:00

A couple of months ago I made a python port of the WebGL shadertoy. The shadertoy website is written in JavaScript and the opengl shading language.

Here is the python port: http://rene.f0o.com/~rene/stuff/shadertoy.py

Maybe it'll be useful for someone learning shading languages through python. Or not.

It's a fun demo anyway. Press the 'Anykey' to change between shader. You'll find the 'Anykey' on your keyboard. If the Anykey you press is Escape, then the program will 'quit'.

Requires 1 tsp of pygame, and a cup of pyopengl.

cherrypy daemontools

2011-07-02T10:50:00.008+01:00

Cherrypy and daemontools work nicely together. However, I have not found any instructions on how to use them together on the interwebs... so I've written up some basic notes for people who love cherrypy and daemontools.

daemontools is a collection of tools for managing UNIX services. It can be used to automatically start, and restart processes on a unix system.

The instructions to run services in daemontools are here: http://cr.yp.to/daemontools/faq/create.html#run. I'm assuming you have daemontools set up already.

Cherrypy is a python library for making website applications. There are many ways to deploy cherrypy apps, but I will describe how to deploy cherrypy apps with daemontools.

Firstly, you do not need the cherryd tool to use with daemontools. Daemontools does the daemonising for you. But cherryd can be used if you want, since it provides some nice options (just don't use the -d daemonize flag). You may want to use cherryd with daemontools if you'd like to use FastCGI or SCGI. I won't cover cherryd any further in this post.

I'll now show you how to setup daemontools for an example cherrypy application.

Here is a normal helloworld cherrypy app... which I'll put in the file exampleapp.py

import cherrypy

class HelloWorld(object):
    def index(self):
        return "Hello World!"
    index.exposed = True

cherrypy.quickstart(HelloWorld())

This runs on 127.0.0.1 on port 8080 by default. It also runs in development mode (which you should not deploy with, but only use for testing).

Now create a 'theservicedir' directory somewhere.
Also create the following directories: 'theservicedir/log' and 'theservicedir/log/main'

Then create a 'theservicedir/run' file.

#!/bin/sh
exec setuidgid rene /usr/bin/python /home/rene/cherrypydaemontools/exampleapp.py

Note that I specified full paths to python and to your python script. Also see that I got the app to run with the setuidgid program as the user 'rene'.

Then create a 'theservicedir/run' file for logging.

#!/bin/sh
exec setuidgid rene multilog t ./main

Now to start it up we symlink the file into the service directory.

sudo ln -s /home/rene/cherrypydaemontools/theservicedir /service/theservicedir

To stop it:

svc -d /service/theservicedir

To restart it:

svc -t /service/theservicedir

To start it:

svc -u /service/theservicedir

Your app should start up the next time your server reboots. Happy cherrypy and daemontools usage. Please leave any related tips or fixes to these instructions in the comments.

hello jquery.mobile

2011-06-18T12:11:00.005+01:00

Was playing around with jquery mobile the other day, which has planned it's first beta release sometime next week.

jQuery mobile is what jquery+jquery-ui are to desktop browsers. It abstracts a lot of the parts away so that you can write your code without having to figure out all the different mobile browser quirks yourself. It offers a structure to make web apps which are cross platform across current mobile devices and which as a bonus work in desktop web browsers as well.

Mobile style events are provided as well. Like touch and swipe which work across the various mobile browsers is a nice touch.

Another great thing it provides is media queries. CSS media queries are buggy on some browsers so this abstracts out a lot of the bugs on the various different devices.

Supported are most of the mobile devices. Some are supported better than others, and some provide certain features that others do not provide. For example, the ajax style page loading is not done on opera mini. But works on android, and safari mobile browsers.

I'm quite looking forward to the feature complete beta release. It's been a long time baking, and smells quite nice.

Running jslint on saving JavaScript with vim

2011-05-17T09:48:00.002+01:00

autocmd BufWritePost,FileWritePost *.js !jslint <afile>

This will run jslint on your JavaScript files after you save them.

http://vimdoc.sourceforge.net/htmldoc/autocmd.html
http://vimdoc.sourceforge.net/htmldoc/autocmd.html#FileWritePost

You can add the command to your ~/.vimrc

Hotmail, xbox and microsoft live have been hacked.

2011-04-29T15:59:00.012+01:00

update: hotmail have added a my friend has been hacked button, so people can report when one of their friends accounts have been hacked. Also I got my account back after 5 days or so. I don't think any of my contacts lost any money from the scam. Other people who got their accounts hijacked have not been so lucky.

Many people have been having their hotmail accounts been broken into and stolen.

Microsoft writes about it on their security blog.

I've meet a few people over the last week who know of people who have had their accounts stolen.

The fraudsters are sending everyone in their contact list and telling people they have been robbed - and to send money. They say they are in a hotel and the hotel will not let them leave.

A friend told me about how some people call up, and someone answers pretending to be the hotel manager mentioned in the scam email.

I haven't read any media coverage of this, but have heard first hand of people who it has happened to. Could this be related to the Sony break-in the other week?

Unfortunately Microsoft do not seem to answer when contacted about accounts being used for fraud. Their account reset procedure is so slow that by the time an account is recovered the damage could very well be done.

Microsoft are also recommending people create new hotmail accounts rather than go through the verification process to recover a stolen account. Unfortunately this is a rather dangerous attitude to take, since many people have been sent emails and all of the people in the contact list could be victims of crimes.

The xbox live network uses the same authentication as hotmail and other Microsoft online properties and stores. Microsoft uses a single sign on system - so this is a very big security break down.

I have no idea how large the break-in is. However, if I hear about multiple people in real life having their accounts stolen then I think this is MASSIVE.

Sony took their system offline whilst they investigated the hackers. I'm not sure what if anything Microsoft has done.

Please spread the word, and warn others about this break-in.

Example scam email

Here is an example of one of the emails they are sending out:

Subject: My Plight!!! Help

I'm sorry for this odd request because it might get to you too urgent but it's 
because of the situation of things right now, I am stuck in United Kingdom . 
we were robbed at the park of the hotel where we stayed,all cash,credit card and 
cell were stolen off us but luckily for us we still have our passports with us.

We've been to the embassy and the Police here but they're not helping issues at all
 and our flight leaves today but we're having problems settling the hotel bills and 
the hotel manager won't let us leave until we settle the bills.

I need a quick loan?? promise to refund it back once i get home.

Let's make a shit JavaScript interpreter! Part Two.

2011-04-10T11:26:00.011+01:00

Let's make a shit javascript interpreter! Part two.

As a learning exercise, I've begun writing a ~~JavaScript~~ ECMAScript interpreter in python. It doesn't even really exist yet, and when it does it will run really slowly, and not support all js features.

Homework from part One - A simple tokeniser.

We ended "Let's make a shit JavaScript interpreter! Part One." by setting some homework to create a tokeniser for simple expressions like "1 + 2 * 4". Two readers sent in their versions of the tokenisers (ps, put a link to your home work results from Part One in the comments, and I'll link to it here).

Our simple tokeniser

operators = ['/', '+', '*', '-']
class ParseError(Exception):
    pass
def is_operator(s):
    return s in operators
def is_number(s):
    return s.isdigit()

def tokenise(expression):
    pos = 0
    for s in expression.split():
        t = {}
        if is_operator(s):
            t['type'] = 'operator'
            t['value'] = s
        elif is_number(s):
            t['type'] = 'number'
            t['value'] = float(s)
        else:
            raise ParseError(s, pos)
 
        t.update({'from': pos, 'to': pos + len(s)})
        pos += len(s) + 1
        yield t


>>> pprint(list(tokenise("1 + 2 * 4")))
[{'from': 0, 'to': 1, 'type': 'number', 'value': 1.0},
 {'from': 2, 'to': 3, 'type': 'operator', 'value': '+'},
 {'from': 4, 'to': 5, 'type': 'number', 'value': 2.0},
 {'from': 6, 'to': 7, 'type': 'operator', 'value': '*'},
 {'from': 8, 'to': 9, 'type': 'number', 'value': 4.0}]

Code for shitjs.

You can follow along with the code for shit js at:

http://pypi.python.org/pypi/shitjs
pip install shitjs
https://bitbucket.org/illume/shitjs
hg clone https://illume@bitbucket.org/illume/shitjs

After you have installed it, shitjs.part1 is the package for the part1 homework.

What next? Parsing with the tokens of our simple expression.

Since we have some tokens from the input, we can now move onto the parser. Remember that we are not making a parser for all of javascript to start with, we are starting on a simple expressions like "1 + 2 * 4". As mentioned in Part One, we are using an algorithm called "Top Down Operator precedence". Where actions are associated with tokens, and an order of operations. Here you can see the precedence rule (order of operations) with parenthesis around the (1 + 2) addition changes the result.

>>> 1 + 2 * 4
9 
>>> (1 + 2) * 4
12

A number is supplied for the left, and the right of each token. These numbers are used to figure out which order the operators are applied to each other. So we take our token structure from tokenise() above, and we create some Token objects from them, and depending on their binding powers evaluate them.

What's new is old is new.

The "Top Down Operator precedence" paper is from the 70's. In the 70's lisp programmers loved to use three letter variable names, and therefore the algorithm and the variable names are three letter ones. They also wore flares in the 70's (which are back in this season) and I'm not wearing them, and I'm not using three letter variable names!

Sorry, I digress... So we call 'nud' prefix, and 'led' infix. We also call rbp right_binding_power, and lbp left_binding_power.

nud - prefix
led - infix
rbp - right_binding_power
lbp - left_binding_power

Prefix is to the left, and infix is to the right.

Manually stepping through the algorithm.

Let's manually step through the algorithm for the simple expression "1 + 2 * 4".


    >>> pprint(list(tokenise("1 + 2 * 4")))
    [{'from': 0, 'to': 1, 'type': 'number', 'value': 1.0},
     {'from': 2, 'to': 3, 'type': 'operator', 'value': '+'},
     {'from': 4, 'to': 5, 'type': 'number', 'value': 2.0},
     {'from': 6, 'to': 7, 'type': 'operator', 'value': '*'},
     {'from': 8, 'to': 9, 'type': 'number', 'value': 4.0}]

Let's give left binding powers to each of the token types.

number - 0
+ operator - 10
* operator - 20

Ok, so first we have a number token, with the value of 1.0. This is because in our shitjs so far all numbers are floats. Here is a log obtained by stepping through the expression.

('token', Literal({'to': 1, 'type': 'number', 'value': 1.0, 'from': 0}))
('expression right_binding_power: ', 0)
   ('token', OperatorAdd({'to': 3, 'type': 'operator', 'value': '+', 'from': 2}))
   ('left from prefix of first token', 1.0)
   ('token', Literal({'to': 5, 'type': 'number', 'value': 2.0, 'from': 4}))
   ('expression right_binding_power: ', 10)
       ('token', OperatorMul({'to': 7, 'type': 'operator', 'value': '*', 'from': 6}))
       ('left from prefix of first token', 2.0)
       ('token', Literal({'to': 9, 'type': 'number', 'value': 4.0, 'from': 8}))
       ('expression right_binding_power: ', 20)
           ('token', End({}))
           ('left from prefix of first token', 4.0)
           ('leaving expression with left:', 4.0)
       ('left from previous_token.infix(left)', 8.0)
       right_binding_power:10: token.left_binding_power:0:
       ('leaving expression with left:', 8.0)
   ('left from previous_token.infix(left)', 9.0)
   right_binding_power:0: token.left_binding_power:0:
   ('leaving expression with left:', 9.0)

You can see that it is a recursive algorithm. Each indentation is where it is entering a new expression.

Also, see how it manages to use the binding powers to make sure that the multiplication of 2 and 4 is done first before the addition. Otherwise the answer might be (1 + 2) * 4 == 12! Not the correct answer 9 that it gives at the end.

The operations are ordered this way because the + operator has a lower left binding power than the * operator.

You should also be able to see from that trace that a tokens infix and prefix operators are used. The OperatorAdd for example, takes what is on the left and adds it to the expression of what is on the right with it's prefix operator.

Here is an example Operator with prefix(left) and infix(right) methods.

class OperatorAdd(Token):
    left_binding_power = 10
    def prefix(self):
        return self.context.expression(100)
    def infix(self, left):
        return left + self.context.expression(self.left_binding_power)

Pretty simple right? You can see the infix method takes the value in from the left, and adds it to the expression of what comes on the right.

Exercises for next time

Make this work:


>>> evaluate("1 + 2 * 4")
9.0

(ps... if you want to cheat the code repository has my part 2 solution in it if you want to see. The code is very short).

Until next time... Further reading (for the train, or the bathtub).

Below is some further reading about parsers for JavaScript, and parsers in python.

After following some of those links you may realise that we could probably make this shitjs interpreter in an easier way by reusing libraries. However if we wanted to do that, we'd just use an existing JavaScript implementation! Also our JavaScript wouldn't be shit, or from scratch.

Pyweek April 2011, this weekend.

2011-04-01T14:41:00.001+01:00

Pyweek 12 – April 3rd-April 10th

Find out more about PyWeek - "Invites entrants to write a game in one week from scratch either as an individual or in a team. Is intended to be challenging and fun. Will hopefully increase the public body of game tools code and expertise. Will let a lot of people actually finish a game. May inspire new projects (with ready made teams!)"

Hammering nails - the hard way.

2011-03-22T20:55:00.005Z

Do the following 1000 hammering tasks, and you will have learned how to use a hammer the hard way. You can do them from your bedroom. These are just like these guitar tasks that I do - in my spare time as a rock star. Well not really a rock star... but I sure can do some awesome guitar solos. They're also like katas I practice for my martial arts... in my spare time as a ninja.

"pssst..." "the world doesn't like guitar solos any more - and playing in your bedroom will never be like playing to real people. Also martial arts practitioners who use katas loose to fighters who practice fighting in real fights."

"pssst... you there..." "The pros don't use hammers. They use nail guns."

MOTHER FUCKING NAIL GUNS. MOTHER FUCKER.

You don't need a book to learn python the hard way. To learn python the hard way, do some real world projects.

_{This pay out brought to you by the awesome Programming, Motherfuckers manifesto.}

Hacking on android and pygame.

2011-03-14T17:11:00.012Z

For the last couple of weeks I've been hacking on my new android tablet in my spare time.

It's one of those cheap 25cm ones (10 inches). Dual core, nvidia graphics, and all of that stuff. Not a bad little device really.

One project has been to port an old game of mine to it using PyGame subset for android(pgs4a). It's a package of a subset of the pygame modules - along with a bundled port of CPython for android. You can either package your apps standalone, or simply install the launcher from the android market, then upload your game to your SD card. Simples.

So far it's been pretty fun, and the other weekend I got something running. Next up is to work on the UI parts. Already it supports keyboard, mouse, mouse+keyboard, and also joystick mode. However, none of them really work with the touch screen on my tablet. I'm not a very big fan of the touch joypad approach. Where they make a virtual joypad on each side of the screen to touch. There are two main actions in [TheGame]... move, and shoot. You can click somewhere, and [MainCharactor] will try and move there. You can also charge up your steam cannon to fire a cannon ball - which requires aiming. There's a few interfaces which could work I think... I'll just have to try them out and see which I like best.

I hacked on the game to the point where I needed to add some missing python modules into the mix, so had to compile the pgs4a myself. That's taken a few days getting used to the tools, coming up against errors and then trying again the next day. The other day I got it compiling... now the next step is to try uploading the creation to the device.

The game in question is Zanthor, and for it I've done some major refactoring in order to modernise the packaging of it a little bit. Now there is a make, and configure script. The code is in the zanthor directory rather than a lib directory, and of course packaged it so it can run on android... the list of packaging changes is quite large but worthwhile. I hope to use it as a test case to improve packaging for games on python. The idea is to automate packaging for multiple platforms from any of the platforms. So that you can create a package for all supported platforms from any of them. This is a project that I've been talking about for *years*... but releasing an actual game with it is the only way to make it actually happen I think.

  zanthor on my android tablet

I also started messing around with a jack sound driver for SDL 1.3. Jack is the "professional" low latency sound system for linux(+osx). So far I've got it compiling, but still have a bunch more work to do on that. My todo list for that is still quite long. It's going to be tricky. Jack clients have high requirements to not pull down a jack server if they misbehave. I think I probably need to read through the jack developer documentation a few more times. One difficulty will be loading the modules dynamically... even though usually they are linked in with the "pkg-config --libs jack" tool at compilation time. Not too hard though, since SDL provides some nice cross platform dynamic library loading code.

Whilst on the pygame audio side, a really nice patch was posted to the pygame mailing list for the midi module by Christopher Arndt. It cleans up the the midi module some more, fixes some bugs, and adds some more documentation for the C level API. I still have to push his change up to the portmidi repository... which is a todo I've got for this weekend. This is the sort of polish that I hope we can apply to the other modules in the 1.9.2 release.

A pygame release is planned for soonish. The focus will be a polish release. Hopefully mainly polishing up existing stuff, and finishing off modules that are in an experimental or unfinished state. My main task is to get the documentation under control.
There's over 400 comments on the documentation to work through, so I expect this will take at least 40 hours time. I'm going to improve the comment system on the website to improve the work flow. To add the ability to mark a comment as 'dealt with', or 'example'. So when going through them, the options are 'delete', 'dealt with' or ignore for later.

I've ordered a new dedicated webserver to help out with pygame.org tasks. This should allow us to do some more things than with just the current host. I paid for a year up front, so hopefully don't have to worry about it again for a while. It has an old nvidia graphics card in it, so I hope to do some graphical webserver tasks on it, as well as take some other heavy processing tasks off the main web host.

The new webserver should help us work in some countries where our current web host for pygame.org is blocked too. Our current host does some things governments don't like... so I think that's why pygame is banned there. Or else governments find the zombies on the website scary.

Lenard Lindstrom has been working on some cleanups over the last number of months on some pygame modules. Especially the numpy integration, which is much improved and is finally up to a similar speed compared to the old Numeric implementation. Lenard has also explored converting our documentation system to use a different one. He's started a conversion tool for this, so if it turns out good we should be able to convert our docs over easily.

New binary installers for pygame on OSX is another task I've been working on. I mainly did this for a few people who were attending the London Python Dojo - who wanted pygame working on their Apple supplied python. However there are still a few bugs in it that I need to fix up. The main one being that SDL_image now does not link to libpng and libjpeg by default. That is sort of awesome... in that it can reduce file size. However, it's also sort of not awesome... since OSX does not support a few types of png files that are commonly used in pygame applications. Also pygame uses those libraries to save to png and save to jpeg. The other major change is that this package I made works with the Apple supplied version of python. Increasingly people on OSX don't bother getting the python from python.org. So now we're going to have multiple packages for OSX. One for Apple python on x864, and one for 10.4 ppc/x86 python.org python. I'd also like to allow including the frameworks in the site-packages/pygame/Frameworks/ directory rather than in the system or users Framework directory. This will make installing better, since we won't need to ask for a password, and people should be able to bundle pygame more easily... since it won't be spread all over the file systems.

london clojure dojo

2011-02-23T10:52:00.007Z

(Last night I went to the London clojure dojo.)

(. javax.swing.JOptionPane (showMessageDialog nil

  (reduce str ["Hello" "London" "Clojure" "Dojo"])))

This dojo was hosted by @otfrom and @rrees at the ThoughtWorks offices (many thanks to them). They put on a nice spread of quiche and sandwiches. Here's some of the quiche... which was nicer than it looks in this photo.) (

-- quiche picture credit to @otfrom (used without permission ;))

There was a github repo setup https://github.com/ldnclj/footpad before the session with the code from previous weeks.

After grazing, and drinking of softdrinks - the night started with a mini talk on cake (the tasty build tool + enhanced repl(interpreter) + kitchen sink for clojure) by the co-organiser of the dojo (Robert Rees from ThoughtWorks).

A programmer from Deutsche bank was there recruiting clojure programmers, and getting involved in the dojo too. It kind of reminded me of python in the early days... where you could count the number of python jobs around on one hand (and two feet worth of toes). People were talking about how they got a chance to use clojure at their work for a small project, or how they were thinking of sneaking a little bit of clojure into their projects.

Anyone who's been to the London python dojo would be familiar with the format (@otfrom is involved in that one too).

We were split into teams of five, and then huddled around the keyboard... then began to scratch our heads. I'd only started learning clojure earlier that day (well I think I've read about it before, but not in depth) (so there was a lot to learn. Luckily a few of the people in my group had more experience with clojure.

During these sessions, it's great to see how other people work in their environments. It's also interesting to see how people think in clojure. Everyone in our group got a chance to type at the keyboard, and we almost got our task done. In our group we used some tests, map+reduce and doall functions as well as list, array and map data structures. We even used a few cake commands (autotest and killall).

At the end of an hour or so of coding each group did a mini presentation and demo of what they made. Each group got a fair way towards getting the task done, and we got to see how they coded things. It's fun seeing what others got up to, the approach they took and the challenges they faced.

... at the end we left through the dungeon below the building. Those who survived had a few drinks at the pub afterwards. )

JQuery 1.5 promises a better future.

2011-01-17T13:21:00.007Z

One of the new features of the unreleased JQuery 1.5 is a new $.ajax module.

Apart from many bug fixes, the most interesting, and useful part for me is that it now uses promises. Those familiar with python and twisted, or the JavaScript library mockikit, Dojo Deferred, or java.util.concurrent.Future might know these as deferreds or futures. jQuery is using promises, and not futures. They are similar but slightly different. It's a very commonly used technique in concurrency libraries now - so it is good that jQuery has hopped on board.

Since jQuery always returns a promise, you can now add callbacks even after you have done the call... or even change the callback later.

jQuery.getJSON( url ).error( errorCallback )

Pluggable ajax lets us test more easily

The other great thing(from a testing perspective) is that jQuery now allows you to use plug in mock implementations. This lets you make a fake ajax implementation for your unit tests more easily.

This pluggable architecture could also lead to people making other plugins that act the same as the ajax interface. Which could come in handy if you had a strange requirement, or you want to develop an alternative communication method - and be able to reuse much of the jQuery code.

Subclassing jQuery object

Another new area where jQuery helps extensibility, is the ability to subclass the jQuery object, and override methods - without it breaking the public interface to jQuery.

This is useful if you want to make a plugin that changes a few parts of jQuery for it's own uses, and not be worried that it will break 17 other plugins.

(function(globaljQuery, $) {
 // private functions
 $.fn.privateFunction = function(){ };

 // overwrite an existing function without breaking other plugins
 //   which use the global jQuery function.
 $.fn.popularExistingFunction = function(){ };

 // Expose some public functions
 globaljQuery.fn.myPublicFunction = function(){ };

})(jQuery, jQuery.subClass());

You can try the jQuery 1.5 beta 1 yourself if you want to have a play. I'm sure they will appreciate any bug reports.

London python dojo pygame write up.

2011-01-14T15:43:00.011Z

Last night I went to the London python dojo at the Fry IT offices.

The night started with pizza, beer, and nerdy chats with everyone.

Then two lightning talks. The first by @tomviner (Tom), on some funny and useful python modules. `pip install oo && python -moo` Python goes moo (dad joke). `pip install e && python -me 1+2`

Next up was @otfrom (Bruce). He has a cunning plan to work on an engine that could be used later in a cross language dojo to display a game world. @otfrom is trying to see if other language dojos in london are interested in having a bot war tournament in the future. So far the clojure dojo, and perhaps the scala dojo are interested in participating.

@otfrom also announced the London dojos mailing list at: http://groups.google.com/group/london-dojos A group for people interested in programming dojos in london (not just python).

Then 30 of us took an open source game from pygame.org, forked it and hacked on it!

I brought along a big bag of joypad controllers - 30 of them. The idea was to hand out one to each person who turned up. Unfortunately, they were two controllers joined together with one USB cable. Which meant that not everyone got a controller, but 15 people got two!

But first let's rewind to the night before... [[insert tape rewinding sound]] where @ntoll and I were discussing what to do... continue on with the xmas card demo we made last time? Or instead, hack on something a bit more complete?

The more complete game chosen is an over head dungeon crawller type game. It's not at all finished, and is a work progress one. @ntoll selected it from the thousands on pygame.org It seemed to have fairly clean modular code, and be roughly what we were aiming to build.

@ntoll github crawlr page: https://github.com/ntoll/crawlr
original crawlr page: https://github.com/axedcode/crawlr

As is usual of the london python dojos, we split up into groups. Five teams of six people this time. 'Pair programming on steroids - and not in a good way' is how the london python dojo has been described. The room was packed, and most teams split up into two groups of three, with 2-4 laptops per group - some laptops hooked up to larger screens so people a bit further away could have a look.

At the beginning we each got 'Quests', which were little tasks made up by @ntoll.

Add NPC characters that walk randomly around the world
Add music and sfx for particular events
Allow the maps to be pre-defined rather than (as is the case) randomly generated each time.
Add static objects to the game world such as gold or food that get added to the players inventory/energy/score/whatever when they bump into them...
Two player mode (sharing the same keyboard or with two controllers)
Menu system (Start Game, Help, Set-up, Quit)

Each group took their Quests, and got stuck into it. I had the pleasure of going around to each group to see how they were going, and to help out where I could with work arounds for pygame issues. I spent a time with most groups... but couldn't work my way into to where one group was, as they were surrounded by other people completely... and I would have had to climb over the top of people.

After coding stopped (@ntoll had to turn the lights off a few times to try and get people to stop)... the groups presented their work. It's a time for reflection and to show off what you did.

Each group managed to accomplish something. However some tasks were a bit harder than others - and they turned into reverse engineering tasks.

So by the end, we had:

joypad control of the main character, starting the game and exiting it.
A start on multiple player abstraction, from single player. With a small issue that the second player was not displayed on screen for some reason. A separate image is loaded and everything... it's just not displayed.
An Non Player Character (NPC), that could move and change direction (currently in two separate code bases).
A start on collision detection of the main character with things on the ground (like mush rooms). As this group was demoing their work, they 'lived coded' up some missing parts of the solution.
Music, which is different for the main menu screen and the in game music.
Sound effects, for different actions in the game... and some 'ambient' screams of FEAR and TERROR as the character walks through the forrest.
Loading and saving of maps into a text based format. Originally the maps are randomly generated, so this needed reverse engineering to figure out how that worked too.

A bloody good result really. Especially since it was code no one there had seen (except ntoll, and me a tiny bit), and the groups only had an hour and a half to hack on things, groups with people they never worked with before, using unfamiliar libraries that some hadn't even installed until the dojo (eg, pygame).

I think some people were a little frustrated with their tasks, especially some which were more of a challenge than others. But most seemed to enjoy it.

Overall I think this Dojo was about something which the organiser didn't intend. It was about quickly figuring out a code base, and modifying it. Reading, and modifying other peoples code is a great skill to master - and if nothing this Dojo gave us some experience with that, and let us see how other people do that too.

I saw some people using class browsers I hadn't seen, some using an API from the interpreter, others weilding their custom made code search scripts - to hunt down variables.

At the end of the dojos we usually decide on what we do next. We decided to continue on with the game idea in future dojos - so we can use it in a cross language dojo bot war. We also decided that 'someone' would try and merge all our bits of code together, and that @tjguk (Tim) would present an over view of the code base if he has time at the next dojo.

Unforuntately I couldn't go to the pub afterwards - hopefully next time.

* Tim wrote about the dojo on his blog too.
* @r4vi uploaded some videos http://www.youtube.com/view_play_list?p=63C1FD922FB37B43

Pyramid does 22, but how do you do zero lines of python?

2011-01-10T15:35:00.005Z

There is an article talking about how in a certain configuration the Pyramid python framework uses 22 lines of python for a hello world app.

How would your python web app run zero lines of code for a hello world? By pre-generating the html at startup - or at deploy time, and serving the generated data.

Beat that Pyramid! ;) 6 minute Abs.

This is a silly blog post - but also slightly serious. Uploading static data to serve is a very valid approach for a web framework. It's performance, security, and complexity benefits are quite big.

Why HD looks weird.

2010-12-28T18:52:00.010Z

Very often I'm sitting next to a HD teevee, and people say 'that looks weird'.

Why?

There's a number of reasons, but the two main ones are:

when compared to SD digital teevee is: Poor colour depth compared to the number of pixels.
For a person used to analog teevee (PAL, or NTSC) - and not digital teevee, a major weirdness is the digital compression used. Digital teevee is compressed

Compared to SD digital teevee - No increase in colour richness.

They increased the number of pixels shown per cm (or inch), but did not increase the richness of colour displayed.

On PAL for example, you may have 422 colour depth - which is 8 bit colour. If you come from the computer world you may think that is 8 bits per channel colour (RGB 888). However in the broadcast world they use the YUV colour space, and this is usually only 8 or 10 bits per pixel. 4 bits Y, 2 bits U, 2 bits V == 8 bits per pixel.

If the colour depth is the same, then why does it look weirder in HD? Imagine a rainbow going from Left->Right. PAL would have 768 pixels wide, and 'HD' 1920 pixels wide.

The same number of colours would be on the rainbow line, only the HD one would be kind of zoomed in.

The error is zoomed in - or enhanced.

There's other issues like how blue ray only does 8bit colour, and then there are cameras that only output 8bit colour... or worse. Then the equipment in various distribution centers, or the center which puts the little logo in left hand side of the screen, might have different resolutions (say one operates in YUV 422... 8bit). Then you have the compression codecs, and settings being used by the broadcasters. Then the link from your broadcast box to your LCD can down-sample to 8bit. Finally the LCD internally may only do 8 bit, or the display only 8bit. If any one link in the chain from the camera to the LCD is 8 bit, then the final result will be 8bit (or possibly even worse, if the conversions are done badly... eg a fast YUV422->RGB->YUV422 conversion are not lossless). Or the camera natively uses bayer colour space, which is converted to YUV422 by the interface, and then to RGB888 in the computer, then back to YUV422 in the encoder, through to the teevee which is shown in bayer colour space again right at the end. Each conversion to a different colour space is lossy.

Other reasons why it looks weird are enhancing the error of the frame intervals. Eg, film at 24fps somehow shown on a screen that only does 50hz. We've gotten used to things been shown at one frame rate compared to another. The movement is different - and human beings can easily tell the difference.

Teevee is now very complex, and a lot of factors complicate what is being shown to you. Everything, from the speed of things being shown to you, to the colour depth used, the bandwidth used per teevee channel, to the size of the display to the type of technology being used to show you... even the glass on the screen is different.

Why does HD look weird?

All of those things have an affect, however the lack of enough extra colour depth is one of the main things.

Compared to analog teevee - digital compression.

For a person used to analog teevee (PAL, or NTSC) - and not digital teevee, a major weirdness is the digital compression used. Digital teevee is compressed, so they can fit more channels in the space available to them. The compression changes the image so to the human eye... little detail is lost.

That's the aim anyway. However, compression is never perfect - and many people can see the weirdness in the compressed images.

The world of teevee is weird - but digital teevee is even weirder.

iphone web app development, from the trenches.

2010-10-12T23:20:00.010+01:00

Dear reader,

I joined the iOS web developers for adventure and a chance to see the world but instead I am working in a mud hole, freezing my arse off, with a constant fear of death. I am writing this note in hope that it makes it out of the trenches. In case I do not.

It's bloody here in the iOS trenches and I feel my days are numbered. With these thoughts on my mind, I hope to share this with you.

Some of this stuff is not documented in the standard issue manual, or disseminated via the standard propaganda channels. I feel it ought be of use to you.

Unfortunately it has been very hectic here, so the words will likely be rushed and detail will be lacking. I apologise for this, but I still think it will be useful (no brain rockets, just some Damn Useful Information).

In case I don't make it,
Your friend from the trenches,
René Dudfield.

ps. if you find this note, please consider commenting on the back with any other useful information your fellow iOS trench mates might find useful.

-------------- -------------- --------------

Going full screen requires a cache manifest on the iphone. Otherwise the files do not save. The cache manifest is a horrible beast, that has quite a few gotchas. It will only go fullscreen when run as a web app, not when run through safari.

You can detect with javascript if your app is running through safari, or as a web app. This is most useful for asking the user if they want to put your app on their home screen.

Changing css opacity quickly is really slow on the iphone. This makes some jquery things slow - like show/hide.

Use the css animations, and transitions if you can. Just like animating opacity is slow with javascript on iphone, so is animating other css attributes.

There is a 5MB limit on the overall app cache size. Big files will not cache (like music and videos).

If you are using cache busting urls, it stuffs up the cache manifest. eg, mystyles.css?v=234 in your html, will make the cache not work correctly. I guess since the cache has just mystyles.css not mystyles.css?v=234.

For cache manifest to work on iphone you need a html 5 doctype, and a html 5 tag without a xmlns attribute.

Google chrome can be a good debugging tool for the cache manifest file. Since it shows you problems in the console.

Restarting the iphone can help clear the safari cache.

Safari 4, and iOS 4 have the audio tag - not earlier. Calling play on an Audio object after calling load does not really work. However if you set the autoplay attribute to true then it plays as soon as it can.

The audio can stutter a little bit if you are doing many other things at the time, or if the network slows, and it's not all loaded yet.

iphone can not play video inline on a webpage. It can only play fullscreen. iPad can play the video inline.

SVG on iphone is pretty quick. Canvas is not so quick... but you can still do basic things ok.

CSS media queries let you supply a different CSS if you are on an iphone/ipad/android etc.

Mouse events are a bit slow to react. You can use mouse events, but learning how to use the touch events will allow you to develop a much better experience. The touch start event, and the touch end event both come before a mouse event arrives. This means that it will always respond faster by using the touch events.

Make touchable things(buttons) big, and make them a similar size. If you have a really small button next to a really big button it might be impossible for the person to press on the really small button. They can zoom in and out, and if the buttons are all a similar size, then they will be zoomed in at the right level to be able to press on the buttons.

Tweeting python packaging tips.

2010-09-29T10:29:00.003+01:00

I've begun 'tweeting' python packaging tips. I'm hoping to go up to 100 useful python packaging tips. If you're a twit too, please feel free to join in on the conversations.

update: I wrote a little script to download all the python packaging related tweets from twitter. For some reason twitter is only giving me 32 of them... but I've written about 40 so far. Going to run through these at the london python dojo tonight in a mini talk.

setup.py build_ext --inplace to compile your extensions inside the source directory. Good for developing inplace

distribute 'setup.py develop' Installed pkg points to dev copy, quicker changes during dev. develop -u to remove

For pyrex/cython/swig packages, include the generated C code so people do not need to install cython/pyrex/swig.

For debugging info set DISTUTILS_DEBUG. os.environ['DISTUTILS_DEBUG'] = "on" OR export DISTUTILS_DEBUG=on

Don't put code in package/__init__.py file. Makes debugging/editing harder as there are lots of __init__.py files

Name the folder your package lives in after the package name, not src or lib. eg ./mypackage/

Make quick and dirty .deb .rpm packaging of any python package with checkinstall

Create man pages(unix docs) for scripts and command line programs you make available. See rst2man and help2man.

Install scripts (cmd line programs) with pkg http://docs.python.org/distutils/setupscript.html#installing-scripts

Including a short summary of changes at end of 'description' metadata lets people quickly see changes on pypi.

put a #hashtag in your description metadata, and your package will turn up on twitter under that.

Read other peoples python packages to see what they do well. Especially packages you use.

Distutils2 is the future of packaging. It's not ready yet though. http://pypi.python.org/pypi/Distutils2

setup.cfg is an (optional) setup config file. http://docs.python.org/distutils/configfile.html

Test on multiple python versions before release. Each py version is different (2.6,3,pypy,ironpython,jython).

Tests are good. See what you are not testing with the coverage package. http://nedbatchelder.com/code/coverage/

Check your package quality with Cheesecake. 'pip install Cheesecake' 'cheesecake_index -v -n yourpackage'.

Give credit, ♥, and props to your contributors in a thanks.txt file.

pep 386 specifies version scheme for Distutils - use it. http://www.python.org/dev/peps/pep-0386/

1 eg. #django package is not at http://pypi.python.org/pypi/django . It is at http://pypi.python.org/pypi/Django

Packages are caseSensitive. Refer to it the same way it's installed, otherwise confusion installing & finding.

Check your package name is available on pypi, and that it follows the pep8 naming convention. All lowercase, etc.

No changes metadata field. Put in long_description field and CHANGES.txt so people know what's new or changed.

MANIFEST.in controls files to include/exclude (.bak .swp) http://docs.python.org/distutils/sourcedist.html

bdist_mpkg is the package required to make Mac OSX installers.

1 In your setup.py from distutils.command.build import build;build.sub_commands.append(('test', None))

Running tests after build helps catch errors in distro packaged versions, and turns all users into testers.

Great guide http://diveintopython3.org/packaging.html Slightly dated (2009) since py packaging moves fast.

pep 345, latest spec for adding metadata to Python distributions. http://www.python.org/dev/peps/pep-0345/

readme.txt - Use .txt extension. Be reStructuredText. Use windows carriage returns/newlines because notepad sux.

The debian/ dir should only be in the Debian packaging repository. The debian maintainer takes care of it.

_module.so _module.dll _module.dynlib - compiled extensions should be named with a leading underscore.

Pokket mixer. A sound mixer from Berlin.

2010-09-26T17:46:00.008+01:00

pokket mixer

I went to the big market at the Berlin park today, and saw this little sound mixer there. This dude and his girlfriend who were at the stall make them! They are his design too. Really cool buying electronics from the people who make them.

Did I mention I like buying small things that can fit in my pocket, and in my carry-on luggage?

It's passive - so it does not need to be powered. It's also very small. Seems to work quite well. The eq seems to work ok. It has the normal Hi, Mid, Lo for each of the two channels. The sound quality seems pretty good (even when going out to high quality studio speakers). I've bought more expensive mixers with worse audio quality.

3g modems in finland? Any plans for a month or one week?

2010-08-09T17:53:00.004+01:00

Dear Lazy web,

Are there any 3G usb modem plans available in Finland that I can get for one week or a month? I'd need to buy the modem too.

Hopefully it should work two hours drive from Oulu.

Update: used a modem by Elisa. Worked well, even 3g. Closing comments because this post is getting spams every day - and blogger.com spam protection seems broken.

javascript (and jquery) templating

2010-07-26T01:37:00.007+01:00

I just put up some code for doing templating with javascript(and jquery). Either on the server side or on the client side. If you open the html in your browser, it runs on the client side. If you first process it server side... then it runs server side(but not on the client side).

http://github.com/illume/nodejs_jquery_templating

This is a followup from you are using the wrong templating language - as a proof of concept.

Why is this useful?

No need for a server to process the templates. Either process them server side or client side.
Data can be stored in a json file. No need for a database for testing. Just create json files in a text file.
Can reuse knowledge of javascript libraries (like jquery), rather than learning one of 798394 different templating languages.
Can keep one html file which front end web developers can edit without them needing a new template file.
Can use jquery plugins on server side too(validation, etc).

Seems to work ok so far... but it still needs a lot of polish before it'll be useful. If it works out ok, I'll try it out on some real projects (where the main server side language is python).

Let's make a shit JavaScript interpreter! Part one.

2010-06-09T10:15:00.028+01:00

Let's make a shit javascript interpreter! Part one.

As a learning exercise, I've begun writing a ~~javascript~~ ECMAScript interpreter in python. It doesn't even really exist yet, and when it does it will run really slowly, and not support all js features.

So... let's make a "from scratch", all parsing, all dancing, shit interpreter of our very own!

Teaching something is a great way to learn. Also writing things on my blog always gets good 'comments', hints, tips, plenty of heart, and outright HATE from people. All useful and entertaining :)

Tokenising

So to start with, we need something to turn the .js files into a list of tokens. This type of program is called a tokeniser.

From some javascript like this:

function i_can_has_cheezbrgr () {return 'yum';};

Into a Token list something like this:

[
{"type":"name",
"value":"function",
"from":0,
"to":8},
{"type":"name",
"value":"i_can_has_cheezbrgr",
"from":9,
"to":28},
{"type":"operator",
"value":"(",
"from":29,
"to":30},
{"type":"operator",
"value":")",
"from":30,
"to":31},
{"type":"operator",
"value":"{",
"from":32,
"to":33},
{"type":"name",
"value":"return",
"from":33,
"to":39},
{"type":"string",
"value":"yum",
"from":40,
"to":45},
{"type":"operator",
"value":";",
"from":45,
"to":46},
{"type":"operator",
"value":"}",
"from":46,
"to":47},
{"type":"operator",
"value":";",
"from":47,
"to":48}
]

Wikipedia has a page on Parsing (also see List_of_unusual_articles for some other background information).

"Tokenization is the process of demarcating and possibly classifying sections of a string of input characters. The resulting tokens are then passed on to some other form of processing. The process can be considered a sub-task of parsing input." -- wikipedia Lexical_analysis#Token page.

We can has vegetarian cheeseburger... but how can we parse javascript?

To the rescue, comes uncle Crockford the javascript guru of jslint fame. He wrote this lovely article: http://javascript.crockford.com/tdop/tdop.html. The ideas come from a 1973 paper called "Top Down Operator Precedence". The Crockford article is great, since it is free, short, and well written javascript. Unlike the 1973 paper it gets the ideas from... which is behind a paywall, long, and uses a 1973 language called "(l,(i,(s,(p))))".

As well as being short and simple... Phil Hassey used "Top Down Operator Precedence" and this article on his journey making tinypy.

Goat driven development

Just as Phil did with tinypy, I'm going to use Goat Driven Development. Well, I'm not even sure what Goat Driven Development is... so maybe not.

Another python using dude, Fredrik Lundh, wrote some articles on "Simple Top-Down Parsing in Python" and Top-Down Operator Precedence Parsing.

Also see Eli Bendersky's article on Top Down Operator Precedence.

So where to begin?

After reading those articles a few times... scratching my head 13 times, making 27 hums, a few haaarrrrs, one hrmmmm, and four lalalas...

light bulb: A brilliant plan!

Eli Bendersky implements a full tokeniser, and parser for simple expressions like "1 + 2 * 4".

Let's copy this approach, but simplify it even more. Our first step is to make a tokeniser for a such an expression. That should be easy right?

A Token data structure.

Uncle Doug Crockford uses this structure for a token.


// Produce an array of simple token objects from a string.
// A simple token object contains these members:
//      type: 'name', 'string', 'number', 'operator'
//      value: string or number value of the token
//      from: index of first character of the token
//      to: index of the last character + 1

Here's an example token from above:


{"type":"name",
"value":"i_can_has_cheezbrgr",
"from":9,
"to":28}

Writing the tokeniser

Often a tokeniser is generated... or written by hand.

Fredrik Lundh writes a simple tokeniser using a regular expression.


>>> import re
>>> program = "1 + 2"
>>> [(number, operator) for number, operator in 
...  re.compile("\s*(?:(\d+)|(.))").findall(program)]
[('1', ''), ('', '+'), ('2', '')]

This is a valid approach... but regexen blow up minds. Instead I'm going to write one using a state machine, in a big while loop with lots of ifs and elses.

Our homework

Write a tokeniser for simple expressions like "1 + 2 * 4". Output a list of tokens like the javascript one does... eg.


{"type":"name",
"value":"i_can_has_cheezbrgr",
"from":9,
"to":28}

Until next time...

Really, I have no idea what I'm doing... but that's never stopped me before! It's going to be a shit javascript, but it will be our shit javascript.

My javascript reading list over the last few months

2010-06-06T15:29:00.006+01:00

Over the last few months I've been fairly deep into javascript land. Both in my full time job, and in most of my coding side projects - I've been mostly doing javascript. Of course, like most web programmers I've been dabbling in javascript over the years... but never so intensely. On the way I've been collecting a 'reading list' of videos, articles, books, and projects.

Here are some of the good links from the last few months.

Crockford on javascript videos:
http://www.yuiblog.com/blog/2010/04/08/video-crockonjs-5/?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed%3A+YahooUserInterfaceBlog+%28Yahoo!+User+Interface+Blog%29

akihabara arcade:
http://www.kesiev.com/akihabara/

canvas javascript games:
http://www.benjoffe.com/code/

javascript 3d engine:
http://github.com/mrdoob/three.js

Aves Engine: HTML/JavaScript Game Engine (youtube.com)
http://www.youtube.com/watch?v=Ol3qQ4CEUTo
http://ajaxian.com/archives/aves-game-engine?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed%3A+ajaxian+%28Ajaxian+Blog%29

css animations and transitions
http://webkit.org/blog/130/css-transforms/
http://webkit.org/blog/138/css-animation/

3d video navigation on iphone
http://ajaxian.com/archives/iads

apple web app documentation
http://developer.apple.com/safari/library/referencelibrary/GettingStarted/GS_iPhoneWebApp/index.html#//apple_ref/doc/uid/TP40008134
http://developer.apple.com/safari/library/samplecode/FingerTips/Introduction/Intro.html

apple visual effects guide
http://developer.apple.com/safari/library/documentation/InternetWeb/Conceptual/SafariVisualEffectsProgGuide/Introduction/Introduction.html#//apple_ref/doc/uid/TP40008032

apple multi touch events
http://developer.apple.com/safari/library/documentation/AppleApplications/Reference/SafariWebContent/HandlingEvents/HandlingEvents.html#//apple_ref/doc/uid/TP40006511-SW22

android, and APIs
http://www.thesearethedroids.com/2009/12/15/creating-android-apps-with-html-css-and-javascript/
http://www.phonegap.com/
http://developer.android.com/resources/articles/using-webviews.html

Safari on iPhone Graphics, Media, and Visual Effects Coding How-To's:
http://developer.apple.com/safari/library/codinghowtos/Mobile/GraphicsMediaAndVisualEffects/index.html

Preparing Your Web Content for iPad:
http://developer.apple.com/safari/library/technotes/tn2010/tn2262.html

gamequery
http://gamequery.onaluf.org/

javascript collision detection
http://www.lukewallin.co.uk/?go=engine

box2d physics javascript:
http://box2d-js.sourceforge.net/index2.html

render engine javascript game engine
http://www.renderengine.com/

vector maths library
http://sylvester.jcoglan.com/

wii javascript
http://en.wikipedia.org/wiki/Wii_Opera_SDK

game js, pygame alike.
http://code.google.com/p/gamejs/

physics:
http://www.queness.com/post/3296/8-amazing-javascript-experiments-of-physic-and-gravity-simulation

how to detect html5 things:
http://diveintohtml5.org/everything.html

javascript audio synth
http://acko.net/blog/javascript-audio-synthesis-with-html-5

audio with js:
http://www.phon.ucl.ac.uk/home/mark/audio/play.htm
- best one.
http://www.javascripter.net/faq/sound/play.htm
http://simplythebest.net/sounds/sound_guide.html

using flash from js for sound
http://www.schillmania.com/content/projects/soundmanager2/

cross browser css image rotation
http://samuli.hakoniemi.net/cross-browser-rotation-transformation-with-css/
http://snook.ca/archives/html_and_css/css-text-rotation

open source iphone game:
http://ajaxian.com/archives/golingo-a-great-titanium-mobile-web-game-open-sourced-for-you?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed%3A+ajaxian+%28Ajaxian+Blog%29

jsdom.
http://github.com/tmpvar/jsdom
http://www.yuiblog.com/blog/2010/05/20/video-insua/?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed%3A+YahooUserInterfaceBlog+%28Yahoo!+User+Interface+Blog%29

a number of interesting projects
http://www.bramstein.com/projects/

image editor
http://pixlr.com/

chrome web development extensions
https://chrome.google.com/extensions/featured/web_dev

html5 ipad app
http://mir.aculo.us/2010/06/04/making-an-ipad-html5-app-making-it-really-fast/

particles
http://spielzeugz.de/html5/liquid-particles.html

You are using the wrong #templating system for #html.

2010-05-14T09:57:00.007+01:00

Use javascript + json + DOM for html templating.

Front end developers know how to use this technology. They know this technology very well, so will be way productive.

No weird new template language to learn and install on every single project.

No dependency on a rapidly changing server side backend... or database, or 3rd party API required.

No taking the html, and turning it into [$%%WEIRD TEMPLATE LANG%%$$] step. Instead you can use javascript, json and the DOM.

Being able to use the templating system either via the browser, or via the server side means there is no dependency between the front end and the back end. Front end developers just need an example json file to work from. Then the back end developer just needs to create the json file.

It does mean you are dependent on a DOM+javascript implementation on the server side if you want to do things like server side caching, and supporting Agents(browsers, bots etc) that do not use javascript on the client side.

Just json files. Just javascript. Just the DOM.

You are (currently) using the wrong templating system.

Validation through html forms.

2010-04-14T06:29:00.002+01:00

Continuing on from a previous article I wrote earlier in the year... 'Using a html form as the model', this post describes using validation.

With html 5 comes a whole bunch of validation you can use in your forms. This makes a html form even more useful as a schema than the html 4 forms. Since you can use types like email, telephone, time, date, and such. You can also use things like minimum, and maximum length. As well as the ever useful 'required' field.

You can read about html 5 forms here.

UX, rsi. ctrl+c considered harmful. If not done properly.

2010-04-06T16:37:00.005+01:00

Ctrl+c, as well as a single handed touchpad move is considered harmful.

Try and do combinations of keys, mouse moves, and gestures one finger at a time.

Why? RSI. RSI hurts people.

Please consider people when designing your user interfaces. Either avoid single handed moves, or warn against them.

Thanks.

When using computers, consider resting regularly, stopping when you hurt(and before you hurt). Also consider using one hand or finger for each action. So rather than shift+A done with one hand, try doing it with two hands. Also consider moving your hand away from qwerty middle keys do get a less stretchy press of the shift/ctrl/alt/cmd keys.

Thanks.

For more details on RSI, and RSI prevention, please see: http://www.rsi.deas.harvard.edu/spread.html.

better search engine

2010-03-15T21:02:00.001Z

http://duckduckgo.com/

Memory usage of processes from python?

2010-03-12T11:28:00.010Z

Is there a way to find the memory usage of python processes?

Trying to find some portable way of doing this. However, so far I think a new module might be needed...

I've got linux mostly covered, but maybe you know how with freebsd, OSX, windows(9x-7)?

So is there something built into python already? Is there a X-platform third party module already? Or a module just for one platform available?

update: here's the linux code I found and cleaned up a bit memory_usage.py if anyone is interested. bytes_resident = memory_usage.resident(). It reads /proc/PID/status... eg, like "$ cat /proc/PID/status | grep VmRSS" would.

pympler: 'Pympler is a development tool to measure, monitor and analyze the memory behavior of Python objects in a running Python application.'

psutil: 'psutil is a module providing an interface for retrieving information on running processes and system utilization (CPU, memory) in a portable way by using Python, implementing many functionalities offered by tools like ps, top and Windows task manager.'

dowser: 'Dowser is a CherryPy application that displays sparklines of Python object counts, and allows you to trace their referents. This helps you track memory usage and leaks in any Python program, but especially CherryPy sites.'

syrupy: 'Syrupy is a Python script that regularly takes snapshots of the memory and CPU load of one or more running processes, so as to dynamically build up a profile of their usage of system resources.'

Some non-pythony memory tools: valgrind memcheck, massif and cachegrind (linux), MallocDebug (osx)

how I recovered a friends MacOSX drive

2010-03-10T08:10:00.005Z

MacOSX can sometimes corrupt a drive (like most OSen).

I've seen it happen a few times, in a couple of different ways. One way is if you interrupt some file transfers. Like by pressing 'stop' on a big transfer - this can trash your partition table. You'll likely see vfs errors in your log at this point. Anyway... linux, ubuntu and 'testdisk' came to the rescue. testdisk found the partition for me, and wrote it back... luckily it worked and my friend and I did a happy dance.

The HFS+ partition was saved.

gnome multimedia keys via dbus-python

2010-03-07T07:56:00.006Z

Ever wanted to get your multi media keys to do something other that play multi media? You could get these key events any number of ways. One way is through dbus. Here is an example: gnome_multimedia_keys.py

Here is a version which does not block your mainloop. So it's useful for integrating with other libraries (like pygame :) I made a dbus with pygame example too.

Ideas for Super Surfaces in pygame.

2010-03-06T12:14:00.014Z

pygame already has a sub-surface, which is part of a larger surface. Sub-surfaces refer to the same pixels as the surface they come from, and share the same Interface as a Surface. It's good for doing sprite sheets, where you save one file with many smaller images - but then being able to manipulate them as if you loaded the images as smaller separate surfaces.

However, sometimes we would like to operate on a whole bunch of smaller surfaces stuck together. This is what I'd like to call a Super Surface - a collection of smaller surfaces which can act as one big surface. It's a complementary idea to the sub-surface, and intuitively it should be there... but it's not yet.

Unfortunately it's a lot harder to code a Super Surface compared to a sub-surface. Since a Super Surface would need to have all the surface affecting routines changed to work with it.

For example, everything in the draw modules would need to be redone. So would all of the surface methods. So when you draw a line on a super surface, it should affect all of the sub-surfaces.

How can Super Surfaces be implemented.

Still trying to think of a simple/clever way of implementing it efficiently, without having to recode all of the drawing/blitting routines. However, just like how sub-surfaces in pygame can simplify code immensely - so too will Super Surfaces.

One method, might be to do a rect translation, and then broadcast the translated method calls over the sub surfaces of the Super Surface. So we translate the rects into the sub-surface coordinate space from the Super Surfaces coordinate space.

I'm not so sure if we can get a function pointer to a method from within the function pointer itself with the python C API. This would simplify it, because the translate-and-broadcast functionality could be implemented in one place and reused within all of the various surface affecting routines automatically. Something to research...

Common methods for implementing tiling engines.

Super Surfaces share a lot of the same properties a tiling engine. So how are tiling engines implemented?

A common way for tiling engines to work is to have the images being placed in a grid, and for each of the sub-surfaces to be the same size. So the image at [0][0] in the array is at those coordinates. This simplifies the implementation, but is not very flexible. A slightly more complex way to implement it is to have each sub surface have its position stored relative to the super surface. This complicates the clipping a little, and for speedy use it really requires a spatial hash... like a quadtree.

So I'm thinking of doing the more free form implementation... probably just a naive implementation to start with... then perhaps a quatree version as an improvement. It would be better to use a spatial hash more efficient at moving elements - but that could be a third improvement.

Use cases for Super Surfaces

Use cases include viewing massive images. Much graphics hardware has texture size limits, so it is common to combine many smaller textures when drawing larger images.

Another common use case is for 'tile engines'. Where tile engines show a larger image made up of many smaller images.

To allow using different image formats for different parts of an image is another use case. For example, a piece of the image which is just black only needs a few bytes to represent the image... it does not need 32bits per pixel. Whereas the part of the image which includes a persons face with 50% transparency will require 32bits per pixel for sure. This will save memory, and increase speed. It increases speed, because it is possible less memory bandwidth means faster operations. Also, less complicated blending - or more optimized blitting can happen (eg, fast 8 bit Run Length Encoding blitters). This should be possible with no extra effort from the programmer too.

Finally, by using smaller sub-surfaces it in effect becomes a tiling engine automatically. Tiling in graphics is used to split operations up easily. This gives memory, and speed advantages. By processing a tile at a time - you only need to have those tiles in memory at that time. It also gives easy parallelism, since the separate tiles are a form of data parallelism (which is the easiest type). For some hardware, it is possible to use parallelism when the surfaces are kept separate.

I'm interested in any comments, especially if you've made a tiling engine like this before?

Why bzr and launchpad? launchpad is open source

2010-03-03T08:15:00.002Z

Why bzr and launchpad? Bzr AND Launchpad are open source.

launchpad: open source
github: closed source
sourceforge: closed source (was open source in the past)
bitbucket: closed source
googlecode: closed source

You can submit changes to launchpad at: dev.launchpad.net. As well as (submit launchpad bugs) and feature requests against it if you don't want to make the patch yourself.

When there is a good open source alternative, I always choose the good open source option. I initially had problems with bzr a couple of years ago... but it has been quite good to me over the last couple of months. So I'm moving over all of my projects from other version control hosting services to launchpad.

Of course bzr and launchpad are also written in python (with selected optional C optimizations), so that makes for happy hacking :)

update: reported a bug here about the 'can not find source code easily on launchpad' issue. Any other issues with launchpad?

London code dojo - 4th March '10 18:30 – 21:30 (ish).

2010-03-01T18:22:00.002Z

Details, and booking here: http://ldnpydojo.eventwax.com/7th-london-python-code-dojo

"""What is a coding dojo? This is a coding dojo.

Last time we attempted to refactor the various adventure game solutions from the January dojo. Whilst interesting, perhaps refactoring isn’t that exciting an activity for a dojo :-). Nevertheless, people seemed to be having fun and we did achieve our goal of a “one true” adventure game code base that allows us to define and navigate around a game world. The code can be found here: http://github.com/ntoll/code-dojo/tree/master/adventure/week2/.

After discussion at the end of the February dojo (and later in the pub) we decided that this time round we’re going to try another small-groups based exercise with a “show and tell” at the end as we continue to build the world’s greatest adventure game. Problems we might want to tackle include: a command parser, keeping track of game state/score/objects, NPCs/AI, authoring tools, turning it into a MUD (and so the list goes on…)

Photos from all the dojos so far can be found here: http://divvyshot.com/event/IsJtx/

Free pizza and beer will be provided.

Participants get the chance to win a cool book (thanks O’Reilly)."""

uh0h, I made a logo. What do you think?

2010-02-25T16:27:00.008Z

Made a quick little logo for a website called...

Tried a few different versions... showed my girl friend a few, and stuck with this one in the end. Made it with a simple DejaVu Sans Mono Bold font. Ya for the DejaVu font project. Modified the zero(0) a bit, then inverted the colors (ya! negative space). Played around with the letter spacing... applied an old school guassian blur filter to the right side, resized for web... and done!

What do you think?

The brief is 'something quick for yet-another-culture blog/zine called uh0h... as in uh oh I dropped a hammer on my foot.'.

update: Added some results in this image below. Which is updated with a pygame/freetype script. The idea is that the results in the image can be updated without the rss feed getting updated. Hoping to clean the script up and add it to pywebsite. The same technique seems to be used by some wordpress blogs with their 'comments:3' image links - so I think it will be useful for all sorts of things shown on blogs where frequent updates are required without having to update the post.

svn merging is easy...

2010-02-24T19:17:00.006Z

Subversion(svn) merging is easy... iff you are using a modern svn version (1.6.x).
Here it is in short:

$ cd /dir/of/your-branch
$ svn merge ^/trunk

Where ^/trunk is the url of the branch you want to merge from. For more details, have a look in the svn book basic merging section.

Also this article on svn merging explains it fairly well.

The svn 1.6 release notes and 1.5 release notes also talk about various svn updates including merge enhancements amongst other goodies (like improved python bindings).

Really, merging is not too bad in subversion now. I know this post won't stop everyone from using 2004 reasons arguing over version control systems... but what ever.

Ok, bzr, hg, and git all have lots of nice features - but merging in svn is pretty easy now so please bash(zsh) it for other reasons if you must.

Now, let's move back to the vi VS emacs argument ;)

The secret to my web development productivity...

2010-02-19T06:14:00.002Z

Can you see the secret to my web development productivity in this photo?

No it's not the red cup of coffee.

It's not the pieces of sticky tape on my laptop.

Follow the cable from my laptop...

and have a look under the desk.

...

... I'll wait whilst you have a look before telling you the answer.

...

That's right!!!

...

it's a joypad.

Using python and pygame, I've made a little app which listens for joystick events, and then deploys the website I'm working on.

With a mash, kick or prod of my foot, up goes my website. Deployed.

Deploy is just a fancy word for 'upload my website, do database migrations, restart app servers, run tests, rollback if there are failures... etc'.

For 5 british pounds, $7.8 USD, or 5.74 euros, you can get one of these joypads delivered to most places these days.

It's giving me too much pleasure pressing it every time I want to upload a new version of the website. Most live updates EVER today - and I have the joypad to thank for it.

The joypad is the secret to my web development productivity. Please don't tell anyone.

update: here's a slightly cleaned up, and slightly silly version of 'Joy to Deploy'...

Check out using bzr revision control:
bzr co lp:joytodeploy
Or you can view joytodeploy.py source code.

joytodeploy.py deployment_program

for example:
joytodeploy.py deploy.sh with great justice

joytodeploy.py echo 'ninjas are better than pirates'

Genshi templates - header/footer templates, and including templates.

2010-02-18T14:06:00.005Z

How to include a Genshi template inside of another genshi template?

Use "py:match" inside the template you want to include into other pages (eg. a sitelayout.html).

Then you can use "xi:include" in the pages you want the files included in. (eg. best_picture_of_my_cat_today.html)

Now, it's not a "simple include this template here" kind of thing. The py:match can find various parts of the file, and then replace them how they like. For example, a page layout template will match the header and footer with "py:match".

It is best explained with examples, and in the documentation:

pylons genshi example

Includes section of the genshi documentation where it explains "xi:include".

Genshi documentation where it explains py:match.

Hopefully this explains the genshi way of including a template into a template.

My fling with engine X.

2010-02-17T08:42:00.007Z

Projects that have X in the name are cool. Engines are cool (especially steam powered ones). Spelling engine 'ngin' gains 17.5 l33t points too. All combined, this makes the name nginx super schwet!

I've been using nginx for a while now, and have found it quite good so far. Been moving a couple of websites to it, fooling around with it a lot.

So far I've been able to use it for everything I've tried. Some of my apache configs are fairly long... so that is saying quite a bit. On my low memory (512MB) server it has saved quite a bit of memory - even though I've only moved over a couple of websites. Along with the cherrypy memory reduction work I did recently, my server has a bit more room to breathe... (and for me to waste memory on hosting other websites! ya!).

Nginx has a good reputation as being rock solid - so I hope it holds true for me. Then perhaps I can get rid of apache completely (on this one server). I've tried to replace apache with other web servers before... but always come up with a reason to move back to apache. Either some application uses a feature that the other server does not support, or the other server is just not as robust as apache. I don't like fixing, or looking at servers... I just want them to work without hassle. I'm not afraid of working, and learning about a new webserver to work... It's just that some webservers are too high maintenance.

Fastcgi is one way nginx allows you to host php and python websites. Nginx can also be used as a reverse proxy server. Personally I like to host python websites with cherrypy and a reverse proxy, and use fastcgi with php. The nginx proxy_pass configuration seems to work quite well... as does its simple-to-setup load balancing.

Like all good software, I love to check out how it was made. Reading the nginx source code is breath of fresh air. Despite Igor[the main author] being Russian, the code is written in english C(with tiny smatterings of asm/perl)... with very few comments. It is a modular, and very clean code base. It doesn't seem to have any unittests... but it's still quality software.

Our relationship is still a fling really. Before I invite nginx to meet my parents, I'll give it few more months. Ok, my love letter to nginx is done now.

python - unifying c types from different packages.

2010-02-04T10:52:00.008Z

Python already has a number of objects to represent c types. However, there is a need to improve interoperability between systems using these c types. Below I explain the need, and discuss existing efforts to address this need. Then ways to transparently translate between the various type systems without each system needing to know about each other are also discussed.

In the ctypes library - you can represent an unsigned 32bit integer with ctypes.c_unit32.

In the array, and struct modules there are different array type codes. For example, 'L' represents unsigned int with a minimum of 4 bytes on 32bit systems and 8 on 64bit systems.

numpy, cython, pyopengl and other python extensions have their own types representing c types too. Most extensions which link up to languages which use static typing represent basic c types to python in some way.

Not only libraries, but various compilers and translation tools also use c types. For example tinypyC++, cython, swig, etc. Also type inference is done from things like shedskin, and rpython - but they represent types internally with their own type objects.

Standardising on one set of c type objects or string codes would give some compatibility advantages. However, that will be hard to change for backwards compatibility reasons. A mapping between the various types should provide plenty of the advantages. For example, to be able to translate from a ctypes to a numpy type should be fairly simple.

Here you can see that numpy, ctypes and the python.array module already have integration:


>>> import numpy, cython, ctypes, OpenGL.GL

>>> numpy.array([1,2,3,4.2], ctypes.c_uint32)
array([1, 2, 3, 4], dtype=uint32)

>>> numpy.array([1,2,3,4.2], numpy.uint32)
array([1, 2, 3, 4], dtype=uint32)

>>> numpy.array([1,2,3,4.2], 'L')
array([1, 2, 3, 4], dtype=uint32)

>>> numpy.array([1,2,3,4.2], OpenGL.GL.GLuint)
array([1, 2, 3, 4], dtype=uint32)

>>> numpy.array([1,2,3,4.2], cython.uint)
------------------------------------------------------------
Traceback (most recent call last):
 File "", line 1, in 
TypeError: data type not understood

Pretty cool hey? Numpy already knows about many of the type variables available in the python ecosystem. With the notable exception of cython.

I think there is a need to try and standardise use of c type variables - so that more code can interoperate without each system needing to know about each other systems type objects. Alternatively a translation layer can be made in place.

For example an adaptor something like this:

# this registers two types which are the same.
>>> type_registry.register_types(numpy.uint32, 
...                              cython.uint)

# here numpy.array does not know about cython directly,
#   but can look at the registered type we just did to get it from there.
>>> numpy.array([1,2,3,4.2], cython.uint)
array([1, 2, 3, 4], dtype=uint32)

# if numpy does not know about the adaptor registry then we can still
# use the registry, if only in a more ugly - non transparent way
# by calling a translate function directly:
>>> numpy.array([1,2,3,4.2],
... type_registry.translate(cython.uint))
array([1, 2, 3, 4], dtype=uint32)

Instead of an adaptor, a magic variable could be used which would contain the 'standard c type variable' from python. For example - cython.uint.__ctype__ == ctype.c_unit32. Then numpy could look for a __ctype__ variable and use that - without having to be extended for every system that is made. One problem with a magic variable over registered types is that some python objects can not have those magic variables assigned. For example, try adding a __ctype__ variable to an int instance - it won't work.

Either the adaptor, or the magic variable would let cython - and other systems use their own type objects and still have a way to translate the types to the standard python c type variables (when/if they are chosen).

A simple mapping (with a dict) from a package to the standard c type objects/type codes is another method that could be used. This will allow a package to fairly easily hook into the eco system. For example cython could have a __c_type_mappings__ magic variable at the top level of its package. Then another package looking to translate the type could look to the package for this __c_type_mappings__ variable. The advantage of this is that many times variables can be injected into a package but not into extension types in the package. On the other hand this feels icky.

The c types from the ctype package seem to be a fairly good choice for this. PyopenGL 3.x series uses the ctypes as its types. eg, OpenGL.GL.GLuint == ctypes.c_uint32. Except ctypes is a fairly big dependency just for a few types.

The buffer pep 3118 being introduced into python/numpy to make buffer sharing between libraries is a similar use case. However it involves sharing instances of differently typed buffers - and has quite clever semantics for many use cases. The formats from that pep could also probably be used to share type information.

The buffer protocol pep specifies extra format strings over the ones specified in the python.array module. So as to be able to specify a more complete set of type, and memory layouts. So rather than using the ctypes types, it probably makes sense to use the new buffer protocol format codes specified in pep 3118. As they are just strings without any further dependencies on the rest of the ctypes machinery (eg libffi etc). Of course, if you are using ctypes already - then depending on it is not a problem.

Of course ctypes.c_uint32 is more descriptive than 'L' to many people, so the format codes (eg 'L') should just be used for specification. People should still use their own type objects - but provide a translation to their format codes as specified in pep 3118 the new buffer protocol.

The codes specified in pep 3118 will probably need to be expanded as more c types need to be described. For example bit fields and bit depths of types are not described in the pep. Many systems specify the bit depth of the type - numpy, ctypes, opengl, etc. For example they use 'uint32' rather than 'unsigned int'. Also bit fields are becoming more common in C so they should be added to the type code formats in someway too.

In conclusion there is a need for interoperability of c types from various python extensions, libraries and python compilers. The pep 3118 format codes, and ctypes types are good candidates to work with for standard c type objects/codes. Adaptor registries, simple mappings, and/or magic variable names could be used to enhance interoperability.

good bye Vivienne

2010-01-17T18:15:00.002Z

will miss my little sister. wish we had more time together. so sad. shit

Jquery 1.4

2010-01-15T10:59:00.003Z

Another quality jquery release... jquery 1.4.

Some very handy changes, my favourite probably being able to use functions to set values. In short, some of the changes are:

setting values with functions

html5 form support

better JSON/JSONP and ajax functionality

easier element construction (just pass a dict of attributes)

reverse indexing like in python, where -1 is the last item.

better animation for multiple attributes at once

performance improvements, code cleanups, and bug fixing

Full details in the jquery 1.4 release notes.

worker pool optimization - batching apis, for some tasks.

2010-01-13T06:40:00.000Z

What are worker pools anyway?

Worker pools are an optimization to the problem of processing many things in parallel. Rather than have a worker for every item you want to process, you spread the load between the available workers. As an example, creating 1,000,000 processes to add 1,000,000 numbers together is a bit heavy weight. You probably want to divide it up between 8 processes for your 8 core machine. This is the optimization worker pools do.

With a usual worker pool there is a queue of jobs/data for the workers to work on. The workers are usually threads, processes or separate machines which do the work passed to them.

So for 1000 items on that queue, there are around 1000 accesses to that queue. Usually the queue has to be thread safe or process safe, so that pieces of data are not sent to many workers at once.

This can be an efficient method to use for some types of data. For example, if each job can take different amounts of time, like IO tasks over the internet... this is not optimal, but pretty good.

Problem work loads for typical worker pools.

Let's assume that the tasks are fairly easy to measure the average(or median if you like) time of each task. So either not IO tasks, or fairly equal length tasks. Then the central queue idea starts to fall down for the following reasons.

What if the cost of starting a new job is quite high? Like if starting each job happened over a machine with a 200ms network latency (say using a HTTP call to the other side of the planet). Or if a new process needs to be spawned for each task ( say with exec or fork ).

Or if the cost of accessing the queue is quite high? Like if you have a lock on the queue (eg a GIL) and lots of workers. Then the contention on that lock will be quite high.

What if there are a lot more items than 1000? Like if there are 10,000,000 items? With so many items, it is worth trying to reduce or avoid that cost of accessing the queue all together.

How to optimize the worker pool for these problems?

The obvious solution is to divide the items up into chunks first, and then feed those big chunks of work to each worker. Luckily the obvious solution works quite well! It's trivial to divide a whole list of things into roughly equal size chunks quite quickly ( a python one liner *1).

An example of how to improve your worker pool.

Here is some example code to transform a pygame.threads.tmap command that uses a worker pool to do its work off a central worker queue, into one that first divides the work into roughly equal parts. Mentally replace pygame.threads.tmap with your own worker pool map function to get the same effect.

#file: divide_and_map.py
import operator

# Here's our one liner divider, as two lines.
def divide_it(l, num_parts):
    return [ l[i:i+num_parts] for i in xrange(0, len(l), num_parts)]

# Here is our_map which transforms a map into 
#   one which takes bigger pieces.
def our_map(old_map, f, work_to_do, num_workers):
    bigger_pieces = divide_it(work_to_do, len(work_to_do)//num_workers+1)
    parts = old_map(lambda parts: map(f, parts), bigger_pieces)
    return reduce(operator.add, parts)

# now an example of how it can speed things up.
if __name__ == "__main__":
    import pygame, pygame.threads, time

    # use 8 worker threads for our worker queue.
    num_workers = 8
    # Use the pygame threaded map function as our 
    #    normal worker queue.
    old_map = pygame.threads.tmap
    pygame.threads.init(num_workers)

    # make up a big list of work to do.
    work_to_do = list(range(100000))

    # a minimal function to run on all of the items of data.
    f = lambda x:x+1

    # We time our normal worker queue method.
    t3 = time.time()
    r = pygame.threads.tmap(f, work_to_do)
    t4 = time.time()

    # We use our new map function to divide the data up first.
    t1 = time.time()
    r = our_map(old_map, f, work_to_do, num_workers)
    t2 = time.time()
    del r

    print "dividing the work up time:%s:" % (t2-t1)
    print "normal threaded worker queue map time:%s:" % (t4-t3)

$ python divide_and_map.py 
dividing the work up time:0.0565769672394:
normal threaded worker queue map time:6.26608109474:

For our contrived example we have 100,000 pieces of data to work through. If you created a thread for each piece of data it would surely take for ever. Which is why people often use a worker queue. However a normal worker queue can still be improved apon.

Results for this contrived example made to make this technique look good?

We get a 100x speedup by dividing the work up in this way. This won't work for all types of data and functions... but for certain cases as mentioned above, it is a great improvement. Not bad for something that could be written in one line of python!*1

It's an interesting case of how massaging your data to use Batching API design techniques gives good results. It also shows how writing parallel code can be sped up with knowledge of the data you are processing.

*1 - Well it could be done in one line if we were functional ninjas... but for sane reading it is split up into 12 lines.

mini languages that non programmers can understand

2010-01-12T07:34:00.005Z

There are hopefully a number of mini text based programming languages that non-programmers can understand. But what are they?

One that I've used in the past is something like this:

Which would parse into a python/javascript data structure like this:

{name: 'Bob',
gender: 'male/female',
age:'22'
}

It's suprisingly common in things like search engines. Grandmas who occasionally check their email might not get it (but many do I'm sure!)... but I think a lot of others do. For things like search it is ok, if people know the magic 'terms'. If they do not know the terms, then they can just enter text to search normally. The mini language is used by advanced users.

This is quite good for single line free form data entry. Since people only need to know the concept that you have 'key:value'. It's slightly easier than using urls, since separators can be different things.

csv files

Next up are comma separated files - csv.
For example:

name,gender,age
Bob,female,22

These are like spread sheets. Many people seem to be able to edit these quite fine. Especially if they have a spread sheet program to do the editing.

URLS

URLs are a mini language. With things like # anchors, query strings, and even paths being used by people all the time.

?name=Bob&age=22&gender=female

ini files

Common as configuration files.

[heading]
key=value

subsitution templates

Common for web sites, and email systems.

Hi ${name},

${age} ${gender}

basic html and other mark up languages

Quite a lot of people know bits of html. 'How do I do a new line? oh, that's right: brrrrrrrr.'

However, I think that html is on the edge of too complicated. Modern html is especially complicated.

Things like markdown, bbcode, and wiki languages all fall into this category. The languages can sometimes have only 5-10 elements - which make them easy to learn the basics.

Older Wiki language text could look just like text someone would write in notepad. However modern ones - like html - now have all sorts of ${}[]==++ characters with special meanings.

Are there any other mini languages which are easy to understand for non-programmers?

pypy svn jit cheat sheet.

2010-01-10T13:06:00.008Z

A quick cheat sheet for trying out pypy jit. This is for a ubuntu/debian 32bit machine translating the jit version of pypy-c.

# install dependencies for debian/ubuntu.
sudo apt-get install python-dev libz-dev libbz2-dev libncurses-dev libexpat1-dev libssl-dev libgc-dev libffi-dev

# download from svn.
svn co http://codespeak.net/svn/pypy/trunk pypy-trunk

# Translate/compile the jit.  This part can take a while.
#   Don't worry. Relax have a home brew.
cd pypy-trunk/pypy/translator/goal
./translate.py -Ojit targetpypystandalone.py

# Or for the low memory pypy-c use this translate instead...
#./translate.py --gcremovetypeptr targetpypystandalone --objspace-std-withsharingdict

The pypy getting started documentation has more about it if you're interested.

pypy has most standard modules up to python2.5... and some from newer versions.

I didn't need any external compiled extension modules... just sqlite, and cherrypy. So for this project I could use pypy! *happy dance*

Didn't take long to port my code to it. I only had a couple of issues, which people in the #pypy channel on freenode helped me with.

There is an alpha version of the sqlite3 which uses ctypes included with pypy as the 'pysqlite2' module. Seemed to work well enough for me, and passed all my tests.

import sqlite3
# for pypy, since the sqlite3 package is empty... but they have a pysqlite2.
if not hasattr(sqlite3, "register_converter"):
from pysqlite2 import dbapi2 as sqlite3

Another issue I had, was I was using 'x is not y' to compare two integers in one function. In cpython they use a hack so that the first 100 numbers or so share the same identity. Since the numbers were always less than 100, this code was working fine in cpython. However, pypy doesn't have that problem/feature - so I just used the != instead.

I think those were the only two things I had to change. All my tests were passing, and it was the end of the day, so I went down the pub to see a friend visiting from Tokyo who was having a birthday.

Unladen swallow review. Still needs work.

2010-01-08T11:58:00.014Z

Tried out unladen swallow on two work loads today. After the announcement they are going to try and bring it into core. So I finally got around to trying it (last time the build failed). An hour or so later, the build finally finished and I could try it out. The C++ llvm takes ages to compile, which is what took most of the extra time. What follows is a review of unladen swallow - as it stands today.

The good part? Extensions work(mostly)! w00t. I could compile the C extension pygame, and run things with it.

Now to run code I care about, my work loads - to see if their numbers hold true for me.

cherrypy webserver benchmark: crash
pygame tests: some crashes, mostly work.
pygame.examples.testsprite : random pauses in the animation.

The crashes I've found so far seem to be thread related I guess. The cherrypy one, and some of the pygame ones both use threads, so I'm guessing that's it.

Random pauses for applications is a big FAIL. Animations fail to work, and user interactions pause or stutter. Web requests can take longer for unknown reasons etc. I'm not sure what causes the pauses, but they be there(arrrr, pirate noise).

LLVM is a big, fast moving dependency written with another language, and a whole other runtime (C++). Unladen swallow uses a bundled version of it, since they often need the latest... and they need the latest fixes to it. This might make it difficult for OS packagers. Or LLVM might stabalise soon, and it could be a non-issue. Depending on C++ is a big issue for some people. Since some applications and environments can not use C++.

The speed of unladen swallow? Slower than normal python for *my* benchmarks. Well, I couldn't benchmark some things because they crash with unladen... so eh. I might be able to track these problems down, but I just can't see the benefit so far. My programs I've tried do not go faster, so I'm not going to bother.

Python 3 seems to be 80% of the speed for IO type programs like web servers (cherrypy) (see benchmarks in my previous post). However unladen-swallow only seems to be 10-20% slower for pygame games, but the random pauses make it unusable.

Python2.x + psyco are way faster still on both these work loads. 20%-100% faster than python2.6 alone. Psyco, and stackless are both still being developed, and both seem to be giving better results than unladen swallow. Using selective optimisation with tools like shedskin, tinypyC++, rpython, cython will give you 20x speedups. So for many, writing code in a subset of python to get the speedups is worth it. Other people will be happy to write the 1% of their program that needs the speed in C. This is the good thing about unladen swallow... you should be able to keep using any C/C++/fortran extensions.

Unladen-swallow has a google reality distortion bubble around it. They only benchmark programs they care about, and others are ignored. There are other peoples reports of their programs going slower, or not faster. However the response seems to be 'that is not our goal'. This is fine for them, as they are doing the work, and they want their own work loads to go faster. However, I'm not sure if ignoring the rest of the python communities work loads is a good idea if they are considering moving it into trunk.

It's too early to declare unladen-swallow done, and good imho. I also think better research needs to go into it before declaring it an overall win at all. Outside review should be done to see if it actually makes things quicker/better for people. For my workloads, and for other peoples workloads it is worse. It also adds dependencies to C++ libraries - which is a nono for some python uses. Extra dependencies also increase the startup time. Startup time with unladen swallow is 33% slower compared to python for me (time python -c "import time").

Let's look at one of their benchmarks - html5lib. See the issue html5lib no quicker or slower than CPython . They arranged the benchmark so unladen-swallow is run 10 times, to allow unladen swallow to warm up. Since Cpython is faster the first time through.

blue - unladen-swallow, red - cpython 2.6. Time(y) for 10 runs(x).

Notice, how jumpy the performance is of unladen on the other runs? This might be related to the random pauses unladen swallow has. I don't like this style of benchmark which does not account for the first run. Many times you only want to run code on a set of data once.

When looking at their benchmark numbers, consider how they structure their benchmarks. It's always good to try benchmarking on your own workloads, rather than believing benchmarks from vendors.

Memory usage is higher with unladen swallow. It takes around two times as much memory just to start the interpreter. The extra C++ memory management libraries, the extra set of byte code, and then extra machine code for everything has its toll. Memory usage is very important for servers, and for embedded systems. It is also important for most other types of programs. The main bottleneck is not the cpu, but memory, disk, and other IO. So they are trading better cpu speed (theoretically) for worse memory. However since memory is often the bottleneck - and not the cpu, the runtimes will often be slower for lots of work loads.

It seems python2.6 will still be faster than unladen swallow for many peoples work loads. If they do not get other peoples programs and workloads working faster, or working at all, it will not be a carrot. As peoples programs work, and go faster with python2.6/2.7 it will be a stick*.

Unladen swallow has not (yet) got to it's 5x faster goal, and for many work loads it is still slower or the same speed. For these reasons, I think it's too early to think about incorporating unladen swallow into python.

* (ps... ok, that made no sense, sorry. Sticks and carrots?!?... donkeys like carrots, but so do ponies. I don't think we should hit people with sticks. Also people don't like carrots as much as perhaps chocolate or beer. Perhaps all this time hitting people with sticks and trying to get them to do things with carrots is the problem. Python 3 has heaps of cool things in it already... but more cool things always helps! Beer and chocolate would probably work best.)

Using a html form as the model.

2010-01-07T06:31:00.001Z

I've mentioned this technique before, but I think it is worth repeating more clearly. Without any other code clouding the main message: 'All you need is ~~love~~ html.

A HTML form can describe a model of your data quite well. This lets you go from HTML form design to a working Create Read Update Delete (CRUD) system. A CRUD can be thought of as a nice admin interface to edit your data (or a CMS if you prefer that acronym).

For example:

<form action='savepage' method='post'>
title:<input type='text' name='title'>
<br>textarea:
<textarea name='content'></textarea>
<input type='submit' name='submit'></form>

That's all you need to create a CRUD. Things like validation can be defined in the html easily enough then implemented in your code to be checked server side, and client side. Parse the form to get the model, and go from there.

The benefits are simplicity, and that designers can make a form, and then pretty quickly go to a working system. No need to edit sql, java, php, python, etc - just normal html forms.

Another benefit is more Rapid Application Design (RAD). From the html design you can quickly move to a working app. Especially in a work flow where the designers and clients mock up various forms - this is quicker. It also stops blockages in the production pipeline. Blockages that happen when waiting for a python/php/java programmer to implement the model.

Multiple file uploads with html 5, cherrypy 3.2 and firefox 3.6beta5.

2010-01-07T01:48:00.006Z

Here's an example for uploading multiple files with HTML 5 and newer browsers, like the firefox 3.6 beta5. You can shift/ctrl select multiple files, and even drag and drop them in. Makes for a much nicer user experience uploading files - rather than having to select one at a time, or having to load some java/flash thing.

It uses the unreleased cherrypy 3.2, with it's new request entity parsing tool hooks. See http://www.cherrypy.org/wiki/RequestBodies for details about the new control allowed over the whole process. It's a lot easier to make custom request entity parsing behaviour now, and in a much less hacky way than before.

http://rene.f0o.com/~rene/stuff/cherry_multi_fileupload.tar.gz

With the tool in there, files come in as a list of files instead.

Oldest python file in your home directory?

2010-01-06T06:42:00.002Z

Feeling just a little nostalgic this time of year.

Just made a little script to find the oldest python files on your hard drive.
http://rene.f0o.com/~rene/stuff/oldest_python.py

oldest_python.py [path]
oldest_python.py mystuff/python
oldest_python.py
oldest_python.py ~

Update: Lennart mentions a unixy way in the comments of finding oldest files with this:
find . -name '*.py' -printf "%T+ %p \n" | sort | more

With that I found some really old python files of mine... woh! the oldest ones dated in 1998. There are older C, C++, haskel, javascript, java, pascal, prolog, asm, sql, perl, etc, etc and heaps of other old junk files, but those are the first ones I could find written in python.

I guess that means I've been programming python for around 12 years now. Python was at version 1.4 or so, and 1.5 was released not long after. New style objects did not exist, and it was not all too uncommon to be able to segfault the interpreter (ping could easily crash linux & windows in those days, so python was doing pretty good).

So what did some of the older python files do?

Cut up writing was the oldest tool I found. Cutup writing was a technique used a lot in the 90s (before then and to this day as well). The idea is that you cutup various pieces of writing, and move the pieces around to get ideas.

After that came a script to randomly ping different web hosts (by http) every few minutes. In 1998 it was common for ISPs to disconnect you from your modem if you were idle for a while. So this script seems to ping one of a random selection of hosts and then go back to sleep. I remember this being the first thing I wrote in python. It took me less than an hour to learn enough python do this little script. That one hour probably saved me $300/year or so in telephone bills.

After that there is a script to convert all file names to lower case. Useful for bringing the contents of FAT drives onto a linux box.

Then there was some thread testing code. Threads these days are way better, with lots better tools, better implementations, and more known about how to use them appropriately. Using threads in those days for IO was pretty crazy... but apache used processes! Python had a wicked set of async IO servers in its toolbox. Which were pretty darn cool in the day.

Finally there were some mp3 making tools - to convert my massive collection of CDs. This was when some machines could barely play mp3s without crackling. Seems my tool would use various linux tools to make my job easier. Rip cds, get their names, convert them to mp3s. Using some old scsi drives I could go about my business without my machine slowing down completely and becoming unusable.

What are your oldest python files? http://rene.f0o.com/~rene/stuff/oldest_python.py

Fifth London code dojo 6.30pm Thursday 7th of Jan.

2010-01-05T16:10:00.007Z

Here's the london dojo google calendar for much of 2010.

More details, and signup here: http://ldnpydojo.eventwax.com/5th-london-python-code-dojo/. The idea is to practice, learn and teach python skills.

python3.2(svn) python2.7(svn) and the new GIL, cherrypy as a test.

2009-12-25T10:52:00.013Z

I've done some quick benchmarks for the unreleased python3.2 on the unreleased cherrypy webserver for python 3. Also with the unreleased python2.7.

Here are the results...

python3.1

Client Thread Report (1000 requests, 14 byte response body, 10 server threads):

threads | Completed | Failed | req/sec | msec/req | KB/sec |
   25 |    1000.0 |    0.0 |  533.32 |    1.875 |  93.75 |
   50 |    1000.0 |    0.0 |  525.86 |    1.902 |  92.69 |
  100 |    1000.0 |    0.0 |  522.96 |    1.912 |   92.1 |
  200 |    1000.0 |    0.0 |  523.83 |    1.909 |  92.25 |
  400 |    1000.0 |    0.0 |  506.92 |    1.973 |  89.27 |
Average |    1000.0 |    0.0 | 522.578 |   1.9142 | 92.012 |

python3.2

Client Thread Report (1000 requests, 14 byte response body, 10 server threads):

threads | Completed | Failed | req/sec | msec/req | KB/sec |
   25 |    1000.0 |    0.0 |  555.72 |    1.799 |  97.78 |
   50 |    1000.0 |    0.0 |  558.86 |    1.789 |  98.52 |
  100 |    1000.0 |    0.0 |  552.87 |    1.809 |  97.45 |
  200 |    1000.0 |    0.0 |  546.09 |    1.831 |  96.27 |
  400 |    1000.0 |    0.0 |  548.64 |    1.823 |  96.53 |
Average |    1000.0 |    0.0 | 552.436 |   1.8102 |  97.31 |

So here you can see a small improvement in the 400 threads version with the new GIL in python3.2.

Python 3.2 threads seem more scalable in this benchmark compared to python 3.1. Also faster overall (20-40 requests per second faster).

python2.6

Python2.6 still beats python3.2 for cherrypy benchmarks. These are a network IO heavy, string processing heavy, with big python thread thread usage. So an ok benchmark for the new GIL work in my opinion.

Note that both python3.2, and cherrypy for python 3 are not released.

Client Thread Report (1000 requests, 14 byte response body, 10 server threads):

threads | Completed | Failed | req/sec | msec/req |  KB/sec |
   25 |    1000.0 |    0.0 |  660.54 |    1.514 |  116.43 |
   50 |    1000.0 |    0.0 |  671.01 |     1.49 |  118.28 |
  100 |    1000.0 |    0.0 |  663.84 |    1.506 |  117.12 |
  200 |    1000.0 |    0.0 |  664.85 |    1.504 |  117.19 |
  400 |    1000.0 |    0.0 |   651.9 |    1.534 |   114.8 |
Average |    1000.0 |    0.0 | 662.428 |   1.5096 | 116.764 |

python2.7

Python2.7 is faster still.

Client Thread Report (1000 requests, 14 byte response body, 10 server threads):

threads | Completed | Failed | req/sec | msec/req | KB/sec |
    25 |    1000.0 |    0.0 |  695.33 |    1.438 | 122.79 |
    50 |    1000.0 |    0.0 |   684.6 |    1.461 | 121.12 |
   100 |    1000.0 |    0.0 |  688.99 |    1.451 | 121.67 |
   200 |    1000.0 |    0.0 |  682.94 |    1.464 | 120.49 |
   400 |    1000.0 |    0.0 |  641.01 |     1.56 | 112.78 |
Average |    1000.0 |    0.0 | 678.574 |   1.4748 | 119.77 |

It's also worth noting that 100-120%(out of 200%) of the two cores cpus are during the run of each version of python tested. Even though the GIL is released for many parts with python and cherrypy, and even though the benchmark is very IO heavy, both cores are not loaded. It's generally a good thing for a webserver to not load up the CPUs - but in a benchmark you want them to go full speed.

Also, during the tests the benchmarking tool 'ab' is run on the same machine skewing results. However ab seems to only use 1% of the CPU during the tests (according to top).

1.6ghz core 2 duo, ubuntu (save the) karmic koala 32bit version.

Shrinking the stack to save some memory.

So how do we reduce the memory usage of threaded programs? Reducing the stack size is one idea. Since threads do not share a stack with each other, each thread takes up a nice big stack. On ubuntu karmic koala it seems to default to 8MB or so.

Note, this is dangerous and can segfault your interpreter if you do not have enough stack space for some operations. So make sure you test things well before doing this.

In 2.6 you do:

import thread
thread.stack_size(32768*2)

Whereas in python 3.x you do:

import threading
threading.stack_size(32768)

3.2
----
(default stack)  * 110MB virt, 10MB resident
(adjusted stack) *  15MB virt, 10MB resident

2.6
----
(default stack)  * 107MB virt, 8MB resident
(adjusted stack) * 12MB virt, 8MB resident

Note, the python3.x versions start using up 2 gigabytes of memory in the benchmark. Compared to a peak of 600MB or so with python2.6. Showing some sort of memory leak in either python or cherrypy (note, both are pre-release).

It seems python3.2 worked with a smaller stack compared to python2.6. python2.6 segfaulted with 32768, but worked with a stack sized 32768*2. However python3.x used more resident memory, and virtual memory.

update:
I've uploaded profile(python cProfile), results and scripts here: http://rene.f0o.com/~rene/stuff/cherrypy_benchmarks. Strangely using python2.7 with the benchmark and cProfile did not finish (well it was still going 30 minutes later). So I guess that is a bug somewhere.

Invent with python - a new pygame/python book.

2009-12-23T11:33:00.002Z

Invent with python 'Teach yourself how to program by making computer games!' is a free book now in its second edition. It is released under the creative commons licence Attribution-Noncommercial-Share Alike 3.0 United States - and the author just asks for a donation if you liked reading it.

The book takes the fun approach of teaching programming through making games... the way many of us learnt(and are still learning) to program.

As a bonus it uses python 3 - which pygame has supported for a while.

Congratulations to Albert Sweigart for finishing his second edition of the book.

structuring modules/packages and the cdb database for websites and python packages

2009-12-22T10:16:00.013Z

Integrated modules are nice, but so are modular packages.

How can we have both? That is, keep all things for a module(or sub package) in one directory, but also have a nice integrated system built on top of that?

Often for one package I will have a file layout like so:


/setup.py
/[package_name]
/[package_name]/[package_module].py
/[package_name]/tests/[package_module]_test.py
/[package_name]/examples/[package_module].py
/[package_name]/docs/[package_module].py

Then each module has its tests, docs, and examples all mixed in the one directory. This is nice if you want to have all of the tests together, and all of the docs, and examples together.

However then all of the modules are mixed in together. Meaning it is harder to separate them, and keep one module in its own directory. Having everything for one thing together is nicer for developers I think.

Using namespace packages through ~~setuptools~~ distribute is one way. There is a draft pep around to put namespace packages into python as well. However, this feels way too heavyweight for me. Especially since they add multiple paths to the interpreter... meaning python has to do stat, and open, calls for every single path - which slows down the startup and import speed of python even when not using those packages.

Another way is to use a sub-package for each module. This turns each module into a sub-package.

However, some people put all of their stuff in an __init__.py file. Which is kind of hard to find, and not very descriptive. Also if you are editing a dozen __init__.py files for a project, it is really quite annoying. It is much better to have it in a 'modulename.py' file, and then import that in the __init.py file. Still slower than using modules directly though, since there are still extra stat calls, and extra file opens and reads.

Using the __init__.py file is still annoying in that you need the extra file, and have to remember what this magic file does.

This one liner can put all of the sub packages modules into the name space:

import os,sys
for f in os.listdir('somepackage'):
  if os.path.isdir(os.path.join('somepackage',f)):
      sys.path.insert(0,os.path.join('somepackage',f))

This is magic itself, and has a pretty bad code smell.

Other issues are playing nicely with things like cx_freeze, py2app, py2exe and the many other tools for managing python packages and modules... who don't know about namespace packages, or my custom magic.

I could write an import hook, or try out importlib in python3.1, but those would probably have similar issues to other non-standard ways.

So in the end, I think I'll stick with a fairly standard method of doing it... using a __init__.py file which imports the proper module into its namespace.

This lets me do import somepackage.somemodule and have it work. I can also "cd somepackage/somemodule/; python somemodule.py" and have it work.


/setup.py
/[package_name]
/[package_name]/[package_module]/
/[package_name]/[package_module]/[package_module]_test.py
/[package_name]/[package_module]/[package_module]_example.py
/[package_name]/[package_module]/[package_module]_docs.txt

This is how I'll structure pywebsite. With a separate directory for each sub module. This gives modularity, making it scalable to many modules, and it will simplify contributors lives. Since it uses standard packaging it should be compatible with most tools. Another bonus is that from within the source download I can just do:

import pywebsite

without having to call the setup.py script or install it. If a module changes from a .py file to a .so or a .dll or the other way around I won't have any issues either.

It is now a little harder for sub-packages within a package to refer to other sub-packages, but can be done with newer versions of python.

Sub-packages are more heavyweight than just using modules in your package... but it does seem cleaner and more extensible. However it is not as heavyweight as using namespace packages.

Any other issues with this approach?

namespace packages

Using the method outlined above, still does not allow me to split up a package into multiple separate files. This is ok though, since to start with I want to keep most of the modules in the one place... in the one repository, with the one bug tracker. However designing for extensibility from the start is useful, so we consider how it can be done.

This is where namespace packages are useful.

There is this draft pep: pep-0382, and also the ~~setuptools~~ distribute namespace packages (which is what everyone is using to do them). Here is the ~~setuptools~~ distribute documentation for namespace packages.

It should be possible to use some sort of hack, so that once you import a package, it searches for other packages with the same namespace.

You can use a packages __path__ attribute to tell it to look in other paths for importing modules.


>>> import pywebsite
>>> pywebsite.__path__
['pywebsite']
>>> pywebsite.__path__.append('lala')
>>> import pywebsite.bla
>>> pywebsite.bla.__file__
'lala/bla.py'

Using __path__ is outlined in the Packages in Multiple Directories part of the python tutorial.

As an example, say I had a 'otherpackage' package, and then someone else wanted to maintain part of that package, or we wanted to separate part of it out. Let's call this namespace package: 'otherpackage_doit'. It could install itself as a directory and package called 'otherpackage_doit'. Then import otherpackage_doit would work fine. However import pywebsite.doit would not work. You can't just call the package 'otherpackage.doit' either - since python will first look in the otherpackage package for the doit package, making import otherpackage.doit fail.

From a users discovery perspective, I would expect otherpackage.doit to be in otherpackage/doit. So that's where I'd look first. Installing into that directory would probably be best then. However that is not a very good method. After that, I'd probably do "print(otherpackage.doit.__file__)". Or I might do a "locate otherpackage.doit" command.

Really I just wish python3.2 could be changed so that 'otherpackage.doit' package is automatically a namespace package - without having to mess around with weird magic .pth files or declaring things in setup files like setuptools does.

So how can we retrofit(hack) existing pythons to do this for us? We need to get the python import machinery to search for otherpackage.* packages outside of the otherpackage directory. I'm sure it's possible with python somehow... Inserting 'otherpackage.doit' into sys.path does not work. You can't even have a package name with a '.' in it.

So I'll give up on namespace packages for now until a suitable option presents itself... or I have more time for research. Separate packages will have to live in the same file, but can still be separate with source control tools - like bzr, svn, github etc.

Still not fast enough... cdb databases for websites and python packages

However, the standard python package method is not the fastest. Supporting cgi operations for a web library is a good idea. This is because many webhosting platforms still only support python through cgi. So loading heaps of files for every cgi request is not an option. It is possible to get acceptable performance out of cgi and python... just many of the large frameworks have poorly optimized loading. Many frameworks rely on long running processes to avoid the slow load times. Using django via cgi in an embedded 130mhz arm with a limit of 10MiB is not going to work very well (or at all).

So how to make it faster for embedded/cgi apps?

Firstly an executable can be made. Using tools like py2exe. This can pack all of your data inside the executable.

One common method people try is to use the zip format. This works fairly well but is not optimal. Zip files are nice as they are supported by OS level tools, and file managers - so this will be one option to use. The downside, is that it makes the files harder to edit. I see .zip files as an optimisation that hinders usability. Especially .egg files(which are just .zip files) are bad, as it makes it harder to debug or change programs. So like .pyc files I think the zip file should be generated as needed - but having the full source tree there to change is very useful. If someone changes the source, the zip file should be regenerated as needed.

Another option is a constant database (cdb). cdb is a very simple constant database format used in things like djbdns, qmail and for other things. cdb happens to be one of the fastest(if not the fastest for constant databases [benchmarks pdf]). cdb is perfect for python packages that are not meant to change since they are so quick.

cdb is also pretty good for serving data from websites. Since often with websites much of the data is mostly static(constant) - a cdb key/value database is a nice optimisation over files on the file system. There are less syscalls, and less latency issues.

Zip files can also be used as .jar files by some browsers(firefox), to reduce latency on websites too. See the jar url scheme for details on how to put all your static files into a zip file(jar file).

There are some of my python module_experiments in:
bzr co http://rene.f0o.com/~rene/stuff/module_experiments
http://rene.f0o.com/~rene/stuff/module_experiments.tar.gz

So I have now refactored pywebsite to use a sub-package for each module, so that all the tests, docs and examples for that module are within its own sub-package. Using a zip/cdb file for imports will be left for later, as will namespace packages.

hashing urls for authorisation, and pywebsite.signed_url

2009-12-21T12:49:00.013Z

Using a hash with a salt is one way of validating urls. With it you can tell that it is quite likely the url was generated by someone with a secret. That is, generated by someone authorised to make that url. This post describes using this technique with the hmac python module applied to urls(and object method calls).

Generating thumbnail urls is a good example. If you have a url which can generate a thumbnail of an image at a specified size that is pretty cool. Except then you can have someone generating images at all sorts of other weird sizes by exploiting the width, and height parameters (eg making 100,000 pixel wide thumbnails). Or perhaps you want to limit it to a certain set of images.

You could always code in your thumb nail generating routine which variables are valid... but this has problems. First it makes your routine less flexible. Separating authorisation is always a nice idea.

Another way (with HMAC like using pywebsite.signed_url) is to generate the urls and add a hash to them with a secret salt. This way you can be fairly sure that it was generated by someone with access to the secret salt.

This system is not as secure as using PKI, but it is a lot quicker to implement and is faster running.

One problem with using hashes is when you have to change the salt or hash scheme... then all your old urls are invalidated. Very annoying (sad panda).

Below is a little example of how you would hash protect some object methods, so only methods which are generated by authorised people are allowed. Note that it only uses a hash with a length of 6 characters, this is to keep urls short.

from pywebsite import signed_url, imageops

class SignedImages(object):
    salt = 'somesecret'
    root_dir = '/tmp/'

    def thumb(self, hash, width, height, rotate, gallery, image):
        """ serves a thumb nail
        http://localhost:9942/thumb/XXXahashXXX/1000/1000/0/test/colorpicker.jpg
        """
        salt = self.salt
        keys = None
        length_used = 6
        values = [width, height, rotate, gallery, image]
        if not signed_url.verify(values, salt, hash, keys, length_used = length_used):
            raise ValueError('not a valid url')
        cache_dir = os.path.join(self.root_dir, "cache/")
        path_to_galleries = os.path.join(self.root_dir,  "galleries/")
        iops = imageops.ImageOps(cache_dir, path_to_galleries)
        path = iops.resize(gallery, image, width, height, rotate)         
        return cherrypy.lib.static.serve_file(path)


    def gen_thumb_url(self, width, height, gallery, image):
        values = [ width, height, '0', gallery, image]
        salt = self.salt
        keys = None
        length_used = 6
        hash = signed_url.sign(values, salt, keys = keys, length_used = length_used)
        url = "/".join(["thumb", hash] + values)
        return url

   thumb.exposed = True
   gen_thumb_url.exposed = True

See the signed_url.py on launchpad file browser, or bzr co lp:pywebsite. Also see pywebsite.signed_url.signed_url_test for the unit tests. It is quite a simple module, and you could roll your own quite easily (with the python hmac module). Well, maybe not well easily... eg, consider the flickr exploits This code is also probably vulnerable( eg, SHA-1 is now vulnerable to collisions), but at least it will stop basic fiddling with unauthorised urls and shouldn't have the same vulnerabilities as the flickr API used to(and the facebook API might still have).

updates: been through a number of changes... the name has changed from hash_url to signed_url.sign, signed_url.verify. It has also moved into its own sub-package (like all pywebsite modules now live in their own self contained sub-package). Fixes for timing attack(unit tested). Uses hmac.

In Switzerland. Lots of snow!

2009-12-21T06:15:00.004Z

In Switzerland. Had fun visiting an old friend and his new girl friend. Lots of snow came whilst we're here which was fun. However, the planes didn't like it, and they got cancelled. So going back to London today instead(if the planes work).

Python the GIL, unladen swallow, reference counting, and atomic operations.

2009-12-17T06:40:00.001Z

The unladen swallow project comments on the global interpreter lock and suggests long term dropping reference counting in favour of GC. http://code.google.com/p/unladen-swallow/wiki/ProjectPlan#Global_Interpreter_Lock. They are also now favouring an incremental GIL removal from python.

However, confusing getting rid of the GIL, with requiring using garbage collection is wrong. This mini-essay will explain why reference counting is good for python, and why you do not need to get rid of it for multiple CPU systems. Then there will be descriptions of other improvements to python for multiple CPU systems that can be made.

Reference counting has a number of advantages over various GC schemes see http://en.wikipedia.org/wiki/Reference_counting#Advantages_and_disadvantages. As a python programmer, optimizing for the programmer is what is most important... and reference counting makes it easier for the programmer. It makes programs much more deterministic, meaning that reference counting code is much easier to think about in our human heads.

Below are two real world examples of atomic reference counting. This makes reference counting thread safe, by using atomic operations.

Here are two mainstream open source object systems that have implemented it:

QT: from qt 4 released June 2005:
http://doc.trolltech.com/qq/qq14-threading.html#atomicreferencecounting

glib: released august 2005.
http://mail.gnome.org/archives/gnome-announce-list/2005-August/msg00048.html

It is possible to make reference counting thread safe in a performant, cross platform manner. Using the argument against reference counting just because of the GIL and thread safeness is a bad argument - as proved by at least these two highly portable, highly performant, open source pieces of code. Not just research papers, but real life code that has been used for a number of years now.

There has been a lot of research and real life code which improves reference counting, which python could take advantage of - rather than moving to a GC.

Simple Direct Media Layer also has a cross platform atomic operations API in SDL 1.3.

Locality of memory is much more important than using GC or reference counting. That is operating on memory worrying about if it is in the caches (L1,L2,L3 etc), also about locality to certain cpus/cores. This is why modern memory allocators have a per thread heap... to help memory allocated on one heap be accessed more easily by threads. See NUMA for details. NUMA factors play a BIG part in modern cpus already... even on single cpu systems. Designing for cache friendliness can give big speedups.

Reference counting can be a problem for cache friendly systems... but it doesn't need to be. There are techniques for reducing, or removing NUMA issues with reference counting memory management.

Reducing memory usage of objects will also mean for faster threaded python programs.
From the pypy project: "The total amount of RAM used on a 32-bit[ED:pypy] Linux is 247 MB, completing in 10.3 seconds. On CPython, it consumes 684 MB and takes 89 seconds to complete... This nicely shows that our GCs are much faster at allocating objects, and that our objects can be much smaller than CPython's."
More details here: http://morepypy.blogspot.com/2009/10/gc-improvements.html. As you can see reducing the memory size of objects can have a massive effect on runtime.

Another memory optimization for python can come from reducing the amount of allocations done by python at run time. Many real time systems have a policy of zero run time memory allocations. For python this is hard, however there are probably a number of places where memory allocations could be removed, or reduced in python. The pypy project tried to reduce the runtime memory allocations and found a number of them which sped things up.

Intelligent sizing of the stack is another memory optimization python could do. Each new thread has a copy of its own stack. So by having a very large stack, python can use less threads than is optimal. An optimization could be to try and use a very small stack, and then resize it if necessary. See the http://docs.python.org/library/thread.html for the thread.stack_size([size]) function.

Also speeding up GIL holding/removing will be good for those parts of python that already remove the GIL - especially on single CPU systems. Currently grabbing the GIL is a fairly costly operation. Using a method like that used by QT with atomic operations would be MUCH more efficient.

An incremental technique of GIL removal was taken by VMs like linux and *bsds (eg freebsd, dragonfly bsd) (and glib, qt). The idea is you make more fine grained locks on individual systems. Then you can slowly remove the GIL from parts incrementally... and you can test them. Removing the GIL all at once is silly... removing it piece by piece is a much better idea. I think the unladen swallow project is now preferring an incremental approach to GIL removal... which is nice.

A plan which approaches GIL removal incrementally is much more sane over all.

Designing APIs to work in manners better for threaded programs is another area python can get better performance on multi cpu systems. For example, by finding calls which block and making sure they release the GIL. Also by making batching APIs, allows not only avoiding the high cost of python function calls, but also makes that code embarrassingly easy to achieve parallelism.

One API that could be easily turned into a batching API is the import command. When you import 10 modules at the top of your program, much of that work could be shared or done in parallel. Likewise making the libraries be able to easily use

Documenting which calls release the GIL, or have fine grained locking/lock free programming can also help programmers more easily take advantage of multiple cpus.

Documenting which objects are safe to pickle will help people use the multiprocessing module, which does not work with objects unless the objects can be pickled. Likewise using forking can allow you to use multiple processes on data that can not be pickled. Since when you fork, the OS does the memory copying for you. The advantage over the multiprocessing library, is that with forking you only need the results to be able to be pickled... not the inputs.

Using wise madvise calls for mmap can also mark data which is read only... making multi processing much more efficient. Using madvise well, can mean much better copy on write behaviour. Improving copy on write behaviour is good for multiple processes... not just multiple threads. Likewise talking to the OS about what the program needs in other ways will also help. For example, if you know your program is going to use 1GB of data... then tell the OS that at the start. If you know it will eventually need 50 threads... then tell it at the start. If you know it needs a very small stack size... then tell it at the start.

Modern CPUs have memory prefetching commands, which can help out with telling the CPU which memory you are likely to access next. Likewise telling python, and the CPU that you want to only read or write to memory can help a lot with performance.

Apart from memory, there are many other OS level tweaks that can make your multi process python code go quicker. For example, setting the process/thread priorities. You can even ask your code to run with real time priority if it needs it. This can reduce the latency of processing, and also speed up the throughput. Easily gaining a 20% increase in throughput, or a very big drop in latency. If you do not care how fast your python program goes, then telling the OS so will help other processes in the system go faster.

Improving the performance of the built in primitives like the Queue/Mutex can also help python a lot. I've found that avoiding the python thread safe Queue can give you pretty good performance increases. Instead just using a list and using its python level atomic operations (documenting which python primitives are atomic would also help programmers a lot). A multi thread safe event queue can be quite nice. Especially if you can pass things into the queue at C level... avoiding the GIL entirely. There are a number of these available for python already (eg pygame.fastevent).

In fact for python 3.2 the GIL has been reworked: http://mail.python.org/pipermail/python-dev/2009-October/093321.html and the RLock primitive has been rewritten in C. This 'newgil' work has already been put into py3k trunk. However I think there is still quite a lot of room for improvement in the python GIL.

Another API design improvement are optimizations available for worker queues. One is sending similar tasks and data to the same cpus/cores... to take advantage of NUMA effects/cache (eg, always sending the url "/dojob1" to the same cpu).

Another worker queue optimisation is to reduce contention on the worker queue, by sending batches of tasks/data to the worker threads/processes. Rather than one task/piece of data at a time... divide the work up into many pieces and then send that across to the worker queues. eg, sending a list of 1000 images to each of the cpus to process rather than have each cpu ask for a single image to process. Dividing the work up into 16 parts of 100 each requires 16 interactions with the worker queue. Having them take one thing at a time means 16000 interactions with the worker queue - that is 16000 * 16 more opportunities for GIL locking and thread contention.

Many operations in the python stdlib could be made to use multiple threads/processes internally at the library level. Then people wouldn't need to know if the library is using async IO, threads, processes, or alien magic. eg, a url downloader might be given a list of URLs to GET... and return the responses. Underneath the covers it could use an async library, or even multiple threads to get the job done. This library level parallelism can happen if the APIs are given a batching design... that is given multiple inputs, and expecting multiple outputs. Then it is possible for library authors to try and choose the best technique. Taking single inputs, and giving single outputs does not let the library authors implement parallel algorithms as easily.

In conclusion, there is a lot we can do to improve python on multi cpu systems without removing the GIL entirely. An incremental approach to removing it from parts of python is a good idea as shown by other projects doing the same. Removing reference counting from python is not needed, as there are good, proven ways of using reference counting in multi CPU systems.

Writing for the latest python is FUN.

2009-12-16T11:25:00.005Z

Writing for the latest python version is just simply... fun. Even the name 'python 3000' or py3k is amusing.

It's been a joy over the last week plugging away at a few personal website and game projects using the latest python version. Over the last year I've been involved in upgrading existing code to the latest versions of python... but not really in writing it JUST for the latest versions.

No need to consider backwards compatibility problems... older pythons be damned when programming for fun! I'm able to take advantage of the latest python features in deep architectural ways. To fiddle around with the new deep magic python internals. But the new deep magic parts aren't the best bits though... it's all the elegant little improvements that make a difference.

It is of great delight finding python 3 compatible modules. It turns out that the modules available for python 3 are often of quite good quality - and well maintained. There's not all that many of them around, so it can be good just to browse through those python 3 packages that are available.

All of this fun makes me reminisce to the days of playing around with python ten years ago. Perhaps I'm romanticising it a little bit(well I definitely am!). However when first getting into python all those years ago was quite a thing. It was probably the small community, and the idea that optimizing for the programmer was more important than optimizing for the computer(no more squiggly braces :). Like the halcyon days of python2, there is a smaller community of python 3 programmers, and there are a bunch of new techniques and tricks available from python 3. These two factors remind me about what it was like to get into python all those years ago.

Most of the programmers doing python 3 stuff are either doing it for fun, or doing it for the good of the python ecosystem over all, that is to make python better for people rather than better for the machine.

So if you need a reason to get into python 3 it is this: python 3 makes python fun again.

pywebsite.sqlitepickle. sqlite VS pickle

2009-12-09T14:18:00.006Z

The sqlite and pickle modules that come with python are quite useful. So lets mix them together and see what comes out.

SQLITE Vs PICKLE

pywebsite.sqlitepickle is a little module I just made which combines sqlite and pickle for persistence. Useful since it works with python3, and both pickle and sqlite are included with pythons (including pypy).

Import the module.
 >>> from pywebsite import sqlitepickle

An in memory db.
 >>> db = sqlitepickle.SQLPickle()
 >>> db.save('key', 'value')
 >>> db.get('key')
 'value'

Can also save to a file.  So we first get a temp file name.
 >>> import tempfile
 >>> f = tempfile.NamedTemporaryFile()
 >>> fname = f.name

 >>> db = sqlitepickle.SQLPickle(fname)
 >>> db.save('key', 'value')
 >>> db.get('key')
 'value'

 >>> db.close()

The issues with this are that sqlite does not like sharing connections from multiple threads. To get around that I just create a new connection in each thread, or do db stuff from one thread. Also using pickles in persistent data can be a security issue.

Speed? ok speed. Pickle isn't the fastest, and neither is sqlite... but they are both kind of ok.

http://pypi.python.org/pypi/pywebsite
https://launchpad.net/pywebsite
http://www.pywebsite.org/

Play cards. Not business cards.

2009-12-08T15:26:00.008Z

Play cards. They're like business cards, but take 30 minutes to make five. Also, they are for handing out to people you want to play with, rather than to people you want to do business with. Well it could be used for both... but I think the normal style business card is a bit boring to give to friends.

Made some of these the other day. Based on an earlier version I designed a few years ago. They're about the size of a finger, but unfold to show other details (eg, email, website, etc).

--

buildout project that uses numpy

2009-12-06T06:33:00.002Z

Here is an example buildout project that uses numpy:
https://launchpad.net/numpybuildout/

If you have bazaar, you can check it out like so:
cd /tmp/ bzr branch lp:numpybuildout cd numpybuildout/trunk/

To use distribute instead of setuptools use the -d flag of bootstrap.py.
python bootstrap.py -d ./bin/buildout ./bin/py >>> import numpy >>> numpy.__file__ '/tmp/numpybuildout/trunk/eggs/numpy-1.4.0-py2.6-linux-i686.egg/numpy/__init__.pyc'

There you have it, a simple project that can download and build numpy off of the python package index and install it in it's own private directory for use.

update: It was broken for python2.6 and numpy 1.3. However it works again with numpy 1.4

gvim and karmic ubuntu... with a fix for you.

2009-12-05T08:45:00.000Z

gvim in ubuntu karmic koala is really annoying. It prints out a bunch of gtk warnings each time you call it. However thanks to someone making a patch, you can clean it up. It seems that the fix isn't going to come out until the next ubuntu.

** (gvim:13354): CRITICAL **: gtk_form_set_static_gravity: assertion `static_gravity_supported' failed

** (gvim:13354): CRITICAL **: gtk_form_set_static_gravity: assertion `static_gravity_supported' failed

** (gvim:13354): CRITICAL **: gtk_form_set_static_gravity: assertion `static_gravity_supported' failed

** (gvim:13354): CRITICAL **: gtk_form_set_static_gravity: assertion `static_gravity_supported' failed

** (gvim:13354): CRITICAL **: gtk_form_set_static_gravity: assertion `static_gravity_supported' failed

Very annoying to see that every time you edit a file.

https://bugs.launchpad.net/ubuntu/+source/vim/+bug/402188

So instead of switching to arch linux, or gentoo linux... or one of the other developer centric distros there is a patch! ya for patches people make and share :)

Grab the debdiff to apply the patch. Check a look at the packaging guide for details on how to build with a debdiff patch.
https://wiki.ubuntu.com/UbuntuPackagingGuide/BuildFromDebdiff

NOTE: please back up your system before trying anything here. Also DO NOT TRY THIS AT HOME AS SOMETHING MAY GO HORRIBLY WRONG!!!

I'll run through what to do here:

cd ~ $ mkdir vim-gnome $ cd vim-gnome $ wget http://launchpadlibrarian.net/35716623/vim_7.2.245-2ubuntu2.1.debdiff

$ md5sum vim_7.2.245-2ubuntu2.1.debdiff
abdb13517ec59a1a0b74b55b977e0139 vim_7.2.245-2ubuntu2.1.debdiff

Check that your md5sum is the same as this. Hey, why not look at the patch to review it for goodness? Well you don't have to, but it can be interesting looking at bug fixes.

We need to install all the tools required for building ubuntu packages...
$ sudo apt-get install build-essential fakeroot devscripts

Then grab the source for vim-gnome (or vim-gtk if you don't want to use the gnome version... replace vim-gnome with vim-gtk in all things below if you want that instead).
$ apt-get source vim-gnome

When I installed the libraries needed to build the source package...
sudo apt-get build-dep vim-gnome
I got a 'Segmentation fault'. eek. Well, time for a restart... It seems Karmic isn't all that stable for me with apt-get, dpkg and friends.

Ok, back from the restart of the computer.
Let us try it again...
sudo apt-get build-dep vim-gnome

Ya! it's working. I have to download about 100MB of packages. ... wait 10 minutes... then everything is downloaded and installed.

Apply the patch...

$ cd vim-7.2.245/ $ patch -p1 < ../vim_7.2.245-2ubuntu2.1.debdiff

Ok, lets build it again after we've applied this patch. This build step can take a long while!

$ debuild -uc -us

Ok, there's no test step... since ubuntu doesn't really have a way to run automated test suites (that I know of).

At least lintian is run and tells you something.

It should produce a bunch of .deb files for you.


vim_7.2.245-2ubuntu2.1_i386.deb         vim-gtk_7.2.245-2ubuntu2.1_i386.deb
vim-common_7.2.245-2ubuntu2.1_i386.deb  vim-gui-common_7.2.245-2ubuntu2.1_all.deb
vim-dbg_7.2.245-2ubuntu2.1_i386.deb     vim-nox_7.2.245-2ubuntu2.1_i386.deb
vim-doc_7.2.245-2ubuntu2.1_all.deb      vim-runtime_7.2.245-2ubuntu2.1_all.deb
vim-gnome_7.2.245-2ubuntu2.1_i386.deb   vim-tiny_7.2.245-2ubuntu2.1_i386.deb

Now to install the relevant packages...
$ sudo dpkg -i vim_7.2.245-2ubuntu2.1_i386.deb vim-common_7.2.245-2ubuntu2.1_i386.deb vim-gui-common_7.2.245-2ubuntu2.1_all.deb vim-runtime_7.2.245-2ubuntu2.1_all.deb vim-gnome_7.2.245-2ubuntu2.1_i386.deb

Ya! no more annoying message :)

Now, if someone made a PPA other people on different architectures could also easily update their binaries. Or perhaps the ubuntu folks will make it nice for gvim developers, and apply the patch themselves.

From Berlin, back to London again... and python dojo!

2009-12-04T09:15:00.008Z

Arrived back in London a few days ago, after a month living in Berlin.

I really enjoyed Berlin, and thought about staying there for longer... but in the end decided against it for now. Would really like to visit in the summer, when it's apparently quite a different place!


	Some pictures from Berlin

Finding the bustle and life on the streets of London quite refreshing. I think I'll be based in London for the next six months or so at this point(probably longer).

Going to the london python dojo again on the 10th. This is an interesting format for a get together. People take turns pair programming with a projector, whilst the rest of the people in the room offer comments. This time there's going to be a few short interactive presentations too. It was nice meeting python people from around London last time. Fry-it are the hosts, who offer their office, pizza(+salad) and beer(+wine) for everyone at each dojo.

pythons distutils install race condition

2009-11-27T17:03:00.005Z

Pythons distutils has a race condition where it starts to copy files into the python path whilst installing.

This is a race condition, since python programs can be importing the package whilst the package is being installed.

It would be good for distutils to install things in an atomic manner. Where things can be installed, or not installed. Like, on unix by moving the files in from a temporary directory. This would also help reduce breakages. Since if a package breaks half way installing a package then the broken version will not over write the existing version.

It's not a very serious problem, since most people don't install things on live important systems. Also some packaging tools fix the issues with the source installs.

2to3c: an implementation of Python's 2to3 for C code

2009-11-22T18:48:00.004Z

"2to3c: an implementation of Python's 2to3 for C code"

See http://dmalcolm.livejournal.com/3935.html for a description of this tool.

It works on C modules. So it should be easier for people to do ports to py3k for their C modules.

The 2to3c tool uses Coccinelle for transformations on the C code. This program has been used for linux, and other software... for updating code when their APIs change.

The perfect fit for the python C API changes!

rakarrack is a decent effects rack for linux

2009-11-05T21:14:00.006Z

rakarrack is a decent effects rack for linux.

It's not packaged for ubuntu(of course), but is fairly easy to compile. I've been using the cvs version (yes, people are still quite happily using cvs it seems).

Once you get it set up you have access to 80 presets of realtime guitar effects(2-40ms depending on your computer sound card setup).

It works with the jack audio system, so you can route audio into it, and route its audio out to other programs easily enough. You can also control it with midi(alsa, or jack). Meaning you can hook it up to a midi controller of some sort. I've been using it with an maudio axiom 25 controller, and with python scripts via pygame.midi.

It even has a very nice help manual integrated into the program... I wish more programs had a good help section. It details the 17 different effects that come with the program.

One thing which is a bit funny is that it doesn't use LADSPA or lv2 plugins. You can of course still use these other plugins through other jack programs if you want. However the ones it comes with are all fairly good quality from my tinkering. There is a lot you can do with the various plugins in different orders with different settings in each one. The 80 presets show a good variations of sounds you can get out of it.

I've been using it in conjunction with the mixxx mixing program and the hydrogen drum machine. So you don't need to use it just with guitars... any program or audio input will do.

If you are using guitars it also has a tuner which might come in handy... if you've never quite got the hang of playing that song called Tuning that you hear at the start of every gig.

Ich bin in Berlin

2009-11-02T15:33:00.004Z

Berlin seems like a fun and laid back place. Arrived here the other day... haven't had much time to look around so far.

Karmic Koala Ubuntu 9.10 beta review.

2009-10-23T18:13:00.004+01:00

This is for the beta release, and most(I think all) of these bugs have been reported in the bug tracker. (Note the bug tracker is currently reporting errors... so I can not link to bugs).

I do like a lot of things about the release... these are mainly things I don't like. So please don't take this as a bad view of Ubuntu 9.10 overall... just some criticism.

Hibernation seems to not work correctly all the time(for me)... when I close the lid, and open it later it does not seem restore properly. It shows the screen, but does not respond to input on the keyboard/trackpad. Then after a bit the screen goes black. Then I need to press the power button, and then the login box comes back up - and I can log in again. This has worked for the last two ubuntu releases, so it's annoying.

Booting seems slower... and is actually slower(timed it) for me. Perhaps I need to install from scratch for it to be faster or something. Pete Shinners has some benchmarks on his blog of ubuntu booting faster for him... boot benchmarks.

pulseaudio was installed again, stuffing up my sound settings. However after removing pulseaudio, the multimedia keys on my keyboard do not work... as they used to work in the previous release without pulseaudio. Apparently the ubuntu mixer applet only works with pulseaudio now. Yuk. If you go to the sound preferences dialog, it only works if you use pulseaudio.

So pulseaudio is using 3% of your cpu when no sound is playing, or pulseaudio does not work with your screen reader... or many other reasons you do not want to use pulseaudio, and you want to remove it ubuntus gui mixer fails.

The main pulseaudio author even says the ubuntu implementation of pulseaudio is bad on his blog... so I hope this is fixed before the official release.

Changing the speed of the CPUs/cores now requires a authentication... and typing in a password... annoying!! So now I end up leaving them at full cpu... wasting power, as I need full cpu in some situations. The usability is not considered here... I think people will just use more power.

Some of my icons were messed up, along with applets I was using. eg, my firefox icon got changed into a red circled crossed with a line through it.

I leave a memory card in the memory card holder... and now ubuntu pops a message up telling me it's there each time I open the lid. Even if I press the ignore button. Annoying. The last ubuntu did not ignore my request.

Newer versions of many software packages are available... which is nice. The graphics do seem to be faster on my intel machine... again nice :) New gcc, etc etc.

Still no jack/pulseaudio compatibility stuff. Still no OSSv4 sound system packaged :( Ubuntu have made it hard to have alsa, jack and pulseaudio all used side by side. The authors of those systems have worked to make the situation better, but ubuntus implementation is not good. Better to install audio stuff yourself for this release too.

Ubuntu has old SDL, and pygame releases... unlike a lot of other platforms which have the latest stable releases.

So expect Ubuntu karmic koala 9.10 to still have bad sound, and bad support for games. I have hopes that bugs will get fixed before the beta finishes... but I don't have very strong hopes.

update: Pete Shinners has some benchmarks on his blog of ubuntu booting faster for him... boot benchmarks. Added link to pulseaudio authors blog about poor pulseaudio on ubuntu.

web design for robots

2009-10-20T14:22:00.002+01:00

More robots are reading websites than humans - so should we be designing the websites for robots?

Thanks to the search engine wars, most visitors to most websites are by robots. But has anyone asked what these robots enjoy?

There's often a divide in artists between introverted, and extroverted artists.
"Fuck you - I make art for myself and not for others." OR "I make art for people to enjoy, I'm not so selfish and self indulgent that I just make art for myself".

So I decided to interview a few robots to find out what some of their favourite websites are. But where to find a robot to ask questions of?

I had to look no further than an internet enabled fridge. So with my notebook in hand I pull a chair up to the fridge and ask it some questions. I open with a flurry of questions, trying to provoke a response.

"Do you have any favourite websites?" I ask. No answer. Obviously this robot does not know english. I give it another ten minutes of questioning - then give up. I leave my card for the robot in case it feels like answering questions later, and pull away my chair. The fridge seems relieved I have gone.

Luckily I can just query some robots brains directly. No need for tricky questions to extract the information I'm after. I stuff a proxy in between the fridge and the internet. A week later I study the statistics of its favourite websites.

As we all know, just because you go to a website five times a day - does not mean you enjoy it. However, it might give us an idea of what this robot does like. First I noticed it occasionally calls home to it's place of manufacture. I guess to send details back home of what it's up to. However, I guess its family has moved on - 404 - page not found for where its trying to send it's details. I wonder for how many years this will go on? It would be like you sending a post card home to your family for 20 years, and never knowing that your family moved house 19 years ago. Oh well, we keep sending the post cards anyway.

First rule for web design for robots - they are really dumb, and will possibly forever, keep looking at the same urls. Robots do not change with the world around them.

The Egyptians knew about mixing colors around 4500 years ago. Do robots know how to mix colors, and do they care about colors? Some of those early Egyptian paintings and scupltures, and poems are really interesting - and still provide joy, and insight. By designing for robots today, will robots in the future still appreciate our work?

A robot mind can often not change once set. Well, until it is updated by the programmer. What happens when the robot programmer moves on to another project? Will the robot ever be updated again? Will the robot gain a split personality? The result of many different programmers updating its brain, must surely be a variety of personalities mixed in there.

Continuing these thoughts - thinking about how robots think - is not really interesting to me. I am now forcing myself to continue writing, in a workman like fashion. Sorry (you should probably stop reading my robot friends).

What if the robot has some way of asking people to tell it what is nice, good, sane, and lawful? What if the robot can ask the people to tell it things... and reward them in some way for their work? Provide the robot brain programmers with some value, and the robot will evolve - for free.

Oh, so the robots are driven by the programmers - and what the robots can sense about the world around them. Can the robots also be driven by positive and negative feedback? Well certainly, it's possible. Is it possible? Then robots need to process the input - the signals - and make sense of them.

Signals are processed by the robots, but what signals do the robots enjoy the most? Signals that fall into easily recognized patterns surely.

Maybe robots will like other robots art the best? Robots that make things for other robots? Well, robots can make things all by themselves this is true.

What if the robot masters try to tell the robots - "do not care about what the robots make - only care for what the humans make". Well, then the robots try to appear human. There are robots who try to become human, and robots that try find out the difference between a human and a robot.

Signal - human. Signal - robot. Robot, human, robot, robot... 101010101...

We shall call this test the Robot Turing test. The Turing test is the test to see if people can see if a signal is from a human or a robot. An idea from a man dead - at the hands of government lives on in our brains, and in robot brains. Shifting signals henceforth into patterns recognised a long time ago.

Robots can use the human brains as a resource. Humans can use the robots as a resource. Who is in control in the end?

ROBOTS? HUMANS? ROBOTS?

Is anything in control? Or is it stuck in an infinite loop? Are the humans and robots just sending post cards, even though their family has moved house?

Should someone tell them all to stop sending postcards?

game review - Tonk Tanks

2009-10-14T10:47:00.004+01:00

Tonk Tanks is a really small, and fun game. All game play is on one screen, and the controls are simple.

You have arrow keys to move your tank around, and a button to shoot the other tanks. It's like one of those old atari 2600 tank games, but a more modern version. Once you die, you teleport to a different respawn positions around the map.

The game is quite playable single player - but the author is working on a networked multiplayer version. Another nice addition would be multiplayer on one machine - especially with joysticks or mice support (since keyboards are evil).

The tank AI seems different and varied enough, that I have not worked it out from playing it ten times or so. Usually my games only last about five minutes before I move onto something else. It's one of those games I can play for little while when I want a short break.

Tonk Tanks works well on linux, windows and Mac (and probably other platforms supported by python+pygame). There's a windows .exe available, otherwise you need to have python+pygame installed to play. I don't think it's packaged for any linux/*bsd distributions yet.

Obama wins Nobel Peace Prize 2009 - 30 minute drawing competition.

2009-10-09T12:32:00.002+01:00

Spam detection on websites?

2009-09-29T16:56:00.002+01:00

Assume you have a user content site - or you're using software that can somehow get spam links inserted into it.

How do you find out if your website has spam put on it?

It seems a common enough problem these days... people putting spam links on websites. Surely there must be a service or piece of software to detect such a thing?

I can think of a few ways to go about writing one fairly easily (using existing spam detection tools... but applying them to a spiders crawl of your website). It would be much nicer if there's already a tool which does such a thing though.

Alsa midi, timidity, fluidsynth and jack.

2009-09-26T17:16:00.001+01:00

If you don't have a midi output on linux(cause your laptop has crappy audio hardware) you can use timidity or fluidsynth to emulate it.

timidity -iA -B2,8 -Os -EFreverb=0

Well, this piece of html has a bunch of incantations for using timidity on linux... and also gives insight into how to use alsa midi tools.

Like listing midi ports, and connection midi ports with these two commands:

$ pmidi -l
 Port     Client name                       Port name
 14:0     Midi Through                      Midi Through Port-0
 20:0     USB Axiom 25                      USB Axiom 25 MIDI 1
128:0     TiMidity                          TiMidity port 0
128:1     TiMidity                          TiMidity port 1
128:2     TiMidity                          TiMidity port 2
128:3     TiMidity                          TiMidity port 3

To connect the midi input from my usb Axiom 25 keyboard to the timidity synth the aconnect is the command to use.
aconnect 20:0 128:0

The AlsaMidiOverview has more information on things.

#remove all connections...
$ aconnect -x

# list all the connections(without using pmidi)
$ aconnect -o

# a gui for connections
# aconnectgui

Another synth that can be driven by midi is fluidsynth. qsynth is the graphical interface for fluidsynth which makes it easier to tweak. You can use it in pretty much the same way as timidity. It opens up a port (which you can list with pmidi -l), and then connect it to your keyboard with aconnect. fluidsynth is probably a bit nicer than timidity... and you can use soundfonts with it. Heaps of free sound fonts are available from resonance.org and hammersound.net.

This plugin is *very* useful:
http://alsa.opensrc.org/index.php/Jack_(plugin)

It allows all your alsa using programs to be routed through your jack server. This means you can use all of your normal audio programs with low latency, good mixing and synchonised audio - even over the network (with netjack).

Ubuntu does not have the alsa jack plugin included for some brain dead reason... even though debian has had it packaged for a year or so. However building from source is simple. http://www.alsa-project.org/main/index.php/Download (./configure && make && make install). I've gone back to removing pulse audio, as this system works very nicely for me.

Here's my old jackd script, I put in a ~/bin/jack_mine and start with screen at boot.
jackd -R -P 70 -d alsa -p 256 -n 3 -r 44100

I can now use various synths, samplers, and effects racks from my python scripts.

Jamin is a mastering program with a eq, compressors etc. I don't think I'll need it for live work. However it might be useful for lots of instruments. I don't see any reason why it couldn't be controlled by another person with a midi controller.

Next up I need to see if multiple sound cards can work... unfortunately that apparently doesn't work the best. You can create a virtual sound card from multiple sound cards, but they have trouble syncing. It's funny that it's currently easier to sync sound cards on multiple machines than on the same machine... from what I know so far.

For my use I don't think sync will matter too much... that is listening with headphones with one card, and outputting to sound system with another channel. My inbuilt sound card has two output lines already, and one line input(which can also be used for output).

Dell inspirion 1525... the channels are:
1(l) - 2(r) - first headphone plug, or speakers if headphone not in plug 0.
5(r) - 6(l) - second headphone plug.
3(l) - 4(r) - third headphone plug.
7(x) - 8(x) - unused... (not soldered on?)

So looks like 6 usable channels... nice for a crappy cheap latop :) This is much nicer than what windows vista allows me to do, and also way nicer than the pulseaudio/gnome combination. I've used this setup to output 6 channel audio with various different playback libraries including pygame.

screen for ghetto servers and startup scripts.

2009-09-25T12:22:00.009+01:00

GNU screen is a good little tool for server administration, or running things on your own remote machines. It's even good for running things locally.

I hope this is useful for people who want to run scripts every time they login, or reboot... and who need interactive access to those scripts. Or useful for those people who are already using screen, but would like to make their setup a bit better:

scripting sessions, rather than doing them manually at each login or reboot,
finding your screen sessions more easily.
restarting scripts at reboot,
monitoring,
logging,
resource control

Running things as daemons is cool... but if you'd also like interactive control occasionally, running things with screen is useful.

Most servers have screen, watch and crontab(osx is lacking watch though) - including most linux distros, *bsd, osx, windows(with cygwin). Most OSes also have their own style init scripts(scripts to run things at boot or logon). So this screen, watch, crontab combination is ok if you want to use multiple different types of computers - but reuse your scripts. If you want something robust, and good this isn't for you.

Scripting sessions
You can make a shell script to script your screen sessions. So you don't need to do them manually each time you login. This can save you a *lot* of time, and can make you less afraid of a reboot :).

-- /home/me/bin/my_screens.sh --

#!/bin/bash

# start up your 'app1' to be restarted 2 seconds after it dies, every time it dies.
screen -d -m -S app1 -t "my web app one descriptive name" /opt/local/bin/watch -n 2  /home/me/someapp1/run_app.py > /home/me/logs/app1stdout >> /home/me/logs/app1stderr

# start running an application.
screen -d -m -S proxy -t "proxy server" /home/me/someapp2/run_app.py

# connect to my my server setup with ssh keys.
screen -d -m -S server1 -t "my server" ssh myserver.example.com

# monitoring run time test of my app1 every 67 seconds
screen -d -m -S monitor1 -t "my monitor script" /opt/local/bin/watch -n 67 /home/me/bin/app1_monitor.py

Finding screen sessions easily: you can run these commands:

screen -d -r app1
screen -d -r server1
screen -d -r proxy

This will let you connect to that screen from any shell. It detatches any session that's open. This is good as you can easily remember things with just the short names you choose, eg 'app1' 'proxy' etc. Normally you have to do 'screen -ls' look at the output, find the session you want to connect to, then finally 'screen -r random_sessionname'.

At Reboot: Then you can add this to your crontab with crontab -e (be careful not to use -r!!! which is right next to e on a qwerty keyboard). Or your crontab web interface with your hosting account( for example cpanel/whm, webmin, plex etc). Note that each user has a crontab. So to run your apps as different users, just login and change each ones crontab(or use sudo, or 'su username -l -c' from the roots crontab).

@reboot /home/me/bin/my_screens.sh
# this line below can be used from a root account to run as the user 'me'.
@reboot su me -l -c /home/me/bin/my_screens.sh

Restarting: Of course your app might crash or something... Look at the first one... that one has a watch -n 2... which means "run this process, wait for it to finish, then start it again after 2 seconds." It's kind of like a ghetto daemontools. Not as good as something like daemontools... but good enough for some purposes.

Monitoring: You can have a separate tool monitoring your scripts if you like... then if your app has frozen, or is overloaded... just send it some term, then kill signals... and watch will start another one up when it dies. Consider each script a 'runtime test'. If the 'runtime test' fails, then kill the app and restart it. The app could be a ssh proxy for your vpn - in which case the 'runtime test' would see if you can ping the network, and if not kill the ssh connection... so it restarts. A webserver runtime test might see if it can do a GET request... if not kill the server. A ghetto monitoring system for sure... but simple.

Logging: It's pretty easy to add some stderr and stdout redirection with > /home/me/logs/app1stdout >> /home/me/logs/app1stderr. This way you can have ghetto logging too.

Resource control: You can use ulimit in your scripts to limit how many resources your server can use. Then if it uses too much, it will die and be restarted in two seconds. Say you think your python web server should never *ever* take up 500MB of ram, then run it from a .sh file, and put ulimit -m 500000 before it. See ulimit -a for a list of things you can limit. Ghetto-quick resource control. Similarly you can use nice and ionice to make things behave more nicely :).

Debugging: screen doesn't give you an error message with -d -m. So you can either look in your logs or try out your command first with "screen cmd". eg "screen python". You can try out your @reboot command, not just by rebooting but by setting it to run in 10 minutes from now. See cron help pages for how to do that. You might want to use a sh -l in front of your command so it's a 'login shell'. This will setup your paths and environment variables like your login. Or setup explicitly for each app/script which paths and environment variables they need.

Again this isn't for everyone... but some ideas here might be useful for your own ghetto screen usage for servers and startup scripts.

Linux sound is getting better.

2009-09-23T19:57:00.018+01:00

No I'm not talking about the free software song sung by Richard Stallman(very funny, but in a low quality .au format). Or the pronouciation of Linus and linux.

To start on this long-journey-of-a-rambling-diatribe-of-words, there's two good audio patches in the SDL bug tracker for the upcoming SDL 1.2.14 release.

One patch is for the pulse audio driver, and the other is for the alsa backend. These solve some of the high latency or scratchy sound issues some have.

That's right a new SDL release very soon... it's over a year since the last 1.2.13 release, and it seems like forever since the SDL 1.3 series begun. Most new development has been happening on the SDL 1.3 tree in the last year... so the 1.2 releases have slowed to an almost stop.

There's a good article on a x-platform atomic operation API for SDL http://www.thegrumpyprogrammer.com/node/16. That's one of the features that's been evolving over a few years, and is being implemented in svn.

In python terms, SDL 1.3 is like python3000. A refinement, and a promise to break backwards compatibility with the ABI. Note, not so much the API... the API is fairly backwards compatible... but some things must change. Also SDL 1.3 has lots of cool features I'm looking forward to.

Even though the SDL 1.3 tree is improving, and many people are now switching over to it, the SDL 1.2 series has a lot of life left in it.

So the SDL 1.2.14 release is all about fixing bugs, and applying patches. There's a lot of bug reports, and also a lot of patches in the main SDL bug tracker.

With free software and open source there is the mantra 'release early, release often' (the other mantra is 'release early, then abandon on sourceforge'). A stable version, that's used by people needs plenty of bug fixes, and people send in patches. Whereas a development version doesn't get the same kind of attention as released-and-used-by-people software. Many of these fixes done on the stable 1.2 tree will also be ported to the 1.3 tree too.

Now enough SDL 1.3 love... what else is improving in linux audio world? now for something completely different.

Well, pulse audio is frantically making releases. Three releases in september... so far, and five for 2009. Pulse and jack are also playing nicer together now(well, not packaged in ubuntu yet... grrrr, see bugs 198048 and 109659 this is critical for allowing many high end audio programs to work along side the 'beep' sound your terminal makes. Hopefully they'll get a good desktop architect (sound experience) from their job posting to fix things).

Jack is the low latency, synchronised audio system used by many professional audio programs on linux. Think unix pipes applied to audio, but in a way that works with the audio latency requirements. Both jack, and pulse audio have been ported to lots of other operating systems these days. Which can only be good for them getting more developer support... and making the linux audio world better along with it. You can see in their change logs, and repository commits that developers on different platforms other than linux are contributing quite a lot.

Even trusty old Open Sound System(OSS) has gotten better. OSS was removed from the kernel, replaced with alsa a while ago... but OSS kept going anyway. OSS4 has lots of things fixed compared to OSSv3 that most people remember using a long time ago. Including a fast transparent high quality in kernel mixer(good for crappy cards that only support one program outputing sound at a time). It also has a "record-what-you-hear" feature... for recording what is coming out of your sound card (a feature MS disabled in vista... booo!) The commercial version is now available as open source with a mecurial repository too! OSS is also quite x-platform.

What about sound applications?

The drum machine Hydrogen got a new release for the first time in three years... and this time it's not just linux only too.

A great DJ program called mixxx is another high quality multiplatform audio program. It's probably my most favourite audio program... just because it's so fun. You can even hook up real vinyl decks to it for scratch control(and midi ones). Unfortunately you can't pipe music in from other audio programs or in from a sound card... so you can't use the vinyl decks in that way. You have to use specially encoded records which the program then reads to figure out where the record is moving. The latest version features javascript scripting of midi and other parts of the program.

(go on, download it and become a dj ninja)

Guitarix is an amp emulator... it tries to sounds like various vintage guitar amps. Pretty fun to play around with.

(plenty of knobs to play with)

Especially in combination with many of the effect plugins available through the hundreds of LADSPA(guitarix is a LADSPA plugin too) and LV2 plugins. Other plugins available include vocoders and all sorts of weirdness.

Lash uptake has been good, and now lash talks dbus... letting it mix in nicely with the rest of the linux desktop ecosystem. Lash is a session system for linux audio programs. It lets you open your 12 different linux audio programs(remember audio in linux is like pipes... pipes with audio running through them instead of water... let's call them wires... but digital... maybe fiber optics... but not using light... ok whatever... why am I explaining it this way?... you're not five... too many dots. sorry.) and save your settings for later. The alternative is to each time open your 12 programs, set up the wiring between six of them, start messing around and finally... 2 hours later... realise you were supposed to be setting them up in a certain way rather than making stupid minimal beep noises to a house vocals mixed with a recording of a fart noise - filtered down to retro 8bit samples. Without lash, you couldn't save that brilliant setup and play with it later.

Audacity, the simple(yet advanced) audio editing work horse is moving towards a 2.0 by the end of the year. Audacity has been around for ages, and has been multi platform for ages. The 1.3 series seems to have been going on forever... but they do regular beta releases, and nightly builds. So it's pretty easy to get fresh versions. Do proper releases matter that much when new releases are pushed out every day? I guess so.

LiVES reached 1.0 earlier in the year after a long time in development... (since 2002!). LiVES is a video editor(which includes audio). It's actually quite useful for editing video! The other cool part of it, is that it's a VJ tool. So you can do those awesome projections you saw the last time you were in a club rushing around the place. You can control much of LiVES with midi too, which is mice.

(Make home movies of your loved ones. Like grandpa Nelson here.)

In fact lots of audio programs available for linux can be controlled by midi. Which is mice for me since you can easily do midi with python and pygame.midi.

Speaking of things midi and pythony... The vj program freej now has python wrappers! There are even five tutorials which use pygame. Unfortunately this is not in release form yet... but all this good stuff happening in the git repos.
(you too can make video art like this with freej... All you need is a crazy mask)

Both LiVES and freej use the frei0r video plugins. Which has nothing to do with linux sound getting better really. So there. Jerk(why did this guy even write this? I wish I didn't waste my time reading it.).

Comments? Important typos I should fix? Interesting linux audio things you're doing? Want to tell me how your tomatoes are growing in your garden? Gott a picture of your cat you'd like to share with me?

Where did the 'new' module go in python 3?

2009-09-22T10:19:00.004+01:00

Anyone know where the 'new' module went in python 3?

2to3 can't seem to find 'new', and I can't find anywhere with my favourite search engine either... filed bug at: issue6964.

A complete 2to3 tool should know about all modules that are missing at least. It needs to actually know what to do with those modules, but should be able to at least tell you which modules are missing. I'm not sure how to get a complete top level module list sanely... I guess by scanning the libs directory of python.

Or maybe there is a module to find all python modules?

Each platform would be slightly different of course... and there'd be differences based on configure. Also some modules have probably stopped importing or compiling at all these days.

Then you could just find the intersection and differences with the lovely set module :)
# find the difference between the modules.
top_level_modules_not_in_3 = set(top_level_modules3series) - set(top_level_modules1_2series)

Well maybe the 2to3 tool could work a different way. Instead it could find all the modules it *does know about*, and warn you if it encounters modules it doesn't know about. You can already list fixes with: 2to3-3.1 -l

But what about packages with submodules? It seems hard to pin down exactly what's included with python. Or maybe there is an easy way to find out.

update: it is printed as a warning with python2.6 -3 -c "import new" . The types module is the one to use instead. A reminder to always use the python2.6 -3 to warn you of things the 2to3 tool can not fix. python2.6 -3 is a good thing to do on your current code base for preparation for a python 3 future.

py3k(python3) more than one year on - 0.96 % packages supporting py3k.

2009-09-16T11:00:00.013+01:00

Python3 was released more than a year ago so far, and the release candidates and beta releases much before then.

How successful has the transition been to python3 so far?

One way to measure that is to look at how many python packages have been ported to py3k. One way to measure what packages are released is to look at the python package index(aka cheeseshop, aka pypi).

Not all packages ported to python3 are listed on pypi, and not all packages are listed on pypi. Also there are many packages which haven't been updated in a long time. However it is still a pretty good way to get a vague idea of how things are going.

73 packages are listed in the python3 section of pypi, and 7568 packages in total. That's 0.96% of python packages having been ported to python3.

Another large project index for python is the pygame.org website. Where there are currently over 2000 projects which use pygame. I think there are 2 projects ported to python3 on there(but I can't find them at the moment). This shows a section of the python community using it in projects. Most of the things listed on pypi are packages - not projects. It's showing what people are using for their projects - not what their libraries support. In a similar way, it could be good to see how many websites are running on top of python3. I think a lot of the people who have ported to python3 aren't really using it for their projects, but have done the porting work as a good will measure towards moving python forward.

Another way to measure the success of the migration, is to pick a random sampling of some popular packages and see if their python3 support exists.

Pygame(yes), Pyopengl(no), pyglet(no), numpy(no), PIL python imaging library(no), cherrypy(yes), Twisted(no), zope.interface(no), buildout(not sure, I think no), setuptools(no, patches available), django(no), plone(no), psyco(no), cython(yes), swig(yes), sqlalchemy(no).

With some packages being used by 1000s or 10,000s of projects, those popular projects hold back the py3k migration significantly. It would seem that some applied efforts to the right projects would help the py3k migration a lot. Perhaps a py3k porting team could be made to help port important libraries to py3k.

How about other python implementations supporting python3 features? None have full python3 support as of yet. For example jython(no), pypy(no), ironpython(no), tinypy(no), python-on-a-chip(no), unladenswallow(no), shedskin(no). However some implementations support some new python3 features.

How about wsgi? wsgi is the python specification for web gateways... that it specifies how different web frameworks, web servers and applications can talk to each other and out to http. The wsgiref module in python3 is somewhat broken, and the amendments for python3 have not made it into a new wsgi spec. However work is being done towards it with a couple of major wsgi users supporting python3(cherrypy and mod_wsgi).

Another question to ask is: 'are many projects planning to support py3k soon? Or have they decided not to work on py3k at all yet?'. It seems many projects have decided not to put in the work yet. At this point, for many projects they don't see enough benefit towards moving to py3k. Or their dependencies have not been ported, and they are waiting on those to be ported before beginning to port themselves.

How well have the python developers themselves developed the support material for people upgrading their code? It looks like the cporting guide is still incomplete and hasn't been updated in a while. However the CPython API using projects have taken up the slack... so there are now a number of extensions for people to look at for guidance. It's possible to make CPython extensions which support both 2.x and 3.x APIs.

There is now a 3to2 script being worked on. This allows projects to write their code in python3 code, and have it translated into python2 code. The python developers realised that having a 2to3 script was backwards in a way - requiring developers to stick with their python 2 code. However, many projects seem to not use the translation script, since it hasn't worked for them. Instead they seem to have either made separate branches, or made their code so that it works in both 2.x and 3.x.

Support for python3 was dropped, and python3.1 is the new python3. However python3 still exists in some distributions (like ubuntu).

So how are the various OS distributions going with their python3 support? The latest version of OSX to be released (snow leopard) uses a version of python2.6.1. Most unix distributions are using python2.6 as their main python at the moment. However most of them have also packaged python3.x as well. So it's fairly easy for people to try out python3 alongside their python2.x installation(s). macports currently has py25(286 ports), py26(206 ports), py30(0 ports... since py3.0 isn't supported by python.org), and py31(4 ports). So for macports, it has 1.9% of py31 packages ported compared to py25. This shows similar percentage to the ratio of ported packages in the pypi index(0.96%).

This is not mirrored by the number of windows downloads from python.org. Python2.6.2 windows installer had 786400 downloads, python2.5.4(104291), python 3.1(241363) 3.1.1(214871) for a total of 456234 for 3.x. This is around 58% comparing 2.6 and (3.0+3.1). Strangely about the same amount of people are downloading 3.0 as 3.1 - even though it states that 3.1 is the new py3k and 3.0 is not supported anymore. This is just windows download counts for August... if you compare it to most unix distributions, they almost all come with python2.5 or python2.6.

So, is the python3 migration going along swimmingly? Or has it failed to reach its goals(what were its goals if any)? What can we do to help? Should we even help at this point? comments?