Monday, December 19, 2011

CherryPy - I love it because it's pyhon.

CherryPy moved to bitbucket a while ago, and also got a website refresh.

I love cherrypy, since I can use it with python 3 - not legacy python.

I love cherrypy, because I can just use python objects.

I love cherrypy because it has a clever name.

I love cherrypy, because it just works.

I love cherrypy, it is minimal.

I love cherrypy.

Wednesday, December 07, 2011

Thoughts on Python 3

It's awesome.

Go get some!

Sunday, November 20, 2011

PyGameZine issue0

Today we are very excited to announce the launch of PyGameZine!

There is more information about it on the website.

Friday, November 11, 2011

Vintage in Paris

Our Vintage in Paris app has been published on the Nokia app store!

The process was pretty simple, since it was a very technically basic but content heavy app.

The main difficulty in this app was walking around the streets of Paris taking pictures, and researching really good places to find Vintage.

Why a Vintage in Paris app?

Finding cool second hand and retro clothes is a hobby of mine, and my partners.   It's always fun finding interesting old styles, rummaging through clothes racks.  Especially in other countries, and especially when you find a bargain.

There's something special about wandering down some back alley, and finding an old shop stuffed to the rafters with old stuff.  It's like taking a journey back in time to some fantastical version of Paris you may have in your head.  If you like Art Deco stuff, then Paris is the place to get it.

What was the signup to the Nokia store like?

I have been signed up as a Nokia developer for quite a while already, so there hasn't been too much paper work, or forms to fill out.  I can't remember what I needed to do originally, so there might be more bureaucracy than I remember.  I'm sure there will be more forms for me to fill out in the future too.

Technical parts.

The app is available as a J2ME Java app, a Meego app, and a WRT app.  Nokia has so many different platforms now!  They are adding a couple more platforms too.  Luckily HTMLish apps "work" on all their platforms.

Our Content Management System(CMS) is written in Python/JavaScript/C/SQLite and uses pygame for image processing.  The CMS part is still in development, but is slowly getting there.

We use a form based system to define schemas.  I wrote about this in a "Validation through html forms" blog post, and mainly in "Using a html form as the model".  Basically this allows us to create a html form, and then that's it for the content type.  No need to mess around with schemas, or python models, just create a html form, and that is it.  Since I usually follow a design led approach, the html forms are usually done before anything else.

The data is stored in a mini database system I built which I detailed in the blog post "sqlitepickle SQLite Vs pickle".  It lets you use a dict like interface to save python objects.  It's basically a very simple key value store built on python stdlib modules.

We're using JavaScript on the server side as I detailed in a blog post "server side jQuery" to generate the html.  The nice thing is I can use jQuery plugins on either the client side or server side depending on the need.  Including form validation, and other goodies.  I see it as a replacement for a templating language.

This app as it stands doesn't need much of a CMS, but future apps we have in development require a beast of a system in order to process the vast amount of data - and squeeze it down onto various different formats ready for distribution on multiple channels.

Well it's out there now.  So hopefully it'll help some fellow vintage treasure hunters find some good stuff in Paris.

This is part of a series of blog posts on "my journey getting published on various app stores".

Friday, October 28, 2011

Halloween pumpkin juggling published to intel appup

So it looks like the little Halloween game we made has been published on the intel appup store.

It is listed here:

Friday, October 21, 2011

Getting into the Samsung store

So, I registered for the Samsung app store 'thingy' the other day.

Samsung have a number of platforms themselves, not just one platform.  Including Bada, Android, Tizen, and windows mobile 7.  They have tablets, TVs, laptops, mobile phones... and probably other things too (maybe internet fridges?).

It seems most of the platforms support web apps of some sort.  Either through wrapping with phonegap (Bada, Android, win7) or just accepting an archive of the files.

Once joined you can distribute free apps through there.  Which is quite lovely really.  They have their own advertising program, and I'm not sure if you're allowed to do In App purchases with them.

It seems for TV, you can only distribute to the US.  Not to other places around the world.  No idea when or if that is going to be possible.  Yes, their SDK does include support for 3D content.  After speaking briefly with someone(I didn't get their name) at a JavaScript meetup I found out you can try out apps on your own TV/DVD player without having to release to the store.

In order to sell applications, you have to submit a bunch of company paperwork - and sign over your first born child*.  They don't currently charge you join though, which is nice compared to some other programs.

* may be a joke.

Thursday, October 20, 2011

Android developer store, problem. No problem.

So, I've got my app working on Android.  It's not even all that slow.  Ok, it is a little slow.  However the new webkit upgrade in Android 4.0 should get it going fast.  Note that Android 4.0 has approximately 0% of the android users on it at the moment - since they haven't really released it yet.

I had to get touch events working with gamejs, which wasn't all that hard.  I had a little chat on the mailing list, about it, then implemented it not long after.  I will send a pull request soon to that project soon.

I'd like to do a phonegap implementation of sound for shitsound too, but haven't got around to that.  This should get around the html5 Audio issues on the android browsers.  But the game is pretty much ready.

So now that it's ready, I went to sign up to the Android market place so that I could try and release the application.  I went through the signup process, and paid my $25USD.  I even got sent a receipt.  But then something weird happened.  I followed the link, to try and login... but it just sent me back to the signup page again.

The help suggested I try all sorts of things, including just signing up again with another credit card... and if all that fails finally to send them a support request.  So now I'm waiting on the famous Google Support to get back to me to fix up my account... or so I thought.

I tried one more time, but this time with another browser.  Somehow it had gotten my personal account mixed up with my business account.  So I cancelled the registration that got mixed up, and then reapplied again after cleaning cache and clearing cookies.

So anyway... long story short, my Android Market account seems to work now.

I linked up my google checkout account to it - the same one I used for the google chrome web store.  Then it told me I can now sell apps on the android store.

So now I have to actually try and release this game, and get it through the validation process.  Luckily I heard the android validation process is really quick.

Monday, October 17, 2011

notes on chome, and android.

One thing I forgot to mention was that my flash sound for chrome did not work after uploading to the chrome web store.  I made the change at the last minute to use flash for sound, because chrome sound was bad for me.  Of course, a change made at the last minute will be a bug!  Luckily the bug was reported to me, and I fixed it quickly.  This made me glad that the chrome webstore allowed very fast updating.  I can deploy updates there as quickly as uploading to a website.  It also made me remember the importance of testing from the users perspective.  Always check you changes on the live version that people use, rather than just on development servers or simulators.  You need to get into the users shoes and experience it the same way they do.

I did a quick port to android, and tested it on my Advent Vega tablet with android 2.2.  Surprisingly the game ran fairly well.  There were a couple of modern javascript features I was using that did not work, but I quickly found those bugs (html5 data attributes was the main one).

I also needed to add touch support to the game, since so far it only works with mouse and keyboard.  You can play it with the mouse emulation that the browsers support by tapping on the screen to move the player around.  However it would be much nicer if it worked by dragging your finger around.

I have an android environment setup from some previous projects, so I'll be doing that port next.  I'm only going to release it for very modern androids that have a fast enough browser in it.  Canvas and javascript have changed so much in the last year, and last years android just isn't good enough (neither is todays really... we wait for android 4 for decent graphics and audio support).  The other point in favour of releasing on android next is that publishing to the android market is a very easy thing to do.

I'm not sure about doing an iphone port yet, since Halloween is 13 days away - and I am doubtful I could get it through their store acceptance process in time.  But I'll make the decision on which store goes next after the android port is done.

I decided to up the $ amount for the Ludumdare October challenge, since the game has made over $1 now.  So my next personal challenge amount is now $1000 in October.

beta testing intel appup apps

So, I've been going around in circles the last few days with the validation our app on the intel appup store.

The support people escalated my request, and it looks like they have fixed a problem with the validation program.  It looks like their automated validation program had a bug.

In the mean time I've gone through the beta testing process.  This is where you can send your app to some beta testers and ask them to test for you.  If they already have the intel appup installed on their computer, your app appears in their "My Apps" section.  Then you can click on it, to try and download+install.

Just like the Mac OSX Application stores, if the download fails halfway through the system fails.  This is 2011 mind you, so you'd hope that people had figured out that internet downloads could possibly fail, and might need resuming.  Especially on mobile netbooks which are using unreliable 3g wireless internet connections.

As far as I can tell, there is no automated feedback collecting system in place.  Which means you will need to manage that feedback yourself.  Either by emailing your beta testers or using your own in game beta feedback forms.  If you're going to be releasing on multiple platforms, you'll want to create your own beta testing process anyway.  Since not all platforms make feedback easy, or possible at all.

It took me quite a while to actually download anything on their store.  Even small programs took a while.  I guess they are having connectivity issues with their download servers.  A 0.42MB app took over 10 minutes and multiple tries to download successfully.

The apps the html5 encapsulator by intel creates all include their own web browser.  Which means that every app is at least 10MB big, not even including your app data.  This is compared to other platforms, you mostly only need to include your actual data, or have a thin wrapper - like phonegap - which doesn't take up too much space.

Anyway... fingers crossed it passes their validation process soon.

Friday, October 14, 2011

'Halloween Juggling Pumpkin' published to the chrome web store... I think.

Well, I clicked the publish button, and it seems the 'Halloween Juggling Pumpkin' app has been published to the chrome web store.  I'm quite happy that it is up!

Here is a screen shot of the menu screen.

Well, that wasn't too hard to do after all.

Here is a screenshot of some game play.

Here is the lovely store URL it gave me for the store:

This was way less work compared to the intel appup store, which I'm still having issues publishing it. To be fair, their store is still in beta for html5 apps.

encapsulator app 'binary validation failed' no other details.

Just got a rejection notice for my intel appup app, with 'binary validation failed'.  There were no other details, which seems to be typical for this program.

I've sent them some messages for more information, so I'm expecting a few days of waiting before anything happens.

chrome audio sucks

Chrome Audio is worse than firefox, safari, and internet explorer Audio.

There are cracks, pops, and sounds cut short.  It seems there's also not many sounds that can be played at once.

Luckily I can use jquery.shitsound and soundmanager2 to prefer using flash audio over a javascript bridge.  This has a slight delay compared to firefox/safari html5 Audio, but is better than the built in chrome Audio.

Thursday, October 13, 2011

export scripts for app

So today I've been writing a couple of export scripts for my October challenge apps.

I set the config in the main html file using html5 data attributes.  Then read that config on app startup.  This seems like a fairly easy way to customise the app depending on platform.  My export script selects the correct html for each platform on export.

I have to create the correct data sets for each platform.  The right sound formats (ogg, mp3, wav), and the right sized images (From 1080 HD to 320x200 screen).  As well as select the correct fonts.  I can't just include all the assets for each platform, since there are size restrictions.  As well the source, full sized raw data can be over 5GB for one app (for my content heavy app).

Then I have to package them up depending on platform, with manifest files and other strangeness.  Or I need to zip them up in a zip archive.

I'm not completely automating the process yet.  I'll wait to see if the apps get any traction before investing time in complete automation.  Completely automating releases is essential for supporting lots of platforms.  Without it, your whole time can be spent releasing, rather than making the apps better.

Wednesday, October 12, 2011

Signed my app with certificate, and submitted for validation again.

So, I managed to sign my app and submit it for validation on the appup store.
It took at bit of messing around, converting certificates, and installing lots of microsoft software, but eventually got my msi signed, and submitted to the store.
That made me happy.  Now I’m off to drink wine!

Intel appup progress. Got certificate, looking forward to webkit upgrade.

Well, some good news on the intel appup front.
I finally managed to get my certificate, so I should be able to now sign my apps to upload.  Unfortunately I used firefox to download the certificate, which means that I need to convert files in order to get into the microsoft fomat needed for code signing.  So I’d recommend anyone else use IE8 to save that hassle.
As well, I’ve been reporting my problems with the webkit they use in their forums.  They just gotten back to me, and say they hope to use the latest webkit (which should include a year of webkit bug fixes and feature additions).
So now, I’m going back to trying to submit my app again.  I hope that it gets through!

Tuesday, October 11, 2011

chrome web store

We’ve started to look into distributing our app via the chrome web store.
I’ve signed up for a developer account, and started reading the documentation on how to publish apps.  So far it has been easy.  All of their forms have worked, and so far the documentation is easy.  This time I’m trying to do as much of the administration up front.  Things like validating my account and setting up payments.  Only after that works will I actually attempt to publish my app.
Let’s hope it’s easier than the intel appup program, which I’m still not able to publish my app through.

Monday, October 10, 2011

Applied for certificate, thinking about what to do next?

I got an email back from Comodo about the code signing certificate, and now their form works.  Then I had to submit my company certificate, and then have to wait for up to 2 days to get the certificate.
I probably should have realised I needed this certificate before, and applied earlier.  Oh well.  For some reason I didn’t remember this was needed.
Next I’ll begin packaging it up for the google web chrome store, so maybe we can sell some on there too.  I don’t think that will take all that long to do.
Finishing the iOS port might be hard since I’ve noticed the performance of JavaScript and canvas on old iOS devices is terrible.  I could just make it available for iOS 5, and the latest iphone 4 phones though… they might be fast enough.  I’ve been exploring possible optimizations for it too.  So I’ll see how that goes.
I’m in the same situation with the android port.  The current web browsers on there are fairly garbage for canvas performance, so I’d need to do some performance work before it would work on there.  Making it available on the latest versions of Android is also one possibility.  I know an android application is likely to be accepted faster than an iOS one, so that might sway me to release there first.
A Mac application is also possible I guess.  For that I’d need to figure out if there is a UIWebKit wrapper that is easy to use, and quick to integrate.
The other option is to finish the other intel appup app I have in development.  But porting this game to other platforms seems easier I think.  Once the whole process is done I’ll have an idea of how long the other app would take to do for the appup platform.

Saturday, October 08, 2011

Battling Intel Appup submission

Well, I managed to prepare my app for submission to the intel appup store.
I first had to figure out bugs: Sound doesn’t work. Also fonts don’t work. This was very disappointing considering that I’ve been working with a musician on sound effects and music. So we’re not able to use any of that work we did until they fix sound. I even tried to use a flash based sound player, and use the embed tag. None of it worked.
Also the lovely font we selected does not work. Trying to use a font-face crashed the browser completely. So instead I had to use ‘Arial’. arg.
I’m not sure what sort of html5 games and apps they expect if fonts and sounds aren’t working?
It seems they are using a 1 year old webkit implementation, so I hope they will upgrade their encapsulator soon. One year in web browser years is about 70 human years.
Anyway, we decided to submit the app anyway. The font makes it look a bit ugly, and without sounds it looses a lot – but at least the game play works.
Then I filled out a few pages of forms, prepared images, wrote copy, and did all sorts of guff so that I could upload the app. For some reason they required silverlight or google gears to upload the file (weird). Not sure why they couldn’t use a normal html upload form, or even a flash upload form.
Anyway… all that got done until I had to apply for a security certificate to sign the files. I fill out another form of my personal details, press submit and I’m greeted with a great error message: “An error has occurred. -1″
Oh well. Sent off an email for support, and now I wait.
Finally blocked after a couple of long days and nights – stopped by the error “-1″.

Friday, October 07, 2011

EULA we can use? (End User License Agreement)

got my app up to the packaging stage now…
Does anyone know of a decent (End User License Agreement) EULA we can use for our games?  If you know of ones we can use please leave a comment.

Thursday, October 06, 2011

I'm in

October came around, and I started working on a game.
It’s javascript/canvas based, instead of my usual pygame. But, I’m using gamejs which uses the pygame API, so I feel pretty much at home.
We got most of our new graphics and sound into the game today, and all of the game play is done.
It’s not running fast enough on my ipod touch 2g, but I think with some work it could be playable. So we’re aiming to release on the various web stores first. Then perhaps improve things for the mobile platforms.
I had to work on improving a sound engine I call jquery.shitsound, or just shitsound for short. Which uses various web browser sound APIs to work. I’ll have to get it using the phonegap framework to play audio too, since html5 Audio on mobile platforms is simply broken.
We’re in, I should say. Since I’m doing a collaboration with Spencer Sternberg ( ) video game audio and music productions.
The second app I’m working on next after this one is released I’m doing as a Powered By Bees production ( ). It’s less game related, and more content related, but I still think it will count for this October Challenge.
Thanks Ludumdare!  This October competition has really given me a big kick up the backside.

Sunday, October 02, 2011

Server Side jQuery - ssjquery

I've got a new server side jQuery project up.  It's an extension on some ideas and projects over the past couple of years.  It uses the node.js javascript interpreter, and node libraries like jsdom, and htmlparser to do all the work.  It can be used within various projects that are not javascript via a command line process or a web service.

This is what the pipeline looks like on the server side.
json + html + server side jquery => rendered html

Instead you can run it on the client side so you don't need a server for development.
json + html + ssjquery run on client first before other scripts => rendered html

A webservice creates the json, and your server side jQuery scripts populate the html.  Think of it as an industrial strength templating language that millions of front end developers know how to use.

Here are my server side jQuery slides I gave at the reject.js conference in Berlin last week.  It wasn't a great talk, but I think it's helped me understand a little more on how to explain the concept to people.  I purposely did not put any code in that talk, so I'm providing it here.  Instead I concentrated on why you'd want to do it.  I might try and write up my talk notes as a separate blog post.

Here is the project page:

There is a python buildout script which downloads node.js, and all required libraries into a directory ready to run (with the npm package manager).  In there it creates a bin directory for the bin/ssjquery script for the command line version and bin/ssjqueryServer for the server version.

As a bonus, I also install the uglifyjs binaries, and coffee script binaries - but they are not required.

Here is how you run the python buildout...

$ hg clone
$ cd ssjquery/pythonbuildout
$ python bootstrap -d
$ ./bin/buildout
$ cd ..

Here is how the command line script ssjquery is called to output rendered html.

$ cd js/example/ 
$ ../../pythonbuildout/bin/ssjquery ./ server.json index.html jquery.js yourServerSide.js yourServerSide2.js > renderedHtml.html

Here is the ssjqueryServer script which listens as a webservice to render html.

$ cd js/example/ 
$ ../../pythonbuildout/bin/ssjqueryServer ./ server.json index.html jquery.js yourServerSide.js yourServerSide2.js 
Server running at try url: http://localhost:8124/?basePath=./&serverJson=server.json&serverJs=jquery.js,,,yourServerSide.js,,,yourServerSide2.js&templateUrl=index.html

That runs a web server which renders the files for you.  It's a bit quicker than calling a process each time you want to do things.  There is a possibility of embedding the javascript interpreter inside python (or php/java/etc) too.

Structure of html

The structure of your html is something like this:
  • include jquery
  • include code which will be used on server side and client side.
  • include server side jquery, and run it if have not processed it server side.
  • include client side jquery
There is a minimal index.html example here if you want to see what it looks like.

This lets you use the same html in either server side processing mode, or client side processing mode and the results should be the same.

Tuesday, September 27, 2011

pygame on raspberrypi, pyconuk was awesome

Thanks to @ntoll I had a chance to play around with the raspberrypi alpha board at the pyconuk conference (which was a great little community conference with a big heart).
That is it on the pink bubble wrap.  It's hooked up to a monitor via the digital HDMI video port, with a keyboard and mouse connected up via USB.

We only had a few hours to play around with it, but managed to try a few things out.  Whilst we were there playing around with it, a whole bunch of people popped there heads in to have a look and ask questions.  Lots of enthusiasm for the project, and also the machine, but also a bit of cautiousness about the project.

In another room, there was a talk going on about how the BBC was going to help bring a programming education project into the schools of the UK again - like they did with the bbc micro.  Tweets started to fly around on the tweet projection screen about it, but then at the end of the talk we found out it was a hoax.  Haha.  It was more a "wouldn't it be awesome if..." kind of talk.

So, what did we find out in our exploration of the raspberrypi?  Firstly, the rasberrypi had a debian linux distribution running on it, and many packages were available to install.  Including an old version of pygame.  Sound wasn't working for us, there were none of the various sound systems running.  It turns out the analog output port is currently really terrible.  With a 1bit DAC, I guess it will sound something like a PC speaker or maybe slightly better.  However, digital audio is going to come from the HDMI port along with the video.  So that should sound really good, assuming it is hooked up to a TV with HDMI.  There's also the option of using a $3 USB sound card - which the project reccommends if you want audio input (eg, a microphone).

One interesting thing is they have not put a real time clock on there.  So when you reboot the system it has the wrong date all the time, which really messes with things like the packaging system, and the file system.  However, once connected to the network it can grab the time from there.  You may be noticing a theme... they've cut back on components to keep the cost low.  I think that it's a great approach in order to get something useful at the price point they are aiming for.

The alpha board had no WIFI hardware, but we found an ethernet cable and plugged it into my laptop, so we were able to share the laptops WIFI internet connection through it.  @ntoll had the pygame package already installed, but it was an old version that didn't include the example programs.

@ntoll has previously blogged about his raspberrypi

So we set the debian package manager the task of installing the gcc compiler, and various other things needed to compile pygame from source(SDL, ffmpeg, portmidi, etc).  It was downloading packages over the network connection at about 1MB/s which is pretty fast for a 10/100 ethernet port (which seems to be attached via USB), and then out over my laptops wireless connection.  It did take a while to install the packages on the SD card though.  Perhaps a faster SD card could be used, or maybe the SD card driver could do with some speed improvements.  However, it wasn't too slow, definitely usable.

All the dependencies installed with no issues... winning!  Then we started up the compiler, and we got pygame to compile with gcc after a fair wait.  This was quite impressive, compiling stuff on the raspberrypi itself, rather than having to use a cross compiler.

So X was running, but so far there was no hardware acceleration of graphics.  I guess in time, someone will be working on the video drivers.  This little machine is designed around a GPU, so it is definitely capable of some fast video.  So web browsing with iceweasle (rebranded firefox) was a bit slow.  It was pretty much unusable whilst pygame was compiling, but usable when the machine wasn't doing much else.

@ntoll got his xmas card pygame program he wrote for the london dojo last year running on the raspberrypi.  It was going a bit slow, but after a few tweaks it ran at an acceptable frame rate (if not exactly smooth).  So this is good news, because even though the graphics drivers are not hardware accelerated yet, the little machine can run simple programs at an acceptable speed.

I also tried running a very CPU and graphics demanding pygame game zanthor.  But it wasn't able to run above 1fps.  The zanthor game has trouble getting 30FPS on a 5 year old laptop with good graphics drivers, so this wasn't unexpected on an alpha board that doesn't even have its hardware acceleration available yet.

We also ran the pygame test suite, and most everything passed except for the sound related tests (and things like the camera, and joystick tests, since it doesn't have a camera or joystick attached).  Apparently there are pins that could be used to attach a camera, since this is a mobile phone chip.  There's a lot of work that could be done to pygame to make it run better on the raspberrypi, but I'm happy that it's mostly working already.

I also tried running an OpenGL demo, but it ran really slowly.  So I'm guessing that hardware accelerated OpenGL was also not working on the distribution that it came with.  But they have shown it working in other demos, so it is only time before it is working in X.  Also, I think the board supports OpenGL ES 2.0 which there is not currently a python wrapper for (please leave a comment if you know of a project for this).

There seemed to be about 189MB of ram on this board according to 'top'.  I guess it was a 256MB ram model that had 60ish MB of ram taken up by the GPU.

@tjguk was there too hacking on his quiz program.  It was interesting listening to him talk about his program, and what he was doing with it.  But I won't go into it here, since he might want to blog about it himself.

Also at the conference was Garry Bulmer who was talking about the ARM board that he has made.  Previously he has been using the arduino micro controller platform as a robotics teaching platform for kids and adults.    I'd like to follow his project, and look forward to seeing it progress.  He talked about wanting more processing power to do things like process cd quality audio, which the arduino can not really do.  His arm cpu board was aimed at also teaching hardware related things, so his board has more usable GPIO inputs and output pins for controlling electronics.  Whereas the raspberrypi board is not currently designed to work well for arduino uses.  It is good to see competition in the cheap educational ARM board space, but I think the two projects have slightly different use cases.

One concern some people had about the raspberrypi project is there seems to be a link between the broadcom employee and the chips they use on the raspberrypi board.  This could be seen as a conflict of interest.  However, maybe it doesn't matter.  Perhaps it is even a good thing!  Since then there might be better support from the company because an employee might have better access to docs, and such things.  At least there will be someone on the project who has access to the tools needed to fix bugs.  There were also concerns about the lack of source code for the chips 18MB binary blob firmware.  What does that code do really?  I think the project wants to open that source code, but so far it is not available because the company does not want to open it.  As a pragmatic option, I still think the raspberrypi is ok.  It's not completely open hardware, but it's open enough to be useful for an educational project for creating software.  The projects goals are to provide a cheap PC to children, so they can use it themselves to learn - it is not about creating an open hardware platform.

I'd love to get a development board, but the raspberrypi foundation only made 50 and they are all distributed to developers so I'll have to wait like everyone else until they are up for general sale.  I think they are aiming to make them available in a few months.

There were plenty of other interesting things going on at pyconuk this year too.  Including talks about testing(mocks, extending unittest and such things), a realtime web socket game (kwizzle) that uses tornado and mongodb, fluiddb (kinda wikipedia for databases), a visual programming environment (The Larch Environment), a django work shop, and the always good 5 minute lightning talks - where random people get up and talk about something for 5 minutes.  Amongst plenty of other things going on.  The volunteers put on a great conference again, with very little resources.  I wish I could have talked with more people at more length, but there's not really enough time to do so.  There's always a creative buzz at these things.  So many creative, passionate people talking about their projects in one spot is quite inspirational.

Wednesday, September 21, 2011

jquery.shitsound.js - a shit sound player for web browsers

Yesterday, I made the first release of shitsound.
jquery.shitsound.js is a shit sound player for web browsers.
Used like this:
$.snd.init({}, function () {
        // we are ready to play sounds!  Brilliant.
shitsound only does a few things with sound::
play, stop, stopAll
jquery.awesound.js might do:
changing volume, pitch, playing with a different tempo, playing in stereo, 3d sound, etc.
jquery.shitsound can not do any of this fancy stuff. It can just play sounds, and stop playing them.
shitsound detects which implementation to use at init time.
  • html5, using the html5 audio tag.
  • embed, for using the embed tag.
  • soundmanager1, for old flash 7.
  • soundmanager2, for modern flash (8,9,10+).
  • empty, pretend implementation that does nothing.
It currently requires jquery, since I use jquery. However it could quite easily drop the jquery dependency with some work. Other dependencies are soundmanager 1 and soundmanager 2, as well as (optionally) swfobject. These are all bundled in ready to use though.
Each implementation is kept separate in the code base, and it is fairly easy to extend with other implementation. The html5 implementation is separate from the soundmanager1 implementation for example. I plan on implementing a phonegap implementation in the future.
If you'd like to see another JavaScript audio engine supported, please let me know.

Yes, it does support netscape navigator 3+. As well as old versions of internet explorer that don't have flash installed, or have old versions of flash installed.

preparing your audio

Because sound support on the web is shit, you need to prepare your audio in various different formats for it to work cross platform.
You need a .wav file, a .mp3 file, and an .ogg file to support all platforms.
It might not be a good idea to include big .wav files for large amounts of audio like songs though. Because they can be 10 times larger than a lossily compressed mp3 or ogg file. But if you want full browser support, then .wav files are required.
The ffmpeg is quite a common powerhouse of a multimedia tool. It is available on OSX through brew, or macports. It can even convert audio for you.
  • ffmpeg -i file.wav file.mp3
  • ffmpeg -i file.wav file.ogg

preparing your webserver

If you use nginx you might need to modify your /etc/nginx/mime.types file for oggs::
audio/ogg ogg;
Other web servers may also need mime types being added.


Maybe these things will be done in the future.
  • volume control. This is possible with all backends.
  • Better browser detection, and blacklisting of bad html5 audio engines.
  • wider testing and debugging.

Wednesday, September 14, 2011

pygame sprint at pyconuk, and virtually. Friday 23rd to Sunday 25th of September

There is going to be a pygame sprint at this years pyconuk. We're going to tackle whatever the participants who turn up want to hack on. That might be porting to a new platform like the $25 rasberrypi computer(there's going to be a couple at pyconuk), or improving the pygame android support. Or it might be polishing off one of the new modules like the new freetype font module, or perhaps implementing a brand new module.  Or maybe it's jazzing up the documentation, or working on a new import pygame.examples. I'm getting there on Friday, so I hope to do some hacking then, and we might be able to get some sprint space for the Friday(conference propper starts Saturday, but some people are getting there early) - otherwise it will have to be in a cafe or something.
Also going on during the conference is the python core sprint and CKAN sprint.  It should be fun to join in on those sprints too.  Of course there will be talks, work shops, an unconference room, lightning talks (5 minute talks) and coffee fueled hallway conversations.  There's even going to be a Code Dojo - london style, and of course - most importantly: lunch/dinner.

pygame recently migrated to mercurial and bitbucket:  We'll be able to commit changes to there or work off branches and submit pull requests later.  There is a pygame hacking guide which we will make notes on any questions people have with working on the pygame code base.

It will also run as a virtual sprint in the irc chat room (freenode #pygame), so if anyone wants to join in remotely they can.  If you only write python code, there are plenty of parts of pygame that can be worked on.  For anyone who turns up I can show them the basics of CPython modules if they already know C.

We'll try and tweet sprint goings on via the @pygame_org twitter account.  Hope to see you there.

Wednesday, August 24, 2011

Let's make a shit JavaScript interpreter! Part Three.

Let's put the Research into R&D. I guess it should be called Shit Research to go along with the name of this series (p1) of articles (p2) I started one year ago. It takes a lot of time to read over hundreds of thousands of lines of undocumented C++ and java code, so part three took much longer than expected.

Let's explore existing JavaScript implementations.

In this part we are going to take a small digression to look over other JavaScript implementations. I'll provide a short description of how each JavaScript implementation is made. We can use the Architectural knowledge from each implementation to inspire our own implementation.

Another benefit of researching each implementation is that we can find all the different tools they use. For example different test suites.

Make sure to read the URL's for each implementation to find out more information about each one.

There is a list of ECMAScript implementations at wikipedia. We will not cover all of the ones listed there. If like me, you may spend a few hours or even days reading through the links from there.

narcissus - js in js.

Narcissus is a javascript implementation written in javascript (with some SpiderMonkey language extensions). It is written by the same author as the original JavaScript implementation back in 2005-2007 (Brendan Eich). In mid 2010 the Narcissus code has been taken up again by the Mozilla project to make researching JavaScript changes easier.

It is a good implementation to study, since it is fairly small, and is meant to be easy enough to read.

It uses a hand written parser, not a generated one.

The original narcissus source repository.

The new repository of narcissus, and two articles about it.

There is a port to python of the narcissus parser called pynarcissus. The authors of pynarcissus found it difficult to port the rest of the interpreter since it relies on JavaScript features itself.

It weighs in at about 7000 lines of code counted with wc -l. I had to count lines in this way since my SLOC counter does not seem to count javascript.

Spider monkey.

Spider Monkey is the original JavaScript implementation used by Netscape, and the Mozilla project.

SpiderMonkey is a production grade, high quality JavaScript implementation.

It also has excellent documentation. Especially the js/src/README.html file which includes a design walk through.

"""The compiler consists of a recursive-descent parser and a random-logic rather than table-driven lexical scanner. Semantic and lexical feedback are used to disambiguate hard cases such as missing semicolons, assignable expressions ("lvalues" in C parlance), etc. The parser generates bytecode as it parses, using fixup lists for downward branches and code buffering and rewriting for exceptional cases such as for loops. It attempts no error recovery. The interpreter executes the bytecode of top-level scripts, and calls itself indirectly to interpret function bodies (which are also scripts). All state associated with an interpreter instance is passed through formal parameters to the interpreter entry point; most implicit state is collected in a type named JSContext. Therefore, all API and almost all other functions in JSRef take a JSContext pointer as their first argument.

The decompiler translates postfix bytecode into infix source by consulting a separate byte-sized code, called source notes, to disambiguate bytecodes that result from more than one grammatical production.

Google V8.

Written in C++, javascript and assembler. Uses hidden classes, and generates machine code at run time.

The parser is hand written parser in C++. It's not generated.

'Preparses', which creates tokens. Then creates an AST by parsing. Finally compiling the AST. parser.css is where all the parsing happens.

The projects documentation is quite limited - so reading the source is the best way to get in there. There are some videos which describe the architecture, and engineering decisions behind the choices taken.

V8 weighs in at a mighty 605,962 SLOC. Making it the largest, biggest, badest V8 JavaScript engine around.


JSLint is written in JavaScript and uses a TDOP approach to the parser like we are using. I won't discuss this, since it is documented well elsewhere.


Rhino can run as either an interpreter or a java byte code generator. It's hosted by the mozilla project (who make firefox). It complements their C++ implementation (spidermonkey) and their JavaScript implementation (narcissus).

There are a few things which show it is a very mature as well as modern implementation. The project was started in 1999, and has been developed ever since. It has partial support for JavaScript 1.8, and ECMAScript 5 standards. Weighing in at 50,551 source lines of Java code (SLOC), it can be considered a very large code base. Another modern feature is that it supports the CommonJS javascript module standard. Despite still being hosted in cvs, it's still being maintained, and features are being added regularly.

The Rhino JavaScript team maintains a library of tests and other documentation for JavaScript. Tests are being shared with the other JavaScript implementations within the mozilla spidermonkey project. There have also been parts which have been ported from spidermonkey - such as the hand written parser.

A handwritten scanner they call TokenStream creates tokens, and then parses those into an AST. Despite being mostly hand written, some parts are generated. Specifically the stringToKeyword method, which detects keywords is generated somehow.

The documentation of the architecture of the project is limited. There is however some API documentation. With a couple of modifications to some ant build files I was able to build it, as well as even make a few small modifications.

The wikipedia Rhino page has some great information on the rhino javascript engine.


KJS is the KDE JavaScript implementation for the konqueror browser. It was the parent of the JavaScript implementations done by Apple Computer, inc. I won't go into any detail on this one, since I'll cover JavaScriptCore instead. KJS is written in C++, for the QT library.

JavaScriptCore, Squirrelfish, Nitro

You can browse the source here:

This uses a hand written lexer(tokeniser), and a hand written parser. The code structure of the parser and lexer looked eerily familiar. The code base is mostly written in C++ and is quite massive. 140,837 SLOC

There is lots of platform specific code, but it also has a jit, uses byte code, and an interpreter. There is also lots of development code in there for things like debuggers, and profilers.

Closed source JavaScript implementations

There are a few JavaScript implementations that are closed source. The two main ones in widespread use are the ones from Microsoft, and the ones from Opera.

They have however published papers and blog posts about their implementations. I won't cover them any more, because not as much can be learned without the source code.

pypy javascript

The pypy project started a javascript interpreter now too.

The description of the project mentions it's currently using the spider monkey parser, but it appears to generate one using a parser generator provided by pypy. Using a EBNF grammar file. It also creates an AST.

It works for some simple javascript programs that don't use the javascript standard library. I'm not sure of the future of the project, since it appears it was a GSOC project which has now finished, so there might not be any full time developers left on it.

It's written in RPython (a restricted subset of python) and python. Running on top of pypy, it should theoretically be able to take advantage of that platforms jit and garbage collector.

This project makes use of some JavaScript tests and benchmarks from other projects. Specifically some benchmarks from v8, and the language shootout website. It also includes the "ECMA 262 Edition 1" tests.

It weighs in at 5452 Source Lines of Code (SLOC). Which is much smaller, but the implementation is also not complete, so that is to be expected.

So what have we learned then?

We see that most of the implementations use a hand written parser. We also see that the implementations in js and python are much smaller. So despite them being incomplete, I think it proves that it should be feasible to make our shit interpreter in python. We don't need half a million lines of C++ to do our project.

We have also learned that there are test suites available, which should help us out a lot. In fact, many of the implementations share the test suites. Having a test suite already available makes it way easier to write an implementation of something yourself. It acts as a guide to development, and also reduces the time for testing since a lot of it can be automated.

Exercise for next time

Choose One(1) of the implementations, build it, run it, and modify it slightly to do something different. Try and run the tests that come with it.

Further reading.

This whole article is "further reading", but we can never have too much to read. Can we!?

This time, instead of reading it on the train or in the bath tub - may I suggest reading these on a couch?

Sunday, August 21, 2011

bugzilla bugs to bitbucket issues

As part of the pygame move to bitbucket, we are converting a bug database stored in bugzilla.

Work is progressing on the issues migration:

Bugzilla is more comprehensive compared to bitbucket. So not all things can be migrated from the bugzilla bugs to the bitbucket issues.

Also, the bitbucket API is limited in what it can do. You can't add user comments or add attachments.

To get around the attachments part, I made the migration script create a directory of attachments, and then link to them in the bug text. Then I've extracted the attachments from bugzilla and put them online.

To hack around the lack of comment adding in the bitbucket API, I'm just combining all of the comments into the content text.

Other features bitbucket issues API does not have are priorities, and platforms. The issues features _does_ support priorities, but it doesn't look like the API is set up yet. You can't add platforms (like 'Windows XP SP2') at all it seems. You also can not add free form tags of any sort. Another couple of basic things you can not set with bitbucket issues are the date, and the bug id.

So there you have it. This code worked ok for us.

This code might be a useful start for anyone else wanting to start on a migration. Feel free to go fork it yourself:

Sunday, July 17, 2011

bye bye launchpad

I'm starting to move most of my projects off launchpad onto bitbucket.

launchpad just hasn't gone in the direction that is useful for me, or my projects.

It still has some nice features, such as packaging farms (PPAs) and of course bzr integration. However, github and bitbucket have just been moving more towards making it easy for developers compared to launchpad which is more geared towards for distributors who work with an upstream. Last year I hoped launchpad would improve after it went open source, and it has improved. Just not really in the direction useful for my projects.

Many people these days are able to use hg, or git... but still very many people have never even heard of bzr. I feel bzr+launchpad was holding one project back a little from gaining collaborators because of this. By that logic, I should be using svn for it... but I don't want to do that :)

The documentation to convert repositories to mecurial is on the hg wiki. The conversion seems to mostly have worked ok for me on my python newsletter project. Except it did not put the correct email address attached to each of the commits. Oh well.

Monday, July 04, 2011 opengl shading language toy shader

A couple of months ago I made a python port of the WebGL shadertoy. The shadertoy website is written in JavaScript and the opengl shading language.

Here is the python port:

Maybe it'll be useful for someone learning shading languages through python. Or not.

It's a fun demo anyway. Press the 'Anykey' to change between shader. You'll find the 'Anykey' on your keyboard. If the Anykey you press is Escape, then the program will 'quit'.

Requires 1 tsp of pygame, and a cup of pyopengl.

Saturday, July 02, 2011

cherrypy daemontools

Cherrypy and daemontools work nicely together. However, I have not found any instructions on how to use them together on the interwebs... so I've written up some basic notes for people who love cherrypy and daemontools.

daemontools is a collection of tools for managing UNIX services. It can be used to automatically start, and restart processes on a unix system.

The instructions to run services in daemontools are here: I'm assuming you have daemontools set up already.

Cherrypy is a python library for making website applications. There are many ways to deploy cherrypy apps, but I will describe how to deploy cherrypy apps with daemontools.

Firstly, you do not need the cherryd tool to use with daemontools. Daemontools does the daemonising for you. But cherryd can be used if you want, since it provides some nice options (just don't use the -d daemonize flag). You may want to use cherryd with daemontools if you'd like to use FastCGI or SCGI. I won't cover cherryd any further in this post.

I'll now show you how to setup daemontools for an example cherrypy application.

Here is a normal helloworld cherrypy app... which I'll put in the file
import cherrypy

class HelloWorld(object):
def index(self):
return "Hello World!" = True


This runs on on port 8080 by default. It also runs in development mode (which you should not deploy with, but only use for testing).

Now create a 'theservicedir' directory somewhere.
Also create the following directories: 'theservicedir/log' and 'theservicedir/log/main'

Then create a 'theservicedir/run' file.
exec setuidgid rene /usr/bin/python /home/rene/cherrypydaemontools/

Note that I specified full paths to python and to your python script. Also see that I got the app to run with the setuidgid program as the user 'rene'.

Then create a 'theservicedir/run' file for logging.
exec setuidgid rene multilog t ./main

Now to start it up we symlink the file into the service directory.
sudo ln -s /home/rene/cherrypydaemontools/theservicedir /service/theservicedir

To stop it:
svc -d /service/theservicedir

To restart it:
svc -t /service/theservicedir

To start it:
svc -u /service/theservicedir

Your app should start up the next time your server reboots. Happy cherrypy and daemontools usage. Please leave any related tips or fixes to these instructions in the comments.

Saturday, June 18, 2011


Was playing around with jquery mobile the other day, which has planned it's first beta release sometime next week.

jQuery mobile is what jquery+jquery-ui are to desktop browsers. It abstracts a lot of the parts away so that you can write your code without having to figure out all the different mobile browser quirks yourself. It offers a structure to make web apps which are cross platform across current mobile devices and which as a bonus work in desktop web browsers as well.

Mobile style events are provided as well. Like touch and swipe which work across the various mobile browsers is a nice touch.

Another great thing it provides is media queries. CSS media queries are buggy on some browsers so this abstracts out a lot of the bugs on the various different devices.

Supported are most of the mobile devices. Some are supported better than others, and some provide certain features that others do not provide. For example, the ajax style page loading is not done on opera mini. But works on android, and safari mobile browsers.

I'm quite looking forward to the feature complete beta release. It's been a long time baking, and smells quite nice.

Tuesday, May 17, 2011

Running jslint on saving JavaScript with vim

autocmd BufWritePost,FileWritePost *.js !jslint <afile>
This will run jslint on your JavaScript files after you save them.

You can add the command to your ~/.vimrc

Friday, April 29, 2011

Hotmail, xbox and microsoft live have been hacked.

update: hotmail have added a my friend has been hacked button, so people can report when one of their friends accounts have been hacked. Also I got my account back after 5 days or so. I don't think any of my contacts lost any money from the scam. Other people who got their accounts hijacked have not been so lucky.

Many people have been having their hotmail accounts been broken into and stolen.

Microsoft writes about it on their security blog.

I've meet a few people over the last week who know of people who have had their accounts stolen.

The fraudsters are sending everyone in their contact list and telling people they have been robbed - and to send money. They say they are in a hotel and the hotel will not let them leave.

A friend told me about how some people call up, and someone answers pretending to be the hotel manager mentioned in the scam email.

I haven't read any media coverage of this, but have heard first hand of people who it has happened to. Could this be related to the Sony break-in the other week?

Unfortunately Microsoft do not seem to answer when contacted about accounts being used for fraud. Their account reset procedure is so slow that by the time an account is recovered the damage could very well be done.

Microsoft are also recommending people create new hotmail accounts rather than go through the verification process to recover a stolen account. Unfortunately this is a rather dangerous attitude to take, since many people have been sent emails and all of the people in the contact list could be victims of crimes.

The xbox live network uses the same authentication as hotmail and other Microsoft online properties and stores. Microsoft uses a single sign on system - so this is a very big security break down.

I have no idea how large the break-in is. However, if I hear about multiple people in real life having their accounts stolen then I think this is MASSIVE.

Sony took their system offline whilst they investigated the hackers. I'm not sure what if anything Microsoft has done.

Please spread the word, and warn others about this break-in.

Example scam email

Here is an example of one of the emails they are sending out:
Subject: My Plight!!! Help

I'm sorry for this odd request because it might get to you too urgent but it's
because of the situation of things right now, I am stuck in United Kingdom .
we were robbed at the park of the hotel where we stayed,all cash,credit card and
cell were stolen off us but luckily for us we still have our passports with us.

We've been to the embassy and the Police here but they're not helping issues at all
and our flight leaves today but we're having problems settling the hotel bills and
the hotel manager won't let us leave until we settle the bills.

I need a quick loan?? promise to refund it back once i get home.

Sunday, April 10, 2011

Let's make a shit JavaScript interpreter! Part Two.

Let's make a shit javascript interpreter! Part two.

As a learning exercise, I've begun writing a JavaScript ECMAScript interpreter in python. It doesn't even really exist yet, and when it does it will run really slowly, and not support all js features.

Homework from part One - A simple tokeniser.

We ended "Let's make a shit JavaScript interpreter! Part One." by setting some homework to create a tokeniser for simple expressions like "1 + 2 * 4". Two readers sent in their versions of the tokenisers (ps, put a link to your home work results from Part One in the comments, and I'll link to it here).

Our simple tokeniser

operators = ['/', '+', '*', '-']
class ParseError(Exception):
def is_operator(s):
return s in operators
def is_number(s):
return s.isdigit()

def tokenise(expression):
pos = 0
for s in expression.split():
t = {}
if is_operator(s):
t['type'] = 'operator'
t['value'] = s
elif is_number(s):
t['type'] = 'number'
t['value'] = float(s)
raise ParseError(s, pos)

t.update({'from': pos, 'to': pos + len(s)})
pos += len(s) + 1
yield t

>>> pprint(list(tokenise("1 + 2 * 4")))
[{'from': 0, 'to': 1, 'type': 'number', 'value': 1.0},
{'from': 2, 'to': 3, 'type': 'operator', 'value': '+'},
{'from': 4, 'to': 5, 'type': 'number', 'value': 2.0},
{'from': 6, 'to': 7, 'type': 'operator', 'value': '*'},
{'from': 8, 'to': 9, 'type': 'number', 'value': 4.0}]

Code for shitjs.

You can follow along with the code for shit js at: After you have installed it, shitjs.part1 is the package for the part1 homework.

What next? Parsing with the tokens of our simple expression.

Since we have some tokens from the input, we can now move onto the parser. Remember that we are not making a parser for all of javascript to start with, we are starting on a simple expressions like "1 + 2 * 4". As mentioned in Part One, we are using an algorithm called "Top Down Operator precedence". Where actions are associated with tokens, and an order of operations. Here you can see the precedence rule (order of operations) with parenthesis around the (1 + 2) addition changes the result.
>>> 1 + 2 * 4
>>> (1 + 2) * 4

A number is supplied for the left, and the right of each token. These numbers are used to figure out which order the operators are applied to each other. So we take our token structure from tokenise() above, and we create some Token objects from them, and depending on their binding powers evaluate them.

What's new is old is new.

The "Top Down Operator precedence" paper is from the 70's. In the 70's lisp programmers loved to use three letter variable names, and therefore the algorithm and the variable names are three letter ones. They also wore flares in the 70's (which are back in this season) and I'm not wearing them, and I'm not using three letter variable names!

Sorry, I digress... So we call 'nud' prefix, and 'led' infix. We also call rbp right_binding_power, and lbp left_binding_power.

nud - prefix
led - infix
rbp - right_binding_power
lbp - left_binding_power

Prefix is to the left, and infix is to the right.

Manually stepping through the algorithm.

Let's manually step through the algorithm for the simple expression "1 + 2 * 4".

>>> pprint(list(tokenise("1 + 2 * 4")))
[{'from': 0, 'to': 1, 'type': 'number', 'value': 1.0},
{'from': 2, 'to': 3, 'type': 'operator', 'value': '+'},
{'from': 4, 'to': 5, 'type': 'number', 'value': 2.0},
{'from': 6, 'to': 7, 'type': 'operator', 'value': '*'},
{'from': 8, 'to': 9, 'type': 'number', 'value': 4.0}]

Let's give left binding powers to each of the token types.
  • number - 0
  • + operator - 10
  • * operator - 20
Ok, so first we have a number token, with the value of 1.0. This is because in our shitjs so far all numbers are floats. Here is a log obtained by stepping through the expression.
('token', Literal({'to': 1, 'type': 'number', 'value': 1.0, 'from': 0}))
('expression right_binding_power: ', 0)
('token', OperatorAdd({'to': 3, 'type': 'operator', 'value': '+', 'from': 2}))
('left from prefix of first token', 1.0)
('token', Literal({'to': 5, 'type': 'number', 'value': 2.0, 'from': 4}))
('expression right_binding_power: ', 10)
('token', OperatorMul({'to': 7, 'type': 'operator', 'value': '*', 'from': 6}))
('left from prefix of first token', 2.0)
('token', Literal({'to': 9, 'type': 'number', 'value': 4.0, 'from': 8}))
('expression right_binding_power: ', 20)
('token', End({}))
('left from prefix of first token', 4.0)
('leaving expression with left:', 4.0)
('left from previous_token.infix(left)', 8.0)
right_binding_power:10: token.left_binding_power:0:
('leaving expression with left:', 8.0)
('left from previous_token.infix(left)', 9.0)
right_binding_power:0: token.left_binding_power:0:
('leaving expression with left:', 9.0)

You can see that it is a recursive algorithm. Each indentation is where it is entering a new expression.

Also, see how it manages to use the binding powers to make sure that the multiplication of 2 and 4 is done first before the addition. Otherwise the answer might be (1 + 2) * 4 == 12! Not the correct answer 9 that it gives at the end.

The operations are ordered this way because the + operator has a lower left binding power than the * operator.

You should also be able to see from that trace that a tokens infix and prefix operators are used. The OperatorAdd for example, takes what is on the left and adds it to the expression of what is on the right with it's prefix operator.

Here is an example Operator with prefix(left) and infix(right) methods.

class OperatorAdd(Token):
left_binding_power = 10
def prefix(self):
return self.context.expression(100)
def infix(self, left):
return left + self.context.expression(self.left_binding_power)

Pretty simple right? You can see the infix method takes the value in from the left, and adds it to the expression of what comes on the right.

Exercises for next time

Make this work:

>>> evaluate("1 + 2 * 4")

(ps... if you want to cheat the code repository has my part 2 solution in it if you want to see. The code is very short).

Until next time... Further reading (for the train, or the bathtub).

Below is some further reading about parsers for JavaScript, and parsers in python. After following some of those links you may realise that we could probably make this shitjs interpreter in an easier way by reusing libraries. However if we wanted to do that, we'd just use an existing JavaScript implementation! Also our JavaScript wouldn't be shit, or from scratch.

Friday, April 01, 2011

Pyweek April 2011, this weekend.

Pyweek 12 – April 3rd-April 10th

Find out more about PyWeek
- "Invites entrants to write a game in one week from scratch either as an individual or in a team. Is intended to be challenging and fun. Will hopefully increase the public body of game tools code and expertise. Will let a lot of people actually finish a game. May inspire new projects (with ready made teams!)"

Tuesday, March 22, 2011

Hammering nails - the hard way.

Do the following 1000 hammering tasks, and you will have learned how to use a hammer the hard way. You can do them from your bedroom. These are just like these guitar tasks that I do - in my spare time as a rock star. Well not really a rock star... but I sure can do some awesome guitar solos. They're also like katas I practice for my martial arts... in my spare time as a ninja.

"pssst..." "the world doesn't like guitar solos any more - and playing in your bedroom will never be like playing to real people. Also martial arts practitioners who use katas loose to fighters who practice fighting in real fights."

"pssst... you there..." "The pros don't use hammers. They use nail guns."


You don't need a book to learn python the hard way. To learn python the hard way, do some real world projects.

This pay out brought to you by the awesome Programming, Motherfuckers manifesto.

Monday, March 14, 2011

Hacking on android and pygame.

For the last couple of weeks I've been hacking on my new android tablet in my spare time.

It's one of those cheap 25cm ones (10 inches). Dual core, nvidia graphics, and all of that stuff. Not a bad little device really.

One project has been to port an old game of mine to it using PyGame subset for android(pgs4a). It's a package of a subset of the pygame modules - along with a bundled port of CPython for android. You can either package your apps standalone, or simply install the launcher from the android market, then upload your game to your SD card. Simples.

So far it's been pretty fun, and the other weekend I got something running. Next up is to work on the UI parts. Already it supports keyboard, mouse, mouse+keyboard, and also joystick mode. However, none of them really work with the touch screen on my tablet. I'm not a very big fan of the touch joypad approach. Where they make a virtual joypad on each side of the screen to touch. There are two main actions in [TheGame]... move, and shoot. You can click somewhere, and [MainCharactor] will try and move there. You can also charge up your steam cannon to fire a cannon ball - which requires aiming. There's a few interfaces which could work I think... I'll just have to try them out and see which I like best.

I hacked on the game to the point where I needed to add some missing python modules into the mix, so had to compile the pgs4a myself. That's taken a few days getting used to the tools, coming up against errors and then trying again the next day. The other day I got it compiling... now the next step is to try uploading the creation to the device.

The game in question is Zanthor, and for it I've done some major refactoring in order to modernise the packaging of it a little bit. Now there is a make, and configure script. The code is in the zanthor directory rather than a lib directory, and of course packaged it so it can run on android... the list of packaging changes is quite large but worthwhile. I hope to use it as a test case to improve packaging for games on python. The idea is to automate packaging for multiple platforms from any of the platforms. So that you can create a package for all supported platforms from any of them. This is a project that I've been talking about for *years*... but releasing an actual game with it is the only way to make it actually happen I think.

  zanthor on my android tablet
I also started messing around with a jack sound driver for SDL 1.3. Jack is the "professional" low latency sound system for linux(+osx). So far I've got it compiling, but still have a bunch more work to do on that. My todo list for that is still quite long. It's going to be tricky. Jack clients have high requirements to not pull down a jack server if they misbehave. I think I probably need to read through the jack developer documentation a few more times. One difficulty will be loading the modules dynamically... even though usually they are linked in with the "pkg-config --libs jack" tool at compilation time. Not too hard though, since SDL provides some nice cross platform dynamic library loading code.

Whilst on the pygame audio side, a really nice patch was posted to the pygame mailing list for the midi module by Christopher Arndt. It cleans up the the midi module some more, fixes some bugs, and adds some more documentation for the C level API. I still have to push his change up to the portmidi repository... which is a todo I've got for this weekend. This is the sort of polish that I hope we can apply to the other modules in the 1.9.2 release.

A pygame release is planned for soonish. The focus will be a polish release. Hopefully mainly polishing up existing stuff, and finishing off modules that are in an experimental or unfinished state. My main task is to get the documentation under control.
There's over 400 comments on the documentation to work through, so I expect this will take at least 40 hours time. I'm going to improve the comment system on the website to improve the work flow. To add the ability to mark a comment as 'dealt with', or 'example'. So when going through them, the options are 'delete', 'dealt with' or ignore for later.

I've ordered a new dedicated webserver to help out with tasks. This should allow us to do some more things than with just the current host. I paid for a year up front, so hopefully don't have to worry about it again for a while. It has an old nvidia graphics card in it, so I hope to do some graphical webserver tasks on it, as well as take some other heavy processing tasks off the main web host.

The new webserver should help us work in some countries where our current web host for is blocked too. Our current host does some things governments don't like... so I think that's why pygame is banned there. Or else governments find the zombies on the website scary.

Lenard Lindstrom has been working on some cleanups over the last number of months on some pygame modules. Especially the numpy integration, which is much improved and is finally up to a similar speed compared to the old Numeric implementation. Lenard has also explored converting our documentation system to use a different one. He's started a conversion tool for this, so if it turns out good we should be able to convert our docs over easily.

New binary installers for pygame on OSX is another task I've been working on. I mainly did this for a few people who were attending the London Python Dojo - who wanted pygame working on their Apple supplied python. However there are still a few bugs in it that I need to fix up. The main one being that SDL_image now does not link to libpng and libjpeg by default. That is sort of awesome... in that it can reduce file size. However, it's also sort of not awesome... since OSX does not support a few types of png files that are commonly used in pygame applications. Also pygame uses those libraries to save to png and save to jpeg. The other major change is that this package I made works with the Apple supplied version of python. Increasingly people on OSX don't bother getting the python from So now we're going to have multiple packages for OSX. One for Apple python on x864, and one for 10.4 ppc/x86 python. I'd also like to allow including the frameworks in the site-packages/pygame/Frameworks/ directory rather than in the system or users Framework directory. This will make installing better, since we won't need to ask for a password, and people should be able to bundle pygame more easily... since it won't be spread all over the file systems.

Wednesday, February 23, 2011

london clojure dojo

(Last night I went to the London clojure dojo.)
(. javax.swing.JOptionPane (showMessageDialog nil 
  (reduce str ["Hello" "London" "Clojure" "Dojo"])))
This dojo was hosted by @otfrom and @rrees at the ThoughtWorks offices (many thanks to them). They put on a nice spread of quiche and sandwiches. Here's some of the quiche... which was nicer than it looks in this photo.) (

-- quiche picture credit to @otfrom (used without permission ;))

There was a github repo setup before the session with the code from previous weeks.

After grazing, and drinking of softdrinks - the night started with a mini talk on cake (the tasty build tool + enhanced repl(interpreter) + kitchen sink for clojure) by the co-organiser of the dojo (Robert Rees from ThoughtWorks).

A programmer from Deutsche bank was there recruiting clojure programmers, and getting involved in the dojo too. It kind of reminded me of python in the early days... where you could count the number of python jobs around on one hand (and two feet worth of toes). People were talking about how they got a chance to use clojure at their work for a small project, or how they were thinking of sneaking a little bit of clojure into their projects.

Anyone who's been to the London python dojo would be familiar with the format (@otfrom is involved in that one too).

We were split into teams of five, and then huddled around the keyboard... then began to scratch our heads. I'd only started learning clojure earlier that day (well I think I've read about it before, but not in depth) (so there was a lot to learn. Luckily a few of the people in my group had more experience with clojure.

During these sessions, it's great to see how other people work in their environments. It's also interesting to see how people think in clojure. Everyone in our group got a chance to type at the keyboard, and we almost got our task done. In our group we used some tests, map+reduce and doall functions as well as list, array and map data structures. We even used a few cake commands (autotest and killall).

At the end of an hour or so of coding each group did a mini presentation and demo of what they made. Each group got a fair way towards getting the task done, and we got to see how they coded things. It's fun seeing what others got up to, the approach they took and the challenges they faced.

... at the end we left through the dungeon below the building. Those who survived had a few drinks at the pub afterwards. )

Monday, January 17, 2011

JQuery 1.5 promises a better future.

One of the new features of the unreleased JQuery 1.5 is a new $.ajax module.

Apart from many bug fixes, the most interesting, and useful part for me is that it now uses promises. Those familiar with python and twisted, or the JavaScript library mockikit, Dojo Deferred, or java.util.concurrent.Future might know these as deferreds or futures. jQuery is using promises, and not futures. They are similar but slightly different. It's a very commonly used technique in concurrency libraries now - so it is good that jQuery has hopped on board.

Since jQuery always returns a promise, you can now add callbacks even after you have done the call... or even change the callback later.
jQuery.getJSON( url ).error( errorCallback )

Pluggable ajax lets us test more easily

The other great thing(from a testing perspective) is that jQuery now allows you to use plug in mock implementations. This lets you make a fake ajax implementation for your unit tests more easily.

This pluggable architecture could also lead to people making other plugins that act the same as the ajax interface. Which could come in handy if you had a strange requirement, or you want to develop an alternative communication method - and be able to reuse much of the jQuery code.

Subclassing jQuery object

Another new area where jQuery helps extensibility, is the ability to subclass the jQuery object, and override methods - without it breaking the public interface to jQuery.

This is useful if you want to make a plugin that changes a few parts of jQuery for it's own uses, and not be worried that it will break 17 other plugins.

(function(globaljQuery, $) {
// private functions
$.fn.privateFunction = function(){ };

// overwrite an existing function without breaking other plugins
// which use the global jQuery function.
$.fn.popularExistingFunction = function(){ };

// Expose some public functions
globaljQuery.fn.myPublicFunction = function(){ };

})(jQuery, jQuery.subClass());

You can try the jQuery 1.5 beta 1 yourself if you want to have a play. I'm sure they will appreciate any bug reports.

Friday, January 14, 2011

London python dojo pygame write up.

Last night I went to the London python dojo at the Fry IT offices.

The night started with pizza, beer, and nerdy chats with everyone.

Then two lightning talks. The first by @tomviner (Tom), on some funny and useful python modules. `pip install oo && python -moo` Python goes moo (dad joke). `pip install e && python -me 1+2`

Next up was @otfrom (Bruce). He has a cunning plan to work on an engine that could be used later in a cross language dojo to display a game world. @otfrom is trying to see if other language dojos in london are interested in having a bot war tournament in the future. So far the clojure dojo, and perhaps the scala dojo are interested in participating.

@otfrom also announced the London dojos mailing list at: A group for people interested in programming dojos in london (not just python).

Then 30 of us took an open source game from, forked it and hacked on it!

I brought along a big bag of joypad controllers - 30 of them. The idea was to hand out one to each person who turned up. Unfortunately, they were two controllers joined together with one USB cable. Which meant that not everyone got a controller, but 15 people got two!

But first let's rewind to the night before... [[insert tape rewinding sound]] where @ntoll and I were discussing what to do... continue on with the xmas card demo we made last time? Or instead, hack on something a bit more complete?

The more complete game chosen is an over head dungeon crawller type game. It's not at all finished, and is a work progress one. @ntoll selected it from the thousands on It seemed to have fairly clean modular code, and be roughly what we were aiming to build.

As is usual of the london python dojos, we split up into groups. Five teams of six people this time. 'Pair programming on steroids - and not in a good way' is how the london python dojo has been described. The room was packed, and most teams split up into two groups of three, with 2-4 laptops per group - some laptops hooked up to larger screens so people a bit further away could have a look.

At the beginning we each got 'Quests', which were little tasks made up by @ntoll.

  1. Add NPC characters that walk randomly around the world
  2. Add music and sfx for particular events
  3. Allow the maps to be pre-defined rather than (as is the case) randomly generated each time.
  4. Add static objects to the game world such as gold or food that get added to the players inventory/energy/score/whatever when they bump into them...
  5. Two player mode (sharing the same keyboard or with two controllers)
  6. Menu system (Start Game, Help, Set-up, Quit)
Each group took their Quests, and got stuck into it. I had the pleasure of going around to each group to see how they were going, and to help out where I could with work arounds for pygame issues. I spent a time with most groups... but couldn't work my way into to where one group was, as they were surrounded by other people completely... and I would have had to climb over the top of people.

After coding stopped (@ntoll had to turn the lights off a few times to try and get people to stop)... the groups presented their work. It's a time for reflection and to show off what you did.

Each group managed to accomplish something. However some tasks were a bit harder than others - and they turned into reverse engineering tasks.

So by the end, we had:
  • joypad control of the main character, starting the game and exiting it.
  • A start on multiple player abstraction, from single player. With a small issue that the second player was not displayed on screen for some reason. A separate image is loaded and everything... it's just not displayed.
  • An Non Player Character (NPC), that could move and change direction (currently in two separate code bases).
  • A start on collision detection of the main character with things on the ground (like mush rooms). As this group was demoing their work, they 'lived coded' up some missing parts of the solution.
  • Music, which is different for the main menu screen and the in game music.
  • Sound effects, for different actions in the game... and some 'ambient' screams of FEAR and TERROR as the character walks through the forrest.
  • Loading and saving of maps into a text based format. Originally the maps are randomly generated, so this needed reverse engineering to figure out how that worked too.

A bloody good result really. Especially since it was code no one there had seen (except ntoll, and me a tiny bit), and the groups only had an hour and a half to hack on things, groups with people they never worked with before, using unfamiliar libraries that some hadn't even installed until the dojo (eg, pygame).

I think some people were a little frustrated with their tasks, especially some which were more of a challenge than others. But most seemed to enjoy it.

Overall I think this Dojo was about something which the organiser didn't intend. It was about quickly figuring out a code base, and modifying it. Reading, and modifying other peoples code is a great skill to master - and if nothing this Dojo gave us some experience with that, and let us see how other people do that too.

I saw some people using class browsers I hadn't seen, some using an API from the interpreter, others weilding their custom made code search scripts - to hunt down variables.

At the end of the dojos we usually decide on what we do next. We decided to continue on with the game idea in future dojos - so we can use it in a cross language dojo bot war. We also decided that 'someone' would try and merge all our bits of code together, and that @tjguk (Tim) would present an over view of the code base if he has time at the next dojo.

Unforuntately I couldn't go to the pub afterwards - hopefully next time.

* Tim wrote about the dojo on his blog too.
* @r4vi uploaded some videos