Wednesday, August 30, 2006

x86 the VM.

It's been really interesting to see VMs over the last couple of years.

Now there are emulators, and virtualisers which are capable of running x86 really quickly. The processors themselves don't run x86 natively anymore, it's a VM.

Now Apple are using x86, and x86 is getting more common in the embedded world too.

So now, rather than creating a VM like python does it seems to make sense to use the standard VM, and that is x86. Of course x86 is really complex, and still fairly slow to emulate on slow hardware. So using a simpler VM still has its advantages.

However writing directly to the most common VM has its advantages too. You can make software which is 400 bytes big which can do almost the same as a 8000 byte program. That's a 10x saving in program size. The same program will run in 12KiB of memory, instead of 1.7MiB of memory. That's a 141x memory usage saving. Because the code size, and memory size is so much smaller you can get a lot more done with the same amount of memory. You can run an entire OS and programs in less memory than a python process uses on debian linux. You can also use a lot more of these tiny processes. eg. for a single purpose webserver you can handle 40,000 connections using fork and separate processes in around 640MiB of ram with a duron 850. Or 100
connections using 1.6MiB, which is less than a single apache process.

I find it very interesting, and think that assumptions about processors and architectures that I learned 10 years ago are perhaps changing. The standard thinking is that fork is slow, and that you should use event driven async for high speed. Well it's not slow if your processes are only 800 bytes worth of code. Just enough code to do the exact task at hand.

Communication between processes has become really quick now. With system calls on linux like slice, tee, and vmsplice you can get two sockets transfering data directly without having to read data into user space. So you can take the output of a mpeg encoder and send it directly to a file on the HD not even copying the data, at the same time send it out on a socket through the network card. All without going into user space. How about compressing a stream of data, and sending it to a socket and also saving it in a file cache.

Processes and VMs can be transported between machines. Mosix can move processes amongst machines. With emulators you can store the whole state of the system in a file. You can store your x86 system image in a file, and move it to a different machine.

Then there's all this GPU general purpose computing stuff. You can run batch processes at 100x the speed of a cpu on these things. There's still a lot of science being done to find good ways to reimplement algorithms that work well, but lots has already been learnt. With the internet I think they have progressed much more quickly than in the 50s-70s. Chips like those found in the GP2X have multiple cpus, DSPs, and all their IO devices all on the same chip.

- tiny programs.
- emulators and virtualisers
- mosix, distributing processes.
- general purpose computing use of GPUs
- tee, splice, and vmsplice in linux 2.6.17

There's lots to be excited about with the new ways people are approaching VMs and computing.

Monday, August 28, 2006

pyweek 3 theme voting has begun.

The theme voting has begun. Here are the themes for the week long python game jam.

* Pick a card, any card
* Watch me pull a rabbit out from this hat
* Sawn in half
* Spoon bending
* The Disappearing Act

A very short list of themes to choose from this time.

Thanks to Richard for organising pyweek again! If you have some spare time coming up, and want to finish a game then you should enter pyweek. If you don't have much time, find a team and join that.

Let's hope the server stays up mostly :) These competitions all ways result in a server going down. Lots of users, and wide spread attention amongst smart, young and demanding hacker types means it gets challenges lots of other websites do not get. Strain on the software, and on the hardware are two of the causes. Otherwise it is caused by a simple cracking in to the machine to take it down.

Thursday, August 24, 2006

Qwerty rhymes. With Flirty.

not a bit of good food
for got home gut
jill had her kill

Oh, how I love bad poetry. I just had to share. You need to type it to appreciate how bad it is.

Saturday, August 12, 2006

Django security.

I replied to a blog post by Jason Huggins on his blog asking for a web frame work evaluation check list. I listed a number of common security issues, and finished my post with:

"I can not see any python web frameworks that do not have a history of security problems. I also can not see one which was designed from the beginning with security in mind."

Then James Bennett (a Django contributor) asked about problems with Django specifically. Saying 'if there’s some long bounding history of security flaws in Django, I sure haven’t seen it'.

Unfortunately my reply seems to be caught up in the moderators queue, so I guess I'll have to put my answer below. A couple of posts after mine are there, so I am not sure why my post was not approved to be posted.

ps. since my post on the 11th of August I have reported four security issues to the django project. Not one has even been replied to. Not even with a simple response like 'I got your email, we are looking at it'. It's great having a policy, but if you don't follow it then what's the point?
UPDATE: Adrian has been notified, and is looking into it. There was a problem with the email alias.

Have fun!

# Rene Dudfield Says: Your comment is awaiting moderation.
August 11th, 2006 at 3:37 am

James. You’ve answered your question about the Django security history for me. I can not see a list of security incidents listed on the Django site. I think a good idea would be to list them on the front page in fairly large type. However those ones you mentioned are only a few months old, so I reckon there’s more problems.

For XSS you need to start with white listing, and go from there. There’s a bunch of good articles on the subject. Just escaping stuff is not enough. is a good source of information about writing secure unix network software.

If you assume that user submitted content can not be cleaned, then is the system still secure? Feedparser, with 3000 unittests and heaps of high traffic sites still got it wrong.

Not only is Django itself not secure, but the pieces it is built on are not secure. Just search for apache, mod_python, and python security issues.

If you are sure about the current state of Django then maybe put up an offer for $1000 if no holes are found ;) I’ll give you $1000 if Django makes it through the next year without a hole found.

Jason. I’m not sure of a web framework which has security considered in the design from the start. It’s really good that the ruby developers have started to put an emphasis on it. I’m going to take this oportunity to evaluate my own security short comings, and I’m glad that others are too.

Saturday, August 05, 2006

Pyweek 3, one week game jam in september.

The third pyweek competition is starting in September.

You can have group entries, or go solo. It is a competition of sorts, but it is all just for fun really. You have a week to work on your game. Most people work on it in their spare time. Other people take a week off to do it.