Friday, December 25, 2009

python3.2(svn) python2.7(svn) and the new GIL, cherrypy as a test.

I've done some quick benchmarks for the unreleased python3.2 on the unreleased cherrypy webserver for python 3. Also with the unreleased python2.7.

Here are the results...

python3.1

Client Thread Report (1000 requests, 14 byte response body, 10 server threads):

threads | Completed | Failed | req/sec | msec/req | KB/sec |
25 | 1000.0 | 0.0 | 533.32 | 1.875 | 93.75 |
50 | 1000.0 | 0.0 | 525.86 | 1.902 | 92.69 |
100 | 1000.0 | 0.0 | 522.96 | 1.912 | 92.1 |
200 | 1000.0 | 0.0 | 523.83 | 1.909 | 92.25 |
400 | 1000.0 | 0.0 | 506.92 | 1.973 | 89.27 |
Average | 1000.0 | 0.0 | 522.578 | 1.9142 | 92.012 |

python3.2

Client Thread Report (1000 requests, 14 byte response body, 10 server threads):

threads | Completed | Failed | req/sec | msec/req | KB/sec |
25 | 1000.0 | 0.0 | 555.72 | 1.799 | 97.78 |
50 | 1000.0 | 0.0 | 558.86 | 1.789 | 98.52 |
100 | 1000.0 | 0.0 | 552.87 | 1.809 | 97.45 |
200 | 1000.0 | 0.0 | 546.09 | 1.831 | 96.27 |
400 | 1000.0 | 0.0 | 548.64 | 1.823 | 96.53 |
Average | 1000.0 | 0.0 | 552.436 | 1.8102 | 97.31 |

So here you can see a small improvement in the 400 threads version with the new GIL in python3.2.

Python 3.2 threads seem more scalable in this benchmark compared to python 3.1. Also faster overall (20-40 requests per second faster).

python2.6

Python2.6 still beats python3.2 for cherrypy benchmarks. These are a network IO heavy, string processing heavy, with big python thread thread usage. So an ok benchmark for the new GIL work in my opinion.

Note that both python3.2, and cherrypy for python 3 are not released.
Client Thread Report (1000 requests, 14 byte response body, 10 server threads):

threads | Completed | Failed | req/sec | msec/req | KB/sec |
25 | 1000.0 | 0.0 | 660.54 | 1.514 | 116.43 |
50 | 1000.0 | 0.0 | 671.01 | 1.49 | 118.28 |
100 | 1000.0 | 0.0 | 663.84 | 1.506 | 117.12 |
200 | 1000.0 | 0.0 | 664.85 | 1.504 | 117.19 |
400 | 1000.0 | 0.0 | 651.9 | 1.534 | 114.8 |
Average | 1000.0 | 0.0 | 662.428 | 1.5096 | 116.764 |

python2.7

Python2.7 is faster still.
Client Thread Report (1000 requests, 14 byte response body, 10 server threads):

threads | Completed | Failed | req/sec | msec/req | KB/sec |
25 | 1000.0 | 0.0 | 695.33 | 1.438 | 122.79 |
50 | 1000.0 | 0.0 | 684.6 | 1.461 | 121.12 |
100 | 1000.0 | 0.0 | 688.99 | 1.451 | 121.67 |
200 | 1000.0 | 0.0 | 682.94 | 1.464 | 120.49 |
400 | 1000.0 | 0.0 | 641.01 | 1.56 | 112.78 |
Average | 1000.0 | 0.0 | 678.574 | 1.4748 | 119.77 |

It's also worth noting that 100-120%(out of 200%) of the two cores cpus are during the run of each version of python tested. Even though the GIL is released for many parts with python and cherrypy, and even though the benchmark is very IO heavy, both cores are not loaded. It's generally a good thing for a webserver to not load up the CPUs - but in a benchmark you want them to go full speed.

Also, during the tests the benchmarking tool 'ab' is run on the same machine skewing results. However ab seems to only use 1% of the CPU during the tests (according to top).

1.6ghz core 2 duo, ubuntu (save the) karmic koala 32bit version.

Shrinking the stack to save some memory.


So how do we reduce the memory usage of threaded programs? Reducing the stack size is one idea. Since threads do not share a stack with each other, each thread takes up a nice big stack. On ubuntu karmic koala it seems to default to 8MB or so.

Note, this is dangerous and can segfault your interpreter if you do not have enough stack space for some operations. So make sure you test things well before doing this.


In 2.6 you do:

import thread
thread.stack_size(32768*2)


Whereas in python 3.x you do:
import threading
threading.stack_size(32768)

3.2
----
(default stack) * 110MB virt, 10MB resident
(adjusted stack) * 15MB virt, 10MB resident

2.6
----
(default stack) * 107MB virt, 8MB resident
(adjusted stack) * 12MB virt, 8MB resident

Note, the python3.x versions start using up 2 gigabytes of memory in the benchmark. Compared to a peak of 600MB or so with python2.6. Showing some sort of memory leak in either python or cherrypy (note, both are pre-release).

It seems python3.2 worked with a smaller stack compared to python2.6. python2.6 segfaulted with 32768, but worked with a stack sized 32768*2. However python3.x used more resident memory, and virtual memory.

update:
I've uploaded profile(python cProfile), results and scripts here: http://rene.f0o.com/~rene/stuff/cherrypy_benchmarks. Strangely using python2.7 with the benchmark and cProfile did not finish (well it was still going 30 minutes later). So I guess that is a bug somewhere.


Wednesday, December 23, 2009

Invent with python - a new pygame/python book.

Invent with python 'Teach yourself how to program by making computer games!' is a free book now in its second edition. It is released under the creative commons licence Attribution-Noncommercial-Share Alike 3.0 United States - and the author just asks for a donation if you liked reading it.

The book takes the fun approach of teaching programming through making games... the way many of us learnt(and are still learning) to program.

As a bonus it uses python 3 - which pygame has supported for a while.

Congratulations to Albert Sweigart for finishing his second edition of the book.

Tuesday, December 22, 2009

structuring modules/packages and the cdb database for websites and python packages

Integrated modules are nice, but so are modular packages.

How can we have both? That is, keep all things for a module(or sub package) in one directory, but also have a nice integrated system built on top of that?

Often for one package I will have a file layout like so:

/setup.py
/[package_name]
/[package_name]/[package_module].py
/[package_name]/tests/[package_module]_test.py
/[package_name]/examples/[package_module].py
/[package_name]/docs/[package_module].py

Then each module has its tests, docs, and examples all mixed in the one directory. This is nice if you want to have all of the tests together, and all of the docs, and examples together.

However then all of the modules are mixed in together. Meaning it is harder to separate them, and keep one module in its own directory. Having everything for one thing together is nicer for developers I think.

Using namespace packages through setuptools distribute is one way. There is a draft pep around to put namespace packages into python as well. However, this feels way too heavyweight for me. Especially since they add multiple paths to the interpreter... meaning python has to do stat, and open, calls for every single path - which slows down the startup and import speed of python even when not using those packages.

Another way is to use a sub-package for each module. This turns each module into a sub-package.

However, some people put all of their stuff in an __init__.py file. Which is kind of hard to find, and not very descriptive. Also if you are editing a dozen __init__.py files for a project, it is really quite annoying. It is much better to have it in a 'modulename.py' file, and then import that in the __init.py file. Still slower than using modules directly though, since there are still extra stat calls, and extra file opens and reads.

Using the __init__.py file is still annoying in that you need the extra file, and have to remember what this magic file does.

This one liner can put all of the sub packages modules into the name space:
import os,sys
for f in os.listdir('somepackage'):
if os.path.isdir(os.path.join('somepackage',f)):
sys.path.insert(0,os.path.join('somepackage',f))
This is magic itself, and has a pretty bad code smell.

Other issues are playing nicely with things like cx_freeze, py2app, py2exe and the many other tools for managing python packages and modules... who don't know about namespace packages, or my custom magic.

I could write an import hook, or try out importlib in python3.1, but those would probably have similar issues to other non-standard ways.

So in the end, I think I'll stick with a fairly standard method of doing it... using a __init__.py file which imports the proper module into its namespace.

This lets me do import somepackage.somemodule and have it work. I can also "cd somepackage/somemodule/; python somemodule.py" and have it work.


/setup.py
/[package_name]
/[package_name]/[package_module]/
/[package_name]/[package_module]/[package_module]_test.py
/[package_name]/[package_module]/[package_module]_example.py
/[package_name]/[package_module]/[package_module]_docs.txt
This is how I'll structure pywebsite. With a separate directory for each sub module. This gives modularity, making it scalable to many modules, and it will simplify contributors lives. Since it uses standard packaging it should be compatible with most tools. Another bonus is that from within the source download I can just do:
import pywebsite
without having to call the setup.py script or install it. If a module changes from a .py file to a .so or a .dll or the other way around I won't have any issues either.

It is now a little harder for sub-packages within a package to refer to other sub-packages, but can be done with newer versions of python.

Sub-packages are more heavyweight than just using modules in your package... but it does seem cleaner and more extensible. However it is not as heavyweight as using namespace packages.

Any other issues with this approach?

namespace packages


Using the method outlined above, still does not allow me to split up a package into multiple separate files. This is ok though, since to start with I want to keep most of the modules in the one place... in the one repository, with the one bug tracker. However designing for extensibility from the start is useful, so we consider how it can be done.

This is where namespace packages are useful.

There is this draft pep: pep-0382, and also the setuptools distribute namespace packages (which is what everyone is using to do them). Here is the setuptools distribute documentation for namespace packages.

It should be possible to use some sort of hack, so that once you import a package, it searches for other packages with the same namespace.

You can use a packages __path__ attribute to tell it to look in other paths for importing modules.

>>> import pywebsite
>>> pywebsite.__path__
['pywebsite']
>>> pywebsite.__path__.append('lala')
>>> import pywebsite.bla
>>> pywebsite.bla.__file__
'lala/bla.py'

Using __path__ is outlined in the Packages in Multiple Directories part of the python tutorial.

As an example, say I had a 'otherpackage' package, and then someone else wanted to maintain part of that package, or we wanted to separate part of it out. Let's call this namespace package: 'otherpackage_doit'. It could install itself as a directory and package called 'otherpackage_doit'. Then import otherpackage_doit would work fine. However import pywebsite.doit would not work. You can't just call the package 'otherpackage.doit' either - since python will first look in the otherpackage package for the doit package, making import otherpackage.doit fail.

From a users discovery perspective, I would expect otherpackage.doit to be in otherpackage/doit. So that's where I'd look first. Installing into that directory would probably be best then. However that is not a very good method. After that, I'd probably do "print(otherpackage.doit.__file__)". Or I might do a "locate otherpackage.doit" command.

Really I just wish python3.2 could be changed so that 'otherpackage.doit' package is automatically a namespace package - without having to mess around with weird magic .pth files or declaring things in setup files like setuptools does.

So how can we retrofit(hack) existing pythons to do this for us? We need to get the python import machinery to search for otherpackage.* packages outside of the otherpackage directory. I'm sure it's possible with python somehow... Inserting 'otherpackage.doit' into sys.path does not work. You can't even have a package name with a '.' in it.

So I'll give up on namespace packages for now until a suitable option presents itself... or I have more time for research. Separate packages will have to live in the same file, but can still be separate with source control tools - like bzr, svn, github etc.

Still not fast enough... cdb databases for websites and python packages


However, the standard python package method is not the fastest. Supporting cgi operations for a web library is a good idea. This is because many webhosting platforms still only support python through cgi. So loading heaps of files for every cgi request is not an option. It is possible to get acceptable performance out of cgi and python... just many of the large frameworks have poorly optimized loading. Many frameworks rely on long running processes to avoid the slow load times. Using django via cgi in an embedded 130mhz arm with a limit of 10MiB is not going to work very well (or at all).

So how to make it faster for embedded/cgi apps?

Firstly an executable can be made. Using tools like py2exe. This can pack all of your data inside the executable.

One common method people try is to use the zip format. This works fairly well but is not optimal. Zip files are nice as they are supported by OS level tools, and file managers - so this will be one option to use. The downside, is that it makes the files harder to edit. I see .zip files as an optimisation that hinders usability. Especially .egg files(which are just .zip files) are bad, as it makes it harder to debug or change programs. So like .pyc files I think the zip file should be generated as needed - but having the full source tree there to change is very useful. If someone changes the source, the zip file should be regenerated as needed.

Another option is a constant database (cdb). cdb is a very simple constant database format used in things like djbdns, qmail and for other things. cdb happens to be one of the fastest(if not the fastest for constant databases [benchmarks pdf]). cdb is perfect for python packages that are not meant to change since they are so quick.

cdb is also pretty good for serving data from websites. Since often with websites much of the data is mostly static(constant) - a cdb key/value database is a nice optimisation over files on the file system. There are less syscalls, and less latency issues.

Zip files can also be used as .jar files by some browsers(firefox), to reduce latency on websites too. See the jar url scheme for details on how to put all your static files into a zip file(jar file).

There are some of my python module_experiments in:
bzr co http://rene.f0o.com/~rene/stuff/module_experiments
http://rene.f0o.com/~rene/stuff/module_experiments.tar.gz

So I have now refactored pywebsite to use a sub-package for each module, so that all the tests, docs and examples for that module are within its own sub-package. Using a zip/cdb file for imports will be left for later, as will namespace packages.

Monday, December 21, 2009

hashing urls for authorisation, and pywebsite.signed_url

Using a hash with a salt is one way of validating urls. With it you can tell that it is quite likely the url was generated by someone with a secret. That is, generated by someone authorised to make that url. This post describes using this technique with the hmac python module applied to urls(and object method calls).

Generating thumbnail urls is a good example. If you have a url which can generate a thumbnail of an image at a specified size that is pretty cool. Except then you can have someone generating images at all sorts of other weird sizes by exploiting the width, and height parameters (eg making 100,000 pixel wide thumbnails). Or perhaps you want to limit it to a certain set of images.

You could always code in your thumb nail generating routine which variables are valid... but this has problems. First it makes your routine less flexible. Separating authorisation is always a nice idea.

Another way (with HMAC like using pywebsite.signed_url) is to generate the urls and add a hash to them with a secret salt. This way you can be fairly sure that it was generated by someone with access to the secret salt.

This system is not as secure as using PKI, but it is a lot quicker to implement and is faster running.

One problem with using hashes is when you have to change the salt or hash scheme... then all your old urls are invalidated. Very annoying (sad panda).

Below is a little example of how you would hash protect some object methods, so only methods which are generated by authorised people are allowed. Note that it only uses a hash with a length of 6 characters, this is to keep urls short.

from pywebsite import signed_url, imageops

class SignedImages(object):
salt = 'somesecret'
root_dir = '/tmp/'

def thumb(self, hash, width, height, rotate, gallery, image):
""" serves a thumb nail
http://localhost:9942/thumb/XXXahashXXX/1000/1000/0/test/colorpicker.jpg
"""
salt = self.salt
keys = None
length_used = 6
values = [width, height, rotate, gallery, image]
if not signed_url.verify(values, salt, hash, keys, length_used = length_used):
raise ValueError('not a valid url')
cache_dir = os.path.join(self.root_dir, "cache/")
path_to_galleries = os.path.join(self.root_dir, "galleries/")
iops = imageops.ImageOps(cache_dir, path_to_galleries)
path = iops.resize(gallery, image, width, height, rotate)
return cherrypy.lib.static.serve_file(path)


def gen_thumb_url(self, width, height, gallery, image):
values = [ width, height, '0', gallery, image]
salt = self.salt
keys = None
length_used = 6
hash = signed_url.sign(values, salt, keys = keys, length_used = length_used)
url = "/".join(["thumb", hash] + values)
return url

thumb.exposed = True
gen_thumb_url.exposed = True


See the signed_url.py on launchpad file browser, or bzr co lp:pywebsite. Also see pywebsite.signed_url.signed_url_test for the unit tests. It is quite a simple module, and you could roll your own quite easily (with the python hmac module). Well, maybe not well easily... eg, consider the flickr exploits This code is also probably vulnerable( eg, SHA-1 is now vulnerable to collisions), but at least it will stop basic fiddling with unauthorised urls and shouldn't have the same vulnerabilities as the flickr API used to(and the facebook API might still have).


updates: been through a number of changes... the name has changed from hash_url to signed_url.sign, signed_url.verify. It has also moved into its own sub-package (like all pywebsite modules now live in their own self contained sub-package). Fixes for timing attack(unit tested). Uses hmac.

In Switzerland. Lots of snow!

In Switzerland. Had fun visiting an old friend and his new girl friend. Lots of snow came whilst we're here which was fun. However, the planes didn't like it, and they got cancelled. So going back to London today instead(if the planes work).

Thursday, December 17, 2009

Python the GIL, unladen swallow, reference counting, and atomic operations.

The unladen swallow project comments on the global interpreter lock and suggests long term dropping reference counting in favour of GC. http://code.google.com/p/unladen-swallow/wiki/ProjectPlan#Global_Interpreter_Lock. They are also now favouring an incremental GIL removal from python.

However, confusing getting rid of the GIL, with requiring using garbage collection is wrong. This mini-essay will explain why reference counting is good for python, and why you do not need to get rid of it for multiple CPU systems. Then there will be descriptions of other improvements to python for multiple CPU systems that can be made.

Reference counting has a number of advantages over various GC schemes see http://en.wikipedia.org/wiki/Reference_counting#Advantages_and_disadvantages. As a python programmer, optimizing for the programmer is what is most important... and reference counting makes it easier for the programmer. It makes programs much more deterministic, meaning that reference counting code is much easier to think about in our human heads.

Below are two real world examples of atomic reference counting. This makes reference counting thread safe, by using atomic operations.

Here are two mainstream open source object systems that have implemented it:

QT: from qt 4 released June 2005:
http://doc.trolltech.com/qq/qq14-threading.html#atomicreferencecounting

glib: released august 2005.
http://mail.gnome.org/archives/gnome-announce-list/2005-August/msg00048.html

It is possible to make reference counting thread safe in a performant, cross platform manner. Using the argument against reference counting just because of the GIL and thread safeness is a bad argument - as proved by at least these two highly portable, highly performant, open source pieces of code. Not just research papers, but real life code that has been used for a number of years now.

There has been a lot of research and real life code which improves reference counting, which python could take advantage of - rather than moving to a GC.

Simple Direct Media Layer also has a cross platform atomic operations API in SDL 1.3.

Locality of memory is much more important than using GC or reference counting. That is operating on memory worrying about if it is in the caches (L1,L2,L3 etc), also about locality to certain cpus/cores. This is why modern memory allocators have a per thread heap... to help memory allocated on one heap be accessed more easily by threads. See NUMA for details. NUMA factors play a BIG part in modern cpus already... even on single cpu systems. Designing for cache friendliness can give big speedups.

Reference counting can be a problem for cache friendly systems... but it doesn't need to be. There are techniques for reducing, or removing NUMA issues with reference counting memory management.

Reducing memory usage of objects will also mean for faster threaded python programs.
From the pypy project: "The total amount of RAM used on a 32-bit[ED:pypy] Linux is 247 MB, completing in 10.3 seconds. On CPython, it consumes 684 MB and takes 89 seconds to complete... This nicely shows that our GCs are much faster at allocating objects, and that our objects can be much smaller than CPython's."
More details here: http://morepypy.blogspot.com/2009/10/gc-improvements.html. As you can see reducing the memory size of objects can have a massive effect on runtime.

Another memory optimization for python can come from reducing the amount of allocations done by python at run time. Many real time systems have a policy of zero run time memory allocations. For python this is hard, however there are probably a number of places where memory allocations could be removed, or reduced in python. The pypy project tried to reduce the runtime memory allocations and found a number of them which sped things up.

Intelligent sizing of the stack is another memory optimization python could do. Each new thread has a copy of its own stack. So by having a very large stack, python can use less threads than is optimal. An optimization could be to try and use a very small stack, and then resize it if necessary. See the http://docs.python.org/library/thread.html for the thread.stack_size([size]) function.

Also speeding up GIL holding/removing will be good for those parts of python that already remove the GIL - especially on single CPU systems. Currently grabbing the GIL is a fairly costly operation. Using a method like that used by QT with atomic operations would be MUCH more efficient.

An incremental technique of GIL removal was taken by VMs like linux and *bsds (eg freebsd, dragonfly bsd) (and glib, qt). The idea is you make more fine grained locks on individual systems. Then you can slowly remove the GIL from parts incrementally... and you can test them. Removing the GIL all at once is silly... removing it piece by piece is a much better idea. I think the unladen swallow project is now preferring an incremental approach to GIL removal... which is nice.

A plan which approaches GIL removal incrementally is much more sane over all.

Designing APIs to work in manners better for threaded programs is another area python can get better performance on multi cpu systems. For example, by finding calls which block and making sure they release the GIL. Also by making batching APIs, allows not only avoiding the high cost of python function calls, but also makes that code embarrassingly easy to achieve parallelism.

One API that could be easily turned into a batching API is the import command. When you import 10 modules at the top of your program, much of that work could be shared or done in parallel. Likewise making the libraries be able to easily use

Documenting which calls release the GIL, or have fine grained locking/lock free programming can also help programmers more easily take advantage of multiple cpus.

Documenting which objects are safe to pickle will help people use the multiprocessing module, which does not work with objects unless the objects can be pickled. Likewise using forking can allow you to use multiple processes on data that can not be pickled. Since when you fork, the OS does the memory copying for you. The advantage over the multiprocessing library, is that with forking you only need the results to be able to be pickled... not the inputs.

Using wise madvise calls for mmap can also mark data which is read only... making multi processing much more efficient. Using madvise well, can mean much better copy on write behaviour. Improving copy on write behaviour is good for multiple processes... not just multiple threads. Likewise talking to the OS about what the program needs in other ways will also help. For example, if you know your program is going to use 1GB of data... then tell the OS that at the start. If you know it will eventually need 50 threads... then tell it at the start. If you know it needs a very small stack size... then tell it at the start.

Modern CPUs have memory prefetching commands, which can help out with telling the CPU which memory you are likely to access next. Likewise telling python, and the CPU that you want to only read or write to memory can help a lot with performance.

Apart from memory, there are many other OS level tweaks that can make your multi process python code go quicker. For example, setting the process/thread priorities. You can even ask your code to run with real time priority if it needs it. This can reduce the latency of processing, and also speed up the throughput. Easily gaining a 20% increase in throughput, or a very big drop in latency. If you do not care how fast your python program goes, then telling the OS so will help other processes in the system go faster.

Improving the performance of the built in primitives like the Queue/Mutex can also help python a lot. I've found that avoiding the python thread safe Queue can give you pretty good performance increases. Instead just using a list and using its python level atomic operations (documenting which python primitives are atomic would also help programmers a lot). A multi thread safe event queue can be quite nice. Especially if you can pass things into the queue at C level... avoiding the GIL entirely. There are a number of these available for python already (eg pygame.fastevent).

In fact for python 3.2 the GIL has been reworked: http://mail.python.org/pipermail/python-dev/2009-October/093321.html and the RLock primitive has been rewritten in C. This 'newgil' work has already been put into py3k trunk. However I think there is still quite a lot of room for improvement in the python GIL.

Another API design improvement are optimizations available for worker queues. One is sending similar tasks and data to the same cpus/cores... to take advantage of NUMA effects/cache (eg, always sending the url "/dojob1" to the same cpu).

Another worker queue optimisation is to reduce contention on the worker queue, by sending batches of tasks/data to the worker threads/processes. Rather than one task/piece of data at a time... divide the work up into many pieces and then send that across to the worker queues. eg, sending a list of 1000 images to each of the cpus to process rather than have each cpu ask for a single image to process. Dividing the work up into 16 parts of 100 each requires 16 interactions with the worker queue. Having them take one thing at a time means 16000 interactions with the worker queue - that is 16000 * 16 more opportunities for GIL locking and thread contention.

Many operations in the python stdlib could be made to use multiple threads/processes internally at the library level. Then people wouldn't need to know if the library is using async IO, threads, processes, or alien magic. eg, a url downloader might be given a list of URLs to GET... and return the responses. Underneath the covers it could use an async library, or even multiple threads to get the job done. This library level parallelism can happen if the APIs are given a batching design... that is given multiple inputs, and expecting multiple outputs. Then it is possible for library authors to try and choose the best technique. Taking single inputs, and giving single outputs does not let the library authors implement parallel algorithms as easily.


In conclusion, there is a lot we can do to improve python on multi cpu systems without removing the GIL entirely. An incremental approach to removing it from parts of python is a good idea as shown by other projects doing the same. Removing reference counting from python is not needed, as there are good, proven ways of using reference counting in multi CPU systems.

Wednesday, December 16, 2009

Writing for the latest python is FUN.

Writing for the latest python version is just simply... fun. Even the name 'python 3000' or py3k is amusing.

It's been a joy over the last week plugging away at a few personal website and game projects using the latest python version. Over the last year I've been involved in upgrading existing code to the latest versions of python... but not really in writing it JUST for the latest versions.

No need to consider backwards compatibility problems... older pythons be damned when programming for fun! I'm able to take advantage of the latest python features in deep architectural ways. To fiddle around with the new deep magic python internals. But the new deep magic parts aren't the best bits though... it's all the elegant little improvements that make a difference.

It is of great delight finding python 3 compatible modules. It turns out that the modules available for python 3 are often of quite good quality - and well maintained. There's not all that many of them around, so it can be good just to browse through those python 3 packages that are available.

All of this fun makes me reminisce to the days of playing around with python ten years ago. Perhaps I'm romanticising it a little bit(well I definitely am!). However when first getting into python all those years ago was quite a thing. It was probably the small community, and the idea that optimizing for the programmer was more important than optimizing for the computer(no more squiggly braces :). Like the halcyon days of python2, there is a smaller community of python 3 programmers, and there are a bunch of new techniques and tricks available from python 3. These two factors remind me about what it was like to get into python all those years ago.

Most of the programmers doing python 3 stuff are either doing it for fun, or doing it for the good of the python ecosystem over all, that is to make python better for people rather than better for the machine.

So if you need a reason to get into python 3 it is this: python 3 makes python fun again.

Wednesday, December 09, 2009

pywebsite.sqlitepickle. sqlite VS pickle

The sqlite and pickle modules that come with python are quite useful. So lets mix them together and see what comes out.

SQLITE Vs PICKLE

pywebsite.sqlitepickle is a little module I just made which combines sqlite and pickle for persistence. Useful since it works with python3, and both pickle and sqlite are included with pythons (including pypy).

Import the module.
>>> from pywebsite import sqlitepickle

An in memory db.
>>> db = sqlitepickle.SQLPickle()
>>> db.save('key', 'value')
>>> db.get('key')
'value'

Can also save to a file. So we first get a temp file name.
>>> import tempfile
>>> f = tempfile.NamedTemporaryFile()
>>> fname = f.name

>>> db = sqlitepickle.SQLPickle(fname)
>>> db.save('key', 'value')
>>> db.get('key')
'value'

>>> db.close()


The issues with this are that sqlite does not like sharing connections from multiple threads. To get around that I just create a new connection in each thread, or do db stuff from one thread. Also using pickles in persistent data can be a security issue.

Speed? ok speed. Pickle isn't the fastest, and neither is sqlite... but they are both kind of ok.

http://pypi.python.org/pypi/pywebsite
https://launchpad.net/pywebsite
http://www.pywebsite.org/

Tuesday, December 08, 2009

Play cards. Not business cards.

Play cards. They're like business cards, but take 30 minutes to make five. Also, they are for handing out to people you want to play with, rather than to people you want to do business with. Well it could be used for both... but I think the normal style business card is a bit boring to give to friends.

play cards


Made some of these the other day. Based on an earlier version I designed a few years ago. They're about the size of a finger, but unfold to show other details (eg, email, website, etc).

--

Sunday, December 06, 2009

buildout project that uses numpy

Here is an example buildout project that uses numpy:
https://launchpad.net/numpybuildout/

If you have bazaar, you can check it out like so:

cd /tmp/
bzr branch lp:numpybuildout
cd numpybuildout/trunk/


To use distribute instead of setuptools use the -d flag of bootstrap.py.
python bootstrap.py -d
./bin/buildout
./bin/py
>>> import numpy
>>> numpy.__file__
'/tmp/numpybuildout/trunk/eggs/numpy-1.4.0-py2.6-linux-i686.egg/numpy/__init__.pyc'


There you have it, a simple project that can download and build numpy off of the python package index and install it in it's own private directory for use.

update: It was broken for python2.6 and numpy 1.3. However it works again with numpy 1.4

Saturday, December 05, 2009

gvim and karmic ubuntu... with a fix for you.

gvim in ubuntu karmic koala is really annoying. It prints out a bunch of gtk warnings each time you call it. However thanks to someone making a patch, you can clean it up. It seems that the fix isn't going to come out until the next ubuntu.

** (gvim:13354): CRITICAL **: gtk_form_set_static_gravity: assertion `static_gravity_supported' failed

** (gvim:13354): CRITICAL **: gtk_form_set_static_gravity: assertion `static_gravity_supported' failed

** (gvim:13354): CRITICAL **: gtk_form_set_static_gravity: assertion `static_gravity_supported' failed

** (gvim:13354): CRITICAL **: gtk_form_set_static_gravity: assertion `static_gravity_supported' failed

** (gvim:13354): CRITICAL **: gtk_form_set_static_gravity: assertion `static_gravity_supported' failed


Very annoying to see that every time you edit a file.

https://bugs.launchpad.net/ubuntu/+source/vim/+bug/402188


So instead of switching to arch linux, or gentoo linux... or one of the other developer centric distros there is a patch! ya for patches people make and share :)

Grab the debdiff to apply the patch. Check a look at the packaging guide for details on how to build with a debdiff patch.
https://wiki.ubuntu.com/UbuntuPackagingGuide/BuildFromDebdiff


NOTE: please back up your system before trying anything here. Also DO NOT TRY THIS AT HOME AS SOMETHING MAY GO HORRIBLY WRONG!!!

I'll run through what to do here:


cd ~
$ mkdir vim-gnome
$ cd vim-gnome

$ wget http://launchpadlibrarian.net/35716623/vim_7.2.245-2ubuntu2.1.debdiff


$ md5sum vim_7.2.245-2ubuntu2.1.debdiff
abdb13517ec59a1a0b74b55b977e0139 vim_7.2.245-2ubuntu2.1.debdiff

Check that your md5sum is the same as this. Hey, why not look at the patch to review it for goodness? Well you don't have to, but it can be interesting looking at bug fixes.

We need to install all the tools required for building ubuntu packages...
$ sudo apt-get install build-essential fakeroot devscripts

Then grab the source for vim-gnome (or vim-gtk if you don't want to use the gnome version... replace vim-gnome with vim-gtk in all things below if you want that instead).
$ apt-get source vim-gnome


When I installed the libraries needed to build the source package...
sudo apt-get build-dep vim-gnome
I got a 'Segmentation fault'. eek. Well, time for a restart... It seems Karmic isn't all that stable for me with apt-get, dpkg and friends.

Ok, back from the restart of the computer.
Let us try it again...
sudo apt-get build-dep vim-gnome

Ya! it's working. I have to download about 100MB of packages. ... wait 10 minutes... then everything is downloaded and installed.



Apply the patch...

$ cd vim-7.2.245/
$ patch -p1 < ../vim_7.2.245-2ubuntu2.1.debdiff



Ok, lets build it again after we've applied this patch. This build step can take a long while!

$ debuild -uc -us

Ok, there's no test step... since ubuntu doesn't really have a way to run automated test suites (that I know of).

At least lintian is run and tells you something.

It should produce a bunch of .deb files for you.

vim_7.2.245-2ubuntu2.1_i386.deb vim-gtk_7.2.245-2ubuntu2.1_i386.deb
vim-common_7.2.245-2ubuntu2.1_i386.deb vim-gui-common_7.2.245-2ubuntu2.1_all.deb
vim-dbg_7.2.245-2ubuntu2.1_i386.deb vim-nox_7.2.245-2ubuntu2.1_i386.deb
vim-doc_7.2.245-2ubuntu2.1_all.deb vim-runtime_7.2.245-2ubuntu2.1_all.deb
vim-gnome_7.2.245-2ubuntu2.1_i386.deb vim-tiny_7.2.245-2ubuntu2.1_i386.deb


Now to install the relevant packages...
$ sudo dpkg -i vim_7.2.245-2ubuntu2.1_i386.deb vim-common_7.2.245-2ubuntu2.1_i386.deb vim-gui-common_7.2.245-2ubuntu2.1_all.deb vim-runtime_7.2.245-2ubuntu2.1_all.deb vim-gnome_7.2.245-2ubuntu2.1_i386.deb

Ya! no more annoying message :)

Now, if someone made a PPA other people on different architectures could also easily update their binaries. Or perhaps the ubuntu folks will make it nice for gvim developers, and apply the patch themselves.

Friday, December 04, 2009

From Berlin, back to London again... and python dojo!

Arrived back in London a few days ago, after a month living in Berlin.

I really enjoyed Berlin, and thought about staying there for longer... but in the end decided against it for now. Would really like to visit in the summer, when it's apparently quite a different place!


Some pictures from Berlin


Finding the bustle and life on the streets of London quite refreshing. I think I'll be based in London for the next six months or so at this point(probably longer).

Going to the london python dojo again on the 10th. This is an interesting format for a get together. People take turns pair programming with a projector, whilst the rest of the people in the room offer comments. This time there's going to be a few short interactive presentations too. It was nice meeting python people from around London last time. Fry-it are the hosts, who offer their office, pizza(+salad) and beer(+wine) for everyone at each dojo.