collections in python - using less memory

Each object in python takes up far more memory than you might think. An int object for example does not take up 4 bytes.


So creating python objects for each element of a collection of data can use up far more memory than is needed.

A simple pattern for avoiding this wasted memory is to store the data in a array.array() then construct an object from the part of the data as you need it.


Using python classes to store an int.
Virtual 23992
Resident 21176

Constructing python classes to store an int dynamically.
Virtual 6572
Resident 3992


As you can see this method can save a *lot* of memory.

Here's some basic code demonstrating this technique... this isn't necessarily the API to use, but just demonstrates the memory savings. You can make a nicer to use API on top of that... or use your existing api with get magic properties.

wget http://rene.f0o.com/~rene/stuff/collection_memory.py

# using python objects...
python collection_memory.py 100000 -object & sleep 2 ; ps aux | grep python
# using a collection of python objects, which constructs a class from the raw data.
python collection_memory.py 100000 & sleep 2 ; ps aux | grep python

Many uses of python could use this technique to save lots of memory. Anything that operates on a large number of python objects.

Python database drivers are one area which could use this technique.

Sprite classes (like pygame sprites) could use an array of underlying data to store all the attributes.