collections in python - using less memory

Each object in python takes up far more memory than you might think. An int object for example does not take up 4 bytes.

So creating python objects for each element of a collection of data can use up far more memory than is needed.

A simple pattern for avoiding this wasted memory is to store the data in a array.array() then construct an object from the part of the data as you need it.

Using python classes to store an int.
Virtual 23992
Resident 21176

Constructing python classes to store an int dynamically.
Virtual 6572
Resident 3992

As you can see this method can save a *lot* of memory.

Here's some basic code demonstrating this technique... this isn't necessarily the API to use, but just demonstrates the memory savings. You can make a nicer to use API on top of that... or use your existing api with get magic properties.


# using python objects...
python 100000 -object & sleep 2 ; ps aux | grep python
# using a collection of python objects, which constructs a class from the raw data.
python 100000 & sleep 2 ; ps aux | grep python

Many uses of python could use this technique to save lots of memory. Anything that operates on a large number of python objects.

  • Python database drivers are one area which could use this technique.

  • Sprite classes (like pygame sprites) could use an array of underlying data to store all the attributes.
  • Comments

    Popular posts from this blog

    Is PostgreSQL good enough?

    pygame on pypy usable

    Experiments with new low latency PyPy garbage collector in a thread.