Thursday, November 08, 2018

Drawing data. With #flask, and #matplotlib.

Here's a pretty simple example of using the Flask microweb framework, and matplotlib together.
Matplotlib is a graphing library used by a lot of people in python.

But how to serve your graphs up in a web browser as a png or an SVG image?
Code is in this gist: flask matplotlib.

Serving svg and png images with Flask and matplotlib.


flask and matplotlib for drawing svg and png graphs with python
here is what it looks like

Code is in this gist: flask matplotlib.

Thursday, November 01, 2018

Josh Bartlett — pygame Artist in Residence

The pygame Artist in Residence grant celebrates the Python arts community and lends a tiny bit of support to someones art practice. Josh Bartlett was the first recipient.

An artist residency usually works something like; a person spends some time in either a gallery making something to present or in a music club doing a weekly spot. The pygame artist in residence will do it in their own space, but be present on the top of the pygame website in the form of a thumbnail and a link to their patreon/blog/artist statement/website/whatever.

Josh Bartlett was the first recipient, and has been featured on the pygame website in October.  Please see the pygame artist in residence profile page for more info.



What has he been up to?  Go to his blog to find out.

ps. Thank you to everyone who applied, and those who helped with selection. Hopefully the next one will go more smoothly as the process is improved.

Friday, August 31, 2018

pygame artist in residence grant

An artist residency usually works something like; a person spends some time in either a gallery making something to present or in a music club doing a weekly spot.

Artist in residence.

The first "pygame artist in residence" grant will be for a small amount of money (€512). It is for someone who is already doing stuff with pygame or python in their arts practice. There will be a little thumbnail on the website linking towards an artist statement/patreon page/blog or some such. The residency won't be in a physical space. It lasts for a month, from October 1st - November 1st. At the end of the month the website 'exhibition' will take the form of a small update from the artist — in whatever form that takes.


If you'd like to apply(or nominate someone) please write something short with a link about the arts practice and how python or pygame is used in it.
Email: residency@pygame.org
Applications close: September 15th.
Announced: October 1st.
Amount: €512 paid by paypal.




Why this? Art is awesome.

Ok, there are other reasons... I feel the python world largely ignores the contributions that the game development, graphics, education, music and the arts community provides for python. People who are makers, musicicians, artists, teachers, and game devs have contributed really major things to the python world. And the python website doesn't even mention that art, or games are a thing. They don't exist apparently.
According to python.org art does not exist
From Ada Lovelace the writer(and first programmer), to Steve Jobs who started out at Atari, to Bill gates the author of the first ever PC game, to Konrad Zuse the painter (who happened to also make the first digital computer) — the arts and computers are intertwined. Python itself is named after a comedy group, and was started off within a multimedia group at CWI after the author had worked on a programming system intended for teaching. Would python be the same without import this ( the zen of python) or as fun without import antigravity?

Art, education, music, making, multimedia, and games are important to 'computers' and to python. Sure, stop calling pypi the cheese shop (made by a damn fine baker and game developer) because it's too silly... but please don't erase the arts from existence. The python arts community exists.

Monday, August 27, 2018

Draft of, "How to port and market games using #python and #pygame."

This is a collaborative document, and a really early draft. Please feel free to add any tips or links in a comment here or on the reddit post https://www.reddit.com/r/pygame/comments/9aodt7/collaborative_doc_lets_write_pygame_distribution/
You've spent two years making a game, but now want other people to see it?
How do you port it to different platforms, and make it available to others? How do you let people know it is even a thing? Is your game Free Libre software, or shareware?

All python related applications are welcome on www.pygame.org. You'll need a screenshot, a description of your game, and some sort of URL to link people to (a github/gitlab/bitbucket perhaps).  But how and where else can you share it?

a few platforms to port to

  • itch.io and windows
  • windows store?
  • mac (for itch.io)
  • mac store
  • steam
  • linux 'flatpack' (latest fedora/ubuntu etc use this like an app store).
  • pypi (python packages can actually be installed by lots of people)
  • android store
  • web
  • debian
  • redhat/fedora

Make it a python package.

Some of the tools work more easily with your package as a python package. Working with all the different tools is sort of hard, and having a convention for packaging would make things easier.

Python packaging guide - http://packaging.python.org/

So, why don't we do things as a simple python package with one example app to do this? pygame has an example app already, solarwolf - https://github.com/pygame/solarwolf. With work, it could be a good example app to use. We can also link in this guide to other pygame apps that have been distributed on various places.

There are other example apps linked below for different distribution technology.
 

pyinstaller

https://www.pyinstaller.org/
This can make ones for linux, windows, and mac.
pyinstaller --onefile --windowed --icon=icon.ico .py

Windows

pynsist and pyinstaller can be used.
https://github.com/takluyver/pynsist

The benefit of pynsist is that it can create installers. Whereas pyinstaller is for making standalone executables (which is good if you are putting your app on the Steam store for example).

Windows code signing

Flatpak

apps on linux - https://flatpak.org/
Here's an example of making a pygame one. https://github.com/flathub/flathub/pull/478
Developer guide for more detail here - http://docs.flatpak.org/en/latest/

pypi

The python package system can mean your app can be available for everyone who can use pip. Which is an audience in the millions.

Mac

pyinstaller is probably the best option at the moment. If your game is open source, then you could use TravisCI for free to make builds with pyinstaller.

Unfortunately you probably need a Mac to make a mac build, test it, and release on the mac/ios stores. Getting a cheap apple machine off ebay might be the way to go. Or a cloud account perhaps from 'macincloud'. Also the mac developer program costs $100.

Another option might be to borrow a friends machine to make the builds when it's time.
See:

iOS

It's not easy, but possible.

With pygame 2 this should be possible since it uses the new SDL2.
If you use LGPL code on iOS you still have to let your users benefit from the protections the LGPL gives them.

Tom from renpy says... "I've been distributing Ren'Py under LGPL section 6c, which says that you can distribute it along with a written offer to provide the source code required to create the executables. Since Ren'Py has a reasonably strong distinction between the engine and game scripts, the user can then combine the game data from an iOS backup with the newly-linked Ren'Py to get a package they can install through xcode." https://github.com/renpy/pygame_sdl2/issues/109#issuecomment-412156973

An apple developer account costs $100, and selling things costs 30% of the cost of your app. https://developer.apple.com/

Steam

There's a few games released using pygame on steam. Here are two threads of games released:
Costs $100 to join up and sell a game on this store. https://partner.steamgames.com/
Recently someone used pyinstaller to package thier game.
pyinstaller --onefile --windowed --icon=icon.ico .py

SteamworksPy

A python module for the C++ steam sdk. https://github.com/Gramps/SteamworksPy
Made by someone who has released their game (using pygame) on steam.

Itch.io

"itch.io is an open marketplace for independent digital creators with a focus on independent video games."
Quite a few people have released their pygame games on itch.io.

Android

This isn't really possible to do well at the moment without a bit of work.

python-for-android seems the best option, but doesn't work well with pygame. https://github.com/kivy/python-for-android There is an old and unmaintained pygame recipe included (for an old pygame 1.9.1). With some work it should be possible to update the recipe to use the SDL2 support in pygame.

There was an older 'pygame subset for android' which is now unmaintained, and does not work with more recent Android devices.

Web

There's not really an 'export for web' option at the moment. It is possible with both CPython and SDL as well as SDL2 working on emscripten (the compiler for WASM and stuff that goes on the web).
Here is the latest 'cpython on web' project. https://github.com/iodide-project/pyodide

Building if you do not have a windows/mac/linux machine

CI tools

If your game is open source, you can use these systems to build your game remotely for free.
How to do that? Well, that's an exercise left up to the reader. Probably getting it to use pyinstaller, and having them upload the result somewhere.

One python app that uses Travis and Appveyor is the Mu editor. You can see how in their .travis.yml and appveyor.yml files. See https://github.com/mu-editor/mu


Virtualbox

With virtualbox (and other emulators) you can run some systems on your local machine. Which means you do not need to buy a new development machine yourself for those platforms.

Both windows and linux images are available that you could use legally.

https://developer.microsoft.com/en-us/microsoft-edge/tools/vms/

Note, that it is good to do your testing on a free install, rather than testing on the same machine that you made your executables with. This is because perhaps you forgot to include some dependency, and that dependency is on the development machine, but not everyone else's machines.

Writing portable python code

Some old (but still valid) basic advice on making your game portable: https://www.pygame.org/wiki/distributing

Things like naming your files case sensitively.

Announcing your game.

Generic Indie game marketing guides all apply here.

Some python/pygame specific avenues for marketing and announcing...
Of course the python world is a tiny place compared to the entire world.



Icons.

Each platform has slightly different requirements for icons. This might be a nice place to link to all the requirements (TODO).


Making a game trailer (for youtube)

You may not need to make the best trailer, or even a good trailer. Just a screen capture of your game might be 'good enough' and is better than nothing.

How about making a trailer with pygame itself? You could call it 'demo mode', or 'intro mode'.
There's a free iMovie on Mac, the Microsoft video editor on windows, and blender for all platforms. An alternative is to use the python module moviepy and script your game trailer.

OBS is pretty good multi platform free screen capture software. https://obsproject.com/download

Animated gif

These are useful for sharing on twitter and other such places, so people can see game play.
You can save the .png files with pygame, and convert them to a gif with the 'convert' tool from imagemagik.
# brew install imagemagick
# sudo apt-get install imagemagick

# call this in your main loop.
pygame.image.save(surf, 'bla_%05d.png' % frame_idx)

Now you can convert the png files to
convert -delay 20 -loop 0 bla_*png animated.gif

Some solutions on stack overflow.

Thursday, July 19, 2018

pygame 1.9.4 released

pygame 1.9.4
pygame 1.9.4 has been released into the wild!

TLDR; Some highlights.

  • python 3.7 support.
  • beta pypy support. See Are we pypy yet?.
  • pygame.draw fixes
  • pygame.math is not experimental anymore. Speedups and bugfixes.
  • Debian, Mac homebrew, mac virtualenv, manylinux and other platform fixes.
  • documentation fixes, jedi support for type ahead in editors like VSCode and VIM.
  • Surface.blits for blitting many surfaces at once more quickly.

Thanks

A very special thanks to the people who have volunteered commits to pygame since the last release. In alphabetical order...
Adam Di Carlo (@adicarlo) | Christian Bender (@christianbender) | Don Kirkby (@donkirkby) | endolith (@endolith) | hjpotter92 (@hjpotter92) | Ian Mallett (@imallett) | Lenard Lindstrom (@llindstrom) | Mathias Weber (@mweb) | Matti Picus (@mattip) | Nicholas Tollervey (@ntoll) | (@orangudan) | Raymon Skjørten Hansen (@raymonshansen) | René Dudfield (@illume) | Stefan Bethge (@kjyv) | Stuart Axon (@stuaxo) | Thomas Kluyver (@takluyver) | Tobias Persson (@Anisa)

I'm probably missing some people, and also missing some people who contributed in other ways.
For example, in discussions, issue reports, helping out on the wiki, the website, and for helping others
in the community, and providing good vibes. So whilst the commits are easy to use to make a list of people to thank, it's not inclusive of everyone who deserves thanks.

More details.

#451 #460 #467 #468 #469 #470
#444 link to help pages when compile fails.
#443 In set_error get_error tests ignore first error. Could be anything.
#442 Freetype requires pkg-config instead of freetype-config now.
#439 Surface.blits
#435 Adding pypy builds for Mac on travis.
#432 Appveyor pypy and pypy3 windows 32bit.
#431 Implement object alloc caching for rect.c to improve on pypy.
#427 PixelArray.close(), with PixelArray(surf) as px, context manager.
#426 Skip tests that rely on arrinter and pythonapi on pypy.
#420 pypy didn't like tp_dictoffset hack in events. Make our own setter, getter.
#418 draw.aaline should work with ARGB surfaces (like on mac).
#416 Vector cleanup
#415 So virtualenv gets a focused window on Mac too.
#414 Mac Travis homebrew fix
#413 Jedi confused by pygame imports. Make it happy.
#408 pygame.transform.threshold tests, keyword arguments, docs.
#403 pygame.math.Vector2/3 not experimental
#398 Clean up _camera_vidcapture.py unused code, and document a bit.
#394 Add pitch bend to MIDI library
#392 Add pypy builder to travis ci, and allow it to fail.
#391 ppc64le and other Debian fixes
#389 pygame.draw.circle with a thickness had a weird moiré pattern.
#387 test python 3.7 on travis CI.
#386 python 3.7 fixes.
#384 pygame.display doc fixes.
#381 import rect.inflate docs.
#363 Fix several typos, and improve grammar in the introduction.
#361 Add unit test for some key functions.
#360 update math.c for pypy.
#357 add UYVY support for better linux camera support.
#356 Fix aaellipse artifacts
703350f Update Rect slicing for Python 3
6d0e97a bug fix for freetype.Font.render_to()
#78 Add environment PYGAME_EXTRA_BASE to add an extra base directory to the start of the search path.
#77 Build alsa libs ourselves for manylinux builds.
#76 Docs fixup.

Wednesday, July 18, 2018

Draft of, ^Let's write a unit test!^

(BeginDraft)

So, I started writing this for people who want to 'contribute' to Free Libre and Open source projects.
It's not finished yet, but still useful, and I'd like a bit of feedback, and to start linking to it from the pygame developer docs. So there.
(/EndDraft)

A unit test is a piece of code which tests one thing works well in isolation from other parts of software. In this guide, I'm going to explain how to write one using the standard python unittest module, for the pygame game library. You can apply this advice to most python projects, or free/libre open source projects in general.

A minimal test.

What pygame.draw.ellipse should do: http://www.pygame.org/docs/ref/draw.html#pygame.draw.ellipse
Where to put the test: https://github.com/pygame/pygame/blob/master/test/draw_test.py

def test_ellipse(self):
    import pygame.draw
    surf = pygame.Surface((320, 200))
    pygame.draw.ellipse(surf, (255, 0, 0), (10, 10, 25, 20))

All the test does is call the draw function on the surface with a color, and a rectangle. That's it. A minimal, useful test. If you have a github account, you can even edit the test file in the browser to submit your PR. If you have email, or internet access you can email me or someone else on the internet and ask them to do add it to pygame.

But why write a unit test anyway?

Unit tests help pygame make sure things don't break on multiple platforms. When your code is running on dozens of CPUs and just as many operating systems things get a little tricky to test manually. So we write a unit test and let all the build robots do that work for us.

A great way to contribute to libre/free and open source projects is to contribute a test. Less bugs in the library means less bugs in your own code. Additionally, you get some public credit for your contribution.

The best part about it, is that it's a great way to learn python, and about the thing you are testing. Want to know how graphics algorithms should work, in lots of detail? Start writing tests for them.
The simplest test is to just call the function. Just calling it is a great first test. Easy, and useful.

At the time of writing there are 39 functions that aren't even called when running the pygame tests. Why not join me on this adventure?


Let's write a unit test!

In this guide I'm going to write a test for an pygame.draw.ellipse to make sure a thick circle has the correct colors in it, and not lots of black spots. There's a bunch of tips and tricks to help you along your way. Whilst you can just edit a test in your web browser, and submit a PR, it might be more comfortable to do it in your normal development environment.

Grab a fork, and let's dig in.

Set up git for github if you haven't already. Then you'll want to 'fork' pygame on https://github.com/pygame/pygame so you have your own local copy.
Note, we also accept patches by email, or on github issues. So you can skip all this github business if you want to. https://www.pygame.org/wiki/patchesandbugs
  • Fork the repository (see top right of the pygame repo page)
  • Make the change locally. Push to your copy of the fork.
  • Submit a pull request
So you've forked the repo, and now you can clone your own copy of the git repo locally.

$ git clone https://github.com/YOUR-USERNAME/pygame
$ cd pygame/
$ python test/draw_test.py 
...
----------------------------------------------------------------------
Ran 3 tests in 0.007s

OK

You'll see all of the tests in the test/ folder.

Browse the test folder online: https://github.com/pygame/pygame/tree/master/test


If you have an older version of pygame, you can use this little program to see the issue.


There is some more extensive documentation in the test/README file. Including on how to write a test that requires manual interaction.


Standard unittest module.

pygame uses the standard python unittest module. With a few enhancements to make it nicer for developing C code.
Fun fact: pygame included the unit testing module before python did.
We will go over the basics in this guide, but for more detailed information please see:
https://docs.python.org/3/library/unittest.html



How to run a single test?

Running all the tests at once can take a while. What if you just want to run a single test?

If we look inside draw_test.py, each test is a class name, and a function. There is a "DrawModuleTest" class, and there should be a "def test_ellipse" function.

So, let's run the test...

~/pygame/ $ python test/draw_test.py DrawModuleTest.test_ellipse
Traceback (most recent call last):
...
AttributeError: type object 'DrawModuleTest' has no attribute 'test_ellipse'


Starting with failure. Our test isn't there yet.

Good. This fails. It's because we don't have a test called "def test_ellipse" in there yet. What there is, is a method called 'todo_test_ellipse'. This is an extension pygame testing framework has so we can easily see which functionality we still need to write tests for.

~/pygame/ $ python run_tests.py --incomplete
...
FAILED (errors=39)

Looks like there are currently 39 functions or methods without a test. Easy pickings.


Digression: Low hanging fruit, help wanted. 

Something that's easy to do.

A little digression for a moment... what is low hanging fruit?

Low hanging fruit is easy to get off the tree. You don't need a ladder, or robot arms with a claw on the end. So I guess that's what people are talking about in the programming world when they say "low hanging fruit".

pygame low hanging fruit


Many projects keep a list of "low hanging fruit", or "help wanted" issues. Like the pygame low hanging fruit list. Ones other people don't think will be all that super hard to do. If you can't find any on there labeled like this, then ask them. Perhaps they'll know of something easy to do, but haven't had the time to mark one yet.

One little trick is that writing a simple test is quite easy for most projects. So if they don't have any marked "low hanging fruit", go take a look in their test folder and see if you can add something in there.

Don't be afraid to ask questions. If you look at an issue, and you can't figure it out, or get stuck on something, ask a nice question in there for help.

Digression: Contribution guide.

There's usually also a contribution guide.  Like the pygame Contribute wiki page. Or it may be called developer docs, or there may be a CONTRIBUTING.md file in the source code repository. Often there is a separate place the developers talk on. For pygame it is the pygame mailing list, but there is also a chat server. Find the details on the Info wiki page.

Back to the test.

The unittest module arranges tests inside functions that start with "test_" that live in a class.


This is a draft remember? So what is there left to finish in this doc?

[TODO: empty todo_test with whole class]
[TODO: image of what is wrong, moire pattern]
[TODO: show how we test that function]
[TODO: show pull request travis/appveyor running tests]






Wednesday, April 11, 2018

Berlin to Copenhagen by bike.

There's a bike path running from the north of Europe all the way to the south. Well, it's not completely finished yet, but it goes most of the way. It's part of the EuroVelo project, which aims to have a high quality cycling network made of 15 routes all around Europe completed by 2020.

All my clothes and possessions for a seven day trip from Berlin to Copenhagen. I was just going to tie a sack under my seat... but was convinced that perhaps these special bags are a good idea after all.
One part that's done well is the Berlin to Copenhagen leg(I guess not so surprising since they are two cities that are really into supporting bikes).
Scroll down on these photos someone took, if you want a look.
http://www.slowtravelberlin.com/in-photos-berlin-copenhagen-cycle-route/

Except, I'm going to do it at the end of April, not in winter like them. Which is usually the most dry time of year, and with a temperature that's not so hot or cold(usually). Fingers crossed it will be dry and warm. It will probably take 7 days if I can ride fast enough each day (about 100km). Eeek. And this part of the route (like many other parts) has a really nice website http://www.bike-berlin-copenhagen.com/route/brandenburg-stage
Going to go very minimal, and take my fast(ish) road bike. With just three sets of clothes(including the ones I'll be wearing), a netbook, and a toothbrush. It probably would be nice to have a tent, but I'll pass. Instead I'll sleep in the forest. If it rains, I'll just get wet.

Just kidding (or am I?). I'll probably cheat and stay in a hotel some nights rather than sleep in forest. I chose the laptop over the tent, because it's nice to be able to sit by a river in a field with no one around and write some nonsense.

A tiny paper notebook and pencils for drawing will also make it into my bike-pack.

Already I've started training a bit, since I didn't ride my bike for much of the winter. The first 6 hour test ride I did left me sore and tired. But that was going from almost zero riding over the last four months and straight into that. Not the smartest idea to go from nothing into such a long ride... but very enjoyable none the less.

Learning from long practice rides around Berlin. Sitting by a lake.
So what have I learnt so far?
  • Padded pants are a good idea.
  • Water is important.
  • Sunscreen is important even on cloudy days.
  • I am not fit.
  • Riding around is one of life's pleasures.
My next long practice ride is going to be with all of my gear on the bike. I expect to learn more. For example, I'll time how far I can go and at what speed. And I also want to try riding in some heavy rain - to test if my bags are actually waterproof. But unfortunately the weather has been pretty nice recently.

Other than experiments I plan to do, I'm reading about what other people do on their long bike rides. And also talking with people. But it's always hard to learn lessons from others, and the best way to find out what is important to me is to do practice rides.

I think I should be able to ride five hours each day with breaks in between, and still get up to my 100km target. But unsure really. By then the days should be quite long, and the nights with a fair bit of light.
  • Berlin day length: 14:30
  • Copenhagen day light length: 15:30
Spring flowers such as 'Tommies' and 'Snow drops' are popping their heads out for spring. Looking forward to seeing all the different landscapes along the way.
With 14-15 sunlight hour days, there's lots of time for slow riding if wind is in my face, for detours, and for sits under trees drawing. By not using the side bags, and not carrying all that much weight I should be able to go a lot faster than other people I've talked to who have done the ride. Their bikes were pretty bad, and they carried a lot more weight than what I'm going to.

I don't really have a good solution for route planning. I have a route I'd like to go, but I can't really put it on google maps easily, so it's visible on an app. So that's another thing I need to find out about. Luckily I've done lots of map development before... but I hope there's not much coding to do such a simple task, that surely other people have done before.

Also, I'm a bit worried about riding in the rain. On average it rains 8 out of 30 days that time of year. So it's quite likely I'll see rain. Generally, I just don't ride my bike if it rains. But I'll have to I guess, because 'waiting it out' doesn't work if it's raining all day.

Tuesday, April 03, 2018

pygame.Surface.blits

Drawing many things at once is faster than drawing things one thing at a time. So, today I made a function to do this with pygame.

It seems to take between 82%-85% of the time it takes to draw things one at a time. Which considering blit is often the bottleneck in many pygame apps is a pretty nice improvement. It will be landing in the pygame 1.9.4 release.

More info in the blits Pull Request.

Moss box

I've been looking to get more greenery into my studio, and I kind of like moss. It reminds me of when the times in my childhood running around rain forests barefoot.

Luckily, a good friend has a whole bunch of it growing out the back of her apartment. And it's likely to be destroyed soon by workers. Which made me not feel so bad about digging some up.


I scavenged some egg cartons

First I found some egg cartons to use as a base. Underneath the moss is a good few centimeters of dirt.

Unlike most of the wall moss I've seen around the place, this moss is alive. I was sort of horrified to learn that people buy dead preserved moss, and glue gun it into place. But this stuff is alive. Sometimes looking stuff up on Youtube kills the magic. Now I don't think wall moss is anywhere near as cool as I once thought it was. Thanks internet.

Here it is sitting on my standing desk.

Next I found an old wooden box. I lined the bottom of this with some plastic I had about the place. Yuk, I know. However, it should hopefully stop any water leaks.

There's two half egg carton containers I found (to hold 12 eggs) being used to fill the box. With spaces around the edge.

The nice thing about having it in a small box is it's easy to move about the place. So I can put it on top of my desk, or even under the desk where my bare feet are. My next project is probably to grow some green grasses in a box for that grass-on-feet-inside feeling.

Can you see the cardboard carton?
I'm not so sure if it will survive, but I'm hoping it will grow. So far I've just been spraying it with water, and occasionally touching and smelling the moss. Which takes me back to the rain forest a bit.

Friday, March 23, 2018

Investigating pypy frame drop

pypy has spikes in time it takes occasionally when using cpyext, otherwise known as pauses.
This is because it's deallocating lots of CPython objects all in one frame, rather than incrementally. This is likely to be addressed in a future release of pypy.

Mainly it's a problem when creating and deleting lots of CPython objects (like pygame.Rect). Not pure python objects.

To work around it for now,
  • the src/rect.c has been changed to have a free list of python objects, so it doesn't alloc and dealloc them. Instead it maintains it's own list of rect PyObject pointers, and reuses them. This is done in PR #431
  • Use PYPY_GC_NURSERY=1m incminimark environment variables when running pypy.
Below are timings of the default pygame, and default pypy (at time of writing). Then a benchmark showing the improvement with the rect freelist implemented in rect.c inside pygame. Then we also improve things by setting PYPY_GC_NURSERY=1m environment variable to change the behavior of the GC. I'm not aware of an API to do this within pygame itself unfortunately. Finally there are some cpython 3.6 timings for comparison.

hacked up benchmark.

The hacked up benchmark is in the branch pypy-hack-frame-bench of the pygame repo. It's based off examples.testsprite.

Because pypy currently doesn't compile matplot lib you need to run it in two steps. First to do the run (let it run for 20+ seconds). Then to load the data pickle, and show the graph.

pypy examples/testsprite.py -plot -noupdate_rects -width 320 -height 200 100
python examples/testsprite.py -plotpickle 

To run with the GC tweak...

PYPY_GC_NURSERY=1m pypy examples/testsprite.py -plot -noupdate_rects -width 320 -height 200 100
 
 

graph colors

  • Blue is the first 1200 frames
  • orange is the last 1200 frames of the run.
This is useful to see if there are any jit warm up affects happening, or different behaviour over time. (doesn't appear so).

Time per frame on pypy 5.10.0.

For this example we don't see any 'jit warm up' behavior.
screen shot 2018-03-23 at 07 21 05

Time per frame on pypy 5.10.0, rect freelist, standard GC.

screen shot 2018-03-23 at 12 05 42

Time per frame on pypy 5.10.0, rect freelist, PYPY_GC_NURSERY=1m.

screen shot 2018-03-23 at 12 04 42
This final result for pypy 'ok' because we want under 0.0166 seconds used per frame. Improvements inside pypy itself in the future should hope to remove many of these spikes.

Time per frame on python3.6

Here we see both the average time per frame and maximums are smaller on python3.6.
screen shot 2018-03-23 at 12 22 02

Time per frame on python3.6, rect freelist

It seems the freelist for rects on python3.6 has a minimal affect.
screen shot 2018-03-23 at 12 20 12

Time per frame on python3.6, gc.disable()

Here we can see using gc.disable() seems to have no affect in this program (on python 3.6). I also tried this on pypy, and it had no affect either (on this program).
screen shot 2018-03-23 at 12 25 14
Other things I tried was to do a gc.collect(0) just before display.flip(), however this didn't appear to help in this case. This is because we can know if we have 5-10ms free waiting for the display VSYNC to flip (when running at 60Hz). Perhaps this can be attempted again as the cpyext GC is improved.

Thursday, March 22, 2018

Windows pypy pygame build for testing.

How to install pypy (fast python written in python) on windows and use the pygame dev build.

pypy running pygame.examples.aliens on Windows

1) Get pypy. It's just a zip folder with all the stuff inside.
https://bitbucket.org/pypy/pypy/downloads/pypy2-v5.10.0-win32.zip
Unzip, and put into C:\pypy, so C:\pypy\pypy.exe exists.

OR get the pypy3.exe from https://bitbucket.org/pypy/pypy/downloads/pypy3-v5.10.1-win32.zip
Unzip, and put into C:\pypy3, so C:\pypy3\pypy3.exe exists.


2) Set the PATH environment variable, so that pypy.exe is found in your command prompt.
https://www.opentechguides.com/how-to/article/windows-10/113/windows-10-set-path.html

Or you can just do this each time you run a new cmd prompt.
set PATH=%PATH%;C:\pypy\;C:\pypy3\

Or just cd C:\pypy to test it out.

3) Ensure pip is installed (pip is a weirdly named tool for installing things).
pypy.exe -m ensurepip
4) Install a dev build of pygame.
pypy.exe -m pip install pygame --pre

5) Check if pygame is installed properly by running an included example.
pypy.exe -m pygame.examples.aliens

ps. use pypy3.exe for python 3 version of pypy above rather than pypy.exe(python 2) if that's what you like.

Tuesday, March 20, 2018

pygame on pypy usable

Hi,

TLDR; I'm at the pypy sprint, and working through the remaining pygame-on-pypy-cpyext issues.
https://youtu.be/WN1slc5O8os


Surprisingly to me... it's already usable. That is pygame (same one that runs on cpython), works on pypy through its C extension API. pypy has good support for the CPython API (through a recompile) now. PyPy is the python language with a fast JIT, so your code can approach C speeds. And in some cases it can be faster than C.

There was an issue with events stopping keyboard/mouse/etc from working. Lots of details in this issue describing the changes needed, so I hope other extensions encountering this will find it useful.
https://github.com/pygame/pygame/issues/419
But now that's fixed, every pygame app I tried on it has worked.

Cat sitting in the fog and snow of Switzerland

Why is this exciting?

This is exciting to me because:
  • pure python code being fast on pypy(after warmup), also mixed with the fast bits in C/asm.
  • cpyext is getting faster in pypy. There is already work and discussion towards it being faster than CPython.
  • maintaining one pygame code base is easier than maintaining several (pygame cffi/ctypes/cython, ...).
  • with one code base it should be fast on both pygame, and pypy(in time).
Here's our old pal solarwolf from early 2000s running on pypy.
https://youtu.be/WN1slc5O8os
Still lots of work to do (especially around PixelArray buffers and such). Then of course, there is the issue of binary wheels, so that  `pip install pygame`  works without needing to compile things from source.

How is the speed? (when do we use this tool? Is it fast enough?)

If your code is already quite well optimized, and not spending much time in python, you can't expect to see an improvement. However, if you are pushing boundaries in your python code, you can expect very good increases.

Some examples where you can expect it to be faster:
  • if profiling and a pygame function (like blit) isn't at the top of the slow bits.
  • collision detection (if you aren't using fancy algorithms).
  • a pure python ray caster.
  • writing a music synthesizer in python python.
Where it can be slower.
  • if you are going into C code for a lot of small operations. Like when using lots of pygame.Rect in a tight loop. This is because (currently) the cost of going from PyPy code into and out of CPython API code (like pygame) is a bit slow.
For me, I'm interested mostly in this for a physics art project which was really slow, and also for a software music synth written in pure python. Even more interesting is running pypy as a separate process for these tasks, and run the gui process with CPython.


Ray tracing 3D scenes. 

Here I run a python and pygame python ray tracer by Ian Mallett.

Ray Tracer to for realistic 3D lighting.

On PyPy - 18.6 seconds.
On Python 2.7 - 9 minutes, 28.1 seconds

That's 30x faster.  Making many more things possible in python - and at speed.


The fog has lifted at the Leysin pypy sprint.

Thursday, March 15, 2018

Drawing sound (as a waveform in pygame).

I recently gave an example of pygame sound generation examples, and a few people asked for more. So... here we go!

There's an infinite number of ways to visualize sound. A classic way is to draw it as a waveform.
Sound samples drawn as a Waveform. Scaled into a 320x200 sized Surface.

A sound could be made up of 44100 samples per second. Where each sample is often a 16 bit number (or 8bit or a 32bit floating point).
Python comes with a built in array for efficiently storing numbers. We can store samples in there, with integers between -32768 and 32768. This is a signed 16bit(2 byte) number. Two to the power of 16 is 65536, and if we divide that by two we see the minimum and maximum value that can hold. pow(2, 16) / 2 == 32768.

Below is the annotated code, also with an example of an array of samples representing a square wave.
We loop over the samples and draw a line from the previous sample to the current one. 0 to 1, 1 to 2, 2 to 3, ... N-1 to N.


You can also find it in the pygame CookBook at https://www.pygame.org/wiki/DrawWaveform
(maybe easier to read than on this blog)


# python built in array type is useful for storing sound data.
import array
# a square wave tone, with sample values between -32766 and 32766.
samples = array.array('h', [32766, 32766, 32766, 32766, 32766, 32766,
32766, 32766, 32766, 32766, 32766, 32766, 32766, 32766, 32766,
32766, 32766, 32766, 32766, 32766, 32766, 32766, 32766, 32766,
32766, 32766, 32766, 32766, -32766, -32766, -32766, -32766,
-32766, -32766, -32766, -32766, -32766, -32766, -32766, -32766,
-32766, -32766, -32766, -32766, -32766, -32766, -32766, -32766,
-32766, -32766, -32766, -32766, -32766, -32766, -32766])


def scale_samples_to_surf(width, height, samples):
    """ Returns a generator containing (x, y) to draw a waveform.

    :param width: width of surface to scale points to.
    :param height: height of surface to scale points to.
    :param samples: an array of signed 1 byte or signed 2 byte.
    """
    assert samples.typecode in ['h', 'b']
    # precalculate a bunch of variables, so not done in the loop.
    len_samples = len(samples)
    width_per_sample = width / len_samples
    height_1 = height - 1

    if samples.typecode == 'h':
        # for array typecode 'h', -32768 to 32768
        factor = 1.0 / 65532
        normalize_modifier = int(65532 / 2)
    elif samples.typecode == 'b':
        # for array typecode 'b', -127 to 127
        factor = 1.0 / 256
        normalize_modifier = int(256 / 2)

    return ((
        int((sample_number + 1) * width_per_sample),
        int(
            (1.0 -
                (factor *
                    (samples[sample_number] + normalize_modifier)))
            * (height_1)
        ))
    for sample_number in range(len_samples))

def draw_wave(surf,
              samples,
              wave_color = (0, 0, 0),
              background_color = pg.Color('white')):
    """Draw array of sound samples as waveform into the 'surf'.

    :param surf: Surface we want to draw the wave form onto.
    :param samples: an array of signed 1 byte or signed 2 byte.
    :param wave_color: color to draw the wave form.
    :param background_color: to fill the 'surf' with.
    """
    assert samples.typecode in ['h', 'b']
    if background_color is not None:
        surf.fill(background_color)
    width, height = surf.get_size()
    points = tuple(scale_samples_to_surf(width, height, samples))
    pg.draw.lines(surf, wave_color, False, points)

# Here we should how to draw it onto a screen.
waveform = pg.Surface((320, 200)).convert_alpha()
draw_wave(waveform, samples)
screen.fill((255, 255, 255))
screen.blit(waveform, (160, 100))