Tuesday, January 31, 2017

Using a common file layout lets us create distributables more easily. Part 2.

This article is part 2 in a series about improving python game distribution. In part one I suggested that python games should be packaged as python packages. Part three is about where the code & data things are. More discussion is happening on the pygame mailing list.

Making apps for platforms more easily.

Android, windows, mac installers, pip wheels, snappy... the list of platforms goes on. All that is quite time consuming to setup. So how do we make this easier?

Have a standard package layout, and build tools which work on that layout.

The Python Packaging Authority has put out a git repo with a basic python package. I think we should create one based on that and use it for the 'skellington' that pyweek uses.

Skellington is 'base code', or boilerplate code used in the pyweek competitions. It makes doing things like creating windows .exe files, and such easy. Because it does all of the work to configure things correctly. If we have a standard package layout, it is easier for us to build tools to share. Distribution and packaging tools.

The problem with creating packages for many platforms is that you need that platform to create the packages for it. So if a developer is on windows, they find it difficult to package an app for mac and linux.

Free build bots will allow people to have packages created for them.

If their games are in a standard layout, then we can even make a central repository that creates packages for other games automatically (without them having to setup the buildbot accounts themselves).
Travisci - linux, mac, and android.
Appveyor - windows.

Of course people can do all this themselves, using scripts we share. But a centralised build bot will mean that only needs to be setup once for everyone.

What needs to be made?

  1. A fork of pypa sampleproject, merged with skellington.
  2. A library for creating .exe, .dmg etc using different platform packaging tools.
  3. Configuration for the buildbots for different platforms (.travis.yml etc).
  4. Using the sampleproject we make a cookiecutter template from the sampleproject.
Where? Probably on the pygame github organisation.


What will people have to do to build their project?

 These are approximately the steps people will need to do in order to package their app.
  • python3.6 -m venv anenv
  • . ./anenv/bin/activate
  • pip install cookiecutter pygame
  • ...generate base code... cookiecutter pygame/skellington mynewgame
  • ... fill in meta data, like game name, url etc...
  • ...upload project to github...
  • ...create travisci and appveyor accounts...
  • ...enable repo for travisci and appveyor ...
Some of these steps can be automated, and some can be removed.



Continue to Part three where the code & data things are.

  1. https://github.com/pypa/sampleproject/
  2. https://pyweek.org/s/help/#what-to-submit-as-your-entry
  3. https://bitbucket.org/dr0id/pyknic/src/92449d2874e031da948fd7c537df0bb7106b2676/pyknic-skellington/?at=default




Promoting pypi for python game releases. Part 1.

This is a statement of intent similar to what I wrote to the pygame mailing list some weeks ago. That I think the python game community should promote packaging their games as python packages. When I wrote to the mailing list, not everyone was convinced, and some people had other ideas. So I'm going incorporate their feedback, and to try to be more detailed on the full plan of where I think we should go, what benefits this provides the python community as a whole, and the benefits it provides game developers.

This is the first in a series of articles about making python game distribution better.

“ the python game community should use python packages ”

With all the great work from lots of people pygame is often easily installable via pip - the standard python packaging system. We still have some issues, but it works quite well on major platforms. See "How we cobbled together a free Spectacularly Adequate Build Page." for more details on how we package pygame for pip.

Now our games can be installed with pip too! Here's an example game I started packaging up, now available on the python package index. Note, it will install pygame for you.

pip3 install --pre solarwolf


solarwolf - a game I packaged up for testing

Since many people enter the python world via games, it makes sense that they get used to publishing python packages as well. I've sat in python user groups, and still 80% of the room has never published a python package despite many of them working with python every day - for years. Let that sink in for a bit... most python developers never publish packages.
“ newbies to python often come in through game development
Currently most people making games find packaging them up for distribution very time consuming, and also pretty much impossible to do for multiple platforms. For game libraries, people still do not use pypi so much. It's easier for them to just put the source tarball up on a website and tell people to download it.

As a game developer why should I use pip? Firstly, there is a very large audience of people who can install python games - python developers. This is mainly your peers, but python also comes installed already on many platforms (linux, Mac, etc). You can manage dependencies more easily (so you don't need to worry people don't have one of your other libraries installed). You don't need to worry about the platform issues of binaries so much. If they have pip on their platform, then they can install your game. Other benefits of publishing on pypi include syndication, since many people tweet and copy all the releases on pypi. Another benefit is all the infrastructure work that goes into pypi, CDN networks and such.

I suggest efforts should be applied to:
  • updating tutorials, and spreading the idea of publishing python games to the cheeseshop (pyweek, pygame.org tutorials, external tutorials, books, youtube videos)
  • base code for a pygame game in a standard structure (skellington, cookiecutter etc)
  • contacting other python game communities to suggest pypi should be a priority
  • making the cheeseshop/pypi itself a better platform for game publishing needs
  • making python packaging easier, by fixing python packaging warts which stop newbies.
What pypi doesn't do currently? It doesn't do many things that a good game release system would do. Video/youtube links, and even screenshots aren't available. Discussion has been disabled (they found it way too hard to moderate). Even ratings are not on there (which can help for discover-ability). Another issue is that closed source things aren't really looked apon nicely there(but it is allowed). Finally, packaging in python still isn't the easiest thing (it's definitely not as easily as uploading a zip file, but it is waaaaaay nicer now than ever before).

Any work that goes into making the packaging better for games helps out with other python game communities as well. We can perhaps even gain allies from the other communities to help improve things for python games in general.

Here are where the pypi projects live.


The second part of this article series is about what else we can do to improve distribution. Topics coming up in this article series include - a new "skellington" (from pyweek) based on the "sampleproject" (from pypa) with everything setup, and using appveyor/travisci/etc to build binaries on mac/linux/windows/android for everyone, as well as a list of python packaging warts I've gathered from people trying to package their python modules.

Sunday, January 29, 2017

Pyweek 23 - the python community game development competition.

pyweek registration is open. For the biannual game jam. http://www.pyweek.org/23/



Which means you can join, and put your self in a team, or join up as a solo entrant.

Spend a week(part time) finishing a game using python. Sunday 19th of February to Sunday 26th of February.

It is inspired by the ludumdare 48h comps, but people only use python, it is a week long, and there can be teams. Over 100 entrants joined in on the fun previous competitions.

Enter to have a chance to prototype your next game, or see if working in a team will work on a small project. Or just have a break, to get your creative juices going, and to feel all of the energy of 100+ people simultaneously feeding off each others creations. http://www.pyweek.org/23/

It's also a great way to learn, and have fun with python. It's possibly the best way to improve your programming skills, and game making skills there is.

Timetable
Friday 2017/01/20Pre-registration underway
Sunday 2017/02/12Theme voting commences
Sunday 2017/02/19Challenge start
Sunday 2017/02/26Challenge end, judging begins
Sunday 2017/03/12Judging closes, winners announced

Thursday, January 26, 2017

How we cobbled together a free Spectacularly Adequate Build Page.

This is a story of how we cobbled together a collection of free build bot apps for pygame - the python game development library. So the internet could build our software for us. Hey, many Free and Open Source projects might find this useful too... I thought... so here is the story.

Continuous integration is great. It's even greater when you have dozens of platforms, and many versions of a python interpreter you need to have things run on. It means your python 2 contributors on linux can see that their change broke an OS X build on python 3. Every change you make things get built for all the different platforms, and tests are run, installers are built, and packages are made.

In 2008 the pygame project had the great fortune for 'thorbrian' and some other contributors to set up a build page for Windows, and Mac. Two platforms which had sort of tricky python build setups not so commonly understood in the FOSS world.

"The Spectacularly Adequate Automated Pygame Build Page" was born. And it was spectacularly adequate. Everyone rejoiced.

We rejoiced in how spectacularly adequate it was.

So now people could download builds when we made changes. Then they could test them on their weird windows machine with directx N, and some strange AMD card they found in a rubbish bin somewhere.

Occasionally it would connect to the SVN code repository (yes, it was that old) and it would grab the code and do a build. Awesome. This was some years before things like TravisCI provided free infrastructure to the world (massive shoutout and thankyou the the travis mob).

But at some point a few years later the machines stopped building things. They crashed. Unfortunately no one was able to replace them.

So, a couple of years ago we began the slow process of finding a replacement. There were a few missteps. Of course! Trying to setup our own virtual machines. That worked for a little while. But VMs were changing at such a rate, and no one really had space to host machines that worked all that well. We even started on a project to let anyone upload a build. But this never was finished. We tried to get in contact with a few other python projects, and open source projects but no one was all that interested in replying to emails it seems.

So our project sort of stalled. This wasn't the only reason, but one big one. Without builds being made automatically we went back into the world of things breaking all the time, and waiting for random do-gooders of the internet to build packages for us.

Launchpad - Ubuntu GNU/linux.

Launchpad is a sort of code repository in the sky things for Ubuntu. You can use it for other things, but this is the main thing. It used to be based on the Bazaar version control language, but actually supports other things too.

Heard of PPA in the ubuntu world? That's what launchpad does, you can host .deb packages for linux. Not only can you host them, but you can get them built for you. Even on some strange architectures like PPC, and ARM. Pretty damn cool.

You can link up your code to be built by launchpad. Note, you'll need git, and not mecurial (last we tried it didn't work). So, because we are still using mecurial version control (hg) we had to have a git mirror of our hg repo.
https://code.launchpad.net/~pygame/+recipe/pygame-daily

Look at all the builds! This is just python3.
So now we have more than 16 builds happening just for python3. So we have PPA archives people can add to their ubuntu install, and pygame can be updated every time we make a change (which passes tests). Launchpad is a bit rough, but because it's used by Ubuntu I hope it will be around for quite some time.

Of course you need to have debian packaging files setup for your program to be packaged. But if you (or a maintainer) has done that, then it should be quite easy to do. Someone may already have even set up a PPA for their own personal use. So you might have zero work to do.

Another challenge we had is that a 'build badge' doesn't exist. You know the type which tells you if a build is breaking or not? So we had to write a little script which scrapes the build pages for us and creates a little svg file. Another cool thing about launchpad is that it links to the Ubuntu and Debian bug trackers. So you can search in there for any patches and bugs people contribute to those projects. (Often it is far easier for people to submit a patch or bug report to their distro rather than upstream).

Thanks launchpad, and thanks Canonical. You provide a great service to the world. Trillions of industry running on your backs. I owe many {insert_beverage_of_choice} to the people who keep this going.

Appveyor.

We use Appveyor or its windows build support. It also integrates really nicely with bitbucket, so we don't need to use the github mirror for this one.

Our build scripts for appveyor are in the appveyor directory of the pygame repository. We don't build all of the dependencies for windows automatically, since that would take quite some time. Lenard has produced some 'prebuilts', which are downloaded by Appveyor for us (see appveyor/install.ps1).

Thanks very much appveyor, for allowing open source projects to use their build system for free. The windows using world owes you!

Travisci

Travisci works with github, so that when a commit happens the builds are done and the tests are run.

We use Travisci for our mac builds and our manylinux builds.

Manylinux is a fairly new thing, which is a binary that will work on many versions of Linux. It does this by not linking to new library symbols in. You can find binary wheels on pypi marked with either  manylinux1_i686 or manylinux1_x86_64.

All the travisci related config for building pygame on there is held in .travis* files starting with .travis.yml.

homebrew

Homebrew is a package manager for Macs. It provides build scripts and binary packages so people don't need to build everything themselves from source all the time.

We use these scripts in our builds for the macs. This lets us use the mac build scripts in order that we can try to keep maintenance for these scripts inside homebrew itself.

Thanks very much to the homebrew pygame maintainers!

Other distros.

Other software distributions also build pygame for us. From raspbian, Debian, to Fedora, to homebrew, to exotic rewrites of BeOS . Each of these packagings are done by different people. We try to keep in touch with things going on by looking into their bug trackers from time to time.

A list of pygame bugtrackers for different distros is kept here: https://www.pygame.org/wiki/patchesandbugs However, this does not show the build status, or allow you to download packages. We don't maintain a list like that (but it's probably something we should do!).

Bringing it all together?

We put some badges into the readme, so that you can see if things are building or failing the build. You can also go from those links to download binaries. Also the Hacking wiki page is where we keep links to all the build pages and instructions for pygame developers.
The build badges in the readme, linking to travisci, launchpad, and appveyor.

Now, once again people can contribute to pygame without having to set up a dozen build environments. It is Spectacularly Adequate.



Thanks to Lenard Linstrom, Paul Craven, Thomas Kluyver, and others for getting this all up and going. As well as to the people who package pygame for all other platforms.