Gradual Packaging, with python - Part two

Gradual Packaging - allow packaging simple python code with very little effort. Later on (if needed) you can create a more polished package. Follow conventions, support the simple cases. If you need to go out of the simple cases, allow a path towards better packaging.

Evolution of your python code.

1. in your ipython session messing around.
2. paste it into badnamechoice.py -> (everything starts with a bad name in a single .py file)
3. test_badnamechoice.py -> (then you add a test. Or did you do this first?)
4. renamed.py -> (now you rename the file to something better)
5. afolderoffiles/ -> (now there is a folder of files)
6. add docs/ and .travisci CI files
7. configure coverage testing
8. configure an annoying mypy static type checking option
9. add flakes testing config tweak
10. support some weird debian derived distro
11. appveyor CI added next to travisci
12. pytest config (change because... ARG reasons).
13. add a requirements.txt, and modify your setup file to have them too.
14. remove testing stuff out of requirements.txt into requirements.dev.txt
15. ... (config tweaks happen over the weeks and months)...
16. ...
17. Giant ball of 30 config file soup. -> (We have arrived at the modern repo!)

"Cool hackathon! What did you do?  - I packaged my python code, added test support, setup a nice git repo, a pypi page, a readthedocs page, configured travisci, search the internet for hours on how to upload a python package. Nice one - look at my finished app."

Get something out the door really quickly, and gradually improve the packaging. Or maybe your code isn't all that special and it's ok using the default packaging choices.

So, how do we support this workflow?

I started making a tool for doing the zero config releases. It's not done yet, but it is almost able to release itself. This follows on from a series of blog posts and experiments on:

• "What would python packaging zero look like? - part zero?"
• python packaging zero - part one

My mission is to improve python packaging for newbies, those in the digital arts community. People using pygame. Also for 90% of python programmers who never release a python package, and yet work with python every day. This is based on feedback from different people in the python community who teach python programming, my own experience teaching it and mentoring teams. The scientific python community is another group which finds it hard to create python packages.

The humble experiment so far.

https://github.com/illume/pyrelease

 (still not sure if it's a good idea... but we will see)

Usage:

pyreleasase

File layouts (based on simple and common layouts often used):

-----

singlefile.py
----

mygame/game.py

data/image.png-----

singlefile.py

test_singlefile.py

-----

singlefile.py

tests/test_singlefile.py

-----

The basic steps at runtime are these.

1. Gather facts, like author name and email. (much like ansible if you know it)
2. Create setup.py, setup.cfg, MANIFEST.in files in a temp folder
3. build sdist in that temp folder
4. Upload files to pypi (eventually other release targets, including pyweek)
5. tag a new version in git (if they are using git)

Tell the user what is happening.

The tool should also teach people about python packaging. It should allow you to see the generated setup.py file, setup.cfg, and MANIFEST.in. It should point to the Python Packaging User Guide. It should print out the browser for creating a pypi account, and tell people what config files they need to fill in. It shouldn't require git, or a github account, but support that workflow if they do.

Tell them why adding each bit of config is useful.

A single .py file is the smallest package of python code.

The simplest package of python code is a .py file. It can be copied into a project and it can work. Or it can be copied into the site-packages folder, and the module will work. People upload python code to share all the time - on websites all over the internet.

Pip supports installing from git repos, from hg repos, from alternative different package indexes, from requirements.txt files, and is adding support for .toml config files. It supports installing from folders on the file system. It supports a .egg file format. It supports a .whl file.

But it doesn't support the simplest, most elegant, oldest of python packages - the single .py file.

Where does the packaging metadata live?

Technically python doesn't need any metadata to install a simple .py file. I mentioned in the first two parts of this series various places where data can be gathered. ~/.pypirc files, .gitrc files, .hgrc files. It can find versions as git tags. It can find __author__, __license__, __version__ variables inside files. description, and longdescription are found at the top of the .py file in the docstring.


It can use what packaging data that has been added already. If you add a requirements.txt, then we should use that. If you add a README.rst, then use that.

Gradually add packaging information as you go, as you need it. Not when your packaging tool thinks you need it.

Why not template generators like sampleproject and cookiecutter?

These tools have their place. They are good if you want to tweak all the config in there, and you know how all the tools work. However, based on feedback from people, it's all too complex still. They want to a couple of .py files and that's it. Renaming code easily is important for some types of python code - especially when you don't even know where your experiment is going. Naming things is hard!

They came to python for the elegance, not the packaging config files.

But they still want to share code easily.

Where next with the pyrelease experiment?

First I want to get the tool into shape so it can at least release itself. I'm going to Gradually Package the humble pyrelease - by keeping it to one file from the beginning :)

It should support single file modules, packages, and also /data/ folders. The simple layouts that people use. As well it supports making a script automatically if it finds a main(). As well it finds dependencies by parsing the python code (I'll add requirements.txt later). So if you import pygame, click, flask etc... it adds them to install_requires in the setup.py.

• I want to add logging of facts. eg. 'Found author: "Rene" in ~/.gitrc". Could not find author in ~/.hgrc
• Suggest additions for missing things. eg. How to create a pypi account, how to make ~.pypirc, git.
• Have to make uploading to pypi easier, especially when things go wrong.
• thinking of removing the setuptools dependency... only thing I use it for is find_packages so far. I've been writing another more modern implementation of that anyway. Remove distutils dep too? [0]
• Pynsist support (or py2exe, pyinstaller, whatever)
• tests, and tests against test . pypi . python . org
• "add setup files into this repo" for when the tool fails to be good enough.
• notice telling people to go to packaging. python .org if they want more
• decide on convention for screenshots (probably screenshots/ folder)
• bitbucket, and better hg support.
• pyweek upload support.
• try releasing a few more games with it.
• watch some other people try and use it.
• keep blogging, and getting feedback.

​A nice thing so far, is that I was easily able to rename the project three times. Just by renaming files and folders. It was first called 'blabla.py' then 'package.py' then 'release.py', and finally 'pyrelease.py'. Normally you'd need to modify 10 config files when things happen.

Anyway... I'll try some more days on it and see how it turns out.


[0] Thomas Kluyver - an explanation of some philosophy on 'flit' the tool for minimal config distribution that doesn't use distutils or setuptools. https://github.com/takluyver/flit/pull/97#issuecomment-270984130