Sunday, February 25, 2018

Why can't we turn off HTTP?

Currently, I'm sitting in a library not able to access HTTPS (probably... I guess it depends when you're reading this) for a particular website.

I like to spend time in libraries. Especially libraries in exotic far off places. Surrounded by book-smell, and curious people learning. Unfortunately some libraries have shockingly bad internet. Some of them decide they will filter which websites people can look at so that they are safe from viruses and nasty stuff which damages their computers.



It's exciting that 60-70% of internet connections to some websites are encrypted these days. But why not turn HTTP off and force everyone to use HTTPS?
Indeed. Why not turn off HTTP completely? I'll try to answer that here (for pygame.org; your website may be different).



Unfortunately these filters in libraries are sometimes pretty terrible and ancient. And when you travel, and spend time in libraries and use hotel internet you begin to notice that the internet that you know at home or work is kind of not the same all around the world. A shocking revelation I know things are different in different places. Soon there will be research funded to confirm this for you in ten years.

One thing they do is have proxy servers that jump in the middle between the browser and the server. Sometimes in strange ways that don't quite work.



Oh. So this is why I can't internet-all-the-things whilst sitting in a hammock on a small island.



Universities are another type of place I've had bespoke-internet-experiences.

So, this is one reason I like to still offer things over normal HTTP, because some people are blocked from HTTPS. A person can still choose already to use HTTPS (which technically still does have a MITM attacks going on from university/corp/country proxies and the like). We could also use HSTS (https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Strict-Transport-Security) which is the proper solution rather than redirects, but that means HTTP doesn't work. Because leaving HTTP on means it's possible HTTPS can be worked around.

If you're privileged enough to have good internet then, please take our EFF friends advice and use the https-everywhere addon for your browser. Unfortunately some things will still break for you. https://www.eff.org/https-everywhere

However, could we still do better? And still make it accessible for those people who are stuck with horrible internet connections. Yes, of course!

So, now pygame.org URLs permanently redirect to https://www.pygame.org/, and the login URLs even on non-HTTPS point to HTTPS. Additionally, I've updated lots of links in the docs and on the internet to HTTPS URLs, and will continue doing that where possible. Search engines always link to HTTPS (if available) it seems, and adding canonical HTTPS link tags in the headers is being tracked here: #34



I would like to update hundreds of mixed HTTP/HTTPS documents to turn all the HTTP links to new HTTPS links... but instead I chose life. Is there a header for that? Yes there is... pop on over to https://w3c.github.io/webappsec-upgrade-insecure-requests/ and learn all about it.



Browsers are now rolling out HTTPS upgrade headers, where they send a hint that they can go to a HTTPS URL when one is available. https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Upgrade-Insecure-Requests
You can also make a content security policy which allows the browser to upgrade old unecrypted links to HTTPS ones. You're telling the browser "Dear browser, everything should work with HTTPS, but I haven't updated all the URLs yet". Seems to be at 77% of browser support at the moment, and even the latest version of Edge supports it. https://caniuse.com/#feat=upgradeinsecurerequests

But remember, even if a browser supports it, it doesn't mean it hasn't been configured to use a proxy in the middle which has issues with HTTPS. Or that 500ms latency turns those several extra back-and-forths HTTPS requires into an unbearable thing to use. With three 500ms round trips, that turns into more than 1500ms. Compared to 20ms*3==60ms for someone on a fast connection nearby.

The reason we can't have zero round trips with HTTPS everywhere I hear you ask? That should would be good for people in far off lands with bad internet connections. Let me tell you my friend those same proxies, and Idiot of Things with ancient buggy HTTPS stacks. Blame your old printer.
 
But I've enabled 0-rtt and HTTP/2 anyway, and the reason is that those same broken internet connects are going to be broken anyway. Might as well make it slightly faster and safer for everyone else, and the broken connections can use HTTP which they would have to do anyway.



Is TLS fast Yet? Yes. Sort of. https://istlsfastyet.com/



With these changes, there's been a 30%(handwavy-made-up-stat) increase in HTTPS connections, and now the majority of connections are using HTTPS. Hopefully as more links are updated (18 years of links on the interwebs might not ever be upgraded).
Cloudflare (which pygame.org is using for HTTPS) supports HTTPS on some pretty ancient computers (we are paying for the pro plan). Even my android 2 phone works with it. https://support.cloudflare.com/hc/en-us/articles/203041594-What-browsers-work-with-Cloudflare-s-SSL-certificates-
 
www.pygame.org is still open for business fun on HTTP however, even though most people probably won't go there 'soon'.  The people in far off places with weird internet connections, and those retro computing people with the browser user agent string "AmigaVoyager/3.2 (AmigaOS/MC680x0)" will still be able to download their files.

pygame.org is //retro computer friendly// after all. But we're also friendly to people in far off lands sitting in hammocks and to people who like book-smells + learning.

Thanks for reading.

Saturday, February 24, 2018

#RaspberryPi binary #pygame wheels for pip rolling into a #python cheese shop near you.

It seems there's some sort of giant build cluster for Raspberry Pi that builds binary wheels (python package files, that don't need to be compiled). Automatically. We live in the future.

Feels like progress.

Now it's compiling pygame nicely. So `pip install pygame` works with the current release on Raspbian now (the Debian distribution put out by some Raspberry Pi people). Thanks to Ben Nuttall for that!
https://github.com/bennuttall/piwheels/issues/52
These are built automatically when new things are released to pypi. So the pygame 1.9.4 release should go on there without us having to do anything.







ps. Anyone reading this with a binary package they maintain... you can just ask them to install the Debian dependencies... or submit a PR to do so. https://github.com/bennuttall/piwheels/
pps. Speaking of binary wheels. The other day I submitted a PR to https://github.com/skvark/opencv-python which is python opencv packaged up nicely for pip. It should be much easier to do computer vision things that you can actually distribute to others (who aren't fond of C++ compilation fun).

Friday, February 23, 2018

Mu, the little #python editor that could.

Nicholas and the Mu gang have been busy with their python editor, and it now supports pgzero.


Mu is a simple code editor for beginner programmers.


I'm pretty excited about this. Here's a little demo of it in action with pgzero...

Whilst it's still in 'beta' with rough edges... it's pretty neat already. Also, they seem to be quickly moving through issues people in the community encounter with each beta release.

It's been super-lovely to see the python community get behind them and offer their support.

Stop idling, and go have a look: https://github.com/mu-editor/mu

pygame documentation updates

There are some documentation updates on https://www.pygame.org/docs/


The website documentation builder was waiting for updates from bitbucket(our previous code host). lol? oops. Robots running in the cloud doing things no one wants them to do anymore. I had to write a new github integration, so now commits to master on github/pygame/pygame will cause the website docs to be rebuilt automatically again.

And python -m pygame.docs didn't work with the wheel builds... because they don't include docs (for some reason). So now, if the docs can't be found locally, it opens the web browser instead.
There were a bunch of documentation updates from Ian Mallett and Lenard Lindstrom which are finally up on the website. Lots of editing, and improvements to the tutorials.
Additionally a bunch of old links were fixed. Mostly to point to https:// versions of pages.
The docs are being built with a new version of Sphinx, which has nicer output in a few ways. See http://www.sphinx-doc.org/en/master/changes.html#release-1-7-0-released-feb-12-2018

Since it was a major version upgrade of Sphinx, there were some small expected issues to deal with, including their python API being broken/deprecated (just call the command line program instead), and disabling the on-by-default smart quotes(which are terrible for people copy pasting code examples).

Also, the launchpad PPA is building again. It got stuck because someone snuck a gpg field in their git commit, which broke the bzr mirror code. Except the packaging for that ppa is from 2013... and needs updating. Luckily it seems a few people have done the packaging work in various flavours of Debian... but somehow none of them have gone through. But anyway... The badge is getting made again(displayed on the readme) with the github webhook, and will alert us if things break on ubuntu/i386/amd64/arm7f.

Speaking of badges... there's now a 'commits since last release' badge on the https://github.com/pygame/pygame Which links to a diff between the last release.

Finally, I updated all the old bitbucket wiki pages to point to the same link on the pygame wiki (https://www.pygame.org/wiki). eg, https://bitbucket.org/pygame/pygame/wiki/CookBook points to https://www.pygame.org/wiki/CookBook

Thursday, February 15, 2018

Hey! It's work on pygame stuff week.

So, it's been about 9 days since I had "a fun day working on pygame stuff". That morning I woke up, and just started working on pygame things. This is a pattern with me between big freelance contracts. Last year I spent some months on pygame stuff, and also the year before that some months.

It was such a fun day... I just kept going. And here we are 9 days later with a web log of some of the things that happened.
A fun 9 days working on pygame stuff.

New pygame.org website changes.

Got a new version of the pygame website out. It took a couple of days, but I fixed a number of issues, and improved a few things. In the process I found a lot more issues on the website than I fixed. So there are now more issues listed than when I started. Feels like progress.

"it's the schrödinger's bugtracker" -- þeshipu

pygameweb 0.0.1 - Weedy Seadragon

Witness the beauty of the Weedy Seadragon (photo by

#30 https url scheme default(not in DEBUG). https login links even on http.
#29 News feeds (rss, atom) are in the header of every page again.
#28 Improved table of contents on the wiki. Syntax help on edit page. Clear cache after edit.
#27 Changes are now shown again when people do a new project release. Project test improvements.
#31 /admin turned off by default, APP_ADMIN config variable.
#13 Admin interface improvements

"Are pygame releases going to have code names too? :-)" I guess so, but maybe the code name for all of them should be "Weedy Seadragon", because it's such a magical creature.

Debian bugs.

I've investigated a number of Debian reported issues, and even managed to fix some of them. It never ceases to amaze me that testing on other systems uncovers real bugs which also happen on other platforms (but in harder to detect ways usually). So, for me, it's always worth working on weird platforms - if only for this reason. https://github.com/pygame/pygame/pull/391

Thanks to Dominik for bringing them into the pygame issue tracker, and for improving the Debian pygame package generally.

It's important to get these fixes in, as I think they are blocking a Debian release of pygame 1.9.3. ... which also blocks derivatives such as Raspberry Pi rasperian distro (who don't provide any support to pygame, or Debian, and just take from Debian packages mostly). Another important derivative is Ubuntu, which also hasn't updated their pygame package.

Here is the Debian tracker for the pygame package, which lists things that need to be done. If anyone can spot something I could help out on there, please feel free to point it out - as I have some days now to investigate and fix things. https://tracker.debian.org/pkg/pygame

Here is the build status on the various Debian platforms:

Thanks to the OpenPOWER foundation and the University of Campinas who provide Power virtual machines to open source developers. It still feels magical to me when I log into a box on another continent to shell-about.

So, I got some Power PC virtual machines for testing. I fixed a few pygame on Debian bugs, and wrote on their issues telling them they are fixed now. Fingers crossed a newer pygame package will make it into Debian, Ubuntu and Raspberian.


Outdated installation instructions from around the net.

TLDR; If you see anywhere on the net that has outdated pygame install/compile instructions, please let them know, and send them a link to GettingStarted.

One thing I've noticed is that there are quite a few web pages on the interwebs describing 'how to install pygame'. Which were all helpful in the past, but are now out of date. The pygame website itself is one place with outdated instructions too. Now they just cause people to fail to install old versions of pygame, and for them to report the same old bugs again and again.

I'm asking people to:
  1. Link to the Getting Started guide, which should have tested installation methods for many platform/python combinations: https://www.pygame.org/wiki/GettingStarted
  2. Update instructions that tell people to install from the outdated bitbucket repository (we moved to github).
I've been updating the pygame website itself to follow this advice in parts where it wasn't current.
Also soon, the bitbucket repo setup.py will get a message added to it:
In the coming months I'll make that bitbucket repo error, with the same instructions.

Additionally, I'll add a message on compile failure to link to the relevant Compile* page.
Eg, if there's a failure on a Ubuntu machine it will link to the compile page for Ubuntu. https://www.pygame.org/wiki/CompileUbuntu

For all the compile pages, they will be updated to add the versions they were confirmed working (luckily, this info is sort of stored in the wiki version history). Old versions of Ubuntu require different compilation advice to the latest Ubuntu versions. Latest version instructions will be up the top of the page, and old versions underneath. I'll also gather links from around the internet, because sometimes better compilation instructions are listed on peoples blog posts than in the wiki. All the Compile pages are listed here: https://www.pygame.org/wiki/Compilation Further development/contributing instructions are listed here: https://www.pygame.org/wiki/Hacking Hopefully giving the Compile* pages some more structure, they will become better in the future.

Python 3.7.0b1

Some python 3.7.0b1 issues were fixed, and I got it working on Travis CI with linux. There is still work to try it out on Windows, and Mac, and to make binary wheels for for all the platforms.

Are we PyPy yet?

It seems various people have been working on making the PyPy CPython C API handling improve a lot. As well as fixing issues here and there with pygame when compiled with pypy.

Wow. Many thanks to all the people working on that.


Now it's at a stage where a bunch of things actually work. Not enough to play a game on it... but getting much closer.

Learning this good news... I got PyPy building on Travis CI/pygame/pygame (with some tests still failing). Then started a milestone for "Are we PyPy yet?". Stuart Axon has already filed one issue, and asked PyPy developers for help on how to fix it. Stuart wasn't joking saying PyPy devs are super helpful.

This one particular issue has to do with surface locking and reference counting. In CPython when you do "del array" it does what you think... and deletes the array right then and there. However, PyPy has a different type of memory management, and PyPy will decide when to delete things thank you very much!

Like with files in python, it was suggested we gain a method like '.close()' to explicitly deal with these locks. Instead of relying on reference counting semantics. Also, to use a context manager for with.

PixelArray is using Surface locks. It does this so if your Surface is graphics hardware memory underneath your operating system won't crash on you (I hope).
Leave the pixels alone!!!! What did they ever do to you?!?

Next up is to start filing pygame issues for each of the test failures and then slowly fixing them. Some of them should be fairly easy, and others are downright scary (BufferProxy and no ctypes.pythonapi on PyPy makes me afraid... very afraid).

Then we have to figure out how to build binary wheels for the various platforms, and probably many other things... All PyPy related work will be tracked on the Are we PyPy yet? milestone in the pygame issue tracker. Not sure how long this work will take, so this won't be included in the upcoming 1.9.4 release I don't think.

The mysteriously weird 'moiré pattern'.

I was clicking-about on stack overflow and came across this weird thing in popular question there. This strange thing happens when drawing thick circles with pygame.draw.

Before: moiré pattern sort of looks nice

After: just a boring filled bit of red. Nothing interesting. Move along.
That was sort of a fun one to hack a fix for.

I love Pulseaudio. And sarcasm.

I have much respect for the pulseaudio developers... linux audio is hard work.

There's a couple of issues with sound and pygame on linux with the 'manylinux' binary wheels we have. It took me some time to track the causes down, and find ways to reproduce the issues.

TLDR; midi music is broken with pygame installed by wheels.

One has to do with arch linux using a different configuration directory to every other linux of the last 20 years for timidity. And also to do with how we are not installing special config files on random parts of the linux file systems when people install the wheel with pip.

So, there are a few different software music synthesizers included with pygame. One is timidity (linked above) and the other is fluidsynth. The soundfont playing synth.

"Had enough with linux audio issues for today. I've got a sneaking feeling that pulseaudio and fluidsynth are to blame somehow for high cpu on linux systems when installing from pip wheels. https://github.com/pygame/pygame/issues/331#issuecomment-365177184"

So, I just sort of paused at that point. I guess we need to disable timidity (or patch it for arch linux), and for the case when timidity isn't installed. Also, we need to handle soundfonts... either provide a message to the user if they are missing, or see if there are some FLOSS-happy ones we can distribute just on linux.

ps. During these investigations, I found out that fluidsynth has been saved from sourceforge and has had a whole bunch of releases. Awesome. http://www.fluidsynth.org/

Updating old links again...

Another pleasant delight whilst searching-around-the-interwebs... I found a patch logged against an old mirror of the bitbucket repo on github. This had a few really good documentation updates, and an addition of a 'pitch_bend' method... which allows you to do pitch bends, as the name suggests. Luckily @endolith (a hardcore Electronics and DSP engineer doing interesting stuff) made a new PR, and so those improvements will finally make it out into the world. Somehow I overlooked it on bitbucket too.

Additionally, a couple of new issues were filed against out bitbucket repo... which I haven't closed down yet(because reasons). They are for a couple of SRC_ALPHA regressions it seems. I'll add them into github soon, and also then to the list of blockers for a 1.9.4 release.

1.9.4 release planning.

"About time for another release I guess?"
About 5 days ago I asked the mailing list if we could do a release, and that I intended to do a release. To give people time to mention issues, do testing, and let me know if they are still working on things to release that we should wait for.

Which is nice because people let me know about various issues they'd liked fixed and pointed me to the details (including the SRC_ALPHA issues above). I started looking around for things to merge in, looking at other bug trackers (like the Debian one).
Then, I started a checklist of things to do for a pygame release, and also started filing issues with the correct labels and using a milestone.  https://github.com/pygame/pygame/issues/390

I want to get this work out so I can start merging in SDL2 stuff into the master branch. That's what I'm really looking forward to.


A new place to idle the day away chat.

We have a pygame discord server for anyone who wants to chat about pygame projects.

/me sheds a tear for irc.

Feels like progress. 

Every time I look at that bug list it gets bigger! Schrödinger has a lot to answer for.
Stop messing about on your phone and get to work!
It was a fun 9 days in floss. Gonna do more!


Tuesday, February 06, 2018

Hey! It's work on pygame stuff day.

Yasssssssss! Today was work-on-pygame-stuff day.


ROTFL. ASCII renderer in action.

So here is a log of pygame (The game library for python) updates.
  • Removed a s.p.aaaam infestation from the pygame.org website. Planned next round of improvements for spam fighting.
  • Went over issue tracker adding labels, and replying to things. Thinking about, and closing a few issues. Felt a bit bad about closing some issues, mixed with a bit of relief. Shoulders felt a tiny bit lighter.
  • Set up a new linux box for doing pygame stuff with.
  • Mucked around with pypy nightly.
  • Messed with the python beta. Ya to breakpoint(). No real big breaking changes to the C API uncovered yet. Read the TLS API changes. Thought to myself that latency, and low memory benchmarks would be a good thing to submit. time.time_ns is useful but still a workaround for using floats for time, and there is code out in the wild already using a smaller measurement of time than ns. 1GHz is a nanosecond, and 5Ghz CPUs have been around for more than a decade, so it's already out of date when the API was released. Still sort of awesome.
  • Applied for a 64bit Power PC developer virtual machine to help me fix several bugs reported by some fine people from Debian. Got given cloud access. Delicious Power box for compiling. I see bit fiddling 64bit big endian fixes in my future.
  • Did a code review of a chip tunes synth project. Got a little bit too excited. Caught up a bit with some internet friends/acquiescences who lurk in the odd parts of the net.
  • Added some files that github now requires. Before it had some red crosses, and now with these files there are green ticks. Feels like progress.
  • Answered a bunch of newbie questions.
  • Looked around at what is going on. Too much to look at really (oh, two more books... blog posts, ... arggggg tooo muuuuch). Joined up on gitlab, joined a few discord servers, got confused about discourse, discord, and discuss. 1789 unread emails, 64 twitter notifications. Slack, gitter, mastadon... noooooooo. Info overload. Lurked on an irc channel, and didn't even say hi, and no one even said hi back, and that is a perfect day on irc. The best part about irc is that people don't write to you.
  • Did some server admin work. "Let's see what I caught in my spam traps" *look of glee*, updating packages, grepping logs, using vi. Killed some daemons.
  • Found some eco renewable power servers. Considered starting on that migration... not today.
  • Looked at some assembly code for the first time in maybe 12 months. Downloaded some syntax highlighting updates for said file. Closed file.
  • Pulled out some test hardware and arranged it in my studio. Plugged some of it in. (a 24bit sound card, wacom tablet, vibration joypads, android and apple phones+tablets, a fist full of raspberrypi, the shittiest windows laptop, the oldest linux laptop, screens, screens, and more screens). Realised I needed more plants, because now it's a jungle of wires.
  • Decided I should keep a web log of this work.
  • Caught up on some changes to GLSL.

Oh gosh. There's lots of work to do... 
Time to pull up my big boy pants and work.

Did some code review, and merged a bunch of changes that some fine people submitted:
Made some plans for how to progress on the sdl2 branches. I see lots of merging in my future.
  • rebase sdl2 branch ontop of master
  • bring all files back to current state on master (that is SDL1 branch is buildable with no SDL2 files).
  • So now all files have the SDL2 stuff in history.
  • Introduce one file at a time with combined SDL1+SDL2 module.
  • At each step the SDL1 branch should be releasable.
  • Convert another module to SDL1+SDL2 in the SDL2 branch.
There's a bunch of testing in between those steps. Eventually we will get to a stage where the SDL2 branch will build as well on the build robots.

Realised my local build machines finished compiling and running tests before the remote CI machines had even started.

Decided it was time for a new 1.9.x release soon.


Today felt like spring. I sat on my balcony with the sun on my face drinking a coffee and eating an Italian strawberry. It was 0 degrees Celsius, but even that felt refreshing.

A good day in FLOSS.