tag:blogger.com,1999:blog-106780742024-03-18T21:41:02.241+00:00making apps, making webs.A log of some thoughts about freelancing, about indie, about making things. On the web. It's a web log. A blog.Unknownnoreply@blogger.comBlogger349125tag:blogger.com,1999:blog-10678074.post-79023100238573879622020-11-13T08:38:00.000+00:002020-11-13T08:38:09.370+00:00Midi musical intrument controllers and synthesizers with python and pygame<p>Musical Instrument Digital Interface (Midi), is for letting digital musical instruments talk to each other in 'real time'. It came from the early 80s, and is still used today. The main idea is that rather than sending audio data, it sends small control messages. This is great for fast real time low bandwidth communication.<br /></p><p></p><p></p><p></p><div class="separator" style="clear: both; text-align: center;"><a href="https://www.patreon.com/posts/midi-music-pygame-python-43826303" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1280" data-original-width="1280" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhKyJrFRj4U5ves4oo3-0FRxFuHMFGiwgYmO28B0lpPYQSGPsVOn65BcDFihcJZlaAEYbU6VAKRaZxAlfaY5UGM7jc6Ls9d2s2SryxOiwORfxxMwvz2UPG8BlDR0s6MJIJYHlDq/s320/midi-controller-pygame.jpeg" /></a></div><p>I'm going to show in all the different ways midi can be used with python/pygame with three separate programs, and how to set up a virtual synth (I've included instructions for Win, Mac, and Linux).<br /></p><p>Full article is available for patrons first as an "early access" before it becomes available to all in a month: <a href="https://www.patreon.com/posts/midi-music-pygame-python-43826303">https://www.patreon.com/posts/midi-music-pygame-python-43826303</a></p><p><br /></p><p>You can support my pygame development and read my development log at: <a href="https://www.patreon.com/pygame">https://www.patreon.com/pygame</a></p>Unknownnoreply@blogger.com63tag:blogger.com,1999:blog-10678074.post-7413926204245790882020-11-12T10:42:00.003+00:002020-11-12T11:17:13.196+00:00Finger painting with multi-touch in python using pygame 2<p>pygame 2 has support for Multi-Touch. You know, where it can track multiple fingers. I've made an app and tutorial to show off the new Multi-Touch features.<br /></p><p></p><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><a href="https://www.patreon.com/posts/finger-painting-with-multi-touch-pygame2-43786073" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="1008" data-original-width="1282" height="314" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj2v8lanoyO_K3VwBSMrSPL1N5OimdBwLdbdVtn5_DF-5icPd16jdeBjXkOAwAtiGbZ1HKDuy3fMdSQEYLQiT9r_SHFG3gQNHHB8vgEoh-sKvCqN2I3UfYQaQMKPdK6oKXdhTH9/w400-h314/screenshot.png" width="400" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">Finger Painting with Multi-Touch<br /></td></tr></tbody></table><br /> <p></p><p></p><p>Full article is available for patrons first as an "early access" before it becomes available to all in a month: <a href="https://www.patreon.com/posts/finger-painting-with-multi-touch-pygame2-43786073">https://www.patreon.com/posts/finger-painting-with-multi-touch-pygame2-43786073</a> You can support my pygame development and read my development log at: <a href="https://www.patreon.com/pygame">https://www.patreon.com/pygame</a><br /></p>Unknownnoreply@blogger.com73tag:blogger.com,1999:blog-10678074.post-70827121574996010382019-11-06T10:49:00.000+00:002019-11-06T10:49:24.031+00:00post modern C tooling - draft 6<div dir="ltr" style="text-align: left;" trbidi="on">
<div style="text-align: left;">
<div class="K3JSBVB-W-o">
Contemporary <a href="https://en.wikipedia.org/wiki/C_(programming_language)">C</a> tooling for making higher quality C, faster or more safely.<br />
<br />
<b>DRAFT 0 - 10/11/18, </b><br />
<b>DRAFT 1 - 9/16/19, 7:19 PM, </b><b><b>I'm still working on this, but it's already useful and I'd like some feedback - so I decided to share it early.</b> </b></div>
<b>DRAFT 2 - 10/1/19, mostly additions to static analysis tools.</b><br />
<b>DRAFT 3 - 10/4/19, updates on build systems, package management, and complexity analysis. </b> <br />
<b>DRAFT 4 - </b><b>10/6/19, run time dynamic verification and instrumentation, sanitizers (asan/ubsan/etc), performance tools, static analyzers.</b><br />
<b>DRAFT 5 - C interpreter(s). </b><br />
<b>DRAFT 6</b><b><b> - </b><b>11/6/19</b>, mention TermDebug vim, more windows debugging tools, C drawing for intro.</b> <br />
<br />
<br />
<br />
In
2001 or so people started using the phrase "Modern C++". So now that
it's 2019, I guess we're in the post modern era? Anyway, this isn't a
post about C++ code, but some of this information applies there too.<br />
<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgkR6H1_FATmwT-3txz6_JTylvAgJZfo3FLGImzYVp1jWkHELG5YpR7JLbOUPNM5ae1X7gPXXAeHHYYXoQD2rLmOb2Eki9E74rfvAl0MjhfuHxPmtgqVfWIGyBVBegIhH9gEmVQ/s1600/the-c-programming-language-scan.png" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="1600" data-original-width="1533" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgkR6H1_FATmwT-3txz6_JTylvAgJZfo3FLGImzYVp1jWkHELG5YpR7JLbOUPNM5ae1X7gPXXAeHHYYXoQD2rLmOb2Eki9E74rfvAl0MjhfuHxPmtgqVfWIGyBVBegIhH9gEmVQ/s320/the-c-programming-language-scan.png" width="306" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">No logo, but it's used everywhere.<span id="goog_1256792431"></span><span id="goog_1256792432"></span></td></tr>
</tbody></table>
<br />
<br />
<h3 style="text-align: left;">
Welcome to the post modern era. </h3>
Some of the C++ people have pulled off one of the cleverest and sneakiest tricks ever. They required 'modern' <a href="https://en.wikipedia.org/wiki/C99">C99</a> and <a href="https://en.wikipedia.org/wiki/C11_(C_standard_revision)">C11</a>
features in 'recent' C++ standards. Microsoft has famously still clung
onto some 80s version of C with their compiler for the longest time. So
it's been a decade of hacks for people writing portable code in C. For a
while I thought we'd be stuck in the 80s with C89 forever. However, now that some C99 and C11
features are more widely available in the Microsoft compiler, we can
use these features in highly portable code (but forget
about C17/C18 ISO/IEC 9899:2018/C2X stuff!!). Check out the <a href="https://www.youtube.com/watch?v=ieERUEhs910">"New" Features in C</a> talk, and the <a href="https://modernc.gforge.inria.fr/">Modern C</a> book for more details.<br />
<br />
So, we have some pretty new language features in C with C11. But what about tooling?<br />
<br />
<h3 style="text-align: left;">
Tools and protection for our feet. </h3>
C, whilst a work horse being
used in everything from toasters, trains, phones, web browsers, ...
(everything basically) - is also an excellent tool for shooting yourself
in the foot.<br />
<blockquote class="tr_bq">
<h3>
<span class="mw-headline" id="Noun">Noun</span></h3>
<b class="Latn headword" lang="en">footgun</b> (<i>plural</i> <b class="Latn form-of lang-en plural-form-of" lang="en"><a href="https://en.wiktionary.org/wiki/footguns#English" title="footguns">footguns</a></b>)
<br />
<ol>
<li><span class="ib-brac">(</span><span class="ib-content"><a href="https://en.wiktionary.org/wiki/Appendix:Glossary#informal" title="Appendix:Glossary">informal</a><span class="ib-comma">,</span> <a href="https://en.wiktionary.org/wiki/humorous" title="humorous">humorous</a><span class="ib-comma">,</span> <a href="https://en.wiktionary.org/wiki/derogatory" title="derogatory">derogatory</a></span><span class="ib-brac">)</span> Any <a href="https://en.wiktionary.org/wiki/feature" title="feature">feature</a> whose addition to a product results in the user <a href="https://en.wiktionary.org/wiki/shoot_oneself_in_the_foot" title="shoot oneself in the foot">shooting themselves in the foot</a>. C.</li>
</ol>
</blockquote>
<br />
Tools like linters, test coverage
checkers, static analyzers, memory checkers, documentation generators,
thread checkers, continuous integration, nice error messages, ... and
such help protect our feet.<br />
<br />
How do we do continuous
delivery with a language that lets us do the most low level footgunie
things ever? On a dozen CPU architectures, 32 bit, 64bit, little endian,
big endian, 64 bit with 32bit pointers (wat?!?), with multiple
compilers, on a dozen different OS, with dozens of different versions of
your dependencies?<br />
<br />
Surely there won't be enough time to do releases, and have time left to eat my vegan shaved ice desert after lunch?<br />
<br />
<br />
<br />
<h3 style="text-align: left;">
Debuggers</h3>
<blockquote class="tr_bq">
Give me 15 minutes, and I'll change your mind about GDB. -- <br />
<a href="https://www.youtube.com/watch?v=PorfLSr3DDI">https://www.youtube.com/watch?v=PorfLSr3DDI</a></blockquote>
Firstly,
did you know gdb had a curses based 'GUI' which works in a terminal?
It's a quite a bit easier to use than the command line text interface.
It's called TUI. It's built in, and uses emacs key bindings.<br />
<br />
But what if you are used to VIM key bindings? <a href="https://cgdb.github.io/">cgdb</a> to the rescue.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjoW_46llbVrAUw7TBbc8MWjVNSbEGQJCLRynWUoVGwFRm3F4MI6J-7IM_fI4Lrsw1OrhY9tUARvBfJOokb9zNJ3OFiYSloacgDEZmayHtPhVCOyXB8R7PsNtH9wQR9aK4COBvH/s1600/cgdb_debugging.png" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="498" data-original-width="508" height="313" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjoW_46llbVrAUw7TBbc8MWjVNSbEGQJCLRynWUoVGwFRm3F4MI6J-7IM_fI4Lrsw1OrhY9tUARvBfJOokb9zNJ3OFiYSloacgDEZmayHtPhVCOyXB8R7PsNtH9wQR9aK4COBvH/s320/cgdb_debugging.png" width="320" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><a href="https://cgdb.github.io/">https://cgdb.github.io/</a></td></tr>
</tbody></table>
<br />
VIM has integrated gdb debugging with <a href="https://www.dannyadam.com/blog/2019/05/debugging-in-vim/">TermDebug</a> since version 8.1.<br />
<br />
Also, there's a fairly easy to use web based front end for GDB called <a href="https://www.gdbgui.com/">gdbgui</a><br />
(https://www.gdbgui.com/).
For those who don't use an IDE with debugging support built in (such as
Visual studio by Microsoft or XCode by Apple).<br />
<br />
<br />
<br />
<br />
<br />
<h4 style="text-align: left;">
Reverse debugger</h4>
Normally a program runs forwards. But what about when you are debugging and you want to run the program backwards?<br />
<br />
<blockquote class="tr_bq">
Set breakpoints and data watchpoints and quickly reverse-execute to where they were hit. </blockquote>
<br />
How
do you tame non determinism to allow a program to run the same way it
did when it crashed? In C and with threads some times it's really hard
to reproduce problems.<br />
<br />
rr helps with this. It's actual magic.<br />
<br />
<a href="https://rr-project.org/">https://rr-project.org/</a><br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<iframe allowfullscreen="" class="YOUTUBE-iframe-video" data-thumbnail-src="https://i.ytimg.com/vi/hYsLBcTX00I/0.jpg" frameborder="0" height="266" src="https://www.youtube.com/embed/hYsLBcTX00I?feature=player_embedded" width="320"></iframe></div>
<br />
<br />
<br />
<br />
<br />
<h4 style="text-align: left;">
LLDB - the LLVM debugger.</h4>
Apart from the ever improving <a href="https://www.gnu.org/software/gdb/">gdb</a>, there is a new debugger from the LLVM people - <a href="https://lldb.llvm.org/">lldb</a> ( https://lldb.llvm.org/ ). <br />
<br />
<br />
<h4 style="text-align: left;">
IDE debugging</h4>
Visual Studio by Microsoft, and XCode by Apple are the two heavy weights here.<br />
<br />
The free Visual Studio Code also supports debugging with GDB. https://code.visualstudio.com/docs/languages/cpp<br />
<br />
Sublime is another popular editor, and there is good GDB integration for it too in the <a href="https://packagecontrol.io/packages/SublimeGDB">SublimeGDB package</a> (https://packagecontrol.io/packages/SublimeGDB).<br />
<br />
<h4 style="text-align: left;">
Windows debugging</h4>
<div data-reddit-rtjson="{"entityMap":{"0":{"type":"LINK","mutability":"MUTABLE","data":{"url":"https://blogs.msdn.microsoft.com/webdav_101/2018/03/20/how-to-take-a-procdump/"}}},"blocks":[{"key":"52n4l","text":"Thanks for the tip :)\n","type":"unstyled","inlineStyleRanges":[],"entityRanges":[],"data":{}},{"key":"d8v3r","text":"I found this tutorial: \"How to take a procdump\" https://blogs.msdn.microsoft.com/webdav_101/2018/03/20/how-to-take-a-procdump/","type":"unstyled","inlineStyleRanges":[],"entityRanges":[{"offset":48,"length":78,"key":0}],"data":{}}]}">
<div class="" data-block="true" data-editor="772a85" data-offset-key="b70cv-0-0">
<div class="public-DraftStyleDefault-block public-DraftStyleDefault-ltr" data-offset-key="b70cv-0-0">
<span data-offset-key="b70cv-0-0"><span data-text="true">Suppose you want to do post mortem debugging? With </span></span><a class="_3t5uN8xUmg0TOwRCOGQEcU" href="https://docs.microsoft.com/en-us/sysinternals/downloads/procdump" rel="noopener noreferrer" target="_blank">procdump</a> and <a class="_3t5uN8xUmg0TOwRCOGQEcU" href="https://docs.microsoft.com/en-us/windows-hardware/drivers/debugger/debugger-download-tools" rel="noopener noreferrer" target="_blank">WinDbg</a><span data-offset-key="b70cv-0-0"><span data-text="true"> you can.</span></span></div>
<div class="public-DraftStyleDefault-block public-DraftStyleDefault-ltr" data-offset-key="b70cv-0-0">
<span data-offset-key="b70cv-0-0"><span data-text="true">"<a href="https://blogs.msdn.microsoft.com/webdav_101/2018/03/20/how-to-take-a-procdump/">How to take a procdump</a>" (</span></span><a class="_1FRfMxEAy__7c8vezYv9qP" href="https://blogs.msdn.microsoft.com/webdav_101/2018/03/20/how-to-take-a-procdump/"><span data-offset-key="b70cv-1-0"><span data-text="true">https://blogs.msdn.microsoft.com/webdav_101/2018/03/20/how-to-take-a-procdump/</span></span></a>) is a nice tutorial on how to use procdump.</div>
</div>
</div>
<br />
<blockquote class="tr_bq">
Launch a process and then monitor it for exceptions:<br />
<b>C:\>procdump -e 1 -f "" -x c:\dumps consume.exe</b></blockquote>
This makes some process dumps when it crashes, which you can then open with <a href="https://docs.microsoft.com/en-us/windows-hardware/drivers/debugger/debugging-using-windbg-preview">WinDbg</a>(https://docs.microsoft.com/en-us/windows-hardware/drivers/debugger/debugging-using-windbg-preview).<br />
<br />
<br />
<h3 style="text-align: left;">
Portable building, and package management</h3>
C doesn't have a package manager... or does it? <br />
<br />
Ever since Debian dpkg,
Redhat rpm, and Perl started doing package management in the early 90s
people world wide have been able to share pieces of software more
easily. Following those systems, many other systems
like Ruby gems, JavaScript npm, and Pythons cheese shop came into being.
Allowing many to share code easily.<br />
<br />
But what about C?
How can we define dependencies on different 'packages' or libraries and
have them compile on different platforms?<br />
<br />
How do we
build with Microsofts compiler, with gcc, with clang, or Intels C
compiler? How do we build on Mac, on Windows, on Ubuntu, on Arch linux?
Sometimes we want to use an Integrated Development Environment (IDE)
because they provide lots of nice tools. But maybe also three IDEs
(XCode, Microsoft Visual C, CLion, ...) depending on platform. But we
don't want to have to keep several IDE project files up to date. But we
also want to integrate nicely with different OS packagers like Debian,
FreeBSD. We want people to be able to use apt-get install for their
dependencies if they want. We also want to cross compile code on our
beefy workstations to work on microcontrollers or slower low memory
systems (like earlier RaspberryPi systems).<br />
<br />
<br />
<br />
<h4>
The Meson Build System.</h4>
</div>
<div>
If CMake is modern, then <a href="https://mesonbuild.com/index.html">The Meson Build System</a> (https://mesonbuild.com/index.html) is post modern.<br />
<blockquote class="tr_bq">
"Meson is an open source build system meant to be both extremely fast,
and, even more importantly, as <b>user friendly as possible.</b> </blockquote>
<blockquote class="tr_bq">
The main design point of Meson is that every moment a developer spends
writing or debugging build definitions is a second wasted. So is every
second spent waiting for the build system to actually start compiling
code."</blockquote>
It's first major user was GStreamer, a multi
platform multimedia toolkit which is highly portable. Now it is
especially popular in the FreeDesktop world with projects like
gstreamer, GTK, and systemd amongst many others using it.<br />
<br />
The documentation is excellent, and it's very fast compared to autotools or cmake.</div>
<div>
<a href="https://www.youtube.com/watch?v=SCZLnopmYBM">https://www.youtube.com/watch?v=SCZLnopmYBM</a></div>
<div>
<a class="anchor-3Z-8Bb anchorUnderlineOnHover-2ESHQB" href="https://www.youtube.com/watch?v=gHdTzdXkhRY" rel="noreferrer noopener" role="button" tabindex="0" target="_blank" title="https://www.youtube.com/watch?v=gHdTzdXkhRY">https://www.youtube.com/watch?v=gHdTzdXkhRY</a><br />
<br />
<br />
<div class="mb-3">
<div class="f4">
<span class="text-gray-dark mr-2" itemprop="about"> Example <b><span style="font-family: "courier new" , "courier" , monospace;">meson.build</span></b> for example project <a href="https://github.com/jpakkane/polysnake">polysnake</a> (https://github.com/jpakkane/polysnake):</span></div>
<div class="f4">
<span class="text-gray-dark mr-2" itemprop="about">"A Python extension module that uses C, C++, Fortran and Rust"? </span></div>
<div class="f4">
<span class="text-gray-dark mr-2" itemprop="about"> </span>
</div>
</div>
<blockquote class="tr_bq">
<span style="font-family: "courier new" , "courier" , monospace;">project('polysnake', 'c', 'cpp', 'rust', 'fortran',<br /> default_options : ['cpp_std=c++14'],<br /> license : 'GPL3+')<br /><br />py3_mod = import('python3')<br />py3_dep = dependency('python3')<br /><br /># Rust integration is not perfect.<br />rustlib = static_library('func', 'func.rs')<br /><br />py3_mod.extension_module('polysnake',<br /> 'polysnake.c',<br /> 'func.cpp',<br /> 'ffunc.f90',<br /> link_with : rustlib,<br /> dependencies : py3_dep)</span></blockquote>
<br />
IDEs
are supported by exporting to XCode+Visual Studio, and they provide
their own interface (which a few less well known IDEs are starting to
use).<br />
<br />
<br />
<h4 style="text-align: left;">
Conan package manager</h4>
There are several packaging tools for C these days, but one of the top contenders is <a href="https://conan.io/">Conan</a> (<a href="https://conan.io/">https://conan.io/</a>). <br />
<blockquote class="tr_bq">
"Conan, the C / C++ Package Manager for
Developers The open source, decentralized and multi-platform package
manager to create and share all your native binaries."</blockquote>
What does a <a href="https://github.com/conan-io/hello">CMake conan project</a> look like? (https://github.com/conan-io/hello)<br />
What does a <a href="https://docs.conan.io/en/latest/reference/build_helpers/meson.html">conan Meson</a> project look like? (https://docs.conan.io/en/latest/reference/build_helpers/meson.html)<br />
<br />
<br />
<h4 style="text-align: left;">
CMake</h4>
"Modern CMake" is the build tool of choice for many C projects.<br />
<br />
Just don't read the official docs, or the official book - they're quite out of date.<br />
<a href="https://cliutils.gitlab.io/modern-cmake/">An Introduction to Modern CMake</a> (https://cliutils.gitlab.io/modern-cmake/)<br />
<a href="https://cgold.readthedocs.io/en/latest/">CGold: The Hitchhiker’s Guide to the CMake</a> (https://cgold.readthedocs.io/en/latest/)<br />
<br />
<blockquote class="tr_bq">
"<a class="reference internal" href="https://cgold.readthedocs.io/en/latest/glossary/CMake.html#id1"><span class="std std-ref">CMake</span></a> is a meta build system. It can generate real
<a class="reference internal" href="https://cgold.readthedocs.io/en/latest/glossary/Native-build-tool.html#id1"><span class="std std-ref">native build tool</span></a> files from abstracted text configuration.
Usually such code lives in <code class="docutils literal"><span class="pre">CMakeLists.txt</span></code> files."</blockquote>
<br />
Around 2015-2016 a bunch of IDEs got support for CMake: <a href="https://blogs.msdn.microsoft.com/vcblog/2016/10/05/cmake-support-in-visual-studio/">Microsoft Visual Studio</a>, <a href="https://blog.jetbrains.com/clion/2015/04/cmake-support-in-clion/">CLion</a>, <a href="https://blog.qt.io/blog/2016/11/15/cmake-support-in-qt-creator-and-elsewhere/">QtCreator</a>, <a href="https://userbase.kde.org/KDevelop4/Manual/Getting_started:_A_small_CMake_tutorial">KDevelop</a>, and <a href="https://developer.android.com/ndk/guides/cmake.html">Android Studio (NDK)</a>. And a lot of people tried extra hard to like it, and a lot of C projects started supporting it.<br />
<br />
Apart from wide IDE support, it is also supported quite well by package managers like VCPkg and Conan.<br />
<br />
<br />
<h3 style="text-align: left;">
Interpreter and REPL</h3>
Usually C is compiled.<br />
<a href="https://github.com/hexagonal-sun/bic">Bic</a> is an interpreter for C (https://github.com/hexagonal-sun/bic).<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj-9P8_d9LkipmF2r1lEnICtx6wOsqtGDVFnhmoimjj71Dc_8uRvwH4aJrMLNNyEFuuSu2rYAuvVx8E_k8QAkssahuPZHMKPLMXHQEQyr3fYSiN5JMj7PB_4dmxnRqre_SYF00x/s1600/c-interpreter-hello-world.gif" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="336" data-original-width="461" height="233" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj-9P8_d9LkipmF2r1lEnICtx6wOsqtGDVFnhmoimjj71Dc_8uRvwH4aJrMLNNyEFuuSu2rYAuvVx8E_k8QAkssahuPZHMKPLMXHQEQyr3fYSiN5JMj7PB_4dmxnRqre_SYF00x/s320/c-interpreter-hello-world.gif" width="320" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><a href="https://github.com/hexagonal-sun/bic">bic</a>: A C interpreter and API explorer</td></tr>
</tbody></table>
<br />
Additionally there is "<a href="https://github.com/root-project/cling">Cling</a>" which is based on the LLVM infrastructure and can even do C++.<br />
https://github.com/root-project/cling<br />
<br />
<br />
<br />
<br />
<h3 style="text-align: left;">
Testing coverage.</h3>
Tests let us know that some certain function is running ok. Which code do we still need to test?<br />
<br />
<a href="https://gcc.gnu.org/onlinedocs/gcc/Gcov.html">gcov</a><code>,</code> a tool you can use in conjunction with GCC to
test code coverage in your programs.<br />
<a href="http://ltp.sourceforge.net/coverage/lcov.php">lcov,</a> LCOV is a graphical front-end for GCC's coverage testing tool gcov.<br />
<br />
<br />
Instructions from codecov.io on how to use it with C, and clang or gcc. (codecov.io is free for public open source repos).<br />
<a href="https://github.com/codecov/example-c">https://github.com/codecov/example-c</a><br />
<br />
<br />
Here's documentation for how CPython gets coverage results for C.<br />
<a href="https://devguide.python.org/coverage/#measuring-coverage-of-c-code-with-gcov-and-lcov">https://devguide.python.org/coverage/#measuring-coverage-of-c-code-with-gcov-and-lcov</a><br />
<br />
Here is the CPython Travis CI configuration they use.<br />
https://github.com/python/cpython/blob/master/.travis.yml#L69<br />
<pre> - os: linux
language: c
compiler: gcc
env: OPTIONAL=true
addons:
apt:
packages:
- lcov
- xvfb
before_script:
- ./configure
- make coverage -s -j4
# Need a venv that can parse covered code.
- ./python -m venv venv
- ./venv/bin/python -m pip install -U coverage
- ./venv/bin/python -m test.pythoninfo
script:
# Skip tests that re-run the entire test suite.
- xvfb-run ./venv/bin/python -m coverage run --pylib -m test --fail-env-changed -uall,-cpu -x test_multiprocessing_fork -x test_multiprocessing_forkserver -x test_multiprocessing_spawn -x test_concurrent_futures
after_script: # Probably should be after_success once test suite updated to run under coverage.py.
# Make the `coverage` command available to Codecov w/ a version of Python that can parse all source files.
- source ./venv/bin/activate
- make coverage-lcov
- bash > (curl -s https://codecov.io/bash)
</pre>
<br />
<br />
<br />
<br />
<h3 style="text-align: left;">
Static analysis</h3>
<blockquote class="tr_bq">
"Static analysis has
not been helpful in finding
bugs in SQLite. More bugs have been introduced into SQLite while trying
to get it to compile without warnings than have been found by static
analysis." -- <a href="https://www.sqlite.org/testing.html">https://www.sqlite.org/testing.html</a></blockquote>
<br />
According to David Wheeler in "How to Prevent the next Heartbleed" (<a href="https://dwheeler.com/essays/heartbleed.html#static-not-found">https://dwheeler.com/essays/heartbleed.html#static-not-found</a>
the security problem with a logo, a website, and a marketing team) only
one static analysis tool found the Heartbleed vulnerability before it
was known. This tool is called CQual++. One reason for projects not
using these tools is that they have been (and some still are) hard to
use. The LLVM project only started using the clang static analysis tool
on it's <a href="http://lists.llvm.org/pipermail/cfe-dev/2019-August/063195.html">own projects recently</a> for example. However, since Heartbleed in 2014 tools have improved in both usability and their ability to detect issues.<br />
<br />
I
think it's generally accepted that static analysis tools are
incomplete, in that each tool does not guarantee detecting every problem
or even always detecting the same issues all the time. Using multiple
tools can therefore be said to find multiple different types of
problems. <br />
<br />
<br />
<h4 style="text-align: left;">
Compilers are kind of smart</h4>
The most basic of static analysis
tools are compilers themselves. Over the years they have been getting
more and more tools which used to only be available in dedicated Static
Analyzers and Lint tools. <br />
<blockquote class="tr_bq">
Variable shadowing and format-string
mismatches can be detected reliably and quickly is because both gcc and
clang do this detection as part of their regular compile. -- <a href="http://randomascii.wordpress.com/2013/09/09/vote-for-the-vc-improvements-that-matter/">Bruce Dawson</a> </blockquote>
Here we see two issues (which used to be) very common in C being detected by the two most popular C compilers themselves.<br />
<br />
Compiling code with gcc "<code>-Wall -Wextra -pedantic</code>" options catches quite a number of potential or actual problems (<a href="https://gcc.gnu.org/onlinedocs/gcc/Warning-Options.html" rel="nofollow">https://gcc.gnu.org/onlinedocs/gcc/Warning-Options.html</a>).
Other compilers check different things as well. So using multiple
compilers with their warnings can find plenty of different types of
issues for you.<br />
<h4 style="text-align: left;">
Compiler warnings should be turned in errors on CI.</h4>
By getting
your errors down to zero on Continuous Integration there is less chance
of new warnings being introduced that are missed in code review. There
are problems with distributing your code with warnings turned into
errors, so that should not be done.<br />
<br />
Some points for people implementing this:<br />
<ul>
<li><code>-Werror</code> can be used to turn warnings into errors</li>
<li><code>-Wno-error=unknown-pragmas</code></li>
<li>should run only in CI, and not in the build by default. See <a href="https://embeddedartistry.com/blog/2017/5/3/-werror-is-not-your-friend" rel="nofollow">werror-is-not-your-friend</a> (https://embeddedartistry.com/blog/2017/5/3/-werror-is-not-your-friend).</li>
<li>Use most recent gcc, and most recent clang (change two travis linux builders to do this).</li>
<li>first have to fix all the warnings (and hopefully not break something in the process).</li>
<li>consider adding extra warnings to gcc: <code>"-Wall -Wextra -Wpedantic"</code> See <a href="http://renesd.blogspot.com/2019/09/post-modern-c-tooling.html" rel="nofollow">C tooling</a></li>
<li>Also the Microsoft compiler MSVC on Appveyor can be configured to treat warnings as errors. The <code>/WX</code> argument option treats all warnings as errors. See <a href="https://docs.microsoft.com/en-us/cpp/build/reference/compiler-option-warning-level?view=vs-2019" rel="nofollow">MSVC warning levels</a></li>
<li>For MSVC on Appveyor, <code>/wdnnnn</code> Suppresses the compiler warning that is specified by nnnn. For example, /wd4326 suppresses compiler warning C4326.</li>
</ul>
If you run your code on different CPU architectures, these
compilers can find even more issues. For example 32bit/64bit Big Endian,
and Little Endian.<br />
<br />
<br />
<h3 style="text-align: left;">
Static analysis tool overview.</h3>
Static analysis can be much slower than the analysis
usually provided by compilation with gcc and clang. It trades off more CPU time for (hopefully)
better results.<br />
<br />
<br />
<a href="http://cppcheck.sourceforge.net/">cppcheck</a> focuses of low false positives and can find many actual problems.<br />
<a href="https://www.coverity.com/" rel="nofollow">Coverity</a>, a commercial static analyzer, free for open source developers</div>
<div>
<a href="http://www.cppdepend.com/" rel="nofollow">CppDepend</a>, a commercial static analyzer based on Clang</div>
<a href="https://github.com/Ericsson/codechecker">codechecker</a>, https://github.com/Ericsson/codechecker<br />
<a href="https://github.com/cpplint/cpplint">cpplint</a>, Cpplint is a command-line tool to check C/C++ files for style issues following <a href="http://google.github.io/styleguide/cppguide.html" rel="nofollow">Google's C++ style guide</a>.<br />
<a href="https://github.com/mre/awesome-static-analysis#cc">Awesome static analysis</a>, a page full of static analysis tools for C/C++. https://github.com/mre/awesome-static-analysis#cc<br />
<a href="https://www.viva64.com/en/pvs-studio/">PVS-Studio</a>, a commercial static analyzer, free for open source developers.<br />
<br />
<br />
<h4 style="text-align: left;">
The Clang Static Analyzer</h4>
<a href="http://clang-analyzer.llvm.org/" rel="nofollow">The Clang Static Analyzer</a> (<a href="http://clang-analyzer.llvm.org/" rel="nofollow">http://clang-analyzer.llvm.org/</a>) is a free to use static analyzer that is quite high quality.<br />
<blockquote>
The Clang Static Analyzer is a source code analysis tool that finds
bugs in C, C++, and Objective-C programs. Currently it can be run either
as a standalone tool or within Apple Xcode. The standalone tool is
invoked from the command line, and is intended to be run in tandem with a
build of a codebase.</blockquote>
The talk "Clang Static Analysis" (<a href="https://www.youtube.com/watch?v=UcxF6CVueDM">https://www.youtube.com/watch?v=UcxF6CVueDM</a>) talks about an LLVM tool called codechecker (https://github.com/Ericsson/codechecker).<br />
<br />
On MacOS an up to date scan-build and scan-view is included with the <code>brew install llvm</code>.<br />
<br />
<pre><code>$SCANBUILD=`ls /usr/local/Cellar/llvm/*/bin/scan-build`
$SCANBUILD -V python3 setup.py build
</code></pre>
<br />
On Ubuntu you can install scan-view with <code>apt-get install clang-tools.</code><br />
<br />
<br />
<h4 style="text-align: left;">
cppcheck </h4>
<blockquote>
Cppcheck is an analysis tool for C/C++ code. It provides unique code
analysis to detect bugs and focuses on detecting undefined behaviour and
dangerous coding constructs. The goal is to detect only real errors in
the code (i.e. have very few false positives).</blockquote>
<br />
The
quote below was particularly interesting to me because it echos the
sentiments of other developers, that testing will find more bugs. But
here is one of the static analysis tools saying so as well.<br />
<blockquote>
"You will find more bugs in your software by testing your software carefully, than by using Cppcheck."</blockquote>
<div style="text-align: left;">
<b>To Install cppcheck:</b></div>
<a href="http://cppcheck.sourceforge.net/" rel="nofollow">http://cppcheck.sourceforge.net/</a> and <a href="https://github.com/danmar/cppcheck">https://github.com/danmar/cppcheck</a><br />
The manual can be found here: <a href="http://cppcheck.net/manual.pdf" rel="nofollow">http://cppcheck.net/manual.pdf</a><br />
<br />
<div class="highlight highlight-source-shell">
<pre>brew install cppcheck bear
sudo apt-get install cppcheck bear</pre>
</div>
<div style="text-align: left;">
<br /></div>
<div style="text-align: left;">
<b>To run cppcheck on C code:</b></div>
You can use <a href="https://github.com/rizsotto/Bear">bear</a> (the
build ear) tool to record a compilation database
(compile_commands.json). cppcheck can then know what c files and header
files you are using.<br />
<br />
<div class="highlight highlight-source-shell">
<pre><span class="pl-c"><span class="pl-c">#</span> call your build tool, like `bear make` to record. </span>
<span class="pl-c"><span class="pl-c">#</span> See cppcheck manual for other C environments including Visual Studio.</span>
bear python setup.py build
cppcheck --quiet --language=c --enable=all -D__x86_64__ -D__LP64__ --project=compile_commands.json</pre>
</div>
<br />
It does seem to find some errors, and style improvements that other
tools do not suggest. Note that you can control the level of issues
found to errors, to portability and style issues plus more. See <code>cppcheck --help</code> and the manual for more details about <code>--enable</code> options.<br />
<br />
For example these ones from the pygame code base:<br />
<pre><code>[src_c/math.c:1134]: (style) The function 'vector_getw' is never used.
[src_c/base.c:1309]: (error) Pointer addition with NULL pointer.
[src_c/scrap_qnx.c:109]: (portability) Assigning a pointer to an integer is not portable.
[src_c/surface.c:832] -> [src_c/surface.c:819]: (warning) Either the condition '!surf' is redundant or there is possible null pointer dereference: surf.
</code></pre>
<br />
<h4 style="text-align: left;">
/Analyze in Microsoft Visual Studio</h4>
Visual studio by Microsoft can do static code analysis too. ( <a href="https://docs.microsoft.com/en-us/visualstudio/code-quality/code-analysis-for-c-cpp-overview?view=vs-2017">https://docs.microsoft.com/en-us/visualstudio/code-quality/code-analysis-for-c-cpp-overview?view=vs-2017</a>)<br />
<br />
"Using SAL annotations to reduce code defects." (<a href="https://docs.microsoft.com/en-us/visualstudio/code-quality/using-sal-annotations-to-reduce-c-cpp-code-defects?view=vs-2019">https://docs.microsoft.com/en-us/visualstudio/code-quality/using-sal-annotations-to-reduce-c-cpp-code-defects?view=vs-2019</a>)<br />
<blockquote class="tr_bq">
"In GNU C and C++, you can use function attributes to specify certain
function properties that may help the compiler optimize calls or
check code more carefully for correctness."</blockquote>
https://gcc.gnu.org/onlinedocs/gcc/Function-Attributes.html<br />
<br />
<h4 style="text-align: left;">
Custom static analysis for API usage</h4>
Probably
one of the most useful parts of static analysis is being able to write
your own checks. This allows you to do checks specific to your code base
in which general checks will not work. One example of this is the <a href="https://gcc-python-plugin.readthedocs.io/en/latest/cpychecker.html">gcc cpychecker</a>
(https://gcc-python-plugin.readthedocs.io/en/latest/cpychecker.html).
With this, gcc can find API usage issues within CPython extensions written in C. Including reference
counting bugs, and NULL pointer de-references, and other types of
issues. You can write custom checkers with LLVM as well in the "<a href="https://clang-analyzer.llvm.org/checker_dev_manual.html">Checker Developer Manual</a>" (https://clang-analyzer.llvm.org/checker_dev_manual.html)<br />
<br />
There is a list of <a href="https://gcc.gnu.org/wiki/plugins">GCC plugins</a> (https://gcc.gnu.org/wiki/plugins) among them are some Linux security plugins by <a class="https" href="https://www.grsecurity.net/features.php#tabs-gcc">grsecurity</a>.<br />
<br />
<br />
<h3 style="text-align: left;">
Runtime checks and Dynamic Verification</h3>
Dynamic verification
tools examine code whilst it is running. By running your code under
these dynamic verification tools you can help detect bugs. Either by
testing manually, or by running your automated tests under the watchful
eye of these tools. Runtime dynamic verification tools can detect
certain errors that static analysis tools can't.<br />
<br />
Some
of these tools are quite easy to add to a build in Continuous
Integration(CI). So you can run your automated tests with some extra
dynamic runtime verification enabled.<br />
<br />
Take a look at how easy they are to use?<br />
<blockquote class="tr_bq">
<pre>./configure CFLAGS="-fsanitize=address,undefined -g" LDFLAGS="-fsanitize=address,undefined"
make
make check</pre>
</blockquote>
<br />
<h4 style="text-align: left;">
Address Sanitizer</h4>
Doing a test run with an address sanitizer apparently helps to detect various types of bugs.<br />
<blockquote>
<a href="https://clang.llvm.org/docs/AddressSanitizer.html" rel="nofollow">AddressSanitizer</a>
is a fast memory error detector. It consists of a compiler
instrumentation module and a run-time library. The tool can detect the
following types of bugs:</blockquote>
<ul>
<li>Out-of-bounds accesses to heap, stack and globals</li>
<li>Use-after-free</li>
<li>Use-after-return (runtime flag ASAN_OPTIONS=detect_stack_use_after_return=1)</li>
<li>Use-after-scope (clang flag -fsanitize-address-use-after-scope)</li>
<li>Double-free, invalid free</li>
<li>Memory leaks (experimental)</li>
</ul>
How to compile a python C extension with clang on MacOS:<br />
<div class="highlight highlight-source-shell">
<pre>LDFLAGS=<span class="pl-s"><span class="pl-pds">"</span>-g -fsanitize=address<span class="pl-pds">"</span></span> CFLAGS=<span class="pl-s"><span class="pl-pds">"</span>-g -fsanitize=address -fno-omit-frame-pointer<span class="pl-pds">"</span></span> python3 setup.py install</pre>
</div>
<br />
<br />
<br />
<h4 style="text-align: left;">
Undefined Behaviour Sanitizer</h4>
From <a href="https://clang.llvm.org/docs/UndefinedBehaviorSanitizer.html" rel="nofollow">https://clang.llvm.org/docs/UndefinedBehaviorSanitizer.html</a><br />
<div class="preview-content">
<div class="comment js-suggested-changes-container" data-thread-side="">
<div class="comment-body markdown-body js-preview-body" style="min-height: 638px;">
<blockquote>
<a href="https://clang.llvm.org/docs/UndefinedBehaviorSanitizer.html" rel="nofollow">UndefinedBehaviorSanitizer (UBSan)</a>
is a fast undefined behavior detector. UBSan modifies the program at
compile-time to catch various kinds of undefined behavior during program
execution, for example:</blockquote>
<ul>
<li>Using misaligned or null pointer</li>
<li>Signed integer overflow</li>
<li>Conversion to, from, or between floating-point types which would overflow the destination</li>
</ul>
You can use the Undefined Behaviour Sanitizer with clang and gcc. Here is the <a href="https://gcc.gnu.org/onlinedocs/gcc/Instrumentation-Options.html" rel="nofollow">gcc documentation for Instrumentation Options and UBSAN</a> (<a href="https://gcc.gnu.org/onlinedocs/gcc/Instrumentation-Options.html" rel="nofollow">https://gcc.gnu.org/onlinedocs/gcc/Instrumentation-Options.html</a>).<br />
<br />
From <a href="https://www.sqlite.org/testing.html" rel="nofollow">https://www.sqlite.org/testing.html</a><br />
<blockquote>
To help ensure that SQLite does not make use of undefined or
implementation defined behavior, the test suites are rerun using
instrumented builds that try to detect undefined behavior. For example,
test suites are run using the "-ftrapv" option of GCC. And they are run
again using the "-fsanitize=undefined" option on Clang. And again using
the "/RTC1" option in MSVC</blockquote>
To compile a python C extension with a UBSAN with clang on Mac do:<br />
<pre><code>LDFLAGS="-g -fsanitize=undefined" CFLAGS="-g -fsanitize=undefined -fno-omit-frame-pointer" python3 setup.py install
</code></pre>
<br />
<h4 style="text-align: left;">
Microsoft Visual Studio Runtime Error Checks</h4>
The Microsoft Visual Studio compiler can use the Run Time Error Checks feature to find some issues. <a href="https://docs.microsoft.com/en-us/cpp/build/reference/rtc-run-time-error-checks?view=vs-2019" rel="nofollow">/RTC (Run-Time Error Checks)</a> (<a href="https://docs.microsoft.com/en-us/cpp/build/reference/rtc-run-time-error-checks?view=vs-2019" rel="nofollow">https://docs.microsoft.com/en-us/cpp/build/reference/rtc-run-time-error-checks?view=vs-2019</a>)<br />
<br />
From <a href="https://docs.microsoft.com/en-us/visualstudio/debugger/how-to-use-native-run-time-checks?view=vs-2019" rel="nofollow">How to: Use Native Run-Time Checks</a> (<a href="https://docs.microsoft.com/en-us/visualstudio/debugger/how-to-use-native-run-time-checks?view=vs-2019" rel="nofollow">https://docs.microsoft.com/en-us/visualstudio/debugger/how-to-use-native-run-time-checks?view=vs-2019)</a><br />
<blockquote class="tr_bq">
<ul>
<li>Stack pointer corruption.</li>
<li>Overruns of local arrays.</li>
<li>Stack corruption.</li>
<li>Dependencies on uninitialized local variables.</li>
<li>Loss of data on an assignment to a shorter variable.</li>
</ul>
</blockquote>
</div>
</div>
</div>
<br />
<h4 style="text-align: left;">
App Verifier</h4>
<blockquote class="tr_bq">
"Any Windows developers that are listening to this: <i><b>if you’re not using App Verifier, you are making a mistake</b></i>." -- Bruce Dawson</blockquote>
Stangely App Verifier does not have very good online documentation. The best article available online about it is: <a href="https://randomascii.wordpress.com/2011/12/07/increased-reliability-through-more-crashes/">Increased Reliability Through More Crashes</a> (https://randomascii.wordpress.com/2011/12/07/increased-reliability-through-more-crashes/)<br />
<blockquote class="tr_bq">
Application Verifier (AppVerif.exe) is a <i>dynamic verification</i>
tool for user-mode applications. This tool monitors application actions
while the application runs, subjects the application to a variety of
stresses and tests, and generates a report about potential errors in
application execution or design.</blockquote>
https://docs.microsoft.com/en-us/windows-hardware/drivers/debugger/application-verifier<br />
https://docs.microsoft.com/en-us/windows/win32/win7appqual/application-verifier<br />
https://docs.microsoft.com/en-us/security-risk-detection/concepts/application-verifier<br />
<br />
<ul style="text-align: left;">
<li>Buffer over runs</li>
<li>Use after free issues</li>
<li>Thread issues including using TLS properly.</li>
<li>Low resource handling</li>
<li>Race conditions</li>
</ul>
If you have a memory corruption bug, App Verifier might be able
to help you find it. If you are using Windows APIs wrong, have some
threading issues, want to make sure you app runs under harsh conditions
-- App Verifier might help you find it.<br />
<br />
Related to App Verifier is the <a href="https://support.microsoft.com/en-us/help/286470/how-to-use-pageheap-exe-in-windows-xp-windows-2000-and-windows-server">PageHeap</a>
tool(https://support.microsoft.com/en-us/help/286470/how-to-use-pageheap-exe-in-windows-xp-windows-2000-and-windows-server)
It helps you find memory heap corruptions on Windows.<br />
<br />
<br />
<br />
<br />
<h3 style="text-align: left;">
Performance profiling and measurement</h3>
<blockquote class="tr_bq">
“The
objective (not always attained) in creating high-performance software
is to make the software able to carry out its appointed tasks so rapidly
that it responds instantaneously, as far as the user is concerned.”
Michael Abrash. “Michael Abrash’s Graphics Programming Black Book.”</blockquote>
Reducing
energy usage, and run time requirements of apps can often be a
requirement or very necessary. For a mobile or embedded application it
can mean the difference of being able to run the program at all.
Performance can directly be related to user happiness but also to the
financial performance of a piece of software.<br />
<br />
But how
to we measure the performance of a program, and how to we know what
parts of a program need improvement? Tooling can help.<br />
<br />
<h4 style="text-align: left;">
Valgrind</h4>
Valgrind has its own section here because it does lots
of different things for us. It's a great tool, or set of tools for
improving your programs. It used to be available only on linux, but is
now also available on MacOS.<br />
<br />
Apparently Valgrind would have caught the heartbleed issue if it was used with a fuzzer.<br />
<br />
http://valgrind.org/docs/manual/quick-start.html <br />
<br />
<h4 style="text-align: left;">
Apple Performance Tools</h4>
Apple provides many <a href="https://developer.apple.com/library/archive/documentation/Performance/Conceptual/PerformanceOverview/PerformanceTools/PerformanceTools.html">performance related development tools</a>. Along with the gcc and llvm based tools, the main tool is called <a href="https://help.apple.com/instruments/mac/current/">Instruments</a>.
Instruments (part of Xcode) allows you to record and analyse programs
for lots of different aspects of performance - including graphics,
memory activity, file system, energy and other program events. By being
able to record and analyse different types of events together can make
it convienient to find performance issues.<br />
<br />
<br />
<h4 style="text-align: left;">
LLVM performance tools.</h4>
Many of the low level parts of the tools
in XCode are made open source through the LLVM project. See "LLVM
Machine Code Analyzer" (
https://llvm.org/docs/CommandGuide/llvm-mca.html) as one example. See
the <a href="https://llvm.org/docs/XRayExample.html">LLVM XRay instrumentation</a> (https://llvm.org/docs/XRayExample.html).<br />
<br />
There's also an interesting talk on XRay here "<a href="https://www.youtube.com/watch?v=jyL-__zOGcU">XRay in LLVM: Function Call Tracing and Analysis</a>" (https://www.youtube.com/watch?v=jyL-__zOGcU) by <i>Dean Michael Berris.</i><br />
<br />
<br />
<h4 style="text-align: left;">
GNU/Linux tools</h4>
<br />
<br />
<h4 style="text-align: left;">
Microsoft performance tools.</h4>
<br />
<h4 style="text-align: left;">
Intel performance tools.</h4>
https://software.intel.com/en-us/vtune<br />
<br />
<br />
<br />
<br />
<h3 style="text-align: left;">
Caching builds</h3>
https://ccache.samba.org/<br />
<br />
ccache
is very useful for reducing the compile time of large C projects.
Especially when you are doing a 'rebuild from scratch'. This is because
ccache can cache the compilation of parts in this situation when the
files do not change.<br />
http://itscompiling.eu/2017/02/19/speed-cpp-compilation-compiler-cache/<br />
<br />
This is also useful for speeding up CI builds, and especially when large parts of the code base rarely change.<br />
<br />
<br />
<h3 style="text-align: left;">
Distributed building.</h3>
<br />
distcc https://github.com/distcc/distcc<br />
icecream https://github.com/icecc/icecream<br />
<br />
<br />
<h3 style="text-align: left;">
Complexity of code.</h3>
<blockquote class="tr_bq">
"Complex is better
than complicated. It's OK to build very complex software, but you don't
have to build it in a complicated way. Lizard is a free open source tool
that analyse the complexity of your source code right away supporting
many programming languages, without any extra setup. It also does code
clone / copy-paste detection."</blockquote>
Lizard is a modern complexity command line tool,<br />
that also has a website UI: <a href="http://www.lizard.ws/">http://www.lizard.ws/</a><br />
<a href="https://github.com/terryyin/lizard">https://github.com/terryyin/lizard</a><br />
<br />
<blockquote class="tr_bq">
<span style="font-family: "courier new" , "courier" , monospace;"># install lizard</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">python3 -m pip install lizard</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"># show warnings only and include/exclude some files.</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">lizard src_c/ -x"src_c/_sdl2*" -w </span></blockquote>
<blockquote class="tr_bq">
<br />
<span style="font-family: "courier new" , "courier" , monospace;"># Can also run it as a python module.</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">python3 -m lizard src_c/ -x"src_c/_sdl2*" -w</span><br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;"># Show a full report, not just warnings (-w). </span><br />
<span style="font-family: "courier new" , "courier" , monospace;">lizard src_c/ -x"src_c/_sdl2*" -x"src_c/_*" -x"src_c/SDL_gfx*" -x"src_c/pypm.c"</span></blockquote>
<br />
Want
people to understand your code? Want Static Analyzers to understand
your code better? Want to be able to test your code more easily? Want
your code to run faster because of less branches? Then <b><i>you may want to find complicated code and refactor it</i></b>.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjH7xX-NRWqQZNDSxtw0Auox2rTY7_bccq43yHKrnp9RexygrdPJYWHI2-A0QK0EDhSCFrHHcj251Nz8hjsKgjzroeawF3_Q2pXLhEEBUwgNzb1a5EU5B7dnBhqOEQ-Xur5Lr-a/s1600/pygame_C_word_cloud.png" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="1414" data-original-width="1418" height="319" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjH7xX-NRWqQZNDSxtw0Auox2rTY7_bccq43yHKrnp9RexygrdPJYWHI2-A0QK0EDhSCFrHHcj251Nz8hjsKgjzroeawF3_Q2pXLhEEBUwgNzb1a5EU5B7dnBhqOEQ-Xur5Lr-a/s320/pygame_C_word_cloud.png" width="320" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Lizard can also make a pretty word cloud from your source.</td></tr>
</tbody></table>
<br />
Lizard complexity analysis can be run in
Continuous Integration (CI). You can also give it lists of functions to
ignore and skip if you can't refactor some function right away. Perhaps
you want to stop new complex functions from entering your codebase? To
stop new complex functions via CI make a supression list of all the
current warnings and then make your CI use that and fail if there are
new warnings.<br />
<br />
<br />
<br />
<h3 style="text-align: left;">
Testing your code on different OS/architectures.</h3>
Sometimes you
need to be able to fix an issue on an OS or architecture that you don't
have access to. Luckily these days there are many tools available to
quickly use a different system through emulation, or container
technology. <br />
<br />
<br />
Vagrant<br />
Virtualbox<br />
Docker<br />
Launchpad, compile and run tests on many architectures.<br />
Mini cloud (ppc machines for debugging)<br />
<br />
If you pay Travis CI, they allow you to connect to the testing host with ssh when a test fails.<br />
<br />
<br />
<h3 style="text-align: left;">
Code Formatting</h3>
clang-format<br />
<br />
clang-format - rather than manually fix
various formatting errors found with a linter, many projects are just
using clang-format to format the code into some coding standard.<br />
<br />
<br />
<br />
<h3 style="text-align: left;">
Services</h3>
LGTM is an
'automated code review tool' with github (and other code repos) support.
https://lgtm.com/help/lgtm/about-automated-code-review<br />
<br />
Coveralls provides a store for test coverage results with github (and other code repos) support. https://coveralls.io/<br />
<br />
<br />
<br />
<br />
<h3 style="text-align: left;">
Coding standards for C</h3>
There are lots of coding standards for C, and there are tools to check them.<br />
<br />
<br />
An older set of standards is the <a href="https://en.wikipedia.org/wiki/MISRA_C">MISRA_C</a> (https://en.wikipedia.org/wiki/MISRA_C) aims to facilitate code safety, security, and portability for embedded systems.<br />
<br />
The <a href="https://www.kernel.org/doc/html/v4.10/process/coding-style.html">Linux Kernel Coding standard</a>
(https://www.kernel.org/doc/html/v4.10/process/coding-style.html) is
well known mainly because of the popularity of the Linux Kernel. But
this is mainly concerned with readability.<br />
<br />
A newer one is the <a href="https://wiki.sei.cmu.edu/confluence/display/seccode/SEI+CERT+Coding+Standards">CERT C coding standard</a>
(https://wiki.sei.cmu.edu/confluence/display/seccode/SEI+CERT+Coding+Standards),
and it is a secure coding standard (not a safety one). <br />
<br />
The
website for the CERT C coding standard is quite amazing. It links to
tools that can detect each of the problems automatically (when they can
be). It is very well researched, and links each problem to other
relevant standards, and gives issues priorities. A good video to watch
on CERT C is "<a href="https://www.youtube.com/watch?v=awY0iJOkrg4">How Can I Enforce the SEI CERT C Coding Standard Using Static Analysis?</a>"
(https://www.youtube.com/watch?v=awY0iJOkrg4). They do releases of the
website, which is edited as a wiki. At the time of writing the last
release into book form was in 2016.<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<h3 style="text-align: left;">
How are other projects tested?</h3>
We can learn a lot by how other C projects are going about their business today.<br />
Also,
thanks to CI testing tools defining things in code we can see how
automated tests are run on services like Travis CI and Appveyor.<br />
<h4 style="text-align: left;">
SQLite</h4>
"<a href="https://www.sqlite.org/testing.html">How SQLite Is Tested</a>" <br />
<h4 style="text-align: left;">
Curl</h4>
"<a href="https://daniel.haxx.se/blog/2017/10/12/testing-curl/">Testing Curl</a>" <br />
https://github.com/curl/curl/blob/master/.travis.yml <br />
<h4 style="text-align: left;">
Python</h4>
"<a href="https://discuss.python.org/t/how-is-cpython-tested/2256">How is CPython tested?</a>"<br />
https://github.com/python/cpython/blob/master/.travis.yml<br />
<h4 style="text-align: left;">
OpenSSL</h4>
"<a href="https://github.com/openssl/openssl/issues/9831">How is OpenSSL tested?</a>" <br />
<br />
https://github.com/openssl/openssl/blob/master/.travis.yml<br />
They use Coverity too: https://github.com/openssl/openssl/pull/9805<br />
https://github.com/openssl/openssl/blob/master/fuzz/README.md<br />
<h4 style="text-align: left;">
libsdl </h4>
"<a href="https://discourse.libsdl.org/t/how-is-sdl-tested/26576">How is SDL tested?</a>" [No response]<br />
<h4 style="text-align: left;">
Linux</h4>
<a href="https://stackoverflow.com/questions/3177338/how-is-the-linux-kernel-tested">https://stackoverflow.com/questions/3177338/how-is-the-linux-kernel-tested</a><a href="https://www.linuxjournal.com/content/linux-kernel-testing-and-debugging">https://www.linuxjournal.com/content/linux-kernel-testing-and-debugging</a><br />
<br />
<h4 style="text-align: left;">
Haproxy</h4>
<a href="https://github.com/haproxy/haproxy/blob/master/.travis.yml">https://github.com/haproxy/haproxy/blob/master/.travis.yml</a><br />
<br />
<br />
<br />
There is some discussion of <a href="https://www.reddit.com/r/C_Programming/comments/dey8bq/post_modern_c_tooling/">Post Modern C Tooling</a> on the "C_Programming" reddit forum.<br />
<br />
<br />
<br />
<h2 style="text-align: left;">
<a href="https://pygame.itch.io/pygame-book">pygame book</a></h2>
This
article will be part of a book called "pygame 4000". A book dedicated
to the joy of making software for making. Teaching collaboration, low
level programming in C, high level programming in Python, GPU graphics
programming with a shader language, design, music, tools, quality, and
shipping(software).<br />
<br />
It's a bit of a weird book. There's a little
bit of swearing in it (consider yourself fucking warned), and all
profits go towards pygame development (the library, the community, and
the website).<br />
</div>
Unknownnoreply@blogger.com166tag:blogger.com,1999:blog-10678074.post-49233759913741573222019-11-06T10:47:00.002+00:002020-12-12T17:29:56.012+00:00Draft 3 of, ^Let's write a unit test!^<div dir="ltr" style="text-align: left;" trbidi="on">
<blockquote class="tr_bq">
<span style="font-family: "courier new" , "courier" , monospace;">So, I started writing this for people who want to 'contribute' to Community projects, and also Free Libre or Open source projects. Maybe you'd like to get involved, but are unsure of where to begin? Follow along with this tutorial, and peek at the end in the "what is a git for?" section for explanations of what some of the words mean. </span></blockquote>
Draft 1, 2018/07/18 - initial draft.<br />
Draft 2, 2019/11/04 - two full unit test examples, assertions, making a pull request, use python 3 unittest substring search, "good first issue" is a thing now. Started "What is a git for? Jargon" section. <br />
Draft 3, 2020/12/12 - default branch name is main.</div><div dir="ltr" style="text-align: left;" trbidi="on"><br />
</div><div dir="ltr" style="text-align: left;" trbidi="on"></div><div dir="ltr" style="text-align: left;" trbidi="on"><br />
<h3 style="text-align: left;">
What's first? A test is first.</h3>
A unit test is a piece of code which tests one thing works well in
isolation from other parts of software. In this guide, I'm going to
explain how to write one using the standard python unittest module, for
the <a href="https://www.pygame.org/">pygame</a> game library. You can apply this advice to most python projects, or free/libre open source projects in general.<br />
<h3 style="text-align: left;">
A minimal test. </h3>
<i><b>What pygame.draw.ellipse should do</b></i>: <a href="http://www.pygame.org/docs/ref/draw.html#pygame.draw.ellipse">http://www.pygame.org/docs/ref/draw.html#pygame.draw.ellipse</a><br />
<i><b>Where to put the test</b></i>: <a href="https://github.com/pygame/pygame/blob/main/test/draw_test.py">https://github.com/pygame/pygame/blob/main/test/draw_test.py</a><br />
<blockquote class="tr_bq">
<br />
<pre>def test_ellipse(self):
import pygame.draw
surf = pygame.Surface((320, 200))
pygame.draw.ellipse(surf, (255, 0, 0), (10, 10, 25, 20))
</pre>
</blockquote>
<br />
All the test does is call the draw function on the surface with a color,
and a rectangle. That's it. A minimal, useful test. If you have a
github account, you can even edit the test file in the browser to submit
your PR. If you have email, or internet access you can email me or
someone else on the internet and ask them to do add it to pygame.<br />
<br />
An easy test to write... but it provides really good value.<br />
<ul style="text-align: left;">
<li>Shows an example of using the code.</li>
<li>Makes sure the function arguments are correct.</li>
<li>Makes sure the code runs on 20+ different platforms and python versions.</li>
<li>No "regressions" (Code that starts failing because of a change) can be introduced in the future. The code for draw ellipse with these arguments should not crash in the future.</li>
</ul>
<br />
<h3 style="text-align: left;">
But why write a unit test anyway?</h3>
Unit tests help pygame make
sure things don't break on multiple platforms. When your code is running
on dozens of CPUs and just as many operating systems things get a
little tricky to test manually. So we write a unit test and let all the
build robots do that work for us.<br />
<br />
A great way to
contribute to libre/free and open source projects is to contribute a
test. Less bugs in the library means less bugs in your own code.
Additionally, you get some public credit for your contribution.<br />
<br />
The
best part about it, is that it's a great way to learn python, and about
the thing you are testing. Want to know how graphics algorithms should
work, in lots of detail? Start writing tests for them.<br />
<blockquote class="tr_bq">
<span style="font-size: medium;"><b><i>The simplest test is to just call the function. Just calling it is a great first test. Easy, and useful.</i></b></span></blockquote>
<br />
At
the time of writing there are 39 functions that aren't even called when
running the pygame tests. Why not join me on this adventure?<br />
<br />
<br />
<h3 style="text-align: left;">
Let's write a unit test!</h3>
In this guide I'm going to write a test for an pygame.draw.ellipse to
make sure a thick circle has the correct colors in it, and not lots of
black spots. There's a bunch of tips and tricks to help you along your
way. Whilst you can just edit a test in your web browser, and submit a
PR, it might be more comfortable to do it in your normal development
environment.<br />
<br />
<h4 style="text-align: left;">
Grab a fork, and let's dig in.</h4>
<a href="https://help.github.com/articles/set-up-git/">Set up git for github</a> if you haven't already. Then you'll want to 'fork' pygame on <a href="https://github.com/pygame/pygame">https://github.com/pygame/pygame</a> so you have your own local copy. <br />
<blockquote class="tr_bq">
<div style="text-align: left;">
<i>Note, we also accept patches by email, or on github issues. So you can <b>skip all this github business</b> if you want to. <a href="https://www.pygame.org/wiki/patchesandbugs">https://www.pygame.org/wiki/patchesandbugs</a></i></div>
</blockquote>
<ul>
<li>Fork the repository (see top right of the <a href="https://github.com/pygame/pygame">pygame repo page</a>) <div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhEUgHEePZnf0BJwwzQ6O5cYBbRcYkjV_-vOAIVPaGefj-RmLKaYK8jZO_uXkwBe9eHDXV3WnuCLJyQfOHs51-nxIZJbUYiPniQfrsRbF3YOgdXDJXZZD3rp7PzUpMqkS0oPA2d/s1600/Screen+Shot+2018-02-19+at+17.38.48.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="250" data-original-width="652" height="122" src="https://web.archive.org/web/20191007225130im_/https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhEUgHEePZnf0BJwwzQ6O5cYBbRcYkjV_-vOAIVPaGefj-RmLKaYK8jZO_uXkwBe9eHDXV3WnuCLJyQfOHs51-nxIZJbUYiPniQfrsRbF3YOgdXDJXZZD3rp7PzUpMqkS0oPA2d/s320/Screen+Shot+2018-02-19+at+17.38.48.png" width="320" /></a></div>
</li>
<li>Make the change locally. Push to your copy of the fork.</li>
<li>Submit a <i>pull request</i> </li>
</ul>
So you've forked the repo, and now you can clone your own copy of the git repo locally.<br />
<br />
<blockquote class="tr_bq">
<pre>$ git clone https://github.com/YOUR-USERNAME/pygame
$ cd pygame/
$ python test/draw_test.py
...
----------------------------------------------------------------------
Ran 3 tests in 0.007s
OK
</pre>
</blockquote>
<br />
You'll see all of the tests in the test/ folder.<br />
<br />
Browse the test folder online: <a href="https://github.com/pygame/pygame/tree/main/test">https://github.com/pygame/pygame/tree/main/test</a><br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi1ilFqr7MZEyGQMeyqrbGGVOwNstQkNo_D3XPgCgFG5qM8QlfBtLUKEsbNUwnrU-_99eSbTmT3JhEm7ulSBqj897XvuOWF_wO5Ji07Ntl5ciffh-xmB1E14TXl7Vfw9tXlukSi/s1600/Screen+Shot+2018-02-19+at+17.53.39.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1188" data-original-width="564" height="320" src="https://web.archive.org/web/20191007225130im_/https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi1ilFqr7MZEyGQMeyqrbGGVOwNstQkNo_D3XPgCgFG5qM8QlfBtLUKEsbNUwnrU-_99eSbTmT3JhEm7ulSBqj897XvuOWF_wO5Ji07Ntl5ciffh-xmB1E14TXl7Vfw9tXlukSi/s320/Screen+Shot+2018-02-19+at+17.53.39.png" width="150" /></a><br />
<br />
If you have an older version of pygame, you can use this little program to see the issue.<br />
<br />
<br />
There is some more extensive documentation in the test/README file.
Including on how to write a test that requires manual interaction.<br />
<br />
<br />
<h3 style="text-align: left;">
Standard unittest module.</h3>
pygame uses the standard python unittest module. With a few enhancements to make it nicer for developing C code.<br />
<blockquote class="tr_bq">
<i>Fun fact: pygame included the unit testing module before python did.</i></blockquote>
We will go over the basics in this guide, but for more detailed information please see:<br />
<a href="https://docs.python.org/3/library/unittest.html">https://docs.python.org/3/library/unittest.html</a><br />
<br />
<br />
<br />
<h3 style="text-align: left;">
How to run a single test?</h3>
Running all the tests at once can take a while. What if you just want to run a single test?<br />
<br />
If we look inside draw_test.py, each test is a class name, and a
function. There is a "DrawModuleTest" class, and there should be a "def
test_ellipse" function.<br />
<br />
<h4 style="text-align: left;">
So, let's run the test... </h4>
<pre>~/pygame/ $ python test/draw_test.py DrawModuleTest.test_ellipse
Traceback (most recent call last):
...
AttributeError: type object 'DrawModuleTest' has no attribute 'test_ellipse'
</pre>
<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhNwcXYHfjlaho1IhIqUvWli5U-G05zUjHW9HXwxgjsZnaorigQWZ8dCkyWP5dIDdEPTRl822zEdVySC8QnCm1HG1ntmklyF5IUZ62lht99onG9J00X2xRRuDmc0763sIxfvdUw/s1600/Screen+Shot+2018-03-05+at+09.20.52.png" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="422" data-original-width="1600" height="84" src="https://web.archive.org/web/20191007225130im_/https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhNwcXYHfjlaho1IhIqUvWli5U-G05zUjHW9HXwxgjsZnaorigQWZ8dCkyWP5dIDdEPTRl822zEdVySC8QnCm1HG1ntmklyF5IUZ62lht99onG9J00X2xRRuDmc0763sIxfvdUw/s320/Screen+Shot+2018-03-05+at+09.20.52.png" width="320" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Starting with failure. Our test isn't there yet.</td></tr>
</tbody></table>
<br />
Good. This fails. It's because we don't have a test called "def
test_ellipse" in there yet. What there is, is a method called
'todo_test_ellipse'. This is an extension pygame testing framework has
so we can easily see which functionality we still need to write tests
for.<br />
<br />
<pre>~/pygame/ $ python -m pygame.tests --incomplete</pre>
... <br />
FAILED (errors=39)<br />
<br />
Looks like there are currently 39 functions or methods without a test. Easy pickings.<br />
<br />
<h4 style="text-align: left;">
Python 3 to the rescue.</h4>
Tip: Python 3.7 makes it easier to run tests with the magic "-k" argument. With this you can run tests that match a substring. So to run all the tests with "ellipse" in their name you can do this:<br />
<br />
<pre>~pygame/ $ python3 test/draw_test.py -k ellipse</pre>
<br />
<br />
<br />
<h3 style="text-align: left;">
Digression: Good first issue, low hanging fruit, and help wanted. </h3>
<div style="text-align: center;">
<blockquote class="tr_bq">
<i>Something that's easy to do.</i></blockquote>
</div>
<br />
A little digression for a moment... what is a good first issue?<br />
<br />
Low hanging fruit is easy to get off the tree. You don't need a ladder,
or robot arms with a claw on the end. So I guess that's what people are
talking about in the programming world when they say "low hanging
fruit".<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://github.com/pygame/pygame/issues?q=is%3Aissue+is%3Aopen+label%3A%22Low+Hanging+Fruit%22" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="984" data-original-width="1600" height="196" src="https://web.archive.org/web/20191007225130im_/https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhSPxoTmGx7TUQgdbnm69t5MR5IkrGpGUzAxXHUgkOmaV2yQUwEAOco1_odZVzZutkJxIW0mVVJIXYb9hZFLl0fhhlj6RMlanrlbQYucHROm9WlM-POReSo1A0Eg7bJvijYFOqb/s320/Screen+Shot+2018-03-05+at+10.28.10.png" width="320" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><a href="https://github.com/pygame/pygame/issues?q=is%3Aissue+is%3Aopen+label%3A%22Low+Hanging+Fruit%22">pygame low hanging fruit</a></td></tr>
</tbody></table>
<br />
<br />
Many projects keep a list of "good first issue", "low hanging fruit", or "help wanted" labeled issues. Like the <a href="https://github.com/pygame/pygame/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22">pygame "good first issue"</a>
list. Ones other people don't think will be all that super hard to do.
If you can't find any on there labeled like this, then ask them. Perhaps
they'll know of something easy to do, but haven't had the time to mark
one yet.<br />
<br />
One little trick is that writing a simple test is quite easy for most
projects. So if they don't have any marked "low hanging fruit", or "good first issue" go take a
look in their test folder and see if you can add something in there.<br />
<br />
Don't be afraid to ask questions. If you look at an issue, and you can't
figure it out, or get stuck on something, ask a nice question in there
for help.<br />
<h4 style="text-align: left;">
Digression: Contribution guide.</h4>
There's usually also a contribution guide. Like the <a href="https://www.pygame.org/wiki/Contribute">pygame Contribute</a> wiki page. Or it may be called developer docs, or there may be a
CONTRIBUTING.md file in the source code repository. Often there is a
separate place the developers talk on. For pygame it is the pygame
mailing list, but there is also a chat server which is a bit more informal.<br />
<br />
<h3 style="text-align: left;">
A full example of a test.</h3>
The unittest module arranges tests inside functions that start with "test_" that live in a class.<br />
<br />
Here is a full example:<br />
<br />
<pre>import unittest
class TestEllipse(unittest.TestCase):
def test_ellipse(self):
import pygame.draw
surf = pygame.Surface((320, 200))
pygame.draw.ellipse(surf, (255, 0, 0), (10, 10, 25, 20))
if __name__ == '__main__':
unittest.main()
</pre>
<br />
You can save that in a file yourself(test_draw1.py for example) and run it to see if it passes.<br />
<br />
<h3 style="text-align: left;">
Committing your test, and making a Pull Request.</h3>
Here you need to make sure you have "git" setup. Also you should have "forked" the repo you want to make changes on, and done a 'git clone' of it.<br />
<br />
<pre># create a "branch"
git checkout -b my-draw-test-branch
# save your changes locally.
git commit test/draw_test.py -m "test for the draw.ellipse function"
# push your changes
git push origin my-draw-test-branch
</pre>
<br />
<br />
Here we see a screenshot of a terminal running these commands.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhEQUUiONHYtybLTOmFKFCPqZW8vL_WIW4UQbuEwDlGdKH3SvcamyOFB1ZwPYjp-vuMTq2a0YHdrHyKJW-449fXx8N-WYzgDNj_pNS4rPg8MslKkWbCtNYH74RwQMIkQaXJmaya/s1600/Screen+Shot+2019-11-04+at+19.48.45.png" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="684" data-original-width="984" height="277" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhEQUUiONHYtybLTOmFKFCPqZW8vL_WIW4UQbuEwDlGdKH3SvcamyOFB1ZwPYjp-vuMTq2a0YHdrHyKJW-449fXx8N-WYzgDNj_pNS4rPg8MslKkWbCtNYH74RwQMIkQaXJmaya/s400/Screen+Shot+2019-11-04+at+19.48.45.png" width="400" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Here we see the commands to commit something and push it up to a repo.</td></tr>
</tbody></table>
When you push your changes, it will print out some progress, and then give you a URL at which you can create a "pull request".<br />
<br />
When you git push it prints out these instructions:
<br />
<pre>remote: Create a pull request for 'my-draw-test-branch' on GitHub by visiting:
remote: https://github.com/YOURUSERNAME/pygame/pull/new/my-draw-test-branch
</pre>
<br />
<br />
You can also go to your online fork to create a pull request there.<br />
<br />
<h3 style="text-align: left;">
Writing your pull request text.</h3>
When you create a pull request, you are saying "hey, I made these changes. Do you want them? What do you think? Do you want me to change anything? Is this ok?"<br />
<br />
It's usually good to link your pull request to an "issue". Maybe you're starting to fix an existing problem with the code.<br />
<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg9hsi6AfZWNYe8gDES-6aQiXxuCV2pPeIMlMEQISkitfE7cIFBUNKSd0j7p_85g2DsIedN8GQ9pyFCR3ecm6EHRVUuIaoz-BW8QprTijlPVttviP9-DniqXj8_iOT6IweiP4fd/s1600/Screen+Shot+2019-11-04+at+18.51.37.png" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="790" data-original-width="1510" height="167" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg9hsi6AfZWNYe8gDES-6aQiXxuCV2pPeIMlMEQISkitfE7cIFBUNKSd0j7p_85g2DsIedN8GQ9pyFCR3ecm6EHRVUuIaoz-BW8QprTijlPVttviP9-DniqXj8_iOT6IweiP4fd/s320/Screen+Shot+2019-11-04+at+18.51.37.png" width="320" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Different "checks" are run by robots to try and catch problems before the code is merged in.</td></tr>
</tbody></table>
<br />
<br />
<br />
<h4 style="text-align: left;">
Testing the result with assertEquals.</h4>
<br />
How about it we want to test if the draw function actually draws something?<br />
Put this code into test_draw2.py <br />
<br />
<br />
<pre>import unittest
class TestEllipse(unittest.TestCase):
def test_ellipse(self):
import pygame.draw
black = pygame.Color('black')
red = pygame.Color('red')
surf = pygame.Surface((320, 200))
surf.fill(black)
# The area the ellipse is contained in, is held by rect.
#
# 10 pixels from the left,
# 11 pixels from the top.
# 225 pixels wide.
# 95 pixels high.
rect = (10, 11, 225, 95)
pygame.draw.ellipse(surf, red, rect)
# To see what is drawn you can save the image.
# pygame.image.save(surf, "test_draw2_image.png")
# The ellipse should not draw over the black in the top left spot.
self.assertEqual(surf.get_at((0, 0)), black)
# It should be red in the middle of the ellipse.
middle_of_ellipse = (125, 55)
self.assertEqual(surf.get_at(middle_of_ellipse), red)
if __name__ == '__main__':
unittest.main()
</pre>
<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgVkezopuZF_soYLHr8TN5d2sXuJXdWV9b3DxuluaxvokgAefbRVg9xja17ddhayPPuMvfR6lHA4Zz7OnNL9MmPTWuKFxstDcn1LAoOquOhlQ-QgyXTZFb4tFefvTYRdszLRgfP/s1600/test_draw2_image.png" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="200" data-original-width="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgVkezopuZF_soYLHr8TN5d2sXuJXdWV9b3DxuluaxvokgAefbRVg9xja17ddhayPPuMvfR6lHA4Zz7OnNL9MmPTWuKFxstDcn1LAoOquOhlQ-QgyXTZFb4tFefvTYRdszLRgfP/s1600/test_draw2_image.png" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Red ellipse drawn at (10, 11, 225, 95)</td></tr>
</tbody></table>
<br />
<br />
<br />
<h3 style="text-align: left;">
What is a git for? Jargon.</h3>
<a href="https://en.wikipedia.org/wiki/Jargon_File">jargon</a> - internet slang used by programmers. Rather than use a paragraph to explain something, people made up all sorts of strange words and phrases.<br />
<a href="https://en.wikipedia.org/wiki/Git">git</a> - for sharing versions of source code. It lets people work together, and provides tools for people to.<br />
<a href="https://en.wikipedia.org/wiki/Distributed_version_control#Pull_requests">pull request</a> (PR) - "Dear everyone, I request that you <b>git pull</b> my commits.". A pull request is a conversation starter. "Hey, I made a PR. Can you have a look?". When you "<b>git push</b>" your commits (upload your changes). <br />
<a href="https://en.wikipedia.org/wiki/Unit_testing">unit test</a> - <b>does this thing</b>(<i>unit</i>) <b>even work</b>(<i>test</i>)?!!? A program to test if another program works (how you think it should). Rather than test manually over and over again, a unit test can be written and then automatically test your code. A unit test is a nice example of how to use what you've made too. So when you do a pull request the people looking at it know what the code is supposed to do, and that the machine has already checked the code works for them.<br />
<a href="https://en.wikipedia.org/wiki/Assertion_(software_development)">assert</a> - "assert 1 == 1". An assert is saying something is true. "I assert that one equals one!". You can also assert variables.<br />
<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">This is a draft remember? So what is there left to finish in this doc?</span><br />
<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">Any feedback? Leave an internet comment. Or send me an electronic mail to: rene@pygame.org</span><br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span>
<br />
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span>
<br />
<h2 style="text-align: left;">
<a href="https://pygame.itch.io/pygame-book">pygame book</a></h2>
This article will be part of a book called "pygame 4000". A book dedicated to the joy of making software for making. Teaching collaboration, low level programming in C, high level programming in Python, GPU graphics programming with a shader language, design, music, tools, quality, and shipping.<br />
<br />
It's a bit of a weird book. There's a little bit of swearing in it (consider yourself fucking warned), and all profits go towards pygame development (the library, the community, and the website).</div>
Unknownnoreply@blogger.com195tag:blogger.com,1999:blog-10678074.post-65794474796471670622019-10-02T17:49:00.003+01:002019-10-03T07:18:26.571+01:00Using PostgreSQL as a cache?<div dir="ltr" style="text-align: left;" trbidi="on">
In the article on his blog Peter asks "<a href="https://www.peterbe.com/plog/redis-vs-postgres-blob-of-json">How much faster is Redis at storing a blob of JSON compared to PostgreSQL?</a>". Answer: 14x slower. <br />
<br />
Seems about right. Usually Redis is about 4x faster for a simple query like that compared to using PostgreSQL as a cache in my experience. It's why so many people use Redis as a cache. But I'd suggest <a href="http://renesd.blogspot.com/2017/02/is-postgresql-good-enough.html">PostgreSQL is good enough</a> to act as a cache for many people.<br />
<br />
Django is pretty slow at fetching from PostgreSQL compared to other python options, so this could explain part of the 14x VS 4x difference.<br />
<br />
Note that Django should be adding an index because of ForeignKey. However it's possible it isn't being used, or the table may need to analyze stats again. Also note that Django does not support prepared statements built in, and when you don't use prepared statements you do not use the PostgreSQL query cache. Prepared statements can give a 50% speedup often, because they don't have to do the query parsing and do a new query plan each time (and the query cache can actually be used). Additionally, you need to turn the Django config CONN_MAX_AGE from the default of <b>0</b>, otherwise it does not even reuse PostgreSQL database connections.<br />
<br />
PG read optimization tips for readers who may be interested.<br />
<ol style="text-align: left;">
<li>Run VACUUM ANALYZE, or have auto vacuuming on.</li>
<li>Check the query plan with EXPLAIN to confirm it is using an index (.explain(verbose=True) on the query in Django). Paste your explain result into <a href="https://explain.depesz.com/">https://explain.depesz.com/</a> if you can not understand it. If there are RED (horribly wrong) results shown, then you need to tweak something.</li>
<li>Tuning PostgreSQL config. Default homebrew PostgreSQL on Mac isn't really configured for speed for example. Easiest is to use a tool like pgtune and plump in your workload numbers <a href="https://pgtune.leopard.in.ua/">https://pgtune.leopard.in.ua/</a> Make your changes and try the EXPLAIN again (or run your benchmarks). Make sure to VACUUM ANALYZE between changes for best effect. This is a very application, OS, and hardware dependent task. If you don't know much about this, first try pgtune.</li>
<li>Consider table layout, column statistics optimizations, and rewriting the table in the order of the index with CLUSTER. REINDEX may also be needed for tables used as caches where they are changed very often.</li>
<li>'Index only scan' can be used where all the data is in the index, so the row doesn't need to be looked at. Not applicable in this case probably, since indexing on the JSON data probably wouldn't be a good idea. If your cache hit ratio is very low(mostly you are doing network lookups) then it could still be a performance increase.</li>
<li>PREPARE statements. In Django there are external packages available for this.</li>
<li>If you are already using a postgresql connection pool like <span class="PRODUCTNAME">Pgpool-II</span>, then you can use a query cache there pretty easily. See "<a href="https://www.pgpool.net/docs/latest/en/html/runtime-in-memory-query-cache.html">in memory query cache</a>".</li>
</ol>
<br />
I wouldn't be surprised if these could take it down to 4x-10x with the Django ORM compared to Redis.<br />
<br />
Of course you should probably just cache the views at the CDN/web proxy level, or even at the Django view or template level. So you probably won't even hit the Django app most times. [Edit: and this is exactly <a href="https://www.peterbe.com/plog/update-to-speed-comparison-for-redis-vs-postgresql-storing-blobs-of-json">what Peter does as mentioned in an update</a>.]</div>
Unknownnoreply@blogger.com72tag:blogger.com,1999:blog-10678074.post-63668111216560430542019-01-04T10:49:00.002+00:002019-01-04T10:49:48.699+00:00Experiments with new low latency PyPy garbage collector in a thread.<div dir="ltr" style="text-align: left;" trbidi="on">
<blockquote class="tr_bq">
TLDR; less performance spikes, faster average performance, PyPy is a good Python for games and multimedia apps on desktop systems, running GC in a thread is a good idea for GIL releasing apps.</blockquote>
<br />
In March 2018 at the PyPy winter sprint I spent some days working on improving pygame support on PyPy. Part of this was <a href="http://renesd.blogspot.com/2018/03/investigating-pypy-frame-drop.html">investigating pypy frame drop</a>. In there I found PyPy had both lower average performance in this pygame-using-CPython-API benchmark, and had spikes which caused frame drops. Remember that in latency sensitive applications performance spikes are not allowed. Note that not very much of this code is pure python (where PyPy is fastest), so it's almost a worst case benchmark for PyPy performance.<br />
<br />
Thanks to Antonio, the PyPy team and their supporters <span class="ILfuVd">—</span> things have much improved since then. One improvement in particular is that PyPy now allows more control of when the Garbage Collection does its work. Read more about it in the blog post "<a href="https://morepypy.blogspot.com/2019/01/pypy-for-low-latency-systems.html">PyPy for low latency systems</a>".<br />
<br />
In this post I'm going to compare my experiments with pygame with the experiments done last year. Additionally I'm going to try a new thing to improve performance which wasn't possible last year.<br />
<br />
<h3 style="text-align: left;">
Running garbage collection in a thread.</h3>
Now that we can call garbage collection when we want, how about we do it when we release the GIL and do some other processing in C? PyPy doesn't run the garbage collector in a thread by default, but we can do that ourselves.<br />
<br />
Remember that python has a GIL that can be released when you want to do IO or CPU intensive work. Here is an example program which has a thread waiting to be told when to run the incremental garbage collection step.<br />
<br />
<pre># Run pypy GC in background.</pre>
<pre>from Queue import Queue
from threading import Thread
import pygame
def gc_collect_step(q):
""" this calls the gc in a thread.
"""
while True:
print '5) GC THREAD: Waiting to do gc.'
ready_to_go = q.get()
print '4) GC THREAD: calling gc_collect_step'
if hasattr(gc, "collect_step"):
gc.collect_step()
q.task_done()
gc_queue = Queue()
worker = Thread(target=gc_collect_step, args=(gc_queue,))
worker.setDaemon(True)
worker.start()
pygame.display.set_mode((640, 480))
clock = pygame.time.Clock()
while True:
print('1) at top of loop')
pygame.event.get()
gc_queue.put(True)
print('2) tell GC thread to collect')
print('3) flipping display. GIL released')
pygame.display.flip()
print ('6) finished flipping. GIL grabbed')
clock.tick()
print(clock.get_fps())
</pre>
<br />
<br />
This is how the program executes. Notice how it does display.flip and waits on the OS to do VSYNC at some refresh rate (usually 60FPS). In this spot many games and multimedia apps have some spare time to do other work.<br />
<br />
<blockquote class="tr_bq">
<span style="font-size: large;">Waiting for VSYNC is the perfect spot to do GC!</span></blockquote>
<br />
Trace of program execution when doing GC in background thread.<br />
<blockquote class="tr_bq">
1) at top of loop<br />
2) tell GC thread to collect<br />
3) flipping display. GIL released<br />
4) <b>GC THREAD:</b> calling gc_collect_step<br />
5) <b>GC THREAD:</b> Waiting to do gc.<br />
6) finished flipping. GIL grabbed<br />
1) at top of loop<br />
2) tell GC thread to collect<br />
3) flipping display. GIL released<br />
4) <b>GC THREAD:</b> calling gc_collect_step<br />
5) <b>GC THREAD:</b> Waiting to do gc.<br />
6) finished flipping. GIL grabbed<br />
1) at top of loop<br />
2) tell GC thread to collect<br />
3) flipping display. GIL released<br />
4) <b>GC THREAD:</b> calling gc_collect_step<br />
5) <b>GC THREAD:</b> Waiting to do gc.<br />
6) finished flipping. GIL grabbed</blockquote>
On my bench-marking setup this gives an increase of <b>11% frames per second on average</b>. From 42 FPS to 47 FPS on average. Note, for this benchmark python2.7 gets 51 FPS on average.<br />
<br />
Running garbage collection in a thread should be able to improve the performance on pypy of many apps which wait on the GIL.<br />
<br />
<br />
<h3 style="text-align: left;">
Our performance goal.</h3>
<br />
Our performance goal is very little variance between frames and no
spikes above 0.013. 0.013 is about the amount of time we have to do our
work before we ask the frame to be flipped at 60 times per second.<br />
<br />
<h3 style="text-align: left;">
Python 2.7 baseline measurement.</h3>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhzl0LsIbm-4GS-EkteuQNy6mNod0mK5UTHWGWFkZXKsWyyyvqCYUz5xPjTr7k3eUmM9FiupbdQ0gRDN8KMkKLWJF2QwJHnL3AtXFLuAr2OmbCcFAeXVLwIgC5FtBhILzepURHZ/s1600/Screen+Shot+2019-01-04+at+08.46.39.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="820" data-original-width="1170" height="224" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhzl0LsIbm-4GS-EkteuQNy6mNod0mK5UTHWGWFkZXKsWyyyvqCYUz5xPjTr7k3eUmM9FiupbdQ0gRDN8KMkKLWJF2QwJHnL3AtXFLuAr2OmbCcFAeXVLwIgC5FtBhILzepURHZ/s320/Screen+Shot+2019-01-04+at+08.46.39.png" width="320" /></a></div>
<br />
So we have something to compare against, here is a measurement of Python 2.7.<br />
<br />
Note various spikes in performance, 51 FPS on average with many frames falling in the 0.005 to 0.007 range (below the required 0.013). The the funny spike at the start. I'm not sure why this is happening now compared to when I measured last year. The underlying OS has changed, or perhaps some other software is running on my computer.<br />
<br />
<b>Remember:</b> Our performance goal is very little variance between frames and no spikes above 0.013. 0.013 is about the amount of time we have to do our work before we ask the frame to be flipped.<br />
<br />
Python 2.7 isn't that bad here. It doesn't spike over 0.013 but it does have quite a lot of variance in some frames.<br />
<br />
<br />
<h3 style="text-align: left;">
PyPy 6.0.0 (last release, does not include new GC stuff).</h3>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj_aqveXGnc3AMWo8U9jKyda033cisYaZ1bLZZbO5vvqpfkM0sHFt08UqjAv5pWrtoN1YQOG3yVifbP073qcEIoIrvFpyq3O_rpoHyjbH2-KP94tO6cAMK1SJEdWTX8WTyHzfby/s1600/Screen+Shot+2019-01-04+at+08.56.50.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="826" data-original-width="1172" height="225" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj_aqveXGnc3AMWo8U9jKyda033cisYaZ1bLZZbO5vvqpfkM0sHFt08UqjAv5pWrtoN1YQOG3yVifbP073qcEIoIrvFpyq3O_rpoHyjbH2-KP94tO6cAMK1SJEdWTX8WTyHzfby/s320/Screen+Shot+2019-01-04+at+08.56.50.png" width="320" /></a></div>
What
about PyPy 6.0.0 which came out some months after my experiments last
year. Here we see 0.005 to 0.010 with some spikes of around 0.0125 and
higher. This older version of PyPy fails the performance goal.<br />
<h3 style="text-align: left;">
PyPy 6.0.0 gc.disable(). </h3>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhTa5EpMaJBN5nBpudhophezQOskU5q5l6a9LLP7nGS6AXJpVdEBbiGd0uvFv8RuP3UNBYjg3wxgpnA7a8Z36OTjkKbeg8qhhoDyvigUFtKMaTtOy086j2RFxnP1fOm55q1V34T/s1600/Screen+Shot+2019-01-04+at+08.57.51.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="826" data-original-width="1174" height="225" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhTa5EpMaJBN5nBpudhophezQOskU5q5l6a9LLP7nGS6AXJpVdEBbiGd0uvFv8RuP3UNBYjg3wxgpnA7a8Z36OTjkKbeg8qhhoDyvigUFtKMaTtOy086j2RFxnP1fOm55q1V34T/s320/Screen+Shot+2019-01-04+at+08.57.51.png" width="320" /></a></div>
What
if we just disable the garbage collector entirely? (Apart from memory
growing until the program starts swapping or crashes). After a few
seconds it just starts to slow down a lot.<br />
<br />
<br />
<h3 style="text-align: left;">
Time per frame on PyPy nightly (6.1.0 alpha0). gc.disable(), gc.collect_step().</h3>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjvCnlGLA2DDMSchPgkaJc-5lE8dfA8T4IqQzI7F0eZMM0lG4VfNXQxIwMO_N80j7ad5WWGkY4QbIYtfTazCRmcRiy13jAZhM5n9xnoRQCLC5FtXLuvv0xKes2gxjBdTxdiCoWm/s1600/Screen+Shot+2019-01-04+at+08.48.02.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="900" data-original-width="1168" height="246" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjvCnlGLA2DDMSchPgkaJc-5lE8dfA8T4IqQzI7F0eZMM0lG4VfNXQxIwMO_N80j7ad5WWGkY4QbIYtfTazCRmcRiy13jAZhM5n9xnoRQCLC5FtXLuvv0xKes2gxjBdTxdiCoWm/s320/Screen+Shot+2019-01-04+at+08.48.02.png" width="320" /></a></div>
This is using gc.disable() at the top of the main loop, and calling the new <b>gc.collect_step()</b> just before display.flip when most of the work has been done already. Remember that display.flip releases the GIL and waits for VSYNC at 60FPS.<br />
<br />
Here we see the average FPS at around 42 with times mostly between 0.006 and 0.008. At the beginning and end there are some spikes. But mostly things stay below the magical 0.015 seconds required to get below 60 FPS. Here is another measurement.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh2aBJ6Y8IeiWQxtjqY52tMpYD9N-o_AYuxjB7xoyyRgFTd0dtgdYhF2_Krng9DGdu8ZpWtBJK1lJOUF-GfM7Z1axRi86uDtj_C8CvUCV4IK8h7yJpZsmh8Ujs7QZwF7mIzApck/s1600/Screen+Shot+2019-01-04+at+08.48.51.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="904" data-original-width="1166" height="248" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh2aBJ6Y8IeiWQxtjqY52tMpYD9N-o_AYuxjB7xoyyRgFTd0dtgdYhF2_Krng9DGdu8ZpWtBJK1lJOUF-GfM7Z1axRi86uDtj_C8CvUCV4IK8h7yJpZsmh8Ujs7QZwF7mIzApck/s320/Screen+Shot+2019-01-04+at+08.48.51.png" width="320" /></a></div>
With this second measurement we see that something is going on at the start and at the end again. Again, mostly there are few spikes above 0.015 and things mostly stay between 0.006 and 0.008. This capture does show one spike, but I found this hard to reproduce.<br />
<br />
This is a great improvement over CPython in all except average FPS. The timings in the graphs are the work that is done before display.flip is called.<br />
<br />
<h3 style="text-align: left;">
PyPy nightly (6.1.0 alpha) GC in a thread, gc.disable().</h3>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjLq7AAAnqywCqpcQ0BN1OeHiDyJynJDaRP42KGEavS8Z5yftp3EZVLH8VF5F12Mdul1JopYLMTxKkaKQIOGovHaHs7LunmXg0hDH-OxmHBoHEkbDNFuZnFdeF24r5tdSTZHGmm/s1600/Screen+Shot+2019-01-04+at+10.25.50.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="880" data-original-width="1170" height="240" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjLq7AAAnqywCqpcQ0BN1OeHiDyJynJDaRP42KGEavS8Z5yftp3EZVLH8VF5F12Mdul1JopYLMTxKkaKQIOGovHaHs7LunmXg0hDH-OxmHBoHEkbDNFuZnFdeF24r5tdSTZHGmm/s320/Screen+Shot+2019-01-04+at+10.25.50.png" width="320" /></a></div>
<br />
Finally we have the GC running in a thread. This gives us 47.5 FPS on average. Most times stay between 0.6 and 0.8, however now there is a large number of spikes. None of them go above 0.015, but quite a few go above our goal of 0.013.<br />
<br />
Why are these spikes here? I'm not sure. It needs to be investigated some more.<br />
I ran the benchmark a few more times, and it seemed to spike less.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi1gsPZ6Veaz9qQTaLkyp8ayuw9nHhDqcSoCC4LwNAkD9R_p956vxZMwgz2GbABe97t1GpvIrNbSxT6WIUYhdc5zkhfqDudL8HxNJQZ9z7lUigu97ARiDNyRLcWL3k8ICuifUjh/s1600/Screen+Shot+2019-01-04+at+10.37.08.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="860" data-original-width="1170" height="235" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi1gsPZ6Veaz9qQTaLkyp8ayuw9nHhDqcSoCC4LwNAkD9R_p956vxZMwgz2GbABe97t1GpvIrNbSxT6WIUYhdc5zkhfqDudL8HxNJQZ9z7lUigu97ARiDNyRLcWL3k8ICuifUjh/s320/Screen+Shot+2019-01-04+at+10.37.08.png" width="320" /></a></div>
<br />
<br />
<br />
So then I switched to using threading.Event instead of threading.Queue, and there seemed to be less spikes over multiple runs. Probably using a lower level synchronization primitive (like threading.Lock) could improve things more. Also using a lower level primitive at C level would probably work even better. Additionally I'm not using the pypy36 which has some (hopefully) improved threading.Event and threading.Queue implementations.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjH-bu7m9cTA02SqmNJmahzrY6nXiExXbTxRVft7iGvWmyQx45OpHzobzF77pTJd3nu7558XCs5l3-I3K__Quuz8YecUpQ7S4w8PC21DuK8u0dVgjfEeWoYo_u_sBvcH8vsXoZM/s1600/Screen+Shot+2019-01-04+at+10.47.37.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="872" data-original-width="1162" height="240" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjH-bu7m9cTA02SqmNJmahzrY6nXiExXbTxRVft7iGvWmyQx45OpHzobzF77pTJd3nu7558XCs5l3-I3K__Quuz8YecUpQ7S4w8PC21DuK8u0dVgjfEeWoYo_u_sBvcH8vsXoZM/s320/Screen+Shot+2019-01-04+at+10.47.37.png" width="320" /></a></div>
<br />
<br />
Here's the example using Event instead of Queue.<br />
<br />
<pre>from threading import Event, Thread, Lock
import pygame
def gc_collect_step(event):
""" this calls the gc in a thread.
"""
while True:
print '5) GC THREAD: Waiting to do gc.'
event.wait()
print '4) GC THREAD: calling gc_collect_step'
if hasattr(gc, "collect_step"):
gc.collect_step()
event.clear()
gc_event = Event()
worker = Thread(target=gc_collect_step, args=(gc_event,))
worker.setDaemon(True)
worker.start()
pygame.display.set_mode((640, 480))
clock = pygame.time.Clock()
while True:
print('1) at top of loop')
pygame.event.get()
gc_event.set()
print('2) tell GC thread to collect')
print('3) flipping display. GIL released')
pygame.display.flip()
print ('6) finished flipping. GIL grabbed')
clock.tick()
print(clock.get_fps())
</pre>
<br />
<br />
Also, you can find the <a href="http://pypy-bench-testsprite.py/">pypy-bench-testsprite.py</a> in a gist. It needs to be run within the pygame examples folder. <br />
<br />
<h3 style="text-align: left;">
PyPy meets performance goals for games </h3>
In conclusion:<br />
<blockquote class="tr_bq">
<b><i>/Happy dance/</i></b></blockquote>
The results that Antonio posted in "<a href="https://morepypy.blogspot.com/2019/01/pypy-for-low-latency-systems.html">PyPy for low latency systems</a>" also appear in the experiments I made. Worst case performance can be increased, and the performance spikes can be removed.<br />
<br />
PyPy meets the performance goals for games now if you disable the gc at the top of the main loop, and call the gc collection every frame. The spikes that were there before are gone. The variance in performance is actually less than CPython.<br />
<br />
Calling GC in a thread is a viable option which increases the average FPS very close to CPython levels. However some more work needs to be done to try and find a way that avoids the increased performance jitter. Note, that this jitter is still at acceptable levels.<br />
<br />
Running the GC in a thread can probably be used for a peak performance increase by many apps which release the GIL for IO or CPU intensive operations.<br />
<br />
I'd like to remind people that this benchmark is a worst case for PyPy and games. In that there is also CPython API used rather than CFFI(fasted ffi on PyPy), and the benchmark contained not much pure python code. If you were also doing physics, collision detection and other stuff in pure python PyPy would pull ahead. Also this was using pygame with SDL1 so that I could sort-of compare performance to the previous time. Pygame 2 with SDL2 runs this benchmark much faster because of hardware acceleration. Another reminder that PyPy 6.1.0 isn't released yet, so the results will differ compared to the nightly version.<br />
<br />
The combination of PyPy's JIT with the Cython ahead of time compiler makes a formidable combination. I'm looking forward to using them all with the hardware acceleration that pygame 2 provides (Vulkan, Metal, OpenGL, Direct3d, ...).</div>
Unknownnoreply@blogger.com40tag:blogger.com,1999:blog-10678074.post-91229945578720862902019-01-02T12:05:00.000+00:002019-01-02T16:52:38.845+00:00Year of the Desktop #yearofthedesktop<div dir="ltr" style="text-align: left;" trbidi="on">
<div class="mentions-texteditor__content">
<b class="mentions-texteditor__hashtag" data-id="urn:li:hashtag:yearofthedesktop" dir="ltr">#yearofthedesktop</b> is a long running joke in FLOSS communities. When will Linux be more popular than Windows on the Desktop?</div>
<div class="mentions-texteditor__content">
<br /></div>
<div class="mentions-texteditor__content">
(please hold back your laughter until the end).</div>
<div class="mentions-texteditor__content">
<br /></div>
<div class="mentions-texteditor__content">
Let me explain why this is the Year of the Desktop, and why community developed software will win (and in fact has already).</div>
<div class="mentions-texteditor__content">
<br /></div>
<div class="mentions-texteditor__content">
TLDR; most PC hardware vendors are shipping Linux machines, Linux had the best cheap laptop of the year, Linux is a good platform for games, Steam now supports 1000s of Windows apps(+ 5000 linux native) and is contributing to Wine, the web is a good high quality desktop software target, government support for Linux desktop is increasing, and even Microsoft is supporting running Linux software on Windows, cheap hardware like Raspberry Pi(and more) is available and usable, community software cares about usability and has a supporting financial model, community development is a kinder place, fragmentation of the Desktop is no more, Gnome receives $1,000,000 donation, governments spend money on community desktop software, IBM the original PC company bought the first Linux Desktop company, less fragmentation and distributing Desktop apps to all Linux users is now very easy, punch line at end.</div>
<div class="mentions-texteditor__content">
<br /></div>
<h3 class="mentions-texteditor__content" style="text-align: left;">
Financial and project based development</h3>
<div class="mentions-texteditor__content">
Two major community graphics applications have
achieved something great in the last years. Two pieces of software widely know to be effective but not the most user friendly. </div>
<ul style="text-align: left;">
<li>The Gnu Image Manipulation Program (GIMP) 2.10 series (released this year).</li>
<li><a href="https://www.blender.org/2-8/">Blender 3d</a> 2.8 series (close to release)</li>
</ul>
<div class="mentions-texteditor__content">
First they have implemented <b>effective financial
models</b> to support development. Secondly they both have <b>User eXperience
focused</b> releases where their major features based on using their
software in production. So not only is it, 'scratch my own itch'
development driving what gets fixed. Ease of use and matters
important to production users are getting fixed too. </div>
<div class="mentions-texteditor__content">
<br /></div>
<div class="mentions-texteditor__content">
Both run projects
using their software which aim to link their developers up with real projects. These projects are films using 3d, and image processing. Both are
also accepting money directly from users. Incentives are aligned.</div>
<div class="mentions-texteditor__content">
<br /></div>
<div class="mentions-texteditor__content">
It's not just these two projects either. Many community developers are starting to be supported directly by the community, and many projects are getting professional management in order to raise funds. <a href="https://numfocus.org/">Numfocus</a> is one example of an organisation that is helping various scientific communities with fund raising and marketing. This is probably one of the reasons why notebooks and python are so popular in research now in places where Excel used to be popular.</div>
<br />
<br />
If you look at the <a href="https://twitter.com/GIMP_Official/status/1072123374434963457">GIMP image editors twitter news</a> feed you will notice they talk a lot about user experience improvements rather than technical items. <br />
<blockquote class="tr_bq">
<div class="js-tweet-text-container" style="-webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; background-color: white; caret-color: rgb(20, 23, 26); color: #14171a; font-family: "Helvetica Neue", Helvetica, Arial, sans-serif; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-decoration: none; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px;">
<div class="TweetTextSize TweetTextSize--jumbo js-tweet-text tweet-text" data-aria-label-part="0" lang="en" style="font-size: 27px; letter-spacing: 0.01em; line-height: 32px; margin: 0px; white-space: pre-wrap; word-wrap: break-word;">
"New in development: you can now select the default file format for exporting newly created images. No more need to manually change PNG to JPEG etc. every time you do the first export in a new GIMP session. Patch by Richard McLean and <a class="twitter-atreply pretty-link js-nav" data-mentioned-user-id="16255320" dir="ltr" href="https://twitter.com/mitchfoo" style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; color: #1da1f2; text-decoration: none;"><s style="color: #1da1f2; text-decoration: none;">@</s><b style="color: #1da1f2; font-weight: normal;">mitchfoo</b></a>. Will be available in GIMP 2.10.10."</div>
</div>
</blockquote>
The <a href="https://www.blender.org/2-8/">blender 3d 2.8 release video</a> is the same. It's mostly about user experience improvements.<br />
<br />
Also see the <a href="https://www.darktable.org/2018/12/darktable-26/">Darktable release notes</a>. It's mostly about usability improvements important to photographers.<br />
<br />
<h3 class="mentions-texteditor__content" style="text-align: left;">
Better behaviour </h3>
<div class="mentions-texteditor__content">
Now we get
onto the next point... Toxic people online. <b>Codes of Behavior</b> have taken
hold, and collaboration tools have improved. It's a much nicer
atmosphere out there. Guess what happens when you make it a nicer place
to hang out for 51% of the population? And it's not just the 51%, <b>jerks
drive away everyone</b>. Creativity in a toxic environment is destroyed because talking about ideas needs a safe space. And remember the UX improvements to the
software itself? This is also happening to development materials too. The
"good first issue" movement isn't just in the rails and python
communities anymore... it's world wide. Making the development experience easier for newbies is a requirement of good projects. In short, <b>we have "leveled up"
in community collaboration</b>. As a community people are learning to be kinder to each other online (it could be better still!).</div>
<div class="mentions-texteditor__content">
<br /></div>
<div class="mentions-texteditor__content">
It used to be that we needed a tough-as-shit-authoritarian-type person who could withstand the abuse for community software to be successful. 2018 saw two major projects (python, and linux) grow in that area. Because toxic behaviour is not acceptable (even from leadership) these types of people aren't as necessary. Community software needed people who could withstand daily toxic abuse in order to merely function. Other methods for decision making have been proven to work (thanks again Debian!), and are now being used.</div>
<div class="mentions-texteditor__content">
<br /></div>
<div class="mentions-texteditor__content">
Better behaviour and more inclusion in 2019 with community developed software will result in more usable software.</div>
<h3 class="mentions-texteditor__content" style="text-align: left;">
In more human languages.</h3>
<div class="mentions-texteditor__content">
Many projects are now getting better support for other human languages other than English. The tooling, but also the will exists now to support more human languages in the development of software.</div>
<div class="mentions-texteditor__content">
<br /></div>
<div class="mentions-texteditor__content">
python for example supports unicode in code, and also has documentation available in other languages other than English now. Sure, gcc still doesn't support unicode literals (clang does).</div>
<div class="mentions-texteditor__content">
<br /></div>
<div class="mentions-texteditor__content">
Whilst English is widespread and allows people to collaborate with a common human language, lots of other human languages are used in the world. And lots of the best software developers in the world don't even speak English. With more contributors from different backgrounds working on issues that affect them, it means the software will continue to improve for those people.</div>
<h3 class="mentions-texteditor__content" style="text-align: left;">
Better tooling.</h3>
<div class="mentions-texteditor__content">
It's not just the social aspects that have improved. Tooling has also gotten much better. Community projects now have access to free high quality continuous integration. Sharing our software has gotten much better. Now using, and distributing packages is not as bad as it used to be (yes, is still not perfect). Reusing and sharing code more easily is a win for everyone.</div>
<div class="mentions-texteditor__content">
<br /></div>
<div class="mentions-texteditor__content">
Other tooling is gitlab, which both Gnome and the freedesktop projects switched to away from adhoc software tooling and development processes that included things like bugzilla.<br />
<br />
The better collaboration has resulted in two things happening within the Gnome desktop. The first is that LibreOffice is planned to be used as a component of the Document preview app. Meaning there is very good document preview support. Another area of collaboration is the Gimp component CEGL is going to be used in the Photo app for better photo editing for non-professional photo editing.</div>
<div class="mentions-texteditor__content">
<br /></div>
<div class="mentions-texteditor__content">
App distribution on Linux with <a href="https://flatpak.org/">flatpak</a> is now really easy since the 1.0 release. With a pull request containing a small config file you can submit your app which will then be available to 10s of millions of people. The app will be able to update itself, and will securely run within a sandbox, and be usable across multiple Linux distributions. Yes, <b>you can release an app to the entire Linux Desktop market</b> with very little trouble. This means Linux distribution fragmentation is less of a big deal for app developers. Already popular apps like Slack, Spotify, Visual Studio Code, and Steam use flatpak.</div>
<div class="mentions-texteditor__content">
<br /></div>
<h3 class="mentions-texteditor__content" style="text-align: left;">
$1,000,000 reasons, and less fragmentation.</h3>
<div class="mentions-texteditor__content">
This year the <a href="https://www.gnome.org/news/2018/05/anonymous-donor-pledges-1m-donation-over-two-years/">Gnome foundation</a> who is responsible for the main linux desktop received a million dollar donation. </div>
<div class="mentions-texteditor__content">
<br /></div>
<div class="mentions-texteditor__content">
Now the Gnome desktop is used on the most popular community distributions by default. Ubuntu moved to supporting Gnome, rather than developing their own stuff in 2018. This has already meant more resources have gone into one Desktop rather than having resources spread over multiple environments. Now there is less fragmentation on the desktop. </div>
<div class="mentions-texteditor__content">
</div>
<div class="mentions-texteditor__content">
Along with this some hardware companies have joined with the Gnome foundation as well. This again aligns community software with production uses. Hopefully government and educational users will also join in with the fun - so their voices are heard more too.</div>
<div class="mentions-texteditor__content">
<br /></div>
<div class="mentions-texteditor__content">
Other hardware companies using Linux include of course Android - the most popular personal computing environment in the world. This means most personal computing hardware companies are supporting Linux (see below about how all major PC vendors are now shipping Linux laptops in great volume).</div>
<div class="mentions-texteditor__content">
<br /></div>
<div class="mentions-texteditor__content">
Another component that now has less fragmentation in is the startup system which runs on boot and in the background - systemd. All major Linux distros now use systemd. And systemd has better support for desktop needs.</div>
<div class="mentions-texteditor__content">
<br /></div>
<h3 class="mentions-texteditor__content" style="text-align: left;">
Government and Commercial support.</h3>
<div class="mentions-texteditor__content">
The government is starting to support desktop software more. Including a bug bounty program run by the EU for some desktop software like the VLC video player, and Notepad++.</div>
<div class="mentions-texteditor__content">
<br /></div>
<div class="mentions-texteditor__content">
Millions of government PCs are also using things like LibreOffice and have some financial support from government organisations.</div>
<div class="mentions-texteditor__content">
<br /></div>
<div class="mentions-texteditor__content">
The original PC company - IBM bought Red Hat. This is the first Desktop Linux company, and one of the most successful Linux companies. IBM will continue to sell Linux into the enterprise.</div>
<div class="mentions-texteditor__content">
<br /></div>
<h3 class="mentions-texteditor__content" style="text-align: left;">
Apple sucking, games, and graphics.</h3>
<div class="mentions-texteditor__content">
It's not just because macbook pros don't have
an Escape key anymore. Apple used to have an open source friendly
development environment. They used to ship a current gcc, and other open source tools (what would we do with the community
developed Homebrew system on mac?). Apple used to support open graphics standards. It used to support the needs of CG professionals.</div>
<div class="mentions-texteditor__content">
<br /></div>
<div class="mentions-texteditor__content">
However, the most recent release of MacOS has pissed off graphics and audio programmers so much <a href="https://twitter.com/tonroosendaal/status/1075693022627708929">they are leaving the platform</a>. Apple used to be the best platform for video, audio and graphics developers. But in recent years it's support for this type of developer has been getting worse and worse.</div>
<div class="mentions-texteditor__content">
<br /></div>
<div class="mentions-texteditor__content">
The latest release of MacOS broke OpenGL and SDL apps. They announced on the release that OpenGL would be deprecated... but really they just broke it more. People are reporting bugs, making hacks, and crossing their fingers... but they broke lots of stuff. They made software just bad enough that people are considering changing platforms.</div>
<br />
<div class="mentions-texteditor__content">
Apple simply doesn't care as much about Desktop anymore. They put most of their resources into their mobile platform. Desktop application development and distribution on Linux is now better than what Apple provides.</div>
<div class="mentions-texteditor__content">
<br /></div>
<div class="mentions-texteditor__content">
But if you keep breaking everyone else's software except your own, more and more applications will stop working on new releases. Developers will stop caring, especially since Mac sales are becoming less and less.</div>
<div class="mentions-texteditor__content">
<br /></div>
<div class="mentions-texteditor__content">
Even professional graphics solution providers like <a href="https://www.blackmagicdesign.com/">Blackmagic design</a> (a long time Apple supporter) provide first class Linux support. So TV stations, and movie productions can(and do!) run their operations on Linux. Apple doesn't even provide good pro hardware anymore. The macpro had a last update in 2013, and only recently did the iMac pro turn up.</div>
<div class="mentions-texteditor__content">
<br /></div>
<h3 class="mentions-texteditor__content" style="text-align: left;">
Games and graphics on linux.</h3>
<div class="mentions-texteditor__content">
Video, graphics, audio, and games developers are exactly the type of developers responsible for great desktop software. It used to be a sea of mac laptops at graphics and game developer conferences... but now they are now a minority.</div>
<div class="mentions-texteditor__content">
<br /></div>
<div class="mentions-texteditor__content">
Vulkan graphics is the next graphics standard from OpenGL. And Linux is probably the best platform for Vulkan graphics. </div>
<div class="mentions-texteditor__content">
<br /></div>
<div class="mentions-texteditor__content">
What platform has the most and best desktop games at the moment? Most people would probably say Windows, and the web. But most games are probably played on mobile and game platforms like Android. Now you can run Android, Web, and many Windows games on the Linux desktop. Many games written for the mac platform don't even work on the latest release of MacOS.</div>
<div class="mentions-texteditor__content">
<br /></div>
<div class="mentions-texteditor__content">
Steam(a major market place for games) just brought out better support for <a href="https://steamcommunity.com/games/221410/announcements/detail/1696055855739350561">running windows games on the Linux desktop</a>. Many games run good enough that they are happy to support their customers who run their windows games on Linux. Valve (makers of Steam) are also supporting development of Linux software including high quality not-emulation software for windows applications. However, <a href="https://store.steampowered.com/hwsurvey/Steam-Hardware-Software-Survey-Welcome-to-Steam">95% of steam users are on windows</a>. Along with the 5000 titles that support linux natively, more than 2600 windows games (as of Oct 2018) also work on Linux with Steam with no effort. They can be played with a single click from inside the steam client.</div>
<div class="mentions-texteditor__content">
<br /></div>
<div class="mentions-texteditor__content">
With a major application distributor contributing to first class Windows application support, more and more windows software will just work on Linux.</div>
<div class="mentions-texteditor__content">
<br /></div>
<div class="mentions-texteditor__content">
Now that Linux can run many games from Android and Windows -- <b>Linux is a good platform for games</b>.</div>
<div class="mentions-texteditor__content">
<br /></div>
<h3 class="mentions-texteditor__content" style="text-align: left;">
The web is ready for desktop class software</h3>
<div class="mentions-texteditor__content">
The web now has pretty good support for WebASM allowing desktop class software through the web. </div>
<div class="mentions-texteditor__content">
This means that for most application domains there is a web version available. Even application domains that require high performance or require millions of lines of C++ to work can now run on the web. </div>
<div class="mentions-texteditor__content">
<br /></div>
<div class="mentions-texteditor__content">
And since there is a web version available for most application areas, it makes the underlying desktop less important.</div>
<div class="mentions-texteditor__content">
<br /></div>
<div class="mentions-texteditor__content">
For example, many designers are now using Figma instead of Sketch for UI design... and Figma runs on the web. The most popular Code editor Microsoft distributes is now using web technology (VScode).</div>
<div class="mentions-texteditor__content">
<br /></div>
<h3 class="mentions-texteditor__content" style="text-align: left;">
Cheap hardware is ready</h3>
<div class="mentions-texteditor__content">
As well as Android hardware, other companies have continued and sprung up. This includes Raspberry Pi the group who makes millions of cheap computers meant for education.</div>
<div class="mentions-texteditor__content">
<br /></div>
<div class="mentions-texteditor__content">
2018 saw two other chip sets used in lots of TV devices that are more than powerful enough to be used on the desktop. These devices can be bought for less than $40. And they all support Linux. And not just with some patches on some weird ftp site or obscure github repo... they're supported on upstream Linux and available to be used in distros like Armbian and even Debian.</div>
<div class="mentions-texteditor__content">
<br /></div>
<div class="mentions-texteditor__content">
These devices have 4-8 cores, and can run 4K screens at 60fps. They also have great OpenGL drivers thanks to Android, and WebGL requiring that. They also have things like robust eMMC storage (SD cards like used in the Raspberry PI were never designed to be robust... they were meant to store photos temporarily).</div>
<div class="mentions-texteditor__content">
<br /></div>
<div class="mentions-texteditor__content">
TV hardware needs to be very usable and reliable. It needs to just work. The graphics and the sound needs to just work. It needs to work in high definition well.</div>
<div class="mentions-texteditor__content">
<br /></div>
<div class="mentions-texteditor__content">
Having a loud voice for newbies, and the requirements of education in the community Desktop world has really helped usability for everyone (thanks Raspberry Pi!). Having TV hardware companies contributing to usability has been a great improvement last year.</div>
<div class="mentions-texteditor__content">
<br /></div>
<div class="mentions-texteditor__content">
Cheap phone and TV hardware is going to be even more powerful this year. Already we have ARM devices that are fast enough to be used as development machines.</div>
<div class="mentions-texteditor__content">
<br /></div>
<div class="mentions-texteditor__content">
The other popular hardware are Chromebooks. Laptops that run Googles ChromeOS, that also support Android apps. Because of this all the main PC hardware developers are shipping ChromeOS compatible hardware that runs Linux. Brands like Acer, HP, Dell, Asus, and Lenovo are all shipping Linux based laptops.</div>
<div class="mentions-texteditor__content">
<br /></div>
<div class="mentions-texteditor__content">
How popular are these Linux based laptops? Half of PCmags best cheap laptops of 2018 were running ChromeOS rather than Windows. The best cheap laptop of 2018 was a ChromeOS based laptop from Asus. Often a ChromeOS based laptop is the best selling laptop on Amazon.</div>
<div class="mentions-texteditor__content">
<br /></div>
<div class="mentions-texteditor__content">
<b>Most major hardware vendors are shipping Linux desktop devices</b> in large volumes that run Linux.</div>
<div class="mentions-texteditor__content">
<br /></div>
<h3 class="mentions-texteditor__content" style="text-align: left;">
Linux software on Chrome OS.</h3>
<div class="mentions-texteditor__content">
ChromeOS is one of the most popular Linux based operating systems used on desktops today.</div>
<div class="mentions-texteditor__content">
<br /></div>
<div class="mentions-texteditor__content">
Now it supports running Debian Linux software on many of those Chromebooks. This support will continue to improve in 2019.</div>
<div class="mentions-texteditor__content">
<br /></div>
<h3 class="mentions-texteditor__content" style="text-align: left;">
Internet explorer is dead</h3>
<div class="mentions-texteditor__content">
Admitting that the free software community web browser components are better than their own, one big company is now <a href="https://blogs.windows.com/windowsexperience/2018/12/06/microsoft-edge-making-the-web-better-through-more-open-source-collaboration/#GmSJg4uFjBM5y8Hz.97">contributing to the most popular web browser components</a>. Microsoft is going to be using Chromium.</div>
<div class="mentions-texteditor__content">
<br /></div>
<div class="mentions-texteditor__content">
And with most desktop software running via the web, once MS start shipping their new browser that platform this means the software running most software will be developed by the community.</div>
<div class="mentions-texteditor__content">
<br /></div>
<div class="mentions-texteditor__content">
Microsoft has also shipped <a href="https://docs.microsoft.com/en-us/windows/wsl/install-win10">Windows subsystem for Linux</a> for the last couple of years. This is a Microsoft supported way to let you use Linux software within Windows.</div>
<div class="mentions-texteditor__content">
<br /></div>
<div class="mentions-texteditor__content">
If your OS supports using Linux software, and runs a web browser with community developed components is it not a duck?</div>
<div class="mentions-texteditor__content">
<br /></div>
<div class="mentions-texteditor__content">
2019 will see even Windows use community developed components as part of the most popular part of their Desktop OS.</div>
<div class="mentions-texteditor__content">
<br /></div>
<h3 class="mentions-texteditor__content" style="text-align: left;">
Most web browsing done on Linux based OSes.</h3>
<div class="mentions-texteditor__content">
Already most web browsing is done on community developed web browsers.</div>
<div class="mentions-texteditor__content">
<br /></div>
<div class="mentions-texteditor__content">
But what OS is used to do that web browsing on? According to <a href="http://gs.statcounter.com/os-market-share">Statcounter</a> Windows and Android have been very close all of 2018. Sometimes Android was the most popular, sometimes Windows was the most popular.</div>
<div class="mentions-texteditor__content">
<br /></div>
<div class="mentions-texteditor__content">
Until that is... right at the end Linux surged ahead (and Android dropped).</div>
<div class="mentions-texteditor__content">
<br /></div>
<div class="mentions-texteditor__content">
Windows 37.43%</div>
<div class="mentions-texteditor__content">
<b>Android 36.49%</b></div>
<div class="mentions-texteditor__content">
iOS 13.16%</div>
<div class="mentions-texteditor__content">
OS X 6.22%</div>
<div class="mentions-texteditor__content">
<b>Unknown 4.01%</b></div>
<div class="mentions-texteditor__content">
<b>Linux 0.85%</b></div>
<div class="mentions-texteditor__content">
<br /></div>
If you add Linux and Android together from these statistics, then Windows is not the most popular OS used for web browsing. Also note: "unknown". What is this mysterious unknown? This is the OS that has increased by the same amount that Android has decreased in the recent months. These are Linux based machines.<br />
<div class="mentions-texteditor__content">
<br /></div>
<div class="mentions-texteditor__content">
A world wide growth of 4% in just two months?<br />
</div>
<div class="mentions-texteditor__content">
<b>Community developed OSes in 2019 are now the leading</b> OSes used by people to do web browsing.</div>
<div class="mentions-texteditor__content">
<br />
<h3 style="text-align: left;">
Cheap TV boxes used as PCs.</h3>
I was in a discord chat when a programmer-musician mentioned he was getting a RK3328 based board to replace his laptop. Apparently quite a few people are doing it already he said. But he was going to run Armbian or Arch (I can't remember which). The RK3328 is made for smart TV boxes, but already people are using them as PCs.<br />
<br />
Maybe those $20-$60 Android
'TV' boxes from ebay/amazon/aliexpress are actually useful for desktop
tasks too? They are. Search for RK3328, S905X2, or S912 and you will find many
different models selling like crazy. These either 4 or 8 cores, 4k video machines that can
run office, can be used with a keyboard (some even come with one), have
USB ports (even USB 3!) and they cost from $20-$60. And they are supported by upstream Linux.<br />
<div class="mentions-texteditor__content">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<iframe width="320" height="266" class="YOUTUBE-iframe-video" data-thumbnail-src="https://i.ytimg.com/vi/X3FTEgH8Rs8/0.jpg" src="https://www.youtube.com/embed/X3FTEgH8Rs8?feature=player_embedded" frameborder="0" allowfullscreen></iframe></div>
<br /></div>
<h3 class="mentions-texteditor__content" style="text-align: left;">
The funny bit.</h3>
<div class="mentions-texteditor__content">
There is no punchline. Community software won (or did it?).</div>
<div class="mentions-texteditor__content">
<br /></div>
<div class="mentions-texteditor__content">
Now that all major desktop software supports Linux(including Windows), and Linux laptops are better (according to PCMag) and outselling windows ones on Amazon... does it matter? Probably not.</div>
<div class="mentions-texteditor__content">
<br /></div>
<div class="mentions-texteditor__content">
The nice part about this being Year of the Desktop is that we don't need to talk about it anymore. Or do we?</div>
<div class="mentions-texteditor__content">
<br /></div>
<div class="mentions-texteditor__content">
Happy hacking!</div>
</div>
Unknownnoreply@blogger.com43tag:blogger.com,1999:blog-10678074.post-71391481413253805042018-11-27T19:03:00.000+00:002018-11-27T19:03:09.541+00:00🐱🏍 — the first pygame 2 community game. Starting now! Are you in?<div dir="ltr" style="text-align: left;" trbidi="on">
What is a 🐱🏍? *[0].<br />
<br />
<br />
There was this email thread on the pygame mailing list.<br />
<div>
<div>
"<a href="https://www.mail-archive.com/pygame-users@seul.org/msg20134.html">About Pygame development</a>".</div>
<div>
<br /></div>
<div>
One topic of that conversation was doing a community game in there for reasons(see below).</div>
<div>
<br /></div>
<div>
I would like to do a <b>pygame 2 community game</b> to submit in the:</div>
<div>
- <a href="https://ldjam.com/" target="_blank">https://ldjam.com/</a></div>
<div>
- <a href="https://itch.io/jam/game-off-2018" target="_blank">https://itch.io/jam/game-off-<wbr></wbr>2018</a> (GameOff github jam)</div>
<div>
<br /></div>
<div>
<div>
Ludumdare(ldjam) starts in 3 days, 7 hours. Theme not selected yet.</div>
<div>
GameOff finishes in 4 days 2 hours. <b>Theme is "Hybrid</b>", Jam already started.</div>
<div>
<br /></div>
<div>
So, the Jams finish in 4.1 days.</div>
<div>
</div>
<div style="text-align: left;">
<h3>
Are you in? </h3>
</div>
<div>
If so join the <a href="https://discord.gg/r8yreB6" target="_blank">web based chatroom</a>(discord) in the "#communitygame" channel.</div>
<div>
Our repo: <a href="https://github.com/pygame/stuntcat">https://github.com/pygame/stuntcat</a></div>
</div>
<div>
<br /></div>
<div>
I'm trying to form a team now.</div>
<div>
<br /></div>
<div>
... read more?...</div>
<div>
<br /></div>
<div>
click....</div>
<div>
loading...</div>
<div>
loading...</div>
<div>
buffering...</div>
<div>
loading...</div>
<div>
refresh...</div>
<div>
loading...</div>
<div>
<br /></div>
<div>
<br /></div>
<div style="text-align: left;">
<h3>
So why do this?</h3>
</div>
My reasons for doing this are to push me to get pygame 2 features done that are actually useful in apps (<a href="https://geometrian.com/programming/tutorials/write-games-not-engines/">Write Games, Not Engines</a>).</div>
<div>
<div>
It will be a good testbed for prototyping as well.</div>
<div>
<br /></div>
</div>
<div>
I will also try to get the game into Steam, and any other app stores.</div>
<div>
<div>
Because distributing pygame 2 games is also important,</div>
<div>
and improving the tooling and documentation around that is also important.</div>
</div>
<div>
<br /></div>
<div>
The pygame 2 community game serves these purposes.</div>
<ul>
<li>guiding and motivating pygame 2 development (make games not engines)</li>
<li>raising funds towards pygame 2 developments (or losing money on steam signup costs)</li>
<li>making an open source pygame 2 game</li>
<li>improving the work flow for people releasing games/apps.</li>
<li>helping to guide improvements on the pygame website </li>
<li>trying out new technology and techniques</li>
</ul>
It's about 4-5 days, and then there will be some more days to try and distribute the game further.<br /><div>
<br /></div>
<div>
<li>License
for code will be the pygame license (LGPL, but you can keep your parts
of course!) </li>
<li>Art assets license will be some form of permissive creative
commons. So technically anyone should be able to distribute it following those licenses (and even sell it).</li>
</div>
<div>
<br /></div>
<div>
Contributors will also:</div>
<ul>
<li>be in the credits on the pygame website, game website, in game, promo pages</li>
<li>have their link (to patreon twitter github etc) in their too</li>
<li>hopefully enjoy themselves, and maybe learn something</li>
</ul>
<div>
Also interested in people who want this game to use their library.</div>
Especially if you will us use the library or improve it as part of the Jam.<br /><div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
Anyone who wants to be involved can join the discord channel <br /><div>
<a href="https://discord.gg/r8yreB6" target="_blank">web based chatroom</a>(discord) in the "#communitygame" channel.</div>
</div>
<div>
Please let us know if you want to be involved and how :)</div>
<div>
*[0] (We came up with a repo name... we were perhaps thinking something like "Speedy the stunt cat" or "stuntcat" Did you know there is this whole weird genre of stunt cat games, and that stunt cat emojis are a thing? 🐱🏍 Also, @claudeb's first cat was called Speedy, and was a stunt cat. So that's the repo name. )</div>
</div>
Unknownnoreply@blogger.com10tag:blogger.com,1999:blog-10678074.post-76478778407679221882018-11-27T09:43:00.000+00:002018-11-27T10:42:49.189+00:00pygame 2 update - the examples all work<div dir="ltr" style="text-align: left;" trbidi="on">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div>
<div>
<div style="text-align: center;">
<br />
<br />
<span style="font-size: large;">🎮👾👽🎮👾👽🎮👾👽🎮👾👽</span></div>
<div>
<div style="text-align: center;">
<br />
examples/<b>chimp.py works</b><br />
<br />
<h3>
<span style="font-size: large;">👽🎮👾👽🎮👾👽🎮👾👽🎮👾</span></h3>
</div>
<div>
</div>
<div>
</div>
<div>
<br />
<br />
That's a nice milestone for pygame 2 development.</div>
<div>
(and also, now these examples also work...</div>
<div>
python -m pygame.examples.aliens</div>
<div>
python -m pygame.examples.scroll</div>
<div>
)</div>
<div>
<br /></div>
<div>
I'm pretty sure all the examples are now working with SDL2/pygame 2.<br />
<br /></div>
<div>
</div>
</div>
<div>
<div style="text-align: left;">
<h2>
<span style="font-size: medium;"><span style="font-size: large;">New SDL2 functionality.</span></span></h2>
</div>
</div>
<div>
Also we have some new SDL2 functionality exposed</div>
<div>
(experimentally, the APIs are not done yet).</div>
<div>
Multitouch, and AudioDevice, along with an example of recording audio.</div>
<div>
@dlon has been fixing a number of bugs(implementing key repeat, fixing unicode issues)</div>
<div>
and also prototyped some Window/Renderer classes (using ctypes and python).</div>
<div>
<br />
<div>
The new events are done in a backwards compatible way, </div>
<div>
so that games using them with pygame 1/SDL1 will still work.</div>
<div>
You just won't get the events.</div>
<div>
<span style="font-family: monospace , monospace;"><br /></span></div>
<div>
<span style="font-family: monospace , monospace;">if event.type == pg.AUDIODEVICEADDED:</span></div>
<div>
<span style="font-family: monospace , monospace;"> print('a new sound card plugged in... let's offer the user to use it')</span></div>
<div>
<br /></div>
<div>
On pygame SDL1 AUDIODEVICEADDED is defined as -1, so you just never get these events.</div>
See <a href="https://github.com/pygame/pygame/blob/master/examples/audiocapture.py">examples/audiocapture.py</a><br />
<div>
<br /></div>
</div>
<div style="text-align: left;">
<h2>
<span style="font-size: large;">Making an instrument, Free and Open Source</span></h2>
</div>
<div>
<div>
<div>
With a friend we made a loop recorder instrument/video thingo with microphone input.</div>
It records the input and then lets you play back sounds as loops.</div>
<div>
Then you can record over tracks making music in a performance.</div>
<div>
<div>
The video synth reacts to beats and pitch changes driving video.</div>
<div>
(pitch/beat detection not included in pygame... yet?)<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjTpzNxI7h-sC2Z-QjUzZgPLHe7rm-yKa8hRMh-4uIVbOXNY6MERv7vf-ALaxYgr8LQEq_krbLJle45SPhiG8AJEM7N49SnyB3wONKJbYdasccPTnaLMk_H91nPehe72RcRLUEa/s1600/pygame-open-source-diversity.gif" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="198" data-original-width="360" height="176" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjTpzNxI7h-sC2Z-QjUzZgPLHe7rm-yKa8hRMh-4uIVbOXNY6MERv7vf-ALaxYgr8LQEq_krbLJle45SPhiG8AJEM7N49SnyB3wONKJbYdasccPTnaLMk_H91nPehe72RcRLUEa/s320/pygame-open-source-diversity.gif" width="320" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">(Demonstrating the app for a pygame presentation at an <a href="https://www.meetup.com/opensourcediversity/events/255369540/" target="_blank">Open Source diversity</a> event in Berlin)</td></tr>
</tbody></table>
<br />
Working on a project that uses pygame 2 has been good for me to push features forward.</div>
<div>
Now you can ask pygame.mixer to use things like 32bit floats (good for audio apps).</div>
<div>
As well as tell it to open a specific audio device for output (and input!).</div>
<div>
<br />
Writing your own mixer/synth is now much easier.</div>
<div>
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh9eoDzph6niCF6k9pSpufKqjmYx7paCF0uv83mGZmFow3aLqjVKs9vzLZRKRqtVQ1ZwdETzPR_jRoMdJtjYmW8InSpWPFYVZgT0sed7xqV-XMRbK2rS49sbx7-RvC5xfJAiOy8/s1600/meow-anim.gif" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="768" data-original-width="1024" height="240" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh9eoDzph6niCF6k9pSpufKqjmYx7paCF0uv83mGZmFow3aLqjVKs9vzLZRKRqtVQ1ZwdETzPR_jRoMdJtjYmW8InSpWPFYVZgT0sed7xqV-XMRbK2rS49sbx7-RvC5xfJAiOy8/s320/meow-anim.gif" width="320" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Music making and performance app for pygame 2, looping and mixing sounds recorded in the microphone.</td></tr>
</tbody></table>
<br />
I think I learnt a lot from that event about making projects more accessible to people, and made some contacts with people from other projects and communities.</div>
<div>
<div>
</div>
<div>
For contributors, I've written about how to debug pygame and other python extensions with gdb,</div>
<div>
and also how to get started with SDL2 in C (strangely this did not exist on the internet). These will be tested and then merged into the pygame contributors guide.</div>
</div>
<div>
<br /></div>
</div>
<div>
<div>
<h2 style="text-align: left;">
<span style="font-size: medium;"><span style="font-size: large;">Python, Cython(Pyrex), and C.</span></span></h2>
</div>
</div>
</div>
<div>
Now we have the start of a module using Cython (_sdl2.pyx). Why Cython?</div>
<ul>
<li>We discussed on the mailing list already that it would be good for new features.</li>
<li>It also works with python 2.6+/3.</li>
<li>Cython can be used on platforms where ctypes can't, and is faster than ctypes.</li>
<li>Cython is more readable to python people.</li>
<li>Cython is also ok on pypy (cffi is the faster option on pypy still).</li>
<li>We already had one Cython/Pyrex module (the portmidi module that the midi.py module uses internally).</li>
</ul>
<div>
<br /></div>
<div>
If we implement things in python using a fairly straight wrapping of SDL2 in Cython,</div>
<div>
then it would be much easier for people in the future to make a CFFI enhancement for pypy.</div>
<div>
If we keep the C parts separate from the python parts, then it will be easier to maintain.</div>
<div>
What I mean, is that C+SDL code should not have any Python.h code in the file.<br />
<br />
<div>
See <a href="https://github.com/pygame/pygame/blob/master/src_c/_sdl2.pyx">src_c/_sdl2.pyx</a><br />
</div>
</div>
<div>
</div>
<div style="text-align: left;">
<h2>
<span style="font-size: large;">What language to use when? A summary.</span></h2>
</div>
<ul>
<li>New
features in python. Python implementations are easier to debug, change
and read. They run faster on pypy. They are a good educational resource,
because the python code is very readable. They can be used to
cross-check test implementations in C. The work done by e1000 on the
line drawing code has shown the value of this approach. Also the pypy
and Cpython projects use a 'python first' approach.</li>
<li>C code
should be kept separate from the Python.h parts. This is 'cleaner'
easier to read, and maintain. This opens the possibility to share the C
code more easily with other projects.</li>
<li>Keep SDL2 wrapping
close to the SDL2 C API. This makes a CFFI implementation easier if
someone wants to do so in the future. And it is clean layering. With
Cython we don't have as much of an overhead as with C+Python, because of
the compiler.</li>
<li>new SDL2 functionality can be wrapped with Cython.</li>
<li>Accelerators
of python code can be done in Cython (the sprite.py code is a great
example of something that could use Cython to speed up collision
detection and dirty rect handling).</li>
<li>We should generate .c files so that the user does not need to use Cython when installing pygame from source.</li>
</ul>
<div>
I'm excited that Cython will allow us to speed up sprite.py, collision handling and dirty rect handling.</div>
<div>
It's one of the "<a href="http://renesd.blogspot.com/2017/03/four-new-pygame-things-for-slow.html" target="_blank">Four new pygame things for slow computers.</a>" I wrote about in March 2017.</div>
<div>
Note, we already have 2 of 4 done :)</div>
<div>
<br /></div>
<div style="text-align: left;">
<h2>
<span style="font-size: medium;"><span style="font-size: large;">How to document with Cython?</span></span></h2>
</div>
<div>
There are some open questions with Cython however. The main one being...</div>
<div>
<ul>
<li>How do we use the documentation from inside a .pyx file and have it included in sphinx?</li>
</ul>
</div>
<div>
If we don't do this, then we need to maintain two sets of documentation.</div>
<div>
(Like we do for midi.py)</div>
<div>
The other way is a possibility too... maintaining the documentation in the .rst files,</div>
<div>
and somehow inserting the docs at build time.</div>
<div>
Hopefully someone else in the Cython/Pyrex community knows how to do this documentation workflow well.</div>
<div>
<br /></div>
<div style="text-align: left;">
<h2>
<span style="font-size: medium;"><span style="font-size: large;">Using projects to guide development.</span></span></h2>
</div>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0px 0px 0px 0.8ex; padding-left: 1ex;">
<div>
"<a href="https://geometrian.com/programming/tutorials/write-games-not-engines/">Write Games, Not Engines</a>"</div>
</blockquote>
<div>
<br /></div>
<div>
Working on the looper/video synth project was good to get some audio code working.</div>
<div>
But also, I've been working on fixing issues with '<a href="https://github.com/pygame/solarwolf" target="_blank">solarwolf</a>' as a test game,</div>
<div>
and it's playable but still with a couple of graphical glitches left.</div>
<div>
<br /></div>
<div>
<div>
<div>
Since we don't have unit tests which cover absolutely everything,</div>
<div>
using apps I know well to test is a pretty good alternative.</div>
</div>
<div>
<br /></div>
<div>
The approach of looking for issues, and then going into the debugger to find the</div>
</div>
<div>
Surface blit/transform combinations that are failing in the <a href="https://github.com/pygame/solarwolf" target="_blank">solarwolf</a> game is a good one.</div>
</div>
<div>
<br /></div>
</div>
<div>
<div style="text-align: left;">
<h2>
<span style="font-size: medium;"><span style="font-size: large;">The first 'game' using pygame 2. A community game.</span></span></h2>
</div>
<div>
Speaking of using projects to drive development.</div>
<div>
I'm starting this "pygame community game" thingo we talked about a while ago... now.</div>
<div>
<br /></div>
<div>
To enter it into the Ludumdare/Github GameOff jams which end in about 4.5 days.</div>
<div>
<br /></div>
<div>
I'll start another email thread and website announcement but...</div>
<div>
Anyone who wants to be involved can join the discord channel </div>
<div>
<a href="https://discord.gg/r8yreB6" target="_blank">web based chatroom</a>(discord) in the "#communitygame" channel.</div>
<div>
<br />
<br />
<br />
<br />
<br /></div>
</div>
</div>
</div>
</div>
</div>
</div>
Unknownnoreply@blogger.com30tag:blogger.com,1999:blog-10678074.post-794787662416151852018-11-08T09:27:00.001+00:002018-11-08T09:41:12.891+00:00Drawing data. With #flask, and #matplotlib.<div dir="ltr" style="text-align: left;" trbidi="on">
Here's a pretty simple example of using the Flask microweb framework, and matplotlib together.<br />
Matplotlib is a graphing library used by a lot of people in python.<br />
<br />
But how to serve your graphs up in a web browser as a png or an SVG image?<br />
Code is in this gist: <a href="https://gist.github.com/illume/1f19a2cf9f26425b1761b63d9506331f">flask matplotlib</a>.<br />
<br />
<h3 style="text-align: left;">
Serving svg and png images with Flask and matplotlib.</h3>
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgN0AZoViGfkQfkJYgdXmrTlLUIxsxL_5-mBwot_gRljQmX0Z07MFAnGuQ-2pxQ8gCaZmJyFeVDcMGAzFD-tzqZRIyo0hDjwBj7YvDPk7vPi5t0ikwzYRSMviJP6uw0LtHtuvBj/s1600/Screen+Shot+2018-11-08+at+10.20.55.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img alt="flask and matplotlib for drawing svg and png graphs with python" border="0" data-original-height="1338" data-original-width="822" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgN0AZoViGfkQfkJYgdXmrTlLUIxsxL_5-mBwot_gRljQmX0Z07MFAnGuQ-2pxQ8gCaZmJyFeVDcMGAzFD-tzqZRIyo0hDjwBj7YvDPk7vPi5t0ikwzYRSMviJP6uw0LtHtuvBj/s320/Screen+Shot+2018-11-08+at+10.20.55.png" width="196" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">here is what it looks like</td></tr>
</tbody></table>
<br />
Code is in this gist: <a href="https://gist.github.com/illume/1f19a2cf9f26425b1761b63d9506331f">flask matplotlib</a>.</div>
Unknownnoreply@blogger.com30tag:blogger.com,1999:blog-10678074.post-6779197375987684452018-11-01T10:13:00.001+00:002018-11-01T10:13:48.679+00:00Josh Bartlett — pygame Artist in Residence<div dir="ltr" style="text-align: left;" trbidi="on">
<div class="panel-body dont-break-out">
The pygame Artist in Residence grant <b>celebrates the
Python arts community</b> and lends a tiny bit of support to someones art
practice. Josh Bartlett was the first recipient.<br />
<br />
An artist residency usually works something like; a person spends
some time in either a gallery making something to present or in a music
club doing a weekly spot. The pygame artist in residence will do it in
their own space, but be present on the top of the pygame website in the
form of a thumbnail and a link to their patreon/blog/artist
statement/website/whatever.<br />
<br />
Josh Bartlett was the first recipient, and has been featured on the pygame website in October. Please see the <a href="https://www.pygame.org/artist-in-residence/1">pygame artist in residence profile page</a> for more info.<br />
<br />
<div class="row row-list">
<img class="col-md-4 col-md-offset-4 img-responsive" height="266" src="https://www.pygame.org/ftp/artist-in-residence/1/artist-in-residence-4-640.jpg" width="400" />
</div>
<br />
<br />
What has he been up to? <a href="https://sqizit.bartletts.id.au/">Go to his blog</a> to find out.<br />
<br />
<small>ps. Thank you to everyone who applied, and those who helped
with selection. Hopefully the next one will go more smoothly as the
process is improved.</small></div>
</div>
Unknownnoreply@blogger.com5tag:blogger.com,1999:blog-10678074.post-23732838054645883142018-08-31T23:15:00.002+01:002018-08-31T23:44:15.320+01:00pygame artist in residence grant<div dir="ltr" style="text-align: left;" trbidi="on">
<div>
An <span class="il">artist</span>
residency usually works something like; a person spends some time in
either a gallery making something to present or in a music club doing a
weekly spot.</div>
<div>
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://www.pygame.org/images/logo.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="317" data-original-width="800" height="126" src="https://www.pygame.org/images/logo.png" width="320" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center; vertical-align: top;">Artist in residence.</td></tr>
</tbody></table>
<br />
The first "pygame artist in residence" grant will be for a small amount of money (€512). It is for someone who is already doing stuff with <a href="https://www.pygame.org/">pygame</a> or python in their arts practice. There will be a little thumbnail on the website linking towards an <span class="il">artist</span> statement/patreon page/blog or some such. The residency won't be in a physical space. It lasts for a month, from October 1st - November 1st. At the end of the month the website 'exhibition' will take the form of a small update from the artist <span class="ILfuVd yZ8quc">— in whatever form that takes</span>.</div>
<div>
<br /></div>
<br />
<div>
If you'd like to apply(or nominate someone) please write something short with a link about the arts practice and how python or pygame is used in it.</div>
<div>
<b>Email: residency@pygame.org</b></div>
<div>
<b>Applications close: September 15th.</b><br />
<b>Announced: October 1st. </b><br />
<b>Amount: €512 paid by paypal.</b></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<hr />
<div>
Why
this? Art is awesome.<br />
<br />
Ok, there are other reasons... I feel the python world largely ignores the contributions that
the game development, graphics, education, music and the arts community provides
for python. People who are makers, musicicians, artists, teachers, and game devs have contributed
really major things to the python world. And the python website doesn't
even mention that art, or games are a thing. They don't exist apparently.</div>
<div>
</div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://pbs.twimg.com/media/DZ7b7lhW4AACqsX.jpg:large" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="420" data-original-width="800" height="168" src="https://pbs.twimg.com/media/DZ7b7lhW4AACqsX.jpg:large" width="320" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">According to python.org art does not exist</td></tr>
</tbody></table>
<div>
From Ada Lovelace the writer(and first programmer), to Steve Jobs who started out at Atari, to Bill gates the author of the first ever PC game, to Konrad Zuse the painter (who happened to also make the first digital computer) <span class="ILfuVd yZ8quc">— the arts and computers are intertwined</span>. Python itself is named after a comedy group, and was started off within a <a href="http://python-history.blogspot.com/2009/01/personal-history-part-1-cwi.html">multimedia group at CWI</a> after the author had worked on a programming system intended for teaching. Would python be the same without <span style="font-family: "courier new" , "courier" , monospace;">import this </span>( the zen of python) or as fun without <span style="font-family: "courier new" , "courier" , monospace;">import antigravity</span>?</div>
<div>
<br /></div>
<div>
Art, education, music, making, multimedia, and games are important to 'computers' and to python. Sure, stop calling pypi the cheese shop (made by a damn fine baker and game developer) because it's too silly... but please don't erase the arts from existence. <b>The python arts community exists.</b></div>
</div>
Unknownnoreply@blogger.com4tag:blogger.com,1999:blog-10678074.post-75797266588721776682018-08-27T16:41:00.001+01:002018-08-27T18:21:37.279+01:00Draft of, "How to port and market games using #python and #pygame."<div dir="ltr" style="text-align: left;" trbidi="on">
<div class="usertext-body may-blank-within md-container ">
<div class="md">
This is a collaborative document, and a really early draft. Please feel free to add any tips or links in a comment here or on the reddit post <a href="https://www.reddit.com/r/pygame/comments/9aodt7/collaborative_doc_lets_write_pygame_distribution/">https://www.reddit.com/r/pygame/comments/9aodt7/collaborative_doc_lets_write_pygame_distribution/</a> </div>
<div class="md">
</div>
You've spent two years making a game, but now want other people to see it?</div>
<div class="usertext-body may-blank-within md-container ">
How do you port it to different platforms, and make it available to others? How do you let people know it is even a thing? Is your game Free Libre software, or shareware?</div>
<div class="usertext-body may-blank-within md-container ">
</div>
<div class="usertext-body may-blank-within md-container ">
</div>
<div class="usertext-body may-blank-within md-container ">
<br />
All python related applications are welcome on <a href="https://www.pygame.org/">www.pygame.org</a>. You'll need a screenshot, a description of your game, and some sort of URL to link people to (a github/gitlab/bitbucket perhaps). But how and where else can you share it?</div>
<div class="usertext-body may-blank-within md-container ">
<br />
<div class="md">
<h2>
a few platforms to port to</h2>
<ul>
<li>itch.io and windows</li>
<li>windows store?</li>
<li>mac (for itch.io)</li>
<li>mac store</li>
<li>steam</li>
<li>linux 'flatpack' (latest fedora/ubuntu etc use this like an app store).</li>
<li>pypi (python packages can actually be installed by lots of people)</li>
<li>android store</li>
<li>web</li>
<li>debian</li>
<li>redhat/fedora</li>
</ul>
<h2>
Make it a python package.</h2>
Some of the tools work more easily with your package as a python
package. Working with all the different tools is sort of hard, and
having a convention for packaging would make things easier.<br />
<br />
Python packaging guide - <a href="http://packaging.python.org/">http://packaging.python.org/</a><br />
<br />
So, why don't we do things as a simple python package with one
example app to do this? pygame has an example app already, solarwolf - <a href="https://github.com/pygame/solarwolf">https://github.com/pygame/solarwolf</a>.
With work, it could be a good example app to use. We can also link in
this guide to other pygame apps that have been distributed on various
places.<br />
<br />
There are other example apps linked below for different distribution technology.<br />
<br />
<h3>
pyinstaller</h3>
<a href="https://www.pyinstaller.org/">https://www.pyinstaller.org/</a><br />
This can make ones for linux, windows, and mac.<br />
<blockquote class="tr_bq">
<pre><code>pyinstaller --onefile --windowed --icon=icon.ico <name game="" of="" your="">.py
</name></code></pre>
</blockquote>
<h2>
Windows</h2>
pynsist and pyinstaller can be used.<br />
<a href="https://github.com/takluyver/pynsist">https://github.com/takluyver/pynsist</a><br />
<br />
The benefit of pynsist is that it can create installers. Whereas pyinstaller is for making standalone executables (which is good if you are putting your app on the Steam store for example).<br />
<br />
<h3>
Windows code signing</h3>
<ul>
<li><a href="https://github.com/pyinstaller/pyinstaller/wiki/Recipe-Win-Code-Signing">https://github.com/pyinstaller/pyinstaller/wiki/Recipe-Win-Code-Signing</a></li>
<li><a href="https://docs.microsoft.com/en-us/windows/uwp/packaging/create-certificate-package-signing">https://docs.microsoft.com/en-us/windows/uwp/packaging/create-certificate-package-signing</a></li>
</ul>
<h2>
Flatpak</h2>
apps on linux - <a href="https://flatpak.org/">https://flatpak.org/</a><br />
Here's an example of making a pygame one.
<a href="https://github.com/flathub/flathub/pull/478">https://github.com/flathub/flathub/pull/478</a><br />
Developer guide for more detail here - <a href="http://docs.flatpak.org/en/latest/">http://docs.flatpak.org/en/latest/</a><br />
<h2>
pypi</h2>
The python package system can mean your app can be available for everyone who can use pip. Which is an audience in the millions.<br />
<h2>
Mac</h2>
pyinstaller is probably the best option at the moment. If your game
is open source, then you could use TravisCI for free to make builds with
pyinstaller.<br />
<br />
Unfortunately you probably need a Mac to make a mac build, test it,
and release on the mac/ios stores.
Getting a cheap apple machine off ebay might be the way to go. Or a
cloud account perhaps from 'macincloud'. Also the mac developer program
costs $100.<br />
<br />
Another option might be to borrow a friends machine to make the builds when it's time.<br />
See:<br />
<ul>
<li><a href="https://github.com/pyinstaller/pyinstaller/wiki/Recipe-OSX-Code-Signing">https://github.com/pyinstaller/pyinstaller/wiki/Recipe-OSX-Code-Signing</a></li>
<li><a href="https://developer.apple.com/macos/distribution/">https://developer.apple.com/macos/distribution/</a></li>
<li><a href="https://developer.apple.com/support/code-signing/">https://developer.apple.com/support/code-signing/</a></li>
</ul>
<h2>
iOS</h2>
It's not easy, but possible.<br />
<br />
With pygame 2 this should be possible since it uses the new SDL2.<br />
If you use LGPL code on iOS you still have to let your users benefit from the protections the LGPL gives them.<br />
<br />
Tom from renpy says... "I've been distributing Ren'Py under LGPL
section 6c, which says that you can distribute it along with a written
offer to provide the source code required to create the executables.
Since Ren'Py has a reasonably strong distinction between the engine and
game scripts, the user can then combine the game data from an iOS backup
with the newly-linked Ren'Py to get a package they can install through
xcode."
<a href="https://github.com/renpy/pygame_sdl2/issues/109#issuecomment-412156973">https://github.com/renpy/pygame_sdl2/issues/109#issuecomment-412156973</a><br />
<br />
An apple developer account costs $100, and selling things costs 30% of the cost of your app. <a href="https://developer.apple.com/">https://developer.apple.com/</a><br />
<h2>
Steam</h2>
There's a few games released using pygame on steam. Here are two threads of games released: <br />
<ul>
<li><a href="https://www.reddit.com/r/pygame/comments/87q9fr/i_just_published_my_game_the_four_colour_theorem/">https://www.reddit.com/r/pygame/comments/87q9fr/i_just_published_my_game_the_four_colour_theorem/</a></li>
<li><a href="https://www.reddit.com/r/pygame/comments/4ck5zv/released_a_pygame_game_on_steam_after_3_years_of/">https://www.reddit.com/r/pygame/comments/4ck5zv/released_a_pygame_game_on_steam_after_3_years_of/</a></li>
<li><a href="https://www.reddit.com/r/pygame/comments/3j6zvr/with_help_from_you_guys_my_first_game_launched_on/">https://www.reddit.com/r/pygame/comments/3j6zvr/with_help_from_you_guys_my_first_game_launched_on/</a></li>
</ul>
Costs $100 to join up and sell a game on this store.
<a href="https://partner.steamgames.com/">https://partner.steamgames.com/</a><br />
Recently someone used pyinstaller to package thier game.<br />
<blockquote class="tr_bq">
<pre><code>pyinstaller --onefile --windowed --icon=icon.ico <name game="" of="" your="">.py
</name></code></pre>
</blockquote>
<h3>
SteamworksPy</h3>
A python module for the C++ steam sdk.
<a href="https://github.com/Gramps/SteamworksPy">https://github.com/Gramps/SteamworksPy</a><br />
Made by someone who has released their game (using pygame) on steam.<br />
<br />
<h2>
Itch.io</h2>
"itch.io is an open marketplace for independent digital creators with a focus on independent video games."<br />
<ul>
<li>creators FAQ - <a href="https://itch.io/docs/creators/faq">https://itch.io/docs/creators/faq</a></li>
</ul>
Quite a few people have released their pygame games on itch.io.<br />
<br />
<h2>
Android</h2>
This isn't really possible to do well at the moment without a bit of work.<br />
<br />
python-for-android seems the best option, but doesn't work well with pygame. <a href="https://github.com/kivy/python-for-android">https://github.com/kivy/python-for-android</a>
There is an old and unmaintained pygame recipe included (for an old pygame 1.9.1). With some work it should be possible to update the recipe to use the SDL2 support in pygame.<br />
<br />
There was an older 'pygame subset for android' which is now unmaintained, and does not work with more recent Android devices.<br />
<h2>
Web</h2>
There's not really an 'export for web' option at the moment.
It is possible with both CPython and SDL as well as SDL2 working on
emscripten (the compiler for WASM and stuff that goes on the web).<br />
Here is the latest 'cpython on web' project. <a href="https://github.com/iodide-project/pyodide">https://github.com/iodide-project/pyodide</a><br />
<h2>
Building if you do not have a windows/mac/linux machine</h2>
<h3>
CI tools</h3>
If your game is open source, you can use these systems to build your game remotely for free.<br />
<ul>
<li>Appveyor (for windows) <a href="https://www.appveyor.com/">https://www.appveyor.com/</a></li>
<li>Travis (for linux and mac) <a href="https://travis-ci.org/">https://travis-ci.org/</a></li>
</ul>
How to do that? Well, that's an exercise left up to the reader.
Probably getting it to use pyinstaller, and having them upload the
result somewhere.<br />
<br />
One python app that uses Travis and Appveyor is the Mu editor. You can see how in their .travis.yml and appveyor.yml files. See <a href="https://github.com/mu-editor/mu">https://github.com/mu-editor/mu</a><br />
<br />
<br />
<h3>
Virtualbox</h3>
With virtualbox (and other emulators) you can run some systems on your local machine. Which means you do not need to buy a new development machine yourself for those platforms.<br />
<br />
Both windows and linux images are available that you could use legally.<br />
<br />
<a href="https://developer.microsoft.com/en-us/microsoft-edge/tools/vms/">https://developer.microsoft.com/en-us/microsoft-edge/tools/vms/</a><br />
<br />
Note, that it is good to do your testing on a free install, rather than testing on the same machine that you made your executables with. This is because perhaps you forgot to include some dependency, and that dependency is on the development machine, but not everyone else's machines.<br />
<br />
<h2>
Writing portable python code</h2>
Some old (but still valid) basic advice on making your game portable: <a href="https://www.pygame.org/wiki/distributing">https://www.pygame.org/wiki/distributing</a><br />
<br />
Things like naming your files case sensitively.<br />
<h2>
Announcing your game.</h2>
Generic Indie game marketing guides all apply here.<br />
<br />
Some python/pygame specific avenues for marketing and announcing...<br />
<ul>
<li>post a 'release' on the pygame website</li>
<li>mention it on the pygame reddit (and python reddit perhaps)</li>
<li>announce on the python announce mailing list (<a href="https://mail.python.org/mailman/listinfo/python-announce-list">https://mail.python.org/mailman/listinfo/python-announce-list</a>)</li>
<li>get your blog listed on planet with your 'python' tag. (open an issue <a href="https://github.com/python/planet/issues">https://github.com/python/planet/issues</a>)</li>
<li>mention @pygame_org or #pygame for retweets</li>
</ul>
Of course the python world is a tiny place compared to the entire world.<br />
<br />
<ul style="text-align: left;">
<li><a href="https://www.reddit.com/r/gamedevexpo/">https://www.reddit.com/r/gamedevexpo/</a></li>
</ul>
<br />
<br />
<h2 style="text-align: left;">
Icons.</h2>
Each platform has slightly different requirements for icons. This might be a nice place to link to all the requirements (TODO).<br />
<br />
<br />
<h2>
Making a game trailer (for youtube)</h2>
You may not need to make the best trailer, or even a good trailer.
Just a screen capture of your game might be 'good enough' and is better
than nothing.<br />
<br />
How about making a trailer with pygame itself? You could call it 'demo mode', or 'intro mode'.<br />
There's a free iMovie on Mac, the Microsoft video editor on windows,
and blender for all platforms. An alternative is to use the python
module <a href="https://zulko.github.io/moviepy/">moviepy</a> and script your game trailer.<br />
<br />
OBS is pretty good multi platform free screen capture software. <a href="https://obsproject.com/download">https://obsproject.com/download</a><br />
<ul>
<li><a href="https://www.youtube.com/watch?v=NOx3uvUyHGs">the top 10 best indie game trailers 2018</a></li>
<li><a href="https://gamedevelopment.tutsplus.com/tutorials/how-to-make-an-indie-game-trailer-with-no-budget--cms-20825">How to Make an Indie Game Trailer With No Budget</a> </li>
</ul>
<h2>
Animated gif</h2>
These are useful for sharing on twitter and other such places, so people can see game play.<br />
You can save the .png files with pygame, and convert them to a gif with the 'convert' tool from imagemagik. <br />
<blockquote class="tr_bq">
<pre><code># brew install imagemagick
# sudo apt-get install imagemagick
# call this in your main loop.
pygame.image.save(surf, 'bla_%05d.png' % frame_idx)
</code></pre>
</blockquote>
<br />
Now you can convert the png files to <br />
<blockquote class="tr_bq">
<pre><code>convert -delay 20 -loop 0 bla_*png animated.gif
</code></pre>
</blockquote>
<br />
Some solutions on stack overflow.<br />
<ul>
<li><a href="https://stackoverflow.com/questions/10922285/is-there-a-simple-way-to-make-and-save-an-animation-with-pygame">https://stackoverflow.com/questions/10922285/is-there-a-simple-way-to-make-and-save-an-animation-with-pygame</a></li>
<li><a href="https://stackoverflow.com/questions/753190/programmatically-generate-video-or-animated-gif-in-python">https://stackoverflow.com/questions/753190/programmatically-generate-video-or-animated-gif-in-python</a></li>
</ul>
</div>
</div>
</div>
Unknownnoreply@blogger.com18tag:blogger.com,1999:blog-10678074.post-90246789212063507432018-07-19T00:00:00.000+01:002018-07-19T10:14:25.696+01:00pygame 1.9.4 released<div dir="ltr" style="text-align: left;" trbidi="on">
<div class="preview-content">
<div class="comment-body markdown-body js-preview-body" style="min-height: 600px;">
<div align="center">
<a href="https://www.pygame.org/wiki/about" rel="nofollow"><img alt="pygame 1.9.4" data-canonical-src="https://www.pygame.org/images/logo.png" src="https://camo.githubusercontent.com/1971c0a4f776fb5351c765c37e59630c83cabd52/68747470733a2f2f7777772e707967616d652e6f72672f696d616765732f6c6f676f2e706e67" style="max-width: 100%;" width="300px" /></a></div>
pygame 1.9.4 has been released into the wild!<br />
<h2>
TLDR; Some highlights.</h2>
<ul>
<li>python 3.7 support.</li>
<li>beta <a href="https://www.pypy.org/" rel="nofollow">pypy</a> support. See <a href="https://github.com/pygame/pygame/issues?q=is%3Aopen+is%3Aissue+milestone%3Apypy">Are we pypy yet?</a>.</li>
<li>pygame.draw fixes</li>
<li>pygame.math is not experimental anymore. Speedups and bugfixes.</li>
<li>Debian, Mac homebrew, mac virtualenv, manylinux and other platform fixes.</li>
<li>documentation fixes, jedi support for type ahead in editors like VSCode and VIM.</li>
<li>Surface.blits for blitting many surfaces at once more quickly.</li>
</ul>
<h2>
Thanks</h2>
A very special thanks to the people who have volunteered commits to pygame since the last release. In alphabetical order...<br />
Adam Di Carlo (<a class="user-mention" data-hovercard-user-id="6961386" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/adicarlo">@adicarlo</a>) | Christian Bender (<a class="user-mention" data-hovercard-user-id="23243382" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/christianbender">@christianbender</a>) | Don Kirkby (<a class="user-mention" data-hovercard-user-id="1639148" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/donkirkby">@donkirkby</a>) | endolith (<a class="user-mention" data-hovercard-user-id="58611" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/endolith">@endolith</a>) | hjpotter92 (<a class="user-mention" data-hovercard-user-id="3393533" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/hjpotter92">@hjpotter92</a>) | Ian Mallett (<a class="user-mention" data-hovercard-user-id="8649790" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/imallett">@imallett</a>) | Lenard Lindstrom (<a class="user-mention" data-hovercard-user-id="26581447" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/llindstrom">@llindstrom</a>) | Mathias Weber (<a class="user-mention" data-hovercard-user-id="882006" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/mweb">@mweb</a>) | Matti Picus (<a class="user-mention" data-hovercard-user-id="823911" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/mattip">@mattip</a>) | Nicholas Tollervey (<a class="user-mention" data-hovercard-user-id="37602" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/ntoll">@ntoll</a>) | (<a class="user-mention" data-hovercard-user-id="23108442" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/orangudan">@orangudan</a>) | Raymon Skjørten Hansen (<a class="user-mention" data-hovercard-user-id="16223275" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/raymonshansen">@raymonshansen</a>) | René Dudfield (<a class="user-mention" data-hovercard-user-id="9541" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/illume">@illume</a>) | Stefan Bethge (<a class="user-mention" data-hovercard-user-id="12260" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/kjyv">@kjyv</a>) | Stuart Axon (<a class="user-mention" data-hovercard-user-id="179677" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/stuaxo">@stuaxo</a>) | Thomas Kluyver (<a class="user-mention" data-hovercard-user-id="327925" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/takluyver">@takluyver</a>) | Tobias Persson (<a class="user-mention" data-hovercard-user-id="733271" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/Anisa">@Anisa</a>)<br />
<br />
I'm probably missing some people, and also missing some people who contributed in other ways.<br />
For example, in discussions, issue reports, helping out on the wiki, the website, and for helping others<br />
in the community, and providing good vibes. So whilst the commits are
easy to use to make a list of people to thank, it's not inclusive of
everyone who deserves thanks.<br />
<h2>
More details.</h2>
<a href="https://github.com/pygame/pygame/pull/451">#451</a> <a href="https://github.com/pygame/pygame/pull/460">#460</a> <a href="https://github.com/pygame/pygame/pull/467">#467</a> <a href="https://github.com/pygame/pygame/pull/468">#468</a> <a href="https://github.com/pygame/pygame/pull/469">#469</a> <a href="https://github.com/pygame/pygame/pull/470">#470</a><br />
<a href="https://github.com/pygame/pygame/pull/444">#444</a> link to help pages when compile fails.<br />
<a href="https://github.com/pygame/pygame/pull/443">#443</a> In set_error get_error tests ignore first error. Could be anything.<br />
<a href="https://github.com/pygame/pygame/pull/442">#442</a> Freetype requires pkg-config instead of freetype-config now.<br />
<a href="https://github.com/pygame/pygame/pull/439">#439</a> Surface.blits<br />
<a href="https://github.com/pygame/pygame/pull/435">#435</a> Adding pypy builds for Mac on travis.<br />
<a href="https://github.com/pygame/pygame/pull/432">#432</a> Appveyor pypy and pypy3 windows 32bit.<br />
<a href="https://github.com/pygame/pygame/pull/431">#431</a> Implement object alloc caching for rect.c to improve on pypy.<br />
<a href="https://github.com/pygame/pygame/pull/427">#427</a> PixelArray.close(), with PixelArray(surf) as px, context manager.<br />
<a href="https://github.com/pygame/pygame/pull/426">#426</a> Skip tests that rely on arrinter and pythonapi on pypy.<br />
<a href="https://github.com/pygame/pygame/pull/420">#420</a> pypy didn't like tp_dictoffset hack in events. Make our own setter, getter.<br />
<a href="https://github.com/pygame/pygame/pull/418">#418</a> draw.aaline should work with ARGB surfaces (like on mac).<br />
<a href="https://github.com/pygame/pygame/pull/416">#416</a> Vector cleanup<br />
<a href="https://github.com/pygame/pygame/pull/415">#415</a> So virtualenv gets a focused window on Mac too.<br />
<a href="https://github.com/pygame/pygame/pull/414">#414</a> Mac Travis homebrew fix<br />
<a href="https://github.com/pygame/pygame/pull/413">#413</a> Jedi confused by pygame imports. Make it happy.<br />
<a href="https://github.com/pygame/pygame/pull/408">#408</a> pygame.transform.threshold tests, keyword arguments, docs.<br />
<a href="https://github.com/pygame/pygame/pull/403">#403</a> pygame.math.Vector2/3 not experimental<br />
<a href="https://github.com/pygame/pygame/pull/398">#398</a> Clean up _camera_vidcapture.py unused code, and document a bit.<br />
<a href="https://github.com/pygame/pygame/pull/394">#394</a> Add pitch bend to MIDI library<br />
<a href="https://github.com/pygame/pygame/pull/392">#392</a> Add pypy builder to travis ci, and allow it to fail.<br />
<a href="https://github.com/pygame/pygame/pull/391">#391</a> ppc64le and other Debian fixes<br />
<a href="https://github.com/pygame/pygame/pull/389">#389</a> pygame.draw.circle with a thickness had a weird moiré pattern.<br />
<a href="https://github.com/pygame/pygame/pull/387">#387</a> test python 3.7 on travis CI.<br />
<a href="https://github.com/pygame/pygame/pull/386">#386</a> python 3.7 fixes.<br />
<a href="https://github.com/pygame/pygame/pull/384">#384</a> pygame.display doc fixes.<br />
<a href="https://github.com/pygame/pygame/pull/381">#381</a> import rect.inflate docs.<br />
<a href="https://github.com/pygame/pygame/pull/363">#363</a> Fix several typos, and improve grammar in the introduction.<br />
<a href="https://github.com/pygame/pygame/pull/361">#361</a> Add unit test for some key functions.<br />
<a href="https://github.com/pygame/pygame/pull/360">#360</a> update math.c for pypy.<br />
<a href="https://github.com/pygame/pygame/pull/357">#357</a> add UYVY support for better linux camera support.<br />
<a href="https://github.com/pygame/pygame/pull/356">#356</a> Fix aaellipse artifacts<br />
<a class="commit-link" href="https://github.com/pygame/pygame/commit/703350f68182641fe72ef1f41d81998fb6d4b385"><tt>703350f</tt></a> Update Rect slicing for Python 3<br />
<a href="https://github.com/pygame/pygame/commit/6d0e97a38b5d2d549f1fee8b912a072731ea33bb"><tt>6d0e97a</tt></a> bug fix for freetype.Font.render_to()<br />
<a href="https://github.com/pygame/pygame/issues/78">#78</a> Add environment PYGAME_EXTRA_BASE to add an extra base directory to the start of the search path.<br />
<a href="https://github.com/pygame/pygame/issues/77">#77</a> Build alsa libs ourselves for manylinux builds.<br />
<a href="https://github.com/pygame/pygame/issues/76">#76</a> Docs fixup.</div>
</div>
</div>
Unknownnoreply@blogger.com15tag:blogger.com,1999:blog-10678074.post-16020918281991010662018-04-11T15:33:00.001+01:002018-04-12T09:04:23.538+01:00Berlin to Copenhagen by bike.<div dir="ltr" style="text-align: left;" trbidi="on">
<div>
There's a bike path running from the north of Europe all the way to the south. Well, it's not completely finished yet, but it goes most of the way. It's part of the <a href="http://www.eurovelo.org/">EuroVelo</a> project, which aims to have a high quality cycling network made of 15 routes all around Europe completed by 2020.</div>
<div>
<br /></div>
<div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEglpLe4kUxs75mscMEXnHy5F-6O6LoZV8tkMPmI8Ep7AP4CXgn9D-pUsYg7GNDtb2EBJNuFO5uILaGXy42MUjBkRZGG39Czl3r-RIU_K4wYu5gKtw7dT5vhfo94yuYPyX2jKCCF/s1600/IMG_9494.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="1600" data-original-width="1600" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEglpLe4kUxs75mscMEXnHy5F-6O6LoZV8tkMPmI8Ep7AP4CXgn9D-pUsYg7GNDtb2EBJNuFO5uILaGXy42MUjBkRZGG39Czl3r-RIU_K4wYu5gKtw7dT5vhfo94yuYPyX2jKCCF/s320/IMG_9494.jpg" width="320" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">All my clothes and possessions for a seven day trip from Berlin to Copenhagen. I was just going to tie a sack under my seat... but was convinced that perhaps these special bags are a good idea after all.</td></tr>
</tbody></table>
One part that's done well is the <span class="il">Berlin</span> to <span class="il">Copenhagen</span> leg(I guess not so surprising since they are two cities that are really into supporting bikes).</div>
<div>
Scroll down on these <span class="il">photos</span> someone took, if you want a look.<br />
<a data-saferedirecturl="https://www.google.com/url?hl=en&q=http://www.slowtravelberlin.com/in-photos-berlin-copenhagen-cycle-route/&source=gmail&ust=1523541017387000&usg=AFQjCNGvuQRLqh-JxBBKtkZLVckCr3Nswg" href="http://www.slowtravelberlin.com/in-photos-berlin-copenhagen-cycle-route/" target="_blank"><span class="il">http</span>://<span class="il">www</span>.<span class="il">slowtravelberlin</span>.co<wbr></wbr>m/<span class="il">in</span>-<span class="il">photos</span>-<span class="il">berlin</span>-<span class="il">copenhagen</span>-<wbr></wbr><span class="il">cycle</span>-<span class="il">route</span>/</a><br />
<br />
Except, I'm going to do it at the end of April, not <span class="il">in</span>
winter like them. Which is usually the most dry time of year, and with a temperature that's not so hot or cold(usually). Fingers crossed it will be dry and warm. It will probably take 7 days if I can ride fast enough each day (about 100km). Eeek. And this part of the route (like many other parts) has a really nice website <a data-saferedirecturl="https://www.google.com/url?hl=en&q=http://www.bike-berlin-copenhagen.com/route/brandenburg-stage&source=gmail&ust=1523541017387000&usg=AFQjCNEsv10zjyeLvNFSRbnSUooNXSKONQ" href="http://www.bike-berlin-copenhagen.com/route/brandenburg-stage" target="_blank"><span class="il">http</span>://<span class="il">www</span>.bike-<span class="il">berlin</span>-copenha<wbr></wbr>gen.<span class="il">com</span>/<span class="il">route</span>/brandenburg-<wbr></wbr>stage</a></div>
<div>
Going
to go very minimal, and take my fast(ish) road bike. With just three sets of clothes(including the ones I'll be wearing), a netbook, and a toothbrush. It probably
would be nice to have a tent, but I'll pass. Instead I'll sleep <span class="il">in</span> the forest. If it rains, I'll just get wet.<br />
<br />
Just kidding (or am I?). I'll probably cheat and stay <span class="il">in</span> a hotel some nights rather than sleep <span class="il">in</span> forest. I chose the laptop over the tent, because it's nice to be able to sit by a river in a field with no one around and write some nonsense.<br />
<br />
A tiny paper notebook and pencils for drawing will also make it into my bike-pack.</div>
<div>
<br />
Already I've started training a bit, since I didn't ride my bike for much of the winter. The first 6 hour test ride I did left me sore and tired. But that was going from almost zero riding over the last four months and straight into that. Not the smartest idea to go from nothing into such a long ride... but very enjoyable none the less.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh65McSpe31V5EL2EkfBkDrMfjxe8GM9lPpd4wkCkpYG5hv0ZnZlHi5FPM2o_h460R9PafJ1RlgWrysVCfmllWoLEaxIG9_nWbi3UP2msPbYhpadeVY-BNcK8qV7XvhCyj6db47/s1600/IMG_9477.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="853" data-original-width="1280" height="213" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh65McSpe31V5EL2EkfBkDrMfjxe8GM9lPpd4wkCkpYG5hv0ZnZlHi5FPM2o_h460R9PafJ1RlgWrysVCfmllWoLEaxIG9_nWbi3UP2msPbYhpadeVY-BNcK8qV7XvhCyj6db47/s320/IMG_9477.jpg" width="320" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Learning from long practice rides around Berlin. Sitting by a lake.</td></tr>
</tbody></table>
So what have I learnt so far?<br />
<ul style="text-align: left;">
<li>Padded pants are a good idea.</li>
<li>Water is important.</li>
<li>Sunscreen is important even on cloudy days.</li>
<li>I am not fit.</li>
<li>Riding around is one of life's pleasures.</li>
</ul>
My next long practice ride is going to be with all of my gear on the bike. I expect to learn more. For example, I'll time how far I can go and at what speed. And I also want to try riding in some heavy rain - to test if my bags are actually waterproof. But unfortunately the weather has been pretty nice recently. <br />
<br />
Other than experiments I plan to do, I'm reading about what other people do on their long bike rides. And also talking with people. But it's always hard to learn lessons from others, and the best way to find out what is important to me is to do practice rides.<br />
<br />
I think I should be able to ride five hours each day with breaks in between, and still get up to my 100km target. But unsure really. By then the days should be quite long, and the nights with a fair bit of light.<br />
<ul style="text-align: left;">
<li>Berlin day length: 14:30</li>
<li>Copenhagen day light length: 15:30</li>
</ul>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjgtj0mkc8-wP-kiQjDxtGlz_E_MByjo0nDo_l-Csz14ULyg7YDH1_logQH0mws4o5VWASwMhz5T9yL00J0rh6olwz2AeNNNcJatf63C3S-Pwxol2zBWnq8v6rhJcLGVUoDq97G/s1600/IMG_9483.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="853" data-original-width="1280" height="213" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjgtj0mkc8-wP-kiQjDxtGlz_E_MByjo0nDo_l-Csz14ULyg7YDH1_logQH0mws4o5VWASwMhz5T9yL00J0rh6olwz2AeNNNcJatf63C3S-Pwxol2zBWnq8v6rhJcLGVUoDq97G/s320/IMG_9483.jpg" width="320" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Spring flowers such as 'Tommies' and 'Snow drops' are popping their heads out for spring. Looking forward to seeing all the different landscapes along the way.</td></tr>
</tbody></table>
With 14-15 sunlight hour days, there's lots of time for slow riding if wind is in my face, for detours, and for sits under trees drawing. By not using the side bags, and not carrying all that much weight I should be able to go a lot faster than other people I've talked to who have done the ride. Their bikes were pretty bad, and they carried a lot more weight than what I'm going to.<br />
<br />
I don't really have a good solution for route planning. I have a route I'd like to go, but I can't really put it on google maps easily, so it's visible on an app. So that's another thing I need to find out about. Luckily I've done lots of map development before... but I hope there's not much coding to do such a simple task, that surely other people have done before.<br />
<br />
Also, I'm a bit worried about riding in the rain. On average it rains 8 out of 30 days that time of year. So it's quite likely I'll see rain. Generally, I just don't ride my bike if it rains. But I'll have to I guess, because 'waiting it out' doesn't work if it's raining all day.<br />
<br /></div>
</div>
Unknownnoreply@blogger.com4tag:blogger.com,1999:blog-10678074.post-34074602932494040262018-04-03T17:05:00.002+01:002018-04-03T17:05:33.164+01:00pygame.Surface.blits<div dir="ltr" style="text-align: left;" trbidi="on">
Drawing many things at once is faster than drawing things one thing at a time. So, today I made a function to do this with pygame.<br />
<br />
It seems to take between 82%-85% of the time it takes to draw things one at a time. Which considering blit is often the bottleneck in many pygame apps is a pretty nice improvement. It will be landing in the pygame 1.9.4 release.<br />
<br />
More info in the <a href="https://github.com/pygame/pygame/pull/439">blits Pull Request</a>.</div>
Unknownnoreply@blogger.com2tag:blogger.com,1999:blog-10678074.post-49014689670034227912018-04-03T09:26:00.002+01:002018-04-03T09:50:41.863+01:00Moss box<div dir="ltr" style="text-align: left;" trbidi="on">
I've been looking to get more greenery into my studio, and I kind of like moss. It reminds me of when the times in my childhood running around rain forests barefoot.<br />
<br />
Luckily, a good friend has a whole bunch of it growing out the back of her apartment. And it's likely to be destroyed soon by workers. Which made me not feel so bad about digging some up.<br />
<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhpw4t1y9ZeK3IOGOjycxa_rz0-5sNuRLWmPzD5-7j7QZWkNDe-1XKa2tY4i0r8C7fo3Fwrxo8egW6nZKYWFSQch8y7pphHmKWSmqaXxCraR300J7mn6gfFpdryZELLOz13k1o_/s1600/IMG_9458.JPG" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="1067" data-original-width="1600" height="213" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhpw4t1y9ZeK3IOGOjycxa_rz0-5sNuRLWmPzD5-7j7QZWkNDe-1XKa2tY4i0r8C7fo3Fwrxo8egW6nZKYWFSQch8y7pphHmKWSmqaXxCraR300J7mn6gfFpdryZELLOz13k1o_/s320/IMG_9458.JPG" width="320" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">I scavenged some egg cartons</td></tr>
</tbody></table>
<br />
First I found some egg cartons to use as a base. Underneath the moss is a good few centimeters of dirt.<br />
<br />
Unlike most of the wall moss I've seen around the place, this moss is alive. I was sort of horrified to learn that people buy dead preserved moss, and glue gun it into place. But this stuff is alive. Sometimes looking stuff up on Youtube kills the magic. Now I don't think wall moss is anywhere near as cool as I once thought it was. Thanks internet.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiJXjFPub_NEVYPNc02xKxEek8WooUtV7idWcO-s4VNU4r6yzq-hCvRPTI6tRR4wjdn-Fx0HYph-niulBOnD2MTSnyB22jqvcUC2RFwph-psgp_U8ERJeXy0s6ccd9Cp4-HqMwK/s1600/IMG_9463.JPG" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="1067" data-original-width="1600" height="213" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiJXjFPub_NEVYPNc02xKxEek8WooUtV7idWcO-s4VNU4r6yzq-hCvRPTI6tRR4wjdn-Fx0HYph-niulBOnD2MTSnyB22jqvcUC2RFwph-psgp_U8ERJeXy0s6ccd9Cp4-HqMwK/s320/IMG_9463.JPG" width="320" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Here it is sitting on my standing desk.</td></tr>
</tbody></table>
<br />
Next I found an old wooden box. I lined the bottom of this with some plastic I had about the place. Yuk, I know. However, it should hopefully stop any water leaks.<br />
<br />
There's two half egg carton containers I found (to hold 12 eggs) being used to fill the box. With spaces around the edge.<br />
<br />
The nice thing about having it in a small box is it's easy to move about the place. So I can put it on top of my desk, or even under the desk where my bare feet are. My next project is probably to grow some green grasses in a box for that grass-on-feet-inside feeling.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjAsovh1Jxm0fGN53wm6wYDszr89yeTSF_Oj1178jZCL6eXDiWWCxy_mwJRkAVr2-APQhk7EImYuTUadoiDLhGcm5UKKZ6f8kUYk3A4A0jhjZ4xNGPSELTf4P_bf5I6BMv-VM9Y/s1600/IMG_9468.JPG" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="1067" data-original-width="1600" height="213" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjAsovh1Jxm0fGN53wm6wYDszr89yeTSF_Oj1178jZCL6eXDiWWCxy_mwJRkAVr2-APQhk7EImYuTUadoiDLhGcm5UKKZ6f8kUYk3A4A0jhjZ4xNGPSELTf4P_bf5I6BMv-VM9Y/s320/IMG_9468.JPG" width="320" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Can you see the cardboard carton?</td></tr>
</tbody></table>
I'm not so sure if it will survive, but I'm hoping it will grow. So far I've just been spraying it with water, and occasionally touching and smelling the moss. Which takes me back to the rain forest a bit.<br />
<br /></div>
Unknownnoreply@blogger.com1tag:blogger.com,1999:blog-10678074.post-44377748118713862012018-03-23T12:02:00.001+00:002018-03-23T12:03:20.644+00:00Investigating pypy frame drop<div dir="ltr" style="text-align: left;" trbidi="on">
pypy has spikes in time it takes occasionally when using cpyext, otherwise known as pauses.<br />
This is because it's deallocating lots of CPython objects all in one
frame, rather than incrementally. This is likely to be addressed in a
future release of pypy.<br />
<br />
<div class="markup">
Mainly it's a problem when creating and deleting lots of CPython objects (like pygame.Rect). Not pure python objects. </div>
<br />
To work around it for now,<br />
<ul>
<li>the <a href="https://github.com/pygame/pygame/blob/master/src/rect.c">src/rect.c</a>
has been changed to have a free list of python objects, so it doesn't
alloc and dealloc them. Instead it maintains it's own list of rect
PyObject pointers, and reuses them. This is done in PR <a class="issue-link js-issue-link" data-error-text="Failed to load issue title" data-id="307989931" data-permission-text="Issue title is private" data-url="https://github.com/pygame/pygame/issues/431" href="https://github.com/pygame/pygame/pull/431">#431</a></li>
<li>Use PYPY_GC_NURSERY=1m <a href="http://doc.pypy.org/en/latest/gc_info.html?highlight=incminimark#environment-variables" rel="nofollow">incminimark environment variables</a> when running pypy.</li>
</ul>
Below are timings of the default pygame, and default pypy (at time of
writing). Then a benchmark showing the improvement with the rect
freelist implemented in rect.c inside pygame. Then we also improve
things by setting PYPY_GC_NURSERY=1m environment variable to change the
behavior of the GC. I'm not aware of an API to do this within pygame
itself unfortunately. Finally there are some cpython 3.6 timings for
comparison.<br />
<h2>
hacked up benchmark.</h2>
The hacked up benchmark is in the branch pypy-hack-frame-bench of the
pygame repo. It's based off examples.testsprite.<br />
<br />
Because pypy currently
doesn't compile matplot lib you need to run it in two steps. First to
do the run (let it run for 20+ seconds). Then to load the data pickle,
and show the graph.<br />
<br />
<pre><code>pypy examples/testsprite.py -plot -noupdate_rects -width 320 -height 200 100
python examples/testsprite.py -plotpickle </code></pre>
<br />
To run with the GC tweak...<br />
<br />
<code>PYPY_GC_NURSERY=1m pypy examples/testsprite.py -plot -noupdate_rects -width 320 -height 200 100
</code><br />
<pre><code> </code></pre>
<pre><code> </code></pre>
<h2>
graph colors</h2>
<ul>
<li><b>Blue</b> is the first 1200 frames</li>
<li><b>orange</b> is the last 1200 frames of the run.</li>
</ul>
This is useful to see if there are any jit warm up affects happening, or different behaviour over time. (doesn't appear so).<br />
<h2>
Time per frame on pypy 5.10.0.</h2>
For this example we don't see any 'jit warm up' behavior.<br />
<a href="https://user-images.githubusercontent.com/9541/37825690-03f9aae0-2e91-11e8-93ba-caa3933b5635.png" target="_blank"><img alt="screen shot 2018-03-23 at 07 21 05" height="240" src="https://user-images.githubusercontent.com/9541/37825690-03f9aae0-2e91-11e8-93ba-caa3933b5635.png" style="max-width: 100%;" width="320" /></a><br />
<h2>
Time per frame on pypy 5.10.0, rect freelist, standard GC.</h2>
<a href="https://user-images.githubusercontent.com/9541/37826129-96fe33aa-2e92-11e8-9c01-cc714b0bee7d.png" target="_blank"><img alt="screen shot 2018-03-23 at 12 05 42" height="227" src="https://user-images.githubusercontent.com/9541/37826129-96fe33aa-2e92-11e8-9c01-cc714b0bee7d.png" style="max-width: 100%;" width="320" /></a><br />
<h2>
Time per frame on pypy 5.10.0, rect freelist, PYPY_GC_NURSERY=1m.</h2>
<a href="https://user-images.githubusercontent.com/9541/37826127-946490bc-2e92-11e8-9946-c7ae9e522e04.png" target="_blank"><img alt="screen shot 2018-03-23 at 12 04 42" height="235" src="https://user-images.githubusercontent.com/9541/37826127-946490bc-2e92-11e8-9946-c7ae9e522e04.png" style="max-width: 100%;" width="320" /></a><br />
This final result for pypy 'ok' because we want under 0.0166 seconds
used per frame. Improvements inside pypy itself in the future should
hope to remove many of these spikes.<br />
<h2>
Time per frame on python3.6</h2>
Here we see both the average time per frame and maximums are smaller on python3.6.<br />
<a href="https://user-images.githubusercontent.com/9541/37826718-e7aed49c-2e94-11e8-8358-3d6919718163.png" target="_blank"><img alt="screen shot 2018-03-23 at 12 22 02" height="233" src="https://user-images.githubusercontent.com/9541/37826718-e7aed49c-2e94-11e8-8358-3d6919718163.png" style="max-width: 100%;" width="320" /></a><br />
<h2>
Time per frame on python3.6, rect freelist</h2>
It seems the freelist for rects on python3.6 has a minimal affect.<br />
<a href="https://user-images.githubusercontent.com/9541/37826713-e54b070c-2e94-11e8-86d6-d8a758722062.png" target="_blank"><img alt="screen shot 2018-03-23 at 12 20 12" height="231" src="https://user-images.githubusercontent.com/9541/37826713-e54b070c-2e94-11e8-86d6-d8a758722062.png" style="max-width: 100%;" width="320" /></a><br />
<h2>
Time per frame on python3.6, gc.disable()</h2>
Here we can see using gc.disable() seems to have no affect in this
program (on python 3.6). I also tried this on pypy, and it had no affect
either (on this program).<br />
<a href="https://user-images.githubusercontent.com/9541/37826821-4fd1d556-2e95-11e8-9872-4a26aa08f4bb.png" target="_blank"><img alt="screen shot 2018-03-23 at 12 25 14" height="234" src="https://user-images.githubusercontent.com/9541/37826821-4fd1d556-2e95-11e8-9872-4a26aa08f4bb.png" style="max-width: 100%;" width="320" /></a><br />
Other things I tried was to do a gc.collect(0) just before
display.flip(), however this didn't appear to help in this case. This is
because we can know if we have 5-10ms free waiting for the display
VSYNC to flip (when running at 60Hz). Perhaps this can be attempted
again as the cpyext GC is improved.</div>
Unknownnoreply@blogger.com2tag:blogger.com,1999:blog-10678074.post-66388545355392898722018-03-22T11:43:00.006+00:002018-03-24T07:54:37.606+00:00Windows pypy pygame build for testing.<div dir="ltr" style="text-align: left;" trbidi="on">
How to install <a href="https://pypy.org/">pypy</a> (fast python written in python) on windows and use the pygame dev build.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj9d9L6NG2kfOx_07463JtQmTZC599LbcYl7lxLCo5dshHPVLqiySqdNGyvZ5VRoMRUu3D0DpwplosGPdnXBrGdFRH2taZF7jhls9yDAam_7hs0i8EllyKUth-qErHZUVNwt7QJ/s1600/Screen+Shot+2018-03-24+at+08.41.56.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="1242" data-original-width="1600" height="248" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj9d9L6NG2kfOx_07463JtQmTZC599LbcYl7lxLCo5dshHPVLqiySqdNGyvZ5VRoMRUu3D0DpwplosGPdnXBrGdFRH2taZF7jhls9yDAam_7hs0i8EllyKUth-qErHZUVNwt7QJ/s320/Screen+Shot+2018-03-24+at+08.41.56.png" width="320" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">pypy running pygame.examples.aliens on Windows </td></tr>
</tbody></table>
<br />
1) Get pypy. It's just a zip folder with all the stuff inside.<br />
<a href="https://bitbucket.org/pypy/pypy/downloads/pypy2-v5.10.0-win32.zip">https://bitbucket.org/pypy/pypy/downloads/pypy2-v5.10.0-win32.zip</a><br />
Unzip, and put into C:\pypy, so C:\pypy\pypy.exe exists.<br />
<br />
<b>OR</b> get the pypy3.exe from <a href="https://bitbucket.org/pypy/pypy/downloads/pypy3-v5.10.1-win32.zip">https://bitbucket.org/pypy/pypy/downloads/pypy3-v5.10.1-win32.zip</a><br />
Unzip, and put into C:\pypy3, so C:\pypy3\pypy3.exe exists.<br />
<br />
<br />
2) Set the PATH environment variable, so that pypy.exe is found in your command prompt.<br />
<a href="https://www.opentechguides.com/how-to/article/windows-10/113/windows-10-set-path.html">https://www.opentechguides.com/how-to/article/windows-10/113/windows-10-set-path.html</a><br />
<br />
Or you can just do this each time you run a new cmd prompt. <br />
<blockquote class="tr_bq">
<div style="text-align: left;">
<span style="font-family: "courier new" , "courier" , monospace;">set PATH=%PATH%;C:\pypy\;C:\pypy3\</span></div>
</blockquote>
<br />
Or just cd C:\pypy to test it out.<br />
<br />
3) Ensure pip is installed (pip is a weirdly named tool for installing things).<br />
<blockquote class="tr_bq">
<span style="font-family: "courier new" , "courier" , monospace;">pypy.exe -m ensurepip</span></blockquote>
<span style="font-family: "courier new" , "courier" , monospace;">4</span>) Install a dev build of pygame.<br />
<blockquote class="tr_bq">
<div style="text-align: left;">
<span style="font-family: "courier new" , "courier" , monospace;">pypy.exe -m pip install</span><span style="font-family: "courier new" , "courier" , monospace;"><span style="font-family: "courier new" , "courier" , monospace;"> pygame --pre</span></span></div>
</blockquote>
<br />
5) Check if pygame is installed properly by running an included example.<br />
<blockquote class="tr_bq">
<span style="font-family: "courier new" , "courier" , monospace;">pypy.exe -m pygame.examples.aliens</span></blockquote>
<br />
ps. use pypy3.exe for python 3 version of pypy above rather than pypy.exe(python 2) if that's what you like.<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
</div>
Unknownnoreply@blogger.com2tag:blogger.com,1999:blog-10678074.post-43653071040698284732018-03-20T08:19:00.003+00:002018-03-20T09:00:43.097+00:00pygame on pypy usable<div dir="ltr" style="text-align: left;" trbidi="on">
<div style="text-align: left;">
<div>
<div>
Hi,<br />
<br />
TLDR; I'm at the <a href="https://pypy.org/">pypy</a> sprint, and working through the remaining pygame-on-pypy-cpyext issues.<br />
<a data-saferedirecturl="https://www.google.com/url?hl=en&q=https://youtu.be/WN1slc5O8os&source=gmail&ust=1521616456714000&usg=AFQjCNFtxsJIye4GKDXYuSksnTSnpqpPyA" href="https://youtu.be/WN1slc5O8os" target="_blank">https://youtu.be/WN1slc5O8os</a></div>
<div>
</div>
<div>
</div>
<div>
</div>
<div>
</div>
<div class="separator" style="clear: both; text-align: center;">
<iframe allowfullscreen="" class="YOUTUBE-iframe-video" data-thumbnail-src="https://i.ytimg.com/vi/WN1slc5O8os/0.jpg" frameborder="0" height="266" src="https://www.youtube.com/embed/WN1slc5O8os?feature=player_embedded" width="320"></iframe></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
Surprisingly
to me... it's already usable. That is pygame (same one that runs on
cpython), works on pypy through its C extension API. pypy has good
support for the CPython API (through a recompile) now. PyPy is the python language with a fast JIT, so your code can approach C speeds. And in some cases it can be faster than C.<br />
<br />
There was
an issue with events stopping keyboard/mouse/etc from working. Lots of
details in this issue describing the changes needed, so I hope other
extensions encountering this will find it useful.</div>
<a data-saferedirecturl="https://www.google.com/url?hl=en&q=https://github.com/pygame/pygame/issues/419&source=gmail&ust=1521616456714000&usg=AFQjCNGcVuLZVFdGBIbEnxlA1WGgg4Fl5Q" href="https://github.com/pygame/pygame/issues/419" target="_blank">https://github.com/pygame/<wbr></wbr>pygame/issues/419</a></div>
But now that's fixed, every pygame app I tried on it has worked.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjDuTmurjkqguW1YlBJeOSzTp-K9k1GayIy4oTURxqulAV4BKb_7EiDQLW2mOyZMLJ2K1dKtz0AD5bgT_sK3ba3kmlvChDzDgYnWXr-jKi6XVhnhnxU6jgJjGYp-wJRHLwlQ0hd/s1600/IMG_2314.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="640" data-original-width="640" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjDuTmurjkqguW1YlBJeOSzTp-K9k1GayIy4oTURxqulAV4BKb_7EiDQLW2mOyZMLJ2K1dKtz0AD5bgT_sK3ba3kmlvChDzDgYnWXr-jKi6XVhnhnxU6jgJjGYp-wJRHLwlQ0hd/s320/IMG_2314.jpg" width="320" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Cat sitting in the fog and snow of Switzerland</td></tr>
</tbody></table>
<h3>
<span style="font-size: medium;">Why is this exciting?</span></h3>
This is exciting to me because:</div>
<ul>
<li>pure python code being fast on pypy(after warmup), also mixed with the fast bits in C/asm.</li>
<li>cpyext is getting faster in pypy. There is already work and discussion towards it being faster than CPython.</li>
<li>maintaining one pygame code base is easier than maintaining several (pygame cffi/ctypes/cython, ...).</li>
<li>with one code base it should be fast on both pygame, and pypy(in time).</li>
</ul>
<div>
Here's our old pal solarwolf from early 2000s running on pypy.<br />
<a data-saferedirecturl="https://www.google.com/url?hl=en&q=https://youtu.be/WN1slc5O8os&source=gmail&ust=1521616456715000&usg=AFQjCNHXt2025mYow-MjJobcK91cQ_g2gQ" href="https://youtu.be/WN1slc5O8os" target="_blank">https://youtu.be/WN1slc5O8os</a></div>
<div>
Still
lots of work to do (especially around PixelArray buffers and such).
Then of course, there is the issue of binary wheels, so that `<b><span style="font-family: monospace , monospace;">pip install pygame</span></b>` works without needing to compile things from source.</div>
<div>
<br /></div>
<div>
<h3>
<span style="font-size: medium;">How is the speed? (when do we use this tool? Is it fast enough?)</span></h3>
If
your code is already quite well optimized, and not spending much time
in python, you can't expect to see an improvement. However, if you are
pushing boundaries in your python code, you can expect very good
increases.</div>
<div>
<br /></div>
<div>
Some examples where you can expect it to be faster:<br />
<ul>
<li>if profiling and a pygame function (like blit) isn't at the top of the slow bits.</li>
<li>collision detection (if you aren't using fancy algorithms).</li>
<li>a pure python ray caster.</li>
<li>writing a music synthesizer in python python.</li>
</ul>
</div>
<div>
Where it can be slower.<br />
<ul>
<li>if you are going into C code for a lot of small operations. Like when using lots of pygame.Rect in a tight loop. This is because (currently) the cost of going from PyPy code into and out of CPython API code (like pygame) is a bit slow.</li>
</ul>
</div>
For
me, I'm interested mostly in this for a physics art project which was
really slow, and also for a software music synth written in pure python.
Even more interesting is running pypy as a separate process for these
tasks, and run the gui process with CPython.<br />
<br />
<br />
<h3 style="text-align: left;">
Ray tracing 3D scenes. </h3>
Here I run a python and pygame <a href="https://geometrian.com/programming/projects/index.php?project=Ray-Tracer">python ray tracer</a> by Ian Mallett.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiZtby8fuO1rEcdrlKkLq9jwxI7QkU6KHmGI_yq92zzD7ES8cqy1bHgziTYgI995xf-TtFJqRCBdwlUAke_ktfb1DiDbM9WTtbAHi-IcM1c85DXahPHd8Fzn4jH6QKUuzeppezn/s1600/Screen+Shot+2018-03-20+at+08.54.10.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="688" data-original-width="1600" height="137" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiZtby8fuO1rEcdrlKkLq9jwxI7QkU6KHmGI_yq92zzD7ES8cqy1bHgziTYgI995xf-TtFJqRCBdwlUAke_ktfb1DiDbM9WTtbAHi-IcM1c85DXahPHd8Fzn4jH6QKUuzeppezn/s320/Screen+Shot+2018-03-20+at+08.54.10.png" width="320" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Ray Tracer to for realistic 3D lighting.</td></tr>
</tbody></table>
<br />
On PyPy - 18.6 seconds.<br />
On Python 2.7 - 9 minutes, 28.1 seconds<br />
<br />
That's <b>30x faster</b>. Making many more things possible in python - and at speed.<br />
<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh-pmTPoXbX1-WqctI5IhXNkv3D3A1OLtUyARGJehtKbZ8OkD7lDbf960I248Fb1idIWoAC3txbKhtsibld_JhCYq6az184dCC-DpNRxnf7AOulsd1umFn8esAGqm_Q8mOIu9Y3/s1600/IMG_2317.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="640" data-original-width="640" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh-pmTPoXbX1-WqctI5IhXNkv3D3A1OLtUyARGJehtKbZ8OkD7lDbf960I248Fb1idIWoAC3txbKhtsibld_JhCYq6az184dCC-DpNRxnf7AOulsd1umFn8esAGqm_Q8mOIu9Y3/s320/IMG_2317.jpg" width="320" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">The fog has lifted at the Leysin pypy sprint.</td></tr>
</tbody></table>
</div>
Unknownnoreply@blogger.com27tag:blogger.com,1999:blog-10678074.post-27019453336103865542018-03-15T19:48:00.000+00:002018-03-15T22:51:43.302+00:00Drawing sound (as a waveform in pygame).<div dir="ltr" style="text-align: left;" trbidi="on">
I recently gave an example of <a href="http://renesd.blogspot.com/2018/03/sound-generation-pygame-examples.html">pygame sound generation examples</a>, and a few people asked for more. So... here we go!<br />
<br />
There's an infinite number of ways to visualize sound. A classic way is to draw it as a <a href="https://en.wikipedia.org/wiki/Waveform">waveform</a>.<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg9AXkpa9S4pSFivVg7L00qdyZ_PDRLMHZErJIjflc_Am06gwahlbRxUhhdXVY3bGcW6Ck9W2wCWpAv3dC6xHH3HIiI9-sPmF6XYVINeoOzDyqNj7zxJK8urohUnUi7hfDjMNaU/s1600/Screen+Shot+2018-03-15+at+20.38.04.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="1234" data-original-width="1582" height="249" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg9AXkpa9S4pSFivVg7L00qdyZ_PDRLMHZErJIjflc_Am06gwahlbRxUhhdXVY3bGcW6Ck9W2wCWpAv3dC6xHH3HIiI9-sPmF6XYVINeoOzDyqNj7zxJK8urohUnUi7hfDjMNaU/s320/Screen+Shot+2018-03-15+at+20.38.04.png" width="320" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Sound samples drawn as a Waveform. Scaled into a 320x200 sized Surface.</td></tr>
</tbody></table>
<br />
A sound could be made up of 44100 samples per second. Where each sample is often a 16 bit number (or 8bit or a 32bit floating point).<br />
Python comes with a built in <a href="https://docs.python.org/3/library/array.html">array</a> for efficiently storing numbers. We can store samples in there, with integers between -32768 and 32768. This is a signed 16bit(2 byte) number. Two to the power of 16 is 65536, and if we divide that by two we see the minimum and maximum value that can hold. pow(2, 16) / 2 == 32768.<br />
<br />
Below is the annotated code, also with an example of an array of samples representing a square wave. <br />
We loop over the samples and draw a line from the previous sample to the current one. 0 to 1, 1 to 2, 2 to 3, ... N-1 to N.<br />
<br />
<br />
You can also find it in the <a href="https://www.pygame.org/wiki/CookBook">pygame CookBook</a> at <a href="https://www.pygame.org/wiki/DrawWaveform">https://www.pygame.org/wiki/DrawWaveform</a><br />
(maybe easier to read than on this blog)<br />
<br />
<br />
<pre style="line-height: 125%;"><span></span><span style="color: #408080; font-style: italic;"># python built in array type is useful for storing sound data.</span>
<span style="color: green; font-weight: bold;">import</span> <span style="color: blue; font-weight: bold;">array</span>
<span style="color: #408080; font-style: italic;"># a square wave tone, with sample values between -32766 and 32766.</span>
samples <span style="color: #666666;">=</span> array<span style="color: #666666;">.</span>array(<span style="color: #ba2121;">'h'</span>, [<span style="color: #666666;">32766</span>, <span style="color: #666666;">32766</span>, <span style="color: #666666;">32766</span>, <span style="color: #666666;">32766</span>, <span style="color: #666666;">32766</span>, <span style="color: #666666;">32766</span>,
<span style="color: #666666;">32766</span>, <span style="color: #666666;">32766</span>, <span style="color: #666666;">32766</span>, <span style="color: #666666;">32766</span>, <span style="color: #666666;">32766</span>, <span style="color: #666666;">32766</span>, <span style="color: #666666;">32766</span>, <span style="color: #666666;">32766</span>, <span style="color: #666666;">32766</span>,
<span style="color: #666666;">32766</span>, <span style="color: #666666;">32766</span>, <span style="color: #666666;">32766</span>, <span style="color: #666666;">32766</span>, <span style="color: #666666;">32766</span>, <span style="color: #666666;">32766</span>, <span style="color: #666666;">32766</span>, <span style="color: #666666;">32766</span>, <span style="color: #666666;">32766</span>,
<span style="color: #666666;">32766</span>, <span style="color: #666666;">32766</span>, <span style="color: #666666;">32766</span>, <span style="color: #666666;">32766</span>, <span style="color: #666666;">-32766</span>, <span style="color: #666666;">-32766</span>, <span style="color: #666666;">-32766</span>, <span style="color: #666666;">-32766</span>,
<span style="color: #666666;">-32766</span>, <span style="color: #666666;">-32766</span>, <span style="color: #666666;">-32766</span>, <span style="color: #666666;">-32766</span>, <span style="color: #666666;">-32766</span>, <span style="color: #666666;">-32766</span>, <span style="color: #666666;">-32766</span>, <span style="color: #666666;">-32766</span>,
<span style="color: #666666;">-32766</span>, <span style="color: #666666;">-32766</span>, <span style="color: #666666;">-32766</span>, <span style="color: #666666;">-32766</span>, <span style="color: #666666;">-32766</span>, <span style="color: #666666;">-32766</span>, <span style="color: #666666;">-32766</span>, <span style="color: #666666;">-32766</span>,
<span style="color: #666666;">-32766</span>, <span style="color: #666666;">-32766</span>, <span style="color: #666666;">-32766</span>, <span style="color: #666666;">-32766</span>, <span style="color: #666666;">-32766</span>, <span style="color: #666666;">-32766</span>, <span style="color: #666666;">-32766</span>])
<span style="color: green; font-weight: bold;">def</span> <span style="color: blue;">scale_samples_to_surf</span>(width, height, samples):
<span style="color: #ba2121; font-style: italic;">""" Returns a generator containing (x, y) to draw a waveform.</span>
<span style="color: #ba2121; font-style: italic;"> :param width: width of surface to scale points to.</span>
<span style="color: #ba2121; font-style: italic;"> :param height: height of surface to scale points to.</span>
<span style="color: #ba2121; font-style: italic;"> :param samples: an array of signed 1 byte or signed 2 byte.</span>
<span style="color: #ba2121; font-style: italic;"> """</span>
<span style="color: green; font-weight: bold;">assert</span> samples<span style="color: #666666;">.</span>typecode <span style="color: #aa22ff; font-weight: bold;">in</span> [<span style="color: #ba2121;">'h'</span>, <span style="color: #ba2121;">'b'</span>]
<span style="color: #408080; font-style: italic;"># precalculate a bunch of variables, so not done in the loop.</span>
len_samples <span style="color: #666666;">=</span> <span style="color: green;">len</span>(samples)
width_per_sample <span style="color: #666666;">=</span> width <span style="color: #666666;">/</span> len_samples
height_1 <span style="color: #666666;">=</span> height <span style="color: #666666;">-</span> <span style="color: #666666;">1</span>
<span style="color: green; font-weight: bold;">if</span> samples<span style="color: #666666;">.</span>typecode <span style="color: #666666;">==</span> <span style="color: #ba2121;">'h'</span>:
<span style="color: #408080; font-style: italic;"># for array typecode 'h', -32768 to 32768</span>
factor <span style="color: #666666;">=</span> <span style="color: #666666;">1.0</span> <span style="color: #666666;">/</span> <span style="color: #666666;">65532</span>
normalize_modifier <span style="color: #666666;">=</span> <span style="color: green;">int</span>(<span style="color: #666666;">65532</span> <span style="color: #666666;">/</span> <span style="color: #666666;">2</span>)
<span style="color: green; font-weight: bold;">elif</span> samples<span style="color: #666666;">.</span>typecode <span style="color: #666666;">==</span> <span style="color: #ba2121;">'b'</span>:
<span style="color: #408080; font-style: italic;"># for array typecode 'b', -127 to 127</span>
factor <span style="color: #666666;">=</span> <span style="color: #666666;">1.0</span> <span style="color: #666666;">/</span> <span style="color: #666666;">256</span>
normalize_modifier <span style="color: #666666;">=</span> <span style="color: green;">int</span>(<span style="color: #666666;">256</span> <span style="color: #666666;">/</span> <span style="color: #666666;">2</span>)
<span style="color: green; font-weight: bold;">return</span> ((
<span style="color: green;">int</span>((sample_number <span style="color: #666666;">+</span> <span style="color: #666666;">1</span>) <span style="color: #666666;">*</span> width_per_sample),
<span style="color: green;">int</span>(
(<span style="color: #666666;">1.0</span> <span style="color: #666666;">-</span>
(factor <span style="color: #666666;">*</span>
(samples[sample_number] <span style="color: #666666;">+</span> normalize_modifier)))
<span style="color: #666666;">*</span> (height_1)
))
<span style="color: green; font-weight: bold;">for</span> sample_number <span style="color: #aa22ff; font-weight: bold;">in</span> <span style="color: green;">range</span>(len_samples))
<span style="color: green; font-weight: bold;">def</span> <span style="color: blue;">draw_wave</span>(surf,
samples,
wave_color <span style="color: #666666;">=</span> (<span style="color: #666666;">0</span>, <span style="color: #666666;">0</span>, <span style="color: #666666;">0</span>),
background_color <span style="color: #666666;">=</span> pg<span style="color: #666666;">.</span>Color(<span style="color: #ba2121;">'white'</span>)):
<span style="color: #ba2121; font-style: italic;">"""Draw array of sound samples as waveform into the 'surf'.</span>
<span style="color: #ba2121; font-style: italic;"> :param surf: Surface we want to draw the wave form onto.</span>
<span style="color: #ba2121; font-style: italic;"> :param samples: an array of signed 1 byte or signed 2 byte.</span>
<span style="color: #ba2121; font-style: italic;"> :param wave_color: color to draw the wave form.</span>
<span style="color: #ba2121; font-style: italic;"> :param background_color: to fill the 'surf' with.</span>
<span style="color: #ba2121; font-style: italic;"> """</span>
<span style="color: green; font-weight: bold;">assert</span> samples<span style="color: #666666;">.</span>typecode <span style="color: #aa22ff; font-weight: bold;">in</span> [<span style="color: #ba2121;">'h'</span>, <span style="color: #ba2121;">'b'</span>]
<span style="color: green; font-weight: bold;">if</span> background_color <span style="color: #aa22ff; font-weight: bold;">is</span> <span style="color: #aa22ff; font-weight: bold;">not</span> <span style="color: green;">None</span>:
surf<span style="color: #666666;">.</span>fill(background_color)
width, height <span style="color: #666666;">=</span> surf<span style="color: #666666;">.</span>get_size()
points <span style="color: #666666;">=</span> <span style="color: green;">tuple</span>(scale_samples_to_surf(width, height, samples))
pg<span style="color: #666666;">.</span>draw<span style="color: #666666;">.</span>lines(surf, wave_color, <span style="color: green;">False</span>, points)
<span style="color: #408080; font-style: italic;"># Here we should how to draw it onto a screen.</span>
waveform <span style="color: #666666;">=</span> pg<span style="color: #666666;">.</span>Surface((<span style="color: #666666;">320</span>, <span style="color: #666666;">200</span>))<span style="color: #666666;">.</span>convert_alpha()
draw_wave(waveform, samples)
screen<span style="color: #666666;">.</span>fill((<span style="color: #666666;">255</span>, <span style="color: #666666;">255</span>, <span style="color: #666666;">255</span>))
screen<span style="color: #666666;">.</span>blit(waveform, (<span style="color: #666666;">160</span>, <span style="color: #666666;">100</span>))</pre>
<pre style="line-height: 125%;"> </pre>
<pre style="line-height: 125%;"> </pre>
</div>
Unknownnoreply@blogger.com1tag:blogger.com,1999:blog-10678074.post-75707569438856585392018-03-14T14:12:00.000+00:002018-03-14T14:25:45.610+00:00Sound generation pygame examples.<div dir="ltr" style="text-align: left;" trbidi="on">
<br />
Here's a few sound generation examples with pygame (and no numpy/scipy).<br />
<br />
If there's interest I'll expand this into a bigger example? Let me know.<br />
<br />
<br />
All the basics for making a music program (sampler/synth).<br />
<div style="text-align: left;">
<ul>
<li>some sample rate conversion, </li>
<li>bit rate conversion</li>
<li>tone generation using generators (square wave)</li>
<li>python arrays used as buffers for pygame.Sound (zero copy).</li>
</ul>
</div>
<br /><div>
<br />
<br />
<pre style="line-height: 125%;"><span style="color: #ba2121; font-style: italic;">""" Some examples for generating and converting sounds for pygame.</span>
<span style="color: #ba2121; font-style: italic;">Python 2.7, 3.6</span>
<span style="color: #ba2121; font-style: italic;">Shows:</span>
<span style="color: #ba2121; font-style: italic;"> - a simple 'square wave' generated</span>
<span style="color: #ba2121; font-style: italic;"> - resampling sample rates (eg, 8363 to 44100)</span>
<span style="color: #ba2121; font-style: italic;"> - using built in python array for making pygame.Sound samples.</span>
<span style="color: #ba2121; font-style: italic;"> - samples at different bit sizes</span>
<span style="color: #ba2121; font-style: italic;"> - converting from signed 8 to signed 16bit</span>
<span style="color: #ba2121; font-style: italic;"> - how initializing the mixer changes what samples Sound needs.</span>
<span style="color: #ba2121; font-style: italic;"> - Using the python stdlib audioop.ratecv for sample rate conversion.</span>
<span style="color: #ba2121; font-style: italic;">Square Wave</span>
<span style="color: #ba2121; font-style: italic;"> https://en.wikipedia.org/wiki/Square_wave</span>
<span style="color: #ba2121; font-style: italic;">MOD (file format)</span>
<span style="color: #ba2121; font-style: italic;"> https://en.wikipedia.org/wiki/MOD_(file_format)</span>
<span style="color: #ba2121; font-style: italic;">pygame.mixer.get_init</span>
<span style="color: #ba2121; font-style: italic;"> https://www.pygame.org/docs/ref/mixer.html#pygame.mixer.get_init</span>
<span style="color: #ba2121; font-style: italic;">pygame.Sound</span>
<span style="color: #ba2121; font-style: italic;"> https://www.pygame.org/docs/ref/mixer.html#pygame.mixer.Sound</span>
<span style="color: #ba2121; font-style: italic;">array (python stdlib)</span>
<span style="color: #ba2121; font-style: italic;"> https://docs.python.org/3/library/array.html</span>
<span style="color: #ba2121; font-style: italic;">wave (python stdlib)</span>
<span style="color: #ba2121; font-style: italic;"> https://docs.python.org/3/library/wave.html</span>
<span style="color: #ba2121; font-style: italic;">audioop.ratecv (python stdlib)</span>
<span style="color: #ba2121; font-style: italic;"> https://docs.python.org/3/library/audioop.html?highlight=audio#audioop.ratecv</span>
<span style="color: #ba2121; font-style: italic;">"""</span>
<span style="color: green; font-weight: bold;">from</span> <span style="color: blue; font-weight: bold;">array</span> <span style="color: green; font-weight: bold;">import</span> array
<span style="color: green; font-weight: bold;">import</span> <span style="color: blue; font-weight: bold;">pygame</span> <span style="color: green; font-weight: bold;">as</span> <span style="color: blue; font-weight: bold;">pg</span>
<span style="color: green; font-weight: bold;">class</span> <span style="color: blue; font-weight: bold;">Tone</span>(pg<span style="color: #666666;">.</span>mixer<span style="color: #666666;">.</span>Sound):
<span style="color: #ba2121; font-style: italic;">"""This generates a 'Square wave' with a generator.</span>
<span style="color: #ba2121; font-style: italic;"> Then creates an array of samples, and passes that into pygame.Sound.</span>
<span style="color: #ba2121; font-style: italic;"> """</span>
<span style="color: green; font-weight: bold;">def</span> <span style="color: blue;">__init__</span>(<span style="color: green;">self</span>, frequency, array_type, volume<span style="color: #666666;">=.1</span>):
<span style="color: green;">self</span><span style="color: #666666;">.</span>frequency <span style="color: #666666;">=</span> frequency
<span style="color: green; font-weight: bold;">if</span> array_type <span style="color: #666666;">==</span> <span style="color: #ba2121;">'b'</span>:
<span style="color: #408080; font-style: italic;"># we have to convert the 1 byte 'b' samples to 2 byte 'h'.</span>
samples <span style="color: #666666;">=</span> <span style="color: green;">self</span><span style="color: #666666;">.</span>signed_char_to_signed_short(
<span style="color: green;">self</span><span style="color: #666666;">.</span>make_samples_b()
)
<span style="color: green; font-weight: bold;">elif</span> array_type <span style="color: #666666;">==</span> <span style="color: #ba2121;">'h'</span>:
samples <span style="color: #666666;">=</span> <span style="color: green;">self</span><span style="color: #666666;">.</span>make_samples_h()
<span style="color: green; font-weight: bold;">else</span>:
<span style="color: green; font-weight: bold;">raise</span> <span style="color: #d2413a; font-weight: bold;">ValueError</span>(<span style="color: #ba2121;">'array_type not supported'</span>)
pg<span style="color: #666666;">.</span>mixer<span style="color: #666666;">.</span>Sound<span style="color: #666666;">.</span><span style="color: blue;">__init__</span>(<span style="color: green;">self</span>, <span style="color: green;">buffer</span><span style="color: #666666;">=</span>samples)
<span style="color: green;">self</span><span style="color: #666666;">.</span>set_volume(volume)
<span style="color: green; font-weight: bold;">def</span> <span style="color: blue;">make_samples_b</span>(<span style="color: green;">self</span>):
<span style="color: #ba2121; font-style: italic;">""" Builds samples array between -127 and 127.</span>
<span style="color: #ba2121; font-style: italic;"> Array type 'h'.</span>
<span style="color: #ba2121; font-style: italic;"> """</span>
mixer_frequency <span style="color: #666666;">=</span> pg<span style="color: #666666;">.</span>mixer<span style="color: #666666;">.</span>get_init()[<span style="color: #666666;">0</span>]
mixer_format <span style="color: #666666;">=</span> pg<span style="color: #666666;">.</span>mixer<span style="color: #666666;">.</span>get_init()[<span style="color: #666666;">1</span>]
period <span style="color: #666666;">=</span> <span style="color: green;">int</span>(<span style="color: green;">round</span>(mixer_frequency <span style="color: #666666;">/</span> <span style="color: green;">self</span><span style="color: #666666;">.</span>frequency))
max_amplitude <span style="color: #666666;">=</span> <span style="color: #666666;">2</span> <span style="color: #666666;">**</span> (<span style="color: green;">abs</span>(mixer_format) <span style="color: #666666;">-</span> <span style="color: #666666;">1</span>) <span style="color: #666666;">-</span> <span style="color: #666666;">1</span>
max_amplitude <span style="color: #666666;">=</span> <span style="color: green;">int</span>(max_amplitude <span style="color: #666666;">/</span> <span style="color: #666666;">256</span>)
<span style="color: #408080; font-style: italic;"># print(f'mixer_frequency:{mixer_frequency}, mixer_format:{mixer_format}')</span>
<span style="color: #408080; font-style: italic;"># print(f'period:{period}, max_amplitude:{max_amplitude}')</span>
<span style="color: #408080; font-style: italic;"># 'b' array is signed char, 1 byte</span>
<span style="color: #408080; font-style: italic;"># https://docs.python.org/3/library/array.html</span>
samples <span style="color: #666666;">=</span> array(<span style="color: #ba2121;">'b'</span>,
(max_amplitude <span style="color: green; font-weight: bold;">if</span> time <span style="color: #666666;"><</span> period <span style="color: #666666;">/</span> <span style="color: #666666;">2</span> <span style="color: green; font-weight: bold;">else</span> <span style="color: #666666;">-</span>max_amplitude
<span style="color: green; font-weight: bold;">for</span> time <span style="color: #aa22ff; font-weight: bold;">in</span> <span style="color: green;">range</span>(period))
)
<span style="color: green; font-weight: bold;">return</span> samples
<span style="color: green; font-weight: bold;">def</span> <span style="color: blue;">signed_char_to_signed_short</span>(<span style="color: green;">self</span>, b_samples):
<span style="color: #ba2121; font-style: italic;">""" Converts 1 byte signed char samples to 2 byte signed short.</span>
<span style="color: #ba2121; font-style: italic;"> 127 to 32767</span>
<span style="color: #ba2121; font-style: italic;"> """</span>
<span style="color: #408080; font-style: italic;"># just a simple linear conversion.</span>
factor <span style="color: #666666;">=</span> <span style="color: green;">int</span>(<span style="color: #666666;">32767</span> <span style="color: #666666;">/</span> <span style="color: #666666;">127</span>)
<span style="color: green; font-weight: bold;">return</span> array(<span style="color: #ba2121;">'h'</span>, (sample <span style="color: #666666;">*</span> factor <span style="color: green; font-weight: bold;">for</span> sample <span style="color: #aa22ff; font-weight: bold;">in</span> b_samples))
<span style="color: green; font-weight: bold;">def</span> <span style="color: blue;">make_samples_h</span>(<span style="color: green;">self</span>):
<span style="color: #ba2121; font-style: italic;">""" Builds samples array between -32767 snd 32767.</span>
<span style="color: #ba2121; font-style: italic;"> Array type 'h'.</span>
<span style="color: #ba2121; font-style: italic;"> """</span>
mixer_frequency <span style="color: #666666;">=</span> pg<span style="color: #666666;">.</span>mixer<span style="color: #666666;">.</span>get_init()[<span style="color: #666666;">0</span>]
mixer_format <span style="color: #666666;">=</span> pg<span style="color: #666666;">.</span>mixer<span style="color: #666666;">.</span>get_init()[<span style="color: #666666;">1</span>]
period <span style="color: #666666;">=</span> <span style="color: green;">int</span>(<span style="color: green;">round</span>(mixer_frequency <span style="color: #666666;">/</span> <span style="color: green;">self</span><span style="color: #666666;">.</span>frequency))
max_amplitude <span style="color: #666666;">=</span> <span style="color: #666666;">2</span> <span style="color: #666666;">**</span> (<span style="color: green;">abs</span>(mixer_format) <span style="color: #666666;">-</span> <span style="color: #666666;">1</span>) <span style="color: #666666;">-</span> <span style="color: #666666;">1</span>
<span style="color: #408080; font-style: italic;"># print(f'mixer_frequency:{mixer_frequency}, mixer_format:{mixer_format}')</span>
<span style="color: #408080; font-style: italic;"># print(f'period:{period}, max_amplitude:{max_amplitude}')</span>
<span style="color: #408080; font-style: italic;"># 'h' array is signed short, 2 bytes</span>
<span style="color: #408080; font-style: italic;"># https://docs.python.org/3/library/array.html</span>
samples <span style="color: #666666;">=</span> array(<span style="color: #ba2121;">'h'</span>,
(max_amplitude <span style="color: green; font-weight: bold;">if</span> time <span style="color: #666666;"><</span> period <span style="color: #666666;">/</span> <span style="color: #666666;">2</span> <span style="color: green; font-weight: bold;">else</span> <span style="color: #666666;">-</span>max_amplitude
<span style="color: green; font-weight: bold;">for</span> time <span style="color: #aa22ff; font-weight: bold;">in</span> <span style="color: green;">range</span>(period))
)
<span style="color: green; font-weight: bold;">return</span> samples
<span style="color: green; font-weight: bold;">class</span> <span style="color: blue; font-weight: bold;">Sample</span>(pg<span style="color: #666666;">.</span>mixer<span style="color: #666666;">.</span>Sound):
<span style="color: #ba2121; font-style: italic;">""" For playing a sample.</span>
<span style="color: #ba2121; font-style: italic;"> Takes a file, and reads it in as 8bit signed data.</span>
<span style="color: #ba2121; font-style: italic;"> Then converts it to the 16bit signed size the pygame.mixer needs.</span>
<span style="color: #ba2121; font-style: italic;"> """</span>
<span style="color: green; font-weight: bold;">def</span> <span style="color: blue;">__init__</span>(<span style="color: green;">self</span>, fname, volume<span style="color: #666666;">=.1</span>):
<span style="color: green; font-weight: bold;">with</span> <span style="color: green;">open</span>(fname, <span style="color: #ba2121;">'rb'</span>) <span style="color: green; font-weight: bold;">as</span> f:
samples <span style="color: #666666;">=</span> <span style="color: green;">self</span><span style="color: #666666;">.</span>signed_char_to_signed_short (
array(<span style="color: #ba2121;">'b'</span>, f<span style="color: #666666;">.</span>read())
)
pg<span style="color: #666666;">.</span>mixer<span style="color: #666666;">.</span>Sound<span style="color: #666666;">.</span><span style="color: blue;">__init__</span>(<span style="color: green;">self</span>, <span style="color: green;">buffer</span><span style="color: #666666;">=</span>samples)
<span style="color: green;">self</span><span style="color: #666666;">.</span>set_volume(volume)
<span style="color: green; font-weight: bold;">def</span> <span style="color: blue;">signed_char_to_signed_short</span>(<span style="color: green;">self</span>, b_samples):
<span style="color: #ba2121; font-style: italic;">""" Converts 1 byte signed char samples to 2 byte signed short.</span>
<span style="color: #ba2121; font-style: italic;"> 127 to 32767</span>
<span style="color: #ba2121; font-style: italic;"> """</span>
<span style="color: #408080; font-style: italic;"># just a simple linear conversion.</span>
<span style="color: green; font-weight: bold;">import</span> <span style="color: blue; font-weight: bold;">time</span>
t0<span style="color: #666666;">=</span>time<span style="color: #666666;">.</span>time()
factor <span style="color: #666666;">=</span> <span style="color: green;">int</span>(<span style="color: #666666;">32767</span> <span style="color: #666666;">/</span> <span style="color: #666666;">127</span>)
samples <span style="color: #666666;">=</span> array(<span style="color: #ba2121;">'h'</span>, (
<span style="color: green;">max</span>(sample, <span style="color: #666666;">-127</span>) <span style="color: #666666;">*</span> factor <span style="color: green; font-weight: bold;">if</span> sample <span style="color: #666666;"><</span> <span style="color: #666666;">0</span> <span style="color: green; font-weight: bold;">else</span>
<span style="color: green;">min</span>(sample, <span style="color: #666666;">127</span>) <span style="color: #666666;">*</span> factor
<span style="color: green; font-weight: bold;">for</span> sample <span style="color: #aa22ff; font-weight: bold;">in</span> b_samples))
t1<span style="color: #666666;">=</span>time<span style="color: #666666;">.</span>time()
<span style="color: green; font-weight: bold;">print</span>(t1<span style="color: #666666;">-</span>t0)
<span style="color: green; font-weight: bold;">return</span> samples
<span style="color: green; font-weight: bold;">def</span> <span style="color: blue;">fetch_example_mod_file</span>(mod_fname):
<span style="color: #ba2121; font-style: italic;">""" Grab a file that has a sound samples in it from the net.</span>
<span style="color: #ba2121; font-style: italic;"> 'MOD is a computer file format used primarily to represent music,</span>
<span style="color: #ba2121; font-style: italic;"> and was the first module file format. MOD files use the ".MOD"</span>
<span style="color: #ba2121; font-style: italic;"> file extension, except on the Amiga which doesn't rely on</span>
<span style="color: #ba2121; font-style: italic;"> filename extensions, instead it reads a file's header to</span>
<span style="color: #ba2121; font-style: italic;"> determine filetype. A MOD file contains a set of instruments in</span>
<span style="color: #ba2121; font-style: italic;"> the form of samples, a number of patterns indicating how and when</span>
<span style="color: #ba2121; font-style: italic;"> the samples are to be played, and a list of what</span>
<span style="color: #ba2121; font-style: italic;"> patterns to play in what order.'</span>
<span style="color: #ba2121; font-style: italic;"> https://en.wikipedia.org/wiki/MOD_(file_format)</span>
<span style="color: #ba2121; font-style: italic;"> """</span>
<span style="color: green; font-weight: bold;">import</span> <span style="color: blue; font-weight: bold;">os</span>
url <span style="color: #666666;">=</span> <span style="color: #ba2121;">'https://api.modarchive.org/downloads.php?moduleid=101996'</span>
<span style="color: green; font-weight: bold;">if</span> <span style="color: #aa22ff; font-weight: bold;">not</span> os<span style="color: #666666;">.</span>path<span style="color: #666666;">.</span>exists(mod_fname):
<span style="color: green; font-weight: bold;">import</span> <span style="color: blue; font-weight: bold;">urllib2</span>
<span style="color: green; font-weight: bold;">print</span> (<span style="color: #ba2121;">'Fetching </span><span style="color: #bb6688; font-weight: bold;">%s</span><span style="color: #ba2121;"> .mod into file: </span><span style="color: #bb6688; font-weight: bold;">%s</span><span style="color: #ba2121;">'</span> <span style="color: #666666;">%</span> (url, mod_fname))
data <span style="color: #666666;">=</span> urllib2<span style="color: #666666;">.</span>urlopen(url)<span style="color: #666666;">.</span>read()
<span style="color: green; font-weight: bold;">with</span> <span style="color: green;">open</span>(mod_fname, <span style="color: #ba2121;">'w'</span>) <span style="color: green; font-weight: bold;">as</span> modf:
modf<span style="color: #666666;">.</span>write(data)
<span style="color: green; font-weight: bold;">def</span> <span style="color: blue;">resample</span>(mod_fname):
<span style="color: #ba2121; font-style: italic;">""" An example of resampling audio to a different framerate.</span>
<span style="color: #ba2121; font-style: italic;"> eg, from 8363 one byte samples per second to</span>
<span style="color: #ba2121; font-style: italic;"> 44100 two byte samples per second.</span>
<span style="color: #ba2121; font-style: italic;"> """</span>
<span style="color: green; font-weight: bold;">import</span> <span style="color: blue; font-weight: bold;">audioop</span>
<span style="color: green; font-weight: bold;">import</span> <span style="color: blue; font-weight: bold;">wave</span>
<span style="color: green; font-weight: bold;">from</span> <span style="color: blue; font-weight: bold;">io</span> <span style="color: green; font-weight: bold;">import</span> BytesIO
in_framerate <span style="color: #666666;">=</span> <span style="color: #666666;">8363</span>
in_sampwidth <span style="color: #666666;">=</span> <span style="color: #666666;">1</span>
in_nchannels <span style="color: #666666;">=</span> <span style="color: #666666;">1</span>
out_framerate <span style="color: #666666;">=</span> <span style="color: #666666;">44100</span>
num_seconds <span style="color: #666666;">=</span> <span style="color: #666666;">5</span>
<span style="color: green; font-weight: bold;">with</span> <span style="color: green;">open</span>(mod_fname, <span style="color: #ba2121;">'rb'</span>) <span style="color: green; font-weight: bold;">as</span> f:
<span style="color: #408080; font-style: italic;"># Throw away the start data of this mod file.</span>
<span style="color: #408080; font-style: italic;"># Better samples later on.</span>
f<span style="color: #666666;">.</span>read(<span style="color: #666666;">8363*2</span>)
in_frame_data <span style="color: #666666;">=</span> f<span style="color: #666666;">.</span>read(in_framerate <span style="color: #666666;">*</span> num_seconds)
<span style="color: #408080; font-style: italic;"># https://docs.python.org/3/library/audioop.html?highlight=audio#audioop.ratecv</span>
newfragment, newstate <span style="color: #666666;">=</span> audioop<span style="color: #666666;">.</span>ratecv(
in_frame_data,
in_sampwidth,
in_nchannels,
in_framerate,
out_framerate,
<span style="color: green;">None</span>)
<span style="color: #408080; font-style: italic;"># print(f'len(newfragment):{len(newfragment)}')</span>
<span style="color: #408080; font-style: italic;"># A perfect conversion is not possible, because the sample</span>
<span style="color: #408080; font-style: italic;"># rates do not divide equally. However, the number</span>
<span style="color: #408080; font-style: italic;"># of samples should be close.</span>
<span style="color: green; font-weight: bold;">assert</span> (out_framerate <span style="color: #666666;">*</span> num_seconds) <span style="color: #666666;">-</span> <span style="color: green;">len</span>(newfragment) <span style="color: #666666;"><</span> <span style="color: #666666;">10</span>
pg<span style="color: #666666;">.</span>mixer<span style="color: #666666;">.</span>Sound(<span style="color: green;">buffer</span><span style="color: #666666;">=</span>newfragment)<span style="color: #666666;">.</span>play(<span style="color: #666666;">-1</span>)
<span style="color: #408080; font-style: italic;"># TODO:</span>
<span style="color: #408080; font-style: italic;"># Converting between modo and stereo?</span>
<span style="color: #408080; font-style: italic;"># audioop.tomono and audioop.tostereo</span>
<span style="color: #408080; font-style: italic;"># https://docs.python.org/3/library/audioop.html?highlight=audio#audioop.tomono</span>
<span style="color: #408080; font-style: italic;"># How to draw a wave form?</span>
<span style="color: #408080; font-style: italic;"># using pygame.draw.lines transforming audio into</span>
<span style="color: #408080; font-style: italic;"># Surface space.</span>
<span style="color: #408080; font-style: italic;"># Meaning, scaling audio samples into a particular</span>
<span style="color: #408080; font-style: italic;"># sized part of the screen.</span>
<span style="color: #408080; font-style: italic;"># More sound generator types.</span>
<span style="color: #408080; font-style: italic;"># Saw tooth.</span>
<span style="color: green; font-weight: bold;">if</span> <span style="color: #19177c;">__name__</span> <span style="color: #666666;">==</span> <span style="color: #ba2121;">"__main__"</span>:
<span style="color: #408080; font-style: italic;"># https://www.pygame.org/docs/ref/mixer.html#pygame.mixer.init</span>
pg<span style="color: #666666;">.</span>mixer<span style="color: #666666;">.</span>pre_init(<span style="color: #666666;">44100</span>, <span style="color: #666666;">-16</span>, <span style="color: #666666;">1</span>, <span style="color: #666666;">1024</span>)
pg<span style="color: #666666;">.</span>init()
pg<span style="color: #666666;">.</span>display<span style="color: #666666;">.</span>set_caption(<span style="color: #ba2121;">'Playing square wave, 808 frequency'</span>)
pg<span style="color: #666666;">.</span>display<span style="color: #666666;">.</span>set_mode((<span style="color: #666666;">320</span>, <span style="color: #666666;">200</span>))
mod_fname <span style="color: #666666;">=</span> <span style="color: #ba2121;">'outrun_3.mod'</span>
fetch_example_mod_file(mod_fname)
<span style="color: #408080; font-style: italic;"># play on repeat, -1 means loop indefinitely.</span>
<span style="color: #408080; font-style: italic;"># https://www.pygame.org/docs/ref/mixer.html#pygame.mixer.Sound.play</span>
<span style="color: green; font-weight: bold;">if</span> <span style="color: #666666;">0</span>:
Tone(frequency<span style="color: #666666;">=808</span>, array_type<span style="color: #666666;">=</span><span style="color: #ba2121;">'b'</span>)<span style="color: #666666;">.</span>play(<span style="color: #666666;">-1</span>)
<span style="color: green; font-weight: bold;">if</span> <span style="color: #666666;">0</span>:
<span style="color: green; font-weight: bold;">try</span>:
Sample(mod_fname)<span style="color: #666666;">.</span>play(<span style="color: #666666;">-1</span>)
<span style="color: green; font-weight: bold;">except</span> <span style="color: #d2413a; font-weight: bold;">IOError</span>:
<span style="color: green; font-weight: bold;">print</span> (<span style="color: #ba2121;">'no </span><span style="color: #bb6688; font-weight: bold;">%s</span><span style="color: #ba2121;">'</span> <span style="color: #666666;">%</span> mod_fname)
<span style="color: green; font-weight: bold;">if</span> <span style="color: #666666;">0</span>:
pg<span style="color: #666666;">.</span>mixer<span style="color: #666666;">.</span>music<span style="color: #666666;">.</span>load(mod_fname)
pg<span style="color: #666666;">.</span>mixer<span style="color: #666666;">.</span>music<span style="color: #666666;">.</span>play()
<span style="color: green; font-weight: bold;">if</span> <span style="color: #666666;">1</span>:
resample(mod_fname)
going <span style="color: #666666;">=</span> <span style="color: green;">True</span>
<span style="color: green; font-weight: bold;">while</span> going:
<span style="color: green; font-weight: bold;">for</span> e <span style="color: #aa22ff; font-weight: bold;">in</span> pg<span style="color: #666666;">.</span>event<span style="color: #666666;">.</span>get():
<span style="color: green; font-weight: bold;">if</span> e<span style="color: #666666;">.</span>type <span style="color: #aa22ff; font-weight: bold;">in</span> [pg<span style="color: #666666;">.</span>QUIT, pg<span style="color: #666666;">.</span>KEYDOWN]:
going <span style="color: #666666;">=</span> <span style="color: green;">False</span></pre>
</div>
</div>
Unknownnoreply@blogger.com2tag:blogger.com,1999:blog-10678074.post-56778890329875982412018-03-13T11:09:00.000+00:002018-03-13T11:09:49.345+00:00pypy sprint<div dir="ltr" style="text-align: left;" trbidi="on">
Been doing a bit of reading this morning on pypy things.<br />
<br />
Trying to prepare myself a bit for the coming '<a href="https://morepypy.blogspot.de/2018/01/leysin-winter-sprint-17-24-march-2018.html">winter sprint</a>'. One topic I'm interested in for the sprint is finishing off remaining issues for pygame running on pypy. Probably not the most common reason for people to go to the Swiss mountains I guess! So this means learning about the cpyext part of pypy... the one which allows it to use the python C API, and modules that use it.<br />
<br />
Open source sprints are great. A bunch of people get together trying to make things. A very rare time when people get to meet other people-from-the-internet. It's a great place to learn, and to see how projects you're interested in are made.<br />
<br />
I guess I'm a little bit too excited, and started playing with pypy early. Below are some notes from my investigations finding out how to do things.<br />
<br />
<h3 style="text-align: left;">
Getting started, finding docs.</h3>
There's a <a href="http://doc.pypy.org/en/latest/getting-started-dev.html">getting started guide</a> for pypy which has some basic notes on running tests, and compiling stuff.<br />
<br />
Also, cython has a pypy section in their user guide - <a href="http://cython.readthedocs.io/en/latest/src/userguide/pypy.html">http://cython.readthedocs.io/en/latest/src/userguide/pypy.html</a><br />
<br />
cpyext is the module which implements the python C API in pypy.<br />
<br />
A wiki page called '<a href="https://bitbucket.org/pypy/compatibility/wiki/c-api">c-api</a>' contains some documentation including a talk on cpyext. It's a little bit dated (from 7 years ago), but still interesting reading(and watching).<br />
<br />
This wiki page was interesting: "In which we document cpyext compatibility progress, this time with cython and pandas."<br />
<a href="https://bitbucket.org/pypy/pypy/wiki/cpyext_2_-_cython%20and%20pandas">https://bitbucket.org/pypy/pypy/wiki/cpyext_2_-_cython%20and%20pandas</a><br />
<br />
It's a blog of development of pypy cpyext getting numpy and pandas features working. From this I noticed a few interesting techniques... like running tests separated processes. Very useful when debugging. There is also an older document called '<a href="https://bitbucket.org/pypy/pypy/wiki/Adventures%20in%20cpyext%20compatibility%20(ended)">Adventures in cpyext compatibility (ended)</a>'. With even more notes on issues that came up, along with their work arounds.<br />
<br />
Of course the <a href="https://docs.python.org/3/c-api/intro.html">Python C API</a> docs are also useful when one wants to do Python C API things.<br /><br />
<h3 style="text-align: left;">
Use the source (Luke).</h3>
But the best source of information on cpyext seems to be the source code itself.<br />
<a href="https://bitbucket.org/pypy/pypy/src/default/pypy/module/cpyext/">https://bitbucket.org/pypy/pypy/src/default/pypy/module/cpyext/</a><br />
<br />
Lots and lots and lots of tests. And lots and lots of code. Bed time reading perhaps.<br />
<br />
It
seems to be a whole implementation of CPython, with bridges into pypy.
But with tests. Lots and lots of tests for the python C API.<br />
<br />
<h3 style="text-align: left;">
What are useful debugging techniques?</h3>
CPython has a thing called a debug build which is useful.<br />
<a href="https://docs.python.org/3/c-api/intro.html#debugging-builds">https://docs.python.org/3/c-api/intro.html#debugging-builds</a><br />
<br />
I don't know of any special debug tools for pypy (gdb extensions or such like).<br />
<br />
<br />
<h3 style="text-align: left;">
Wheels on pypy?</h3>
I found this page "<a href="https://github.com/antocuni/pypy-wheels">pypy binary wheels for some popular packages</a>", and a <a href="https://morepypy.blogspot.de/2017/07/binary-wheels-for-pypy.html">blog post about it</a>.<br /><br />
It seems pypy doesn't compile on the ancient centos 5 that many linux uses. Perhaps the new manylinux which uses a newer linux will work.<br />
<br />
It seems the <a href="https://www.piwheels.hostedpi.com/">piwheels</a> project (compiles wheels for raspberry pi automatically) doesn't have pypy support yet. piwheels is really great, in that it makes lots of ARM builds available so people don't need to compile stuff themselves on the raspberrypi.<br />
<br />
No idea if windows, or mac binary wheels work for pypy. Without binary wheels for these platforms, most pypy users won't be able to install a C based library like pygame on pypy.<br />
<br />
<h3 style="text-align: left;">
Which platform?</h3>
It seems linux amd64 is the best supported, but I'm also interested in Mac, and Windows builds.<br />
Especially setting them up on Appveyor, and Travis /Mac.<br />
<br />
<a href="http://doc.pypy.org/en/latest/windows.html">Translating on windows</a> is the document which describes the windows builds. Not sure my cheap windows laptop has enough memory for it. It's only got 4GB of ram, and I expect it's going to take hours to compile.<br />
<br />
The buildbot contains <a href="http://buildbot.pypy.org/nightly/trunk/">Nightly builds</a> for windows as well as other platforms. So I might try using them to debug stuff, and I guess do extension development. Failing that, maybe I can copy a build off someones computer!<br />
<br />
Hopefully I won't need to hire some remote servers for compiling stuff. It doesn't seem like I'd need to be recompiling all of pypy each time for extension development.<br /><br /></div>
Unknownnoreply@blogger.com2tag:blogger.com,1999:blog-10678074.post-33073438225086216402018-03-08T12:42:00.000+00:002018-03-08T12:42:07.983+00:00pygameweb - 0.0.4 - Northern Rockhopper Penguin<div dir="ltr" style="text-align: left;" trbidi="on">
<a href="https://github.com/pygame/pygameweb">pygameweb</a> is the source code for the <a href="https://www.pygame.org/" rel="nofollow">pygame website</a> at <a href="https://www.pygame.org/" rel="nofollow">https://www.pygame.org/</a><br />
<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiUQDS89U80WHc34IHzgRcGaTV1RbGlggdqz-qkQcL6mL2qm0rLxKrn5F1nJG6Qvj2UVvhsF4M24oceU_Ol4xP-6LGOqoffpgbeZiAnInl7eoAJKbaXPuzRnv7hUPJjDtWhMsmN/s1600/6954251406_ed27093c48_o.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="1068" data-original-width="1600" height="213" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiUQDS89U80WHc34IHzgRcGaTV1RbGlggdqz-qkQcL6mL2qm0rLxKrn5F1nJG6Qvj2UVvhsF4M24oceU_Ol4xP-6LGOqoffpgbeZiAnInl7eoAJKbaXPuzRnv7hUPJjDtWhMsmN/s320/6954251406_ed27093c48_o.jpg" width="320" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Northern Rockhopper Penguin <a href="https://www.flickr.com/photos/briangratwicke/6954251406/in/photolist-bAwmEj-eihPBP-eihPEB-eioy7s-eihFEB-eijTDh-eie9kx-eie9s8-eikaeG-eihQqF-eioq9E-eioxRC-eihFxp-eioxUQ-eiknML-N7RLE1-eiopLy-TLNSDY-eihv1F-dhWw6P-dhWvpC-eijTeW-eihQut-eioyRw-eie92H-bPqVyr-dhWwY6-eioywS-eikohy-eihPT4-eihPJc-bPr1vp-eU7Eoh-eieDeT-eihPNF-eihuSx-eijtGC-eieD5n-eie9K6-eiopoA-eiopPd-eie9V2-eieCst-eieD2M-eioecy-eiopjY-eie9Bk-eieCTv-eihFrD-eidJL4" rel="nofollow">(photo by Brian Gratwicke)</a></td></tr>
</tbody></table>
<ul>
<li><a class="issue-link js-issue-link" data-error-text="Failed to load issue title" data-id="303424977" data-permission-text="Issue title is private" data-url="https://github.com/pygame/pygameweb/issues/52" href="https://github.com/pygame/pygameweb/pull/52">#52</a> Adding ability to deploy through travisci. And docs.</li>
<li><a class="issue-link js-issue-link tooltipped tooltipped-ne" data-error-text="Failed to load issue title" data-id="301544725" data-permission-text="Issue title is private" href="https://github.com/pygame/pygameweb/pull/46">#46</a> Wiki readability. Wiki tools on left out of reading line.</li>
<li>Twitter card for releases. For nicer view when posting to twitter.</li>
<li>Wiki table of contents improvements.</li>
<li>builds docs for pygame, when something lands on pygame github master branch.</li>
<li>project release feeds bugfix.</li>
<li>Only show one recent release for each project. Some projects release a few times in a row.</li>
<li>Wiki, nicer pre code handling.</li>
<li>Wiki code uses inline colors, so copy/paste works with colors too.</li>
<li>use https for login and register buttons, even when on http (see <a href="http://renesd.blogspot.de/2018/02/why-cant-we-turn-off-http.html" rel="nofollow">Why can't we turn off HTTPS?</a>).</li>
<li>Ask more bad robot web crawlers not to be bad.</li>
</ul>
See a full diff of <a href="https://github.com/pygame/pygameweb/compare/0.0.1...0.0.4">changes since last release</a>.</div>
Unknownnoreply@blogger.com1tag:blogger.com,1999:blog-10678074.post-16694884845107242632018-03-06T21:47:00.000+00:002018-03-06T21:54:27.090+00:00On the threshold of a journey with a Jedi, a Sphinx, and a Homebrew.<div dir="ltr" style="text-align: left;" trbidi="on">
<div>
<i>Feels like practice. </i></div>
<div>
</div>
<div>
It all started some days ago, late at night,<br />
with a comment made by someone on the internet...</div>
<div>
<b>"Threshold does not take keyword arguments" </b></div>
<div>
</div>
<div>
As is often the case when you want to correct<br />
something on the internet,<br />
this comment lead me on a <b><i>journey</i></b></div>
<div>
(not one where I sell numpy for 22 million monies,</div>
<div>
a different type of <b><i>journey</i></b><br />
involving text editors and compilers,<br />
and lots of failing tests).<br />
Software archeology -- not with Indiana,<br />
but at least there was a Jedi and a Sphinx.</div>
<div>
<br />
<blockquote class="tr_bq">
"Relax. Don't worry. And have a homebrew." -- <a href="https://en.wikipedia.org/wiki/Charlie_Papazian">Charlie Papazian</a></blockquote>
<br />
And with a warm brew in hand,<br />
herein begins my tale...<br />
<br />
Starting at the end, where all poor stories start,<br />
with the comment I will send "TheBob427" (actual real nick),<br />
right after I publish this blog post... [<a href="https://www.reddit.com/r/pygame/comments/80zyqg/threshold_does_not_take_keyword_arguments/dvae53d/">-1</a>]</div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<hr />
<br />
<blockquote class="tr_bq">
<i>Dear TheBob427,</i><br />
<div>
<i><br /></i></div>
<div>
<i>yes, this function doesn't take keyword arguments and you are right</i></div>
<div>
<i>that it's super buggy. However, now after some days of bashing my keyboard,</i></div>
<div>
<i>it should actually work. Additionally it should be slightly less weird. </i></div>
<div>
<i>It was a fun journey. I guess you'll have to wait for the next release to use it.</i></div>
<div>
<i>Which should be here 'soon'. Which in open source minutes is </i></div>
<i>approximately the same amount of time as 'when it's done'.</i><br />
<div>
<i><br /></i></div>
<div>
<i>yours sincerely,</i></div>
<div>
<i>illume</i></div>
<div>
<i><br /></i></div>
</blockquote>
</div>
<blockquote>
<div>
<i>ps. you could also use PixelArray.replace instead.</i></div>
</blockquote>
<br />
<hr />
<div>
<br /></div>
<div>
</div>
<div>
So, I updated the pygame.transform.threshold function, tests and docs [<a data-saferedirecturl="https://www.google.com/url?hl=en&q=https://github.com/pygame/pygame/pull/408&source=gmail&ust=1520457012449000&usg=AFQjCNHttXX1GiA_k6JjL8_RELJbIZsIzQ" href="https://github.com/pygame/pygame/pull/408" target="_blank">0</a>].</div>
<div>
Apart from that function being buggy and super confusing...</div>
<div>
<br />
It was sort of a test of how we could improve all the pygame functions for pygame 3000.</div>
<div>
Whilst it still doesn't follow the new pygame C API yet[<a data-saferedirecturl="https://www.google.com/url?hl=en&q=https://github.com/pygame/pygame/issues/333&source=gmail&ust=1520457012450000&usg=AFQjCNHADYH55S6d_vRA61rC6qTDa34zNw" href="https://github.com/pygame/pygame/issues/333" target="_blank">1</a>],<br />
I wanted to try a few things like 'new' C99/C11 features <br />
(thanks ffmpeg for tricking MS into dragging their compiler into the 90s),<br />
new docs features (param types, and examples) and such like.</div>
<div>
<br />
It's quite a bit of C and python test changes,<br />
and took me quite a few days longer than I expected.<br />
Simplifying here. Simplifying there. Simplifying everywhere.<br />
<br />
One thing it does now is use <a href="http://www.sphinx-doc.org/en/master/">sphinx</a> type parameters.<br />
Apart from the docs being improved, tools like <br />
<a href="https://jedi.readthedocs.io/en/latest/">jedi</a> can use them for autocomplete and type inference.<br />
<br />
So, now editors like vim, sublime, and vscode can autocomplete with pygame.<br />
<div>
(I had to change a few things for the 'odd' but 'cool' pygame<br />
'missing module' imports feature so the type inference worked in jedi [<a data-saferedirecturl="https://www.google.com/url?hl=en&q=https://github.com/pygame/pygame/pull/413&source=gmail&ust=1520457012450000&usg=AFQjCNHpggX740aCa4isjE7BzLl5ZtHe0w" href="https://github.com/pygame/pygame/pull/413" target="_blank">2</a>])</div>
<div>
<br /></div>
</div>
<div>
Fancy.</div>
<div>
<br /></div>
<div>
For examples, sphinx can now easily include source code.</div>
<div>
So we can include pygame.examples code, or even test cases.</div>
<div>
The 'nulledge' python search code search doesn't seem all that good<br />
for many pygame things, and it lacks https. So i'm thinking of dropping<br />
that and manually linking to relevant examples/test code.</div>
<div>
<br />
Additionally, we can link to github source code, and even<br />
the documentation files themselves. Which is handy for<br />
'edit this'/'fork on github' style buttons.</div>
<div>
<br /></div>
<br />
<i>Feels like practice.</i><br />
<br />
<br />
<br />
[-1] a comment, <a href="https://www.reddit.com/r/pygame/comments/80zyqg/threshold_does_not_take_keyword_arguments/dvae53d/">https://www.reddit.com/r/pygame/comments/80zyqg/threshold_does_not_take_keyword_arguments/dvae53d/</a><br />
[0] threshold function pr, <a data-saferedirecturl="https://www.google.com/url?hl=en&q=https://github.com/pygame/pygame/pull/408&source=gmail&ust=1520457012450000&usg=AFQjCNEWoz5jFCvtvkWYAIjeUKI6F0SUXA" href="https://github.com/pygame/pygame/pull/408" target="_blank">https://github.com/pygame/<wbr></wbr>pygame/pull/408</a><br />
[1] pygame C api issue, <a data-saferedirecturl="https://www.google.com/url?hl=en&q=https://github.com/pygame/pygame/issues/333&source=gmail&ust=1520457012450000&usg=AFQjCNHADYH55S6d_vRA61rC6qTDa34zNw" href="https://github.com/pygame/pygame/issues/333" target="_blank">https://github.com/pygame/<wbr></wbr>pygame/issues/333</a><br />
[2] jedi confused with pygame pr, <a data-saferedirecturl="https://www.google.com/url?hl=en&q=https://github.com/pygame/pygame/pull/413&source=gmail&ust=1520457012450000&usg=AFQjCNHpggX740aCa4isjE7BzLl5ZtHe0w" href="https://github.com/pygame/pygame/pull/413" target="_blank">https://github.com/pygame/<wbr></wbr>pygame/pull/413</a></div>
Unknownnoreply@blogger.com1