Wednesday, September 16, 2009

py3k(python3) more than one year on - 0.96 % packages supporting py3k.

Python3 was released more than a year ago so far, and the release candidates and beta releases much before then.

How successful has the transition been to python3 so far?

One way to measure that is to look at how many python packages have been ported to py3k. One way to measure what packages are released is to look at the python package index(aka cheeseshop, aka pypi).

Not all packages ported to python3 are listed on pypi, and not all packages are listed on pypi. Also there are many packages which haven't been updated in a long time. However it is still a pretty good way to get a vague idea of how things are going.

73 packages are listed in the python3 section of pypi, and 7568 packages in total. That's 0.96% of python packages having been ported to python3.

Another large project index for python is the pygame.org website. Where there are currently over 2000 projects which use pygame. I think there are 2 projects ported to python3 on there(but I can't find them at the moment). This shows a section of the python community using it in projects. Most of the things listed on pypi are packages - not projects. It's showing what people are using for their projects - not what their libraries support. In a similar way, it could be good to see how many websites are running on top of python3. I think a lot of the people who have ported to python3 aren't really using it for their projects, but have done the porting work as a good will measure towards moving python forward.

Another way to measure the success of the migration, is to pick a random sampling of some popular packages and see if their python3 support exists.

Pygame(yes), Pyopengl(no), pyglet(no), numpy(no), PIL python imaging library(no), cherrypy(yes), Twisted(no), zope.interface(no), buildout(not sure, I think no), setuptools(no, patches available), django(no), plone(no), psyco(no), cython(yes), swig(yes), sqlalchemy(no).

With some packages being used by 1000s or 10,000s of projects, those popular projects hold back the py3k migration significantly. It would seem that some applied efforts to the right projects would help the py3k migration a lot. Perhaps a py3k porting team could be made to help port important libraries to py3k.

How about other python implementations supporting python3 features? None have full python3 support as of yet. For example jython(no), pypy(no), ironpython(no), tinypy(no), python-on-a-chip(no), unladenswallow(no), shedskin(no). However some implementations support some new python3 features.

How about wsgi? wsgi is the python specification for web gateways... that it specifies how different web frameworks, web servers and applications can talk to each other and out to http. The wsgiref module in python3 is somewhat broken, and the amendments for python3 have not made it into a new wsgi spec. However work is being done towards it with a couple of major wsgi users supporting python3(cherrypy and mod_wsgi).

Another question to ask is: 'are many projects planning to support py3k soon? Or have they decided not to work on py3k at all yet?'. It seems many projects have decided not to put in the work yet. At this point, for many projects they don't see enough benefit towards moving to py3k. Or their dependencies have not been ported, and they are waiting on those to be ported before beginning to port themselves.

How well have the python developers themselves developed the support material for people upgrading their code? It looks like the cporting guide is still incomplete and hasn't been updated in a while. However the CPython API using projects have taken up the slack... so there are now a number of extensions for people to look at for guidance. It's possible to make CPython extensions which support both 2.x and 3.x APIs.

There is now a 3to2 script being worked on. This allows projects to write their code in python3 code, and have it translated into python2 code. The python developers realised that having a 2to3 script was backwards in a way - requiring developers to stick with their python 2 code. However, many projects seem to not use the translation script, since it hasn't worked for them. Instead they seem to have either made separate branches, or made their code so that it works in both 2.x and 3.x.

Support for python3 was dropped, and python3.1 is the new python3. However python3 still exists in some distributions (like ubuntu).

So how are the various OS distributions going with their python3 support? The latest version of OSX to be released (snow leopard) uses a version of python2.6.1. Most unix distributions are using python2.6 as their main python at the moment. However most of them have also packaged python3.x as well. So it's fairly easy for people to try out python3 alongside their python2.x installation(s). macports currently has py25(286 ports), py26(206 ports), py30(0 ports... since py3.0 isn't supported by python.org), and py31(4 ports). So for macports, it has 1.9% of py31 packages ported compared to py25. This shows similar percentage to the ratio of ported packages in the pypi index(0.96%).

This is not mirrored by the number of windows downloads from python.org. Python2.6.2 windows installer had 786400 downloads, python2.5.4(104291), python 3.1(241363) 3.1.1(214871) for a total of 456234 for 3.x. This is around 58% comparing 2.6 and (3.0+3.1). Strangely about the same amount of people are downloading 3.0 as 3.1 - even though it states that 3.1 is the new py3k and 3.0 is not supported anymore. This is just windows download counts for August... if you compare it to most unix distributions, they almost all come with python2.5 or python2.6.

So, is the python3 migration going along swimmingly? Or has it failed to reach its goals(what were its goals if any)? What can we do to help? Should we even help at this point? comments?

33 comments:

Cory said...

I maintain at least two packages on pypi. Both of them would be trivial to port to Py3k. I just made it my mission to be the one who pushes py3k acceptance over 1%! :-)

Gregory said...

Two corrections:
* Django does not support Py3K and won't for a good while (as per the devs).
* Snow leopard comes with Python 2.6.1

illume said...

@Gregory: thanks, fixed.

j_king said...

I don't foresee many projects shifting to Python 3 any time soon. It offers almost no benefit for the pain of porting. I think breaking backwards compatibility the way they did might have been good for the language and the core team but it was bad for the programmers and developers using it in their work. If it works just fine on Python 2, why bother?

If it had offered performance benefits over Python 2 I think there might have been a more concerted effort to switch, but so far the response to the benefits offered by Python 3 have been underwhelming.

Ville said...

Part of the problem may be that python3 is not universally deployed yet. e.g. Jaunty has no python 3.1 in apt repos.

Important progress is happening; pyqt is now python 3 compatible, so you can now develop a full gui app with python 3 (unless you use Tk ;-). The dependencies just need to land in repositories before you can seriously deploy your app.

Masklinn said...

Libraries were never expected to port to Pyton 3.0, and most people were not expected to switch to it either. That simply wasn't the goal of the 3.0 release.

Library porting efforts are expected to slowly ramp up now that 3.1 has been released and the kinks of 3to2 (note: not 2to3) are being worked out, and to start in earnest by 3.2 (which will probably be released early next year).

There's nothing abnormal to the current low adoption of 3.0, quite the opposite in fact (there's a reason why it's already fully unsupported, less than 10 months after its release), in fact it's all going according to the plan right now.

Masklinn said...

Now please stop concern-trolling Python.

j_king said...

Not saying Python 3 is bad of course.

I just see most folks already writing Python 2 staying with Python 2.

illume said...

@Masklinn: libraries are expected to be ported to py3k. Otherwise it's impossible for others to port their code.

The more libraries that are ported sooner, the more experience people have porting, more people can find bugs in py3k.

I'm not sure where you got the impression that the python developers did not want people to port their libraries? Do you have a link?

It's not written in any peps, and blog posts from guido and other developers discuss issues porting libraries to py3k. How to keep API compatibility during the cross over, and how to try and encourage others to port their libraries to py3k.

Stopping python 3.0 support was unplanned, but is reasonable considering the low amount of users, and how everyone should try and target 3.1.1 now.

Masklinn said...

> @Masklinn: libraries are expected to be ported to py3k. Otherwise it's impossible for others to port their code.

To py3k in general yes, to Python 3.0 no. Never. Ever.

> The more libraries that are ported sooner, the more experience people have porting, more people can find bugs in py3k.

Of course but that's not really the point. The point of 3.0 was to have it out the door and allow people to check it out, ensure that Python 3 was viable, etc…

> I'm not sure where you got the impression that the python developers did not want people to port their libraries? Do you have a link?

Sadly not, I'm sure you can find posts about that on the mailing lists though.

> Stopping python 3.0 support was unplanned

Uh no it wasn't.

Masklinn said...

Re links, on news.yc, someone points out to the title of #python@irc.freenode.net, which to this day says (it's the second item in the title) It's too early to use Python 3.x.

illume said...

@Masklinn: Great, look forward to seeing evidence of where anyone has said 'do not port your libraries to python 3.0'. It does not state in the py3k pep that they'd drop support for the python 3.0 release. There's no evidence for your claims.

Now please stop making shit up about Python.

Marius Gedminas said...

setuptools works on Python 3 now? Are you sure? I don't see it in the "Python 3 packages" section of PyPI.

Having once learned enough of setuptools/virtualenv/zc.buildout, I cannot go back and live without them.

illume said...

@Marius:

there is a page here about how someone has worked on porting it:
http://regebro.wordpress.com/2009/02/01/setuptools-and-easy_install-for-python-3/

However, I'm not sure if it's in the main release yet( the main release wasn't working a couple of months ago ). I expect Distribute(recent setuptools fork) should get py3k support very soon.

illume said...

hi again, Updated to say 'setuptools (no, patches available). Which is more accurate. I looked into Distribute py3k support, and it's still planned... not finished.

regebro said...

Yes, libraries was supposed to be ported to Python 3.0. However, Python 3.0 turned out to be a bit of an "intermediary" release. And this was not supposed to be, but also not unexpected.

Is Python 3 adaptation going on Swimmingly? No. And the reason for that is twofold: As mentioned above, there is little actual benefit. And secondly, almost all software, both applications and libraries, depend on other libraries, which are NOT ported to Python.


I'm sure there are several big bottlenecks in the porting, but the biggest of them are setuptools, which do not have official Python 3 support and is not likely to ever get it either. As a response to this and other concerns about setuptools the Distribute project has been created. It's a fork of setuptools that aim to split up refactor it and yes, support Python 3.

This is going, if not swimmingly, so at least forward. I would hope for a Python 3 release maybe for a Christmas present.

btami said...

Sqlalchemy is coming soon too.
http://www.sqlalchemy.org/news.html#item_1

Lee_B said...

I've been dying to upgrade to 3.x ever since it was launched, but the underlying libraries/frameworks/ORMS are still talking about years of wait, and I can't find any other 3.x libraries/frameworks/ORMS that serve as alternatives. I think you're right that we need to start at the bottom of the python dependency stack, choosing the frameworks that are most popular, and working together to get them upgraded.

But before all that starts, debian really needs 3.x packages. It's the key to getting a whole host of people wanting to install python3.1 over python2.6 and itching enough that they can't.

Roger Binns said...

My extension (apsw) has been source compatible with Python 3 since the betas and I have released Windows binaries since Python 3 was released (yes I do deserve a gold star :-)

Since I release Windows binaries and have an approximately monthly release schedule, you can easily see the adoption profiles for Windows users of the various Python versions. It has only been very recently that Python 2.3 has diminished to almost nothing! Have a look at apsw downloads.

Brandon Craig Rhodes said...

There's a huge issue here that no one is addressing: if anyone does port their package to Python 3, there's nowhere to put it! If you maintain the "foo" package, and put up your shiny new Python 3 version on the Cheeseshop, then — since it will register as the most recent version of your library — it will also be downloaded by everyone still using Python 2.x and horrendously break 90+% of your user base. If, instead, you jury-rig the version number to keep the Python 3 version from being newest, then I have absolutely no idea how Python 3 users will find it.

The situation is intractable until a Python 3 Cheeseshop is opened, and a Python 3 easy_install is available to pull from it. Otherwise everyone in the whole Python community will have to have a "foo3" package in addition to their "foo" package, and won't that look silly after a couple of years? I can hear the newcomers to Python in 2015 now: "It's a cool language... but why does every single package end its name with a '3'?" :-)

Also, this is a great opportunity to replace "pypi" with a name that doesn't conflict audibly with the PyPy project. Let's create a Python 3 Cheeseshop, abbreviate it as something other than "pypi", and then people will have somewhere to start releasing Python 3 packages without breaking their 2.x users.

Graham Dumpleton said...

The WSGI situation is not as cheery as you make out. There is still so no consensus. My official stance on Apache/mod_wsgi is that it does not support WSGI on Python 3.X. Yeah, there is some code in there which will do something on Python 3.X, but you shouldn't be assuming that WSGI for Python 3.X will be anything similar. Although CherryPy is moving on Python 3.X support, can't say for sure that it even works the same as Apache/mod_wsgi. Thus you have different implementations potentially working differently. Until there is proper updated specification, it is going to be an untenable situation.

I will be posting an absolutely huge rant about this soon which I have spent a few weeks putting together.

Neil said...

I ported a module written in C to work with Python 3.x 2 2 months ago and found little helpful information. There are some reasonably mechanical changes you can do to implement tp_richcompare rather than tp_compare, for example, but you have to work this out for yourself.

My opinion is that the incompatibilities between 2.x and 3.x have inserted a multi-year blockage into the development of Python. It is much worse than I expected.

boo said...

Increase python 3 Performance!

Florian said...

The miserable adoption rate and slow library support of python3 you see is the typical growth curve of an entirely new and unknown language. As such, and assuming optimal conditions, python3 will have significant market share under the big 5-6 languages in roughly 8-12 years.

Python3 would maybe have fared better by not associating itself with python, *cough*, yeah that sounds strange, but in actuality, Python3 is a new language/environment, people might be more experimental if they actually perceived it as something new rather then the next iteration.

Chris Arndt said...

As long as there is no Python 3 support in setuptools or a an alternative I won't even start thinking about porting the packages I'm involved with.

As for the PyPI issue: this is actually a setuptools/easy_install problem. It always assumes the latest version is what you want, even it is alpha or beta. This is continually bugging me. Everytime a new version of one of my dependencies is released on PyPI, users installing my package via easy_install will get this new version. If it is a C-extension, and there is no binary package for their system and they have no compiler/dev-packages, the installation breaks. Even though a perfectly good binary package of an older version of the dependency package is still available on PyPI.

This is the main reason why I created EggBasket, a minimal package index server, so you can host your own package repository.


Chris

mike bayer said...

SQLAlchemy trunk runs on Python 3.x though we haven't yet added it's tests to the buildbot, nor have we added documentation on how to run the 2to3 tool, but its very easy. There's an "sa2to3.py" script which you run against ./lib/ (and optionally ./test/) to produce Python 3 code.

Tartley said...

Is it meaningful to construct a dependency graph of packages in PyPI to make visible the priorities for migration.

Things which don't depend on anything, but upon which many other things depend, could be flagged up as significant blockers on the path to migration.

Then the original authors would at least be aware of this situation, and third parties could also make a meaningful decision about whether to step in and lend a hand.

This could be dovetailed with Brandon's idea, making tools such as easy_install / pip, etc 3k-aware could be given an artificially high priority in this scheme.

illume said...

@Tartley: that's a good idea. We know of at least some major blockers at this point - however we can't know all of the important ones.

There's a few python dependency tools around. python comes with modulefinder which can be used for this.

It's probably a good idea to make a list of 'packages which are blocking lots of other python projects from migrating'.

Like you say easy_install, pip, and setuptools, pypi are probably is *the* major thing to update at this point. Distribute is the answer to setuptools lack of py3k support I think.

Personally, I have looked at what my projects use and have found a few projects which I'm trying to help out. I've picked two major projects to help port, because they are important to me. It's the whole think global, act local idea ;)


cheers,

Peter Eisentraut said...

A large-scale move to Python 3 will obviously have to go through Python 2.6 first. That in turn requires that Python 2.5 is no longer relevant for library and application developers. In the Linux world, it usually takes about 5 years before an important software package is phased out (support cycle of enterprise distributions, release cycle of depending packages, etc.). So expect Python 3 to get real traction by then.

Melinda said...
This comment has been removed by a blog administrator.
Louie In Seattle said...

So, has anything improved with porting to Python 3 since 2009 when this was written? The posts have a time and no date, so I can't tell how old this thread is.

illume said...

Louie, yeah things have improved a lot. Most popular packages have been ported over.

There are a few popular packages holding out on their support, like Django and twisted. As well as lots of smaller packages, including ones that are not maintained any more. I know Django has python3 support on their road map. Twisted also has python support on their roadmap.

The tools have gotten better, and python 3 is now the only version getting development (except for security fixes).

However, the majority of popular packages are ported over. Courses are being taught using just python3 for a while now. There is a lot more documentation and community knowledge on porting.

There is a website which lists the most popular pypi packages and shows their porting state: http://py3ksupport.appspot.com/

There are around 550 packages that list themselves as having python3 support out of 16350 packages on pypi total (3.3%).

No other major python implementation has got python 3 support yet. Two of the implementations have lost a lot of development support (Jython and IronPython). I'm not sure if it would be fair to say the python 3 migration contributed to their loss of development funding - I have no idea. pypy now supports python2.7.1 which is makes it much closer to python 3 support. However I'm not sure if there is python 3 support close by on the roadmap for pypy or not.

There are even a couple of linux distros using python3 as the main default python. Because a couple of them got in early, many packages got a lot of porting and testing on python 3 early. The python community owes them a lot of thanks for the pain they went through, so others didn't have to. Also, the people who got stuck in and helped to port popular packages to python 3 deserve many thanks.

OS X (latest release Lion) still does not support python3 out of the box.

There are games coming out on pygame.org all the time which support just python3. However the majority are still python2.

WSGI(the web gateway for python) is now python3 compatible, and the major players in WSGI world are supporting python 3.

The main packaging tools now support python3 well. Including distribute, setuptools, pip, etc.

For me, I do about half of my stuff on python2, because it is better supported in some places. However most of the packages I use support python 3 now.

Quality packages have python 3 support at this stage. It is a great indicator of a well maintained package, and surrounding community.

2011/08/26

Victor Akinyemi said...

Actually, pyglet, PyOpenGL, NumPy, PIL, has all been ported to python 3, HURRAY!