py3k(python3) more than one year on - 0.96 % packages supporting py3k.
Python3 was released more than a year ago so far, and the release candidates and beta releases much before then.
How successful has the transition been to python3 so far?
One way to measure that is to look at how many python packages have been ported to py3k. One way to measure what packages are released is to look at the python package index(aka cheeseshop, aka pypi).
Not all packages ported to python3 are listed on pypi, and not all packages are listed on pypi. Also there are many packages which haven't been updated in a long time. However it is still a pretty good way to get a vague idea of how things are going.
73 packages are listed in the python3 section of pypi, and 7568 packages in total. That's 0.96% of python packages having been ported to python3.
Another large project index for python is the pygame.org website. Where there are currently over 2000 projects which use pygame. I think there are 2 projects ported to python3 on there(but I can't find them at the moment). This shows a section of the python community using it in projects. Most of the things listed on pypi are packages - not projects. It's showing what people are using for their projects - not what their libraries support. In a similar way, it could be good to see how many websites are running on top of python3. I think a lot of the people who have ported to python3 aren't really using it for their projects, but have done the porting work as a good will measure towards moving python forward.
Another way to measure the success of the migration, is to pick a random sampling of some popular packages and see if their python3 support exists.
Pygame(yes), Pyopengl(no), pyglet(no), numpy(no), PIL python imaging library(no), cherrypy(yes), Twisted(no), zope.interface(no), buildout(not sure, I think no), setuptools(no, patches available), django(no), plone(no), psyco(no), cython(yes), swig(yes), sqlalchemy(no).
With some packages being used by 1000s or 10,000s of projects, those popular projects hold back the py3k migration significantly. It would seem that some applied efforts to the right projects would help the py3k migration a lot. Perhaps a py3k porting team could be made to help port important libraries to py3k.
How about other python implementations supporting python3 features? None have full python3 support as of yet. For example jython(no), pypy(no), ironpython(no), tinypy(no), python-on-a-chip(no), unladenswallow(no), shedskin(no). However some implementations support some new python3 features.
How about wsgi? wsgi is the python specification for web gateways... that it specifies how different web frameworks, web servers and applications can talk to each other and out to http. The wsgiref module in python3 is somewhat broken, and the amendments for python3 have not made it into a new wsgi spec. However work is being done towards it with a couple of major wsgi users supporting python3(cherrypy and mod_wsgi).
Another question to ask is: 'are many projects planning to support py3k soon? Or have they decided not to work on py3k at all yet?'. It seems many projects have decided not to put in the work yet. At this point, for many projects they don't see enough benefit towards moving to py3k. Or their dependencies have not been ported, and they are waiting on those to be ported before beginning to port themselves.
How well have the python developers themselves developed the support material for people upgrading their code? It looks like the cporting guide is still incomplete and hasn't been updated in a while. However the CPython API using projects have taken up the slack... so there are now a number of extensions for people to look at for guidance. It's possible to make CPython extensions which support both 2.x and 3.x APIs.
There is now a 3to2 script being worked on. This allows projects to write their code in python3 code, and have it translated into python2 code. The python developers realised that having a 2to3 script was backwards in a way - requiring developers to stick with their python 2 code. However, many projects seem to not use the translation script, since it hasn't worked for them. Instead they seem to have either made separate branches, or made their code so that it works in both 2.x and 3.x.
Support for python3 was dropped, and python3.1 is the new python3. However python3 still exists in some distributions (like ubuntu).
So how are the various OS distributions going with their python3 support? The latest version of OSX to be released (snow leopard) uses a version of python2.6.1. Most unix distributions are using python2.6 as their main python at the moment. However most of them have also packaged python3.x as well. So it's fairly easy for people to try out python3 alongside their python2.x installation(s). macports currently has py25(286 ports), py26(206 ports), py30(0 ports... since py3.0 isn't supported by python.org), and py31(4 ports). So for macports, it has 1.9% of py31 packages ported compared to py25. This shows similar percentage to the ratio of ported packages in the pypi index(0.96%).
This is not mirrored by the number of windows downloads from python.org. Python2.6.2 windows installer had 786400 downloads, python2.5.4(104291), python 3.1(241363) 3.1.1(214871) for a total of 456234 for 3.x. This is around 58% comparing 2.6 and (3.0+3.1). Strangely about the same amount of people are downloading 3.0 as 3.1 - even though it states that 3.1 is the new py3k and 3.0 is not supported anymore. This is just windows download counts for August... if you compare it to most unix distributions, they almost all come with python2.5 or python2.6.
So, is the python3 migration going along swimmingly? Or has it failed to reach its goals(what were its goals if any)? What can we do to help? Should we even help at this point? comments?
How successful has the transition been to python3 so far?
One way to measure that is to look at how many python packages have been ported to py3k. One way to measure what packages are released is to look at the python package index(aka cheeseshop, aka pypi).
Not all packages ported to python3 are listed on pypi, and not all packages are listed on pypi. Also there are many packages which haven't been updated in a long time. However it is still a pretty good way to get a vague idea of how things are going.
73 packages are listed in the python3 section of pypi, and 7568 packages in total. That's 0.96% of python packages having been ported to python3.
Another large project index for python is the pygame.org website. Where there are currently over 2000 projects which use pygame. I think there are 2 projects ported to python3 on there(but I can't find them at the moment). This shows a section of the python community using it in projects. Most of the things listed on pypi are packages - not projects. It's showing what people are using for their projects - not what their libraries support. In a similar way, it could be good to see how many websites are running on top of python3. I think a lot of the people who have ported to python3 aren't really using it for their projects, but have done the porting work as a good will measure towards moving python forward.
Another way to measure the success of the migration, is to pick a random sampling of some popular packages and see if their python3 support exists.
Pygame(yes), Pyopengl(no), pyglet(no), numpy(no), PIL python imaging library(no), cherrypy(yes), Twisted(no), zope.interface(no), buildout(not sure, I think no), setuptools(no, patches available), django(no), plone(no), psyco(no), cython(yes), swig(yes), sqlalchemy(no).
With some packages being used by 1000s or 10,000s of projects, those popular projects hold back the py3k migration significantly. It would seem that some applied efforts to the right projects would help the py3k migration a lot. Perhaps a py3k porting team could be made to help port important libraries to py3k.
How about other python implementations supporting python3 features? None have full python3 support as of yet. For example jython(no), pypy(no), ironpython(no), tinypy(no), python-on-a-chip(no), unladenswallow(no), shedskin(no). However some implementations support some new python3 features.
How about wsgi? wsgi is the python specification for web gateways... that it specifies how different web frameworks, web servers and applications can talk to each other and out to http. The wsgiref module in python3 is somewhat broken, and the amendments for python3 have not made it into a new wsgi spec. However work is being done towards it with a couple of major wsgi users supporting python3(cherrypy and mod_wsgi).
Another question to ask is: 'are many projects planning to support py3k soon? Or have they decided not to work on py3k at all yet?'. It seems many projects have decided not to put in the work yet. At this point, for many projects they don't see enough benefit towards moving to py3k. Or their dependencies have not been ported, and they are waiting on those to be ported before beginning to port themselves.
How well have the python developers themselves developed the support material for people upgrading their code? It looks like the cporting guide is still incomplete and hasn't been updated in a while. However the CPython API using projects have taken up the slack... so there are now a number of extensions for people to look at for guidance. It's possible to make CPython extensions which support both 2.x and 3.x APIs.
There is now a 3to2 script being worked on. This allows projects to write their code in python3 code, and have it translated into python2 code. The python developers realised that having a 2to3 script was backwards in a way - requiring developers to stick with their python 2 code. However, many projects seem to not use the translation script, since it hasn't worked for them. Instead they seem to have either made separate branches, or made their code so that it works in both 2.x and 3.x.
Support for python3 was dropped, and python3.1 is the new python3. However python3 still exists in some distributions (like ubuntu).
So how are the various OS distributions going with their python3 support? The latest version of OSX to be released (snow leopard) uses a version of python2.6.1. Most unix distributions are using python2.6 as their main python at the moment. However most of them have also packaged python3.x as well. So it's fairly easy for people to try out python3 alongside their python2.x installation(s). macports currently has py25(286 ports), py26(206 ports), py30(0 ports... since py3.0 isn't supported by python.org), and py31(4 ports). So for macports, it has 1.9% of py31 packages ported compared to py25. This shows similar percentage to the ratio of ported packages in the pypi index(0.96%).
This is not mirrored by the number of windows downloads from python.org. Python2.6.2 windows installer had 786400 downloads, python2.5.4(104291), python 3.1(241363) 3.1.1(214871) for a total of 456234 for 3.x. This is around 58% comparing 2.6 and (3.0+3.1). Strangely about the same amount of people are downloading 3.0 as 3.1 - even though it states that 3.1 is the new py3k and 3.0 is not supported anymore. This is just windows download counts for August... if you compare it to most unix distributions, they almost all come with python2.5 or python2.6.
So, is the python3 migration going along swimmingly? Or has it failed to reach its goals(what were its goals if any)? What can we do to help? Should we even help at this point? comments?
Comments
* Django does not support Py3K and won't for a good while (as per the devs).
* Snow leopard comes with Python 2.6.1
If it had offered performance benefits over Python 2 I think there might have been a more concerted effort to switch, but so far the response to the benefits offered by Python 3 have been underwhelming.
Important progress is happening; pyqt is now python 3 compatible, so you can now develop a full gui app with python 3 (unless you use Tk ;-). The dependencies just need to land in repositories before you can seriously deploy your app.
Library porting efforts are expected to slowly ramp up now that 3.1 has been released and the kinks of 3to2 (note: not 2to3) are being worked out, and to start in earnest by 3.2 (which will probably be released early next year).
There's nothing abnormal to the current low adoption of 3.0, quite the opposite in fact (there's a reason why it's already fully unsupported, less than 10 months after its release), in fact it's all going according to the plan right now.
I just see most folks already writing Python 2 staying with Python 2.
The more libraries that are ported sooner, the more experience people have porting, more people can find bugs in py3k.
I'm not sure where you got the impression that the python developers did not want people to port their libraries? Do you have a link?
It's not written in any peps, and blog posts from guido and other developers discuss issues porting libraries to py3k. How to keep API compatibility during the cross over, and how to try and encourage others to port their libraries to py3k.
Stopping python 3.0 support was unplanned, but is reasonable considering the low amount of users, and how everyone should try and target 3.1.1 now.
To py3k in general yes, to Python 3.0 no. Never. Ever.
> The more libraries that are ported sooner, the more experience people have porting, more people can find bugs in py3k.
Of course but that's not really the point. The point of 3.0 was to have it out the door and allow people to check it out, ensure that Python 3 was viable, etc…
> I'm not sure where you got the impression that the python developers did not want people to port their libraries? Do you have a link?
Sadly not, I'm sure you can find posts about that on the mailing lists though.
> Stopping python 3.0 support was unplanned
Uh no it wasn't.
Now please stop making shit up about Python.
Having once learned enough of setuptools/virtualenv/zc.buildout, I cannot go back and live without them.
there is a page here about how someone has worked on porting it:
http://regebro.wordpress.com/2009/02/01/setuptools-and-easy_install-for-python-3/
However, I'm not sure if it's in the main release yet( the main release wasn't working a couple of months ago ). I expect Distribute(recent setuptools fork) should get py3k support very soon.
Is Python 3 adaptation going on Swimmingly? No. And the reason for that is twofold: As mentioned above, there is little actual benefit. And secondly, almost all software, both applications and libraries, depend on other libraries, which are NOT ported to Python.
I'm sure there are several big bottlenecks in the porting, but the biggest of them are setuptools, which do not have official Python 3 support and is not likely to ever get it either. As a response to this and other concerns about setuptools the Distribute project has been created. It's a fork of setuptools that aim to split up refactor it and yes, support Python 3.
This is going, if not swimmingly, so at least forward. I would hope for a Python 3 release maybe for a Christmas present.
http://www.sqlalchemy.org/news.html#item_1
But before all that starts, debian really needs 3.x packages. It's the key to getting a whole host of people wanting to install python3.1 over python2.6 and itching enough that they can't.
Since I release Windows binaries and have an approximately monthly release schedule, you can easily see the adoption profiles for Windows users of the various Python versions. It has only been very recently that Python 2.3 has diminished to almost nothing! Have a look at apsw downloads.
The situation is intractable until a Python 3 Cheeseshop is opened, and a Python 3 easy_install is available to pull from it. Otherwise everyone in the whole Python community will have to have a "foo3" package in addition to their "foo" package, and won't that look silly after a couple of years? I can hear the newcomers to Python in 2015 now: "It's a cool language... but why does every single package end its name with a '3'?" :-)
Also, this is a great opportunity to replace "pypi" with a name that doesn't conflict audibly with the PyPy project. Let's create a Python 3 Cheeseshop, abbreviate it as something other than "pypi", and then people will have somewhere to start releasing Python 3 packages without breaking their 2.x users.
I will be posting an absolutely huge rant about this soon which I have spent a few weeks putting together.
My opinion is that the incompatibilities between 2.x and 3.x have inserted a multi-year blockage into the development of Python. It is much worse than I expected.
Python3 would maybe have fared better by not associating itself with python, *cough*, yeah that sounds strange, but in actuality, Python3 is a new language/environment, people might be more experimental if they actually perceived it as something new rather then the next iteration.
As for the PyPI issue: this is actually a setuptools/easy_install problem. It always assumes the latest version is what you want, even it is alpha or beta. This is continually bugging me. Everytime a new version of one of my dependencies is released on PyPI, users installing my package via easy_install will get this new version. If it is a C-extension, and there is no binary package for their system and they have no compiler/dev-packages, the installation breaks. Even though a perfectly good binary package of an older version of the dependency package is still available on PyPI.
This is the main reason why I created EggBasket, a minimal package index server, so you can host your own package repository.
Chris
Things which don't depend on anything, but upon which many other things depend, could be flagged up as significant blockers on the path to migration.
Then the original authors would at least be aware of this situation, and third parties could also make a meaningful decision about whether to step in and lend a hand.
This could be dovetailed with Brandon's idea, making tools such as easy_install / pip, etc 3k-aware could be given an artificially high priority in this scheme.
There's a few python dependency tools around. python comes with modulefinder which can be used for this.
It's probably a good idea to make a list of 'packages which are blocking lots of other python projects from migrating'.
Like you say easy_install, pip, and setuptools, pypi are probably is *the* major thing to update at this point. Distribute is the answer to setuptools lack of py3k support I think.
Personally, I have looked at what my projects use and have found a few projects which I'm trying to help out. I've picked two major projects to help port, because they are important to me. It's the whole think global, act local idea ;)
cheers,
There are a few popular packages holding out on their support, like Django and twisted. As well as lots of smaller packages, including ones that are not maintained any more. I know Django has python3 support on their road map. Twisted also has python support on their roadmap.
The tools have gotten better, and python 3 is now the only version getting development (except for security fixes).
However, the majority of popular packages are ported over. Courses are being taught using just python3 for a while now. There is a lot more documentation and community knowledge on porting.
There is a website which lists the most popular pypi packages and shows their porting state: http://py3ksupport.appspot.com/
There are around 550 packages that list themselves as having python3 support out of 16350 packages on pypi total (3.3%).
No other major python implementation has got python 3 support yet. Two of the implementations have lost a lot of development support (Jython and IronPython). I'm not sure if it would be fair to say the python 3 migration contributed to their loss of development funding - I have no idea. pypy now supports python2.7.1 which is makes it much closer to python 3 support. However I'm not sure if there is python 3 support close by on the roadmap for pypy or not.
There are even a couple of linux distros using python3 as the main default python. Because a couple of them got in early, many packages got a lot of porting and testing on python 3 early. The python community owes them a lot of thanks for the pain they went through, so others didn't have to. Also, the people who got stuck in and helped to port popular packages to python 3 deserve many thanks.
OS X (latest release Lion) still does not support python3 out of the box.
There are games coming out on pygame.org all the time which support just python3. However the majority are still python2.
WSGI(the web gateway for python) is now python3 compatible, and the major players in WSGI world are supporting python 3.
The main packaging tools now support python3 well. Including distribute, setuptools, pip, etc.
For me, I do about half of my stuff on python2, because it is better supported in some places. However most of the packages I use support python 3 now.
Quality packages have python 3 support at this stage. It is a great indicator of a well maintained package, and surrounding community.
2011/08/26