Not My Idea

Why ?

I wanted to work on python packaging because, as the time pass, we were having a sort of complex tools in this field. Each one wanted to add features to distutils, but not in a standard way.

Now, we have PEPs that describes some format we agreed on (see PEP 345), and we wanted to have a tool on which users can base their code on, that’s distutils2.

My job

I had to provides a way to crawl the PyPI indexes in a simple way, and do some installation / uninstallation scripts.

All the work done is available in my bitbucket repository.

Futures plans

As I said, I’ve enjoyed working on distutils2, and the people I’ve met here are really pleasant to work with. So I want to continue contributing on python, and especially on python packaging, because there is still a lot of things to do in this scope, to get something really usable.

I’m not plainly satisfied by the work I’ve done, so I’ll probably tweak it a bit: the installer part is not yet completely finished, and I want to add support for a real REST index in the future.

We’ll talk again of this in the next months, probably, but we definitely need a real REST API for PyPI, as the "simple" index is an ugly hack, in my opinion. I’ll work on a serious proposition about this, maybe involving CouchDB, as it seems to be a good option for what we want here.

Issues

I’ve encountered some issues during this summer. The main one is that’s hard to work remotely, especially being in the same room that we live, with others. I like to just think about a project with other people, a paper and a pencil, no computers. This have been not so possible at the start of the project, as I needed to read a lot of code to understand the codebase, and then to read/write emails.

I’ve finally managed to work in an office, so good point for home/office separation.

I’d not planned there will be so a high number of emails to read, in order to follow what’s up in the python world, and be a part of the community seems to takes some times to read/write emails, especially for those (like me) that arent so confortable with english (but this had brought me some english fu !).

Thanks !

A big thanks to Graine Libre and Makina Corpus, which has offered me to come into their offices from time to time, to share they cheerfulness !
Many thanks too to the Google Summer of Code program for setting up such an initiative. If you’re a student, if you’re interested about FOSS, dont hesitate any second, it’s a really good opportunity to work on interesting projects!

General feelings

First, and it’s a really important point, the GSoC is going very well, for me as for other students, at least from my perspective. It’s a pleasure to work with such enthusiast people, as this make the global atmosphere very pleasant to live.

First of all, I’ve spent time to read the existing codebase, and to understand what we’re going to do, and what’s the rationale to do so.

It’s really clear for me now: what we’re building is the foundations of a packaging infrastructure in python. The fact is that many projects co-exists, and comes all with their good concepts. Distutils2 tries to take the interesting parts of all, and to provide it in the python standard libs, respecting the recently written PEP about packaging.

With distutils2, it will be simpler to make "things" compatible. So if you think about a new way to deal with distributions and packaging in python, you can use the Distutils2 APIs to do so.

Tasks

My main task while working on distutils2 is to provide an installation and an un-installation command, as described in PEP 376. For this, I first need to get informations about the existing distributions (what’s their version, name, metadata, dependencies, etc.)

The main index, you probably know and use, is PyPI. You can access it at http://pypi.python.org.

PyPI index crawling

There is two ways to get these informations from PyPI: using the simple API, or via xml-rpc calls.

A goal was to use the version specifiers defined in PEP 345 and to provides a way to sort the grabbed distributions depending our needs, to pick the version we want/need.

Processes

For now, I'm trying to follow the "documentation, then test, then code" path, and that seems to be really needed while working with a community. Code is hard to read/understand, compared to documentation, and it's easier to change.

While writing the simple index crawling work, I must have done this to avoid some changes on the API, and some loss of time.

Also, I've set up a schedule, and the goal is to be sure everything will be ready in time, for the end of the summer. (And now, I need to learn to follow schedules ...)

A PyPI Server mock

The mock is embeded in a thread, to make it available during the tests, in a non blocking way. We first used WSGI and wsgiref in order control what to serve, and to log the requests made to the server, but finally realised that wsgiref is not python 2.4 compatible (and we need to be python 2.4 compatible in Distutils2).

So, we switched to BaseHTTPServer and SimpleHTTPServer, and updated our tests accordingly. It’s been an opportunity to realize that WSGI has been a great step forward for making HTTP servers, and expose a really simplest way to discuss with HTTP !

You can find the modifications I made, and the related docs about this on my bitbucket distutils2 clone.

The PyPI Simple API

So, back to the main problematic: make a python library to access and request information stored on PyPI, via the simple API. As I said, I’ve just grabbed the work made from Distribute, and played a bit with, in order to view what are the different use cases, and started to write the related tests.

The work to come

So, once all use cases covered with tests, I’ll rewrite a bit the grabbed code, and do some software design work (to not expose all things as privates methods, have a clear API, and other things like this), then update the tests accordingly and write a documentation to make this clear.

Next step is to a little client, as I’ve already started here I’ll take you updated !