Not My Idea » Python

An amazing summer of code working on distutils2

Alexis Metaireau — Mon, 16 Aug 2010 11:54:55 +0000

The Google Summer of Code I’ve spent working on distutils2 is over. It was a really amazing experience, for many reasons.

First of all, we had a very good team, we were 5 students working on distutils2: Zubin, Éric, Josip, Konrad and me. In addition, Mouad have worked on the PyPI
testing infrastructure. You could find what each person have done on the wiki page of distutils2.

We were in contact with each others really often, helping us when possible (in #distutils), and were continuously aware of the state of the work of each participant. This, in my opinion, have bring us in a good shape.

Then, I’ve learned a lot. Python packaging was completely new to me at the time of the GSoC start, and I was pretty unfamiliar with python good practices too, as I’ve been introducing myself to python in the late 2009.

I’ve recently looked at some python code I wrote just three months ago, and I was amazed to think about many improvements to made on it. I guess this is a good indicator of the path I’ve traveled since I wrote it.

This summer was awesome because I’ve learned about python good practices, now having some strong mercurial knowledge, and I’ve seen a little how the python community works.

Then, I would like to say a big thanks to all the mentors that have hanged around while needed, on IRC or via mail, and especially my mentor for this summer, Tarek Ziadé.

Thanks a lot for your motivation, your leadership and your cheerfulness, even with a new-born and a new work!

Why ?

I wanted to work on python packaging because, as the time pass, we were having a sort of complex tools in this field. Each one wanted to add features to distutils, but not in a standard way.

Now, we have PEPs that describes some format we agreed on (see PEP 345), and we wanted to have a tool on which users can base their code on, that’s distutils2.

My job

I had to provides a way to crawl the PyPI indexes in a simple way, and do some installation / uninstallation scripts.

All the work done is available in my bitbucket repository.

Crawling the PyPI indexes

There are two ways of requesting informations from the indexes: using the "simple" index, that is a kind of REST index, and using XML-RPC.

I’ve done the two implementations, and a high level API to query those twos. Basically, this supports the mirroring infrastructure defined in PEP 381.
So far, the work I’ve done is gonna be used in pip (they’ve basically copy/paste the code, but this will change as soon as we get something completely stable for distutils2), and that’s a good news, as it was the main reason for what I’ve done that.

I’ve tried to have an unified API for the clients, to switch from one to another implementation easily. I’m already thinking of adding others crawlers to this stuff, and it was made to be extensible.

If you want to get more informations about the crawlers/PyPI clients, please refer to the distutils2 documentation, especially the pages about indexes.

You can find the changes I made about this in the distutils2 source code .

Installation / Uninstallation scripts

Next step was to think about an installation script, and an uninstaller.
I’ve not done the uninstaller part, and it’s a smart part, as it’s basically removing some files from the system, so I’ll probably do it in a near future.

distutils2 provides a way to install distributions, and to handle dependencies between releases. For now, this support is only about the last version of the METADATA (1.2) (See, the PEP 345), but I’m working on a compatibility layer for the old metadata, and for the informations provided via PIP requires.txt, for instance.

Extra work

Also, I’ve done some extra work. this includes:

working on the PEP 345, and having some discussion about it (about the names of some fields).
writing a PyPI server mock, useful for tests. you can find more information about it on the documentation.

Futures plans

As I said, I’ve enjoyed working on distutils2, and the people I’ve met here are really pleasant to work with. So I want to continue contributing on python, and especially on python packaging, because there is still a lot of things to do in this scope, to get something really usable.

I’m not plainly satisfied by the work I’ve done, so I’ll probably tweak it a bit: the installer part is not yet completely finished, and I want to add support for a real REST index in the future.

We’ll talk again of this in the next months, probably, but we definitely need a real REST API for PyPI, as the "simple" index is an ugly hack, in my opinion. I’ll work on a serious proposition about this, maybe involving CouchDB, as it seems to be a good option for what we want here.

Issues

I’ve encountered some issues during this summer. The main one is that’s hard to work remotely, especially being in the same room that we live, with others. I like to just think about a project with other people, a paper and a pencil, no computers. This have been not so possible at the start of the project, as I needed to read a lot of code to understand the codebase, and then to read/write emails.

I’ve finally managed to work in an office, so good point for home/office separation.

I’d not planned there will be so a high number of emails to read, in order to follow what’s up in the python world, and be a part of the community seems to takes some times to read/write emails, especially for those (like me) that arent so confortable with english (but this had brought me some english fu !).

Thanks !

A big thanks to Graine Libre and Makina Corpus, which has offered me to come into their offices from time to time, to share they cheerfulness !
Many thanks too to the Google Summer of Code program for setting up such an initiative. If you’re a student, if you’re interested about FOSS, dont hesitate any second, it’s a really good opportunity to work on interesting projects!

GSOC Distutils first report

Alexis Metaireau — Fri, 04 Jun 2010 10:27:33 +0000

As I’ve been working on Distutils2 during the past week, taking part of the GSOC program, here is a short summary of what I’ve done so far.

As my courses are not over yet, I’ve not worked as much as I wanted, and this will continues until the end of June. My main tasks are about making installation and uninstallation commands, to have a simple way to install distributions via Distutils2.

To do this, we need to rely on informations provided by the Python Package Index (PyPI), and there is at least two ways to retreive informations from here: XML-RPC and the "simple" API.

So, I’ve been working on porting some Distribute related stuff to Distutils2, cutting off all non distutils’ things, as we do not want to depend from Distribute’s internals. My main work has been about reading the whole code, writing tests about this and making those tests possible.

In fact, there was a need of a pypi mocked server, and, after reading and introducing myself to the distutils behaviors and code, I’ve taken some time to improve the work Konrad makes about this mock.

A PyPI Server mock

The mock is embeded in a thread, to make it available during the tests, in a non blocking way. We first used WSGI and wsgiref in order control what to serve, and to log the requests made to the server, but finally realised that wsgiref is not python 2.4 compatible (and we need to be python 2.4 compatible in Distutils2).

So, we switched to BaseHTTPServer and SimpleHTTPServer, and updated our tests accordingly. It’s been an opportunity to realize that WSGI has been a great step forward for making HTTP servers, and expose a really simplest way to discuss with HTTP !

You can find the modifications I made, and the related docs about this on my bitbucket distutils2 clone.

The PyPI Simple API

So, back to the main problematic: make a python library to access and request information stored on PyPI, via the simple API. As I said, I’ve just grabbed the work made from Distribute, and played a bit with, in order to view what are the different use cases, and started to write the related tests.

The work to come

So, once all use cases covered with tests, I’ll rewrite a bit the grabbed code, and do some software design work (to not expose all things as privates methods, have a clear API, and other things like this), then update the tests accordingly and write a documentation to make this clear.

Next step is to a little client, as I’ve already started here I’ll take you updated !

A Distutils2 GSOC

Alexis Metaireau — Sat, 01 May 2010 14:33:01 +0000

WOW.

I’ve been accepted to be a part of the Google Summer Of Code program, and will work on python distutils2, with a lot of (intersting!) people.

So, it’s about building the successor of Distutils2, ie. « the python package manager» . Today, there is too many ways to package a python application (pip, setuptools, distribute, distutils, etc.) so there is a huge effort to make in order to make all this packaging stuff interoperable, as pointed out by the PEP 376.

In more details, I’m going to work on the Installer / Uninstaller features of Distutils2, and on a PyPI XML-RPC client for distutils2.

Here are the already defined tasks:

Implement Distutils2 APIs described in PEP 376.
Add the uninstall command.
think about a basic installer / uninstaller script. (with deps) — similar to pip/easy_install
in a pypi subpackage;
Integrate a module similar to setuptools’ package_index’
PyPI XML-RPC client for distutils 2: http://bugs.python.org/issue8190

As I’m relatively new to python, I’ll need some extra work in order to apply all good practice, among other things that can make a developper-life joyful.

I’ll post here, each week, my advancement, and my tought about python and especialy python packaging world.

Et c’est reparti pour un tour!

Alexis Metaireau — Sun, 04 Oct 2009 15:54:40 +0000

Un petit billet plein de bonnes nouvelles !

Ce blog

Un « sous blog» consacrée aux différentes astuces, compilations, partage de connaissances, via des billets courts, la partie « snippets» fait son apparition. J’y publie bien plus souvent des mini articles sur des trucs que je rencontre au jour le jour. Allez y jeter un oeil, et pourquoi pas, mettez le dans vos lecteurs de flux

Quand à ce blog , il continuera de décrire l’ensemble des évolutions des divers projets auquel je participe, qu’ils soient ou non liés au monde logiciel, ponctué par quelques articles techniques que j’ai sous le coude.

Spiral

Depuis la publication de l’injecteur de dépendances de Spiral, nous avons travaillé durant une petite semaine à plein temps sur spiral, sur des thématiques telle que le système de packages, ou le modèle de persistance des données. Puis on à recentré nos efforts sur la mise en place d’un environnement de tests digne de ce nom. J’en ai profité pour faire pour la première fois, du *vrai* TDD !

Spiral continue d’être ce que nous souhaitons qu’il soit: un lieu d’expérimentations et de mise en place de bonnes pratiques. En parallèle, nous travaillons dur sur la mise en place d’une documentation, mettant en lumière les nombreux concepts qui nous obligent à réfléchir et à écumer l’horizon des possibles, et … de changer de direction tous les 4 matins. Tant qu’à faire, autant que ce soit accessible au plus grand nombre.

Je ne vais pas retracer toutes les modifications, mais, rapidement, nous sommes arrivés à un système de packages des plus stables¹, et je suis en train de terminer la nouvelle mouture de l’injecteur de dépendances, qui sera cette fois-ci facilement extensible.

Nous avons récupéré le nom de domaine www.spiral-project.org, qui servira à publier la documentation et les diverses expériences en relation avec spiral. Rien de terminé pour le moment (d’ailleurs, vous pouvez voir que c’est bien vide), mais on à un joli site web en préparation, vous pouvez tout suivre en direct sur bitbucket !

Stage

Je termine un stage de pratiquement un an chez Keo Networks, en tant que développeur web, sur du Zend Framework (PHP). De bons souvenirs, mais ce changement démarque aussi une envie d’aller aussi voir ailleurs, pour plusieurs raisons, et la permissivité du langage PHP n’y est pas pour rien.

Je m’en vais donc vers d’autres horizons, puisque j’aurais le plaisir d’aller bosser chez Makina Corpus, dans l’équipe Python/Django/Zope. L’entreprise est très attachée aux valeurs du logiciel libre, ce qui n’est pas vraiment pour me déplaire

J’ai donc déjà commencé à creuser un peu sur django et sur python, pour l’instant, je dois surtout avouer que je ne suis pas aussi à l’aise qu’avec le PHP que j’ai tant l’habitude de manier, mais ça vient progressivement !

Associatif !

Et puis, pour terminer sur une note un peu moins « geek» , cette année, on reprends les mêmes, et on recommence ! Plusieurs projets qui pointent le bout de leur nez, dont un qui arrive au terme de la période de réflexion, pour entrer en action: l’AMAP étudiante Toulousaine, Zest. Les distributions commencent demain (lundi 5 octobre), je croise les doigts pour que tout roule !

La semaine de l’environnement de l’année prochaine commence à pointer le bout de son nez aussi, et les réunions d’organisation vont bientôt débouler. Y’à du pain sur la planche !

Au réseau GRAPPE, aussi, on continue à avoir des réflexions sur la thématique de l’alimentation, d’ailleurs, vous pourrez trouver un document qui résume notre boulot de l’année dernière: « Les étudiants se mettent à table« .

et qui ressemble à celui de Flow3, tiens tiens…