Not My Idea

Carnets Web d'Alexis Métaireau

Archive for the ‘Python’ Category

A distutils2 month — PyPI simple index ready.

2 comments

I’m working for about a month for distutils2, even if I was being a bit busy (as I had some class courses and exams to work on)

I’ll try do sum-up my general feelings here, and the work I’ve made so far. You can also find, if you’re interested, my weekly summaries in a dedicated wiki page.

General feelings

First, and it’s a really important point, the GSoC is going very well, for me as for other students, at least from my perspective. It’s a pleasure to work with such enthusiast people, as this make the global atmosphere very pleasant to live.

First of all, I’ve spent time to read the existing codebase, and to understand what we’re going to do, and what’s the rationale to do so.

It’s really clear for me now: what we’re building is the foundations of a packaging infrastructure in python. The fact is that many projects co-exists, and comes all with their good concepts. Distutils2 tries to take the interesting parts of all, and to provide it in the python standard libs, respecting the recently written PEP about packaging.

With distutils2, it will be simpler to make "things" compatible. So if you think about a new way to deal with distributions and packaging in python, you can use the Distutils2 APIs to do so.

Tasks

My main task while working on distutils2 is to provide an installation and an un-installation command, as described in PEP 376. For this, I first need to get informations about the existing distributions (what’s their version, name, metadata, dependencies, etc.)

The main index, you probably know and use, is PyPI. You can access it at http://pypi.python.org.

PyPI index crawling

There is two ways to get these informations from PyPI: using the simple API, or via xml-rpc calls.

A goal was to use the version specifiers defined in PEP 345 and to provides a way to sort the grabbed distributions depending our needs, to pick the version we want/need.

Using the simple API

The simple API is composed of HTML pages you can access at http://pypi.python.org/simple/.

Distribute and Setuptools already provides a crawler for that, but it deals with their internal mechanisms, and I found that the code was not so clear as I want, that’s why I’ve preferred to pick up the good ideas, and some implementation details, plus re-thinking the global architecture.

The rules are simple: each project have a dedicated page, which allows us to get informations about:

  • the distribution download locations (for some versions)
  • homepage links
  • some other useful informations, as the bugtracker address, for instance.

If you want to find all the distributions of the "EggsAndSpam" project, you could do the following (do not take so attention to the names here, as the API will probably change a bit):

>>> index = SimpleIndex()
>>> index.find("EggsAndSpam")
[EggsAndSpam 1.1, EggsAndSpam 1.2, EggsAndSpam 1.3]

We also could use version specifiers:

>>> index.find("EggsAndSpam (< =1.2)")
[EggsAndSpam 1.1, EggsAndSpam 1.2]

Internally, what’s done here is the following:

  • it process the http://pypi.python.org/simple/FooBar/ page, searching for download URLs.
  • for each found distribution download URL, it creates an object, containing informations about the project name, the version and the URL where the archive remains.
  • it sort the found distributions, using version numbers. The default
    behavior here is to prefer source distributions (over binary ones), and to rely on the last "final" distribution (rather than beta, alpha etc. ones)

So, nothing hard or difficult here.

We provides a bunch of other features, like relying on the new PyPI mirroring
infrastructure
or filter the found distributions by some criterias. If you’re curious, please browse the distutils2 documentation.

Using xml-rpc

We also can make some xmlrpc calls to retreive informations from PyPI. It's a really more reliable way to get informations from from the index (as it's just the index that provides the informations), but cost processes on the PyPI distant server.

For now, this way of querying the xmlrpc client is not available on Distutils2, as I'm working on it. The main pieces are already present (I'll reuse some work I've made from the SimpleIndex querying, and some code already set up), what I need to do is to provide a xml-rpc PyPI mock server, and that's on what I'm
actually working on.

Processes

For now, I'm trying to follow the "documentation, then test, then code" path, and that seems to be really needed while working with a community. Code is hard to read/understand, compared to documentation, and it's easier to change.

While writing the simple index crawling work, I must have done this to avoid some changes on the API, and some loss of time.

Also, I've set up a schedule, and the goal is to be sure everything will be ready in time, for the end of the summer. (And now, I need to learn to follow schedules ...)

Written by Alexis Metaireau

juillet 6th, 2010 at 9:12

A distutils2 sprint in tours

5 comments

Yesterday, as I was traveling to Tours, I’ve took some time to visit Éric, another student who’s working on distutils2 this summer, as a part of the GSoC. Basically, it was to take a drink, discuss a bit about distutils2, our respective tasks and general feelings, and to put a face on a pseudonym.

I’d really enjoyed this time, because Éric knows a lot of things about mercurial and python good practices, and I’m eager to learn about those.

So, we have discussed about things, have not wrote so much code, but have some things to propose so far, about documentation, and I also provides here some bribes of conversations we had.

Documentation

While writing the PyPI simple index crawler documentation, I realized that we miss some structure, or how-to about the documentation. Yep, you read well. We lack documentation on how to make documentation. Heh.

We’re missing some rules to follow, and this lead to a not-so-structured final documentation.

We probably target three type of publics, and we can split the documentation regarding those:

  • Packagers who want to distribute their softwares.
  • End users who need to understand how to use end user commands, like the installer/uninstaller
  • packaging coders who use distutils2, as a base for building a package manager.

We also need to discuss about a pattern to follow while writing documentation. How many parts do we need ? Where to put the API description ? etc. That’s maybe seems to be not so important, but I guess the readers would appreciate to have the same structure all along distutils2 documentation.

Mercurial

I’m really not a mercurial power user. I use it on daily basis, but I lack of basic knowledge about it. Big thanks Éric for sharing yours with me, you’re of a great help.

We have talked about some mercurial extensions that seems to make the life simpler, while used the right way. I’ve not used them so far, so consider this as a personal note.

  • hg histedit, to edit the history
  • hg crecord, to select the changes to commit

We have spent some time to review a merge I made sunday, to re-merge it, and commit the changes as a new changeset. Awesome.

These things make me say I need to read the hg book, and will do as soon as I got some spare time: mercurial seems to be simply great.

So … Great. I’m a powerful merger now !

On using tools

Because we also are hackers, we have shared a bit our ways to code, the tools we use, etc.

Both of us were using vim, and I’ve discovered vimdiff and hgtk, which will completely change the way I navigate into the mercurial history.

We aren’t « power users» , so we have learned from each other about vim tips.

You can find my dotfiles on github, if it could help. They’re not perfect, and not intended to be, because changing all the time, as I learn. Don’t hesitate to have a look, and to propose enhancements if you have !

On being pythonic

My background as an old Java user disserves me so far, as the paradigms are not the same while coding in python. Hard to find the more pythonic way to do, and sometimes hard to unlearn my way to think about software engineering.

Well, it seems that the only solution is to read code, and to re-read import this from times to times ! Coding like a pythonista seems to be a must-read, so, I know what to do.

Conclusion

It was really great. Next time, we’ll need to focus a bit more on distutils2, and to have a bullet list of things to do, but days like this one are opportunities to catch !

We’ll probably do another sprint in a few weeks, stay tuned !

Written by Alexis Metaireau

juillet 6th, 2010 at 2:48

Posted in distutils2

Tagged with ,

GSOC Distutils first report

leave a comment

As I’ve been working on Distutils2 during the past week, taking part of the GSOC program, here is a short summary of what I’ve done so far.

As my courses are not over yet, I’ve not worked as much as I wanted, and this will continues until the end of June. My main tasks are about making installation and uninstallation commands, to have a simple way to install distributions via Distutils2.

To do this, we need to rely on informations provided by the Python Package Index (PyPI), and there is at least two ways to retreive informations from here: XML-RPC and the "simple" API.

So, I’ve been working on porting some Distribute related stuff to Distutils2, cutting off all non distutils’ things, as we do not want to depend from Distribute’s internals. My main work has been about reading the whole code, writing tests about this and making those tests possible.

In fact, there was a need of a pypi mocked server, and, after reading and introducing myself to the distutils behaviors and code, I’ve taken some time to improve the work Konrad makes about this mock.

A PyPI Server mock

The mock is embeded in a thread, to make it available during the tests, in a non blocking way. We first used WSGI and wsgiref in order control what to serve, and to log the requests made to the server, but finally realised that wsgiref is not python 2.4 compatible (and we need to be python 2.4 compatible in Distutils2).

So, we switched to BaseHTTPServer and SimpleHTTPServer, and updated our tests accordingly. It’s been an opportunity to realize that WSGI has been a great step forward for making HTTP servers, and expose a really simplest way to discuss with HTTP !

You can find the modifications I made, and the related docs about this on my bitbucket distutils2 clone.

The PyPI Simple API

So, back to the main problematic: make a python library to access and request information stored on PyPI, via the simple API. As I said, I’ve just grabbed the work made from Distribute, and played a bit with, in order to view what are the different use cases, and started to write the related tests.

The work to come

So, once all use cases covered with tests, I’ll rewrite a bit the grabbed code, and do some software design work (to not expose all things as privates methods, have a clear API, and other things like this), then update the tests accordingly and write a documentation to make this clear.

Next step is to a little client, as I’ve already started here I’ll take you updated !

Written by Alexis Metaireau

juin 4th, 2010 at 12:27

Posted in distutils2

Tagged with , ,

A Distutils2 GSOC

one comment

WOW.

I’ve been accepted to be a part of the Google Summer Of Code program, and will work on python distutils2, with a lot of (intersting!) people.

So, it’s about building the successor of Distutils2, ie. « the python package manager» . Today, there is too many ways to package a python application (pip, setuptools, distribute, distutils, etc.) so there is a huge effort to make in order to make all this packaging stuff interoperable, as pointed out by the PEP 376.

The current state of packaging

In more details, I’m going to work on the Installer / Uninstaller features of Distutils2, and on a PyPI XML-RPC client for distutils2.

Here are the already defined tasks:

  • Implement Distutils2 APIs described in PEP 376.
  • Add the uninstall command.
  • think about a basic installer / uninstaller script. (with deps) — similar to pip/easy_install
  • in a pypi subpackage;
  • Integrate a module similar to setuptools’ package_index’
  • PyPI XML-RPC client for distutils 2: http://bugs.python.org/issue8190

As I’m relatively new to python, I’ll need some extra work in order to apply all good practice, among other things that can make a developper-life joyful.

I’ll post here, each week, my advancement, and my tought about python and especialy python packaging world.

Written by Alexis Metaireau

mai 1st, 2010 at 4:33

Posted in distutils2

Tagged with , ,

Python, go !

5 comments

Cela fait maintenant un peu plus d’un mois que je travaille sur un projet en django, et que, nécessairement, je me forme à Python.

Je prends un plaisir non dissimulé à découvrir ce langage (et à l’utiliser), qui ne cesse de me surprendre. Les premiers mots qui me viennent à l’esprit à propos de Python, sont « logique»  et « simple» . Et pourtant puissant pour autant.

Je ne manque d’ailleurs pas une occasion pour faire un peu d’évangélisation auprès des quelques personnes qui veulent bien m’écouter.

The Zen of Python

Avant toute autre chose, je pense utile de citer Tim Peters, et le PEP20, qui constituent une très bonne introduction au langage, qui prends la forme d’un easter egg présent dans python:

>>> import this
The Zen of Python, by Tim Peters

Beautiful is better than ugly.
Explicit is better than implicit.
Simple is better than complex.
Complex is better than complicated.
Flat is better than nested.
Sparse is better than dense.
Readability counts.
Special cases aren't special enough to break the rules.
Although practicality beats purity.
Errors should never pass silently.
Unless explicitly silenced.
In the face of ambiguity, refuse the temptation to guess.
There should be one-- and preferably only one --obvious way to do it.
Although that way may not be obvious at first unless you'
re Dutch.
Now is better than never.
Although never is often better than *right* now.
If the implementation is hard to explain, it's a bad idea.
If the implementation is easy to explain, it may be a good idea.
Namespaces are one honking great idea -- let'
s do more of those!

J’ai la vague impression que c’est ce que j’ai toujours cherché à faire en PHP, et particulièrement dans le framework Spiral, mais en ajoutant ces concepts dans une sur-couche au langage.

Ici, c’est directement de l’esprit de python qu’il s’agit, ce qui signifie que la plupart des bibliothèques python suivent ces concepts. Elle est pas belle la vie ?

Comment commencer, et par ou ?

Pour ma part, j’ai commencé par la lecture de quelques livres et articles intéressants, qui constituent une bonne entrée en matière sur le sujet (La liste n’est bien évidemment pas exhaustive et vos commentaires sont les bienvenus) :

J’essaye par ailleurs de partager au maximum les ressources que je trouve de temps à autres, que ce soit via twitter ou via mon compte delicious. Allez jeter un œil au tag python sur mon profil, peut être que vous trouverez des choses intéressantes, qui sait!

Un python sexy

Quelques fonctionnalités qui devraient vous mettre l’eau à la bouche:

  • Le chaînage des opérateurs de comparaison est possible (a<b <c dans une condition)
  • Assignation de valeurs multiples (il est possible de faire a,b,c = 1,2,3 par exemple)
  • Les listes sont simples à manipuler !
  • Les list comprehension, ou comment faire des opérations complexes sur les listes, de manière simple.
  • Les doctests: ou comment faire des tests directement dans la documentation de vos classes, tout en la documentant avec de vrais exemples.
  • Les métaclasses, ou comment contrôler la manière dont les classes se construisent
  • Python est un langage à typage fort dynamique: c’est ce qui m’agaçait avec PHP qui est un langage à typage faible dynamique.

Cous pouvez également aller regarder l’atelier donné par Victor Stinner durant le Pyconfr 09.

Have fun !

Written by Alexis Metaireau

décembre 17th, 2009 at 3:07

Posted in Python

Retours sur le scrumpy montpellier

leave a comment

Vendredi dernier, j’ai pu assister à la conférence scrumpy organisée par le collectif particules, sur python et l’agilité. Ce fut l’occasion d’en apprendre un peu plus sur python, django et scrum grâce à David Larlet et Claude Aubry.

Scrum / Agilité

Concept assez récent pour moi puisque j’ai découvert ce qu’était l’agilité lors du passage de l’agile tour sur Toulouse. La présentation de Claude à eu le mérite de reprendre l’ensemble des concepts et de poser les bases d’une nouvelle manière de gérer les projets. Rien de bien sorcier à priori, mais l’agilité remets grandement en question les préceptes appris en cours, notamment durant le BTS ou nous croulions littéralement sous la documentation (inutile?), et ou le temps nous venait fréquemment à manquer.

Après cette présentation, il semble difficile de revenir à des méthodes de gestion de projet standard. On comprends mieux pourquoi l’agilité à le vent en poupe !

Il me tarde réellement de voir en action ce que ça peut donner, et j’en aurais d’ailleurs l’occasion puisque mon stage commence la semaine prochaine, et m’apportera — à priori en tout cas — des exemples concrets de ce que peut apporter l’agilité en entreprise, au niveau de la gestion d’équipe.

Python / Django

Une rapide présentation de python, puis une entrée en matière par David sur Django. On y parles de bonnes pratiques, de gain de temps et d’élégance, puis on passe à la réalisation d’une mini application (un twitter like). Ayant déjà eu l’occasion de toucher un peu à Django, je m’attendais à une présentation un peu plus technique, mais difficile d’aller en profondeur en si peu de temps. On à pu parler entres autres d’internationalisation, de templates, de la mise en place d’un modèle ainsi qu’un bref aperçu de l’administration auto-générée de Django. Suffisamment en tout cas pour mettre l’eau à la bouche. On sent que David maitrise son sujet et ça donne envie de creuser plus loin.

Dans l’ensemble

Présenter la méthodologie agile et django lors du même après-midi nous à permis de parler à la fois de gestion de projet et de choix technologiques. Les deux points ont malheureusement été abordés sans véritable liant, puisque il y a eu peu d’interaction entre les deux intervenants. Difficile alors de se rentre compte si Django est adapté aux méthodes agiles (même si on s’en doute un peu).

Globalement, j’ai beaucoup apprécie les interventions, même si j’aurais bien aimé pourvoir rester discuter un peu plus en fin d’après-midi. Ce fut l’occasion de rencontrer en chair et en os et d’échanger (un peu) avec David, Nicolas et Loïc. Çà fait du bien de mettre un visage derrière des pseudonymes!

Enfin, merci au collectif particul.es, et en particulier à Stephane Langlois d’avoir organisé cet évènement. On en reprendrait volontiers, surtout dans un cadre pareil !

Written by Alexis Metaireau

octobre 27th, 2009 at 8:10

Posted in Python, agilité, django

Tagged with , ,