Archive for the 'Programming' Category

setuptools versioning - wtf?

Saturday, July 19th, 2008

Nowadays I’m taking Debian’s APT software management system pretty much for granted. It probably represents the most advanced and powerful system. So I’m not instantly turned away if I find other systems that don’t work as well. Observing how Windows users struggle with getting the 50 different installed pieces of software to update that all use their own mechanisms thus wasting computing time and hard disk space while often getting annoyed by “Do you want to update now?” requests or sudden reboots is fun.

Anyway, as an addicted Python programmer I obviously had to deal with the “setuptools” already. It’s Python’s pendant to Perl’s CPAN. It can download Python modules with certain versions, satisfy dependencies automatically etc. The central place for Python modules is the Python Package Index (PyPi). Obviously I usually wouldn’t need to deal with the setuptools often because Debian already provides properly packages Python modules but sometimes it’s nice to deploy applications within a virtualenv environment (similar to a chroot with its own set of module versions not conflicting with other versions installed on the system). Still I think that setuptools is pretty pathetic.

  • You can’t uninstall modules properly. This is a no-go - we aren’t in the Windows 95 ages any more.
  • PyPi often just links to the developers’ home pages which tend to go offline when you urgently need to deploy an application and can’t get a certain version downloaded. (Pylons has kindly collected the needed eggs on their download page for that case.)
  • Sometimes certain versions are requested that are no longer available on PyPi that have been released just weeks ago.
  • Outside of the virtualenv you’d have to install modules system-wide which collides with your software management like APT (unless you have an operating system like Windows that doesn’t care anyway).

But the single most pathetic detail about setuptools is their versioning. Gustavo Noronha Silva pointed me to /usr/share/pyshared/pkg_resources.py which contained an explanation of how version numbers are compared:

Strings like “a”, “b”, “c”, “alpha”, “beta”, “candidate” and so on (that come before “final” alphabetically) are assumed to be pre-release versions, so that the version “2.4″ is considered newer than “2.4a1″.

Finally, to handle miscellaneous cases, the strings “pre”, “preview”, and “rc” are treated as if they were “c”, i.e. as though they were release candidates, and therefore are not as new as a version string that does not contain them.

So the sorting goes something like a, b, c, pre, preview, rc, alpha, beta, candidate, final. And “2.4″ is newer than “2.4a1″ but not newer than “2.4f1″? Please people, lay off the drugs. I feel tempted to send that to The daily wtf.

I thought that setuptools and virtualenv is a good combination if a certain Python module is not yet available in Debian. But I think I’d rather spend 30 minutes to create a proper Debian package with a real version numbering system instead of wasting my time with these antics.

Don’t get me wrong. I appreciate the intention of setuptools and easy_install. But the world needs a 2.0 of this work. From what I heard hardly anyone understands setuptools well enough to provide something better. Bummer.

Fun and despair with ExtJS

Tuesday, January 15th, 2008

Since I started playing with Javascript about a year ago I became aware of a pretty large number of Javascript libraries that offer additional functions or even widgets for a nicer user interface. I had started with prototype/script.aculo.us because it was included by Pylons. But when I found out that I’m not really bound to it I started looking around in my usual pointless search for “the perfect tool” ™. Finally I stranded with jQuery and I fell in love with it quickly. Give me results quickly and I’m willing to dig into the more subtle parts of something.

Sure, other people told me that jQuery is lame and Dojo is it or YUI (the Yahoo UI library) or even ExtJS. ExtJS? Yes, I looked at that before. Very impressive demos like the “Web Desktop”. I didn’t want to write anything like that though so I enjoyed it as a proof-of-concept and was done with it. But somehow one of my current projects requires a “grid” - that is a table that acts as a paginator (you get shown chunks of data and can navigate back and forth) but allows you to select rows and run actions. Perhaps I would also need a tree view. ExtJS supported all that so I dumped jQuery for a moment and created a (Mercurial) branch of my project to try out ExtJS.

There were instantly a few things I didn’t like. The whole Javascript was 500 KB large. Hardly something I could use on a production web site. Imagine a >500 KB start page. I also had to throw away my CSS for a moment and use their stylesheet - with 850 not-documented classes. Anyway - I surrendered to that and started playing with ExtJS. Strangely loading the page took 10 seconds in Firefox. After some research I found out that there was a bug in FireBug (a Firefox extension that web developers can hardly live without) with large Javascript files. Yes, 500 KB is large. Finally I found a hacked version of it that disabled a few features just to work better with ExtJS.

So finally my application loaded the Ext-all.js (the documented order and list of files I had to include was wrong) and could copy/paste a grid example into my application. One day later it worked. I had defined Stores, Proxies, ColumnModels, Actions, GridPanels and PagingToolbars and after 150 lines of Javascript I finally loaded JSON data from my web service (running on Pylons) and had the grid fill with it. I better hadn’t looked into the generated HTML - made me dizzy looking at all the nested DIVs and TABLEs. My Pylons code was just 5-10 lines though. It was an adventure of surfing through the API documentation of ExtJS that is complete but unusable to me. It defines all possible parameters and options but those are inherited from one another, some aren’t documented because they are “common” and you hardly get an impression on what they mean or how to use them. There were nice examples and everything they did was well documented. But that didn’t help me do more than that. Even doing server-side sorting of the JSON data meant firing up FireBug and watching the parameters - I had expected that to be documented somewhere.
Finally I decided not to work with ExtJS any more. If my application consists of 90-95% Javascript and 5-10% Python/Pylons then I’m not spending my time as intended. Using ExtJS basically means surrendering to it. Just use their library and style sheet and change as little as possible. 500 KB Javascript and 850 CSS styles are pure bloat. And while their license is basically the LGPL they have some weird additional clauses in their license like you can’t use their theme graphics and styles outside of the ExtJS context. I understand that a library of such a quality can’t be maintained by a handful of volunteers. So I see that they want to make money with it if others start making money with it. But somehow it was too much legal stuff for me. I prefer software that is just free. MIT or BSD. Or even GPL - I don’t mind. (Everytime you complain about the complicated licensing you are pointed to their license consultants to convince you. :) ).

It’s a great library. It has great widgets. It even creates HTML that is strictly valid in all browsers I could find. But I was not ready to surrender my application to ExtJS. I wanted a grid to support my application. Fortunately I didn’t have to reinvent the wheel because Matt Knight has recently published his Ingrid grid plugin for jQuery serving me well. And another (ridiculous) point for jQuery: it has mailing lists. ExtJS has a web forum. I hate web forums. Some people say mailing lists are archaic. But I can delete threads or blacklist people or reply properly in a text format. Morale: the flashiest isn’t always the best - neither for the user nor for the developer.

Security by Obscurity isn’t necessarily bad

Sunday, December 2nd, 2007

Time and again I find myself in a hot discussion with others who claim that security by obscurity is a bad thing. SbO means gaining safety by hiding things. Don’t get me wrong - I love open-source software and although I have never taken a look at the Linux or Firefox source code I trust it way more than Windows and the Internet Explorer (that are even less functional). But it’s with no doubt harder to find security bugs in something that you can’t look into. Playing around with different HTML pages thus finding ways to exploit the application is surely harder with the IE than by analysing Firefox’ source code. And what about your password? You are hiding your password from me. Does it make your data safe? Or the PIN code of your credit card? Or why do people run a knockd to protect access to their servers? Those are just various degrees of security gained by keeping information to yourself. I don’t think there is much difference in whether the secret is a password or some piece of code.

In my humble opinion there are hardly any cases where security is not gained by obscurity. With PGP for example. I believe in mathematical one-way functions. My passwords are way safer in a PGP-encrypted file than by putting them into a file at /usr/lib/nobodyfindsme.so.3. And even with PGP you could argue that it’s just knowledge that someone else does not have that protects your information. But my PGP passphrase is way more complicated than my usual account password. And the private key it protects is insanely large. So I consider these to be an acceptable security measure.
The discussion arised when I received complaints that the source code of the CGIs used for mentors.debian.net was not made public. When I tidied up the code and published it finally I suddenly found a secret password in the code and quickly removed it.
mentors.debian.net hasn’t been hacked in years but now that the sources are publicly available I fear for that a tiny bit. The next projects that will be made open-source will be available publicly right from the start so that people may rather help me close security bugs. It would at least make me feel better.

There is but one argument that I would agree upon: it’s silly to rely on SbO.

kate isn’t just a NOTEPAD.EXE for KDE

Sunday, December 2nd, 2007

I’ve been a VIM lover for at least a decade. Nothing beats it at doing complex text editing. Surely it’s not intuitive (although I frequently claim that VI stands for “very intuitive”) but no other text editor can so quickly repeat the last action or reformat a paragraph or insert text at a certain column in every line etc. I am very fast with it and happily ranted at IDEs like SPE, Eric or WingIDE for having such a bad text editor. They were more like a NOTEPAD.EXE that allowed you to navigate with the cursor keys and insert text where you are. But I have to admit that IDEs have a lot of advantages like code completion (vim doesn’t understand Python - not even with omni-completion), templates or dealing with your repository software.

I love cryptic hardcore tools because once I understand them I’m very fast in accomplishing tasks. But even after a decade of using VIM I don’t really understand its configuration or some more advanced features properly. Recently I tried to tell a Python file that I simply like the textwidth to be 100 by adding

:vim:set tw=100:

at the end of the file. No idea why that didn’t work. The command worked well when typed manually.

In addition many basic features like editing multiple files at the same time became a chaos quickly. Multiple buffers? Nah. Tabs are nice but after a certain amount of open files you have to use the arrows to find your file again. Syntax coloring is nice but customizing it is a pain. VIM is great for editing single files and I will always use it to work on DNS zone files for example. But since I develop Pylons (a Python web framework) applications I have to deal with many open files and complex Python code. I tried “pida” which is very promising but is still lacking. For example it doesn’t remember the window panes as I want to have them arranged and find myself moving around things every time I start it up.

I wouldn’t have expected to hear myself saying that vim is not good for all tasks. I had always condemned editors for not being vim and wasn’t open for all the features that I was missing out. Currently I’m trying out “kate”. I had expected it to be a stupid text editor like NOTEPAD.EXE or joe but although it looks similar it is very powerful. Some things I like about it:

  • Handles multiple open files in sessions gracefully (even shows which files have changed)
  • Shows indentation levels in source code (no more guessing where a Python function ends)
  • Text completion (no worse than what vim can do)
  • Closing brackets automatically
  • Split views
  • Syntax highlighting for all types of text files I care about
  • Proper Python indentation handling (vim doesn’t even add an indentation level after a line ending with “:” unless you use dirty tricks)
  • Code folding
  • Snippets (finally I don’t have to copy/paste templates from my knotes any more)
  • Commenting/Uncommenting of lines without :s/^/#/ magic

Kate doesn’t do all things that vim can. But the common tasks are made simpler and more comfortable. No idea if it will be my new editor. But it appears like other text editors besides vim are also worth being tried.