Christoph’s Blog

Rants, discoveries and my tireless search for perfect Linux tools

Archive for the ‘Open-Source’ Category

ExtJS - better luck this time

with 3 comments

You may remember that I blogged about ExtJS before. I was impressed by ExtJS but somehow didn’t consider it for my owngoing projects.

This time I’m working on a project that would normally become a GUI application. Fat client-style. But I’m still wanting to make it a web application because I wouldn’t like to force the dozens of users to install updates frequently. And I’m also a newbie regarding UI toolkits like Qt or Gtk. So I thought I give ExtJS another try.

You can’t hurry love

ExtJS takes some learning. And with the limited literature on the web dealing with programming applications in ExtJS I felt pretty alone. Many tutorials that I found on the web that deal with what I wanted to know only used version 1.x. And 2.x works differently. So I needed support from #extjs (IRC channel on freenode), read parts of the API documentation and the ExtJS forums (why oh why can’t it just have a simple mailing list - web forums stink). With jQuery you will get results quickly. But ExtJS isn’t just giving you a few JS commands and classes to simplify your life. It’s a whole new paradigm really. It’s like you code a GUI application that works on the web. If you use layouts and panels and not just simple DOM-manipulating operations then you may need to take your time before you get something done that actually works. The API documentation is very complete. But the classes are wired together and subclassed in a way that you sometimes have to guess which documentation page describes certain attributes. (Where the heck are the items documented in tree panels again?)

I’m using Pylons as a backend and usually I have 70% Python code and 30% in static Javascript and HTML templates. But with ExtJS the equation is more like 30% Python code (for dealing with backend calls and JSON) and 70% Javascript with virtually no templates. Javascript is by far not as comfortable as Python so it’s a little drawback. But it’s roughly (very roughly) similar to Python so it should feel familiar.

Licensing chaos

Recently ExtJS turned around the licensing terms a little thus annoying many developers in the community. And I must say that I was close to not even considering ExtJS again because they frightened me with their “commercial license”. I’m (fortunately) not a lawyer but I’m pretty sure that the only case where you would need to pay for ExtJS is when you write applications using ExtJS that you use publicly and do NOT want to put your application/web site under the GPL. The important result for me was: as it’s an intranet application within our organisation we do not need to share the source or put the application under GPL. And for any other cases I usually put my projects under an open-source license anyway. Of course life would be easier for everyone if ExtJS just used the MIT license. But the library is of very high quality and very capable. From what I have seen so far I would happily write commercial applications that depended on ExtJS (then paying for the license and support of course). My point is: if you like ExtJS (perhaps even from just enjoying their programming samples) then see if you really need the commercial license.

What ExtJS isn’t useful for

The spectrum of applications where ExtJS is a good catch is a little limited. For example I wouldn’t make a customer web site using Ext unless they need a kind of GUI-style application. For effects and a little AJAX jQuery is much easier to understand and use and it’s MIT-licensed so I wouldn’t really have to worry about legal stuff. And the design is fixed. You would easily spot an ExtJS-based web site because you usually don’t style the widgets. The icons and widgets come in three different colors but that’s it. I wouldn’t want to start playing with a few hundred lines of CSS that ExtJS comes with unless I really have to. It doesn’t even play nicely with my existing style sheets because there’s little they don’t redefine for their own purposes. For example don’t expect LI tags to produce list items with dots in front any more.

Debug hell

Pylons has a nice online debugger. So if anything fails in your Python code then you can comfortably analyse the situation and quickly fix the problem But if most of your work is done in Javascript then you have to debug Javascript instead. Unfortunately most programming errors that I encountered lead to very strange error messages pointing to lines in ExtJS’ main file ext-all.js. Other developers in #extjs recommended that I use the (1 MB large) extjs-all-debug.js file to help me nail down the origin of the error message. But I hardly ever got more useful messages. Clicking on the error messages in Firebug made my PC slow down like an ant in a mud puddle and after half a minute showed me a line of ExtJS internal code that didn’t help me either. So my current strategy is trying out things slowly. And whenever something works I check in my changes into the Mercurial repository I use for the project. Thus I can exactly see what I changed and what might cause the problem. But this is surely not a very modern way to debug applications. Mercurial and Firebug are the only tools to save my life. By the way many of my problems are giving components the same DOM ID as an existing DIV on the web page. ;)

Validation done twice

I have seen other developers’ web applications already where they fully trust the Javascript to do the right thing. They happily send everything that was deemed good by the Javascript code to the backend. Even SQL injections. That’s obviously a bad thing. So you’ll have to validate the data your backend gets. Just be aware that you will need to do the validation twice - in your JS frontend and then again in the backend. Just don’t trust anything that JS does. Whatever you do: validate what ExtJS sends to the backend! Everything else is optional.

Definition versus instantiation

A central thing I had to learn was the difference of defining some “component” to actually instantiating it. It’s not very complicated from an object-oriented point of view. But my first attempt to get a layout and a couple of panels working made me do everything in the doReady() call that is triggered when all referenced elements of the web page are loaded. I hesitate to use derived/subclassed components because it seems too complicated - although other developers recommended to do that. So to save memory and CPU cycles you rather define Javascript arrays containing the attributes that you would later use to instantiate a certain component. Think of it as ‘just in time instantiation’. Creating instances of everything right from the start is a waste of resources and will probably slow down the application.

Does half a megabyte of JS make things fast?

Yes, surprisingly it does. Half a megabyte is the largest amount of Javascript I had ever dealt with. But it’s loaded once and then usually kept in the browser cache. And looking at the start page of my employer’s web site they also use that much Javascript but accomplish much less than ExtJS. ;) The advantage of ExtJS’ approach is that the application runs on the client/browser and not on the server. So everything is ready to use in the browser and the server just needs to send and receive data in the background. No loading times when you switch from one web to another. It becomes indistinguishable from a fat client application.

May the source be with you

Stay tuned for maybe more experiences once I finish my appliation. I hope I keep the patience because it took me three days of readig, trying, understanding, err’ing and programming to finally have a border layout with a tree view, a panel view and a click handler. What helped me a lot was reading the well-documented source from ExtJS’ layout browser and Saki’s web site. And my personal favorite useless but amazing loading effect can be seen here. :)

Written by Christoph Haas

October 16th, 2008 at 8:59 pm

MySQL to PostgreSQL - a Bacula conversion odyssey

with 2 comments

Why is it that always the seemingly most simple things turn out to be the most annoying? This time I “just” wanted to get rid of one of my last MySQL databases and move it over to PostgreSQL. The Bacula catalog that saves which file I backed up when and to which storage medium. I tried with MySQL’s “mysqldump” and it’s PostgreSQL compatibility option - but apparently MySQL developers know nothing about PostgreSQL. Then I tried with “sqlfairy” - and found myself booting my system hard after it ate 2 GB of swap and died while converting 500 MB of data.

So finally I asked in #bacula and was told to try CSV (comma-seperated values) as an intermediate format. Yuck… that satan-spawned format that reminds me of my dark past? Okay. First dump the catalog from MySQL:

for table in BaseFiles CDImages Client Counters Device File Filename FileSet Job JobMedia Location LocationLog Log Media MediaType Path Pool Status Storage UnsavedFiles Version ; do mysqldump -u root -pmypassword -T. bacula $table; done

(Okay, okay, this is not comma- but tab-seperated. But that’s even better for running the COPY-FROM command later.)

This creates an ‘.sql’ (the schema) and a ‘.txt’ (the rows/records) file for each table in the current directory. Just don’t try to apply the schema to PostgreSQL. Instead better create a new schema. Bacula ships with a script for that purpose.

Unless you have the PostgreSQL database for Bacula ready you should run something like…

/usr/share/bacula-director/make_postgresql_tables -h localhost -U bacula

…and…

/usr/share/bacula-director/grant_postgresql_privileges -h localhost -U bacula

Now on to read the tab-delimited data into PostgreSQL. The importing via the COPY command must be done with administrator privileges! And it’s important to explicitly state which columns correspond to which table columns (see the respective ‘.sql’ files) or otherwise you’ll get chaos. Of course this only has to be done for .txt files larger than 0 bytes. Oh, and the filename has to be absolute. Example:

  • psql -h localhost bacula admin -c “COPY basefiles (baseid,basejobid,jobid,fileid,fileindex) FROM ‘/var/tmp/BaseFiles.txt’”
  • psql -h localhost bacula admin -c “COPY cdimages (mediaid,lastburn) FROM ‘/var/tmp/CDImages.txt’”
  • psql -h localhost bacula admin -c “COPY client (ClientId,Name,Uname,AutoPrune,FileRetention,JobRetention) FROM ‘/var/tmp/Client.txt’”
  • psql -h localhost bacula admin -c “COPY counters (Counter,MinValue,MaxValue,CurrentValue,WrapCounter) FROM ‘/var/tmp/Counters.txt’”
  • psql -h localhost bacula admin -c “COPY device (DeviceId, Name, MediaTypeId, StorageId, DevMounts, DevReadBytes, DevWriteBytes, DevReadBytesSinceCleaning ,DevWriteBytesSinceCleaning, DevReadTime,DevWriteTime, DevReadTimeSinceCleaning, DevWriteTimeSinceCleaning, CleaningDate, CleaningPeriod) FROM ‘/var/tmp/Device.txt’”
  • psql -h localhost bacula admin -c “COPY file (FileId, FileIndex, JobId, PathId, FilenameId, MarkId, LStat, MD5) FROM ‘/var/tmp/File.txt’”
    psql -h localhost bacula admin -c “COPY fileset (FileSetId, FileSet, MD5, CreateTime) FROM ‘/var/tmp/FileSet.txt’”
  • psql -h localhost bacula admin -c “COPY filename (FilenameId, Name) FROM ‘/var/tmp/Filename.txt’”
  • psql -h localhost bacula admin -c “COPY jobmedia (JobMediaId, JobId, MediaId, FirstIndex, LastIndex, StartFile, EndFile, StartBlock, EndBlock, VolIndex, Copy) FROM ‘/var/tmp/JobMedia.txt’”
  • psql -h localhost bacula admin -c “COPY location (LocationId, Location, Cost, Enabled) FROM ‘/var/tmp/Location.txt’”
  • psql -h localhost bacula admin -c “COPY locationlog (LocLogId, Date, Comment, MediaId, LocationId, NewVolStatus, NewEnabled) FROM ‘/var/tmp/LocationLog.txt’”
  • psql -h localhost bacula admin -c “COPY log (LogId,JobId,Time,LogText) FROM ‘/var/tmp/Log.txt’”
  • psql -h localhost bacula admin -c “COPY media (MediaId, VolumeName, Slot, PoolId, MediaType, FirstWritten, LastWritten, LabelDate, VolJobs, VolFiles, VolBlocks, VolMounts, VolBytes, VolErrors, VolWrites, VolCapacityBytes, VolStatus, Recycle, VolRetention, VolUseDuration, MaxVolJobs, MaxVolFiles, MaxVolBytes, InChanger, MediaAddressing, VolReadTime, VolWriteTime, EndFile, EndBlock, LabelType, StorageId, VolParts, MediaTypeId, DeviceId, LocationId, RecycleCount, InitialWrite, ScratchPoolId, RecyclePoolId, Enabled, Comment) FROM ‘/var/tmp/Media.txt’”
  • psql -h localhost bacula admin -c “COPY mediatype (MediaTypeId, MediaType, ReadOnly) FROM ‘/var/tmp/MediaType.txt’”
  • psql -h localhost bacula admin -c “COPY path (PathId, Path) FROM ‘/var/tmp/Path.txt’”
  • psql -h localhost bacula admin -c “COPY pool (PoolId, Name, NumVols, MaxVols, UseOnce, UseCatalog, AcceptAnyVolume, VolRetention, VolUseDuration, MaxVolJobs, MaxVolFiles, MaxVolBytes, AutoPrune, Recycle, PoolType, LabelFormat, Enabled, ScratchPoolId, RecyclePoolId, LabelType, NextPoolId, MigrationHighBytes, MigrationLowBytes, MigrationTime) FROM ‘/var/tmp/Pool.txt’”
  • psql -h localhost bacula admin -c “COPY storage (StorageId, Name, AutoChanger) FROM ‘/var/tmp/Storage.txt’”
  • psql -h localhost bacula admin -c “COPY unsavedfiles (UnsavedId, JobId, PathId, FilenameId) FROM ‘/var/tmp/UnsavedFiles.txt’”

Unfortunately my “Job.txt” and “Media.txt” contained datestamp entries like “0000-00-00 00:00:00″ which are not valid for PostgreSQL. So I went into Vim and replaced it: s/0000-00-00 00:00:00/1970-01-01 00:00:00/g. Clear the table (DELETE FROM job) and import again.

And finally it’s important to get the sequence numbers right as described in the Bacula manual.

Morale: spend two extra-minutes to start with PostgreSQL right away instead of bothering about conversions later. And never assume converting from one database to another would work - just because both have “SQL” in their names.

Written by Christoph Haas

September 16th, 2008 at 11:11 pm

Posted in Debian, Open-Source, Rants

mentors.debian.net’s flakiness

without comments

A lot of users of mentors.debian.net have already mailed, irced and jabbered me. (Fortunately nobody rang at my door.) The service was pretty unreliable in the past weeks. If I’m not mistaken it was due to not enough RAM. Unfortunately it’s a VPS server (sponsored for that purpose by vpslink.com) that has 1 GB RAM but doesn’t allow to set up swap for the rare cases where a 100 MB more RAM would save the world. Took me a while to figure out where the infrequent “MySQL server has gone away” came from. That lead to packages either left in the incoming directory or being thrown away randomly during the import into the repository because not all database error conditions have been properly caught.

The better news is that the debexpo project done during this year’s Google Summer of Code is now done. The Pylons-based software was written to suit the need for easy-to-use web-based Debian package repositories like PPAs or debpool. I am currently working on a few subleties of debexpo before it will be used for mentors.debian.net. I intend to announce the beta version of mentors.debian.net in about a week. And I hope for lots of feedback then.

And let’s hope I get the RAM situation improved on the server. The mentors backend was the only software using the MySQL database. With debexpo (which uses SQLAlchemy for database interaction) I’m free to use PostgreSQL or even SQLite.

Update: I should have taken my logs more seriously. The “OperationalError: (2006, ‘MySQL server has gone away’)” was meant seriously. Now it’s also clear why it didn’t happen all of the time. MySQL’s default connection timeout is 8 hours. It’s now set to 3 days so things shouldn’t break unless maintainer’s stop uploading packages for longer than 3 days. Does anyone know if MySQL can be taught to not cancel idle connections? I read rumors about a reconnect=1 argument to Python’s MySQLdb module but I didn’t find it in the docs. This is really a stupid default behavior.

Written by Christoph Haas

September 8th, 2008 at 11:25 am

Mercurial repositories moved to hg.workaround.org

without comments

I have finally found the time to aggregate the chaotic amount of my mighty open-source Mercurial works into http://hg.workaround.org

So if you are missing a repository on other URLs at workaround.org it’s worth a try to see if it’s now there. :)

Written by Christoph Haas

July 9th, 2008 at 11:14 pm

Posted in Open-Source

Fun and despair with ExtJS

with 4 comments

Since I started playing with Javascript about a year ago I became aware of a pretty large number of Javascript libraries that offer additional functions or even widgets for a nicer user interface. I had started with prototype/script.aculo.us because it was included by Pylons. But when I found out that I’m not really bound to it I started looking around in my usual pointless search for “the perfect tool” ™. Finally I stranded with jQuery and I fell in love with it quickly. Give me results quickly and I’m willing to dig into the more subtle parts of something.

Sure, other people told me that jQuery is lame and Dojo is it or YUI (the Yahoo UI library) or even ExtJS. ExtJS? Yes, I looked at that before. Very impressive demos like the “Web Desktop”. I didn’t want to write anything like that though so I enjoyed it as a proof-of-concept and was done with it. But somehow one of my current projects requires a “grid” - that is a table that acts as a paginator (you get shown chunks of data and can navigate back and forth) but allows you to select rows and run actions. Perhaps I would also need a tree view. ExtJS supported all that so I dumped jQuery for a moment and created a (Mercurial) branch of my project to try out ExtJS.

There were instantly a few things I didn’t like. The whole Javascript was 500 KB large. Hardly something I could use on a production web site. Imagine a >500 KB start page. I also had to throw away my CSS for a moment and use their stylesheet - with 850 not-documented classes. Anyway - I surrendered to that and started playing with ExtJS. Strangely loading the page took 10 seconds in Firefox. After some research I found out that there was a bug in FireBug (a Firefox extension that web developers can hardly live without) with large Javascript files. Yes, 500 KB is large. Finally I found a hacked version of it that disabled a few features just to work better with ExtJS.

So finally my application loaded the Ext-all.js (the documented order and list of files I had to include was wrong) and could copy/paste a grid example into my application. One day later it worked. I had defined Stores, Proxies, ColumnModels, Actions, GridPanels and PagingToolbars and after 150 lines of Javascript I finally loaded JSON data from my web service (running on Pylons) and had the grid fill with it. I better hadn’t looked into the generated HTML - made me dizzy looking at all the nested DIVs and TABLEs. My Pylons code was just 5-10 lines though. It was an adventure of surfing through the API documentation of ExtJS that is complete but unusable to me. It defines all possible parameters and options but those are inherited from one another, some aren’t documented because they are “common” and you hardly get an impression on what they mean or how to use them. There were nice examples and everything they did was well documented. But that didn’t help me do more than that. Even doing server-side sorting of the JSON data meant firing up FireBug and watching the parameters - I had expected that to be documented somewhere.
Finally I decided not to work with ExtJS any more. If my application consists of 90-95% Javascript and 5-10% Python/Pylons then I’m not spending my time as intended. Using ExtJS basically means surrendering to it. Just use their library and style sheet and change as little as possible. 500 KB Javascript and 850 CSS styles are pure bloat. And while their license is basically the LGPL they have some weird additional clauses in their license like you can’t use their theme graphics and styles outside of the ExtJS context. I understand that a library of such a quality can’t be maintained by a handful of volunteers. So I see that they want to make money with it if others start making money with it. But somehow it was too much legal stuff for me. I prefer software that is just free. MIT or BSD. Or even GPL - I don’t mind. (Everytime you complain about the complicated licensing you are pointed to their license consultants to convince you. :) ).

It’s a great library. It has great widgets. It even creates HTML that is strictly valid in all browsers I could find. But I was not ready to surrender my application to ExtJS. I wanted a grid to support my application. Fortunately I didn’t have to reinvent the wheel because Matt Knight has recently published his Ingrid grid plugin for jQuery serving me well. And another (ridiculous) point for jQuery: it has mailing lists. ExtJS has a web forum. I hate web forums. Some people say mailing lists are archaic. But I can delete threads or blacklist people or reply properly in a text format. Morale: the flashiest isn’t always the best - neither for the user nor for the developer.

Written by Christoph Haas

January 15th, 2008 at 4:01 pm