Sphinx FAIL

On my recent Rails project I decided to try Sphinx search engine. Before that I used Ferret and then - SOLR. I abandoned Ferret because it's instability and lack of tools to track problems (like Luke - index browser for Lucene).

So, Sphinx. I have found two plugins for Rails - acts_as_sphinx and Unltrasphinx. Ok, I've heard in mailing lists that Ultrasphinx is better (and Sphinx recipe in Advanced Rails Recipes book also involves it) so I decided to use it.

First I have to install Sphinx from sources because version in MacPorts (although, latest released version) is too old and Ultrasphinx requiresa newer one (at that point - release candidate 2 of next revision, which is 0.9.8rc2 vs 0.9.7). Then I had to do various dances around Sphinx to compile it on my Mac OS X (which are described in my previous post.).

And then it began:

1. It was needed that new data (e.g. new articles) could be searched with Sphinx right after it was added. And then I find out, that it is not encouraged to do such updates often, you'd better run full updates once a day... Wtf? I have heard that there are something called "deltas", but as I found out from plugin, it doesn't install any hooks on models, so I assume that deltas need to be built periodically (which is also unacceptable).
After consulting with other people who have used Sphinx I found out that:
1) they don't use plugins and do all communication with some low level library (Riddle, as far as I remember) manually.
2) they install their own hooks on models and call indexer manually to reindex their models. I have tried to to install after_save hook but it is run BEFORE transaction is commited and indexer can't see inserted/updated data, so I don't know how this can be accomplished easily.

2. Sphinx configuration is one big scary thing. Although Ultrasphinx managed to build it for me, I needed to do some tweaks to it and then next time I needed to build configuration for another model my tweaks were lost.

3. Model on which I have used Ultrasphinx (called ::is_indexed) failed to load by automatic dependency loader. After several hours of tracking this problem I stuck "require 'my_model'" into environment.rb (which helped) and cursed that Sphinx and all it's plugins.

So, for me Sphinx definitely FAIL.

PS
I have tried SOLR and it worked like charm:
1) almost no configuration.
SOLR configuration consists of type definitions (which describe how value of that type should be analyzed), field definitions and dynamic field definitions (acts_as_solr rails plugin uses only dynamic fields). Acts_as_solr comes with default SOLR configuration which contains commonly used types and dynamic field definitions for them.
2) no compilation needed. Acts_as_solr comes with SOLR JAR files, so you just need proper version of Java runtime installed.
3) works like charm. Everything you would expect from full-text search engine.

Installing Sphinx on Mac OS Leopard

One of my ongoing projects required using Sphinx search engine. While I'm more comfortable with Lucene based solutions (I'm evaluating SOLR currently), I decided to give Sphinx a try.

I have installed Sphinx via MacPorts system and was going to use UltraSphinx Rails plugin. But when I tried to run some test query, it complained, that server version is older than the client version. So, I needed to update Sphinx as UltraSphinx depends on some development release of Sphinx. But when I downloaded sources, configured and issued "make" command, the build failed because there were link problems with iconv library:

Undefined symbols:

  "_iconv_close", referenced from:

      xmlUnknownEncoding(void*, char const*, XML_Encoding*)in libsphinx.a(sphinx.o)

  "_iconv", referenced from:

      xmlUnknownEncoding(void*, char const*, XML_Encoding*)in libsphinx.a(sphinx.o)

  "_iconv_open", referenced from:

      xmlUnknownEncoding(void*, char const*, XML_Encoding*)in libsphinx.a(sphinx.o)

ld: symbol(s) not found

I saw some posts on how to overcome and they suggest downloading and installing iconv library. But I was strongly against doing such things (and also iconv library installed on my system was the latest one), so after some trial-and-error I have figured how to build it: just give standard system's library path precedence:

export LDFLAGS="-L/usr/lib"

./configure

make

sudo make install

Hope this will help somebody.

On Prototype vs jQuery

I see more and more posts like this or this recently and I always see that when people are telling about how poor the Prototype framework is they really don't know even half of it's possibilities.

As it always happens, the code comparison in those posts is written by person who is not experienced in one framework or another.

I agree, that jQuery has some nice features: 1) actions on groups of objects (that you have mentioned), 2) more powerful selectors: prototype has support for css3 selectors only, but jQuery can handle more sophisticated queries like selecting hidden elements or inputs of particular type.

I haven't dug into what it takes to write custom effect or plugin (plan to do it soon), but I don't see how you can base your effects on existing ones in a way other than aggregating/compositing (the thing with Prototype effects is that you can inherit your effect from some base effect and have some nice features like configurable 'easing' function, which can control effect speed over time, or some other stuff for free).

But seeing syntax comparisons like that just makes me mad.
First, adding a class name to element in Prototype is just as easy as in jQuery:

  $('element').addClassName('className')
  $('element').removeClassName('classsName')

Second, the Ajax syntax. Prototype's variant is designed to do POST requests by default. You don't need to specify 'method' parameter if you do POST request. As for GET requests, in Prototype you can do just as in jQuery:

  new Ajax.Updater('element', url + '&' + parameters)

(I believe, the '&' thing is needed in jQuery also, but author of code comparison just 'forgot' to mention that)

Moreover, if you don't like the wordy syntax of Ajax.Updater, you can extend Element to support whatever syntax you like:

  Element.addMethods({
    load: function(element, url) {
      new Ajax.Updater(element, url)
    }
  })

and then you can do

  $('element').load(url + params)

just as in jQuery.

And, yes, Prototype doesn't handle working with sets of elements by default. To add a particular class to a set of elements in Prototype you need to do some extra stuff, but not as it is mentioned in code comparison:

  $$('.element').invoke('addClassName', 'className')

And yes, I agree, that Prototype and Scriptaculous have bad documentation and I personally have bought a book on them (see Pragmatic Programmers bookshelf) and my problems with Prototype and Scripaculous have gone since then.

I don't see any big reason to switch to jQuery.


PS There are lots of other javascript frameworks like mooTools (if I spell it correctly), Dojo, Rhino and such and I always haven't paid much attention to those, because usually my web application require custom solutions, not those that are present in such frameworks. But recently I came across some book ("Mastering Dojo" published by Pragmatic Programmers) and a podcast on it and it got my attention. They say that lot's of fortune 500 companies are using their framework because it is very solid and stable. And I'm curious about what particular features of Dojo make them think so. I have visited Dojo's site and tried to see at some of the examples (e.g. fish eye component) and they were running way too slow on my latest Firefox 2.x (which I consider as rather modern web browser). Probably, I need to give it one more chance and, probably, read that book.

ActiveRecord's database shortcut methods

I see more and more people nowadays complain about other people using destroy_all to delete a bunch of records without understanding the performance decrease compared to using delete_all.

Well, I'll tell you what: in general they do the right thing. My opinion is that having such shortcut methods like delete_all (or update_attribute that issues an update command without triggering validation/callbacks) are evil. While ActiveRecord tries to provide a foundation for building your domain model (smart model, not just containers for data) those methods allow to mess everything up. E.g. you have a model, and when you decide to provide a means to delete all objects, you use delete_all because of better performance than destroy_all. Later on you decide to add some on_destroy callbacks, and then you find out that you need to change every call to delete_all to destroy_all.

Calling delete_all just breaks an invariant for your objects.

What it would be nice is to optimize destroy_all to just call delete_all if nothing else is required. The main reason to call destroy_all is the on_destroy callback. So, ActiveRecord could check if there is any callbacks and if there is no callbacks, then just call delete_all operation.

The same could be done for updates (and it was discussed for a long time): update only those fields that have changed. Then, instead of calling "special" method to update just one field, you will just update the required field and call regular #save. AR will detect all changed fields and issue an update command for just those fields.

How this can be implemented ? The ideal solution is to keep a set of "unmodified" data and compare object's data to that "unmodified" set on save. But on big objects it could require two times more memory. Instead, it could be easier to track what fields were assigned values to and set "changed" flag for that field. If the flag wasn't yet set and the assigned value differs from the one that it is now, then set the flag. If the flag was already set, then don't update it. Of course, it will sometimes update the field to the same value if you first set it to some other value and then set the value that was before. But I believe that such cases are rare and it is not a problem if we update it with the same value.
Storing a boolean flag per database column is not an issue also.

Giles Bowkett: IRB: What Was That Method Again?

Giles Bowkett: IRB: What Was That Method Again?
why should someone bother adding just another method to do stuff that is easily done with existing methods ?
"any arbitrary string".methods.grep("ch")