Ruby Text Search

EDIT: A lot of people asked for a plugin of this library. You can find it on GitHub.
Part of my current project screamed for some basic searching. In typical Ruby development fashion, the first thing I think of is “someone else must have done this already.” Sure enough, there were several, but the one the fit the best for me was the query version found here. It looked perfect, but there were a few problems.

  1. The Wiki messed up a lot of characters, making it difficult to get the code.
  2. A few things didn’t work quite right

Time to fix the problems. First, I needed to get the code onto my disk without spending hours finding all the places where the Wiki messed things up. A quick Google search for “search apply_demorgans” located what I was looking for. Now I had something that basically worked. Unfortunately, a few items did not work as advertised.

The first problem I found was that if you added a searches_on to your model object, you could not override the searchable fields. A quick update to self.searchable_fields solved that problem. The second problem was that if you included another table in your search, the searchable fields were not automatically pulled in as advertised. In addition, searchable fields did not include the table name so any searches with an include didn’t always work (need unique column names). Finally, I updated the search routine to also return a count so you can more easily paginate the results.

That’s it for now. The remaining issue I have is that include only works if the association variable name is the same as the table name. However, if you have an association with a different name or finder_sql, it doesn’t work. I don’t have a huge need for this right now, so I haven’t fixed it yet.

Anyone out there care to chime in?

Here is my final code: [REMOVED] Get the plugin from GitHub.






66 responses to “Ruby Text Search”

  1. macographer Avatar

    Dave, I just tried this and I’m getting a few errors. I was getting complaints about the regular expressions in the process_chunk method, it looks like these characters need to be escaped:/^\+/

    Now that I’ve done that, I’m getting this error: “bad value for range”
    The application trace is:
    c:/ruby/lib/ruby/gems/1.8/gems/activesupport-1.4.2/lib/active_support/whiny_nil.rb:35:in `make_chunks’
    #{RAILS_ROOT}/lib/search.rb:205:in `lex’
    #{RAILS_ROOT}/lib/search.rb:283:in `parse’
    #{RAILS_ROOT}/lib/search.rb:361:in `build_text_condition’
    #{RAILS_ROOT}/lib/search.rb:94:in `search’
    #{RAILS_ROOT}/app/controllers/recipes_controller.rb:83:in `search’

    I’ve got this application hosted, but I haven’t updated it with this search library yet. If it would help you debug this (and you have time). This is far beyond my abilities.

  2. Dave Avatar

    Take a look at the orginal post again. I updated the post so search.rb is downloaded instead of in the post. I didn’t notice that WordPress also changed some of the text when I posted it in. Let me know if this works for you now.
    I’m using Ruby 1.8.6 and Rails 1.2.3.

  3. Hans Avatar

    DaveThanks for the ruby text search plugin. I find it very useful. However, I have problems with using the :only options, together with the pagination snippet you proposed. Have you tested that use-case ?
    I have a search field in a table and a drop down box to restrict the search to one ore more fields. However, when I try to limit the search I got the error
    “…MySQL server version for the right syntax to use near ‘))’ at line 1: SELECT count(*) AS count_all FROM people WHERE (())”
    It seems located to the code row — results = [count(diff), find(:all, search_options)]
    It works ok without the :only option. I have defined search_on :all in the table. I am using windows and Ruby 1.8.6 and Rails 1.2.3.
    Any suggestions ?

  4. Dave Avatar

    Thanks for the catch. It looks like you caught a side effect of me using table names to support includes where the column names are not unique. When specifying :only or :except, you need to include the table name. For example, if you have a people table and you are searching, instead of:‘jones’, :only => [‘last_name’])

    use‘jones’, :only => [‘people.last_name’])

    The reason is that Person.searchable_fields returns all field names with the table name prepended (table.attribute). When you specify :only, you need to also be specific with your attribute names by including the table name.

    Hope this helps!

  5. Hans Avatar

    Thanks – That helped !You really answered directly
    The proposed change made it work, so no are all my tables searchable.
    How about indexing the fields in MySQL ? Would that imporve performance in any important way ?

  6. Dave Avatar

    Indexes are usually a good thing, but you need to weigh the extra cost of writes. If your database does a lot or writes, you will need to balance the number of indexes with write performance. However, if there are a few string columns that you do a lot of searching on, then it’s worth adding an index for those.
    In the end, try a few combinations and tune based on how your application accesses the databases. However, I wouldn’t spend too much time tuning without hard data and enough traffic to make it worth it. It’s too easy to spend hours tuning for something that an end user will never realize.

  7. Roland Avatar

    Do I need to “require”, “include” or otherwise let my application know this file exists? I’m still getting “undefined method `searches_on’ for Medium:Class”
    Thanks for your time.

  8. Dave Avatar

    You do need a require “search” for your classes. For example:
    require “search”
    class Contact < ActiveRecord::Base searches_on :first_name, :last_name, :description end

  9. German Avatar

    DAve, first of all, thanks for this amazing add to the rails search, it works great!! one question i need to ask you, is if you can give some examples of how to use the “order” option, for example to order by the field “title” in “desc” order.
    Thanks in advance! 🙂

  10. Dave Avatar

    You can use order just like in a finder method.‘director’, :order => ‘people.title desc’)

    I usually put the table name before the column to avoid any unique column name errors, but in this case it is not necessary.

  11. Dave Avatar

    Glad I could help.

  12. German Avatar

    thanks dave!! i made it this way: (Publication is the class, and @keywords is an array of strings, @c_order is either ‘title’ or ‘author’, and @c_sort is ‘asc’ or ‘desc’)
    @search_result = @keywords, :include => [:author], :order => “#{@c_order} #{@c_sort}”


  13. Elia Avatar

    Thanks for the code. Very helpful as it seems to be doing the trick for me. One question: I need to add additional conditions to the search. For instance, I need to make sure a user has permissions to see the record. With it, I assumed I would type something like this:[:s], :conditions => [“user_id = ?”,session[:user_id]], …

    where session[:user_id] = 3 and it would tack the appropriate conditions on the end. Instead, I get a SQL error:

    Mysql::Error: #42000You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ‘?3)’ at line 1: SELECT count(*) AS count_all FROM tmps WHERE ((tmps.title like ‘%time%’ or tmps.description like ‘%time%’) AND user_id = ?3)

    I don’t know enough of what I am looking at to know how to fix it but can see that the ? isn’t being replaced as appropriate. Any suggestions?

  14. Elia Avatar

    Answer my own question after thinking about it over night (it makes for some very early mornings): looks like the ? option doesn’t work but the placeholder method does. This worked just fine:
    … :conditions => [“user_id = :user”,{:user => session[:user_id]}] …

  15. Elia Avatar

    I lied. It worked fine if not logged in (session[:id] = nil, which the search will turn up certain things) but when logged in I get a similar error as my post from last night. Would still appreciate your input on this.

  16. Dave Avatar

    You are right. The ? substitution isn’t enabled. The code simply joins your conditions to the generated conditions with an AND. Try this:[:s], :conditions => “user_id = #{session[:user_id]}”)

    This works for me.

    Note, I also always try to use the table names (tmps.user_id). It helps the db, and it saves you from issues later if you add an include to your search.

  17. Elia Avatar

    Thanks, Dave. I am new to all this so forgive my ignorance. Is there a risk with this regarding SQL injection?

  18. Dave Avatar

    There should not be. Nothing is ever sent directly to SQL. The search strings are all parsed out first. However, when you build your conditions strings, you should be careful about injections. I haven’t used the extra conditions much, so I haven’t worried about it yet. For things like you are doing, I’m doing something like:[:s])

    This ensures that I am restricted to only the current user’s tmps. Note, @current_user is always setup by a before filter.

  19. Elia Avatar

    Thanks. Using @current_user doesn’t quite work for what I am doing but normally I would agree.

  20. Andy Avatar

    Dear Dave,
    Thank you vey much for the code. It worked well.

    I just want to give a quick warning to about case sensitivity. If you are using Oracle you need to uncomment the following code:

    # DND: no need for this since mysql is already case insensitive.
    unless options[:case] == :sensitive
    text.downcase!! { |f| “LOWER(#{f})” }

    Thanks again for your help.

  21. Dave Avatar

    My pleasure. Thanks for the comment. I don’t know why I bothered to comment section out since I never would have used that option anyway. I’ve always used MySQL or MS SQL Server, and both of those default to case insensitive searches. Therefore, I pulled it out for my purposes.

  22. Diego Avatar

    Hi Dave, I found the code in the rails wiki and gave it a try… It was giving me a little error with the :includes, but I’ve tested your version and works perfect for me!I felt I should leave you a message: Thank you so much 🙂

  23. Dave Avatar

    My pleasure. Glad I could help.

  24. Ryan Svoboda Avatar

    Thanks for the code, working great so far 🙂

  25. Leif Avatar

    Hi Dave!
    I’m new to Rails and trying to build an small test app which uses your search script. At the moment i get the error

    undefined method `title’ for 0:Fixnum

    I don’t know what went wrong. Any hints?

    Thanks in advance.

  26. Dave Avatar

    I need a little more info. Based on your comment, it looks like there is a number being loaded into something that is expected to be a model. Please update a little bit of your code, and I can take a look.

  27. Leif Avatar

    Controller looks like:
    def search
    if params[:query]
    @offers =[:query])
    @offers = []

    require_dependency “search”

    class Offer < ActiveRecord::Base
    searches_on :all

    {:action => :search} do |f| %>


    View for results:

    There were no results for your query



    Thanks for your help!

  28. Dave Avatar

    Looks about the same as what I do. The only difference I see is that use:
    require ‘search’

    instead of

    require_dependency ‘search’

    Is title an attribute of your Offer model?

    Please dump what the log spits out when you hit the search action. I’m curious to see what it’s doing. Feel free to remove anything you don’t want to be seen, or you can email it to me @ dave [at]

  29. Leif Avatar

    Hi Dave,
    i send you an email.
    “Title” is an field in the table “offers”.


  30. Evan Avatar

    Hi everyone, I’m really new to Ruby on Rails, and I’m getting pretty frustrated trying to implement this search. First, I was getting an error saying

    private method `gsub’ called for {“query”=>”COKE”}:HashWithIndifferentAccess

    which i fixed by changing s.gsub(‘%’, ‘\%’).gsub(‘_’, ‘\_’) to
    s.values.join.gsub(‘%’, ‘\%’).gsub(‘_’, ‘\_’)

    this seems to clear up that problem, but now I get an error saying

    SQLite3::SQLException: near “values”: syntax error: SELECT * FROM “values” WHERE ((values.Symbol like ‘%COKE%’ or values.Start like ‘%COKE%’))

    (values is my object, Symbol and Start are variables, COKE is the query) Now, I don’t know any SQL because I’ve been trying to learn all of this html and Ruby and Ruby on Rails (my background is in Java.) is this problem because i’m using sqlite3 instead of MySQL? Any help would be much appreciated.

  31. Dave Avatar

    Please show me some code from your model and how you are calling the search method. You should not be getting the error you are getting.

  32. Evan Avatar

    This is my entire model-
    require_dependency “search”

    class Value < ActiveRecord::Base
    #choose which columns to search
    searches_on :Symbol, :Start
    # #anything else your model does

  33. Evan Avatar

    This is where the method gets called in my view–
    {:controller=>”values”, :action=>”search”}, :html => { :multipart => true } do |f| -%>

    Search? :

  34. Evan Avatar

    and inside my controller-
    require ‘search’

    def search

    if params[:query]
    @values =[:query])
    search.html # search.html.erb
    format.xml { render :xml => @value }
    @values = []
    search.html # search.html.erb
    format.xml { render :xml => @value }


  35. Dave Avatar

    I see several issues to deal with:
    Your model is not following the normal convention, so I doubt Rails will be able to connect. I expected something more like:

    require “search”

    class Value < ActiveRecord::Base #choose which columns to search searches_on :symbol, :start # #anything else your model does end

    Are symbol and start both string or text type? Search only works on strings and text.

    Do not require search in your controller. It only works in models.

    In your view, you have a multipart form. Is there another reason for that?

    Here is a short form example that I have in my view:

    <% form_tag admin_users_path, :id => “f_user_search”, :method => :get do %>
    < %= text_field_tag "q", @criteria, :maxlength => “256”, :title => “Search Users” %>
    < %= submit_tag 'Search', :or => link_to(‘Reset’, admin_users_path) %>
    < % end %>

    One more thing: I’m not sure but values may be a keyword in SQL, so you can’t use that as a table name.

  36. Evan Avatar

    To be honest, I don’t really know what a multipart form is. I just copied and pasted the html from a csv import tutorial I read so that the search method was called correctly and hopefully routed to the search page. Would you suggest rewriting the entire program to see if changing the name from values to something else fixes the problem? I’m really just overwhelmed by Ruby on Rails right now, so sorry that this explanation is so jumbled.
    My database contains dates under the start field, but text search should still work correctly right?

  37. Dave Avatar

    I’m guessing the multipart form was to upload a CSV file. You don’t need one for what you are doing.
    At a minimum, I would change your model name to something other than Value.

    If the database columns are declared as datetime, then search will not look in there. The columns/attributes must be strings. Searching for dates and times is an entirely different process.

  38. Evan Avatar

    Ok, so I just made another application, and a new scaffold called Profit this time so there is no syntax error. I also changed my view, and I’m searching only on strings. Now, when I run the search I get

    SystemStackError in ProfitsController#search

    stack level too deep

    RAILS_ROOT: C:/ruby/ReturnCalc
    Application Trace | Framework Trace | Full Trace

    c:/ruby/lib/ruby/gems/1.8/gems/activerecord-2.2.2/lib/active_record/connection_adapters/abstract/connection_pool.rb:227:in `current_connection_id’
    c:/ruby/lib/ruby/gems/1.8/gems/activerecord-2.2.2/lib/active_record/connection_adapters/abstract/connection_pool.rb:95:in `connection’
    c:/ruby/lib/ruby/gems/1.8/gems/activerecord-2.2.2/lib/active_record/connection_adapters/abstract/connection_pool.rb:326:in `retrieve_connection’
    c:/ruby/lib/ruby/gems/1.8/gems/activerecord-2.2.2/lib/active_record/connection_adapters/abstract/connection_specification.rb:121:in `retrieve_connection’
    c:/ruby/lib/ruby/gems/1.8/gems/activerecord-2.2.2/lib/active_record/connection_adapters/abstract/connection_specification.rb:113:in `connection’
    c:/ruby/lib/ruby/gems/1.8/gems/activerecord-2.2.2/lib/active_record/base.rb:1715:in `add_lock!’
    c:/ruby/lib/ruby/gems/1.8/gems/activerecord-2.2.2/lib/active_record/base.rb:1634:in `construct_finder_sql’
    c:/ruby/lib/ruby/gems/1.8/gems/activerecord-2.2.2/lib/active_record/base.rb:1490:in `find_every’
    c:/ruby/lib/ruby/gems/1.8/gems/activerecord-2.2.2/lib/active_record/base.rb:589:in `find’
    lib/search.rb:114:in `search’
    app/controllers/profits_controller.rb:6:in `search’
    app/controllers/profits_controller.rb:7:in `search’

    What am I doing wrong? I am hopeless

  39. Dave Avatar

    A couple of things to be aware of. First, I goofed. I updated search and made it dependent on a string method that you may not have. That isn’t your problem here, but it will be. I think part of your problem here is that you have search included in your controller. Also, since it expected that there will be a LOT of items to search for, I assume that you have will_paginate installed as a GEM.
    As an example, I uploaded a fully functional application using search. You can get it here:

    Things to note:

    – The two requires at the end of config/environment.rb
    – The require in app/models/profit.rb
    – The app/views/profits/index.html.erb file – look at the form and the will_paginate call at the end.
    – The index method in app/controllers/profits_controller.rb.

    The app is using sqlite3, so you should not have to do anything special for your database.

  40. Evan Avatar

    Thank you so much for the example Dave. Actually seeing some basic code really helped me a lot, and I have almost all the functionality I was looking for now. Thanks so much!

  41. Dave Avatar

    You are very welcome.

  42. André Avatar

    I’m trying to make use of your search library for a project. I’m fairly new to rails unfortunately…
    In ruby on rails wiki, you have this:
    # Step 1: setup the pages
    page = (params[:page] ||= 1).to_i
    items_per_page = 20
    offset = (page – 1) * items_per_page

    # Step 2: get the data
    @num_people, @people = criteria,
    :include => [:company],
    :offset => offset,
    :limit => items_per_page,

    # Step 3: Create the paginator
    @people_pages =, @num_people, items_per_page, page)

    Can you post an example of how to use the @people_pages to display the actual data?
    (Can I use something like will_paginate to create buttons to the next and previous pages?)

  43. Dave Avatar

    Since the wiki posting, I’ve updated search to include support for will_paginate. Interestingly, I just posted an example 3 comments up that shows how to do this with will paginate. See my previous comment (1/7), and check out

  44. André Avatar

    Yes I saw it, and my apologies for bugging you even after you’ve posted that example.The thing is that I’ve tried to move the two files you have under lib on that project (search and extensions) to the project I was developing with your previous version of search. On that project, I already had will_paginate has a plugin (script/plugin install …). But this moving operation resulted in a error:
    undefined method `strip_punctuation’ for “note”:String

    Unfortunately, I should have seen your environment.rb, which “requires” extensions. It’s just a matter of copying this line and all gets fine.

    So, let me thank you doing this small search library, which has turned out to have many uses for me:D

    Before leaving you :), please tell me:
    – extensions.rb was written by you?
    – if you continue to develop the library, where should I look for updates? here, in this blog?

    Thank you very much for the good work.

  45. Dave Avatar

    extensions.rb was written by me. There are several other methods as well, but I wanted to keep things simple for this example.
    I will continue to develop the library. If I get some time this weekend, I will turn it into a plugin and put it up on github. Of course, before I do that I need to add some more tests. I have some but not enough to publically release as a plugin.

  46. D. Claussen Avatar
    D. Claussen

    Dave – it looks like the make_chunks() method blows up if you pass it a string with a trailing space, like this “dogs “.
    The problem is with this bit of code around like 146:

    next_interesting_index = (s =~ /\S/)
    s = s[next_interesting_index..-1]

    To solve it, I added this as the first line of the method:
    s.rstrip! if s

  47. D. Claussen Avatar
    D. Claussen

    One other thing, here’s the stack trace that happens when it blows up due to the trailing space: [RAILS_ROOT]/lib/search.rb:149:in `make_chunks’
    [RAILS_ROOT]/lib/search.rb:207:in `lex’
    [RAILS_ROOT]/lib/search.rb:285:in `parse’
    [RAILS_ROOT]/lib/search.rb:355:in `build_text_condition’
    [RAILS_ROOT]/lib/search.rb:94:in `search’

  48. Dave Avatar

    Yes, there is a problem with that. I have a fix in the latest version that I did not update. I plan on adding a plugin with my latest fixes. I also found that extra spaces between the keywords caused some issues. As a result, I extended String with:
    def strip_extra_whitespace
    self.gsub(/\s+/,’ ‘).strip

    Then, at the end of self.make_chunks, I modified it to be:! { |chunk| chunk.strip_extra_whitespace }

  49. Christophe Avatar

    First, sorry for my bad english (I’m from Belgium)
    I try your app. (profit) and its works great!

    I have a question:
    Is it possible for a tutorial for a advanced search width a pulldown-menu, so you can search in a subCategories (ex. books, dvds, …).
    Based on your example?

  50. Dave Avatar

    I don’t think a tutorial like that would be able this search library. It would be about using pull-downs. Essentially, you would call search using the appropriate model object. For example:[:q])[:q])

    Getting to there would be a tutorial on how to handle controllers and views.

  51. André Avatar

    Hey, in the version that you have in the “profit” example, can I override the value that will paginate uses to limit the results?

  52. Dave Avatar

    Yes, use the standard paginate options.
    1. Add :per_page as an option, and that will be passed on to paginate.
    2. You can also set it in your model like so:

      class Post < ActiveRecord::Base
        cattr_reader :per_page
        @@per_page = 50
  53. André Avatar

    One last question:
    I’m doing a generic search controller, searching over multiple resources/models, using your library. So i’m looping over all the models and doing a =, b = …

    In the end, I wanted to merge all the results, @result = a + b + c… and paginate it, to present them in the view, mixed together…

    I have done some research, and discover that will_paginate supports paginating arrays, so I could do @results.paginate(…).

    My doubt is due to the fact that your search method returns paginated results already.

    Can I still do what I want? What would you advise me to do?

  54. Dave Avatar

    I’m guessing that the models you are searching over are not related. Correct? If not, then you will have to combine the result arrays manually in order to use paginate. Your best bet it to check out the will_paginate code for WillPaginate::Collection and see what’s it is doing. I haven’t done it, but it looks like you can pull the array out of each collections, combine them, and the create a new collection for pagination.
    Please not, however, that you are quickly entering the territory where you should really be looking into Lucene, Ferret, or some other indexing scheme. It would be much better to have an index of searchable items that references all your models rather than search each model individually.

  55. Eyal Avatar

    Hi Dave. I have a pretty simple rails app set up and I want to add search functionality. I’ve included the code in my lib folder and my “Charity” method looks like
    require_dependency “search”

    class Charity :contacted, :dependent => :destroy
    has_many :projects, :dependent => :destroy

    searches_on :all

    def self.charities_options
    Charity.find(:all).collect {|p| “#{}”}.join(“”)

    Charity.find(:all, :conditions => [‘name LIKE ?’, “%#{query}%”])


    But if i’m in my console and I type:“Amy”)

    I get the result with”Amy’s Charity”. However, if I search“Amy “) or try to use the “or” or “-” or parentheses then I get no results. Also, if I search“”) I get all the results instead of none.

    I have included extensions/all and will_paginate in config/environments.rb.

    Any idea what I’m missing?

    Thanks so much,

  56. Dave Avatar

    Your Charity class defined a method, and that overrides the search method in search.rb. If you want to use search.rb, you do not need to define your own search method. search.rb will add it for you by updating ActiveRecord::Base with a few extra methods, including
    What are you using self.charities_options for?

  57. Eyal Avatar

    Of course! Thanks so much. I should have seen that. self.charities_options is for the early stages of my development when I only have a few charities and I want to create a drop down select menu.
    Thanks again!

  58. Dave Avatar

    No problem. Glad I could help.

  59. Eyal Avatar

    Hi Dave, another question:
    I noticed that in both my app and your profit app, a search on only one character gives me back all the results in the table.

    As in“$”) == Profit.find(:all). Any idea why this is?


  60. Dave Avatar

    Good catch! In my production apps, I require more than one character before I allow a search, so I never saw or tested this case. I can’t say exactly why it happens, but I have some ideas. I’m guessing that something about make_chunks doesn’t like a single character. I also know articles and some characters are removed (punctuation, etc.). If you remove everything, you end up with a where clause that is something like “table.col like ‘%%’, and that will match everything.
    The question is whether it would a valid case to do a full text search for a single character. Normally, it’s no, so I would probably lean towards handling conditions being blank the same as the search text being blank and return an empty collection instead of everything.

  61. Josh Avatar

    Great work here BUT it looks like you have added a lot since the last posting of your files.

    Did you ever turn this into a plugin and, if not, where can we get the latest and greated?



  62. Dave Avatar

    I haven’t gotten around to turning this into a plugin yet because I don’t think I have enough tests. Been working on 3 other projects so haven’t had the cycles to give this what it deserves. Let me see what I can do tonight.

  63. Dave Avatar

    OK. I quickly wrapped up what I have as a plugin. You can get it on GitHub. See From the links there, you should be able to easily install search as a plugin.
    As I mentioned, there are no real tests in there. If I get a chance, I will add some.

    Don’t forget to remove your old version of search.lib.

  64. jerry Avatar

    Hi does the plugin include all methods that are needed? Iam getting errors about unknown methods. I think the missing methods were inside the extensions.rb file before .
    Below is the line that gives me errors:

    when /^(\w|(#{String::ARTICLE_WORDS.join(‘|’)}))$/

  65. Dave Avatar

    That’s what happens when there are not enough tests. You miss something. Sorry about that. I tried things in several of my other projects, and everything. However, all those projects had ARTICLE_WORDS define. I will fix ASAP.

  66. Dave Avatar

    I added ARTICLE_WORDS to the github repository. Sorry about that.

Leave a Reply

Your email address will not be published. Required fields are marked *