Big Dave’s Blog

It’s been a while since I read Longitude by Dava Sobel, but I happened to catch another documentary on the story recently. It reminded me again about this amazing story of science and perseverance. As an engineer, I can’t help but admire John Harrison for his scientific method. It’s also a description of an iterative method. Granted the time periods are years vs. weeks, but the idea is the same.

Before I describe why this is such an amazing story, there needs to be some background on longitude and the longitude problem. Using latitude and longitude, you can describe any location on earth. Both latitude and longitude are expressed in degrees — as an angle from a reference line. Latitude is a vertical position relative to the earth’s equator, and north or south is added to differentiate. Longitude is expressed east or west from the prime meridian, which runs vertically through Greenwich, London.

Latitude is relatively easy to find and calculate. Very simply, if you measure the angle between the horizon and the sun at its highest point (noon,) you can figure out your latitude. A sextant is usually the tool of choice here. Remember, I’m talking about the time before GPS devices.

Longitude is another matter, however. Until John Harrison came around, there wasn’t a practical way to calculate longitude from a ship. Most often, navigators would combine frequent speed measurements and a hourglass. Speed was usually measured by dropping a rope over the stern with a wooden plate on the end. This rope had knots tied at precise distances. Using a 30 second sandglass, a sailor would count the number of knots pulled out over the 30 seconds. This count was reported as “knots.” Hence, you get n knots as a nautical speed (1 knot = 1.85166 km/h.) Using the number of turns of the hourglass and the frequent knot readings, the navigators would attempt to calculate their longitude. Unfortunately, neither of these pieces of data were very accurate. As a result, a lot of ships found themselves aground or sunk due to bad longitude information. This is where the Longitude Prize came from.

The British government offered the prize for a simple and practical method of determining a ship’s longitude. It was a HUGE prize in its day — enough to retire. The prize was administered by the Board of Longitude, and this is where the problem came in. The Board of Longitude was made up mostly of astronomers who were convinced that the sky was the key. John Harrison’s realization was that if you had an extremely accurate clock, calculating longitude was pretty simple. If you had a clock set onboard set to noon at your home port, you can figure out noon on your ship at sea using the sun. If it’s noon on the ship, and your clock reads 2:00pm, you know precisely where you are (think time zones which the same idea.) The trouble was that no one knew how to build a clock accurate enough, especially one that would run on a ship that rocked and was often very humid.

John Harrison worked with wood, so his first few clocks were made of wood with no training in clock making. Some of these clocks still work today — without lubrication due to the natural oils in the wood. At the time, these clocks were considered the most accurate built to date. These clocks were pendulum clocks and of no use on a ship. Still, he invented several techniques and mechanisms still in use today.

Moving from wood to metal and putting himself into the running for the Longitude Prize started him on a 40+ year quest to retrieve the prize. No one ever officially received the prize, but John Harrison did eventually receive a monetary award from Parliament for his achievements. What it took to get there is a fascinating story. It’s one of those stories that you can’t make up.

If you can, check out the book or any of the documentaries. Not only is it a great story, but it’s a great description of an iterative, scientific process to solve a very difficult problem.

How many software packages do you use that give you joy? How about simple utility? How about both? My guess is there are far fewer that provide joy and even fewer that provide both. I have dozens of applications that provide utility only. For example, I use OmniFocus and EverNote constantly. Both apps have become part of my daily routine of getting things done and keeping notes on what’s happening. However, they are both far from joyful. There are times when it’s far from it — “to do” list is is too long or the EverNote iPhone app crashed again. Some apps appear to provide joy, but it’s not the app. It’s the data. If I’m checking the weekend weather, it’s not the app that provided joy. It’s the report of perfect weather that gives me joy.

Joy is pretty easy to get, but I find it usually is limited to games or streaming media apps. My personal geeky pleasures for joy are Moon Globe and Starmap for iPhone. There is no utility in knowing that the orange “star” I see every night when I take my dog out is actually Mars or that clear area on the Moon is where Apollo 11 landed, but it does make me stop and check out the night sky. It’s nice to know where the planets are or what’s up on our planetary neighbor. Yes, it is the data that gives me joy, but I also think it’s the ease with which I can get it also contributes.

The ultimate is to figure out a way to build and application that provides both joy and utility. How many of those are there? The only one that comes to mind is Google Earth. I can spend an hour just checking parts of the world out, but I also use it for gain useful utility — how far to the water from here, etc. For your own applications, you can create joy and utility by first doing something useful, and then make it useable. You still might not create joy, but you at least have a chance.

Any apps out there that provide utility and joy? Your browser is another easy example.

BTW, I’ve been away for awhile. Back in December, I started as CTO of Wimba. Wimba focuses on innovative collaboration solutions that empower educators and engage students. It’s all about collaboration in the education market. It’s kept me a little on the busy side these days, and it has greatly reduced my coding time. It has opened up several new topics to discuss, so stay tuned.

How often have you been involved in a project, and someone starts a statement with “It would be really cool if … ?” The second I hear that, I find myself evaluating what comes next with a high degree of skepticism. First of all, it usually would be “really cool,” but that doesn’t mean we should do it. Too often these ideas solve a problem that you won’t ever have or will not have in the foreseeable future.

YAGNI = You’re are not going to need it

Sure, it would be pretty cool to have a full plugin architecture, but do you really need it now? Let’s gain some traction, iterate, and then we’ll determine if it’s really necessary. Doing it because it’s cool only wastes time if you figure out that the users don’t really care. YAGNI.

Always design for current needs, leaving yourself open for the foreseeable future. Forget about using the crystal ball to guess what your users will want a year from now. It’s far better to get something in front of your users sooner and find out what they really want now. Even if a year from now you have to do a major refactor because something in the crystal ball came true, you will have a user base now and a good reason to make the change. You did iterate to get there, didn’t you?

I recently had an instance where I wanted to add full-text search to an application. I’ve used Lucene, Solr, and a few others in past lives, but this time I wanted something just as functional but a little more lightweight. After looking around I settled on Sphinx, and so far it’s worked great. By itself, Sphinx is not hard to use, but since I’m in Rails, I figured someone must have a gem or plugin for this. Sure enough, I found Thinking Sphinx. Now, it’s really simple.

Let’s get things installed.

To install Sphinx on Linux (See doc for others):

1. Download Sphinx 0.9.8
2. tar xzvf sphinx-0.9.8.tar.gz
3. cd sphinx
4. ./configure
5. make
6. sudo make install

To install Thinking Sphinx:

First, install the gem. There is a plugin available, but I prefer the gem.

sudo gem install freelancing-god-thinking-sphinx \
  --source http://gems.github.com

Add to your config/environment.rb:

config.gem(
  'freelancing-god-thinking-sphinx',
  :lib         => 'thinking_sphinx',
  :version     => '1.1.12'
)

Finally, to make all the rake tasks available to your app, add the following to your Rakefile:

require 'thinking_sphinx/tasks'

Now, we need to use it, but before we do that a brief introduction to some Sphinx terms is necessary. Sphinx will build an index based on fields and attributes. Fields are the actual content of your search index. Fields are always strings. If you want to find content by keywords then it must be a field. Attributes are part of the index, but they are only used for sorting and grouping. Attributes are ignored for keyword searches, but they are very powerful when you want to limit a search. Unlike fields, attributes support multiple types. The supported types are integers, floats, datetimes (as Unix timestamps – and thus integers anyway), booleans, and strings. Take note that string attributes are converted to ordinal integers, which is especially useful for sorting, but not much else.

Thinking Sphinx adds the ability to index any one of your models. To setup an index, you simply add a define_index block. For example:

class Company < ActiveRecord::Base
  define_index do
    indexes [:name, sym], :as => :name, :sortable => true
    indexes description
    indexes city
    indexes state
    indexes country
    indexes area_code
    indexes url
    indexes [industry1, industry2, industry3], :as => :industry
    indexes [subindustry1, subindustry2, subindustry3], :as => :subindustry

    has fortune_rank, created_at, updated_at, vendor_updated_at, employee_bucket, revenue_bucket
    has "reviewed_at IS NULL", :as => :unreviewed, :type => :boolean

    set_property :delta => WorklingDelta
  end
end

Most of this should be pretty self explanatory. To index content (fields), you use “indexes” keyword. As you can see, you can have compound fields by using an array. Note that :name and :id must be symbols or Thinking Sphinx will get confused. You can also use some SQL in your indexes statement.

To add attributes, you use the “has” keyword. Thinking Sphinx is pretty good about determining the type of an attribute, but sometimes you need to tell it using :type.

I will explain the set_property :delta => WorklingDelta later.

To build your index, simply run:

rake thinking_sphinx:index

After processing each model, you will see a message like the one below. Ignore it. Everything is working fine. Really.

distributed index 'company' can not be directly indexed; skipping.

However, if you have made structural changes to your index (which is anything except adding new data into the database tables), you’ll need to stop Sphinx, re-index, and then re-start Sphinx – which can be done through a single rake call.

rake thinking_sphinx:rebuild

Once you have your index setup, you can search really easily.

Company.search "International Business Machines"

This will perform a keyword search across all the indexes for Company. If you want to limit your search to a specific field, use :conditions.

Company.search :conditions => { :description => "computers" }

To use your attributes for grouping and such use :with.

Company.search :conditions => { :description => "computers" },
                                :with => { :employee_bucket => 2 }

With can also accept arrays and ranges. See the doc for more information.

Back to the set_property above. One issue with Sphinx vs. Solr or Lucene is that the Sphinx index is fixed. If you update your model, the change will not be reflected in the index until you rebuild the entire index. To get around this, Sphinx supports delta indexes. A delta index allows you to make a change and have it show up in searches without rebuilding the entire index. Although, rebuilding an index is not a big deal with Sphinx. For example, I can rebuild the Company index defined here in under 2 minutes (1.6 million records).

What does set_property :delta => WorklingDelta do? First, it adds an after_save callback to your model that will use WorklingDelta to perform the delta index step. Given that Workling is in the name you’re probably guessing that I hooked this up to use Workling so delta indexing happens asynchronously.

Add lib/workling_delta.rb:

class WorklingDelta < ThinkingSphinx::Deltas::DefaultDelta
  def index(model, instance = nil)
    return true unless ThinkingSphinx.updates_enabled? && ThinkingSphinx.deltas_enabled?
    return true if instance && !toggled(instance)

    doc_id = instance ? instance.sphinx_document_id : nil
    WorklingDeltaWorker.asynch_index(:delta_index_name => delta_index_name(model), :core_index_name => core_index_name(model), :document_id => doc_id)

    return true
  end
end

Add app/workers/workling_delta_worker.rb:

class WorklingDeltaWorker < Workling::Base
  def index(options = {})
    logger.info("WorklingDeltaWorker#index: #{options.inspect}")
    ThinkingSphinx::Deltas::DeltaJob.new(options[:delta_index_name]).perform
    if options[:document_id]
      ThinkingSphinx::Deltas::FlagAsDeletedJob.new(options[:core_index_name], options[:document_id]).perform
    end

    return true
  end
end

Now, whenever a Company object is created, updated, or destroyed, the WorklingDeltaWorker will be called to update the delta index.

If you have a need to perform powerful searches over hundreds of thousands (or even millions) of records give Sphinx and Thinking Sphinx a try. There are some minor feature omissions, but I think the trade-offs for most applications more than make up for them. BTW, scale is not one of the omissions. The largest Sphinx installation, boardreader.com, uses Sphinx to index over 2 billion records. Craigslist.org is probably the biggest with 50 million queries per day.

July 16, 2009 marks the 40 year anniversary of the Apollo 11 launch. Four days later, July 20, the Eagle landed on surface of the moon. It all started with the famous speech by President Kennedy. It’s still one of my favorite presidential quotes.

“We choose to go to the moon in this decade and do the other things, not because they are easy, but because they are hard, because that goal will serve to organize and measure the best of our energies and skills, because that challenge is one that we are willing to accept, one we are unwilling to postpone, and one which we intend to win, and the others, too.”

If you are at all interested, be sure to check out We Choose the Moon. This site will be doing a real-time replay of the entire mission. They will include all the transmissions as well as a lot of great information. I know I will be checking the site out often. There also are a few Twitter feeds to follow: @AP11_SPACECRAFT, @AP11_EAGLE, and @AP11_CAPCOM. [update: sorry, I needed to fix the Twitter links]

I was too young to really remember the early landings when they happened, but throughout my life, I read and watched everything I could on Apollo, Gemini, and Mercury programs. I can’t get enough of it. It’s one of the things that got me into science and computers.

Now, 40 years after the launch and a lot of software projects under my belt, I often ponder whether we could land a man on the moon in same amount of time. Kennedy made his speech on May 25, 1961, and Neil Armstrong stepped foot on the moon July 20, 1969 — a little over 8 years. Never mind the political climate and government project differences between now and then, I’m thinking about whether having all the computing power available today would have helped or hurt. Think about it. Apollo worked because thousands of very smart people did the work computers would do today (ironically, today it’s in an effort to use fewer people). The problem is that all that smarts would require mountains of code and would be nearly untestable for all possible cases. As an example, Boeing’s 777 has 2.5 million lines of code to deal with one aircraft. Add in the 3rd party code, and you’re talking about 4 million lines of code. That software project took 4.5 years to complete. The problem is that as complicated as the the 777 is, it’s nothing compared to a moon mission. The number of contingencies to handle in software would be crazy. So, you can try to build a brain to handle everything, or you can park a very smart brain in front of a computer and let them make a decision. Which is going to be more flexible?

Now, don’t get me wrong. A project like this would be a dream, but I also understand that if you wanted to do it in 8 years, you would not be able to automate everything. Too much code, not enough time to test it. As another example, the F22 Raptor project started in 1981, and the plane did not go into service until 2005. 24 years — ouch. I still would LOVE to see it happen though. Maybe this time Mars will be the goal.

Let me leave with one of my favorite calculations done for Apollo. Obviously, to step foot on the moon, you have to get there first. It’s close to 250,000 miles from Earth, but thanks to space you don’t have to run your engine the whole way. All you need to do is complete a forward pass from one moving object to another, but be sure to take into account the gravitational affects of at least 3 celestial objects. Newton would be proud.

You’re in orbit around the Earth, start your engine and get yourself to what speed? What direction? Once you turn off the engine, you will coast for about 3 days, and if everything goes right you will be in lunar orbit. Too fast or wrong direction, and you miss and go off into space. Too slow and you end up a pancake on the moon. With not much more than a slide rule (anyone even know how to use one of these things?), the Apollo scientists calculated lunar orbit to the exact second. Not bad. I know it impresses the hell of me every time I think about it.

Thoughts? BTW, anyone know the equations?

I’ve been using Hoptoad for about two months now, and I’m sold. Hoptoad is a product by thoughbot, inc. Previously, I’ve always used the excellent exception_notification plugin. The exception_notification plugin is easy to use, and it works great. However, there are a couple of problems with it.

1. You could flood your inbox if there is a bad problem on a popular page.
2. It’s difficult to collect all the emails and track errors.

Enter Hoptoad. Hoptoad solves both problems, and it is super simple to use and setup. It’s free for one project and two users. If that isn’t enough for you, the premium services are very reasonable.

How does it solve both problems? First, Hoptoad will still send you an email when there is an error. However, it won’t flood your inbox if there are hundreds of the same email. Second, Hoptoad provides a simple web-based system that groups your errors by exception class. That makes it easier to track down problems. Finally, you can mark errors resolved so you don’t have to wade through noise to solve the real problems.

Hoptoad installation:

REMOVE EXCEPTION_NOTIFIER

1. In your ApplicationController, REMOVE this line:include ExceptionNotifiable
2. In your config/environment* files, remove all references to ExceptionNotifier.
3. Remove the vendor/plugins/exception_notifier directory.

INSTALL HOPTOAD_NOTIFIER

From your project’s RAILS_ROOT, run:

script/plugin install git://github.com/thoughtbot/hoptoad_notifier.git

CONFIGURATION

You should have something like this in config/initializers/hoptoad.rb.

HoptoadNotifier.configure do |config|
  config.api_key = '1234567890abcdef'  # You get your key when you sign up
end

(Please note that this configuration should be in a global configuration, and is not enrivonment-specific. Hoptoad is smart enough to know what errors are caused by what environments, so your staging errors don’t get mixed in with your production errors.)

Once you do the above, any Exception not caught by your controllers will be sent to Hoptoad where where they can be aggregated, filtered, sorted, analyzed, massaged, and searched.

Now, if you have read anything I’ve done before, you know I do a lot of asynchronous processing with Workling. By default, Hoptoad will not log errors from anything outside of your controllers. Fear not. Hoptoad provides a webservice API to send errors. Here is an example.

In workling/lib/workling/base.rb:

    # takes care of suppressing remote errors but raising Workling::WorklingNotFoundError
    # where appropriate. swallow workling exceptions so that everything behaves like remote code.
    # otherwise StarlingRunner and SpawnRunner would behave too differently to NotRemoteRunner.
    def dispatch_to_worker_method(method, options)
      begin
        self.send(method, options)
      rescue Exception => e
        raise e if e.kind_of? Workling::WorklingError
        logger.error "Workling Error: runner could not invoke #{ self.class }:#{ method } with #{ options.inspect }. error was: #{ e.inspect }\n #{ e.backtrace.join("\n") }"
        # DND: Let HopToad know of the issue
        params = options || {}
        HoptoadNotifier.notify(
          :error_class => "Workling Error - #{e.class.name}",
          :error_message => "Workling Error(#{e.class.name}): #{e.message}",
          :request => { :params => params.merge(:worker_class => self.class.name, :worker_method => method) })
        # DND: end of change
      end
    end

Now, any error not caught by your workers will be sent to Hoptoad for processing. You can use a similar method from Rake tasks or any scripts that run outside of controllers. Remember, Hoptoad will collect errors by :error_class, so you can use different classes to separate errors into bins based on where they came from.

Give Hoptoad a try. You will not be disappointed.

I’ve hired a lot of engineers over the years, and one of the first things I’m always asked to add to the job description is “must have experience in XXXX platform.” I sometimes put it on there to make people happy, but I rarely will disregard an engineer because they do not have a lot of experience in a particular platform. Give me a Rock Star, or even a very good/great engineer, and I guarantee you that person will run circles around the average engineer with platform experience. Now, if you can find the Rock Star with platform experience — BONUS! The only time I deviate from this plan is if the project calls for someone to “just get it done and fast.” Then I need to pay more attention to platform experience because I don’t have time to wait for the new platform to be learned.

I’ve lost track of how many times this simple fact has been proven to me. The qualities of a great engineer carry over to any platform, and a great engineer will pick up a new platform quickly — mostly because they love learning new things. If you’re starting out in software development, concentrate on being a great engineer. That’s far more valuable than an engineer that knows a platform.

What makes a great engineer? To me, it’s pretty simple. You have a passion and ability to craft outstanding, maintainable, and testable code. You know your algorithms, design patterns, and data structures like the back of your hand. Finally, you posses the other skills necessary to round things out — communication, time management, risk assessment, strategic and detailed design, and quick decision making (and sticking to those decisions). You can write a loop in Java? I don’t really care. You understand dependency injection or how an outer join works. Now we’re talking.

What do you think?

UK Snubs Support for Home of WWII Enigma

Here we go again. Bletchley Park continues to get little love. Here we have what is basically one on of the birthplaces of modern computing. On top of that, the group of people that worked here, along with their US partners in Building 26, did more to shorten WWII and save countless lives than just about any other group. Makes your life as a military leader a whole lot easier when you know what your counterpart is up to.

It seems now that the UK will not give Bletchley Park the same status as Imperial War Museum.

“We have no plans at present to associate it with the Imperial War Museum,” Lord Davies said. “The House is all too well aware of the significance of designating any area in association with a museum of that rank, but I want to give an assurance that Bletchley Park will continue to develop under the resources made available to it.”

OK, I know I’m a little biased because I’m a history buff, but I’m also aware of the history of my profession. Without these two groups of scientists, we might not have the same level of computing we have today. This is where Alan Turing (of Turing Machine fame) cut his chops.

Let’s not forget about these people and what they did. I know it was super-secret, but it was almost 70 years ago now. 

As you probably noticed already, I use Workling a lot, and I wrote about it a few times (Part 1, Part 2, Part 3). One minor gotcha to be aware of is that you need to make sure you handle it when Workling is too fast. A common use of Workling is to make a call in an after_save method. For example:

class User < ActiveRecord::Base
  def after_save
    MyWorker.asynch_geocode_address(:user_id => self.id)
  end
end

class MyWorker < Workling::Base
  def geocode_address(options = {})
    user = User.find(options[:user_id])
    user.geocode_address
  end
end

Pretty straight forward, right? Unfortunately, what you will find is that often your worker will get called so fast on create that ActiveRecord hasn’t committed the transaction yet, and User.find will fail. To get around this, you need to write your workers that could be called from after_save/after_create to handle this condition. Instead of the above, you need to do the following:

class MyWorker < Workling::Base
  def geocode_address(options = {})
    retries = 3
    begin
      user = User.find(options[:user_id])
      user.geocode_address
    rescue => err
      if (retries -= 1) > 0
        sleep(0.2)  # Give ActiveRecord a chance to save
        retry
      end
      raise err
    end
  end
end

Now we catch the exception that ActiveRecord throws when it can’t find the new user yet. Waiting a short time and trying a few times gives ActiveRecord a chance to commit the record.

Enjoy asynchronous processing…

Rails 2.3 added a couple of nice new methods – find_each and find_in_batches. Both methods accomplish the same thing in a slightly different way. Unlike a normal finder these methods grab objects in batches instead of all at once. For instance, if you have 500,000 users, you don’t want to do the following:

User.find(:all).each { |user| user.some_method }

The reason is that you just loaded all 500,000 records into memory, and your server is not happy. Instead, you could do:

User.find_each { |user| user.some_method }

By default, the above will only load 1,000 User objects into memory, and your server will thank you. If 1,000 is too big/small for you, use the :batch_size option to change it. The find_in_batches method is similar except that it provides the array to the block instead of one object at a time. For example:

User.find_in_batches do |users|
  users.each { |user| user.some_method }
end

If you ever used the wonderful will_paginate gem, you are probably familiar with the concept from the paginated_each method that will_paginate provided.

So, what is the problem? The problem is that you have to aware that unlike paginate_each, find_each and find_in_batches work by setting up a with_scope block. Therefore, if you need to do any other finds on that same model, the scope will apply. Usually this only affects relationships, but it isn’t hard to forget. Here is an example:

# This is a purely made up example
class User < ActiveRecord::Base
  # There is a last_login_at attribute
  named_scope :recent_login,
              lambda { |*args|
              { :conditions => ["people.last_login_at >= ?", (args.first || 1.week.ago)] } }
  belongs_to :parent, :class_name => "User", :foreign_key => "parent_id"
end 

User.recent_login.find_each do |user|
  parent = user.parent # This will include the recent_login scope.
end 

It’s no different than other with_scope issues, but it isn’t as obvious. You can get around it by doing:

User.recent_login.find_each do |user|
  # Got use send because with_excusive_scope is protected.
  User.send(:with_exclusive_scope)
    parent = user.parent # This will include the recent_login scope
  end
end

Now, go and be kind to your server with find_each and find_in_batches – just remember the scope.