Big Dave’s Blog

UK Snubs Support for Home of WWII Enigma

Here we go again. Bletchley Park continues to get little love. Here we have what is basically one on of the birthplaces of modern computing. On top of that, the group of people that worked here, along with their US partners in Building 26, did more to shorten WWII and save countless lives than just about any other group. Makes your life as a military leader a whole lot easier when you know what your counterpart is up to.

It seems now that the UK will not give Bletchley Park the same status as Imperial War Museum.

“We have no plans at present to associate it with the Imperial War Museum,” Lord Davies said. “The House is all too well aware of the significance of designating any area in association with a museum of that rank, but I want to give an assurance that Bletchley Park will continue to develop under the resources made available to it.”

OK, I know I’m a little biased because I’m a history buff, but I’m also aware of the history of my profession. Without these two groups of scientists, we might not have the same level of computing we have today. This is where Alan Turing (of Turing Machine fame) cut his chops.

Let’s not forget about these people and what they did. I know it was super-secret, but it was almost 70 years ago now. 

As you probably noticed already, I use Workling a lot, and I wrote about it a few times (Part 1, Part 2, Part 3). One minor gotcha to be aware of is that you need to make sure you handle it when Workling is too fast. A common use of Workling is to make a call in an after_save method. For example:

class User < ActiveRecord::Base
  def after_save
    MyWorker.asynch_geocode_address(:user_id => self.id)
  end
end

class MyWorker < Workling::Base
  def geocode_address(options = {})
    user = User.find(options[:user_id])
    user.geocode_address
  end
end

Pretty straight forward, right? Unfortunately, what you will find is that often your worker will get called so fast on create that ActiveRecord hasn’t committed the transaction yet, and User.find will fail. To get around this, you need to write your workers that could be called from after_save/after_create to handle this condition. Instead of the above, you need to do the following:

class MyWorker < Workling::Base
  def geocode_address(options = {})
    retries = 3
    begin
      user = User.find(options[:user_id])
      user.geocode_address
    rescue => err
      if (retries -= 1) > 0
        sleep(0.2)  # Give ActiveRecord a chance to save
        retry
      end
      raise err
    end
  end
end

Now we catch the exception that ActiveRecord throws when it can’t find the new user yet. Waiting a short time and trying a few times gives ActiveRecord a chance to commit the record.

Enjoy asynchronous processing…

Rails 2.3 added a couple of nice new methods – find_each and find_in_batches. Both methods accomplish the same thing in a slightly different way. Unlike a normal finder these methods grab objects in batches instead of all at once. For instance, if you have 500,000 users, you don’t want to do the following:

User.find(:all).each { |user| user.some_method }

The reason is that you just loaded all 500,000 records into memory, and your server is not happy. Instead, you could do:

User.find_each { |user| user.some_method }

By default, the above will only load 1,000 User objects into memory, and your server will thank you. If 1,000 is too big/small for you, use the :batch_size option to change it. The find_in_batches method is similar except that it provides the array to the block instead of one object at a time. For example:

User.find_in_batches do |users|
  users.each { |user| user.some_method }
end

If you ever used the wonderful will_paginate gem, you are probably familiar with the concept from the paginated_each method that will_paginate provided.

So, what is the problem? The problem is that you have to aware that unlike paginate_each, find_each and find_in_batches work by setting up a with_scope block. Therefore, if you need to do any other finds on that same model, the scope will apply. Usually this only affects relationships, but it isn’t hard to forget. Here is an example:

# This is a purely made up example
class User < ActiveRecord::Base
  # There is a last_login_at attribute
  named_scope :recent_login,
              lambda { |*args|
              { :conditions => ["people.last_login_at >= ?", (args.first || 1.week.ago)] } }
  belongs_to :parent, :class_name => "User", :foreign_key => "parent_id"
end 

User.recent_login.find_each do |user|
  parent = user.parent # This will include the recent_login scope.
end 

It’s no different than other with_scope issues, but it isn’t as obvious. You can get around it by doing:

User.recent_login.find_each do |user|
  # Got use send because with_excusive_scope is protected.
  User.send(:with_exclusive_scope)
    parent = user.parent # This will include the recent_login scope
  end
end

Now, go and be kind to your server with find_each and find_in_batches – just remember the scope.

Gizmodo – Why We Need Audiophiles – Audiophiles

This is a great read for anyone that is interested in great sound. I’ve said it a few times, I’m not into expensive audio gear because I like gadgets (although that part is fun too). I’m into expensive gear because I flat-out love music, and I want to hear it like the artist meant it to be heard. I will tolerate MP3′s on my phone, but not when I want to hear some really choice music played like it was meant to. Sadly, between my kids and my schedule, I do not get near enough time for pure listening.

This is Michael Fremer. He’s listening to “Avalon” by Roxy Music on his $350,000 stereo system. It sounds excellent. He’s a bit crazy, but if you love music, you need him.

BTW, “Avalon” is one of those pieces of music that screams for high quality. My system isn’t quite in the league of $350k, but it does pretty well.

I think the following quote sums up my thoughts exactly:

Because the thing is, Fremer loves music first and foremost. The audiophile I had feared was one who cares far more about the overpriced gadgetry than the actual music. This is not who I ended up meeting. This man listens to music and makes sure it was recorded with the best fidelity, that the intents of the artist have been preserved. And thank God he does, because we certainly don’t.

If you ever get a chance to listen to a favorite, well-recorded piece of music on a choice system, take it. You may be surprised at the subtleties and feeling you are missing. Be careful though. Many recent recordings are mixed for MP3 players and will sound pretty bad on a great system. The really scary part is that some recent “Remastered” versions of classic albums were mastered for MP3 players as well. Many times, it is better to stick with the original.

In a future post, I will describe what I built/coded to control my entire media system from one remote. Note, since I have small children, all my equipment is in a locked closet.

Testing code is good, right? No! Testing code is bad. Testing functionality is good. How many of us have written a bunch of code and then turned around and wrote a bunch of tests for that code? Be honest! Everyone has done it, but why is it a bad thing? The problem with this is that far too often you end up testing what the code DOES, not what it SHOULD DO. This has always been one problem with testing your own code. You assume your code is correct, so tests are written that validate the code. Unfortunately, all this does is keep you honest in case you change the code to cause the test to break. You have not proven that the code does what it’s supposed to do.

Here is a quick example in RoR:

app/models/user.rb:
class User < ActiveRecord::Base
  def unsubscribe!
    add_role(Role::UNSUBSCRIBED)
    forget_me # this will save
  end
end

test/unit/user_test.rb:
class UserTest < ActiveSupport::TestCase
  test "should unsubscribe" do
    user = users(:dave)
    assert user.unsubscribe!
    assert user.unsubscribed? # Assume I have a test for this as well
    assert_nil people(:dave).remember_token
  end
end

Great, we have a test that verifies that the user is unsubscribed. The problem is that when a user is unsubscribed, they SHOULD have had all their email preferences turned off as well. The code is wrong, but the tests will not catch that because the test only verified the code as written.

This is where Test Driven Development (TDD) comes in. With TDD, you write the tests for the functionality you want to happen. Once you described all the desired functionality as tests, you write the code necessary to make all the tests pass. Now, you’re testing what the code should do and not what it does. I’m on board with this, but there are a few problems in the real world.

The biggest problem that I have with TDD is that it assumes you know what the code should do in the first place. Unfortunately, when you innovate you don’t always know what you will end up with. You’re experimenting, trying new things, seeing what sticks. This is normal for many projects. Still, you need to have tests to verify things once you do settle on a direction.

How does this fit into TDD? It doesn’t fit directly because you aren’t writing tests first. What I do in this situation is the following:

1. Once I settle on a direction and set of functionality, I attempt to write the tests for what I have without looking at the implementation code. Write the tests to verify your intended results rather than looking at the code to see what it does. I’m trying to avoid the perils of testing the code.
2. Now that I have a direction, I can go back to writing the test followed by making the test pass.
3. If a problem is discovered anywhere, I write a test to reproduce the issue before fixing it in the code.

The other issue I have with TDD is how do I write code to verify what code should do if I don’t know how it is supposed to do it? When writing the tests, I have to leave many method calls blank until I figure out how things are done and what I should check. Often, I go back to the above until I have a set of standard methods to use, or better yet I let the tests determine what the methods should be. Either way, the above scheme still works for me.

This hybrid approach works for me, but YMMV. The bottom line, as always, is that the code isn’t done until the tests are done as well.

Now, go write some tests!

I’ve been using Apache + Mongrel for a while now. I read about Passenger when it first came out, but I never felt the need to bother with it since I have a bunch of recipes for Apache and Mongrel (see a previous posts about background tasks with Workling). The other day, I figured I would give it a shot. Now, I think I found my new way to deploy Rails apps. It is so simple, and it works well.

Some people write off Passenger because it uses Apache and not Nginx. Nginx is a little faster, and it is the latest “best practice,” but I haven’t had issues with Apache in forever. As for the speed issues, I don’t see it being a big factor to the end users. A few milliseconds is not going to be noticeable, at least for my apps.

Installation could not be simpler.

1. sudo gem install passenger
2. sudo passenger-install-apache2-module
3. Follow the instructions the previous script displays
4. Restart Apache
5. Done!

If your development machine is OSX, don’t bother to manually change the Apache conf for your application and your hosts file just yet. You can use the Passenger Preference Pane instead. This little beauty will make all the changes to your Apache config files for you. One of the benefits is that you can have dozens of small Rails projects configured, and all are available with their own url — no need to run script/server for each. For install, the site explains it pretty well, so I won’t go into it here.

I did run into one problem on my Leopard machine. My default Apache config file had the httpd process running as the _www user, but my project files are owned by me. The result is that I kept getting permission errors accessing files in my project. I’m sure there is a solution to the problem by changing the permissions on my project files, but I didn’t find it. I ended up running httpd as me, and that fixed it. I did not have this issue on my test or production systems.

Even with the problem, the entire process took me less than an hour. What I gained is the following:

• I no longer have to run script/server every time I switch to a new application for development.
• All of my apps on my dev machine have their own url (myapp.local, etc.) instead of localhost:3000.
• My god.rb config no longer has to watch Mongrel processes. It’s all in Apache.
• No Mongrel cluster or load balancing in Apache configs among multiple Mongrels. Apache just operates normally, and my config files are a lot simpler.
• Mongrel hasn’t had any updates in while, but Passenger is still under active development.
• An added benefit is that my test/staging server performs better since only the applications being used are running. By default Passenger will drop any process not accessed in 300 seconds (you can change that).

All of the required changes involved removing any references to Mongrel in my god.rb config, apache config, and my deploy.rb. The only thing I needed to actually change is how to start/stop/restart my application. Remember, there is no mongrel process to start or stop. It’s all Apache. Passenger makes it easy, however. All you need to do is touch tmp/restart.txt. If that file is modified, then Passenger will restart your application.

My updated config/deploy.rb:

namespace :deploy do

  desc "Start the application"
  task :start, :roles => :app do
    run "touch #{current_release}/tmp/restart.txt"
    sudo "god start listeners"
  end

  desc "Stop the application"
  task :stop, :roles => :app do
    # Do nothing for application
    sudo "god stop listeners"
  end

  desc "Restart Application"
  task :restart, :roles => :app do
    run "touch #{current_release}/tmp/restart.txt"
    sudo "god restart listeners"
  end

end

Give Passenger a try. I think you will like it.

Music Is Dead: iPods and Young People Have Utterly Destroyed Music

This is depressing for someone like me. A Stanford professor did an informal study over several years, and he found that students each year preferred low-quality MP3 more and more. Like the author, this is disappointing for me because it most likely means it will become harder to find high-quality music. It’s already hard, but it will get worse. Even CDs are being mixed with MP3 players in mind.

I guess it comes down to what you’re used to. If it’s all you ever heard, then you don’t know what you’re missing. Personally, I am spoiled by hundreds of high quality live performances, and I always wanted to recreate that sound at home as much as possible. Low quality MP3s just don’t do it for me. They are fine on my iPhone or small player, but when I sit in front of my home system I want to hear like I was in the studio.

I had a similar experience many years ago when I had a top-end VCR with one of the best TVs available at the time. I thought the picture was amazing. Then I hooked up a cheap DVD player, and I forgot all about the VCR. The problem here is that these low quality recordings are becoming so much the norm that it will be hard for anyone to know what they are missing.

What is a music lover to do?

Gallery – Reflecting on a new generation of mirrors – Image 1 – New Scientist

Continuing on the theme of interesting optics, Andrew Hicks, a mathematician, has created mirrors that accomplish some amazing feats. Imagine looking at text in a mirror, and the text is not backwards. Imagine 360-degree panoramas without distortion. It’s all pretty cool stuff.

All of the surfaces were designed with computers to reflect the light rays across the mirror surface before returning them to the viewer. With the right shapes, you can do some bizarre tricks with light. It’s like Fun House mirrors without all the distortion. Some of the applications of this are in robotics and even automobiles.

I remember back when I was getting my masters, I built a 3D rendering engine for giggles (I wanted to understand the math). If you have ever built one of these you know it’s all about calculating where light rays go and what color and brightness they end up with when they get there. In this case, you need to work backwards. Here is what I want. Now, what is the shape necessary to get there, and that’s a whole different ball game. I wonder if the idea is to solve the same rendering equations in the other direction.

I’ve been an iPhone user since the first weekend the 3G came out, and so far I love it. Prior to the iPhone, I was a Blackberry user for a while. My reasons for waiting to change were:

1. My contract with Verizon conveniently ended the same month the 3G came out.
2. AT&T had lousy coverage at my home and office. However, for some reason AT&T and Verizon swapped in the months leading up to the 3G coming out. Verizon started to have lousy coverage in my home, and I changed job locations. The new location had lousy Verizon coverage.
3. The price drop made it a great time to upgrade my wife as well. There was no way I was going to get away with a 1st gen iPhone without my wife getting one as well. With penalties, that would have been $1550 instead of $400. I like Apple stuff, but I’m not that big a fan boy.

Fast forward several months, and I’m not looking back to my Blackberry (ever), but there are a few minor issues with the iPhone that I wish Apple would fix. It’s been long enough now that I’m starting to wonder when/if these may happen.

No Indicators on the Locked Page

Why make me slide the bar and enter my code just to see whether I have voicemail or email? It doesn’t seem to be that hard to have a couple of indicators on the locked page to let me know. This would be a great time saver and a good compromise away from that annoying blinking LED on the Blackberries.

No Sound Presets

To change any of the sound settings, I have to go in and update each separately. Having something like Night, Office, Silent, etc. would be great. The silent switch is good, but sometimes I really want silence, and others I want the phone to still vibrate.

Related to this, I want different sound settings for each email account as well. For example, I have error emails from my site(s) sent to me. I want those to complain loudly regardless of what I have setup for other volumes.

The Blackberry does this well.

Signatures for Each Email Account

I have three email accounts configured on my iPhone. I prefer to separate my email signatures between work and personal. Again, I could do this on a Blackberry.

One Global Inbox

It would be a lot easier to read my email if there was a single Inbox where all three of my accounts show up. This is what Apple Mail does on my desktop, and I use it all the time. Right now, I have to go through the follow set of interactions for each account:

1. Tap Personal account
2. Tap Inbox
3. Read/Process inbox
4. Tap Personal to go back
5. Tap Accounts to go back
6. Tap Work
7. Tap Inbox
8. Read/Process inbox
9. Tap Work to go back
10. Tap Accounts to go back
11. etc.

Lot of tapping going on here! Give me a consolidated inbox, and life would be a lot easier for those of us with multiple email accounts. Did I mention I could do this on my Blackberry?

Notice I did not say copy/paste. In seven months, I think I could have used it twice. I can see Apple making the correct decision to cater to the 80% of users out there and not the reviewers.

Don’t get me wrong, I will not go back to a Blackberry without a fight, but give me the above, and I will be an even happier camper.

Excuse me, I have to take a call now…

I just finished reading The Secret in Building 26: The Untold Story of How America Broke the Final U-boat Enigma Code. What a fascinating read! I’ve written before about the UK’s contribution to code breaking during WWII (Neglect of Bletchley condemned and Bletchley Park Update), but this is the first time I learned about some of the details of the US contribution. I always thought the majority, if not all, of the work was handled by the British, but this book details how it was much more of a partnership between the British and Americans. Most of this stuff is only recently declassified, and most is still classified. However, the information available is a great introduction to the dawn of the electronic computer.

Computers were born from two basic military needs. First, I want to blow up the other guy so I need to calculate the ballistic trajectory of my shells. Second, I want to read and learn my enemy’s secrets. The first (ballistics) is a fairly straight forward calculation (although a lot harder than you think), but it is such that you can create tables and distribute rather than recalculating every time. It’s not exact, but it gets you pretty close. The second (code breaking) is much harder because the codes and the techniques are changing all the time. This book covers those americans working in Building 26 of the NCR Corporation. With their British counterparts, they help cracked the Nazi Enigma machine, allowing the allies to shorten the war.

What is so interesting about this story? As a computer and math geek, I love learning how computing was born. Most importantly, I’m interested in the people that made it happen. For example, one the foremost code breakers of the time was Agnes Driscoll. In the 20′s and 30′s is was not very common for woman to hold such a high scientific and strategic position. “Madam X” or “Ms Aggie” as she was referred to took a lot of heat because she was a woman, but damn she held her own and is credited with breaking several “unbreakable” codes — all by hand! My favorite story of the crap she had to put up with was when a gung-ho Navy officer dared question her application to renew her security clearance. The young officer barged in and yelled “Ma’am, in this section where you are required to list five references, you have listed the names of five admirals. You are supposed to list the names of people you know well.”

Driscoll looked up at the young man with a squint in her eye and said, “Sonny, I knew all of them when they were ensigns and lieutenants, and if you don’t straighten up, I’m going to tell them to never promote you.”

Another interesting person was Joseph R. Desch. He was the lead for the entire effort. Most of the pressure fell on this man, and no one ever knew what he did. Never mind 14-hour days, 7 days a week, and often 7:30AM to 2:30AM the next day, 7 days a week for months at a time. All of this work to develop a machine call a Bombe. The German Navy’s Enigma machine had 2.0 x 10¹⁴⁵ ciphering possibilities compared to 1.0 x 10⁸⁰ atoms in the known universe, and it was the Bombe’s job to crack the code. It couldn’t do it alone, but by the end of the war through a combination of Bombes, cribs, lax German procedures, and good guesses, the teams could read German messages often before the German recipients did. Fascinating stuff.

In 1947, Joe Desch received the National Medal of Merit in a secret ceremony for his work in developing the U.S. Bombe. Unfortunately, no one ever knew until after his death why he had the highest civilian honor for wartime service. Not even his family had a clue until the 90′s.

Don’t forget Alan Turing was part of the effort to build the British Bombe. You all recognize the name Turing, right?

If you are at all interested in world and computing history, I highly recommend this book. If you do read the book, don’t skip the acknowledgements. It’s impressive to see how much went into gathering the information for this book.

Above is a US Bombe machine with a WAVE operator from 1943. The US Bombe machines were 10 feet wide, 7 feet tall and about 2 feet deep. They weighed over 2 and 1/2 tons. The American machines used mechanical relays, vacuum tubes, various motors and clutches, and were run by the women of the WAVE section (Women Accepted for Volunteer Emergency Service) of the United States military.