I came across this interesting article regarding Blechley Park.  I was surprised to see how rundown the place is.  For those that don’t know, Bletchley Park is where the code breakers lived during WWII.  It is basically the birthplace of modern computing, and it helped shortened the war.  Imagine reading your enemy’s mail.  Unfortunately, even though they knew certain things, the Alies often had to ignore it or risk letting the enemy know their secrets were not safe. Overall, however, the knowledge had a huge impact on the war.

I know the soldiers deserve credit for everything they fought and died for (believe me. I know.), but show a little love for those geeks that worked hard in total secrecy doing their part to secure victory.  They may not have carried a gun, but their information saved thousands and had a huge impact on the Alied victory.  It wasn’t till the 1970s that the place was even acknowledged to exist and the thousands of workers could talk about it publicly.

Bletchley Park needs to preserved both for its place in modern history and for its place in modern computing.

Of course, I had to try my first blog post using the new Wordpress for iPhone app. It works well.

I have a bunch of new material that I will be posting over the next few weeks.

Right now, it’s time to find more fun stuff on this device.

A great product team gets together and builds an amazing product or service. Product Marketing provided valuable customer feedback. Product Design uses that feedback to designed a great product that removes customer pain. Development builds it on-time and under budget, and QA ensures a high-quality product. All of these teams work together to build an awesome product — often with long hours and a few miracles along the way. As a result of these efforts, sales go through the roof. Unfortunately, all that means is Sales gets the most benefit (monetarily anyway) from the the other teams efforts because their commissions go up.

The question is how does this work it’s way back to the other teams? Should it? Unfortunately, there is no easy answer because there isn’t any way to figure out how much an individual contributed to a product. At first, I thought we could come up with a scheme to reward a team for their contributions, but even that doesn’t always work. In the world of Agile and Scrum, there is a concept of BVI (Business Value Index). I often use BVI to help prioritize stories for completion. In a nutshell, BVI is the percentage contribution a story has to the overall value of the release. So, if a release is projected to bring in $1M in revenue, and a story has a BVI of 0.25, then that story’s Business Value (BV) is $250K. In theory, you could then use the combined BV of a scrum team’s story output to come up with that team’s contribution to the final value. Unfortunately, the reality is not that simple. Often, stories have a BVI of 0 even though other stories can’t happen without it. A story to refactor a class to make it more maintainable has no impact on the customer and, therefore, no BVI. It does impact cost, so maybe we could add a CVI (Cost Value Index) to those stories without BVI.

To some extent, CVI can have even more impact long term. A story with a BV of $250K will always be far less on the company’s bottom line. However, a story with a Cost Value (CV) of $250K goes straight to the company bottom line. Plus, developer’s spend less time on maintenance and more time developing, and that makes everyone happy.

Cost Value sounds like a great idea, but it is extremely difficult to calculate. In fact, very few organizations pay much attention to it. “We can’t spend 80 hours automating that process. There isn’t time.” The part that people forget is not having the process automated is costing 10 hours per week. Let’s say one developer takes the two weeks (80 hours) to automated it, then your total cost is 100 hours (you still have the 10 hours/week without automation). For your 100 hours of cost, you saved 480 hours over a year (48 weeks * 10 hours/week). That’s 480 more hours to spend on actual product development.

CV = Net savings per year
CVI = Percentage contribution to the whole

I need to think about this some more, but I’m curious what people think. The biggest problem with this is a breaks one fundamental rule in Scrum. That is to properly calculate CV, you have to track how much time people are spending not doing their tasks, and we know we should only be tracking how much time is left.

What do you think? I don’t like the idea of tracking time, but I do like the idea of being able to prioritize a story that reduces costs along with a story that generates revenue.

I will leave you with a great quote. I don’t know who said it, but it is so very true.

“Organizations don’t get what they want. They get what they measure.”

Imagine all those startups thinking they are secure with a patent to cover their IP. This is one crazy cloud over so that thought. Granted, a bunch of recent patents are completely bogus, but now the patent trolls could have a field day.   Definitely need to keep an eye on this one.

I wonder if that means the patent I just received last month is not valid anymore…

Kelly: 1. The Skunk Works manager must be delegated practically complete control of his program in all aspects. He should report to a division president or higher.

Agile: 1. The Scrum Master must have the power and authority to remove any blocking issues.

Kelly: 2. Strong but small project offices must be provided both by the military and industry.

Agile: 2. Sprint teams must be in the same offices. Virtual teams can work, but it’s best to be in the same area.

Kelly: 3. The number of people having any connection with the project must be restricted in an almost vicious manner. Use a small number of good people (10% to 25% compared to the so-called normal systems).

Agile: 3. Sprint teams must be small (< 7). Although, I like Kelly’s phrasing better.

Kelly: 4. A very simple drawing and drawing release system with great flexibility for making changes must be provided.

Agile: 4. Use Stories instead of detailed design and functional specifications. Document when necessary, but keep it simple.

Kelly: 5. There must be a minimum number of reports required, but important work must be recorded thoroughly.

Agile: 5. Keep the status reports to the waterfall people. We hold daily Scrum Meetings to discuss status. If you’re interested, come to the meeting, but don’t expect a report.

Kelly: 6. There must be a monthly cost review covering not only what has been spent and committed but also projected costs to the conclusion of the program. Don’t have the books ninety days late and don’t surprise the customer with sudden overruns.

Agile: 6. Hold daily Scrum meetings to determine the status of the project. Identify and remove blocking issues before they become a problem.

Kelly: 7. The contractor must be delegated and must assume more than normal responsibility to get good vendor bids for subcontract on the project. Commercial bid procedures are very often better than military ones.

Agile: 7. The Sprint Team must be fully empowered to reach the Sprint Goal.

Kelly: 8. The inspection system as currently used by the Skunk Works, which has been approved by both the Air Force and Navy, meets the intent of existing military requirements and should be used on new projects. Push more basic inspection responsibility back to subcontractors and vendors. Don’t duplicate so much inspection.

Agile: 8. The Sprint Team shall complete all assigned stories, including full testing. Don’t push testing off for a future team.

Kelly: 9. The contractor must be delegated the authority to test his final product in flight. He can and must test it in the initial stages. If he doesn’t, he rapidly loses his competency to design other vehicles.

Agile: 9. The Sprint Team must demonstrate success of the Sprint by showing actual, working product that completes the assigned stories. Testing must be don’t continuously throughout the Sprint.

Kelly: 10. The specifications applying to the hardware must be agreed to well in advance of contracting. The Skunk Works practice of having a specification section stating clearly which important military specification items will not knowingly be complied with and reasons therefore is highly recommended.

Agile: 10. Stories must be committed to in advance of the Sprint.

Kelly: 11. Funding a program must be timely so that the contractor doesn’t have to keep running to the bank to support government projects.

Agile: 11. Keep the Sprint Team well fed, well caffeinated, and unblocked.

Kelly: 12. There must be mutual trust between the military project organization and the contractor with very close cooperation and liaison on a day-to-day basis. This cuts down misunderstanding and correspondence to an absolute minimum.

Agile: 12. There must be mutual trust between upper management and the Sprint Team. Scrum Meetings are held daily to facilitate communication.

Kelly: 13. Access by outsiders to the project and its personnel must be strictly controlled by appropriate security measures.

Agile: 13. Access by outsiders to the Sprint team and its personnel must be strictly controlled by the Scrum Master. That means no going to a Sprint Team member to add more tasks.

Kelly: 14. Because only a few people will be used in engineering and most other areas, ways must be provided to reward good performance by pay not based on the number of personnel supervised.

Agile: 14. Pay should be based on the success of the team and the individual, not on how many people are supervised.

There you have it. Kelly Johnson and the Skunk Works were one of the first Agile Engineering shops. Very few can claim to be as successful or innovative.

Last month, I pointed out an article about salvaging a ship in trouble.  The salvage crew successfully saved the ship and its cargo of 4,703 Mazda vehicles (loosing one life in the process), but now what do you do with the cars?  Turns out, it’s not easy to destroy 4,703 cars and not get in trouble, damage your brand, or get sued in the process.  It took Mazda a year to plan and about another year to actually do it.  Who knew?

Those of you in Boston may remember Tech Cocktail last fall. If you were one of the lucky 300+ people in attendance, you know how successful it was. I know I enjoyed it, and I met some great new contacts there. Well, Brian Balfour and Jay Meattle are at it again, but now it is POPSignal.

POPSignal parties are aimed at bringing together the local tech community in a fun and informal environment. There is no format, presentations, or speeches. However, there is always a free open bar, free food, music, fun activities from sponsors, and great conversation.

The date is May 15th, 6:30-9:30pm, at Tequila Rain near Fenway Park. To RSVP and get more details check out http://popsignal.eventbrite.com/.

Come on by and say hello.

Continuing the theme of the Lockheed Skunk Works, this is an interesting take on the development of the shaft-driven fan on the F-35B. The inventor discusses how he came up with the idea and how the system works. Like any good research project, there was a risk of failure, but in the 9th month of a 9 month study Dr. Paul Bevilaqua had an idea that eventually turned into the F-35’s engine. I hope he got a bonus considering the F-35 contract is worth $200 Billion (that’s a ‘B’).

One comment from Dr. Bevilaqua worth mentioning is how it is more important to do engineering than to do math. In other words, think! As Dr. Bevilaqua says, school may have taught him how to move the pieces, but his mentor (Dr. Hans von Ohain, co-inventor of the jet engine) taught him how to play chess. In my travels, I see far too many “engineers” that don’t think. If someone hasn’t done it before, they are stuck.

We all should take a lesson from this. Good software engineering is more than re-hashing what someone else did. Sometimes it’s the wacky ideas that end up being the best. Don’t be afraid to reach sometimes and see what might work. Don’t re-invent the wheel, but if there is no wheel then figure out a way to make one.

Installing god.rb is as simple as:

sudo gem install god

The god.rb web site has some decent documentation to help you understand how to build a config file. I tend to build my config files for as much as I can using erb templates rather than manually updating the config files for each environment.  I build my apache config file this way as well.

1. Download my god config to /lib/templates/god.conf. This is an example of an actual god.conf file. You will want to update it for paths and to setup the mongrels and the listeners to not run as root — not a good idea to run those as root. My apache and mysql scripts already run as a different user.

2. Now, add the following to your config/deploy.rb file:

namespace :god do

  desc <<-DESC
  Generate the GOD configuration. We will create the appropriate GOD
  configuration stanza for the application and copy it to:

  /shared/path/god.conf

  Once it's there, somebody with the required permission to manage the
  GOD configuration should somehow incorporate that file and restart
  the GOD server. Each time the runtime configuration (ie number of
  mongrels running) changes, the configuration will have to be manually
  updated to match.
  DESC
  task :config, :roles => :app do
    rails_env = fetch(:rails_env, 'production')
    dispatcher_starting_port = fetch(:dispatcher_starting_port, 8000)
    dispatcher_instances = fetch(:dispatcher_instances, 3)
    dispatcher_ending_port = dispatcher_starting_port + dispatcher_instances - 1
    dispatcher_ports = dispatcher_starting_port..dispatcher_ending_port
    god_conf_path = "#{shared_path}/god.conf"
    god_conf_template = File.join(File.dirname(__FILE__), '..', 'lib', 'templates', 'god.conf.erb')
    begin
      god_conf = ERB.new(File.read(god_conf_template)).result(binding)
      put god_conf, god_conf_path, :mode => 0644
    rescue Exception => e
      abort "An error occurred in the GOD config generation: #{e}"
    end
  end
end

3. Execute cap god:config with the desired environment, and a new god.conf file will be copied to the shared directory on your server.  I’m assuming you have capistrano 2.x here.

4. On your server, execute: sudo god -c /shared/path/god.conf

That’s it! Now, all your processes will be monitored by god.rb and restarted whenever there is a problem. Hold on though, we’re not completely done. You want god.rb to start up when your server boots, right? We also have to update our deploy process to use god.rb to start/stop our services now. Try to use you existing spin or spawn tasks, and god.rb will fight with you.

First, let’s get god.rb setup as a service. I found this script in the god.rb sources, and I tweaked it a bit for my system.

#!/bin/bash
# god       startup script for god (like monit, only Ruby)
# Author: Dave Dupre

# Comments to support chkconfig on CentOS
# chkconfig: - 85 15
# description: god - monitor all my processes

CONF_DIR=/shared/path
LOG_DIR=/var/log
PID_DIR=/var/run/god
BIN_DIR=/usr/local/bin

RETVAL=0

# Go no further if config directory is missing.
[ -d "$CONF_DIR" ] || exit 0

case “$1″ in
  start)
    $BIN_DIR/god -P $PID_DIR/god.pid -l $LOG_DIR/god.log -c $CONF_DIR/god.conf
    RETVAL=$?
  ;;
  stop)
    $BIN_DIR/god terminate
    RETVAL=$?
  ;;
  restart)
    $BIN_DIR/god terminate
    $BIN_DIR/god -P $PID_DIR/god.pid -l $LOG_DIR/god.log -c $CONF_DIR/god.conf
    RETVAL=$?
  ;;
  status)
    $BIN_DIR/god status
    RETVAL=$?
  ;;
  *)
  echo “Usage: god {start|stop|restartstatus}”
  exit 1
  ;;
esac

exit $RETVAL

Put the above contents into /etc/init.d/god and make it executable (sudo chmod +x /etc/init.d/god). Lastly, tell your system there is a new service in town. I use CentOS 5, so I run:

sudo chkconfig /etc/init.d/god on

Now, god.rb will start up whenever your server boots, and you can start/stop/restart it using standard service calls.

The last thing we need to do is update our deployment process to use god.rb to stop/start our processes. Add this to your deploy.rb file:

namespace :deploy do
  [ :stop, :start, :restart ].each do |t|
    desc “#{t.to_s.capitalize} mongrels using god”
    task t, :roles => :app do
      sudo “god #{t.to_s} listeners”
      sudo “god #{t.to_s} mongrels”
    end
  end
end

namespace :starling do
  [ :stop, :start, :restart ].each do |t|
    desc “#{t.to_s.capitalize} starling using god”
    task t, :roles => :app do
      sudo “god #{t.to_s} starlings”
    end
  end
end

namespace :workling do
  [ :stop, :start, :restart ].each do |t|
    desc “#{t.to_s.capitalize} workling using god”
    task t, :roles => :app do
      sudo “god #{t.to_s} listeners”
    end
  end
end

OK! Now, we’re done. You deploy as you normally would, and I have full control over Starling and the Workling listener. Notice that there is no mongrel cluster either. God.rb started up all the instances of mongrel I needed, and it will monitor everything so there is no need for mongrel cluster anymore.

That’s it for now. You have a system that will process all your background tasks and stay running. The only thing I didn’t setup here is notifications from god.rb when there is a problem. The god.rb config settings have lots of schemes for email notifications. Take a peak at the docs and make sure god can talk to you.

Most of the changes to Workling, are limited to the lib/workling/starling/poller.rb file. Here is a summary of the changes I made:

• Added threading
• Updated the way ClassAndMethodRouting builds the routing hash. I needed a routing hash per worker. Note, whatever you do don’t ever try to call routing.build from inside another thread. Really strange things happen that were very difficult to track down. Ruby threads are nice, but I’m used to real threads.
• Added handling for MemCache exceptions. During development, I went to dinner and left the listener running without Starling. When I got back, my Mac was complaining the disk was full because my log was > 40Gig. Now, I explicitly catch MemCache exceptions and wait 30 seconds before trying again. The log will still grow, but at least not as fast. Besides, I will show you how to make sure Starling stays running.
• Keep your database connection alive. I found this out after leaving for the night. I thought I had a working system when I left, but nothing worked in the morning. Basically, ActiveRecord will drop your connection if you don’t do anything for a while. Normally, this is not a problem in web apps because the connection is updated every time you access a page, but it doesn’t work that way here. You have to do it yourself. Add this in your loop to be sure:

  
  
  unless ActiveRecord::Base.connection.active?
  
    unless ActiveRecord::Base.connection.reconnect!
  
      RAILS_DEFAULT_LOGGER.fatal("FAILED - Database not available")
  
      break
  
    end
  
  end
  
  

Here is my workling patch. Now, I need to go back and update the tests, but the threads complicate that. It’s a learning experience…

That’s it for now. Next time, I will walk through the deployment and monitoring process as I have it.