In any event, the problem seems fixed now though it had nothing to do
with the DEADLINE_SOON message directly.  However, it very
consistently appeared right after the DEADLINE_SOON response in the
log files.  So I wrote up a test script to try to find out why this
correlation seemed to manifest.  Turns out that the connection between
the server and client is always dropped after a DEADLINE_SOON
response.  I'm not sure as to why this is so but I included my Ruby
test script at the end of this post.

In our code, this dropped connection caused a re-connect on the next
reserve call which then appeared to hang because it was watching the
default tube and not the tubes which actually had jobs waiting.  Thus
the DEADLINE_SOON was not directly related to our initial problem
though because of this odd dropped connection side-effect, it seemed
to cause the problem.  Hence our multitude of questions to try to
understand it and explain this strange correlation.  Thanks for all
the patience BTW.

Here's the test script:

<beanstalk.test.rb>
require 'rubygems'
require 'beanstalk-client'

@beanstalk = Beanstalk::Pool.new('localhost:11300')

@beanstalk.use('test-tube')
@beanstalk.watch('test-tube')
@beanstalk.ignore('default')

# clear the tube of ready jobs
while !...@beanstalk.peek_ready.nil?
  @beanstalk.reserve.delete
end

puts "Ready Jobs: #...@beanstalk.stats['current-jobs-ready']}"
puts "Reserved Jobs: #...@beanstalk.stats['current-jobs-reserved']}"

@beanstalk.put('message 1', 65536, 0, 10)

job = @beanstalk.reserve
puts "Reserved: #{job.body}"

begin
  puts "Open Connections: #...@beanstalk.open_connections.length}"
  puts "Waiting for job..."
  puts @beanstalk.reserve
rescue
  puts "ERROR: #{$!}"
  puts "Open Connections: #...@beanstalk.open_connections.length}"
end
</beanstalk.test.rb>

Running with the 0.11.2 client:

[root@localhost ~]# gem list | grep beanstalk
beanstalk-client (0.11.2)

Here's the output: