"Mysql::Error: MySQL server has gone away: SELECT * FROM" ... in simple-daemonized script?

[Jörg Battermann]

Hello there,

I have a mail-fetcher script which checks for mails in a pop3 account
regularly and which works perfectly fine standalone and being called
via a crontab entry regularly. However, I'd like to daemonize it
properly and therefore basically wrote the stuff below.

It loads the rails env, checks for mails and calls the
ImportFromMail.receive method which basically inherits from
ActionMailer::Base.

However, it seems the combination of rails env + simple-daemon whoes
with AR in the background as basically the connection to the database
is lost somewhere inbetween.... resulting in the error above :-/

Has anyone come across this / knows a proper way around to use simple-
daemon in combination with the rails environment loaded (and actually
keeping it)?


-J


---- the mail fetcher ----
#!/usr/bin/env ruby
RAILS_ENV = ARGV[1] || 'development'

require File.dirname(__FILE__) + '/../../config/boot'
require File.dirname(__FILE__) + '/../../config/environment'
require File.dirname(__FILE__) + '/../../lib/net/pop.rb'

require 'simple-daemon'

# configuring working dir for SimpleDaemon
WORKING_DIR = File.join(RAILS_ROOT, 'tmp', 'importers',
'mail_fetcher')
CONFIG = YAML.load(IO.read("#{RAILS_ROOT}/config/importers/
mail_fetcher.yml"))

class MailFetcher < SimpleDaemon::Base
`mkdir -p #{WORKING_DIR}` # just make sure the working dir exists
SimpleDaemon::WORKING_DIRECTORY = WORKING_DIR

def self.start
MailFetcher.log("Starting up")
sleep(5)

# looping and processing received messages
MailFetcher.log("Entering infinite loop state (checking every
#{CONFIG[RAILS_ENV]["sleep_time"]} seconds)")
loop do
MailFetcher.log("Checking for new mail(s)")
begin
# setting timeout to just a little bit under 10 minutes
timeout(CONFIG[RAILS_ENV]['timeout']) do
MailFetcher.log("Configuring pop3 environment")
@pop = Net::POP3.new(CONFIG[RAILS_ENV]['server'])
@pop.enable_ssl if CONFIG[RAILS_ENV]['use_ssl'] == true
MailFetcher.log("Configuring pop3 environment: done")

MailFetcher.log("Connecting to mail server")
@pop.start(CONFIG[RAILS_ENV]['username'],
CONFIG[RAILS_ENV]['password'])
MailFetcher.log("Connecting to mail server: done")

if @pop.mails.empty?
log "No new mails"
else
@pop.mails.each do |email|
begin
log "Receiving and importing mail"
ImportFromMail.logger = nil # suppressing rails
internal logging
ImportFromMail.debug_logger = @logger
ImportFromMail.receive(email.pop)
log "Done, deleting mail"
email.delete
rescue Exception => e
log "Error receiving mail: #{e.message}"
end
end
end

MailFetcher.log("Disconnecting from mail server")
@pop.finish
MailFetcher.log("Disconnecting from mail server: done")
end
rescue => e
MailFetcher.log("Could not fetch mail(s): #{e.inspect}")
end
MailFetcher.log("Checking for new mail(s): done")

# sleeping for another n seconds
sleep(CONFIG[RAILS_ENV]["sleep_time"])
end
end

def self.stop
MailFetcher.log("Shutting down")
@pop.finish unless @pop.nil?
sleep(5)
MailFetcher.log("Shutting down: done")
end

# Logging relevant methods
def self.logger
@logger ||= Logger.new(File.join(RAILS_ROOT, 'log',
'mail_fetcher.log'))
end

def self.log(message)
MailFetcher.logger.info("#{Time.now.utc}: #{message}")
end

def log(message)
MailFetcher.log(message)
end
end

# start the actual bot
MailFetcher.daemonize

[Jörg Battermann]

A little addition... the ImportFromMail class looks like this

class ImportFromMail < ActionMailer::Base
def receive(email)
# looking up user in rails app database, importing mail content
etc etc
end
end

.. and that's where the db error pops up (oh and the ruby process
looses about 35mb of memory usage) :-/

-J

[Luke Francl]

We solved this problem in one of our apps by catching the error and
retrying. Basically, MySQL disconnects if you've been idle too long.

# Handle lost connection weirdness, and get user
@already_retried = false
begin
user = User.find_by_message_from(email.from.to_s.downcase)
rescue ActiveRecord::StatementInvalid
ActiveRecord::Base.connection.reconnect!
unless @already_retried
@already_retried = true
retry
end
raise
else
@already_retried = false
end

However, lately, I've been favoring the cron-based approach. It does
spawn more processes than a daemon, but it seems more reliable because
the long-running Rails processes don't have time to eat up a ton of
memory.

Here's an example (taken from the new MMS2R PDF: http://peepcode.com/products/mms2r-pdf)
that shows using the Slantwise Fetcher (http://slantwisedesign.com/
rdoc/fetcher/) with the Lockfile gem so it can be run from cron.

begin
Lockfile.new('cron_mail_fetcher.lock', :retries => 0) do
config = YAML.load_file("#{RAILS_ROOT}/config/mail.yml")
config = config[RAILS_ENV].to_options

puts "Running MailFetcher in #{RAILS_ENV} mode"
fetcher = Fetcher.create({:receiver =>
MailReceiver}.merge(config))
fetcher.fetch

puts "Finished running MailFetcher in #{RAILS_ENV} mode"
end
rescue Lockfile::MaxTriesLockError => e
puts "Another fetcher is already running. Exiting."
end

Hope this helps,

Luke

[Jörg Battermann]

Luke,

ahhh ok.. thanks! That definitely helps! And yes - I think I'll stick
with the crontab way for repeating tasks and only use daemons for real
background/everrunning ones (e.g. the jabber bot I've written needs to
be online all the time and react timely fairly close to a message
etc).

Thanks for the heads up!

-J