Queuing emails for delivery

[Andrew Havens]

The company I work for has a system for queuing and delivering emails which was previously some PHP code that would save a record in a MySQL database and send every 5 minutes via cron job. We recently decided to rebuild this service as a RESTful API using Rails and MongoDB. It's one of our company's first official Ruby projects so I'm pretty excited to get the chance to write some ruby code at work! =]

My question for you, as an educated Rails developer, if you were building an application like this, how would you handle the queuing and sending? Would you use cron and write a ruby script to do the sending? Or would you use some job queuing library? I know there are a bunch of job queue libraries out there, but I'm not sure if they are appropriate or overkill for such a simple scenario. I'm curious if you would do things the same or different.

Sorry if this is a silly question. I'm still relatively new to Ruby, and I'm the one with the most Ruby knowledge at work.

--Andrew

[Sam Livingston-Gray]

Pardon my lack of devops cred, but isn't queuing emails exactly what sendmail is for?

--
(Sent from phone; please excuse brevity.)

> --
> You received this message because you are subscribed to the Google Groups "pdxruby" group.
> To post to this group, send email to pdx...@googlegroups.com.
> To unsubscribe from this group, send email to pdxruby+u...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/pdxruby?hl=en.
>

[Jesse Cooke]

If sendmail lives on the machine that's doing the process, then you probably don't need a queuing system. If you use an external smtp, then you'd need something else, like Sidekiq[1].

[1] http://github.com/mperham/sidekiq

--------------------------------------------
Jesse Cooke :: N-tier Engineer
jc00ke.com / @jc00ke

[Matthew Boeh]

Yeah, seconded. Is the generation of these emails particularly
time-consuming? If not, using a local install of
postfix/exim/sendmail/whatever would seem to be the natural choice,
and it requires no data store or queuing system.

You may want to log your outgoing mails. In that case, I'd still
simply send them out as soon as they were requested, but log them
after in whichever format you please.

Matthew Boeh

On Mon, Apr 16, 2012 at 10:32 PM, Sam Livingston-Gray <gee...@gmail.com> wrote:

[Brad Heller]

Or services like SendGrid. I've pumped a TON of mail through that service...and it's very cheap. Seems to be very similar to what you guys are making...

Anyway, depending on what dependencies you're willing to take on, I've found Resque to be the lowest maintenance job queue around. It has a very active developer community to boot. If you're in Rails land, resque_notify (I think that's what it's called?) will automagically queue any ActionMailer for asynchronous delivery. Resque is backed by Redis, which makes some dev ops' hair stand on end but, like Resque itself, Redis has been crazy low maintenance. So much so, in fact, that I've been able to apt-get install redis-server and forget about it in the past!

[Matthew Boeh]

I feel like sidekiq (or any background worker system) is overkill for
what we know of this use case. You might want to background SMTP
connections if they're done inline in an existing web app, but if the
service is completely dedicated to that purpose I'd think a threaded
web server would be preferable.

Or girl_friday, I suppose.

Matthew Boeh

[Jesse Cooke]

On Mon, Apr 16, 2012 at 10:37 PM, Brad Heller <br...@revisu.com> wrote:

Or services like SendGrid. I've pumped a TON of mail through that service...and it's very cheap. Seems to be very similar to what you guys are making...

Possibly. I use SendGrid to send emails, but I still generate them on my end, and I don't want the connection to SendGrid to block my app, so I use something like girl_friday or Sidekiq. 

[Jesse Cooke]

On Mon, Apr 16, 2012 at 10:39 PM, Matthew Boeh <matthe...@gmail.com> wrote:

I feel like sidekiq (or any background worker system) is overkill for
what we know of this use case. You might want to background SMTP
connections if they're done inline in an existing web app, but if the
service is completely dedicated to that purpose I'd think a threaded
web server would be preferable.

Or girl_friday, I suppose.

For simple worker needs, nothing IMO beats girl_friday. 

[Andrew Havens]

I guess I should give a little more background...this Email Queue API is meant to achieve a few different things:

• Delay the sending for 5 minutes to give the user a chance to cancel sending the email if it was a mistake.
• Provide the ability to delay sending the email until a future date/time.
  
• Keep a history of this email so we can keep track of how many emails we send to the recipient and what was communicated to them. This also provides the ability to resend an identical email, or duplicate it to send to someone else.
• Abstract the sending implementation to be able to swap out the implementation or specifically choose which one to use (SendGrid, SMTP, SendMail, etc)

So, in a way, we are sort of building our own SendGrid API to use internally. Our process right now is to add this email to a MySQL table with a "sent" field set to false, and a cron runs every 5 minutes to send any emails that haven't been sent yet, which will update the sent flag to true. Or if you want to cancel the email, it will be deleted from the queue. However, since the cron runs every 5 minutes, if the email gets added to the queue 10 seconds before the cron runs, it doesn't really give you enough time to cancel it, thus defeating the purpose of waiting 5 minutes.

I like the idea of using some sort of background worker to ensure that it's sent at 5 minutes (not before, not after). But I haven't really seen an example of using girl_friday, resque, sidekiq, etc. in this way. Usually they just "fork" themselves to do the work immediately by storing some record in Redis and remove themselves when they are done. However, I have never used any of these so there might already be example code or some obvious thing that I am missing.

Could you provide a high level overview of what I should do in order to implement this sort of scheduled background task while still keeping a history of this email in our MongoDB email history/queue collection?

Thanks!

--Andrew

[Matthew Boeh]

You should take a look at delayed_job. it's intended for exactly this
kind of work.

Is there a reason you're using MongoDB for this? This kind of job
relies heavily on timestamp-based queries on the database, and SQL
databases generally perform better in that situation.

Matthew Boeh

> --
> You received this message because you are subscribed to the Google Groups
> "pdxruby" group.

> To view this discussion on the web visit
> https://groups.google.com/d/msg/pdxruby/-/JXHS32OxRYQJ.

[Smit Shah]

Hey, I might be able to help out with this issue as recently I was implementing something similar a week back and I hope it helps you as well,

Whatever you have given information about points to an wonderful gem called DelayedJob (Actively developed) you first need to setup this gem, which should be easy enough.

So you can queue up jobs in the delayedjob queue, every 5 mins or whatever time you want all you have to do is run this rake jobs:worker as cron job or if you are using engineyard or heroku you can use their scheduler to accomplish it.

This will spawn up a process and it will automatically take care the emailing process for you (We used it to do send bulk mails), and there are lot of optimizations you can do to speed up the process.

Like run the cron job only when the DelayedJob:Count > 0 etc, 

aside from that, sending email is already handle by ActionMailer you can read the railsguide and if you are using any third party service you can quickly setup all the SMTP parameters and if you want to send it from the system you can setup postfix and rDNS to do so but I would use a third party to send the email itself as resolving everything so your email doesn't end up in the spam bin is a pain.


I hope I answered your questions, if not please reply back and I will try my best to help you out (Like others did when I needed help) !

--

You received this message because you are subscribed to the Google Groups "pdxruby" group.

To view this discussion on the web visit https://groups.google.com/d/msg/pdxruby/-/JXHS32OxRYQJ.