Parts of the project - throttling

[Dave]

Hi,

I'm very interested in the throttling aspect of this project. It may
be a bit early to start work on this so I thought I'd kick off a bit
of a discussion about how to implement it.

Having looked at what James has done so far, I've had a couple of
thoughts:

- How do we see the send_all() type function being used? Cron job?
Called by a signal? Constantly running? Some other way?
- How configurable does it need to be? Do we just need to define
emails/(second|minute|hour) or do we need a more complex definition
like exim's ratelimit: <threshold>,<init delay>,<increasefactor>,<max
delay>
- Are there any pre-written SMTP python libraries about that can do
any of this?

That's about all I have at the moment.

Thanks,

Dave

[James Tauber]

On 08/08/2007, at 5:40 PM, Dave wrote:

>
> Hi,
>
> I'm very interested in the throttling aspect of this project. It may
> be a bit early to start work on this so I thought I'd kick off a bit
> of a discussion about how to implement it.
>
> Having looked at what James has done so far, I've had a couple of
> thoughts:
>
> - How do we see the send_all() type function being used? Cron job?
> Called by a signal? Constantly running? Some other way?

Either a cron job calling send_all() or a constantly running send_loop
() were the two ways I was thinking.

A guess another django app could call send_all() as well but I'm
inclined to dissuade multiple calls to send_all(). While, in theory,
it's re-entrant, I'd rather treat it as if it were not.

> - How configurable does it need to be? Do we just need to define
> emails/(second|minute|hour) or do we need a more complex definition
> like exim's ratelimit: <threshold>,<init delay>,<increasefactor>,<max
> delay>

I was initially just thinking of a number of emails per period. Can
you explain more about the exim settings?

> - Are there any pre-written SMTP python libraries about that can do
> any of this?

Good question. It's worth looking.

James

[David Reynolds]

On 9 Aug 2007, at 11:00 am, James Tauber wrote:

> Either a cron job calling send_all() or a constantly running send_loop
> () were the two ways I was thinking.
>
> A guess another django app could call send_all() as well but I'm
> inclined to dissuade multiple calls to send_all(). While, in theory,
> it's re-entrant, I'd rather treat it as if it were not.

While I agree a cron job could be used, I'm wondering if there's a
better way.
Think about a system running 30 sites, having 30 cronjobs or
instances of send_all()
running might prove to be intensive on the system. I'm thinking a bit
selfishly of
my own case here, but I hope it applies to other people.

>> - How configurable does it need to be? Do we just need to define
>> emails/(second|minute|hour) or do we need a more complex definition
>> like exim's ratelimit: <threshold>,<init delay>,<increasefactor>,<max
>> delay>
>
> I was initially just thinking of a number of emails per period. Can
> you explain more about the exim settings?

I think I may have been not thinking too straight when I wrote that.
I think exim's way
of doing it is looking at it from the MTA's end (obviously) so when
the number of emails
hits the threshold it starts applying the init delay and then for
each mail over the threshold
it increases the delay by the increasefactor before hitting the
maxdelay. I think we need to look
at it from the other end, as mentioned and do number of emails per
period.

>> - Are there any pre-written SMTP python libraries about that can do

>> any of this?
>
> Good question. It's worth looking.

I'll try and have a look today.

Thanks,

Dave

--
David Reynolds
da...@reynoldsfamily.org.uk

[James Tauber]

On 09/08/2007, at 3:37 AM, David Reynolds wrote:
> While I agree a cron job could be used, I'm wondering if there's a
> better way.
> Think about a system running 30 sites, having 30 cronjobs or
> instances of send_all()
> running might prove to be intensive on the system. I'm thinking a bit
> selfishly of
> my own case here, but I hope it applies to other people.

To be honest, I hadn't given much thought to the multiple site case,
although I'll likely myself have a system running about 10 sites
fairly soon.

Cron jobs obviously have the advantage over send_loop() of not being
long running which would be important to people hosting on services
like webfaction that have long-running process limits.

I'll be running a separate apache and separate database for each
site, though, so a separate send_loop() process doesn't seem that
marginally onerous.

James

[philp]

On Aug 8, 10:40 pm, Dave <da...@reynoldsfamily.org.uk> wrote:
> - How do we see the send_all() type function being used? Cron job?
> Called by a signal? Constantly running? Some other way?

Django's signal architecture may be worth considering:

http://www.mercurytide.co.uk/whitepapers/django-signals/

Rather than setting up cron jobs, or implementing nasty infinite
loops, you would be able to signal to the message despatcher that it
needs to take action whenever an event occurs - such as a new bulk
message being created, a new user registering etc.

Plus, you ought to be able to decouple the mailer from specific
models, so that you could cater for things like message failures: if a
message fails to be sent, a project could have custom code which
handles that failure, or sending a notification email to an admin; or
deleting the message silently; or flagging the user because their
email address may not be valid.

-Phil

[Andrew McClain]

I agree with the signal architecture... especially for use cases where
there need to be notifications to a single or handle of users.

Also something to consider -- is there reverse communication back to
the main app on success/failure?

-andrew

[David Reynolds]

On 15 Aug 2007, at 12:20 am, Andrew McClain wrote:

>
> I agree with the signal architecture... especially for use cases where
> there need to be notifications to a single or handle of users.
>
> Also something to consider -- is there reverse communication back to
> the main app on success/failure?

I agree that signals would be a good solution in some respects, but
in order to throttle the email to send only $number of emails an hour
there needs to be some sort of timer process running to do this. I
can't see how this could be implemented with a signal - if anyone
could enlighten me I'd be very grateful.

Thanks,

David
--
David Reynolds
da...@reynoldsfamily.org.uk