Timeout

[Andy.vdg]

Hi,

Is the script set to time out after a certain amount of inactivity? I
let it run over night and when I checked it this morning at around
7:30 am it had not processed anything since 3:52 am.
[May 07 03:52:05] [info] copying message: xxxxxx
^C[May 07 07:28:16] [fatal] Interrupted (SIGTERM)

It didn't give me an error message or anything - just sat there. It
happened again this afternoon

[May 07 07:58:25] [info] copying message: xxxxxx
^C[May 07 09:45:59] [fatal] Interrupted (SIGTERM)

Once restarted it just continues from where it left off without any
problems.

Cheers
Andy

[Ryan Grove]

On Thu, May 7, 2009 at 3:49 AM, Andy.vdg <andy...@gmail.com> wrote:
> Is the script set to time out after a certain amount of inactivity? I
> let it run over night and when I checked it this morning at around
> 7:30 am it had not processed anything since 3:52 am.
> [May 07 03:52:05] [info] copying message: xxxxxx
> ^C[May 07 07:28:16] [fatal] Interrupted (SIGTERM)
>
> It didn't give me an error message or anything - just sat there. It
> happened again this afternoon

No, Larch doesn't have any built-in inactivity timeouts, and even if
the server drops the connection for some reason, Larch should
reconnect automatically.

However, there is a nasty deadlock bug in Ruby's Net::IMAP library
that can cause it to hang indefinitely (without sending an exception
to Larch) if the connection is dropped during certain stages of
certain actions, and message transfer is the most common case where
this occurs. There's a warning about this in the Known Issues section
of the Larch readme.

Unfortunately there's nothing I can do about this Net::IMAP bug. Well,
nothing safe, anyway...I could monkeypatch it, but then I'd have to
have some way of ensuring that users only run Larch on a single
specific Ruby version.

- Ryan

[Alex Barsamian]

On May 7, 12:14 pm, Ryan Grove <r...@wonko.com> wrote:
> However, there is a nasty deadlock bug in Ruby's Net::IMAP library
> that can cause it to hang indefinitely (without sending an exception
> to Larch) if the connection is dropped during certain stages of
> certain actions, and message transfer is the most common case where
> this occurs. There's a warning about this in the Known Issues section
> of the Larch readme.
>
> Unfortunately there's nothing I can do about this Net::IMAP bug. Well,
> nothing safe, anyway...I could monkeypatch it, but then I'd have to
> have some way of ensuring that users only run Larch on a single
> specific Ruby version.

What would the "monkeypatch" solution to this problem entail?

I'm running into what I think is this problem quite a bit with the
migration I'm trying to do, and in a reproducible way (always on
particular messages so I can't just CTRL-C and try again). I'm not
sure what I'd be looking for, but I can't find a common thread in the
messages that hang it up, though there seem to be several, and unless
I exclude the containing folder, larch hangs up consistently. The last
[debug] message is "peeking at message" and:

@pos: 3254
@lex_state: EXPR_DATA
@token.symbol: LPAR
@token.value: "("

is an example of the sort of thing I'd see in the last few lines of
output in "insane" mode.

The "to" server is gmail, while the "from" server is Dartmouth
College's BlitzMail system, which is small and obscure enough that I'm
not surprised when it's not well-supported or exhibits bugs.

If you have any insight as to how to get past this, I'd love to hear
your thoughts. Let me know if there's any other experimentation/info
gathering I can do for you. I think larch is awesome and I'd love to
make it work for me and make it stronger in the process if I can.

Thanks!
Alex

[Ryan Grove]

On Wed, Jun 24, 2009 at 2:37 PM, Alex Barsamian
<alexander....@gmail.com> wrote:
> What would the "monkeypatch" solution to this problem entail?

It's been a few months since I looked into this so my memory's a bit
rusty, but I believe you'd need to modify Net::IMAP's
get_tagged_response and get_response methods to be more robust about
handling different kinds of socket errors. There may also be cases
elsewhere in Net::IMAP where the library can get stuck endlessly
waiting for the next character of a malformed response, although I
haven't confirmed this yet.

The problem with monkeypatching a standard lib is that it differs
between Ruby versions, so you'd have to have a patch compatible with
every supported version, which would be a nightmare.

> I'm running into what I think is this problem quite a bit with the
> migration I'm trying to do, and in a reproducible way (always on
> particular messages so I can't just CTRL-C and try again).  I'm not
> sure what I'd be looking for, but I can't find a common thread in the
> messages that hang it up, though there seem to be several, and unless
> I exclude the containing folder, larch hangs up consistently. The last
> [debug] message is "peeking at message" and:
>
> @pos: 3254
> @lex_state: EXPR_DATA
> @token.symbol: LPAR
> @token.value: "("
>
> is an example of the sort of thing I'd see in the last few lines of
> output in "insane" mode.

Hmm. That's interesting. This may be a case where either the message
or the server response is somehow malformed and is causing Net::IMAP
to sit waiting for a token that never arrives.

One way to narrow this down a bit: manually copy one or more of the
problem messages to another server running different IMAP software,
then run Larch against both. If one server works fine and the other
doesn't, then it's likely to be a malformed server response that's the
issue. If both servers fail on the same message, then the message
itself is triggering the issue.

- Ryan

[Alex Barsamian]

> One way to narrow this down a bit: manually copy one or more of the
> problem messages to another server running different IMAP software,
> then run Larch against both. If one server works fine and the other
> doesn't, then it's likely to be a malformed server response that's the
> issue. If both servers fail on the same message, then the message
> itself is triggering the issue.

Thanks for the response!

I took your advice and tried the manual copy thing, using Apple Mail
to make the copy to gmail, delete from "to" and then trying to go the
opposite direction with larch: Worked fine. I dug a little deeper and
discovered that the parser was getting stuck in an infinite loop when
parsing the envelope as presented by BlitzMail, but not by gmail. I
compared how Blitzmail presented the message with how gmail does, and
it turns out BlitzMail's IMAP implementation fails to adhere to the
IMAP standard for envelopes in several ways, all involving either
incorrectly escaped parens or mysterious missing double-quotes around
things the standard specifies as nstrings. Such things confuse the
hell out of Net::Imap's parser.

Since envelope was causing so many problems, I monkeypatched my own
version by simply getting rid of the 'ENVELOPE' part of the FETCH
command in Mailbox.fetch and changing the reference to data.attr
['ENVELOPE'] just below it to nil. The envelope attribute of Message
isn't actually ever used except to get "to" and "from" in the info-log
lines.

Well, good enough for me. I'ma start it running tonight, get some
sleep, and see how it looks in the morning.

Thanks again,
Alex

[Ryan Grove]

On Sun, Jun 28, 2009 at 1:15 AM, Alex
Barsamian<alexander....@gmail.com> wrote:
> it turns out BlitzMail's IMAP implementation fails to adhere to the
> IMAP standard for envelopes in several ways, all involving either
> incorrectly escaped parens or mysterious missing double-quotes around
> things the standard specifies as nstrings. Such things confuse the
> hell out of Net::Imap's parser.

Thanks for following up with the results. I'll make a note of this in
the Larch documentation in case other users run into similar problems.

- Ryan

[Alex Barsamian]

> Thanks for following up with the results. I'll make a note of this in
> the Larch documentation in case other users run into similar problems.

Sure thing. Here is a bit more detail to my patch, you can describe
it in the docs or make an actual patch if you like:

In larch-1.0.1/lib/larch/imap/mailbox.rb, in the fetch function,
replace:

data = imap_uid_fetch([uid], [(peek ? 'BODY.PEEK[]' : 'BODY[]'),
'FLAGS', 'INTERNALDATE', 'ENVELOPE']).first

Message.new(message_id, data.attr['ENVELOPE'], data.attr['BODY[]'],
data.attr['FLAGS'], Time.parse(data.attr['INTERNALDATE']))

with

data = imap_uid_fetch([uid], [(peek ? 'BODY.PEEK[]' : 'BODY[]'),
'FLAGS', 'INTERNALDATE']).first

Message.new(message_id, nil, data.attr['BODY[]'],
data.attr['FLAGS'], Time.parse(data.attr['INTERNALDATE']))


And in larch-1.0.1/lib/larch.rb, in the copy_message function, replace
all of

if msg.envelope.from
env_from = msg.envelope.from.first
from = "#{env_from.mailbox}@#{env_from.host}"
else
from = '?'
end

@log.info "copying message: #{from} - #{msg.envelope.subject}"

with just

@log.info "copying message: #{msg.id}"

It's working fine now. 1.5g of emails transferred and counting.
Thanks for a great application.

Alex