Can't get larch 1.1.0
sidney
very large Gmail folder as described in
http://groups.google.com/group/larch/browse_frm/thread/48268c9fa7af2708
and tried the solution it said there:
"If you're brave, you might want to try installing the very latest
Larch development gem from GitHub:
gem sources -a http://gems.github.com
sudo gem install rgrove-larch
"
In a more recent thread Ryan said
http://groups.google.com/group/larch/browse_frm/thread/8fa3320371e2cb8
"which wasn't added until Larch 1.1.0 (currently in development)"
However, when I install rgrove-larch using the above commands I get
version 1.0.2.3, not 1.1.0, and it does not have the changes to handle
large folders that I can see when I browse the repository on GitHub.
I did an uninstall and verified that no old larch files were still
around, installed again, and that did not help.
Is this related to this announcement on GitHub about no longer
building gems?
http://github.com/blog/515-gem-building-is-defunct
Is there now another way to get the latest development version of
larch?
Ryan Grove
> Is this related to this announcement on GitHub about no longer
> building gems?
>
> http://github.com/blog/515-gem-building-is-defunct
>
> Is there now another way to get the latest development version of
> larch?
Yep, sorry, I haven't updated the readme since GitHub decided to shut
off gem building without warning. You can install the latest Larch dev
gem from Gemcutter:
gem install larch --pre -s http://gemcutter.org
- Ryan
sidney
line in which I mistakenly had a --exclude INBOX option resulted in
the following error:
larch --from imaps://imap.example.com --to imaps://imap.gmail.com --
exclude INBOX --from-folder 'INBOX.ham' --to-folder ham --no-create-
folder --from-user USER --to-user US...@gmail.com -V debug
Library/Ruby/Gems/1.8/gems/larch-1.1.0dev20091006/lib/larch.rb:44:in
`init': undefined method `strip' for ["INBOX"]:Array (NoMethodError)
from /Library/Ruby/Gems/1.8/gems/larch-1.1.0dev20091006/lib/larch.rb:
40:in `map'
from /Library/Ruby/Gems/1.8/gems/larch-1.1.0dev20091006/lib/larch.rb:
40:in `init'
from /Library/Ruby/Gems/1.8/gems/larch-1.1.0dev20091006/bin/larch:90
from /usr/local/bin/larch:19:in `load'
from /usr/local/bin/larch:19
Then when I removed the --exclude INBOX I found that the
Net::IMAP::NoResponseError has gone away if I copy from a big folder
(tens of thousands of messages) to a new or empty folder, but if I
copy to a folder that already has tens of thousnds of messages, even
if I copy from a folder that only has two messages, the
Net::IMAP::NoResponseError is still there. Here is the end of the log
output from that:
[info] US...@imap.example.com: INBOX.ham: fetching message headers 1
through 2...
[debug] US...@gmail.com@imap.gmail.com: ham: getting mailbox status
[debug] US...@gmail.com@imap.gmail.com: ham: examining mailbox
[info] US...@gmail.com@imap.gmail.com: ham: fetching latest message
flags...
[debug] US...@gmail.com@imap.gmail.com: ham: removing 323 deleted
messages from the database...
[info] US...@gmail.com@imap.gmail.com: ham: fetching message headers
10937 through 42979...
[info] US...@gmail.com@imap.gmail.com: Net::IMAP::NoResponseError: Some
messages could not be FETCHed (Failure) (will retry)
[info] US...@gmail.com@imap.gmail.com: Net::IMAP::NoResponseError: Some
messages could not be FETCHed (Failure) (will retry)
[info] US...@gmail.com@imap.gmail.com: Net::IMAP::NoResponseError: Some
messages could not be FETCHed (Failure) (will retry)
[fatal] Net::IMAP::NoResponseError: Some messages could not be FETCHed
(Failure) (giving up)
[info] 0 message(s) copied, 0 failed, 2 untouched out of 2 total
Ryan Grove
>
> It installed, but running it with the following (obfuscated) command
> line in which I mistakenly had a --exclude INBOX option resulted in
> the following error:
>
> larch --from imaps://imap.example.com --to imaps://imap.gmail.com --
> exclude INBOX --from-folder 'INBOX.ham' --to-folder ham --no-create-
> folder --from-user USER --to-user US...@gmail.com -V debug
>
> Library/Ruby/Gems/1.8/gems/larch-1.1.0dev20091006/lib/larch.rb:44:in
> `init': undefined method `strip' for ["INBOX"]:Array (NoMethodError)
Sounds like a bug in the dev build. I'll look into it.
> Then when I removed the --exclude INBOX I found that the
> Net::IMAP::NoResponseError has gone away if I copy from a big folder
> (tens of thousands of messages) to a new or empty folder, but if I
> copy to a folder that already has tens of thousnds of messages, even
> if I copy from a folder that only has two messages, the
> Net::IMAP::NoResponseError is still there. Here is the end of the log
> output from that:
>
> [info] US...@imap.example.com: INBOX.ham: fetching message headers 1
> through 2...
> [debug] US...@gmail.com@imap.gmail.com: ham: getting mailbox status
> [debug] US...@gmail.com@imap.gmail.com: ham: examining mailbox
> [info] US...@gmail.com@imap.gmail.com: ham: fetching latest message
> flags...
> [debug] US...@gmail.com@imap.gmail.com: ham: removing 323 deleted
> messages from the database...
> [info] US...@gmail.com@imap.gmail.com: ham: fetching message headers
> 10937 through 42979...
> [info] US...@gmail.com@imap.gmail.com: Net::IMAP::NoResponseError: Some
> messages could not be FETCHed (Failure) (will retry)
Ugh. The "Some messages could not be FETCHed" error is a bug (or at
least some very bad behavior) in Gmail's IMAP interface. You can read
about it here (among other places):
http://www.google.com/support/forum/p/gmail/thread?tid=3cc1c96a8f210bd3&hl=en
The leading theory is that this error is caused by a message that
Gmail's web interface handles just fine, but which for some reason
causes their IMAP interface to choke. Every time an IMAP client sends
a FETCH command that includes this message, Gmail returns the error
above. The only solution seems to be to narrow down which message is
causing the problem and either delete it or move it to another folder.
I'd like to be able to have Larch automatically handle this problem,
but since the exact cause is unknown (which means I can't
intentionally create the problem in order to test solutions), I don't
have a fix for it right now.
If you're doing a one-time transfer, then a temporary workaround would
be to have Larch transfer mail to a new Gmail folder, then manually
move those messages into the desired folder using the Gmail web
interface.
- Ryan
sidney
> The leading theory is that this error is caused by a message that
> Gmail's web interface handles just fine, but which for some reason
> causes their IMAP interface to choke
when I tried to use Thunderbird to copy an archive of emails on one
IMAP server to a GMail folder. There were some emails that triggered
Google's spam filter really hard -- Nothing I could do would make
GMail put the spam into a folder (It really was spam, from a spamtrap
archive), but worse, GMail returned an error that aborted the copy of
the set of messages. This was not true of all mail that Google would
identify as spam, just certain ones that apparently had something they
consider so bad that they would not allow it into the system. I have a
vague recollection that there is a problem with
I wonder if this is a similar situation, for example maybe an email
that contains what is now recognized as a virus attachment. I'll see
if I can come up with an automated way of identifying the culprit
message out of the 44,000 or so in the folder and get some insight as
to why it is happening.
> If you're doing a one-time transfer, then a temporary workaround would
> be to have Larch transfer mail to a new Gmail folder, then manually
> move those messages into the desired folder using the Gmail web
> interface.
sidney
I followed the suggestion there of using openssl as a client to debug
with IMAP telnet commands. I fetched the UID of the full range that
Larch tried. All the requested messages were fetched except for one
that produced an error, and then there was the message at the end
saying that not all had been fetched. I tracked down the bad message
and tried to look at it in the GMail web interface. I got it by
searching for messages with that label within the date range of the
message that came before and after in the fetched list. It did show up
in the list of search results, but when I clicked to look at the
message, GMail just gave me a series of showing me an error "Oops… the
system encountered a problem (#754) Retrying in" and a time countdown
for endlessly repeating retries.
The message is not spam and I don't think has any attachments.
I had to close the GMail tab to get to stop the repeating error, then
I searched for the date range again and changed the label of the
message to a new one I created. Now that is the only mail message in
that folder and clicking on it to display it still produces the error.
It certainly seems like Google's problem. Would a correct workaround
be for Larch to ignore the error, missing just the one email, given
that all the other headers do fetch ok?
Ryan Grove
> I had to close the GMail tab to get to stop the repeating error, then
> I searched for the date range again and changed the label of the
> message to a new one I created. Now that is the only mail message in
> that folder and clicking on it to display it still produces the error.
>
> It certainly seems like Google's problem. Would a correct workaround
> be for Larch to ignore the error, missing just the one email, given
> that all the other headers do fetch ok?
Thanks for the additional analysis and details. Given what you've
described, it does seem like the best thing for Larch to do is to
ignore messages that trigger this error. The hard part, though, is
identifying them. When Larch scans headers in a folder, it does a
batch fetch of up to 1024 headers at a time, and the entire batch will
fail with this error if just one of the messages is corrupt. In order
to find the corrupt message, Larch would have to gradually decrease
the size of the batch until the fetch succeeds and the culprit is
identified, which could take a while.
Still, it'd be better for Larch to work around the problem slowly than
not work around it at all, so this is probably what I'll implement.
The only blocker now is that I'll need to be able to reproduce the
problem myself before I can test a solution for it. I don't currently
have any Gmail test mailboxes that are experiencing this problem, and
I haven't yet figured out a way to intentionally create a corrupt
message that will trigger it.
I don't suppose you'd be willing to try forwarding me your corrupt
message? Not sure if it'll even be possible if the web UI chokes on
the message, but it's worth a shot. Of course, I'll understand if
you'd prefer not to for privacy reasons.
- Ryan
sidney
> I don't suppose you'd be willing to try forwarding me your corrupt
> message? Not sure if it'll even be possible if the web UI chokes on
> the message, but it's worth a shot. Of course, I'll understand if
> you'd prefer not to for privacy reasons.
I found that I could change the labels on the message in the web
interface. I can't open it, which means that I can't save it as a file
or forward it. Some discussion of the problem talked about downloading
messages using POP3. I don't know if they were successful at downloading
the offending message, or if the POP3 download was a way of getting all
messages except that one. The disadvantage of downloading everything as
POP3 is losing the labels. I found no way to download just one message
via POP3. The only POP3 choices I seem to have are to set the Gmail
account to let me download new messages that are incoming after I make
the setting, or else download all messages.
I created a new label for it to isolate it, and I'm in the process of
using larch to copy all the other folders to another gmail account. Then
I'll delete all the other messages from the original account and see if
POP3 works to read it. When that fails, I suppose I could change the
password on the account and trust you with access to it long enough to
debug this thing, but I hope I can manage to get the message on a new
dummy Gmail account exhibiting the problem so I can just hand that to
you. The larch synching will take a while longer as there are over
70,000 messages to transfer.
sidney
offending message found one other bad message. I isolatd that one too
and restarted the copy. That died by hanging with 18431 messages in the
new account.
Now when I restart the larch run it dies with an illegal instruction
error at the same place each time. The output looks like this (editing
out the real usernames and the [time stamp] and [info] om each line):
$ larch "gmailtogmail2"
U...@imap.gmail.com: connecting...
U...@imap.gmail.com: connected on port 993 using SSL
U...@imap.gmail.com: authenticated using PLAIN
U...@imap.gmail.com: connecting...
U...@imap.gmail.com: connected on port 993 using SSL
U...@imap.gmail.com: authenticated using PLAIN
copying messages from imap.gmail.com/f1 to imap.gmail.com/f1
copying messages from imap.gmail.com/f2 to imap.gmail.com/f2
U...@imap.gmail.com: f2: fetching latest message flags...
U...@imap.gmail.com: f2: fetching latest message flags...
copying messages from imap.gmail.com/spamtrap to imap.gmail.com/f3
U...@imap.gmail.com: f3: fetching latest message flags...
U...@imap.gmail.com: f3: fetching latest message flags...
U...@imap.gmail.com: f3: fetching message headers 18680 through 18698...
Illegal instruction
I'm going to try to exclude the folder that I called f3 in the above log
and see if I can get everything else backed up and then see what is
going on with that folder.
sidney
gmailtogmail2:
from: imaps://imap.gmail.com
from-user: USER1
from-pass: PASSWORD1
to: imaps://imap.gmail.com
to-user: USER2
to-pass: PASSWORD2
all: true
verboaity: debug
exclude:
- "[Gmail]/All Mail"
- "[Gmail]/Spam"
- "[Gmail]/Trash"
- "INBOX"
- "xclude1"
- "f2"
And here is the error I am getting now that I excluded "f2" which is
where the illegal instruction error was happening
[info] copying messages from imap.gmail.com/[Gmail]/Sent Mail to
imap.gmail.com/[Gmail]/Sent Mail
[info] US...@imap.gmail.com: [Gmail]/Sent Mail: fetching message
headers 1 through 637...
[info] copying messages from imap.gmail.com/[Gmail] to imap.gmail.com/
[Gmail]
[fatal] unable to get status of mailbox: Invalid folder: [Gmail]
(Failure)
I'll try adding an exclude of [Gmail] and see if that helps.
sidney
> [fatal] unable to get status of mailbox: Invalid folder: [Gmail]
> (Failure)
>
> I'll try adding an exclude of [Gmail] and see if that helps.
Adding the exclude for [Gmail took care of that problem, and the copy
completed, without the folder that led to the illegal instruction.
I then removed the exclude for that folder, which now is the only one
not fully synched, and that repeated the illegal instruction error.
Using openssl for telnet access to imap I did a fetch 1:n rfc822.header
on that folder to verify that there are no fetch failures in that
folder. So that doesn't explain the illegal instruction trap.
I now have deleted the larch.db file in case the problem is related to
larch caching the flags for the problematic email that I have moved to
its own folder, and I'm running larch again now to see what happens.
This testing has revealed another bug: In config.yaml I have a line
verbosity: debug
but the output only shows info level unless I add -V debug to the
command line.
sidney
On Oct 28, 8:21 am, sidney <sidn...@gmail.com> wrote:
> I now have deleted the larch.db file in case the problem is related to
> larch caching the flags for the problematic email that I have moved to
> its own folder, and I'm running larch again now to see what happens.
if their cached information on a folder is out of date and has to be
retrieved again? I assume that you can detect if a message has been
added to a folder and the cache has to be updated, but it looks like
you don't detect if a message has been removed from the middle of the
folder.
My next steps are to verify that everything except the two problematic
messages were copied, then delete them from the first account, then
see if POP3 succeeds in retrieving the messages, and if it does see if
it reproduces the problem when they are copied to a new Gmail account.
sidney
>
> verbosity: debug
config.yaml file in the word "verbosity". So that one's a false alarm.
On the other hand most of the folders were not copied even thought the
info output shows the message headers being downloaded.
Ryan Grove
>> I now have deleted the larch.db file in case the problem is related to
>> larch caching the flags for the problematic email that I have moved to
>> its own folder, and I'm running larch again now to see what happens.
>
> That did it. Is there a standard method that IMAP clients use to tell
> if their cached information on a folder is out of date and has to be
> retrieved again? I assume that you can detect if a message has been
> added to a folder and the cache has to be updated, but it looks like
> you don't detect if a message has been removed from the middle of the
> folder.
This could indicate a bug in Larch's DB code. The "Illegal
instruction" error certainly isn't coming from Gmail or from the Ruby
Net::IMAP lib itself. Since it's not accompanied by a Ruby stack trace
and since deleting larch.db fixes it, my best guess is that it's
happening somewhere in the native SQLite adapter.
Some quick searching reveals that SQLite tends to throw illegal
instruction errors and die when it runs out of memory, so we may have
a memory exhaustion issue here, which implies that either Larch is
operating on the DB in a memory-inefficient way, or you're running
Larch on a machine with very limited memory.
- Ryan
sidney
> which implies that either Larch is
> operating on the DB in a memory-inefficient way, or you're running
> Larch on a machine with very limited memory.
other applications, so it probably isn't lack of memory on the
machine.
Also, the issue of not copying most of the folders was consistent. I
moved the server configuration portion of config.yaml into default
then used a bash command line like
for i in f1 f2 f3 ; do larch default -F "$i" -T "$i" ; done
and that seems to be working fine copying all the messages that were
skipped over by the configuration I posted earlier in this thread.
Ryan Grove
> It's running on an Intel iMac with 3GB RAM that is not running nay
> other applications, so it probably isn't lack of memory on the
> machine.
Definitely sounds like a Larch problem then. I'll investigate.
> Also, the issue of not copying most of the folders was consistent. I
> moved the server configuration portion of config.yaml into default
> then used a bash command line like
> for i in f1 f2 f3 ; do larch default -F "$i" -T "$i" ; done
> and that seems to be working fine copying all the messages that were
> skipped over by the configuration I posted earlier in this thread.
Are the folders that aren't being copied subscribed? If not, it's
possible that the "all" config option is not being respected properly.
I'll look into this as well.
- Ryan
sidney
> Are the folders that aren't being copied subscribed? If not,
Ryan Grove
> if their cached information on a folder is out of date and has to be
> retrieved again? I assume that you can detect if a message has been
> added to a folder and the cache has to be updated, but it looks like
> you don't detect if a message has been removed from the middle of the
> folder.
Just realized I forgot to respond to this (it's been a busy day).
The method Larch uses is (I think) the same one used by Thunderbird
and most, if not all, other IMAP clients. Each time Larch opens a
mailbox, it requests a complete list of all the message UIDs and flags
in that mailbox. Any UIDs that are in the database but not in the
server's response have been deleted since Larch last saw them, so
Larch deletes them from the database to keep things in sync.
Assuming the server is following the spec and not reassigning UIDs,
this should correctly detect any messages in the mailbox that have
been removed, regardless of their position in the mailbox. If you run
Larch with --verbosity debug, you should see a message like "removing
{number} deleted messages from the database..."
Note that a message that has been flagged \Deleted on the server is
not actually considered removed and will still exist in the mailbox
(and in Larch's database) until it's actually expunged from the
mailbox. This may be what you're seeing.
- Ryan
sidney
> Note that a message that has been flagged \Deleted on the server is
> not actually considered removed and will still exist in the mailbox
> (and in Larch's database) until it's actually expunged from the
> mailbox. This may be what you're seeing.
you speculated about earlier. What I did was find a message that could
not be fetched - it can be seen in Gmail's search results - and
removed its label and gave it another one, in effect moving it from
one folder into its own folder. When I then ran larch again it got the
illegal instruction exception while trying to copy that first folder.
That's the error that went away when I deleted larch.db.
Once the current copy of all the remaining unsubscribed folders is
completed, which looks like quite a few more hours, I'll delete all
those messages from the first account leaving just the two unfetchable
ones. Then I can experiment to see if moving one of them to another
folder after the db is created reproduces that illegal instruction
error. I can also then see if POP3 can retrieve those messages and if
it can, whether copying it to another gmail account reproduces the
original problem and gives you an actual test case.
sidney
> What I did was find a message that could
> not be fetched - it can be seen in Gmail's search results - and
> removed its label and gave it another one
so I guess it is something else. I'm completing the copy by using the
command line I mentioned earlier
for i in f1 f2 f3 ; do rm ~/.larch/larch.db ; larch default -F "$i" -
T "$i" ; done
and we'll see what happens.
sidney
individually did complete without apparent error, but some of the
folders did not contain all the messages that were supposedly copied
to them. For example in one folder the source had over 6,000 messages
and the destination ended up with 359. I ran the command I mentioned
in my last post to delete larch.db and copy each folder individually.
The info and debug output showed that larch found the correct number
of messages in the source folder and the current number of messages in
the destination folder but then did not copy any messages, ending up
by saying that there were 0 failures and 0 copies and the correct
source number of messages left untouched.
I then used Thunderbird to copy some of the messages for one of the
folders, stopping it after around 2,000 of the 6,000 were transfered.
I then ran larch again on that folder and this time it appears to be
copying messages.
Ryan Grove
> The info and debug output showed that larch found the correct number
> of messages in the source folder and the current number of messages in
> the destination folder but then did not copy any messages, ending up
> by saying that there were 0 failures and 0 copies and the correct
> source number of messages left untouched.
Certainly sounds like there could be a problem in the mailbox scanning
code in the dev build. I'm keeping track of the issues you've
reported, but I won't be able to investigate them in depth until this
weekend at the earliest.
- Ryan
sidney
I finally was able to get through processing the tens of thousands of
old emails to filter down to ones that cause problems.
There are two messages that are stuck in my Gmail box that causes
Google to fail when anything tries to fetch anything more substantial
than their UID and some of the header information. Even if I try to
look at the messages in Gmail's web interface, the interface goes into
an infinite loop of displaying an error message. I'm trying to work
out how I can minimize use of that Gmail account and then give you a
password so you can play directly with those emails.
There is also another message that causes larch to fail, and this
problem I can reproduce more simply. I suspect it is a larch bug and
not in Google.
To reproduce the problem create an email with the following incorrect
format of Subject header. This violates RFC2047 in trying to use
charset encoding, but this format does show up, especially in spam,
and should not hang larch like it does. As I read RFC2047, when you
use the =? ?= syntax to embed non-ASCII in a header line, you need to
have a matching ?= on every line that has an opening =?, you are not
allowed to let it cross a line boundary.
Subject: =?ISO-8859-1?Q?Nokia new phone
design?=
The whitespace before the "design" in my test case is a single tab
character. I don't know if the type of whitespace there makes a
difference to the bug.
To test this I created a message in a file in eml format, imported it
into a local mailbox folder in Thunderbird using an add-on named
ImportExportTools, then in Thunderbird copied the message to an IMAP
folder of a Gmail account. Then I deleted larch.db, used larch to copy
that folder to a different Gmail account.
When I did that, larch produced no error messages but the email was
not copied.
I then ran larch again without deleting larch.db. This time, larch
hung after the debug message saying that it was about to peek at the
messsage in the source mailbox. Adding some debug output I fond that
it never returns from the call to fetch (peek at) the message.
If you can't reproduce this I can send you the email file that causes
it.
Note that Gmail has no problem with this email - I can fetch it in
Thunderbird and copy it between accounts. It appears to be larch that
has the problem in this case.
That's unlike the other two messages I have in which Google will not
let them go or make their body text visible. I'll let you know more
about those two when I work out how to get you access to them.
Note when testing like this, that if you delete and even expunge a
message in a Gmail account in Thunderbird, it is not deleted, it still
shows up in [Gmail]/All Mail, just without the folder name as a label.
You delete it my moving it to the [Gmail]/Trash folder and then
deleting it from there.
Ryan Grove
> To reproduce the problem create an email with the following incorrect
> format of Subject header. This violates RFC2047 in trying to use
> charset encoding, but this format does show up, especially in spam,
> and should not hang larch like it does. As I read RFC2047, when you
> use the =? ?= syntax to embed non-ASCII in a header line, you need to
> have a matching ?= on every line that has an opening =?, you are not
> allowed to let it cross a line boundary.
as described, but Larch isn't choking on it.
- Ryan