group-restore failure partway due to API msg size limit - try again and excluding large mail?

[Sean Davis]

Hi there,

I tried to restore a user mailbox to a google-group for archival. There were around 4000 messages present in the mailbox. I came back to the restore to the error "Groups supports restore of messages up to 26214400" This is the 25MB API limit so I began digging.

If I check the .sqlite files, the -restored.sqlite file shows all 4000 message IDs as rows

If I check the msg-db.sqlite file, the count of rows and the message uids match those in the -restored.sqlite file. From this alone I assume all messages were restored. However, if I do a ./gyb --email group-restored-to@blah --action count it tells me around 3100 messages are in the mailbox. This makes me think all messages are NOT restored.

Is there any support for rerunning the restore against the existing group, but excluding all large messages when restoring?

If not, is it supported to edit the msg-db.sqlite, remove the uid and messageid rows for messages over 25MB (note - this is a single message with uid 3003 in my case), delete the .eml from the file structure, and run the restore process again with --noresume?

Thanks in advance for any insight here.
Sean

[Nick]

Hey,

I was also very confused by the info message: "Groups supports restore of messages up to 26214400". I've spent the entire day messing around, deleting emails manually before doing new backups repeatedly and wasting my time because I thought the script had exited due to an error. However, I had 0 emails that were larger than 25 MB. Conversations yes, but emails no and the latter is what matters from my understanding.

As Jay posted somewhere in this group: "Groups supports restore of messages up to 26214400" is not an error at all. But since there's no "Restore completed successfully" or anything else, I mistook it for an error.

I haven't used --action count myself, so I'll refrain from commenting on its functionality. Perhaps it's buggy?

I think you could disable conversation mode and the group inbox should show individual messages.

I also don't know what gyb does with files that are too big. Will it intelligently and silently skip? It might.

26214400 byte is technically 26.2 Megabyte and 25 Mebibyte, so I suppose files under 26.2 MB could be fine? That said, Google says 25 MB including metadata, body and attachments in their documentation, so I'd just stick to this smaller number to be safe.

I can suggest two options:

A) Simply move away that large .eml file and then run the script again. It will continue where it left off. If normal behaviour is to silently skip large files, it will be done quickly and upload nothing.

I don't know whether --noresume will cause duplicates or not, so you can try without it first or try including it and find out.

This is what the harmless skip warning looks like if you move the file but don't edit the database (I could not be bothered to edit it), it continued happily after printing this warning:

WARNING! file GYB-GMail-B...@email.com/2024/12/5/id.eml does not exist for message 165
  this message will be skipped.

B) Run a new, clean backup without too large files.

gyb --email exp...@email.com --service-account --search smaller:25M

That should do it. Restoring this backup will prevent any doubt whether it has to do with email size.

Good luck!

Note: In hindsight, deleting hundreds of emails at a time after a quick vetting was absolutely not a waste of time: the groups UI does not let you select multiple search results, I would have had to open and then delete 3000 emails one by one. Existing labels and filter functionality are lost in groups. Searching by date does not work for me either, I guess due to the metadata loss. So I can absolutely recommend anyone to clean up before restoring to a group, because managing old emails after restoring takes a lot of manual work.

In my opinion Google Calendar emails, (product) newsletters, ToS changes and the like are pointless filler, so I suggest getting rid of emails such as these in a hard-to-navigate environment such as groups.

Op vrijdag 10 januari 2025 om 00:53:48 UTC+1 schreef Sean Davis: