message_num: How is it generated and can it be predicted?

[aaron ackley]

The Situation:

A 27GB mbox restore fails consistently at 59.20%  Restarts inluding from scratch removing the sqlite files fail to resolve

The Goal:

Delete the offending message from the mbox and try again

<username>-restored.sqlite lists a message_num for each of the previously restored emails.  My intent was to use that message num to find the last successful restore in the mbox and just edit/delete the next message.

However I don't see a way to identify or predict the next message_num in sequence.  

Things I've tried:

There's no documented way to give GYB instructions to skip a particular message

Searching this list for message_num related discussions produces nothing relevant

Nothing that i see in the message headers looks similar to message_num

Search the other way also produces nothing

     head -5000  Inbox.mbox  |   grep message_num

[Jay Lee]

Should be a straight up count of how messages GYB has seen in the mailbox. Try searching for regex "^From " (lines that start with exactly From and a space). These are new messages so the Nth occurrence of From(space) would be the problematic message.

Jay

--
--
You received this message because you are subscribed to the Google
Groups "Got Your Back: Gmail Backup" group.
To post to this group, send email to got-yo...@googlegroups.com
To unsubscribe from this group, send email to
got-your-bac...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/got-your-back?hl=en?hl=en

---
You received this message because you are subscribed to the Google Groups "Got Your Back: Gmail Backup" group.
To unsubscribe from this group and stop receiving emails from it, send an email to got-your-bac...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/got-your-back/964e6b32-1410-4258-a764-adeb35e76eef%40googlegroups.com.

[aaron ackley]

Thank you Jay, I did just about that, with a lot more fiddling around than was necessary

For posterity:

•  Open the sqlite file with DB Browser for Sqlite.  Identify the last record, in this case 57720.  Msg 57721 is the problem
•  Find all of the messages and count them and save their numbers  
  cat ./Inbox.mbox | grep @xxx > messageid.txt
  
  
• It should look something like this
  1:From 1618944912136401259@xxx Tue Dec 04 17:56:22 +0000 2018

  2:From 1621025231769352752@xxx Thu Dec 27 17:02:09 +0000 2018

  3:From 1617163843784063062@xxx Thu Nov 15 02:07:03 +0000 2018

  4:From 1617772480382737996@xxx Wed Nov 21 19:21:04 +0000 2018

  5:From 1621725029772087100@xxx Fri Jan 04 10:25:09 +0000 2019

  6:From 1622627734515990089@xxx Mon Jan 14 09:33:15 +0000 2019

  7:From 1622097921414122143@xxx Tue Jan 08 13:12:06 +0000 2019

  8:From 1614685030227585151@xxx Thu Oct 18 17:27:22 +0000 2018

  9:From 1615322143438061011@xxx Thu Oct 25 18:14:00 +0000 2018

  10:From 1617858233825729107@xxx Thu Nov 22 18:04:05 +0000 2018
  
  
  
  
• Find the offending message and the ones near it
   osboxes  ~  Desktop  cat msgidnumber.txt | grep -C 2 ^57721

  57719:From 1500571806527496245@xxx Fri May 08 03:46:31 +0000 2015

  57720:From 1500571802764911647@xxx Fri May 08 03:46:28 +0000 2015

  57721:From 1500366076565314117@xxx Tue May 05 21:16:32 +0000 2015

  57722:From 1500254005614019890@xxx Mon May 04 15:35:13 +0000 2015

  57723:From 1498958271032618853@xxx Mon Apr 20 08:20:04 +0000 2015
  
  
• use csplit to split your file at the next-good message -- 57722 in this case
  csplit -k Inbox.mbox '/1500254005614019890@xxx/'
  
  
• rename xx01 to xx01.mbox and restore that file

On Saturday, February 1, 2020 at 5:34:26 PM UTC-5, Jay Lee wrote:

Should be a straight up count of how messages GYB has seen in the mailbox. Try searching for regex "^From " (lines that start with exactly From and a space). These are new messages so the Nth occurrence of From(space) would be the problematic message.

Jay

On Sat, Feb 1, 2020, 5:16 PM aaron ackley <aaron...@gmail.com> wrote:

The Situation:

A 27GB mbox restore fails consistently at 59.20%  Restarts inluding from scratch removing the sqlite files fail to resolve

The Goal:

Delete the offending message from the mbox and try again

<username>-restored.sqlite lists a message_num for each of the previously restored emails.  My intent was to use that message num to find the last successful restore in the mbox and just edit/delete the next message.

However I don't see a way to identify or predict the next message_num in sequence.  

Things I've tried:

There's no documented way to give GYB instructions to skip a particular message

Searching this list for message_num related discussions produces nothing relevant

Nothing that i see in the message headers looks similar to message_num

Search the other way also produces nothing

     head -5000  Inbox.mbox  |   grep message_num

--
--
You received this message because you are subscribed to the Google
Groups "Got Your Back: Gmail Backup" group.
To post to this group, send email to got-yo...@googlegroups.com
To unsubscribe from this group, send email to

got-yo...@googlegroups.com

For more options, visit this group at
http://groups.google.com/group/got-your-back?hl=en?hl=en

---
You received this message because you are subscribed to the Google Groups "Got Your Back: Gmail Backup" group.

To unsubscribe from this group and stop receiving emails from it, send an email to got-yo...@googlegroups.com.

[Jay Lee]

I'd be curious to see that bad message by the way and understand why GYB can't handle it.

got-your-bac...@googlegroups.com

For more options, visit this group at
http://groups.google.com/group/got-your-back?hl=en?hl=en

---
You received this message because you are subscribed to the Google Groups "Got Your Back: Gmail Backup" group.

To unsubscribe from this group and stop receiving emails from it, send an email to got-your-bac...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/got-your-back/73f24325-bd6e-4bca-bfd4-3005ca895b5c%40googlegroups.com.

[aaron ackley]

Update:  possibly I'm wrong, or there is something else afoot that caused the consistent fail at that point.

Those messages restore fine as individual or as a 2-message mbox.

So, my issue is resolved, and unless you're keen on repeating the whole restore, I'm not sure how to replicate. I will re-run with debug if you like.

To view this discussion on the web visit https://groups.google.com/d/msgid/got-your-back/CA%2BVVBp_4uwTXF7M9B2AozcXyKRmz5Ad8cKe5Ouit6u2rDNwshA%40mail.gmail.com.

[Jay Lee]

What does the actual failure look like? I'm glad to try full reproduction with the file but that'd obviously take some time.

To view this discussion on the web visit https://groups.google.com/d/msgid/got-your-back/CAE9Oe32drs0u6dyAu%3DMg9CMLi3EmTn5YZ4HbfZAgz8x-BJPPhA%40mail.gmail.com.

[aaron ackley]

Since i still need to fast-restore to get the missed messages:

../gyb/gyb --email ryan...@hescoarmor.com --debug --fast-restore --action restore-mbox --local-folder . | grep -v "skipping already restored message" > debug-fastrestore.txt

this failed after a short while with this error to the terminal, the debug log is attached

 osboxes  ~  Desktop  Ryan  ../gyb/gyb --email ryan...@hescoarmor.com --debug --fast-restore --action rtore-mbox --local-folder . | grep -v "skipping already restored message" > debug-fastrestore.txt
Traceback (most recent call last):
  File "gyb.py", line 2157, in <module>
  File "gyb.py", line 1922, in main
  File "gyb.py", line 574, in callGAPI
  File "site-packages/googleapiclient/_helpers.py", line 130, in positional_wrapper
  File "site-packages/googleapiclient/http.py", line 1455, in execute
  File "site-packages/googleapiclient/http.py", line 1386, in _execute
  File "gyb.py", line 103, in wrapped_request_method
  File "site-packages/google_auth_httplib2.py", line 197, in request
  File "site-packages/httplib2/__init__.py", line 1967, in request
  File "site-packages/httplib2/__init__.py", line 1639, in _request
  File "site-packages/httplib2/__init__.py", line 1547, in _conn_request
  File "http/client.py", line 1230, in request
  File "http/client.py", line 1276, in _send_request
  File "http/client.py", line 1225, in endheaders
  File "http/client.py", line 1043, in _send_output
  File "http/client.py", line 965, in send
  File "ssl.py", line 1204, in sendall
  File "ssl.py", line 1173, in send
BrokenPipeError: [Errno 32] Broken pipe
[15345] Failed to execute script gyb
 osboxes  ~  Desktop  Ryan  

In the debug log, the break appeared to happen right around where it was happening previously at 58.20%   

Is that a special char (^M) in the debug log? significant?  The messages I identified as problematic was 57721, but that message restores fine on it's own and as part of a much smaller mbox

message 57724 - 58.20%
 reading message 57724... - 58.20%
 message 57725 - 58.20%
 reading message 57725... - 58.20%
 restoring 5 messages (57725) - 58.20%^Mconnect: (oauth2.googleapis.com, 443)
send: b'POST /token HTTP/1.1\r\nHost: oauth2.googleapis.com\r\nContent-Length: 242\r\ncontent-type: application
/x-www-form-urlencoded\r\nuser-agent: Got Your Back 1.34 | https://git.io/gyb | Jay Lee - jay...@gmail.com | P
ython 3.8.1 64-bit final | google-api-client 1.7.11 | Linux-4.15.0-58-generic-x86_64-with-glibc2.3.4 x86_64\r\n
accept-encoding: gzip, deflate\r\n\r\n'
send: b'grant_type=refresh_token&client_id=1011068670840-kg7vr17vrtvmb7ouv8dsn9ngru43bsqe&client_secret=5iTLYJ-
B8pE2dw-12T9Tfx5m&refresh_token=1%2F%2F0dQHmWl6q-3ygCgYIARAAGA0SNwF-L9Irvv7VNid8y17OCkgmG22zinv4SSPbT8pcCAq1V8Q
3Q7mZtoL6vHrfWHnJQ2EUDOt_CLM'

On Mon, Feb 3, 2020 at 9:07 AM aaron ackley <aaron....@gmail.com> wrote:

It looks pretty typical, but doesn't throw an error message like 400 Single From expected etc.   

Unfortunately it's well past the scrollback in the terminal, and it wasn't remarkable enough to save.  I've munged the original mbox quite a bit, but i could untar a new one.

No guarantee that the problem wasnt with how tar extracted though.   It's not a worry to run it again, just takes some time.

To view this discussion on the web visit https://groups.google.com/d/msgid/got-your-back/CA%2BVVBp9vss1DR51YKcqT4v7iqnH6jOtGqGtnCvFzMVKtGXoEZA%40mail.gmail.com.