Dives disappeared from log after syncing to cloud

[Doug]

iPhoneX on iOS 12.3.1 with subsurface 2.1.6(4.8.4.66)
iPad on iOS 12.2 with same subsurface version

I had synced 8 dives from my suunto eon core and logged the dives in subsurface on my iPhone awhile ago.
Today I opened the app on my iPad and tried to sync from the cloud and the dives didn’t show up. So I opened my iPhone app and when it synced to the cloud it popped up a message saying there was a problem with my local data cache and that I copied it to some /var/... directory that was too long to see the whole thing in the message box. And then it synced with the cloud and the 8 newest dive logs disappeared.
I’m not sure if the iPad sync overwrote the cloud data killing those dive logs or if they had never been synced to the cloud in the first place?
Does anyone know how I can recover this data?

Thanks,
Doug

[Dirk Hohndel]

> On Jun 16, 2019, at 6:01 AM, Doug <doug....@gmail.com> wrote:
>
> iPhoneX on iOS 12.3.1 with subsurface 2.1.6(4.8.4.66)
> iPad on iOS 12.2 with same subsurface version
>
> I had synced 8 dives from my suunto eon core and logged the dives in subsurface on my iPhone awhile ago.

And I assume you made sure that they are synced to the cloud? On mobile it tends to be worth to force a manual sync by pulling down the dive list and holding for like half a second until it forces a sync.
Even more so on iOS as that doesn't allow us to sync in the background (which we try to do on Android - not always successfully).

> Today I opened the app on my iPad and tried to sync from the cloud and the dives didn’t show up. So I opened my iPhone app and when it synced to the cloud it popped up a message saying there was a problem with my local data cache and that I copied it to some /var/... directory that was too long to see the whole thing in the message box. And then it synced with the cloud and the 8 newest dive logs disappeared.

Yeah, so it had them locally, but didn't sync them to the cloud, and then you must have had a change on the cloud side that managed to confuse things (that's exceptionally rare to happen, but it does happen). And it gave up.
Now, if you had an Android phone, I'd know how to get to the lost data. On iOS? No idea.

> I’m not sure if the iPad sync overwrote the cloud data killing those dive logs or if they had never been synced to the cloud in the first place?
> Does anyone know how I can recover this data?

Sadly, not really. They are most likely still happily saved in the iOS file system, but I have no idea how to get to them.

/D

[Doug]

Thanks for the quick response
On Sunday, June 16, 2019 at 1:32:33 PM UTC-4, Dirk wrote:

> > On Jun 16, 2019, at 6:01 AM, Doug wrote:
> >
> > iPhoneX on iOS 12.3.1 with subsurface 2.1.6(4.8.4.66)
> > iPad on iOS 12.2 with same subsurface version
> >
> > I had synced 8 dives from my suunto eon core and logged the dives in subsurface on my iPhone awhile ago.
>
> And I assume you made sure that they are synced to the cloud? On mobile it tends to be worth to force a manual sync by pulling down the dive list and holding for like half a second until it forces a sync.
> Even more so on iOS as that doesn't allow us to sync in the background (which we try to do on Android - not always successfully).
>

I’m not 100% sure whether they were synced to the cloud originally. I’ll be sure to force the sync in the future.

> > Today I opened the app on my iPad and tried to sync from the cloud and the dives didn’t show up. So I opened my iPhone app and when it synced to the cloud it popped up a message saying there was a problem with my local data cache and that I copied it to some /var/... directory that was too long to see the whole thing in the message box. And then it synced with the cloud and the 8 newest dive logs disappeared.
>
> Yeah, so it had them locally, but didn't sync them to the cloud, and then you must have had a change on the cloud side that managed to confuse things (that's exceptionally rare to happen, but it does happen). And it gave up.
> Now, if you had an Android phone, I'd know how to get to the lost data. On iOS? No idea.
>
> > I’m not sure if the iPad sync overwrote the cloud data killing those dive logs or if they had never been synced to the cloud in the first place?
> > Does anyone know how I can recover this data?
>
> Sadly, not really. They are most likely still happily saved in the iOS file system, but I have no idea how to get to them.

Yeah I figured they probably were lost.
I’m wondering if it’s possible to re-download the dives from my dive computer. Obviously I wouldn’t get the details/notes I’d hand entered, but I can probably reconstruct some of it from my brain.
I tried to redownload them but it said no new dives were present
Not sure whether it’s the app or the DC keeping track of what’s been downloaded already.
Any ideas how to redownload?
>
> /D

[Shenji]

Hi there,

I just encountered a similar problem but on Android. After entering some dive details in the program, I clicked on "Manually sync to Cloud".  Guess I hadn't synced to the Cloud for a while and I lost about a 100 dives after it said there was a problem and it stored the local log updates somewhere on the device.

Any clue where the location is and how to restore what I had locally prior to the sync?

Thanks a bunch!

S

[Dirk Hohndel]

Unsurprisingly, in situations like this, details MATTER.

Had you IMMEDIATELY after this happened sent a support request from within the app, the log would likely have told me all I needed to know. Sadly, that chance has been lost.

Please help me better understand what happened.

You manually entered a 100 dives (why would anyone do this on mobile???) and then started a manual sync and the dives were lost?

I'm pretty sure it did more than just "say that there was a problem" - there almost certainly was a more specific error message. Did you take a screen shot?

Looking at the meta data of your cloud account, I can see that there has been no attempt to write data to that account in 20 months. It has been accessed from Windows 7 (which isn't officially supported anymore) and Android 7.1 (which also likely will cause you problems with current versions of Subsurface).

There's a chance that you can restore all the data, but the details depend a bit on what exactly happened.

Can you tell me which dives are visible on your Android device right now? The ones that you entered, or the ones that were in the cloud account (as written by the Windows system)?

If you go to settings and enable developer mode at the very bottom of the advanced features section, you'll get a new menu entry in the main menu called (surprise!) Developer. If you open that menu, the bottom entry is "Access local cloud cache dirs". When you tap on that menu entry, what does it tell you?

Please try to be precise when responding - I'm trying to help, and the more information I have, the better the chances that we can recover the dives that appear to have disappeared.

/D

[Shenji Schaeppi]

Hi Dirk,

Thanks for responding so quickly :) So, couple more details. My issue is very similar to the issue the person I responded to had:

1. I opened Subs Mobile. It already had 326 or so dives, mostly downloaded from my DC and edited on my PC.

2. I then downloaded another 60 odd dives from my DC yesterday and added some details (don't worry, I didn't enter 100 dives manually :))

3. I then saw the Manual sync option and thought it'd be good to backup my dives to the cloud

4. Then an error popped up (no screenshot) saying something like "Problem loading local log?...moving to local storage... [some cut off directory path]

5. After the sync finished, I only had 230 odd dives, what I believe to be the latest which was on the Cloud from a sync 20 months or so ago

6. Tried the developer option, I get a screen with no button (see screenshot)

Hope this helps and thanks for your help!

S

P.S. I didn't yet update to the latest SubS just in case

--
You received this message because you are subscribed to a topic in the Google Groups "Subsurface Divelog" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/subsurface-divelog/n1HvtAl33H0/unsubscribe.
To unsubscribe from this group and all its topics, send an email to subsurface-dive...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/subsurface-divelog/1AED76AC-97E0-4043-BA7C-088AED4A2F11%40hohndel.org.

[Willem Ferguson]

Your computer has a local copy of your dive log (that is if you actually started your dive log with a local copy) and another copy in the cloud. If the copy in the cloud has been damaged, you still have a local copy on your machine. After the cloud copy has been damaged, it is best to abandon the cloud data and to restart with your local copy which is (possibly) not damaged.

Your cloud copy is extremely unlikely to irretrievably damaged. When it goes wonky all that is likely to have happened is that an error record has been added to the dive log on the cloud. All the data are till there. BUT, to recover the data you need a good idea of when your various edits (e.g. dive uploads and manual editing) were done to the dive log.

In general I feel it is risky to edit your dive log from more than one device. Most of the time editing your dive log on both one's phone and desktop computer just works, but the frequency of problems is too high for my liking. I do all edits on my divelog from my desktop workstation. This is because the fully-featured subsurface is not available on the phone but the workstation version is fully featured. The dive log on my phone is just for looking up information and for dive companies that want to see my dive history. If you would anyhow like to edit your dive log from both devices, I would suggest also saving all edits to the local copy on your laptop or workstation BEFORE syncing with the cloud. If you work using your local copy, that should almost always automatically happen. If working with a local dive log is not easy, then consider using 3rd party storage such as dropbox, just for backup purposes.

Kind regards,

willem

You received this message because you are subscribed to the Google Groups "Subsurface Divelog" group.
To unsubscribe from this group and stop receiving emails from it, send an email to subsurface-dive...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/subsurface-divelog/CANNi6ND5tri_oUrONMD29bei0okmnjgE_i5rgpgrPd_z6XnTsw%40mail.gmail.com.

This message and attachments are subject to a disclaimer.
Please refer to http://upnet.up.ac.za/services/it/documentation/docs/004167.pdf for full details.

[Shenji Schaeppi]

Thanks Willem,

Do you know where and under what file name SubS stores this local log on Android when it encounters a problem syncing from the Cloud?

Thanks and kind regards,

S

To view this discussion on the web visit https://groups.google.com/d/msgid/subsurface-divelog/39b9d622-17ba-cb6c-4b16-7affea2f5b4a%40zoology.up.ac.za.

[Dirk Hohndel]

Hi Shenji,

A few clarifications. 

Anything that was ever saved to the cloud is still there.

Anything that was ever edited on your device, is also still there - but it may be tricky to get to, depending on what exactly went wrong and "how wrong" it went.

I appreciate the details that you have given, but want to make sure I understand you correctly.

- you believe that on your phone there were about 380+ dives (the "326 or so" dives that were already there, plus the "60 odd" new ones)

- you tried a manual sync, and after that you got an error message and after that you had only 230 dives

- those 230 dives are the ones that you can see on your Windows system

Is all this correct?

The message that you describe is a bit inconsistent with that behavior, but we'll figure that out as well.

Can you open Subsurface-mobile on your phone, go to settings and send me a screen shot of that?

Next, can you please create a support request from within the app (Help->Ask for support) so I get to see the logs of the current state?

Also, while I know that this is frustrating - please don't try to uninstall / reinstall / update / do anything else that might mess with device storage. Right now the dives are in only exactly one place - the local storage of your phone. And it's unfortunately frustratingly easy to damage this.

One last question... how "technical" are you... i.e., are you a computer savvy person, comfortable with the command line and installing apps on your computer, configuring things, etc - or are you more of an "end user" who happily follows instructions, but really wants to have something to click on instead of typing in commands and command line options and stuff?

Thanks for your patience - I'm still reasonably optimistic that we'll be able to recover your data.

/D

You received this message because you are subscribed to the Google Groups "Subsurface Divelog" group.

To unsubscribe from this group and stop receiving emails from it, send an email to subsurface-dive...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/subsurface-divelog/CANNi6ND5tri_oUrONMD29bei0okmnjgE_i5rgpgrPd_z6XnTsw%40mail.gmail.com.
<截屏_20220827_091152.jpg>

[Dirk Hohndel]

Hi Willem

I love your enthusiasm and your support for Subsurface... but there are a few things in this message that I feel I have to correct.

- it's almost impossible to actually "damage" the copy of the data in the cloud. Unless you are using git on the command line and try to mess with the repo directly -- and even then git tries to make it painfully obvious if you are doing some damaging: you need to use the '--force' option to really delete things on the server... and even THEN, given the redundant servers that I have built, the backup server would still hold the old version because I don't allow force pushes between the sibling servers... so you'd have to manually force push to both upstream servers. At which point... yeah, you clearly asked for it.

- I would never ever ever suggest to a user to "abandon the cloud and restart with your local copy". The only situation where that is good advice is if someone really has never used the cloud or has managed to make changes on two different systems (and kept making diverging changes without ever syncing) and then had a catastrophic merge error. In the eight years of maintaining our cloud servers and with more than 20000 active users by now, there are two people who managed to do that... and one later admitted that he was using the command line to try to 'fix' things which resulted in the problem being irrecoverable. But the general advice to give to users should always be "STOP WHAT YOU ARE DOING AND TALK TO DIRK"

- Editing dive logs from multiple devices isn't risky at all. Making conflicting changes from multiple devices without syncing can be problematic. What do I mean by that? The following work flow will create no risk. Period.

PC: download dives

PC: sync to cloud

Mobile: sync to cloud

Mobile: edit

Mobile: download dives

Mobile: edit

Mobile: sync to cloud

PC: sync to cloud

PC: edit

etc.

As long as you are careful to sync on both devices (or on all 12 in my case - I currently use my live, real cloud storage with my real dives in it from 12 different computers, VMs, phones, and iPads -- with absolutely zero issues) before and after making ANY changes, you are perfectly safe.

I repeat. You go to a new device that isn't the last one you used, you sync to the cloud. You are done doing stuff, even if you aren't sure if you made any changes? You sync to the cloud. I believe it is that second sync that is most problematic and that people keep forgetting.

The workflows that can create problems are the ones where you allow the devices to get out of sync

PC: download dives

PC: edit dives

Mobile: edit dives

Mobile: sync

PC: sync

In almost all sane cases, this STILL will work just fine. But there can be issues. And the most likely way to create issues is if you make conflicting edits. So on both the PC and the Mobile, while they are in out of sync state, you edit the SAME dive. Or the same dive site. Etc.

There is this old saying.

"Doctor, it hurts when I poke a spoon into my eye"

Do you know the doctor's recommendation?

"Don't do that."

I rest my case.

As to the rest of your advice. I respectfully and strongly disagree with it. Asking people to store locally, sync things to dropbox (which doesn't work from mobile), do any of these things, will significantly increase the likelihood of people making mistakes, of people getting systems out of sync, and of people losing dives that were never saved to the cloud because they followed the advice to save locally and then didn't sync them.

I would appreciate if you didn't give that advice in our forum. I end up being the person who tries to help users recover their lost data. And I'd prefer the maximum chance of success for that. And things that are on dropbox or distributed in random exports on people's file system are generally not things I can help with. But unless someone makes intentionally dangerous changes from the command line to the remote git repos, anything that was ever in the cloud is still there and can be recovered. And I am happy to spend the time to do that for our users. Hundreds and hundreds of hours over the years... but saving someone's dive log is worth it to me -- and THAT is the reason why I am so consistently refusing requests to allow other cloud backends... because there the users are out of luck if things go wrong

Cheers

/D

To view this discussion on the web visit https://groups.google.com/d/msgid/subsurface-divelog/39b9d622-17ba-cb6c-4b16-7affea2f5b4a%40zoology.up.ac.za.

[Dirk Hohndel]

PLEASE don't try this unless you are working with me.

The moment you mess with the local file system you can truly damage things and lose things.
Subsurface doesn't save "a dive log" to your file system. It saves a local copy that caches the git repo on the cloud server. So that's tons and tons of files that contain what looks to any regular tool like random garbage. And messing with those files is likely to have permanently damaging results.

/D

[Shenji Schaeppi]

Thanks Dirk,

Please see my response inline below.

Kind regards,

Shenji

On Sat, Aug 27, 2022, 16:54 Dirk Hohndel <di...@hohndel.org> wrote:

Hi Shenji,

A few clarifications. 

Anything that was ever saved to the cloud is still there.

Anything that was ever edited on your device, is also still there - but it may be tricky to get to, depending on what exactly went wrong and "how wrong" it went.

I appreciate the details that you have given, but want to make sure I understand you correctly.

- you believe that on your phone there were about 380+ dives (the "326 or so" dives that were already there, plus the "60 odd" new ones)

- you tried a manual sync, and after that you got an error message and after that you had only 230 dives

- those 230 dives are the ones that you can see on your Windows system

Is all this correct?

YES - Correct, except I see the 230 dives on SubS mobile. I haven't opened SubS on my PC yet.

The message that you describe is a bit inconsistent with that behavior, but we'll figure that out as well.

Can you open Subsurface-mobile on your phone, go to settings and send me a screen shot of that?

DONE - Attached

Next, can you please create a support request from within the app (Help->Ask for support) so I get to see the logs of the current state?

DONE 

Also, while I know that this is frustrating - please don't try to uninstall / reinstall / update / do anything else that might mess with device storage. Right now the dives are in only exactly one place - the local storage of your phone. And it's unfortunately frustratingly easy to damage this.

UNDERSTOOD

One last question... how "technical" are you... i.e., are you a computer savvy person, comfortable with the command line and installing apps on your computer, configuring things, etc - or are you more of an "end user" who happily follows instructions, but really wants to have something to click on instead of typing in commands and command line options and stuff?

Comfortable with command line and configuring things if this makes resolving this easier.

To view this discussion on the web visit https://groups.google.com/d/msgid/subsurface-divelog/0BAC5CFE-9D60-4966-A19C-080EF11C0757%40hohndel.org.

[Dirk Hohndel]

Hi Shenji

On Aug 27, 2022, at 10:05 AM, Shenji wrote:

I appreciate the details that you have given, but want to make sure I understand you correctly.

- you believe that on your phone there were about 380+ dives (the "326 or so" dives that were already there, plus the "60 odd" new ones)

- you tried a manual sync, and after that you got an error message and after that you had only 230 dives

- those 230 dives are the ones that you can see on your Windows system

Is all this correct?

YES - Correct, except I see the 230 dives on SubS mobile. I haven't opened SubS on my PC yet.

That's what I meant. On mobile you see those 230 dives. So essentially, when you synced, Subsurface-mobile decided to grab only the remote data and deleted the local data. That's bad and shouldn't happen, but let's see if that's the last word in this saga.

The message that you describe is a bit inconsistent with that behavior, but we'll figure that out as well.

Can you open Subsurface-mobile on your phone, go to settings and send me a screen shot of that?

DONE - Attached

Thanks - the data there is consistent with the log that you sent me in the next step

Next, can you please create a support request from within the app (Help->Ask for support) so I get to see the logs of the current state?

DONE 

I received that. Thanks.

Also, while I know that this is frustrating - please don't try to uninstall / reinstall / update / do anything else that might mess with device storage. Right now the dives are in only exactly one place - the local storage of your phone. And it's unfortunately frustratingly easy to damage this.

UNDERSTOOD

:)

One last question... how "technical" are you... i.e., are you a computer savvy person, comfortable with the command line and installing apps on your computer, configuring things, etc - or are you more of an "end user" who happily follows instructions, but really wants to have something to click on instead of typing in commands and command line options and stuff?

Comfortable with command line and configuring things if this makes resolving this easier.

Good. Let me dig into this for a moment and come up with a theory what happened (and based on that, with an idea how to proceed)...

/D

[Dirk Hohndel]

Quick update - I sent you a request to allow me to access your cloud data to the email address from which you sent the support request.
Please respond to that email and give me access.
There are basically two different ways to continue from here.
The first one is relatively pain-free for you (I'll make some changes on the server and then, AFTER I TELL YOU, you initiate a manual sync on the phone)
This may or may not work

The second one requires you to install adb on your PC (that's why I asked about working on the command line). Then I'll send you a debug apk (the default app is a release build and unless your phone is rooted, it refuses to allow access to the file system) that you need to install via adb.
And then I'll send you some magic commands to use with adb in order to figure out what has actually been saved on your device and if we can find the data that way.
That one is quite painful, to be honest :)

(and in eight years you'd be the third person with whom I ended up having to go down that path)

/D

[Dirk Hohndel]

And another quick update here for anyone reading along.

We figured out that it was a bug in our code that offers the local cache repos for import.
Basically, if the local repo and the server go completely out of sync, we move the local repo away and sync with the server. Which looks to the user like "I lost all of these dives". But then there's an easy way to simply re-import that moved-away repo on the mobile device and push the result back to cloud storage. Except there was a bug that made us not show any of the cache repos (as seen in the screen shot posted earlier).
I hacked around the bug and Shenji has all dives back.

/D