Problems importing from Garmin Descent Mk1

[Andrew Trevor-Jones]

Subsurface 4.9.6 on Mac Mini 10.12.6

For a couple of years I have been importing my dives from my Mk1 using a mount point on the drive of my main computer.  I normally download the FIT file from Garmin Connect (as a ZIP file and unzip it).  These files load just fine and I've not had any problems.

Last weekend, Garmin were off the air and so I had to get the files directly from the Mk1 via USB.  I copied to FIT files to the exact same folder.  Subsurface doesn't see them.  The only difference is the format of the filename.

For example, the file I copied from the Mk1 for the dive last Thursday is 2020-07-23-07-53-35.fit.

The file I downloaded from Garmin Connect is 5278291798.fit

The contents of the two files are identical but I can only import the second one.

I remember when support for the Mk1 was first added it Subsurface it only supported the filename format from the Mk1 itself and not the format of the download.  The code was updated to handle any filename. Has this changed?

[Miika Turkia]

I have used the download from divecomputer feature and given Subsurface the mountpoint where the Mk1 is mounted. In my case that is /media/mturkia/GARMIN. The actual dive files are then under Garmin/Activity/<date-time>.fit . This should work similarly on Mac, but obviously the mount point is different.

I have not been diving in a few months so I don't have any dives on the Mk1 to test this out. Anyway, a backup from last year shows that the file naming format on the device has been 2020-07-23-07-53-35.fit even then. So this should still work.

[Andrew Trevor-Jones]

I agree it SHOULD work.  I'm pointing out that it does not.

Even if I tell Subsurface to "Force download of all dives", it only shows the dives for files with the nnnnnnnnnn.fit filename format.  The YYYY-MM-DD-hh-mm-ss.fit filename format are ignored.

The dive from 23/07/20 shows because of 5278291798.fit

[Andrew Trevor-Jones]

I dived again today.  It looks like Garmin Connect is still not working properly and so I couldn't download my dive from there.  I copied it from the Mk1.  

With the file as 2020-08-01-07-28-20.fit Subsurface does not see it.  If I rename it to 5555555555.fit Subsurface does see it.

[Anton Lundin]

Try to generate a libdivecomputer debug log, and that will probably say "name too long" or something. Anyway it will clue us in to what's happening.

My guess is that mac os readdir returns the string in some other format which we tink is to long for a dive file.

//Anton

[Linus Torvalds]

On Fri, Jul 31, 2020 at 10:42 PM Andrew Trevor-Jones
<andrew.tr...@gmail.com> wrote:
>
> I agree it SHOULD work. I'm pointing out that it does not.

I think I found the bug. I'm surprised it hasn't hit us before. I
think it happened to work depending on almost random contents on the
stack.

It's related to some of the code from early in development when we
_only_ accepted the exact name length that Garmin provided, I'll need
to clean that up too, but it looks straightforward.

I'm assuming you're not building your own version of subsurface? I'll
have a fix for libdivecomputer shortly, but then you'll have to wait
for Dirk to build an OS X binary..

Linus

[Linus Torvalds]

On Sat, Aug 1, 2020 at 9:30 AM Linus Torvalds
<torv...@linux-foundation.org> wrote:
>
> I think I found the bug. I'm surprised it hasn't hit us before.

Oh, never mind. I don't understand the bug after all. The code is a
bit fragile, but should have worked fine unless OS X is doing
something _really_ odd with readdir.

Admittedly, OSX has done that before, so maybe...

Linus

[Linus Torvalds]

I'll do the filename handling cleanup regardless, but I'll echo Anton
Lundin: please check the "Save libdivecomputer logfile" thing, and
send the resulting log file to me or the list.

It *should* have debug messages for all the files it finds, something like

[0.000000] INFO: Open: name=/your/mount/point/goes/here
[0.000033] INFO: Read: size=xy, data=<hexnoise representing the mountpoint>
[0.000044] DEBUG: Iterating over Garmin files
[0.000060] DEBUG: .
[0.000065] DEBUG: . - name lacks FIT suffix
[0.000066] DEBUG: ..
[0.000067] DEBUG: .. - name lacks FIT suffix
[0.000068] DEBUG: somerandomname.tar
[0.000069] DEBUG: somerandomname.tar - name lacks FIT suffix
[0.000069] DEBUG: someotherlongrandomname.fit
[0.000070] DEBUG: someotherlongrandomname.fit - name too long

or similar.

And, as usual, it all does want that "Garmin/Activities" pathname, but
you must be having that part right already since it works when you
rename the file.

Linus

[Dirk Hohndel]

So I played with this and my Garmin Descent. No new dives on there (hopefully in a day or two), but I noticed an interesting change in the behavior of the Garmin Express app on Mac.

The current version now grabs the Garmin and prevents it from mounting as a file system. I had to quit (and tell it not to go into the background) before Finder would see /Volumes/Garmin

Once I did that, Subsurface happily downloaded all 85 dives on the Descent

/D

On Jul 31, 2020, at 10:57 PM, Andrew Trevor-Jones <andrew.tr...@gmail.com> wrote:

I dived again today.  It looks like Garmin Connect is still not working properly and so I couldn't download my dive from there.  I copied it from the Mk1.  

With the file as 2020-08-01-07-28-20.fit Subsurface does not see it.  If I rename it to 5555555555.fit Subsurface does see it.

--
You received this message because you are subscribed to the Google Groups "Subsurface Divelog" group.
To unsubscribe from this group and stop receiving emails from it, send an email to subsurface-dive...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/subsurface-divelog/6b405542-cb3d-40e0-b4e2-73a2e55c7dfbo%40googlegroups.com.

[Dirk Hohndel]

Oh, and I should have mentioned that the files of course have the regular format:

-rwxrwxrwx  1 hohndel  staff  156944 Oct 26  2019 2019-10-26-10-14-42.fit

-rwxrwxrwx  1 hohndel  staff  111181 Oct 26  2019 2019-10-26-11-33-51.fit

-rwxrwxrwx  1 hohndel  staff  116168 Oct 26  2019 2019-10-26-12-37-26.fit

So among the questions to ask... what do you see if in Terminal.app you run

ls -lrt /Volumes/GARMIN/Garmin/Activity | tail -10

/D

To view this discussion on the web visit https://groups.google.com/d/msgid/subsurface-divelog/9EC405B4-FE98-45E3-9276-C57545D7E3DE%40hohndel.org.

[Andrew Trevor-Jones]

This is what it says in the log, which is all good:

[476.554940] DEBUG:   2020-07-23-07-53-35.fit

[476.554941] DEBUG:   2020-07-23-07-53-35.fit - adding to list

[476.554941] DEBUG:   2020-07-24-08-36-34.fit

[476.554942] DEBUG:   2020-07-24-08-36-34.fit - adding to list

[476.554942] DEBUG:   2020-07-25-09-23-12.fit

[476.554943] DEBUG:   2020-07-25-09-23-12.fit - adding to list

[476.554943] DEBUG:   2020-08-01-07-28-20.fit

[476.554944] DEBUG:   2020-08-01-07-28-20.fit - adding to list

However, this is what gets displayed on the screen:

The only reason 23/07/2020 shows up is because 5278291798.fit is present. 

Note that I am not loading directly from the Mk1 but an appropriately named folder.

[Linus Torvalds]

On Sat, Aug 1, 2020 at 8:44 PM Andrew Trevor-Jones
<andrew.tr...@gmail.com> wrote:
>
> This is what it says in the log, which is all good:
>
> [476.554940] DEBUG: 2020-07-23-07-53-35.fit
> [476.554941] DEBUG: 2020-07-23-07-53-35.fit - adding to list
> [476.554941] DEBUG: 2020-07-24-08-36-34.fit
> [476.554942] DEBUG: 2020-07-24-08-36-34.fit - adding to list
> [476.554942] DEBUG: 2020-07-25-09-23-12.fit
> [476.554943] DEBUG: 2020-07-25-09-23-12.fit - adding to list
> [476.554943] DEBUG: 2020-08-01-07-28-20.fit
> [476.554944] DEBUG: 2020-08-01-07-28-20.fit - adding to list

Ok, so it actually sees all files. So it's not readdir().

> However, this is what gets displayed on the screen:

.. but decides that it has seen three of them already.

Ok, I have a suspicion.

Check the "Force download all dives" checkbutton before downloading.

Do you see all dives then?

So what might be going on is that subsurface remembers "I've seen this
dive already" (that's done by the "fingerprint" of the dive, which is
just the filename). So then it ignores that dive, and any dives after
it..

Renaming the file to something that looks newer (and "55555.fit" does,
because the sorting is strictly alphabetical, and 5555.. is larger
than 2020...) will then fool that logic.

Linus

[Andrew Trevor-Jones]

On Sunday, 2 August 2020 14:01:27 UTC+10, Linus Torvalds wrote:

Check the "Force download all dives" checkbutton before downloading.

Do you see all dives then?

No.  The "missing" dives are still missing:

What is weird is that if I attempt to get the dives directly from the Mk1, not only do I get the dives but it lists a bunch of other dives that I already have:

 Perhaps the issue is sort of "fingerprint" related.

 

[Linus Torvalds]

On Sat, Aug 1, 2020 at 10:28 PM Andrew Trevor-Jones
<andrew.tr...@gmail.com> wrote:
>
> On Sunday, 2 August 2020 14:01:27 UTC+10, Linus Torvalds wrote:
>>
>> Check the "Force download all dives" checkbutton before downloading.
>>
>> Do you see all dives then?
>
> No. The "missing" dives are still missing:

Ugh,

Ok. Can you send me the *full* log file from a download (with the
"Force download all files"), and also a full directory listing of the
"Garmin/Activities" folder you use.

You might want to do it as an email directly to me because the log
might be a bit big for the list.

None of this makes sense, since the files seem to be found according
to your earlier log snippet.

Linus

[Andrew Trevor-Jones]

On Sunday, 2 August 2020 15:33:31 UTC+10, Linus Torvalds wrote:

You might want to do it as an email directly to me because the log
might be a bit big for the list.

log fils is 3.5GB! Where do I find your email address? 

[Linus Torvalds]

On Sat, Aug 1, 2020 at 10:42 PM Andrew Trevor-Jones
<andrew.tr...@gmail.com> wrote:
>
> log fils is 3.5GB! Where do I find your email address?

That's a bit excessive. Just how many dives do you have? But I guess
the real verbose part is the parsing side, which isn't even very
interesting.

Actually, don't do the "Force download" thing, that should cut down
the parsing a lot, and it should still contain the important parts.

And if you have gzip or something installed, compress before sending
it to <torv...@linux-foundation.org>.

Linus

[Linus Torvalds]

On Sat, Aug 1, 2020 at 11:50 PM Linus Torvalds
<torv...@linux-foundation.org> wrote:
>
> And if you have gzip or something installed, compress before sending

> it [..]

Ok, thanks, got it.

Yeah, so subsurface finds 339 *.fit files in that directory of yours.

And then sorts them alphabetically (reversed), and starts parsing files:

5278291798.fit
5252935268.fit

and on the second file, it decides to stop.

And we haven't even gotten close to the files you *want* it to parse,
because of the reverse sorting.

When you actually just download from the Garmin itself, all the
tilenames will be date-and-time, like this:

2020-08-02-07-40-35.fit
2020-08-01-07-28-20.fit
2020-07-25-09-23-12.fit
2020-07-24-08-36-34.fit
2020-07-23-07-53-35.fit
2020-02-10-07-20-30.fit
2019-02-17-15-11-24.fit

and then that "sort alphabetically in reverse" works beautifully.

But the other filenames are some odd timestamp format. It's not the
normal "seconds since Jan 1, 1970" epoch thing.

And yeah, the parser is _very_ verbose. I guess I should turn that
down a lot, it goes back to me just learning to parse the Garmin FIT
file format, and I never removed the debug stuff.

Sadly, the one thing it _doesn't_ say is why it stopped parsing it.
But it does smell like the "you already had that one dive, so now
we'll stop".

Except then you should have gotten all 339 fit files as dives (except
for the ones that aren't dives at all, I'm assuming many of them are
other activities).

Ugh. I'll have to stare at this some more.

Linus

[Andrew Trevor-Jones]

On Sunday, 2 August 2020 17:14:47 UTC+10, Linus Torvalds wrote:

But the other filenames are some odd timestamp format. It's not the
normal "seconds since Jan 1, 1970" epoch thing.

Not a timestamp.  It's a unique number for each activity across all Garmin users worldwide.  10 digits allows for 10,000,000,000 different numbers.  I guess it just increments by one for each new activity that is loaded to Garmin Connect. 

[Andrew Trevor-Jones]

If I rename the files copied from the Mk1 from 2020-..... to 6020-..... they get picked up.

This suggests that it is related to the way they are sorted and probably "you already had that one dive, so now we'll stop". But why doesn't it pick them up when I "Force download all files"?

[Linus Torvalds]

Yes. That confuses me. A lot.

The only reason we stop downloading (that I can see) is

(a) user cancel

(b) actual read errors from the files

(c) the fingerprint code (that should be disabled by "Force download
all dives").

I wonder if (b) might trigger. It might not be an actual IO error -
things like permission errors would also make us abort.

I wonder if you could pinpoint (by renaming the files) where in the
list they stop working for you..

Linus

[Andrew Trevor-Jones]

On Tuesday, 4 August 2020 10:35:52 UTC+10, Linus Torvalds wrote:

I wonder if you could pinpoint (by renaming the files) where in the
list they stop working for you..

If I rename the ones direct from the Mk1 (2020...) so they sort above the others (in a reverse sort) they show up.

If I rename the ones from Garmin Connect (10 digit number) so they sort below the others (or are at least out of order) they don't show up even if "Force download all files".

It appears it is sorting them in reverse alphanumeric order and then reading them in that order.  If a file in the list is out of order it gets ignored. 

[Andrew Trevor-Jones]

On Tuesday, 4 August 2020 11:26:16 UTC+10, Andrew Trevor-Jones wrote:

It appears it is sorting them in reverse alphanumeric order and then reading them in that order.  If a file in the list is out of order it gets ignored. 

Ah... it doesn't ignore them, it loads them in the order they are found not in the order the dives were done:

The dives are there but just displayed out of order.

 

[Linus Torvalds]

Ahh. A light appears.

On Mon, Aug 3, 2020 at 7:31 PM Andrew Trevor-Jones
<andrew.tr...@gmail.com> wrote:
>
> On Tuesday, 4 August 2020 11:26:16 UTC+10, Andrew Trevor-Jones wrote:
>>
>> It appears it is sorting them in reverse alphanumeric order and then reading them in that order. If a file in the list is out of order it gets ignored.
>
> Ah... it doesn't ignore them, it loads them in the order they are found not in the order the dives were done:

Yes, I thought that was clear.

What happens is that we always sort the dives by that reverse
alphabetical order, which gets us the dives in the right order on the
actual Gramin mount.

libdivecomputer doesn't actually see the real dates until it parses
the files, but by then they have already been presented to subsurface
in that original order (that's how the libdivecomputer interfaces
work: parsing is a separate phase from downloading).

> The dives are there but just displayed out of order.

Ok, that explains the confusion, and explains the situation.

Yeah, I don't know how to sort the oddly formatted stuff downloaded
from Garmin in-order.

In general, I'd suggest that when copying from the Garmin Descent to
the local filesuystem, you _don't_ mix the ones you've downloaded from
garmin.com up with the ones that are gotten directly from the dive
computer, because clearly the namespaces are completrey different.

Linus