Download form a Deepblu COSMIQ+ computer?

[Larry Michaels]

My brother just got a Deepblu COSMIQ+ dive computer and I'm trying to find a dive log he can use on his PC which is windows 10.  It uses Bluetooth do phone app but we want a dive log on his PC. 

Any ideas?

[MikeM24108]

On Thursday, January 11, 2018 at 7:24:52 PM UTC-5, Larry Michaels wrote:

My brother just got a Deepblu COSMIQ+ dive computer and I'm trying to find a dive log he can use on his PC which is windows 10.  It uses Bluetooth do phone app but we want a dive log on his PC. 

Any ideas?

Is there any planned support for DeepBlu COSMIQ dive longs with this software?  I'd love to use this but it is not supported on my new DeepBlu dive computer 

[Robert C. Helling]

My understanding is that DeepBlu so far has not shared the information how to talk to their dive computers directly and nobody has figured it out via reverse engineering, yet. But what you can do is to use their smartphone software to transfer the data from the Cosmiq to their cloud website and then use the tools by Sander available on https://github.com/bluppfisk/deepblu-tools or via the online service on http://worldofnonging.com/deepblu-tools/index.php to convert that information to uddf which then can be read by Subsurface.

BTW, the sheer existence of that web service that does not require a password is an indication that you should not enter too much sensitive information on the deepblu web site...

[Michael Werle]

BTW, the sheer existence of that web service that does not require a password is an indication that you should not enter too much sensitive information on the deepblu web site...

Well, it only accesses the information which you made public if you don't enter a password. But in general you should NEVER enter any information you don't want in the public domain into an online service over which you don't have full control or at the very least an SLA with. Only because an online service is password-protected does not imply it is in any way secure.

I only found out about the "deepblu-tools" today and the downloader works great - although I didn't use the web service provided.  I copied the source locally then ran that. The downloaded data integrated nicely into Subsurface which lets me compare the Cosmiq+ with my Suunto Mosquito.

[Michael Werle]

In the meantime I have (mostly) reverse-engineered the Bluetooth protocol between the Cosmiq+ and my phone.

I've also managed to connect to it using my PC, but not had time to try and get it to talk via the protocol.   I've also had a really quick look at the libdivecomputer source code, but right now that's more effort than I can afford to put in to work out how to add support.

Is there someone who can pick up from my work? Ideally someone who has some experience with Subsurface/libdivecomputer source code? If so I'll see about writing up my findings so far and sharing them somewhere.

[Linus Torvalds]

On Fri, Aug 9, 2019, 03:09 Michael Werle <mwe...@gmail.com> wrote:

In the meantime I have (mostly) reverse-engineered the Bluetooth protocol between the Cosmiq+ and my phone.

I've also managed to connect to it using my PC, but not had time to try and get it to talk via the protocol.   I've also had a really quick look at the libdivecomputer source code, but right now that's more effort than I can afford to put in to work out how to add support.

So with BLE, you need bit just reverse-engineer the protocol, you also need to reverse-engineer the rules for figuring out the right BLE GATT characteristic to use, and sometimes even for control.

It sounds like you've done this all from the Bluetooth packet dumps, in which case you've already presumably figured out the basic rules. 

But those rules end to going not in libdivecomputer (which doesn't really know about BLE connection details) but into the subsurface BLE connection code itself.

It turns out that most BLE serial encapsulation is very similar - not standardized per se, but there are just "this is how people have done it" and almost everybody uses mostly the same approach. Sometimes they do that because they use the same underlying BLE chipset.

Is there someone who can pick up from my work? Ideally someone who has some experience with Subsurface/libdivecomputer source code? If so I'll see about writing up my findings so far and sharing them somewhere.

If you have packet dumps and a good write-up, I can try to take a look. Without hardware, I can almost certainly not get it working all the way (unless it's really obvious in all the details, which basically never happens), so you'd probably need to then help testing several times. 

So no promises, but I've done enough of the BLE serial things that the low-level connection part is probably not a big deal (and we've set it up so that the most common connection models JustWork(tm), but it's about fifty-fifty).

The libdivecomputer side is the part where depending on how well you've reverse engineered it, and how complex it is, things might be more iffy.  I've never even seen one of the COSMIQ+ dive computers, and I suspect my local dive shop doesn't have them either (I've been able to get loaners from them for testing).

Long story short: possible if no big road blocks appear (and if I don't end up having kernel issues come up - they always take precedence)

      Linus

[Michael Werle]

On Saturday, 10 August 2019 03:18:51 UTC+9, Linus Torvalds wrote:

So with BLE, you need bit just reverse-engineer the protocol, you also need to reverse-engineer the rules for figuring out the right BLE GATT characteristic to use, and sometimes even for control.

It sounds like you've done this all from the Bluetooth packet dumps, in which case you've already presumably figured out the basic rules. 

I'm afraid I've never really played with Bluetooth before, let alone BLE, so not entirely sure how to drive the GATT characteristics. It seems the phone and computer set up a Nordic UART connection over which everything else then runs.  

Haven't figured out the Bluetooth side of things, but did make some inroads into the actual data protocol.

If you have packet dumps and a good write-up, I can try to take a look. Without hardware, I can almost certainly not get it working all the way (unless it's really obvious in all the details, which basically never happens), so you'd probably need to then help testing several times. 

Bluetooth captures (can be loaded into wireshark) as well as typed up some analysis on the actual dive computer protocol (readme.txt for list of files and what's in each): 

https://drive.google.com/open?id=1iyNLJY3bhjbdftDta5XT2EtmEbg6DgOB

I can easily create new Bluetooth captures, should also get some more dives on it in the next couple of weeks.

If the writeup isn't sufficient, please point at an example of a "good writeup".

Long story short: possible if no big road blocks appear (and if I don't end up having kernel issues come up - they always take precedence)

Awesome, and, of course! :)

 

- Micha.

 

[Jef Driesen]

I can help you with that.

A friend of mine also has a Cosmiq, and I can probably borrow that for testing.
That will certainly help too.

Jef

[Linus Torvalds]

On Fri, Aug 9, 2019 at 11:18 AM Linus Torvalds
<torv...@linux-foundation.org> wrote:
>
> So with BLE, you need bit just reverse-engineer the protocol, you also
> need to reverse-engineer the rules for figuring out the right BLE GATT
> characteristic to use, and sometimes even for control.

I decided to just get one of those dive computers so that I can
verify, and it turns out that the BLE side is not a problem at all.
It's a Nordic Semiconductor BLE serial chip, and the logic we have for
it in subsurface works as-is.

To make Subsurface recognize it automatically as a dive computer (so
that you don't have to do the "show all BLE devices" and pick it out
from possibly hundreds of other random BLE sensor things), a trivial
patch like this

diff --git a/core/btdiscovery.cpp b/core/btdiscovery.cpp
index 2a3eff580..175ecf932 100644
--- a/core/btdiscovery.cpp
+++ b/core/btdiscovery.cpp
@@ -98,6 +98,11 @@ static dc_descriptor_t *getDeviceType(QString btName)
product = "iX3M GPS Easy"; // we don't know which of
the GPS models, so set one
}

+ if (btName == "COSMIQ") {
+ vendor = "Deepblu";
+ product = "Cosmiq+";
+ }
+
if (!vendor.isEmpty() && !product.isEmpty())
return descriptorLookup.value(vendor + product);

will make things more convenient, but that's literally the only change
we need to subsurface itself.

libdivecomputer is a different issue. That's where all the query/reply
protocol is and where we'll need to parse the end result, but I'll
hopefully look at that this weekend and see if I can get something at
least limping along.

No promises, but at least one part of the equation was pain free.

Linus

[Linus Torvalds]

On Fri, Aug 9, 2019 at 7:14 PM Michael Werle <mwe...@gmail.com> wrote:
>
> Bluetooth captures (can be loaded into wireshark) as well as typed up some analysis on the actual dive computer protocol (readme.txt for list of files and what's in each):
>
> https://drive.google.com/open?id=1iyNLJY3bhjbdftDta5XT2EtmEbg6DgOB
>
> I can easily create new Bluetooth captures, should also get some more dives on it in the next couple of weeks.
>
> If the writeup isn't sufficient, please point at an example of a "good writeup".

Looks useful already.

The first packet actually looks like a setup packet. Your writeup says

#202e0c190730223004
# - request / command
20 - packet type : ?
2e - ?
0c - packet payload length in characters (12)
190730223004 - packet payload data : ?

19 - matches number of dives below; coincidence?

but I think the 19 is actually "19" as the _year_ 2019: that looks
like the date and time of the connection: 19/07/30 at 22:30:04.

Do you think that 10:30pm sounds like a likely time for you to have
done that dump? That's *kind* of what the dump data itself says, I see
a packet timestamp that claims

Jul 30, 2019 15:30:05.134311000 PDT

(so off by a second, and showing your 10:30pm PDT with what looks like
a -0700 timezone correction - I think that may be a wireshark problem,
where it applies the timezone correction to something that was already
a local time).

Your first dump ("hci_log_A") has a similar pattern: Wireshark says

Jul 16, 2019 02:02:47.853062000 PDT

but I doubt you did it at 2:02AM, and it's likely 9:02AM local time.
Which matches the first setup packet in the dump:

#204C0C190716090247

which would be 19/07/16 at 09:02:47.

So the "20" seems to be the command, and I agree that the "0c" might
be the length of the remaining data.

I'm not sure what that middle byte there is ("4C" in the first dump,
"2E" in the second one). It seems largely random. It does look like
the pattern is "command/response byte", "random byte", "data length
byte" all encoded as HEX numbers in the ASCII stream of data.

I wonder if the "random byte" is some kind of parity or CRC byte.

It's a very odd format. It's all ASCII, with some of it seemingly
being real ASCII data (like that date and time), and other parts being
hex-encoded bytes.

With that #/$ for command/response marker, and newlines at the end.

And almost always, the BLE packetization matches the ASCII line
exactly (so one line per packet). But I see one case of the line being
split, so that the first packet only contains the "$", and the next
packet contains the ASCII line of data (and newline). I suspect that's
because the Nordic Semiconductor BLE serial logic simply has a small
buffer and a timeout, and what happened once was that the writing side
of the dive computer started with the "$" byte and then had a small
delay before the rest of the line made it to the BLE serial converter
logic, and so that sent it out as two packets.

So it _looks_ packetized, but you actually have to treat it as an
ASCII stream with newlines.

Some of the HEX encoding is also confused about the case of the hex
characters too (ie "a-f" vs "A-F").

Anyway, I think I can make sense of most of this, and I can parse the
insanity. It's a quite odd format, but the actual amount of data there
seems to be pretty limited and simple.

Famous last words.

Linus

[Linus Torvalds]

On Fri, Aug 16, 2019 at 11:20 AM Linus Torvalds
<torv...@linux-foundation.org> wrote:
>
> I'm not sure what that middle byte there is ("4C" in the first dump,
> "2E" in the second one). It seems largely random. It does look like
> the pattern is "command/response byte", "random byte", "data length
> byte" all encoded as HEX numbers in the ASCII stream of data.
>
> I wonder if the "random byte" is some kind of parity or CRC byte.

Yup.

So to take a random packet:

#41BB0202

we have:

# - request
41 - request code
BB - two's complement modular sum, so 0x41+0xbb+0x02+0x02 = 0x00 (modulo 256)
02 - bytes of data
02 - data itself

and that literally seems to be the command for "give me metadata for dive #2".

The replies have the same format, just with '$' as the first
character, but following the same "reply code/checksum/length" ASCII
HEX encoding, followed by ASCII data and a newline. But the "ASCII
data" is that odd mix of "sometimes actual ASCII, but mostly
hex-encoded data".

Although the only _real_ ASCII data I've seen is that date. Which I
*guess* you could call hex-encoded too, if you treat the date numbers
as BCD.

Strange, but seems fairly straightforward. I'm sure I'll have
something by the end of the week.

Famous last words.

Linus

[Linus Torvalds]

On Fri, Aug 16, 2019 at 11:20 AM Linus Torvalds
<torv...@linux-foundation.org> wrote:
>

> On Fri, Aug 9, 2019 at 7:14 PM Michael Werle <mwe...@gmail.com> wrote:
> >
> > If the writeup isn't sufficient, please point at an example of a "good writeup".
>
> Looks useful already.
>
> The first packet actually looks like a setup packet. Your writeup says
>
> #202e0c190730223004
> # - request / command
> 20 - packet type : ?
> 2e - ?
> 0c - packet payload length in characters (12)
> 190730223004 - packet payload data : ?
>
> 19 - matches number of dives below; coincidence?
>
> but I think the 19 is actually "19" as the _year_ 2019: that looks
> like the date and time of the connection: 19/07/30 at 22:30:04.

Confirmed. I can set the time and date by just changing those numbers.

So the format is:
- '#' cmd byte marker
- 20 cmd number
- 2e checksum byte
- 0c data size
- 190730223004 - date and time: 19/07/30 at 22:30:04

Looking around some more, I think I can fill in some of the other
question marks you have in your writeup.

For example, cmd 41 (dive metadata), gets the data size as a reply,
and then gets further replies of type 42 with the actual data.

That actual data you already mostly seem to have decoded:

- two bytes dive number
- one byte dive type, I think (ie scuba/freedive/etc)
- one byte O2 percentage
- two bytes LE unknown
- two bytes LE year
- one byte day/month/minute/hour each
- two bytes LE dive time
- two bytes unknown
- two bytes LE guess: surface pressure (millibar)
- four bytes unknown
- two bytes max pressure (millibar). Is this ambient?
- two bytes water temp in tenth of a Celsius

You call millibar "cm h2o", which is pretty much the same thing, but I
would expect millibar to be the more natural one. Also, the dive
pressures at the end very much look like surface pressure in millibar
(ie looks like it hovers around "1013").

You said "centi-centigrade", but then make clear that you *meant*
tenth of C, ie deci-centigrade.

And you didn't guess the surface pressure, but I'm pretty sure that's
what it is because you need a surface pressure to turn the ambient
pressure in the samples into a real depth.

I guess it could take the surface pressure from the end of the dive
too, but since you need it _during_ the dive to get the depth, I'd be
surprised if it's not in that dive header thing.

I think most of the data is decoded. I don't have any dives on my own
Cosmiq+, so I'm just looking at your dumps so far, but I have a
pressure chamber and can do fake dives that way.

But I'm done for the day. I have verified that I can talk to mine with
the time/date command, so the packet format is good, and I think you
have all the basics pretty much figured out for the core packet
commands.

Linus

[Linus Torvalds]

On Fri, Aug 16, 2019 at 2:43 PM Linus Torvalds
<torv...@linux-foundation.org> wrote:
>
> - two bytes LE guess: surface pressure (millibar)

No, this was not the case, I'm not finding surface pressure
information anywhere.

If I use a fixed surface pressure of 1013 millibar, and a specific
water weight of 1.024kg/l, then when I convert from the pressures in
millibar to meters, I seem to get depth values that match the ones
that the dive computer was showing in my little testing pressure
chamber.

So I've successfully downloaded the basic information from my two
dives with the libdivecomputer code that I've pushed out to the normal
subsurface libdivecomputer repo.

As mentioned, subsurface itself works "out of the box", except that
adding the matching for the Cosmiq bluetooth name makes it a lot more
usable. I'll make a PR for that too.

Micha - are you able to build your own libdivecomputer and subsurface
binaries? Dirk is off on vacation with his family, so the daily builds
might not happen.

Jef - feel free to take a look at my Deepblu Cosmiq changes in the
usual subsurface libdivecomputer repo. It's a bit rough, but it does
seem to give the basic profile. You had a friend that you could get a
loaner for testing?

Dirk - I'm doing the PR just for the BLE name recognition, I have
*not* done the libdivecomputer update itself. I'll let my pushed-out
version simmer a bit first, to hopefully avoid the Travis issues.

I don't think this is hugely important. But it does kind of seem to work.

Linus

Linus

[Linus Torvalds]

On Sat, Aug 17, 2019 at 3:21 PM Linus Torvalds
<torv...@linux-foundation.org> wrote:
>
> So I've successfully downloaded the basic information from my two
> dives with the libdivecomputer code that I've pushed out to the normal
> subsurface libdivecomputer repo.

I fixed up parsing a bit more - it now gets freedives right too (which
have different sample intervals) and filled in the gasmix information
too.

There are some other fields in the dive header that I have no idea
what they are, but on the whole the Cosmiq+ download seems to work
fairly well. As well as it can - considering that apparently there
really aren't any events or anything else in the log - only
temperature and pressure.

It is fairly noisy with fake "errors" etc, and I should add progress
reporting etc during the download, but the basic functionality seems
to be there.

I also don't know if my pressure -> depth conversion is really
matching exactly what Deepblu does, but it's close.

Linus

[Michael Werle]

Wow!  Thanks so much for all the effort and decoding some of the additional fields!  I was expecting a checksum in one of the "random bytes" somewhere, but hadn't spent a lot of time yet trying to figure out which algorithm, the couple I checked didn't match.

Sorry for having gone quiet - was just away for a few days diving and climbing MtFuji (ouch!).  Still have a friend visiting until Sunday but then I can take a look.  And yes, I can build libdivecomputer and subsurface.  I've also got new dives on the computer to test the download.

[Jef Driesen]

On 2019-08-18 00:21, Linus Torvalds wrote:
> Jef - feel free to take a look at my Deepblu Cosmiq changes in the
> usual subsurface libdivecomputer repo. It's a bit rough, but it does
> seem to give the basic profile. You had a friend that you could get a
> loaner for testing?

Yes, I'll ask my friend to borrow his Cosmiq and give it a try. That
will probably be something for next week.

Jef

[Michael Werle]

Ok, so I got everything compiled and it "just worked" - bluetooth connection and download of dive data.  Only issue is it said "Invalid O2 percentage - 21000" or similar (forgot to take screenshot, sorry).  Briefly checked the libdivecomputer code and it "looks" ok so not sure why it ended up giving a warning in Subsurface.
When I reload the Subsurface file containing the downloaded dives everything looks correct, showing "air" as the gas used.

Somewhat annoyingly I no longer have the computer so in the short term I can't do more testing. I'll try to borrow one from another member in our club to keep working on this though.

[Linus Torvalds]

On Tue, Aug 27, 2019 at 4:53 AM Michael Werle <mwe...@gmail.com> wrote:
>
> Ok, so I got everything compiled and it "just worked" - bluetooth
> connection and download of dive data.

Good.

> Only issue is it said "Invalid O2 percentage - 21000" or similar

That's because I have the mental capacity of a slightly retarded chipmunk.

In subsurface, we try to avoid using floating point for various (very
valid) reasons: odd rounding, data structure size etc. And we have the
units named as part of the data structures, so when you set the oxygen
percentage, you see code like

dive->cylinder[i].gasmix.o2.permille = o2;

which is a bit verbose, but there's no question about what the value
is. It's permille (tenths of a percent), and you can't by mistake just
say "o2 = 21" or similar.

But libdivecomputer doesn't have that model, and some things are
integers and some things are floating point, and I didn't check which
was which.

In libdivecomputer, the gas percentages are floating point, but I just
put the unmodified percentage in there. So I basically reported 0.21
as "21.0". And then subsurface wasn't happy, turning it into "21000
permille".

> When I reload the Subsurface file containing the downloaded dives everything looks correct, showing "air" as the gas used.

That just happens to work for air. We see the crazy number, say
"that's clearly BS", and ignore it. And the default value for gas is
air, so it happens to work for the 21% case.

But had you been diving Nitrox, we'd also have ignored a value like
32000 permille, and said that your 32% nitrox mix was air.

> Somewhat annoyingly I no longer have the computer so in the short term
> I can't do more testing. I'll try to borrow one from another member in
> our club to keep working on this though.

No need for testing, the bug (and the fix) is obvious:

- deepblu->cache.gasmix[0].oxygen = data[3];
+ deepblu->cache.gasmix[0].oxygen = data[3] / 100.0;

I'll fix things up. Embarrassing - that error must have showed for my
downloads too, and I just didn't look or care.

There are a few other (known) weaknesses in the current Cosmiq
downloader - it doesn't stop downloading when we see a dive we've
already seen and subsurface says "hey, I've already got that, stop
already".

And it also doesn't support the "dive fingerprint" feature to avoid
downloading things that we told it earlier we knew about, so it really
always downloads things over and over.

Subsurface will then ignore the duplicate dives (unless you explicitly
ask for "download all dives", of course), so it's not _that_
noticeable outside of wasted time, but BLE can be pretty slow.

It's probably not a huge deal since the Cosmiq can only hold a fairly
limited number of dives, but I just didn't get around to finishing up
those kinds of details.

I'll fix the oxygen percentage thing, and maybe I'll be shamed into
testing it too and even fix the other nagging issues.

Thanks for the report,

Linus

[Linus Torvalds]

On Tue, Aug 27, 2019 at 11:07 AM Linus Torvalds
<torv...@linux-foundation.org> wrote:
>
> No need for testing, the bug (and the fix) is obvious:
>
> - deepblu->cache.gasmix[0].oxygen = data[3];
> + deepblu->cache.gasmix[0].oxygen = data[3] / 100.0;

I've pushed this out to the libdivecomputer repo (but not updated the
subsurface repo).

And yes, I even tested it with a simulated nitrox dive this time. So
now I can report that nitrox definitely works. At least for that one
test dive.

I'll look at the "don't download everything every time" issue next.

Linus