Average depth from CSV import and other problems

[darryl...@gmail.com]

I'm running Subsurface 4.5.6 on OS X 10.8.5. My CSV files containing log info are being produced (in tab separated format) with OpenOffice.org 3.4.1 and imported in imperial mode.

I'm trying to transfer old pre-computer dives from my paper logbook and am having a number of problems. I'm using CSV importing. A few things do not seem to be importing correctly, despite the "blue bubbles" correctly matching all of my headings via autodetection.

- The most serious problem is average depth. Max depth is read correctly, and a reasonable guess at a dive profile is created as a result. However, the avg. depth field in the Info tab reads 0. If I now attempt to edit the dive via Log > Edit dive and save my changes, the entire dive profile is wiped out. That is, the graph becomes a horizontal line a zero depth and my max depth also changes to zero for the dive. Oddly, this does not happen if I edit the dive by simply clicking on a field in the log to enter editing mode. The yearly statistics are also affected by the average problem.

- I cannot seem to import my tank size information correctly. I used AL80's on these dives. Putting "80" or "80cuft" into the CSV file under cylinder size imports as 2.83cuft. Putting "AL80" doesn't import at all.

- The O2% field for a cylinder does not get populated. While the type of gas used based on this information is correctly recognized in the "Gases used" field under Info tab, the field under the Equipment tab stays blank. This persists even after manually editing the O2% field. The gas type under Info changes properly in response to this manual change after saving, but the O2% field again becomes blank. (Annoyingly, when I tested this by entering an O2% of 29—to which the the Gases used field changed from air to EAN29—I couldn't change it *back* to 21% nor could I delete the cylinder and had to delete and re-import the entire dive). 

- I can't edit my dives in dive planner mode. Nothing happens after selecting a dive and clicking Log > Edit dive in planner. This doesn't particularly matter for these old dives but could be problematic for more recent ones.

- On the whole, the program is behaving very unstably for me. Since installing it last night and importing four dives, it's crashed at least half a dozen times.

[Miika Turkia]

Thank you for your detailed report. It should help us to tackle the issues. Could you send me a sample log to test the import. No need for full log, nor to post it publicly. It would be great to be able to tackle these issues before our next release... (As I know what the import expects, I usually fail to generate sample files that do not work :D)

The equipment tab does not show the mix if it is air. However, other mixes should be shown. The value change might fail if you do not hit enter key when changing the value. We are having some trouble with the GUI framework we use and this annoyance has persisted through our attempts to fix it. Luckily it is quite simple to live with if you just remember to use the enter or tab key after modifying the value.

The intention (as far as I can tell) of the edit dive in planner mode is to plan a new dive. You can use this feature to take an existing dive as a template and then plan a new dive based on that. I was able to open a computer dive on this mode, but failed with a dive that I just imported from elsewhere, so this feature might need some love.

Hearing that Subsurface is unstable for you is quite bad news for us. Have you noticed any specific steps that might be causing the problems and how we could reproduce the bug? And of course there are a number of bugs that are already fixed but not yet released, so hopefully the next version will be more stable, and if not, it would be nice to be able to tackle these as well before the next release.

[Robert Helling]

Hi,

Am 08.11.2016 um 18:19 schrieb Miika Turkia <miika....@gmail.com>:

The intention (as far as I can tell) of the edit dive in planner mode is to plan a new dive. You can use this feature to take an existing dive as a template and then plan a new dive based on that. I was able to open a computer dive on this mode, but failed with a dive that I just imported from elsewhere, so this feature might need some love.

I can reproduce this. This is due to the fact that the dives that are imported as CSV of course don’t have depth samples. Only for the display, we make up the fake profile (using depth and average depth). The proper thing to do would be to produce samples for such dives.

Best

Robert

[Linus Torvalds]

On Tue, Nov 8, 2016 at 11:28 AM, Robert Helling <hel...@atdotde.de> wrote:
>
> I can reproduce this. This is due to the fact that the dives that are
> imported as CSV of course don’t have depth samples. Only for the display, we
> make up the fake profile (using depth and average depth). The proper thing
> to do would be to produce samples for such dives.

No, we should not produce fake samples for such dives. That would then
save completely bogus data when you try to save the dive.

Yes, we create a fake plot, but that's different from actually
creating and saving complete garbage data.

Of course, when you actually go into the *planner*, we should probably
create a similar "fake plan", the same way we create a fake profile.
But that's for planner data only. Maybe that's what you meant.

Linus

[Robert Helling]

Hi everybody,

Am 08.11.2016 um 20:33 schrieb Linus Torvalds <torv...@linux-foundation.org>:

Of course, when you actually go into the *planner*, we should probably
create a similar "fake plan", the same way we create a fake profile.
But that's for planner data only. Maybe that's what you meant.

yes, that’s what I meant. We should start the planner/edit dive (and not silently do nothing as currently for replanning a dive without samples or actually without a dive computer model name) and we should not start with an empty profile but offer the user the fake profile as a starting point.

This patch implements that.

Best

Robert

[Robert Helling]

Hi,

Am 08.11.2016 um 21:22 schrieb Robert Helling <hel...@atdotde.de>:

 or actually without a dive computer model name

with my patch this will no longer prevent us to „replan“ such dives. But maybe also the CSV import should set the dive computer name to something like „imported dive“ or similar.

Best

Robert

[Miika Turkia]

yes it should. Some imports already does that, but not all :(

I am working on a couple of issues with csv import, so could do that change as well

miika

[Miika Turkia]

Seems that the CSV import does that already.

<divecomputer deviceid="ffffffff" model="Manual import">

miika

[Robert Helling]

Miika,

Am 09.11.2016 um 20:35 schrieb Miika Turkia <miika....@gmail.com>:

Seems that the CSV import does that already.

<divecomputer deviceid="ffffffff" model="Manual import">

not for me. I import

[darryl...@gmail.com]

I just emailed Miika my CSV files, along with a video captured from my screen as I reproduce the problems I described, but here's the file in case anyone else wants the CSV to try to duplicate my issues.

[Miika Turkia]

Thanks a lot. This was helpful. The O2 problem you have is due to not hitting tab or enter key after inserting the number. We have not been able to work around that bug so you will need to do the extra step to get the value actually stored.

In general, we do not store information that was not read from dive computer or imported file. So the average depth on the crafted profile is calculation. And since it is just a guess, we do not show it on the info tab. The average depth is wrongly imported without unit conversion. (Internally Subsurface uses metric units. And we do not apparently get enough testing from people using imperial units and doing the imports.)

The long safety stop you see is calculated to show a safe ascent. And since the average depth was wrong, the safety stop got excessively long.

[Darryl H]

In general, we do not store information that was not read from dive computer or imported file. So the average depth on the crafted profile is calculation. And since it is just a guess, we do not show it on the info tab. The average depth is wrongly imported without unit conversion. (Internally Subsurface uses metric units. And we do not apparently get enough testing from people using imperial units and doing the imports.)

Fair enough, but then anyone using Subsurface for a mixture of both manual and computer logged dives (as I am, at least for the early dives) is going to have statistics like the year's average depth be incorrect. It would make more sense if profileless dives were simply ignored for that purpose, rather than entering into the calculation as zero.

 

The long safety stop you see is calculated to show a safe ascent. And since the average depth was wrong, the safety stop got excessively long.

OK, that makes sense. That still leaves the editing bug I mentioned at the beginning. When editing a manually entered dive via Log > Edit dive, the max. depth gets set to zero and the guessed profile is wiped out. 

[Darryl H]

Oh, and one more thing I discovered since starting to import dives from my computer (a Suunto Vyper). I don't know if this is a general issue with Subsurface or specific to Suunto computers, but Subsurface is incorrectly reading my Vyper's *surface* temperature as the *air* temperature. My computer doesn't store the air temperature so I guess since Subsurface is looking for an air temperature somewhere, it's grabbing the only available datum as using that.

[Miika Turkia]

This grabbing of air temperature is intentional. Suunto is very slow
in updating the temperature due to averaging it. Thus the first dive
sample should generally be quite close to the actual air temperature.
Of course, if you float/swim a while before starting the dive, this
assumption is incorrect.

> --
> You received this message because you are subscribed to the Google Groups
> "Subsurface Divelog" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to subsurface-dive...@googlegroups.com.
> To post to this group, send email to subsurfac...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/subsurface-divelog/9deb1b69-4fca-47d4-b6b4-1c19db20cb8d%40googlegroups.com.
>
> For more options, visit https://groups.google.com/d/optout.

[Darryl H]

On Thursday, November 10, 2016 at 12:16:41 AM UTC-5, Miika Turkia wrote:

This grabbing of air temperature is intentional. Suunto is very slow
in updating the temperature due to averaging it. Thus the first dive
sample should generally be quite close to the actual air temperature.
Of course, if you float/swim a while before starting the dive, this
assumption is incorrect.

Ah, I see. Yes, I have a number of a shore dives with surface swims that jumped out at me as problems. Nothing to do about that, I guess.

By the way, I tried tabbing after entering an O2% and it did not fix the problem. I think I was doing that already (including in the video I sent you). The result is the same: for any non-air mixture, the O2% field stays populated. However, if I entire 21% then, though the "Gases used" field once again becomes "air", the O2% field goes back to being blank.

[Robert C. Helling]

Hi,

OK, that makes sense. That still leaves the editing bug I mentioned at the beginning. When editing a manually entered dive via Log > Edit dive, the max. depth gets set to zero and the guessed profile is wiped out. 

for that I already sent a patch that fixes that. Should be in the daily build soon.

Best

Robert 

[Robert Helling]

[Miika Turkia]

> --
> You received this message because you are subscribed to the Google Groups
> "Subsurface Divelog" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to subsurface-dive...@googlegroups.com.
> To post to this group, send email to subsurfac...@googlegroups.com.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/subsurface-divelog/F00C201A-392D-46FD-90E1-26BDA1293FDA%40atdotde.de.

>
> For more options, visit https://groups.google.com/d/optout.
>

> This patch does exactly this. Alternatively, we could make up some average
> depth for dives for which we only know the max depth and duration. But
> making stuff up is not what we usually do. What do people here thing?

No made up values, please. At least I cannot think of any way to guess
sensible average depth based on max depth and duration.

One should also note that the year's average depth is actually the
average of maximum depths (IIRC). Maybe we should change the title in
the yearly statistics to:
Maximum Depth (m)
Average Minimum Maximum

We have been bitten by the definition of average depth before and that
is why we have disabled it from the Stats tab. But similar change
might be in order to the Stats tab as well. Thoughts/comments?

miika

[Darryl H]

No, I'm pretty sure it's the weighted average of the averages (i.e. the true average for the year). Or else there's another bug somewhere. Currently, Subsurface is telling me in the yearly stats that my average depth for 2009 (a year that all my dives were manually input for) is zero.

[Miika Turkia]

> On 10 Nov 2016, at 21.00, Robert C. Helling <rob...@euve10195.vserver.de> wrote:
>
> Miika,

>
>> Am 10.11.2016 um 18:04 schrieb Miika Turkia <miika....@gmail.com>:
>>
>> One should also note that the year's average depth is actually the
>> average of maximum depths (IIRC). Maybe we should change the title in
>

> I looked at the code yesterday: it is the average of the average depths weighted by duration. So it is really the average depth of all the time spent in the water. I would say this makes sense to me.

Excellent, I would say. (And no surprise I remembered this incorrectly :D)

miika

[Darryl H]

I don't know if this is a Mac specific problem, or something general with the last release, but the Facebook sharing functionality seems to be disabled. I don't have a Facebook tab under my preferences to input my login info.