CSV import problem from DataPlus 2 file

[Etsuko Moriyama]

I'm trying to import old dive records generated by Oceanic DataPlus 2. The original file was in .mdb format. The downloader software ran only on Windows98, but I could read the file using a newer OceanLog (2.4.9 Built 06.12.15) running on Windows 7 (using Parallels). OceanLog saved the file in .dlg format. From here I tried two ways to import the file into Subsurface 4.5.6 (running on Mac OS 10.10.5). First I tried to directly import .dlg file using divelogs.de account. Following the instruction on their website, I could import 150 dives. Then I exported these records in both .dld and .udcf format so I can import them into Subsurface. But when I looked at these exported files, they didn't include temperature information. It shows only the single minimum temp, but not all temp for each 30 seconds. When I exported the original .dlg file using Oceanlog, both TXT and ZXU files include all these temps (as well as many more) information.

Then I tried to import a single diving in a cvs file into Subsurface. I'm attaching both the originalTXT file exported by OceanLog (directly from the .dlg file) and the csv file (I deleted the top header lines and keep only the csv format). Using the "Import log files", the csv file could be read by Subsurface. The csv file has a lot of columns as below (showing on the header line):

Elapsed Dive Time (hr:min),Depth(FT),Nitrogen Bar Graph,Oxygen Bar Graph,Ascent Rate(FPM),Air Time Remaining,Dive Time Remaining,Deco Time (hr:min),Stop Depth(FT),Temperature(F),PO2,Tank,Pressure Reading(PSI),Link Status,Dive Status

I allocated "Sample time", "Sample depth", and "Sample temperature" to the columns 1, 2, and 10. I wasn't sure what to do with other columns. So I kept them empty. After clicking "OK", the depths are plotted, but the temps are not. I also tried to remove all columns but only the above three (so all columns are assigned the labels). It still doesn't show the temp record.

 

How can I import both depth and temp records from a csv file? It would be also great if the original TXT file can be read directly. I can try to write a small program to convert this TXT format to something else that can be read by Subsurface, too. Is there any good dive log file format where I can store those information found in the TXT file?

Thank you very much for your help!!

Etsuko

[Etsuko Moriyama]

To update, I asked divelogs.de about missing temperature information and was told that divelogs doesn't store per sample temp information. No wonder I didn't see it. So my only option seems to be using csv file only if it works!

Etsuko

[Miika Turkia]

Configuration as shown on the attachment works for me to import your
dive profile along with the temperatures. The rest of the columns you
can just leave be. Temperature should also be imported from UDDF
format. If you could provide that as well I can make sure there is no
bug in our import.

If you want to write a converter, our XML log file format would be the
best option. If you look at our XML file, you should be able to figure
it out quite easily. We have some test dives in the logs that should
illustrate most of the features and you can also save your own log
file and take a look at the XML. Also, adding option to parse the TXT
file directly shouldn't be too hard...at least to skip the header and
parse the actual content.

[Etsuko Moriyama]

Thank you, Miika, for replying to my question! Looking at the configuration you used, I realized that I forgot to change the unit from "metric" to "imperial". That solved my problem!  Thank you!

I'll also take a look at the XML file format as you suggested. In the mean time, I also tried to simply change the header lines into another csv file. The manual says two types of csv files can be imported: dive details and dive profiles. But after importing these two csv files, I'm not sure how I can merge those information? Can I for example, import the dive details csv file first, then add the corresponding dive profiles from another csv file? Or for this, writing a converter to XML format is better?

By the way, when I tried to import a ZXU file, subsurface crashes. FYI, I'm attaching the file I tried.

Etsuko

On Wednesday, May 4, 2016 at 12:12:30 AM UTC-5, Etsuko Moriyama wrote:

[Miika Turkia]

The crash should now be fixed. However, it will take a couple of hours
before the patch ends up int our source repository and some more
before there will be daily binaries available to test it out.

Merging of dive details and the profile should occur automatically
when certain criteria is met. See:
https://groups.google.com/d/msg/subsurface-divelog/KsTUvwy6zJk/W94UuVqvBAAJ

Note also the fact that the display is not refreshed properly when
merging the dives.

> --
> You received this message because you are subscribed to the Google Groups
> "Subsurface Divelog" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to subsurface-dive...@googlegroups.com.
> To post to this group, send email to subsurfac...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/subsurface-divelog/3eded1c8-650b-44da-be79-780703ad1eef%40googlegroups.com.
>
> For more options, visit https://groups.google.com/d/optout.

[Dirk Hohndel]

On Thu, May 05, 2016 at 11:22:43AM +0300, Miika Turkia wrote:
> The crash should now be fixed. However, it will take a couple of hours
> before the patch ends up int our source repository and some more
> before there will be daily binaries available to test it out.

The Mac binary with Miika's latest patches is here

https://subsurface-divelog.org/downloads/daily/Subsurface-4.5.6-12-g4d8d7c2a0fa1.dmg

/D

[Etsuko Moriyama]

Thank you, Miika and Dirk!  ZXU files work now! Between ZXU and csv files, I may not have to write a converter to import those old DataPlus 2 files.

Thank you very much again!!

Etsuko

On Wednesday, May 4, 2016 at 12:12:30 AM UTC-5, Etsuko Moriyama wrote:

[Etsuko Moriyama]

I converted the original Oceanic DataPlus 2 text file to a csv file and now I can import these old dive logs. I have a few more questions regarding the csv format. 

Attached TXT file is the original from DC. The csv file is the converted version. Importing this file works except for "Air temp", "Tags", and "O2" columns. For the air temp, I took the first temperature sample and populated this column. For this dive, I used Nitrox with 32% O2. So I wanted to add a tag "Nitrox". Import dive log file dialog window shows all columns with correct label assignments including "Air temp", "Tags", and "O2". But after importing, "Notes" page shows empty "Air temp" and "Tags" fields.. In the Equipment page, O2 information is not shown either. How can I include these information in the csv file?

Also I'm thinking about adding extra information using "Extra data" page. What column headers should be used so I can import those data from csv files? Manual editing doesn't seem to be allowed for this page?

Thank you!

Etsuko

On Wednesday, May 4, 2016 at 12:12:30 AM UTC-5, Etsuko Moriyama wrote:

[Miika Turkia]

The missing fields are not imported currently. Originally the profile
import only supported the samples. Then I added date field as constant
column, and it never occurred to me to add some other similar fields..
But now that the issue is brought up, I'll add these fields to be
imported. (Dive details import uses the same import dialog, and that
is why the "extra" column headers have been available.)

I am quite sure the extra data import needs to be implemented as well.
I am thinking of a column called "extra data" that would be parsed
similarly to the tags column. The content could be something like:

"serial=123456,battery level=3,misc=true"

miika

> --
> You received this message because you are subscribed to the Google Groups
> "Subsurface Divelog" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to subsurface-dive...@googlegroups.com.
> To post to this group, send email to subsurfac...@googlegroups.com.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/subsurface-divelog/b10673c4-07c4-4745-bb03-329eed23086d%40googlegroups.com.

[Etsuko Moriyama]

Thank you, Miika, for considering to add these columns to be imported! Reading your reply, I realized that if I prepare two files for each diving, dive profile and dive details, those "extra" columns could have been imported using the "details" file. Seeing those extra labels, I just assumed that those columns can be used also in the profile file. It is still really useful, though, if two files don't need to be prepared for each diving to be imported. It'd be also very nice if "Extra data", especially multiple of them, can be imported using a csv file, too!

Thank you!

Etsuko
On Wednesday, May 4, 2016 at 12:12:30 AM UTC-5, Etsuko Moriyama wrote:

[Etsuko Moriyama]

I'm sorry for many times...  But I'm still having some problems with trying to import csv files. I created two files, one for dive details and one for dive profiles (both files are attached). From the info.csv file, the "Air temp" and "Tags" are now correctly imported but O2 information is not. Also these two entries are not merged as one diving entry even though both files have the same "Dive #', "Date" and "Max depth." I also added "Time" column in the profile file because I found that without this column, the imported profile shows a wrong time (the current time instead of just keeping it empty). But with or without "Time" column, after importing the two files, I get two separated entries instead of one single merged record. Then I realized "merge selected dive" popup menu. Using this, I get a single record. But now I lost the "Air temp" information. What is the proper way to merge the dive information and dive profile csv files?

Thank you very much for all your help!

Etsuko

On Wednesday, May 4, 2016 at 12:12:30 AM UTC-5, Etsuko Moriyama wrote:

[Miika Turkia]

You need to add duration field to the info.csv. That field must also
match between the dives in the two import runs. And of course you can
also use the manually merging, if the automerge does not occur
properly.

I just sent patches to fix the issue with O2 (and He), and the
disappearing air temperature on merge.

> --
> You received this message because you are subscribed to the Google Groups
> "Subsurface Divelog" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to subsurface-dive...@googlegroups.com.
> To post to this group, send email to subsurfac...@googlegroups.com.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/subsurface-divelog/a2f97a2e-3b96-428a-8ed9-0db6620690fd%40googlegroups.com.

[Etsuko Moriyama]

Thank you, Miika!

I added "Duration" to the info file and now it's working. Actually both "Duration" and "Max depth" fields seem to be needed in addition to "Date" and "Time". Then the two imported files can be automatically merged. With this, "Air temp" is also kept! I'm looking forward to see the patched version, hopefully appearing soon on the daily directory, then O2 information can be imported. Then I think I can import all dive record from my old DataPlus DC!

Etsuko

> email to subsurface-divelog+unsub...@googlegroups.com.

[Etsuko Moriyama]

I'm sorry but I think I got excited too early...

With both "Duration" and "Max depth" included in both files, the merging worked automatically. But then I realized that it lost the dive profile data! So here is what's happening:

1) With both "Duration" and "Max depth": Automatic merging seems to work (becoming one single record). But profile data are lost if importing is done profile.csv, then info.csv. If importing info.csv first then profile.csv next, profiles are kept but Air temp and Tags data are lost.

2) With only "Duration" but no "Max depth": No automatic merging. After manually merging, Tags data is kept, but Air temp is lost. Depth and temp profiles are kept.

Maybe these problems may have been already fixed with the patch. Once I can try it, I'll let you know how it looks.

Etsuko