BUG: CSV Import issue

[Tony G]

Dan, I'm testing Export and Import of CSV and noticed the following:

Use any task list, go to Export as CSV, select All, comma-delimited, then Export.
Now Import that list back in, to the top of the active tasklist.
The data is spread out through the tasklist and it looks corrupted - I haven't looked into too much detail but it looks like some or all rich text comments are converted to plain text. The imported tasks are placed after the top task - it doesn't actually get inserted at the very top.

I've seen this sort of weirdness before but now I'm focused on it and it's 100% repeatable.

I can provide detail if required.

HTH
T

[.dan.g.]

>> I can provide detail if required.

Yes please. Screenshots and steps pls.

[Tony G]

The attached .tdl is complete rubbish, just used for testing in v7.0.x.

This shows my export settings.


In  the "Exporting Csv File" dialog (CSV should be caps on that page) shows comma for delimiter and the "Always export..." is checked.
The task attributes are shown here, compare to the import below.



Excel is launched (I wish it wasn't) and no changes are made. But note that there are no column headings for time units.




Wait a few seconds to avoid file lock issues, then Close Excel.

Now go to Import Tasks, set file name. Note the initial state of the task data.


Note import fields with no mapping to target attributes:


Here is the result.


1) New ID 226 from 9 removed leading empty lines from its rich text comment.
2) New 227 from 8 has all of its embedded images removed from the comment, and the remaining text is also moved up.
3) The order of Level 2 tasks 223,224,225 is different from the source 4,13,14.
4) Other level 1 tasks are ordered differently.
5) After an import the "file changed" asterisk on the list's tab is removed.

My original notes about mis-ordering were due to the list being sorted by a field other than the position, so it's not as bad as I thought. But this test was done with tasks sorted by position, as shown. I'm not sure if that's a factor.

HTH
T

[.dan.g.]

Thx for extra details Tony.

>> -1) Excel is launched (I wish it wasn't) 

You can disable this in the Import/Export preferences

>> 0) '--unnamed--' attributes when importing

i) Originally I had the bright idea of putting the units in a separate column so that the column could be accumulated.

However, it doesn't really make to do this because often units are mixed and the parent has an aggregate total of its subtasks, both of which would mess up any summing.

I will fix this in 7.0.12 by merging the unnamed units columns with their respective time column.

ii) The '--unnamed--' after the comments field is spurious and I can safely remove it.

iii) The '--unnamed--' below the filelink I need to investigate

Note: The custom attributes will always require manual mapping for now.

>> 1) New ID 226 from 9 removed leading empty lines from its rich text comment.

>> 2) New 227 from 8 has all of its embedded images removed from the comment, and the remaining text is also moved up.

Rich-text comments are not exported to csv, only the simple-text equivalent.

As a consequence, importing from csv can only ever produce simple text comments.

>> 3) The order of Level 2 tasks 223,224,225 is different from the source 4,13,14.
>> 4) Other level 1 tasks are ordered differently.

When I import with no sorting active, it works correctly. Can you verify if this is also the case for you?

>> 5) After an import the "file changed" asterisk on the list's tab is removed.

I can't reproduce this. Might the asterisk have disappeared because of auto-saving?

Dan

On Thursday, 14 January 2016 06:36:35 UTC+11, Tony G wrote:

[Martin Lemburg]

Dear Tony,

CSV is a pure plain text format, so the comments written in RTF should not be exported to CSV!

There are mostly no CSV importing applications, that will detect RTF text and will reuse it, so the RTF text will be displayed like loading a RTF as plain text into an editor!

I expect similar problems with date/time fields! Because if there is a string exported, the format must match some (internationally ruled) rules, but if they match the rules of the importing application and if the data saved by the importing application is readable again on re-import, that is not guaranteed!

Excel needs integers as date/times or an internationally defined string format and will save integers,r ight?

Exporting to CSV means to loose data, so importing should never bring the original data back! This export is no "backup"!

And this export is not useful to re-import!

If you need to export to allow others to reuse it, you can't expect to get really useful data back!

But I agree, the tasks structure and the basic tasks data of exported data should be the same imported back!

So, what do you want to achieve with ex-/importing CSV files?

Best regards,

Martin

[Tony G]

It's looking much better in 7.0.12. Thanks!!!!

dan.g. wrote:

Note: The custom attributes will always require manual mapping for now.

I'll create a separate thread for that and understand that it may not get processed anytime soon.

 

>> 1) New ID 226 from 9 removed leading empty lines from its rich text comment.

>> 2) New 227 from 8 has all of its embedded images removed from the comment, and the remaining text is also moved up.

Rich-text comments are not exported to csv, only the simple-text equivalent.

As a consequence, importing from csv can only ever produce simple text comments.

Why not export the CUSTOMCOMMENTS node? It's pure text, can be loaded right back into the same field, and then the plain text version can be re-generated from that on import.


 

When I import with no sorting active, it works correctly. Can you verify if this is also the case for you?

Yeah, that was it. I thought I turned off sorting but it was still sorting by Position.


 

>> 5) After an import the "file changed" asterisk on the list's tab is removed.

I can't reproduce this. Might the asterisk have disappeared because of auto-saving?

Possibly, I don't see this right now. Will keep an eye on it.

Best,
T
 

[Tony G]

Thanks for your feedback, Martin!!

On Thursday, January 14, 2016 at 12:54:56 AM UTC-8, Martin Lemburg wrote:

Dear Tony,

CSV is a pure plain text format, so the comments written in RTF should not be exported to CSV!

There are mostly no CSV importing applications, that will detect RTF text and will reuse it, so the RTF text will be displayed like loading a RTF as plain text into an editor!

You are correct. However, let's define the "intent" of an export. We can export to CSV or another format for immediate processing by another tool. Or we can export to another format merely as a vehicle for some other kind of processing. In this case the format doesn't need to be human-legible, it just needs to contain the required data in some documented format.

Since the .tdl XML file is already entirely plain text, I'm hoping that an export of "All attributes" really will contain ALL attributes which are available, and that includes the compressed, Base64-encoded, plain-text CUSTOMCOMMENTS node.

"That will make the CSV file messy."  If someone intends to open the file in Excel, sure, that one column will be a mess. But by asking for ALL attributes we understand that some of the data may not be human-readable, and not intended to be viewed as-is with a rendering engine like Excel.

And that's really my focus, that an Export should provide all of the data that we ask for, and not just the data that is expected to be viewed with specific tools.

 

I expect similar problems with date/time fields! Because if there is a string exported, the format must match some (internationally ruled) rules, but if they match the rules of the importing application and if the data saved by the importing application is readable again on re-import, that is not guaranteed!

Excel needs integers as date/times or an internationally defined string format and will save integers,r ight?

You're making your own argument that CSV is a plain text format - CSV does not equal Excel! The export says CSV, not XLSX. I can write an export for XLSX if someone wants "Excel", but when asking for CSV, I don't want the exporter to convert the data into a format that Excel understands.  I do want a plain text format. And anyone post-processing this data will need to convert their external date/time format into some other format that they require.


 

Exporting to CSV means to loose data, so importing should never bring the original data back! This export is no "backup"!

And this export is not useful to re-import! If you need to export to allow others to reuse it, you can't expect to get really useful data back!

That's the way it works now but there's no reason for it. Again, if the .tdl file is plain text there's no reason why the .csv can't contain the exact same data. And if I modify the data in the CSV with some program, I hope that on Import that it should just accept all fields as well - including a modified Base64 block of text in the CUSTOMCOMMENTS field.

 

But I agree, the tasks structure and the basic tasks data of exported data should be the same imported back!

So, what do you want to achieve with ex-/importing CSV files?

I think it's clear now. The definition of "Import and Export" currently implies that the software knows what's going to be done with the data, so decisions are made for us. I don't think that should be the case.

I'm hoping that from this thread the changes documented in this other new thread can be implemented.

Regards,
T