csv_unpack plugin: Import CSV rows as tiddlers/fields

[Evan Balster]

Hey, all —

Made another plugin for my own use; some here might find it helpful:

http://evanbalster.com/tiddlywiki/csv_unpack.html

It parses imported CSV files, converting each row into a tiddler.  It works as part of TiddlyWiki's own parse mechanism.

CSV files are expected to have a header row.  Header fields may be "re-mapped" to different tiddler fields using the plugin's settings.  Default fields can also be applied to all tiddlers created from CSVs (this is useful for applying tags).

This is a quickie and I don't expect to do substantially further development on it.  That said, let me know about any bugs you find.

[ste...@gmail.com]

Hi Evan,

On Wednesday, December 20, 2017 at 8:45:40 AM UTC+1, Evan Balster wrote:

 

Made another plugin for my own use; some here might find it helpful:

http://evanbalster.com/tiddlywiki/csv_unpack.html

It parses imported CSV files, converting each row into a tiddler.  It works as part of TiddlyWiki's own parse mechanism.

This looks like a really nice idea. However, I wasn't able to import any of my CSV files, no matter whether I used a comma or a semicolon as a delimiter, and no matter whether I added quotation marks or not. Could you maybe publish a sample CSV file that works for you? This way, it might be easier for us to debug your plugin and/or to adjust our own files.

Best,

Stef

[Evan Balster]

Hey, Stef —

A first row specifying column names is required by this plugin.

I use the "Papa Parse" library, which has a demo you can test with here:  http://papaparse.com/demo

I've had trouble with some of my own CSV files not importing; the files in question came from a program called DB Browser and contained NULL characters as well as malformed line breaks (though the plugin should be able to handle the latter).  These errors showed up in the papa parse demo as well.

[ste...@gmail.com]

On Wednesday, December 20, 2017 at 7:21:59 PM UTC+1, Evan Balster wrote:

I use the "Papa Parse" library, which has a demo you can test with here:  http://papaparse.com/demo

Thank you! This example works. Amazingly, I was able to import a considerable number of CSV files generated by LibreOffice on my Linux PC, too. A very nifty plugin, indeed!

However, all attempts at importing CSV files created by Excel on MS Windows failed. Maybe something to do with Excel itself, and/or the charset (German locale)? If I find an explanation, I will report back.

Cheers,

Stef

  

[Evan Balster]

Excel might be exporting its CSV in UTF-16 encoding?  I could stand to add some mechanism to display Papa Parse's errors (and I'm asking about this in TiddlyWikiDev).

Myself, I'm importing two years' worth of work logs so that I can visualize and datamine them for my end-of-year self-review.  I've been envious of my colleague Devine and his Horaire for some time now...

[ste...@gmail.com]

On Wednesday, December 20, 2017 at 9:04:09 PM UTC+1, Evan Balster wrote:

Myself, I'm importing two years' worth of work logs so that I can visualize and datamine them for my end-of-year self-review.  I've been envious of my colleague Devine and his Horaire for some time now...

Maybe of interest for some readers here: With your plugin, it's also straightforward to import the "tweets.csv" file from the official Twitter archive, which contains all the tweets (and a lot of metadata) from one's personal account. Still don't know what to do with this in TiddlyWiki, but there is certainly potential...

[ste...@gmail.com]

Hi Evan,

It's me again - today, I had the chance to test this once more on my Windows machine. However, I had no luck - I wasn't even able to import "normal.csv" from the "Papa Parse" library you linked to. In both browsers (Firefox and Chrome) TiddlyWiki treated this file as a regular text file, not as a .csv. I even tried to import "normal.csv" into a local copy of your public wiki (http://evanbalster.com/tiddlywiki/csv_unpack.html) - same result.

However, on my Linux machine, everything works fine (including the import of "normal.csv", which I had emailed to myself from the Windows machine). 

Any idea what might be causing those problems? Some particular security settings in Windows, maybe?

Best,

Stef

[Evan Balster]

Hey, Stef —

Works OK under Chrome on OS X.  Could be a newlines issue, though I'm not sure how that would work.  Probably the best thing to do for debugging would be to have the deserializer prepare an informative tiddler in the imports if any errors occur.  Currently error messages generated by Papa Parse don't show up at all...