Help to import .csv as separate tiddlers

[Jon]

Hi,

I'm importing a .csv file and I want each row to be a separate tiddler.

What's the best way to do this?

I have tried TW5-JsonMangler/ where "each individual row is rendered to a tiddler, then these tiddlers are packaged as shadow-tiddlers into a plugin."

but then there doesn't appear to be any instructions as to what to do with the resulting plugin (sorry, I'm only an end user).

Any guidance about using this method or an alternative much appreciated.

Regards

Jon 

[springer]

Have you tried the excel importer: https://tiddlywiki.com/prerelease/editions/xlsx-utils/

?

-ESpringer

[Jon]

Yes, I did come across that after posting but I've not had much success with it.

When trying the demo in the online edition, it produces a javascript error so I downloaded the plug-in to my wiki. I then managed to get rid of the Presidents and Justices demo import specification from the drop down and can't get it back so don't know how to use it - not having a good day!

Regards

Jon

[springer]

Jon, I do remember that it took some poking around before I got the spreadsheet import process to work for me — there are steps to make sure that you're specifying the right worksheet and handling the various fields properly. But I was thrilled when it did work; it was a tremendously efficient way of importing loads of data.

I'm not able to put time into it now, but perhaps there are folks here with a better sense of how to explain it clearly.

-Springer

[Joshua Fontany]

Hi Jon, author of JsonMangler here. I setup the import to package the imported tiddlers into a plugin to have a simple way of overwriting.updating/deleting a whole import at once.

The way that tiddlywiki is setup, "shadow" tiddlers are full tiddlers in the system (beware of confusing this with "system" tiddlers that start with "$:/" because something can be both a shadow and a system tiddler).

Meaning that you can search for and/or filter-query any "shadow" tiddler just like any other tiddler, BUT you have to start your filter runs with `all[shadows+tiddlers]` instead of the default (asumed if missing) `all[tiddlers]`. The one special feature of shadow tiddlers is that you can edit them, creating a "regular" tiddler of that name to store the edits, and then if you delete the edited version the "backup/shadow" version is immediately referenced in its place.

For example, after importing the Video Games CSV in my example wiki, you can hit the Filter tab of $:/AdvancedSearch and enter `[all[shadows+tiddlers]search:*[Mario]]` to display links to all references to "Mario".

If you want to promote a shadow tiddler to a regular one, the easiest way to do that is a weird little hack I picked up, where you setup a button to delete a field that DOES NOT EXIST from each shadow tiddler, like so:
```

\define plugin-target() Data/vgsales

\define unpack-actions()

<$list variable="shadowtiddler" filter="[<plugin-target>indexes[/tiddlers]]">

<$action-setfield $tiddler=<<shadowtiddler>> $field="does.not.exit.in.this.wiki452369084306093845760894253" />

</$list>

\end

<$button actions=<<unpack-actions>> >

Press here to unpack <<plugin-target>>

</$button>

<$list variable="test" filter="[<plugin-target>indexes[/tiddlers]]">

</$list>

```

This creates a regular tiddler that is identical to the shadow, for each one in the plugin-target. Change the definition of that line to your target plugin containing yuor imported tiddlers and it will "unpack" it for you (you will no longer need `all[shadows+tiddlers]` in filters to target the "unpacked" tiddlers). You will see the links to your tiddlers go from being Bold to normal font to show that they are now "overwritten shadow tiddlers", and exist as normal tiddlers in the wiki.

Best,
Joshua F.

[Jon]

Hi Springer, 

That's encouraging if it worked so well for you - I'll investigate it further for future use.

I've got the .csv all cleaned up and ready to go and I'm nearly there with JsonMangler.

Regards

Jon

[Jon]

Hi Joshua,

many thanks for getting back to me with the missing piece.

I'll have a go later and report back.

Regards

Jon

[Jon]

Hi Joshua,

The unpacking worked really well and things are looking promising but there are few glitches I can't get past.

For the .csv file to import, I included the fieldnames 'title' 'text' & 'created'.

The text field appeared in the body of the tiddler as it should but the created & title fields appear to be absent.

Instead, the name of the tiddler is derived from the name of the plug-in.

I noticed there is an option to define the primary key and as the title field is in the first row of the spreadsheet,  I chose the column 0 option. 

This created a name for each tiddler which includes the contents of the 'title' field but combined with the name of the plugin - so perhaps a bit of tweaking will solve this easily.

Also, I notice that for the unpacked tiddlers, there is still a message which appears in edit mode:

"This is a modified shadow tiddler. You can revert to the default version in the plugin Data/Metrics - HH by deleting this tiddler"

Regards

Jon

This is a modified shadow tiddler. You can revert to the default version in the plugin Data/Metrics - HH by deleting this tiddler

[Jon]

Just played around with this a bit and having the 'title' in the 1st column of the spreadsheet, setting the primary key to 0 and then putting [<primaryKey>] in the box for the shadow name filter names the resulting tiddlers with the 'title' field - so all good!

With regard to the creation date,  if I gave it a different name (so that it is imported), I suppose I could still perform ordering  and searching etc. in the same way.

Good to have some confirmation of this before I go any further.

Thanks

Jon

[Jon]

Hi Joshua,

A bit more experimenting and yes, it does appear that using another field for the created date does allow for correct sorting etc.

I also found that adding a 'tags' column in the spreadsheet is respected for the import and so a tag is added to each tiddler.

Thank you!

Regards

Jon

[Joshua Fontany]

Excellent work figuring out the Filter syntax needed to set the title!

Handling all the possibilities around CSV data in a flexible way was a complicated problem, so thank you for the feedback (notes for the next version's UI, which is a while out).

In order to get rid of the shadow tiddler warning, once you have confirmed that the source plugin is unpacked to your specs, you can simply delete the source plugin tiddler. It will leave all the "modified" tiddlers behind.

Best,
Joshua Fontany

[Jon]

Thanks Joshua and for the extra bit about the message. 

After using your suggestion for the filtering syntax, I actually like the idea of keeping everything bundled as a plugin - what I've imported is 19 years' worth of diary entries, so I'm not going to edit them anyway and I like the warning message which pops up if I attempt to (but it's good to know I can unpack them if needed). After opening, they also don't appear in the recent tab which also suits me by keeping that free.

Just as an aside, I noticed that I wasn't able to drag and drop the diary plugin into another wiki - nothing is imported.

Thanks again.

Jon

[Osin]

Hi Jon,

How did you initially import the CSV, just drag and drop?

I can't seem to find the demo or documentation. When I import, the csv is a "type: application/vnd.ms-excel"

Thank you

[Jon]

Hi Osin,

Sorry, I've not been on the site for a bit so missed your post.

I actually imported the .csv file using Joshua's Json Mangler as referenced above.

I figured out a few things that were needed, but they're referenced in the thread above as well.

Regards

Jon