CSV Import

[TonyM]

Folks,

I note the addition of a CSV Parser in TW5.1.18 which includes the csvtiddlers macro for exporting.

Is there a new or existing method to import csv files and generate tiddlers ?

I know there is the excel handling tool but wondering if there is a simpler title - tiddler and all columns to fields 

Regards

Tony

[Joshua Fontany]

Evan built one a while back: https://evanbalster.com/tiddlywiki/csv_unpack.html

[TonyM]

Thank you

[SteveH]

Hi there,

I've just tried the unpacker.

Firstly, I used CSV Export to export a couple of tiddlers, then dragged and dropped the CSV file, and imported it.  Rather than end up with a  couple of new tiddlers a new 'combined' tiddler  was created.

The plugin documentation says 'simply drag a valid CSV file into your wiki'.  - Is the format described anywhere in the plugin?

Thanks

Steve H

On Tuesday, 5 March 2019 14:23:49 UTC+11, TonyM wrote:

Thank you

[Joshua Fontany]

Hmmm, yah, just tried this. Doesn't seem to work as described anymore. I'd have to scan through the source to see what is going on.

[TonyM]

I require this once again.

Is there a way to use it manually without the drag and drop process which is not working.

Regards

Tony

[TonyM]

This was an old request, but I still need to resolve it.

• I have built my own macro csv column to fieldname process
• But I do not know how to feed a row at a time from my csv/datatiddler tiddler to the macro
• I tried to make it look like a data tiddler
• full name:,firstname,surname,type,email,location
• But I still could not get each row with indexes

I am lost. All I want to do is process a row of a tiddler at a time. This should not be rocket science.

Thanks in advance if anyone can get me out of this desperate state.

Tony

[Mark S.]

Hi Tony,

The 5.1.20 prerelease has new split operators.

You could make your CSV tiddler a data tiddler as you suggested. Navigate through all the indexes, and split the line by the comma. Obviously this will fail if you have any embedded commas, but it should work with "regular" data.

-- Mark

[TonyM]

Mark,

Thanks, I am using the prerelease and split operator already. After a nights sleep I tried to take another approach and there is clearly something I do not understand.

Here is what I am doing, 

• I hope others can use successfully, for processing CSV files, 
• but only when I resolve a bug. 
• Oddly I do find various macros but not widgets in TiddlyWiki that supposed to parse csv. None of which seems documented at tiddlywiki.com, only the export process.

I added a tag "<eol>" for end of line to each row in the csv

\define each-row(row)
$row$<br>
\end
<$list filter="[{StaffList.csv}split[<eol>]]" variable=input-row>
  <$macrocall $name=each-row row=<<input-row>>/>
</$list>

and I get a list of each row.

Then inside the each-row macro above I am adding the following for each field, where the nth[] is the position in each row.

<$set name=fieldname value={{{ [[$row$]split[,]nth[1]] }}} >

The full detail for each row is

<$set name=person-tiddler value={{{ [[$row$]split[,]nth[1]] }}} >
<$set name=first-name value={{{ [[$row$]split[,]nth[2]] }}} >
<$set name=surname value={{{ [[$row$]split[,]nth[3]] }}} >
<$set name=person-role value={{{ [[$row$]split[,]nth[4]] }}} >
<$set name=email-address value={{{ [[$row$]split[,]nth[5]] }}} >
<$set name=import-region value={{{ [[$row$]split[,]nth[6]] }}} >
Person=<<person-tiddler>>
first-name=<<first-name>>
surname=<<surname>>
person-role=<<person-role>>
email-address=<<email-address>>
import-region=<<import-region>>
<br><br>
</$set></$set></$set></$set></$set></$set>

The idea being to extract the nth item in each row and assign it to a variable

What is ODD is the above actually works for the first row

I get a nice line of field=value key/value pairs.

However it displays all rows, but never sets the variables ever again, only for the first case.

This is seriously confusing me.

Any feedback would be welcome.

Regards

Tony

[TonyM]

I think I found the problem

Watch this space

Tony

[TonyM]

There is possibly a bug, which I am yet to work Around.

I changed the method used in the set statements and and the bug is now more evident

<$set name=person-tiddler filter="[<row>splitbefore[:]removesuffix[:]]" >
<$set name=first-name filter="[[$row$]split[,]nth[2]]" >

The first person-tiddler value is always

[[first surname]]

the first time

then ALL subsequent rows it is 

[[ firstsurname]]

where a mysterious space is inserted in the result.

In my previous set statements using {{{ filter }}} this space seems to have caused a failure of sorts
Using the 

filter="[<row>splitbefore[:]removesuffix[:]]"

Now shows the unwanted space

Perplexed

[Mark S.]

Where is <eol> coming from?

[TonyM]

I added it as the value in the last column in the csv because I seemed to have no way to split on new row/line

In other apps one would say split[\n] to indicate newline

Regards

Tony

[Mark S.]

I don't know why your approach didn't work, but switching

[$row$]

to

<__row__>

Seems to fix it.

It seems like it must be caching [$row$] inside of the triple bracket transclusion.

[Mark S.]

In my test, I'm using a pound sign (#) as the eol.

[TonyM]

Mark

I had changed to use the filter on the set statements

<$set name=person-tiddler filter="[<__row__>splitbefore[:]removesuffix[:]]" >
<$set name=first-name filter="[<__row__>split[,]nth[2]]" >

But <__row__> did not work for me here

I returned to the {{{ }}} method and used your suggestion

<$set name=person-tiddler value={{{ [<__row__>split[,]nth[1]removesuffix[:]] }}} >
<$set name=first-name value={{{ [<__row__>split[,]nth[2]] }}} >

it now works, thanks so much

My normal practice would be to take this moment and try an evaluate what led me to this?

But I am so frustrated by it, I want to move on. Thanks so much for your support

Regards

Tony

[Mark S.]

Glad I could help ;-)

[TonyM]

For those following wanting a final solution to process csv this is what worked for me

\define each-row(row)

<$set name=person-tiddler value={{{ [<__row__>split[,]nth[1]removesuffix[:]] }}} >
<$set name=first-name value={{{ [<__row__>split[,]nth[2]] }}} >

<$set name=surname value={{{ [<__row__>split[,]nth[3]] }}} >
<$set name=person-role value={{{ [<__row__>split[,]nth[4]] }}} >
<$set name=email-address value={{{ [<__row__>split[,]nth[5]] }}} >
<$set name=import-region value={{{ [<__row__>split[,]nth[6]] }}} >
person=<<person-tiddler>>
first-name=<<first-name>>
surname=<<surname>>
person-role=<<person-role>>
email-address=<<email-address>>
import-region=<<import-region>>
<br>

</$set></$set></$set></$set></$set></$set>

\end
<$list filter="[{StaffList.csv}split[<eol>]removesuffix[,]]" variable=input-row>

  <$macrocall $name=each-row row=<<input-row>>/>
</$list>

• Each row in the csv ends with a column containing "<eol>" to indicate the end of line
• The set statements set a variable to the value in the nth[n] column in the csv

Now I will use these variables, an action create tiddler for each row, a template tiddler, and wrap it all in a button to bulk generate tiddlers from my csv file.

I will see if I can share that as well

Thanks soooo much to Mark S

Regards

Tony

[TonyM]

A final issue to bulk create new tiddlers from CSV

This works however it has the unpleasant need to open every tiddler for edit, and demand I save each.

In the above solution rather than display the key value pairs, I have added a action-sendmessage to create my new tiddlers

<$action-sendmessage $message="tm-new-tiddler" $param="$:/PSaT/WRRef/templates/objects/person"
title=<<person-tiddler>>
first-name=<<first-name>>
surname=<<surname>>
person-role=<<person-role>>
email-address=<<email-address>>
import-region=<<import-region>>
tags="batch"
/>

I am using tm-new-tiddler because unlike 

ActionCreateTiddlerWidget

it permits the use of a template 

GitHub Issue raised here

Then I invoke my each row for all csv rows by wrapping the invocation in a button

<$button>
<$list filter="[{StaffListTest.csv}split[<eol>]removesuffix[,]]" variable=input-row>
  <$macrocall $name=each-row row=<<input-row>>/>
</$list>
Create tiddlers from Staff List
</$button>

Any idea how to;

• Use template tiddlers with  ActionCreateTiddlerWidget
• Create but not open new tiddlers with <$action-sendmessage $message="tm-new-tiddler"

Regards

Tony

[Mark S.]

I've been down this road. Consider whether you actually need a template. You could just add whatever additional fields/lists you need from tm-new-tiddler. You could

even iterate over the fields in your template tiddler and add them from there.

[TonyM]

Mark,

I have used the template and saved every tiddler in this case. I had hundreds not thousands and it was quite quick.

Yes I am aware of that work around, but hope my Git Hub Request gets honored before being faced with thousands.

I have however planned to build a way to iterate all the fields in one tiddler and apply them to another existing tiddler, its a way of changing a tiddler from one to another "type", and allows for a few other tricks.

eg task to contact, task to reference etc...

Thanks again