[TW5] Is there a way to import multiple tiddlers from a single file?

[Arkady Grudzinsky]

I have a text file where each line is a list item.  I would like to import this file in my TW5 and create a separate tiddler for each line.  I also would like to specify tags and some fields for each tiddler.

.tid files do not allow multiple tiddlers in one file.

I thought, HTML would work with something like this
<div title="Tiddler 1" tags="task" start="201311080800" priority="5">Tiddler 1 text</div>
<div title="Tiddler 2" tags="task" start="201311080800" priority="6">Tiddler 2 text</div>
etc.

But it did not work.  I get one tiddler with type="text/html" with file name as a title and this HTML code inside.

I could, of course, create a Perl or an awk script to produce one .tid file per line, but it seems like too much effort.

Any suggestions?

[Arkady Grudzinsky]

I did crate a Perl script and generated 322 .tid files - one for each tiddler.  But, to my disappointment, I cannot select multiple files from the file selection interface.

[Arkady Grudzinsky]

Found a workaround.  Copied all tiddlers in "editions/mytw5/tiddlers" directory, copied tiddlywiki.info from editions/tw5.com and ran bld.sh after adding to it a section to create "mytw5.html" file.  Whew...  Learned something in the process.  But still, would like to see an easier way.

[Jeremy Ruston]

Hi Arkady

The "browse" button in TiddlyWiki5 should let you select multiple files - what browser are you using?

The easiest way to import a bunch of tiddlers at once is to use a .json file containing an array of tiddler field objects. For example:

[

{"title":"Tiddler1","text":"Text of tiddler},

{"title":"Tiddler2","text":"Text of tiddler}

]

You can import a JSON file with drag and drop in the browser, or via the --load command under node.js.

Can Perl conveniently create JSON files?

Best wishes

Jeremy.

On Sun, Nov 10, 2013 at 7:53 AM, Arkady Grudzinsky <agrud...@gmail.com> wrote:

Found a workaround.  Copied all tiddlers in "editions/mytw5/tiddlers" directory, copied tiddlywiki.info from editions/tw5.com and ran bld.sh after adding to it a section to create "mytw5.html" file.  Whew...  Learned something in the process.  But still, would like to see an easier way.

--
You received this message because you are subscribed to the Google Groups "TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tiddlywiki+...@googlegroups.com.
To post to this group, send email to tiddl...@googlegroups.com.
Visit this group at http://groups.google.com/group/tiddlywiki.
For more options, visit https://groups.google.com/groups/opt_out.

--
Jeremy Ruston
mailto:jeremy...@gmail.com

[Arkady Grudzinsky]

On Sunday, November 10, 2013 09:02, Jeremy Ruston wrote
regarding Re: [tw] Re: [TW5] Is there a way to import multiple tiddlers from a single file?:

> The "browse" button in TiddlyWiki5 should let you select multiple files - what
> browser are you using?

Firefox 24.0 for Linux. Just double-checked. I cannot select multiple
files using the Browse button, for some reason. It could be some
parameter of the button which has different default value in different
browsers.

> The easiest way to import a bunch of tiddlers at once is to use a .json file
> containing an array of tiddler field objects. For example:
>
> [
> {"title":"Tiddler1","text":"Text of tiddler},
> {"title":"Tiddler2","text":"Text of tiddler}
> ]

This worked. (you missed closing quotation marks). I thought, there
should be a solution like that, but have not come across anything like
that.

Thanks a lot.

> You can import a JSON file with drag and drop in the browser, or via the --load
> command under node.js.
>
> Can Perl conveniently create JSON files?

Yes. I'm very familiar with Perl. I often use it at work. I think, it
was originally created to process text files. Most likely, there is a
Perl module to handle JSON files, but the syntax is simple enough to
create them manually. The JSON file you describe seems to be very easy
to create from a CSV file using any text editor. My favorite Vim would
work just fine.

--
Arkady

[Jeremy Ruston]

Hi Arkady

On Sun, Nov 10, 2013 at 10:32 PM, Arkady Grudzinsky <agrud...@gmail.com> wrote:

Firefox 24.0 for Linux.  Just double-checked.  I cannot select multiple
files using the Browse button, for some reason.  It could be some
parameter of the button which has different default value in different
browsers.

I've just tested Firefox 25 for OS X, and it does the multiple selection properly. Maybe it's a Linux thing?

 

> Can Perl conveniently create JSON files?

Yes.  I'm very familiar with Perl.  I often use it at work.  I think, it
was originally created to process text files.  Most likely, there is a
Perl module to handle JSON files, but the syntax is simple enough to
create them manually.  The JSON file you describe seems to be very easy
to create from a CSV file using any text editor.  My favorite Vim would
work just fine.

OK, do be aware that browsers are generally extremely strict about JSON syntax. Watch out for the string escaping rules:

http://www.json.org

Good luck!

Best wishes

Jeremy

 

--
Arkady

--
You received this message because you are subscribed to the Google Groups "TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tiddlywiki+...@googlegroups.com.
To post to this group, send email to tiddl...@googlegroups.com.
Visit this group at http://groups.google.com/group/tiddlywiki.
For more options, visit https://groups.google.com/groups/opt_out.

[Arkady Grudzinsky]

On Sunday, November 10, 2013 22:54, Jeremy Ruston wrote

regarding Re: [tw] Re: [TW5] Is there a way to import multiple tiddlers from a single file?:
>

> I've just tested Firefox 25 for OS X, and it does the multiple selection
> properly. Maybe it's a Linux thing?

It could be. I'll keep an eye on it. I know, I can upload multiple
sites for some sites (Google Drive, for example).

> OK, do be aware that browsers are generally extremely strict about JSON syntax.
> Watch out for the string escaping rules:
>
> http://www.json.org

Thanks for the reference. JSON seems like a good format to learn. It
seems much leaner and readable than XML. The syntax is very close to
Perl where [] denotes array references and {} denotes hash references
or objects with "string - value" pairs.

For my application (task management) I would not need to enter complex
tiddler contents - just the title and a few tags and fields to import
tasks from spreadsheets. Once again, TW5 seems to fit my needs very
well.

Thanks.

--
Arkady

[Arkady Grudzinsky]

On Sunday, November 10, 2013 09:02, Jeremy Ruston wrote

regarding Re: [tw] Re: [TW5] Is there a way to import multiple tiddlers from a single file?:
>

> You can import a JSON file with drag and drop in the browser, or via the --load
> command under node.js.

Jeremy,

Sorry, I'm not very familiar with node.js. Can you give me an example
of a command for how to add a JSON file named tiddlers.json into a TW5
file named mytw5.html? Is it just

node mytw5.html --load tiddlers.json text/json ?

Can I pipe STDOUT to node instead of importing a file?
E.g.

echo "<json code>" | node mytw5.html --load - text/json

or something like that?

This would be very useful. Most of my tasks come from emails and if I can
write a script to add tiddlers to TW5 from command line, I can create a
macro in my mutt mail client to create tasks from emails. This would be
great because integration with email client is #1 feature I look for in
a task management system. I did something similar when I used "task"
project management application (terminal based) and it worked well.

Thanks.

--
Arkady

[Jeremy Ruston]

Hi Arkady

Sorry, I'm not very familiar with node.js.  Can you give me an example
of a command for how to add a JSON file named tiddlers.json into a TW5
file named mytw5.html?  Is it just

node mytw5.html --load tiddlers.json text/json ?

Close, but on the server TW5 needs a WikiFolder to store it's tiddlers, and cannot use a standalone HTML file as it's working store. So you'd need to do something like this:

node ./tiddlywiki.js \

./path/to/my/wiki/folder \

--verbose \

--load ./path/to/my/wikifile \

--rendertiddler $:/core/templates/tiddlywiki5.template.html ./test.html text/plain \

|| exit 1

Line 1 invokes TW5, line 2 points to your wiki folder (take a copy of editions/empty to get started). Line 4 loads the (non-system) tiddlers from the specified TiddlyWiki file, and line 5 saves the result as a standalone wiki.

If you're working this way you can make your scripts deposit tiddlers as .tid files in the tiddlers folder of your wiki folder. Then they will be automatically loaded when the wiki starts up.

But you might be better off running a local server under node.js; see ./serve.sh. If you're running like that then your scripts should use CURL to save tiddlers directly to the server.

Can I pipe STDOUT to node instead of importing a file?
E.g.

echo "<json code>" | node mytw5.html --load - text/json

or something like that?

No, at the moment all the command line stuff in TW5 is file-based, not stream based.

 

This would be very useful.  Most of my tasks come from emails and if I can
write a script to add tiddlers to TW5 from command line, I can create a
macro in my mutt mail client to create tasks from emails.  This would be
great because integration with email client is #1 feature I look for in
a task management system.  I did something similar when I used "task"
project management application (terminal based) and it worked well.

Yes, that would be great, and I think perfectly possible as things stand,

Best wishes

Jeremy

 

Thanks.

--
Arkady

--
You received this message because you are subscribed to the Google Groups "TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tiddlywiki+...@googlegroups.com.
To post to this group, send email to tiddl...@googlegroups.com.
Visit this group at http://groups.google.com/group/tiddlywiki.
For more options, visit https://groups.google.com/groups/opt_out.

[Neil Griffin]

Just resurrecting this old thread...

I am trying to do something very similar to Arkady: in my case convert a list of items from an excel file to one tiddler for each row in the list (ultimately with the aim of making a searchable database).  I can write a python script to make a suitably formatted file to import into TW, but while JSON works fine with standard fields, it doesn't work with my non-standard fields.  The only other format described in http://tiddlywiki.com/#TiddlerFiles that appears to support multiple tiddlers in a single file is a 'TiddlyWiki HTML file'.  Does this mean an entire TW file, or is there some reduced format with not much more than a collection of <div>'s that TW will recognise it as a collection of tiddlers to import?

Thanks,

Neil.

[Jeremy Ruston]

Hi Neil

I am trying to do something very similar to Arkady: in my case convert a list of items from an excel file to one tiddler for each row in the list (ultimately with the aim of making a searchable database).  I can write a python script to make a suitably formatted file to import into TW, but while JSON works fine with standard fields, it doesn't work with my non-standard fields.  The only other format described in http://tiddlywiki.com/#TiddlerFiles that appears to support multiple tiddlers in a single file is a 'TiddlyWiki HTML file'.  Does this mean an entire TW file, or is there some reduced format with not much more than a collection of <div>'s that TW will recognise it as a collection of tiddlers to import?

The minimal requirement for an HTML file to be recognised as an unencrypted TW file is that it has a <div id="storeArea"> immediately preceding the tiddler DIVs, which must be followed by a </div>. For example:

<div id="storeArea">

<div created="20130302085406905" modified="20130302084548184" tags="Examples" title="A tiddler title">

<pre>HTML encoded text of tiddler

</pre>

</div>

<div created="20140315085406905" modified="20140321084548184" tags="One Two [[Three with Space]]" title="Another title" customfield="field value">

<pre>Text of this tiddler

</pre>

</div>

I've updated the docs to include this information:

https://github.com/Jermolene/TiddlyWiki5/commit/9a36dc61ec3ed48e069e55b5dced53846a841521

Best wishes

Jeremy.

 

For more options, visit https://groups.google.com/d/optout.

[Neil Griffin]

Thanks.  That did the trick (once I added type="text/vnd.tiddlywiki" to prevent the tiddlers being mistaken for TW classic format).

N.

[stevesuny]

Resurrecting an old thread, yet again...I have the same challenge (a spreadsheet file where each line becomes a tiddler, with separate column for title, text, tags, etc.. I need to help students import a large number of tiddlers (5 to 200 or more) that they create through spreadsheet manipulation. In TWClassic on tiddlyspace, we use the Tiddlytools plugin http://tiddlytools.com/#CreateTiddlersFromCSV.

As I contemplate moving classes from TWClassic to TW5, I need to replicate that functionality, for non-technical users. The responses above reference scripts etc. that are beyond my expertise, and certainly that of my students.

How do to this in TW5. Maybe I just don't know how to run the load-file command?

Thanks!

//steve.

[Måns]

+1

My workflow has been to import csv file to TWc via Eric's CreateTiddlersFromCsv and then drop the TWc on a TW5 to import all tiddlers at once.

The tedious part is then to click/accept to convert every imported tiddler from TW classic format to a TW5 format.

It is tedious when you have 300+ tiddlers to convert. I guess it should be ok with a few tiddlers...

There is an online csv to JSON converter here: http://www.convertcsv.com/csv-to-json.htm I haven't tried it out yet...

If you drag a correctly formated JSON file on to a TW5 it should generate tiddlers.... - again I haven't tried it out.

Good luck - and please tell how it goes.

Cheers Måns Mårtensson

[stevesuny]

Of course, writing out the problems leads to its solution (at least partially).

I've discovered that, as promised above, a properly formatted json file, as:

[
    {"title":"Tiddler1","text":"This is the text of tiddler 1","tags":"Tag1 Tag2 [[Tag 3]]"},
    {"title":"Tiddler2","text":"This is the text of tiddler 2, with a link to [[Tiddler1]]","tags":"Tag1 Tag4"}
]

saved as, for example, tiddlers.json

can then be imported using the "native" import in TW5, and then click on import, and all tiddlers get created.


I'm now working on generating json files from spreadsheet. Here is a link for a script for google sheets:

http://blog.pamelafox.org/2013/06/exporting-google-spreadsheet-as-json.html

It looks reasonable, and I'll work on it today.

As always, once discovered, it seems straightforward. Thanks, Jeremy, for the file structure that you put in a few years ago which got me headed in the right direction.

//steve.

[Jeremy Ruston]

Hi Steve

You'll probably have seen the current built-in support for CSV files in TW5: dragging or importing a file results in a single tiddler containing the entire CSV file. There's a special parser for text/csv tiddlers so that the content gets displayed as a nicely formatted HTML table.

The approach I favour for making the CSV importing more flexible is actually to add facilities for accessing and manipulating the content of CSV tiddlers, rather than extending the import process itself. I like the idea of maintaining CSV as a native format, not just an import format - we could integrate a JavaScript spreadsheet engine like SocialCalc (see http://tiddlywiki.com/rboue's%20plugins%20for%20TiddlyWiki)/

Adding basic support for referencing rows and cells in CSV tiddlers would be fairly straightforward - it would largely mirror the existing implementation for access to property indexes within data tiddlers (aka JSON tiddlers). I'd be happy to guide anyone interested in working on a pull request.

Best wishes

Jeremy.

--

You received this message because you are subscribed to the Google Groups "TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tiddlywiki+...@googlegroups.com.
To post to this group, send email to tiddl...@googlegroups.com.
Visit this group at http://groups.google.com/group/tiddlywiki.

To view this discussion on the web visit https://groups.google.com/d/msgid/tiddlywiki/54ca5c29-ef5f-40b7-91f2-e749e6bf9078%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[stevesuny]

Reviving an old thread. Am again interested in importing CSV as separate tiddlers.

I use google sheets to generate CSV.

I'd like to use http://www.convertcsv.com/csv-to-json.htm to create json, but none of the output formats work - when I use the import, the json file doesn't return any tiddlers available for import.

Anyone have suggestions or experience with this? Or, alternatives?

Thanks,

//steve.

[Jeremy Ruston]

Hi Steve

Funnily enough, I've been working with some CSV files in my dayjob. I've got a very noddy little Node.js script that converts CSV files into JSON files. It's not very polished - eg one has to edit the JS source to tell it which columns to use.

But, if you're comfortable with Node.js it might make a good starting point. If you're interested I can post it to GitHub.

Best wishes

Jeremy.

[stevesuny]

Thanks, Jermey. I'm not quite to node.js yet. I installed but don't really get it. I don't do much at command line.

As for CSV, the tool I'm trying to use (http://www.convertcsv.com/csv-to-json.htm) offers these options:

Use this tool to convert CSV data to JSON. There are 4 types of JSON conversions:

• CSV to JSON - array of JSON structures matching your CSV
• CSV to Keyed JSON - Structure of JSON with the specified key field as the key value to a structure of the remaining fields
• CSV to JSON Array - An array of CSV values where the CSV values are in an array, or a structure with column names and data as an array
• CSV to JSON Column Array - An array of CSV values where each column of values are in an array

Which of these would be best to use to import JSON file?

Thanks!

//steve.

[Andrew Sanchez]

Dear Jeremy,

I'd be interested in playing around with the Node.js script.  Sounds useful!

Thanks,

Andrew

[Jeremy Ruston]

Hi Andrew

I'd be interested in playing around with the Node.js script.  Sounds useful!

I've put the source on GitHub:

https://github.com/Jermolene/csv2tiddlers

It's not for the faint of heart; you'll need to hack the source code to customise it for your own CSV files.

Let me know how you get on,

Best wishes

Jeremy.

[Marc J. Cawood]

Currently TW5 does not seem to import CSVs as indicated in this thread:

There's a special parser for text/csv tiddlers so that the content gets displayed as a nicely formatted HTML table.

When I import a csv I just get a text/plain tiddler.

Is the parser part of the core?

test.csv:

"Name","Length"
"dir.op.json","118480"
"file.list.cmdlet","294"

[Jeremy Ruston]

Hi Marc

When I import a csv I just get a text/plain tiddler.

Is the parser part of the core?

Try setting the type of the tiddler to "text/csv”. Does you CSV file have a ‘.csv’ extension?

Best wishes

Jeremy

test.csv:

"Name","Length"
"dir.op.json","118480"
"file.list.cmdlet","294"

--
You received this message because you are subscribed to the Google Groups "TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tiddlywiki+...@googlegroups.com.
To post to this group, send email to tiddl...@googlegroups.com.

Visit this group at https://groups.google.com/group/tiddlywiki.
To view this discussion on the web visit https://groups.google.com/d/msgid/tiddlywiki/26b00b42-145c-459e-b4d0-89a0f219286f%40googlegroups.com.