How to batch import journal entries?

[Radish Records]

I've kept a digital journal on google drive for 2 years or so, and want to move everything onto tiddlywiki. I've done the basic setup, and exporting my google drive files into docx and pdf. 

Whats the fastest way to get everything on? 

[TW Tones]

Maxhi,

I think you need to share more information about the arrangement of your data. 

PDF and docx are less easy to transfer from than other formats, do you use one or more files? or spreadsheets, are these files based on some time period or every day, or by subject. However you organise these is it easy to identify the breaks such as the difference between two days? Can you download all your data in a zip where they can be accessed as files?....

What is the total size of files, does it include images or Audio, are they plain text or something else.

My experience has taught me you start doing it and find ways to make it faster or more automated if possible. But if you don't explain the first step - understand your data we can help much.

Never the less you can use special import tools, import multiple files at once, import JSON, CSV, TEXT and more, and manipulate content after import.

Tones

[Radish Records]

It's two separate zip files one with pdfs and one with docxs, separate files for every entry i wanna put in, naming scheme is like (Month Day - title.pdf) although not everything is formatted identically.

The timestamps for date of modification are correct, so ideally i'd sort using those. 

It's not strictly one per day, sometimes they'll be multiple entries per day.

All plaintext, very few have images and i'm okay losing them in the transfer. 

All in all it's only a few megabytes. 600 unique files. 

[Radish Records]

UPDATE : I've managed to convert everything to txt using pandoc. Now i mostly just need to figure out how to get the date modified to match up with the date in tiddlywiki. Anyone know how? 

[Jon]

Hi,

I did something similar a while ago. I can't just post the exact instructions now, but you should find this conversation useful https://groups.google.com/g/tiddlywiki/c/qbnKoBoGYfw/m/06ZgEuPgBQAJ

Regards

Jon

[TW Tones]

maxhi,

If you have any consistent information to can use this to match data. an example may be to import all the files such as there filename is the tiddler name, then separately use the operating system eg windows Dir command to list the files with created and modified dates along with the filename into a text file. Now import that perhaps as a csv file and use JSON mangler to generate a plugin containing a tiddler (in a plugin) for each file name containing your "original-created and original-modified dates" you can then write a list to look up the dates for each filename and apply it to the tiddler for each text file. We can parse text files as well if necessary.

I strongly urge to to use new fields for dates and retain the integrity of the tiddlywikis created and modified dates. I expect the dates will be in a different format and need conversion anyway, ideally convert them to tiddlywikis date serial number format, even if you have to zero fill the time or milliseconds.

later you can build a method to spilt existing files into each day eg excise 

At every step try to retain and import any information you have, it may easily prove useful later, for example if the files were already in folders try and capture the folder name as well. if you need to reformat retain the old and the new values. If some filenames are duplicates (but not their content) there are tricks you can employ.

Of course you will still have some way to go but do lean on us as much as you need, if you can generate some sample or dummy content to share, we can help you further.

Regards

Tones

[Radish Records]

After configuring pandoc i've managed to get this
https://imgur.com/a/QP0z4xh

All my files, names aren't particularly consistent but they are numbered and the dates are all there. No folders or anything. 

Here's a sample zip that shows the way my files are formatted.
https://drive.google.com/file/d/195fmBEM8X8fTEfKFW9rlMS-kY5eCFU9V/view?usp=sharing.

If i make a new field for dates, will it be compatible with the base system or will i have to fiddle with making fields for every tiddler i make going forward to keep it consistent? 

[TW Tones]

maxhi,

It is approaching midnight here in Australia, So I hope to reply tomorrow if others don't)

• Am I correct in saying every filename is unique? It looks like you have what you need. 
• Can you import the list of filenames and their dates?
• If you use  <$browse multiple> you can import multiple files at once. I recommend batches. Drag and drop multiple files also works

If you make new fields its easy to automate their use or do a bulk conversion (list inside a button wrapping action widgets). Typically however we setup a list widget to review tiddlers or use search and these can be designed to accommodate anything. The best approach is to try and keep it simple to start with, and we can create custom journal buttons etc... for new entries going forward and make them comply with what ever fields you have in use. We need not destroy anything just create different views of it.

Regards

Tones

[Radish Records]

Approaching midnight for me too, so i'll leave the answers here since i won't be able to reply for a hilwe

- yes every file name is unique
- i'm not exactly sure how to do that, but i'll try to get it done tommorow morning

Making custom buttons sounds like a good plan! 

[TW Tones]

Morning.

Errata: use this  <$browse multiple/> I forgot the "/"

I would consider as a windows user, using dir > file.txt

then import this to excel and use the text to columns, then save as a csv.

If you are not confident to do this send the text file to me and I can prepare it and return it. Just the filenames list,  not the files.

Regards

Tones

[TW Tones]

Oh,

Another idea. I know you have 600 or so files but sometimes people try and over simplify things. If you were to import 10 at a time and manually select a related date , divide the file and make us of it as an opportunity to review your files and content you may find it is a valuable learning experience. Add to this insight to tagging, fields and field values and searching  the manual approach may be more powerful. Perhaps a week or fortnight every day?

Also to consider tiddly tools https://tiddlytools.com/timer.htmlas a platform on which to build a time and date related solution. 

Regards

Tones

[Radish Records]

I've got the filenames, but i'm not particularly good with excel, could you help? 

https://drive.google.com/file/d/1HTziGPLdvb6paFfrz93ME4-nYxfW2ynq/view?usp=sharing

[TW Tones]

Here is the csv attached

You can drop it on your wiki and import it, then once you have the text files imported (with matching filenames) we can use this to lookup the dates and save it to the matching tiddler.
If fieldnames include special characters, or comas quotes you may need to handle a few manually.

• Unfortunately the size includes commas, but we don't need the size, we can get the first items and the last items (filename)

If something like this works we can use it to update fields in the matching tiddlers. 

<$list filter="[[dir.csv]get[text]splitregexp[\n]]" variable=each-line>

   <$set name="date" value={{{ [<each-line>split[,]first[]] }}}>

   <$set name="time" value={{{ [<each-line>split[,]nth[2]] }}}>

   <$set name="am-pm" value={{{ [<each-line>split[,]nth[3]] }}}>

   <$set name="filename" value={{{ [<each-line>split[,]last[3]first[]] }}}>

         <<filename>>,<<date>>,<<time>>,<<am-pm>> <br>

   </$set></$set></$set></$set>

</$list>

Regards

]Tones

[Radish Records]

i've imported a chunk of my stuff to test but i'm not quite sure how to run the code? Sorry, i've never used tiddlywiki before this and all the documentation seems really dense. 

[TW Tones]

maxhi,

if you imported the file dir.csv then put the code I gave you in another tiddler. And you could see how it lists the details for each text file.

If you have imported the files with the same name already you are almost done.

Let me know where you are, but I clearly assumed a little more experience with TiddlyWiki.

Regards

Tones

[Radish Records]

https://imgur.com/a/Hnrrk4L

Oh i see, my content type was set to plaintext. It does indeed work, with a few hiccups but i can do manual correction on those. 

[TW Tones]

Good to here,

Once you develop a set of action widgets, to use the variables in my example to set fields in the matching field tiddlers, you can then wrap a list in a button to trigger it. click and the various actions are executed for each tiddler.

If you separate the date YYYY MM and DD hh and mm (add 12 to hh if PM) you can make a new date time stamp and apply that to your custom field.

• Look for the date picker if you want to interactively set a date
• remember excise allows to to split one tiddler into two
• use the viewWidget to display a tiddlywiki serial date in any date format with the template paramter

You will get there.

Feel free to ask more questions.

Tones