How to save multiple lines into a field ?

[Mark S.]

I just discovered that attempting to store multiple lines into a field (other than the text field) will either

1. Lose the line feeds on standalone TW ... or ..

2. Corrupt the ".tid" file on node.js (becomes obvious when you restart node)

Is there any special technique for saving multiple lines into a regular (not text) field ?

Thanks!

[TW Tones]

Mark,

I have a method to do 1, I need to check if it avoids 2.

However it works well on single file wikis. I will test it on top of a Bob Instance and share back.

Are you editing the tid file outside tiddlywiki?

The main trick is to not use the default field editor. To do this you can use an edit-text widget including in edit mode, and hide the fields visibility in the stqandard field edit.

I will share back soon if this is helpful?

Regards

Tony

[Mark S.]

Hi Tony,

All editing happens in the browser.

I can see what happens inside of node.js. When node parses the tiddler, it perceives the line break as meaning the end of the "fields" section (top section) of the tiddler. It interprets everything else that remains as "text" -- including the title! Then, since it doesn't have a real title, it guesses at the title name. The upshot is that the entire tiddler is corrupted when re-loaded.

In single-file TW, once you save and reload, the field is stripped of the line feeds.

I kind of think that the only solution would be to encode the linefeeds with my own set of symbols.

Thanks!

[TW Tones]

Mark

I will test your assumption and agree or contradict them soon. I have built single file wikis that do not have this problem. In fact I designed a hack once to do something similar intentionaly.

I will test my working solution to see if I can cause your outcomes.

Regards
Tony

[Saq Imtiaz]

Mark: on node.js that sounds like a regression. As of TW 5.1.20, tiddlers with multiline fields are supposed to be saved as JSON.

See:

• https://github.com/Jermolene/TiddlyWiki5/blob/master/core/modules/utils/filesystem.js#L225
  
• https://github.com/Jermolene/TiddlyWiki5/issues/1947#issuecomment-491532661
  

[Saq Imtiaz]

In single file TW, the following will create a multiline field that saves properly.

<$edit-text tag="textarea" tiddler="testing" field="myfield"/>

Note that the resulting field can look like all linebreaks are stripped due to how its rendered. Especially if viewing the field in the info section of the tiddler, as it is rendered in a table cell instead of a paragraph.

Use something like {{testing!!myfield}} instead to confirm linebreaks are properly saved.

Also recall that:

This is
another line
and another one

if entered in the text field of a tiddler will display as all one line.

While

This is

another line

and another one

Will display as 3 lines.

Hope this helps,

Saq

PS: To get linebreaks to always display when viewing a field, use the following CSS:

white-space: pre-wrap;

Note that this is inconsistent with default TW handling where to \n are needed to get a linebreak.

[Mark S.]

On Thursday, July 16, 2020 at 11:57:45 PM UTC-7, Saq Imtiaz wrote:

Note that the resulting field can look like all linebreaks are stripped due to how its rendered. Especially if viewing the field in the info section of the tiddler, as it is rendered in a table cell instead of a paragraph.

Use something like {{testing!!myfield}} instead to confirm linebreaks are properly saved.

Well, that's a relief!

But on node, the generated tiddler file (prerelease 5.1.23) is not being saved as JSON.

On node, create a test tiddler like:

<$button>Do transfer
<$list filter="[[That Tiddler]]" >
<$action-setfield saveme={{!!text}} />
</$list>
</$button>

{{That Tiddler!!text}}

Create a tiddler "That Tiddler" and populate it with multiple lines.

When you run the button on node, the output will be a tid file (see attached screen shot).

Interestingly, if I do the test on stand-alone TW, and then drag the resulting tiddler to TW, the generated file will be a JSON.

This suggests that if you start with a tid file on node.js, and then modify it, it doesn't know to change to a JSON.

Thanks!

[TW Tones]

Or the setfield widget assumes its not a text area but just text.
?

test it without that step.

Regards
Tony

[Mark S.]

Here's a longer version of the test, where you can restore the "saved" text. The multiple lines are restored as expected:

<$button>Do transfer
<$list filter="[[That Tiddler]]" >
<$action-setfield saveme={{!!text}} />
</$list>
</$button>

{{That Tiddler!!text}}

<$button>Transfer back
<$list filter="[[That Tiddler]]" >
<$action-setfield text={{!!saveme}} />
</$list>
</$button>

So, setfield seems to be working OK.

[Saq Imtiaz]

@Mark 

This suggests that if you start with a tid file on node.js, and then modify it, it doesn't know to change to a JSON.

This is a known issue with a PR in progress, where once created the file type for a tiddler does not change:

https://github.com/Jermolene/TiddlyWiki5/pull/4630

Personally I find the JSON saving for multiline fields on node.js lacking and a deal breaker for use of multiline fields, it negates the advantage of having content in a format easily editable in external editors which is why I run node.js in the first place.

[UBi]

I don't know if this works with node.js, but in standalone TWs you can modify $:/core/ui/EditTemplate/fields.

If you add 'tag="textarea"' to the edit-text widget located in the <td class="tc-edit-field-name"> element, multiline edits will work.

[Mark S.]

On Friday, July 17, 2020 at 6:53:58 AM UTC-7, Saq Imtiaz wrote:

@Mark 

Personally I find the JSON saving for multiline fields on node.js lacking and a deal breaker for use of multiline fields, it negates the advantage of having content in a format easily editable in external editors which is why I run node.js in the first place.

Thanks for the info!

What is the occasion or use case for which you need to externally edit tiddlers? 

I'm thinking that maybe what TW should do is encode just a field as JSON.

In my case, I need the multi-lines, or else my node project will be breaking all the time. I'm thinking the simplest solution would be to encode line endings with my own code structure. The only problem being that someone could accidentally use those same symbols in their own text. For instance, if documenting what replacement symbols were used to indicate end of line, that documentation would have the symbols in it.

[Saq Imtiaz]

Hi Mark,

What is the occasion or use case for which you need to externally edit tiddlers? 

What has worked best for me over the years is having my notes/content in flat files that can be edited with any text editor. It is the only approach that I have found that I will persist with. I have well developed workflows using everything from command line tools to OS dependent apps to mobile apps that work with synced flat files.  So for me TiddlyWiki is just another tool that fits into those workflows for interacting with and creating my content.

What actually brought me back to using TiddlyWiki after a gap of several years was the TW5 node.js implementation where each tiddler is saved as a separate file in an easy to read and edit format.

I wanted to use multiline fields both while working on Streams, and for another personal project (notes library for tasting notes), and in both cases I would prefer to have the content in .tid or .txt files on the disk rather than JSON.

This a personal preference though and shouldn't be a consideration for others.

 

I'm thinking that maybe what TW should do is encode just a field as JSON.

I suspect the purists would not like that approach as being adhoc and inconsistent but it would be a step in the right direction for me!

 

In my case, I need the multi-lines, or else my node project will be breaking all the time. I'm thinking the simplest solution would be to encode line endings with my own code structure. The only problem being that someone could accidentally use those same symbols in their own text. For instance, if documenting what replacement symbols were used to indicate end of line, that documentation would have the symbols in it.

You could also merge that patch in a local copy of the repo and run that until it is officially merged. 

https://github.com/Jermolene/TiddlyWiki5/pull/4630

[Mark S.]

You could also merge that patch in a local copy of the repo and run that until it is officially merged. 

https://github.com/Jermolene/TiddlyWiki5/pull/4630

Is there any sane, easy way to navigate in GH and get just the two files?

GH is so opaque. I can see a commit on the files in the branch was made only a few days ago. But when I try to view (and then download) the files, it says the latest commit was 22 days ago. Huh?

So I tried to download the two files. Added them to the tiddlers dir as ext .js. Checking the shadow tiddlers, they seem to be there (the shadow tiddler is overwritten).

But it doesn't seem to change anything -- files with multiple lines added are still created as text files, not JSON.

Thanks!

[Saq Imtiaz]

Mark: did you restart the server after adding the two files, and then also reload the wiki in the browser?

[Mark S.]

Yes, that's SOP. It's unclear to me that the problem addressed in the PR is the same problem I'm experiencing.

Thanks!

[Saq Imtiaz]

The PR addresses an issue where the filepath and extension for a tiddler wont change after its been created.  So it is my suspicion that is why the tiddler doesnt get switched from a .tid file to a .json file when we add a multiline field.

It is possible there has been a regression elsewhere that is causing your issue. I can attempt to take a closer look tomorrow if this is still unresolved at that time.

[Saq Imtiaz]

@Mark I suspect it may be the case that it is the same underlying issue, but the PR only fixes it in the situation where $:/config/FileSystemPaths is being used.

[Saq Imtiaz]

@Mark as a temporary fix try this:

With the patch files in place, create $:/config/SystemFileExtensions

where the text is a filter like

[has[multiline]then[.json]]

Edit: replace multiline with some other criteria to identify tiddlers with mutiline fields

[Mark S.]

Hi Saq,

I don't think it's possible to test for multiple lines in a single filter.

For the purposes of my experiments, I set a field by hand on the tiddlers with multiple lines and then set this per your example in $:/config/SystemFileExtensions:

[has[ismulti]then[.json]]

But that didn't do anything (even after saving, stopping, restarting, and reloading). It continued to produce tid files with the broken format.

Thanks!

[Saq Imtiaz]

Hi Mark,

Firstly apologies for the trial and error approach at debugging yesterday. I was trying to help but didn't have access to my dev machine.

I've done some debugging and have confirmed that as you suspected, there is a bug where only new tiddlers with multiline fields are saved as .json

This bug is the same underlying issue as the PR we have been discussing but that PR only addresses it for situations where $:/config/SystemFileExtensions or $:/config/FileSystemPaths has been set.

https://github.com/Jermolene/TiddlyWiki5/issues/4761

I'll investigate next if there is an easy fix/patch you can run in the interim.

Regards,

Saq

Edit: added link to bug report

[Saq Imtiaz]

Mark,

If you unzip the below file and drop the fileinfo-patch folder into the tiddlers folder and restart your server, multiline fields should correctly trigger saving tiddlers as .json

https://github.com/saqimtiaz/sq-tw/raw/master/fileinfo-patch.zip

The files:

• filesystem.js and filesystemadaptor.js from Joshua's patch
• FileSystemExtensions has the filter: [type[text/vnd.tiddlywiki]then[.tid]] which does nothing mostly, as it enforces the .tid extension for text/vnd.tiddlywiki which is already the default. However this forces fileInfo and the filePath to be reevaluated for each tiddler matching the filter. So adding a multiline field correctly switches the fileformat to .json
  
  
• the fourth tiddler is testmutiline, open it and type a few characters into the textarea. A tiddler called HelloThere2a will be created as .tid. Add a linebreak into the textarea, the tiddler will switch to being saved as .json.

Hope this helps until we get a proper fix.

Cheers,
Saq

[Mark S.]

Thanks very much for that, Saq!

What I don't understand (actually there's a lot I don't understand) is why my filter extension

[has[ismulti]then[.json]]

didn't work (assuming I added the "ismulti" field).

It's counter-intuitive that you have to tell the system to use "tid" all the time in order to force it to use JSON some of the time.

Thanks!

[Saq Imtiaz]

@Mark I didn't test that particular filter as I realized it would cause other issues. The filters in this tiddler only change the file extension, not the format its saved in. So you would get a .json file with plain text body and a .meta file. Not what we want.

The filter I used actually does nothing in this case and therefore doesn't make sense, we aren't using it in the way it is meant. This is just a way to use the fact that any tiddler matching the filter has its info/path recalculated to solve our problem.

I am also unsure if the server needs restarting after the filter is added/changed, and if that might be why your filter was not working yesterday.

I've added some comments on both tickets about these things, so hopefully we will end up with a more robust and consistent solution. 

https://github.com/Jermolene/TiddlyWiki5/issues/4761

https://github.com/Jermolene/TiddlyWiki5/pull/4630

[clutterstack]

Whoop, just ran into this yesterday (and understood it this morning).