Automatically applying tags to imported tiddlers

[Michael Wiktowy]

Greetings,

I am looking for some guidance. I have figured out how to apply tags to tiddlers already imported by dragging and dropping them into a <$droppable> zone. I am not sure how to tie into the Import system to apply these tags while a file is being imported and made into a tiddler.

I am interested in adding a tm-add-tag message into the flow in both a drag-and-drop import into a specific <$droppable> zone as well as an import initiated by the <$browse> widget.

Can someone point me to an example as I don't know how to get the tiddler titles from a resulting import or trigger an action while they are being processed?

/Mike

[TonyM]

Michael,

I have being playing with the import mechanism of late and may be able to help. Can you share your dropzone tagging so I can test some ideas for you, but the way I would like to develop custom drop zones myself.

Regards

Tony

[Michael Wiktowy]

Hi Tony,

The ImportHere tiddler here: http://mwiktowy.tiddlyspot.com/#ImportHere gives an idea of what I would like to do but I am not sure of the sequence of actions that I would need to put in the droppable-import-actions macro or in the messages parameter of the $browse button.

The only thing that it works for is internal resources dragged over from within the wiki (from the Open or Recent tab or a Title list in a tag pill). The idea is to put one of these mechanisms into tiddlers (that might be buried in tabs) that would allow me to attach notes and resources to it while I import them. A $dropzone nested in a $droppable widget does not do it ... although I am not sure that the tiddler-specific $dropzone is even working due to the global $dropzone covering the entire wiki.

Any pointer in the right direction would be appreciated.

Thanks,

/Mike

[TW Tones]

Mike,

That functionality it something I have wanted my self, tagging or other actions on internal tiddlers, so thanks.

• I notice when importing external items that the standard import process takes over. 
• Perhaps there is a way to toggle this behaviour off/on so yours gets honoured?
• I am working on improving this standard import with addons

• A view template that appears on field:plugin-type[import] can be used to work with the standard import, which contains a json of the import tiddlers

Regards

Tony

[Michael Wiktowy]

Thanks for the pointer ... I wasn't aware of how to identify the tiddlers being imported ... listing the field:plugin-type[import] and acting on them with a tm-tag-add message might be a feasible thread to pull on to do what I want in at least a semi-automatic way.

Will have to do a deep dive into $:/core/ui/ViewTemplate/import and $:/core/ui/ImportListing to see if I can decipher it all. Hooking into the import functionality of the $browse button macro contained in a particular tiddler is my priority. Getting drag and drop working on a tiddler is secondary.

Thanks again,

/Mike

[Mark S.]

Adding a field and value to incoming imports is pretty easy (if you want to hack the core). Hacking the core has downsides, especially in terms of upgrades and maintenance.

Look for NavigatorWidget.prototype.handleImportTiddlersEvent   in  $:/core/modules/widgets/navigator.js .

In between these lines:

incomingTiddlers.push(title);
importData.tiddlers[title] = tiddlerFields;

You can put something like:

tiddlerFields.myimport = "imported" ;

(Be sure to make a backup if you try this on anything valuable).

I think you can get a local variable by using this.getVariable(variablename), allowing you to wrap a browse widget with the name of the value you want to be inserted. But I didn't test this.

Have fun.

[Michael Wiktowy]

Hi Mark,

Hacking js is outside my expertise and comfort-zone. Like you said, future maintenance would be a challenge. I wanted to stretch vanilla tiddlywiki to do this but it may not be possible without a RFE. I will see what I can do with the "message" parameter of the $browse macro but maybe that is just a hook to trigger some custom js plugin and I would be back outside my wheel-house/comfort-zone again. :]

Thanks for your help,

/Mike

[Saq Imtiaz]

If you're willing to write JavaScript, a far cleaner way is to use the th-importing-tiddlers hook

https://github.com/Jermolene/TiddlyWiki5/blob/d832cb93ce48f3cfaefb18809930e2448917bc93/core/modules/widgets/navigator.js#L559

[Joshua Fontany]

I was able to hack together a custom ViewTemplate section for the $:/Import tiddler (or any tiddler with `plugin-type[import]`) that will do this, but it requires my JsonMangler plugin to be installed. The attached tiddler adds a tag-pill with "DROP TAG-TITLE HERE" on it to the $:/Import tiddler. The tag-pill is wrapped in a $droppable widget that invokes the actions to add (or remove with the SHIFT key held down) the title of any tiddler-link dragged onto it to ALL tiddlers waiting to be imported. *phew*

The roadblock to doing this in vanilla TW is that you cannot address nested JSON (which is what a plugin/import tiddler is). So you have to install my JsonMangler plugin from here:

https://joshuafontany.github.io/TW5-JsonMangler/

And then drag and drop the attached json file with the ViewTemplate into your wiki. Import a bunch of tiddlers (for example, the attached "tiddlers.json" which is a few document tiddlers from Tiddlywiki.com). Set the import options to view the Fields. Drag a tiddler-link (from the Open tab, or anywhere) onto the tag-pill.

All awaiting import tiddlers will have the tag added. Shift+Drag a tiddler-link (from the Open tab, or anywhere) onto the tag-pill. That title will be removed from the tags in the subtiddlers. I would have to work on the logic to selectively apply or remove the tag from only the Checked subtiddlers, but that would be a bit more complicated.

Best,

Joshua Fontany

[TW Tones]

Joshua

Lovely work like usual. I knew the json mangler plugin would start to pay off with particular applications like this.

I would be keen to apply filters and actions to the json tiddlers during import to add or remove title prefix/suffix or field value pairs.

This would allow importing tiddlers that would overwrite existing ones under a different namespace and keeping them there or allowing a difference check and commit process. This could help with multiple contributors submissions such as via the comments plugin.

I expect even just a subset of json mangler features to handle import and json files would go a long way.

Thanks
Tony

[TW Tones]

Folks,

I just revisited this thread with a clear mind, 

This is a technical musing which may provide some interesting options.

I realised there are two kinds of manipulations we need to do on incoming tiddlers. One is pre-import, acting on them before import, or at least before unpacking the import JSON and those post import.

• The first pre-import needs interrogation of the JSON file created with the import process; 
• The Second post import needs to act on the tiddlers imported.

So we need json handling, eg Joshua's tools for the first, and with the second we can use any tool for post import processing as long as we have access to the list of imported tiddlers.

It seems to me if we can enhance the import process to generate a list of imported tiddlers, that can be used programaticaly, and use this list for any post processing that will be handled.

In the pre-processing step it seems we need to be able apply these minimum functions

1. Import all tiddlers with a prefix or suffix added, so they do not overwrite and existing tiddlers, or can be bundled for later processing.
2. Be able to apply a filter to the incoming tiddlers and chose which in the import list is to really be imported, perhaps even by comparison to existing tiddlers.

• However if the 1st step is available we can do the 2nd in a post processing with a prefix awareness.

In fact one way of doing the equivalent to item 1 above is to turn the import tiddlers into a plugin (rename plugin-type to plugin) , then the imported tiddlers become shadow tiddlers. But shadow tiddlers are never the less tiddlers and may override and existing tiddler when accessing the imported tiddler name.

Another way is to save the import to ar JSON tiddler (Which it is) and process that later.

Regards

Tony

[TW Tones]

Further discovery,

If you rename $:/import to anyname and change plugin-type to plugin, then later you can import the contents (to regular tiddlers) with 

<$button>
<$action-sendmessage $message="tm-perform-import" $param="anyname"/>
Import from Plugin
</$button>

Regards

Tony

[TW Tones]

/Mike,

The variable tv-enable-drag-and-drop is set using the tiddler $:/config/DragAndDrop/Enable which does not exist unless created, setting its text field to "no" will stop the standard drop zone being active thus stop interfering with you own.

In $:/core/ui/PageMacros you can see how the standard dropzone wraps all items within the page template items tagged $:/tags/PageTemplate so any thing in the the story and further will trap the drop. 

I can now only drop in your defined area but it is not applying the tag.

Regards

Tony

On Thursday, June 11, 2020 at 5:08:37 AM UTC+10, Michael Wiktowy wrote:

[Michael Wiktowy]

Hi Tony, et al.,

I have come to similar conclusions. While JSON manipulation looks to be a powerful tool (once I figure out how to use it), but I have been continuing to try to make things work with vanilla TW and finally come up with a solution for the $browse macro at least. Having a list of imported tiddlers after import and having some dedicated post import action trigger would be incredibly useful. But I managed to fake it and force the right order of processing.

I reused parts of the import template from within the tiddler and triggered the tm-perform-import from within the $button body (as you indicated) and then scrape the resulting list of imported tiddlers that pops up in $:/Import and add tags to the list in the "actions" parameter that is triggered after the actions in the body.

To that end, cut and pasted the Import tempate into my local tiddler and added a button:

<$button actions=<<tag-import-actions>>>Import and TagHere
<$action-sendmessage $message="tm-perform-import" $param="$:/Import"/>
</$button>

where tag-import-actions is:

 

\define tag-import-actions()
 
<$set name="titleprefix" value="[[">
<$set name="titlesuffix" value="]]">
<$list filter="[[$:/Import]get[text]split[# ]removeprefix<titleprefix>removesuffix<titlesuffix>]" variable=imported>
<$fieldmangler tiddler=<<imported>>>
<$action-sendmessage $message="tm-add-tag" $param=<<currentTiddler>>/>
</$fieldmangler>
</$list>
</$set>
</$set>
\end

Lessons learned/remaining issues:

1) If I had two $list widgets in the actions, the second one would not process a field set by the first one. I initially tried to scrape the list directly into a Title List that I stuck into a field that I would later access with a list operator but second $list widget filter would fire first. Only having one $list widget with all the removeprefix/suffix mess worked.

2) removeprefix cares about leading spaces and split does not remove them unless told to :]

3) Whether to use a "$param" vs a "param" is annoyingly unclear ... $field=name sometimes means $<name of field>=<value to assign to field> and sometimes $field=<name of field> and some subsequent $value=<value to assign to field> and othertimes <name of field>=<value to assign to field> is sufficient. Not sure how many times I got caught up in that subtle syntax change. Maybe there is a hard and fast rule for when to use which syntax but I ended up with many head-slapping moments.

4) <$set name="tv-auto-open-on-import" value="no"> seems to get ignored by <$browse> to suppress the $:/Import tiddler from popping in to the Story list. Either I am misinterpretting what that is supposed to do, mis-applying it or there is a bug.

5) While the intent of the $list widget is to be able to process multiple files, there seems to be something wrong where it only processes and tags one of the imported tiddlers. {{{ [[$:/Import]get[text]split[# ]rest[1]]}}} seems to list both. Still scratching my head on that one so for the time being $browse does not have "multple" in it.

I think using your $:/config/DragAndDrop/Enable might fix that easily but I think what I have is in a usable state at http://mwiktowy.tiddlyspot.com/#ImportHere

Thanks for everyones help!

/Mike

[Michael Wiktowy]

An update:

I have tagging on import working for multiple files now also.

For the record, I replaces the split and removeprefix/suffix with some splitregexp 's

\define tag-import-actions()
 
<$set name="preregfilter" value="# \[\[">
<$set name="sufregfilter" value="\]\]">
<$list filter="[[$:/Import]get[text]splitregexp<preregfilter>splitregexp<sufregfilter>each[]]" variable=imported>

<$fieldmangler tiddler=<<imported>>>
<$action-sendmessage $message="tm-add-tag" $param=<<currentTiddler>>/>
</$fieldmangler>
</$list>
</$set>
</$set>
\end

There is a lot of extra cruft that get generated by the splitregexp but the each[] filters out everything that is not a valid tiddler title and should keep things from being too brittle with $:/Import format changes.

I will get to drag and drop of external resources and maybe integrate in canonical_URIs in the future but right now, this fits 99.99999% of my needed use-case.

I am still wondering about proper use of the "tv-auto-open-on-import" variable. What Tony wrote about "tv-enable-drag-and-drop" makes me think there is a tiddler that I should be globally poking that value into rather than it being a locally controlled thing.

/Mike

[Saq Imtiaz]

@Michael  some thoughts below:

 

I am still wondering about proper use of the "tv-auto-open-on-import" variable. What Tony wrote about "tv-enable-drag-and-drop" makes me think there is a tiddler that I should be globally poking that value into rather than it being a locally controlled thing.

So the issue is that when you set tv-enable-drag-and-drop, it is only set in that widget and its children.

The import message is handled by the navigator widget which is an ancestor of your widget.

I use something like this:

A tiddler called $:/sq/macros/tv-auto-open-on-import with the tag $:/tagsMacro and the content:

\define tv-auto-open-on-import()
{{$:/config/tv-auto-open-on-import}}
\end

And a second tiddler called $:/config/tv-auto-open-on-import, in which you can set the text to no

You will also need to set it to yes, or delete this tiddler, after your post-import handling is completed.

Regarding actions, it is best to think of all actions executed by a button as being invoked in parallel, that is you cannot assume one executes before the others. So if you set a field in one action, you cannot read that value in the next. (The underlying issue has to do with refresh and how actions are parsed and invoked)

I run a couple of small tweaks of the import related code in Navigator that I think might be worth a core pull request based on this discussion:

- a post import hook, invoked after a tiddler is imported.

- the names of imported tiddlers are saved to a field in the $:/Import tiddler as a title list, making it easy to re-use via wikitext.

Cheers,
Saq

[Michael Wiktowy]

Thank you Saq.

I had just discovered how to do it globally by looking at upgrade.html but was still banging my head as to how to do it locally. Your solution of transcluding into the macro/variable an editable tiddler would work great.

I agree wrt your recommendations. It would make things much cleaner and easier to have the import Title List available and have some way of triggering actions but I managed to kludge something together for my needs with everyone's help.

Much appreciation.

/Mike

[Saq Imtiaz]

Michael: something like this should also work for parsing $:/Import

<$list filter="[enlist{$:/Import}butfirst[6]]" variable="imported">
 <!--actions go here-->
</$list>

[TW Tones]

/Mike,

This sent me off on my own import tools development, Still a way to go. 

I attached some incomplete research notes into the import mechanism for others wishing to further this.

I do now think we can save any import as a JSON and we can add all the bells and whistles we want.

Regards

Tony.

[Michael Wiktowy]

On Wednesday, June 17, 2020 at 5:00:58 PM UTC-3, Saq Imtiaz wrote:

So the issue is that when you set tv-enable-drag-and-drop, it is only set in that widget and its children.

The import message is handled by the navigator widget which is an ancestor of your widget.

I use something like this:

A tiddler called $:/sq/macros/tv-auto-open-on-import with the tag $:/tagsMacro and the content:

\define tv-auto-open-on-import()
{{$:/config/tv-auto-open-on-import}}
\end

I tried doing this without success. Not sure what I am doing wrong but it seemed to ignore the transcluded "no" setting and opened $:/Import anyways. I trying something similar by not using the transclusion and adding and removing the $:/tags/Macro. But it seems that macros are a bit sticky and once enabled, it stays that way until reload.

Your idea of using enlist is more elegant since it is list delimiter and spacing agnostic. I initially tried to use enlist but couldn't get it to work since I was using [] rather than {} ... it is the subtle things that get you ... I will likely forget about that the next time that I want to use enlist :]

However, I will end up blending our solutions to form:

<$list filter="[enlist{$:/Import}each[]]" variable=imported>
...
</$list>

I used to use "rest" (aka "bf" aka "butfirst") however switched to "each" since it will cull any number of preamble words/lines that appear in $:/Import in the future and not just a fixed quantity and will test the existence of the tiddlers before tagging them in case something goes sideways in the import.

/Mike

[Saq Imtiaz]

@Michael

I tried doing this without success. Not sure what I am doing wrong but it seemed to ignore the transcluded "no" setting and opened $:/Import anyways. I trying something similar by not using the transclusion and adding and removing the $:/tags/Macro. But it seems that macros are a bit sticky and once enabled, it stays that way until reload.

I suspect that what you are running into is a variation of the problem you encountered earlier, that the results of an action are not picked up by subsequent actions. Are you setting the transclusion and triggering the import in the same button?

 

Does adding the tag work, in that the new value is picked up in time? If so, perhaps in addition to removing the tag, you tag another tiddler that sets the value to yes.

Note that you can also use action-listops instead of fieldmangler and tm-add-tag:

<$action-listops $tiddler=<<imported>> $tags="mytag" />

[Saq Imtiaz]

@Michael I am working on a few import related things myself this morning and realized we were grossly over-complicating parsing the $:/Import tiddler

[[$:/Import]links[]]

Did the technique to flag another $:/tags/Macro tiddler to reset tv-auto-open-on-import workout for you?

[TW Tones]

Saq

I must try and publish something on import tools I am developing as well for a little more colaboration.

I wish we had a more robust but uncomplicated way to collaborate on things inside a wiki.

I am sure a check in out process for serial editing a single wiki would be enough. Perhaps with a user aware streams.

Regards
Tony

[Michael Wiktowy]

Yes. That links filter seems much more rational. There are quite a few filter operators and every time I have to write a filter I have to relearn them:)

Setting the macro tag worked better for me in that it did suppress the $/:Import but the problem is when the tag is removed, the macro stays active for some time ... The only reliable way of resetting things seems to be to save and reload. There must be some caching happening. It doesn't seem browser dependent since it does the same thing in Chrome and Firefox (at least in Linux).