Presenting: Batch Manipulator - mass tagging

[Mat]

Yeah, the TWaddler gives thee

Batch Manipulator

~ bigly manipulate the masses ~

a tool to simplify tagging/untagging,

setting/deleting fields and/or their (full) content

...en masse as well as single tids

In particular, it simplifies identifying tiddlers that are distantly related so to group (i.e tag) them - see the "green tags" feature.

I've previously made some half measures for this but the tipping point came as I'm currently in the process of converting a big TWC into TW5 (good use case!)

Cred: Some early pieces for this were picked from things by TWizards @Mario and @Jed. 

<:-)

P.S I'm involved in a few other threads here that I have not properly responded to. I hope to do so eventually and I really appreciate your kind replies to my various questions. As someone with a constant tornado of ideas in my head, many of my TW projects float into one another so there are typically several parallel issues. This also means that your kind replies often serve as pieces in several puzzles :-)

[TonyM]

Mat,

• Thanks very much for your work, this will be a valuable tool. It is very powerful and responsive, perhaps this could be an issue.
• Constructive Feedback: It feels "sensitive", with immediate responses we could loose information, I need to use it cautiously.
• However I think it is very elegant and useful with the all drag-able a very useful feature.
  

I empathise with your words "a constant tornado of ideas in my head, many of my TW projects float into one another so there are typically several parallel issues", in fact this challenge is making me look into ways of addressing complex feedback loops, process bootstrapping and iteration in a rapid development process. TiddlyWiki inspires more than new tiddlers.

Thanks very much

On Friday, August 25, 2017 at 10:49:51 PM UTC+10, Mat wrote:

Yeah, the TWaddler gives thee

Batch Manipulator

~ bigly manipulate the masses ~

a tool to simplify tagging/untagging 

[TonyM]

Post Script;

I too have a plan to convert a big TWC into TW5. I Think it may result in multiple TW5 so I am working on interaction between wikis,

A 17mb TWC with my life embedded in it.

Regards

Tony 

[Mat]

Hi Tony and thanks for kind words and feedback!

• Constructive Feedback: It feels "sensitive", with immediate responses we could loose information, I need to use it cautiously.

Anything specific you're thinking of?

One problem is that in a complex wiki with many tags, the generated table can get really big - and it can take a long time to generate, including "page unresponsive" warnings. That could be a problem if one doesn't save often.

A 17mb TWC with my life embedded in it.

17mb... you must be 200 years old ;-)

<:-) 

[TonyM]

Mat,

I have been building a single TWC for more than 10 years. It does much of what I need but is not so "responsive" for mobile use, and yes the firefox timeout is set from 10 up to 80 and occasionally I loose a days work (every 6-12 months), It actually contains around 16,000 tiddlers. Actually I think my biggest issue will be reproducing functionality, replacing plugins and specially designed view tiddlers. In fact it is such a deep rabbit hole, I am avoiding it for now.  A trial attempt to import it all into node.js as an experiment, seems to have ended sadly, now I need to reboot my Synology NAS its running on. The reason its so big is it has included dozens of large projects, dozens of small ones as a Professional IT solutions designer, as a Knowledge and Information Manager, as a sustainability and community activist not to mention creative and entrepreneurial ideas all with Operational, Tactical and Strategic review processes built around it. Fortunately it is well structured and the key information is in references and Projects.

I expect I will handle Contacts, Projects, references and to do lists and similar information sets separately and maintain historical information in TWC. I think the trick is to bundle these different resources into their own new systems that can be moved from one wiki to another, and use a phased transition. I expect I should do something with Node.JS so it is easier to include or exclude tiddlers, and deal with the size.

I have appreciated the bespoke nature of building exactly what I want when I want into a single effective tool, but it appears this means it is only I who can do a migration. I will be posting on one effort of mine to prepare the stage for this migration, to this group for feedback soon. What I call Object tiddlers.

I am working on rapid Tiddlywiki development processes, generic tools and solutions, building a technical understanding for three reasons, 1. To Turbo Charge how I work, 2. provide solutions for clients and 3. move my old tiddlywiki.

I expect I will get to know Batch Manipulator well.

Perhaps we can compare notes on this migration.

Regards
Tony

[Mat]

Tony, yeah, that sounds like it would make for a lot of tiddlers and complexity ;-)

I also procrastinated conversion of my main TWC because if was way deep and complex and with special solutions and plugins.

 

I expect I will get to know Batch Manipulator well.

Just FYI, I'm continuing to dev it. Tables sized "filtered tids times collective sets of tags" means it is very easy to mistakenly render overly big tables that cause time-outs, so I'm making an optional table that is more limited. Also, it should handle any field, not just tags. For example, TWC to TW5 conversion gives a "changecount" field in every converted tid and these should be easy to remove.

<:-)

[Mat]

 Updated  bigly to manipulate even more masses even.

The main news (yes, real news!) is that you can now batch manipulate fields in a similar way as the tags.

Believe me, Batch Manipulator will be great when you want to convert imported TWC tids to TW5 format. But that is just one use case. There are billions and billions more. Believe me.

Note: It is not perfected yet but I will now not do more on it before I stumble on problems with it myself. Unless there are serious bugs of course, in which case I hope you report them here.

So much winning.

<:-)

[@TiddlyTweeter]

Mat

Will it be able to find all instances of "Perry Mason" in Tiddler texts and delete him (for the sake of humanity)?

Josiah

[Mat]

@TiddlyTweeter wrote:

Will it be able to find all instances of "Perry Mason" in Tiddler texts and delete him (for the sake of humanity).

He he! If you mean to delete the full tiddlers then yes, that's doable. Or, if you'd want to clear out the text fields (fully) then that's also doable with a minor tweak. But it is not possible to "cut out" only segments of a field content. If it is a one-time job you have in mind, then the easiest is probably in a real text editor.

<:-)

[@TiddlyTweeter]

Allora, Mat, thanks for the clarification. It looks like I will have to incite EXTERNAL FORCES to cleanse me of Perry Mason. J.

@TiddlyTweeter wrote:

Will it be able to find all instances of "Perry Mason" in Tiddler texts and delete him (for the sake of humanity).

 

Mat replied:

... it is not possible to "cut out" only segments of a field content. If it is a one-time job you have in mind, then the easiest is probably in a XXXX text editor.

[TonyM]

Mat,

Wow leaps and bounds ahead. I will be depending on it to migrate my Personal organiser from TWC to TW5, when I "get around to it", lets hope I can find "a round tooit" because those tooits are quite rare :)

I am a little less trepidatious (apprehensive or nervous; filled with trepidation) with your "batch Manipulator" by my side.

Regards
Tony

[@TiddlyTweeter]

Ciao Mat

I took a closer look. The BATCH MANIPULATOR is seriously useful for maintenance.

It did take me a while to figure out how to use it. That's likely because of its flexibility combined with my slowness in learning.

I do a few more tests & then, all being well, will replace several batch tools I pickup up along the way with this one tool.

TX.

Best wishes
Josiah

p.s. My inline Perry Mason problem I leave for another day.

[@TiddlyTweeter]

Ciao Mat

Footnote ...

Have to admit I was slightly thrown by the thread title "Batch Manipulator - mass tagging" ...

I think that sells it short of what it does.

Best
Josiah

[Mat]

@TiddlyTweeter

I am, in spite of what I said, doing some tweaking on it so you may want to wait just a little with evaluation.

<:-)

[Mat]

Have to admit I was slightly thrown by the thread title "Batch Manipulator - mass tagging" ...

You're right. That was the original premise but it has since expanded. Unfortunately it's not possible to edit thread titles AFAIK 

@moderator - if possible, could it be changed into "Presenting: Batch Manipulator - mass tag/field manipulation"

<:-)

[Mat]

Updated

main change: quick buttons to add e.g a title to the filter (the blue arrows) and buttons to empty out the filter.

<:-)

[@TiddlyTweeter]

Ciao Mat

I get a character encoding problem with the latest on Firefox ...

Best wishes
Josiah

[Mat]

I get a character encoding problem with the latest on Firefox ...

Josiah, thanks for reporting. 

This is a bit surprising as I'm using the utf hex format, specifically

&#x2BA5;

would you mind seeing if the HTML decimal format works for you: i.e simply put this in a tiddler

&#11173;

it should show a little angled arrow.

I picked it up from this neat site (do a search for 2ba5 to find that particular arrow). Does that site show a lot of faulty characters for you? Is there any "systemic" problem, i.e are perhaps all characters in specific tables there problematic? Most importantly; which ones work?

<:-)

[@TiddlyTweeter]

Neither work on tiddlywiki.com in Firefox OR Chrome for me.

Windows 7 Pro.

Weird. Are they dependent on LOCAL font files?

J, x

[@TiddlyTweeter]

Mat,

there is a Unicode guide to font support for arrows here ...

https://www.key-shortcut.com/en/writing-systems/35-symbols/arrows/

Maybe it shed some light on the issue?

Hex 2BA5 has virtually no wide support in common fonts according to the charts there.

J.

[@TiddlyTweeter]

would &#10548; / &#x2934; do? they both work for me in Chrome & Firefox and seem to be in most common fonts.

[Mat]

Updated

Hopefully everyone can see the "add to filter arrows" now (thanks Josiah!)

A function I realized I needed but still miss is a button next to each listed title to remove it from the filter (i.e add -mytitle or -[[my title]] to the filter). Experiments with listops have failed thus far and I will probably post separately on this.

<:-)

[Mat]

 Updated 

I believe the final piece - for now - is in place, thanks to help from Mark S: It is now possible to remove individual items from the list. This is useful when you e.g want to delete all of [tag[foo]] ...except for X and Y that you note in the listing.

<:-)

[Joshua Fontany]

Hmm, I seem to be breaking something when I click the "up arrow" icon next to tags in the table. It adds the "[tag[bb bb]]" text to the filter, but the table header breaks and displays code (breaking the layout). Tested on Chrome 3 times with different tags added to the filter.

Best,
Joshua Fontany

[Mat]

Joshua, thanks for reporting. Do you think you could make a screen dump and post here so I know which table (the input table, the single data table or the relations table) and also so I can see more in detail what happens and what tags it is you're applygin? 

Again, thanks for reporting!

<:-)

[Mat]

Updated

I had fallen into my usual favourite trap of not dealing with tiddlers or tags etc that contain quote marks.

@Joshua - does it work now?

<:-)

[Furicle]

This is a nifty tool - I've got all these mis matched case tags....
Thanks Mat!

[Diego Mesa]

This is a good tool - any thoughts of contributing some/most/all of this functionality into the "tiddler manager"?

On Friday, August 25, 2017 at 7:49:51 AM UTC-5, Mat wrote:

Yeah, the TWaddler gives thee

Batch Manipulator

~ bigly manipulate the masses ~

a tool to simplify tagging/untagging,

setting/deleting fields and/or their (full) content

...en masse as well as single tids

In particular, it simplifies identifying tiddlers that are distantly related so to group (i.e tag) them - see the "green tags" feature.

I've previously made some half measures for this but the tipping point came as I'm currently in the process of converting a big TWC into TW5 (good use case!)

Cred: Some early pieces for this were picked from things by TWizards @Mario and @Jed. 

<:-)

P.S I'm involved in a few other threads here that I have not properly responded to. I hope to do so eventually and I really appreciate your kind replies to my various questions. As someone with a constant tornado of ideas in my head, many of my TW projects float into one another so there are typically several parallel issues. This also means that your kind replies often serve as pieces in several puzzles :-)

[HC Haase]

Using your fileds2table plugin I thought this would be a useful tool with tags instead of fields, and then remembered you already made batch manipulator. I then remembered the reason I dislike using it... the wacky colours.

why do you define all the colours in the plugin, why not rely more on the tw colour? the fields2table plugin mostly use the TW colours and are Pleasant to use, the batch manipulator look like this when using a dark theme and are quite hard to use (especially the dropdowns).

would it be possible for you to make it use the colours from the theme instead of locally defined colors as much as possible please :)

[Mat]

Thanks for input HC Haase

I assume you're talking about the green (and nothing else), right?

The color of the active checkboxes... I guess I chose that color to indicate the active state (as in "green light / go")

And the "links" are non-blue to clearly indicate that they are not links. What would you suggest intead? I chose green to not introduce yet another color but stick with the checkbox one.

The unfortunate blue-boxed arrows are an annoying browser styling of some characters. They were originally a subtle grey arrow but some browser updates changed it into that totally ugly thing. I really want a small upward bending subtle arrow.

<:-)

[Mat]

...but what is that totally blank batchmanipulator "head" where you define the filters etc? Is that how it actually looks on you screen? Does it get that way from the theme you've chosen (which one in that case)?

<:-)

[HC Haase]

fredag den 21. juni 2019 kl. 14.03.20 UTC+2 skrev Mat:

...but what is that totally blank batchmanipulator "head" where you define the filters etc? Is that how it actually looks on you screen? Does it get that way from the theme you've chosen (which one in that case)?

<:-)

The green is fine.. but yes that is how it i looks at my screen. If i choose a white palette then no problem. it is all he white background and unreadable text that's my problem. I found some "background:white" code that I have commented out, but the dropdown is still not nice.

my palette is spartan night made by jd

[HC Haase]

hmm.. I found out, that the dropdown thing is from the palette somewhere. when I use another variation of this palette the dropdown is fine

(BTW this is witht the outcommented bacground:white modifications)