Best way to use TW as text comparision

[Danielo Rodríguez]

Hello everyone.

I want to build a batch text processing tool on top of TW5. The first tool I have "implemented" is a column compare one. It is a macro that gets the text on two tiddlers and give you the unique values in one side of the comparison. The problem with  using a macro is that it is evaluated on every keystroke, navigation event and so on. This is extremely slow and unefficent. Which kind of element should I use to make this more usable? A widget? if so, how? I would like to process just once and then show the result.

This is what I have currently:

/*\

title: $:/macros/danielo515/get_unique_values.js

type: application/javascript

module-type: macro

This macro returns the list of values that exists only on tiddler A.

It expects a list of \n separated values on each tiddler.

\*/

(function(){

  /*jslint node: true, browser: true */

  /*global $tw: false */

  "use strict";

  /*

  Information about this macro

  */

  exports.name = "getunique";

  exports.params = [

  {name: "A"},

  {name: "B"}

  ];

  /*

  Run the macro

  */

  exports.run = function(A,B){

    var a=($tw.wiki.getTiddlerText(A) ||""),

        b=($tw.wiki.getTiddlerText(B) ||""),

        a_values= a.split('\n'),

        b_values= b.split('\n'),

        result = [];

        

        a_values.forEach(function(a_value){

            if(b_values.indexOf(a_value) === -1)

                result.push(a_value);

        });

        console.log(result.length," unique values remains");

        return "```\n" + result.join('\n') + "\n```";

  };

})();

[Danielo Rodríguez]

Offtopic:

I just saw the autoHeight option of the edit-text widget. When did this came out? Very useful and needed.

[Felix Küppers]

Hi Danielo,

there is no way to prevent your macro is refreshed on every keystroke when it appears in a tiddler's preview window. Doesn't matter if you use macro or widget. Your only chance to control the behaviour if you put it somewhere else where the parent does not completely refresh all the time. Then you can control the refreshes via the refresh function of your widget by simply ignoring all the tiddlers in changedTiddlers that are drafts (have a field "draft.of").

-Felix

--
You received this message because you are subscribed to the Google Groups "TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tiddlywiki+...@googlegroups.com.
To post to this group, send email to tiddl...@googlegroups.com.
Visit this group at http://groups.google.com/group/tiddlywiki.
To view this discussion on the web visit https://groups.google.com/d/msgid/tiddlywiki/dc66884b-c46a-4664-bbe3-0ae8dd3fcc4f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Felix Küppers]

Also interesting for you may be this topic:

https://groups.google.com/d/topic/tiddlywiki/V18SC8SDNOw/discussion

And Jeremy's comment:

https://groups.google.com/d/msg/tiddlywiki/V18SC8SDNOw/moR6BNo8kCEJ

On 07.04.2015 18:16, Danielo Rodríguez wrote:

[Danielo Rodríguez]

Hello Felix,

thank you for your comments.

I just remembered how I overcome this previously: I put the macro invocation on a different tiddler. Then when I want to get the results, I just navigate to that tiddler. This case was a bit trickier because I have to change the macro invocation, but this was easy to fix using some macros and the set-field widget (the first time the nest macros behavior is useful!!)

\define unique-a() <<getunique "$:/temp/col-a" "$:/temp/col-b">>

\define unique-b() <<getunique "$:/temp/col-b" "$:/temp/col-a">>

<$button>Get unique values on A

<$action-setfield $tiddler="$:/status/compare-result" text=<<unique-a>>/>

<$action-navigate $to="$:/status/compare-result" $scroll="true"/>

</$button>

<$button>Get unique values on B

<$action-setfield $tiddler="$:/status/compare-result" text=<<unique-b>>/>

<$action-navigate $to="$:/status/compare-result"/>

</$button>

<$edit-text tiddler="$:/temp/col-a" size=20 default="" minHeight="80px" placeholder="A - column" autoHeight="no"/>

<$edit-text tiddler="$:/temp/col-b" size=20 default="" minHeight="50px" placeholder="B - column" autoHeight="no"/>

But tiddlywiki goes very slow as soon as some thousand of values are written in any input. I'm not sure why does this happens but I think it is also related to the "refresh" mechanism.

[Jeremy Ruston]

Hi Danielo

I want to build a batch text processing tool on top of TW5. The first tool I have "implemented" is a column compare one. It is a macro that gets the text on two tiddlers and give you the unique values in one side of the comparison. The problem with  using a macro is that it is evaluated on every keystroke, navigation event and so on. This is extremely slow and unefficent. Which kind of element should I use to make this more usable? A widget? if so, how? I would like to process just once and then show the result.

A reasonable approach might be to have a daemon (ie startup module) that performs the calculations in response to a message, and deposits the results in a tiddler.

Best wishes

Jeremy.

 

--
You received this message because you are subscribed to the Google Groups "TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tiddlywiki+...@googlegroups.com.
To post to this group, send email to tiddl...@googlegroups.com.
Visit this group at http://groups.google.com/group/tiddlywiki.
To view this discussion on the web visit https://groups.google.com/d/msgid/tiddlywiki/dc66884b-c46a-4664-bbe3-0ae8dd3fcc4f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Jeremy Ruston
mailto:jeremy...@gmail.com

[Felix Küppers]

A reasonable approach might be to have a daemon (ie startup module) that performs the calculations in response to a message, and deposits the results in a tiddler.

Yes that is possible. This daemon would then need to register its message handlers to the root widget which is available via "$tw.rootWidget". Also, if you use this technique, make sure your startup exports AFTER "rootWidget" ( exports.after = [ "rootwidget" ];)

-Felix

[Danielo Rodríguez]

Hello Jeremy and Felix,

I though about that also, but it is much more complicated than I wanted at first instance. It is more complicated to start with, because once you have it setup is really easy to add more methods.

Thank you Felix for the explanation, anyway I have some experience with daemons and TW (remember TiddlyDrive? :P )

Anyway, as I said moving the "macro execution" to a different tiddler did the trick for this particular case. Regarding the bad performance while editiong big lists, it has improved a lot bu switching to vanilla theme and doing some theme tweaks. I think for the moment this is good enough, but sooner or later I will be forced to implement your suggestion.

[Felix Küppers]

Hi Danielo

> Thank you Felix for the explanation, anyway I have some experience
> with daemons and TW (remember TiddlyDrive? :P )

Sure I remember! Just ignore stuff you already know ;) Just wanted to
add my thoughts to Jeremy's comment.

> Anyway, as I said moving the "macro execution" to a different tiddler
> did the trick for this particular case.

Nice

> Regarding the bad performance while editiong big lists, it has
> improved a lot bu switching to vanilla theme and doing some theme
> tweaks. I think for the moment this is good enough, but sooner or
> later I will be forced to implement your suggestion.

Or your creative mind comes up with a better idea. As it is often the
case, many brilieant solutions are not discovered at first sight :)

Good night

[Danielo Rodríguez]

I think I misunderstanding the TW's refresh mechanism again .

Since I moved the macro call to a different tiddler it is only updated when I change that tiddler content. Because I do this from a button it looks like it is only reacting to "button press" events. If I edit the content of one of the "temp" tiddlers from an edit-widget it is not being re-evaluated on every key press. Since this is exactly what I wanted I expected the opposite.

Could you please throw some light on this? (yes once again)

[Danielo Rodríguez]

Sure I remember! Just ignore stuff you already know ;) Just wanted to
add my thoughts to Jeremy's comment.

And you did in an excellent way, as usual. That answer is not only useful for my but for anybody reading this thread.

Or your creative mind comes up with a better idea. As it is often the
case, many brilieant solutions are not discovered at first sight :)

Thank you for your kind words.

 

Good night

Gute nacht. 

[Danielo Rodríguez]

And here I am, again.

I created a new macro, this time for search and replacing text. It is, as usual very simple.

I'm using it like this:

<$button class="button button-tiny button-primary">Clean

<$action-setfield $tiddler="$:/temp/col-a" text=<<replace "$:/temp/col-a" "," "x">>/>

</$button>

The problem? The same as allways, on every keystroke the macro is re-evaluated. I did not expected this because it is part of a widget parameter (the action widget) and I though it will be only evaluated when the widget "requires" it. Maybe an option to do this would be useful, to do not evaluate macros until they are required as parameter when the widget is executed.

[Jeremy Ruston]

Hi Danielo

 

The problem? The same as allways, on every keystroke the macro is re-evaluated. I did not expected this because it is part of a widget parameter (the action widget) and I though it will be only evaluated when the widget "requires" it. Maybe an option to do this would be useful, to do not evaluate macros until they are required as parameter when the widget is executed.

The logic actually has to be the other way around: during the refresh cycle, the attributes of widgets are recalculated and then if any of them have changed the widget itself is refreshed.

Best wishes

Jeremy.

 

--
You received this message because you are subscribed to the Google Groups "TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tiddlywiki+...@googlegroups.com.
To post to this group, send email to tiddl...@googlegroups.com.
Visit this group at http://groups.google.com/group/tiddlywiki.

To view this discussion on the web visit https://groups.google.com/d/msgid/tiddlywiki/320a75dc-9832-41a8-ae2d-3d6b10633dd5%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Danielo Rodríguez]

Hello,

For the search and replace thing I ended using a daemon. It is working fairly well.

Just some doubts that have arise from doing this:

•  at which TW version do we started to use event.paramObject notation for messages? I think in 5.1.4, but I'm not sure
  
• It is the same as the old param holding the object, right? I see this much cleaner.
• $tw.wiki.setText function is useful, but the way you want to call it could be odd. For example, you want to change just the text field, making use of the default values that this function tooks you have to $tw.wiki.setText("title",null,null,"new value"). Since the function is called setText I would expect it to change it only the text field. But instead it is able to change any field and any index of the given tiddler. Since this is very powerful I think it should be spliced into two functions. setText (only for text field) and setField (for any field)
  

Regards.

[Felix Küppers]

Hola

•  at which TW version do we started to use event.paramObject notation for messages? I think in 5.1.4, but I'm not sure
  

As far as I remember since 5.1.6 where the action-setfield widget was heavily improved and the new-tiddler message. The advantage here is that we can pass more than one attribute.
 

• It is the same as the old param holding the object, right? I see this much cleaner.

For one parameter it is the same, otherwise the paramObject is better.
 

• $tw.wiki.setText function is useful, but the way you want to call it could be odd. For example, you want to change just the text field, making use of the default values that this function tooks you have to $tw.wiki.setText("title",null,null,"new value"). Since the function is called setText I would expect it to change it only the text field. But instead it is able to change any field and any index of the given tiddler. Since this is very powerful I think it should be spliced into two functions. setText (only for text field) and setField (for any field)
  

I agree in the utils library for the tiddlymap plugin, I also "split" them as it confused me.

Since I moved the macro call to a different tiddler it is only updated when I change that tiddler content. Because I do this from a button it looks like it is only reacting to "button press" events. If I edit the content of one of the "temp" tiddlers from an edit-widget it is not being re-evaluated on every key press.

Sorry maybe I did not understand your question here but generally speaking, a macro that is contained in a tiddler's body is not reevaluated simply because you change a field of the tiddler. It only gets evaluated once you wikitext-parse the body. Only then the macro is actually executed. So e.g. when you transclude a field, then it gets wikiparsed.
 

Since this is exactly what I wanted I expected the opposite.

Sorry, I do not really understand you here.

-Felix