The Many Macros Question - design and performance

[TonyM]

Folks,

This Question is for the more advanced TiddlyWiki user.

The Many Macros Question - design and performance

I am hard at work on a large wiki with many macros driving its operation. As I hope to generalise my code I am calling for any advice on your practices with managing lots of macros.

Background

• I have a number macro tiddlers containing a couple of dozen macro definitions each. 
• I also document each macro in a field on the tiddler macro-macroname with the field value containing the syntax
• When I need to review a macro its easy to find the tiddler its in, but it is hard work locating the definition within the tiddler, especially in edit, and when I have similar names, a search is slow too.
• You have to be careful because a key stroke in the wrong place in macro definitions will break things

I am thinking of doing something like as follows; but not sure if it will add unnecessary overheads/performance;

• I could excise every macro to its own tiddler and tag that with $:/tags/Macro
• This will give more opportunities to document the macro, add keywords
• If one macro depends on another it could make things harder to troubleshoot
• Can tiddlywiki cope with hundreds of $:/tags/Macro tiddlers?
• I could bundle some macros together to reduce the number of tiddlers but does this help anyway?
• Unfortunately I started this design before the recent versions that introduced SystemTag: $:/tags/Macro/View
• even then I am not sure how to divide my macros between view and global as they are often but not always both
• In future wikis how can I use this opportunity to use $:/tags/Macro/View vs $:/tags/Macro
• Using import variables in appropriate places may allow me to selectively include a set of macros in different places. 
• How best will I divide my macros between macro tiddlers this way?
• But where are the best places to import variables (not in the currentTiddler)
• Over lapping import variables are not a problem if its in a good cause, but they could make performance impacts.
• Would conditional transcluding or using view templates to choose when to transclude or import macros at appropriate times help?
• Does tiddlywiki's native indexing lend a hand in any of the above cases to improve performance?

I hope to generate some helpful guidelines from what I learn here.

Thanks in advance

Tony

[Mohammad]

My two cents

1. keep each macro in separate tiddler
2. group them with tag
3. import on demand a group or one macro where you need
4. add a tiddler to search and list quickly (you can use searchable table example exist in Shiraz)
5. bundle and publish as a plugin. You can use Tinka here to do it on the fly

--Mohammad 

[TonyM]

Mohammad

Thanks for your responce. How do you include groups or individual macros as needed?

If we define global macros vs local it sounds like you are proposing mostly local?

Your answers are very helpful to keep this clear, but I wonder about performance?

thanks in advance
Tony

[PMario]

Hi Tony,

I think it depends, what your macros do. .. Eg. If they define global functionality, which is useful for every "user" tiddler, or used in the ViewTemplate, you pretty much have to make them global, tagging them $:/tags/Macro

If you use them in something like a Dashboard tiddler, it can make sense, to \import them in the Dashboard itself. .. It limits the scope of the macros, so you won't have problems with "name collisions", which can be an issue for global macros.

There are some differences in macro/variable lookup performance, but I think you can safely ignore them. If your application uses a filter somewhere, they will outweight the js macro/variable lookup by far. JS variable lookup is highly optimized by browsers.

-------------

I personally have several macro definitions in 1 tiddler. IMO this makes debugging much easier. It also makes it easier to convert a macro definition tiddler into a template, which isn't tagged or imported at, but transcluded.

Transcluding macros uses more memory, since every transclusion defines new macro variables. Calling those macros = variables, has a very minimal performance advantage, since the variable lookup is faster. ... BUT every transclusion needs time to build those variables. ... So in the end the penalties will compensate the advantages.

Global macros are imported in $:/core/ui/PageTemplate ... So those variables will only be built, if the page template or one of the macros are changed.

$:/tags/Macro/View variables will be built everytime the ViewTemplate is shown. On the other hand you can use the same macro name per tiddler view. This gives you the possibility to change macro behavior per tiddler view in the story river.

So again ... It depends, on what you want to do. ...

I think $:/tags/Macro/View is only needed for very special cases. I would go with a global macro definition. It makes them simple to use.

-------------

Having every macro in 1 tiddler can improve the possibility to document them. The text below a macro definitions is ignored by the \import process. So it doesn't have any memory or performance penalty if there is a lot of text there.

But .. I personally think, it makes debugging more complex.

Adding comments like <!-- comment --> inside the macro body, _will_ negatively impact macro performance, since the comment is processed everytime the macro is executed. (... we should try to improve this )

The [tag[abcd]] filter index does speed up finding tiddlers tagged "abcd" quite a bit. So yes. \import filter is improved since the indexing exists.

In 5.1.22 there have also been some improvements, with internal variable handling, which has improved macro handling.

Just some thoughts

have fun!

mario

[TonyM]

Mario

Thanks for your knowledgeable response. I have placed some question inline with your comments. This will be a good opportunity to write some guidance for others, 

I think it depends, what your macros do. .. Eg. If they define global functionality, which is useful for every "user" tiddler, or used in the ViewTemplate, you pretty much have to make them global, tagging them $:/tags/Macro 

 

Yes, its learning how to decide with each macro developed how best to implement it.

If you use them in something like a Dashboard tiddler, it can make sense, to \import them in the Dashboard itself. .. It limits the scope of the macros, so you won't have problems with "name collisions", which can be an issue for global macros.

Good example,  

There are some differences in macro/variable lookup performance, but I think you can safely ignore them. If your application uses a filter somewhere, they will outweigh the js macro/variable lookup by far. JS variable lookup is highly optimized by browsers.

Good to know, this suggests to me that the most important need is self documentation and ease of lookup and use, rather than being concerned with performance.

 

-------------

I personally have several macro definitions in 1 tiddler. IMO this makes debugging much easier. It also makes it easier to convert a macro definition tiddler into a template, which isn't tagged or imported at, but transcluded.

I would love an example of this if possible. That is convert a macro definition tiddler into a template.

 

Transcluding macros uses more memory, since every transclusion defines new macro variables. Calling those macros = variables, has a very minimal performance advantage, since the variable lookup is faster. ... BUT every transclusion needs time to build those variables. ... So in the end the penalties will compensate the advantages.

Very important to know. So transclusion redefines, import variables makes the variables available, $:/tags/Macro makes them always available ?

Global macros are imported in $:/core/ui/PageTemplate ... So those variables will only be built, if the page template or one of the macros are changed.

So this makes them quite efficient, but some of my macros refer to another value eg 

\define wiki-mode() {{$:/config/wiki-mode}}

so will all the the page macros be built every time $:/config/wiki-mode changes.

 

$:/tags/Macro/View variables will be built everytime the ViewTemplate is shown. On the other hand you can use the same macro name per tiddler view. This gives you the possibility to change macro behavior per tiddler view in the story river.

So again ... It depends, on what you want to do. ...

The word "reusability" comes to mind here as a design consideration for macros.

 

I think $:/tags/Macro/View is only needed for very special cases. I would go with a global macro definition. It makes them simple to use.

-------------

Thanks so much for this advice, I almost started using it wholesale, I will not now.

 

Having every macro in 1 tiddler can improve the possibility to document them. The text below a macro definitions is ignored by the \import process. So it doesn't have any memory or performance penalty if there is a lot of text there.

But .. I personally think, it makes debugging more complex. 

Good to know, So If I placed in the body macro documentation is will be visible on the macro tiddler but not add too much to slow down?

For example?

\define first-tiddler(filter:"[tag<currentTiddler>]") {{{ $filter$ +[first[]] }}}
\define last-tiddler(filter:"[tag<currentTiddler>]") {{{ $filter$ +[last[]] }}}
\define next-tiddler(filter:"[tag<currentTiddler>]") {{{ $filter$ +[after[]] }}}
\define prev-tiddler(filter:"[tag<currentTiddler>]") {{{ $filter$ +[before[]] }}}

<h2>Macro definition tiddler</h2>

;Obtain the first/last/next/prev member in a list defined by filter
:Defaults to tiddlers tagged with the current tiddler, `[tag<currentTiddler>]`
*`<<first-tiddler [filter]>>`
*`<<last-tiddler [filter]>>`
*`<<next-tiddler [filter]>>`
*`<<prev-tiddler [filter]>>`

<!-- comment here ok? search keywords etc... -->

 

Adding comments like <!-- comment --> inside the macro body, _will_ negatively impact macro performance, since the comment is processed everytime the macro is executed. (... we should try to improve this )

But if I understand correctly it is ok as in the example above?

 

The [tag[abcd]] filter index does speed up finding tiddlers tagged "abcd" quite a bit. So yes. \import filter is improved since the indexing exists.

This is when the filters are used in the import widget and pragma?

 

Thanks for this feedback. I will summarise it all at the end of this thread.

Regards

tony

[Mark S.]

Since your question seems to be mainly about documenting macros, why not put your macro definitions in a data tiddler. Then as you edit, if you need a reminder just type

{{macs##mymacro1}}

and see something like

mymacro1(size of tiddler, name of field)

over in the preview window.

Putting all the definitions in one place also has the advantage that you can quickly test at any time to make sure you're not accidentally creating a tiddler with the same name as an existing one.

[TonyM]

Mark,

On Monday, 4 May 2020 08:20:20 UTC+10, Mark S. wrote:

Since your question seems to be mainly about documenting macros, why not put your macro definitions in a data tiddler. Then as you edit, if you need a reminder just type

{{macs##mymacro1}}

and see something like

mymacro1(size of tiddler, name of field)

Interesting idea. Extending it further with a special excise button in the editor, to highlight and push a macro syntax to a data tiddler could be very efficient.

 

over in the preview window.

I don't use preview much at this point. You have a different workflow to me.

 

Putting all the definitions in one place also has the advantage that you can quickly test at any time to make sure you're not accidentally creating a tiddler with the same name as an existing one.

 My macros are typically in a tiddler called $:/PSaT/projectname/macros so I dont see this problem. 

Until now I add macro-macroname fields containing the syntax of each macro defined in a given tiddler, I have a utility window that lists them including syntax and a link to the source tiddler.

I may revisit this but it works very well.

Thanks for your ideas

Tony

[PMario]

On Monday, May 4, 2020 at 12:07:32 AM UTC+2, TonyM wrote:...

I personally have several macro definitions in 1 tiddler. IMO this makes debugging much easier. It also makes it easier to convert a macro definition tiddler into a template, which isn't tagged or imported at, but transcluded.

I would love an example of this if possible. That is convert a macro definition tiddler into a template.

In October 2017 I did a video series, where I did create a "Script / ToDo Manager", which was requested in the GG. I did record 17 videos, that basically explain every line of code, that has been created and a lot of concepts, behind the whole app.

The code can be seen at: https://wikilabs.github.io/editions/script-manager/ 

Open Right sidebar : More : System ... It's a bundle and not a plugin.

The intro thread was: https://groups.google.com/forum/#!msg/tiddlywiki/_VQw8yIKWn8/G8TO8h6MCAAJ which also contains a detailed overview, about the videos, and what is shown. ...

The first post also contains the bundle version 0.0.3, which is part of the edition linked above.

If you open the edition link, the first tiddler you see contains a transclusion to https://wikilabs.github.io/editions/script-manager/#%24%3A%2F_bundle%2Fwikilabs%2Fscript-manager%2Fscript-list-template ... which was intended to implement several macros.

In the end it turned out, that it just calls the main macro at the end of the tiddler and is used as a transclusion instead.

There are some more -template tiddlers, which contain several macros and directly call them. ... Other tiddlers are used as macros. ...

have fun!

-mario

[PMario]

On Monday, May 4, 2020 at 12:07:32 AM UTC+2, TonyM wrote:

...

Global macros are imported in $:/core/ui/PageTemplate ... So those variables will only be built, if the page template or one of the macros are changed.

So this makes them quite efficient, but some of my macros refer to another value eg 

\define wiki-mode() {{$:/config/wiki-mode}}

so will all the the page macros be built every time $:/config/wiki-mode changes.

I think I wasn't clear enough. .. It will need to recalculate all macros, if you edit the tiddler, that contains the \define line. The wiki-mode macro will be evaluated if you use it with <<wiki-mode>> 

The global wiki-mode variable contains: {{$:/config/wiki-mode}} ... as a string





You can see all variables if you use: <<dumpvariables>> macro see: https://tiddlywiki.com/#dumpvariables%20Macro%20(Examples)





-mario


[PMario]

On Monday, May 4, 2020 at 12:07:32 AM UTC+2, TonyM wrote:

...

Having every macro in 1 tiddler can improve the possibility to document them. The text below a macro definitions is ignored by the \import process. So it doesn't have any memory or performance penalty if there is a lot of text there.

But .. I personally think, it makes debugging more complex. 

Good to know, So If I placed in the body macro documentation is will be visible on the macro tiddler but not add too much to slow down?

For example?

\define first-tiddler(filter:"[tag<currentTiddler>]") {{{ $filter$ +[first[]] }}}
\define last-tiddler(filter:"[tag<currentTiddler>]") {{{ $filter$ +[last[]] }}}
\define next-tiddler(filter:"[tag<currentTiddler>]") {{{ $filter$ +[after[]] }}}
\define prev-tiddler(filter:"[tag<currentTiddler>]") {{{ $filter$ +[before[]] }}}

<h2>Macro definition tiddler</h2>

;Obtain the first/last/next/prev member in a list defined by filter
:Defaults to tiddlers tagged with the current tiddler, `[tag<currentTiddler>]`
*`<<first-tiddler [filter]>>`
*`<<last-tiddler [filter]>>`
*`<<next-tiddler [filter]>>`
*`<<prev-tiddler [filter]>>`

<!-- comment here ok? search keywords etc... -->

<h2> ... is already comment. So you can see it, if open the tiddler in view mode.

If you the following line to a tiddler tagged: $:/tags/Macro you'll see the code in view mode.

<pre><$view field="text"/></pre>

So your comment could be text only. .. no need to use <!-- comment --> 

All kudos for the <pre> ... line above go to Eric Shulman.

The <pre>... definition isn't possible, if you use a tiddler as a template, or if you want to transclude it.

have fun!

[PMario]

On Monday, May 4, 2020 at 12:07:32 AM UTC+2, TonyM wrote:

...

Adding comments like <!-- comment --> inside the macro body, _will_ negatively impact macro performance, since the comment is processed everytime the macro is executed. (... we should try to improve this )

But if I understand correctly it is ok as in the example above?

The example you used isn't part of a macro. It needs to be like this:

title: test-macro

tags: $:/tags/Macro

\define test("xx")

$xx$  <!-- this comment will slow you down -->

\end

Text here in a tiddler tagged: $:/tags/Macro doesn't influence macro performance.

[PMario]

On Monday, May 4, 2020 at 12:07:32 AM UTC+2, TonyM wrote:

...

The [tag[abcd]] filter index does speed up finding tiddlers tagged "abcd" quite a bit. So yes. \import filter is improved since the indexing exists.

This is when the filters are used in the import widget and pragma?

Yes.

-m

[TonyM]

Thanks Mario,

I will turn this into some notes and republish.

Thanks a lot.

Regards

Tony