systemConfig load order? How to extend a plugin?

[Conal Elliott]

I've written a simple language spec for Haskell to add to Bob's SyntaxifyPlugin.  It works fine as a modification to the plugin, but I'd rather have it in a separate systemConfig tiddler, so I can share it with others and conveniently get updates from SyntaxifyPlugin.

My puzzle: how do I get my plugin to be executed after SyntaxifyPlugin, which provides needed code?

Thanks,  - Conal

[Bob McElrath]

Conal Elliott [conal....@gmail.com] wrote:
> I've written a simple language spec for Haskell to add to Bob's

> SyntaxifyPlugin <http://bob.mcelrath.org/syntaxify.html>. It works fine as

> a modification to the plugin, but I'd rather have it in a separate
> systemConfig tiddler, so I can share it with others and conveniently get
> updates from SyntaxifyPlugin.
>
> My puzzle: how do I get my plugin to be executed after SyntaxifyPlugin,
> which provides needed code?

Cool!

I added syntaxify.addLanguages to do this, but I can see now that the
order of plugin loading is important.

The simple answer is to name your Haskell plugin something that comes
after the name of your syntaxify plugin ("Plugin: Syntaxify"). (And
call syntaxify.addLanguages with a data structure similar to that in the
plugin)

This is hardly satisfactory though. It would be very nice to have a
depends-on: qualifier of some sort. But that in turn requires some way
to identify the plugin that is independent of what you named the
tiddler. (Surely not everyone likes my choice of "Plugin: Syntaxify" --
I know lots of plugins named SomethingPlugin instead)

Anyone want to chime in on this? Perhaps it would be useful for plugin
writers to register themselves with the PluginManager?

Conal Elliott [conal....@gmail.com] wrote:
> I want to create a simple formatter that takes tiddler content like
>
> \begin{code}
> >my code goes here
> >\end{code}
> >
>
> and treats it exactly as
>
> {{haskell{
> >my code goes here
> >}}}
>
>
> The reason I want to enter the former instead of the latter is so that my
> tiddler content can be interpreted as a literate Haskell program (where text
> is treated as comments unless surrounded by \begin{code} and \end{code}).
>
> I'm guessing this sort of thing is trivial for someone who knows their way
> around the TW source, but I haven't been able to figure it out.
>
> I'm using ZiddlyWiki, and my html says version 2.0.11.
>
> I'd love some suggestions.

Another good question... (you don't hang out on axiom-developer, do you?
;)

Take a look at http://bob.mcelrath.org/tiddlyjsmath.html which creates
\begin{equation}...\end{equation}. I would be quite interested in a
faux-latex wikifier, if you want to work on that. But anyway the markup
is pretty trivial, the only non-trivial thing is that you really should
introduce the backslash as an escape character (as is done in that
plugin).

--
Cheers,
Bob McElrath [Univ. of California at Davis, Department of Physics]

"The reasonable man adapts himself to the world; the unreasonable one
persists in trying to adapt the world to himself. Therefore all progress
depends on the unreasonable man."
-- George Bernard Shaw, Maxims for Revolutionists

[Saq Imtiaz]

I believe specifying plugin load orders and dependencies was penned for TW 2.1 but got pushed back. There should be an open ticket for it.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)

iD8DBQFFlLOljwioWRGe9K0RAi1LAKCvyNUMd8jDLDy9H3O1YqJCmYl4vQCgxfOh
QIDjdmci/Bh+7NVz/SCMPq4=
=RpNx
-----END PGP SIGNATURE-----

[Jeremy Ruston]

> My puzzle: how do I get my plugin to be executed after SyntaxifyPlugin,
> which provides needed code?

As you've gathered, there's currently no decent way of doing this
apart from using the alphabetical ordering of tiddler titles, as Bob
suggests.

There is another approach I've considered that may solve the problem
quite neatly, though, and I'd be interested to see what people think.

The idea is to allow plugins to have a return value. Usually, the
return value will be something indicating "OK" (and indeed for
backwards compatibility, plugins that don't return a value would be
assumed to return "OK"). It is also possible for plugins to return the
value "NOTYET" which means "I haven't yet executed because something
I'm dependent upon hasn't happened yet". In that case, the plugin
framework will move onto the remaining plugins and try to execute
those before returning to the original plugin and trying again.

So, for example, your plugin could start with code like this:

if(!bobsPluginIsLoaded)
return NOTYET
// Do our processing here
return OK

This is pretty easy to implement - the only point to watch is that the
plugin framework should bail out if a complete cycle of plugins return
NOTYET.

Not sure if that makes sense, but it appears to allow for fairly
complex chains of dependency with reasonably robust deadlock recovery.

Cheers

Jeremy

--
Jeremy Ruston
mailto:jer...@osmosoft.com
http://www.tiddlywiki.com

[Daniel Baird]

On 12/29/06, Jeremy Ruston <jeremy...@gmail.com> wrote:
>
>
> There is another approach I've considered that may solve the problem
> quite neatly, though, and I'd be interested to see what people think.
>

> [..]

>
> Not sure if that makes sense, but it appears to allow for fairly
> complex chains of dependency with reasonably robust deadlock recovery.

That sounds like a great idea -- simple to code and understand, and
flexible enough for just about anything.

There's only two things it seems not to cover:
- plugin A wants to be loaded after plugin B, but only if plugin B is
present. If plugin B isn't there, then plugin A can still be loaded.
- plugin A wants to be loaded _before_ plugin B.

They're pretty obscure, just figured they should be out there. If
plugins had reliably unique identifiying strings, you could solve the
first case by providing each plugin with the list of plugins loaded
and plugins yet-to-load.

I think the deadlock detection will still hold true even if you're
passing plugins a plugin list.

And come to think of it, then you could solve the second situation by
the author of A asking the author of B to load after A, if A is
present.

I haven't really thought this through so maybe I've missed something..

;Daniel

--
Daniel Baird
http://tiddlyspot.com (free, effortless TiddlyWiki hosting)
http://danielbaird.com (TiddlyW;nks! :: Whiteboard Koala :: Blog ::
Things That Suck)

[Bob McElrath]

I like this idea, but the logic behind "is plugin X loaded?" is a bit
fuzzy.

Given that several different versions of a plugin, slightly modified by
several different authors may exist... we also need a reliable way to
answer the question "is plugin X loaded".

I guess we want to use here
if(version.extensions.Syntaxify == undefined) {
return NOTYET;
}
if(version.extensions.Syntaxify.major != 1) {
return INCOMPATIBLE;
}

The scheme you suggest also needs a well-defined termination point.
That is, imagine you have plugins A and B both of which return NOTYET.
TW doesn't know whether A depends on B or vice-versa. It might run A,
then B, then A, then B... I suppose a good way to define the endpoint is
if the list of plugins loaded is the same after running all un-run
plugins, and having them return NOTYET. Of course this indicates some
dependency is simply not satisfied.

Any chance we can get this into the next 2.2 beta?

Cheers,
-- Bob

> --~--~---------~--~----~------------~-------~--~----~
> You received this message because you are subscribed to the Google Groups
> "TiddlyWiki" group.
> To post to this group, send email to Tiddl...@googlegroups.com
> To unsubscribe from this group, send email to
> TiddlyWiki-...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/TiddlyWiki?hl=en
> -~----------~----~----~----~------~----~------~--~---

[Jeremy Ruston]

> I guess we want to use here
> if(version.extensions.Syntaxify == undefined) {
> return NOTYET;
> }
> if(version.extensions.Syntaxify.major != 1) {
> return INCOMPATIBLE;
> }

That was what I reckoned too.

> The scheme you suggest also needs a well-defined termination point.
> That is, imagine you have plugins A and B both of which return NOTYET.
> TW doesn't know whether A depends on B or vice-versa. It might run A,
> then B, then A, then B... I suppose a good way to define the endpoint is
> if the list of plugins loaded is the same after running all un-run
> plugins, and having them return NOTYET. Of course this indicates some
> dependency is simply not satisfied.

I think that's the right approach: drop out after 'n' sweeps of the
un-run plugins if no plugin changes it's stage. As you say, 'n' must
be greater than 1; it seems as though n should be 2, what do you
reckon?

> Any chance we can get this into the next 2.2 beta?

Sure, I'll take patches in the usual way :)

Best wishes

Jeremy

> > > --
> Cheers,
> Bob McElrath [Univ. of California at Davis, Department of Physics]
>
> "The reasonable man adapts himself to the world; the unreasonable one
> persists in trying to adapt the world to himself. Therefore all progress
> depends on the unreasonable man."
> -- George Bernard Shaw, Maxims for Revolutionists
>
>

> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.6 (GNU/Linux)
>

> iD8DBQFFlWWQjwioWRGe9K0RAjVmAJ0d9frdUJJowOnAkEv03L96gkn8JwCePIK9
> AQ2RAfjuFHltcoHfNWhQ3yE=
> =U8Wn
> -----END PGP SIGNATURE-----

[Conal Elliott]

How about the following simple, direct, declarative approach: have plugins list the tiddlers they depend on, either as tags or via a custom field.  Sounds easier to write, to read, and to implement.  No wondering about why NOTYET.

  - Conal

On 12/29/06, Jeremy Ruston <jeremy...@gmail.com> wrote:

[Jeremy Ruston]

> How about the following simple, direct, declarative approach: have plugins
> list the tiddlers they depend on, either as tags or via a custom field.
> Sounds easier to write, to read, and to implement. No wondering about why
> NOTYET.

I thought about that idea, but figured on a couple of factors:

- it requires that plugins have their official title; at the moment,
TW lets users rename their plugins quite freely. It's not so much that
renaming the plugins is a hugely useful feature, more that it's a
level of robustness that makes the overall system more forgiving.
- it changes the semantics of plugins by adding metadata (whether tags
or extended fields) outside of the plugin content itself.
- it restricts the dependencies that can be expressed. The beauty of
the NOTYET mechanism is that it very flexibly allows complex
dependencies (such as "install if pluginA OR pluginB is installed" or
"if we're on Windows, require pluginW"). Under the declarative
approach, each of these scenarios would need to be specifically
catered for in the syntax used in the tag or field.
- from an implementation perspective, detecting and dealing with
cyclic dependencies seems more complex under your proposal - one would
have to pre-compute some kind of dependency graph in order to figure
out where to start

Cheers

Jeremy

[Conal Elliott]

Hi Bob,

Thanks very much for the name ordering tip.  It worked.  I gather what you're implying is that tiddlers are loaded in lexical order of their names.  Is that ordering property true of ZiddlyWiki in particular or universal in TWs?

I happened to rename your plugin SyntaxifyPlugin, so I renamed my extension "SyntaxifyPlugin: Haskell", which works fine for me.    Not sure what names I like for plugins, and maybe I prefer "Plugin: Syntaxify: Haskell", but I'm fine with just appending.  Of course that trick only works for single dependencies.  Otherwise, I might list the dependencies in reverse lexical order at the start of my plugin's name.  Workable until there's another solution.  I love declarative methods (which is why I work mostly in Haskell), so I hope that's how we go.

Thanks also for the jsMath pointer.  Besides the simple & relevant TW-programming example, I'm looking forward to playing with using LaTeX math in my TiddlyWiki.

Best regards,  - Conal

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)

[Conal Elliott]

Hi Jeremy,

I see what you mean about the flexibility of the NOTYET approach. It does have a performance problem, namely O(n²) worst case for n plugins. Making dependencies explicit allows a standard topological sort algorithm, which is linear in the number of plugins + actual dependencies.

About your metadata point: plugins already do rely on a piece of metadata, namely the systemConfig tag.

I'm confused about some aspects of the NOTYET idea:

• Suppose my plugin can work with "pluginA OR pluginB installed", but prefers pluginB. I don't know how it could decide when to settle for pluginA and when to hold out in hope of pluginB.
• Mechanical: how can a plugin return NOTYET or any value at all? Isn't plugin code simply evaluated for its side-effect?

I'm wondering whether CSS has a similar dependency issue. When I want to override a CSS spec in some tiddler (like StyleSheetLayout), I don't know how to do that reliably. (Well, maybe I do, now with Bob's lexical ordering tip.) I think the one that shows up last in the html wins. And I don't know how to guarantee that mine ends up last. So I have the same problem as plugin dependencies. The explicit dependency approach (with top-sort) would apply to CSS as well as plugins, but the NOTYET solution depends on there being executable code. Right?

BTW, while we're on the topic of plugins & dependencies, I'd love to have specification and mechanism for pulling in tiddlers (plugins, CSS, marked up text) from external sources. That way, I can save a lot of duplication and consistency management. Ideally, I'd like my TWs to be really tight -- free of any redundant information, like my programs are. Anything reusable would be put in a library and imported. Rather than simply ordering the processing of my plugins (as we've been discussing in this thread), the listed dependencies (or the plugin code) would be used to actively load depended-on code if it's not local to the TW and if it hasn't already been loaded.

One thing I really love about TiddlyWiki is that it stimulates my imagination whenever I use it. I start out just wanting to type in my journal and the tool itself inspires my creativity. Last night I got the idea of organizing my Haskell program modules asTWs and using Haskell's module discipline for TWs in general. The code would be rendered beautifully and have automatic links from name uses to defining tiddlers. I got so excited about this idea I couldn't sleep.

All the best,

[Udo Borkowski]

For more flexibility in the future I suggest not just return a value but an object. E.g. like this:

> I guess we want to use here
>     if(version.extensions.Syntaxify == undefined) {

>         return {state: NOTYET};
>     }
>     if(version.extensions.Syntaxify.major != 1) {
>         return {state: INCOMPATIBLE};
>     }

This gives us many options how we could control the plugin load process.

Example:

we may (later) allow a result like this:

    return
        {
            state: NOTYET,
            pluginsToLoad: ["http://tiddlywiki.abego-software.de/#DataTiddlerPlugin"]
        };

I.e. using the property "pluginsToLoad" a plugin may either trigger the dynamic loading/installation of other plugins (if this is allowed) or TiddlyWiki may just report the list of missing plugins at the end of loading process.

An other example that are extends the PluginLoadResult object is a "message" property. When defined the given string is displayed to the user when the plugin is (finally) not loaded.


Udo

----------
Udo Borkowski
http://www.abego-software.de

On 12/29/06, Jeremy Ruston < jeremy...@gmail.com> wrote:

> I guess we want to use here

>     if( version.extensions.Syntaxify == undefined) {

[Udo Borkowski]

I think that's the right approach: drop out after 'n' sweeps of the
un-run plugins if no plugin changes it's stage. As you say, 'n' must
be greater than 1; it seems as though n should be 2, what do you
reckon?

2 sweeps with unchanged states of unrun plugin are sufficient (assuming that unchanged states also mean the plugin hasn't done any changes).

Actually one needs less than 2 complete sweeps: an unrun plugin just needs to be re-run if there as any state change since the last attempt to run it. But I guess this extra coding is not worth the effort and just doing 2 complete sweeps  should be OK.

Udo

----------
Udo Borkowski
http://www.abego-software.de

On 12/29/06, Jeremy Ruston <jeremy...@gmail.com> wrote:

[Udo Borkowski]

I've got one unsolved issue with the "return" approach of plugin.

When developing plugins I typically write the JavaScript code in a separate file and "include" it with a <script> tag in a Markup... tiddler. This way I can better debug the code.

In this scenario using "return NOTYET" or "return OK" doesn't really work, I think.

May somebody has an idea? Eg. is there a way to detect if plugin code is executed by the "plugin loader" or by the browser?

Udo

----------
Udo Borkowski
http://www.abego-software.de

On 12/29/06, Jeremy Ruston <jeremy...@gmail.com> wrote:

[Udo Borkowski]

Hi Conal,

you are correct with your worst case complexity analysis, but actually this worst case only happens when we  a chain of plugins that depend on each other (e.g. A -> B -> C -> D -> ... -> Z). I would assume that in most cases the dependency level will be 1 or maybe 2. Most plugins will not depend on any other plugins. So I would complexity is more like O(n*d), with n plugins and d the maximal level of dependencies. Given this and the other advantages of the procedural approach I also advocate the "return" approach.

• Suppose my plugin can work with "pluginA OR pluginB installed", but prefers pluginB. I don't know how it could decide when to settle for pluginA and when to hold out in hope of pluginB.

If both pluginA and pluginB are available both will be installed, independet on whether you plugin prefers pluginA or pluginB. I guess you will the decision what plugin to use (A or B) to the first time you really need it. But because of the load process you can be sure that either pluginA or pluginB is installed (otherwise your plugin would not be "loaded").

Mechanical: how can a plugin return NOTYET or any value at all? Isn't plugin code simply evaluated for its side-effect?

The plugin code is executed using "window.eval(... the plugin code...)", giving us the option to get a return value. Up to now this return value was just discarded.

Hope this clears things up.

Udo

----------
Udo Borkowski
http://www.abego-software.de

[Daniel Baird]

On 12/30/06, Conal Elliott <co...@conal.net> wrote:

> [..]

> I'm wondering whether CSS
> has a similar dependency issue. When I want to override a CSS spec in some
> tiddler (like StyleSheetLayout), I don't know how to do that reliably.
> (Well, maybe I do, now with Bob's lexical ordering tip.) I think the one
> that shows up last in the html wins. And I don't know how to guarantee that
> mine ends up last. So I have the same problem as plugin dependencies. The
> explicit dependency approach (with top-sort) would apply to CSS as well as
> plugins, but the NOTYET solution depends on there being executable code.
> Right?

Somewhat off-topic, but TW explicitly loads StyleSheet last, after
StyleSheetLayout etc, so that you put your overriding CSS into
StyleSheet and it's guaranteed to override, if you get your
specificity right. Unfortunately CSS specificity is quite hard to get
right..

;D

[Conal Elliott]

Thanks for this tip, Daniel.  I wondered why the overriding worked when the specificities where the same as in StyleSheetLayout.

Still, I'd like the same CSS ordering guarantee in other situations also.  Suppose I have a tiddler to share that overrides some CSS in StyleSheetLayout and which a TW can further override vis StyleSheet.

Cheers, - Conal

[Conal Elliott]

Does the plugin name ordering trick work for IE?  I'm getting "'syntaxify' undefined" there when loading my plugin.  Also, is lexically ordered loading limited to ZiddlyWiki? - Conal

On 12/28/06, Bob McElrath <bob.mc...@gmail.com> wrote:

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)

[Daniel Baird]

The lexical ordering works because TW orders tiddlers alphabetically
when it's saving the tiddler store. So it should work everywhere, not
just in ZW.

I'm guessing you've confirmed that the Syntaxify plugin is actually
getting loaded properly itself.. maybe try looking for something you
know is in versions.extensions, to see if you are looking at the right
thing?

And is it version.extensions.Syntaxify or version.extensions.syntaxify
that you are checking?

Otherwise maybe post a URL so we can have a fiddle :)

;Daniel

[Conal Elliott]

Thanks, Daniel.  I do see that the Syntaxify plugin is getting loaded and indeed loaded before my extension to it, which is at http://journal.conal.net/#%5B%5BSyntaxifyPlugin%3A%20Haskell%5D%5D .

I am not checking any version.extensions at all.  What is the purpose of defining and version.extensions objects?

Now there's a second error that shows up (first) under IE (" 'RewritePlugin: math/Haskell' -- Object unexpected").  I have no idea what's broken there.  Can anyone recommend a debugging tool for Javascript?

An unrelated problem is that my layout under IE gets very messed up when I change the view text size.

Any suggestions appreciated.  Cheers,

  - Conal

[Saq Imtiaz]

try window.syntaxify

I came across a bug yesterday to do with IE scoping as of TW 2.1.3.
Dont know if it applies to ZW but its worth a shot.