Parsing other format to tiddlers: in build step or within TW?

[Scott Sauyet]

I'm working on a new programming library and want to generate a tiddlywiki version of the documentation to serve from Node. I'm looking for advice on the best way to do this.  I will be maintaining the documentation in a markdown-file-per-function format; this will work well with our GitHub environment.  I will need to convert that to tiddlers.  I'd like to break this down into smaller parts in keeping with the TW philosophy,  But I don't know if I should do this in some server-side build step or whether it would be straightforward to do that inside the TW client environment.

I will be doing this conversion in JS.  Either it will take place in a Node-based build step or in a custom plugin vaguely similar to the Markdown plugin.  So the core code would probably be similar either way.  I know little of plugin development., but I'm more than willing to learn.  And if I can manage it, I would much rather have the intact content (with just some field headers prepended) passed down to the client and interpreted there.  But I just don't know how possible that is.

The project is behind a corporate wall, and I can't share it, but it's similar to work I've done on the open-source Ramda (https://ramdajs.com/).  Taking a function like `map` (https://ramdajs.com/docs/#map), I would start with this (sorry fairly long) markdown:

======================================================================

R.map

=====

*"It is not down on any map; true places never are."* - Herman 

Melville

Signature

---------

  

    map :: Functor f => (a -> b) -> f a -> f b

    map :: (a -> b) -> [a] -> [b]

    map :: (a -> b) -> {String : a} -> {String: b}

    map :: (a -> b) -> (r -> a) -> (r -> b)

    

    

Description

-----------

Takes a function and a [Functor][fu], applies the function to each of

the functor's values, and returns a functor of the same shape.

Ramda provides suitable map implementations for `Array` and `Object`

and `Function`, so this function may be applied to `[1, 2, 3]`, to 

`{x: 1, y: 2, z: 3}` or to `n => n + 1`.  It delegates to other 

Functors.

Symbolic

--------

    R.map(f, [a, b, c])           //=> [f(a), f(b), f(c)]

    R.map(f, {x: a, y: b, z: c})  //=> {x: f(a), y: f(b), z: f(c)}

    R.map(f, g)                   //=> (...args) => f(g(...args))

    R.map(f, functor)             //=> functor.map(f)

    

    

Examples

--------

*     const double = n => n + n

      const square = n => n * n

    

      R.map(double, [1, 2, 3])           //=> [2, 4, 6]

      R.map(double, {x: 2, y: 3, z: 5})  //=> {x: 4, y: 6, z: 10}

      R.map(double, square)(3)           //=> 18

*     const increment = n => n + 1

      R.map(increment, Maybe.Just(42)    //=> Maybe.Just(43)

      R.map(increment, Maybe.Nothing()   //=> Maybe.Nothing()

See Also

--------

  - R.transduce

  - R.addIndex

  

Since

-----

    v0.1

  

  [fu]: ../types/Functor

======================================================================

And I would like to generate a tiddler such as

======================================================================

created: 20180928141411712

modified: 20180928170503224

tags: 

title: R.map

type: text/vnd.tiddlywiki

{{$:/api/R.map/intro}}

!! Signature

{{$:/api/R.map/sig}}

!! Description

{{$:/api/R.map/desc}}

!! Symbolic

{{$:/api/R.map/symb}}

!! Examples

<$list filter="[tag[fn-Example]tag[R.map]]">

   <$transclude field="text" mode="block"/>

</$list>

!! See Also

{{$:/api/R.map/seeAlso}}

!! Since

```

v0.1.0

```

======================================================================

and a collection of supporting tiddlers like

======================================================================

created: 20180928141421990

modified: 20180928170615408

tags: R.map fn-Signature

title: $:/api/R.map/sig

type: text/vnd.tiddlywiki

```

map :: Functor f => (a -> b) -> f a -> f b

map :: (a -> b) -> [a] -> [b]

map :: (a -> b) -> {String : a} -> {String: b}

map :: (a -> b) -> (r -> a) -> (r -> b)

```

======================================================================

I guess the first question is whether a plugin's code can be run at startup against a collection of tiddlers and generate more tiddlers.  If that can't happen, then I'm pretty sure I'm stuck doing this in Node.

But then, if I can, *should* I?  I really like both TiddlyWiki's small-content-is-better philosophy and the single-markdown-file-per-function documentation style.  Perhaps I need to live with just the one and include these simply as Markdown tiddlers.  That would mean that I don't get the ability to reuse smaller parts of the documentation in other interesting places.  But I don't have any immediate need to do that.  It's purely speculative.  And yet, if I simply put it off, it probably won't happen. I have time now to work on this that I probably won't get later.

So I'm looking for advice.  Would you do this in a server-side build step?  Would you do it inside TiddlyWiki?  Is there some better approach I'm not even seeing? 

Thanks for any suggestions you might have.

  -- Scott

[Jeremy Ruston]

Hi Scott

I’ve also been thinking about something similar: using TW5 to create a reusable tool that interleaves documentation files in a standard git repository, threading them together to construct a TiddlyWiki that can be used to view/edit the docs, or to generate static renderings. I think the round trip editing could be a killer feature: most existing documentation-baking tools expect editors/writers to navigate the source tree of MD files directly.

To your specific question about whether the documentation files might be sliced into multiple tiddlers  as part of the process of threading them together, I suspect that that might make it a little bit harder to make a round trip editor. There’s also the question you raise about whether chunking into sections will actually lead to reusability.

Perhaps there’s more scope for reusability by centralising definitions and descriptions, and transcluding them at the point of use?

Anyhow, it’s all quite ambitious, but one could build it in stages:

* An MVP to get things started might be a tool that scans a repo for documentation files and does the most basic aggregation of those files as tiddlers into a TiddlyWiki that includes a plugin containing the templates needed to view and interact with those tiddlers

* Next, one might try to get things working under Node.js so that the documentation files can be edited within TiddlyWiki. There would be a startup process reading the tiddlers from the repo, and then passing them to the syncer/filesystemadaptor so as to have future changes to those tiddlers synced back to the original file

* Finally, one could explore reusing some existing open source documentation templates for output

Best wishes

Jeremy

--
You received this message because you are subscribed to the Google Groups "TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tiddlywiki+...@googlegroups.com.
To post to this group, send email to tiddl...@googlegroups.com.
Visit this group at https://groups.google.com/group/tiddlywiki.
To view this discussion on the web visit https://groups.google.com/d/msgid/tiddlywiki/09b73b62-4cbd-4316-9398-02d70b213643%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Scott Sauyet]

Thanks for the great feedback!

Jeremy Ruston wrote:

> I’ve also been thinking about something similar: using TW5 to create
> a reusable tool that interleaves documentation files in a standard
> git repository, threading them together to construct a TiddlyWiki that
> can be used to view/edit the docs, or to generate static renderings.
> I think the round trip editing could be a killer feature: most
> existing documentation-baking tools expect editors/writers to navigate
> the source tree of MD files directly.

Ah, I think I forgot to mention that I was planning on the standard view
of this to be read-only.  :-)

What you're talking about sounds like a terrific extension of my ideas,
and I had considered some of it.  I was thinking that core team members
might be able to edit the tiddlers in place for a round-trip experience.
But that was not central to my idea, and definitely was not meant for
a first pass.

Although I love TiddlyWiki's save-in-place mechanism (now slightly
altered for me by running with Node, but still awesome), this has little
to do with my reason for using TW in this case.  The point was the UI
that TW presents: the microcontent, the ability to have open just the
content I want, the ability to link to that collection of open tiddlers.

Your notion, though, is making me think again.  I was considering TW
only as an output format that I generated from server-side markdown
files.  Inverting that so that the microcontent is the main source of
truth would probably be easier to do.  The trouble is that this library
will be maintained by a reasonably large community, and I also really
like the idea of a workflow that involves something like: change one
test file, one source file, and one markdown documentation file.  Most
people working with this will be familiar with the standard way of
doing things in GitHub, which means Pull Requests from one's own copy of
the markdown files to the central one.  That doesn't seem to mesh well
with using TW as the source of truth.

So I'm torn.

 
> To your specific question about whether the documentation files might
> be sliced into multiple tiddlers  as part of the process of threading
> them together, I suspect that that might make it a little bit harder
> to make a round trip editor. There’s also the question you raise about
> whether chunking into sections will actually lead to reusability.

The thing is, there are parts of this I definitely want at some point to
be able to reuse.  For example, I really would like to be able at some
point to search the Signatures section.

I don't know if those Hindley-Milner-inspired signatures mean anything
to you, but to those using libraries like this one (or like Ramda or
Sanctuary) and to those from many FP languages such as ML and Haskell,
this is simply how functions are described.  Haskell, whose
documentation overall does not impress me, has one killer documentation
feature: Hoogle.  This API search feature lets you specify a signature
and it finds all the functions that match, in various ways.  A search
results page for `(x -> y) -> [x] -> [y]` not only returns the `map`
function for lists (`map :: (a -> b) -> [a] -> [b]`), but also for more
generic types (`fmap :: Functor f => (a -> b) -> f a -> f b`) and more
specific related but not quite compatible types (`all :: (a -> Bool) ->
[a] -> Bool`).  You can see these and many more at

    <https://www.haskell.org/hoogle/?hoogle=%28x-%3Ey%29-%3E%5Bx%5D-%3E%5By%5D>

I think I might be able to make something like this, but only if I have
a way of distinguishing the signatures from the rest of the the
documentation.

And there are a few other similar ways I might want to use this
microcontent.

> Perhaps there’s more scope for reusability by centralising definitions
> and descriptions, and transcluding them at the point of use?

I would definitely centralize what I could.  But that sample I supplied
before was all specifically to do with the function `map`.  Like with
Ramda, there are some core concepts that might be mentioned many places,
but those would probably be with links.  The point was not that these
API pages would use external content, but that other documentation might
use pieces of these pages.

> Anyhow, it’s all quite ambitious, but one could build it in stages:

> * An MVP to get things started might be a tool that scans a repo for
>   documentation files and does the most basic aggregation of those
>   files as tiddlers into a TiddlyWiki that includes a plugin
>   containing the templates needed to view and interact with those
>   tiddlers

Yes, that would definitely be my first pass.  And doing that does not
concern me at all.  I think I can do this quite easily with Node.  I
can also split the content the way I was suggesting; I'm just still
not sure whether doing so would be a good idea.

> * Next, one might try to get things working under Node.js so that the
>   documentation files can be edited within TiddlyWiki. There would be
>   a startup process reading the tiddlers from the repo, and then
>   passing them to the syncer/filesystemadaptor so as to have future
>   changes to those tiddlers synced back to the original file

This sounds much more complex.  I'd have to think about this quite a
bit.

> * Finally, one could explore reusing some existing open source
>   documentation templates for output

To my mind, I would only design two ways to view the documentation:
The TiddlyWiki would be exposed and viewable, perhaps both in a
read-only mode for most and an editable mode for the maintainers.  And
the plain markdown files would be part of the repository, viewable
through the GitHub UI or locally through whatever Markdown viewer you
choose.

----------

I guess there is one other possibility that I hadn't really thought
about.  If the GitHub markdown is the source, I could generate tiddlers
with a newly-created content type, and move all the structured data from
the text of the tiddler into data fields.  That would presumably make it
easy to add the sort of searching I want and to reuse the content
elsewhere.  Are there downsides to this that I'm not seeing?


Cheers,

  -- Scott

[TonyM]

Alot of this is over my head, but can I suggest in your readonly view you make use of the new comments plugin in 5.1.18 or a version of it so than non editors can comment on any tiddler and send those comment tiddlers to an editor.

Once I see the final comnent plugin I plan to build an import tool for editors to process imported comments against each tiddler.

Tony

[Scott Sauyet]

TonyM wrote:

> Alot of this is over my head, but can I suggest in your readonly view
> you make use of the new comments plugin in 5.1.18 or a version of it
> so than non editors can comment on any tiddler and send those comment
> tiddlers to an editor.

Thank you.  While this wouldn't be necessary for my first use of this
technique in a corporate environment -- everyone using this library
would probably just email my team with comments -- if it works well, I
would like to extend it to a public library I'm starting to work on.
There, a comment system would be wonderful.

> Once I see the final comnent plugin I plan to build an import tool for
> editors to process imported comments against each tiddler.

I know nothing about any comment plugins.  I see the one in the TW
repository [1].  Is that the one you're talking about?  I have some
concern about this: "Comments are attributed to the username stored in
the system tiddler [[$:/status/UserName]]."  I wouldn't expect users of
such documentation to be logged in in any given way.  But if they were
to log in, I would assume it would be through GitHub integration.  I
wonder if that could work.

In any case, thanks for the pointer!


  -- Scott

  [1]: <https://github.com/Jermolene/TiddlyWiki5/tree/master/plugins/tiddlywiki/comments>