Are the downloadable TWs from tw.com assembled at download or pre-made?

[Mat]

When clicking Download on empty or the populated tw.com version or an edition etc, how is this file actually "served"? Is it a pre-made copy that is downloaded or is it assembled, presumably using nodejs, from individual tiddlers and then downloaded?

I'm asking because I'm curious if there is an opportunity to introduce customization already at this stage. For instance let the user select components and then click download, upon which assembly takes place.

Thanx

<:-)

[Ben H.]

The download link on tiddlywiki.com is for a pre-compiled html file. When you use the "export all tiddlers" option to save as html, that compiles the current wiki in its current state. By default, node will also save all changes to disk.

You can setup a template html file, load it, make any customizations, and then export the changed wiki as its own html file.

[PMario]

hi everyone,
 
please don't use tw(.)com in posts. It creates a link, that has nothing to do with tiddlywiki.com

-m

[PMario]

The $:/editions/tw5.com/download-empty contains a filter, that is used by: http://tiddlywiki.com/#%24%3A%2Feditions%2Ftw5.com%2Fsnippets%2Fdownload-empty-button

The download full version uses a different filter. So if you modify the filters, you can create customized downloadable versions.

mario

[Mat]

Ben, Mario

- thank you for your answers! (And I will from now on write twcom! Good point Mario)

My thoughts arose apropos Erwans tw-aggregator - i.e if it would be possible to let the user design his own version before downloading it. Your replies make me think this might well be possible.

A bit of fantasy:

...but first I should mention that the only instance currently available of tw-aggregators is of course Erwans own so the following thoughts, while based on a tw-aggregator mechanism, are not particularly about his personal instance.

It seems reasonable to let the user filter+manually select tiddler titles in a community aggregator... and then have the result be used as a download filter somewhere!  To get a mere filter would be possible even if an aggregator is set up like Erwans to only display "skinny tiddlers" (i.e not the actual tiddlers). 

The filter could then be used for the downloading, like pmario mentions.

The question is where to download from and, where the current aggregator work is performed, i.e to split up the downloaded TWs and then concoct the good parts into the desired TW. Maybe some server hosting an aggregator could offer a temporary space for this baking to take place.... 

Anyway, I brought up part of this issue on the tw-aggregator github.

<:-)

[Erwan]

Hi Mat,

On 15/09/15 22:53, Mat wrote:

The question is where to download from and, where the current aggregator work is performed, i.e to split up the downloaded TWs and then concoct the good parts into the desired TW. Maybe some server hosting an aggregator could offer a temporary space for this baking to take place.... 

I don't understand everything, but I think there is an easy answer to this particular question, *assuming*:

1) that the user doesn't mind waiting a long time "for the cake to bake" (delayed service)

2) that the service is not abused or overloaded

The way I see it, a request would be processed when the update takes place (e.g. you might have to wait up to 24h in the current setting), and the resulting wiki would simply be created like the CommunitySearch wiki on github (with a predetermined name indicated in the request, I guess). Basically github already plays half the role of a server in my system: the computation takes place on a local computer which transmits the result to github at the end, from where users download it.

Erwan

[Mat]

  Hi Erwan

Again, I wasn't talking about your particular instance (simply because it'd be rude of me to put some kind of "public pressure" on you personally. You're incredibly generous in creating this to begin with!). So, regarding the "delayed service", in another instance of a tw-aggregator this might not be an issue. For example, one can imagine a user triggered initiation.

Another idea is for users to have a personal aggregator set up so it does things locally. I don't know what this would require but I'm guessing something akin to those bash scripts should be functional also locally. Then a user would go to the community aggregator to produce the filtered list of updated titles and then use this list locally. Or he'd download the full skeleton list and create his filter locally.

Yes, talking is easier than coding ;-)

...

Another fantasy; if those user created filters are saved, they could be analyzed to eventually get a picture of which "combos of tiddlers" are popular. And if the users provided some meta-data info such as what the generated TW is intended for, we would have indications for what should be premade "suggested editions". For example, the combo "empty+pluginX+pluginY+themeZ" meta-tagged "collections" and "wines" is a basis for a "wine connoisseur" edition.

<:-)

[Erwan]

Again, I wasn't talking about your particular instance (simply because it'd be rude of me to put some kind of "public pressure" on you personally. You're incredibly generous in creating this to begin with!).

Thank you for your kind words, but technically my contribution is very small compared to what Jeremy, the other contributors and the plugin developers do!

So, regarding the "delayed service", in another instance of a tw-aggregator this might not be an issue. For example, one can imagine a user triggered initiation.

Another idea is for users to have a personal aggregator set up so it does things locally. I don't know what this would require but I'm guessing something akin to those bash scripts should be functional also locally. Then a user would go to the community aggregator to produce the filtered list of updated titles and then use this list locally. Or he'd download the full skeleton list and create his filter locally.

Even though I agree that the idea of a personal aggregator is indeed completely feasible (btw anybody on linux can try mine), In my opinion the creation of a customized TW is not an application worth the bother of installing this kind of program for a user, even assuming a good packaging and user-friendly interface. To me this is typically much more of a server-side service, like Andreas' customizer that you mentioned (http://twguides.org/customizer.html). I said that with my current aggregator it would be delayed simply because I don't have a real server, but otherwise the creation process is certainly fast enough to be carried out on demand (as opposed to the aggregation process with multiple wikis, which is harder on the CPU and bandwidth).

Yes, talking is easier than coding ;-)

...

Another fantasy; if those user created filters are saved, they could be analyzed to eventually get a picture of which "combos of tiddlers" are popular. And if the users provided some meta-data info such as what the generated TW is intended for, we would have indications for what should be premade "suggested editions". For example, the combo "empty+pluginX+pluginY+themeZ" meta-tagged "collections" and "wines" is a basis for a "wine connoisseur" edition.

This is another good reason why the process should be done on the server side. I'm not sure however that a lot of users would bother filling the meta-data info... or even know exactly what they want and how to describe it! ;)

Erwan

[Tobias Beer]

Hi Erwan,

I think a local version of (something like) your aggregator has quite more potential than "just" being the basis for a great community resource.

With a basic ui, even in TW, maybe some extension to TiddlyDesktop, users could define feed collections in the context of which they add channels, essentially other TiddlyWikis, e.g. from a predefined list like the ones indexed by community search, or add new channels, or delete / deactivate others to ones interests.

Then all you need to know is the right commandline, perhaps even OS integration via some addon, that lets you pull updated contents onto your local machine, at the desired location corresponding to the collection, e.g. "dev resources", "my school project", custom to your preferences, e.g. all created in the last two weeks. No need for a server at all. All you want and get is custom TiddlyWiki "news collections" being the result of your aggregators output.

One step would be to leverage TiddlyWiki's save or export mechanisms to export the collection / subscription lists and preferences so that a given commandline or other executable environment can pick it up and construct the output.

Best wishes,

— tb

[PMario]

On Tuesday, September 15, 2015 at 11:53:58 PM UTC+2, Mat wrote:

- thank you for your answers! (And I will from now on write twcom! Good point Mario)

imo use tiddlywiki.com .. not relly much more to type, but that's the name
-m

[Erwan]

Hi Tobias,

Yes, I agree that it would make sense for the kind of application that
you describe.

I only meant that for this application alone (generating a customized
edition) it seems like an overkill, since I don't see any big technical
difficulty in doing this server-side.

> One step would be to levererage TiddlyWiki's save or export mechanisms

> to export the collection / subscription lists and preferences so that
> a given commandline or other executable environment can pick it up and
> construct the output.
>

I think this is more or less what my aggregator does already:

1) the wiki addresses are defined in the "skeleton" wiki (as node.js,
but anyway conversion can be done both ways)
2) a command asks the node.js TW to render a list of the wiki tiddlers
which match a filter (e.g. those for which the author agreed to have
their content indexed); this step could be done from .tid files as well.
3) download these wikis as standalone html
4) convert the individual wikis to node.js
5) process tiddlers as .tid files, build the output wiki
6) convert the output wiki back to standalone html

I don't see what is missing in TW functionality for this kind of
scenario, it's already perfect :-)
The only minor issue is that any command-line call to tiddlywiki is
quite long to execute, I assume that this is because some node.js
initialization must take place every time (?).

Regards
Erwan

[Tobias Beer]

Hi Erwan,

 

I don't see what is missing in TW functionality for this kind of
scenario, it's already perfect :-)

You sure have done a great job here, no question. :-)

What I was suggesting was, that if this ever became some sort of desktop application,

it would be good to not only have a single "channel subscription list" extracted from tiddlers from which to construct,

but to rather be able to define "collections" or "interests", i.e. different sets of subscriptions for different purposes,

e.g. one for the latest "Plugin News" and another for "My Project Team Wikis",

which could, theoretically, even overlap.

Not sure if you're familiar with it, but think of "interests" in Facebook.

There you can add users or pages to special subscription lists for which you can then see a dedicated news feed.

Best wishes,

— tb

[Erwan]

Hi Tobias,
 
With Mat we had a discussion on github which is related to this kind of idea: https://github.com/erwanm/tw-aggregator/issues/77

It would indeed make sense to have this as a local application, but first I'd like to try something based on reading input parameters from the indexed wikis, and creating the collection/channel/customized wiki as usual, either as a specific tiddler in the community-search (so that it can be bookmarked) or maybe even as an independent wiki. The idea in this case is that many users can be interested in following the same "channel".

It seems to me that there is no big technical obstacle with this kind of feature/application, however the exact design is not clear at all in my mind: for example what would be the input parameters that a user can choose? set of wikis, tags, filters, ... How are these parameters represented? How is the result represented? As a tiddler in the CommunitySearch wiki (similar to the tiddler created for every tag), in another custom wiki... There are many possibilities and I don't know what is best, so of course suggestions are welcome!

Erwan

--
You received this message because you are subscribed to the Google Groups "TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tiddlywiki+...@googlegroups.com.
To post to this group, send email to tiddl...@googlegroups.com.
Visit this group at http://groups.google.com/group/tiddlywiki.
To view this discussion on the web visit https://groups.google.com/d/msgid/tiddlywiki/5f9a258b-99c7-4bd7-b482-029ab99f22a5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Tobias Beer]

Hi Erwan,

First of all, thanks for your amazing contribution. Needs to be said, again, and again. :-)

 

There are many possibilities and I don't know what is best, so of course suggestions are welcome!

In general, I would design that application / workflow in a way that everyone could use it in their environment just the way you do ...however, easily customized to personal preferences.

As I see it, right now, you have something like the following workflow...

1. you maintain some sort of "channel-list" / "subscription list"
   
2. you maintain certain filters, either global or with respect to every channel
   
3. via commandline you batch-download all channels
   
4. via commandline you extract a channel's data according to either a global or per-channel filter or both into tid files
5. via commandline you build an output wiki from those tid files alongside a set of dedicated template components

All that is great and working.

Any required options can be associated with the corresponding step above. Right now, you are possibly the only one who knows what needs configuration or what can be configured and how. Unless you already have, you can start by publishing a minimal documentation on what you (can) do, at the moment, grouped by step... and then we can give feedback on how that can be utilized or enhanced or whichever workflow to best construct around all that. Parameters should be simple text-based, preferably tiddlers. No fancy UIs or anything... except for what TiddlyWiki delivers out of the box. But maybe all that doesn't even need TiddlyWiki in the beginning, just plain text-editing.

One thing I would like to see around all this is the ability to adapt the entire workflow to "collections", i.e.  being able to do 1..5 for different collections of configurations rather than just one. So, for any "collection" you could do 1..5 independently from other collections, perhaps with a fallback to some global defaults.

Eventually, you may have...

      6. via commandline you execute the update process for all collections at once

It would be good if we can reserve the term "collection" or something equivalent for meaning that: running the aggregator process against a distinct set of necessary configuration options. So, right now, your aggregator only caters for one "collection", being an "all-subscriptions", "all-channels" aggregator.

Although later on of interest, what I would not discuss or develop right now is some sort of community process / workflow to design "shared aggregator configurations". What I can perhaps see is that you wish to expose your configuration for "community search" or "community news" more so that others know where and how to contribute. I don't know if that needs actual development, rather than communication / documentation / a suitable workflow.

Best wishes,

— tb

[Erwan]

Hi Tobias,

Thank you for this, that helps me greatly with getting a more general perspective.

In fact, I originally developed the aggregator without thinking further than aggregating content for users to search in it. It's only recently that I slowly started realizing that the main idea can be used for various applications. So currently it's difficult to distinguish the parameters from the process, in particular in my code where some parameters are hard-coded, simply because they didn't make sense as "parameters" in the original application.

But even if it takes a long time, my brain is slowly integrating this potential for various applications. this is actually the reason why I decided to separate the aggregator code from the community-search wiki: although the former is still 99% about generating the latter, it's now not the sole purpose anymore.

Regards,
Erwan

On 20/09/15 16:54, Tobias Beer wrote:

Hi Erwan,

First of all, thanks for your amazing contribution. Needs to be said, again, and again. :-)

 

There are many possibilities and I don't know what is best, so of course suggestions are welcome!

In general, I would design that application / workflow in a way that everyone could use it in their environment just the way you do, however easily customized to personal preferences.

As I see it, right now, you the following workflow...

1. you maintain some sort of "channel-list" / "subscription list"
   
2. you maintain certain filters, either global or with respect to every channel
   
3. via commandline you batch-download all channels
   
4. via commandline you extract a channel's data according to either a global or per-channel filter or both into tid files
5. via commandline you build an output wiki from those tid files alongside a set of dedicated template components

All that is good and working. The required options would correspond to each step. Right now, you are possibly the only one who knows what needs configuration or what can be configured and how. Unless you already have, can start by publishing documentation on what you (can) do, step-wise, at the moment... and then we can give feedback on how that can be utilized or enhanced or whichever workflow to best construct around all that. Parameters should be simple text-based, preferably tiddlers. No fancy UIs or anything... except for what TiddlyWiki delivers out of the box. But maybe all that doesn't even need TiddlyWiki in the beginning, just plain text-editing.

One thing I would like to see around all this is the ability to adapt the entire workflow to "collections", i.e.  being able to do 1..5 for different collections of configurations rather than just one. So, for any "collection" you could do 1..5 independent from other collections, perhaps with a fallback to some global defaults.

Eventually, you may have...

     6. via commandline execute the update process for all collections at once

It would be good if we can reserve the term "collection" or something equivalent for meaning that: running the aggregator process against a distinct set of necessary configuration options. So, right now, your aggregator only caters for one "collection", being an "all-subscriptions", "all-channels" aggregator.

Although later on of interest, what I would not discuss or develop right now is some sort of community process / workflow to design "shared aggregator configurations". What I can perhaps see is that you wish to expose your configuration for "community search" or "community news" more so that others know where and how to contribute. I don't know if that needs actual development, rather than communication / documentation / a suitable workflow.

Best wishes,

— tb

--
You received this message because you are subscribed to the Google Groups "TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tiddlywiki+...@googlegroups.com.
To post to this group, send email to tiddl...@googlegroups.com.
Visit this group at http://groups.google.com/group/tiddlywiki.

To view this discussion on the web visit https://groups.google.com/d/msgid/tiddlywiki/5a9c2bfe-211c-4736-ace3-65e59fba7859%40googlegroups.com.