Why "sharing of tiddlers" is a linchpin for success

[Mat]

[Note: fellow Diego Mesa just started a thread that seems highly relevant for this.]

In another thread PMario wrote:

So users use what they know best, or they introduce a 3rd party platform [...] causes fragmentation! 

...and @Jeremy wrote:

In terms of submitting plugins to the library [...] we need an API 

Many people want to contribute and share stuff but we lack good infrastructure for it. An API makes much sense and/but this post is IMO about an even more fundamental matter for contributing. It is also about a remedy to the problems with 3'd party solutions:

The smallest common denominator for every TW community member is... TiddlyWiki! This is the only system where every potential contributor knows how to create "posts" and "code"; i.e tiddlers! We should take advantage of this.

If we had a basic TW implementation for sharing of tiddlers, then this would be the LINCHPIN for alternative ways to:

• contribute code to e.g Github (probably via an API)
  
• co-develop TW stuff without e.g Github
  
• collectively build repositories (for e.g plugins)
  
• collectively build documentation
  
• discuss TW stuff
  
• discuss non-TW stuff

If there was a TW implementation for sharing of tiddlers, then any member could create custom interfaces for these special areas of concern.

What is missing for implementing this? i.e to push or fetch or download individual tiddlers? Is there something fundamental beyond someone actually coding it that is missing? - I'm asking @Jeremy @Mario @Jed whom I know have pondered over these matters but also any other competent coder here.

And I'm curious as to why so few seem to think in these terms? What am I missing? Instead people keep suggesting 3d party platforms, seemingly disregarding the lessons from 10+ years of failures with that approach. And people keep creating heartfelt projects that require continuous curation by single individuals to be useful in spite of such projects failing time after time.

Side note: My own SetUp project is in deed such a "requires continuous curation" project but was, in fact, partially created as a helping guide for anyone eventually implementing such a "sharing of tiddlers" system, because the latter will likely involve such things as savers/servers etc. I'm not a coder so this is one of the few things I can contribute with in this matter.

Another side note: Most of you know that @Jed and I did some work on TWederation and Jed actually succeeded in creating a solution where people could fetch tiddlers from other peoples wikis. Unfortunately it did not scale well as it had to download all the (full) wikis that one "subscribed" to in order to pluck out the relevant tiddlers, so as more wikis connected the procedure quickly became too slow. This fundamental lack halted development so it never reached a stage where it would make sense to perfect the UI nor to promote it.

So, what are your thoughts on this? I'm particularly interested in hearing what actual coders have to say about it. In this unpaid project nobody can expect - not to mention demand - anything, but I'm wondering why this linchpin matter just seems to not get any attention?

<:-)

[Jed Carty]

I am working on the external server component needed by Bob to make it into the optional nodejs server component we had discussed as an extension of twederation. I have a publicly accessible website set up serving Bob right now and it works well enough to use, but I don't have the SSL certificates set up correctly yet (my brother is in charge of our domains and he hasn't had time to get me the proper ssl certificate so it is self-signed for now).

I am not sure if this counts as a basic implementation because it requires an external server and setting up SSL, but with the server in place we should be able to make plugins that let single file wikis connect to this server (or other servers running similar code) as a method of sharing tiddlers natively.

I have been in the US to get married so I have been working slowly. Also some other work is picking up so I don't know how quickly I will be working on any of this.

[@TiddlyTweeter]

Ciao Mat

I think concrete examples help.

My use case. I like to try promote  TW (though promotion is not on your list :-) on Twitter because its effective and Twitter I know well. There is an infinite listening audience there. I am doing 7,000% less than I could because every bl**dy TW resource I want to point to I have to research, find the address for documentation and the install,  know if the author needs acknowledging on Twitter (i.e. are on Twitter or not) etc, find them there, craft a post, check it.

PLEASE give me standardly laid out tiddler for every resource worth posting about ...

1 - download address
2 - documentation address(es)
3 - support address
4 - author address(es)
5 - data of release or version # if available
6 - description

Then recycling info over Twitter would be a doddle and we could begin to dance it.

Ideally I'd like to generate a resource list from a Tiddler set of, say, 600 resources, output it to a Twitter robot poster and leave it get on with it for six months at a time. Then I can concentrate on responding to questions about them, not the hand-crafting of them.

As is its unviable. And a consequence is TW is not getting  the important exposure it deserves.

Right now I'm looking to use David Gifford's TiddlyToolmap, since its the most comprehensive list around. But that is huge work as the entries don't come up in a browser in any standard way (as there is NO standard way defined for listing a resource). That is NOT usable for a wider audience who need a standard approach so it doesn't get confusing. So I still have to check every entry and re-write it. That's too much to do often.

Regarding your own depiction of example scopes ... IMO I think its important to define the Final Usage of "Data" as precisely as possible in order to layout what you talking about in practice--down to the level of fields and whatnot. I think your list may be currently too broad...

Mat ...

• contribute code to e.g Github (probably via an API)
  
• co-develop TW stuff without e.g Github
  
• collectively build repositories (for e.g plugins)
  
• collectively build documentation
  
• discuss TW stuff
  
• discuss non-TW stuff

Delimited basic resource finding I suspect is critical across the board at the moment? (Not saying no to anything else, just trying to find a viable handle to work forward from.)

Just thoughts
Josiah

[@TiddlyTweeter]

Ciao Mat

Further to my last, and hopefully on-point.

IF I, say converted David's list, WHAT would be the best format (basically fields defined and filled for every Tiddler?) that is optimal for sharing? Where's the list :-)?

This immediately brings up the issue of legacy. IMO getting legacy a bit more ordered is a good step? The other route is automation "in-future" with smart new methods. But I do think the fragmentation of resources we have now IS a large part of what we have now so there needs to be some enfolding the old into a new approach?

Just thoughts
Josiah

[Diego Mesa]

I have long thought we need a way to upgrade plugins, which would require fetching tiddlers! I agree its super important! 

[Mat]

Jed Carty wrote:

I am working on the external server component needed by Bob to make it into the optional nodejs server component we had discussed as an extension of twederation. [...]

While pondering over how to interpret this I realized I have this basic question:

With a public TW based on "TW on node.js", or Bob, is it possible for a visitor to access individual tiddlers (.tid files) or is the output always an "integrated" wiki? I mean, can somehow individual tiddlers (or sets of tiddlers) be extracted directly from where they are served? I believe the now defunct TiddlySpace (based on TiddlyWeb) featured this, right? 

If this single tid extraction is possible then this would of course speed up fetching. Considering how Bob can convert single file TWs into individual tiddlers then this would in deed be very exciting!

For anyone reading this who is not familiar with TWederation: While the basic "federated wiki" idea is to allow individual wikis to fetch tiddlers from other peoples wikis, one additional idea - at least in TWederation - is to have a central server hosting a directory over all public wikis (as reported by the wiki owner or by anyone else). The server frequently roams all connected wikis for updates and can potentially fetch these updates and serve them on the server. The public directory also lets users directly find TWs to "subscribe" to directly, i.e without going via the server. The server could also function as what you see on the current TWederation tiddlyspot, i.e as a hub for e.g discussions - or for any of the bullets listed in the OP.

I am not sure if this counts as a basic implementation because it requires an external server and setting up SSL, but with the server in place we should be able to make plugins that let single file wikis connect to this server (or other servers running similar code) as a method of sharing tiddlers natively.

Well, if it enables "sharing of tiddlers" then maybe there is no need - or at least much less need - for what I refer to as the basic implementation. 

I have been in the US to get married so I have been working slowly. 

 

Hey! Congrats man! 

Also some other work is picking up so I don't know how quickly I will be working on any of this.

If the code is public (...and I guess somewhat documented/commented) then maybe others could fiddle with it...?

<:-)

[Mat]

TiddlyTweeter/Josiah

My bullet list is only to give a general idea of what should be possible. It is in deed broad but it is also very unspecific because not only could each bullet probably be implemented in several ways, it might also depend on how the sharing-of-tiddlers is actually implemented.

This should also answer your request for:

IF I, say converted David's list, WHAT would be the best format (basically fields defined and filled for every Tiddler?) that is optimal for sharing? Where's the list :-)?

...i.e; I have no idea. Nobody can know this yet. It may depend on how the sharing is implemented. It may also depend on what people want to do with it: For example, Twitter has that fixed number of characters limit so if, for some reason, sharing typically ends up as tweets then that would of course affect what is "best format". Just not possible to know at this stage.

<:-)

[@TiddlyTweeter]

Exactly why TW in the wild inclines to the "centrifugal".

The "centripetal" is MISSING. That lack maybe part causes the fragmentation you are talking of.

I don't think this is about waiting around more ... I mean how many years you (everyone reading) been here already?

GIVE me a format. Lets get on with it :-)

Josiah

[@TiddlyTweeter]

Mat: Nobody can know this yet.

I disagree. I DO think it is important a few people, TW "experts", examine it to make sure its broad enough. But to say "Nobody can know" is just silly.

 

Twitter has that fixed number of characters limit so if, for some reason, sharing typically ends up as tweets then that would of course affect what is "best format". Just not possible to know at this stage.

FYI, Twitter in now 280. It will fit fine. I'd make sure in my case. IF somebody else made another 500 I could use that were longer that 280 total, those I'd edit in Tweet. Easy.

It may also depend on what people want to do with it:

No. A definition of what data is essential to adequately capture a resource should be as wide as seems doable. That leaves end users free to do what they want. They don't define it. They use it.

As I said, lets get on with it.

Josiah

[Jed Carty]

I tried to start a listing of plugins that I was hoping to extend to other tiddlywiki resources that is similar to what you are describing but then I got distracted and I don't think anyone else really started using it. What I made is here: http://inmysocks.tiddlyspot.com/#Plugin%20twCard

Mat,

The idea is to set it up so that you can use the same filters as in our implementation of twederation and get just the tiddlers back instead of loading the full wiki. So it would take much less time and would let you get tiddlers from multiple wikis if they are on the server and it would be much faster than the non-server assisted way. You could set up something that serves wikis like this without ssl but I wouldn't trust a system for editing the wikis online without https.

[@TiddlyTweeter]

Jed Carty wrote:

I tried to start a listing of plugins that I was hoping to extend to other tiddlywiki resources that is similar to what you are describing but then I got distracted and I don't think anyone else really started using it. What I made is here: http://inmysocks.tiddlyspot.com/#Plugin%20twCard

Its a really good resource.

I agree that getting it really used likely needs other steps. Part of the issue is, I think, "what are we doing" -- I mean is this an exercise in one-off excavation? Or is it a viable, sustainable thing?

Part of MY interest is to document the mess (since David did it already). Get it over with and maybe help develop a shared data structure such that TW resources are more easily found.

One thing that goes on that is rarely mentioned is Erwan's aggregator. What I seen of it its quite a neat idea. Under subscribed and not that developed, but it works. I think there is mileage in the approach?

J.

[Mat]

Jed Carty wrote:

I tried to start a listing of plugins that I was hoping to extend to other tiddlywiki resources that is similar to what you are describing but then I got distracted and I don't think anyone else really started using it. What I made is here: http://inmysocks.tiddlyspot.com/#Plugin%20twCard

With the risk of sounding patronizing: A main idea is of course that individuals should not do those kinds of things - for the exact reason you state; we get distracted. Instead, that list ought be built by the central server, listing all fetched tiddlers that, say, have the tag "Plugin" (or, probably better, some form of meta-data tiddlers that extract data from fetched plugin tiddlers and then ditch the actual plugin tiddlers).

The idea is to set it up so that you can use the same filters as in our implementation of twederation and get just the tiddlers back instead of loading the full wiki. So it would take much less time and would let you get tiddlers from multiple wikis if they are on the server and it would be much faster than the non-server assisted way. 

 

From where would I actually fetch the tiddlers - from the actual wikis or from the central server? Because if it is from the actual wikis... then how do I extract individual tiddlers, and if it is from the server then will this mean the server will continually collect and keep copies of the full wikis? Or would the server fetch the wikis on demand and splice out the tiddlers that I can then fetch... which would not be fast.

Or maybe the server only keeps copies of tiddlers subscribed to as specified by e.g tags or some subscriber-defined filter... "subscribe to [tag[@mat]has[bar]]"...

You could set up something that serves wikis like this without ssl but I wouldn't trust a system for editing the wikis online without https.

If I recall, there was some conflict that http wikis could fetch from both http and https, but https wikis could only fetch from https. But was there also some issue that the current TWederation implementation could not handle https? ...or was that only because it was hosted on tiddlyspot (http)? Other than as a generally good tip, I am unsure how this relates to what we're discussing (and I don't mean that it doesn't, I just mean that I don't understand).

<:-)

[TonyM]

I would like to add something here and I hope I may be able to state it clearly.

• We can already effectively store and share tiddlers and groups of tiddlers, we can drag and drop, import and export. 
  
• It seems to me this thread is considering the "Ultimate" integrated transfer or subscription process" when 98% of what is required already exists.
  
• I am all for the ultimate solution but this is to me putting the cart before the horse, 
  
• In building my own tiddlyWiki network, to share and distribute plugins and other wikis I have used existing tools to do this 

It seems to me that a large part of this solution can occur at the wiki implementation level, without plugins or core changes.

Of course my implementation has limits but actually there are a few features that would allow it to become a full solution.  Here is a non exhaustive list of the top of my head.

• At TiddlyWiki load time checking for the existence of a named file such as a Json file and importing it,
• Use file time stamps to avoid doing it more than once until it changes.
• Provide a logout or exit process, or within the save process to export a json file with selected tiddler once a tiddler changes.
• Mark tiddlers as owned by a specific wiki, and allow edit in that wiki and avoid edit in other wikis.
• This entails a method to register wikis
• Using iframe embedding external wikis, works well to open a window and permit drag drop of tiddler such as plugtins between TiddlyWikis
• This however requires selecting import in the destination wiki
• Or still seeing the side bar etc.. when displaying a foreign tiddler
• Providing a way to open a Wiki with the focus on a single tiddler would help here.

Each of these features should be fairly easy to implement

If we then combined the above with a set of practices or plugin support people could then build loosely coupled networks of wikis with shared tiddlers.

In conclusion, a lot could be done in the implementation space with support from a few tools. This can then be extended with features in platforms such as Node, Bob to build highly scale-able tiddler distribution hubs.

 

Regards

Tony

[TonyM]

Post Script,

Allowing people to annotate or comment on tiddlers, perhaps part of this need to share tiddlers could also be achieved with NoteSelf using the local pouchDB to save changes, the user can then export their annotations and send an email, post etc... a json file to the site owner who can import them, it would take very little to design an alternate import process that allowed a difference review of each inbound tiddler, and an acceptance, rejection or hold response to each tiddler. And while this seems a manual process the fact is the user may not want it any other way because they wish to accept or reject the submitted tiddlers.

Regards

Tony

[Mat]

Tony

First, if you're up for it; let us (you and I) set up something together to see how far we can get. I could start with this in about a week or two from now. How about you? Anyone else interested?

But generally; I agree that what I am calling for is the, as you put it, "ultimate integrated transfer or subscriptions process" because that, actually, is what I think IS needed. It may well be that we're 98% there but then what I'm asking is what are those last 2%? What is missing? And why there is so little discussion about it? I believe it is the key to a new era of creativity in the TiddlyVerse but maybe I'm naive and disregard fundamental problems?

<:-)

[@TiddlyTweeter]

Mat: maybe I'm naive and disregard fundamental problems?

I think you are Hot On -- EXCEPT on the question of legacy. On which you don't seem to see the issue.

The future will come not just with a shiny new system (I am in favour of) it will also be because of a BRIDGE to the past.

My concern is HOW to document the past work (that is huge) so that it can be included. As IS FOUND now. Not requiring a user to change anything.

So I see this as about two things ...

(1) shiny new system

(2) a data STRUCTURE under which old stuff can be organised.

I hope this is clear.
Josiah

[Mat]

@TiddlyTweeter wrote:

I hope this is clear.

Honestly, it sounds interesting but I don't understand what you're talking about. Could you perhaps start another thread about it?

<:-)

[TonyM]

Mat,

I can loosely commit some time to furthering this. If I could ask you to Join Yammer at https://www.yammer.com/tiddlywiki and we continue the conversation in a group,

However I am sure a few of the above requirements will need some deeper developer skills.

Another thing I realised the other day is it is the individual TiddlyWiki, who wishes to subscribe to a source, thus the only place a subscription action need exist is there, all a server or core wiki needs is to serve the shared wikis. However we could share skeleton tiddlers which contain the url to where they are hosted. 

If we can trigger a recognition that a change has taken place in tiddler(s) that this wiki is subscribed to with low overheads then I think with a logical layer on top we will have it.

Regards

Tony

[TonyM]

Josiah,

I believe you are correct, but someone building a solution now that they can see the end of the tunnel. are possibly already informed by the past, but it is important that people with a "naive disregard [for perceived] fundamental problems" just have a go, and the wise and experienced should look on and help where they can.

Regards

Tony

[Jed Carty]

Tony,

Yammer isn't sending me a confirmation email so I can't join there.

The way that you describe subscribing to something is used by twederation. Subscribing is done completely on the subscribing wiki and doesn't have any effect on the wiki that is being subscribed to. And each wiki has a tiddler that describes the wiki information and that is all any other wiki needs to copy to subscribe to it.

All of this can be much faster now that I have am working on node version, but the general idea is the same as it was before. 

[TonyM]

Jed,

Sorry, the invitations are slow because I am asleep at the time. I sent it recently

Good to see the federation connection.

In relation to the node version, I just found I may be able to use hosting accounts to serve nodeJS and they are working on making it a more user configurable choice. I have a wholesale hosting service here in sydney.

Of Course I will try Bob on it, keeping in mind some potential security risks, I think Bob is potentially an excellent platform for a group of federated wikis but I am also interested in the universal form of federation as well.

Yes, I know a lot of water has passed under the federation bridge, but what I have read often confused me because of a lack of knowledge I am slowly resolving. In the mean time I see value in following my naive ideas first.

Regards

tony

[bingo]

Is the idea of shared tiddlers plugin that was part of TwC useful in this context ? It seems if we could get it to work in TW5 context it would also help with reducing the tiddlywiki file size and help organising information better ...