Fetching tiddlers from one wiki to another

[Saq Imtiaz]

I've been contemplating a project where if I were to use TiddlyWiki, I would need to be able to fetch content from other wikis on startup, and preferably without needing to have extra plugins installed on the wiki that is the content source.

I've created a quick and messy proof of concept. This has only a very narrow use case in that it requires that both wikis either be on the same domain, or that the wiki that is the source of content be served with CORS headers. I am sharing it in case anyone else finds it of interest.

Warning: this is very raw and incomplete. I spent about 40 minutes hacking this into place - with a lot of déjà vu moments thinking of working with FND and prototyping a plugin update system back in the TW 2.x days - and then maybe just as long preparing a demo:

https://saqimtiaz.github.io/sq-tw/temp/get-remote-tiddlers.html

What does make this a little exciting is that Github pages supports CORS and with a little luck we might get CORS support at TiddlyHost too. So it opens the door to some of the workflow that was commonplace in the TWC days, like auto-importing content, updates to certain tiddlers, plugin auto updates etc.

In preparing the demo I also realized that it would be useful to be able to bypass the cache when fetching the content. 

Cheers,
Saq

[Mohammad Rahmani]

Hi Saq,

 This is very exciting and opens a lot of new windows for Tiddlywiki.

One question: In Searchwikis (https://github.com/kookma/TW-Searchwikis) it uses index tiddlers from target wikis to search them

The question is: Can I use Get Remote Tiddlers plugin to automatically update the indeed tiddlers in my main wiki i.e to fetch latest changes?

A more radical question is can " Get Remote Tiddlers " deprecate searchwikis and lets users search other wikis from a central wiki?

Best wishes

Mohammad

--
You received this message because you are subscribed to the Google Groups "TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tiddlywiki+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/tiddlywiki/20ade4fa-4f1e-4569-8133-3bf3cd4ad1e4n%40googlegroups.com.

[Saq Imtiaz]

Hi Mohammad,

So a lot of things are still up in the air. As I mentioned this is a quick prototype without a lot of pre-planning and I haven't had the chance to figure out a good workflow yet even for my specific needs. I'll try to answer your questions as well as I can at this point, but some things we may need to discuss and figure out together.

 

One question: In Searchwikis (https://github.com/kookma/TW-Searchwikis) it uses index tiddlers from target wikis to search them

The question is: Can I use Get Remote Tiddlers plugin to automatically update the indeed tiddlers in my main wiki i.e to fetch latest changes?

Yes that should be possible as long as the wikis are on the same domain or served with CORS headers. The catch is that for every wiki, the code will be fetching the entire TW file to extract the tiddlers, which means more bandwidth used and also possibly some lag in the wiki depending on the number of wikis being fetched and the hardware. I want to experiment with using the same technique with an uglified TW with Flibbles' plugin, or a TW with the core in an external js file. Especially the second option should load considerably faster.

Another option that might make the most sense for Searchwikis is to generate and save the index as a separate file. This would need far less bandwidth to fetch and should overall be considerably faster. For wikis served by Github pages we could even try to implement an option to save the index file automatically each time the wiki is saved.

 

A more radical question is can " Get Remote Tiddlers " deprecate searchwikis and lets users search other wikis from a central wiki?

Fetching tiddlers from a remote wiki already entails loading the entire wiki and extracting all its tiddlers and creating an array of tiddler titles. So yes, theoretically you could skip the part of Searchwikis that creates the index and just use Get Remote Tiddlers to create the index for the other wikis. However, the same caveats apply in terms of bandwidth usage and that with many wikis being fetched in the background it could slow down the wiki when that is happening. Using TW files with an external core.js file would be very helpful here.

Having to rely on CORS headers being present is indeed a limitation but this overall approach (rather than the code I posted specifically) opens some interesting possibilities as well.

If you're interested, there's a small discussion here about enabling CORS on TiddlyHost: https://github.com/simonbaird/tiddlyhost/issues/112

I am also curious to see if TW on Fission opens up for similar workflows.

Cheers,
Saq

[Mohammad Rahmani]

On Sun, Apr 4, 2021 at 1:38 PM Saq Imtiaz <saq.i...@gmail.com> wrote:

Hi Mohammad,

Hi Saq,

So a lot of things are still up in the air. As I mentioned this is a quick prototype without a lot of pre-planning and I haven't had the chance to figure out a good workflow yet even for my specific needs. I'll try to answer your questions as well as I can at this point,

 

but some things we may need to discuss and figure out together.

Sure!

 

One question: In Searchwikis (https://github.com/kookma/TW-Searchwikis) it uses index tiddlers from target wikis to search them

The question is: Can I use Get Remote Tiddlers plugin to automatically update the indeed tiddlers in my main wiki i.e to fetch latest changes?

Yes that should be possible as long as the wikis are on the same domain or served with CORS headers. The catch is that for every wiki, the code will be fetching the entire TW file to extract the tiddlers, which means more bandwidth used and also possibly some lag in the wiki depending on the number of wikis being fetched and the hardware. I want to experiment with using the same technique with an uglified TW with Flibbles' plugin, or a TW with the core in an external js file. Especially the second option should load considerably faster.

I did not know the whole wiki is fetched, then yes for many external wikis or large wikis we will have some time lags! I use the Flibbles uglify, I am very happy with that, but for this case still the empty.html is around 1.2MB, the external.js seems promising here!

Another option that might make the most sense for Searchwikis is to generate and save the index as a separate file. This would need far less bandwidth to fetch and should overall be considerably faster. For wikis served by Github pages we could even try to implement an option to save the index file automatically each time the wiki is saved.

Great! Right now I have to open each target wiki separately, update the index, export it, and import to the central wiki!

 

A more radical question is can " Get Remote Tiddlers " deprecate searchwikis and lets users search other wikis from a central wiki?

Fetching tiddlers from a remote wiki already entails loading the entire wiki and extracting all its tiddlers and creating an array of tiddler titles. So yes, theoretically you could skip the part of Searchwikis that creates the index and just use Get Remote Tiddlers to create the index for the other wikis. However, the same caveats apply in terms of bandwidth usage and that with many wikis being fetched in the background it could slow down the wiki when that is happening. Using TW files with an external core.js file would be very helpful here.

Having to rely on CORS headers being present is indeed a limitation but this overall approach (rather than the code I posted specifically) opens some interesting possibilities as well.

If you're interested, there's a small discussion here about enabling CORS on TiddlyHost: https://github.com/simonbaird/tiddlyhost/issues/112

I will!

I am also curious to see if TW on Fission opens up for similar workflows.

Cheers,
Saq

Thank you Saq!

--
You received this message because you are subscribed to the Google Groups "TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tiddlywiki+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/tiddlywiki/a30adfd0-8d9f-42d8-aea6-e0824c777db6n%40googlegroups.com.

[Ste]

Jumps in, mentions twederation, jumps out again. 

[Saq Imtiaz]

Thanks @Ste

 I briefly looked at twederation but it didn't meet this particular requirement which would be very helpful for my intended purposes: "...preferably without needing to have extra plugins installed on the wiki that is the content source".

It also wasn't clear to me from the ReadMe or the demo whether fetching tiddlers silently in the background with no user interaction was possible, which is a must for the workflow I will need.

This is really just a quick hack to understand if what I am envisioning is technically possible before I start working out the workflow. Now I need to find the time to flesh out the desired user workflow which is the real challenge and then I'll return to the technical implementation, which may involve use of the core syncer. I decided to share this demo in case it was interesting for others.

 
Cheers,
Saq

[Jeremy Ruston]

Hi Saq

 I briefly looked at twederation but it didn't meet this particular requirement which would be very helpful for my intended purposes: "...preferably without needing to have extra plugins installed on the wiki that is the content source".

It also wasn't clear to me from the ReadMe or the demo whether fetching tiddlers silently in the background with no user interaction was possible, which is a must for the workflow I will need.

It did indeed fetch without user interaction. The Twederation prototype was all about using iframes to get around CORS restrictions.

Those restrictions still exist. For example, the XMLHttpRequest approach described here generally won’t work from a file: URI, which was a major design goal for the plugin library and Twederation.

This is really just a quick hack to understand if what I am envisioning is technically possible before I start working out the workflow. Now I need to find the time to flesh out the desired user workflow which is the real challenge and then I'll return to the technical implementation, which may involve use of the core syncer. I decided to share this demo in case it was interesting for others.

It’s definitely time we explored dynamic content loading via XMLHttpRequest in more detail. The constraints get less onerous as we gain better HTTP/HTTPS solutions.

Best wishes

Jeremy

To view this discussion on the web visit https://groups.google.com/d/msgid/tiddlywiki/ea4c8e50-7e7d-40b5-9a28-7906d7cee2ban%40googlegroups.com.

[Saq Imtiaz]

Hi Jeremy,

A little background: I have a couple of ideas in mind for projects to do with education with some of the organizations I work with, for which I am considering TiddlyWiki. One of them would be very similar to the syncing educator and student notebooks project that was based on TiddlyWiki classic over 10 years ago. The other involves providing a sandbox for exploration and developing skills to do with using and potentially creating/customizing digital tools to aid one's own learning. While TiddlyWiki sounds like a good fit, it will also be essential that it is hard for the students to break the tool itself.

Both projects, if they proceed, will entail quite a bit of custom coding to get the user experience just right, so as a first step I wanted to understand what was possible today in terms of browser restrictions and loading content dynamically in TW5. As mentioned in my original post, I realize the use case for this particular prototype is very narrow. However I find that sharing ideas and prototypes is rarely a bad idea as it can often inspire other ideas in the community, so I try my best to do so whenever possible.

 

Those restrictions still exist. For example, the XMLHttpRequest approach described here generally won’t work from a file: URI, which was a major design goal for the plugin library and Twederation.

That's interesting as I've found it to work without problems to fetch content into a local file wiki as long as the remote content is served with the correct CORS headers (and the request is made without the X-Requested-With header). Or do you mean when the remote content is also accessed via a file: URI ? I have not tested that scenario.

Hosting all the wikis with CORS support is an easy requirement to satisfy for the projects I have in mind.

 

It’s definitely time we explored dynamic content loading via XMLHttpRequest in more detail. The constraints get less onerous as we gain better HTTP/HTTPS solutions.

Agreed. I quite miss the possibilities offered by the quite rich ecosystem of adaptors that we had in TiddlyWiki Classic and even the sync mechanism, despite all the clunkiness inherent in the design and the later difficulties imposed by greater browser restrictions. 

 

Regards,
Saq

[Jan Johannpeter]

Hi Saq,

thank you very much for this widget! This is a great help, I have looked for something like this for a long time. 

This is a very important puzzlepiece to use TW as an interactive Learning Management System. 

For this usecase it is great that it has a silent mode, which makes it mor unobstrusive than the fetch-function of Twederation.

For the interactity of an LMS I would love three other features:

-Can it handle filters to import only certain tiddlers? So far I could not see how to implement them in the action-widget.

-It would be great if it could tag them imported tiddlers like "temp" or "imported"

-It would be very important to have an "Overwrite Filter" determining which tiddlers can be overwritten by the imported tiddlers. (For example al those whose modifier is not the current user).

Best wishes  and nany thanks for this great improvement!

Jan

--

You received this message because you are subscribed to the Google Groups "TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tiddlywiki+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/tiddlywiki/7ea4cca2-c07a-486d-a60c-b59c9838c0bdn%40googlegroups.com.

[TW Tones]

Folks,

I have long sought something similar, and wonder if the following approach would help with remote tiddlers, loading/importing and performance.

• Provide a utility to save filtered tiddlers (json) to another filename on the host, not index.html or wikiname.html, a bit like rss
  • This is the source wiki publishing a subset of tiddler for remote load, it can be added to the save process.
• Provide a plugin that given a url (inc file://) to the above address will silently import (although log it) the tiddlers found in the JSON file if they have changed, in the current wiki.

I am confident this is possible, I have seen the various requirements satisfied previously I just to not have the skills to code it, needing a saver etc... 

This same mechaisium could be used to generate multiple files to use a filter to publish a library. If necessary this can be done through a zip file, however it would be nice if the remote wiki could unzip and import. I am a strong believer that a library or importing remote tiddlers would be a far simpler way to intergrrating tiddlywiki than other suggestions so far, which with all due respect to those at the leading edge can be hard to follow.

My current workaround is a package Icon (tiddlers in a json format) that I can drag from one Wiki in an iFrame to another to install or update.

Tones

[Saq Imtiaz]

Hi Jan,

I'll reiterate from my first post that this is just a quick and messy POC with a very narrow usage window because of the CORS requirement.

Also note that apparently Twederation is supposed to be able to silently import content in the background as well, as pointed out by Jeremy above. If that works for your intended workflow, it may be a better choice for you as it wont impose the same requirement to serve the wiki files with the correct CORS headers.

The demo I posted does support using a filter to only import certain tiddlers. The other two requirements you mention are feasible though not implemented in the demo. I am still at the exploration stage of seeing what is possible, and need to work on the project proposal I have in mind which involves using TW in an LMS context too. If that goes well I will revisit this topic later this year.

Regards,
Saq

[Jean-Pierre Rivière]

I have got a look at https://saqimtiaz.github.io/sq-tw/temp/get-remote-tiddlers.html but in the "Interactive import" tiddler, the button reads "fetch Undefined widget 'action-getremotetiddler'" and does nothing. I tested it on a recent chromium.

I can't see any plugin or macro for the magic. And I'm not so sure at what is happening. As for me, I would like to be able to transclude tiddlers from other wikis, but not import them per se. Icbnnot see if that would be possible within the scope of this projects (and it should also be noted the alrady announced technical limitations liki CORS).

regards,

-- 

Jean-Pierre

[Saq Imtiaz]

 

I have got a look at https://saqimtiaz.github.io/sq-tw/temp/get-remote-tiddlers.html but in the "Interactive import" tiddler, the button reads "fetch Undefined widget 'action-getremotetiddler'" and does nothing. I tested it on a recent chromium.

That is pretty odd, it works for me in both current Chrome and FF on Windows 10. Do you see any error messages in the browser console?

 

 As for me, I would like to be able to transclude tiddlers from other wikis, but not import them per se.

That is not the aim with this POC.  TiddlyWiki does have facilities to load remote .tid files as tiddlers which might be closer to what you need. It makes use of the _canonical_uri field but has the same limitations regarding CORS.

[TiddlyTweeter]

Ciao Saq,

VERY interesting idea indeed!

ONE roadblock will be policy. 

I think it is a real issue.

WHAT IF a TW has a rule you CAN'T REUSE it contents in a web page?

HOW would you determine and honor that?

Best wishes

TT

[Jean-Pierre Rivière]

Thank you for your answer. Well, now the POC is working from this link with chromium and firefox, both on linux. 

If I get back the bug, I'll look at the console. I should have thought about it. I had tried a few things before discovering the "imteractive import" tiddlers but not this time.

As for loading tiddlers, to be honest, this is not the proper way to describe what I want. The tiddlers I want to show suppose a lot of macros and data tiddlers for their rendering. What I would like is getting a snapshot of their final html code within the requiring wiki, which would have imported the sylesheet beforehand. This being stated, I also have some plain tiddlers I would like to see. importing them within the $:/temp/ hierarchy would be fine. All those tiddlers are helpers for reference documentation. They are too numerous to include. Thiy must be seen on an ad hoc basis and get ridden off ASAP.

[Jan Johannpeter]

Hi Saq, 

I think that your plugin is great for educational scenarios - and there CORS won't be a problem because as student's  and teacher's wikis are likely to be hosted in the same domain. 

I found the $filter in the widget - thanks for the hint.

I would like to build / use a set of wikis where  

-the teacher (me) has a wiki has a wiki which is serving material and tasks. 

-the students work on the tasks an build a portfolio doing this. There your plugin would be great to import the tiddlers with the current material

-the teacher has a anotherwiki to automatically collect the tiddlers marked to be sent in as solved tasks. (Again your plugin could be very usefull.)

(What is still missing in my thought is an easy way for the sutdents to interact in this construction)

So thanks for helping me with your valuable tool.

Jan

I found the way to implement a filter which is great. 

[Ste]

This is sounding exciting!  All I need to do now is get my IT to host....:( :D