Import website as tiddlers

[Diego Mesa]

Hello,

I frequently check Cosma Shalizi's "Notebooks": 

http://bactra.org/notebooks

I find myself copying links from them, and copying/pasting etc. I thought it would be excellent for me to "import" each notebook into its own tiddler in my wiki, but I have no idea how to accomplish this. For one, Im using the single file version, which I like. Second, I'd have to change all of the <mathjax> calls to "$$". Third, I'd have to change all links like "bactra.org/noteboks/***" to be [[***]]", but keep all other bactra and non-bactra links intact. Fourth, I'd have to develop this script in such a way as to be able to be periodically run and import new changes to notebooks, while preserving the changes I've made to the tiddlers in the meantime. 

This seems like quite a task. Does anyone know of anything similar done in the past that I can use as a starting off point?

Thank you,

Diego

[Diego Mesa]

In case its useful to anyone, below is some regular expressions to turn (at least some) of the local-relative links to tiddly pretty links:

import re


regex = r"<a.+?href=\"(?!https?)([^\"]+?)(?:\.html)?\"[^>]*>([^<]+)<\/a>"


test_str = "I want to turn <a href=\"local-link.html\">title</a> to a particular wiki format: [[title|local-link]]"


subst = "[[$2|$1]]"


# You can manually specify the number of replacements by changing the 4th argument
result = re.sub(regex, subst, test_str, 0)


if result:
    print (result)

See explanation here: https://regex101.com/r/vkN20z/3

[@TiddlyTweeter]

That's lovely regex.

In my opinion we should be able to do it in TW eventually.

It needs work to develop a framework to cope with bulk.

However BJ's FLEXITY plugin (a version of his Flexitype Plugin that allows arbitrary regex) is a great step in the right direction.

http://flexibility.tiddlyspot.com/

Best wishes
Josiah