TiddlyWiki+Python

[Neil Griffin]

A little over a year ago, I discovered both TW and Python and since then, it seems that the solution to pretty much every problem has been to use one or other of them.  It was only a matter of time before I felt the need to get them working together.

As a result, I have written a little Python module that allows me to perform the essential functions of reading, searching and writing to tiddlers in the store area of a local single-file TiddlyWiki.  I thought this might be of interest to some others here, so I have put it up on Github for people to play with if they like.  There appears to have been something similar for TW Classic, but I've not seen anything for TW5.
https://github.com/ng110/TiddlPy  ('TiddlyPy' was already used by the TWC Python library).


PLEASE NOTE: Because this has the ability to edit a TW file and overwrite the original, there may well be unforeseen side-effects that I have not yet discovered, so BACK UP any valuable data before using.  There are quite likely to be bugs, so I make no claims for the reliability.

   
So far, it has worked well for me in my first application, which is as a news reader.  I am scraping news pages from the website of my local newspaper (which is horribly overloaded with intrusive adverts) and dropping them into a dedicated TiddlyWiki.  A scheduled task runs my Python script to keep it regularly updated.  As well as avoiding the ads, it also allows me to browse the news offline.

The ability to automate moving data in and out of a local TW file opens up all sorts of possibilities.  Here are some ideas:

• Extracting information from websites for formatting/viewing/searching etc in TW
• Tools for importing various data formats into TW
• Exchanging information between different TW files
• Communicating between different computers (using a file synchronised with Dropbox etc).  This could allow you to use TW on one device as a control panel that sends instructions / reads data from one or more remote machines that might be controlling an experiments / a home security system, etc.
• New WikiText markup that provides instructions to an external Python script.  For example, it could instruct the script to download an image or other file from a specified URL, drop the file into a local directory and then replace the markup with a link to the saved file. Possible security issues there, but you'd have to hack both my TW and my Python code.
  

... of course, any of this could be done with the language of your choice, if you are not a Python fan.

I would be interested to know if anyone finds this useful, or if they can spot any potential problems in writing to a TW file in this way.  Obviously there is the risk of lost changes if Python tries to write at the same time as someone is editing, though this already exists as a problem when using a file synchronised between different computers.

Enjoy,

Neil.

[Tobias Beer]

Maybe one day I will do python, for now I would just ask,

shouldn't the wikiedit function also parse the wiki using beautifulsoup?

I take it, you never use html within tiddlers.

Best wishes, Tobias. 

[Neil Griffin]

There's nothing to stop anyone using BeautifulSoup or any other method to generate html content to write into the text field of a tiddler. 

I have previously used BS to write an html tiddler file for manual importing, but I assume that the import process 'normalises' the formatting in that case.  Because I was writing the to TW file directly, I avoided BS for the wikiedit function to make sure I reproduced the original formatting when writing the output. 

I haven't tried using BS to parse and then output a whole TW file.  It would be interesting to know whether it mangles it at all.  At the very least it would change all the layout, so it might look a bit unfamiliar to anyone looking through the source code.

N.

[Tobias Beer]

There's nothing to stop anyone using BeautifulSoup or any other method to generate html content to write into the text field of a tiddler.

The reason why I am asking is because you only check for something like "<div".

However, a tiddler may well contain <div>... or...

Ah, forget it! Just realizing: It would be html-escaped wouldn't it?

So, nothing to worry about. ^^

Best wishes, Tobias. 

[BJ]

very good idea!
Anki is written in python - and has a plugin architecture, a bridge to tiddlywiki would be a great thing.
cheers
BJ

[Dom Gittins]

Neil,

I have just discovered TiddlPy and wanted to drop you a quick note to say thank you.

I have been trying to work out how to post TiddlyWiki pages into graph database (neo4j) nodes for ages ... and you have made it immeasurably easier !

One quick question - you look like a bit of a regex wizard.

Have you come up with a simple bit of python to convert a TiddlyWiki list (e.g. tags) into a nice clean comma-delimited list of text items without square brackets ?

I've done it with a bit of regex and some looping, but can't help thinking I'm missing some super-simple one-step regex substitution or something ...

Wondered if you had already cracked this ???

Thanks again - this is really handy,

Dom Gittins

[Neil Griffin]

Hi Dom,

Glad you found it useful.

I'm definitely not a regex wizard... on the rare occasions I use it, I pretty much have to learn it again from scratch!  However, I was tempted by the challenge.  Give this a try:

[s for s in re.split(r"[\[\]]", twlist) if s.strip()]

N.

On 3 March 2018 at 23:19, Dom Gittins <dom.g...@gmail.com> wrote:

Neil,

I have just discover TiddlPy and wanted to drop you a quick note to say thank you.

Dom Gittins

--
You received this message because you are subscribed to a topic in the Google Groups "TiddlyWiki" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/tiddlywiki/BBM5IyiLGsI/unsubscribe.
To unsubscribe from this group and all its topics, send an email to tiddlywiki+unsubscribe@googlegroups.com.
To post to this group, send email to tiddl...@googlegroups.com.
Visit this group at https://groups.google.com/group/tiddlywiki.
To view this discussion on the web visit https://groups.google.com/d/msgid/tiddlywiki/f0b77574-f913-4326-8ff5-09c17e18100a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.