Introducing TiddlyRemember, a tool for integrating TiddlyWiki with Anki spaced-repetition flashcards

[Soren Bjornstad]

I'm pleased to announce the release of version 1.0 of TiddlyRemember, a tool to make it easy to remember the most important bits of your notes. You call out specific bits of your tiddlers that you want to remember using TiddlyWiki macros, and an associated Anki add-on then pulls these snippets into Anki where you can review them regularly. The Anki cards link back to the tiddler they came from so you can find the context if you need it, and if you edit the content in TiddlyWiki or move it around, your Anki collection gets updated to match on your next sync, maintaining your review history.

Here's a little snip of the syntax, the rendering in a TiddlyWiki, and the card in Anki (permalink to this page in the docs):

There are cloze deletion cards too (if you're not a memory geek, you might call these "fill in the blanks cards").

Credit for this idea goes to Andy Matuschak, who reported building a private tool that worked similarly for his notes system. I cannot for the life of me find this note now (if someone knows where it is, please post the link!).

Check it out and install from here: https://sobjornstad.github.io/TiddlyRemember/.

I'm hoping this is a reasonably stable and functional version (thus the 1.0.0 moniker), but I've only been using it myself for a couple of weeks as I've been developing, and there are definitely improvements possible, so suggestions and pull requests are welcome! The two biggest areas for improvement I see that I'm not sure how to attack are:

• You have to have TiddlyWiki installed on Node on your system to use TiddlyRemember. (You can use TiddlyRemember fine with a single-file wiki though. It just has to be there for TiddlyRemember to call out to.) There's no way around this I can think if if you want a fully automated sync (the Anki add-on needs some way to run TiddlyWiki commands unless it wants to duplicate the wiki parser), but I could imagine a way to use the new browser-based static site generator tool as part of a two-step process where you export.
• In order to extract the questions, TiddlyRemember renders all the tiddlers in your wiki and then parses the resulting HTML, which means it will end up pulling multiple copies of a question if the tiddler containing a question is transcluded into another tiddler that's included in the sync. It will happily remove duplicates, but there's no way for it to tell which tiddler is the "original source", so it picks whichever one it happens to process first, which isn't great. I don't tend to use transclusion much aside from adding templates to things and building up system tiddlers and sidebar tabs, so this isn't a big deal for me, but it might be for others.
• I originally planned to make each question a tiddler of its own; while this seemed more philosophically pure, I eventually decided against this approach because I didn't like the idea of having to separately edit a bunch of subtiddlers for all of my notes, and because having the question included in a tiddler allows the source to be automatically determined without having to link back to it and update it if you move the question around (aside from the transclusion issue). But I could see an argument for doing it as transcluded question tiddlers, and maybe we could support that as an alternative method eventually (especially if it becomes easier to edit transcluded tiddlers in a future version of TiddlyWiki).

[Henrik Hansen]

This is really cool and a rather neat solution :)

I've written something similar for myself, because I also wanted to have my flashcards within the context of my notes. It is quite powerful once you get the hang of it.

How do you handle cloze deletions and updating those? If I have a

<<remembercz "20200523175253647"
    "TiddlyRemember integrates TiddlyWiki with {Anki}.">>

then later change it to

<<remembercz "20200523175253647"
    "TiddlyRemember integrates {TiddlyWiki} with {Anki}.">>

What will then happen with the progress of the clozes in Anki?

[Soren Bjornstad]

Good question. This is one of the little warts in the current solution. :) If you made exactly the change you showed, you end up with the "new" card (the one with "TiddlyWiki" occluded) having the scheduling of your previous card with "Anki" occluded, and a new card for the existing occlusion! You can however avoid that by switching to numbering the occlusions explicitly, Anki-style:

<<remembercz "20200523175253647"
    "TiddlyRemember integrates {c2::TiddlyWiki} with {c1::Anki}.">>

In the absence of these explicit numbers, it just numbers them left to right. It probably wouldn't be too hard in the future to parse the note that's currently in Anki and attempt to catch when the implicit numbering has changed and is now wrong.

[David Gifford]

Added this to the TiddlyWiki Toolmap under the educational aids section!

[si]

Wow Soren this is a fantastic contribution to TiddlyWiki!

I agree that there are advantages to both the inline and separate tiddler approaches, so it would be cool to see that one day. I often find myself 'working backwards' - creating Anki cards first and then later relating them to a TW note by adding the name of the tiddler to a field called "parent". I had hoped to be able to one day sync these Anki notes with my Wiki and have them transcluded to the bottom of their parent tiddler.

Anyway thanks this is awsome! I hope I will be able to fit it into my Anki/TiddlyWiki workflow.

[si]

Hi Soren.

When I tested this yesterday I used a new empty Wiki and it worked perfectly. I've just tried it with my main personal Wiki and I get this error message in Anki:

Debug info:
Anki 2.1.26 (70784154) Python 3.8.0 Qt 5.14.1 PyQt 5.14.1
Platform: Windows 10
Flags: frz=True ao=True sv=2
Add-ons, last update check: 2020-05-25 17:11:24

Caught exception:
Traceback (most recent call last):
  File "C:\Users\Si\AppData\Roaming\Anki2\addons21\60456529\twimport.py", line 57, in _invoke_tw_command
    proc = subprocess.run(cmd, cwd=wiki_path, stdout=subprocess.PIPE,
  File "subprocess.py", line 512, in run
subprocess.CalledProcessError: Command '['C:\\Users\\Si\\AppData\\Roaming\\npm\\tiddlywiki.cmd', '--output', 'C:\\Users\\Si\\AppData\\Local\\Temp\\tmpbkihbek9\\render', '--render', '[type[text/vnd.tiddlywiki]] [type[]] +[!is[system]]', '[is[tiddler]addsuffix[.html]]', 'text/html', '$:/plugins/sobjornstad/TiddlyRemember/templates/TiddlyRememberParseable']' returned non-zero exit status 1.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\Si\AppData\Roaming\Anki2\addons21\60456529\__init__.py", line 141, in join_thread
    raise self.extract_thread.exception
  File "C:\Users\Si\AppData\Roaming\Anki2\addons21\60456529\__init__.py", line 59, in run
    self.notes = twimport.find_notes(
  File "C:\Users\Si\AppData\Roaming\Anki2\addons21\60456529\twimport.py", line 186, in find_notes
    _render_wiki(tw_binary, wiki_folder, render_location, filter_)
  File "C:\Users\Si\AppData\Roaming\Anki2\addons21\60456529\twimport.py", line 143, in _render_wiki
    _invoke_tw_command(cmd, wiki_path, "render wiki")
  File "C:\Users\Si\AppData\Roaming\Anki2\addons21\60456529\twimport.py", line 68, in _invoke_tw_command
    raise Exception(f"Failed to {description}: return code {proc.returncode}.\n"
Exception: Failed to render wiki: return code 1.
$ C:\Users\Si\AppData\Roaming\npm\tiddlywiki.cmd --output C:\Users\Si\AppData\Local\Temp\tmpbkihbek9\render --render [type[text/vnd.tiddlywiki]] [type[]] +[!is[system]] [is[tiddler]addsuffix[.html]] text/html $:/plugins/sobjornstad/TiddlyRemember/templates/TiddlyRememberParseable

fs.js:114

    throw err;

    ^


Error: ENOENT: no such file or directory, open 'C:\Users\Si\AppData\Local\Temp\tmpbkihbek9\render\Aardvark.html'

    at Object.openSync (fs.js:438:3)

    at Object.writeFileSync (fs.js:1189:35)

    at $:/core/modules/commands/render.js:58:6

    at Object.$tw.utils.each (C:\Users\Si\AppData\Roaming\npm\node_modules\tiddlywiki\boot\boot.js:126:12)

    at Command.execute ($:/core/modules/commands/render.js:43:12)

    at Commander.executeNextCommand ($:/core/modules/commander.js:107:14)

    at Commander.executeNextCommand ($:/core/modules/commander.js:111:12)

    at Commander.execute ($:/core/modules/commander.js:64:7)

    at Object.exports.startup ($:/core/modules/startup/commands.js:34:12)

    at Object.$tw.boot.executeNextStartupTask (C:\Users\Si\AppData\Roaming\npm\node_modules\tiddlywiki\boot\boot.js:2397:10)

It's a single file wiki, the most recent version. TiddlyRemember still works perfectly with the empty wiki I originally tested it with.

I've tried removing all other Anki and TW plugins but still get the same message. Any idea what's going on here?

On Monday, 25 May 2020 03:26:29 UTC+1, Soren Bjornstad wrote:

[Soren Bjornstad]

Si, this is odd -- for some reason the TiddlyWiki Node app is failing to write out HTML files to a temp folder. A couple things to try:

• Restart your computer. I doubt this is going to help since your other wiki was working, but there could be some crud stuck in your temp folder, and you never know, that can sometimes help.
• What's the output of tiddlywiki --version at a command prompt? There might be different behavior than I'm seeing if it's not the latest version.
• What version of Windows are you running?
• Open C:\Users\Si\AppData\Roaming\Anki2\addons21\60456529\twimport.py in Notepad or another text editor on your computer, go to line 185, which says "render_location = os.path.join(tmpdir, 'render')", and add a new line right after that which says:
          os.mkdir(render_location)
• Be sure it has the same 8 spaces of indentation ahead of it that the surrounding lines do.
• Then try running again.
  

If none of that helps, can you send me a copy of your wiki to see if it happens on my computer with your wiki? You can email me if you don't want to post it publicly (hit sorenbjornstad.com for the address), and feel free to delete content from that copy as long as you're still having the problem.

On Tuesday, May 26, 2020 at 6:18:15 AM UTC-5, si wrote:

Hi Soren.

When I tested this yesterday I used a new empty Wiki. I've just tried it with my main Wiki and I get this error message in Anki:

[si]

Hi Soren,

Thanks for your reply.

I believe I have identified the source of the bug: tiddlers with ? or :: in their titles.

When I rename these tiddlers the plugin works as expected - I guess they are treated as special characters in the command line?

I found that I had to actually remove the titles from my wiki. If I try to exclude them in Anki with a filter like [all[tiddlers]!is[system]type[]!regexp[(?i)\?]] I get the same error message.

Does this explain the bug?

Just in case you still need to know tiddlywiki --version gives the output 5.1.22 and I'm running Windows 10.

[Soren Bjornstad]

Yes, this helps a lot! This is a Windows filename bug -- ? and :, among others, are both invalid characters in Windows filenames. The docs say TW will replace both these characters with _ in titles, so I assumed it would take care of things for me...but it turns out it applies this only when saving tiddlers in Node.js, not when rendering them to HTML.

If I try to exclude them in Anki with a filter like [all[tiddlers]!is[system]type[]!regexp[(?i)\?]] I get the same error message.

I'm not entirely sure why this didn't work, but I'm sure there's some explanation. :)

I released a new version of the add-on that fixes this on my Windows machine. If you choose "Check for Updates" in the add-ons window in Anki and then restart Anki, you should be good (no changes were made to the TiddlyWiki plugin, so no need to update that).

On Tuesday, May 26, 2020 at 5:26:50 PM UTC-5, si wrote:

Hi Soren,

[si]

Awesome it's working fully for me now. Thanks for the quick fix!

If you would permit me to make a feature request: it would be cool if the add-on also worked with encrypted wikis. There is a password command on TW for Node.js so it seems like this would be possible.

Anyway thanks again.

[Sebastián Ortega]

You make my day when I read Anki + Tiddlywiki!

However syncing doesn't work for me. I've posted the details to a github issue: https://github.com/sobjornstad/TiddlyRemember/issues/6

Anyway, thanks a lot for the project

[Soren Bjornstad]

Si, I added an issue for this. If you have thoughts on how the password should be requested and/or stored, let me know.

Sebastian, answering on the issue you posted.

[si]

Si, I added an issue for this. If you have thoughts on how the password should be requested and/or stored, let me know.

Being forced to use my wiki without a password has made me realise how much faster it runs without encryption, so I'm retracting my vote for this feature (sorry)! However I'm sure that it would be useful to others.

Credit for this idea goes to Andy Matuschak, who reported building a private tool that worked similarly for his notes system. I cannot for the life of me find this note now (if someone knows where it is, please post the link!).

I have just come across the notes where he talks about the "mnemonic medium" for personal notes, which look like they might be what you are referring to:

https://notes.andymatuschak.org/The_mnemonic_medium_can_be_extended_to_one%E2%80%99s_personal_notes

https://notes.andymatuschak.org/My_implementation_of_a_personal_mnemonic_medium

[Productive Master]

Is there a way to add image in Tiddly remember which will show in Anki card?

Something like this?

--
You received this message because you are subscribed to the Google Groups "TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tiddlywiki+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/tiddlywiki/6edba8ae-aea6-4af3-acdd-4dfd44f05727%40googlegroups.com.

[Soren Bjornstad]

Thanks, that's it!

[Soren Bjornstad]

Ooh. Not currently. I can see the value of this, but I can say right now it would be really tricky to implement since it would have to render image tiddlers as well, or otherwise retrieve them, and then hook into Anki's media syncing functionality while simultaneously editing the reference to the image. If you want to post an issue, feel free...you never know, maybe someone else will come along with a brilliant architecture idea that will make it easier.

To unsubscribe from this group and stop receiving emails from it, send an email to tiddl...@googlegroups.com.

[TonyM]

Build the whole solution in tiddlywiki would do it.

Tony

[Sebastián Ortega]

That might not be a solution for people using Anki for years.

--
Sebastián

On Sat, 30 May 2020 at 12:07, TonyM <anthony...@gmail.com> wrote:

Build the whole solution in tiddlywiki would do it.

Tony

--
You received this message because you are subscribed to a topic in the Google Groups "TiddlyWiki" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/tiddlywiki/UD6VyV_r-94/unsubscribe.
To unsubscribe from this group and all its topics, send an email to tiddlywiki+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/tiddlywiki/f0037706-1329-4c5e-af28-6c7ba99f461b%40googlegroups.com.

[Soren Bjornstad]

It's a thought. A simple SR algorithm is not particularly hard, so that would be doable. But recently I've been seeing sites pop up that all do their own spaced-repetition implementation, and the result is if you use more than one of them, pretty soon you have 8 places to go every day to review flashcards. I'd rather not be responsible for a fifteenth competing standard.

Anki also has a really nice mobile app, which is something TW still struggles with.

[si]

Hi Soren.

This might be more of an Anki question, but I have been unable to get permalinks to work in the way that you describe here.

It seems to me that Anki won't open files that are outside of the "collection.media" folder. I've been trying all sorts of permutations of <a href="file:///C:\path\to\file.html">link</a> but when I click the link nothing happens.

Is there some trick to get this to work that I'm not aware of?

[soren.b...@gmail.com]

Si,

 
It works for me when I just paste an <a href> tag into the HTML editor directly on a card (Ctrl-Shift-X while you have a field selected). The URL you list is not a valid file: URL, however -- it needs to use forward slashes even on Windows. It's possible that is the issue, but I don't have a Windows machine handy at the moment to test on and see if that breaks something.

It does appear that the permalink generation in TiddlyRemember applied during the sync is incorrectly putting a backslash at the end of URLs that can't accept one there. On my first try I ended up with "/home/soren/test.html/#TiddlerName" as the permalink, which is invalid since an HTML file is not a directory, and this resulted in nothing happening when I clicked the link. I opened an issue for that. To see if this is what's causing your problem, you can try manually changing the text in the Permalink field of a card and removing that extra slash.

[TW Tones]

Si,

The protection on browsers is the issue here I believe. If you run your wiki in TiddlyDesktop, as a hta file (some limitations) and using twexe referencing file: addresses is not so limited. Similarly if you use the files feature of bob you can access those files at a URL

Relative links to files relative to files under node may also work for you but not file from the browser.

You can put file:///C:\path\to\file.html in the browser address bar with the right permissions, even browse the folders. and you can copy link address and import to a wiki. Otherwise it is hard now to get this interaction with the local machine unless tiddlywiki is running in A non internet browser like the one built into TiddlyDesktop.

My feeling is others have overcome this limitation like Office 365 browser apps, but 

[si]

Hi Soren thanks for your reply.

>>> The URL you list is not a valid file: URL, however -- it needs to use forward slashes even on Windows.

Thanks, I wasn't sure which to use here but I had tried every combination and it still didn't work.

>>> It does appear that the permalink generation in TiddlyRemember applied during the sync is incorrectly putting a backslash at the end of URLs that can't accept one there.

I had noticed this, but I have tested with the correct permalink and it still won't work. I've also tried putting the HTML directly into the card (<a href="file:///C:/Users/Si/Desktop/file.html">link</a> using Ctrl+Shift+X), but again when I click the link while viewing the card nothing happens.

I've noticed that there is an add-on for opening files in an external program. I got it to work with an HTML file, but because it looks for a specific extension at the end of the path, it doesn't work with permalinks that end in "#tiddler".

Is it possible that Anki just can't open external HTML files on Windows?

[soren.b...@gmail.com]

I just tried it on Windows and it works fine for me. Downloaded an HTML to my desktop and added to the card:

<a href="file://C:/Users/soren/Desktop/test.html">link</a>

Click the link and test.html opens in the browser.

I'm on Anki 2.1.34 (a couple versions behind, but can't think of anything that would have changed here).

Do other links to websites work in Anki? Perhaps your default browser is configured wrong and Anki can't figure out what to do with links.

[si]

Hi Soren.

>>> Do other links to websites work in Anki? Perhaps your default browser is configured wrong and Anki can't figure out what to do with links.

I have no problem with Anki opening websites. Anything with the prefix "https://" will open in the browser, but when I use "file://" nothing happens.

I also tried this on a totally different Windows 10 machine with an almost fresh install and still it wouldn't work. It also didn't work when I tried clicking the link from AnkiWeb. This was all with me creating the link exactly as you describe above (the file opens if I copy the path directly into the browser).

Did you try it on a 64 bit Windows 10, or some earlier version (no idea if that would matter)?

[soren.b...@gmail.com]

Pretty sure it’s 64-bit. VM on Parallels on my MacBook with the default license and settings. Can’t imagine why it would matter anyway.

I’m stumped. I’d suggest you post this on the Anki forums at https://forums.ankiweb.net since it’s an Anki issue at this point.

[si]

>>> I’d suggest you post this on the Anki forums at https://forums.ankiweb.net since it’s an Anki issue at this point.

Will do. Thanks for your help.

[Prestige]

Any way to add tags to the cards?

[Sebastián Ortega]

You can add an anki_tag field to any tiddler. Then, questions in tiddlers tagged with the first tiddler would get synced.

For example, in TiddlyRemember I have:

Then I get this synced:

--
Sebastián

--
You received this message because you are subscribed to a topic in the Google Groups "TiddlyWiki" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/tiddlywiki/UD6VyV_r-94/unsubscribe.
To unsubscribe from this group and all its topics, send an email to tiddlywiki+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/tiddlywiki/7d3054ca-5ca5-4d5c-aaa5-f3bf71ecec09n%40googlegroups.com.

[Prestige]

Thanks

This is cool.

[Prestige]

Does TiddlyRemember support images in cards? does it have to be links or I can drag drop?