Wikitext img failing under node.js (5.1.23)

[Mark S.]

I have an odd problem that's showing up when attempting to use img[] links and canonical uri's with locally hosted (static) image files. 

I have a tiddler with an image:

with canonical external file:

But when used as wikitext image, it fails:

I haven't tried it yet to see if it works as a local file.

[Saq Imtiaz]

I suspect this is a standalone TW file and not node.js?

I believe this only works on node.js, due to path resolution relative to the server root, as well as needing a static file server that serves the images at that URI (which TW on node.js does for the files folder)

[Mark S.]

It's being served up by standard node.js. 

[Saq Imtiaz]

Mark: I just tried to reproduce this and did not encounter the same problem. Where the broken/non-existent image displays, can you inspect it to see what the src attribute is set to?

[Mark S.]

Well, that is the problem:

<img src="data:image/png;base64,undefine">

So something in the system thinks that it is a base 64 image, even though it is a relative link image (as shown in the screenshots).

[Mark S.]

The problem turned out to be that the way the _canonical_uri files were created was also creating 6byte image files in the tiddler folder which apparently confuses the system into thinking that it is dealing with base64 image files.

[Saq Imtiaz]

Hmm. That's interesting. I'm seeing 0 byte image files but the images do otherwise load correctly.

What OS and browser are you using? 

It would also be interesting to check how TW on node.js on your system, with no plugins saves _canonical_uri tiddlers. Are there any image files created at all, or just a tid file?

I've been working on this here and there over the last week, so with a few more details I should be able to address this.

[Saq Imtiaz]

It would helpful to determine what the default behaviour of TW on node.js is for _canonical_uri tiddlers and saving to file.

On Windows 10, latest Chrome and using latest pre-release with the server edition and no plugins, I tried manually creating a tiddler with:

- title: test.png

- type: image/png, 

- and an arbitrary _canonical_uri value of files/test.png

In the tiddlers directory I get a 0 bytes file test.png and test.png.meta.

I am not sure if the 0 byte expected behaviour on node.js for canonical_uri tiddlers or something new in 5.1.23

Using [img[test.png]] in another tiddler, the img is correctly resolved to have a src attribute files/test.png, both before and after restarting the node server.

So I cannot reproduce the problem. It could be OS specific. 

It is worth considering that 5.1.23 had significant changes to the file saving code, so there may have been a regression.

Also I am using the latest pre-release, which might be relevant as at least one bugfix since 5.1.23 seems to deal with missing text fields of tiddlers, see https://github.com/Jermolene/TiddlyWiki5/pull/5416/files

If you can confirm that creating a _canonical_uri tiddler by hand works well for you on node.js, that will help narrow down whether this is a core issues or a plugin issue.

Regards,
Saq

[Mark S.]

I explained that poorly. The way that I created the png files (actually using your external files tool) generated 6 byte file images. These images do not display correctly.

The zero-byte images generated when making external uri's by hand do display correctly. 

So to be clear, AFAIK, there is nothing wrong with img[] nomenclature under node.js. If there was still a way to mark an issue as resolved, I would do that.

Thanks!

[Saq Imtiaz]

Understood. As I'm unable to reproduce the problem of the 6 byte images being created, there isn't much I can do to further debug this without more information.

Saq

[Mark S.]

Just to be clear, the 6 byte images were created using the kit from your sandbox.

Just in case it was an OS thing, I fired up my old Windows 7 machine, created a new node.js instance (5.1.21 is what I have there), dragged and dropped the items from your sandbox including the plugin, restarted the instance. Then dropped in an image. It made the _canonical_uri tiddler. I was able to insert the image as img[[ wikitext. Then I restarted the instance. Now the img[] image is replaced by the "missing" icon. On the disk, the image file in the tiddlers directory is 1k rather than 6bytes. (1k is too small to be the image size, BTW). 

Then, just in case it related to TW version, I upgraded to 5.1.23 and tested again. Once again it makes 1k files. But they won't display via img[] after restart.

So the same problem appears to exist when making files on Windows and on Linux. The only difference is that in one system it's making 6byte image files and in the other 1k byte files. Either way the image doesn't display with img[] after the node instance has been restarted.

[Saq Imtiaz]

Hi Mark, appreciate the additional testing and feedback. It at least helps rule out some potential causes.

 I'm still quite perplexed as I have been using this for several months now, with no issues across several server restarts. I consistently get 0 bytes image files in the tiddlers folder, which I presume is the expected behaviour.

Anyway, I've actually worked on this some the last week, including  rewriting portions and adding better redundancy and error handling in case of write/upload errors. I'll publish that once I get the chance, it will be interesting to see if the problem you are encountering still persists. 

[Mark S.]

I just booted into Windows 10 and experienced the same error. In the case of one of the image files, it corrected itself after I temporarily changed the text in the canonical_uri tiddler. But the other two images that I imported continued to have a non-zero byte file (1kib) in the tiddlers directory. In addition, there was an oddity that the imported image files have no mod dates, and so don't appear in the "recent" list.

So I've tried it now on 3 file systems, 3 operating systems and two different physical machines. 

Thanks! 

[Saq Imtiaz]

Hi Mark,

Very peculiar! This suggests that there is something about my environment that makes this work, whereas it otherwise fails. I've been using this as part of a highly customized TW, where the image import is via a custom UI and part of a larger workflow, so it is possible that there is some other part of it that is needed for this to work properly.

However, I did realize that when I was testing yesterday to recreate your problem in a fresh wiki, I made the mistake of using the version of the plugin that I have been working on the last week. Since this is the code base I will be working on going forwards, it would be very helpful to know if this version still gives you the same issue on 5.1.23, before I try to do a deeper dive and identify and resolve the issue.

https://saqimtiaz.github.io/sq-tw/temp/import-to-external-file.html

Thanks again for the feedback and time invested in testing this.

Regards,

Saq

[Mark S.]

Ok! That seems to fix it -- it creates zero byte stub files! (Linux, Chromium).

Thank you!

[Saq Imtiaz]

That's good to hear Mark. I wasn't looking forward to having to debug this if it was still an issue with the newer code.

I'm still actively working on this, in fact I put in a little time on it even today, so there isn't any documentation as things are apt to change as I explore different implementation approaches.

One caveat to be aware of: The way the code now works is that it saves the imported image as a normal base 64 tiddler in the wiki tiddler store in memory, while uploading it to the files folder in the background. Once it has been uploaded and successfully saved to the files folder, the base 64 image tiddler is converted into a _canonical_uri tiddler. All of this happens without any user interaction and doesn't seem to disrupt normal usage in any manner. This workflow guards against losing data because of any issues writing the file to the files folder, and also works around the fact that import process in TW is designed to be synchronous. 

Working on a local node.js server it is all almost instantaneous and you don't notice that the tiddler is first saved as base 64, but just a heads up that this is what happens in the background. On a remote server over a slow connection, or with very large media files, this could be an issue as the normal sync mechanism may start saving the base 64 tiddler while the file is uploading to the files folder. However, best dealing with that may require some core modifications and support in the file saving adaptor. The long term goal is to identify what changes would be needed and hopefully get support for saving to the files folder in the core. 

Not present in the version I uploaded for you this morning but now implemented, is a toolbar button and a tm-externalise-tiddler message that can be used to convert an existing base 64 image into a _canonical_uri tiddler with the actual image saved in the files folder. If that's something you need, I can publish that.

Saq

[Adithya B]

Hi Saq, 

This plugin is very nice. Thanks for the contribution!

It runs beautifully - I had to make a small change in the JS to make it work on my TiddlyServer installation though!

I changed "var uploadURL = '/api/upload';" to "var uploadURL = 'api/upload';" Because the initial / was making it call the api on the home directory instead of the subdirectory. Once I did that it worked perfectly!

Thanks again,

Adithya

[Joshua Fontany]

This is very cool stuff Saq. Mahalo for sharing your experiments.

Best,
Joshua Fontany

[Saq Imtiaz]

Thank you.

I've created a separate thread and posted the latest version here:

https://groups.google.com/g/tiddlywiki/c/T0MP1Adzzk0

[Pumin Wuvanich]

hello Adithya,

May you please elaborate how to change uploadURL variable?  I need to do the same, and couldn't find it.

Best Regards,

Pumin

[Adithya B]

You have to change the "var uploadURL = '/api/upload';" to "var uploadURL = 'api/upload';" in  $:/plugins/sq/ImportToExternalFile/startup.js tiddler in the latest version of the plugin ( here: https://saqimtiaz.github.io/sq-tw/temp/import-to-external-file.html) 

You can  make the edit yourself. Otherwise, I have also made a pull request with this change - lets see.

[Pumin Wuvanich]

Thanks Adithya (and to Saq) , this fix works great at least for now.