external files not saved back to original location

[RA]

I got my collection of scripts to show up in my wiki as plain text tiddlers. ( I am using an absolute path, and the scripts have various extensions like ".sql",".ps" etc.). When I edit them, however, they get saved not at the original location, but along with regular tiddlers. Their extensions are stripped and replaced with ".txt", and the absolute path added to the name with slashes encoded as "%2F". I'd like the scripts to be saved back to the original location with the name and extension unchanged. Is that possible?

[RA]

Forgot to mention I'm running it on Node.js.

[Joshua Fontany]

To clarify, what method are you using to load/import these scripts with the Node.js Tiddlywiki server?

Best,
Joshua F

[RA]

Am I hitting this? Does it mean I have to use relative path if I want the original file saved unders same path and name?

" Any error saving a tiddler to disk, with a logical path that does not start with the wiki folder's path the most common error, causes the filepath to be encoded via Javascript's encodeURIComponent() method and the tiddler is saved as this file in the wiki folder's default-tiddler-location"

(from https://tiddlywiki.com/#Customising%20Tiddler%20File%20Naming)

On Saturday, February 6, 2021 at 1:16:01 PM UTC-8 RA wrote:

[RA]

I am using `tiddlywiki.files` method.

tiddlywiki.files:

{
    "directories": [
        {
            "path": "/srv/git/scripts/t-sql",
            "isTiddlerFile" :false,
            "isEditableFile": true,
            "fields": {
                "title": {"source": "filename", "prefix": "scripts/t-sql/"},
                "created": {"source": "created"},
                "modified": {"source": "modified"},
                "type": "text/plain",
                "tags": ["t-sql", "external"],
                "linux-path":  {"source": "filename", "prefix":        "/srv/git/scripts/t-sql/"},
                "win-path": {"source": "filename", "prefix": "c:\\vm\\srv\\git\\scripts\\t-sql\\"}
            }
        }
    ]
}iddler

$:/config/FileSystemPaths:

[is[system]!has[draft.of]removeprefix[$:/]addprefix[_system/]]
[is[draft]search-replace:g:regexp[/|\\],[_]addprefix[drafts/]]
[tag[task]addprefix[mytasks/]]
[!tag[external]]   -- Tried without this filter, same result. (This comment is for the posting, it's not in the tiddler.)

Wiki directory structure:

.

+-- tiddlers
¦   +-- %2Fsrv%2Fgit%2Fscripts%2Ft-sql%2Fdetach.txt               -- these two appear after I edit and save the external tiddler
¦   +-- %2Fsrv%2Fgit%2Fscripts%2Ft-sql%2Fdetach.txt.meta    -- and the original file `/srv/git/scripts/t-sql/detach.sql` is deleted
¦   +-- …
¦   +-- scripts
¦   ¦   +-- t-sql
¦   ¦       +-- tiddlywiki.files
¦   +-- _system
¦   ¦   +-- config
¦   ¦   ¦   +-- …
¦   ¦   ¦   +-- FileSystemPaths.txt
¦   ¦   ¦   +-- FileSystemPaths.txt.meta
¦   ¦   +-- …
¦   +-- …
+-- tiddlywiki.info

[RA]

In the wiki, the external tiddler's name is `scripts/t-sql/detach.sql`, and I like it exactly that way.

[Mark S.]

I'm not an expert on this, but I don't see anything in your $:/config/FileSystemPaths that would match the files you imported. It looks to me like you need one that starts [tag[external]... or that matches prefix scripts/t-sql/ . Also, if you read further here , you need to specify the text extension in $:/config/FileSystemExtensions with something like:

[tag[external]then[.sql]] (or whatever your matching/extension logic is).

[Joshua Fontany]

Thank you for your config/setup. When we were updating the `tiddlywiki.files` behavior for 5.1.23 we were still experimenting with how it would interact with the $:/config/FileSystem* tiddlers.

I believe that the final implementation was that as long as your `tiddlywiki.files` config is setup correctly, you do not need to mess with any $:/config/FileSystem* filters.

The `isEditableFile: true` flag should cause TW to save the location-on-disk for that tiddler and write it back to that location automatically. Changes to non *.tid files will cause a *.meta file to be generated storing the non-text-body "metadata" (all other fields).

I will attempt to recreate your setup to see if we have a bug or what else is going on.

Thanks!
Best,
Joshua Fontany

On Sunday, February 7, 2021 at 2:50:25 PM UTC-8 RA wrote:

The example of $:/config/FileSystemPaths in https://tiddlywiki.com/#Customising%20Tiddler%20File%20Naming does not include a filter matching the external notes, so I am questioning whether your recommendation is correct.

On Sunday, February 7, 2021 at 1:51:09 PM UTC-8 RA wrote:

Added a matching filter to $:/config/FileSystemPaths, same result. Tried both absolute and relative path. I know the filter is correct because it generates a filename with the correct absolute path encoded (all slashes replaced by %2F).

Added $:/config/FileSystemExtensions as well, the extension part is now ok. Here's a universal filter that will keep the extension as is: [split[.]last[]addprefix[.]]

[Joshua Fontany]

Ah, we did finally settle on using the  $:/config/FileSystem* filters, possibly to set custom extensions..... so you appear to have everything setup correclty.

But it also looks like we do have a bug with the current behavior. Any tiddlers loaded from OUTSIDE the Wiki directory will try to write back to that location, and the last check for this always encoded those file-paths and writes the file to the `tiddlers/` folder instead.

There is a security concern this has uncovered, but I will write up a bug report and we will try to get a fix in by the 5.1.24 release.

For now, I can suggest moving the source-directory into your Wiki directory (it can be parallel to your `tiddlers/` directory).

Best,
Joshua Fontany

[Mark S.]

On Sunday, February 7, 2021 at 5:15:23 PM UTC-8 joshua.... wrote:

But it also looks like we do have a bug with the current behavior. Any tiddlers loaded from OUTSIDE the Wiki directory will try to write back to that location, and the last check for this always encoded those file-paths and writes the file to the `tiddlers/` folder instead.

? I seem to have it working for an experimental .sql file inside a directory outside the Wiki folder. The path is relative, but definitely outside the wiki. The file saves back to the original location as hoped for. This after wasting an hour not realizing I needed to be in 5.1.23.

In file tiddlywiki.files, located in tiddlers/findme :

 "path": "../../../../sql",

So sql directory is two whole steps above the tiddlywiki root folder. I want to try it later with an absolute directory.

Something that isn't documented. It appears that your folder with the tiddlywiki.files file has to be below ./tiddlers or ./plugins ?? Is that correct?

 Thanks!

[RA]

I am running the latest from GitHub, so I volunteer to test the fix when it comes out :).

I was also wondering if I needed to add something to the build section of `tiddlywiki.info` for this.

I have "externalimages" section in there, but in the latest code only empty edition has it. Is it not needed in client-server anymore?

[RA]

Not sure if the current implementation allows this, but it would be nice if `*.meta` for external files could be stored inside wiki directory (next to *.tid files or in their own subdirectory).

[Joshua Fontany]

I have opened a Bug report for these concerns here: https://github.com/Jermolene/TiddlyWiki5/issues/5483

[RA]

@Mark, so you have it working with 5.1.23? Are you able to zip it up and share here?

[Mark S.]

So, it turns out that I did need the $:/config/FileSystemExtensions tiddler in order to prevent TW from assigning the ".txt" extension. This may be because I used text/plain, since TW doesn't know about text/sql. When importing the tiddlers I used tag "sql" and then added this to the extensions tiddler:

[tag[sql]then[.sql]

In the attached, zip open some place. Go into directory Wikis and run

tiddlywiki test2 --listen port=8090

(or whatever port you want). You should be able to edit myscript.sql and see it changed on the drive.

Looking at your prior post, it looks like you have tiddlywiki.files inside of scripts/t-sql . As far as I could tell, the tiddlywiki.files file would not work anywhere except below "tiddlers/". But this shouldn't be a problem, because it's the actual sql files you care about, and not the configuration of TW.

[RA]

@Mark, here's what happened when I edited and saved `myscript.sql`. Does it work for you with the release version? I'm running latest as of yesterday. I'll try rolling it back to release.

[Mark S.]

Yes, I just set it up for the release version today. I was having no luck when using the 5.1.22 .

I'm running on Linux, btw. 

[RA]

Confirmed, saving back to an external file outside wiki folder works in 5.1.23 release, broken in the current GitHub repo.

[RA]

I deleted a couple of my messages as they had a very wrong filter suggestion. Unfortunately, one of them was quoted by another person, so the filter is still visible. Please don't use it. Use something like this one instead: [regexp[\.[A-Za-z0-9_-]{1,5}]split[.]last[]addprefix[.]]

[RA]

Another correction. Currently it's not possible to use sets and ranges in regexp filter, since they require another pair of square brackets. This, for example, [regexp[[a-z]]] gives "Filter error: Syntax error in filter expression".