Request for Comment Tip 430 - Adding Zipfs support to the core

[Hypnotoad]

Tclers,

I am requesting community feedback for Tip 430, which adds zipfs support to the core:

Tip 430: http://core.tcl.tk/tips/doc/trunk/tip/430.md

The quick and dirty explaination:

1) The TIP embeds a zipfs implementation into the core sources
2) With this implementation scripts can mount and unmount zip archives
3) When tclAppInit.c is compiled without shims for kits, the core now provides some default kit-like behaviors
4) The Tcl sources now compile a locally executable copy of the minizip program (The sources are already distributed in /compat)

The kit-like behaviors are as follows:

1) If the tcl executable has an attached zip file system, mount it
2) If a file named main.tcl is the root of that file system, treat that as the tcl startup script
3) If a file named tcl_library/init.tcl is in that file system, make the directory it is in the value of TCL_LIBRARY

If the TCL_LIBRARY is not found in the executable, the routine then looks to see if the tcl dynamic library has an attached file system, and looks for tcl_library/init.tcl in there.

The implementation is checked in as the core_zip_vfs branch on http://core.tcl.tk/tcl, and ready for testing on Unix and Windows builds using MSYS/Mingw

--Sean "The Hypnotoad" Woods

[Rich]

Hypnotoad <yo...@etoyoc.com> wrote:
> The kit-like behaviors are as follows:
>
> 1) If the tcl executable has an attached zip file system, mount it
> 2) If a file named main.tcl is the root of that file system, treat
> that as the tcl startup script
> 3) If a file named tcl_library/init.tcl is in that file system, make
> the directory it is in the value of TCL_LIBRARY

Are #2 & #3 only performed if #1 occurs? Your posting implies "yes" but
is not 100% clear on this fact.

Because if #2 and/or #3 always occur upon mounting a zip, then these
two would lead to a MSWin style "autorun" security problem with using
the zipfs in general to access zip files.

[undro...@gmail.com]

Hypnotoad,

some questions:

1) what is the purpose of minizip if the tclsh already provides a "zipfs mkzip" to create a ZIP archive by itself?
2) is it possible to do a "cd zipfs:/somedir" on Windows and/or a "cd //zipfs:/somedir" on UN*X? What are the implications?
3) does the result of "glob /*" on UN*X include "//zipfs:"?

All the best,
Christian

[Hypnotoad]

1)

We need a separate executable for wrapping because there is no guarantee that the platform you are compiling is the same you are compiling on. And on Windows there is no "binary of record" for building zip files. And on Unix just because a program is named "zip" is no guarantee it will process command line arguments in the way that the Makefile is expecting.

Now, with a zip enabled core installed in your environment you could reliably use the zip making facilities for wrapping extensions and executables. But you can't make Tcl depend on Tcl to install itself, because Tcl is often the first tool installed when porting to a new platform.

2) Yes.

3) No. The entire pattern has to match for the Tcl file system to embark on a search through the Zipfs mounted volumes. (And yes, I did just fire off a freshly minted tclsh on Macos just to make sure.)

I'm not actually 100% sold on //zipfs:/ as the prefix for Unix. The concern is that the user could potentially have a directory named zipfs: in the local folder and MAY want to find files inside of it without that folder being confused for a zipfs: mounted system.

[Hypnotoad]

Sorry about that. Yes indeed, steps 2 and 3 are predicated on a successful mounting of the file system in step 1.

The kit-like behaviors are ONLY performed at process startup, and ONLY on the actual executable that spawned the process. Just mounting a volume does not trigger any of these behaviors. And the search for TCL_LIBRARY is only done once at process startup. And only if the application doesn't specify a TCL_LOCAL_MAIN_HOOK.

I was kicking around the idea of having a convention to automatically search for a pkgIndex.tcl file on any mount. But the security issues you describe makes that sound like a terrible idea so let us continue to NOT do that.

[Rich]

Hypnotoad <yo...@etoyoc.com> wrote:
> On Saturday, November 18, 2017 at 10:31:14 AM UTC-5, Rich wrote:
>> Hypnotoad <yo...@etoyoc.com> wrote:
>> > The kit-like behaviors are as follows:
>> >
>> > 1) If the tcl executable has an attached zip file system, mount it
>> > 2) If a file named main.tcl is the root of that file system, treat
>> > that as the tcl startup script
>> > 3) If a file named tcl_library/init.tcl is in that file system,
>> > make the directory it is in the value of TCL_LIBRARY
>>
>> Are #2 & #3 only performed if #1 occurs? Your posting implies "yes"
>> but is not 100% clear on this fact.
>>
>> Because if #2 and/or #3 always occur upon mounting a zip, then these
>> two would lead to a MSWin style "autorun" security problem with
>> using the zipfs in general to access zip files.
>
> Sorry about that. Yes indeed, steps 2 and 3 are predicated on a
> successful mounting of the file system in step 1.

Good. That is the more secure path.

> The kit-like behaviors are ONLY performed at process startup, and
> ONLY on the actual executable that spawned the process. Just
> mounting a volume does not trigger any of these behaviors. And the
> search for TCL_LIBRARY is only done once at process startup. And
> only if the application doesn't specify a TCL_LOCAL_MAIN_HOOK.

Might I suggest that when you do write the manpage/docs for this, that
you make this behavior 100% explicitly clear in the docs.

> I was kicking around the idea of having a convention to automatically
> search for a pkgIndex.tcl file on any mount. But the security issues
> you describe makes that sound like a terrible idea so let us continue
> to NOT do that.

It is not a good idea at all. MS had tons of security problems back in
the day of CDROM's due to their OS'es of the day automatically running
an "autorun.bat" (or something like that) by simply inserting a CD/DVD
into a drive. There was an override (hold down shift key while
inserting CD/DVD) but the very people most in danger of the possible
exploit were also the very ones least likely to know the override.

If I (or another dev.) want to search for and load modules/packages
from random zip's we mount, let us explicitly choose to add paths to
such places in the zips ourselves, fully aware of the implications.

[Rich]

undro...@gmail.com wrote:
> 2) is it possible to do a "cd zipfs:/somedir" on Windows and/or a "cd
> //zipfs:/somedir" on UN*X? What are the implications?

I can't answer #1 or #3 (which is why they are missing) but I do know
the answer to this one.

The TCL interpreter can see the VFS mounted zip contents.

The host OS has no idea anything weird is going on and also cannot see
inside the TCL VFS mounted zips (or whatever other container format of
choice can be TCL VFS mounted).

[Rich]

Hypnotoad <yo...@etoyoc.com> wrote:
> I'm not actually 100% sold on //zipfs:/ as the prefix for Unix. The
> concern is that the user could potentially have a directory named
> zipfs: in the local folder and MAY want to find files inside of it
> without that folder being confused for a zipfs: mounted system.

TCLKit's 'mount' the filesystem from inside the kit to appear to be "on
top of" the executable itself, which avoids the issues of overriding an
existing directory path in the host filesystem.

And on Unix/Linux, "//zipfs:/" could be a valid filesystem path. Runs
of slashes get collapsed, and colons are valid characters in filenames,
so it is not an error to have:

/zipfs:/

as a FS root level directory.

Now, this is highly unlikely, but sadly the "highly unlikely"
situations are always the ones that seem to eventually trip someone,
somewhere, up.