load after package require

[Oleg O. Nemanov]

Hi, all.

load right after tclsh is started(libmy.so is in current directory):
% load libmy.so
%

load after package require:
% package require test
can't find package test
% load libmy.so
couldn't load file "libmy.so": libmy.so: cannot open shared object file: No such file or directory
% load ./libmy.so
%

Why this is happen?

[Conor Williams]

comrade -- a basic kestion...
dot is not in your LD_LIBRARY_PATH
export LD_LIBRARY_PATH=.:$LD_LIBRARY_PATH
.profile .bashrc .tclsh etc etc almost ad infinium mate
for most shells
or in your autoexec.bat on dos
or maybe graphically...
rc shell on plan 9 is similar
/c:2021141018:02002

[Mike Griffiths]

Check your current working directory (via [pwd]) at each stage; it may be that a package is changing it?

[Oleg O. Nemanov]

четверг, 14 октября 2021 г. в 20:53:38 UTC+3, conor.w...@gmail.com:

> comrade -- a basic kestion...
> dot is not in your LD_LIBRARY_PATH
> export LD_LIBRARY_PATH=.:$LD_LIBRARY_PATH

This helps, but why loading works without manipulation with
LD_LIBRARY_PATH before package require command?

[Oleg O. Nemanov]

пятница, 15 октября 2021 г. в 08:28:28 UTC+3, Mike Griffiths:

> Check your current working directory (via [pwd]) at each stage; it may be that a package is changing it?

I didn't show [pwd] results in my example, but i done it - a current working directory is not changed.

[Uwe Klein]

Am 15.10.21 um 09:26 schrieb Oleg O. Nemanov:

IMU:
loading a lib from inside the interpreter does not look at LD_LIBRARY_PATH.

man n load ( or Windows equivalent. M$ seems to use some heuristics ..)

best practice probably is handing a fully resolved path/filename
to [load].

semi OT:
there is a search path defined for the [package require ... ] mechanism.

Uwe

[Harald Oehlmann]

Dear Oleg,

may I allow to give my opinion? I am not sure of the search path on
Linux to find the library.
I can only tell that in the TCL world, it is common practice to load a
DLL with a full path.
This path is normally generated within the pckIndex.tcl mecahnism.

Typically, you have a pckIndex.tcl file in a folder or subfolder listed
by the TCL variable "auto_path".

This file typically is in the same folder as your library and contains
the line:
package ifneeded my 1.0.0 "[list load [file join $dir libmy.so]]"

Remark that the "dir variable contains the path of the current folder.
So, you can relocate that and it still works.

This works on the harddisk, in starkits and hopefully also within the
TCL 8.7 zip file.

If you don't like the pckIndex mechanism, you may load it in the
following way from a TCL script in the same folder:
source [file join [file dirname [info script]] libmy.so]

Take care,
Harald

[Ralf Fassel]

* "Oleg O. Nemanov" <oleg.o....@gmail.com>

Never ever put "." in LD_LIBRARY_PATH, this opens up all kinds of
security and other problems.

As others have said, use the full path name for loading. Also, on
Linux, the .so needs to have proper execute permissions, but I think
you've checked that.

HTH
R'

[Oleg O. Nemanov]

пятница, 15 октября 2021 г. в 10:50:47 UTC+3, Uwe Klein:

> IMU:
> loading a lib from inside the interpreter does not look at LD_LIBRARY_PATH.

If so why then LD_LIBRARY_PATH manipulation helps?

> man n load ( or Windows equivalent. M$ seems to use some heuristics ..)

I can't found any info related to a lib search path in load man page.

> semi OT:
> there is a search path defined for the [package require ... ] mechanism.

Do you talk about $auto_path?

[Oleg O. Nemanov]

пятница, 15 октября 2021 г. в 11:18:33 UTC+3, Harald Oehlmann:

> If you don't like the pckIndex mechanism, you may load it in the
> following way from a TCL script in the same folder:
> source [file join [file dirname [info script]] libmy.so]

No-no. I like pkgIndex mechanism. I only noted an inconsistency
in load behaviour before and after package require call.

[Oleg O. Nemanov]

пятница, 15 октября 2021 г. в 12:15:48 UTC+3, Ralf Fassel:

> Never ever put "." in LD_LIBRARY_PATH, this opens up all kinds of
> security and other problems.

Yeah. I know this. Thanks.

> As others have said, use the full path name for loading. Also, on
> Linux, the .so needs to have proper execute permissions, but I think
> you've checked that.

Yes. The permissions are right.

[Ralf Fassel]

* "Oleg O. Nemanov" <oleg.o....@gmail.com>

| No-no. I like pkgIndex mechanism. I only noted an inconsistency
| in load behaviour before and after package require call.

I would suggest to recompile TCL with debugging information and
trace the calls to TclpDlopen() (unix/tclLoadDl.c) in a debugger.
That way you could check which pathnames are tried in each situation.
I can't see any obvious reason in the code for the behaviour you
describe...

R'

[Oleg O. Nemanov]

пятница, 15 октября 2021 г. в 13:24:02 UTC+3, Ralf Fassel:

Hm. It's funny, but in 8.6.11 version of tcl i can't reproduce this. Before i
tried 8.6.9.

[Oleg O. Nemanov]

пятница, 15 октября 2021 г. в 13:54:24 UTC+3, Oleg O. Nemanov:

Sorry. The wrong info :-). In 8.6.11 the same stranges.

[Schelte]

On 15/10/2021 13:25, Oleg O. Nemanov wrote:
> Sorry. The wrong info :-). In 8.6.11 the same stranges.
>

There must be some misbehaving package on your system. Executing
[package require test] causes all pkgIndex.tcl files found under the
directories in $auto_path to be executed. These files are not supposed
to have any side-effects, other than updating the list of available
packages via [package provide] commands. But apparently one of the files
does something more.

You can investigate by running [package names] before and after you do
[package require test]. For each of the packages that newly appear, run
[package versions <pkg>]. Then for each of the available versions of a
suspected package, find the pkgIndex.tcl file and examine it.

Or easier, run: exec find {*}$auto_path -name pkgIndex.tcl

You can narrow down the culprit by removing directories from $auto_path
before doing the [package require test]. If that solves the problem, the
rogue package is below one of the directories you removed.


Schelte.

[Oleg Nemanov]

пятница, 15 октября 2021 г. в 15:40:05 UTC+3, Schelte:

> On 15/10/2021 13:25, Oleg O. Nemanov wrote:
> > Sorry. The wrong info :-). In 8.6.11 the same stranges.
> >
> There must be some misbehaving package on your system. Executing
> [package require test] causes all pkgIndex.tcl files found under the
> directories in $auto_path to be executed. These files are not supposed
> to have any side-effects, other than updating the list of available
> packages via [package provide] commands. But apparently one of the files
> does something more.
>
> You can investigate by running [package names] before and after you do
> [package require test]. For each of the packages that newly appear, run
> [package versions <pkg>]. Then for each of the available versions of a
> suspected package, find the pkgIndex.tcl file and examine it.

Before ```llength [package names]``` returns 4, after - 448 :-).

> You can narrow down the culprit by removing directories from $auto_path
> before doing the [package require test]. If that solves the problem, the
> rogue package is below one of the directories you removed.

I set auto_path to only one empty dir and load behaviour isn't changed(before
package require and after it load successfully load a lib in current dir).

After that, i set auto_path to only /usr/lib/tcl8.6 and load behaviour is changed.
But here only 2 pkgIndex.tcl files:

/usr/lib/tcl8.6/http1.0/pkgIndex.tcl
/usr/lib/tcl8.6/opt0.4/pkgIndex.tcl

~$ cat /usr/lib/tcl8.6/http1.0/pkgIndex.tcl
# Tcl package index file, version 1.0
# This file is generated by the "pkg_mkIndex" command
# and sourced either when an application starts up or
# by a "package unknown" script. It invokes the
# "package ifneeded" command to set up package-related
# information so that packages will be loaded automatically
# in response to "package require" commands. When this
# script is sourced, the variable $dir must contain the
# full path name of this file's directory.

package ifneeded http 1.0 [list tclPkgSetup $dir http 1.0 {{http.tcl source {httpCopyDone httpCopyStart httpEof httpEvent httpFinish httpMapReply httpProxyRequired http_code http_config http_data http_formatQuery http_get http_reset http_size http_status http_wait}}}]
~$ cat /usr/lib/tcl8.6/opt0.4/pkgIndex.tcl
# Tcl package index file, version 1.1
# This file is generated by the "pkg_mkIndex -direct" command
# and sourced either when an application starts up or
# by a "package unknown" script. It invokes the
# "package ifneeded" command to set up package-related
# information so that packages will be loaded automatically
# in response to "package require" commands. When this
# script is sourced, the variable $dir must contain the
# full path name of this file's directory.

if {![package vsatisfies [package provide Tcl] 8.2]} {return}
package ifneeded opt 0.4.6 [list source [file join $dir optparse.tcl]]

In http pkgIndex.tcl file i placed right after comments:

puts "1"
load libsetval.so

In opt pkgIndex.tcl file i placed right after comments:

puts "2"
load libsetval.so

After that i do:

~$ tclsh
% set auto_path /usr/lib/tcl8.6
/usr/lib/tcl8.6
% package require test
1
error reading package index file /usr/lib/tcl8.6/http1.0/pkgIndex.tcl: couldn't load file "libsetval.so": libsetval.so: cannot open shared object file: No such file or directory
2
error reading package index file /usr/lib/tcl8.6/opt0.4/pkgIndex.tcl: couldn't load file "libsetval.so": libsetval.so: cannot open shared object file: No such file or directory

can't find package test
%

I.e. load behaviour is changed before any statements in pkgIndex.tcl.

[Ralf Fassel]

* Oleg Nemanov <oleg.o....@gmail.com>

| In http pkgIndex.tcl file i placed right after comments:
>
| puts "1"
| load libsetval.so

Make that:

puts "1: pwd {[pwd]} glob {[glob -nocomplain libsetval.so]}"
load libsetval.so

and also in 2, verify that pwd and the glob indeed indicate the expected
dir and file existence.

| I.e. load behaviour is changed before any statements in pkgIndex.tcl.

Another possibility is that the 'load' command is replaced by something
else by the 'package require'. So insert a printing version of load:

rename load load_orig
proc load {args} {
puts stderr "load: $args"
load_orig {*}$args
}

If after the 'package require' the printout is not there, 'load' has
been replaced.

R'

[Oleg Nemanov]

пятница, 15 октября 2021 г. в 18:37:59 UTC+3, Ralf Fassel:
> ...[CUT]...
Done:

~$ tclsh
% set auto_path /usr/lib/tcl8.6
/usr/lib/tcl8.6

% rename load load_orig
% proc load {args} { puts stderr "load: $args"; load_orig {*}$args; }
% package require test
1: pwd {/home/lego/work/libs/tcl/setval} glob {libsetval.so}
load: libsetval.so

error reading package index file /usr/lib/tcl8.6/http1.0/pkgIndex.tcl: couldn't load file "libsetval.so": libsetval.so: cannot open shared object file: No such file or directory

2: pwd {/home/lego/work/libs/tcl/setval} glob {libsetval.so}
load: libsetval.so

error reading package index file /usr/lib/tcl8.6/opt0.4/pkgIndex.tcl: couldn't load file "libsetval.so": libsetval.so: cannot open shared object file: No such file or directory
can't find package test

% load libsetval.so
load: libsetval.so

[Ralf Fassel]

* Oleg Nemanov <oleg.o....@gmail.com>

| ~$ tclsh
| % set auto_path /usr/lib/tcl8.6
| /usr/lib/tcl8.6
| % rename load load_orig
| % proc load {args} { puts stderr "load: $args"; load_orig {*}$args; }

Obviously what's missing here is the

load libsetval.so

right before the

package require test

;-)

| % package require test
| 1: pwd {/home/lego/work/libs/tcl/setval} glob {libsetval.so}
| load: libsetval.so
| error reading package index file /usr/lib/tcl8.6/http1.0/pkgIndex.tcl:
| couldn't load file "libsetval.so": libsetval.so: cannot open shared
| object file: No such file or directory
| 2: pwd {/home/lego/work/libs/tcl/setval} glob {libsetval.so}
| load: libsetval.so
| error reading package index file /usr/lib/tcl8.6/opt0.4/pkgIndex.tcl:
| couldn't load file "libsetval.so": libsetval.so: cannot open shared
| object file: No such file or directory
| can't find package test

Ok, looks good, 'load' has obviously not been replaced and gives
consistent results.

| % load libsetval.so
| load: libsetval.so
| couldn't load file "libsetval.so": libsetval.so: cannot open shared object file: No such file or directory

This is what I would have expected from the manpage of dlopen(): the
filename does not contain a "/", so the default search path is applied
which only includes the system defaults (and notably not "."), thus the
library is not found. If you do "load ./libsetval.so" instead, it
contains a "/" and so the name is used as-is.

R'

[Schelte]

On 15/10/2021 16:38, Oleg Nemanov wrote:
> I.e. load behaviour is changed before any statements in pkgIndex.tcl.
>

I tried to reproduce your scenario and managed to easily do so.

First observation: The initial load only works in an interactive
session. Running the same commands from a file immediately throws the
"couldn't load file" error.

Executing [package require] for a package that is not yet known is
handled by procs registered to [package unknown]. On my system, [package
unknown] returns "::tcl::tm::UnknownHandler ::tclPkgUnknown". The first
proc is for modules, the second for packages. Getting rid of the command
for modules didn't change anything. Then I made my own simplified
version of tclPkgUnknown. Reducing that step by step, I found that the
changed behavior happens after running a [glob] command.

So then I just executed a glob myself, instead of the package require.
That also triggered the change in behavior:

% glob -nocomplain foo
% load libfswatch2.0.1.so
couldn't load file "libfswatch2.0.1.so": libfswatch2.0.1.so: cannot open

shared object file: No such file or directory

The same happens with [pwd] and [cd]. It seems to me those commands
initialize some filesystem access, after which [load] no longer looks in
the current directory.

This is curious. But since it already never works in a script, I don't
think it is really anything to worry about.


Schelte.

[Oleg Nemanov]

пятница, 15 октября 2021 г. в 21:02:04 UTC+3, Schelte:

> % glob -nocomplain foo
> % load libfswatch2.0.1.so
> couldn't load file "libfswatch2.0.1.so": libfswatch2.0.1.so: cannot open
> shared object file: No such file or directory
> The same happens with [pwd] and [cd]. It seems to me those commands
> initialize some filesystem access, after which [load] no longer looks in
> the current directory.
>
> This is curious. But since it already never works in a script, I don't
> think it is really anything to worry about.

Ok :-). Thanks for your research and explanation!