is modulefinder.ModuleFinder working at all?

[Wolfgang Maier]

Hi,

I just used the stdlib's modulefinder.ModuleFinder (intended to find
modules used by a script) for the first time in my life and it just
doesn't seem to work like documented at all.
Not sure what is going on, but if I try the usage example from
https://docs.python.org/3/library/modulefinder.html
it's reporting every single module from the stdlib whether imported or
not! I tried with Python3.5, 3.4, 3.3, 3.2 and 2.7, but no success.
Has anybody here used this successfully and, if so, how?

Thanks for any help,
Wolfgang

[Skip Montanaro]

I've not used it before, but I suspect it's meant to be used in
"freeze" type environments. In that situation, you really do want
everything reachable, whether the script imported it or not. It
doesn't take much to wind up importing much of the stdlib either. Try:

python -m modulefinder -m /path/to/os.py

This is from Python 2.7, YMMV slightly in Py3.

Skip

[Wolfgang Maier]

On 09.11.2016 22:48, Skip Montanaro wrote:
> I've not used it before, but I suspect it's meant to be used in
> "freeze" type environments. In that situation, you really do want
> everything reachable, whether the script imported it or not.

Hmm, but that's exactly the problem. It *is* supposed to report only
what's imported and *not* everything that's available. At least that's
how I understand the docs.

> It doesn't take much to wind up importing much of the stdlib either. Try:
>
> python -m modulefinder -m /path/to/os.py
>
> This is from Python 2.7, YMMV slightly in Py3.
>

Yeah, I have tried this as well (with other stdlib modules), but you
shouldn't wind up things with the example bacon.py from the docs.

Best,
Wolfgang

[Steve D'Aprano]

On Thu, 10 Nov 2016 08:08 am, Wolfgang Maier wrote:

> Hi,
>
> I just used the stdlib's modulefinder.ModuleFinder (intended to find
> modules used by a script) for the first time in my life and it just
> doesn't seem to work like documented at all.
> Not sure what is going on, but if I try the usage example from
> https://docs.python.org/3/library/modulefinder.html
> it's reporting every single module from the stdlib whether imported or
> not!

I see what you mean, but it's not quite *every* module. After I run the
example from the docs, I get 197 modules, and here's at least two that
aren't included:

py> len(finder.modules)
197
py> 'statistics' in finder.modules
False
py> 'cmath' in finder.modules
False


Curiously, it includes modules that aren't cached in sys.modules:

py> len(finder.modules.keys() - sys.modules.keys())
107


So I'm not sure how that's possible.

According to the example module, it imports re and itertools. Both of those
have already been imported, and so will be cached in sys.modules, as will
all their dependencies. So I don't see how it is possible that ModuleFinder
can find dependencies that aren't cached.

Theoretically, of course some dependency might import a bunch of modules,
then delete them from sys.modules. If it were only one or two, I'd believe
that. But not 107 of them.

After running the module finder on the sample file, I ran the report()
method to get a nicer display of the imported modules:

py> finder = ModuleFinder()
py> finder.run_script('/tmp/bacon.py')
py> finder.report()

Name File
---- ----
m __future__ /usr/local/lib/python3.5/__future__.py
m __main__ /tmp/bacon.py
m _ast
m _bootlocale /usr/local/lib/python3.5/_bootlocale.py
m _bz2 /usr/local/lib/python3.5/lib-dynload/_bz2.cpython-35m-i386-linux-gnu.so
[...]


which shows the unittest package being loaded, which is pretty dubious.


On the other hand, here's a simpler example which seems to work fine:


py> with open('/tmp/do_little.py', 'w') as f:
... f.write('import math\n')
...
12
py> finder = ModuleFinder()
py> finder.run_script('/tmp/do_little.py')
py> finder.report()

Name File
---- ----
m __main__ /tmp/do_little.py
m math /usr/local/lib/python3.5/lib-dynload/math.cpython-35m-i386-linux-gnu.so


So I'm not really sure what's going on.





--
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

[Wolfgang Maier]

Thanks for the detailed analysis.
You are right that it seems to work with certain imports. I verified
your example with import math and found a few additional examples.

Things seem to work with imports of any of the following (alone or in
combination):
io, itertools, math, sys

OTOH, I doubt that it reports correctly with:
os, heapq, operator, re

Personally, I fail to see a pattern here, but maybe somebody else can
explain that behaviour.
I'd really like to report this to the bug tracker, but at the moment I'm
not sure as what exactly. It seems that at least the documentation is
incorrect since the usage example there is not working.

Best,
Wolfgang