Not clear where python modules are imported from.

[Michael Jensen]

Greetings!

Almost there!

It now appears that I have a functioning clawpack installation!

(roughly speaking, I followed this: https://github.com/clawpack/clawpack/issues/238)

Now, my objective with all this is to figure out how the source code actually works, not only to run it.  So, with that in mind, (I'm specifically interested in the euler equations), I am looking at examples solving them in pyclaw.

(Note that in this, I am not referring to the documentation, since it isn't the same.  Documentation is out-of-date frequently enough to be treated, always, with a grain of salt)

The first uncommented line of shocktube.py is:

from clawpack import riemann

then later,

def setup(use_petsc=False, outdir='./_output', solver_type='classic',
          kernel_language='Python',disable_output=False):

(in other words, kernel_language='Python')

then..

    if kernel_language =='Python':
        rs = riemann.euler_1D_py.euler_hllc_1D

however, (first confusion), riemann does not contain a program called euler_1D_py.  that code is in riemann/riemann.  So.. I have to assume that this is a feature of python imports, that nested folders of the same name are treated as the same folder?

Slightly further down is:

    elif solver_type=='classic':
        solver = pyclaw.ClawSolver1D(rs)

..however, pyclaw does not contain ClawSolver1D.  in fact, I search for it in the entire clawpack directory, and it isn't there.  ..but when I try to run the program, it runs!

..So there is clearly something fundamental I'm not understanding.  Equally clearly, this is not so much an issue with clawpack itself, but an issue with my understanding of python.  That said, unless it is already there and I just haven't found it, it might be useful to have a section of the instructions claritying this.  I'll even volunteer to write it :-)

Many thanks!

-Mike :-)

[David Ketcheson]

Dear Mike,

As you yourself say, what you are missing is an understanding of the basics of how Python modules are structured.  I don't think it makes sense for Clawpack's documentation to include a tutorial on this topic, since it is explained already in hundreds of places on the internet and has little to do with Clawpack specifically.  I recommend you spend time learning this material from one of those many available and excellent sources.  For instance, you could start here:

https://towardsdatascience.com/understanding-python-imports-init-py-and-pythonpath-once-and-for-all-4c5249ab6355

I guess that it would also be useful for you to learn to use a tool like grep or ack, since you say you aren't able to find terms that are in fact present in (files in subdirectories of) your Clawpack directory.

I will give you one quick tip: if you type the name of an imported module within a Python interpreter, then it will give you the location of the file for that module.

David Ketcheson

[Michael Jensen]

ok, you're clearly a paying user of medium.  Or somebody who works for them.  I'm not.  I'll see if I can track down another equivalent source, you've said there are many.  I'll message later with the link once/if I do. :-)

[Michael Jensen]

Thank you very much David for the tips!

I know of the existence of grep, and have codes built around the concept, but I'm not yet in the habit of using it regularly.  When I said I couldn't find them anywhere in the directory, what I did was to do a search with.. whatever it is that ubuntu uses to display your files.  In my experience, if you are in a given directory, and you search for a given set of characters, it will display for you all the files, anywhere in that directory and it's sub-directories, that has that combination of characters in it's name.  I did that search, and got nothing.  I'll report back later if ever I figure out why.. :-)

On Saturday 17 February 2024 at 00:25:42 UTC-5 David Ketcheson wrote:

[Michael Jensen]

I'm sorry.  I wish I could take back that last message.  The next best thing I can do is apologize, thank you again for all your efforts to help, and move on :-)  I'm looking deeper into grep, and into a long list of tutorials.  I'll probably pay the fee, and it'll likely be worth it.  I have a lot of learning to do. :-)

On Saturday 17 February 2024 at 00:25:42 UTC-5 David Ketcheson wrote:

[David Ketcheson]

No worries.  But I think you can get most of the information you want easily in the Python interpreter, once you know a little more.  For instance:

> from clawpack import pyclaw

> pyclaw.ClawSolver1D

returns

> clawpack.pyclaw.classic.solver.ClawSolver1

which tells you exactly where to find it.

[Michael Jensen]

Greetings again :-)

I'm mostly just feeling tired and responding somewhat at random.  It's been a while because I got distracted by another project for several weeks.  I've read enough tutorials, and explored enough code to be able, at the very least, to find most of the codes that the particular example I'm trying to run actually uses.  One thing is very clear now:  I'm a python novice.  At least by the standards of this software.  The only word I can use to describe the architecture so far is "extremely sophisticated", and I can claim at best a vague understanding of what it's doing.  I think I had maybe one course on object oriented programming way back when, and haven't needed it since.  I'm still at the stage of working my way through the "user interface".  I was last looking at the __init__ of the "Solution" class, for example.  It motivated me to attempt running the code and injecting print-statements so that it'll tell me what the different variables actually are, allowing me to trace what it's going to do.  Assuming that that works.

..so I tracked down instructions on how to run the examples from the command-line.  I followed the installation instructions way back when, and the examples it told me to run seemed to work. ..now.. (it's been a while)

the instruction is the following:

$ cd clawpack/pyclaw/examples/acoustics_1d_homogeneous

$ python acoustics.py iplot=1

In my case, acoustics doesn't exist, but acoustics_1d does.

it gives me the following:

(base) eriadar@eriadar-Latitude-E7440:~/clawpack/pyclaw/examples/acoustics_1d_homogeneous$ python acoustics_1d.py iplot=1
Traceback (most recent call last):
  File "/home/eriadar/clawpack/pyclaw/examples/acoustics_1d_homogeneous/acoustics_1d.py", line 22, in <module>
    from clawpack import riemann
ModuleNotFoundError: No module named 'clawpack'

..so I assume I must have a non-standard installation somehow.

for kicks, I also tried your instructions,

"> from clawpack import pyclaw

> pyclaw.ClawSolver1D"

and got..

(base) eriadar@eriadar-Latitude-E7440:~$ python
Python 3.11.5 (main, Sep 11 2023, 13:54:46) [GCC 11.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.

>>> from clawpack import pyclaw
>>> pyclaw.ClawSolver1D

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: module 'clawpack.pyclaw' has no attribute 'ClawSolver1D'
>>>

..I'm in anaconda -> [you can see the (base)], would that mess things up?

I found ClawSolver1D somehow.. I think maybe grep helped me with that..

it's in:

/clawpack/pyclaw/src/pyclaw/classic   ..though of course you know that..  (it's a class, right?)

..I've now gone back to the top of the "testing a pyclaw installation", and tried running:

cd pyclaw/examples

nosetests

, seeing as my browser seemed to indicated that I had previously only tested the fortran installation.  I ended up having to install nosetests again (is there a different version for python than for fortran?), and now it's giving me:

(base) eriadar@eriadar-Latitude-E7440:~/clawpack/pyclaw/examples$ nosetests
E
======================================================================
ERROR: Failure: ImportError (No module named numpy)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/nose/loader.py", line 418, in loadTestsFromName
    addr.filename, addr.module)
  File "/usr/lib/python2.7/dist-packages/nose/importer.py", line 47, in importFromPath
    return self.importFromDir(dir_path, fqname)
  File "/usr/lib/python2.7/dist-packages/nose/importer.py", line 94, in importFromDir
    mod = load_module(part_fqname, fh, filename, desc)
  File "/home/eriadar/clawpack/pyclaw/examples/__init__.py", line 1, in <module>
    from .advection_1d import advection_1d
  File "/home/eriadar/clawpack/pyclaw/examples/advection_1d/advection_1d.py", line 19, in <module>
    import numpy as np
ImportError: No module named numpy

----------------------------------------------------------------------
Ran 1 test in 0.000s

FAILED (errors=1)

...so.. maybe I need to install numpy for this particular virtual environment of anaconda?

(base) eriadar@eriadar-Latitude-E7440:~/clawpack/pyclaw/examples$ python
Python 3.11.5 (main, Sep 11 2023, 13:54:46) [GCC 11.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>
(base) eriadar@eriadar-Latitude-E7440:~/clawpack/pyclaw/examples$ pip3 install numpy
Requirement already satisfied: numpy in /home/eriadar/anaconda3/lib/python3.11/site-packages (1.26.3)

..I see in the list of directories a mention of python 2.7.  ..does that mean I should install python2?  If I were convinced of that, I'd try it, but I'm not.  The whole point of virtual environments, as I understand it, is to keep these versions separate..

..I'll keep poking it.  I'll probably figure it out in a few days.. :-)

[Michael Jensen]

I have now tried the next most obvious thing, importing numpy from where it's claimed to be using

import sys
sys.path.append()

that didn't work.  I got some error about a syntax error in the numpy __init__ file.  obviously the wrong solution to the problem.  I checked, and at least one person said "don't do that".  (importing numpy directly from it's directory).  I currently have no other ideas for what to try.  switching to other things for now. :-)

[Aron Ahmadia]

I suspect you don't have "nose/nosetests" installed in the environment that you're calling pyclaw/clawpack from :)

I haven't installed pyclaw recently, but you should try the "pip install nose" command and also "which nosetests" to figure out if the binaries don't line up.

--
You received this message because you are subscribed to the Google Groups "claw-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to claw-users+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/claw-users/bd5eab72-f1b3-40bb-a988-c50ad5c75f43n%40googlegroups.com.

[Michael Jensen]

thanks!  ..I can't help but wonder if I failed to report something I had done..

I tried using both pip and pip3 to install nose, as suggested, and pip -did- do something, following which pip3 claimed it was installed, so I guess those two versions are the same, and that, in this case, apparently, nose -wasn't- installed.  I'm sure I installed it before, but.. maybe that was the pip3 version.  -that- would(could) make sense.  the program is still unable to find numpy.  nose is installed to:

(base) eriadar@eriadar-Latitude-E7440:~/clawpack/pyclaw/examples$ which nosetests
/home/eriadar/anaconda3/bin/nosetests

my attempt to grasp "line up the binaries", so far has given the following:

(base) eriadar@eriadar-Latitude-E7440:~/clawpack/pyclaw/examples$ which clawpack
(base) eriadar@eriadar-Latitude-E7440:~/clawpack/pyclaw/examples$ which pyclaw
(base) eriadar@eriadar-Latitude-E7440:~/clawpack/pyclaw/examples$ which numpy
(base) eriadar@eriadar-Latitude-E7440:~/clawpack/pyclaw/examples$

searching the net, I'm of course finding all kinds of things, but I'm wary of diving too deep, seeing as what I'll likely find will be generic advice that may or may not apply to this particular situation.  I currently have clawpack installed to: /home/eriadar/clawpack.

..numpy is in: /home/eriadar/anaconda3/lib/python3.11/site-packages ...

..so does that mean I should re-install (or move) clawpack inside the anaconda directory?  if so, to where precisely?

[Marc Kjerland]

Michael,

I suspect you’re struggling with multiple versions of Python on your machine (which is a common frustration for Python novices) and I also suspect you may not have assigned environment variables to point your system to Clawpack.

Anaconda manages a local Python 3 installation. Your error messages suggest nosetests it is using a system-wide Python 2.7 version.

Try some commands such as:

which pip

which pip3

which python

which python3

which nosetests

echo $CLAW

The latter prints an environment variable that should list your Clawpack location; if it’s blank you need to update ~/.bashrc based on the Clawpack installation instructions.

See my email to this group yesterday for some guidance; it uses system-wide ‘apt install’ rather than Anaconda, but you can modify it for your case if you prefer Anaconda (or delete

/home/eriadar/anaconda3 and start fresh).

Also, search Stack Overflow as many of these errors are classic Python versioning issues.

 

Marc

To view this discussion on the web visit https://groups.google.com/d/msgid/claw-users/eb2a1da0-78ff-4d33-a32e-4a4c4aeebf42n%40googlegroups.com.

[Michael Jensen]

Thanks!  Something in this message or your other one helped clue me in as to what was wrong.

If you look back at the error messages I reported, you'll see I was in the (base) environment.  I don't remember what was installed in there (I guess the default base system python installation, 2.7?), but when I ran my tests of clawpack before, I instead used another environment (which is what anaconda was created to allow one to do), called py3p11 (for the python version it uses).  so I switched to that one, and that particular problem vanished.  I got a new error, something related to "collections.Callable", but that one I've seen before, know how to fix.

A new one now, for the reccord, is something related to Rossby_wave:

  File "/home/eriadar/clawpack/pyclaw/examples/shallow_sphere/Rossby_wave.py", line 444, in setup
    auxtmp = problem.setaux(mx,my,num_ghost,mx,my,xlower,ylower,dx,dy,auxtmp,Rsphere)
             ^^^^^^^
NameError: name 'problem' is not defined

More full error:

======================================================================
ERROR: Test solution of shallow water equations on the sphere.

----------------------------------------------------------------------
Traceback (most recent call last):

  File "/home/eriadar/anaconda3/envs/py3p11/lib/python3.11/site-packages/nose/case.py", line 199, in runTest
    self.test(*self.arg)
  File "/home/eriadar/clawpack/pyclaw/examples/shallow_sphere/test_shallow_sphere.py", line 23, in test_shallow_sphere
    return test_app(Rossby_wave.setup,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/eriadar/clawpack/pyclaw/src/pyclaw/util.py", line 235, in test_app
    claw = application(**kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/eriadar/clawpack/pyclaw/examples/shallow_sphere/Rossby_wave.py", line 444, in setup
    auxtmp = problem.setaux(mx,my,num_ghost,mx,my,xlower,ylower,dx,dy,auxtmp,Rsphere)
             ^^^^^^^
NameError: name 'problem' is not defined
-------------------- >> begin captured stdout << ---------------------
{'disable_output': True}

--------------------- >> end captured stdout << ----------------------

----------------------------------------------------------------------
Ran 49 tests in 37.113s

FAILED (SKIP=2, errors=4)

--------------------------------------------------------------------------------------------

..I haven't tried that hard to solve it, and it probably isn't relevant until I try to run that particular example, if I ever do.

  Thanks!