Installation process

[Craig Warren]

We are looking at streamlining the installation process for gprMax, i.e. trying to make the process for users and developers the same, and as simple as possible. Please note it is not going to be a double-clickable .exe file any time soon(!), but we would like some feedback on how you find the current process and your thoughts on our proposed changes. 

It would be great if you could complete the Doodle poll to tell as which operating system you are using http://doodle.com/poll/4vqw5tn9a4f8zye7

The current installation process (for Windows users) is:

1. Download source code and binary extensions from GitHub
2. Expand zip files and copy binary extensions into gprMax folder
3. Download and install Miniconda
4. Create the Python environment with packages
5. Download and install C library redistributables

We would like to avoid ourselves having to build binary extensions for multiple platforms for every release. This would also mean you can easily get the most up-to-date code all the time. So, the proposed new install procedure (for Windows users and developers) would be:

1. Download and install Miniconda
2. Install Git and checkout latest gprMax source code
3. Create the Python environment with packages
4. Download and install Microsoft Visual Studio Build Tools
5. Run ``python setup.py install`` to build and install gprMax

The beauty of this is that to update gprMax you would simply run ``git pull`` and then ``python setup.py install`` again.

What do you think? We don't think this is a more difficult procedure, and it offers advantages for everyone.

Kind regards,

Craig

[Craig Warren]

I have updated the Installation instructions, which now describe this new procedure: http://docs.gprmax.com/en/latest/include_readme.html#installation

Grateful for any feedback.

Craig

[Ivan Vasconcelos]

Hey Craig,

Congrats on a spectacular job putting all this together! (though I haven't managed to get it working yet... see below)

On the installation, your notes are fine and clear, but I did run into one problem: though I had installed gcc-6 (on Mac OS 10.11 El Capitan), when building the gprMax environment, the package mpi4py would not compile. I then tried compiling it with by hand pointing to the right path to gcc-6 (using sudo also), but no dice. Ultimately I had to install mpicc to be able to compile the mpi4py package. I then installed the remaining packages, had to remove the gprMax enviroment and then build it again. That seems to have worked.

However, when running you first example on your notes, I get this error:
(gprMax) Geo00494:gprMax ivanvasconcelos$ python -m gprMax user_models/cylinder_Ascan_2D.in
Traceback (most recent call last):
  File "/Users/ivanvasconcelos/miniconda3/envs/gprMax/lib/python3.5/runpy.py", line 174, in _run_module_as_main
    mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
  File "/Users/ivanvasconcelos/miniconda3/envs/gprMax/lib/python3.5/runpy.py", line 133, in _get_module_details
    return _get_module_details(pkg_main_name, error)
  File "/Users/ivanvasconcelos/miniconda3/envs/gprMax/lib/python3.5/runpy.py", line 109, in _get_module_details
    __import__(pkg_name)
  File "/Users/ivanvasconcelos/gprMax/gprMax/__init__.py", line 2, in <module>
    from .gprMax import api as run
  File "/Users/ivanvasconcelos/gprMax/gprMax/gprMax.py", line 40, in <module>
    from .fields_update import update_electric, update_magnetic, update_electric_dispersive_multipole_A, update_electric_dispersive_multipole_B, update_electric_dispersive_1pole_A, update_electric_dispersive_1pole_B
ImportError: dlopen(/Users/ivanvasconcelos/gprMax/gprMax/fields_update.cpython-35m-darwin.so, 2): Symbol not found: ___emutls_get_address
  Referenced from: /usr/local/opt/gcc6/lib/gcc/6/libgomp.1.dylib (which was built for Mac OS X 10.11)
  Expected in: /usr/lib/libSystem.B.dylib
 in /usr/local/opt/gcc6/lib/gcc/6/libgomp.1.dylib

It seems to suggest some type of mismatch of where to find the right OMP library? I'll try and figure it out, but if anyone knows the answer please let me know.

Cheers,
Ivan.

 

[Craig Warren]

Thanks Ivan.

Sounds like you have had a bit of a struggle so far! My own machine runs macOS Sierra and previously Mac OS X El Capitan, both with gcc6, so I know it can work ;-)

The mpi4py package is only really required if you intend to use our MPI based task farm, aimed at HPC environments. So if you find problems installing mpi4py you can edit the 'conda_env.yml' file in the gprMax package and remove the mpi4py line. This will stop it being installed as part of the gprMax conda environment.

But it seems the error you are getting is more likely related to your gcc6 installation and the OpenMP library. How did you install gcc6? I have successfully installed it through Homebrew on my machine. To install this way you usually have to run 'brew install gcc-6 --without-multilib' to ensure you get the OpenMP support.

Kind regards,

Craig

[Ivan Vasconcelos]

Hi Craig,

Thanks for the quick feed back.

Yes I installed gcc6 with homebrew, though the brew command on your page did not work for me (brew didn't know the 'gcc-6' formula name). Instead I followed the Homebrew instructions from:
https://deadline.today/sysadmin/GCC6-on-OSX/
and added the --without-multilib option.

Running
brew search gcc
for me gives
Geo00494:local ivanvasconcelos$ brew search gcc
gcc                                      homebrew/versions/gcc48
homebrew/versions/gcc43                  homebrew/versions/gcc49
homebrew/versions/gcc44                  homebrew/versions/gcc5
homebrew/versions/gcc45                  homebrew/versions/gcc6 ✔
homebrew/versions/gcc46                  homebrew/versions/llvm-gcc28
homebrew/versions/gcc47
homebrew/dupes/apple-gcc42               Caskroom/cask/gcc-arm-embedded

So no gcc-6 formula (explicitly).

Cheers,
Ivan.

[Craig Warren]

Thanks, the gcc-6 is a typo (I'll correct it in the install instructions). Either 'brew install gcc --without-multilib' or 'brew install gcc6 --without-multilib' should both currently install GCC 6.x using Homebrew.

What output do you get from the command 'brew info gcc'?

Craig

[Ivan Vasconcelos]

OK, thanks.

'brew info gcc' gives
Geo00494:local ivanvasconcelos$ brew info gcc
gcc: stable 6.2.0 (bottled), HEAD
GNU compiler collection
https://gcc.gnu.org
Not installed
From: https://github.com/Homebrew/homebrew-core/blob/master/Formula/gcc.rb
==> Dependencies
Required: gmp ✔, libmpc ✔, mpfr ✔, isl ✔
==> Options
--with-all-languages
    Enable all compilers and languages, except Ada
--with-java
    Build the gcj compiler
--with-jit
    Build the jit compiler
--with-nls
    Build with native language support (localization)
--without-fortran
    Build without the gfortran compiler
--without-multilib
    Build without multilib support
--HEAD
    Install HEAD version
==> Caveats
GCC has been built with multilib support. Notably, OpenMP may not work:
  https://gcc.gnu.org/bugzilla/show_bug.cgi?id=60670
If you need OpenMP support you may want to
  brew reinstall gcc --without-multilib

I have run 'brew reinstall gcc --without-multilib' ... now the issue on installing it over is
Geo00494:local ivanvasconcelos$ brew reinstall gcc --without-multilib
==> Reinstalling gcc with --without-multilib
Error: You must `brew link isl` before gcc can be installed
Geo00494:local ivanvasconcelos$ brew link isl
Linking /usr/local/Cellar/isl/0.17.1...
Error: Could not symlink lib/pkgconfig/isl.pc
/usr/local/lib/pkgconfig is not writable.
Geo00494:local ivanvasconcelos$ brew info isl
isl: stable 0.17.1 (bottled), HEAD
Integer Set Library for the polyhedral model
http://isl.gforge.inria.fr
/usr/local/Cellar/isl/0.17.1 (79 files, 3.9M)
  Poured from bottle on 2016-10-19 at 13:45:49
From: https://github.com/Homebrew/homebrew-core/blob/master/Formula/isl.rb
==> Dependencies
Required: gmp ✔

This isl linking seems to be a common problem on OS X. I'm going through searches to see if I can sort it out...

Thanks again!

[Ivan Vasconcelos]

P.S.: not sure why I currently don't have OMP support as I did add the --without-multilb option on originally....

[Ivan Vasconcelos]

Ok, if I do

Geo00494:local ivanvasconcelos$ brew info gcc6
homebrew/versions/gcc6: stable 6.2.0 (bottled)
The GNU Compiler Collection
https://gcc.gnu.org
/usr/local/Cellar/gcc6/6.2.0 (5,156 files, 382.2M)
  Built from source on 2016-10-18 at 14:11:02 with: --without-multilib --with-all-languages --with-java
From: https://github.com/Homebrew/homebrew-versions/blob/master/gcc6.rb
==> Dependencies
Required: gmp ✔, libmpc ✔, mpfr ✔, isl014 ✔

==> Options
--with-all-languages
    Enable all compilers and languages, except Ada
--with-java
    Build the gcj compiler
--with-jit
    Build the jit compiler
--with-nls
    Build with native language support (localization)

--with-profiled-build
    Make use of profile guided optimization when bootstrapping GCC

--without-fortran
    Build without the gfortran compiler
--without-multilib
    Build without multilib support

Everything appears to be fine.... It seems 'brew install gcc' and 'brew install gcc6' don't do the same thing?

[Craig Warren]

I think can do the same thing, and currently both install GCC 6.x, just at slightly different paths, which I don't think this makes any difference. 

Can you run 'gcc-6 -v' and paste the output.

Craig

[Ivan Vasconcelos]

Ok , now I got
Geo00494:~ ivanvasconcelos$ brew info gcc

gcc: stable 6.2.0 (bottled), HEAD
GNU compiler collection
https://gcc.gnu.org

/usr/local/Cellar/gcc/6.2.0 (1,358 files, 242.6M)
  Built from source on 2016-10-19 at 15:18:26 with: --without-multilib
From: https://github.com/Homebrew/homebrew-core/blob/master/Formula/gcc.rb
==> Dependencies
Required: gmp ✔, libmpc ✔, mpfr ✔, isl ✔

==> Options
--with-all-languages
    Enable all compilers and languages, except Ada
--with-java
    Build the gcj compiler
--with-jit
    Build the jit compiler
--with-nls
    Build with native language support (localization)

--without-fortran
    Build without the gfortran compiler
--without-multilib
    Build without multilib support

--HEAD
    Install HEAD version

This is after sorting out 2 problems:
1) the read/write settings of lib/pkgconfig, by 'sudo chown ivanvasconcelos /usr/local/lib/pkgconfig'
2) I had to manually go into Xcode preferences and set the latest version of command line tools (CLT) under the Locations tab, even though in the bash terminal I was being told Xcode CLT was installed.

Then I ran 'brew install gcc --without-multilib'.

So now I get a different run-time error in the gprMax environment:

(gprMax) Geo00494:gprMax ivanvasconcelos$ python -m gprMax user_models/cylinder_Ascan_2D.in
Traceback (most recent call last):
  File "/Users/ivanvasconcelos/miniconda3/envs/gprMax/lib/python3.5/runpy.py", line 174, in _run_module_as_main
    mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
  File "/Users/ivanvasconcelos/miniconda3/envs/gprMax/lib/python3.5/runpy.py", line 133, in _get_module_details
    return _get_module_details(pkg_main_name, error)
  File "/Users/ivanvasconcelos/miniconda3/envs/gprMax/lib/python3.5/runpy.py", line 109, in _get_module_details
    __import__(pkg_name)
  File "/Users/ivanvasconcelos/gprMax/gprMax/__init__.py", line 2, in <module>
    from .gprMax import api as run
  File "/Users/ivanvasconcelos/gprMax/gprMax/gprMax.py", line 40, in <module>
    from .fields_update import update_electric, update_magnetic, update_electric_dispersive_multipole_A, update_electric_dispersive_multipole_B, update_electric_dispersive_1pole_A, update_electric_dispersive_1pole_B

ImportError: dlopen(/Users/ivanvasconcelos/gprMax/gprMax/fields_update.cpython-35m-darwin.so, 2): Library not loaded: /usr/local/opt/gcc6/lib/gcc/6/libgomp.1.dylib
  Referenced from: /Users/ivanvasconcelos/gprMax/gprMax/fields_update.cpython-35m-darwin.so
  Reason: image not found

I wonder if I need to make the enviroment again? ...

So sorry for this mess: but thanks for the help!

[Ivan Vasconcelos]

Oh I forgot to mention that between 1) and 2), and uninstalled and reinstalled isl.

[Craig Warren]

I wouldn't have thought you need to rebuild the Python environment again, it still looks like a gcc install error to me. You could try rebuilding gprMax with 'python setup.py build'.

Craig

[Ivan Vasconcelos]

Yup, so, I had to additionally install with 'brew install gcc6 --without-multilib', then build it again. Even though brew installed gcc6 with 'brew install gcc --without-multilib', and the Cellar is there when doing 'find Cellar' on /usr/local/ , gcc -v always points to clang instead of the brew-installed one. Now at least gcc6 points to the brew-installed one.

Now the modelling code runs! :-D But plotting is failing:
Output file: /Users/ivanvasconcelos/gprMax/user_models/cylinder_Ascan_2D.out

Running simulation, model 1 of 1: 100%|█████| 637/637 [00:00<00:00, 979.95it/s]
Memory (RAM) used: ~36.3MB

=== Simulation completed in [HH:MM:SS]: 0:00:00 ===============================

(gprMax) Geo00494:gprMax ivanvasconcelos$ python -m tools.plot_Ascan user_models/cylinder_Ascan_2D.out

Traceback (most recent call last):

  File "/Users/ivanvasconcelos/miniconda3/envs/gprMax/lib/python3.5/runpy.py", line 184, in _run_module_as_main
    "__main__", mod_spec)
  File "/Users/ivanvasconcelos/miniconda3/envs/gprMax/lib/python3.5/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/Users/ivanvasconcelos/gprMax/tools/plot_Ascan.py", line 25, in <module>
    import matplotlib.pyplot as plt
  File "/Users/ivanvasconcelos/miniconda3/envs/gprMax/lib/python3.5/site-packages/matplotlib/__init__.py", line 1131, in <module>
    rcParams = rc_params()
  File "/Users/ivanvasconcelos/miniconda3/envs/gprMax/lib/python3.5/site-packages/matplotlib/__init__.py", line 975, in rc_params
    return rc_params_from_file(fname, fail_on_error)
  File "/Users/ivanvasconcelos/miniconda3/envs/gprMax/lib/python3.5/site-packages/matplotlib/__init__.py", line 1100, in rc_params_from_file
    config_from_file = _rc_params_in_file(fname, fail_on_error)
  File "/Users/ivanvasconcelos/miniconda3/envs/gprMax/lib/python3.5/site-packages/matplotlib/__init__.py", line 1018, in _rc_params_in_file
    with _open_file_or_url(fname) as fd:
  File "/Users/ivanvasconcelos/miniconda3/envs/gprMax/lib/python3.5/contextlib.py", line 59, in __enter__
    return next(self.gen)
  File "/Users/ivanvasconcelos/miniconda3/envs/gprMax/lib/python3.5/site-packages/matplotlib/__init__.py", line 1000, in _open_file_or_url
    encoding = locale.getdefaultlocale()[1]
  File "/Users/ivanvasconcelos/miniconda3/envs/gprMax/lib/python3.5/locale.py", line 558, in getdefaultlocale
    return _parse_localename(localename)
  File "/Users/ivanvasconcelos/miniconda3/envs/gprMax/lib/python3.5/locale.py", line 486, in _parse_localename
    raise ValueError('unknown locale: %s' % localename)
ValueError: unknown locale: UTF-8

Possibly some typo in the plotting script in passing the 'localename' variable? I'll sort it out tomorrow.

Thanks  for all the help, Craig!

[Craig Warren]

So, 'gcc' should always point to Apple installed clang. 'gcc-6' (note the dash) should point to /usr/local/bin/gcc-6 which in turn points to /usr/local/Cellar/gcc/6.x. For me on different Macs 'brew install gcc --without-multilib' and 'brew install gcc6 --without-multilib' have had the same overall effect but YMMV.

The error you are getting with plotting is to do with Python somehow not recognising the region of the world (locale) your are in. What is the output when you run the command 'locale' from a Terminal? Mine is:

LANG="en_GB.UTF-8"

LC_COLLATE="en_GB.UTF-8"

LC_CTYPE="en_GB.UTF-8"

LC_MESSAGES="en_GB.UTF-8"

LC_MONETARY="en_GB.UTF-8"

LC_NUMERIC="en_GB.UTF-8"

LC_TIME="en_GB.UTF-8"

LC_ALL=

 I suspect you may have to set these either in Terminal prefs (see attached screenshot) by selecting 'Set locale environment variable on startup', or in your .bashrc or .profile.

Craig

[Ivan Vasconcelos]

Thanks, Craig,

I ended having to manually set the locale variable $LANG in my .bash_profile .

That seems to have done the trick. Now, still no plot:

(gprMax) Geo00494:gprMax ivanvasconcelos$ python -m tools.plot_Ascan user_models/cylinder_Ascan_2D.out

Traceback (most recent call last):
  File "/Users/ivanvasconcelos/miniconda3/envs/gprMax/lib/python3.5/runpy.py", line 184, in _run_module_as_main
    "__main__", mod_spec)
  File "/Users/ivanvasconcelos/miniconda3/envs/gprMax/lib/python3.5/runpy.py", line 85, in _run_code
    exec(code, run_globals)

  File "/Users/ivanvasconcelos/gprMax/tools/plot_Ascan.py", line 231, in <module>
    plt = mpl_plot(args.outputfile, args.outputs, fft=args.fft)
  File "/Users/ivanvasconcelos/gprMax/tools/plot_Ascan.py", line 165, in mpl_plot
    raise CmdInputError('Output(s) requested to plot: {}, but available output(s) for receiver {} in the file: {}'.format(', '.join(outputs), rx, ', '.join(availableoutputs)))
gprMax.exceptions.CmdInputError: Output(s) requested to plot: Ex, Ey, Ez, Hx, Hy, Hz, Ix, Iy, Iz, but available output(s) for receiver 1 in the file: Ex, Ey, Ez, Hx, Hy, Hz

That seems to be an actual error in the script (the I-field components not output by the modelling script)?

Otherwise, I'm nearly there! :-)

[Craig Warren]

Ha! Glad you are nearly there.

Pull the latest version from the master branch. The error you are seeing was from a bug in the plotting script introduced a few weeks ago and fixed a day ago.

Craig

[Ivan Vasconcelos]

Success! :-D

Thanks *very* much for all your help, Craig.

In the coming days, I'll contact you through email to discuss some science.

Cheers,
Ivan.

[Mohammed Mustapha]

Good Sir.
Thanks for this wonderful job.  I have a challenge with 
 gprMax installation Am at the
 stage of installing of Microsoft

Visual C++ Build Tools 2015. I keep on receiving a message that " a setup package is Missing or damage" Please, help me out? Thanks

[Craig Warren]

Hi Mohammed,

Welcome to the forum!

Have you followed the install instructions and tried watching the screencast (http://docs.gprmax.com/en/latest/screencasts.html#installation)?

Can you post a screenshot of the error you are receiving?

Kind regards,

Craig

[Mohammed Mustapha]

Thank u very much Sir, I have followed Have the installation instructions and watched the screencast. That was what enable me to reach the stage I am presently.
below is the message i have been receiving. I tried the download package from internet option but to no avail.

[Craig Warren]

Hi Mohammed,

This is obviously an issue with installing the Visual Studio C compiler on Windows. It is not a problem we have seen before with other users, and is not related to the gprMax software itself. Therefore all I can suggest is that you have a look at the info in this thread http://stackoverflow.com/questions/32445763/a-setup-package-is-either-missing-or-damaged-error-while-installing-visual-stu Perhaps there was a problem with Microsofts server when you tried it, or you can find an offline installer to download.

Kind regards,

Craig

[AI]

Hi Craig,

I already updated miniconda and installed git, and also changed the path in Environment Variables. I'm getting an error message that git is not recognized though. 

I then manual downloaded the latest package from gitHub, but then I can't create the environment. What am I missing?

Thanks for your help.

[Craig Warren]

Hi Al,

There should be no need to be installing or working inside the Miniconda directory. Once you have Miniconda downloaded and installed, close and re-launch the Command Prompt. Then you should install git with conda as per the instructions. It is highly recommended to use conda/miniconda with gprMax as it keeps all the required Python packages installed in a single environment that is separate from any other Python install on your machine.

Kind regards,

Craig

[AI]

I already installed miniconda + git per the instructions. I also closed and relaunched the command prompt. Please see the attached screenshot with my comments. git is still not recognized for some reason. I even restarted my laptop. What am I missing?

[Craig Warren]

Looks like Miniconda is not correctly set on your path, which is why git is not found. Can you post a screenshot of your PATH environment variable settings.

Craig

[AI]

Please see attached the screen shot of the environment path. 

On a different note, when I type "conda env create -f conda_env.yml", I got this error message:

"SpecNotFound: Can't process without a name"

[Craig Warren]

So indeed Miniconda is not correctly installed, as there should be more items to other Miniconda directories that are automatically added to your path. I suggest deleting and re-installing Miniconda. Watch the screencast carefully (switch on HD to read all the text) if you are still unsure.

Also you should not be operating inside the Miniconda scripts directory - install (or clone) gprMax elsewhere, i.e. your desktop. Finally, when you try to create the conda environment you must be inside the top gprMax directory, i.e. not the Miniconda scripts directory.

Craig

[AI]

Actually, I had manually changed the path to Scripts directory. Please see attached the original path. 

What is the right path to set? Should it be the top GPRMax directory?

I have additionally attached a snapshot of my miniconda directory.

[Craig Warren]

Just let the Miniconda install set the path - there are a number of them to various bin directories - I would not try to do this manually.

Craig

[Trevor Hillebrand]

Hi Craig,

I updated using the commands from the Getting Started page, and now I am unable to plot.

When I run the command

python -m tools.plot_Ascan user_models/cylinder_Ascan_2D.out

It gives me a long traceback, ending with

ImportError: cannot import name 'get_backend'

I am running Python 3.6.7 on Mac OS 10.12.6.

Thank you for your help,

Trevor

[Craig Warren]

Hi Trevor,

Yes, I just spotted a similar error occurred on my Mac after I updated my conda environment. There seems to be issues between Python 3.6.7 and Python 3.7.x and the mkl package. These can be fixed by ensuring Python 3.6.6 and mkl 2018.0.0 are used. I have now specified these in the environment file. So if you pull the latest from with git, and then update your conda environment with:

conda env update -f conda_env.yml

This should sort the issue, until I get some time to investigate more the combination of the versions of packages that is causing problems.

Kind regards,

Craig

[Trevor Hillebrand]

Hi Craig,

That worked! I am now able to plot results. However, my results using the provided cylinder_Ascan_2D.in result in what looks like a numerical instability of some type. I am using the exact input file provided in the tutorial and running the commands as described there. I am attaching a .png of the results. Note that they are missing the third column shown in the tutorial, and that the Ez, Hx, and Hy field solutions blow up.

I tried going back through the process of updating gprMax outlined on the Getting Started page, and I noticed that

python setup.py install

ends in the following error:
ValueError: bad marshal data (unknown type code)

I'm attaching a screenshot of the rest of the traceback.

Thank you,
Trevor

[Craig Warren]

Thanks. Yeh there were a couple of further gotchas I had to iron out:

1. Adding removal of .pyc files to our 'cleanall' procedure. A bad/corrupt .pyc file was causing your install error.
2. Further updated the environment file. Double equals is required to force use of a specific version of a package. In this case mkl==2018.0.0, which should work with Python 3.6.6 or Python 3.6.7 but does not work with Python 3.7.x

I need to look further at what has changed with mkl to cause the problem, but if you pull the latest code (git pull), update your environment (conda env update -f conda_env.yml), then clean (python setup.py cleanall), and finally build (python setup.py build) and install (python setup.py install), it should now work correctly.

Let me know if you still have problems.

Kind regards,

Craig

[Trevor Hillebrand]

Hi Craig,

Thanks! That solution seems to get me most of the way there. I'm now getting the solution shown in the tutorial, except that my plot only has the first two columns (E and H), but not the third (I).

When I run

python setup.py install

it results in the same error (ValueError: bad marshal data (unknown type code)) as before, though this time the traceback leads to line 183 in setup.py.

Thanks for all your help.

Trevor

[Craig Warren]

Good.

The 'I' column in the example image was plotting currents (Ix, Iy, Iz). This use to happen by default which is why it is shown in the example image, but we changed this behaviour and made it an option. I've updated the example image now to avoid confusion in the future, but it is nothing to worry about. You can specify currents as an optional output for receivers, see http://docs.gprmax.com/en/latest/input.html#rx

Strange that you are still getting the 'bad marshal data' error. Did you definitely pull the latest setup.py code and run the cleanall step?

Kind regards,

Craig

[Trevor Hillebrand]

Okay, good! Then it looks like I'm up and running.

Regarding the bad marshal data error: I performed all the steps as you outlined them. However, since the simulation is working and plotting, it doesn't seem to be affecting anything yet.

Thanks,

Trevor

[Trevor Hillebrand]

Actually, perhaps I have found what is missing from the installation. I am trying to run the jupyter notebook example_Ascan_2D, and I get the following error every time it calls to import from gprMax:

---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
<ipython-input-10-32f7855d4488> in <module>
      1 get_ipython().run_line_magic('matplotlib', 'inline')
      2 
----> 3 from gprMax.waveforms import Waveform
      4 from tools.plot_source_wave import check_timewindow, mpl_plot
      5 

ModuleNotFoundError: No module named 'gprMax'


These commands work when I run them in ipython.

Sorry to be so much trouble!

Trevor

[Craig Warren]

Hi Trevor,

The Jupyter notebooks are working OK for me. I am fairly sure the reason for them not working for you is because of the 'bad marshal' error you get when you try to install gprMax. If the installation phase is not completing then gprMax will not be installed as a module that can be imported into the Jupyter notebook. Can you post the full traceback of the 'bad marshal' error again.

Thanks,

Craig

[Felipe Augusto]

Hello Trevor,

Did you try activate gprMax on the anaconda (or miniconda) prompt before run Jupyter Notebook?

Try this e tell me if works!

[Trevor Hillebrand]

Hi Felipe,

Yes, I activated gprMax before running Jupyter Notebook.

Craig,

Here is the full traceback:

Traceback (most recent call last):
  File "setup.py", line 183, in <module>
    include_dirs=[np.get_include()])
  File "/Users/trevorhillebrand/miniconda3/envs/gprMax/lib/python3.6/site-packages/setuptools/__init__.py", line 140, in setup
    return distutils.core.setup(**attrs)
  File "/Users/trevorhillebrand/miniconda3/envs/gprMax/lib/python3.6/distutils/core.py", line 148, in setup
    dist.run_commands()
  File "/Users/trevorhillebrand/miniconda3/envs/gprMax/lib/python3.6/distutils/dist.py", line 955, in run_commands
    self.run_command(cmd)
  File "/Users/trevorhillebrand/miniconda3/envs/gprMax/lib/python3.6/distutils/dist.py", line 974, in run_command
    cmd_obj.run()
  File "/Users/trevorhillebrand/miniconda3/envs/gprMax/lib/python3.6/site-packages/setuptools/command/install.py", line 67, in run
    self.do_egg_install()
  File "/Users/trevorhillebrand/miniconda3/envs/gprMax/lib/python3.6/site-packages/setuptools/command/install.py", line 109, in do_egg_install
    self.run_command('bdist_egg')
  File "/Users/trevorhillebrand/miniconda3/envs/gprMax/lib/python3.6/distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/Users/trevorhillebrand/miniconda3/envs/gprMax/lib/python3.6/distutils/dist.py", line 974, in run_command
    cmd_obj.run()
  File "/Users/trevorhillebrand/miniconda3/envs/gprMax/lib/python3.6/site-packages/setuptools/command/bdist_egg.py", line 220, in run
    os.path.join(archive_root, 'EGG-INFO'), self.zip_safe()
  File "/Users/trevorhillebrand/miniconda3/envs/gprMax/lib/python3.6/site-packages/setuptools/command/bdist_egg.py", line 273, in zip_safe
    return analyze_egg(self.bdist_dir, self.stubs)
  File "/Users/trevorhillebrand/miniconda3/envs/gprMax/lib/python3.6/site-packages/setuptools/command/bdist_egg.py", line 383, in analyze_egg
    safe = scan_module(egg_dir, base, name, stubs) and safe
  File "/Users/trevorhillebrand/miniconda3/envs/gprMax/lib/python3.6/site-packages/setuptools/command/bdist_egg.py", line 422, in scan_module
    code = marshal.load(f)

ValueError: bad marshal data (unknown type code)

Thanks,

Trevor

[Craig Warren]

Hi Trevor,

I have done a bit of searching and the 'bad marshal' error seems to point to a corrupt .pyc file somewhere in your environment. The 'cleanall' step should have deleted these but I think the next step is to delete and rebuild the gprMax conda environment, i.e.

conda env remove --name gprMax
conda clean --all
conda env create -f conda_env.yml
conda activate gprMax
python setup.py cleanall
python setup.py build
python setup.py install

Let me know if this fixes it.

Kind regards,

Craig

[Trevor Hillebrand]

Hi Craig,

I'm sorry to say that did not fix it.

Trevor

[Craig Warren]

Hi Trevor,

Thats annoying! I don't have any other suggestions at the moment, but will continue to look into as and when I have time.

Kind regards,

Craig

[Trevor Hillebrand]

Hi Craig,

Here's an update. As per usual, I fixed one thing and broke another.

I was able to get through the installation steps you suggested above by removing my old gprMax directory and starting all the way at the beginning with

git clone https://github.com/gprMax/gprMax.git

It no longer gives the bad marshal data error when I run

python setup.py install

However, now when I run any command in the gprMax environment (from command line or Jupyter Notebooks), I get the following error:

Traceback (most recent call last):

  File "/Users/trevorhillebrand/miniconda3/envs/gprMax/lib/python3.6/runpy.py", line 183, in _run_module_as_main
    mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
  File "/Users/trevorhillebrand/miniconda3/envs/gprMax/lib/python3.6/runpy.py", line 142, in _get_module_details
    return _get_module_details(pkg_main_name, error)
  File "/Users/trevorhillebrand/miniconda3/envs/gprMax/lib/python3.6/runpy.py", line 109, in _get_module_details
    __import__(pkg_name)
  File "/Users/trevorhillebrand/Documents/Antarctica/Radar/gprMax/gprMax/__init__.py", line 2, in <module>
    from .gprMax import api as run
  File "/Users/trevorhillebrand/Documents/Antarctica/Radar/gprMax/gprMax/gprMax.py", line 38, in <module>
    from gprMax.model_build_run import run_model
  File "/Users/trevorhillebrand/Documents/Antarctica/Radar/gprMax/gprMax/model_build_run.py", line 44, in <module>
    from gprMax.fields_updates_ext import update_electric
ImportError: dlopen(/Users/trevorhillebrand/Documents/Antarctica/Radar/gprMax/gprMax/fields_updates_ext.cpython-36m-darwin.so, 2): Symbol not found: _GOMP_parallel
  Referenced from: /Users/trevorhillebrand/Documents/Antarctica/Radar/gprMax/gprMax/fields_updates_ext.cpython-36m-darwin.so
  Expected in: flat namespace
 in /Users/trevorhillebrand/Documents/Antarctica/Radar/gprMax/gprMax/fields_updates_ext.cpython-36m-darwin.so

It seems this issue has been raised before (here and here), but setup.py seems to have already incorporated these workarounds, i.e.:

linker_args = ['-fopenmp', '-Wl,-rpath,' + rpath]

I am using gcc7 on Mac OS 10.13.2.

Thanks,

Trevor

[Craig Warren]

Hi Trevor,

Glad one problem is fixed!

Did you install gcc7 via Homebrew? Can you check if 'libgomp.1.dylib' exists at '/usr/local/opt/gcc/lib/gcc/7/'? Also print the output of:

otool -L /Users/trevorhillebrand/Documents/Antarctica/Radar/gprMax/gprMax/fields_updates_ext.cpython-36m-darwin.so

Thanks,

Craig

[Trevor Hillebrand]

Hi Craig,

This is interesting.

otool -L /Users/trevorhillebrand/Documents/Antarctica/Radar/gprMax/gprMax/fields_updates_ext.cpython-36m-darwin.so

returns

/Users/trevorhillebrand/Documents/Antarctica/Radar/gprMax/gprMax/fields_updates_ext.cpython-36m-darwin.so:    @rpath/libgomp.1.dylib (compatibility version 2.0.0, current version 2.0.0)
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1252.0.0)
    /usr/local/lib/gcc/7/libgcc_s.1.dylib (compatibility version 1.0.0, current version 1.0.0)

But there is no /usr/local/opt/gcc/lib/gcc/7. I do, however, have  /usr/local/opt/gcc/lib/gcc/8, which does contain 'libgomp.1.dylib',  but the 'build' step does not seem to find that.

I did install gcc7 using homebrew:

brew install gcc@7

Thanks,

Trevor

[Craig Warren]

Hmmm...looks like you have a strange mixture of gcc7 and gcc8. I see 'libgcc' is coming from your gcc7 install. The 'setup.py' simply looks for the latest gcc in /usr/local/bin and sets the rpath accordingly. You could try to resolve your mixture of gcc's or modify the setup script to look for 'libgomp.1.dylib' in another location, but I'm not sure I'd recommend that.

Kind regards,

Craig

[Trevor Hillebrand]

Okay, I just uninstalled my older versions of gcc (6&7, it turns out) using Homebrew and reinstalled gprMax. Now everything works, including Jupyter Notebooks! Thank you for your help and patience!

Best,

Trevor

[Craig Warren]

Great. Glad it is all solved. 

It would be great to hear how you are using or plan to use gprMax - it helps us fund and develop the code if we know how people use it, and the potential impact it may have in different areas of research.

Kind regards,

Craig

[Heloisa Moraes]

Hi everyone.

I have a new computer and to install gprMax I followed all the steps listed in: http://docs.gprmax.com/en/latest/include_readme.html#installation

Everything went well during the process, but when I try to produce the geometry view of my model I get the ModuleNotFoundError: No module named 'gprMax.fields_updates_ext' error. This is the third time I repeat the installation.

I still have my old machine, so I updated the gprMax on it and I'm using it.  One thing caught my attention: on both machines I can only update gprMax before activating it  as you will see in the image bellow (from my new computer)

The message is in portuguese and says: 'git' is not recognized as an internal or external command, an operable program, or a batch file

What can be (or am I doing) wrong?

Grateful for any feedback.

Heloisa

[Craig Warren]

Hi Heloisa,

Once you have activated the gprMax environment make sure you then do the build and install steps:

(gprMax)$ python setup.py build

(gprMax)$ python setup.py install

The error you are receiving is because the built (Cython) modules cannot be found.

The git error is most probably caused by the fact git is only installed in your base conda environment. gprMax can be updated without having its environment activated.

Kind regards,

Craig

[Heloisa Moraes]

Hi Craig,

I followed your instructions and everything worked out. I was able to produce my geometry view. Thank you very much.

Best regards,

Heloisa