Running MOOSE on HPC

440 views
Skip to first unread message

William Harris

unread,
Jul 20, 2015, 12:30:05 PM7/20/15
to moose...@googlegroups.com
I am trying to use the HPC-provided moose module. Here are the commands I ran:

ssh -X (my-username)@falcon1
cd projects/moose
module load use.projects
module load moose-dev/5.6.0-gmvolf-5.5.4
scripts/update_and_rebuild_libmesh.sh

I then went into the modules/phase_field/ and ran make -j8, and then I went into tests/image_function and opened one of the tests with peacock. I got an error message:

ImportError: no module named vtk

I am unsure what to do at this point. Can anyone help with this so that I can start running my code on the HPC?

Thanks!

Garvey, Cormac T

unread,
Jul 20, 2015, 6:45:50 PM7/20/15
to moose...@googlegroups.com
Hi William,
I had moved the moose-dev modules to the projects location (to accommodate the moose team), in doing so I introduced an error in
the "moose-dev/5.6.0-gmvolf-5.5.4" module. This error is now corrected.

I confirmed that I could bring up peacock with the phase_field image_function test. (See the attached screen shot).

So, please try again.

module load use.projects
module load moose-dev/5.6.0-gmvolf-5.5.4

cd moose/scripts
export JOBS=8
update_and_rebuild_libmesh.sh
cd modules/phase_field
make -j 8
cd phase_field/tests/image_function
../../../../gui/peacock

Let me know if you still have any problems.

Thanks,
Cormac.

--
You received this message because you are subscribed to the Google Groups "moose-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to moose-users...@googlegroups.com.
Visit this group at http://groups.google.com/group/moose-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/moose-users/CAJQhoru_gXWhTE9pyDoWXi13BEFOG9m2HDOSZe3%2BAZ7wBq3JZg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.



--
Cormac Garvey
HPC Software Consultant
Scientific Computing
Idaho National Laboratory
Ph: 208-526-6294

Screenshot-3.png

William Harris

unread,
Jul 22, 2015, 4:04:58 PM7/22/15
to moose-users
Thank you for the help. However, I now get the error message peacock: cannot connect to X server localhost:55.0. Have you seen this before?

Cody Permann

unread,
Jul 22, 2015, 4:40:48 PM7/22/15
to moose-users
On Macs you sometime need to use "-Y" instead of "-X" when doing X-forwarding.

In general though, I find it much easier to build input files on my local system and use the clusters for running them. Developing interactively over a forwarded X session is really not all that much fun.

Cody

To view this discussion on the web visit https://groups.google.com/d/msgid/moose-users/c89fca65-6b39-436c-9b11-3eece3d04cc1%40googlegroups.com.

Garvey, Cormac T

unread,
Jul 22, 2015, 4:59:34 PM7/22/15
to moose...@googlegroups.com

The INL HPC environment now also has the sgi visServer (create a virtual terminal), it has high end graphics cards and 512GB of memory, this environment does not require X forwarding and should be good especially for GUI based interactive work and working with large visualization datasets

Just another option.

Thanks,
Cormac.

To view this discussion on the web visit https://groups.google.com/d/msgid/moose-users/CAK06EuXbAn8moan2ELjco-1De3_0VZC2PCDPP5pQF9cfap0T9w%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

William Harris

unread,
Jul 23, 2015, 12:29:36 PM7/23/15
to moose...@googlegroups.com
The command 'module load moose-dev/5.6.0-gmvolf-5.5.4' does not seem to work when I use the sgi visServer, so 'make -j 8' won't work either.

You received this message because you are subscribed to a topic in the Google Groups "moose-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/moose-users/C50VgJZhCRE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to moose-users...@googlegroups.com.

Visit this group at http://groups.google.com/group/moose-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/moose-users/CACw_P3XmSgQW%3DLZ78zCux%2BgLfk2qaa5wKS%3DJsK3v_t8Os77ThA%40mail.gmail.com.

Garvey, Cormac T

unread,
Jul 23, 2015, 1:01:10 PM7/23/15
to moose...@googlegroups.com
It works for me, Share your vizServer session with me (garvct), so I can see your terminal.

Thanks,
Cormac.

To view this discussion on the web visit https://groups.google.com/d/msgid/moose-users/CAJQhorufTFv0bU0B5XsGF0URJiDbnBWFsA1y%3DSBw3OOKbeN4NQ%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

Garvey, Cormac T

unread,
Jul 23, 2015, 1:12:08 PM7/23/15
to moose...@googlegroups.com

I worked with William using a shared visServer session, peacock is up and running on HPC.

Thanks,
Cormac.

William Harris

unread,
Jul 23, 2015, 2:11:24 PM7/23/15
to moose...@googlegroups.com
The problem is that, although peacock runs and I can run some moose programs, I cannot run the moose files that I need to run because I keep getting error messages that VTK is not enabled (I am using the image function in the phase field module), even though I ran the command:

./update_and_rebuild_libmesh.sh --with-vtk-include=$VTKINCLUDE_DIR --with-vtk-lib=$VTKLIB_DIR


To view this discussion on the web visit https://groups.google.com/d/msgid/moose-users/CACw_P3W1YL4Z85SpiFwbfhMK-RDgY7XdmdReC2WzsJrUFmSmVA%40mail.gmail.com.

Cody Permann

unread,
Jul 23, 2015, 2:22:41 PM7/23/15
to moose...@googlegroups.com
Debugging why this isn't working is not all that daunting. You need to look at your libMesh config.log file and see what it says. In there it attempts to look for vtk and it tells you where it's looking (i.e. it'll be in the path you passed on the command line) If those paths don't exist or are just wrong then it will fail.

To view this discussion on the web visit https://groups.google.com/d/msgid/moose-users/CAJQhoruL9GzysiQcoEdMvFRsuf6JP77oFYj32wA1R8xdFwJ1zA%40mail.gmail.com.

Miller, Jason M

unread,
Jul 23, 2015, 3:10:51 PM7/23/15
to moose...@googlegroups.com
William,

The VTK module that is included in the moose-dev module does not export those VTK environment variables you are attempting to use. However, you can determine where the lib directory is and then therefor the include directory by studying the LD_LIBRARY_PATH environment variable that does get modified when the VTK module is loaded, and then set the variables your self.

   module load use.projects moose-dev/5.6.0-gmvolf-5.5.4
   module list   (to see all modules loaded, and determine which VTK module was loaded)
   module display VTK/6.1.0-gmvolf-5.5.4-Python-2.7.8

Studying the output of the module display command reveals that VTK is located at:

   /apps/local/easybuild/software/VTK/6.1.0-gmvolf-5.5.4-Python-2.7.8

So in order to use VTK in the manner you are asking:

   export VTKINCLUDE_DIR=/apps/local/easybuild/software/VTK/6.1.0-gmvolf-5.5.4-Python-2.7.8/include/vtk-6.1
   export VTKLIB_DIR=/apps/local/easybuild/software/VTK/6.1.0-gmvolf-5.5.4-Python-2.7.8/lib
   ./update_and_rebuild_libmesh.sh --with-vtk-include=$VTKINCLUDE_DIR --with-vtk-lib=$VTKLIB_DIR


Hope this helps!
Jason


William Harris

unread,
Jul 27, 2015, 4:25:31 PM7/27/15
to moose...@googlegroups.com
Unfortunately, I still get the error:

*** ERROR ***
libMesh must be configured with VTK enabled to utilize ImageFunction



To view this discussion on the web visit https://groups.google.com/d/msgid/moose-users/CALgZTAf4R0yTw6qoXa_JzCPp1JddpzR9uMwzFWaSOSgtTMWCUQ%40mail.gmail.com.

William Harris

unread,
Jul 27, 2015, 4:27:26 PM7/27/15
to moose...@googlegroups.com
I even checked to make sure that VTK is located in the same place as you found before I entered in the export commands.

Miller, Jason M

unread,
Jul 28, 2015, 10:49:59 AM7/28/15
to moose...@googlegroups.com
I admit I do not know exactly what you are modifying (if anything) in the phase_field module tests. But all mine pass, with no skipped tests due to VTK missing:


[milljm@falcon1][~/projects/moose/modules/phase_field]> ./run_tests --re=image_function -j6
image_function.2d......................................................................................... OK
image_function.2d_elemental............................................................................... OK
image_function.threshold.................................................................................. OK
image_function.3d_subset.................................................................................. OK
image_function.threshold_adapt............................................................................ OK
image_function.3d......................................................................................... OK
image_function.component.................................................................................. OK
image_function.shift_and_scale............................................................................ OK
image_function.subset..................................................................................... OK
image_function.threshold_adapt_parallel...................................................... [MIN_CPUS=3] OK
image_function.flip....................................................................................... OK
image_function.flip_dual.................................................................................. OK
image_function.threshold_adapt_parallel_check_files.......................................... [MIN_CPUS=3] OK
image_function.crop....................................................................................... OK
image_function.flip_quad.................................................................................. OK
image_function/error.no_vtk............................................................. skipped (VTK!=false)
image_function.image_mesh_2d.............................................................................. OK
image_function.image_mesh_3d.............................................................................. OK
image_function/error.component............................................................................ OK
image_function/error.file_suffix.......................................................................... OK
image_function/error.invalid_file......................................................................... OK
image_function/error.threshold_values..................................................................... OK
-------------------------------------------------------------------------------------------------------------
Ran 21 tests in 8.1 seconds
21 passed, 1 skipped, 0 pending, 0 failed
[milljm@falcon1][~/projects/moose/modules/phase_field]>



This is on Falcon, using the moose-dev module/s described in a previous email on this thread. If you are running phase_field differently, can you explain how? I will attempt to duplicate your issue.

Thanks!
Jason




To view this discussion on the web visit https://groups.google.com/d/msgid/moose-users/CAJQhors-f6pLkmLSrqjsH_0O6eaorvjxKvS%2BvhktW-tFft%2BfxQ%40mail.gmail.com.

William Harris

unread,
Aug 8, 2015, 6:54:14 PM8/8/15
to moose...@googlegroups.com
Finally I resolved this issue by deleting my moose directory and starting from scratch using Jason's instructions. Thank you so much!

To view this discussion on the web visit https://groups.google.com/d/msgid/moose-users/CALgZTAceA8PugSU0hgt7YX44yRKz9Ca186QUrpTpsHe_oSOH8Q%40mail.gmail.com.

William Harris

unread,
Jun 7, 2017, 9:49:42 AM6/7/17
to moose...@googlegroups.com
I seem to have this problem every year for the past 3 years. I freshly cloned moose into my directory on the HPC. I loaded the necessary modules (use.projects, moose-dev/5.6.0-gmvolf-5.5.4). My goal is to use the Image Reader. I followed the instructions here


but those modules are not available on the HPC, so I then tried to follow the old instructions I was given 2 years ago (earlier in this post), so I executed module list, saw VTK/6.1.0-gmvolf-5.5.4-Python-2.7.8 in the module list, then executed

module display VTK/6.1.0-gmvolf-5.5.4-Pyton-2.7.8

which output

/apps/local/easybuild/modules/vis/VTK/6.1.0-gmvolf-5.5.4-Python-2.7.8.

So then I executed the following commands:


 export VTKINCLUDE_DIR=/apps/local/easybuild/software/VTK/6.1.0-gmvolf-5.5.4-Python-2.7.8/include/vtk-6.1
 export VTKLIB_DIR=/apps/local/easybuild/software/VTK/6.1.0-gmvolf-5.5.4-Python-2.7.8/lib
 ./update_and_rebuild_libmesh.sh --with-vtk-include=$VTKINCLUDE_DIR --with-vtk-lib=$VTKLIB_DIR

When that finished, I went to moose/modules/phase_field and executed make -j8. Last year there was an image_function test in phase_field/tests, but there isn't anymore. So I went to my own app and executed make -j8 to compile it. 

I must have broken something in the process above, because I received errors, including the following:

Linking Library /home/harrwh/projects_2017/moose/modules/module_loader/lib/libmodule_loader_with-opt.la...
In file included from /home/harrwh/projects_2017/moose/scripts/../libmesh/installed/include/libmesh/perf_log.h:25:0,
                 from /home/harrwh/projects_2017/moose/framework/include/base/Moose.h:19,
                 from /home/harrwh/projects_2017/moose/framework/include/base/MooseError.h:18,
                 from /home/harrwh/projects_2017/moose/framework/include/utils/InputParameters.h:19,
                 from /home/harrwh/projects_2017/moose/framework/include/base/MooseObject.h:19,
                 from /home/harrwh/projects_2017/moose/framework/include/kernels/KernelBase.h:18,
                 from /home/harrwh/projects_2017/moose/framework/include/kernels/Kernel.h:18,
                 from /home/harrwh/projects_2017/liger/include/kernels/kernelA.h:18,
                 from /home/harrwh/projects_2017/liger/src/kernels/kernelA.c:15:
/home/harrwh/projects_2017/moose/scripts/../libmesh/installed/include/libmesh/libmesh_common.h:36:19: fatal error: cstdlib: No such file or directory
 #include <cstdlib>
                   ^

I'm not sure how to proceed. Any help would be greatly appreciated! 

To unsubscribe from this group and stop receiving emails from it, send an email to moose-users+unsubscribe@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "moose-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to moose-users+unsubscribe@googlegroups.com.



--
Cormac Garvey
HPC Software Consultant
Scientific Computing
Idaho National Laboratory
Ph: 208-526-6294

--
You received this message because you are subscribed to a topic in the Google Groups "moose-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/moose-users/C50VgJZhCRE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to moose-users+unsubscribe@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "moose-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to moose-users+unsubscribe@googlegroups.com.



--
Cormac Garvey
HPC Software Consultant
Scientific Computing
Idaho National Laboratory



--
Cormac Garvey
HPC Software Consultant
Scientific Computing
Idaho National Laboratory

--
You received this message because you are subscribed to a topic in the Google Groups "moose-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/moose-users/C50VgJZhCRE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to moose-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/moose-users/CACw_P3W1YL4Z85SpiFwbfhMK-RDgY7XdmdReC2WzsJrUFmSmVA%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "moose-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to moose-users+unsubscribe@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "moose-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to moose-users+unsubscribe@googlegroups.com.

--
You received this message because you are subscribed to a topic in the Google Groups "moose-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/moose-users/C50VgJZhCRE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to moose-users+unsubscribe@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "moose-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to moose-users+unsubscribe@googlegroups.com.

--
You received this message because you are subscribed to a topic in the Google Groups "moose-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/moose-users/C50VgJZhCRE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to moose-users+unsubscribe@googlegroups.com.

Cody Permann

unread,
Jun 7, 2017, 6:13:20 PM6/7/17
to moose...@googlegroups.com
On Wed, Jun 7, 2017 at 7:49 AM William Harris <whha...@ncsu.edu> wrote:
I seem to have this problem every year for the past 3 years. I freshly cloned moose into my directory on the HPC.

Generally you don't need to do this if you already have an old copy sitting around. Just update the old copy, but no big deal if it's fresh either.
 
I loaded the necessary modules (use.projects, moose-dev/5.6.0-gmvolf-5.5.4).

Why this module? Just load moose-dev-gcc!
 
My goal is to use the Image Reader. I followed the instructions here


but those modules are not available on the HPC, so I then tried to follow the old instructions I was given 2 years ago (earlier in this post), so I executed module list, saw VTK/6.1.0-gmvolf-5.5.4-Python-2.7.8 in the module list, then executed

module display VTK/6.1.0-gmvolf-5.5.4-Pyton-2.7.8

which output

/apps/local/easybuild/modules/vis/VTK/6.1.0-gmvolf-5.5.4-Python-2.7.8.

So then I executed the following commands:


 export VTKINCLUDE_DIR=/apps/local/easybuild/software/VTK/6.1.0-gmvolf-5.5.4-Python-2.7.8/include/vtk-6.1
 export VTKLIB_DIR=/apps/local/easybuild/software/VTK/6.1.0-gmvolf-5.5.4-Python-2.7.8/lib
 ./update_and_rebuild_libmesh.sh --with-vtk-include=$VTKINCLUDE_DIR --with-vtk-lib=$VTKLIB_DIR

Wait, so you didn't load the module? This might be the problem. Loading the module does more than set a path. It sets several variables and loads other modules too
 

When that finished, I went to moose/modules/phase_field and executed make -j8. Last year there was an image_function test in phase_field/tests, but there isn't anymore. So I went to my own app and executed make -j8 to compile it. 

Those tests aren't gone, they've just been moved around. You can grep in the input files for ImageReader to find a few uses of it. I personally like the examples here:
 

I must have broken something in the process above, because I received errors, including the following:

Linking Library /home/harrwh/projects_2017/moose/modules/module_loader/lib/libmodule_loader_with-opt.la...
In file included from /home/harrwh/projects_2017/moose/scripts/../libmesh/installed/include/libmesh/perf_log.h:25:0,
                 from /home/harrwh/projects_2017/moose/framework/include/base/Moose.h:19,
                 from /home/harrwh/projects_2017/moose/framework/include/base/MooseError.h:18,
                 from /home/harrwh/projects_2017/moose/framework/include/utils/InputParameters.h:19,
                 from /home/harrwh/projects_2017/moose/framework/include/base/MooseObject.h:19,
                 from /home/harrwh/projects_2017/moose/framework/include/kernels/KernelBase.h:18,
                 from /home/harrwh/projects_2017/moose/framework/include/kernels/Kernel.h:18,
                 from /home/harrwh/projects_2017/liger/include/kernels/kernelA.h:18,
                 from /home/harrwh/projects_2017/liger/src/kernels/kernelA.c:15:
/home/harrwh/projects_2017/moose/scripts/../libmesh/installed/include/libmesh/libmesh_common.h:36:19: fatal error: cstdlib: No such file or directory
 #include <cstdlib>
                   ^

I bet this error stems from not loading the other module you found. You definitely need to load both the compiler stack and the VTK stack. These modules should be configured so that they load the right dependencies. You'll still need to run those commands to export the VTK include and library directories.
 

--
You received this message because you are subscribed to the Google Groups "moose-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to moose-users...@googlegroups.com.
Visit this group at http://groups.google.com/group/moose-users.



--
Cormac Garvey
HPC Software Consultant
Scientific Computing
Idaho National Laboratory
Ph: 208-526-6294

--
You received this message because you are subscribed to a topic in the Google Groups "moose-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/moose-users/C50VgJZhCRE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to moose-users...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "moose-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to moose-users...@googlegroups.com.
Visit this group at http://groups.google.com/group/moose-users.



--
Cormac Garvey
HPC Software Consultant
Scientific Computing
Idaho National Laboratory



--
Cormac Garvey
HPC Software Consultant
Scientific Computing
Idaho National Laboratory

--
You received this message because you are subscribed to a topic in the Google Groups "moose-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/moose-users/C50VgJZhCRE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to moose-users...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "moose-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to moose-users...@googlegroups.com.
Visit this group at http://groups.google.com/group/moose-users.

--
You received this message because you are subscribed to the Google Groups "moose-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to moose-users...@googlegroups.com.
Visit this group at http://groups.google.com/group/moose-users.

--
You received this message because you are subscribed to a topic in the Google Groups "moose-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/moose-users/C50VgJZhCRE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to moose-users...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "moose-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to moose-users...@googlegroups.com.
Visit this group at http://groups.google.com/group/moose-users.

--
You received this message because you are subscribed to a topic in the Google Groups "moose-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/moose-users/C50VgJZhCRE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to moose-users...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "moose-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to moose-users...@googlegroups.com.

William Harris

unread,
Jun 12, 2017, 12:11:43 PM6/12/17
to moose-users
Cody,

When I load use.projects, use.moose, and moose-dev-gcc, I can't even get the tests to run. So unless there are additional modules I need to load at the same time, then it seems that moose-dev/5.6.0-gmvolf-5.5.4 is the one that works for me.

I just tried re-doing my steps from last week, but with the additional step of loading the VTK module. Specifically, this is what I did.

1. Loaded the modules use.projects and moose-dev/5.6.0-gmvolf-5.5.4
2. Loaded the module VTK/6.1.0-gmvolf-5.5.4-Python-2.7.8
3. exported VTKINCLUDE_DIR and VTKLIB_DIR as I did last week
4. ran the update_and_rebuild_libmesh script with the --vtk-include flags, as I did last week.

Once this was finished, I went to /phase_field, did a make -j8, and then went into the tests/boundary_intersecting_features directory and opened the input file in peacock, then ran the simulation. I got the same error as before:

*** ERROR ***
libMesh must be configured with VTK enabled to utilize ImageSampler [39m
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".

So I still am not sure where I'm going wrong. Any more suggestions?

Thanks,
Wil 

William Harris

unread,
Jun 12, 2017, 12:19:36 PM6/12/17
to moose-users
Clarification: 

When I load moose-dev-gcc, I can't open peacock. When I unload it and load moose-dev/5.6.0-gmvolf-5.5.4, then I can open it.

Wil 

Alger, Brian K

unread,
Jun 12, 2017, 1:25:45 PM6/12/17
to moose...@googlegroups.com
I don't think any of the VTK modules on falcon are going to work with a current libmesh. Libmesh relatively recently required VTK to be compiled with MPI.
You probably have the message when libmesh is getting configured:

<<< libMesh requires VTK to be configured with -DVTK_Group_MPI:BOOL=ON >>>
<<< Configuring library without VTK support >>>

Creating new VTK modules with this flag set will likely be required for this to work.


To unsubscribe from this group and stop receiving emails from it, send an email to moose-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/moose-users/c9770f2b-0591-4b8f-a674-064701a1f29b%40googlegroups.com.

Cody Permann

unread,
Jun 12, 2017, 1:28:25 PM6/12/17
to <moose-users@googlegroups.com>
We are coordinating with the INL HPC administrators to create a new software stack to support this workflow on the Falcon cluster. I'll report back on this thread when this work is complete with the instructions you'll need.

Stay tuned...

--
You received this message because you are subscribed to the Google Groups "moose-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to moose-users...@googlegroups.com.
Visit this group at https://groups.google.com/group/moose-users.

William Harris

unread,
Jun 12, 2017, 1:29:56 PM6/12/17
to moose...@googlegroups.com
Ok, thanks!

Stay tuned...

To unsubscribe from this group and stop receiving emails from it, send an email to moose-users+unsubscribe@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "moose-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to moose-users+unsubscribe@googlegroups.com.

--
You received this message because you are subscribed to a topic in the Google Groups "moose-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/moose-users/C50VgJZhCRE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to moose-users+unsubscribe@googlegroups.com.

Visit this group at https://groups.google.com/group/moose-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/moose-users/CAK06EuXizAdAC2Ryf33Amw-wTXa4AbH9Ykf-36ytw%2B-b4e%2Bpjw%40mail.gmail.com.

William Harris

unread,
Jun 14, 2017, 1:41:21 PM6/14/17
to moose-users
I am finding that I get the same error message that I mentioned above (copied below) even when I start completely from scratch, create a new app, and the only files I add to my app are the ExampleConvection.h and ExampleConvection.c files from the MOOSE Wiki.

[harrwh@falcon1 liger3]$ make -j8
MOOSE Compiling C++ (in opt mode) /home/harrwh/projects_TEST_2/liger3/src/base/liger3App.C...
MOOSE Compiling C (in opt mode) /home/harrwh/projects_TEST_2/liger3/src/kernels/ExampleConvection.c...
In file included from /home/harrwh/projects_TEST_2/moose/scripts/../libmesh/installed/include/libmesh/perf_log.h:25:0,
                 from /home/harrwh/projects_TEST_2/moose/framework/include/base/Moose.h:19,
                 from /home/harrwh/projects_TEST_2/moose/framework/include/base/MooseError.h:18,
                 from /home/harrwh/projects_TEST_2/moose/framework/include/utils/InputParameters.h:19,
                 from /home/harrwh/projects_TEST_2/moose/framework/include/base/MooseObject.h:19,
                 from /home/harrwh/projects_TEST_2/moose/framework/include/kernels/KernelBase.h:18,
                 from /home/harrwh/projects_TEST_2/moose/framework/include/kernels/Kernel.h:18,
                 from /home/harrwh/projects_TEST_2/liger3/include/kernels/ExampleConvection.h:18,
                 from /home/harrwh/projects_TEST_2/liger3/src/kernels/ExampleConvection.c:15:
/home/harrwh/projects_TEST_2/moose/scripts/../libmesh/installed/include/libmesh/libmesh_common.h:36:19: fatal error: cstdlib: No such file or directory
 #include <cstdlib>
                   ^
compilation terminated.
make: *** [/home/harrwh/projects_TEST_2/liger3/src/kernels/ExampleConvection.x86_64-unknown-linux-gnu.opt.lo] Error 1
make: *** Waiting for unfinished jobs....

To clarify, I am not doing anything with VTK as I was previously and I have run update_and_rebuild_libmesh.sh without the vtk flags.

Wil

Slaughter, Andrew E

unread,
Jun 14, 2017, 1:50:48 PM6/14/17
to moose...@googlegroups.com
                 from /home/harrwh/projects_TEST_2/liger3/src/kernels/ExampleConvection.c:15:
Use big C for the file extension.

--
You received this message because you are subscribed to the Google Groups "moose-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to moose-users+unsubscribe@googlegroups.com.
 Visit this group at https://groups.google.com/group/moose-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/moose-users/6523ded6-0f72-470c-9fd1-21c508100bb7%40googlegroups.com.

William Harris

unread,
Jun 14, 2017, 3:29:41 PM6/14/17
to moose-users
Thank you! That solved the problem.
To unsubscribe from this group and stop receiving emails from it, send an email to moose-users...@googlegroups.com.

Alger, Brian K

unread,
Jun 21, 2017, 11:20:55 AM6/21/17
to moose...@googlegroups.com
Cormac built a module with VTK 7.1.1 :
module load use.projects moose-dev/5.7.0-gmvolf-5.5.4

or equivalently

module load use.moose moose-dev-gcc-vtk

I verified that the image reader tests in MOOSE work with this module stack.
You can try it out and let us know if it works for you.

William Harris

unread,
Jun 28, 2017, 5:27:01 PM6/28/17
to moose-users
Does this require compiling libMesh with VTK enabled? If so, is the following procedure correct...


Find the VTK module’s information using

module display VTK/7.1.1-gmvolf-5.5.4-Python-2.7.9

For me, the output of this command was (among other things)

                /apps/local/easybuild/modules/vis/VTK/7.1.1-gmvolf-5.5.4-Python-2.7.9.lua:

Then, execute the following commands to compile libMesh with VTK enabled so we can use the ImageFunction.

export VTKINCLUDE_DIR= /apps/local/easybuild/modules/vis/VTK/7.1.1-gmvolf-5.5.4-Python-2.7.9.lua/include/vtk-6.1
export VTKLIB_DIR= /apps/local/easybuild/modules/vis/VTK/7.1.1-gmvolf-5.5.4-Python-2.7.9.lua/lib
./update_and_rebuild_libmesh.sh --with-vtk-include=$VTKINCLUDE_DIR --with-vtk-lib=$VTKLIB_DIR

Peterson, JW

unread,
Jun 28, 2017, 5:33:34 PM6/28/17
to moose-users
On Wed, Jun 28, 2017 at 3:27 PM, William Harris <whha...@ncsu.edu> wrote:
Does this require compiling libMesh with VTK enabled? If so, is the following procedure correct...


Find the VTK module’s information using

module display VTK/7.1.1-gmvolf-5.5.4-Python-2.7.9

For me, the output of this command was (among other things)

                /apps/local/easybuild/modules/vis/VTK/7.1.1-gmvolf-5.5.4-Python-2.7.9.lua:

Then, execute the following commands to compile libMesh with VTK enabled so we can use the ImageFunction.

export VTKINCLUDE_DIR= /apps/local/easybuild/modules/vis/VTK/7.1.1-gmvolf-5.5.4-Python-2.7.9.lua/include/vtk-6.1
export VTKLIB_DIR= /apps/local/easybuild/modules/vis/VTK/7.1.1-gmvolf-5.5.4-Python-2.7.9.lua/lib
./update_and_rebuild_libmesh.sh --with-vtk-include=$VTKINCLUDE_DIR --with-vtk-lib=$VTKLIB_DIR


Yes, once the VTK module is loaded, you just have to do that last line (the module exports the environment vars for you).

--
John

Alger, Brian K

unread,
Jun 28, 2017, 5:43:43 PM6/28/17
to moose...@googlegroups.com
Hmm, not on falcon apparently. It only exports VTK_DIR.
So the sequence would be:
module load use.moose moose-dev-gcc-vtk
export VTKINCLUDE_DIR=$VTK_DIR/include/vtk-7.1
export VTKLIB_DIR=$VTK_DIR/lib
scripts/update_and_rebuild_libmesh.sh

The update_and_rebuild_libmesh.sh script seems to auto detect the VTK options if those two environment variables are not empty.

--
You received this message because you are subscribed to the Google Groups "moose-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to moose-users+unsubscribe@googlegroups.com.

Visit this group at https://groups.google.com/group/moose-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/moose-users/CANj%2BmwAGqetVJgqgqVw3_ofnrcMT5rx9k7vO_ZN5AHfoWgLoHw%40mail.gmail.com.

William Harris

unread,
Jun 29, 2017, 2:49:31 PM6/29/17
to moose...@googlegroups.com
I must be doing something wrong. The modules I have loaded are:

Currently Loaded Modules:
  1) pbs
  2) use.projects
  3) use.moose
  4) GCC/4.9.2
  5) cppunit/1.12.1-GCC-4.9.2
  6) tbb/4.3.0.090
  7) MVAPICH2/2.0.1-GCC-4.9.2
  8) moose-dev-gcc
  9) OpenBLAS/0.2.13-GCC-4.9.2-LAPACK-3.5.0
 10) gmvapich2/5.5.4
 11) FFTW/3.3.4-gmvapich2-5.5.4
 12) ScaLAPACK/2.0.2-gmvapich2-5.5.4-OpenBLAS-0.2.13-LAPACK-3.5.0
 13) gmvolf/5.5.4
 14) PETSc/3.6.4-gmvolf-5.5.4
 15) Python/2.7.9-gmvolf-5.5.4
 16) zlib/1.2.8-GCC-4.9.2
 17) GMP/6.1.1-GCC-4.9.2
 18) nettle/3.2-GCC-4.9.2
 19) libpng/1.6.21-GCC-4.9.2
 20) freetype/2.6.3-GCC-4.9.2
 21) expat/2.2.0-GCC-4.9.2
 22) fontconfig/2.12.1-GCC-4.9.2
 23) bzip2/1.0.6-GCC-4.9.2
 24) X11/20160819-GCC-4.9.2
 25) libdrm/2.4.70-GCC-4.9.2
 26) ncurses/6.0-GCC-4.9.2
 27) LLVM/3.8.1-GCC-4.9.2
 28) Mesa/12.0.2-GCC-4.9.2
 29) libGLU/9.0.0-GCC-4.9.2-Mesa-12.0.2
 30) VTK/7.1.1-gmvolf-5.5.4-Python-2.7.9
 31) PyYAML/3.11-gmvolf-5.5.4-Python-2.7.9
 32) SIP/4.16.4-gmvolf-5.5.4-Python-2.7.9
 33) Qt/4.8.6-gmvolf-5.5.4
 34) PyQt/4.11.3-gmvolf-5.5.4-Python-2.7.9
 35) moose-dev/5.7.0-gmvolf-5.5.4

With these modules loaded, I then went to moose/scripts and executed 

export VTKINCLUDE_DIR=$VTK_DIR/include/vtk-7.1
export VTKLIB_DIR=$VTK_DIR/lib
./update_and_rebuild_libmesh.sh 

When this didn't work, I tried

export VTKINCLUDE_DIR=$VTK_DIR/include/vtk-7.1
export VTKLIB_DIR=$VTK_DIR/lib
./update_and_rebuild_libmesh.sh --with-vtk-include=$VTKINCLUDE_DIR --with-vtk-lib=$VTKLIB_DIR

In both cases, when I go to moose/modules/phase_field/tests/boundary_intersecting_features and run this, I get the error

*** ERROR ***
libMesh must be configured with VTK enabled to utilize ImageSampler [39m

Additionally, in the error message there are the following lines:

'--with-vtk-include=/apps/local/easybuild/software/VTK/7.1.1-gmvolf-5.5.4-Python-2.7.9/include/vtk-7.1'

'--with-vtk-lib=/apps/local/easybuild/software/VTK/7.1.1-gmvolf-5.5.4-Python-2.7.9/lib'      

'VTK_DIR=/apps/local/easybuild/software/VTK/7.1.1-gmvolf-5.5.4-Python-2.7.9'                                 
Then, thinking that the issue could be solved by re-compiling, I went to the moose/modules/phase_field directory and exeucing make -j8. Then when I went back to the boundary_intersecting_features, I couldn't even open peacock. The error was:

/home/harrwh/projects_2017/moose/modules/phase_field/phase_field-opt: error while loading shared libraries: libpetsc.so.3.5: cannot open shared object file: No such file or directory
Traceback (most recent call last):
  File "../../../../gui/peacock", line 256, in <module>
    main_window = UiBox(options,args,executable,app)
  File "../../../../gui/peacock", line 69, in __init__
    self.initUI()
  File "../../../../gui/peacock", line 81, in initUI
    self.tabs = self.application.tabs(self)
  File "/home/harrwh/projects_2017/moose/gui/plug_ins/PeacockApplication.py", line 30, in tabs
    main_ui.input_file_widget = InputFileWidget(main_ui.app_path, main_ui.options, main_ui, main_ui.qt_app, main_ui.application)
  File "/home/harrwh/projects_2017/moose/gui/gui/InputFileWidget.py", line 42, in __init__
    self.recache()
  File "/home/harrwh/projects_2017/moose/gui/gui/InputFileWidget.py", line 291, in recache
    self.yaml_data = YamlData(self.qt_app, self.app_path, force_recache or self.options.recache, self.options.use_cached_syntax)
  File "/home/harrwh/projects_2017/moose/gui/utils/YamlData.py", line 28, in __init__
    self.yaml_data = self.gen_syntax.GetSyntax(recache)
  File "/home/harrwh/projects_2017/moose/gui/utils/GenSyntax.py", line 89, in GetSyntax
    raw_data = self.getRawDump()
  File "/home/harrwh/projects_2017/moose/gui/utils/GenSyntax.py", line 138, in getRawDump
    data = data.split('**START YAML DATA**\n')[1]
IndexError: list index out of range

Any suggestions?

Thanks!
Wil 

--
You received this message because you are subscribed to a topic in the Google Groups "moose-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/moose-users/C50VgJZhCRE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to moose-users+unsubscribe@googlegroups.com.

Visit this group at https://groups.google.com/group/moose-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/moose-users/CAAjL3cFfqaSm653YqPMZ1otWq4%3DY-tzbFk0Yzv4fTC2Xw%2B4T%3Dg%40mail.gmail.com.

Peterson, JW

unread,
Jun 29, 2017, 2:56:52 PM6/29/17
to moose-users
Did you confirm that $VTK_DIR is actually set to something that exists?
  

In both cases, when I go to moose/modules/phase_field/tests/boundary_intersecting_features and run this, I get the error

*** ERROR ***
libMesh must be configured with VTK enabled to utilize ImageSampler [39m

We need to see your libmesh/build/config.log file to understand why VTK is not being found... it could be a linking error or something even simpler. 

--
John

William Harris

unread,
Jun 29, 2017, 3:02:32 PM6/29/17
to moose-users
I didn't change VTK_DIR, so I'm assuming that it is set correctly when I loaded the module, as suggested by Brian Alger in a previous reply.

Alger, Brian K

unread,
Jun 29, 2017, 3:10:04 PM6/29/17
to moose...@googlegroups.com
I just did your steps and libmesh does detect VTK and uses it.
When building libmesh, you should see that it picked up VTK version 7.1.1 (at the end of when it configures and before it starts compiling).
After building libmesh you do have to rebuild all of MOOSE, including the phase field module.


--
You received this message because you are subscribed to the Google Groups "moose-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to moose-users+unsubscribe@googlegroups.com.
Visit this group at https://groups.google.com/group/moose-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/moose-users/1cafb159-58cd-4f18-bd4a-3ebb14ddb064%40googlegroups.com.

Peterson, JW

unread,
Jun 29, 2017, 3:11:53 PM6/29/17
to moose-users
On Thu, Jun 29, 2017 at 1:02 PM, William Harris <whha...@ncsu.edu> wrote:
I didn't change VTK_DIR, so I'm assuming that it is set correctly when I loaded the module, as suggested by Brian Alger in a previous reply.

Why assume anything? Just echo $VTK_DIR and make sure...

--
John

Miller, Jason M

unread,
Jun 29, 2017, 3:17:49 PM6/29/17
to moose...@googlegroups.com
Currently Loaded Modules:
  1) pbs
  2) use.projects
  3) use.moose
  4) GCC/4.9.2
  5) cppunit/1.12.1-GCC-4.9.2
  6) tbb/4.3.0.090
  7) MVAPICH2/2.0.1-GCC-4.9.2
  8) moose-dev-gcc <-- Conflicts
 35) moose-dev/5.7.0-gmvolf-5.5.4 <-- with this one



You only need to load the following modules to get VTK functionality:

module load use.moose moose-dev-gcc-vtk

or

module load use.projects moose-dev

But not both. The reason, is because of the potential conflict the two moose-dev-$$$ modules create with PETSc:

module load use.projects moose-dev yields:

echo $PETSC_DIR
/apps/local/easybuild/software/PETSc/3.6.4-gmvolf-5.5.4

while module load use.moose moose-dev-gcc yields:

echo $PETSC_DIR
/apps/moose/gnu/petsc/petsc-3.6.3-hypre

Granted the last module loaded will overwrite the previous one's settings... So you won't see a conflict with PETSc if you have always loaded these two modules together since the dawn of time and have not changed your module loading habits ever... However, if you loaded one of them, then built MOOSE, and then loaded another one just to get VTK going... conflicts. Thats where your libpetsc error is coming from.

Cheers!
Jason


--
You received this message because you are subscribed to the Google Groups "moose-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to moose-users+unsubscribe@googlegroups.com.
Visit this group at https://groups.google.com/group/moose-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/moose-users/1cafb159-58cd-4f18-bd4a-3ebb14ddb064%40googlegroups.com.

Miller, Jason M

unread,
Jun 29, 2017, 3:28:21 PM6/29/17
to moose...@googlegroups.com
William,

Sorry about this! I just realized when I created that module for folks to use (moose-dev-gcc-vtk) and how/what not to do, I mentioned it _only_ to Cody... thinking that I emailed the entire group:



To Cody

For those of you using the traditional `use.moose` subset of our modules, you can load load the afore mentioned VTK modules Cormac built by loading the following two modules:

module load use.moose moose-dev-gcc-vtk

VERY IMPORTANT NOTE: This is just a simple wrapper to the above listed modules provided by Cormac. The moose-dev-gcc-vtk module is in _no way_ related to, or even compatible with the moose-dev-gcc module. In other words: You can't use one and then another with out first performing a `make clobberall` of your repo.


Cheers!



Sorry about that!
Jason

William Harris

unread,
Jun 29, 2017, 3:35:05 PM6/29/17
to moose...@googlegroups.com
If you don't mind, can someone please provide step-by-step instructions for what I should do? I'm lost, and since it takes a long time to compile libmesh I don't want to do it incorrectly again.

--
You received this message because you are subscribed to a topic in the Google Groups "moose-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/moose-users/C50VgJZhCRE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to moose-users+unsubscribe@googlegroups.com.

Visit this group at https://groups.google.com/group/moose-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/moose-users/CALgZTAc5CYAdcLa27wVP41s-F7MV6UyQUQTPTb8j8sktFrExzQ%40mail.gmail.com.

Peterson, JW

unread,
Jun 29, 2017, 4:15:43 PM6/29/17
to moose-users
Also, the config.log file that you sent me off-thread contains this:

configure:40451: result: <<< Configuring library with VTK version 7.1.1 support >>>

so it looks to me like libmesh successfully configured itself to use VTK.


If you are getting this error

*** ERROR ***
libMesh must be configured with VTK enabled to utilize ImageSampler [39m

from MOOSE, it must be because you are using some libmesh other than the one whose config.log you sent me.

--
John

Miller, Jason M

unread,
Jun 29, 2017, 6:09:19 PM6/29/17
to moose...@googlegroups.com
Where things went wrong and what you need to fix first:
Somehow, you are loading the moose-dev-gcc module. You will need to figure out how in your profile(s) you are doing that, and... well don't do it anymore. When you figure it out, it would be best to log off the machine and then log back on, perform a `module list` and verify, the moose-dev-gcc module is _not_ loading anymore.

Next,
You will need to start over, clean up any previous builds (`make clobberall`), and rebuild everything again (including libMesh) using _only_ the following modules loaded before hand:

module load use.projects moose-dev

or

module load use.moose moose-dev-gcc-vtk

Just pick one of those (preferably the first one, as the second one is simply an alias to the first one).

Thanks!
Jason



--
You received this message because you are subscribed to the Google Groups "moose-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to moose-users+unsubscribe@googlegroups.com.

Visit this group at https://groups.google.com/group/moose-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/moose-users/CAJQhortk4jwEhXqsjwO9NKMyBnCQfx9JcuNOHPQMrJh98vWsMA%40mail.gmail.com.

Miller, Jason M

unread,
Jun 30, 2017, 8:57:35 AM6/30/17
to moose...@googlegroups.com
Hmmm there is still an issue with the modules. Looks like there is a difference between what moose-dev, vs moose-dev-gcc-vtk now give you.  At the time of my alias module creation, I was pointing moose-dev-gcc-vtk directly at:  moose-dev/5.7.0-gmvolf-5.5.4, which is setting all the VTK directories.

But loading moose-dev by default _now_ gives you: 5.7.0-gmvolf-5.5.7, which does _not_ set all the VTK directories. This module only sets VTK_DIR. And based on this large email thread, I gather that is not enough? That we need VTKINCLUDE/VTKLIB  directories set as well? Its an easy fix to add these attributes to the 5.7.0-gmvolf-5.5.7 module.

Brian,
what module did you use for you testing?

William,
If you have already started over building everything again using one of these modules, no worries. We can make either of them work. We just need to know what module you choose to use. My original advice still stands however: Once you 'pick' a module to use, you need to stick with that module (aka Environment).

Thanks all!
Jason



To view this discussion on the web visit https://groups.google.com/d/msgid/moose-users/CAAjL3cFwvmAwp5VFx9nfeEq7tnGdmDec6w%3D92V5ihkKqz5Aneg%40mail.gmail.com.

Miller, Jason M

unread,
Jun 30, 2017, 9:08:31 AM6/30/17
to moose...@googlegroups.com
William,
Ok... I'm digging into this a bit more and I apologize for having to say this, but there will be an issue by loading just the moose-dev module. Don't do it. You will get the incorrect version of VTK, and not enough environment variables to just have it work with out any extra steps (setting additional vtklib, vtkinclude directories).

Just use:

module load use.moose moose-dev-gcc-vtk

And you should be fine. If you have already started rebuilding everything using moose-dev, you will need to start over using moose-dev-gcc-vtk :/

Apologies again!
Jason

Cody Permann

unread,
Jun 30, 2017, 10:30:08 AM6/30/17
to moose...@googlegroups.com
I appears that there is too much room for error. Can we add the appropriate "conflict" option to one of these modules to prevent users from even running into this possibility?

Thanks,
Cody

To unsubscribe from this group and stop receiving emails from it, send an email to moose-users...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "moose-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to moose-users...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "moose-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to moose-users...@googlegroups.com.

Visit this group at https://groups.google.com/group/moose-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/moose-users/CALgZTAcBgY6LZ3i7_bjNvctjcKbTMdWO62V%3DzeUgP5%3D1Mj4VEA%40mail.gmail.com.

Miller, Jason M

unread,
Jul 3, 2017, 9:45:26 AM7/3/17
to moose...@googlegroups.com
I am not sure how I would accomplish this. I would have for every one of our modules, 200+ of their modules (HPC's) that I would have to add to ours as a conflict.

To see what I mean, log into falcon, and perform a:

module load use.projects
module av

97% of the things listed, will have some different version of GCC than our use.moose modules (thats 200+ modules).

I _could_ create an all encompassing conflict for the `use.projects` module. But that would render that useful 3% of modules that does not conflict from being loadable.


To unsubscribe from this group and stop receiving emails from it, send an email to moose-users+unsubscribe@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "moose-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to moose-users+unsubscribe@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "moose-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to moose-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/moose-users/CALgZTAcBgY6LZ3i7_bjNvctjcKbTMdWO62V%3DzeUgP5%3D1Mj4VEA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "moose-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to moose-users+unsubscribe@googlegroups.com.

Visit this group at https://groups.google.com/group/moose-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/moose-users/CAK06EuXSZEmxf39CT%2BxqkYyX65fwwk0r2m%3DWAEcHxgeVixMG2g%40mail.gmail.com.
Reply all
Reply to author
Forward
0 new messages