Moose on Windows Subsystem Linux Updated

[co...@oregonstate.edu]

Hey Moose community,

I attempted to install MOOSE on my windows machine using an insider build a few months ago with the windows subsystem linux (WSL).  It didn't work then so I waited for the anniversary edition to come out which included the most stable build for WSL.  I have since started trying to reinstall moose and have run into a few problems.  The good thing is that the majority of the tests run using

./run_tests -j8

In fact only 19 tests fail.  But when each of these tests fail it cites mpiexec as a reason for failure. 

I have since tried updating my MPI.  I have not remade the

make -j8

command from the appropriate folder but I figured that if it is strictly an MPI problem I wouldn't need to.

I am wondering if anyone out there is trying the same thing and has figured out a work around or fix.  I know windows is not the popular system for MOOSE but I would like to get it up and running on here.  Attached are a few screen shots of the output I see and I would love some help.

As a background I am somewhat familiar with MPI.  If I had to guess I would imagine that the ability to use multi-threading is somewhat limited by the fact that linux is running only in a windowed bash and I am unsure how to bridge that gap. 

The first image is the overall output of the ./run_tests command

Here is an example of an individual test failure where the mpiexec command can be seen.

Finally here is an example of a test case run for the simple_diffuse test.

Hopefully these pictures can be downloaded and expanded.

Jordan

[Daniel Schwen]

Hey Jordan,

can you check if adding your hostname to the windows hosts file will fix this? The mpi error looks familiar and on Unix systems it can be fixed by editing /etc/hosts and adding the current hostname (with the 127.0.0.1 ip).

Cheers,

Daniel

--
You received this message because you are subscribed to the Google Groups "moose-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to moose-users...@googlegroups.com.
Visit this group at https://groups.google.com/group/moose-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/moose-users/f30823a3-36e6-4aa3-b451-b72de39b7910%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Cody Permann]

Right, to be clear this isn't related to MOOSE. You most likely can't run MPI Hello World in your current configuration. If Daniel's suggestion doesn't work make sure you can run a very simple example.

Cody

To view this discussion on the web visit https://groups.google.com/d/msgid/moose-users/CAPxoKqeDOia9oMimwoBE1oq_JJDrVDF64iUtCUnSBEivXH1oWA%40mail.gmail.com.

[co...@oregonstate.edu]

So I attempted to add the host to my windows host files which for windows are located at 

 C:\Windows\System32\drivers\etc\hosts

Below is a screen shot, but based on the hosts file in linux, I added the following lines

127.0.0.1 localhost

# The following lines are desirable for IPv6 capable hosts

::1 ip6-localhost ip6-loopback

fe00::0 ip6-localnet

ff00::0 ip6-mcastprefix

ff02::1 ip6-allnodes

ff02::2 ip6-allrouters

ff02::3 ip6-allhosts

However I am unsure if the windows hosts file also accepts the IPv6 commands as the commented lines in the windows hosts file doesn't suggest it can.

Either way I re-ran the make -j8 command and ran the ./run_tests -j8 and ran into the same mpi problems as before.

Thanks,

Jordan

[co...@oregonstate.edu]

Yeah I am definitely aware that if this is an mpi problem, it is more related to my environment than it is to moose. 

[Daniel Schwen]

Add
127.0.0.1 DESKTOP-1L17VV2

Daniel

On Wed, Aug 3, 2016 at 4:43 PM <co...@oregonstate.edu> wrote:

Yeah I am definitely aware that if this is an mpi problem, it is more related to my environment than it is to moose. 

--
You received this message because you are subscribed to the Google Groups "moose-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to moose-users...@googlegroups.com.
Visit this group at https://groups.google.com/group/moose-users.

To view this discussion on the web visit https://groups.google.com/d/msgid/moose-users/9ea622c8-f8c5-4f61-be4f-d3c25d276013%40googlegroups.com.

[co...@oregonstate.edu]

Daniel,

I was unsure whether you meant to add it into the windows hosts file or the WSL hosts file.  I added it first to the windows to no effect, and then added it to the linux hosts file (in /etc/hosts).  Attached is a side-by-side of both hosts files.  This successfully allowed the ./run_tests -j8 command to run some tests, but now they are timing out.  An image of this is shown below.  

Thank you again for contributing so much to this and if you have time and further advice I would really appreciate it.

Jordan

[Daniel Schwen]

can you run ./run_tests -p 8 

The -j 8 option just means run 8 tests at the same time, -p 8 means try to run each test on 8 cores. I'd really like to see some tests reporting success in parallel. I'm not the expert here, but it looks like some of the parallel tests that fail above are pretty hard core (i.e. forcing MPI errors and making sure they error out in a specific way). I would not be surprised if those are hard to get to work. But this might not impact day to day MOOSE simulations with MPI.

Daniel

--
You received this message because you are subscribed to the Google Groups "moose-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to moose-users...@googlegroups.com.
Visit this group at https://groups.google.com/group/moose-users.

To view this discussion on the web visit https://groups.google.com/d/msgid/moose-users/0b5f600f-42da-483f-96e7-32f830e4e32e%40googlegroups.com.

[co...@oregonstate.edu]

Yes I will do that next.  Before I start it though just a quick question.  My machine has 2 physical cores with up to 4 virtual cores (as far as I understand system architecture).  Should I use a lower number such as -p4 or -p2 to make sure it doesn't try to access more cores than is possible?  From my understanding of the command -p8 means up to 8 cores, and includes a lower number.  But I am thinking since it having trouble with parallelization I should use the exact number of physical cores??

[Daniel Schwen]

Sure go for -p2 or -p4 then.

To view this discussion on the web visit https://groups.google.com/d/msgid/moose-users/9ca3ed4c-28eb-432c-bd01-fa416bffb347%40googlegroups.com.

[co...@oregonstate.edu]

The -p8 command and all others produce the same timeout error.

[co...@oregonstate.edu]

As an update, instead of running the intense system tests, I ran the simple diffusion case with 2 processors, and that case ran just fine.  So it appears that MPI is working my machine might just be incredibly slow no?

[Daniel Schwen]

Do you have _any_ test running successfully in parallel? Could you go into modules/phase_field (build the executable) and do the same run tests command there?

Daniel

On Wed, Aug 3, 2016 at 5:44 PM <co...@oregonstate.edu> wrote:

The -p8 command and all others produce the same timeout error.

--

You received this message because you are subscribed to the Google Groups "moose-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to moose-users...@googlegroups.com.
Visit this group at https://groups.google.com/group/moose-users.

To view this discussion on the web visit https://groups.google.com/d/msgid/moose-users/b31fdfd5-aab3-4fec-941c-8a2bab3149a1%40googlegroups.com.

[co...@oregonstate.edu]

I will head to the modules/phase_field and run the test command. 

If I understand the command correctly I am running the following test in parallel. 

-Directory

~/projects/moose/test/tests/kernels/simple_diffusion

Command

mpiexec -n 4 ../../../moose_test-opt -I simple_diffusion.i

with the following output:

Framework Information:
MOOSE version:           git commit 9e87977 on 2016-08-02
PETSc Version:           3.6.4
Current Time:            Wed Aug  3 16:48:33 2016
Executable Timestamp:    Wed Aug  3 14:15:34 2016

Parallelism:
  Num Processors:          4
  Num Threads:             1

Mesh:
  Parallel Type:           replicated
  Mesh Dimension:          2
  Spatial Dimension:       2
  Nodes:
    Total:                 121
    Local:                 35
  Elems:
    Total:                 100
    Local:                 24
  Num Subdomains:          1
  Num Partitions:          4
  Partitioner:             metis

Nonlinear System:
  Num DOFs:                121
  Num Local DOFs:          35
  Variables:               "u"
  Finite Element Types:    "LAGRANGE"
  Approximation Orders:    "FIRST"

Execution Information:
  Executioner:             Steady
  Solver Mode:             Preconditioned JFNK
  Preconditioner:          hypre boomeramg

 0 Nonlinear |R| = 3.316625e+00
      0 Linear |R| = 3.316625e+00
      1 Linear |R| = 7.729528e-02
      2 Linear |R| = 5.163994e-03
      3 Linear |R| = 1.688774e-04
      4 Linear |R| = 8.914807e-06
 1 Nonlinear |R| = 8.915525e-06
      0 Linear |R| = 8.915525e-06
      1 Linear |R| = 5.708712e-07
      2 Linear |R| = 2.116487e-08
      3 Linear |R| = 6.820533e-10
      4 Linear |R| = 1.666996e-11
 2 Nonlinear |R| = 1.665721e-11

[Daniel Schwen]

Ok, that looks good. Did you compare the exodus outputs? If that works chances are good that most test in the modules will also work.

--

You received this message because you are subscribed to the Google Groups "moose-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to moose-users...@googlegroups.com.
Visit this group at https://groups.google.com/group/moose-users.

To view this discussion on the web visit https://groups.google.com/d/msgid/moose-users/fe4b0b22-c561-43e1-9a8b-33fb5f659fe9%40googlegroups.com.

[Miller, Jason M]

Jordan,

Nice job!  I remember you took a stab at this back in May, coupled with what we thought we had to do at the time:

Windows 10 Ubuntu bash

On Windows:

 install xming   (allows X-windows apps to display, like xclock, xeyes, etc)

On Ubuntu (automatically logs you in as root):

 apt-get remove upstart udev

 apt-get autoremove

Note: Some how the Ubuntu package manager was broken and not able to install _anything_ immediately after it became available on the Windows machine. Meaning we didn't do anything to break it ourselves. Ubuntu came inherently broken for us. The exact error (and solution) was found here: http://superuser.com/questions/1064801/i-broke-the-windows-linux-subsystem-bash-on-ubuntu-on-windows-10-linux-help And so the above apt-get commands solved our initial issue.

 apt-get install build-essential <among other things>

Optional but highly recommended:

 Create a normal user and grant that user sudo rights

 su <normal user>

 add `export DISPLAY=:0` to your ~/.bashrc

 profit.

Windows/Ubuntu bash automatically logs you in as 'root, and thus, MPI voices some serious concerns and warnings. It was necessary to create a new regular user, and `su` to that user.

Can you explain what it was you did to get to where you are now? Any of the above? Does Peacock work as well (by installing that xming package)?

Thanks!

Jason

To view this discussion on the web visit https://groups.google.com/d/msgid/moose-users/CAPxoKqemx6FxOUnR%3DMLpY9imRXpvNaKtxBQLczguFZQ2ngzNmA%40mail.gmail.com.

[Jordan Cox]

Hey Daniel,

So I did attempt to compare the exodus files but using the diff command it says that my output and the gold output are different though I do not know how different they are.

I apologize I am newer at moose so if you could help me understand a bit better how to interpret their differences that would be awesome.

[Daniel Schwen]

Show us the exodiff output!

To view this discussion on the web visit https://groups.google.com/d/msgid/moose-users/1cd36b54-3669-42f1-adc1-4613a1e41fe7%40googlegroups.com.

[Jordan Cox]

Hey Jason,

Yes my goal was to produce full documentation of how to install this on WSL.  I documented it and will do a full write up sometime later this week. 

For peacock I am working on testing it now based on the MOOSE peacock page.

[co...@oregonstate.edu]

Attached is the exodiff output

[Daniel Schwen]

Uhm... that says "Files are the same" at the bottom! (which is good)

On Thu, Aug 4, 2016 at 11:17 AM <co...@oregonstate.edu> wrote:

Attached is the exodiff output

--
You received this message because you are subscribed to the Google Groups "moose-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to moose-users...@googlegroups.com.
Visit this group at https://groups.google.com/group/moose-users.

To view this discussion on the web visit https://groups.google.com/d/msgid/moose-users/f0820022-ee80-40ec-a281-619e0ca20a5c%40googlegroups.com.

[Jordan Cox]

Yeah I noticed that on the exodiff command, but when I just used the plain "diff" command it said they differ, which is why I assumed they differed in some way.  But other than that, it sounds like to some degree moose is running in WSL just with less than full MPI support.

To answer your above questions and I ran the tests both in the main directory and in the phase_field, in the main directory 19 tests failed, all with a reference to MPI, and in the phase_field 1 test failed with a reference to MPI but of course the simple diffusion test was run in parallel as can be seen.  Let me know if you would like any additional output and thank you very much for your help!

[Daniel Schwen]

Diff is not the right way to compare this kind of data. Diff is a bitwise comparison. The floating point numbers 0.238764328764872 and 0.238764328764871 differ bitwise but the difference will come down to truncation errors (which can depend on floating point operation order). Exodiff compares the actual values down to a given tolerance to give a more meaningful concept of "differing".

On Thu, Aug 4, 2016 at 11:36 AM Jordan Cox <rae...@gmail.com> wrote:

Yeah I noticed that on the exodiff command, but when I just used the plain "diff" command it said they differ, which is why I assumed they differed in some way.  But other than that, it sounds like to some degree moose is running in WSL just with less than full MPI support.

To answer your above questions and I ran the tests both in the main directory and in the phase_field, in the main directory 19 tests failed, all with a reference to MPI, and in the phase_field 1 test failed with a reference to MPI but of course the simple diffusion test was run in parallel as can be seen.  Let me know if you would like any additional output and thank you very much for your help!

--
You received this message because you are subscribed to the Google Groups "moose-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to moose-users...@googlegroups.com.
Visit this group at https://groups.google.com/group/moose-users.

To view this discussion on the web visit https://groups.google.com/d/msgid/moose-users/bf82a936-01e1-4092-be11-73c88276875c%40googlegroups.com.

[Jordan Cox]

Hey Jason,

As an update I am working on testing peacock.  I followed the steps found here

http://mooseframework.org/wiki/Peacock/

Upon the "usage" of peacock it said that Peacock  command wasn't found.  I have not installed xming but I am unsure if we should because we are running in the WSL which is considered "native" linux.  I was planning on trying the xming package next and trying to work with that.  However if you are interested in me doing further testing with Peacock and can provide some guidance I would love to provide all the info I can about how WSL works with Peacock. 

Jordan

[Miller, Jason M]

Jordan,

The xming application is something you install on the Windows side... which allows the Linux side (WSL) to have a display port to attach to (for X related applications like Peackock).

I will need to figure out what exactly Peacock needs v's what is _not_ installed by default in WSL. Unfortunately, thats a tall order... I will need to obtain my own WSL machine and go from there...

However... it may be as easy as trying to get `xeyes` to work in WSL. Ubuntu is pretty good at telling you what programs to install (apt-get install). I would try that for starters... start your WSL and try to run:  xeyes, and follow the on screen instructions from there... But you _will_ need xming installed on the Windows side of things first.

Hope that helps!

Jason

--

You received this message because you are subscribed to the Google Groups "moose-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to moose-users+unsubscribe@googlegroups.com.

Visit this group at https://groups.google.com/group/moose-users.

To view this discussion on the web visit https://groups.google.com/d/msgid/moose-users/540268e0-09f9-4472-b870-73fb567a65b8%40googlegroups.com.

[Jordan Cox]

Jason,

So I have installed xming on my windows side, and then in WSL typed xeyes.  I had to install x11-apps using the sudo apt-get command. 

After installing and retrying the xeyes command it says

Error: Can't open display:

Jordan

[Miller, Jason M]

Once you install all that (including xming), I think we had to reboot... or something... but I do remember getting xeyes going!

--

You received this message because you are subscribed to the Google Groups "moose-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to moose-users+unsubscribe@googlegroups.com.
Visit this group at https://groups.google.com/group/moose-users.

To view this discussion on the web visit https://groups.google.com/d/msgid/moose-users/fe176cea-4943-49ef-80cd-f35e34124c88%40googlegroups.com.

[co...@oregonstate.edu]

Hey Jason,

I tried several times in several configurations and couldn't get xeyes to work.  Are there any special setting changes you remember I need to make for the xeyes command to work?  I am looking through the xming launcher and trying out several permutations and seeing what happens.

Jordan

[co...@oregonstate.edu]

Jason,

I have completed my write up of the steps I took to install moose on WSL.  I will attach it here but let me know if I should e-mail it to anyone else at INL.

Jordan

[Miller, Jason M]

Hmmm... I do not remember... However, I came across the following site: https://www.slightfuture.com/how-to/x-on-wsl which among their steps included:

 `export DISPLAY=:0`

and then run the `xeyes` binary. I do not remember having to do that before, but perhaps the owner of the machine I was 'borrowing' at the time had already set that up in his bash profile or something. I really wish I had a dedicated Windows 10 machine up here sorry!

Those instructions make no mention of xming as well :/  except during the comments from other uses, which were more like answers to the original poster's reports of WSL 'Not quite ready yet'. So, installing xming seems like the right thing to still be doing.

Good luck! And thank you for your documentation! I especially enjoy the detail and time you obviously put into these instructions. They won't go to waste I can promise you that.

Now that Microsoft has released a build of Windows 10 which includes WSL, perhaps I can finally get my hands on it.

Jason

--

You received this message because you are subscribed to the Google Groups "moose-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to moose-users+unsubscribe@googlegroups.com.
Visit this group at https://groups.google.com/group/moose-users.

To view this discussion on the web visit https://groups.google.com/d/msgid/moose-users/9b3ed22c-4a9d-4580-b0a5-af30fa56b7cc%40googlegroups.com.

[Miller, Jason M]

I was able to acquire WSL on my test Windows 10 box.. finally!

I got xeyes to work... The magic certainly was that `export DISPLAY=:0` command. What I did:

Install xming (for Windows)

apt-get update; apt-get upgrade

apt-get install x11-apps

export DISPLAY=:0

xeyes

I was also able to run Peacock, but I first had to install another library:

apt-get install libglu1-mesa

So in a nut shell, this is what I did to make everything work...

1. enable Windowes Subsytem for Linux (WSL)

2. install Xming

3. modify the /etc/hosts file and add the results of `hostname` to resolve to 127.0.0.1

   * example:

  [ ]> hostname

  DESKTOP-L7BGA7L

  [ ]> sudo vi /etc/hosts

  127.0.0.1   localhost

  127.0.0.1   DESKTOP-L7BGA7L    <---- ADD THAT

4. apt-get update; apt-get upgrade

5. install the pre-req stuff (including the two newly discovered required items; x11-apps and libglu1-mesa):

  [ ]> apt-get install build-essential \

gfortran \

tcl \

git \

m4 \

freeglut3 \

doxygen \

libblas-dev \

liblapack-dev \

libx11-dev \

libnuma-dev \

libcurl4-gnutls-dev \

zlib1g-dev \

libhwloc-dev \

x11-apps \

libglu1-mesa

6. dpkg -i moose-environment-40_Ubuntu-14.04_x86_64.deb

7. restart Bash session (so the moose-environment gets loaded)

8. export DISPLAY=:0

  * Should probably add that to our ~/.bashrc file to be permanent

9. build libMesh/MOOSE

10. profit:

Peacock also refreshes instantly. I can feel no 'software rendering lag' at all.

However, I get artifacts when viewing 3D objects (as apposed to 2D). Its as if all the polygons are 'flipped' and I am only seeing the back of them:

Not trying to turn this email into a benchmark run, but I thought I would also post some numbers:

(GCC, moose_test-opt, run_tests -j 4)

WSL machine running 4 Haswell cores took 128 seconds

Mac OS X 10.11 running 4 Ivy Bridge cores took 116 seconds

The WSL machine is operating on a OCZ Revo 3 X2 drive (no longer available but you can still read about the specs), while the Mac OS X machine is running on a comparable PCi Express drive (Mac Cylinder).  I have performed some benchmarks using `dd` in WSL, and I appear to hitting an average of 555 MB/s (keep in mind that megabytes, not megabits). What I am trying to say here, is the hard drive speed doesn't seem to be the limiting factor on run_tests. Its most likely the emulation going on that is slowing it down.

Hope this helps some folks!

Jason

[Jordan Cox]

Hey Jason,

That looks really awesome.  I am actually in the process of re-compiling Moose because I wanted to check my instructions (and I had to wipe my computer) and I will definitely follow your steps, and add them to my instructions once I get it up and running.

Jordan

[Peterson, JW]

On Tue, Aug 9, 2016 at 12:52 PM, Miller, Jason M <jason....@inl.gov> wrote:

I was able to acquire WSL on my test Windows 10 box.. finally!

Hope this helps some folks!

Great job, Jason!

--

John

[Derek Gaston]

Awesome Jason!

Can you start a Wiki page on this?  We need to head towards actual instructions on this... before we do our next training we should have it in order.

Derek

--

You received this message because you are subscribed to the Google Groups "moose-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to moose-users...@googlegroups.com.

Visit this group at https://groups.google.com/group/moose-users.

To view this discussion on the web visit https://groups.google.com/d/msgid/moose-users/CANj%2BmwDJyb3t8RBG2YqT6xNWOgGWQO4rtMq8SmKtMOgw3GOYSA%40mail.gmail.com.

[Jordan Cox]

Hey Jason,  I was able to get xeyes running as well for the simple diffusion case.  It worked great. 

If you would like me to help with the write or append these directions to the previous document let me know.

Jordan

[Miller, Jason M]

Glad to hear it!

I am currently working on a set of instructions based on my previous email. But they won't be published until I get the go-ahead that we can official support Windows...

For now, we will probably reference this thread for those onesies twosies WSL requests.

Jason

--

You received this message because you are subscribed to the Google Groups "moose-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to moose-users+unsubscribe@googlegroups.com.
Visit this group at https://groups.google.com/group/moose-users.

To view this discussion on the web visit https://groups.google.com/d/msgid/moose-users/7dc368ee-b69e-4812-85d5-5c698f5346e8%40googlegroups.com.

[Heather Sheldon]

Any update on the WSL instructions? I'd be really interested to know how people have found using MOOSE with WSL. I would only be running small test problems so not concerned if there is not full MPI functionality.

On Thursday, August 11, 2016 at 1:52:49 AM UTC+10, jason.miller wrote:

Glad to hear it!

I am currently working on a set of instructions based on my previous email. But they won't be published until I get the go-ahead that we can official support Windows...

For now, we will probably reference this thread for those onesies twosies WSL requests.

Jason

On Wed, Aug 10, 2016 at 9:39 AM, Jordan Cox <rae...@gmail.com> wrote:

Hey Jason,  I was able to get xeyes running as well for the simple diffusion case.  It worked great. 

If you would like me to help with the write or append these directions to the previous document let me know.

Jordan

--
You received this message because you are subscribed to the Google Groups "moose-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to moose-users...@googlegroups.com.

Visit this group at https://groups.google.com/group/moose-users.

[Miller, Jason M]

Heather,

Obtaining several machines with which to test WSL is our hold up. As I recall, MPI worked just fine after following the instructions within this thread: https://groups.google.com/d/msg/moose-users/-Vwl42fZds0/aPrk5kQFDQAJ (specifically step 3)

Not that answer you were looking for I know. But right now, I am not sure the group is ready to support Windows.

Thanks!

Jason

To unsubscribe from this group and stop receiving emails from it, send an email to moose-users+unsubscribe@googlegroups.com.

Visit this group at https://groups.google.com/group/moose-users.

To view this discussion on the web visit https://groups.google.com/d/msgid/moose-users/3d7415de-3887-4c53-a54b-573bffe164ba%40googlegroups.com.