Issue with Spack on Stampede2

[Bili Dong]

Hi,

Is there anyone using Spack on Stampede2? I'm having trouble setting it up properly. My compilers.yaml and packages.yaml are shown below. I'm using system default modules if that's relevant. With these settings, `spack install intel-mpi` would err with:

==> inte...@2018.2.199 : has external module in impi/18.0.2

==> inte...@2018.2.199 : is actually installed in /opt/intel/compilers_and_libraries_2018.2.199/linux/mpi/intel64

==> inte...@2018.2.199 : generating module file

==> Error: Error: unterminated ${ in format:'() {  if [ -z "${LMOD_SH_DBG_ON+x}" ]; then

 case "$-" in 

 *v*x*)

 __lmod_sh_dbg='vx'

 ;;

 *v*)

 __lmod_sh_dbg='v'

 ;;

 *x*)

 __lmod_sh_dbg='x'

 ;;

 esac;

 fi;

 if [ -n "${__lmod_sh_dbg'

The only clue I have is that this piece of code is from an environment variable:

BASH_FUNC_module()=() {  if [ -z "${LMOD_SH_DBG_ON+x}" ]; then

 case "$-" in 

 *v*x*)

 __lmod_sh_dbg='vx'

 ;;

 *v*)

 __lmod_sh_dbg='v'

 ;;

 *x*)

 __lmod_sh_dbg='x'

 ;;

 esac;

 fi;

 if [ -n "${__lmod_sh_dbg:-}" ]; then

 set +$__lmod_sh_dbg;

 echo "Shell debugging temporarily silenced: export LMOD_SH_DBG_ON=1 for Lmod's output";

 fi;

 eval $($LMOD_CMD bash "$@") && eval $(${LMOD_SETTARG_CMD:-:} -s sh);

 local _lmod_my_status=$?;

 if [ -n "${__lmod_sh_dbg:-}" ]; then

 echo "Shell debugging restarted";

 set -$__lmod_sh_dbg;

 unset __lmod_sh_dbg;

 fi;

 return $_lmod_my_status

}

There is another weird thing that when I do `spack install intel-mpi`, before `==> inte...@2018.2.199 ...`, there would be a large block of output showing all the environment variables set. I think it's from my compiler setting with `modules: [intel/18.0.2]`.

compilers.yaml

compilers:

- compiler:

    environment: {}

    extra_rpaths: []

    flags: {}

    modules: [intel/18.0.2]

    operating_system: centos7

    paths:

      cc: /opt/intel/compilers_and_libraries_2018.2.199/linux/bin/intel64/icc

      cxx: /opt/intel/compilers_and_libraries_2018.2.199/linux/bin/intel64/icpc

      f77: /opt/intel/compilers_and_libraries_2018.2.199/linux/bin/intel64/ifort

      fc: /opt/intel/compilers_and_libraries_2018.2.199/linux/bin/intel64/ifort

    spec: in...@18.0.2

    target: x86_64

packages.yaml

packages:

  all:

    compiler: [intel]

    providers:

      mpi: [intel-mpi]

  intel-mpi:

    buildable: False

    modules:

      inte...@2018.2.199: impi/18.0.2

In summary, I'm looking for someone with experience using Spack on Stampede2 to help me. Thanks a lot in advance!

Best,

Bili

[Erik Schnetter]

Bili

I am using Spack on Stampede2, but with a configuration that might not
be of interest to you. In particular, I issue these module commands
before using Spack:

module unload intel impi
module load gcc/7.1.0

which means I'm using GCC and not any of the Intel software.

-erik

> --
> You received this message because you are subscribed to the Google Groups "Spack" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to spack+un...@googlegroups.com.
> To post to this group, send email to sp...@googlegroups.com.
> Visit this group at https://groups.google.com/group/spack.
> To view this discussion on the web visit https://groups.google.com/d/msgid/spack/7acdd8c8-bf42-4b80-b6a2-480b1762a076%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.



--
Erik Schnetter <schn...@gmail.com>
http://www.perimeterinstitute.ca/personal/eschnetter/

[Elizabeth A. Fischer]

---------- Forwarded message ---------
From: Bili Dong - Gmail <qobi...@gmail.com>
Date: Mon, May 13, 2019 at 12:11 PM
Subject: Re: [spack] Issue with Spack on Stampede2
To: <elizabet...@columbia.edu>

Hi Elizabeth,

Thanks for taking a look at this. I've attached all env vars in env.txt below. It seems there is no spack_build.out generated yet.

Another finding is that $SPACK_ROOT/share/spack/setup-env.sh seems to have set BASH_FUNC_module() which is causing the error. Probably that says something?

Best,

Bili

On Mon, May 13, 2019 at 4:31 AM Elizabeth A. Fischer <elizabet...@columbia.edu> wrote:

Please run printenv and share all your env vars.  Please share the spack_build.out file as well, rename it to .txt when you attach.

[Bili Dong]

Hi Erik,

Thanks for the information. It's good to know it works with gcc.

Could you do me a favor and see if you could reproduce the error with my configs? That could help me figure out if this is a general issue on Stampede2 or just me.

Thanks,

Bili

> To unsubscribe from this group and stop receiving emails from it, send an email to sp...@googlegroups.com.

[Bili Dong]

My original question might be a little bit vague as to what I want. Ultimately I want to solve this issue. But even if you don't see where the issue is, there are two other things that would be extremely helpful to me:

1. If you use Spack on Stampede2, what's your setup? What's your compilers.yaml and packages.yaml like?

2. If you have access to Stampede2, could you reproduce my error?

I'm not sure if this is an issue with Spack or just me being stupid. If I could get someone else reproduces the error, then it's probably worth reporting this as an issue on GitHub. But I hope it's just me because I really look forward to using Spack on Stampede2 now. I've already succeeded in using it on some other HPC sites.

Thanks for whoever could give me some help.

Best wishes,

Bili

[Bili Dong]

Hi Erik,

Could you share your config files on Stampede2 with me? Upon further investigation, I found the issue probably related to the module system, but not specific to the intel compiler. More specifically:

1. How do you set your compilers.yaml and packages.yaml?

2. Do you use a modules.yaml as well? I'm not sure if I should set it.

3. Do you use other config files not mentioned above?

Thanks a lot in advance! Wish you have a great day!

Best regards,

Bili

On Monday, May 13, 2019 at 5:15:32 AM UTC-7, Erik Schnetter wrote:

> To unsubscribe from this group and stop receiving emails from it, send an email to sp...@googlegroups.com.

[Bili Dong]

I have opened an issue on GitHub. Leave a note here for future reference.

Best,

Bili

On Sunday, May 12, 2019 at 11:02:40 PM UTC-7, Bili Dong wrote:

[Kenneth Moreland]

Bili,

I happen to be playing with Spack on Stampede2 today and was able to find someone who had the same problem and worked around it. Although the intel-mpi package is not working, you can fake out Spack by using the mvapich2 package and pointing it to the impi module on the system. My packages.yaml file looks as follows:

packages:

  all:

    compiler: 

      - in...@18.0.2

    providers:

      mpi:

        - mvapich2

  mvapich2:

    buildable: False

    modules:

      mvapich2: impi

-Ken

[Bili Dong]

Hi Ken,

Thanks a lot for sharing the information!

FYI, I later spent some time to track down where the issue is, and that process is recorded on GitHub. If you'd like to have a look, I'd suggest from the bottom. In summary, the bug is triggered through the IntelPackage class during the environment modification detection process, and a PR is underway to fix it.

But before that PR gets merged, I could use your clever workaround.

Thanks,

Bili