SNOPT Trial with AMPL for Pyomo

[Mike T]

Hello,

I am attempting to use SNOPT with Pyomo for my research. It is my understanding that I need a compiled executable built against the ASL, as this is how I am using IPOPT currently. Unfortunately, I have been having a lot of difficulties generating this executable.

I have downloaded the AMPL Solver Library interface, along with the wrapper for SNOPT through Netlib, and followed the instructions accordingly. I was successful in compiling the ASL, however I am running into errors while compiling the SNOPT sub directory. I placed the trial static Fortran library and license file under/solvers/snopt/src. 

This is the error I am given when attempting the make command (after successfully running ./configure) in solvers/snopt:

cd ${OBJDIR=sys.`uname -m`.`uname -s`}; make

make[1]: Entering directory '/home/mike/My_Stuff/Snopt/solvers/snopt/sys.x86_64.Linux'

cc -c -O -I../../sys.x86_64.Linux snopt.c

make[1]: *** No rule to make target 'src/libsnopt.a', needed by 'snopt'.  Stop.

make[1]: Leaving directory '/home/mike/My_Stuff/Snopt/solvers/snopt/sys.x86_64.Linux'

make: *** [makefile:2: snopt] Error 2

I noticed that the SNOPT library file was named differently in the makefile, (libsnopt.a vs libsnopt7.a), so I tried renaming the library file, as well as modifying the makefile to match the different name. Neither of these changes fixed the issue.

I am not sure how to proceed. Is there documentation to assist this installation? Unfortunately, the READMEs from AMPL are not very helpful in the details. 

Is it even possible to generate this executable using the trial version of SNOPT?

If so, what could I be doing incorrectly?

Thank you for the help,

Mike T.

[SNOPT]

Hi Mike,

You can do it but you'll have to modify several things.  I'm going to list all of the steps for completion even though you've done some of them already.

1.  Download solvers.tgz and snopt.tgz from netlib, uncompress.  Should create directories 'solvers' and 'snopt'.

2.  Create a file 'fmain.c'  inside the solvers directory with the following contents:

        char **xargv;
        extern void MAIN__(void);
        int main(int argc, char **argv)
        {
            xargv = argv;
            MAIN__();
            return 0;
        }
Add this to the list of files built by ASL. (look for "a = ..." inside makefile in the solvers dir).  (This is also documented in ASL's instructions in README.f77).

3. In solvers directory, run "./configurehere; make".  You should see amplsolver.a now.

4.  Copy 'makefile_ampl', which I've placed here:  https://gist.github.com/gnowzil/4bfc0b6c0ec725d0188bded45c885162

3.  Modify SNOPT_LIB to point to the snopt7 library.  If necessary, modify the other locations as well.  By default, it assumes that the directories 'solvers' and 'snopt' are on the same level as the file.

SNOPT_LIB   should point to the libsnopt7.a file

AMPL_SOLVERS_DIR should point to the 'solvers' directory
SNOPT_AMPL_DIR should point to the 'snopt' asl directory.

4.  Run makefile_ampl with the command "make -f makefile_ampl".  It should create the executable 'snopt_ampl'.

--Elizabeth

[kingb...@gmail.com]

Elizabeth, 

Thank you so much for the quick reply! 

I did as you instructed, and everything seemed to be working until the final step, but unfortunately I get an error when I run makefile_ampl as instructed. I get the following error:

f77  -o snopt_ampl snopt/snopt.o solvers/amplsolver.a libsnopt7.a

/usr/bin/ld: solvers/amplsolver.a(funcadd1.o): in function `dl_close':

funcadd1.c:(.text+0xe): undefined reference to `dlclose'

/usr/bin/ld: solvers/amplsolver.a(funcadd1.o): in function `dl_open':

funcadd1.c:(.text+0x194): undefined reference to `dlopen'

/usr/bin/ld: funcadd1.c:(.text+0x25e): undefined reference to `dlopen'

/usr/bin/ld: funcadd1.c:(.text+0x28e): undefined reference to `dlerror'

/usr/bin/ld: solvers/amplsolver.a(funcadd1.o): in function `libload_ASL':

funcadd1.c:(.text+0x38c): undefined reference to `dlsym'

/usr/bin/ld: funcadd1.c:(.text+0x502): undefined reference to `dlsym'

collect2: error: ld returned 1 exit status

make: *** [makefile_ampl:22: snopt_ampl] Error 1

Currently, I have the directories 'solvers' and 'snopt', and libsnopt7.a on the same level as the makefile, and modified the pointers accordingly, but this resulted in the error above. Is the placement of libsnopt7.a correct? I tried placing libsnopt7.a under 'snopt/src' (what the instructions I had found before had instructed), but get the same error. 

Unfortunately, this is not my area of expertise and I am starting to get lost in the weeds debugging the issue. What could I be doing incorrectly?

Thank you again for your help!

Mike T.

[SNOPT]

Hi Mike,

Try adding  -ldl  to the end of line 22 in makefile_ampl.  (I also updated the gist file with this change).

I was testing on a mac and I think you're on Linux so sometimes minor adjustments are needed.

--Elizabeth

[Mike T]

Elizabeth, 

Thank you! That fixed the problem and it created the executable snopt_ampl.

However, after I tried using this new executable in Pyomo, I get a new error. It appears that Pyomo recognizes that this executable is SNOPT 7.7.4, and the SolverFactory was able to build the solver, but here is there error:

SNOPT 7.7.4 : ERROR: Solver (asl) returned non-zero return code (-11)

ERROR: See the solver log above for diagnostic information.

Traceback (most recent call last):

  File "DoublePendulum.py", line 857, in <module>

    newM = runOpt(data)

  File "DoublePendulum.py", line 652, in runOpt

    results = opt.solve(m, tee=True)

  File "/home/mike/anaconda3/lib/python3.7/site-packages/pyomo/opt/base/solvers.py", line 600, in solve

    "Solver (%s) did not exit normally" % self.name)

pyutilib.common._exceptions.ApplicationError: Solver (asl) did not exit normally

This same code works using an IPOPT executable built against the ASL, and I am confident in the problem setup in Pyomo. 

Pyomo seems to refer to the SNOPT solver as (asl) which doesn't seem right, and also the error return code is -11, which as I understand is not a SNOPT exit condition. I know in IPOPT -11 is IP_INVALID_PROBLEM_DEFINITION, but as the problem works with IPOPT, I am not sure what this means.

Do you think that this could be an issue with the SNOPT executable? If not, do you have an idea about what could be wrong here?

Thank you again, its almost working!

Mike

[Zico Da Silva]

Hello,

I'm doing the same, and cannot build SNOPT for compatibility with Pyomo. I have obtained a license to use the software through my university, and wondering what are the steps to get this working? I followed the above, but it did not work. In particular, I'm confused by this sentence from Mike " I placed the trial static Fortran library and license file under/solvers/snopt/src. "? Do I need to provide source files from snopt for this to work?

I get an error during compilation using "make -f makefile_ampl`. Something to do with the "ld: Undefined symbols for architecture x86_64: .."

Any guidance would be appreciated? I'm using MacOS.

Best,

Zico

[SNOPT]

Hi Zico,

Where did you acquire SNOPT and what version is it?  (7.x?  source code? libraries?)

What does the rest of the error message say?  Knowing the name of the undefined symbol would be helpful.

--Elizabeth

[Zico Da Silva]

Hi Elizabeth,

Thanks for the prompt reply.

I'm actually unsure of the exact version - how would I find that? I was simply given a link to download the libraries (the link included my license ID). I downloaded the MacOS (gfortran) version for C++/C/fortran i.e. linking against "lsnopt7_cpp".

This is the rest of the error message:

gfortran  -o snopt_ampl snopt/snopt.o solvers/amplsolver.a /Users/zico/Downloads/libsnopt7_cpp/libsnopt7_cpp.a -ldl

Undefined symbols for architecture x86_64:

  "_main", referenced from:
     implicit entry/start for main executable
     (maybe you meant: _mainexit_ASL)
  "_xargv", referenced from:
      _MAIN__ in snopt.o
ld: symbol(s) not found for architecture x86_64

collect2: error: ld returned 1 exit status

make: *** [snopt_ampl] Error 1

I feel like I am missing something here. I do not have the source code, besides whatever I downloaded from Netlib. The download link did not allow me to download the source files.

Best,

Zico

[Philip E. Gill]

Hi Zico,

Elizabeth is asking in part where you got SNOPT from. Was it from the download site? From your advisor?

Cheers,
--Philip

On 6/29/22 12:28 PM, Zico Da Silva wrote:
> Hi Elizabeth,
>
> Thanks for the prompt reply.
>
> I'm actually unsure of the exact version - how would I find that? I was simply given a link to download the libraries (the link included my license ID). I downloaded the MacOS (gfortran) version for C++/C/fortran i.e. linking against "lsnopt7_cpp".
>
> This is the rest of the error message:
>

> *gfortran  -o snopt_ampl snopt/snopt.o solvers/amplsolver.a /Users/zico/Downloads/libsnopt7_cpp/libsnopt7_cpp.a -ldl

> Undefined symbols for architecture x86_64:
>   "_main", referenced from:
>      implicit entry/start for main executable
>      (maybe you meant: _mainexit_ASL)
>   "_xargv", referenced from:
>       _MAIN__ in snopt.o
> ld: symbol(s) not found for architecture x86_64
> collect2: error: ld returned 1 exit status

> make: *** [snopt_ampl] Error 1*
> *
> *

> I feel like I am missing something here. I do not have the source code, besides whatever I downloaded from Netlib. The download link did not allow me to download the source files.
>
> Best,
> Zico
> On Wednesday, 29 June 2022 at 17:38:06 UTC+1 SNOPT wrote:
>
> Hi Zico,
>
> Where did you acquire SNOPT and what version is it?  (7.x?  source code? libraries?)
>
> What does the rest of the error message say?  Knowing the name of the undefined symbol would be helpful.
>
> --Elizabeth
>
> On Wednesday, June 29, 2022 at 6:45:53 AM UTC-7 Zico Da Silva wrote:
>
> Hello,
>
> I'm doing the same, and cannot build SNOPT for compatibility with Pyomo. I have obtained a license to use the software through my university, and wondering what are the steps to get this working? I followed the above, but it did not work. In particular, I'm confused by this sentence from Mike " I placed the trial static Fortran library and license file under/solvers/snopt/src. "? Do I need to provide source files from snopt for this to work?
>

> I get an error during compilation using "make -f makefile_ampl`. Something to do with the "*ld: Undefined symbols for architecture x86_64: ..*"

>
> Any guidance would be appreciated? I'm using MacOS.
>
> Best,
> Zico
>
> On Friday, 19 June 2020 at 19:02:16 UTC+1 Mike T wrote:
>
> Elizabeth,
>
> Thank you! That fixed the problem and it created the executable snopt_ampl.
>
> However, after I tried using this new executable in Pyomo, I get a new error. It appears that Pyomo recognizes that this executable is SNOPT 7.7.4, and the SolverFactory was able to build the solver, but here is there error:
>
> SNOPT 7.7.4 : ERROR: Solver (asl) returned non-zero return code (-11)
> ERROR: See the solver log above for diagnostic information.
> Traceback (most recent call last):
>   File "DoublePendulum.py", line 857, in <module>
>     newM = runOpt(data)
>   File "DoublePendulum.py", line 652, in runOpt
>     results = opt.solve(m, tee=True)
>   File "/home/mike/anaconda3/lib/python3.7/site-packages/pyomo/opt/base/solvers.py", line 600, in solve

>     "Solver (%s) did not exit normally" % self.name <https://urldefense.com/v3/__http://self.name__;!!Mih3wA!AlDmaH-qvu-u6KMZXoJK_2BYneMA5VljSptVgxOTHBZQDzbyFPjBT8C6-hiz0Nu38NudCJvZlTvBeQ$>)

> 4.  Copy 'makefile_ampl', which I've placed here: https://gist.github.com/gnowzil/4bfc0b6c0ec725d0188bded45c885162 <https://urldefense.com/v3/__https://gist.github.com/gnowzil/4bfc0b6c0ec725d0188bded45c885162__;!!Mih3wA!AlDmaH-qvu-u6KMZXoJK_2BYneMA5VljSptVgxOTHBZQDzbyFPjBT8C6-hiz0Nu38NudCJuq2ZVw8g$>

>
> 3.  Modify SNOPT_LIB to point to the snopt7 library.  If necessary, modify the other locations as well.  By default, it assumes that the directories 'solvers' and 'snopt' are on the same level as the file.
> SNOPT_LIB   should point to the libsnopt7.a file
> AMPL_SOLVERS_DIR should point to the 'solvers' directory
> SNOPT_AMPL_DIR should point to the 'snopt' asl directory.
>
> 4.  Run makefile_ampl with the command "make -f makefile_ampl".  It should create the executable 'snopt_ampl'.
>
> --Elizabeth
>
>
> On Wednesday, June 17, 2020 at 9:09:38 AM UTC-7, Mike T wrote:
>
> Hello,
>
> I am attempting to use SNOPT with Pyomo for my research. It is my understanding that I need a compiled executable built against the ASL, as this is how I am using IPOPT currently. Unfortunately, I have been having a lot of difficulties generating this executable.
>
> I have downloaded the AMPL Solver Library interface, along with the wrapper for SNOPT through Netlib, and followed the instructions accordingly. I was successful in compiling the ASL, however I am running into errors while compiling the SNOPT sub directory. I placed the trial static Fortran library and license file under/solvers/snopt/src.
>

> This is the error I am given when attempting the *make* command (after successfully running *./configure)* in solvers/snopt:

>
> cd ${OBJDIR=sys.`uname -m`.`uname -s`}; make
> make[1]: Entering directory '/home/mike/My_Stuff/Snopt/solvers/snopt/sys.x86_64.Linux'
> cc -c -O -I../../sys.x86_64.Linux snopt.c
> make[1]: *** No rule to make target 'src/libsnopt.a', needed by 'snopt'.  Stop.
> make[1]: Leaving directory '/home/mike/My_Stuff/Snopt/solvers/snopt/sys.x86_64.Linux'
> make: *** [makefile:2: snopt] Error 2
>
> I noticed that the SNOPT library file was named differently in the makefile, (libsnopt.a vs libsnopt7.a), so I tried renaming the library file, as well as modifying the makefile to match the different name. Neither of these changes fixed the issue.
>
> I am not sure how to proceed. Is there documentation to assist this installation? Unfortunately, the READMEs from AMPL are not very helpful in the details.
>
> Is it even possible to generate this executable using the trial version of SNOPT?
>
> If so, what could I be doing incorrectly?
>
> Thank you for the help,
>
> Mike T.
>

> --
> You received this message because you are subscribed to the Google Groups "SNOPT" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to snopt+un...@googlegroups.com <mailto:snopt+un...@googlegroups.com>.
> To view this discussion on the web visit https://groups.google.com/d/msgid/snopt/5614a98d-7b76-4de5-aab5-8750fc184494n%40googlegroups.com <https://urldefense.com/v3/__https://groups.google.com/d/msgid/snopt/5614a98d-7b76-4de5-aab5-8750fc184494n*40googlegroups.com?utm_medium=email&utm_source=footer__;JQ!!Mih3wA!AlDmaH-qvu-u6KMZXoJK_2BYneMA5VljSptVgxOTHBZQDzbyFPjBT8C6-hiz0Nu38NudCJscCF9VSA$>.

[SNOPT]

Hi Zico,

Ok.  It looks like you're trying to compile the AMPL interface and linking with the precompiled SNOPT library.  It should be possible but I will need to verify the steps to do this.  If I recall correctly, you have to add an extra file, say fmain.c, with the following contents when you compile the AMPL files:

char **xargv;
extern void MAIN__(void);

main(int argc, char **argv)
{
  xargv = argv;
  MAIN__();
  return 0;
}

I will get back to you as soon as I can.

--Elizabeth

[Zico Da Silva]

Hi,

Yes, I got SNOPT from here: https://ccom.ucsd.edu/~optimizers/downloads/ but using a license I was able to download the precompiled files.

I have included a file called fmain.c from the instructions you provided above to Mike.

Thanks,

Zico

[SNOPT]

Did you add fmain.c to the list of files for compilation?  (Step 2 on this page https://gist.github.com/gnowzil/4bfc0b6c0ec725d0188bded45c885162)

You need to modify "makefile" in the solvers directory.  Should look like this:

...

a = \
        fmain.c \
        asldate.c \
        atof.c \

...

--Elizabeth

[Zico Da Silva]

Hi Elizabeth,

Thank you very much for the assitance. I got it to work by doing the above. However, I did do this previously (as stated in the original steps) but the makefile would be overwritten when you execute "./configurehere". So instead of running "./configurehere ; make", I did the following:

- ./configurehere

- Edit makefile to include the fmain.c source file.

- make

And that worked :)

Zico

[Yen-An Lu]

Hi,

I am getting the same issue as Mike:

ERROR: Solver (asl) returned non-zero return code (-11)

ERROR: Solver log: SNOPT 7.7.7 :

Traceback (most recent call last):

  File "snopt_test.py", line 18, in <module>
    results = SolverFactory('snopt_ampl').solve(model)
  File "/home/qizh/lu000285/.conda/envs/myenv_py371/lib/python3.7/site-packages/pyomo/opt/base/solvers.py", line 600, in solve
    "Solver (%s) did not exit normally" % self.name)

pyutilib.common._exceptions.ApplicationError: Solver (asl) did not exit normally

I followed the instructions on the GitHub page listed above. I got the snopt_ampl executable file, and it has been added to the sys path. I am wondering if there is anything else I should do to link snopt to pyomo? Or is there anything I should check?

Thanks,

Yen-An