AMPL cannot be executed under musl

[marc...@gmail.com]

People using Alpine Linux, Void Linux, Morpheus Linux, etc. aren't able to run AMPL.
We currently have to install less secure and more resource-hungry libcs (http://www.etalabs.net/compare_libcs.html) just to be able to get the AMPL binary to run, which isn't either an easy or elegant approach.
musl-based AMPL releases would be good for these users and maybe not too troublesome for you developers?
I heard you depend on dynamic-linking to make AMPL work, maybe that issue should be tackled at its core (I can't see how being enforced to dynamic-link would be any good?) or maybe it could just be dynamic linked against musl for the users that don't have a way out.
The musl libc developer is a chill and helpful person. If you need anything, I'm sure he would be eager to help.

Hope to hear from you soon and to run AMPL natively too,
Best wishes,

Marcus C.

[victor.z...@gmail.com]

You can run AMPL or any other program that uses glibc on Alpine Linux by following the steps described in http://wiki.alpinelinux.org/wiki/Running_glibc_programs.

For example, here's a (somewhat suboptimal) script that creates a simple chroot environment for running AMPL:

  wget http://archlinux.my-universe.com/community/os/i686/busybox-1.23.2-1-i686.pkg.tar.xz

  wget http://archlinux.my-universe.com/core/os/i686/glibc-2.21-2-i686.pkg.tar.xz

  mkdir -p ~/chroot/bin ~/chroot/dev ~/chroot/proc ~/chroot/root ~/chroot/etc ~/chroot/lib ~/chroot/tmp

  for i in *.pkg.tar.xz;do

    tar xJf $i -C ~/chroot

  done

  cp /etc/resolv.conf ~/chroot/etc/

  ln -s /usr/bin/busybox ~/chroot/bin/sh

  ln -s /usr/bin/busybox ~/chroot/bin/tar

  ln -s /usr/bin/busybox ~/chroot/bin/wget

  ln -s /usr/lib/ld-linux.so.2 ~/chroot/lib

  chroot ~/chroot/ /bin/sh

After running this script, you'll be able to install and run AMPL:

# wget http://ampl.com/dl/demo/ampl-demo-linux32.tar.gz

# tar xzf ampl-demo-linux32.tar.gz 

# cd ampl-demo

# ./ampl

ampl:

without affecting the main system or compromising security (even more secure way would be to use containers).

Dynamic linking is used in AMPL for loading custom function libraries among other things so it's not possible to completely replace it with static linking. Also since you are concerned about resources, you should probably prefer shared libraries because static linking increase binary sizes and, possibly, memory usage.

HTH,

Victor

--
You received this message because you are subscribed to the Google Groups "AMPL Modeling Language" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ampl+uns...@googlegroups.com.
To post to this group, send email to am...@googlegroups.com.
Visit this group at http://groups.google.com/group/ampl.
For more options, visit https://groups.google.com/d/optout.

[marc...@gmail.com]

Thanks for the reply, Victor.

I may do that, but don't you agree the process is a bit troublesome and especially hard for people that aren't computer-smart like me?
Would it be out of question for AMPL to have a musl release (dynamic link)? I think it would be as simple as compiling it against musl for AMPL devs and releasing the binary?
For end users, setting up another libc on their system and all that isn't friendly.

Best wishes,

Marcus C.

[victor.z...@gmail.com]

I agree that setting up AMPL on a musl-based system requires some technical knowledge, but I'd expect it from users of such systems. Less technically advanced users are likely to choose one of the more popular Linux distributions that are mainly based on glibc or compatible implementations of the C standard library.

Unfortunately it's not as easy as compiling AMPL itself against musl, because we should also take into account solvers and function libraries. AMPL and solvers should use binary compatible versions of libc at least because of the function libraries that they load. However, to the best of my knowledge, major solver vendors only provide libraries compiled against glibc.

Best regards,

Victor

--

[marc...@gmail.com]

Victor, I got your chroot method working and ampl ran.
It's troublesome to do "chroot ~/chroot/ /bin/sh" then "cd ampl-demo" and then "./ampl" every time just to run it and to go outside of chroot to use emacs, create my model.mod, copy to the chroot folder, etc. It would be really good to have native dynamic linked musl builds. Do you think that's possible?

Anyway, I still can't use ampl inside this temporary fix you gave me (using chroot). It hangs after solve; which makes me believe there's not enough memory?

Best wishes,
Marcus

[victor.z...@gmail.com]

You can use a simple script to run ampl in a chrooted environment instead of running chroot manually:

  #!/bin/sh

  chroot ~/chroot/ /ampl-demo/ampl $@

We'll consider adding a musl-based version of ampl, but I would't expect it anytime soon for the reasons I mentioned in my previous emails.

Could you post the problem AMPL hangs on as well as the complete output?

HTH,

Victor

[marc...@gmail.com]

cd ./chroot
cat prod0.mod
>var XB;
>var XC;
>maximize Profit: 25 * XB + 30 * XC;
>subject to Time: (1/200) * XB + (1/140) * XC <= 40;
>subject to B_limit: 0 <= XB <= 6000;
>subject to C_limit: 0 <= XC <= 4000;
chroot ~/chroot/ /ampl-demo/ampl $@
model prod0.mod;
solve;

Then it hangs after solve;
If I try option solver cplex; or any other it still hangs when I try to solve;

[victor.z...@gmail.com]

Unfortunately I couldn't reproduce it on my Alpine Linux box. Could you post the complete output that you get from AMPL including the solver output if any?

HTH,

Victor

--

[marc...@gmail.com]

There is no output.
It just hangs after I type solve;

$ sudo ampl
ampl: reset;
ampl: model prod0.mod;
ampl: solve;
^Z[1]+  Stopped                    sudo ampl

I have to ^Z eventually, it just hangs there without doing anything for as long as I leave it.

[victor.z...@gmail.com]

Could you try writing an .nl file and passing it to the solver from command line? You can write the file test.nl by replacing the solve command with the following:

  write gtest;

If this succeeds, try passing the file to the solver:

  minos test.nl

This will help in localizing the issue.

Best regards,

Victor

[marc...@gmail.com]

$ sudo ampl
ampl: reset;
ampl: model prod0.mod;

ampl: write gtest;
ampl: minos test.nl
syntax error
context:  minos  >>> test. <<< nl
ampl?

[victor.z...@gmail.com]

I meant running minos from the command line, not from AMPL:

  $ minos test.nl

You can also get verbose output from minos by setting the outlev option to 2:

  $ minos_options='outlev 2' minos test

Since the write command succeeds, it looks that the problem is with running the solver.

HTH,

Victor

[marc...@gmail.com]

Then it works? The problem looks like it's ampl calling the solver.

~$ sudo chroot ~/chroot/ /bin/sh
/# cd ampl-demo/
/ampl-demo# ./ampl

ampl: reset;
ampl: model prod0.mod;
ampl: write gtest;

ampl: quit;
/ampl-demo# ./minos test.nl         
MINOS 5.51: optimal solution found.
2 iterations, objective 191999.99999999997
/ampl-demo# minos_options='outlev 2' ./minos test.nl
outlev 2
 Nonzeros allowed for in LU factors    10147
 Scale option  2,    Partial price   1
 
 Itn      0 -- linear E rows feasible.   Obj =   0.000000000E+00
 Itn      0 -- feasible solution.  Objective =   0.000000000E+00
 
 EXIT -- optimal solution found
 Problem name                        
 No. of iterations                   2   Objective value      1.9200000000E+05
 No. of degenerate steps             0   Percentage                        .00
 Max x       (scaled)        1 5.0E+03   Max pi      (scaled)        2 3.0E+01
 Max x                       1 6.0E+03   Max pi                      1 4.2E+03
 Max Prim inf(scaled)        0 0.0E+00   Max Dual inf(scaled)        0 0.0E+00
 Max Primal infeas           0 0.0E+00   Max Dual infeas             0 0.0E+00
 Time for MPS input                            .00 seconds
 Time for solving problem                      .00 seconds
 Time for solution output                      .00 seconds
 Time for constraint functions                 .00 seconds
 Time for objective function                   .00 seconds
MINOS 5.51: optimal solution found.
2 iterations, objective 191999.99999999997

[marc...@gmail.com]

cplex works too (and actually gives the exact solution, not approximate like minos).

# ./cplex test.nl
CPLEX 12.6.1.0: optimal solution; objective 192000
1 dual simplex iterations (0 in phase I)

The problem is indeed with ampl calling the solver and printing their result, right?
Also, why isn't cplex ampl's default solver? minos got this really easy problem wrong (sort of, it got really close, but not exact).

[victor.z...@gmail.com]

Yes, it looks like something is wrong with running the solver from AMPL. Do you get any output in AMPL when setting outlev to higher value in minos_options:

  ampl: option minos_options 'outlev=2';

  ampl: solve;

Also it might help if you could run the following AMPL script:

model prod0.mod;

print 'writing problem';

write 'b/tmp/test';

print 'running solver';

shell 'minos /tmp/test -AMPL';

print 'reading solution';

solution '/tmp/test.sol';

and post the complete output. It closer corresponds to the behavior of the solve command but split into multiple commands to make localizing the problem easier.

I'm not sure why MINOS is the default in AMPL, perhaps because, unlike CPLEX, it can handle general nonlinear problems.

Best regards,

Victor

--

[marc...@gmail.com]

Here's the first thing you asked. No results.

    $ sudo ampl
    ampl: reset;
    ampl: model prod0.mod;

    ampl: option solver minos;

    ampl: option minos_options 'outlev=2';
    ampl: solve;

Here's the output from the script. I don't think it helps, does it?

    $ sudo ampl
    ampl: include script.run;
    writing problem
    running solver

Do you have access to ampl source code?
Maybe you could try compiling it under musl and releasing the binary? Then only the solvers would be chrooted and I may have a better chance at this.
I wonder if any solvers could work natively on musl... Maybe the open-source ones, but they aren't that good. Anyway, I digress...

[victor.z...@gmail.com]

Something is apparently broken in Alpine Linux when it comes to the machinery AMPL uses for history processing. If you invoke "ampl -vi2 ..." rather than "ampl ..." (with -vi2 as the first command-line argument), history processing is disabled and "solve;" should work as expected.

I don't do AMPL releases and unfortunately, as I mentioned earlier, it is not enough to just recompile AMPL itself with musl.

Regards,

Victor

[marc...@gmail.com]

It can't find the solver with that flag.

$ sudo ampl -vi2

ampl: reset;
ampl: model prod0.mod;

ampl: solve;
Cannot find "minos"
ampl: option solver cplex;
ampl: solve;
Cannot find "cplex"
ampl:

minos and cplex are available. I tried option solver "./cplex"; but that didn't work either. I also made sure to chmod +x them, to no avail.

Also, you say "something is apparently broken in Alpine Linux when it comes to the machinery AMPL uses for history processing".
Who's at fault here? AMPL, musl or Alpine Linux? Should I forward this talk to either Rich Felker (musl) or Natanael Copa (Alpine)? I'm sure if it's an issue on their end they would love to track it down and fix it. They are really good guys.

[victor.z...@gmail.com]

Is the solver in the $PATH in the chroot environment? You can either add the directory containing solvers to the PATH environment variable or specify the (full) path in the solver option:

  option solver 'path/to/solver';

Keep in mind that you need to adjust your paths for the chroot environment as they will be different from paths in your host environment. For example, if you have /tmp/chroot as your chroot directory and the path to the minos executable is /tmp/chroot/ampl/minos, you should specify

  option solver '/ampl/minos';

rather than

  option solver '/tmp/chroot/ampl/minos';

As for the history processing issue, I'll try to find out more details. In the meantime, you can use the -vi2 option.

HTH,

Victor

[marc...@gmail.com]

Got it to work. Finally.
But this was a lot of workarounds and paths are troublesome for me when saving models, etc.
I hope AMPL eventually works natively and only the solver chrooted (or maybe not even it, depending)
Thanks for the support, I hope we can eventually get even further in simplifying the process. musl is really good and AMPL is the only program I can't use with it, unfortunately.

[victor.z...@gmail.com]

The issue with history processing and solve has been fixed in AMPL version 20150422, so if you update to this version you won't need the -vi2 option any more.

HTH,

Victor