Compiled MATLAB app not including all libs.

[harry mangalam]

Starting a new subject line since this is now the next step..

 

The previous was just a lead-up to the real reason for trying Sing.

I have a MATLAB script that requires a lib that has a GLIBC incompatibility with our current cluster OS but that the PI is desperate to run.

 

I've compiled the mcode on my laptop to 'image2pce' and it runs as required, but as you might guess, has a zillion direct lib dependencies and zillions more indirect lib deps.

 

When I try to Sing-wrap it, Sing goes thru multiple resolution phases, but eventually it is built (all 225MB), but when trying to launch, keeps complaining about missing libs.

 

the short form is:

# set the LD_LIBRARY_PATH to include the nec. MATLAB paths

# mcc-compile the mcode into a native executable

 

#set the sspec file to:

====

Name: image2pce

Exec: /home/hjm/nacs/zeba/pct_hpc/standalone/image2pce

====

 

#create the sapp:

$ time singularity build image2pce.sspec image2pce.sapp > sapp.output

 

real 5m36.986s

 

# try to run the sapp:

$ ./image2pce.sapp

An Error has occurred while trying to initialize the MATLAB Runtime.

The error is: Fatal error loading library /home/hjm/MATLAB/R2015b/bin/glnxa64/libmat.so Error: /home/hjm/MATLAB/R2015b/bin/glnxa64/libmat.so: cannot open shared object file: No such file or directory

Error:mclmcr initialization failed

 

 

# and it's true, the file libmat.so is not included in the build output

$ grep libmat.so sapp.output

# nothing

 

# !!! BUT !!! it's also true that it's not included in the ldd output from the compiled app:

 

ldd image2pce | grep libmat.so

# nothing

 

!! so this is a problem - if the app doesn't acknowledge the requirement for libmat.so, how will Singularity resolve it?

 

# however, when I try to include it explicitly in the sspec file

====

Name: image2pce

Exec: /home/hjm/nacs/zeba/pct_hpc/standalone/image2pce

%files

/home/hjm/MATLAB/R2015b/bin/glnxa64/libmat.so

====

 

# and then rebuild

time singularity build image2pce.sspec image2pce.sapp > sapp.output

 

# and check:

$ grep libmat.so sapp.output

Installing file: /home/hjm/MATLAB/R2015b/bin/glnxa64/libmat.so

 

# and now it's something else....

 

$ ./image2pce.sapp

An Error has occurred while trying to initialize the MATLAB Runtime.

The error is: Fatal error loading library /home/hjm/MATLAB/R2015b/bin/glnxa64/libmwmclbase.so Error: /home/hjm/MATLAB/R2015b/bin/glnxa64/libmwmclbase.so: cannot open shared object file: No such file or directory

Error:mclmcr initialization failed

 

there are 1.8GB of files in /home/hjm/MATLAB/R2015b/bin/glnxa64/, so adding all of them explicitly is a non-starter.

 

and at 5.5min per re-wrapping file-by-file additions, that's going to be tiring..

 

Is there a way to pre-resolve the deps? Maybe this is more a MATLAB problem than a Sing problem..

 

hjm

 

 

---

Harry Mangalam - Research Computing, OIT, Rm 225 MSTB, UC Irvine

[m/c 2225] / 92697 Google Voice Multiplexer: (949) 478-4487

415 South Circle View Dr, Irvine, CA, 92697 [shipping]

XSEDE 'Campus Champion' - ask me about your research computing needs.

Map to Office | Map to Data Center Gate

[the command line is the new black]

---

 

[Gregory M. Kurtzer]

Try to start off with using "singularity specgen ...."

It works really well at identifying dependencies of the primary thread (not forks) and will write a new specfile for you. 

Other the. That, the more that you can include in the spec he shorter the amount of dependency loops will need to occur. 

Hope that helps!!

Sent from my iPhone

[harry mangalam]

Thanks Greg,

 

The use of the 'specgen' option seemed to help quite a bit. It did not insert the full path of the executable (which I think may qualify as a bug), and inserted the requirements of some temporary lock files that were not available when the matlab app was not-running, but otherwise seemed to catch ~140 files that were referenced and explicitly included them in the '%files' stanza.

 

The build took a very long time (~32m) to complete and resulted in a 640MB sapp file(!). When that sapp file was executed, 'top' shows a few seconds of gzip & cpio, followed by an unending stream of only this line:

 

utLoadlibrary mcos_impl failed, error = 1

 

using about 100% of a CPU, until killed (after waiting about 10m)

 

the top line:

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND

31879 hjm 20 0 504272 66612 50944 S 98.9 0.4 8:06.44 image2pce

 

so I'm further along in the process but not quite there. this is an especially hairy use case and it will be amazing if it actually works..

 

hjm

[Gregory M. Kurtzer]

Harry,

One thing additionally you can try is to add a %test scriptlet in your spec. It will try to execute this scriptlet on the temporary container before packaging everything up.

The bit about using specgen as well is that it will grab every file touched... which includes locales and archives which may not be specifically necessary (and will add a lot of space). 

You can also try a brute force method of including everything from your Matlab path instead of referencing single paths of each of the libraries. Maybe that will help?

Greg

--

Gregory M. Kurtzer
Technical Lead and HPC Systems Architect
High Performance Computing Services (HPCS)
University of California
Lawrence Berkeley National Laboratory
One Cyclotron Road, Berkeley, CA 94720