Singularity and the RStudio Parallel Package

[Ping Luo]

I can run the following code in RStudio without Singularity, but it hangs at makeCluster(4) when RStudio is running in Singularity. How can I resolve the issue?

> library(parallel)
> cl <- makeCluster(4)

Ping

[David Trudgian]

Hi Ping,

This will need a bit more information to look at.

What is the Singularity container, or recipe you are using? What kind of environment are you running Singularity under - on a single machine, cluster through a scheduler etc?

Is there any debug output / errors?

Thanks.

DT

--
You received this message because you are subscribed to the Google Groups "singularity" group.
To unsubscribe from this group and stop receiving emails from it, send an email to singularity...@lbl.gov.

[Ping Luo]

David,

The Singularity container is created from the recipe attached. Singularity is running under CentOS 6.10, on a single machine. It is running through Open OnDemand (https://osc.github.io/ood-documentation/master/), which I think can be considered as running through a scheduler. 

There is no error in rsession.log file. Where should I check logs for Singularity?

Thanks,

 

Ping

[David Trudgian]

Hi Ping,

There is no error in rsession.log file. Where should I check logs for Singularity?

Am afraid in this case there is not going to be anything useful in Singularity logs. The issue is something to do with how the R parallel package is working. However, detailed logs can be obtained by adding the `-d` option to the singularity command, e.g. `singularity -d run myimage.sif`

On my test machine, building your container and running under singularity I get success...

> library(parallel)

> cl <- makeCluster(4)

> cl

socket cluster with 4 nodes on host ‘localhost’

By default R parallel makeCluster sets up an R cluster with socket communication. Looking around on the net there is a lot that can potentially go wrong, and the most common failure mode is just hanging indefinitely. It’s possible to force a bit more info with…

> cl <- makeCluster(4,  outfile=“")

… which will ensure any errors from the workers spinning up is sent to standard error/output.

You can also get the command that is used to startup workers - and then manually try to run it, to see what is going on. For me the the command looks like...

> cl <- makeCluster(4, manual=T, outfile="")

Manually start worker on localhost with

     '/usr/lib/R/bin/Rscript' --default-packages=datasets,utils,grDevices,graphics,stats,methods -e 'parallel:::.slaveRSOCK()' MASTER=localhost PORT=11156 OUT= TIMEOUT=2592000 XDR=TRUE

Do you have any detail on how Singularity is installed in the Open OnDemand environment you are using, and how exactly you are running the container? i.e. what is the `singularity run …..` or `singularity exec ….` command being used?

Thanks 

DT

<singularity>

[Ping Luo]

David,

I think I found out the cause of the problem. The environment variable SINGULARITY_CONTAIN is set to 1 before OOD runs 'singularity exec rserver ...'. If I comment it out and then start RStudio in OOD, I can run makeCluster without any problem.

You can reproduce this without OOD by setting SINGULARITY_CONTAIN=1 before you run 'singularity exec rserver' from the command line.

I don't know what's the use of this environment variable. I was guessing it was used to limit number of workers to 1. So I tried to set it to some number larger than 1. It always hung no matter how large SINGULARITY_CONTAIN was specified. So my guess seems not right. Do you know what it does?

Thanks,

Ping

[David Trudgian]

Ping,

That’s interesting - if I set that it still works okay….

$ SINGULARITY_CONTAIN=1 singularity run r.simg

R version 3.4.4 (2018-03-15) -- "Someone to Lean On"

Copyright (C) 2018 The R Foundation for Statistical Computing

Platform: x86_64-pc-linux-gnu (64-bit)

R is free software and comes with ABSOLUTELY NO WARRANTY.

You are welcome to redistribute it under certain conditions.

Type 'license()' or 'licence()' for distribution details.

  Natural language support but running in an English locale

R is a collaborative project with many contributors.

Type 'contributors()' for more information and

'citation()' on how to cite R or R packages in publications.

Type 'demo()' for some demos, 'help()' for on-line help, or

'help.start()' for an HTML browser interface to help.

Type 'q()' to quit R.

> library(parallel)

> cl <- makeCluster(4,  outfile="")

starting worker pid=31864 on localhost:11735 at 22:11:37.305

starting worker pid=31873 on localhost:11735 at 22:11:37.495

starting worker pid=31882 on localhost:11735 at 22:11:37.685

starting worker pid=31891 on localhost:11735 at 22:11:37.871

>

Do you have pointer to what exact singularity command OOD is using, what version of Singularity is in play here etc? I’ve had a quick look in some OOD code and it seems like a lot depends on how its installed, and I didn’t find exactly where Singularity is being used to run R.

Thanks.

[Ping Luo]

David,

That's interesting. Mine still hangs

$ singularity exec 3.4.3/rstudio.simg R

R version 3.4.3 (2017-11-30) -- "Kite-Eating Tree"
Copyright (C) 2017 The R Foundation for Statistical Computing

Platform: x86_64-pc-linux-gnu (64-bit)

R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.

  Natural language support but running in an English locale

R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.

Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.

> library(parallel)

> makeCluster(4, outfile="")
starting worker pid=20154 on localhost:11709 at 22:24:33.362
Error in socketConnection(master, port = port, blocking = TRUE, open = "a+b",  :
  cannot open the connection
Calls: <Anonymous> ... doTryCatch -> recvData -> makeSOCKmaster -> socketConnection
In addition: Warning message:
In socketConnection(master, port = port, blocking = TRUE, open = "a+b",  :
  localhost:11709 cannot be opened
Execution halted

I think it is not a problem of OOD, as I am running this without OOD. The singularity version I am using is 2.4.2. Which version are you using?

Ping

[David Trudgian]

Hi Ping,

I am using 2.6.0 - it’s *highly* recommended to use the latest stable version. See the release page on GitHub for security and bug fix information.

https://github.com/singularityware/singularity/releases

Cheers,

DT

[Ping Luo]

David,

Singularity 2.6.0 doesn't work either when SINGULARITY_CONTAIN=1

> library(parallel)
> makeCluster(2, outfile="")
starting worker pid=27479 on localhost:11250 at 17:32:12.794

Error in socketConnection(master, port = port, blocking = TRUE, open = "a+b",  :
  cannot open the connection
Calls: <Anonymous> ... doTryCatch -> recvData -> makeSOCKmaster -> socketConnection
In addition: Warning message:
In socketConnection(master, port = port, blocking = TRUE, open = "a+b",  :

  localhost:11250 cannot be opened
Execution halted

(it hangs ...)

The R version I am using is 3.4.3. And the host OS is CentOS 6.10.

Thanks,

Ping

[David Trudgian]

Hi Ping,

Looks like this will need me to fire up a CentOS 6 VM to take a look at this as I can’t reproduce on my CentOS 7 and Ubuntu or Fedora environments. I’ll try and get around to this in the early part of the week. At least we have an error to go on r.e. the failure to open a socket.

DT

[Ping Luo]

David,

I tested Singularity on CentOS 7 and R hangs when calling makeCluster if SINGULARITY_CONTAIN=1. It is so strange why your test worked. Below please find my test environment.

System information:
Host OS: CentOS 7.5.1804

Singularity image OS: ubuntu 16.04

Singularity version: 2.6.0-dist

R: 3.4.4

[pingluo@portal containers]$  SINGULARITY_CONTAIN=1 ../2.6.0/bin/singularity run rstudio-new

R version 3.4.4 (2018-03-15) -- "Someone to Lean On"
Copyright (C) 2018 The R Foundation for Statistical Computing
Platform: x86_64-pc-linux-gnu (64-bit)

R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.

  Natural language support but running in an English locale

R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.

Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.

> library(parallel)

> makeCluster(4, outfile="")
starting worker pid=45787 on localhost:11652 at 18:41:49.386

Error in socketConnection(master, port = port, blocking = TRUE, open = "a+b",  :
  cannot open the connection
Calls: <Anonymous> ... doTryCatch -> recvData -> makeSOCKmaster -> socketConnection
In addition: Warning message:
In socketConnection(master, port = port, blocking = TRUE, open = "a+b",  :

  localhost:11652 cannot be opened
Execution halted

Thanks,

Ping

[David Trudgian]

Hi Ping,

From your output here it doesn’t look like it did hang in the same way… it ran through to reporting a socket error?

In socketConnection(master, port = port, blocking = TRUE, open = "a+b",  :
  localhost:11652 cannot be opened
Execution halted

This is the same thing I saw on CentOS 7 - when you say it still hangs, how do you experience the hang - it freezes, or are you just referring to this socket error?

DT

[Ping Luo]

David,

I am experiencing the same thing on CentOS7 as on CentOS6. The following test is done on CentOS 6.

[pingluo@ada7 rstudio]$ SINGULARITY_CONTAIN=1 ../../2.6.0/bin/singularity exec rstudio-07182018-2.6.0.simg R
> library(parallel)
> makeCluster(4, outfile="")
starting worker pid=2883 on localhost:11470 at 20:24:53.231

Error in socketConnection(master, port = port, blocking = TRUE, open = "a+b",  :
  cannot open the connection
Calls: <Anonymous> ... doTryCatch -> recvData -> makeSOCKmaster -> socketConnection
In addition: Warning message:
In socketConnection(master, port = port, blocking = TRUE, open = "a+b",  :

  localhost:11470 cannot be opened
Execution halted

I thought you were able to run on makeCluster without problem from one of your previous email. I am copying the test result from your email and pasted it below:

$ SINGULARITY_CONTAIN=1 singularity run r.simg

...

> library(parallel)

> cl <- makeCluster(4,  outfile="")

starting worker pid=31864 on localhost:11735 at 22:11:37.305

starting worker pid=31873 on localhost:11735 at 22:11:37.495

starting worker pid=31882 on localhost:11735 at 22:11:37.685

starting worker pid=31891 on localhost:11735 at 22:11:37.871

>

Ping

[David Trudgian]

Apologies - I misread my own emails there... yes I can get the workers on CentOS 7 (just re-ran it).

Is this all still through Open OnDemand? Or is it on standalone Singularity on straight CentOS 7 now?

DT

[Ping Luo]

It is on standalone Singularity on straight CentOS 7, nothing to do with OOD.

[David Trudgian]

Hi Ping,

There definitely appears to be something unusual on your setup that is blocking opening that port on localhost - I’ve just replicated again below. Anything at all unusual about your CentOS installs you can think of? 

If anyone else is able to replicate success / failure that might also be useful. I’m afraid I’m out of ideas here as I can’t replicate the failure on a hardware CentOS install, or a VM.

dave@petroc:~/Git/singularity

03:49 PM $ cat /etc/redhat-release

CentOS Linux release 7.5.1804 (Core)

dave@petroc:~/Git/singularity

03:47 PM $ singularity --version

2.6.0-HEAD.579c4150

dave@petroc:~/Git/singularity

03:47 PM $ singularity build r.simg docker://r-base

Docker image path: index.docker.io/library/r-base:latest

Cache folder set to /home/dave/.singularity/docker

[6/6] |===================================| 100.0%

Importing: base Singularity environment

Exploding layer: sha256:8f9234435f64947bcad1104e14346b296e1b87f45a1afe2f95e3bf1cb0864926.tar.gz

Exploding layer: sha256:4768ed7b65fb84476d16e30ced8a0318191752cdf6488299dad3db72c1500cce.tar.gz

Exploding layer: sha256:315aef06947a63106f1b991c5aaf93985a3cb03c55da613fd72dac13dc85fd40.tar.gz

Exploding layer: sha256:646915aa41c68863854026530acca13cabd24e5a85f992f3532101bf70471306.tar.gz

Exploding layer: sha256:430db1c15e9318ed42284202243e072dd74e0249b333bcbc74ca908ddb168a9c.tar.gz

Exploding layer: sha256:402b231c832395be460ac7c9a55a825933cfc55ba49f3a4fd1104061f22736a1.tar.gz

Exploding layer: sha256:1980f7a9ddb8074fff5d1d18b580325882246fc86b94bc55b4fc06744a1a180b.tar.gz

WARNING: Building container as an unprivileged user. If you run this container as root

WARNING: it may be missing some functionality.

Building Singularity image...

Singularity container built: r.simg

Cleaning up...

dave@petroc:~/Git/singularity

03:49 PM $ SINGULARITY_CONTAIN=1 singularity run r.simg

R version 3.5.1 (2018-07-02) -- "Feather Spray"

Copyright (C) 2018 The R Foundation for Statistical Computing

Platform: x86_64-pc-linux-gnu (64-bit)

R is free software and comes with ABSOLUTELY NO WARRANTY.

You are welcome to redistribute it under certain conditions.

Type 'license()' or 'licence()' for distribution details.

  Natural language support but running in an English locale

R is a collaborative project with many contributors.

Type 'contributors()' for more information and

'citation()' on how to cite R or R packages in publications.

Type 'demo()' for some demos, 'help()' for on-line help, or

'help.start()' for an HTML browser interface to help.

Type 'q()' to quit R.

> library(parallel)

> cl <- makeCluster(4,  outfile="")

starting worker pid=12459 on localhost:11908 at 20:49:27.009

starting worker pid=12482 on localhost:11908 at 20:49:27.236

starting worker pid=12491 on localhost:11908 at 20:49:27.421

starting worker pid=12501 on localhost:11908 at 20:49:27.607

>

Save workspace image? [y/n/c]: n

[Ping Luo]

David,

Looks like we have built R from different source and with different version. I replicated all your steps and please see my test below. This time it hangs after starting a worker process:

[pingluo@portal containers]$ cat /etc/redhat-release

CentOS Linux release 7.5.1804 (Core)

[pingluo@portal containers]$ ../2.6.0/bin/singularity build r.simg docker://r-base

Docker image path: index.docker.io/library/r-base:latest

Cache folder set to /general/home/pingluo/.singularity/docker

Importing: base Singularity environment
Exploding layer: sha256:8f9234435f64947bcad1104e14346b296e1b87f45a1afe2f95e3bf1cb0864926.tar.gz
Exploding layer: sha256:4768ed7b65fb84476d16e30ced8a0318191752cdf6488299dad3db72c1500cce.tar.gz
Exploding layer: sha256:315aef06947a63106f1b991c5aaf93985a3cb03c55da613fd72dac13dc85fd40.tar.gz
Exploding layer: sha256:646915aa41c68863854026530acca13cabd24e5a85f992f3532101bf70471306.tar.gz
Exploding layer: sha256:430db1c15e9318ed42284202243e072dd74e0249b333bcbc74ca908ddb168a9c.tar.gz
Exploding layer: sha256:402b231c832395be460ac7c9a55a825933cfc55ba49f3a4fd1104061f22736a1.tar.gz
Exploding layer: sha256:1980f7a9ddb8074fff5d1d18b580325882246fc86b94bc55b4fc06744a1a180b.tar.gz
WARNING: Building container as an unprivileged user. If you run this container as root
WARNING: it may be missing some functionality.
Building Singularity image...
Singularity container built: r.simg
Cleaning up...

[pingluo@portal containers]$ SINGULARITY_CONTAIN=1 ../2.6.0/bin/singularity run r.simg

R version 3.5.1 (2018-07-02) -- "Feather Spray"
Copyright (C) 2018 The R Foundation for Statistical Computing
Platform: x86_64-pc-linux-gnu (64-bit)

R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.

  Natural language support but running in an English locale

R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.

Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.

> library(parallel)

> makeCluster(4, outfile="")
starting worker pid=36308 on localhost:11967 at 14:30:32.178
^C
> q()

Save workspace image? [y/n/c]: n

[pingluo@portal containers]$

[pingluo@portal containers]$ ps -ef|grep pingluo
pingluo  34001 34000  0 09:13 pts/1    00:00:00 -bash
pingluo  36272 34001  0 09:30 pts/1    00:00:00 /usr/lib/R/bin/exec/R
pingluo  36308     1  0 09:30 pts/1    00:00:00 /usr/lib/R/bin/exec/R --slave --no-restore -e parallel:::.slaveRSOCK() --args MASTER=localhost PORT=11967 OUT= SETUPTIMEOUT=120 TIMEOUT=2592000 XDR=TRUE


As comparison, I pulled Rstudio from shub (this is how we got our Rstudo and R) and did the same tests with the newer version of R (3.5.1) as yours, as well as with the version of R (3.4.4) we are currently using. With singularity-rstudio:3.5.1, I had the exact same problem as shown above , ie, R hangs after starting one worker. With singularity-rstudio:3.4.4, it halts after reporting not able to open some local port, the same behavior as I mentioned in my previous emails. 

[pingluo@portal containers]$ ../2.6.0/bin/singularity pull --name rstudio.img shub://nickjer/singularity-rstudio:3.5.1
Progress |===================================| 100.0%
Done. Container is at: /scratch/user/pingluo/projects/singularity/containers/rstudio.img
[pingluo@portal containers]$ SINGULARITY_CONTAIN=1 ../2.6.0/bin/singularity exec rstudio.img R

R version 3.5.1 (2018-07-02) -- "Feather Spray"
Copyright (C) 2018 The R Foundation for Statistical Computing
Platform: x86_64-pc-linux-gnu (64-bit)

R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.

  Natural language support but running in an English locale

R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.

Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.

> library(parallel)

> makeCluster(4, outfile="")
starting worker pid=40204 on localhost:11152 at 15:09:43.931
^C
> q()

Save workspace image? [y/n/c]: n

[pingluo@portal containers]$

[pingluo@portal containers]$ ../2.6.0/bin/singularity pull --name rstudio.img shub://nickjer/singularity-rstudio:3.4.4
Progress |===================================| 100.0%
Done. Container is at: /scratch/user/pingluo/projects/singularity/containers/rstudio.img
[pingluo@portal containers]$ SINGULARITY_CONTAIN=1 ../2.6.0/bin/singularity exec rstudio.img R

R version 3.4.4 (2018-03-15) -- "Someone to Lean On"

Copyright (C) 2018 The R Foundation for Statistical Computing
Platform: x86_64-pc-linux-gnu (64-bit)

R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.

  Natural language support but running in an English locale

R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.

Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.

> library(parallel)

> makeCluster(4, outfile="")
starting worker pid=41169 on localhost:11738 at 15:19:31.999

Error in socketConnection(master, port = port, blocking = TRUE, open = "a+b",  :
  cannot open the connection
Calls: <Anonymous> ... doTryCatch -> recvData -> makeSOCKmaster -> socketConnection
In addition: Warning message:
In socketConnection(master, port = port, blocking = TRUE, open = "a+b",  :

  localhost:11738 cannot be opened
Execution halted
^C 
> q()

Save workspace image? [y/n/c]: n

[pingluo@portal containers]$

In summery, I had a problem of starting multiple workers from R parallel using singularity when SINGULARITY_CONTAIN=1 on our clusters running CentOS 6 and 7. With 3.5.1, R hangs after 1 worker is started. With 3.4.4, R halts without starting any workers.

Thanks,

Ping