Anvi-refine / Anvi-interactive on SGE cluster (OGE)

[Robert Kwapich]

Hi community!

I have user access to a lab SGE cluster (or rather Oracle Grid Engine, as it is now called). I am trying to run anvi-refine is server mode, for the simplicity I've tried to run it on example infant gut. When I run it locally (anvio 2.4.0) it works like charm, load the data in a couple of seconds. But when working on "--server-mode" I get constantly running process, and the "log-output" is like this:

Contigs DB ...................................: Initialized: CONTIGS.db (v. 8)

                                                                                

[0;30m [46m

[11 Sep 17 23:06:45 Initializing splits taxonomy] ...                          [0m

                                                                                

                                                                                

[0;30m [46m

[11 Sep 17 23:06:46 Initializing functions class] ...                          [0m

                                                                                

                                                                                

[0;30m [46m

[11 Sep 17 23:06:46 Initializing the profile database superclass] Loading  (...) [0m

                                                                                

[0;30m [46m

[11 Sep 17 23:06:46 Initializing the profile database superclass] Creating (...) [0m

                                                                                

[0;30m [46m

[11 Sep 17 23:06:46 Initializing the profile database superclass] Setting  (...) [0m

                                                                                

[0;30m [46m

[11 Sep 17 23:06:46 Initializing the profile database superclass] Accessin (...) [0m

                                                                                

Auxiliary Data ...............................: Found: AUXILIARY-DATA.h5 (v. 1)

Profile Super ................................: Initialized with all 4784 splits: PROFILE.db (v. 20)

                                                                                

[0;30m [46m

[11 Sep 17 23:06:48 Loading contig sequences] Identifying contigs shorter  (...) [0m

                                                                                

[0;30m [46m

[11 Sep 17 23:06:48 Loading contig sequences] Reading contig sequences          [0m

                                                                                

[0;30m [46m

[11 Sep 17 23:06:48 Loading contig sequences] Filtering out shorter contigs    [0m

                                                                                

                                                                                

[0;30m [46m

[11 Sep 17 23:06:48 Computing split sequences from contigs] Discarding spl (...) [0m

                                                                                

[0;30m [46m

[11 Sep 17 23:06:48 Computing split sequences from contigs] Generating spl (...) [0m

                                                                                

                                                                                

[0;30m [46m

[11 Sep 17 23:06:49 Pruning view dicts] ...                                    [0m

                                                                                

[0;30m [46m

[11 Sep 17 23:06:49 Pruning view dicts] processing view "variability"          [0m

                                                                                

[0;30m [46m

[11 Sep 17 23:06:49 Pruning view dicts] processing view "mean_coverage_Q2Q3"    [0m

                                                                                

[0;30m [46m

[11 Sep 17 23:06:49 Pruning view dicts] processing view "detection"            [0m

                                                                                

[0;30m [46m

[11 Sep 17 23:06:49 Pruning view dicts] processing view "relative_abundance"    [0m

                                                                                

[0;30m [46m

[11 Sep 17 23:06:49 Pruning view dicts] processing view "abundance"            [0m

                                                                                

[0;30m [46m

[11 Sep 17 23:06:49 Pruning view dicts] processing view "mean_coverage"        [0m

                                                                                

[0;30m [46m

[11 Sep 17 23:06:49 Pruning view dicts] processing view "std_coverage"          [0m

                                                                                

[0;30m [46m

[11 Sep 17 23:06:49 Pruning view dicts] processing view "max_normalized_ratio" [0m

                                                                                

 

And then nothing more is outputted. 

I've tried different options for ports:

anvi-interactive -p PROFILE.db -c CONTIGS.db --server-only -I 127.0.0.1 -P 8888

anvi-interactive -p PROFILE.db -c CONTIGS.db --server-only -I 0.0.0.0 -P 80808

And then when I ssh into my computing cluster through terminal, and load http://127.0.0.0:8888 (or other combinations localhost, 0.0.0.0 etc.. ) I get:

channel 4: open failed: connect failed: Connection refused

channel 3: open failed: connect failed: Connection refused

Is it SGE access problem? But in that case shouldn't I get some "errors" in my log file?

What do you think?

Best,

Robert.

[Özcan Esen]

Hi Robert,

Did you setup port forwarding (ssh tunnel) when you SSH into your cluster? There are some instructions here[1] if you didn't.

[1]: http://merenlab.org/2015/11/28/visualizing-from-a-server/

All the Best,
Ozcan

--
Anvi'o Paper: https://peerj.com/articles/1319/
Project Page: http://merenlab.org/projects/anvio/
Code Repository: https://github.com/meren/anvio
---
You received this message because you are subscribed to the Google Groups "Anvi'o" group.
To unsubscribe from this group and stop receiving emails from it, send an email to anvio+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/anvio/219daefe-0b55-4520-be85-2bc8d679a025%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Robert Kwapich]

Yes, of course. So for our server (let's say X) I type:

ssh 

ssh -L 8080:127.0.0.1:8080 kwapichr@X

when I run 

anvi-interactive -p PROFILE.db -c CONTIGS.db --server-only -I 127.0.0.1 -P 8080

I assume SGE is not configured to allow using port 8080, or the 127.0.0.1 address. I'll contact our admin. However, shouldn't anvio report an error when it's unable (due to lack of permission) to set-up a server?

I'll update post if I find something out.

To unsubscribe from this group and stop receiving emails from it, send an email to anvio+un...@googlegroups.com.

[Robert Kwapich]

Alright, I totally got this. Here is my solution how to run anvi-interactive or anvi-refine on SGE cluster. It is somewhat simple. First you need to open TWO terminal windows:

(1 terminal windows) 

1. First you login into your "master" server (let's say X), from which you usually submit .PBS scripts:

           ssh user@X

2. Then you run an interactive session request interactive session through qlogin (or qrsh) asking gently for some resources, let's say:

            qlogin -l h_vmem=35G,mem_free=45G -pe threaded 2

Here you have to notice the node number you were delegated to, ex. node052. However you have to know the full address of the node you are running anvio. You can check that with "qstat" command, in the QUEUE field.

So you could get  al...@node052.internal.server.com. Note that address (be sure it's not truncated due to formatting of qstat output)

3. Then you load anvio (If you have a conda installation) and run the analysis

                  source activate anvio240

                  anvi-interactive -p PROFILE.db -c CONTIGS.db  --server-only -P 8080 

Alright, done here. You switch back to second terminal window

(2 TERMINAL WINDOW)

you ssh into you server X however specifying port forwarding and putting specifically  the node address

ssh -L 8080:node052.internal.server.com:8080 X 

(where X is the server name or address)

Then you can open your browser and type localhost:8080. Voila! 

I hope this will help somebody.