Error runing ParallelStructure

[prs...@gmail.com]

Good morning,

Can someone help me.

I am doing some trial runs with ParallelStructure, but so far with no sucess.

I have used my data file in my PC, so I now it can be read and use by Structure.

The error I get is this:
/projects/ps-ngbt/home/cipres/ngbw/contrib/scripts/workbench.rc: line 18: ulimit: file size: cannot modify limit: Operation not permitted
Unloading compiler-dependent module mvapich2_ib/2.1
Unloading compiler-dependent module openmpi_ib/1.8.4
Unloading compiler-dependent module openmpi_ib/1.8.4
Loading required package: parallel
Error in file(file, "rt") : cannot open the connection
Calls: parallel_structure -> plot_str -> as.matrix -> read.table -> file
In addition: Warning messages:
1: In read.table(joblist, colClasses = "character") :
  incomplete final line found by readTableHeader on 'joblist.txt'
2: In file(file, "rt") :
  cannot open file './results_job_T1_0_1_q': No such file or directory
Execution halted
srun: error: comet-03-12: task 0: Exited with exit code 1


Is it a problem with my run or with ParallelStructure?

Thanks!

Pedro

[Mark Miller]

Hi Pedro,

Thanks for reporting that. It sounds like there is something in your input file PS does not like, but the error messaging in the code is fairly generic and not super informative.

The code is working when everything is perfect.

I am still learning the structure of PS input myself, so I don't know exactly wat this error mean yet, but solving it is key.:

1: In read.table(joblist, colClasses = "character") :
  incomplete final line found by readTableHeader on 'joblist.txt'

So it sounds like the format of the joblist.txt is slightly off, or maybe it has windows line endings, or something like that.

What I would do is compare the format of joblist with the format in the manual, and see if it is something obvious to you,

and if not, send the _jobinfo.txt file to me for this job and I will see what I can learn.

Best,

Mark

[prs...@gmail.com]

Thanks Mark.

Structure is very new to me.

But when running it in a single core, my data file is read without problems and the software runs.


As for the joblist, as far I can tell there is no extra character there (I had considered that).

What I am not sure is if I need to have populations defined a priori to run ParallelStructure. This is not required for Structure.

I have made this question in the Structure group. If I have a reply I will update it here.

Pedro

[Mark Miller]

Thanks Pedro,

If Structure accepts your input file fine, then I expect the issue to be in the joblist file, which the error message also suggests.

The person responsible for ParallelStructure is Francois Besnier.

I have found him to be quite helpful, so you might run the question past him as well, since he will be the expert on the joblist file.

Mark

[Nathan Johnson]

Pedro, Mark, and others -

Did anyone find a solution to this problem?  I'm receiving the same error message (see below) when running ParallelStructure on Cipres. Here is what I have in my joblist.txt file:

T1 1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17 15 100 1000

Any help will be greatly appreciated!

- Nate

Unloading compiler-dependent module mvapich2_ib/2.1
Loading required package: parallel
Error in file(file, "rt") : cannot open the connection
Calls: parallel_structure -> plot_str -> as.matrix -> read.table -> file
In addition: Warning messages:
1: In read.table(joblist, colClasses = "character") :
  incomplete final line found by readTableHeader on 'joblist.txt'
2: In file(file, "rt") :
  cannot open file './results_job_T1_q': No such file or directory
Execution halted
srun: error: comet-05-36: task 0: Exited with exit code 1

[Mark Miller]

Hi Nathan

The way the PS code is organized, there aren't explicit error messages for a number of types of problems.

When these problems are encountered, the program aborts before it gets to creating those files, and that is the "default" error message, if you will.

So there a number of possible problems with the input file that will produce this exact error message.

Please see if there is anything you can detect in your input file that might be amiss, or try the dataset with Structure locally and see if it throws a more helpful error message.

If you are stumped, please send me the jobinfo.txt file for the job, and I will see if I can help...

Mark

[Nathan Johnson]

Hi Mark -

Many thanks for the response and offer to help.  I've successfully completed a Structure run on my machine using the input file but no luck using ParallelStructure on Cipres.  I've also tried several different jobinfo files.  Attached is the simplest one I've tried.  I have 17 populations in my snp dataset and the data are organized on a single row. These options were selected correctly using the Cipres parameters interface and Structure on my machine.  Please let me know if you need additional details about my dataset or want to see the input file.

Gratefully,

- Nate 

[Mark Miller]

Hi Nate,

I am sorry for the very slow response time. I spent a while on this with no real success.
I believe presently that somehow the tool broke, or breaks intermittently on our side.

I am still working on it and will let you know when it is working again....

Mark

[Mark Miller]

Ok, it took me forever to figure this out, but it seems the option -printqhat is critical to the success of the program. So far this is consistent for me. I will hard code this so it cant be turned off.

[Jim Cohen]

Hi Mark-

Good morning, how are you?  I was wondering if the above problem had been solved yet, and, perhaps, if it may have re-emerged?  I am getting the same error when I'm running parallelstructure (below is the stderr, stdout, and term output).  I'm hoping that the issue is with me and the manner in which my files are formatted, but I'm not quite sure.  Any help would be greatly appreciated.  Thank you.

Jim Cohen

Unloading compiler-dependent module mvapich2_ib/2.1
Loading required package: parallel
Error in file(file, "rt") : cannot open the connection
Calls: parallel_structure -> plot_str -> as.matrix -> read.table -> file
In addition: Warning message:
In file(file, "rt") :
  cannot open file './results_job_T1_q': No such file or directory
Execution halted
srun: error: comet-07-62: task 0: Exited with exit code 1
srun: error: _server_read: fd 16 n 0 got error or unexpected eof reading header: Resource temporarily unavailable
srun: error: step_launch_notify_io_failure: aborting, io error with slurmstepd on node 0

[1] "starting work at 2017-10-11 04:14:01"
[1] "job T1 was calculated  at 2017-10-11 04:14:03"
[1] "job T2 was calculated  at 2017-10-11 04:14:05"
[1] "job T3 was calculated  at 2017-10-11 04:14:07"
[1] "job T4 was calculated  at 2017-10-11 04:14:09"
[1] "job T5 was calculated  at 2017-10-11 04:14:11"
[1] "job T6 was calculated  at 2017-10-11 04:14:13"
[1] "job T7 was calculated  at 2017-10-11 04:14:15"
[1] "job T8 was calculated  at 2017-10-11 04:14:17"
[1] "was calculated  at 2017-10-11 04:14:17"

1507720736 Wed Oct 11 04:18:56 PDT 2017
Wed Oct 11 04:18:56 2017
             JOBID PARTITION     NAME     USER    STATE       TIME TIME_LIMI  NODES NODELIST(REASON)
          11706209    shared NGBW-JOB   cipres  RUNNING       3:16 7-00:00:00      1 comet-07-65
This file created by srun of: ./_batch_command.cmdline

[Mark Miller]

Hi Jim,

I am well, thanks. And thanks for reporting this. As far as I can tell, the tool is generically working, but what you are seeing is (unfortunately) what most job failures in PS report, i.e. not super descriptive.
There is almost certainly some minor or major :) formatting issue with your input file.
If you have done what you can with it, you can send me the _jobinfo.txt file for this run.

I will see what I can learn.

Mark

[João Vieira]

Hi Mark,

I'm getting the same error. I can confirm that the analysis run flawless in my computer, so probably the error isn't due an error in my files. Please find attached my matrix, _jobinfo.txt and the joblist file.

Thanks for your help

[Mark Miller]

Dear Joao,

I looked through my notes, and some folks who had this issue did not have a population column.

Besnier states that a population column in necessary to run parallel structure, a constraint that does not exist when running Structure alone.

It looks to me like your data set has this feature, however. I also not that the third column in your dataset has non-integer values. Perhaps this is not allowed in ParallelStructure?

It might be efficient to contact Francois Besnier (the codes creator) directly. I have found him to be quite helpful.

Mark

[João Vieira]

Thanks for your answer. In fact, this data set run with no erros in ParallelStructure when running in my slow linux machine. May you please point me to a exemple data set already vefied running in ParallelStructure via CIPRES?

Thanks in advance

[Mark Miller]

Oh I see. I did not realize you were already running ParallelStructure locally. That is quite different. I am retesting my data set here, and will send it to you shortly. I agree it is worrying that the same data fails on different installations of the same code. Mark

[João Vieira]

Thanks in advance. If it helps to identify the problem, this is the command I'm running locally:

parallel_structure(joblist="joblist_test.txt", n_cpu=4, structure_path="Structure/bin/", infile="project_data", outpath="0_results/", numinds=128, numloci=90, plot_output=0, label=1, popdata=1, popflag=0, phenotypes=0, markernames=1, mapdist=0, onerowperind=1, phaseinfo=0, extracol=1, missing=-9, ploidy=1, usepopinfo=0, revert_convert=1, printqhat=1, locdata=1, recessivealleles=0, phased=1, noadmix=0, linkage=0, locprior=1, inferalpha=1, alpha=1, popalphas=1, unifprioralpha = 1, alphamax = 10, alphapropsd = 0.025, freqscorr = 1, onefst = 0, fpriormean = 0.01, fpriorsd = 0.05, inferlambda = 1, lambda = 1, computeprob = 1,pfromflagonly = 0, ancestdist = 0, startatpopinfo = 0, metrofreq = 10)

[Mark Miller]

Thanks, I will test that. Also, attached are some files that work here.

[Mark Miller]

I have the interface just about ready so you can urn this command line.. But I am curious, the documentation says phased is for when linkage=1. But you used it with linkage=0. I am not sure if that is right. Can you comment? Also, I cant find documentation for the revert_convert=1 command. Do you have a doc for that?

Mark

[Mark Miller]

The new interface will be in production tomorrow. Please let me know if you require the revert_convert command.

Best,

Mark

[Mark Miller]

I added the revert_convert option, but according to the manual it can only be used when usepopinfo=1.

[Rodrigo Ignacio Marín Nahuelpi]

Hi guys. Got the same issue...

[1] "starting work at 2024-01-23 17:34:19.369878" [1] "job T1 was calculated at 2024-01-23 17:34:36.10918" [1] "job T2 was calculated at 2024-01-23 17:34:52.259003" [1] "job T3 was calculated at 2024-01-23 17:35:07.983892" [1] "was calculated at 2024-01-23 17:35:07.98459" Error in file(file, "rt") : no se puede abrir la conexión Además: Warning message: In file(file, "rt") : no fue posible abrir el archivo 'results/results_job_T1_q': No such file or directory
This is the joblist.

T1 1,2,3 1 50000 250000
T2 1,2,3 2 50000 250000
T3 1,2,3 3 50000 250000

:( any ideas?

[Mark Miller]

Hi there,

Can you provide the username on the account. I do not see any user associated with your email address.

Mark

[Rodrigo Ignacio Marín Nahuelpi]

Hi Mark. I just wrote to ask for access. My lab email es rodrig...@ug.uchile.cl (btw, best regards from Chile!).

I've just noticed that, even though the program was running, this error appeared because I miswritte the loci number. Now the software run OK !

So hope this helps for the other colleagues. Best regards.

[Rodrigo Ignacio Marín Nahuelpi]

Hi Mark. I just asked for access and it was granted. My email es rodrig...@ug.uchile.cl (btw, best regards from Chile!).

I've just noticed that, even though the program was running, this error appeared because I miswrote the loci number. Now the package run OK !

So hope this helps for the other colleagues. Best regards.

El martes, 23 de enero de 2024 a las 17:54:57 UTC-3, mmi...@sdsc.edu escribió:

[Mark Miller]

Hi Rodrigo,

Thanks very much for letting me know.

The parallelstructure code has pretty weak error messaging, so every error produces the same output.

I am glad you were able to manage the solution.

Mark