Problems resuming a Phylosift run

[Zachary Stewart]

Hi guys,

I've recently been running Phylosift on a number of samples, though one was stopped after performing alignment of a chunk (#23) due to an issue with the high power computer cluster I was running this on. I've since tried to resume this by using the command...

"phylosift place --continue --threads 4 --output ./PS_temp/output_directory --paired file.1.fq file.2.fq"

...which to my understanding should work (?), but when I attempt this I receive an error as below.

"Can't call method "unlock" on an undefined value at /pkg/suse11/phylosift/1.0.1/bin../lib/Phylosift/Phylosift.pm line 187".

I am able to initiate a fresh run, so I was wondering if someone knew why I am receiving this error when I attempt to --continue this run. I noticed someone else has received this error previously (https://groups.google.com/forum/#!topic/phylosift/fQZgGZTfX94) though they received it at the end of a run where it was apparently not a problem. Can anyone lend a hand?

Thanks,
Zac.

[Guillaume Jospin]

Could you share a complete output using the --debug flag to know when the error occurs?

It is possibly though I haven't looked at the code in a while that using --continue skips some variable population which creates the undefined value later in the code execution.

--
You received this message because you are subscribed to the Google Groups "PhyloSift" group.
To unsubscribe from this group and stop receiving emails from it, send an email to phylosift+unsubscribe@googlegroups.com.
To post to this group, send email to phyl...@googlegroups.com.
Visit this group at https://groups.google.com/group/phylosift.
For more options, visit https://groups.google.com/d/optout.

[Zachary]

Thanks for looking into this, I'll need to recreate this issue and will update that here soon.

On Thursday, September 29, 2016 at 2:33:41 AM UTC+10, Guillaume Jospin wrote:

Could you share a complete output using the --debug flag to know when the error occurs?

It is possibly though I haven't looked at the code in a while that using --continue skips some variable population which creates the undefined value later in the code execution.

On Mon, Sep 26, 2016 at 9:11 PM, Zachary Stewart <mrhue...@gmail.com> wrote:

Hi guys,

I've recently been running Phylosift on a number of samples, though one was stopped after performing alignment of a chunk (#23) due to an issue with the high power computer cluster I was running this on. I've since tried to resume this by using the command...

"phylosift place --continue --threads 4 --output ./PS_temp/output_directory --paired file.1.fq file.2.fq"

...which to my understanding should work (?), but when I attempt this I receive an error as below.

"Can't call method "unlock" on an undefined value at /pkg/suse11/phylosift/1.0.1/bin../lib/Phylosift/Phylosift.pm line 187".

I am able to initiate a fresh run, so I was wondering if someone knew why I am receiving this error when I attempt to --continue this run. I noticed someone else has received this error previously (https://groups.google.com/forum/#!topic/phylosift/fQZgGZTfX94) though they received it at the end of a run where it was apparently not a problem. Can anyone lend a hand?

Thanks,
Zac.

--
You received this message because you are subscribed to the Google Groups "PhyloSift" group.

To unsubscribe from this group and stop receiving emails from it, send an email to phylosift+...@googlegroups.com.

[Zachary]

Okay, because starting the program with 'place --continue' results in immediate failure without much --debug output, I started a run at the search step with the following script (I substituted some of the file names etc. to generic names throughout this post):

"phylosift search --continue --debug --threads 4 --output ./output_dir --paired file.1.fq file.2.fq"

This produces the output file below...

READSFILE    file.1.fq
force : 0
PhyloSift -- Phylogenetic analysis of genomes and metagenomes
(c) 2011, 2012 Aaron Darling and Guillaume Jospin

CITATION:
        PhyloSift. A. E. Darling, G. Jospin, E. Lowe, F. A. Matsen, H. M. Bik, J. A. Eisen. Submitted to PeerJ


PhyloSift incorporates several other software packages, please consider also citing the following papers:


        pplacer: linear time maximum-likelihood and Bayesian phylogenetic placement of sequences onto a fixed reference tree.
        Frederick A Matsen, Robin B Kodner, and E Virginia Armbrust
        BMC Bioinformatics 2010, 11:538
       
        Adaptive seeds tame genomic sequence comparison.
        SM Kielbasa, R Wan, K Sato, P Horton, MC Frith
        Genome Research 2011.
       
        Infernal 1.0: Inference of RNA alignments
        E. P. Nawrocki, D. L. Kolbe, and S. R. Eddy
        Bioinformatics 25:1335-1337 (2009)
        
         Bowtie: Ultrafast and memory-efficient alignment of short DNA sequences to the human genome.
         Langmead B, Trapnell C, Pop M, Salzberg SL. Genome Biol 10:R25.
        
         HMMER 3.0 (March 2010); http://hmmer.org/
         Copyright (C) 2010 Howard Hughes Medical Institute.
         Freely distributed under the GNU General Public License (GPLv3).
        
         Phylogenetic Diversity within Seconds.
        Bui Quang Minh, Steffen Klaere and Arndt von Haeseler
        Syst Biol (2006) 55 (5): 769-773.
Before START 2016-09-30 17:38:25
All systems are good to go, continuing the screening
MODE : search
Using updated markers
Using a marker list file /pkg/suse11/phylosift/1.0.1/share/phylosift/markers/marker_list.txt
Before runBlast 2016-09-30 17:44:26
USING 738
Input type is dna, fastq
Making fifos
Launching search process 1
Running /pkg/suse11/phylosift/1.0.1/bin/lastal -F15 -e75 -f0 /pkg/suse11/phylosift/1.0.1/share/phylosift/markers/replast "/home/my_folder/output_dir/blastDir/last_0.pipe" |Launching search process 3
Running /pkg/suse11/phylosift/1.0.1/bin/lastal -F15 -e75 -f0 /pkg/suse11/phylosift/1.0.1/share/phylosift/markers/replast "/home/my_folder/output_dir/blastDir/last_2.pipe" |Launching search process 2
Running /pkg/suse11/phylosift/1.0.1/bin/lastal -F15 -e75 -f0 /pkg/suse11/phylosift/1.0.1/share/phylosift/markers/replast "/home/my_folder/output_dir/blastDir/last_1.pipe" |Launching search process 4
Running /pkg/suse11/phylosift/1.0.1/bin/lastal -F15 -e75 -f0 /pkg/suse11/phylosift/1.0.1/share/phylosift/markers/replast "/home/my_folder/output_dir/blastDir/last_3.pipe" |Launching search process 5
Running /pkg/suse11/phylosift/1.0.1/bin/lastal -e75 -Q2 -f0 /pkg/suse11/phylosift/1.0.1/share/phylosift/markers/rnadb "/home/my_folder/output_dir/blastDir/last_rna.pipe" |Writing candidates from process 3
ReadsFile:  file.1.fq
.lastal Got 0 markers with hits
.lastal Got 0 nucleotide markers with hits
Writing candidates from process 4
ReadsFile:  file.1.fq
.lastal Got 0 markers with hits
.lastal Got 0 nucleotide markers with hits
Writing candidates from process 2
ReadsFile:  file.1.fq
.lastal Got 0 markers with hits
.lastal Got 0 nucleotide markers with hits
Writing candidates from process 5
ReadsFile:  file.1.fq
.lastal.rna Got 0 markers with hits
.lastal.rna Got 0 nucleotide markers with hits
Writing candidates from process 1
ReadsFile:  file.1.fq
.lastal Got 0 markers with hits
.lastal Got 0 nucleotide markers with hits
Opening /home/my_folder/output_dir/blastDir/reads.fasta.1
Octopus is handing out sequences
Octopus handed out 1000000 sequences
Debug lvl : 1
After runBlast 2016-09-30 17:45:40
MODE :: align
Before Alignments 2016-09-30 17:45:40
After Alignments 2016-09-30 17:45:40
Before runPPlacer 2016-09-30 17:45:40
After runPPlacer 2016-09-30 17:45:40
Before taxonomy assignments 2016-09-30 17:45:40
After taxonomy assignments 2016-09-30 17:45:40

-----
PBS Job 1107451.pbs
CPU time  : 00:01:10
Wall time : 00:07:16
Mem usage : 44092kb

____________________

...and an error file below

Phylosoft (v1.0.1) module loaded.
Located in /pkg/suse11/phylosift/1.0.1
Can't call method "unlock" on an undefined value at /pkg/suse11/phylosift/1.0.1/bin/../lib/Phylosift/Phylosift.pm line 187.
____________________

Finally, this is the last few lines of the run_info.txt file

Chunk 29 sequences processed 22a841cff3a1975b2cccbb7872588b3d

Chunk 29 Search completed

Chunk 29 Align completed

Chunk 29 Place completed

Chunk 29 Summarize completed

Chunk 30 sequences processed d3393615d5188505feb5a5c8671a048d

Chunk 30 Search completed

Chunk 30 Align completed

Program run as:

phylosift place --continue --debug --threads 4 --output ./output_dir --paired  file.1.fq file.2.fq

Marker database version:

http://edhar.genomecenter.ucdavis.edu/~koadman/phylosift_markers/markers.tgz

1413946442

____________________

I can't think of much else that might be of relevance. I've tested smaller runs where I've cancelled Phylosift after completing different stages (i.e., after searching, alignment, and placement) which doesn't seem to matter, I still receive the 'unlock' error. I thought it might have to do with having left over files from when a stage was cancelled mid-way, but deleting these files then restarting seems to make things worse. At this point I've simply opted to re-run my Phylosift jobs. If you can see a solution to this problem easily that would be appreciated as its possible these jobs won't fully complete again and knowing how to continue could be of use (they take a really, really long time as the datasets are huge). If trouble shooting this would be difficult though then don't stress, it seems like this error is very rare and ideally my runs should complete without fault this time around!

[Zachary]

Update on this situation, my second attempt to run this on two of my samples has also crashed. The first time this crash was caused by not allocating enough RAM with my PBS job submission script, now it appears as though my job is crashing with the same "unlock" error after I processed 80-95 hours of these samples. The start of the error file for both of these are as such:

Phylosoft (v1.0.1) module loaded.

Located in /pkg/suse11/phylosift/1.0.1

Unable to read from /pkg/suse11/phylosift/1.0.1/share/phylosift/markers/18s_reps/18s_reps.cm at /pkg/suse11/phylosift/1.0.1/bin/../lib/Phylosift/Utilities.pm line 85
	Phylosift::Utilities::ps_open('/pkg/suse11/phylosift/1.0.1/share/phylosift/markers/18s_reps/...') called at /pkg/suse11/phylosift/1.0.1/bin/../lib/Phylosift/Utilities.pm line 840
	Phylosift::Utilities::get_marker_length('self', 'Phylosift::Phylosift=HASH(0x2649f80)', 'marker', '18s_reps') called at /pkg/suse11/phylosift/1.0.1/bin/../lib/Phylosift/FastSearch.pm line 986
	Phylosift::FastSearch::write_candidates('self', 'Phylosift::Phylosift=HASH(0x2649f80)', 'hitsref', 'HASH(0x232fbd8)', 'searchtype', '.lastal.rna', 'reads', '/home/my_folder/output_dir/bla...', 'process_id', ...) called at /pkg/suse11/phylosift/1.0.1/bin/../lib/Phylosift/FastSearch.pm line 319
	Phylosift::FastSearch::launch_searches('self', 'Phylosift::Phylosift=HASH(0x2649f80)', 'readtype', 'HASH(0x120a168)', 'dir', '/home/my_folder/output_dir/bla...', 'contigs', 0, 'chunk', ...) called at /pkg/suse11/phylosift/1.0.1/bin/../lib/Phylosift/FastSearch.pm line 136
	Phylosift::FastSearch::run_search('self', 'Phylosift::Phylosift=HASH(0x2649f80)', 'marker_reference', 'ARRAY(0x24e49f0)') called at /pkg/suse11/phylosift/1.0.1/bin/../lib/Phylosift/Phylosift.pm line 418
	Phylosift::Phylosift::run_search('self', 'Phylosift::Phylosift=HASH(0x2649f80)', 'cont', 0, 'marker', 'ARRAY(0x24e49f0)') called at /pkg/suse11/phylosift/1.0.1/bin/../lib/Phylosift/Phylosift.pm line 175
	Phylosift::Phylosift::run('Phylosift::Phylosift=HASH(0x2649f80)', 'force', undef, 'custom', undef, 'cont', undef) called at /pkg/suse11/phylosift/1.0.1/bin/../lib/Phylosift/Command/all.pm line 129
	Phylosift::Command::all::execute('Phylosift::Command::all=HASH(0x2661f18)', 'Getopt::Long::Descriptive::Opts::__OPT__::2=HASH(0x272d938)', 'ARRAY(0xbfa678)') called at /pkg/suse11/phylosift/1.0.1/bin/../lib/App/Cmd.pm line 243
	App::Cmd::execute_command('Phylosift=HASH(0x2638370)', 'Phylosift::Command::all=HASH(0x2661f18)', 'Getopt::Long::Descriptive::Opts::__OPT__::2=HASH(0x272d938)', 'file.1.fq', 'file.2.fq') called at /pkg/suse11/phylosift/1.0.1/bin/../lib/App/Cmd.pm line 171
	App::Cmd::run('Phylosift') called at /pkg/suse11/phylosift/1.0.1/bin/phylosift line 7

---and---

Phylosoft (v1.0.1) module loaded.

Located in /pkg/suse11/phylosift/1.0.1

Unable to read from /pkg/suse11/phylosift/1.0.1/share/phylosift/markers/16s_reps_bac/16s_reps_bac.cm at /pkg/suse11/phylosift/1.0.1/bin/../lib/Phylosift/Utilities.pm line 85
	Phylosift::Utilities::ps_open('/pkg/suse11/phylosift/1.0.1/share/phylosift/markers/16s_reps_...') called at /pkg/suse11/phylosift/1.0.1/bin/../lib/Phylosift/Utilities.pm line 840
	Phylosift::Utilities::get_marker_length('self', 'Phylosift::Phylosift=HASH(0x2649f80)', 'marker', '16s_reps_bac') called at /pkg/suse11/phylosift/1.0.1/bin/../lib/Phylosift/FastSearch.pm line 986
	Phylosift::FastSearch::write_candidates('self', 'Phylosift::Phylosift=HASH(0x2649f80)', 'hitsref', 'HASH(0x232fbd8)', 'searchtype', '.lastal.rna', 'reads', '/home/my_folder/output_dir...', 'process_id', ...) called at /pkg/suse11/phylosift/1.0.1/bin/../lib/Phylosift/FastSearch.pm line 319
	Phylosift::FastSearch::launch_searches('self', 'Phylosift::Phylosift=HASH(0x2649f80)', 'readtype', 'HASH(0x120a168)', 'dir', '/home/my_folder/output_dir...', 'contigs', 0, 'chunk', ...) called at /pkg/suse11/phylosift/1.0.1/bin/../lib/Phylosift/FastSearch.pm line 136
	Phylosift::FastSearch::run_search('self', 'Phylosift::Phylosift=HASH(0x2649f80)', 'marker_reference', 'ARRAY(0x24e49f0)') called at /pkg/suse11/phylosift/1.0.1/bin/../lib/Phylosift/Phylosift.pm line 418
	Phylosift::Phylosift::run_search('self', 'Phylosift::Phylosift=HASH(0x2649f80)', 'cont', 0, 'marker', 'ARRAY(0x24e49f0)') called at /pkg/suse11/phylosift/1.0.1/bin/../lib/Phylosift/Phylosift.pm line 175
	Phylosift::Phylosift::run('Phylosift::Phylosift=HASH(0x2649f80)', 'force', undef, 'custom', undef, 'cont', undef) called at /pkg/suse11/phylosift/1.0.1/bin/../lib/Phylosift/Command/all.pm line 129
	Phylosift::Command::all::execute('Phylosift::Command::all=HASH(0x2661f18)', 'Getopt::Long::Descriptive::Opts::__OPT__::2=HASH(0x272d938)', 'ARRAY(0xbfa678)') called at /pkg/suse11/phylosift/1.0.1/bin/../lib/App/Cmd.pm line 243
	App::Cmd::execute_command('Phylosift=HASH(0x2638370)', 'Phylosift::Command::all=HASH(0x2661f18)', 'Getopt::Long::Descriptive::Opts::__OPT__::2=HASH(0x272d938)', 'file.1.fq', 'file.2.fq') called at /pkg/suse11/phylosift/1.0.1/bin/../lib/App/Cmd.pm line 171
	App::Cmd::run('Phylosift') called at /pkg/suse11/phylosift/1.0.1/bin/phylosift line 7

_________________________

The tail of these two are as such:

Unable to read from /pkg/suse11/phylosift/1.0.1/share/phylosift/markers/18s_reps/18s_reps.cm at /pkg/suse11/phylosift/1.0.1/bin/../lib/Phylosift/Utilities.pm line 85
	Phylosift::Utilities::ps_open('/pkg/suse11/phylosift/1.0.1/share/phylosift/markers/18s_reps/...') called at /pkg/suse11/phylosift/1.0.1/bin/../lib/Phylosift/Utilities.pm line 840
	Phylosift::Utilities::get_marker_length('self', 'Phylosift::Phylosift=HASH(0x2649f80)', 'marker', '18s_reps') called at /pkg/suse11/phylosift/1.0.1/bin/../lib/Phylosift/FastSearch.pm line 986
	Phylosift::FastSearch::write_candidates('self', 'Phylosift::Phylosift=HASH(0x2649f80)', 'hitsref', 'HASH(0x232fbd8)', 'searchtype', '.lastal.rna', 'reads', '/home/my_folder/output_dir/bla...', 'process_id', ...) called at /pkg/suse11/phylosift/1.0.1/bin/../lib/Phylosift/FastSearch.pm line 319
	Phylosift::FastSearch::launch_searches('self', 'Phylosift::Phylosift=HASH(0x2649f80)', 'readtype', 'HASH(0x120a168)', 'dir', '/home/my_folder/output_dir/bla...', 'contigs', 0, 'chunk', ...) called at /pkg/suse11/phylosift/1.0.1/bin/../lib/Phylosift/FastSearch.pm line 136
	Phylosift::FastSearch::run_search('self', 'Phylosift::Phylosift=HASH(0x2649f80)', 'marker_reference', 'ARRAY(0x24e49f0)') called at /pkg/suse11/phylosift/1.0.1/bin/../lib/Phylosift/Phylosift.pm line 418
	Phylosift::Phylosift::run_search('self', 'Phylosift::Phylosift=HASH(0x2649f80)', 'cont', 0, 'marker', 'ARRAY(0x24e49f0)') called at /pkg/suse11/phylosift/1.0.1/bin/../lib/Phylosift/Phylosift.pm line 175
	Phylosift::Phylosift::run('Phylosift::Phylosift=HASH(0x2649f80)', 'force', undef, 'custom', undef, 'cont', undef) called at /pkg/suse11/phylosift/1.0.1/bin/../lib/Phylosift/Command/all.pm line 129
	Phylosift::Command::all::execute('Phylosift::Command::all=HASH(0x2661f18)', 'Getopt::Long::Descriptive::Opts::__OPT__::2=HASH(0x272d938)', 'ARRAY(0xbfa678)') called at /pkg/suse11/phylosift/1.0.1/bin/../lib/App/Cmd.pm line 243
	App::Cmd::execute_command('Phylosift=HASH(0x2638370)', 'Phylosift::Command::all=HASH(0x2661f18)', 'Getopt::Long::Descriptive::Opts::__OPT__::2=HASH(0x272d938)', 'file.1.fq', 'file.2.fq') called at /pkg/suse11/phylosift/1.0.1/bin/../lib/App/Cmd.pm line 171
	App::Cmd::run('Phylosift') called at /pkg/suse11/phylosift/1.0.1/bin/phylosift line 7

Can't call method "unlock" on an undefined value at /pkg/suse11/phylosift/1.0.1/bin/../lib/Phylosift/Phylosift.pm line 187.

---and---

Unable to read from /pkg/suse11/phylosift/1.0.1/share/phylosift/markers/18s_reps/18s_reps.cm at /pkg/suse11/phylosift/1.0.1/bin/../lib/Phylosift/Utilities.pm line 85
	Phylosift::Utilities::ps_open('/pkg/suse11/phylosift/1.0.1/share/phylosift/markers/18s_reps/...') called at /pkg/suse11/phylosift/1.0.1/bin/../lib/Phylosift/Utilities.pm line 840
	Phylosift::Utilities::get_marker_length('self', 'Phylosift::Phylosift=HASH(0x2649f80)', 'marker', '18s_reps') called at /pkg/suse11/phylosift/1.0.1/bin/../lib/Phylosift/FastSearch.pm line 986
	Phylosift::FastSearch::write_candidates('self', 'Phylosift::Phylosift=HASH(0x2649f80)', 'hitsref', 'HASH(0x232fbd8)', 'searchtype', '.lastal.rna', 'reads', '/home/my_folder/output_dir...', 'process_id', ...) called at /pkg/suse11/phylosift/1.0.1/bin/../lib/Phylosift/FastSearch.pm line 319
	Phylosift::FastSearch::launch_searches('self', 'Phylosift::Phylosift=HASH(0x2649f80)', 'readtype', 'HASH(0x120a168)', 'dir', '/home/my_folder/output_dir...', 'contigs', 0, 'chunk', ...) called at /pkg/suse11/phylosift/1.0.1/bin/../lib/Phylosift/FastSearch.pm line 136
	Phylosift::FastSearch::run_search('self', 'Phylosift::Phylosift=HASH(0x2649f80)', 'marker_reference', 'ARRAY(0x24e49f0)') called at /pkg/suse11/phylosift/1.0.1/bin/../lib/Phylosift/Phylosift.pm line 418
	Phylosift::Phylosift::run_search('self', 'Phylosift::Phylosift=HASH(0x2649f80)', 'cont', 0, 'marker', 'ARRAY(0x24e49f0)') called at /pkg/suse11/phylosift/1.0.1/bin/../lib/Phylosift/Phylosift.pm line 175
	Phylosift::Phylosift::run('Phylosift::Phylosift=HASH(0x2649f80)', 'force', undef, 'custom', undef, 'cont', undef) called at /pkg/suse11/phylosift/1.0.1/bin/../lib/Phylosift/Command/all.pm line 129
	Phylosift::Command::all::execute('Phylosift::Command::all=HASH(0x2661f18)', 'Getopt::Long::Descriptive::Opts::__OPT__::2=HASH(0x272d938)', 'ARRAY(0xbfa678)') called at /pkg/suse11/phylosift/1.0.1/bin/../lib/App/Cmd.pm line 243
	App::Cmd::execute_command('Phylosift=HASH(0x2638370)', 'Phylosift::Command::all=HASH(0x2661f18)', 'Getopt::Long::Descriptive::Opts::__OPT__::2=HASH(0x272d938)', 'file.1.fq', 'file.2.fq') called at /pkg/suse11/phylosift/1.0.1/bin/../lib/App/Cmd.pm line 171
	App::Cmd::run('Phylosift') called at /pkg/suse11/phylosift/1.0.1/bin/phylosift line 7
Uncaught exception: Failure("Sequence 925132.2.1.99 doesn't overlap any reference sequences.")
Fatal error: exception Failure("Sequence 925132.2.1.99 doesn't overlap any reference sequences.")
Uncaught exception: Failure("Sequence 301650.1.2.88 doesn't overlap any reference sequences.")
Fatal error: exception Failure("Sequence 301650.1.2.88 doesn't overlap any reference sequences.")

Can't call method "unlock" on an undefined value at /pkg/suse11/phylosift/1.0.1/bin/../lib/Phylosift/Phylosift.pm line 187.

______________________

I did not run these with the --debug tag so I don't have detailed output logs. I will need to contact my computer cluster's support to see if this is something they might understand. For now it seems like Phylosift might not be capable of running these jobs, as many of them are expected to take upwards of 800 hours and I cannot be certain they won't crash at any moment. If you can help with figuring this out still that would help as I think Phylosift provides the best output for my study.

Thanks,
Zac.

[Zachary]

Have been reading more into this issue, it seems like this issue might be something to do with lock files that are being employed by the computing cluster to allow for multiple jobs to be run on the same Phylosift program files. I don't understand why the runs I am doing are conflicting with one another, as the lock file system should allow for many jobs to run without incident, but unless you guys see something that conflicts with this interpretation I will need to continue this troubleshooting with the computing cluster.

Thanks,
Zac.

[Guillaume Jospin]

Hi Zach,

The following error means that PS can't find the particular file listed OR the permissions are not set properly.

Can you check if it can be read at least by everybody? I think the parent directories also need to be accessible.

Looking at my own database, I am noticing that some of the files in the markers for 16s_reps_bac, 16s_reps_arc and 18s_reps do not have the readable bit set for everybody. 

chmod 644 <filename>

should change that.

I never ran into that problem because I was always the one downloading the database for my account.

I hope this helps. 

Please let us know if this fixes things.

Unable to read from /pkg/suse11/phylosift/1.0.1/share/phylosift/markers/18s_reps/18s_reps.cm

To unsubscribe from this group and stop receiving emails from it, send an email to phylosift+unsubscribe@googlegroups.com.

[Guillaume Jospin]

Can you look in your phylosift database directory share/phylosift and see if there are lock files in there.

What I suspect might be happening is in the event of a crash, the lock file isn't released and when you come back it then tries to get a new lock and is denied.

I would recommend removing those files and trying the continued run.

I also would suggest that you just use the all command in addition to the continue flag. PS should scan your run_info file and decide where to pick up the screening on its own.

I think that using continue with an intermediate step screws up the normal operation because it'll think it's supposed to only do that one step.

To unsubscribe from this group and stop receiving emails from it, send an email to phylosift+unsubscribe@googlegroups.com.

[Zachary]

Hi Guillaume,

It appears as though it was a permissions issue. The support staff of the computing cluster didn't say if they called chmod 644 to resolve this issue, I was just advised that adjustments had been made so the shared marker database can be read by all users which is allowing me to --continue my runs now. Thanks for looking into this issue for me, it was much appreciated.

Zac.

[Zachary]

Guess I wouldn't be so lucky. I'm still having issues with a "Can't call method "unlock" on an undefined value at /pkg/suse11/phylosift/1.0.1/bin../lib/Phylosift/Phylosift.pm line 187" error, but the output log files aren't giving me any strong hint as to what the root cause is other than having something to do with the lock files (based on line 187). At this stage it seems as though there is something wrong with how it has been installed on the computing cluster. I will provide an update with regards to whether this gets fixed or not.

I also recently noticed that more RAM and more CPU power is needed for each successive chunk. As my largest file has approx. 273 chunks to process I think it might use an excessive (but unknown) amount of these resources while also taking upwards of 900 hours to run. If these numbers don't seem wrong to you, then perhaps Phylosift won't be cut out to analyse my data?

Zac.

[Guillaume Jospin]

That is quite a lot of data and I definitely think that PS might be outmatched against that. 

I just ran a bit over 10M reads (7 chunks of 1.5M pairs of reads ish) and it took 88 hours. That's rather slow when comparing to tools like kraken or kaiju. Unfortunately we are lacking resources to improve on the current state of things. 

What I might consider if I were you though is to randomly subsample your reads to a reasonable amount to at least get an idea that if what the results might look like and see if it would even be worth the effort to run everything. 

Sent from my iPhone