Does the 3D-DNA pipeline has an option to set tmp dir?
Juan Cerda
Moshe Olshansky
Olga Dudchenko
Juan Cerda
export _JAVA_OPTIONS=-Djava.io.tmpdir=/path/to/tmp/
As for the parallel, I do have parallel installed an on the path. Parallel is being picked by the earlier sub-scripts, but not by the vizualiser and seems like it prevets the program from continuing as well. Can you expand on the env variable set up for the GNU Parallel path? Oddly enough, I feel like everytime I run the program, I get a different error even if run exactly the same. What gives?
Here is some of the log file, not all of it though
/data/users/juan/Hi-C-Assembly/opt/juicer/work/MC3413_Hi-C/3d-dna/run-asm-pipeline.sh /scratch/users/juan/Hi-C-Assembly/opt/juicer/references/Cuscuta_gronovii.hifiasm.purged.cleaned.fasta /scratch/users/juan/Hi-C-Assembly/opt/juicer/work/MC3413_Hi-C/aligned/merged_nodups.txt
version: 180922
###############
Starting iterating scaffolding with editing:
...starting round 0 of scaffolding:
:) -p flag was triggered. Running LIGer with GNU Parallel support parameter set to true.
:) -s flag was triggered, starting calculations with 15000 threshold starting contig/scaffold size
:) -q flag was triggered, starting calculations with 1 threshold mapping quality
...Using cprops file: Cuscuta_gronovii.hifiasm.purged.cleaned.0.cprops
...Using merged_nodups file: Cuscuta_gronovii.hifiasm.purged.cleaned.mnd.0.txt
...Scaffolding all scaffolds and contigs greater or equal to 15000 bp.
...Starting iteration # 1
...Starting iteration # 2
...Starting iteration # 3
...Starting iteration # 4
...Starting iteration # 5
...Starting iteration # 6
...Starting iteration # 7
...Starting iteration # 8
...Starting iteration # 9
...Starting iteration # 10
...Starting iteration # 11
...Starting iteration # 12
...Starting iteration # 13
...Starting iteration # 14
...Starting iteration # 15
...Starting iteration # 16
...Starting iteration # 17
...Starting iteration # 18
...Starting iteration # 19
...Starting iteration # 20
...Starting iteration # 21
...Starting iteration # 22
...Starting iteration # 23
...Starting iteration # 24
...Starting iteration # 25
...Starting iteration # 26
...Starting iteration # 27
:) DONE!
...visualizing round 0 results:
:) -p flag was triggered. Running with GNU Parallel support parameter set to true.
:) -q flag was triggered, starting calculations for 1 threshold mapping quality
:) -i flag was triggered, building mapq without
:) -c flag was triggered, will remove temporary files after completion
...Remapping contact data from the original contig set to assembly
...Building track files
...Building the hic file
Picked up _JAVA_OPTIONS: -Djava.io.tmpdir=/scratch/users/juan/Hi-C-Assembly/opt/juicer/work/MC3413_Hi-C/tmp/
Not including fragment map
Start preprocess
Writing header
Writing body
..
Writing footer
Finished preprocess
HiC file version: 8
Calculating norms for zoom BP_2500000
Calculating norms for zoom BP_1000000
Calculating norms for zoom BP_500000
Calculating norms for zoom BP_250000
Calculating norms for zoom BP_100000
Calculating norms for zoom BP_50000
Calculating norms for zoom BP_25000
Calculating norms for zoom BP_10000
Calculating norms for zoom BP_5000
Calculating norms for zoom BP_1000
Writing expected
Writing norms
Finished writing norms
...detecting misjoins in round 0 assembly:
:) -p flag was triggered. Running with GNU Parallel support parameter set to true.
:) -c flag was triggered, starting calculations with 5% saturation level
:) -w flag was triggered, performing cursory search for mismatches at 25000 resolution
:) -d flag was triggered, depletion score will be averaged across a region bounded by 125000 superdiagonal
:) -k flag was triggered, starting calculations with 55% depletion as mismatch threshold
:) -n flag was triggered, performing mismatch region thinning at 1000 resolution
...Dumping 25000 resolution matrix
Picked up _JAVA_OPTIONS: -Djava.io.tmpdir=/scratch/users/juan/Hi-C-Assembly/opt/juicer/work/MC3413_Hi-C/tmp/
HiC file version: 8
...Estimating necessary saturation level for requested misassembly sensitivity
...Coarse resolution saturation level = 5.31539
...Analyzing near-diagonal mismatches
...Filtering mismatch calls without better alternative: Not functional yet, skipping
...Thinning mismatch region boundaries.
...Dumping 1000 resolution matrix
Picked up _JAVA_OPTIONS: -Djava.io.tmpdir=/scratch/users/juan/Hi-C-Assembly/opt/juicer/work/MC3413_Hi-C/tmp/
HiC file version: 8
...Estimating necessary saturation level for requested misassembly sensitivity
...Fine resolution saturation level = 2
...Analyzing near-diagonal mismatches
:) -w flag was triggered, performing cursory search for repeat at 25000 resolution
:) -t flag was triggered, flagging regions with coverage higher than 2
...Dumping 25000 resolution coverage track
Picked up _JAVA_OPTIONS: -Djava.io.tmpdir=/scratch/users/juan/Hi-C-Assembly/opt/juicer/work/MC3413_Hi-C/tmp/
HiC file version: 8
...-p flag was triggered. Running with GNU Parallel support parameter set to true.
...-r flag was triggered, output will be labeled as *.1.*
...applying edits to cprops file
...applying edits to mnd file
...starting round 1 of scaffolding:
:) -p flag was triggered. Running LIGer with GNU Parallel support parameter set to true.
:) -s flag was triggered, starting calculations with 15000 threshold starting contig/scaffold size
:) -q flag was triggered, starting calculations with 1 threshold mapping quality
...Using cprops file: Cuscuta_gronovii.hifiasm.purged.cleaned.1.cprops
...Using merged_nodups file: Cuscuta_gronovii.hifiasm.purged.cleaned.mnd.1.txt
:( GNU Parallel support is set to true (default) but GNU Parallel is not in the path. Please install GNU Parallel or set -p option to false. Exiting!
...visualizing round 1 results:
:) -p flag was triggered. Running with GNU Parallel support parameter set to true.
:) -q flag was triggered, starting calculations for 1 threshold mapping quality
:) -i flag was triggered, building mapq without
:) -c flag was triggered, will remove temporary files after completion
:( Not sure how to parse your input or input files not found at intended locations. Exiting!
*****************************************************
Visualizing draft genomes in juicebox: 18 July 2016
USAGE: ./run-asm-visualizer.sh [options] <path_to_cprops> <path_to_asm> <path_to_contig_mnd>
DESCRIPTION:
This is a script to visualize draft assemblies (represented in input by their cprops and asm files) from pairwise contact data represented by Juicer merged_nodups.txt file. The script will produce hic files for viewing in Juicebox as well as scaffold annotation files. Metadata can be attached to the map by passing -i and -g flags with paths to stats and graph files.
ARGUMENTS:
path_to_cprops Specify cprops file path.
path_to_asm Specify asm file.
path_to_contig_mnd Specify path to mnd file describing pairwise Hi-C contacts between assembly contigs.
OPTIONS:
-q mapq Build map for a specific mapq threshold (default is 1).
-p true/false Use GNU Parallel to speed up computation (default is true).
-z zoom Build map with hardcoded zoom level. By default this is calculated based on the cprops file and applied only to genomes >= 2.1 Gb.
**unprompted**
#-l gap_file Path to gap bed file - necessary to build contig annotation track.
-m path_to_asm_mnd Path to mnd already lifted from input to assembly chromosome: used to skip the remapping step.
-n Skip normalization.
-r Build for specific resolutions (default is -r 2500000,1000000,500000,250000,100000,50000,25000,10000,5000,1000)
-c Clean up when done (default: no cleanup.)
-i Ignore mapq suffix.
-h Shows this help
*****************************************************
/scratch/users/juan/Hi-C-Assembly/opt/juicer/work/MC3413_Hi-C/3d-dna/run-asm-pipeline.sh: line 683: /afs/bx.psu.edu/user/j/jdc81/miniconda3/bin/rm: Permission denied
...detecting misjoins in round 1 assembly:
:) -p flag was triggered. Running with GNU Parallel support parameter set to true.
:) -c flag was triggered, starting calculations with 5% saturation level
:) -w flag was triggered, performing cursory search for mismatches at 25000 resolution
:) -d flag was triggered, depletion score will be averaged across a region bounded by 125000 superdiagonal
:) -k flag was triggered, starting calculations with 55% depletion as mismatch threshold
:) -n flag was triggered, performing mismatch region thinning at 1000 resolution
:( GNU Parallel support is set to true (default) but GNU Parallel is not in the path. Please install GNU Parallel or set -p option to false. Exiting!
:) -w flag was triggered, performing cursory search for repeat at 25000 resolution
:) -t flag was triggered, flagging regions with coverage higher than 2
...Dumping 25000 resolution coverage track
Picked up _JAVA_OPTIONS: -Djava.io.tmpdir=/scratch/users/juan/Hi-C-Assembly/opt/juicer/work/MC3413_Hi-C/tmp/
Mar 30, 2022 3:48:31 PM java.util.prefs.FileSystemPreferences$1 run
WARNING: Couldn't create user preferences directory. User preferences are unusable.
Mar 30, 2022 3:48:31 PM java.util.prefs.FileSystemPreferences$1 run
WARNING: java.io.IOException: Permission denied
Failed to create user directory!
Exception in thread "main" java.lang.ExceptionInInitializerError
at juicebox.tools.HiCTools.main(HiCTools.java:66)
Caused by: java.lang.NullPointerException
at juicebox.DirectoryManager.getHiCDirectory(DirectoryManager.java:113)
at juicebox.HiCGlobals.<clinit>(HiCGlobals.java:52)
... 1 more
Exception in thread "Thread-1" java.lang.SecurityException: Could not lock User prefs. Lock file access denied.
at java.util.prefs.FileSystemPreferences.checkLockFile0ErrorCode(FileSystemPreferences.java:937)
at java.util.prefs.FileSystemPreferences.lockFile(FileSystemPreferences.java:925)
at java.util.prefs.FileSystemPreferences.sync(FileSystemPreferences.java:729)
at java.util.prefs.FileSystemPreferences.flush(FileSystemPreferences.java:824)
at java.util.prefs.FileSystemPreferences.syncWorld(FileSystemPreferences.java:464)
at java.util.prefs.FileSystemPreferences.access$1200(FileSystemPreferences.java:50)
at java.util.prefs.FileSystemPreferences$4$1.run(FileSystemPreferences.java:442)
:( Juicebox coverage dump is empty! Perhaps something is wrong with the hic file or the requested resolution is too high. Exiting!
/scratch/users/juan/Hi-C-Assembly/opt/juicer/work/MC3413_Hi-C/3d-dna/run-asm-pipeline.sh: line 696: /afs/bx.psu.edu/user/j/jdc81/miniconda3/bin/mv: Permission denied
/scratch/users/juan/Hi-C-Assembly/opt/juicer/work/MC3413_Hi-C/3d-dna/run-asm-pipeline.sh: line 697: /afs/bx.psu.edu/user/j/jdc81/miniconda3/bin/mv: Permission denied
/scratch/users/juan/Hi-C-Assembly/opt/juicer/work/MC3413_Hi-C/3d-dna/run-asm-pipeline.sh: line 698: /afs/bx.psu.edu/user/j/jdc81/miniconda3/bin/mv: Permission denied
/scratch/users/juan/Hi-C-Assembly/opt/juicer/work/MC3413_Hi-C/3d-dna/run-asm-pipeline.sh: line 699: /afs/bx.psu.edu/user/j/jdc81/miniconda3/bin/mv: Permission denied
/scratch/users/juan/Hi-C-Assembly/opt/juicer/work/MC3413_Hi-C/3d-dna/run-asm-pipeline.sh: line 701: /afs/bx.psu.edu/user/j/jdc81/miniconda3/bin/mv: Permission denied
/scratch/users/juan/Hi-C-Assembly/opt/juicer/work/MC3413_Hi-C/3d-dna/run-asm-pipeline.sh: line 702: /afs/bx.psu.edu/user/j/jdc81/miniconda3/bin/mv: Permission denied
cat: mismatch_narrow.at.step.1.bed: No such file or directory
cat: repeats_wide.at.step.1.bed: No such file or directory
awk: fatal: cannot open file `Cuscuta_gronovii.hifiasm.purged.cleaned.1_asm.superscaf_track.txt' for reading (No such file or directory)
awk: /scratch/users/juan/Hi-C-Assembly/opt/juicer/work/MC3413_Hi-C/3d-dna/edit/overlay-edits.awk:20: fatal: cannot open file `Cuscuta_gronovii.hifiasm.purged.cleaned.1_asm.scaffold_track.txt' for reading (No such file or directory)
:) No more input edits to be done. Moving to polishing!
/scratch/users/juan/Hi-C-Assembly/opt/juicer/work/MC3413_Hi-C/3d-dna/run-asm-pipeline.sh: line 719: /afs/bx.psu.edu/user/j/jdc81/miniconda3/bin/rm: Permission denied
:( Not sure how to parse your input or input files not found at intended locations. Exiting!