Juicer results issues

[Lu Yin]

Hello,

I ran into issues with Juicer for the results produced. Here are the results I've got in my aligned/ folder without specifying any stages (see below). Because I'm missing some of the result files such as merged_nodups.txt, I re-run the pipeline with -S merge but in the error output it says "Move or remove directory "/home/lyin25/juicer/HIC003_4/aligned" before proceeding". Could you help me how I can deal with this?

a1656632655_msplit0000  a1656632655_msplit0004  inter_30.hic               merged1.txt       split0000  split0004

a1656632655_msplit0001  a1656632655_msplit0005  inter_30.txt               merged30.txt      split0001  split0005

a1656632655_msplit0002  inter.hic               inter_30_contact_domains/  merged_dedup.bam  split0002  tmp

a1656632655_msplit0003  inter.txt               inter_30_loops/            merged_sort.sam   split0003

Thanks!

Lu

[Lu Yin]

I re-run with -S dedup and now it ran (not failed) but I still do not have the merged_nodups.txt file. 

Here are some errors that I can find in my debug folder:

hic-16416689.err: Error while reading graphs file: java.io.FileNotFoundException: /home/lyin25/juicer/HIC003_4/aligned/inter_hists.m (No such file or directory)

hic30-16416690.err: Error while reading graphs file: java.io.FileNotFoundException: /home/lyin25/juicer/HIC003_4/aligned/inter_30_hists.m (No such file or directory)

Thanks so much!

Lu

[Moshe Olshansky]

Hi Lu,

Could you please do

ls -ltrh /home/lyin25/juicer/HIC003_4/aligned

and share the outcome??

Thank you,

Moshe.

[Lu Yin]

Hello Moshe!

Thanks for your reply. Here is what I got by doing ls -ltrh

-rw-rw-r-- 1 lyin25 lyin25 2.2G Jun 30 16:49 merged_sort.sam

-rw-rw-r-- 1 lyin25 lyin25 404M Jun 30 16:49 a1656632655_msplit0000

-rw-rw-r-- 1 lyin25 lyin25 427M Jun 30 16:49 a1656632655_msplit0001

-rw-rw-r-- 1 lyin25 lyin25 426M Jun 30 16:49 a1656632655_msplit0002

-rw-rw-r-- 1 lyin25 lyin25  69M Jun 30 16:50 a1656632655_msplit0005

-rw-rw-r-- 1 lyin25 lyin25 424M Jun 30 16:50 a1656632655_msplit0003

-rw-rw-r-- 1 lyin25 lyin25 425M Jun 30 16:50 a1656632655_msplit0004

drwxrwxr-x 2 lyin25 lyin25    0 Jun 30 17:14 inter_30_contact_domains/

drwxrwxr-x 2 lyin25 lyin25    0 Jun 30 17:14 inter_30_loops/

-rw-rw-r-- 1 lyin25 lyin25 404M Jul  1 11:37 split0000

-rw-rw-r-- 1 lyin25 lyin25 427M Jul  1 11:37 split0001

-rw-rw-r-- 1 lyin25 lyin25 404M Jul  1 11:37 a1656700599_msplit0000

-rw-rw-r-- 1 lyin25 lyin25 426M Jul  1 11:37 split0002

-rw-rw-r-- 1 lyin25 lyin25 427M Jul  1 11:37 a1656700599_msplit0001

-rw-rw-r-- 1 lyin25 lyin25 424M Jul  1 11:37 split0003

-rw-rw-r-- 1 lyin25 lyin25 425M Jul  1 11:37 split0004

-rw-rw-r-- 1 lyin25 lyin25 426M Jul  1 11:37 a1656700599_msplit0002

-rw-rw-r-- 1 lyin25 lyin25  69M Jul  1 11:37 split0005

-rw-rw-r-- 1 lyin25 lyin25 424M Jul  1 11:37 a1656700599_msplit0003

-rw-rw-r-- 1 lyin25 lyin25  69M Jul  1 11:37 a1656700599_msplit0005

-rw-rw-r-- 1 lyin25 lyin25 425M Jul  1 11:37 a1656700599_msplit0004

-rw-rw-r-- 1 lyin25 lyin25    5 Jul  1 11:38 tmp

-rw-rw-r-- 1 lyin25 lyin25 1.5K Jul  1 11:38 inter.txt

-rw-rw-r-- 1 lyin25 lyin25 1.5K Jul  1 11:38 inter_30.txt

-rw-rw-r-- 1 lyin25 lyin25  80M Jul  1 11:38 merged1.txt

-rw-rw-r-- 1 lyin25 lyin25  74M Jul  1 11:38 merged30.txt

-rw-rw-r-- 1 lyin25 lyin25 594M Jul  1 11:39 merged_dedup.bam

-rw-rw-r-- 1 lyin25 lyin25 1.6G Jul  1 11:49 inter_30.hic

-rw-rw-r-- 1 lyin25 lyin25 1.7G Jul  1 11:50 inter.hic

[Lu Yin]

Actually this is what I have by doing the command on the align/ folder (after fixing another issue on QOS):

[lyin25@agave3:~/juicer/HIC003_5/aligned]$ ls -lhrt

total 4.0G

-rw-rw-r-- 1 lyin25 lyin25    5 Jul  5 16:14 tmp

-rw-rw-r-- 1 lyin25 lyin25 1.5K Jul  5 16:14 inter.txt

-rw-rw-r-- 1 lyin25 lyin25 1.5K Jul  5 16:14 inter_30.txt

-rw-rw-r-- 1 lyin25 lyin25  74M Jul  5 16:15 merged30.txt

-rw-rw-r-- 1 lyin25 lyin25  80M Jul  5 16:15 merged1.txt

-rw-rw-r-- 1 lyin25 lyin25 594M Jul  5 16:15 merged_dedup.bam

-rw-rw-r-- 1 lyin25 lyin25 1.7G Jul  5 16:27 inter.hic

-rw-rw-r-- 1 lyin25 lyin25 1.6G Jul  5 16:27 inter_30.hic

drwxrwxr-x 2 lyin25 lyin25    0 Jul  5 16:27 inter_30_contact_domains/

drwxrwxr-x 2 lyin25 lyin25    0 Jul  5 16:27 inter_30_loops/

[Moshe Olshansky]

Hi Lu,

Why do you think that you are missing files? What version of juicer are you using?

[Lu Yin]

Moshe,

I'm using juicer 1.6, which I think should be the latest version of juicer. 

I'm not sure why I'm getting the results. It could be because the tech help at my University (we use SLURM) helped changed the juicer.sh script. Particularly, when I first ran the script, it did not work on SLURM, and the error output showed "Invalid qos specification". Now after the tech help's changes (he helped added the #SBATCH -q qos in several places in juicer.sh; -p on our system is partition, -q on our system is qos, which are different from these specifications in the juicer program), it ran fine and it did not produce this error anymore. I'm not sure if this could have contributed to the incomplete results. After all, if you have any other ways you can make this work on SLURM, that would be much appreciated.

Lu

[Olga Dudchenko]

Hey Lu,

Sorry things are a bit confusing as we are transitioning from Juicer 1 to Juicer 2. Looks like you are trying to generate the merged_nodups.txt file, correct? Based on your output files you seem to be running Juicer 1. There is a special flag now (--assembly) to generate the merged_nodups.txt in addition to the merged_dedup.bam file. You can also run something like this manually:

samtools view -@8 -O SAM -F 1024 $outputdir/merged_dedup.*am | awk -v mnd=1 -f ${juiceDir}/scripts/sam_to_pre.awk > ${outputdir}/merged_nodups.txt

Hope this helps,

Olga  

[Lu Yin]

Yes Olga! That was part of the issue. Thanks for your reply! But when I added the --assembly to the command (on the existing folder with HIC003_5/aligned which contained merged_dedup.bam), it failed and error output showed "echo '***! Move or remove directory "/home/lyin25/juicer/HIC003_5/aligned" before proceeding.'" I'm just confused as to how could I convert the the file to merged_nodups.txt if the merged_dedup.bam does not exist? Anyway, am I missing anything? This is my code: 

/home/lyin25/juicer/scripts/juicier.sh 

-q publicgpu # partition                                                      

-l publicgpu # partition                                                      

-c wildfire  # QOS                                                            

-p hg19 #path for chr.sizes file. this must be specified                      

-g hg19 #genomeID                                                             

-d /home/lyin25/juicer/HIC003_5 #directory containing the fastq/              

-a "hg19" #description of exp.                                                

-z /home/lyin25/juicer/references_eg/Homo_sapiens_assembly19.fasta #path for ref seq file, bwa indexed

-D /home/lyin25/juicer/ #juicer directory                                     

-y /home/lyin25/juicer/restriction_sites_eg/hg19_MboI.txt #path for restriction_site.txt file; include this in HIC003_2 run

-s MboI  # "HindIII" or "MboI"; include this option in HIC003_2 run

--assembly #generate merged_nodups.txt from merged_dedup.bam, on existing HIC003_5, failed, aligned/ exists

I also tried the second way you provided (samtools view ....), it ran fine but in the error output it says,

/var/spool/slurmd/job16507160/slurm_script: line 17: -q: command not found

/var/spool/slurmd/job16507160/slurm_script: line 18: -l: command not found

/var/spool/slurmd/job16507160/slurm_script: line 19: -c: command not found

/var/spool/slurmd/job16507160/slurm_script: line 20: -p: command not found

/var/spool/slurmd/job16507160/slurm_script: line 21: -g: command not found

/var/spool/slurmd/job16507160/slurm_script: line 22: -d: command not found

/var/spool/slurmd/job16507160/slurm_script: line 23: -a: command not found

/var/spool/slurmd/job16507160/slurm_script: line 24: -z: command not found

/var/spool/slurmd/job16507160/slurm_script: line 25: -D: command not found

/var/spool/slurmd/job16507160/slurm_script: line 26: -y: command not found

/var/spool/slurmd/job16507160/slurm_script: line 27: -s: command not found

[Moshe Olshansky]

Your error output explains what happened. bash thinks that every line is a new command.  Everything should be in one line. You can split the one line command into several lines, but then you need to remove the comments and put \ (preceded by space) at the end of every line except for the last line.

[Lu Yin]

Thanks! Now it works for solving this issue (using the samtools view xxx command you provided). However, I have another issue, which shows in the mergesort-xxx.err file in the debug/ folder. See below. Also I do not have merged_sort.txt file in the aligned/ folder. Any thoughts on where this might have gone wrong?

[bam_sort_core] merging from 0 files and 8 in-memory blocks...

20.73user 2.86system 0:09.11elapsed 259%CPU (0avgtext+0avgdata 2290132maxresident)k

12496inputs+4447416outputs (1major+17169minor)pagefaults 0swaps

Lu

[Moshe Olshansky]

Do you have merged_nodups.txt in your aligned folder?

[Lu Yin]

Yes. But I don’t have merged_sort.txt and there is this error in the debug folder during the merge step..

Get Outlook for iOS

---

From: 3d-ge...@googlegroups.com <3d-ge...@googlegroups.com> on behalf of Moshe Olshansky <moshe.o...@gmail.com>
Sent: Thursday, July 7, 2022 6:12:38 PM
To: 3D Genomics <3d-ge...@googlegroups.com>
Subject: 3D Genomics Forum Re: Juicer results issues

 

--
You received this message because you are subscribed to a topic in the Google Groups "3D Genomics" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/3d-genomics/8uNuH0010Vc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to 3d-genomics...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/3d-genomics/8fae902c-8b46-4de1-b939-a5d06c0d3c1cn%40googlegroups.com.

[Lu Yin]

I have merged_nodups.txt only because I ran the samtools view step that you shared with me earlier to convert the merged_dedup.bam to merged_nodups.txt. My point is this step ran fine but I still think my results are incomplete.

Get Outlook for iOS

---

From: Lu Yin <Lu.Y...@asu.edu>
Sent: Thursday, July 7, 2022 6:16:14 PM
To: 3d-ge...@googlegroups.com <3d-ge...@googlegroups.com>
Subject: Re: 3D Genomics Forum Re: Juicer results issues

 

[Moshe Olshansky]

Yes, you have merged_nodups.txt because on the samtools view ... command. The message: [bam_sort_core] merging from 0 files and 8 in-memory blocks... is not an error. If the bam/sam file you were sorting is small enough it is kept in RAM and does not need temporary files.

How big your merged_nodups.txt is?

[Lu Yin]

My merged_nodups.txt is 753 M. But in my inter_30.txt, the alignables and Unique + Duplicates do not match. For example, my 2 alignments and 3 or more alignments = 2,327,130 and the Unique + Duplicates = 2,251,836. I saw from one of Neva's posts that these two numbers should match, otherwise I'm missing some reads from the hic file. That's another reason why I say my results might not be complete.

--

Lu Yin

Postdoctoral Research Scholar

Plant Genetics and Meiosis

School of Life Sciences, LSE 651

Arizona State University

---

From: 3d-ge...@googlegroups.com <3d-ge...@googlegroups.com> on behalf of Moshe Olshansky <moshe.o...@gmail.com>

Sent: Thursday, July 7, 2022 6:26 PM
To: 3D Genomics <3d-ge...@googlegroups.com>

To view this discussion on the web visit https://groups.google.com/d/msgid/3d-genomics/1c9aa34a-74f8-48d3-a3bf-53d2d5076550n%40googlegroups.com.

[Moshe Olshansky]

Does this mean that you had less than 3 million reads to start with?

[Moshe Olshansky]

To follow on my previous email, if your merged_nodups.txt is 753M, you probably have more than 10M contacts, which is impossible with ~3M reads.

How many reads have you started with?

[Lu Yin]

I started with the test data (hg19) which has pair-end reads and each is about 160 M

--

Lu Yin

Postdoctoral Research Scholar

Plant Genetics and Meiosis

School of Life Sciences, LSE 651

Arizona State University

---

From: 3d-ge...@googlegroups.com <3d-ge...@googlegroups.com> on behalf of Moshe Olshansky <moshe.o...@gmail.com>

Sent: Thursday, July 7, 2022 8:37 PM

To view this discussion on the web visit https://groups.google.com/d/msgid/3d-genomics/12a31735-dd98-48bd-8153-f2d9af6db507n%40googlegroups.com.

[Moshe Olshansky]

If you started with 160M reads you should have seen much higher numbers in inter.txt and inter_30.txt, so indeed something has gone wrong.

For 160M reads you do not really need the cluster. You can run the CPU version of juicer on a single cluster node. 

[Lu Yin]

For my actual data that I'd like to run, I have pair-end data and each is about 40G. Could you offer some insights as to where might have gone wrong? I do want to get this worked out on the cluster to run my actual data. And the cluster we have here is SLURM. Appreciate it!

--

Lu Yin

Postdoctoral Research Scholar

Plant Genetics and Meiosis

School of Life Sciences, LSE 651

Arizona State University

---

From: 3d-ge...@googlegroups.com <3d-ge...@googlegroups.com> on behalf of Moshe Olshansky <moshe.o...@gmail.com>

Sent: Thursday, July 7, 2022 8:50 PM

To view this discussion on the web visit https://groups.google.com/d/msgid/3d-genomics/d7128c03-1531-4dfd-89f1-7f6d5d0e42ddn%40googlegroups.com.

[Moshe Olshansky]

I see. I am also still in the process of learning the SLURM version of juicer. Neva is the definite reference here.

I would have removed (or renamed) the aligned folder and run the script again, waiting for it to finish (successfully or not) and without doing anything manually.

Before you do this, can you please do ls -ltrh in your splits folder and send the outcome?

[Lu Yin]

Below is my splits/ outcome. By remove/rename the aligned/ folder, do I just start from scratch again (which I have done a couple of times, and the results are similar) or is there is a particular stage you recommend? Maybe the "merge" stage?

[lyin25@agave1:~/juicer/HIC003_5/splits]$ ls -lhtr

total 2.2G

lrwxrwxrwx 1 lyin25 lyin25   65 Jul  5 16:06 HIC003_S2_L001_R1_001.fastq.gz -> /home/lyin25/juicer/HIC003_5/fastq/HIC003_S2_L001_R1_001.fastq.gz

lrwxrwxrwx 1 lyin25 lyin25   65 Jul  5 16:06 HIC003_S2_L001_R2_001.fastq.gz -> /home/lyin25/juicer/HIC003_5/fastq/HIC003_S2_L001_R2_001.fastq.gz

-rw-rw-r-- 1 lyin25 lyin25    8 Jul  5 16:06 HIC003_S2_L001_001.fastq.gz_linecount.txt

-rw-rw-r-- 1 lyin25 lyin25   55 Jul  5 16:12 HIC003_S2_L001_001.fastq.gz_norm.txt.res.txt

-rw-rw-r-- 1 lyin25 lyin25 2.2G Jul  5 16:12 HIC003_S2_L001_001.fastq.gz.sam

--

Lu Yin

Postdoctoral Research Scholar

Plant Genetics and Meiosis

School of Life Sciences, LSE 651

Arizona State University

From: 3d-ge...@googlegroups.com <3d-ge...@googlegroups.com> on behalf of Moshe Olshansky <moshe.o...@gmail.com>

Sent: Thursday, July 7, 2022 9:13 PM

To view this discussion on the web visit https://groups.google.com/d/msgid/3d-genomics/bb08e3e5-af4b-4b63-9a75-9aaaf7c7d551n%40googlegroups.com.

[Moshe Olshansky]

Can you please cat HIC003_S2_L001_001.fastq.gz_linecount.txt and HIC003_S2_L001_001.fastq.gz_norm.txt.res.txt?

2.2GB sam file is way too small for 160M reads. How large are your fastq.gz files?

On Friday, July 8, 2022 at 2:18:55 PM UTC+10 lyi...@asu.edu wrote:

Below is my splits/ outcome. By remove/rename the aligned/ folder, do I just start from scratch again (which I have done a couple of times, and the results are similar) or is there is a particular stage you recommend? Maybe the "merge" stage?

[lyin25@agave1:~/juicer/HIC003_5/splits]$ ls -lhtr

total 2.2G

lrwxrwxrwx 1 lyin25 lyin25   65 Jul  5 16:06 HIC003_S2_L001_R1_001.fastq.gz -> /home/lyin25/juicer/HIC003_5/fastq/HIC003_S2_L001_R1_001.fastq.gz

lrwxrwxrwx 1 lyin25 lyin25   65 Jul  5 16:06 HIC003_S2_L001_R2_001.fastq.gz -> /home/lyin25/juicer/HIC003_5/fastq/HIC003_S2_L001_R2_001.fastq.gz

-rw-rw-r-- 1 lyin25 lyin25    8 Jul  5 16:06 HIC003_S2_L001_001.fastq.gz_linecount.txt

-rw-rw-r-- 1 lyin25 lyin25   55 Jul  5 16:12 HIC003_S2_L001_001.fastq.gz_norm.txt.res.txt

-rw-rw-r-- 1 lyin25 lyin25 2.2G Jul  5 16:12 HIC003_S2_L001_001.fastq.gz.sam

--

Lu Yin

Postdoctoral Research Scholar

Plant Genetics and Meiosis

School of Life Sciences, LSE 651

Arizona State University

[Lu Yin]

Below are the cat results of the two files. By fastq.gz files, I think you meant the input fastq.gz, which are the hg19 test files. One pair-end is 157M and the other pair-end is 167M.

[lyin25@agave1:~/juicer/HIC003_5/splits]$ cat  HIC003_S2_L001_001.fastq.gz_linecount.txt

9394312

[lyin25@agave1:~/juicer/HIC003_5/splits]$ cat HIC003_S2_L001_001.fastq.gz_norm.txt.res.txt

895318 2348578 21448 1865150 386686 75294 0 391.662010

--

Lu Yin

Postdoctoral Research Scholar

Plant Genetics and Meiosis

School of Life Sciences, LSE 651

Arizona State University

---

From: 3d-ge...@googlegroups.com <3d-ge...@googlegroups.com> on behalf of Moshe Olshansky <moshe.o...@gmail.com>

Sent: Thursday, July 7, 2022 9:35 PM

To view this discussion on the web visit https://groups.google.com/d/msgid/3d-genomics/69f4a073-f28d-4f62-bded-4609c27de3c1n%40googlegroups.com.

[Moshe Olshansky]

Oh, so your imput fies are ~160MB, not 160M reads. Then it makes sense.

If you open the juicer.sh script in an editor and look for version, what does it say there?

[Lu Yin]

Oh yeah 160 Mb are my input fastq.gz files. Sorry about the confusion. In juicer.sh, it actually says Juicer 2.0 (sorry previously I told you 1.6 which seemed to be wrong).

--

Lu Yin

Postdoctoral Research Scholar

Plant Genetics and Meiosis

School of Life Sciences, LSE 651

Arizona State University

From: 3d-ge...@googlegroups.com <3d-ge...@googlegroups.com> on behalf of Moshe Olshansky <moshe.o...@gmail.com>

Sent: Thursday, July 7, 2022 9:50 PM

To view this discussion on the web visit https://groups.google.com/d/msgid/3d-genomics/14c0d731-2c84-49ef-a185-f3293f78043dn%40googlegroups.com.

[Moshe Olshansky]

I suspected it was juicer2. This version is not supposed to create merged_nodups.txt. Istead, it should create merged1.txt and merged30.txt files (which I think you got).

On Friday, July 8, 2022 at 2:58:27 PM UTC+10 lyi...@asu.edu wrote:

Oh yeah 160 Mb are my input fastq.gz files. Sorry about the confusion. In juicer.sh, it actually says Juicer 2.0 (sorry previously I told you 1.6 which seemed to be wrong).

--

Lu Yin

Postdoctoral Research Scholar

Plant Genetics and Meiosis

School of Life Sciences, LSE 651

Arizona State University

[Lu Yin]

Okay! Thanks! So glad to know it actually works! But merged_nodups.txt is needed for 3D-DNA pipeline (chapter 3 in your cookbook) and the Juicebook Assembly Tools (chapter 4). Will it work if I use merged1.txt or merged30.txt? If so, which one should I use?

--

Lu Yin

Postdoctoral Research Scholar

Plant Genetics and Meiosis

School of Life Sciences, LSE 651

Arizona State University

---

From: 3d-ge...@googlegroups.com <3d-ge...@googlegroups.com> on behalf of Moshe Olshansky <moshe.o...@gmail.com>

Sent: Thursday, July 7, 2022 10:24 PM

To view this discussion on the web visit https://groups.google.com/d/msgid/3d-genomics/86ef666c-e6a0-47bc-bba5-d1f686923b18n%40googlegroups.com.

[Moshe Olshansky]

Since it is a small example and does not take much time to run, I would run it anew with the --assemnly flag, as per Olga's suggestion.

[Lu Yin]

I've finished running it from anew with the --assembly flag. Below is my outputs in aligned/. Does this look normal to you? The good thing is that I do have merged_nodups.txt, but I do not have hic files though. Also again in the inter_30.txt file, the 2 or 3 or more alignments = 2327130 while the Unique+Duplicates = 2251836, not matching. Let me know! Thanks again!

[lyin25@agave1:~/juicer/HIC003_6/aligned]$ ls -ltrh

total 1.5G

-rw-rw-r-- 1 lyin25 lyin25    5 Jul  7 22:49 tmp

-rw-rw-r-- 1 lyin25 lyin25 1.5K Jul  7 22:49 inter.txt

-rw-rw-r-- 1 lyin25 lyin25 1.5K Jul  7 22:49 inter_30.txt

-rw-rw-r-- 1 lyin25 lyin25  80M Jul  7 22:50 merged1.txt

-rw-rw-r-- 1 lyin25 lyin25  74M Jul  7 22:50 merged30.txt

-rw-rw-r-- 1 lyin25 lyin25 594M Jul  7 22:50 merged_dedup.bam

-rw-rw-r-- 1 lyin25 lyin25 753M Jul  7 22:53 merged_nodups.txt

[Olga Dudchenko]

You are not supposed to have hic files. Please follow the instructions in the genome assembly cookbook. -Olga

[Lu Yin]

If I'm not supposed to have hic files, how am I supposed to visualize my data? According to Chapter 5 of the Genome assembly cookbook, to use the Juicebox Assembly Tools, I need both the assembly file and the hic file to view the HIC contacts. The assembly file is the output from the 3D-DNA pipeline using the merged_nodups.txt, whereas the hic file is what produced out of juicer. 

--

Lu Yin

Postdoctoral Research Scholar

Plant Genetics and Meiosis

School of Life Sciences, LSE 651

Arizona State University

---

From: 'Olga Dudchenko' via 3D Genomics <3d-ge...@googlegroups.com>
Sent: Friday, July 8, 2022 11:48 AM

To view this discussion on the web visit https://groups.google.com/d/msgid/3d-genomics/288018c9-dcc0-4453-b582-4cfb40112d3bn%40googlegroups.com.

[Moshe Olshansky]

Hi Lu,

The hic files will be created later by the 3d-dna pipeline.. If you really want to get the hic files now you can use juicer tools pre, something like 

java -jar juicer_tools.jar pre -r 2500000,1000000,500000,250000,100000,50000,25000,10000,5000 merged30.txt inter_30.hic hg19.chromosomes.sizews

[Lu Yin]

That answers my question! Thank you so much for all your help!

--

Lu Yin

Postdoctoral Research Scholar

Plant Genetics and Meiosis

School of Life Sciences, LSE 651

Arizona State University

---

From: 3d-ge...@googlegroups.com <3d-ge...@googlegroups.com> on behalf of Moshe Olshansky <moshe.o...@gmail.com>

Sent: Friday, July 8, 2022 6:30 PM

To view this discussion on the web visit https://groups.google.com/d/msgid/3d-genomics/ebd71511-dff4-4935-b485-21684ff998e2n%40googlegroups.com.

[Lu Yin]

Dear Aiden Lab,

Going back to this discussion, I tried 3d-dna and it did not produce the expected .hic file for me. When I loaded PurgeHiFiPolishSnLHiFiZing1_3G.rawchrom.hic into Juicebox, it says error loading file. There are my output:

lrwxrwxrwx 1 lyin25 lyin25   48 Nov 22 10:47 PurgeHiFiPolishSnLHiFiZing1_3G.mnd.txt -> /run010/aligned/merged_nodups.txt

-rw-rw-r-- 1 lyin25 lyin25  95K Nov 22 10:47 PurgeHiFiPolishSnLHiFiZing1_3G.cprops

lrwxrwxrwx 1 lyin25 lyin25   37 Nov 22 10:47 PurgeHiFiPolishSnLHiFiZing1_3G.0.cprops -> PurgeHiFiPolishSnLHiFiZing1_3G.cprops

-rw-rw-r-- 1 lyin25 lyin25  14K Nov 22 10:48 PurgeHiFiPolishSnLHiFiZing1_3G.0.asm

-rw-rw-r-- 1 lyin25 lyin25 349K Nov 22 10:50 PurgeHiFiPolishSnLHiFiZing1_3G.0_asm.superscaf_track.txt

-rw-rw-r-- 1 lyin25 lyin25 422K Nov 22 10:50 PurgeHiFiPolishSnLHiFiZing1_3G.0_asm.scaffold_track.txt

-rw-rw-r-- 1 lyin25 lyin25 112K Nov 22 10:50 PurgeHiFiPolishSnLHiFiZing1_3G.0.assembly

-rw-rw-r-- 1 lyin25 lyin25  155 Nov 22 11:27 PurgeHiFiPolishSnLHiFiZing1_3G.0.hic

lrwxrwxrwx 1 lyin25 lyin25   56 Nov 22 11:27 PurgeHiFiPolishSnLHiFiZing1_3G.resolved_asm.superscaf_track.txt -> PurgeHiFiPolishSnLHiFiZing1_3G.0_asm.superscaf_track.txt

lrwxrwxrwx 1 lyin25 lyin25   55 Nov 22 11:27 PurgeHiFiPolishSnLHiFiZing1_3G.resolved_asm.scaffold_track.txt -> PurgeHiFiPolishSnLHiFiZing1_3G.0_asm.scaffold_track.txt

lrwxrwxrwx 1 lyin25 lyin25   36 Nov 22 11:27 PurgeHiFiPolishSnLHiFiZing1_3G.resolved.hic -> PurgeHiFiPolishSnLHiFiZing1_3G.0.hic

lrwxrwxrwx 1 lyin25 lyin25   39 Nov 22 11:27 PurgeHiFiPolishSnLHiFiZing1_3G.resolved.cprops -> PurgeHiFiPolishSnLHiFiZing1_3G.0.cprops

lrwxrwxrwx 1 lyin25 lyin25   36 Nov 22 11:27 PurgeHiFiPolishSnLHiFiZing1_3G.resolved.asm -> PurgeHiFiPolishSnLHiFiZing1_3G.0.asm

-rw-rw-r-- 1 lyin25 lyin25   47 Nov 22 11:27 PurgeHiFiPolishSnLHiFiZing1_3G.polished.suspect_2D.txt

-rw-rw-r-- 1 lyin25 lyin25   47 Nov 22 11:27 PurgeHiFiPolishSnLHiFiZing1_3G.polished.mismatches_2D.txt

-rw-rw-r-- 1 lyin25 lyin25   47 Nov 22 11:27 PurgeHiFiPolishSnLHiFiZing1_3G.polished.edits_2D.txt

-rw-rw-r-- 1 lyin25 lyin25  95K Nov 22 11:27 PurgeHiFiPolishSnLHiFiZing1_3G.polished.cprops

-rw-rw-r-- 1 lyin25 lyin25  14K Nov 22 11:28 PurgeHiFiPolishSnLHiFiZing1_3G.polished.asm

-rw-rw-r-- 1 lyin25 lyin25 112K Nov 22 11:30 PurgeHiFiPolishSnLHiFiZing1_3G.resolved.polish.assembly

-rw-rw-r-- 1 lyin25 lyin25 349K Nov 22 11:30 PurgeHiFiPolishSnLHiFiZing1_3G.polished_asm.superscaf_track.txt

-rw-rw-r-- 1 lyin25 lyin25 422K Nov 22 11:30 PurgeHiFiPolishSnLHiFiZing1_3G.polished_asm.scaffold_track.txt

-rw-rw-r-- 1 lyin25 lyin25  155 Nov 22 12:06 PurgeHiFiPolishSnLHiFiZing1_3G.polished.hic

-rw-rw-r-- 1 lyin25 lyin25   47 Nov 22 12:06 h.edits.txt

-rw-rw-r-- 1 lyin25 lyin25  95K Nov 22 12:06 PurgeHiFiPolishSnLHiFiZing1_3G.split.cprops

-rw-rw-r-- 1 lyin25 lyin25   47 Nov 22 12:06 PurgeHiFiPolishSnLHiFiZing1_3G.polished.split.suspicious_2D.txt

-rw-rw-r-- 1 lyin25 lyin25   47 Nov 22 12:06 PurgeHiFiPolishSnLHiFiZing1_3G.polished.split.mismatches_2D.txt

-rw-rw-r-- 1 lyin25 lyin25   47 Nov 22 12:06 PurgeHiFiPolishSnLHiFiZing1_3G.polished.split.edits_2D.txt

-rw-rw-r-- 1 lyin25 lyin25  14K Nov 22 12:07 PurgeHiFiPolishSnLHiFiZing1_3G.split.asm

-rw-rw-r-- 1 lyin25 lyin25 349K Nov 22 12:09 PurgeHiFiPolishSnLHiFiZing1_3G.split_asm.superscaf_track.txt

-rw-rw-r-- 1 lyin25 lyin25 422K Nov 22 12:09 PurgeHiFiPolishSnLHiFiZing1_3G.split_asm.scaffold_track.txt

-rw-rw-r-- 1 lyin25 lyin25 112K Nov 22 12:09 PurgeHiFiPolishSnLHiFiZing1_3G.polished.split.assembly

-rw-rw-r-- 1 lyin25 lyin25  155 Nov 22 12:33 PurgeHiFiPolishSnLHiFiZing1_3G.split.hic

-rw-rw-r-- 1 lyin25 lyin25 112K Nov 22 12:33 PurgeHiFiPolishSnLHiFiZing1_3G.split.assembly

-rw-rw-r-- 1 lyin25 lyin25  95K Nov 22 12:33 PurgeHiFiPolishSnLHiFiZing1_3G.rawchrom.cprops

-rw-rw-r-- 1 lyin25 lyin25  14K Nov 22 12:33 PurgeHiFiPolishSnLHiFiZing1_3G.rawchrom.asm

-rw-rw-r-- 1 lyin25 lyin25 349K Nov 22 12:37 PurgeHiFiPolishSnLHiFiZing1_3G.rawchrom_asm.superscaf_track.txt

-rw-rw-r-- 1 lyin25 lyin25 422K Nov 22 12:37 PurgeHiFiPolishSnLHiFiZing1_3G.rawchrom_asm.scaffold_track.txt

-rw-rw-r-- 1 lyin25 lyin25 112K Nov 22 12:37 PurgeHiFiPolishSnLHiFiZing1_3G.rawchrom.assembly

-rw-rw-r-- 1 lyin25 lyin25  155 Nov 22 13:04 PurgeHiFiPolishSnLHiFiZing1_3G.rawchrom.hic

-rw-rw-r-- 1 lyin25 lyin25 1.9G Nov 22 15:09 PurgeHiFiPolishSnLHiFiZing1_3G.rawchrom.fasta

-rw-rw-r-- 1 lyin25 lyin25    0 Nov 22 15:14 alignments.txt

-rw-rw-r-- 1 lyin25 lyin25   43 Nov 22 15:14 overlaps_2D_input.txt

-rw-rw-r-- 1 lyin25 lyin25   43 Nov 22 15:14 overlaps_2D_asm.txt

-rw-rw-r-- 1 lyin25 lyin25  14K Nov 22 15:14 PurgeHiFiPolishSnLHiFiZing1_3G.rawchrom_tiled.asm

drwxrwxr-x 2 lyin25 lyin25    3 Nov 22 15:47 PurgeHiFiPolishSnLHiFiZing1_3G.rawchrom/

-rw-rw-r-- 1 lyin25 lyin25  14K Nov 22 15:47 PurgeHiFiPolishSnLHiFiZing1_3G.final.asm

lrwxrwxrwx 1 lyin25 lyin25   89 Nov 22 15:47 PurgeHiFiPolishSnLHiFiZing1_3G.final.fasta -> PurgeHiFiPolishSnLHiFiZing1_3G.rawchrom/merged_PurgeHiFiPolishSnLHiFiZing1_3G.rawchrom.fa

-rw-rw-r-- 1 lyin25 lyin25  95K Nov 22 15:48 PurgeHiFiPolishSnLHiFiZing1_3G.final.cprops

-rw-rw-r-- 1 lyin25 lyin25 112K Nov 22 15:48 PurgeHiFiPolishSnLHiFiZing1_3G.final.assembly

-rw-rw-r-- 1 lyin25 lyin25 1.9G Nov 22 15:58 PurgeHiFiPolishSnLHiFiZing1_3G.FINAL.fasta

-rw-rw-r-- 1 lyin25 lyin25 112K Nov 22 15:58 PurgeHiFiPolishSnLHiFiZing1_3G.FINAL.assembly

Because I double-checked with a collaborator, he mentioned that the inter.txt file as an output from my juicer pipeline is incomplete. Basically I'm expected to get something like:

Sequenced Read Pairs:  310,000,000
 Normal Paired: 199,086,110 (64.22%)
 Chimeric Paired: 79,079,046 (25.51%)
 Chimeric Ambiguous: 27,473,877 (8.86%)
 Unmapped: 4,360,967 (1.41%)
 Ligation Motif Present: 56,264,573 (18.15%)
Alignable (Normal+Chimeric Paired): 278,165,156 (89.73%)
Unique Reads: 200,881,515 (64.80%)
PCR Duplicates: 74,635,669 (24.08%)
Optical Duplicates: 2,647,972 (0.85%)
Library Complexity Estimate: 411,814,896
Intra-fragment Reads: 30,817,577 (9.94% / 15.34%)
Below MAPQ Threshold: 19,468,503 (6.28% / 9.69%)
Hi-C Contacts: 150,595,435 (48.58% / 74.97%)
 Ligation Motif Present: 22,195,082  (7.16% / 11.05%)
 3' Bias (Long Range): 67% - 33%
 Pair Type %(L-I-O-R): 25% - 25% - 25% - 25%
Inter-chromosomal: 24,325,570  (7.85% / 12.11%)
Intra-chromosomal: 126,269,865  (40.73% / 62.86%)
Short Range (<20Kb): 93,324,676  (30.10% / 46.46%)
Long Range (>20Kb): 32,944,589  (10.63% / 16.40%)

But my inter.txt looks like:

Read type: Paired End

Sequenced Read Pairs:  151,666,666

No chimera found: 1,781,655 (1.17%)

 One or both reads unmapped: 1,781,655 (1.17%)

2 alignments: 97,134,534 (64.04%)

 2 alignments (A...B): 96,949,989 (63.92%)

 2 alignments (A1...A2B; A1B2...B1A2): 184,545 (0.12%)

3 or more alignments: 52,750,477 (34.78%)

Ligation Motif Present: 517,451 (0.34%)

Average insert size: 650.97

Total Unique: 80,534,224 (82.91%, 53.10%)

Total Duplicates: 16,600,310 (17.09%, 10.95%)

My code to run juicer was:

 43 #opts=(                                                                         

 44 #  -q publicgpu # partition                                                     

 45 #  -l publicgpu # partition                                                     

 46 #  -c wildfire  # QOS                                                           

 47 #  -p /juicer/references/PurgeHiFiPolishSnLHiFiZing1_3G.fa.chrom.sizes #path for chr    .sizes file. this must be specified

 48 #  -d /run010/ #directory containing the fastq/ using the reduced-size files, remove the original

 49 #  -a "Zingeria" #description of exp.                                           

 50 #  -z /juicer/references/PurgeHiFiPolishSnLHiFiZing1_3G.fa #path for ref seq file, b    wa indexed

 51 #  -D /juicer/ #juicer directory         

 52 #  --assembly #generate merged_nodups.txt                                       

 53 #  -y /juicer/restriction_sites/PurgeHiFiPolishSnLHiFiZing1_3G_MboI.txt

 54 #  -s "MboI"                                                                    

 55 #)                                                                              

 56 #                                                                               

 57 ##juicier.sh is juicer.sh but with QOS support                                  

 58 #/juicer/scripts/juicier.sh "${opts[@]}"

And my code to run the 3d-dna was:

 68 /home/lyin25/3d-dna/run-asm-pipeline.sh -m diploid -r 0 -i 50000 --merger-lastz-options \"--gfextend\ --gapped\ --chain=200    ,200\ --exact\" /data/cchen347/ZingeriaProject2021-/HiC/juicer/references/PurgeHiFiPolishSnLHiFiZing1_3G.fa /scratch/lyin25    /run010/aligned/merged_nodups.txt

So it seems there was definately something wrong in the juicer pipeline because the output was not complete. I'm looking forward to your reply!

Lu

[Olga Dudchenko]

Hello Lu,

You'll need to explore the debug folder of your Juicer run (look through .err files for hints). 3d-dna output is not important for debugging Juicer.

Best,

Olga

[Lu Yin]

Hello Olga!

These are what in my debug folder. Am I missing anything?

[lyin25@agave:/run010/debug]$ ls -lhtr

total 314K

-rw-rw-r-- 1 lyin25 lyin25  691 Nov  8 14:10 head-18038797.out

-rw-rw-r-- 1 lyin25 lyin25   58 Nov  8 14:52 count_ligation-18038798.out

-rw-rw-r-- 1 lyin25 lyin25  687 Nov  9 09:34 align1-18038799.out

-rw-rw-r-- 1 lyin25 lyin25 153K Nov  9 09:34 align1-18038799.err

-rw-rw-r-- 1 lyin25 lyin25   58 Nov  9 12:29 merge-18038800.out

-rw-rw-r-- 1 lyin25 lyin25   49 Nov  9 12:29 merge-18038800.err

-rw-rw-r-- 1 lyin25 lyin25  236 Nov  9 13:55 mergesort-18038801.err

-rw-rw-r-- 1 lyin25 lyin25   51 Nov  9 13:55 fragmerge-18038803.err

-rw-rw-r-- 1 lyin25 lyin25  156 Nov  9 13:55 aligncheck-18038802.out

-rw-rw-r-- 1 lyin25 lyin25  117 Nov  9 14:20 fragmerge-18038803.out

-rw-rw-r-- 1 lyin25 lyin25  46K Nov  9 15:20 dedup-18038805.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:39 dup-split-109.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:39 dup-split-158.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:39 dup-split-13.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:39 dup-split-115.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:39 dup-split-186.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:39 dup-split-183.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:39 dup-split-176.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:39 dup-split-174.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:39 dup-split-152.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:39 dup-split-139.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:39 dup-split-138.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:39 dup-split-187.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:39 dup-split-180.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:39 dup-split-112.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:39 dup-split-5.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:39 dup-split-170.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:39 dup-split-111.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:39 dup-split-106.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:39 dup-split-147.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:39 dup-split-75.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:39 dup-split-25.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:39 dup-split-172.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:39 dup-split-90.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:39 dup-split-40.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:39 dup-split-20.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:39 dup-split-17.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:39 dup-split-155.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:39 dup-split-153.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:39 dup-split-149.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:39 dup-split-134.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:39 dup-split-68.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:39 dup-split-29.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:39 dup-split-178.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:39 dup-split-173.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:39 dup-split-171.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:39 dup-split-15.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:39 dup-split-130.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:39 dup-split-118.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:39 dup-split-1.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:39 dup-split-89.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:39 dup-split-53.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:39 dup-split-38.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:39 dup-split-123.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:39 dup-split-10.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:39 dup-split-41.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:39 dup-split-27.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:39 dup-split-24.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:39 dup-split-93.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:39 dup-split-52.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:39 dup-split-99.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:39 dup-split-2.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:39 dup-split-98.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:39 dup-split-7.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:39 dup-split-61.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:39 dup-split-50.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:39 dup-split-102.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:39 dup-split-95.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:39 dup-split-21.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:39 dup-split-11.out

-rw-rw-r-- 1 lyin25 lyin25   71 Nov  9 15:39 dup-split-195.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:39 dup-split-91.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:39 dup-split-160.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:39 dup-split-136.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:39 dup-split-189.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:39 dup-split-135.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:39 dup-split-129.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:39 dup-split-125.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:39 dup-split-120.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:39 dup-split-114.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:39 dup-split-107.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:39 dup-split-105.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:39 dup-split-182.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:39 dup-split-177.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:39 dup-split-145.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:39 dup-split-141.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:39 dup-split-131.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:39 dup-split-110.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:39 dup-split-193.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:39 dup-split-190.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:39 dup-split-168.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:39 dup-split-166.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:39 dup-split-117.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:39 dup-split-82.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:39 dup-split-185.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:39 dup-split-162.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:39 dup-split-84.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:39 dup-split-85.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:39 dup-split-12.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:39 dup-split-48.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:39 dup-split-36.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:39 dup-split-34.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:39 dup-split-51.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:39 dup-split-49.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:39 dup-split-43.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:39 dup-split-19.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:39 dup-split-59.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:39 dup-split-28.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:39 dup-split-22.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:39 dup-split-192.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:39 dup-split-96.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:39 dup-split-87.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:39 dup-split-58.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:39 dup-split-55.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:39 dup-split-39.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:39 dup-split-100.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:39 dup-split-83.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:39 dup-split-179.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:39 dup-split-175.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:39 dup-split-116.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:39 dup-split-184.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:39 dup-split-169.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:39 dup-split-188.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:39 dup-split-181.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:39 dup-split-161.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:39 dup-split-146.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:39 dup-split-144.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:39 dup-split-124.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:39 dup-split-108.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:39 dup-split-159.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:39 dup-split-143.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:39 dup-split-45.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:39 dup-split-4.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:39 dup-split-23.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:39 dup-split-163.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:39 dup-split-137.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:39 dup-split-126.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:39 dup-split-47.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:39 dup-split-3.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:39 dup-split-127.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:39 dup-split-101.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:39 dup-split-46.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:39 dup-split-44.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:39 dup-split-37.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:39 dup-split-33.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:39 dup-split-165.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:39 dup-split-154.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:39 dup-split-150.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:39 dup-split-132.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:39 dup-split-128.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:40 dup-split-16.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:40 dup-split-60.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:40 dup-split-194.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:40 dup-split-18.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:40 dup-split-156.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:40 dup-split-119.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:40 dup-split-113.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:40 dup-split-77.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:40 dup-split-64.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:40 dup-split-63.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:40 dup-split-62.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:40 dup-split-6.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:40 dup-split-57.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:40 dup-split-42.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:40 dup-split-26.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:40 dup-split-157.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:40 dup-split-151.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:40 dup-split-140.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:40 dup-split-14.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:40 dup-split-133.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:40 dup-split-122.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:40 dup-split-86.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:40 dup-split-80.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:40 dup-split-76.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:40 dup-split-74.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:40 dup-split-73.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:40 dup-split-72.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:40 dup-split-56.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:40 dup-split-35.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:40 dup-split-32.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:40 dup-split-30.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:40 dup-split-191.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:40 dup-split-167.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:40 dup-split-164.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:40 dup-split-148.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:40 dup-split-142.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:40 dup-split-121.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:40 dup-split-104.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:40 dup-split-94.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:40 dup-split-97.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:40 dup-split-92.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:40 dup-split-88.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:40 dup-split-81.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:40 dup-split-31.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:40 dup-split-103.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:40 dup-split-9.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:40 dup-split-70.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:40 dup-split-8.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:40 dup-split-66.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:40 dup-split-65.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:40 dup-split-71.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:40 dup-split-69.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:40 dup-split-67.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:40 dup-split-54.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:40 dup-split-78.out

-rw-rw-r-- 1 lyin25 lyin25   72 Nov  9 15:40 dup-split-79.out

-rw-rw-r-- 1 lyin25 lyin25   74 Nov  9 16:07 dup-split-0.out

-rw-rw-r-- 1 lyin25 lyin25   59 Nov  9 16:46 dup-merge.out

-rw-rw-r-- 1 lyin25 lyin25 1.5K Nov  9 16:51 dup-guard-trigger.out

-rw-rw-r-- 1 lyin25 lyin25   29 Nov  9 16:51 dedupguard-18038804.out

-rw-rw-r-- 1 lyin25 lyin25 1.4K Nov  9 16:51 post_dedup-18038806.out

-rw-rw-r-- 1 lyin25 lyin25    0 Nov  9 16:51 dupcheck-18038807.err

-rw-rw-r-- 1 lyin25 lyin25    0 Nov  9 16:53 dup-mail.err

-rw-rw-r-- 1 lyin25 lyin25  115 Nov  9 16:53 dup-mail.out

-rw-rw-r-- 1 lyin25 lyin25   20 Nov  9 18:42 dupcheck-a1667941836

-rw-rw-r-- 1 lyin25 lyin25  115 Nov  9 18:42 dupcheck-18038807.out

-rw-rw-r-- 1 lyin25 lyin25    0 Nov  9 18:42 dup-rm.err

-rw-rw-r-- 1 lyin25 lyin25    0 Nov  9 18:42 prestats-18038810.err

-rw-rw-r-- 1 lyin25 lyin25    0 Nov  9 18:42 merged30-18038809.err

-rw-rw-r-- 1 lyin25 lyin25    0 Nov  9 18:42 merged1-18038808.err

-rw-rw-r-- 1 lyin25 lyin25   58 Nov  9 18:42 dup-rm.out

-rw-rw-r-- 1 lyin25 lyin25   58 Nov  9 19:13 prestats-18038810.out

-rw-rw-r-- 1 lyin25 lyin25   29 Nov  9 19:39 merged30-18038809.out

-rw-rw-r-- 1 lyin25 lyin25   29 Nov  9 19:39 merged1-18038808.out

-rw-rw-r-- 1 lyin25 lyin25    0 Nov  9 19:39 bamrm-18038811.out

-rw-rw-r-- 1 lyin25 lyin25    0 Nov  9 19:39 bamrm-18038811.err

-rw-rw-r-- 1 lyin25 lyin25   58 Nov  9 19:39 stats-18038812.out

-rw-rw-r-- 1 lyin25 lyin25  134 Nov  9 19:39 stats-18038812.err

-rw-rw-r-- 1 lyin25 lyin25   58 Nov  9 19:59 stats30-18038813.out

-rw-rw-r-- 1 lyin25 lyin25  134 Nov  9 19:59 stats30-18038813.err

-rw-rw-r-- 1 lyin25 lyin25    0 Nov  9 19:59 mnd-18038814.err

-rw-rw-r-- 1 lyin25 lyin25   58 Nov  9 21:22 mnd-18038814.out

-rw-rw-r-- 1 lyin25 lyin25  194 Nov  9 21:22 fincln1-18038815.out

-rw-rw-r-- 1 lyin25 lyin25  119 Nov  9 21:22 fincln1-18038815.err

Thanks,

Lu

--

Lu Yin

Postdoctoral Research Scholar

Plant Genetics and Meiosis

School of Life Sciences, LSE 651

Arizona State University

---

From: 'Olga Dudchenko' via 3D Genomics <3d-ge...@googlegroups.com>

Sent: Thursday, December 1, 2022 4:04 PM

To view this discussion on the web visit https://groups.google.com/d/msgid/3d-genomics/3ba31924-b4dc-47c0-939f-8784a06ac919n%40googlegroups.com.

[Olga Dudchenko]

Hello Lu,

You need to look for actual reports in the err files. Do smth like tail debug/*err | less -S which would allow you to check the last 10 lines of all err files.

Olga

[Lu Yin]

Hi Olga!

This is what I have for .err files based on your suggested code:

[lyin25@agave3:/run010/debug]$ tail *.err | less -S

[M::mem_pestat] skip orientation FR as there are not enough pairs

[M::mem_pestat] skip orientation RF as there are not enough pairs

[M::mem_pestat] skip orientation RR as there are not enough pairs

[M::mem_process_seqs] Processed 153974 reads in 439.695 CPU sec, 32.046 real sec

[main] Version: 0.7.17-r1188

[main] CMD: bwa mem -SP5M -t 14 -R @RG\tID:reduced_Zingeria_biebersteiniana_OmniC_I1209_L2\tSM:HiC_sample\tPL:ILM\tLB:HiC_library /data/cchen347/ZingeriaProject2021-/HiC/jui

[main] Real time: 63087.553 sec; CPU: 865731.619 sec

==> bamrm-18038811.err <==

==> dupcheck-18038807.err <==

==> dup-mail.err <==

==> dup-rm.err <==

==> fincln1-18038815.err <==

/data/cchen347/ZingeriaProject2021-/HiC/juicer//scripts/check.sh: line 45: [: 1.51667e+08: integer expression expected

==> fragmerge-18038803.err <==

Currently Loaded Modulefiles:

  1) samtools/1.15.1

==> merge-18038800.err <==

real    175m19.697s

user    139m25.435s

sys     4m13.069s

==> merged1-18038808.err <==

==> merged30-18038809.err <==

==> mergesort-18038801.err <==

[bam_sort_core] merging from 96 files and 8 in-memory blocks...

6070.01user 2303.69system 1:24:53elapsed 164%CPU (0avgtext+0avgdata 17020380maxresident)k

1224763177inputs+536611875outputs (32055617major+264626297minor)pagefaults 0swaps

==> mnd-18038814.err <==

==> prestats-18038810.err <==

==> stats-18038812.err <==

Exception in thread "main" java.lang.RuntimeException: Unknown command: statistics

        at juicebox.tools.HiCTools.main(HiCTools.java:98)

==> stats30-18038813.err <==

Exception in thread "main" java.lang.RuntimeException: Unknown command: statistics

        at juicebox.tools.HiCTools.main(HiCTools.java:98)

This does not show my mnd-18038814.err file which is empty and fincln1-18038815.err which says 

/juicer//scripts/check.sh: line 45: [: 1.51667e+08: integer expression expected

Thanks for your reply! I'm looking forward to your reply!

Lu

--

Lu Yin

Postdoctoral Research Scholar

Plant Genetics and Meiosis

School of Life Sciences, LSE 651

Arizona State University

From: 'Olga Dudchenko' via 3D Genomics <3d-ge...@googlegroups.com>

Sent: Wednesday, December 7, 2022 4:35 PM

To view this discussion on the web visit https://groups.google.com/d/msgid/3d-genomics/5ca7f915-a4ba-41ba-bbc4-9012adf66b7fn%40googlegroups.com.

[Olga Dudchenko]

There are probably folks not he threads who can give you better advice about running Juicer, but I can see some unexpected reports from the alignment. Are your reads super-short or something? Not sure what line 45 is in specific version you are running. You can try to look yourself. Probably if [ $wctotal -ne $check2 ] So, it probably means that it fails to calculate one of the following

wctotal=`cat ${splitdir}/*_linecount.txt | awk '{sum+=$1}END{print sum/4}'`

check2=`cat ${splitdir}/*norm*res* | awk '{s2+=$2;}END{print s2}'`

You can try to run manually and see what happens.

 -Olga

[小韩]

Dear Olga!

 My name is Qiao Du, and I am a user of Juicer. I am writing to seek your help regarding an issue I encountered while using the juicer.

I kindly request your assistance in guiding me on how to rectify this problem. Firstly, I am using juicer 2. I cloned the github repo for Juicer. (git clone https://github.com/aidenlab/juicer.git)

My error output log file content is as follows:
[main] Version: 0.7.17-r1188
[main] CMD: bwa mem -K 320000000 -SP5M -t 90 -R @rg\tID:13271\tSM:HiC_sample\tPL:ILM\tLB:HiC_library /juicer_work/references/genomic.fna /juicer_work/work/splits/13271_R1.fastq.gz /juicer_work/work/splits/13271_R2.fastq.gz
[main] Real time: 29331.845 sec; CPU: 2567696.332 sec
[bam_sort_core] merging from 17 files and 90 in-memory blocks...
Picked up _JAVA_OPTIONS: -Xmx1024m -Xms1024m

Exception in thread "main" java.lang.RuntimeException: Unknown command: statistics
at juicebox.tools.HiCTools.main(HiCTools.java:98)

Picked up _JAVA_OPTIONS: -Xmx1024m -Xms1024m

Exception in thread "main" java.lang.RuntimeException: Unknown command: statistics
at juicebox.tools.HiCTools.main(HiCTools.java:98)

Picked up _JAVA_OPTIONS: -Xmx150000m -Xms150000m
Error while reading graphs file: java.io.FileNotFoundException: /juicer_work/work/aligned/inter_hists.m (No such file or directory)

I know this error is because of using an old version of Juicer Tools jar 1.9.9. However, I found that the file juicer/work/aligned/inter.hic is still being generated continuously, and other files have not been generated new content after the error is reported, such as "inter.txt". I hope you don't mind me asking you to make sure that my actions produce the correct results. Any help or suggestions you can provide would be greatly appreciated.

My requirement is just to get the  merged_nodups.txt file in order to run the next 3d-dna command. I found a solution in the discussion above,  so I used a command in juicer.sh to generate the merged_nodups.txt file.

According to my output log , the previous juicer command has successfully run the steps of alignment, merging and deduplication, so can I manually generate the txt file. Here is my command and files in folder /work/aligned/. Could these be correct? 

~/software/miniconda3/envs/juicer/bin/samtools view -@ 90 -O SAM -F 1024 /roach_data/dq//juicer/work/aligned/merged_dedup.bam | awk -v mnd=1 -f /roach_data/dq//juicer/scripts/common/sam_to_pre.awk > /roach_data/dq//juicer/work/aligned/merged_nodups.txt

 

（Among these files, the last update time of inter.txt and inter_30.txt files (Aug 7 17:16) is consistent with the time when the error was reported due to the version of juicer tools jar. The hic file is generated normally, but I stopped the juicer）

There are my input files:

My output log file content is as follows:
Using /juicer_work/restriction_sites/DpnII.txt as site file
(-: Looking for fastq files...fastq files exist
(-: Created /juicer_work/work/splits.
(-: Aligning files matching /juicer_work/work/fastq/_R.fastq*
to genome /juicer_work/references/genomic.fna with no fragment delimited maps.
Running command bwa mem -SP5M -t 90 -R @rg\tID:13271\tSM:HiC_sample\tPL:ILM\tLB:HiC_library /juicer_work/references/genomic.fna /juicer_work/work/splits/13271_R1.fastq.gz /juicer_work/work/splits/13271_R2.fastq.gz > /juicer_work/work/splits/13271.fastq.gz.sam
(-: Align of /juicer_work/work/splits/13271.fastq.gz.sam done successfully
(-: Finished sorting all sorted files into a single merge.
(-: Mark duplicates done successfully
Not including fragment map
Start preprocess
Writing header
Writing body

I have downloaded the latest version of juicer tools jar. I also noticed that the -S flag seems to re-run juicer on the previous basis. It takes a lot of time to run the task once, so I'm not sure if I need to re-run the task with -S "early" or "final". Or can I use the merged_nodups.txt generated by samtools above for the next 3d-dna operation?

I truly appreciate your willingness to help. May your days be filled with happiness!

Best regards,

Qiao Du

[Olga Dudchenko]

Hi Qiao,

I believe Juicer2 has the --assembly flag to generate the merged_nodups.txt file. This woudl invoke the line you ran directly:

samtools view $sthreadstring -O SAM -F 1024 $outputdir/merged_dedup.*am | awk -v mnd=1 -f ${juiceDir}/scripts/sam_to_pre.awk > ${outputdir}/merged_nodups.txt

to dump the old format from the bam. hic file will fail if there are a lot of pieces, hence the -S early/--assembly flag case.

Hope this helps.

Olga

[小韩]

Dear Olga!

Thank you for your reply!

Your answer helped me a lot!

Best regards,

Qiao

[Alicia]

Dear Olga,

My name is Alicia, and I am trying to use your software to scaffold a draft genome. I am using Juicer version 1.6, BWA 0.7.17-r1198-dirty, 128 threads, and Java version "1.8.0_121". I ran the script using the -S early option. I wanted to use 3-DNA, and I obtained the merged_nodups.txt file. In the log file, I can read the following:

(-: Pipeline successfully completed (-:
Run cleanup.sh to remove the splits directory

However, I have observed the following information in the log file:

Exception in thread "main" java.lang.RuntimeException: Unknown command: LibraryComplexity
        at juicebox.tools.HiCTools.main(HiCTools.java:99)

Could you please inform me if I should be worried about this error?

Thank you,

Best regards,

Alicia