64-bit 2bit files

25 views
Skip to first unread message

Jeffrey Rosenfeld

unread,
Aug 14, 2017, 4:21:08 PM8/14/17
to gen...@soe.ucsc.edu
Hi,


Is there a way to get a 64-bit 2bit file so that the size can be above
4GB? I was some talk about it on message boards, but have not seen the
file.


--
Thanks,

Jeffrey

------------------------------------------------------
Jeffrey Rosenfeld, Ph. D

Assistant Professor of Pathology and Laboratory Medicine
Rutgers Cancer Institute of New Jersey

jeffrey....@rutgers.edu
732-235-8565 (voice)

Jairo Navarro Gonzalez

unread,
Aug 14, 2017, 6:16:06 PM8/14/17
to Jeffrey Rosenfeld, gen...@soe.ucsc.edu

Hello Jeffrey,

Thank you for using the UCSC Genome Browser and your question about creating 2bit files.

You can create a 64-bit 2bit file by using the faToTwoBit utility:

faToTwoBit - Convert DNA from fasta to 2bit format
usage:
   faToTwoBit in.fa [in2.fa in3.fa ...] out.2bit
options:
   -long          use 64-bit offsets for index.   Allow for twoBit to contain more than 4Gb of sequence. 
                  NOT COMPATIBLE WITH OLDER CODE.
   -noMask        Ignore lower-case masking in fa file.
   -stripVersion  Strip off version number after '.' for GenBank accessions.
   -ignoreDups    Convert first sequence only if there are duplicate sequence
                  names.  Use 'twoBitDup' to find duplicate sequences.

The -long flag can be used to create files larger than 4Gb.

You can download the faToTwoBit utility from our utilities directory.

I hope this is helpful. If you have any further questions, please reply to gen...@soe.ucsc.edu.
All messages sent to that address are archived on a publicly-accessible Google Groups forum.
If your question includes sensitive data, you may send it instead to genom...@soe.ucsc.edu.

Jairo Navarro 
UCSC Genomics Institute



--

---
You received this message because you are subscribed to the Google Groups "UCSC Genome Browser Public Support" group.
To unsubscribe from this group and stop receiving emails from it, send an email to genome+un...@soe.ucsc.edu.
To post to this group, send email to gen...@soe.ucsc.edu.
Visit this group at https://groups.google.com/a/soe.ucsc.edu/group/genome/.
To view this discussion on the web visit https://groups.google.com/a/soe.ucsc.edu/d/msgid/genome/7011afaf-bbb7-a41f-36f3-1824b395a456%40cinj.rutgers.edu.
For more options, visit https://groups.google.com/a/soe.ucsc.edu/d/optout.

Jeffrey Rosenfeld

unread,
Aug 15, 2017, 11:43:33 AM8/15/17
to Jairo Navarro Gonzalez, gen...@soe.ucsc.edu

Hi,


I have been able to create large 2bit files, but when I try to blat against them, I get a malloc error.  This machine has a few hundred GB of RAM, so I don't think that is the cause of the error.  What do you suggest?

*** Error in `blat': malloc(): memory corruption: 0x000000000402e168 ***
======= Backtrace: =========
/lib/x86_64-linux-gnu/libc.so.6(+0x777e5)[0x7fe811ef17e5]
/lib/x86_64-linux-gnu/libc.so.6(+0x8213e)[0x7fe811efc13e]
/lib/x86_64-linux-gnu/libc.so.6(__libc_malloc+0x54)[0x7fe811efe184]
blat[0x431cf3]
blat[0x431e6e]
blat[0x40fd16]
blat[0x40fee9]
blat[0x412f36]
blat[0x40b77e]
blat[0x40b80e]
blat[0x40b96c]
blat[0x40bbd7]
blat[0x40bf1d]
blat[0x40c42c]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0)[0x7fe811e9a830]
blat[0x40ade9]
======= Memory map: ========
00400000-0062d000 r-xp 00000000 ca:01 44040194                           /home/ubuntu/bin/blat
0082d000-0085c000 rw-p 0022d000 ca:01 44040194                           /home/ubuntu/bin/blat
0085c000-00874000 rw-p 00000000 00:00 0
01eed000-124cba000 rw-p 00000000 00:00 0                                 [heap]
7fe75c000000-7fe75c021000 rw-p 00000000 00:00 0
7fe75c021000-7fe760000000 ---p 00000000 00:00 0
7fe761226000-7fe810a18000 rw-p 00000000 00:00 0
7fe811003000-7fe811019000 r-xp 00000000 ca:01 396802                     /lib/x86_64-linux-gnu/libgcc_s.so.1
7fe811019000-7fe811218000 ---p 00016000 ca:01 396802                     /lib/x86_64-linux-gnu/libgcc_s.so.1
7fe811218000-7fe811219000 rw-p 00015000 ca:01 396802                     /lib/x86_64-linux-gnu/libgcc_s.so.1
7fe811219000-7fe811230000 r-xp 00000000 ca:01 398138                     /lib/x86_64-linux-gnu/libresolv-2.23.so
7fe811230000-7fe811430000 ---p 00017000 ca:01 398138                     /lib/x86_64-linux-gnu/libresolv-2.23.so
7fe811430000-7fe811431000 r--p 00017000 ca:01 398138                     /lib/x86_64-linux-gnu/libresolv-2.23.so
7fe811431000-7fe811432000 rw-p 00018000 ca:01 398138                     /lib/x86_64-linux-gnu/libresolv-2.23.so
7fe811432000-7fe811434000 rw-p 00000000 00:00 0
7fe811434000-7fe811437000 r-xp 00000000 ca:01 396813                     /lib/x86_64-linux-gnu/libkeyutils.so.1.5
7fe811437000-7fe811636000 ---p 00003000 ca:01 396813                     /lib/x86_64-linux-gnu/libkeyutils.so.1.5
7fe811636000-7fe811637000 r--p 00002000 ca:01 396813                     /lib/x86_64-linux-gnu/libkeyutils.so.1.5
7fe811637000-7fe811638000 rw-p 00003000 ca:01 396813                     /lib/x86_64-linux-gnu/libkeyutils.so.1.5
7fe811638000-7fe811642000 r-xp 00000000 ca:01 5344                       /usr/lib/x86_64-linux-gnu/libkrb5support.so.0.1
7fe811642000-7fe811841000 ---p 0000a000 ca:01 5344                       /usr/lib/x86_64-linux-gnu/libkrb5support.so.0.1
7fe811841000-7fe811842000 r--p 00009000 ca:01 5344                       /usr/lib/x86_64-linux-gnu/libkrb5support.so.0.1
7fe811842000-7fe811843000 rw-p 0000a000 ca:01 5344                       /usr/lib/x86_64-linux-gnu/libkrb5support.so.0.1
7fe811843000-7fe811846000 r-xp 00000000 ca:01 396787                     /lib/x86_64-linux-gnu/libcom_err.so.2.1
7fe811846000-7fe811a45000 ---p 00003000 ca:01 396787                     /lib/x86_64-linux-gnu/libcom_err.so.2.1
7fe811a45000-7fe811a46000 r--p 00002000 ca:01 396787                     /lib/x86_64-linux-gnu/libcom_err.so.2.1
7fe811a46000-7fe811a47000 rw-p 00003000 ca:01 396787                     /lib/x86_64-linux-gnu/libcom_err.so.2.1
7fe811a47000-7fe811a4a000 r-xp 00000000 ca:01 398135                     /lib/x86_64-linux-gnu/libdl-2.23.so
7fe811a4a000-7fe811c49000 ---p 00003000 ca:01 398135                     /lib/x86_64-linux-gnu/libdl-2.23.so
7fe811c49000-7fe811c4a000 r--p 00002000 ca:01 398135                     /lib/x86_64-linux-gnu/libdl-2.23.so
7fe811c4a000-7fe811c4b000 rw-p 00003000 ca:01 398135                     /lib/x86_64-linux-gnu/libdl-2.23.so
7fe811c4b000-7fe811c77000 r-xp 00000000 ca:01 5341                       /usr/lib/x86_64-linux-gnu/libk5crypto.so.3.1
7fe811c77000-7fe811e76000 ---p 0002c000 ca:01 5341                       /usr/lib/x86_64-linux-gnu/libk5crypto.so.3.1
7fe811e76000-7fe811e78000 r--p 0002b000 ca:01 5341                       /usr/lib/x86_64-linux-gnu/libk5crypto.so.3.1
7fe811e78000-7fe811e79000 rw-p 0002d000 ca:01 5341                       /usr/lib/x86_64-linux-gnu/libk5crypto.so.3.1
7fe811e79000-7fe811e7a000 rw-p 00000000 00:00 0
7fe811e7a000-7fe81203a000 r-xp 00000000 ca:01 398146                     /lib/x86_64-linux-gnu/libc-2.23.so
7fe81203a000-7fe81223a000 ---p 001c0000 ca:01 398146                     /lib/x86_64-linux-gnu/libc-2.23.so
7fe81223a000-7fe81223e000 r--p 001c0000 ca:01 398146                     /lib/x86_64-linux-gnu/libc-2.23.so
7fe81223e000-7fe812240000 rw-p 001c4000 ca:01 398146                     /lib/x86_64-linux-gnu/libc-2.23.so
7fe812240000-7fe812244000 rw-p 00000000 00:00 0
7fe812244000-7fe81225c000 r-xp 00000000 ca:01 398129                     /lib/x86_64-linux-gnu/libpthread-2.23.so
7fe81225c000-7fe81245b000 ---p 00018000 ca:01 398129                     /lib/x86_64-linux-gnu/libpthread-2.23.so
7fe81245b000-7fe81245c000 r--p 00017000 ca:01 398129                     /lib/x86_64-linux-gnu/libpthread-2.23.so
7fe81245c000-7fe81245d000 rw-p 00018000 ca:01 398129                     /lib/x86_64-linux-gnu/libpthread-2.23.so
7fe81245d000-7fe812461000 rw-p 00000000 00:00 0
7fe812461000-7fe812569000 r-xp 00000000 ca:01 398141                     /lib/x86_64-linux-gnu/libm-2.23.so
7fe812569000-7fe812768000 ---p 00108000 ca:01 398141                     /lib/x86_64-linux-gnu/libm-2.23.so
7fe812768000-7fe812769000 r--p 00107000 ca:01 398141                     /lib/x86_64-linux-gnu/libm-2.23.so
7fe812769000-7fe81276a000 rw-p 00108000 ca:01 398141                     /lib/x86_64-linux-gnu/libm-2.23.so
7fe81276a000-7fe812783000 r-xp 00000000 ca:01 396873                     /lib/x86_64-linux-gnu/libz.so.1.2.8
7fe812783000-7fe812982000 ---p 00019000 ca:01 396873                     /lib/x86_64-linux-gnu/libz.so.1.2.8
7fe812982000-7fe812983000 r--p 00018000 ca:01 396873                     /lib/x86_64-linux-gnu/libz.so.1.2.8
7fe812983000-7fe812984000 rw-p 00019000 ca:01 396873                     /lib/x86_64-linux-gnu/libz.so.1.2.8
7fe812984000-7fe8129a8000 r-xp 00000000 ca:01 396847                     /lib/x86_64-linux-gnu/libpng12.so.0.54.0
7fe8129a8000-7fe812ba7000 ---p 00024000 ca:01 396847                     /lib/x86_64-linux-gnu/libpng12.so.0.54.0
7fe812ba7000-7fe812ba8000 r--p 00023000 ca:01 396847                     /lib/x86_64-linux-gnu/libpng12.so.0.54.0
7fe812ba8000-7fe812ba9000 rw-p 00024000 ca:01 396847                     /lib/x86_64-linux-gnu/libpng12.so.0.54.0
7fe812ba9000-7fe812c6c000 r-xp 00000000 ca:01 5343                       /usr/lib/x86_64-linux-gnu/libkrb5.so.3.3
7fe812c6c000-7fe812e6c000 ---p 000c3000 ca:01 5343                       /usr/lib/x86_64-linux-gnu/libkrb5.so.3.3
7fe812e6c000-7fe812e79000 r--p 000c3000 ca:01 5343                       /usr/lib/x86_64-linux-gnu/libkrb5.so.3.3
7fe812e79000-7fe812e7b000 rw-p 000d0000 ca:01 5343                       /usr/lib/x86_64-linux-gnu/libkrb5.so.3.3
7fe812e7b000-7fe812ea1000 r-xp 00000000 ca:01 398124                     /lib/x86_64-linux-gnu/ld-2.23.so
7fe812ebf000-7fe81308d000 rw-p 00000000 00:00 0
7fe81309d000-7fe8130a0000 rw-p 00000000 00:00 0
7fe8130a0000-7fe8130a1000 r--p 00025000 ca:01 398124                     /lib/x86_64-linux-gnu/ld-2.23.so
7fe8130a1000-7fe8130a2000 rw-p 00026000 ca:01 398124                     /lib/x86_64-linux-gnu/ld-2.23.so
7fe8130a2000-7fe8130a3000 rw-p 00000000 00:00 0
7ffe59c42000-7ffe59c63000 rw-p 00000000 00:00 0                          [stack]
7ffe59dc7000-7ffe59dc9000 r--p 00000000 00:00 0                          [vvar]
7ffe59dc9000-7ffe59dcb000 r-xp 00000000 00:00 0                          [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]
Aborted (core dumped)

-Jeffrey

Brian Lee

unread,
Aug 16, 2017, 6:53:41 PM8/16/17
to Jeffrey Rosenfeld, Jairo Navarro Gonzalez, gen...@soe.ucsc.edu
Dear Jeffrey,

Thank you for reporting this experience. We have had some success replicating this error and we are currently investigating it further and plan to write back when we have more information.

Thank you again for using and helping improve the UCSC Genome Browser by sharing this information. If you have any further questions, please reply to gen...@soe.ucsc.edu. All messages sent to that address are archived on a publicly-accessible Google Groups forum. If your question includes sensitive data, you may send it instead to genom...@soe.ucsc.edu.

Brian Lee
UCSC Genomics Institute

Jeffrey Rosenfeld

unread,
Aug 18, 2017, 11:43:08 AM8/18/17
to Brian Lee, Jairo Navarro Gonzalez, gen...@soe.ucsc.edu

Hi Brian,


Do you think that you will be able to get a quick fix for this?


-Jeffrey

Matthew Speir

unread,
Aug 22, 2017, 11:18:54 AM8/22/17
to Jeffrey Rosenfeld, Brian Lee, Jairo Navarro Gonzalez, gen...@soe.ucsc.edu

Hi Jeffrey,

We are still looking into this issue. Unfortunately, we don't have an estimate on when it may be fixed. Thank you, again, for reporting it to us.



If you have any further questions, please reply to gen...@soe.ucsc.edu. All messages sent to that address are archived on a publicly-accessible Google Groups forum. If your question includes sensitive data, you may send it instead to genom...@soe.ucsc.edu.

Matthew Speir
UCSC Genome Bioinformatics Group

Reply all
Reply to author
Forward
0 new messages