memory error with bedGraphToBigWig when converting a huge bedGraph file to bigwig

[David]

Hello,

I am trying to convert a huge bedGraph file (59GB ) to bigwig file for loading into UCSC genome browser using bedGraphToBigWig.   An error message,  "needLargeMem: trying to allocate 28413240288 bytes (limit: 17179869184) " , always showed up after it ran about 1 hour, when a bigwig file of 6.2GB was generated.  I followed the suggestions at https://lists.soe.ucsc.edu/pipermail/genome/2011-March/025337.html,  and increased the maximum memory to 64 GB by adding a factor of 4 to the line "static size_t maxAlloc = ..." in the file src/lib/memalloc.c. After re-compiling the bedGraphToBigWig command, I ran it again. However, it stopped at the same position with the same error message. Do you guys have any suggestions how to solve this problem.

Thanks very much. Really appreciate it.

-- 

Zhiguang "David" Li, Ph.D.

Laboratory of Human Environmental Epigenomes

Department of Environmental Health Sciences

The Epigenetic Center

Bloomberg School of Public Health

Johns Hopkins University

615 N. Wolfe St., W7710, Baltimore, MD 21205

Phone: 410-614-5049

[Hiram Clawson]

Good Morning David:

This isn't a problem in the source code for the program. You do not
need to modify the code for this type of error.

The error is from your memory limit settings in your login shell.
If you are working with a bash shell, see if this command indicates
a memory limitation:
$ ulimit -1

Or, in csh/tcsh, this command:
$ limit

to discover the memory limits in your shell environment. You can
increase these limits with these same commands. For example, in
csh/tcsh, raise your limit to 160Gb:
$ limit datasize 163840m
$ limit vmemoryuse 163840m

or in bash:

$ ulimit -d 163840000
$ ulimit -v 163840000

I'm curious, how many lines does your 59Gb bedGraph file contain ?
See also: http://genomewiki.ucsc.edu/index.php/Big_file_converters

--Hiram

[Hiram Clawson]

The bash test command should read:

$ ulimit -a

[David]

Hiram,

Thanks very much for your response.

After playing around it for quite a while, I finally figured out it was
indeed the memory limit set at src/lib/memalloc.c that caused this
problem. I added a factor of 4, let it become "static size_t maxAlloc =
Š*4;". Everything works now. The trick is that after making the above
change, you have to go to the directory "src/lib", and type "make"
command. Then, go to "src/utils/bedGraphToBigWig" to recompile the
bedGraphToBigWig program. Everything would work perfectly and take the
changes you have made.

Using "ulimit -a", I found my max memory limit is 56GB on this machine. It
is enough for processing the 59GB bedGraph file. The peak memory usage is
about 20GB.

The 59GB file has 2,367,330,140 lines.


David