> I have a similar problem that many new python users might encounter. I > would really appreciate if you could help me fix the error. > I have a big text file with size more than 2GB. It turned out memory > error when reading in this file. Here is my python script, the error > occurred at line -- self.fh.readlines().
[snip code] Your 'error' is that you're running it on a computer with insufficient memory.
phoebe> I have a big text file with size more than 2GB. It turned out phoebe> memory error when reading in this file. Here is my python phoebe> script, the error occurred at line -- self.fh.readlines().
Don't do that. The problem is that you are trying to read the entire file into memory. Learn to operate a line (or a few lines) at a time. Try something like:
a = open("/home/sservice/nfbc/GenoData/CompareCalls3.diff") for line in a: do your per-line work here
-- Skip Montanaro - s...@pobox.com - http://www.smontanaro.net/ when i wake up with a heart rate below 40, i head right for the espresso machine. -- chaos @ forums.usms.org
> I have a similar problem that many new python users might encounter. I would > really appreciate if you could help me fix the error. > I have a big text file with size more than 2GB. It turned out memory error > when reading in this file. Here is my python script, the error occurred at > line -- self.fh.readlines().
> start=0 > snp=0 > for i in range(lfile): > if (i==0): continue > after=a.lines[i].split() > before=a.lines[i-1].split() > if (before[0]==after[0]): > if (i!=(lfile-1)): continue > else: > end=lfile-1 > myfun(before[0],start,end) > snp=snp+1 > else: > end=i-1 > myfun(before[0],start,end) > snp=snp+1 > start=i > if (i ==(lfile-1)): > myfun(after[0],start,start) > snp=snp+1
> result.close()
> sincerely, phoebe
Others have pointed out that you have too little memory for a 2gig data structure. If you're running on a 32bit system, chances are it won't matter how much memory you add, a process is limited to 4gb, and the OS typically takes about half of it, your code and other data takes some, and you don't have 2gig left. A 64 bit version of Python, running on a 64bit OS, might be able to "just work."
Anyway, loading the whole file into a list is seldom the best answer, except for files under a meg or so. It's usually better to process the file in sequence. It looks like you're also making slices of that data, so they could potentially be pretty big as well.
If you can be assured that you only need the current line and the previous two (for example), then you can use a list of just those three, and delete the oldest one, and add a new one to that list each time through the loop.
Or, you can add some methods to that 'textfile' class that fetch a line by index. Brute force, you could pre-scan the file, and record all the file offsets for the lines you find, rather than storing the actual line. So you still have just as big a list, but it's a list of integers. Then when somebody calls your method, he passes an integer, and you return the particular line. A little caching for performance, and you're good to go.
Anyway, if you organize it that way, you can code the rest of the module to not care whether the whole file is really in memory or not.
BTW, you should derive all your classes from something. If nothing else, use object. class textfile(object):
> BTW, you should derive all your classes from something. If nothing > else, use object. > class textfile(object):
Just out of curiousity... why is that? I've been coding in Python for a long time, and I never derive my base classes. What's the advantage to deriving them?
2009/7/13 Aaron Scott <aaron.hildebra...@gmail.com>:
>> BTW, you should derive all your classes from something. If nothing >> else, use object. >> class textfile(object):
> Just out of curiousity... why is that? I've been coding in Python for > a long time, and I never derive my base classes. What's the advantage > to deriving them?
On Mon, Jul 13, 2009 at 2:51 PM, Vilya Harvey<vilya.har...@gmail.com> wrote: > 2009/7/13 Aaron Scott <aaron.hildebra...@gmail.com>: >>> BTW, you should derive all your classes from something. If nothing >>> else, use object. >>> class textfile(object):
>> Just out of curiousity... why is that? I've been coding in Python for >> a long time, and I never derive my base classes. What's the advantage >> to deriving them?
Note that Python 3.0 makes explicitly subclassing `object` unnecessary since it removes old-style classes; a class that doesn't explicitly subclass anything will implicitly subclass `object`.
On Mon, 13 Jul 2009 14:20:13 -0700, Aaron Scott wrote: >> BTW, you should derive all your classes from something. If nothing >> else, use object. >> class textfile(object):
> Just out of curiousity... why is that? I've been coding in Python for a > long time, and I never derive my base classes. What's the advantage to > deriving them?
"Old style" classes (those whose base classes aren't derived from anything) have a few disadvantages: