Re: Memory error due to big input file
MRAB
> Hi All,
>
> I have a similar problem that many new python users might encounter. I
> would really appreciate if you could help me fix the error.
> I have a big text file with size more than 2GB. It turned out memory
> error when reading in this file. Here is my python script, the error
> occurred at line -- self.fh.readlines().
>
[snip code]
Your 'error' is that you're running it on a computer with insufficient
memory.
sk...@pobox.com
phoebe> I have a big text file with size more than 2GB. It turned out
phoebe> memory error when reading in this file. Here is my python
phoebe> script, the error occurred at line -- self.fh.readlines().
phoebe> import math
phoebe> import time
phoebe> class textfile:
phoebe> def __init__(self,fname):
phoebe> self.name=fname
phoebe> self.fh=open(fname)
phoebe> self.fh.readline()
phoebe> self.lines=self.fh.readlines()
Don't do that. The problem is that you are trying to read the entire file
into memory. Learn to operate a line (or a few lines) at a time. Try
something like:
a = open("/home/sservice/nfbc/GenoData/CompareCalls3.diff")
for line in a:
do your per-line work here
--
Skip Montanaro - sk...@pobox.com - http://www.smontanaro.net/
when i wake up with a heart rate below 40, i head right for the espresso
machine. -- chaos @ forums.usms.org
Dave Angel
> Hi All,
>
> I have a similar problem that many new python users might encounter. I would
> really appreciate if you could help me fix the error.
> when reading in this file. Here is my python script, the error occurred at
> line -- self.fh.readlines().
>
> import math
> import time
>
> class textfile:
> def __init__(self,fname):
> self.name=fname
> self.fh=open(fname)
> self.fh.readline()
> self.lines=self.fh.readlines()
>
> a=textfile("/home/sservice/nfbc/GenoData/CompareCalls3.diff")
>
> lfile=len(a.lines)
>
> def myfun(snp,start,end):
> subdata=a.lines[start:end+1]
> NEWmiss=0
> OLDmiss=0
> DIFF=0
> for row in subdata:
> k=row.split()
> if (k[3]=="0/0") & (k[4]!="0/0"):
> NEWmiss=NEWmiss+1
> elif (k[3]!="0/0") & (k[4]=="0/0"):
> OLDmiss=OLDmiss+1
> elif (k[3]!="0/0") & (k[4]!="0/0"):
> DIFF=DIFF+1
> result.write(snp+" "+str(NEWmiss)+" "+str(OLDmiss)+" "+str(DIFF)+"\n")
>
> result=open("Summary_noLoop_diff3.txt","w")
> result.write("SNP NEWmiss OLDmiss DIFF\n")
>
> start=0
> snp=0
> for i in range(lfile):
> if (i==0): continue
> after=a.lines[i].split()
> before=a.lines[i-1].split()
> if (before[0]==after[0]):
> if (i!=(lfile-1)): continue
> else:
> end=lfile-1
> myfun(before[0],start,end)
> snp=snp+1
> else:
> end=i-1
> myfun(before[0],start,end)
> snp=snp+1
> start=i
> if (i ==(lfile-1)):
> myfun(after[0],start,start)
> snp=snp+1
>
> result.close()
>
> sincerely, phoebe
>
>
Others have pointed out that you have too little memory for a 2gig data
structure. If you're running on a 32bit system, chances are it won't
matter how much memory you add, a process is limited to 4gb, and the OS
typically takes about half of it, your code and other data takes some,
and you don't have 2gig left. A 64 bit version of Python, running on a
64bit OS, might be able to "just work."
Anyway, loading the whole file into a list is seldom the best answer,
except for files under a meg or so. It's usually better to process the
file in sequence. It looks like you're also making slices of that data,
so they could potentially be pretty big as well.
If you can be assured that you only need the current line and the
previous two (for example), then you can use a list of just those three,
and delete the oldest one, and add a new one to that list each time
through the loop.
Or, you can add some methods to that 'textfile' class that fetch a line
by index. Brute force, you could pre-scan the file, and record all the
file offsets for the lines you find, rather than storing the actual
line. So you still have just as big a list, but it's a list of
integers. Then when somebody calls your method, he passes an integer,
and you return the particular line. A little caching for performance,
and you're good to go.
Anyway, if you organize it that way, you can code the rest of the module
to not care whether the whole file is really in memory or not.
BTW, you should derive all your classes from something. If nothing
else, use object.
class textfile(object):
Aaron Scott
> else, use object.
> class textfile(object):
Just out of curiousity... why is that? I've been coding in Python for
a long time, and I never derive my base classes. What's the advantage
to deriving them?
Vilya Harvey
class Foo:
uses the old object model.
class Foo(object):
uses the new object model.
See http://docs.python.org/reference/datamodel.html (specifically
section 3.3) for details of the differences.
Vil.
Chris Rebert
> 2009/7/13 Aaron Scott <aaron.hi...@gmail.com>:
>
> uses the old object model.
>
> class Foo(object):
>
> uses the new object model.
>
> See http://docs.python.org/reference/datamodel.html (specifically
> section 3.3) for details of the differences.
Note that Python 3.0 makes explicitly subclassing `object` unnecessary
since it removes old-style classes; a class that doesn't explicitly
subclass anything will implicitly subclass `object`.
Cheers,
Chris
--
http://blog.rebertia.com
Steven D'Aprano
"Old style" classes (those whose base classes aren't derived from
anything) have a few disadvantages:
(1) Properties don't work correctly:
>>> class Parrot: # old-style class
... def __init__(self):
... self._x = 3
... def _setter(self, value):
... self._x = value
... def _getter(self):
... print "Processing ..."
... return self._x + 1
... x = property(_getter, _setter)
...
>>> p = Parrot()
>>> p.x
Processing ...
4
>>> p.x = 2
>>> p.x
2
In general, anything that uses the descriptor protocol, not just
property, will fail to work correctly with old-style classes.
(2) Classes using multiple inheritance with diamond-shaped inheritance
will be broken.
(3) __slots__ is just an attribute.
(4) super() doesn't work.
And, depending on whether you consider this a disadvantage or an
advantage:
(5) Special methods like __len__ can be over-ridden on the instance, not
just the class:
>>> class K:
... def __len__(self):
... return 0
...
>>> k = K()
>>> len(k)
0
>>> k.__len__ = lambda : 42
>>> len(k)
42
In their favour:
(1) Less typing.
(2) If you're not using descriptors, including property(), or multiple
inheritance with diamond diagrams, they work fine.
(3) They're (apparently) a tiny bit faster.
--
Steven