"Too many open files" error

Caroline Alexander

unread,

Apr 22, 2016, 12:30:13 PM4/22/16

to SunPy

Hi everyone,

I was wondering if anyone has encountered (and fixed!) an issue I'm having with SunPy. I have a directory with 1600 AIA fits files (these have already been subarrayed so I don't think it's a memory issue) and I'm using:

dirname='AIA304files_b'

mc=list()

for filename in sorted(os.listdir(dirname)):

if '.fits' in filename:

file=os.path.join(dirname,filename)

print file

md=sunpy.map.Map(file)

mc.append(md)

But after 1015 files it crashes out and says "OSError: [Errno 24] Too many open files". I'm using a mac and my colleagues on Linux have encountered the same problem. The code works fine if you use less files.

The issue seems to be with the md=sunpy.map.Map(file) line.... maybe it's unable to close the files properly??

Any advice or thoughts would be appreciated as I'm hoping to use SunPy on large file numbers.

Cheers,

Caroline

Jack Ireland

unread,

Apr 22, 2016, 12:35:38 PM4/22/16

to su...@googlegroups.com

This can be fixed by changing defaults on your system. Look up the man page for ulimit on your OS. You can change the number of open files there.

Although that will allow you to do what you want to do, there may be a bug somewhere in SunPy - perhaps files are not being closed properly???

--
You received this message because you are subscribed to the Google Groups "SunPy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sunpy+un...@googlegroups.com.
To post to this group, send email to su...@googlegroups.com.
Visit this group at https://groups.google.com/group/sunpy.
For more options, visit https://groups.google.com/d/optout.