SunPy Maps Question

[Ellis Avallone]

Hi all,

I'm new to sunpy/solar data analysis/working with large datasets, and I've run into this error

OSError: [Errno 24] Too many open files

while trying to create a mapcube. I was wondering if creating a map/mapcube leaves files open, and if so, how would I close them so I can work with another set of data?

Thank you,

Ellis Avallone

[Shih, Albert Y. (GSFC-6710)]

Hi, Ellis,

     The default behavior is that a FITS file is opened via memory mapping, so the data array in the Map object keeps the file handle open for as long as that data array exists.  Thus, yes, creating many Maps will eventually hit the operating-system cap on the number of files that can be open at once.  See issue #1745 on our repository – https://github.com/sunpy/sunpy/issues/1745 – for more discussions about the use of memory mapping.

     In theory, it should be possible to release a file handle by replace the memory-mapped data array with a non-memory-mapped copy of itself, but I haven’t tested that.  There was a preliminary implementation to allow a user to disable memory mapping when creating Maps, but it didn’t work correctly, and I don’t know what obstacles still need to be overcome.  Stuart?

 

Albert

--
You received this message because you are subscribed to the Google Groups "SunPy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sunpy+un...@googlegroups.com.
To post to this group, send email to su...@googlegroups.com.
Visit this group at https://groups.google.com/group/sunpy.
For more options, visit https://groups.google.com/d/optout.

[Steven Christe]

I think perhaps the easiest solution would be to remember to delete maps you are not using by using the del command on them. Can someone confirm that this would work and would thus solve your problem?

Sent from my iPhone

[Shih, Albert Y. (GSFC-6710)]

Hi, Steven,

     Yes, deleting Map or MapCube instances that are no longer being used does allow the associated files to close.  However, that is (presumably) not helpful for the issue here, as Ellis described trying to create a single MapCube for a large dataset.

 

Albert

[Shih, Albert Y. (GSFC-6710)]

Hi, all,

     It turns out there’s a much better answer for this whole too-many-open-files issue.  We use Astropy’s FITS reader, and Astropy provides a configuration option to enable/disable memory mapping.  To disable memory mapping, which then allows FITS files to close after the data is read in, go to your Astropy configuration file (typically `$HOME/.astropy/config/astropy.cfg`) and uncomment the line `# use_memmap = True` and change it to `use_memmap = False`.

 

Albert