Preserving file size

[Phil Morefield]

I'm reading two netCDF files as Cubes, calculating the average, and then saving the result.

The two input files are each 1.8 GB. The output file is 5.7 GB, which is quite unexpected. 

I've check the shape, data type, fill value, etc. and I don't find any differences. I've confirmed that the output data are indeed averages.

Any ideas on why the output file is so much larger? 

[Tony Phillips - UKRI BAS]

Is it possible that the input netCDF files are stored using netCDF compression but that the output file is not? Iris will only write compressed netCDF output if you explicitly ask it to by including zlib=True in the call to iris.save. In my experience, netCDF file sizes generally match what I’d expect in terms of shape times data type size in the absence of compression, so that’s what is making me wonder if this is the cause. You can check whether the input files use compression by running ncdump -hs on them and checking for the presence of the attribute _DeflateLevel in the output. Of course, you can also check whether the 1.8 GB matches what you would expect based on the data shape and data type size in the input files. These are all just random thoughts - I hope this is useful.

--
You received this message because you are subscribed to the Google Groups "SciTools (iris, cartopy, cf_units, etc.) - https://github.com/scitools" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scitools-iri...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/scitools-iris/08bd68f7-acb1-4b5d-88ec-d330e36ee50cn%40googlegroups.com.

This email and any attachments are intended solely for the use of the named recipients. If you are not the intended recipient you must not use, disclose, copy or distribute this email or any of its attachments and should notify the sender immediately and delete this email from your system. UK Research and Innovation (UKRI) has taken every reasonable precaution to minimise risk of this email or any attachments containing viruses or malware but the recipient should carry out its own virus and malware checks before opening the attachments. UKRI does not accept any liability for any losses or damages which the recipient may sustain due to presence of any viruses. Opinions, conclusions or other information in this message and attachments that are not related directly to UKRI business are solely those of the author and do not represent the views of UKRI.

[Phil Morefield]

>Is it possible that the input netCDF files are stored using netCDF compression but that the output file is not? Iris will only write compressed netCDF output if you explicitly ask it to by including zlib=True in the call to iris.save.

This was the issue. Adding zlib=True gave me an output of 1.8 GB as expected. Thanks for your help!