potential bug introduced with importcalc.py

[jaime]

Hello, 

My team uses openquake and frequently needs to store the HDF5 files elsewhere in remote storage (shared drive) so that more than one person can access calculation results.

It appears that the importcalc.py file has been updated not too long ago, but in doing so, it removes the ability to import calculations from a remote filepath. 

e.g., 

     path = 'full path to hdf5 file'
     from openquake.commands import importcalc

     importcalc.main(path)

     OSError: File not found: C:\[userdirectory]\oqdata\calc_987.hdf5

The same error occurs with the console usage, e.g.,

    oq importcalc "full path the filename" 

I believe this is a problem with line 39, where the path gets over written by the calc_id number. Maybe it's not a big but a feature, but it would require us to copy the hdf5 files to our own oq_data folders, which can be quite space intensive. 

A way around this is to use the old import_calc.py file, but that's less ideal. 

Thanks for reading!

[Michele Simionato]

Can you find out the pull request that broke the compatibility? I am willing to restore the old behavior if possible.

         Michele

[jaime]

Thanks for your help!

It looks like it's changed a few times since v3.16.7, which was the version we were using for importcalc.py. 

There are several changes to the file, so I'm not sure which ones should stay to maintain functionality for full paths.

This commit converted to the calc_id path to an integer (older change):  

SHA: ec286c58ac820e983e2ac01709a9377ab0d5d65b

commit: ec286c5

This commit adds the calc_id integer to the datadir path (newer change): 

full SHA:  b2af31f6c1ad02700480f1a426426e15a79e68da

commit: b2af31f

Cheers, Jaime

[Michele Simionato]

I see that if the file is not inside oqdata the engine tries to copy it into oqdata:

           # inside importcalc.py
            print(f'Moving into {fname}')
            os.rename(calc_id, fname)

Perhaps the problem is that you cannot copy the file? What if you try to make a symlink, i.e.

if you change in the code os.rename -> os.symlink?

               Michele

[Michele Simionato]

Also notice that there is the environmente variable OQ_DATADIR to set if you want to change the default location of the calculation files, then you would avoid any need to copy.

     Michele