potential bug introduced with importcalc.py

[jaime]

Hello, 

My team uses openquake and frequently needs to store the HDF5 files elsewhere in remote storage (shared drive) so that more than one person can access calculation results.

It appears that the importcalc.py file has been updated not too long ago, but in doing so, it removes the ability to import calculations from a remote filepath. 

e.g., 

     path = 'full path to hdf5 file'
     from openquake.commands import importcalc

     importcalc.main(path)

     OSError: File not found: C:\[userdirectory]\oqdata\calc_987.hdf5

The same error occurs with the console usage, e.g.,

    oq importcalc "full path the filename" 

I believe this is a problem with line 39, where the path gets over written by the calc_id number. Maybe it's not a big but a feature, but it would require us to copy the hdf5 files to our own oq_data folders, which can be quite space intensive. 

A way around this is to use the old import_calc.py file, but that's less ideal. 

Thanks for reading!

[Michele Simionato]

Can you find out the pull request that broke the compatibility? I am willing to restore the old behavior if possible.

         Michele

[jaime]

Thanks for your help!

It looks like it's changed a few times since v3.16.7, which was the version we were using for importcalc.py. 

There are several changes to the file, so I'm not sure which ones should stay to maintain functionality for full paths.

This commit converted to the calc_id path to an integer (older change):  

SHA: ec286c58ac820e983e2ac01709a9377ab0d5d65b

commit: ec286c5

This commit adds the calc_id integer to the datadir path (newer change): 

full SHA:  b2af31f6c1ad02700480f1a426426e15a79e68da

commit: b2af31f

Cheers, Jaime

[Michele Simionato]

I see that if the file is not inside oqdata the engine tries to copy it into oqdata:

           # inside importcalc.py
            print(f'Moving into {fname}')
            os.rename(calc_id, fname)

Perhaps the problem is that you cannot copy the file? What if you try to make a symlink, i.e.

if you change in the code os.rename -> os.symlink?

               Michele

[Michele Simionato]

Also notice that there is the environmente variable OQ_DATADIR to set if you want to change the default location of the calculation files, then you would avoid any need to copy.

     Michele

[jaime]

Sorry for taking so long to respond, it's been a busy month.

Maybe I'm misunderstanding the code, but to me it looks like the path is simply renamed:

os.rename(calc_id, fname)

and then calc ID is changed from the path to the base name (integer):

calc_id =  datastore.extract_calc_id_datadir(calc_id)[0]

And there is no copying. And that's a problem, because the file isn't stored in fname (oq_data) and the file isn't found. 

I do not want to copy the file to oqdata, that is the primary issue. The reason we would need to use importcalc at all is because the files are stored elsewhere. The files are so large that it's cumbersome to keep multiple projects at a time in oqdata. 

Changing the OQ_DATADIR sounds like it could be an option for our situation. Is this something that can only be configured without the console, or can it be set while running OQ scripts from a virtual environment? Can you point me to documentation for changing OQ_DATADIR?

-Jaime

[Michele Simionato]

OQ_DATADIR is simply an environment variable that you can set as any other environment variable.

--
You received this message because you are subscribed to a topic in the Google Groups "OpenQuake Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/openquake-users/ors93ib8oyo/unsubscribe.
To unsubscribe from this group and all its topics, send an email to openquake-use...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/openquake-users/7171bed2-274e-4845-ad85-81faa2b981a7n%40googlegroups.com.