Repost from email question

[Alexander Dunn]

This is a copy of a question I got over email, thought I'd put it (and the answer) here so others could find it.

---------------------------------------------------------------------------------------------------------------------------------------------------------

Hi Alex,

 

Thanks for integrating the previous fix so fast, and again for providing Rocketsled.

 

I wanted to also let you know about a couple changes I had to make to get it to run:

1)      Run on my Windows sandbox, and,

2)      Run RS jobs that were not located in my Python install directory (I wanted to store my RS jobs in a separate work folder)

 

The changes I made were all in utils.py’s serialize() / deserialize() functions (see attached):

1)      Change the “/” to Python’s “os.sep” OS-dependent filepath separator so it work son windows

2)      Modify serialize() to try getting the “wf_creator” function from the RS Job’s module first

 

Steps to reproduce:

1)      Copy basic.py somewhere outside the Python installation dir

a.       In this case, I used: C:\Users\abe\rmsit\workspaces\projects\ers3_rocketsled_scripts

2)      Rename “basic.py” to “basic2.py” (or any other name that is not already used by a module in your Python installation dir)

3)      Run basic2.py

 

It errors out with:

Traceback (most recent call last):

 

  File "<ipython-input-1-5a380a7f1dd2>", line 1, in <module>

    runfile('C:/Users/abe/rmsit/workspaces/projects/ers3_rocketsled_scripts/basic2.py', wdir='C:/Users/abe/rmsit/workspaces/projects/ers3_rocketsled_scripts')

 

  File "C:\ProgramData\Anaconda3\lib\site-packages\spyder_kernels\customize\spydercustomize.py", line 786, in runfile

    execfile(filename, namespace)

 

  File "C:\ProgramData\Anaconda3\lib\site-packages\spyder_kernels\customize\spydercustomize.py", line 110, in execfile

    exec(compile(f.read(), filename, 'exec'), namespace)

 

  File "C:/Users/abe/rmsit/workspaces/projects/ers3_rocketsled_scripts/basic2.py", line 96, in <module>

    mc.configure(wf_creator=wf_creator, dimensions=x_dim)

 

  File "C:\ProgramData\Anaconda3\lib\site-packages\rocketsled\control.py", line 219, in configure

    wf_creator = serialize(wf_creator)

 

  File "C:\ProgramData\Anaconda3\lib\site-packages\rocketsled\utils.py", line 122, in serialize

    "you sure it's module is in your PYTHONPATH?".format(fun_path))

 

ImportError: Administrator.rmsit.workspaces.projects.ers3_rocketsled_scripts.basic2.wf_creator couldn't be serialized to be imported. Are you sure it's module is in your PYTHONPATH?

 

 

I added a print statement to print out the function path it was trying to resolve:

        for _ in range(5):

            try:

                # if we couldn't find the func, try adding the parent folder name to the import path

                full_import_path = all_pkgs[-1] + "." + full_import_path

                all_pkgs = all_pkgs[:-1]

                fun_path = full_import_path + "." + name

                print("fun_path: {}".format(fun_path))

                deserialize(fun_path)

                return fun_path

            except ImportError:

                continue

        else:

            raise ImportError("{} couldn't be serialized to be imported. Are "

                              "you sure it's module is in your PYTHONPATH?".format(fun_path))

 

 

This is the output of the print statement:

2019-06-06 11:33:31,422 INFO Optimization collection opt_default hard reset.

fun_path: ers3_rocketsled_scripts.basic2.wf_creator

fun_path: projects.ers3_rocketsled_scripts.basic2.wf_creator

fun_path: workspaces.projects.ers3_rocketsled_scripts.basic2.wf_creator

fun_path: rmsit.workspaces.projects.ers3_rocketsled_scripts.basic2.wf_creator

fun_path: abe.rmsit.workspaces.projects.ers3_rocketsled_scripts.basic2.wf_creator

 

 

So it never tried to look up the base case “basic2.wf_creator”. I rearranged the code in the for-loop like so, but will defer to you on whether this is the right thing to do:

        for _ in range(5):

            try:

                fun_path = full_import_path + "." + name

                print("fun_path: {}".format(fun_path))

                deserialize(fun_path)

                return fun_path

            except ImportError:

                # if we couldn't find the func, try adding the parent folder name to the import path

                full_import_path = all_pkgs[-1] + "." + full_import_path

                all_pkgs = all_pkgs[:-1]

                continue

        else:

            raise ImportError("{} couldn't be serialized to be imported. Are "

                              "you sure it's module is in your PYTHONPATH?".format(fun_path))

 

 

The output of the print statement now gets “basic2.wf_creator” on the first iteration of the for-loop:

fun_path: basic2.wf_creator

 

I tested this out with Anaconda3 on a Windows 7 VM. Like I said, I will defer to you on whether these are the right code changes. I just know I had to make these changes to get it to work on my Win7 box, and I figured I’d share so hopefully it makes it easier for you. I would have done a pull request on Github, but I need to check on our corporate policies before I can do that, so I figured I would just email you to get it to you faster. Sorry about that.

 

If you have any feedback, please feel free to let me know!

 

Thanks,

Abe

[Alexander Dunn]

Hi Abe, 

Thanks for mentioning the separators for Windows compatibility. I will update the code accordingly.

---

The "not being able to find the parent module" is a tricky and persistent issue because it is not well defined how is best to access the module outside the rocketsled directory given only some free-floating function object. But there are ways around it on Linux/Unix without having to further modify serialize()/deserialize().

Here is a short primer on defining wf_creator functions (though this extends to get_z, and predictor as well):

You can pass either in a function object or string representing a function to the wf_creator argument.

Function object If you choose to pass a function object, the serializer has to serialize it (basically turn your function object into a string). This is the easiest but probably most error prone way. The serializer iteratively tries to import based on directory names and hopes that eventually it will find a fully defined importable. This works well for modules in packages, but when the function object is not in an importable package's module but rather in a free module in some folder, you will get an error like the one shown in your stacktrace. 

String - in a package: If your wf_creator is in a module which is a part of some package, you can simply do 'my_package.my_subpackage.my_module.wf_creator'.

String - not in a package: If your wf_creator is free floating in some non-package directory (e.g., C:\Abe\Downloads\basic.py), you can use the syntax "C:\Abe\Downloads\basic.wf_creator". This basically allows rocketsled to add this folder to your pythonpath env variabe so rocketsled can find the module containing wf_creator. I have just personally tested this on Linux/Unix and it works;  on windows, you might encounter some unforseen issues but it is worth a try. 

You can find more details at the end of the wf_creator section in the comprehensive guide: https://hackingmaterials.lbl.gov/rocketsled/guide.html

 

[Alexander Dunn]

P.S. I have an idea in mind for allowing function objects to work even if they are outside of registered python package in your environment. If I implement it in the next rocketsled version, I'll update this thread.

[Abe Wu]

Thanks for the response Alex.

I tried the "String - not in a package" method, by specifying wf_creator="C:\\Abe\\Downloads\\basic.wf_creator", and it did in fact work on Windows. Thanks for letting me know of this other way of doing it.

However, I think the utils.py fix I submitted might do what you're looking for, if I'm understanding what you're saying correctly. For a couple reasons:

1. It allows you to run a "free floating" RS job without having to hardcode the filepath to the basic.py, and,
2. If it doesn't find the wf_creator in the RS job, it still works in the same way as you describe. Namely, if it doesn't find the wf_creator function in the RS job, it will look in the parent folder recursively. So if the script is in "C:\Abe\Downloads\basic.py", it will look in these places in order:
• basic.wf_creator
• downloads.basic.wf_creator
• abe.downloads.basic.wf_creator

No worries though if you have a better way to do it. In any case, thanks for all the work you put in to creating/maintaining Rocketsled!

Abe

[Abe Wu]

Just to follow up on my previous email, I did confirm that I run into the same error on Linux with it not finding the wf_creator function. It was fixed after I applied the same change to utils.py.

Also, I realized I misspoke when I said "it will look in the parent folder recursively". It doesn't recurse, it just does it in the for-loop (same as in the original code).

Thanks,

Abe