Rez Python API Cache Issue

[Benjamin Rhoades]

Hello all,

I'm running into an issue with the caching system that Rez uses to manage its packages.

I'm using the python API in a launcher application to create resolved contexts for various tools and environments. 

It seems as though while the tool is open, if a new package family, or new package version, etc. is added to a package repository that Rez will not pick it up if the package resources have already been searched.

An example of how I'm grabbing all of the available packages to display to the user:

from rez.packages_ import iter_package_families

def get_all_packages():

    it = iter_package_families(config.packages_path)

    seen = set()

    for family_resource in it:

        for package_resource in family_resource.iter_packages():

            key = (package_resource.name, package_resource.version)

            if key in seen:

                continue

            seen.add(key)

            yield package_resource

The goal of the launcher is to be a tool that can be up at all times, and will always try to resolve the context with access to the latest package information.

I've found that I can force Rez to check for new packages by clearing the cache:

from rez.packages_import package_repository_manager
package_repository_manager.clear_caches()

This seems like a somewhat inefficient way to make sure that Rez is resolving to the current state of the package repositories.

Is this behavior intended, or is the cache actually more intelligent that I'm giving it credit for and there's something that I need to be doing differently?

Has anyone found a way of solving this, or is clearing the cache the correct way?

Thanks all!

[Allan Johns]

Hey Benjamin,

This is the intended behaviour, for now, and you're doing the right thing. But I do want to change this.

Specifically, you should be able to create a package repo manager that is able to detect new package releases and perform new resolves when necessary, automatically. This work is actually needed for part of another big feature, if I ever get the time to do it (rez-as-a-service).

Having to clear the cache entirely isn't quite as bad as it sounds however. Rez itself still has its own cache (in the form of memcached). Clear_caches is only clearing the in-memory caches in your client application. But it would still be better to avoid this when possible.

Thx

A

--
You received this message because you are subscribed to the Google Groups "rez-config" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rez-config+unsubscribe@googlegroups.com.
To post to this group, send email to rez-c...@googlegroups.com.
Visit this group at https://groups.google.com/group/rez-config.
For more options, visit https://groups.google.com/d/optout.

[Benjamin Rhoades]

Thanks for the response, Allan. Glad to know I wasn't missing anything.

The reason why I brought this up is that just fetching a list of packages available (see example code in my first post), actually takes much longer than I would expect. This slowness is not tied into a context resolve. In the GUI I've created, these operations are threaded in such a way as to have minimal impact on usability, but there are still times when the user is waiting for a list to be populated. Ideally, as you mention, the package manager would detect new packages and add them to the cache so that refreshing to the current state of the package repository should be much quicker.

To unsubscribe from this group and stop receiving emails from it, send an email to rez-config+...@googlegroups.com.

[Allan Johns]

Are you running memcached? If not, you will be getting package load times that are far longer than if you are, because rez is fetching the definition off disk each time.

For production use it is highly recommended that you run a memcached instance. All you need to do is point rez at it with the memcached_uri setting (for testing you can simply export REZ_MEMCACHED_URI, which does the same thing).

Hth

A

To unsubscribe from this group and stop receiving emails from it, send an email to rez-config+unsubscribe@googlegroups.com.