numba with ray and cache sharing

[Ryan Skene]

Not sure how familiar people may be with Ray or how common it is to use them both, but

I am building a project that utilizes both Numba and Ray and the two separately are great but paired together are even better in terms of performance improvements.

Except for one area: caching.

Just working from my 16-cpu local machine, Numba caches each function at the launch of each worker ... even though each worker uses the same functions and even though the functions are already cached locally.

I am aware that you can store cached function in a custom directory via NUMBA_CACHE_DIR and I am aware that there is NUMBA_CPU_NAME envvar that can be set.

But I'm unsure of how to get each Ray worker to look to the CPU_NAME instead building the cache itself. That sort of network coms between isolated processes is a bit beyond my skill set at this point.

This might be a question better suited to the Ray community but thought I'd try asking here as well.

[Valentin Haenel]

Hi Ryan,

this mailnglist is in the process of being phased out. I suggest opening a post on our discourse instead: https://numba.discourse.group/ as you are much more likely to get a good response there.

Best wishes,

V-

--
You received this message because you are subscribed to the Google Groups "Numba Public Discussion - Public" group.
To unsubscribe from this group and stop receiving emails from it, send an email to numba-users...@continuum.io.
To view this discussion on the web visit https://groups.google.com/a/continuum.io/d/msgid/numba-users/0b2eee71-8ed2-4aab-a4ab-847164b8cb52n%40continuum.io.

[Ryan Skene]

thank you!