Using NumPy random within a nogil block

[Pedro Camargo]

Hi, 

I searched the list and StackOverflow on this topic and couldn't find anything, so here it goes.

I am working on a piece of software that depends on random number generation, and I would like to be able to do that within a nogil block.

I understand the implications on reproducibility (or lack of thereof) that would come with sharing a single random number generator across multiple threads, but I could live with that.

On that note, is there a way to access the NumPy random number generator from a nogil block?

If not, does anyone know if the the standard c random number generator can be used to reproduce a random number sequence from MatLab (the tests instances are coming from MatLab, so reproducing that sequence is critical)...

Any other suggestions?

Cheers,

Pedro

[da-woods]

I don't think Numpy provide an interface access their random generator at a C level, so you're stuck with it being a Python function call. You could always put it in a small "with gil" block of course.

The C random generator doesn't look guaranteed to be thread-safe (so the worst case is, you may just end up corrupting its state). It also isn't specified as a specific generator so there's no way to reproduce a sequence (at least across compilers/standard library implementations, etc.)

You could always use one of the C++ generators from Cython. They're objects so you create one per thread for perfect thread safety. They also each have a specified algorithm so you should get a reproducible sequence (although if you want to match Matlab you're very dependent on knowing exactly what Matlab has done here...)

David

--

---
You received this message because you are subscribed to the Google Groups "cython-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cython-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cython-users/0b6ded0e-17bb-4f0b-bbba-dd116a66983en%40googlegroups.com.

[Jérôme Kieffer]

Hi Pedro,

I had the same issue last year and found several solutions:
* use a RNG written in C, the default one is not the best, by far!
* Use some of the ones found in stdlib from C++: not the fastest
* Use the C-API of numpy (with it s ABI incompatibilities: this is fairly recent in Numpy)
* I found a Mersenne-twister implementation in Cython which fits my needs
I had also multi-threading issues, and distribution constrains.

I summarized my peregrinations in:
https://gist.github.com/kif/ace15da5acfa495efaf3d37feb701a17

The final code is available here:
https://github.com/silx-kit/fabio/blob/master/fabio/ext/dense.pyx

Cheers,

Jerome

[Pedro Camargo]

That was EXACTLY what I was looking for!  Thanks, Jerome!

Cheers,
Pedro

---- On Thu, 03 Feb 2022 17:52:45 +1000 goo...@terre-adelie.org wrote ----

--

---
You received this message because you are subscribed to a topic in the Google Groups "cython-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/cython-users/kcr7M6py8bQ/unsubscribe.
To unsubscribe from this group and all its topics, send an email to cython-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cython-users/20220203085245.61012455%40antarctica.fournet.lan.

[mattip]

There is now (from 1.19 released June 2020) a C-API to the NumPy random module. It is documented https://numpy.org/devdocs/reference/random/c-api.html, and there are even some cython examples https://numpy.org/devdocs/reference/random/extending.html. If something is missing or not clear please raise an issue https://github.com/numpy/numpy/issues, we don't get much feedback on what works and what doesn't.

Matti

[Jérôme Kieffer]

Hi Matti,

I tested the C-interface to numpy and it works, thanks, the
performances are also good ...

It was discarded because I fear ABI-change and imposed a
compatibility with a numpy which was far too recent (at that time) for
my target audiance (ubuntu 20.04). Let it mature a year
or two and get it packaged and it will be perfect.

Cheers,

> > <https://groups.google.com/d/msgid/cython-users/0b6ded0e-17bb-4f0b-bbba-dd116a66983en%40googlegroups.com?utm_medium=email&utm_source=footer>
> > .

> >
> >
> >
>
> --
>
> ---
> You received this message because you are subscribed to the Google Groups "cython-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to cython-users...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/cython-users/54c80b77-168a-4899-99a4-8c2c24a32080n%40googlegroups.com.


--