CUDA stream support

[tklei...@gmail.com]

The ArrayFire 3.1.0 release notes mention:

CUDA Backend: Use streams in CUDA (no longer using default stream)

Using async cudaMem ops

How exactly are streams being used?

For example, if you call an ArrayFire method on array1 and then a method on array2 and then another method on array1, does ArrayFire put the two operations on array1 in the same stream and the operation on array2 in a different asynchronous stream or does it put them all on one stream?

If you start a memory transfer from CPU to GPU by declaring an array with a host pointer and then launch a method on a different array does it do this on different asynchronous streams?

One final question.  What Is the purpose of getStream()?  If ArrayFire uses multiple streams, what stream is it giving you?

Thanks for any info you can provide, Troy.

[Pradeep Garigipati]

Hello,

I have answered your questions in place, please see below.

How exactly are streams being used?

Each CUDA device exposed through the library uses an explicit stream (created during library initialization) instead of the default(0) on that device.

For example, if you call an ArrayFire method on array1 and then a method on array2 and then another method on array1, does ArrayFire put the two operations on array1 in the same stream and the operation on array2 in a different asynchronous stream or does it put them all on one stream?

All operations after a `af::setDevice` call will launch those operations/functions on that device's stream. What you asked for is currently not possible.

If you start a memory transfer from CPU to GPU by declaring an array with a host pointer and then launch a method on a different array does it do this on different asynchronous streams?

As of now, host to device copies are synchronous (on which ever device this copy is happening) since they involve the data pointers provided by the user. This might change in the future.

One final question.  What Is the purpose of getStream()?  If ArrayFire uses multiple streams, what stream is it giving you?

As i have said earlier, each device has it's own stream. If you notice the getStream function signature, it is takes in an integer parameter. This integer is the device identifier which you can get using the function `af::getDevice()` (returns the current active device). You can also set a device as active device using `af::setDevice`. All information regarding available devices can be queries using the functions listed in this section.

Regards,

Pradeep.

--
You received this message because you are subscribed to the Google Groups "ArrayFire Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to arrayfire-use...@googlegroups.com.
To post to this group, send email to arrayfi...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/arrayfire-users/ee3324b5-02f5-4847-bcc5-244df085346f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[tklei...@gmail.com]

Thanks Pradeep.

What is the purpose of providing the CUDA stream to the user through getStream()?  Is the intent to allow the user to insert their own kernels into the stream ArrayFire is using or for some other use?

Are there plans for ArrayFire methods to accept a CUDA stream as input?  This would allow me to mix in ArrayFire methods with my own CUDA kernels in a multi-stream CUDA application.

Thanks, Troy.

[Pavan Yalamanchili]

Hi,

We will eventually have support for user provided streams. The current set up allows the users to use arrayfire stream as well as synchronize the stream as needed.

--

Pavan

To view this discussion on the web visit https://groups.google.com/d/msgid/arrayfire-users/1cd3b4c5-672a-46e0-ac6b-5b523ab75acf%40googlegroups.com.

[Pradeep Garigipati]

What is the purpose of providing the CUDA stream to the user through getStream()?  Is the intent to allow the user to insert their own kernels into the stream ArrayFire is using or for some other use?

Yes, the idea is to allow users to run their custom kernels on the streams ArrayFire kernels are running.

Are there plans for ArrayFire methods to accept a CUDA stream as input?  This would allow me to mix in ArrayFire methods with my own CUDA kernels in a multi-stream CUDA application.

yes.

On Fri, Oct 30, 2015 at 10:30 AM <tklei...@gmail.com> wrote:

To view this discussion on the web visit https://groups.google.com/d/msgid/arrayfire-users/1cd3b4c5-672a-46e0-ac6b-5b523ab75acf%40googlegroups.com.