Would ANGLE CL numerical accuracy (OpenCL on SPIR-V) match native driver accuracy?

[Lorenzo Rutayisire]

Hello,

I'm asking the above question as I'm experiencing a difference in the computations performed by OpenCL (native driver) vs Slang => SPIR-V. Same computations, on a NVIDIA RTX 3090, Driver 580.105.08.

Interestingly, if I compile and run the same code on a Adreno (TM) 830, the results are equal. And close to the same comptuations run on CPU too.

I'm wondering what is the NV driver doing here :)
Do you have any guess? Is it something you address in ANGLE? I'd be curious to know how.

My experiments:

- RoundingModeRTE 32, SignedZeroInfNanPreserve 32 doesn't help
- NoContraction on every OP doesn't help

- FloatControl2 + FPFastMathDefault %none doesn't help

(I tried all the above combined too)

I'm a bit out of ideas...

[Shahbaz Youssefi]

Hi,

Could you please expand above what it is you are testing? The description says "OpenCL (native driver)" vs "Slang => SPIR-V". Where does ANGLE fit in this? I presume the latter (slang) means you are using Vulkan compute shaders, i.e. unrelated to ANGLE.

Does "OpenCL (native driver)" mean you are testing ANGLE with it's OpenCL backend? We do currently implement OpenCL with two implementations, one that forwards to the native driver (i.e. the OpenCL backend) and one that implements it over Vulkan (i.e. the Vulkan backend). Both are works in progress FWIW. So how are you exactly using ANGLE?

[David L. Riley]

Could you please explain to me what all of this is? I’m a retired simple man and I don’t know why all this keeps coming to my email. How do I get it to stop?

--
You received this message because you are subscribed to the Google Groups "angleproject" group.
To unsubscribe from this group and stop receiving emails from it, send an email to angleproject...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/angleproject/0eaa2f93-2947-49a3-b9d2-505681794960n%40googlegroups.com.

[Lorenzo Rutayisire]

> Could you please expand above what it is you are testing? The description says "OpenCL (native driver)" vs "Slang => SPIR-V". Where does ANGLE fit in this?

> I presume the latter (slang) means you are using Vulkan compute shaders, i.e. unrelated to ANGLE.

Exactly, I'm not experimenting with ANGLE.
I'm asking in the ANGLE community as one of your goals is to port any API to Vulkan Compute, so I was wondering whether you had encountered such precision issues.

> Does "OpenCL (native driver)" mean you are testing ANGLE with it's OpenCL backend? We do currently implement OpenCL with two implementations, one that forwards to the native driver (i.e. the OpenCL backend) and one that implements it over Vulkan (i.e. the Vulkan backend). Both are works in progress FWIW. So how are you exactly using ANGLE?

I would be curious to know the differences between ANGLE CL with OpenCL backend and ANGLE CL with Vulkan Compute backend, on the NVIDIA Hardware.
If it's what I'm observing, their outcome should be different. 

Let me explain the exact test I'm running:
I'm running an optimization which is made of 4000 sequential kernels whose result of one kernel is fed to the next kernel (gradient descent).

What I'm seeing is that:

- VK (Adreno 830) converges at Step 4000; Loss: -0.8004418
- VK (NV RTX 3090) converges at Step 4000; Loss: -0.6021383 <= this is wildly different!
- CPU (Desktop) converges at Step 4000; Loss: -0.79429066
- CL (Adreno 830) Step 4000; Loss: -0.79901075

- CL (NV RTX 3090) Step 4000; Loss: -0.79938555

As the code and the SPIR-V fed to each backend is the same (a part from CPU; while for OpenCL I have exact same kernels),
the culprit to me is the driver. I've read online that the NV driver does aggressive optimizations on floats, that are allowed by the SPIR-V spec, and it makes sense as the principal use-case for Vulkan Compute is real-time graphics (hence performance over accuracy).

I tried decorating SPIR-V ops in several way but the driver seems just to ignore them and the loss keeps converging at Loss: -0.6021383.

This said, is this something you've experienced in ANGLE? Could you suggest anything?

[Shahbaz Youssefi]

Got it. Yes, there are absolutely precision-related differences between vendors. It's often not so much the driver, but the hardware itself that calculates things at different precisions. For example, a RelaxedPrecision operation might be done in 16-bit on some hardware, or 20-bit or 32-bit on some other hardware. If your shader uses full-width fp32 types (no fp16, no RelaxedPrecision), you _might_ get more consistent results. But still _might_, even between different CPUs float precision is often inconsistent. Another source of precision differences is the compiler; reordering float operations lead to precision differences even if mathematically they are equivalent, so we can't really know if the CL code compiles to really the same thing as the SPIR-V code does. See https://www.phys.uconn.edu/~rozman/Courses/P2200_15F/downloads/floating-point-guide-2015-10-15.pdf for an interesting read.

Given your algorithm has a feedback loop, it's particularly prone to error propagation with floating point math (see 4.6 Error Propagation in the pdf linked above). That means slight precision differences turn into not-so-slight differences after many iterations. There's a whole subject dedicated to writing algorithms in such a way that minimize floating point errors during iteration instead of accumulating them.