Re: [julia-users] Fused-Multiply-Add function?

Stefan Karpinski

unread,

Jul 23, 2016, 1:50:30 PM7/23/16

to Julia Users

Yes: https://github.com/JuliaLang/julia/issues/6330. In short, there are both fma and muladd operations with different purposes:

help?> fma

search: fma findmax @fastmath UniformScaling

fma(x, y, z)

Computes x*y+z without rounding the intermediate result x*y. On some systems this is significantly more expensive than x*y+z. fma is used to improve accuracy in certain algorithms. See muladd.

help?> muladd

search: muladd

muladd(x, y, z)

Combined multiply-add, computes x*y+z in an efficient manner. This may on some systems be equivalent to x*y+z, or to fma(x,y,z). muladd is used to improve performance. See fma.

On Sat, Jul 23, 2016 at 1:40 PM, Oliver Schulz <oliver...@tu-dortmund.de> wrote:

Does Julia have a standardized FMA (Fused-Multiply-Add) function? Like fma(A, B, factor) and fma!(dest, A, B, factor), so that that GPU libraries, etc. can provide optimized versions?

Oliver Schulz

unread,

Jul 23, 2016, 1:54:05 PM7/23/16

to julia-users

Hi Stefan,

sorry, yes, I had somehow overlooked fma. Mainly I was looking for an in-place version though, like fma! and muladd!. Is there a reason those don't exist?

Stefan Karpinski

unread,

Jul 23, 2016, 2:04:31 PM7/23/16

to Julia Users

They don't make sense for scalars but they could be added for matrices.

Oliver Schulz

unread,

Jul 23, 2016, 2:31:42 PM7/23/16

to julia-users

Oh, sure - I was actually mainly thinking about matrices. I was looking for something like fma! in ArrayFire (without success), and then wondered what it might be called in Base.

Stefan Karpinski

unread,

Jul 23, 2016, 2:47:57 PM7/23/16

to Julia Users

I would file an issue about having and fma! function. Since it's only sensible for matrices (as far as I can tell), there doesn't seem to be a corresponding need for muladd! since the fusion is at the collection level.

Jutho

unread,

Jul 23, 2016, 6:24:02 PM7/23/16

to julia-users

Did you check axpy! which originates from the BLAS tradition?

Oliver Schulz

unread,

Jul 24, 2016, 3:59:01 AM7/24/16

to julia-users

Yes, I know axpy. However - as far a I understand - that is explicitly a BLAS function. So it's not something that, e.g., a GPU library would provide specialized methods for. Also, axpy always results in a call, even for small arrays (please correct me if I'm wrong). I think in-place multiply-add for collections is something so basic and general that it would to well in Base.

Oliver Schulz

unread,

Jul 24, 2016, 4:04:02 AM7/24/16

to julia-users

Uh, sorry, I don't quite get that. I thought muladd was basically that same as fma - with the difference that fma has to do it without rounding the intermediate result, while muladd is free to do whatever is most efficient on the given hardware? But why wouldn't make muladd! make sense for collections then?

Stefan Karpinski

unread,

Jul 24, 2016, 12:11:00 PM7/24/16

to julia...@googlegroups.com

Maybe it does. It's unclear to me whether operations like BLAS apxy implement fma or muladd.

Erik Schnetter

unread,

Jul 24, 2016, 4:17:00 PM7/24/16

to julia...@googlegroups.com

fma is a very specific operation defined by an IEEE standard. BLAS axpy does not implement it.

If you extend the fma definition to linear algebra, then you'd arrive at a definition such as "no intermediate result will be rounded". The BLAS definition of axpy does not state that, and it's unlikely that any real-world implementation would do this.

I'd thus that that axpy is equivalent to muladd, without any requirement on intermediate results, or the order in which things are evaluated.

-erik

--

Erik Schnetter <schn...@gmail.com> http://www.perimeterinstitute.ca/personal/eschnetter/

Oliver Schulz

unread,

Jul 24, 2016, 4:46:25 PM7/24/16

to julia-users

I've opened an issue (https://github.com/JuliaLang/julia/issues/17593) - and realized that with Julia v0.5 we may not need this at all. :-)

Reply all

Reply to author

Forward