Breeze and Scala 3

[David Hall]

Hi everyone,

It's been a long time. Hope everyone is safe and well. I've been doing other stuff mostly and neglecting Breeze, as is my wont, but I recently decided to pick it up and see about porting to Scala 3/Dotty.

There's a huge stumbling block which is that Breeze depends heavily on macro annotations for code generation, and codegen is important for performance, but Scala 3 doesn't allow them. I think what this means is that the story for Breeze on Scala 3 is going to be "use the 2.13 builds in compatibility mode," at least for a long while.

Now, that aside, I think I really like Scala 3. The new metaprogramming facility (inline) is really cool and refreshingly simple, and combined with things like literal types mean that a lot of things I had to do kludge-ily (Axis._2) or avoided altogether (like matrix sizes in types or named dimensions) might actually be ergonomic. So, I think I'm likely to work on a "Breeze 2", or maybe some other name, that is not backwards compatible with Breeze, but might be near enough to backwards compatible that it won't be a huge pain to port.

No promises though.

-- David

[quaf...@gmail.com]

I had a good experience using Breeze - thank you for your work. 

You mention above that you jumped through a lot of hoops on performance. Wondering if this can make life a lot easier you?

new Vector API in JDK 16

But guessing you might know better than I!

[David Hall]

Thanks for the pointer! I was only really dimly aware of it and I should take more of a look. This will definitely help with some things. A lot of what I ended up fighting was basically boxing and dispatch related, though trying to write loops to maximize performance was definitely something I fought a bunch.

-- David

--
You received this message because you are subscribed to the Google Groups "Scala Breeze" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scala-breeze...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/scala-breeze/9dc6b26d-b51b-433b-9051-5de78e15f675n%40googlegroups.com.

[Darren Wilkinson]

David,

A "Breeze 2" for Scala 3 sounds awesome!

Regarding "Breeze 1", I think there is a suggestion that Scala 3 libraries should not include dependencies on 2.13 libraries, so the lack of a Breeze for Scala 3 could be a bit of an issue. Would it be possible to ditch the code gen and accept the performance hit until Breeze 2 comes on line?

But regarding Breeze 2, my impression from this group and the issue tracker is that 90% of the uses of Breeze are core linear algebra (a Scala-friendly wrapper around netlib-java), and probability distributions. They certainly cover over 90% of my requirements. Do you think it would be a lot of effort to get core linear algebra up and running? Would you still wrap around netlib-java?

Regards,

Darren

[David Hall]

On Thu, Apr 8, 2021 at 9:12 AM 'Darren Wilkinson' via Scala Breeze <scala-...@googlegroups.com> wrote:

David,

A "Breeze 2" for Scala 3 sounds awesome!

Regarding "Breeze 1", I think there is a suggestion that Scala 3 libraries should not include dependencies on 2.13 libraries, so the lack of a Breeze for Scala 3 could be a bit of an issue. Would it be possible to ditch the code gen and accept the performance hit until Breeze 2 comes on line?

Yeah it's doable, but I would expect the performance hit to be pretty massive for the codegened ops unless Hotspot pulls off some serious magic because a whole lot of stuff is going to bounce through type classes in inner loops. I also thought about just moving the codegen out to a separate system since it's more or less just token substitution a la the C preprocessor. It just seemed like a lot of pain for not that much of a gain given that Breeze is more or less in maintenance mode.

But regarding Breeze 2, my impression from this group and the issue tracker is that 90% of the uses of Breeze are core linear algebra (a Scala-friendly wrapper around netlib-java), and probability distributions. They certainly cover over 90% of my requirements. Do you think it would be a lot of effort to get core linear algebra up and running? Would you still wrap around netlib-java?

It's definitely doable and a lot less work of course since it's a prefix. I was mostly trying to figure out a way to make it so I didn't need to essentially maintain two codebases to get the performance I'd want, and I couldn't figure out a way other than extracting the codegen to another build step with some separate templating system, which I'm not the most excited about doing.

Why is depending on scala 2.13 libs bad? I thought binary backwards compat was a major promise to ease fears about a major language change.

 

Regards,

Darren

On Tuesday, February 16, 2021 at 11:13:31 PM UTC David Hall wrote:

Hi everyone,

It's been a long time. Hope everyone is safe and well. I've been doing other stuff mostly and neglecting Breeze, as is my wont, but I recently decided to pick it up and see about porting to Scala 3/Dotty.

There's a huge stumbling block which is that Breeze depends heavily on macro annotations for code generation, and codegen is important for performance, but Scala 3 doesn't allow them. I think what this means is that the story for Breeze on Scala 3 is going to be "use the 2.13 builds in compatibility mode," at least for a long while.

Now, that aside, I think I really like Scala 3. The new metaprogramming facility (inline) is really cool and refreshingly simple, and combined with things like literal types mean that a lot of things I had to do kludge-ily (Axis._2) or avoided altogether (like matrix sizes in types or named dimensions) might actually be ergonomic. So, I think I'm likely to work on a "Breeze 2", or maybe some other name, that is not backwards compatible with Breeze, but might be near enough to backwards compatible that it won't be a huge pain to port.

No promises though.

-- David

--

You received this message because you are subscribed to the Google Groups "Scala Breeze" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scala-breeze...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/scala-breeze/5dbd432f-fae7-4c29-b401-0e29dcb3d455n%40googlegroups.com.

[Darren Wilkinson]

Why is depending on scala 2.13 libs bad? I thought binary backwards compat was a major promise to ease fears about a major language change.

 

I have a limited understanding of this myself, so someone please correct me if I get something wrong. I think the issue is transitive dependency. I don't think it is possible for any library or application to depend on both a Scala 2 and Scala 3 version of the same library, and that it is not possible for tooling to automatically resolve such conflicts. This would be an issue for a library like Breeze, which has lots of dependencies. eg. if I depend on a Scala 2 version of Breeze, then I'm forced to use Scala 2 versions of Cats, Spire, etc. This could already be an issue for application code, but is worse for library code. eg. if I produce a Scala 3 library which depends on Breeze, I'm forcing users of that library to use Scala 2 versions of many other libraries. That obviously wouldn't end well, and hence is not recommended.

This blog post might help:

https://scala-lang.org/blog/2021/04/08/scala-3-in-sbt.html

Regards,

Darren

 

[David Hall]

ugh, right, thanks. Ok, I think I figured out a reasonable path through scala meta to preserve codegen. I'll give that a go.

--

You received this message because you are subscribed to the Google Groups "Scala Breeze" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scala-breeze...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/scala-breeze/ae1256e0-a6a2-4321-babd-c612a36e69a5n%40googlegroups.com.