Would OpenVDB benefit from Cray's Chapel language?

[Joshua Olson]

Of course, the animation industry is the main user of OpenVDB right now, but Chapel is designed to combine the major parallel/high-performance computing paradigms out there today. Would something that runs on clusters like, say, RenderMan, benefit? I've been told that anything other than C/C++ is unlikely to speed up our favorite data structure, but this isn't some random Nim. It's Cray versus IBM and originally funded by DARPA for many years.

I'm going to try to port this for my raytraced game most likely regardless, but I'd appreciate some of the community's thoughts.

[Joshua Olson]

I really wish I could edit posts. If I recall, IBM dropped out of the High Productivity Computing Systems program.

[edward]

That might be interesting. Does Chapel have generics?

Do you know if they have updated benchmarks? The first paper I found
(published in 2013) showed that Chapel still needed optimizations:
http://se.inf.ethz.ch/people/west/benchmark-multicore-esem-2013.pdf

On the other hand, that benchmark puts TBB in a pretty good light, which
OpenVDB currently uses.

-Edward

[Joshua Olson]

Yes, generics are there. As for the benchmarks, their TBB comparisons haven't been updated since then in the repo. This led me to realize that the only way to report bugs is to email the team directly or to a SourceForge list. Ouch.

Nevermind, not gonna touch Chapel.

[Joshua Olson]

While my opinion has not changed, I'd like to share how a team member addressed my concerns:

Thanks for your interest in Chapel as an alternative to TBB, Rust, Julia, and the like.

Someone in the OpenVDB forum wanted to know if there's a recent comparison with Intel TBB. It seems there is not.

Offhand, I'm not aware of a recent Chapel-vs-TBB comparison (and wouldn't trust anything older because our performance has been improving by leaps and bounds over the past few years).

Here on the core development team, we tend to focus our efforts on work that others could not easily accomplish themselves, like modifications to the compiler and language.  By comparison, benchmarking Chapel vs. parallel programming model 'xyz' is relatively easy for anyone in the open-source/academic programming communities to do without special knowledge, so we tend to leave that to others.

Practically speaking, given that the space of {programming models x benchmarks} that someone _might_ be interested in seeing results for is vast, I consider attempting to satisfy that desire to be a bottomless pit, particularly given my team's finite/limited resources.  Better for us to focus on making Chapel perform better than to try and flesh out this huge cross-product for the community (besides which, if we published the numbers, you'd likely take them with a healthy dash of salt anyway).

Internally, our focus is increasingly turning to distributed memory execution since that's Chapel's primary reason for being, so we tend to be focused more on comparisons to other distributed memory models like MPI rather than on shared memory models like TBB.  That said, there are some ongoing shared memory comparison efforts that you might be interested in:

One comparison that we have been focused on in recent years is the Computer Language Benchmarks Game (formerly "the language shootout"), hosted at http://benchmarksgame.alioth.debian.org/.  We like it because it's a good example of a neutrally-managed system that does cross-model comparisons where we'd only need to do the work for Chapel, not everyone else.  Unfortunately, it does not look like TBB has an entry on the official website. (Nor do we, yet.  We've been working on fixing up a few final issues before submitting our codes.  But these codes are available in our repo if people wanted to do their own comparisons).

A second case that we're focused on currently is the LCALS loop kernel suite from Livermore.  I'm not seeing a TBB version of this offhand either.  In practice, our comparisons have been against OpenMP (which is the dominant shared memory parallel programming model in our community).

The Intel PRK suite (https://github.com/ParRes/Kernels) is a third set of benchmarks (for shared and/or distibuted memory) that we've recently started looking at, though I'm not seeing a TBB entry there.  (And frankly, neither of these last two cases are set up to easily support automated cross-language comparisons as well as the first).

Anyway, if you are aware of a language comparison system similar to the Computer Language Benchmarks game that includes Intel TBB that you think we should participate in, please let us know.  We have reasonably limited resources, so might not get to it in short order, but I'm always interested in hearing about such comparison sites, and ports like these make for good summer intern projects.


Reorganizing your mail:

Does the Chapel team have a justification for only benchmarking against itself...

We don't benchmark just against ourselves, but we do focus on programming models that are used most dominantly by our community (MPI and OpenMP being key examples).  If, in this question, you're referring to the fact that you don't see other programming models in our nightly performance graphs:

        http://chapel.sourceforge.net/perf/

...the historical reason for this has been that it's a nontrivial legal hassle for us to commit others' benchmark codes to our repository (not to mention that it would result in SCM bloat) combined with the fact that our automated performance testing only uses code in our repo.  We've recently been brainstorming ways to work around these issues (say, by pulling benchmark suites and overlaying them dynamically), but haven't gotten that up and working yet.  If you're interested in contributing to that effort, let us know.

Second, I just found out where to report issues, and it's nothing as nice as GitHub or JIRA.

We've recently launched a Chapel JIRA site, though we're definitely still easing into it.  Specifically, for the time being, we're using it primarily for internal issue tracking purposes as the team gets familiar with it and establishes best practices for its use.  It is publicly-readable, though, and the intention is to increasingly reflect bugs reported to the (public) mailing lists as JIRA issues:

        https://chapel.atlassian.net/projects/CHAPEL/issues/

Specifically, we have not yet created a portal for the community to open their own JIRA issues and have been kicking around different approaches for filtering signal from noise.

Beefing up our support for JIRA as our primary issue tracker has been called out as a priority for the current release cycle.  You can see this (and other current priorities) in our release notes for version 1.13:

        http://chapel.cray.com/releaseNotes/1.12/10-Priorities.pdf
        http://chapel.cray.com/download.html#releaseNotes

Does the Chapel team have a justification for ... using disparate SF lists riddled with spam?

We've been looking for a replacement for SF for our mailing lists, but haven't found anything that's felt satisfying enough to justify the effort of switching horses and migrating our archives.  If you have a proposal, please let us know -- maybe you're aware of a hosting site that we're not. Note that there's no need to be subscribed to chapel-bugs to mail to it, so hopefully the spam only affects us and not those trying to report bugs.

As someone wanting something more mature than Julia, Rust, or Nim to port the sparse data structure behind much of 3D animation simulation,

Sounds interesting -- let us know if you end up pursuing it.  I'll mention that our "built-in" sparse domain/array implementation is functional but in need of a lot more performance-tuning work, so if your goal is to find something that is currently performance competitive with TBB today, I'd guess that you wouldn't want to use Chapel's sparse domains/arrays at present.  Building your own sparse data structures from dense ones would likely result in more competitive performance, but lose a lot of the productivity features that Chapel was designed for.

Back on your first question, I'll note that I'm not aware offhand of a good suite of shared memory sparse array/matrix benchmarks that would be appropriate for Chapel vs. TBB or OpenMP comparisons. So if you are, that would be of particular interest to me.  I think part of the reason that optimizing our sparse implementation hasn't received more attention is due to a lack of having written such benchmarks in Chapel.

Improving sparse performance and capabilities is also a priority over the next few release cycles (under the umbrella "array/domain improvements" in the deck I pointed you to above), so if you're interested in seeing Chapel step up its sparse game here, we'd be curious what real-world use cases like yours look like in Chapel, where the pain points are, and where our performance is lagging the competition.

[jono]

You seem very interesting in alternate programming languages. 

I would say that for 99% of OpenVDB's audience, the only place where a non-C++ implementation is interesting on it's face is for code running on the GPU. While I think porting OpenVDB to an alternate language for running on the CPU is certainly an interesting project, I suspect matching the performance will be very very hard. OpenVDB is carefully crafted C++ with lots and lots of attention paid to performance.

If you can beat the performance of the existing implementation (using an alternative language or not), I'm sure the community would be interested. But it will be very hard to do.

--jono

[Joshua Olson]

The main reason I am unsatisfied with the current C++ version is because it takes special effort to compile on Windows. Add in the fact that the best C++ compilers are proprietary and none of the adequate ones are well-designed and all I see is a software dead-end.

[Joshua Olson]

Serieously, I understand for supercomputing and animation industries the focus on Linux, but I'd rather port and gain an understanding of how OpenVDB works than be stuck with a nonportable codebase. I want to use this to make a game. I seriously doubt gcc will ever transcompile to console SDKs.

[Mihai Aldén]

"The main reason I am unsatisfied with the current C++ version is because it takes special effort to compile on Windows."

Is it not easier to make the current implementation compile on Windows than rewriting everything in a new programming language?

"I want to use this to make a game."

Ok, but you probably don't need the entire library. Also, there is never a single software solution that is optimal for all problems and hardware platforms. I recommend that you explore multiple data structures and algorithms and make the most appropriate choices for your game.

"I seriously doubt gcc will ever transcompile to console SDKs."

Not sure I understand how that relates to the current C++ implementation.

Best,

Mihai

PS

If you want a truly challenging port, take a look at Ook! http://esolangs.org/wiki/ook! ? - Just kidding :) 

[edward]

On Wednesday, January 6, 2016 at 7:28:32 PM UTC-5, Joshua Olson wrote:

The main reason I am unsatisfied with the current C++ version is because it takes special effort to compile on Windows. Add in the fact that the best C++ compilers are proprietary and none of the adequate ones are well-designed and all I see is a software dead-end.

So this is traded off with the effort it takes to implement it in another language? Visual C++ may be proprietary on Windows but effectively free for commercial usage the last time I checked.

If the effort is because of dependencies, I think it's worthwhile work to try to lift them. Offhand, only Boost and TBB are the only real strong dependencies. I forget what other uses OpenEXR is used for but it's mostly just for the half type, which is a single .h/.cpp pair. Some of the boost features are also now available in modern C++ libraries now as well.

-Edward

[jono]

As I said, I think a port of OpenVDB to another language is a great exercise, and will be a very big challenge. I'm sure you will learn a lot. I don't think that itch is very common.

If the goal is to make a game, and for that game you need something like OpenVDB, I would think the above is a pretty big detour you might not want to take. I don't understand why you can't use the C++ version on modern consoles. 

That said, give it a try, I'd be curious if you are able to even approach the performance of the C++ version.

The OpenVDB team has put a lot of blood, sweat and tears into making it scale well, I imagine the low-hanging fruit is hard to come by.

--jono

[Joshua Olson]

Luckily I never planned to use a lot of OpenVDB anyway. And I'm talking about the Intel C++ compiler. Basically, I'm seeing too much Intel-specific stuff here (premature future-proofing optimization, maybe?). And as I see it, using a programming language designed to transcompile (like Haxe to C# for Unity someday, but Nim is faster now), and a less mess/messily standardized one at that, would make mods easier for users to make.

[Joshua Olson]

I would hope the cumulative historical effort to fix OpenVDB bottlenecks would be irrelevant to a current source-to-source port.