Parallel compilation in Stan?

[Jökull Snæbjarnarson]

Is this possible? has anyone done it here that can give me some advice? if so where do I start? 

I'm fitting very large models at the moment and the compilation time is killing me slowly.

Thanks in advance,

Jökull Snæbjarnarson

[Daniel Lee]

At the end of it all, it's header only so unless your compiler can parallelize, there aren't many options. 

But, it only takes my laptop a few minutes to compile a single model on my machine (< 5 mins). 

What operating system are you on? What compiler? How much ram?

Daniel

--
You received this message because you are subscribed to the Google Groups "Stan users mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to stan-users+...@googlegroups.com.
To post to this group, send email to stan-...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Jökull Snæbjarnarson]

I have a mac os, 16 gb ram,  I'm using RStan so i guess some default c++ compiler, I could try on an even better machine but my gut is that it won't make a huge difference.

You're model is probably not 10.000 lines of stan code? 

For  small models it's 1-10 min but then the compilation time scales with lines of code/number of characters/size of model in some way. And then all of a sudden you have a model compiling for more than a/ hour/s. 

But maybe I can try some more advanced parallelize-able compilers? 

-Jökull

[Ben Goodrich]

I've seen this happen once

https://groups.google.com/d/msg/stan-users/_Vu2iWAilMA/0zcksPQl_AQJ

but I don't know anything that can make clang++ go faster on a huge file. The only solution in that case was to have a smaller Stan program.

Ben
 

-Jökull

[Jökull Snæbjarnarson]

Hmmm this is going to be a serious problem for me.

Any ideas to solve this without making the Stan program smaller ? Because i'm about to make much larger models. It seems that Stan doesn't scale very well at the moment.

You think pymc does a better job ? 

-Jökull

[Krzysztof Sakrejda]

On Tuesday, February 2, 2016 at 8:25:02 AM UTC-5, Jökull Snæbjarnarson wrote:

So is it Stan -> C++ that's taking so long or C++ compilation?  

 

 

-Jökull

[Bob Carpenter]

No idea bout PyMC --- I don't even know if it's compiled
at some point or all interpreted in Python.

The Stan to C++ step should be fast, even with 10K lines of code.

The C++ compilation takes a long time because of the way the
C++ preprocessor expands templates. It generates efficient code,
but the preprocessor itself isn't so efficient and it can generate
a ton of code that itself takes forever to compile.

You are using clang++, right? And not running into memory
swapping issues?

The only way to solve this problem is to generate independent
translation units and link. Or to reduce the template dependency
depth. But for that, we need to know the instantiations. As is,
each of our densities with polymorphic vectorization and autodiff
yields on the order of 10^3 possible instantiations, with four-argument
ones being 10 times worse.

We honestly weren't anticipating people trying to compile
10K lines of Stan code. It'd be interesting at this point
to profile the C++ compiler and see where all the time's
going.

- Bob

[Jökull Snæbjarnarson]

Krzysztof, it's the compilation.

Bob, Yes clang++ and no i'm not running into memory swapping issues.

-Jökull

[Bob Carpenter]

We may take up rewriting some of our code to allow
some of it to be precompiled and linked, but it'd be a lot
of work and involve rather large object files to link against.

- Bob

[Jökull Snæbjarnarson]

That would be great! It's much needed for what i'm doing now and even more important for what I will be doing in the future.

-Jökull

[Daniel Lee]

Jökull, question for you: is your Stan program written by hand or generated somehow?

Daniel

[Jökull Snæbjarnarson]

It's generated, how ever I don't think I would have handwritten it any differently.

Why? 

- Jökull

[Krzysztof Sakrejda]

I'm curious if the reason for the giant Stan file is the result of a missing language feature. We have loops so it's not that but maybe something like that?

[Daniel Lee]

Hi Jökull,

I asked because if it's auto-generated, there may be some tricks to help. If you're taking a lot of time compiling, I take it you're packing things into matrices and using multiplication? If it's not a requirement, you might want to unpack that for the sake of compilation. At the point you're at, you'll want to look at the generated C++ code, which may explain some things.

It's hard to help you blind -- would you want to share a smaller version of your Stan program? (Since it's auto-generated, can you make sure it's formatted reasonably before sending?)

Daniel

[Jökull Snæbjarnarson]

Hi Daniel, 

So is there a speed up in compilation time if things are not packed into matrices? Could you explain why that is?

-Jökull

[Daniel Lee]

It's down to the complexity of the template metaprogramming in C++. It's between the complexity of the Stan math library, Boost, and Eigen.

If you're really interested in finding out what's causing the C++ compiler to be incredibly slow, take a look at the generated C++ header file. 

Also, try changing compiler optimization flags. I've seen a lot of variation in compile time (and run time) with the different levels. 

Daniel

[Bob Carpenter]

If the model runs fast and compiles slowl, that might
make sense. But moving to -O0 will make it compile in
1/10th the time but it'll take 10 times longer to run.

How does the C++ header file help understand the compile
complexity? Are there flags in the compiler or something?

- Bob

[Daniel Lee]

Getting back to what's going on:
- Jökull has a 10k line Stan program that is computer generated.

- The stanc compiler correctly translates from Stan to C++.

- He's not able to get the C++ to compile because it takes too long.

- He hasn't gotten to running the program yet.

- He hasn't shared the actual Stan prorgram.

- He's using a Mac with the "standard C++ compiler" (in other words, we don't know what compiler he's using.)

(Jökull, please correct me if I'm wrong)

Jökull, if you're using g++ 4.2, then you're going to get much better performance with clang++ for compilation time. There isn't any blanket advice we can give you -- at this point, you'll have to figure out how the C++ interacts with the compiler you are using. Even point changes in compiler versions have some non-trivial differences in performance.

On Mon, Feb 15, 2016 at 11:05 AM, Bob Carpenter <ca...@alias-i.com> wrote:

If the model runs fast and compiles slowl, that might
make sense.  But moving to -O0 will make it compile in
1/10th the time but it'll take 10 times longer to run.

How does the C++ header file help understand the compile
complexity?  Are there flags in the compiler or something?

If the goal is to understand how to do parallel compilation of the C++ (see the subject line), then I think you have to get to the C++ level in order to understand how to deal with the limitations of compilers and complexity of template metaprogramming.


Jökull, if you want more help, here's what you need to provide:

- Mac OS version and compiler version

- the Stan program

- a summary of what the Stan program is / does

(I don't have the time to comb through a 10k computer-generated Stan program unless there's clear guidance.)

Hopefully we can help you get it compiling.

Daniel

[Jökull Snæbjarnarson]

Hi again Daniel.

The code is 10 k lines. 

The stanc compiler correctly translates from Stan to C++.

I am able to get the C++ to compile it how ever takes very long 90 min +. 

I have gotten the program(s) to run.

I have not and can not share the Stan program. Maybe I'll give some example code though.

I'm using g++ 4.2 and clang++ on a mac with Yosemite 10.10.2.

Running the model takes a few hours, and with this compilation time I'm getting worried about how it will scale since I have plans on building much bigger models.

I have how ever not tried changing the compiler optimization flags and will take a look at that as soon as possible.

Anyway thanks for the replies guys :)

-Jökull

[Daniel Lee]

Thanks for the update.

g++ 4.2's compile time is very slow relative to a more recent version of g++ (4.6 or 4.8) and clang++, so I'll definitely suggest compiling with clang++ if you can.

90+ mins doesn't seem normal. Is there a difference in time with clang++?

Daniel

[Bob Carpenter]

I'm not surprised that a 10K line program would take 90+
minutes to compile. A 100 line program takes over a minute
with g++.

I think rather than generating ever-larger Stan programs,
you'd be better off generating some C++ and integrating that way.
Or pushing more work into functions and less into straight
generated code.

But if you can't share the code or concepts, we can't help
beyond generalities.

- Bob