C preprocessor for handling order-dependent includes

[Daniel Lee]

Joshua or anyone else...

- are there examples of this being done?

- are there references to explain how the C preprocessor works?

I haven't done this before and what I'm finding online isn't helping me understand how we could possibly do this.

Help?

Daniel

On Sun, Sep 4, 2016 at 6:07 AM, Bob Carpenter <ca...@alias-i.com> wrote:

I think that's a great idea if we can isolate when things are
out of order.  I'm not sure how easy it'll be to predefine these
things.  Daniel --- any idea?

We were thinking just having higher-level includes would help
the problem, but then we went and wrote tests that violated our
general principles, so clearly the API exposes the danger as is.

I created the issue and attributed it to @jpritikin

https://github.com/stan-dev/math/issues/383

Thanks for the suggestion.

- Bob

> On Sep 3, 2016, at 8:22 PM, Joshua N Pritikin <jpri...@pobox.com> wrote:
>
> On Fri, Sep 02, 2016 at 08:32:11PM -0400, Daniel Lee wrote:
>>   The order dependency really happens when we instantiate our traits. If
>>   certain files get included, and they typically have to do with Eigen or
>>   Boost, it instantiates templates that then screw up the rest of our
>>   compiles.
>
> Is there a way to catch this kind of thing with the C preprocessor? For
> example, a trait could #define I_AM_A_TRAIT and then if some other
> header got included later in the wrong order then you could #error
>
> ?
>
> --
> Joshua N. Pritikin, Ph.D.
> Virginia Institute for Psychiatric and Behavioral Genetics
> Virginia Commonwealth University
> PO Box 980126
> 800 E Leigh St, Biotech One, Suite 1-133
> Richmond, VA 23219
> http://people.virginia.edu/~jnp3bc
>
> --
> You received this message because you are subscribed to the Google Groups "stan development mailing list" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to stan-dev+unsubscribe@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>

--
You received this message because you are subscribed to the Google Groups "stan development mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to stan-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Bob Carpenter]

I'm not exactly sure how to do it, but I think
if we require A to be included before B, then
we #define a variable in B, then we ifdef on it
in A. Need to make things fail at that point, I
think and I'm not sure how to do that.

Eigen uses this pattern to flag things.

It may be discussed in either the Josuttis and Vandevoorde
book or Alexandrescu.

- Bob

> > To unsubscribe from this group and stop receiving emails from it, send an email to stan-dev+u...@googlegroups.com.

> > For more options, visit https://groups.google.com/d/optout.
> >
>
> --
> You received this message because you are subscribed to the Google Groups "stan development mailing list" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to stan-dev+u...@googlegroups.com.

> For more options, visit https://groups.google.com/d/optout.
>
>
> --
> You received this message because you are subscribed to the Google Groups "stan development mailing list" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to stan-dev+u...@googlegroups.com.

[Joshua N Pritikin]

On Thu, Sep 08, 2016 at 11:15:54PM -0400, Bob Carpenter wrote:
> I'm not exactly sure how to do it, but I think
> if we require A to be included before B, then
> we #define a variable in B, then we ifdef on it
> in A.

Yeah

> Need to make things fail at that point, I think and I'm not sure how
> to do that.

#ifdef WRONG_ORDER
#error "You included things in the wrong order"
#endif

> It may be discussed in either the Josuttis and Vandevoorde
> book or Alexandrescu.

Dunno. I don't think you need a book. It should be easy to figure out.

[Daniel Lee]

On Fri, Sep 9, 2016 at 8:23 AM, Joshua N Pritikin <jpri...@pobox.com> wrote:

On Thu, Sep 08, 2016 at 11:15:54PM -0400, Bob Carpenter wrote:
> I'm not exactly sure how to do it, but I think
> if we require A to be included before B, then
> we #define a variable in B, then we ifdef on it
> in A.

Yeah

I don't think that's our use case. I'm trying to wrap my head around more complex uses of the C preprocessor.

I think the best case scenario here is that we get to write our own error message instead of having the compiler spit out template instantiation errors but it can't solve the problem. But it's all speculation until I get to understand things how this is all held together.

 

> Need to make things fail at that point, I think and I'm not sure how
> to do that.

#ifdef WRONG_ORDER
#error "You included things in the wrong order"
#endif

That makes sense if there's only one way to instantiate our library. We have 12. The right order is different for those 12 different use cases. If you're wondering how we have 12, it's {primitive, reverse mode, forward mode, mixed mode} x {scalar, array, matrix}. And if you instantiate the traits for the wrong thing, it won't work.

We definitely need the first distinction: primitive, reverse mode, forward mode, mixed mode. We have trouble compiling models with mixed mode under different compilers, so we can't just blanket include everything.

We might be able to do away with the second distinction: scalar, array, matrix. But... we've got it working now. It came out of early users' desire to have minimal library dependencies. Right now, if you include scalar, it shouldn't bring std::vector (but it probably does). If you bring in array, it won't bring in Eigen::Matrix. If you bring in matrix, it'll bring in Eigen::Matrix and std::vector. All three depend on Boost.

 

> It may be discussed in either the Josuttis and Vandevoorde
> book or Alexandrescu.

Dunno. I don't think you need a book. It should be easy to figure out.

Lol. Things are easy to people that know how to do them! 

I'm looking for any direction here. If you wanted to put up a working example, I'd be happy to take it from there. If you wanted to point me to any references, online or print, that would be helpful. I don't know much about the C preprocessor and how it works.

Daniel

--
Joshua N. Pritikin, Ph.D.
Virginia Institute for Psychiatric and Behavioral Genetics
Virginia Commonwealth University
PO Box 980126
800 E Leigh St, Biotech One, Suite 1-133
Richmond, VA 23219
http://people.virginia.edu/~jnp3bc

--
You received this message because you are subscribed to the Google Groups "stan development mailing list" group.

To unsubscribe from this group and stop receiving emails from it, send an email to stan-dev+unsubscribe@googlegroups.com.

[Joshua N Pritikin]

On Fri, Sep 09, 2016 at 09:15:23AM -0400, Daniel Lee wrote:
> > Need to make things fail at that point, I think and I'm not sure
> how
> > to do that.
> #ifdef WRONG_ORDER
> #error "You included things in the wrong order"
> #endif
>
> That makes sense if there's only one way to instantiate our library. We
> have 12. The right order is different for those 12 different use cases.
> If you're wondering how we have 12, it's {primitive, reverse mode,
> forward mode, mixed mode} x {scalar, array, matrix}.

So in stage 1 {primitive, reverse mode, forward mode, mixed mode} you do:

#ifndef STAGE_1
#define STAGE_1
#endif

And then in stage 2 {scalar, array, matrix} you do:

#ifndef STAGE_2
#define STAGE_2
#endif

Since you don't want STAGE_2 to come before STAGE_1, you also need for
stage 1:

#ifdef STAGE_2
#error "All stage 1 needs to come before stage 2"
#endif

> And if you instantiate the traits for the wrong thing, it won't
> work.
> We definitely need the first distinction: primitive, reverse mode,
> forward mode, mixed mode. We have trouble compiling models with mixed
> mode under different compilers, so we can't just blanket include
> everything.
> We might be able to do away with the second distinction: scalar, array,
> matrix. But... we've got it working now. It came out of early users'
> desire to have minimal library dependencies. Right now, if you include
> scalar, it shouldn't bring std::vector (but it probably does). If you
> bring in array, it won't bring in Eigen::Matrix. If you bring in
> matrix, it'll bring in Eigen::Matrix and std::vector. All three depend
> on Boost.

Hm, well ...

> > It may be discussed in either the Josuttis and Vandevoorde
> > book or Alexandrescu.
>
> Dunno. I don't think you need a book. It should be easy to figure
> out.
>
> Lol. Things are easy to people that know how to do them!
> I'm looking for any direction here. If you wanted to put up a working
> example, I'd be happy to take it from there. If you wanted to point me
> to any references, online or print, that would be helpful. I don't know
> much about the C preprocessor and how it works.

It's just a matter of describing the requirements precisely. If you can
explain when you need then it is easy to translate into CPP macros.

[Daniel Lee]

Nope. That's not that right way we instantiate traits. I wish it were that simple! More below.

This is actually pretty straightforward. We have existing working header files. The traits need to be instantiated in this order:

• prim x scal. lines 5-36: https://github.com/stan-dev/math/blob/develop/stan/math/prim/scal.hpp
• prim x arr. lines 4-12, then it's really the prim x scal lines: https://github.com/stan-dev/math/blob/develop/stan/math/prim/arr.hpp
• prim x mat. lines 4-21, then the lines from prim x arr, then prim x scal: https://github.com/stan-dev/math/blob/develop/stan/math/prim/mat.hpp
• rev x scal. lines 4-7, then prim x scal. https://github.com/stan-dev/math/blob/develop/stan/math/rev/scal.hpp
• rev x arr. lines 4-6, then prim x arr, then rev x scal. https://github.com/stan-dev/math/blob/develop/stan/math/rev/arr.hpp
• rev x mat. lines 4-8, then prim x mat, then rev x arr. https://github.com/stan-dev/math/blob/develop/stan/math/rev/mat.hpp

Hopefully you get the idea... we're obviously missing the traits for forward and mix, but it's all laid out in the same order. So the issue is going to be if you instantiate the traits in prim x scal first, then include a function that uses Eigen::Matrix and stan::math::var (rev x mat), then everything will crash and burn.

I think Bob's thrown out the dependencies in the past. It looks like something like this (for the first line prim doesn't depend on anything, rev depends on prim):

prim <- rev  

     <- fwd

rev + fwd <- mix

scal <- arr <- mat

I'm having trouble thinking about the actual requirements because I don't know what the C preprocessor can do and can't do. Is that enough of a functional description to get it going? If you can write out actual requirements from this, would you mind putting it up on the issue?

Daniel

[Bob Carpenter]

Saw the issue before commenting here. Thanks, Joshua,
that makes it clear to me what we need to do.

I don't think we need to account for someone wanting
to include fwd/scal.hpp and rev/mat.hpp, say. They get
one choice from each line:

prim, fwd, rev, mix
scal, arr, or mat

and that determines the single header they should include.
I'm pretty sure Daniel coded the includes sufficiently
that this will work.

Option 1: Take a hard line and only allow users to
include a single high-level header. There's no reason
not to do it this way, because once the right higher-level
header's included, including another header is either
an ordering error or redundant.

Option 2: Allow redundant includes.

- Bob

> To unsubscribe from this group and stop receiving emails from it, send an email to stan-dev+u...@googlegroups.com.

> For more options, visit https://groups.google.com/d/optout.
>
>

> --
> You received this message because you are subscribed to the Google Groups "stan development mailing list" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to stan-dev+u...@googlegroups.com.

[Joshua N Pritikin]

On Fri, Sep 09, 2016 at 10:37:15AM -0400, Bob Carpenter wrote:
> Saw the issue before commenting here. Thanks, Joshua,
> that makes it clear to me what we need to do.
>
> I don't think we need to account for someone wanting
> to include fwd/scal.hpp and rev/mat.hpp, say. They get
> one choice from each line:
>
> prim, fwd, rev, mix
> scal, arr, or mat
>
> and that determines the single header they should include.
> I'm pretty sure Daniel coded the includes sufficiently
> that this will work.
>
> Option 1: Take a hard line and only allow users to
> include a single high-level header. There's no reason
> not to do it this way, because once the right higher-level
> header's included, including another header is either
> an ordering error or redundant.
>
> Option 2: Allow redundant includes.

Hm, so the level of granularity is stan/math/prim/scal.hpp vs
stan/math/prim/arr.hpp ?

Or do you want to guard individual files like,

#include <stan/math/prim/scal/prob/uniform_ccdf_log.hpp>
#include <stan/math/prim/scal/prob/uniform_cdf.hpp>
#include <stan/math/prim/scal/prob/uniform_cdf_log.hpp>
#include <stan/math/prim/scal/prob/uniform_log.hpp>
#include <stan/math/prim/scal/prob/uniform_rng.hpp>

?

[Bob Carpenter]

We want to encourage people to use the top-level includes.
If we have to do this file-by-file, we'll never be done with
it.

- Bob

[Joshua N Pritikin]

On Fri, Sep 09, 2016 at 01:24:34PM -0400, Bob Carpenter wrote:
> We want to encourage people to use the top-level includes.
> If we have to do this file-by-file, we'll never be done with
> it.

Good, then the problem is fairly simple.

> > Hm, so the level of granularity is stan/math/prim/scal.hpp vs
> > stan/math/prim/arr.hpp ?
> >
> > Or do you want to guard individual files like,
> >
> > #include <stan/math/prim/scal/prob/uniform_ccdf_log.hpp>
> > #include <stan/math/prim/scal/prob/uniform_cdf.hpp>
> > #include <stan/math/prim/scal/prob/uniform_cdf_log.hpp>
> > #include <stan/math/prim/scal/prob/uniform_log.hpp>
> > #include <stan/math/prim/scal/prob/uniform_rng.hpp>