need help with R

[Bob Carpenter]

R and Rcpp are winning the wrestling match and I'm
at my wit's end. I seriously don't think I can build
the Stan-R interface any time soon without some help.
I just don't understand enough about R.

I managed to spend yesterday getting the Rcpp hello
world installed again, but then it took hours for
me to track down a build bug vs. Stan that turned
out to be because they're defining a macro PI (at
least I think that's what's going on) that's messing
with the C++ parser when it sees stan::math::PI being
defined. I'm about to commit a patch to Stan that
will get rid of that constant.

Today, I installed the latest R (2.14) and latest
Rcpp (0.9.10) at work, and I can't get the Rcpp
hello world example to compile. Yet everything
worked just dandy when I did the same thing last
night at home.

This is all starting from

* fresh install of R 2.14
* fresh install.packages('Rcpp') of Rcpp 0.9.10

All I'm doing is following the breadcrumbs I left
last time I slogged through this (stan/doc/notes/rcpp.txt)
and get stalled at:

R
> setwd('temp')
> install.packages('Rcpp') // should get Rcpp_0.9.6.tgz
> require('Rcpp') // should load Rcpp
> Rcpp.package.skeleton(name='mypkg',module=TRUE)
// creates temp/mykg

SHELL
$ R CMD build mypkg // creates temp/mypkg_1.0.tar.gz
$ R CMD check mypkg // OK, with some warnings

It's no longer OK with some warnings, but rather
tells me:

=========================================================
~/stan> R CMD build mypkg
* checking for file �mypkg/DESCRIPTION� ... OK
* preparing �mypkg�:
* checking DESCRIPTION meta-information ... OK
* cleaning src
* checking for LF line-endings in source and make files
* checking for empty or unneeded directories
* building �mypkg_1.0.tar.gz�

~/stan> R CMD check mypkg
* using log directory �/Users/carp/stan/mypkg.Rcheck�
* using R version 2.14.2 (2012-02-29)
* using platform: x86_64-apple-darwin9.8.0 (64-bit)
* using session charset: UTF-8
* checking for file �mypkg/DESCRIPTION� ... OK
* checking extension type ... Package
* this is package �mypkg� version �1.0�
* checking package namespace information ... OK
* checking package dependencies ... OK
* checking if this is a source package ... OK
* checking if there is a namespace ... OK
* checking for executable files ... OK
* checking whether package �mypkg� can be installed ... ERROR
Installation failed.
See �/Users/carp/stan/mypkg.Rcheck/00install.out� for details.

~/stan> cat /Users/carp/stan/mypkg.Rcheck/00install.out
* installing *source* package �mypkg� ...
** libs
*** arch - i386
g++ -arch i386 -I/Library/Frameworks/R.framework/Resources/include
-I/Library/Frameworks/R.framework/Resources/include/i386
-I/usr/local/include
-I"/Library/Frameworks/R.framework/Versions/2.14/Resources/library/Rcpp/include"
-fPIC -g -O2 -c rcpp_hello_world.cpp -o rcpp_hello_world.o
g++ -arch i386 -I/Library/Frameworks/R.framework/Resources/include
-I/Library/Frameworks/R.framework/Resources/include/i386
-I/usr/local/include
-I"/Library/Frameworks/R.framework/Versions/2.14/Resources/library/Rcpp/include"
-fPIC -g -O2 -c rcpp_module.cpp -o rcpp_module.o
g++ -arch i386 -dynamiclib -Wl,-headerpad_max_install_names -undefined
dynamic_lookup -single_module -multiply_defined suppress
-L/usr/local/lib -o mypkg.so rcpp_hello_world.o rcpp_module.o
/Library/Frameworks/R.framework/Versions/2.14/Resources/library/Rcpp/lib/i386/libRcpp.a
-F/Library/Frameworks/R.framework/.. -framework R -Wl,-framework
-Wl,CoreFoundation
ld: warning: in /usr/local/lib/libstdc++.dylib, file was built for
unsupported file format which is not the architecture being linked (i386)
installing to /Users/carp/stan/mypkg.Rcheck/mypkg/libs/i386
*** arch - x86_64
g++ -arch x86_64 -I/Library/Frameworks/R.framework/Resources/include
-I/Library/Frameworks/R.framework/Resources/include/x86_64
-I/usr/local/include
-I"/Library/Frameworks/R.framework/Versions/2.14/Resources/library/Rcpp/include"
-fPIC -g -O2 -c rcpp_hello_world.cpp -o rcpp_hello_world.o
g++ -arch x86_64 -I/Library/Frameworks/R.framework/Resources/include
-I/Library/Frameworks/R.framework/Resources/include/x86_64
-I/usr/local/include
-I"/Library/Frameworks/R.framework/Versions/2.14/Resources/library/Rcpp/include"
-fPIC -g -O2 -c rcpp_module.cpp -o rcpp_module.o
g++ -arch x86_64 -dynamiclib -Wl,-headerpad_max_install_names -undefined
dynamic_lookup -single_module -multiply_defined suppress
-L/usr/local/lib -o mypkg.so rcpp_hello_world.o rcpp_module.o
/Library/Frameworks/R.framework/Versions/2.14/Resources/library/Rcpp/lib/x86_64/libRcpp.a
-F/Library/Frameworks/R.framework/.. -framework R -Wl,-framework
-Wl,CoreFoundation
installing to /Users/carp/stan/mypkg.Rcheck/mypkg/libs/x86_64
** R
** preparing package for lazy loading
** help
Warning: /Users/carp/stan/mypkg/man/mypkg-package.Rd:30: All text must
be in a section
*** installing help indices
** building package indices ...
** testing if installed package can be loaded
*** arch - i386
*** arch - x86_64
R(1222) malloc: *** error for object 0x7fff704bd500: pointer being freed
was not allocated
*** set a breakpoint in malloc_error_break to debug
sh: line 1: 1222 Abort trap
'/Library/Frameworks/R.framework/Resources/bin/R' --arch=x86_64
--no-save --slave <
/var/folders/ju/juD0VflHENew03RUV+bbPU+++TM/-Tmp-//RtmpRJ7Xbd/file47a4ecefc40
ERROR: loading failed for �x86_64�
* removing �/Users/carp/stan/mypkg.Rcheck/mypkg�
~/stan> R --version
R version 2.14.2 (2012-02-29)
Copyright (C) 2012 The R Foundation for Statistical Computing
ISBN 3-900051-07-0
Platform: x86_64-apple-darwin9.8.0/x86_64 (64-bit)

R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under the terms of the
GNU General Public License version 2.
For more information about these matters see
http://www.gnu.org/licenses/.
===============================================================

And I don't know if it's relevant, because I don't see any compiler
called out in the dump, but

==============================
~/stan> g++ --version
i686-apple-darwin10-g++-4.2.1 (GCC) 4.2.1 (Apple Inc. build 5666) (dot 3)
Copyright (C) 2007 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
===============================

- Bob

[Ben Goodrich]

Building / checking / installing mypkg is more or less fine on my laptop. So, this is going to be difficult for me to help you resolve what appears to be a Mac issue. My first thought is to pass --no-multiarch to circumvent trying to generate the 32 bit code

R CMD build mypkg
R CMD INSTALL --no-multiarch mypkg_*

which pertains to the -arch i386 stuff below.

Ben

[Ben Goodrich]

A second thing to try is to use clang++ to compile it for now, and we'll figure out the g++ 4.2 issues at half past later.

mkdir $HOME/.R
echo -e "CC=clang\nCXX=clang++\n" > $HOME/.R/Makevars

R CMD build mypkg/
R CMD INSTALL mypkg_*

for reference, see

http://cran.r-project.org/doc/manuals/R-admin.html#Customizing-package-compilation

Ben

[Ben Goodrich]

Unrelated to the compiler issue, here is an email from Dirk to the r-devel list

https://stat.ethz.ch/pipermail/r-devel/2012-March/063648.html

that gives a simple example of how to use the inline and Rcpp packages to create and call a C++ function (with a comparison to the same C function that uses the R API). The email to which it refers has an attached PDF that is a cheatsheet for the R API, but Rcpp makes it so that you don't actually need to know the R API that thoroughly.

In our case, the C++ function just needs to #include "user_model.cpp" produced by a system call to stanc, to translate all the command line options (except --data) to passed integers, to invoke the driver, and to return a list that has a matrix of samples, any diagnostics, etc.

Ben

[Bob Carpenter]

Thanks. I'll give it a shot when I get back to the office.

The weird thing is that it works on my Macbook Air, just
not on the iMac. Both are running the same Snow Leopard version
of the OS (the current is "Lion" -- these names are just too
cutesy for me). I'll have to check on versions of g++.

I also got some pointers from Dirk Eddelbuettel and others
on the list. They suggested I compile my own version of Rcpp.
Somehow using install.packages() may have messed things up.
(Though that's also what I did at home, and the non-module
Rcpp thing still works.)

- Bob

[Bob Carpenter]

Exac

Thanks. Some of those docs are on the CRAN page, too.
I have the cheat sheet. What I need is a more thorough
Rcpp API doc.

What I don't see how to do is get the row-major sequence
of values corresponding to an R multi-dimensional array.

- Bob

Exactly. There's a proto-implementation of
var_context in the rstan subdirectory of Stan.
I figured out how to get the dimensions, but I
can't see how to get the values for a 3+ dimensional
array in R. I haven't looked that hard, yet.

Instead, I'm working on the manual, so I can
at least feel like I'm making progress. Every time
I touch R, Windows, Linux, a new gcc, etc. etc., it
feels like two steps back.

- Bob

[Ben Goodrich]

On Saturday, March 24, 2012 1:48:34 PM UTC-4, Bob Carpenter wrote:

What I don't see how to do is get the row-major sequence
of values corresponding to an R multi-dimensional array.

I'll look into it. What do we need a multi-dimensional array for? Maybe you want to return an iterations x parameters x chains array of samples? If that is the only thing, it would probably be easier to return a list of length chains where each element is an iterations x parameters matrix. We could reshape that to a 3D thing inside R if necessary.

[Ben Goodrich]

Either way, I think we'll be okay on the array thing. See

http://comments.gmane.org/gmane.comp.lang.r.rcpp/2753

Rcpp acts like R in this case where an array is really a long vector that happens to have an attribute specifying its dimensions.
 
Ben

[Bob Carpenter]

If users declare a Stan variable as

double x[I,J,K];

then it's going to need to be initialized from
a 3-D array from R. All I need from R are
the dimensions (that's in the example that Malecki
helped me write) and the values in last-index-major
order. Stan will do all the rest with the var_context
automatically.

- Bob

[Bob Carpenter]

I knew that. We discussed this before and that's why
I set up var_context the way I did, anticipating reading
in from R (and R-style dump files).

Right -- I just need to figure out how to get
that array. I didn't see it in the Rcpp doc,
but it must be there somewhere.

- Bob

[Ben Goodrich]

On Saturday, March 24, 2012 4:49:02 PM UTC-4, Bob Carpenter wrote:

If users declare a Stan variable as

    double x[I,J,K];

then it's going to need to be initialized from
a 3-D array from R.  All I need from R are
the dimensions (that's in the example that Malecki
helped me write) and the values in last-index-major
order.  Stan will do all the rest with the var_context
automatically.

We can do that, although the ordering will be a little tricky. In R, if you do

> ( myarray <- array(c(letters, LETTERS), dim = c(13, 2, 2)) )
, , 1

      [,1] [,2]
 [1,] "a"  "n"
 [2,] "b"  "o"
 [3,] "c"  "p"
 [4,] "d"  "q"
 [5,] "e"  "r"
 [6,] "f"  "s"
 [7,] "g"  "t"
 [8,] "h"  "u"
 [9,] "i"  "v"
[10,] "j"  "w"
[11,] "k"  "x"
[12,] "l"  "y"
[13,] "m"  "z"

, , 2

      [,1] [,2]
 [1,] "A"  "N"
 [2,] "B"  "O"
 [3,] "C"  "P"
 [4,] "D"  "Q"
 [5,] "E"  "R"
 [6,] "F"  "S"
 [7,] "G"  "T"
 [8,] "H"  "U"
 [9,] "I"  "V"
[10,] "J"  "W"
[11,] "K"  "X"
[12,] "L"  "Y"
[13,] "M"  "Z"

It fills rows, then column, then shelves, ... Are you saying that stan wants the order to be

"a", "A", "b", "B", ... "z", "Z"

If so, then

c(apply(myarray, MARGIN = 1:2, FUN = function(x) x))

does that.

Ben

[Ben Goodrich]

You might also have a look at Dirk's newly blogged R package RcppSMC, which implements Sequential Monte Carlo via SMC. Other than using a different Monte Carlo algorithm, that package is probably pretty similar to what we need to do with stan.

http://cran.r-project.org/web/packages/RcppSMC/index.html

Ben

[Bob Carpenter]

Nice example -- that's exactly the order I meant.
I called it "last-index major" to parallel column major
in matrices. Is there a way to say that more idiomatically
in R? I don't recall ever hearing "shelf" before, but it
sort of corresponds to how R prints it out.

> If so, then
>
> c(apply(myarray, MARGIN = 1:2, FUN = function(x) x))

Cool. I don't think I'll ever understand R. That
looks like you're applying an identity function.

- Bob

[Andrew Gelman]

Once it gets to close to runnable, you can send some examples by me, and I will play the role of "typical R user".

Also, I thought we had decided to use "real", not "double"?

A

[Bob Carpenter]

On 3/24/12 8:41 PM, Andrew Gelman wrote:
> Once it gets to close to runnable, you can send some examples by me, and I will play the role of "typical R user".

You're definitely the first guinea pig in line for
testing the R interface.

> Also, I thought we had decided to use "real", not "double"?

Sorry, yes, it's "real". I told you
I'd have trouble with it because it doesn't
match other programming languages!

Don't worry, it's right in the doc! That,
I can check automatically.

- Bob