Fwd: [eigen] Eigen 3.2.2 is released!

[Bob Carpenter]

Looks like some useful things for us, like LDLT numerical improvements and

performance improvements.

Since interfaces didn't change, can we upgrade Stan C++ and

CmdStan without impeding RStan or PyStan?

- Bob

Begin forwarded message:

From: Gael Guennebaud <gael.gu...@gmail.com>

Subject: [eigen] Eigen 3.2.2 is released!

Date: August 4, 2014 at 9:29:50 AM EDT

To: eigen <ei...@lists.tuxfamily.org>

Reply-To: ei...@lists.tuxfamily.org

Eigen 3.2.2 has been released today, on August 4, 2014.

This is a maintenance release with many bug fixes since the release of 3.2.1 five months ago. In particular, this release includes various numerical improvements in JacobiSVD, LDLT, BiCGSTAB, and ColPivHouseholderQR. There are also some limited performance improvements and added functionality in the 3.2.2 release.

The source archive is at http://bitbucket.org/eigen/eigen/get/3.2.2.tar.bz2


The full changelog is at http://eigen.tuxfamily.org/index.php?title=ChangeLog#Eigen_3.2.2, and reproduced below:

Core:
 * Relax Ref such that Ref<MatrixXf> accepts a RowVectorXf which can be seen as a degenerate MatrixXf(1,N)
 * Fix performance regression for the vectorization of sub columns/rows of matrices.
 * EIGEN_STACK_ALLOCATION_LIMIT: Raise its default value to 128KB, make use of it to assert on maximal fixed size object, and allows it to be 0 to mean "no limit".
 * Bug 839: Fix 1x1 triangular matrix-vector product.
 * Bug 755: CommaInitializer produced wrong assertions in absence of Return-Value-Optimization.

Dense solvers:
 *  Add a rank() method with threshold control to JacobiSVD, and make solve uses it to return the minimal norm solution for rank-deficient problems.
 * Various numerical fixes in JacobiSVD, including:bug 843, and the move from Lapack to Matlab strategy for the default threshold.
 * Various numerical fixes in LDLT, including the case of semi-definite complex matrices.
 * Fix ColPivHouseholderQR::rank().
 * Bug 222: Make temporary matrix column-major independently of EIGEN_DEFAULT_TO_ROW_MAJOR in BlockHouseholder.

Sparse:
 *  Bug 838: Fix dense * sparse and sparse * dense outer products and detect outer products from either the lhs or rhs.
 * Make the ordering method of SimplicialL[D]LT configurable.
 * Fix regression in the restart mechanism of BiCGSTAB.
 * Bug 836: extend SparseQR to support more columns than rows.
 * Bug 808: Use double instead of float for the increasing size ratio in CompressedStorage::resize, fix implicit conversions from int/longint to float/double, and fix set_from_triplets temporary matrix type.
 * Bug 647: Use smart_copy instead of bitwise memcpy in CompressedStorage.
 * GMRES: Initialize essential Householder vector with correct dimension.

Geometry:
 * Bug 807: Missing scalar type cast in umeyama()
 * Bug 806: Missing scalar type cast in Quaternion::setFromTwoVectors()
 * Bug 759: Removed hard-coded double-math from Quaternion::angularDistance.

OS, build system and doc:
 * Fix compilation with Windows CE.
 * Fix some ICEs with VC11.
 * Check IMKL version for compatibility with Eigen
 * Bug 754: Only inserted (!defined(_WIN32_WCE)) analog to alloc and free implementation.
 * Bug 803: Avoid char* to int* conversion.
 * Bug 819: Include path of details.h file.
 * Bug 738: Use the "current" version of cmake project directories to ease the inclusion of Eigen within other projects.
 * Bug 815: Fix doc of FullPivLU wrt permutation matrices.
 * Bug 632: doc: Note that dm2 = sm1 + dm1 is not possible
 * Extend AsciiQuickReference (real, imag, conjugate, rot90)

[Ben Goodrich]

RStan would be fine if you upgrade Eigen under lib/.

[Allen Riddell]

I don't think there should be any problems for PyStan.

On 08/04, Ben Goodrich wrote:
> RStan would be fine if you upgrade Eigen under lib/.
>
> On Monday, August 4, 2014 6:25:16 PM UTC-4, Bob Carpenter wrote:
> >
> > Looks like some useful things for us, like LDLT numerical improvements and
> > performance improvements.
> >
> > Since interfaces didn't change, can we upgrade Stan C++ and
> > CmdStan without impeding RStan or PyStan?
> >
> > - Bob
> >
> > Begin forwarded message:
> >

> > *From: *Gael Guennebaud <gael.gu...@gmail.com>
> > *Subject: **[eigen] Eigen 3.2.2 is released!*
> > *Date: *August 4, 2014 at 9:29:50 AM EDT
> > *To: *eigen <ei...@lists.tuxfamily.org>
> > *Reply-To: *ei...@lists.tuxfamily.org

> --
> You received this message because you are subscribed to the Google Groups "stan development mailing list" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to stan-dev+u...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Daniel Lee]

I think that's the the three major interfaces.

Can we upgrade boost?

[Allen Riddell]

Is there any interest in getting Stan, CmdStan, RStan, or PyStan into
a Debian/Ubuntu package? That's the only reason I can think of for pinning
a specific Boost and Eigen.

Best,

-a

[Bob Carpenter]

We could also make Debian/Ubuntu packages for Stan that are
out of date w.r.t. the current Stan C++ branch. That's what
all the other packaged packages on Linux, R, etc., seem to
do.

Daniel --- for Boost, I think we're stuck with whatever the
BH package is at as far as R goes, which is currently on
Boost 1.54, despite Boost 1.55 coming out over 9 months ago.
Boost is way late on 1.56 by the way.

As Ben's noted, BH isn't complete, but we can add our own extra libs
as needed.

So this all means we can either update for Stan C++ and CmdStan,
which don't have all these dependencies, and then start testing
multiple versions.

I really don't want to have to start tagging and maintaining
multiple builds, though. So anything on an outdated build would
just lag in all of its bug fixes. We just don't have the people power
for the alternative, which is continuing to maintain multiple
versions and patch them all.

- Bob

[Ben Goodrich]

On Tuesday, August 5, 2014 8:04:15 AM UTC-4, Daniel Lee wrote:

I think that's the the three major interfaces.

Can we upgrade boost?

I would hold off on upgrading boost for now for the reasons Bob mentioned. We should be able to do it not too long after Boost 1.56 is released.

Ben

[Ben Goodrich]

On Tuesday, August 5, 2014 8:56:49 AM UTC-4, Allen Riddell wrote:

Is there any interest in getting Stan, CmdStan, RStan, or PyStan into
a Debian/Ubuntu package? That's the only reason I can think of for pinning
a specific Boost and Eigen.

I think Debian / Ubuntu packages might be viable once we split Stan up a little more (nomad, stanc, libstan). Ross was talking about doing so a few months ago but decided it was too much trouble at the time.

For official Debian / Ubuntu packages, you can depend on Boost or Eigen >= $VERSION, but it is frowned upon to make $VERSION higher than actually necessary to use Stan.

Ben

[Bob Carpenter]

That's a chicken-and-egg problem. We can make sure Stan never uses features
from Eigen or Boost beyond some version.

I believe the feeling is that it's worth tying ourselves to older versions
of Boost and/or Eigen in order to be able to have dependencies on things
in CRAN or Ubuntu or whatever. I've already said I'm willing to do this
as long as we don't wind up introducing dependencies on multiple versions
we have to test.

- Bob

[Ross Boylan]

On Tue, 2014-08-05 at 08:56 -0400, Allen Riddell wrote:
> Is there any interest in getting Stan, CmdStan, RStan, or PyStan into
> a Debian/Ubuntu package? That's the only reason I can think of for pinning
> a specific Boost and Eigen.
>
> Best,
>

I'm interested and took a run at it, but it was too big a project for me
given my unfamiliarity with stan internals and Debian packaging.

For reasons described more fully below, the existence of a downstream
package, e.g., for Debian, does not necessarily mean you need to pin on
a specific library version.

The way stan is packaged as a giant collection, including most external
dependencies, was a major barrier to making a Debian package, although
the split into cmdstan, rstan, etc, which came after my efforts, may
help. The Debian package would need to rely on the already packaged
versions of boost, eigen, the compilers, R and python in Debian.

stan itself would probably be split into several packages. With luck
there would be a common core set of libraries (and maybe tools and
documentation) used by all the stan flavors.

stan, especially rstan, is also kind of weird in that it is itself
effectively a compiler, so that the usual model (compile the code,
package the result, and distribute it) is not quite apt for it. Usually
there is a division into "development" and "application" packages, with
the former having stuff needed if you are developing software to use the
packaged software and the latter having what you need to run it. Since
stan builds and compiles software in routine operations, this
distinction is difficult to apply in its case.

I'm sure packaging is possible; it's just not easy.

As for the issue of maintaining multiple versions (raised by Bob), that
could be the responsibility of the (currently non-existent!) Debian
packager rather than the core stan team. Although it would obviously be
easier on the packagers if it were done upstream (i.e., by the stan
group), I'm not even sure that's the norm. There are a lot of
distributions of linux--for that matter, Debian itself alway has 3,
stable (=current official release), development (=next release), and old
stable--and I doubt upstreams typically maintain all 3. The bar is
usually pretty high (i.e., security and major bug fixes only) for
modifying a package once it's in stable, and so it's really for the
distribution (e.g., Debian) to decide exactly what code to change.

I suspect in stan a middle ground between "we will do all the packaging
for Debian" and "we will ignore packaging--someone else's problem" would
be best. The middle ground would be making some modifications to stan,
especially the build system, to make it more manageable for downstream,
and then letting packagers to do the rest. Such changes may be
desirable in their own right, and could facilitate things that would be
good on their own, like having a package for R that would be acceptable
on CRAN because it wasn't huge and scary.

The version in Debian stable, let alone old stable, tends to get old
quickly. It is often not even the current version of the software at
the time Debian makes a release, because it gets frozen at some earlier
state.

2 last wrinkles: There are also backports, which are the current version
(or at least the one in Debian development branch) built to work on the
stable distribution. Only some packages are in backports. Only some of
those are completely current with upstream. In some cases they can't
be; for example julia is not in stable and never will be backported
because it requires a more recent version of LLVM than is in stable.
Second, there is also Debian "experimental" for cutting edge stuff
that's ahead of the development release.

Ross Boylan

[Bob Carpenter]

Thanks for the rundown. The issue with being able to compile
is also why it won't ever be possible for us to have a one-touch
CRAN install, because at the very least, the user needs to install
a recent-enough C++ toolchain.

We've been thinking about moving to C++11. Will that make it
impossible to release on Linux?

I'm not sure what you mean by breaking Stan down further.

Right now, there are four repos (this was a recent change to
split out cmdstan). The core Stan functionality is in:

stan-dev/stan

Then the interfaces are in:

stan-dev/{ cmdstan, rstan, pystan }

The Eigen and Boost libraries are included in full in stan-dev/stan,
and included as is as part of the cmdstan distribution. But a simple
makefile flag lets you use other versions.

RStan trims the libs down to just what's needed for RStan and I
believe now depends on the BH library for boost (and RcppEigen
for Eigen?). But again, RStan can be built and run with other lib
versions.

I'm not sure what PyStan does for library dependencies.

What specifically would we need to change about all of this?

- Bob

[Ben Goodrich]

On Tuesday, August 5, 2014 2:03:16 PM UTC-4, Bob Carpenter wrote:

What specifically would we need to change about all of this?

C++11 would be fine, except perhaps for oldstable.

When a Debian package depends on "Boost", it really means that it depends on a sublibrary of Boost. Taken as a whole, Stan depends on a lot of Boost sublibraries, but if Stan itself (not the interfaces per se) were more separatable, each part of Stan could become a Debian package that itself depends on fewer sublibraries of Boost. So, for example, nomad would be something stand-alone that maybe depends on a little bit of Boost. And stanc depends on Spirit and some other Boost stuff but not the Boost math stuff that libstan depends on. The documentation could be a separate Debian package that depends on LaTeX but not Boost or Eigen, etc.

Ben

[Bob Carpenter]

As a largely header only, Stan can't be separately compiled
and then linked later.

Compiling a model file is the worst --- it depends on everything
but the parser and code generator (Spirit Qi and its bind and
variant library dependencies).

Running RStan requires having all these things at hand.

So you're thinking we have a separately compiled compiler
that does the work of stanc and then R just calls that as
an executable?

Or that we distribute RStan code that doesn't work unless
combined with a bunch of other distributions? My worry is that
this will make it just too hard to maintain and compile for
users who don't want a pre-packaged Debian version.

- Bob

>
> Ben

[Ross Boylan]

On Tue, 2014-08-05 at 14:02 -0400, Bob Carpenter wrote:
> Thanks for the rundown. The issue with being able to compile
> is also why it won't ever be possible for us to have a one-touch
> CRAN install, because at the very least, the user needs to install
> a recent-enough C++ toolchain.

Can't you just say "if you want to use this library (= R package for
rstan) you must have a compiler [with capabilities x or versions y] and
an installation of CmdStan [or at least core stan libraries]"?
Or have the R build procedure check for the existence of necessary tools
and alert the user if they are not found. If ambitious, have the R
build procedure download and install the tools.

For example, I know that Rmpi requires you to have installed mpi before
you use it; mpi installation is not part of the process in which R
builds rmpi.

Admittedly this is not one touch. It seems to me if you want one touch
it would be best to have something that behaves like a regular R package
with external dependencies, and then have a wrapper/tarball that
includes the R library, a bunch of other stuff, and scripts to install
it. Or just scripts that download everything as needed.

AFAIK even the current distribution for R does not include a compiler
with it.

>
> We've been thinking about moving to C++11. Will that make it
> impossible to release on Linux?

Why would it? I don't know what the current state of C++11 support is
on linux, but even if it's not good, it will improve. Though if current
C++11 support is not good, I guess it would be a major barrier to
current adoption, especially if the use of new features is pervasive and
the code doesn't have versions for the old standard (either in #ifdef's
or in a separate branch).

>
> I'm not sure what you mean by breaking Stan down further.

There are 2 distinct senses of "breaking down."

One concerns the way the stan code base (interpreted broadly to include
wrappers for R, python, ...) is organized and its build process is
structured. The main modification I was thinking of here was a change
in the build procedure, not the breakdown into repositories.

The second sense of "breakdown" concerns the Debian packaging. In the
typical case, header files and library development files go in one
Debian binary package and executables go in a separate Debian binary
package. There might also be a debug version of the binary packages.
All binary packages are built from the same Debian source package, and
the breakdown of the Debian binary packages has no corresponding
breakdown in the upstream source.

The general flow is
upstream source + debian patches -> debian source package -> one or more
debian binary packages
The binary packages are the ones regular users install on their systems.
In a trivial sense there are always multiple binary packages: one for
each hardware architecture Debian supports.

If upstream sources contain material that is not freely distributable
that's usually stripped out before creating the debian source.

The debian patches always include a makefile that knows how to build the
debian packages.

>
> Right now, there are four repos (this was a recent change to
> split out cmdstan). The core Stan functionality is in:
>
> stan-dev/stan
>
> Then the interfaces are in:
>
> stan-dev/{ cmdstan, rstan, pystan }
>
> The Eigen and Boost libraries are included in full in stan-dev/stan,
> and included as is as part of the cmdstan distribution. But a simple
> makefile flag lets you use other versions.
>
> RStan trims the libs down to just what's needed for RStan and I
> believe now depends on the BH library for boost (and RcppEigen
> for Eigen?). But again, RStan can be built and run with other lib
> versions.
>
> I'm not sure what PyStan does for library dependencies.
>
> What specifically would we need to change about all of this?

I don't know. With the separation of the core stan functionality and
the ability to use external libraries things may be unsnarled enough.
If they do not actually share a common core, for example rstan uses
different libraries in place of core Stan, it might be good to change
that.

Both R libraries and Debian packages have ways of expressing dependence
on other libraries and packages; I think R packages also have a way to
at least note dependencies on things (e.g., compilers) other than R
packages.

Ross

[Ross Boylan]

Here's the current granularity of boost packaging on Debian:
libboost1.49-doc, libboost-chrono1.49-dev, libboost-date-time1.49-dev,
libboost-filesystem1.49-dev, libboost-graph-parallel1.49-dev,
libboost-graph1.49-dev, libboost-iostreams1.49-dev,
libboost-locale1.49-dev, libboost-math1.49-dev, libboost-mpi1.49-dev,
libboost-program-options1.49-dev, libboost-python1.49-dev,
libboost-random1.49-dev, libboost-regex1.49-dev,
libboost-serialization1.49-dev, libboost-signals1.49-dev,
libboost-system1.49-dev, libboost-test1.49-dev, libboost-thread1.49-dev,
libboost-timer1.49-dev, libboost-wave1.49-dev, xsltproc, doxygen,
docbook-xml (>= 4.2), docbook-xsl (>= 1.73.2), default-jdk (>= 1.4), fop
(>= 0.94)

You can get these all at once by installing the libboost-dev package,
which depends on these parts.
Ross

[Ben Goodrich]

On Tuesday, August 5, 2014 3:03:18 PM UTC-4, Bob Carpenter wrote:

As a largely header only, Stan can't be separately compiled
and then linked later.  

Compiling a model file is the worst --- it depends on everything
but the parser and code generator (Spirit Qi and its bind and
variant library dependencies).

Yes, but not on the documentation or the tests or the example models, which could also be separate Debian packages that are suggested but not required. Making separate packages out of those wouldn't entail changing the Stan repo though. Having nomad be in a separate repo would be good because it might be useful without the rest of Stan. Maybe stanc too.

Part of the reason for this is that if there were a need to update one package (whether by us or by the Debian package maintainer) then you don't need to update everything. For example, the Debian package maintainer might choose to backport a fix from the develop branch for a parser bug without rereleasing libstan.
 

Running RStan requires having all these things at hand.

RStan and PyStan Debian packages are a bit of a separate issue. If there were official Stan Debian packages, then making RStan and PyStan Debian packages would be easy because they would depend on the Stan packages and basically just add the interface code. But making official Stan Debian packages is pretty involved. We could have unofficial Debian packages that basically do what the interfaces do now by each bundling all of Stan as a git submodule, but I don't think there is much benefit to unofficial packages.

Ben

[Bob Carpenter]

Ah, I see what they did. Boost distributes those
packages and builds them separately. Most of Boost
is header-only, but not all of it. Those libs standing
alone are the ones that aren't header-only. That means they
need be compiled on a per-plaform basis and linked rather than
just included in code that uses them.

So far, Stan only depends on the header-only part of Boost.

- Bob