[ADMB Users] ABMD & DLLs

[Laurie Kell]

I´m compiling DLLs using both MSVC and MinGW, this is because of the
better debugging facilities of MSVC and the need to use MinGW for
compiling ADMB code in an R package.

However, there is a problem compiling and linking a DLL with MSVC. I´m
using "admb-9.1-windows-vc9-32bit.exe" and Microsoft Visual C++ 2008 i.e.

Microsoft Visual Studio 2008
Version 9.0.30729.1 SP
Microsoft .NET Framework
Version 3.5 SP1

As adlink.bat goes into an infinite loop at lines 31 to 37 i.e.

:STARTLOOP
if [%2]==[] goto ENDLOOP
if %1==-r set re=1& shift
REM if %1==-s set s=& set adlib=ads32.lib& set
df1b2lib=df1b2stubs.lib& shift
if %1==-s set s=& shift
goto STARTLOOP
:ENDLOOP

since there is no DLL option for "-d" as there is for MinGW version
which includes a "-d" option i.e

if %1==-d set linker=dllwrap& shift

Laurie
_______________________________________________
Users mailing list
Us...@admb-project.org
http://lists.admb-project.org/mailman/listinfo/users

[Arni Magnusson]

Hi Laurie,

When I wrote the compilation scripts for GCC (Windows and Linux) and
Borland (Windows), I was striving for a uniform interface. Before those
scripts, there was a wide variety of scripts with different names, and
some worked while others didn't.

There are still a couple of remaining issues:

1. I don't have MSVC, so at the time I asked if someone could create
similar scripts for MSVC, that would be great. I was using MSVC ten years
ago, and I agree its debugging facilities are much better than GDB. The
main reason I'm using GCC is so I can compile the same code in Windows and
Linux.

I'm guessing Johnoel Ancheta created the compilation scripts for MSVC, but
I'm not sure. In other words, if someone finds out the incantation to
create DLLs with MSVC, please email updated compilation scripts to this
mailing list. Johnoel will then commit them to the repository, and the
ADMB community will thank you.

2. The ability to create DLLs in ADMB is limited. Here's my experience:

Windows GCC: yes
Windows Borland: no
Windows MSVC6: yes (old ADMB version 5.01 from 2000)
Windows MSVC8: ?
Linux GCC: no

When I was writing the compilation scripts, I inquired about this and Dave
Fournier said that implementing the DLL support had been a pain in the
neck in the past, and he didn't sound enthusiastic about it. You can use
normal executables instead of DLLs in an R package, as demonstrated by the
glmmADMB package
(http://otter-rsch.com/admbre/examples/glmmadmb/glmmADMB.html).

I've created ADMB DLLs using Windows GCC and MSVC6 and called them from R
to demonstrate what ADMB can do, but it requires special dll_* objects as
you have already found out. I'm honestly not sure how much work it is to
rewrite an existing model (say 1000 lines of TPL code) to use dll_*
objects. Maybe it's less work than I imagine.

The DLL support for Linux GCC and Windows Borland are almost working, so a
skilled C++ programmer could probably fix the parts that are broken. I'm
not sure what the status of DLL support is for MSVC8.

Both of these issues (checking if MSVC can create ADMB DLLs, and adding
DLL support for other compilers and platforms) are opportunities to
contribute to the ADMB project.

Hope this helps,

Arni

[Laurie Kell]

Thanks Arni for your quick response.

My interest is in running ADMB DLLs from R under Windows, Linux and Mac
operating systems,mainly using GCC. I also have both the full and the
free express version of MSVC and I have been able to compile ADMB DLLs
using these so I should be able to create a script. However, I have had
other problems in linking and in running with some versions of the ADMB
libraries . But as I can compile the DLLs then it should be possible to
track down runtime errors especially as we now have access to the ADMB
source.

I have had a bit of experience in running ADMB DLLs, well at least I
know the problems and have found some solutions. Modifying the TPL files
is not too difficult and I can include my own C++ classes within the
ADMB C++ code. The main problem is that most of the work is in modifying
the C++ code not the TPL. Maybe if I can find some generic solutions
then it will be possible at a future date to make these part of the TPL
code.

Thanks Laurie

_______________________________________________

[Arni Magnusson]

I haven't been able to create working ADMB DLLs in Linux, but it may be
possible in some ADMB releases. To make your R package run in Linux, you
may have to include the model as an executable like the glmmADMB example,
unless you can get the Linux DLL to work.

I just remembered an additional positive entry in my DLL success table,
with an old Borland-ADMB setup:

Windows GCC: yes
Windows BCC52: yes (old ADMB version 5.0.1 from 2000)
Windows BCC55: no
Windows MSVC6: yes (old ADMB version 5.0.1 from 2000)

Windows MSVC8: ?
Linux GCC: no

The fact that Windows BCC52 could create working DLLs makes me believe
that this capability can be revived for the current Windows BCC55.
Likewise, I'm sure DLLs once worked for Linux GCC and can be fixed. I can
create non-ADMB DLLs with Linux GCC that run fine in R, so the problem is
in ADMB.

Laurie, it sounds like you have the ambition and skills to contribute in
this area. Please share your findings and improvements!

Arni

On Tue, 16 Feb 2010, Laurie Kell wrote:

> Thanks Arni for your quick response.
>
> My interest is in running ADMB DLLs from R under Windows, Linux and Mac
> operating systems,mainly using GCC. I also have both the full and the
> free express version of MSVC and I have been able to compile ADMB DLLs
> using these so I should be able to create a script. However, I have had
> other problems in linking and in running with some versions of the ADMB
> libraries . But as I can compile the DLLs then it should be possible to
> track down runtime errors especially as we now have access to the ADMB
> source.
>
> I have had a bit of experience in running ADMB DLLs, well at least I
> know the problems and have found some solutions. Modifying the TPL files
> is not too difficult and I can include my own C++ classes within the
> ADMB C++ code. The main problem is that most of the work is in modifying
> the C++ code not the TPL. Maybe if I can find some generic solutions
> then it will be possible at a future date to make these part of the TPL
> code.
>
> Thanks Laurie
>

[Laurie Kell]

Yes, I´ll try and help. I´m already talking and working with Anders and
I am also working with Iago Mosqueira on Linux and automatic compilation
as part of R packages.

I think we have a good idea of how to progress, might just need a bit of
help with the ADMB classes that handle IO.

But I think having ADMB as a tool in R will be extremely useful

Laurie

[Laurie Kell]

I have managed to compile windows MSVC9 DLLs, both on the command line
and using the IDE with debugging info. I will update the adlink.bat for
MSVC9

However, using the simple example to create the DLL there is a problem
when calling the DLL from R for versions compiled in MSVC9 and GCC
(windows and linux).

The MSVC9 dll crashes at line

mp.computations(argc,argv);

While under GCC (both windows and linux) although the DLL runs i.e.
.C("simpleDLL", ...)

it crashes R if you try to save the results i.e.
res<-.C("simpleDLL", ...)

I think this points to a memory allocation or a calling convention
problem in the ADMB code. I´ve had a quick look at the code on
http://code.google.com/p/admb-project

But think it might be easier if somebody who is more familar with the
source than me could have a look at this problem.

Laurie

[Arni Magnusson]

I had experimented with DLL compilation, but never enough to notice this
bug. By the way, the assignment call does work if it's the first thing in
the session,

x <- rep(0,100)
f <- 0
dyn.load("simpleDLL.dll")
res <- .C("simpleDLL", as.integer(length(x)), as.double(x),
as.double(f), "")

but then a plain subsequent run will crash:

.C("simpleDLL", as.integer(length(x)), as.double(x), as.double(f), "")

It is therefore conceivable that an R package would work fine, since
package functions would always make an assignment call, and never make a
plain call without assigning the result to an object name.

R does not have this problem with non-ADMB DLLs I have tried, so it
appears that all ADMB DLLs are at least half broken. I still believe this
can be fixed by a skilled C++ programmer, with hints from Dave Fournier.
Until then, Johnoel will probably tag this bug report with a number.

Laurie, as an encouragement to improve ADMB in this area, can you describe
the benefits of using DLLs as opposed to plain executables? I have only
experimented with this a bit, and tried out the R package 'glmmADMB' which
uses plain executables. I'm guessing DLLs are a tidier connection than
passing text files to and from executables, but the latter approach can
achieve the same goal by adding a couple of lines to the R function to
take care of temporary files, right? The time penalty should be
negligible; I just tried writing and reading a 5 column x 1000 row dataset
in R,

write.table(quakes, "quakes.txt", row.names=FALSE)
quakes2 <- read.table("quakes.txt", header=TRUE)

and it takes 0.06 sec on Windows and 0.02 sec on Linux 64, even when using
the inefficient read.table and write.table functions. File handling on the
ADMB side should be very fast as well.

The benefit of using plain executables is of course that the same model
can be used either as stand-alone or called by other software via text
files. If the medium-term goal is to fix the ADMB DLLs, then a long-term
goal could be to make it possible to compile both DLLs and executables
without modifying the TPL code.

Thanks for exploring this, and for improving the MSVC compilation scripts.

Arni

[Laurie Kell]

Hi

I´ll download the ADMB source code and try and track the bug It might be
something simple such as an errant print statement. I agree that a
worked example of an ADMB/R package with a discussion of benefits would
be useful eg.
The benefits of using ADMB rather than R non-linear solvers (e.g.
optim/minpack.lm), the benefits of using R for graphics, data-bases, and
linking with other tools. Then a discussion of object orientated
programming in R, i.e. S4 classes and FLR.

I think a candidate for this example could be Pella-T example.

Laurie

_______________________________________________

[dave fournier]

You could try unloading and reloading the the DLL before you call it
the second time.
A problem with ADMB is that a lot of global variables etc are
initialized when you
execute an exe or load it as a DLL. However when you leave the DLL and
then
reenter it the global variables are not reinitialized. I think this
is the origin of most of the problems. There may also be some other
more subtle issues.

Dave

> > problem in the ADMB code. I�ve had a quick look at the code on

> >http://code.google.com/p/admb-project
>
> > But think it might be easier if somebody who is more familar with the
> > source than me could have a look at this problem.
>
> > Laurie
>
>
>

> _______________________________________________
> Users mailing list
> Us...@admb-project.orghttp://lists.admb-project.org/mailman/listinfo/users

[Arni Magnusson]

Laurie raised a few points besides DLL compiling:

> ... a worked example of an ADMB/R package with a discussion of benefits

> would be useful eg. The benefits of using ADMB rather than R non-linear
> solvers (e.g. optim/minpack.lm), the benefits of using R for graphics,
> data-bases, and linking with other tools. Then a discussion of object
> orientated programming in R, i.e. S4 classes and FLR.
>
> I think a candidate for this example could be Pella-T example.

The benefits of using ADMB rather than R nonlinear solvers like
optim/minpack.lm are mainly speed, reliability, and built-in features like
random effects and MCMC.

Benchmark results like http://otter-rsch.ca/SchnuteRichardsOlsen1998.pdf,
http://otter-rsch.com/admbre/examples/logistic/logistic.html, and
http://otter-rsch.com/admbre/examples/sdv/sdv.html indicate that ADMB is
faster than Gauss, Matlab, S-Plus, and BUGS. I agree with Laurie, that
given the current variety of nonlinear solvers available in R, it would be
interesting to benchmark them against ADMB using a Pella-Tomlinson model.

If you want random effects and/or MCMC inside R, the functions are
limiting compared to ADMB. R packages like nlme, lme4, MCMCglmm, MCMCpack,
and mcmc are practical for many things, but can only express a small
subset of the models that can be built with ADMB. I have found this
especially true for models that are time-structured but not basic
time-series models.

R is excellent for many things, but there are some tasks that are better
performed by more specialized software, like databases or spreadsheets, or
a programming language with particularly strong support for specific
tasks, like ADMB and Perl. Even expert R programmers recognize its
limitations, and delegate some tasks to other languages, including
C/C++/Fortran to crunch numbers, and Perl to build packages.

If I was to do a Pella-Tomlinson assessment, I would probably start with a
quick Excel prototype, to get some initial parameter values and to verify
later that I get the same numbers in ADMB. Then I would write the model in
ADMB (probably borrow from the existing pella-t.tpl example), diagnose and
plot the model fit in R, run ADMB -mcmc, diagnose and plot the MCMC
results in R, evaluate possible transformations and reparametrizations,
and then plot and summarize the final results in R. Even though R is my
preferred programming language, I would use Excel and ADMB for the things
they do better. There are many other candidate tools, but these are the
ones I would use.

The benefits of ADMB become more important when dealing with models with
hundreds of parameters and very large datasets.

I have only used optim() and other nonlinear solvers in R to satisfy my
curiosity, never for work. The approach is very similar to ADMB: prepare
data objects, define parameters, make predictions, and evaluate the
objective function. Even the code will look quite similar, but in my
opinion I have a more reliable and versatile tool if I implement a model
in ADMB.

Of course, it would be nice to back up this opinion with a Pella-Tomlinson
benchmark. To test the "reliable" part, we would need a difficult dataset
with local minima. Does anyone know of a suitable dataset?

Arni

P.S. I'm looking forward to trying out the FLR
(http://r-forge.r-project.org/R/?group_id=318) way to do Pella-Tomlinson
and other assessment models. The FLash package is particularly interesting
to ADMB users, as it implements automatic differentiation for other FLR
packages using ADOL-C, although it appears not quite ready for testing.
FLR is an ambitious project, and even if the S4 classes take some of the
playfulness out of R, they're probably the best approach to manage a
project of this scope.

[Arni Magnusson]

Yes, the Pella-Tomlinson example
(http://code.google.com/p/admb-project/source/browse/trunk/examples/admb/pella-t/)
highlights a relevant issue: how hard is it to rewrite an existing ADMB
model to compile to DLL instead of plain executable? I don't know the
answer, but it looks like there's more to it than simply renaming all the
data and parameter objects (init_int to dll_init_int, etc.), because that
doesn't compile. I'm not sure if dll_init_bounded_dev_vector can be used.

A smaller example, converting the random-effects simple.tpl (ADMB-RE
manual, section 2.3) to compile to DLL does not appear straightforward
either, although tpl2rem
(http://code.google.com/p/admb-project/source/browse/trunk/src/df1b2-separable/tpl2rem.lex)
recognizes many dll_ objects, including dll_random_effects_vector. I
couldn't get it to compile.

I'm still optimistic that DLL support can be improved in future versions
of ADMB, if the community thinks it's an important goal. Admittedly, DLLs
seem more nifty than necessary ... tempting to fix, though, since they're
only half-broken. DLLs make it easier for other software to interact with
ADMB in ways we cannot imagine yet. Open source tends to spark emerging
properties that way.

Out of historical curiosity, I have kept ADMB version 5 from the year 2000
on my computer, to benchmark against current versions. I just tested
compiling DLLs from those old versions (Borland 5.2, GCC 2.95.2, MSVC 6.0)
and they all work perfectly in R. Dave Fournier knows best whether it may
be useful to release ancient ADMB source code, as a study guide for those
who want to improve the current DLL support, or if it's better to go
bug-hunting without the ghosts of ADMB past.

Arni

[Laurie Kell]

I think Arni´s idea is worth puruing and for a test dataset I recommend
that used in the Polacheck at al 1993 paper, this has data sets for rock
lobster, Namibian hake and albacore and I already have it in a file, I
can also generate data sets with known parameters for testing,

It would also be useful to compare the various Bayesian options

Laurie