Source code of MCDS.exe (the one under Distance 7.3 software) ?

20 views
Skip to first unread message

Hypolais Polyglotte

unread,
Jan 30, 2021, 8:50:04 AM1/30/21
to distance...@googlegroups.com
Hello.

As already explained last year, I'm currently developing a python library to make it simpler and safer to run numerous CDS analyses (Distance is great, but not that easy when you want to run numerous identical analyses (same parameters, at least tens) on multiple samples (tens or hundreds)), while keeping at the same time manual access to all the analyses results and especially output curves for keeping an eye on what's happening (statistically and "birdly" speaking).

Of course, this library is based on calls of the MCDS executable shipped with Distance 7.3 (the file has this timestamp : 2014-03-27, and it is version 6.2.0 according to the log files).

It works great under windows (7, 10) but I'd like to port it to NATIVE Linux (avoiding virtualisation software for obvious simplicity and performances reasons).

So ... would it be possible to get access to MCDS 6.2 source code in order to compile it under Linux ?

Of course, I would give you back any script or makefile that I would build for that (my work has no commercial purpose at all, only for "science", in my free time, for a small cummunity of French Massif Central birders).
And I would also strictly follow your instructions, whatever they are, about how this source code, or else MCDS linux binary, could be shared (or not) with my fellow birders and "distance samplers".

Thanks in advance for your help.

Jean-Philippe MEURET
(long time professional developer and data analyst, free time birder and "distance sampler" in Clermont-Ferrand, France).

PS: I know that older versions of MCDS have been released as source code ... but they are too old :-)

Len Thomas

unread,
Jan 30, 2021, 12:54:29 PM1/30/21
to distance-sampling, hypol...@gmail.com
Hi Hypolais and list,

We're very open to making the source code public again, and will do so
when time allows. Meantime I'm happy to send it to anyone who asks. It
is written mostly in FORTRAN 77 with a small piece of FORTRAN 90.
However, you should be aware of one fundamental limitation in using it:
compiling it requires a commercial numerical library (the IMSL Fortran
Library) -- we have permission to distribute the software for free with
the routine we use from this library compiled in, but not to distribute
the library itself.

Given the above, it one possibly simpler options would be for you to
call the Distance or mrds packages in R from Python (I believe the
Python package rpy2 https://rpy2.github.io/ enables this). Even
simpler would be to work in R rather than Python ;)

I'm happy to correspond further with you off-list, and we can post back
to the list whatever solution we come up with.

Best wishes, Len Thomas
> --
> You received this message because you are subscribed to the Google
> Groups "distance-sampling" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to distance-sampl...@googlegroups.com
> <mailto:distance-sampl...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/distance-sampling/CAP00E0EA6d71Dwe4pHbX5WifS9pcYOS0wZ6p3GarjPLasKYf_Q%40mail.gmail.com
> <https://groups.google.com/d/msgid/distance-sampling/CAP00E0EA6d71Dwe4pHbX5WifS9pcYOS0wZ6p3GarjPLasKYf_Q%40mail.gmail.com?utm_medium=email&utm_source=footer>.

--
Len Thomas len.t...@st-andrews.ac.uk lenthomas.org @len_thom
Centre for Research into Ecological and Environmental Modelling
and School of Mathematics and Statistics
The Observatory, University of St Andrews, Scotland KY16 9LZ
Office: UK+1334-461801 Admin: UK+1334-461842

While I may be sending this email outside of my normal office hours,
I have no expectation to receive a reply outside of yours.

The University of St Andrews is a charity
registered in Scotland, No SC013532.

Len Thomas

unread,
Feb 2, 2021, 3:24:56 PM2/2/21
to distance-sampling, Jean-Philippe
Hi Jean-Philippe

Regarding (A). The IMSL routine provides the optimiser that does CDS
key function + adjustment terms analyses with monotonicity constraints
(it's a bounded nonlinear constraints problem), so it's pretty key. I
guess it would be possible to remove this and only do analyses without
monotonicity constraints, but this would require a bit of engineering of
the code. Any changes to the FORTRAN code are a bit tricky as it is
quite convoluted for historical reasons (e.g., one design criterion when
it was first written was that it should fit within 640k of computer
memory, so there's lots of variable re-use). This is one major
motivation for us to undertake our future improvements in design and
analysis capabilities in R, using newer easier-to-maintain code.

It would likely be much easier to either run it in linux on an emulator
such as WINE, or switch to Windows operating system.

Regarding (B). It is quite possible to call R from the command line and
run batch scripts, in a similar way that mcds is called. (Indeed, this
is what Distance for Windows does - it uses the mcds.exe program for CDS
analyses and R for the other analysis engines.) You're right that it
would take some work to get the same outputs -- but it's quite easy in R
to provide outputs directly as html, Excel, pdf, etc if that was preferred.

Another, different, option would be to move the whole workflow into R.
A simple graphical interface could be built if required using Shiny;
formatted outputs can be created with the knitr package. But I
appreciate that does involve a lot of re-engineering.

Best wishes, Len

(P.S. Steve Buckland tells me that Hypolais is the scientific name of a
genus of birds, so sorry for calling you that by mistake!)

On 02/02/2021 20:05, Jean-Philippe wrote:
> Hello, Len.
>
> Thanks for your kind answer.
>
> A) Regarding MCDS sources :
>
> Well, if IMSL is needed for compiling MCDS, it's a big problem for us,
> as we don't have any financial support for this associative spare-time
> work (not an academic wrok, but still in the interest of the public good).
>
> Because there don't seem to be any alternative open source library able
> to do the same ...
>
> Unless MCDS source code use of IMSL is very limited, only to a few
> simple routines ?
>
> B) Regarding moving to MRDS
>
> Well, there is a big step between these 2 approaches, isn't it ?
> 1) use MCDS (even in a automated way, through the command line) to make
> CDS analyses,
> 2) recode MCDS features in order to produce the same kind of results for
> the same CDS analyses, even with the big and welcome help of the
> "Distance" R package (to speak only about the simplest one of the set).
>
> Unless some clever guys already did this ?
>
> And ... it's been a long and painful work for interfacing to MCDS
> results output formats and build well-formated HTML or Excel reports:
> wouldn't the 2nd approach (recode MCDS on top or Distance R package)
> simply drop this work to the trash bin and force us to do it again from
> scratch ?
>
> Thanks in advance for your answers and advices :-)
>
> Best regards,
>
> Jean-Philippe Meuret.
> Bonjour .
>
> A bientôt.
>
> Jean-Philippe.

Mei-Ling Bai

unread,
Feb 3, 2021, 1:05:03 AM2/3/21
to distance-sampling
Dear distance-sampling group members,

I am using distance in R to analyse our bird survey data. I would like to know the confidence interval of the ESWs for each species. Should I bootstrap them? Or is there some function that can do the job? 

Thanks in advance for your help.

Mei-Ling

Jean-Philippe

unread,
Feb 3, 2021, 2:03:29 AM2/3/21
to Len Thomas, distance-sampling
Hello, Len.

Thanks for your kind answer.

A) Regarding MCDS sources :

Well, if IMSL is needed for compiling MCDS, it's a big problem for us, as we don't have any financial support for this associative spare-time work (not an academic wrok, but still in the interest of the public good).

Because there don't seem to be any alternative open source library able to do the same ...

Unless MCDS source code use of IMSL is very limited, only to a few simple routines ?

B) Regarding moving to MRDS

Well, there is a big step between these 2 approaches, isn't it ?
1) use MCDS (even in a automated way, through the command line) to make CDS analyses,
2) recode MCDS features in order to produce the same kind of results for the same CDS analyses, even with the big and welcome help of the "Distance" R package (to speak only about the simplest one of the set).

Unless some clever guys already did this ?

And ... it's been a long and painful work for interfacing to MCDS results output formats and build well-formated HTML or Excel reports: wouldn't the 2nd approach (recode MCDS on top or Distance R package) simply drop this work to the trash bin and force us to do it again from scratch ?

Thanks in advance for your answers and advices :-)

Best regards,

Jean-Philippe Meuret.

Le 30/01/2021 à 18:54, Len Thomas a écrit :
Hi Hypolais and list,

We're very open to making the source code public again, and will do so when time allows. Meantime I'm happy to send it to anyone who asks.  It is written mostly in FORTRAN 77 with a small piece of FORTRAN 90. However, you should be aware of one fundamental limitation in using it: compiling it requires a commercial numerical library (the IMSL Fortran Library) -- we have permission to distribute the software for free with the routine we use from this library compiled in, but not to distribute the library itself.

Given the above, it one possibly simpler options would be for you to call the Distance or mrds packages in R from Python (I believe the Python package rpy2  https://rpy2.github.io/ enables this).  Even simpler would be to work in R rather than Python ;)

I'm happy to correspond further with you off-list, and we can post back to the list whatever solution we come up with.

Best wishes, Len Thomas

On 30/01/2021 13:49, Hypolais Polyglotte wrote:
Hello.

As already explained last year, I'm currently developing a python library to make it simpler and safer to run numerous CDS analyses (Distance is great, but not that easy when you want to run numerous identical analyses (same parameters, at least tens) on multiple samples (tens or hundreds)), while keeping at the same time manual access to all the analyses results and especially output curves for keeping an eye on what's happening (statistically and "birdly" speaking).

Of course, this library is based on calls of the MCDS executable shipped with Distance 7.3 (the file has this timestamp : 2014-03-27, and it is version 6.2.0 according to the log files).

It works great under windows (7, 10) but I'd like to port it to NATIVE Linux (avoiding virtualisation software for obvious simplicity and performances reasons).

So ... would it be possible to get access to MCDS 6.2 source code in order to compile it under Linux ?

Of course, I would give you back any script or makefile that I would build for that (my work has no commercial purpose at all, only for "science", in my free time, for a small cummunity of French Massif Central birders).
And I would also strictly follow your instructions, whatever they are, about how this source code, or else MCDS linux binary, could be shared (or not) with my fellow birders and "distance samplers".

Thanks in advance for your help.

Jean-Philippe MEURET
(long time professional developer and data analyst, free time birder and "distance sampler" in Clermont-Ferrand, France).

PS: I know that older versions of MCDS have been released as source code ... but they are too old :-)

-- 
You received this message because you are subscribed to the Google Groups "distance-sampling" group.
To unsubscribe from this group and stop receiving emails from it, send an email to distance-sampl...@googlegroups.com <mailto:distance-sampl...@googlegroups.com>.
To view this discussion on the web visit https://groups.google.com/d/msgid/distance-sampling/CAP00E0EA6d71Dwe4pHbX5WifS9pcYOS0wZ6p3GarjPLasKYf_Q%40mail.gmail.com <https://groups.google.com/d/msgid/distance-sampling/CAP00E0EA6d71Dwe4pHbX5WifS9pcYOS0wZ6p3GarjPLasKYf_Q%40mail.gmail.com?utm_medium=email&utm_source=footer>.


-- 
Bonjour .

A bientôt.

Jean-Philippe.

Eric Rexstad

unread,
Feb 3, 2021, 3:11:18 AM2/3/21
to Mei-Ling Bai, distance-sampling

Mei-Ling

Thanks for your question; we received a very similar question via Github a month ago asking for confidence intervals for Effective Detection Radius for point counts.  Getting confidence intervals for Effective Strip Width is a bit simpler, because the output from the 'ds()' function provides you with measures of precision for P_a.  The coefficient of variation (CV) for P_a and ESW are identical.

Here is the rest of the calculation to produce the confidence interval bounds from the result of a call to the 'ds()' function

alpha <- 0.05
conv <- convert_units("meter", "kilometer", "square kilometer")
data("LTExercise")
d <- ds(LTExercise, convert.units = conv)
sumd <- summary(d)
p_a <- sumd$ds$average.p
se.p_a <- sumd$ds$average.p.se
cv.p_a <- se.p_a / p_a
w <- sumd$ds$width
esw <- p_a * w
degfree <- sumd$ds$n - length(sumd$ddf$par)
t.crit <- qt(1 - alpha/2, degfree)

se.log.esw <- sqrt(log(1 + cv.p_a^2))
c.mult <- exp(t.crit * se.log.esw)
(ci.esw <- c(esw / c.mult, esw * c.mult))

--
You received this message because you are subscribed to the Google Groups "distance-sampling" group.
To unsubscribe from this group and stop receiving emails from it, send an email to distance-sampl...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/distance-sampling/CAESF0TJZLY%2BiM4jfOiAYF8ccd48W%2BbOtpfD5-nuST4C4O10mrQ%40mail.gmail.com.
-- 
Eric Rexstad
Centre for Ecological and Environmental Modelling
University of St Andrews
St Andrews is a charity registered in Scotland SC013532

Len Thomas

unread,
Feb 3, 2021, 4:13:28 AM2/3/21
to distance...@googlegroups.com
In case it's helpful: on Github the question and solution code (which
Eric wrote in the form of a function, and has an accompanying pdf
illustrating its use) is at

https://github.com/DistanceDevelopment/mrds/issues/36

and we've undertaken to include this functionality in release of the
mrds package due on 1st December this year
(https://github.com/DistanceDevelopment/mrds/milestone/3) which will
then automatically feed through to the release of the Distance package
due on the same day.

Cheers, Len
>> <mailto:distance-sampl...@googlegroups.com>.
>> <https://groups.google.com/d/msgid/distance-sampling/CAESF0TJZLY%2BiM4jfOiAYF8ccd48W%2BbOtpfD5-nuST4C4O10mrQ%40mail.gmail.com?utm_medium=email&utm_source=footer>.
>
> --
> Eric Rexstad
> Centre for Ecological and Environmental Modelling
> University of St Andrews
> St Andrews is a charity registered in Scotland SC013532
>
> --
> You received this message because you are subscribed to the Google
> Groups "distance-sampling" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to distance-sampl...@googlegroups.com
> <mailto:distance-sampl...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/distance-sampling/ec37cfce-5b11-28f9-cbf3-19c28b9ca4a7%40st-andrews.ac.uk
> <https://groups.google.com/d/msgid/distance-sampling/ec37cfce-5b11-28f9-cbf3-19c28b9ca4a7%40st-andrews.ac.uk?utm_medium=email&utm_source=footer>.

--
Len Thomas len.t...@st-andrews.ac.uk lenthomas.org @len_thom
Centre for Research into Ecological and Environmental Modelling
and School of Mathematics and Statistics
The Observatory, University of St Andrews, Scotland KY16 9LZ
Office: UK+1334-461801 Admin: UK+1334-461842

While I may be sending this email outside of my normal office hours,
I have no expectation to receive a reply outside of yours.

The University of St Andrews is a charity
registered in Scotland, No SC013532.

David Lawrence Miller

unread,
Feb 16, 2021, 8:49:12 AM2/16/21
to distance-sampling, Jean-Philippe
Hi Jean-Philippe, hi listfolk,

Just wanted to follow-up on this as I've spent the last few weeks
running MCDS.exe via wine on macOS and Linux (Debian 10). Generally
speaking this seems to work fine (though note that for more recent
versions of macOS that have dropped 32-bit support, this version of wine
is needed https://github.com/Gcenx/homebrew-wine). I ran about 600
models with no big issues. There is some cost to "spinning-up" the wine
session, but it's only a few seconds (and lower on Linux than macOS,
likely because of the 32/64-bit issue). That could probably be minimised
by writing all of the calls to MCDS.exe into a batch file and calling
that once from wine (I've not tested this).

I've added a note on this to the Distance website here:
http://distancesampling.org/distanceextras.html#running-distance-on-other-operating-systems

Hope this it helpful.

best,
--dave
Reply all
Reply to author
Forward
0 new messages