Hessian is singular

[Jorge A. Ahumada]

Hello,

I am trying to estimate detection probability and occupancy for a community of species from camera trap data and I tried to do it initially for three species as a test. However, I keep getting this error:

ufmodel<-occu(~sp.fac2 ~sp.fac2,uftest)
Error: Hessian is singular.  Try using fewer covariates.

My only covariate is species, so not really an option. My y matrix is large (58 sites x 73 days) and the 58 sites were not sampled simultaneously but sequentially in two groups, so there are a lot of NAs in the matrix. Naïve occupancy is pretty low (~0.04), so I guess this might be contributing to the problem.

Should I group the data in groups of  3-6 days?  How can I specify starting values for the parameters in a 0-1 scale? Any other ideas?

I can send the data upon request..

Thanks,

Jorge

--
Jorge A. Ahumada
Technical Director
Tropical Ecology Assessment and Monitoring Network (TEAM)
Science and Knowledge Division
Conservation International
2011 Crystal Dr. Suite 500
Arlington, VA 22202
Phone: 703-341-2543
Mobile: 202-716-3374
http://www.teamnetwork.org/

[Ian Fiske]

The problem might be that you have specified the same covariate for both levels of the model.  That could cause some parameters to become non-identifiable.  Also, did you try fitting the model without any covariates to see if that works?

ufmodel<-occu(~1 ~1,uftest)

Ian

[Jorge A. Ahumada]

No, I tried that; same problem..

Jorge

[Richard Chandler]

Hi Jorge,

Can you post the results of

summary(uftest)

With a naive occupancy estimate of 0.04, you might be out of luck. But feel free to send me the data.

Richard

[Richard Chandler]

Hi Jorge,

It looks like you just need to provide starting values. This worked for me:

fm1 <- occu(~1 ~1, uftest, starts=c(0, -3))
fm2 <- occu(~1 ~sp.fac, uftest, starts=c(0, 0, 0, -3))
fm3 <- occu(~sp.fac ~1, uftest, starts=c(0, -3, 0, 0))
fm4 <- occu(~sp.fac ~sp.fac, uftest, starts=c(2, 0, 0, -3, 0, 0))

# Species-specific psi and p from fm4
newdata <- data.frame(sp.fac = levels(siteCovs(uftest)$sp.fac))
predict(fm4, type="state", newdata=newdata)
predict(fm4, type="det", newdata=newdata)


Richard

On Tue, Nov 2, 2010 at 10:03 AM, Jorge A. Ahumada <j.ah...@conservation.org> wrote:

Hi Richard,

Yes occupancy is very low. The dataset is in the workspace file attached.

> summary(uftest)

unmarkedFrame Object

174 sites
Maximum number of observations per site: 73
Mean number of observations per site: 31.48
Sites with at least one detection: 82

Tabulation of y observations:
   0    1 <NA>
5327  151 7224

Site-level covariates:
                      sp.fac  
 Macaca nemestrina       :58  
 Muntiacus muntjak       :58  
 Symphalangus syndactylus:58  

Interestingly when I fit this using my own code (I specified the conditional probability and then used mle2()
to find the parameters) I get a solution. I fitted each species separately though – here is an example of the output from the model fit:

$`Macaca nemestrina`

Call:
mle2(minuslogl = f.like.ss, start = list(phi = phi, p = p), method = "L-BFGS-B",
    data = list(d = data), lower = c(phi = 0.001, p = 0.001),
    upper = c(phi = 0.999, p = 0.999))

Coefficients:
       phi          p
0.86286712 0.04641301

Log-likelihood: -305.68

[Jorge A. Ahumada]

Thanks, so I guess when p or phi are really low, the program needs starting values.. The se=FALSE option does put me in lala land.
In my case I can use some other code, but what if you don’t have any? How would you go about providing starting values? I guess this is the ‘art’  part of statistical modeling...

Jorge


On 11/2/10 10:52 AM, "Richard Chandler" <richard....@gmail.com> wrote:

Yes, I applied a logit transformation to your estimates from mle2.

> qlogis(0.04)
[1] -3.178054

> qlogis(0.86)
[1] 1.81529

You could also try using something like:

occu(~1~1, uftest, se=FALSE)


Although you might wind up in lala land.

Richard




On Tue, Nov 2, 2010 at 10:43 AM, Jorge A. Ahumada <j.ah...@conservation.org> wrote:

Thanks..!
Really stupid question: How do I know which values to provide? Are these logit transforms for occ and dp? exp(starting value?)

Jorge

[Richard Chandler]

Hi Jorge,

If you need to provide starting values, you can generally come up with a good guess by taking a look at the raw data and starting with a simple model. If it looks like p will be low, try a low starting value. Then you should make sure that other starting values get you to the same MLEs.

Richard

[Richard Chandler]

---------- Forwarded message ----------
From: Carlos <carlosh...@gmail.com>
Date: Thu, Jun 16, 2011 at 6:39 PM
Subject: Re: Hessian is singular
To: Richard Chandler <richard....@gmail.com>


Dear all,
I´ve gotten the same problem (“Error: Hessian is singular.  Try using
fewer covariates.”). Nevertheless, the situation is quite different.
It happened in pcount function and by using continuous covariable.
The null model worked properly as well as other models. Nevertheless,
the covarible “forest” is similar with other ones and it doesn´t work,
even using only it. It is not possible use less covariable otherwise
it becomes the null model.
Regard to Richard´s advice for dummy covariable (provide starting
values), the solution I tried for continuous covariable neither work
nor I found specific post in this forum about it. Sorry for the basic
questions and for recovering this subject again. Every comment for
this case (detailed below) would be very helpful.
Thank you very much, Carlos.

THE PROBLEM:
> javaMforest=pcount(~1~forest.nat,java.winter,K=24)

Error: Hessian is singular.  Try using fewer covariates.

> javaMforest=pcount(~1 ~forest.nat,java.winter,K=24,starts=c(0,1))
Error in pcount(~1 ~ forest.nat, java.winter, K = 24, starts = c(0,
1)) :
 The number of starting values should be 49
> javaMforest=pcount(~1 ~forest.nat,java.winter,K=24,starts=c(0,rep(1,48)))

Error: Hessian is singular.  Try using fewer covariates.

DATA:
> summary(java.winter)
unmarkedFrame Object
91 sites
Maximum number of observations per site: 30
Mean number of observations per site: 27.25
Sites with at least one detection: 25
 DETAILED DATA:
Point count data of wild boar per day with camera trap , during at
maximum of 30 days on. That is, max. 30 observed replication
(occasions). The missing data (NA) happened due to camtrap
malfunctions.  The total independent site was 91.
The sites were grouped in 6 regions with different conditions. An
important detail: The project was replicated 2 times within 1 year
interval. The “year” was treated as a dummy covariable. So that, all
data set was treated as closed population, because I´m interested in
the avagered population size of each region/year instead of
colonization and extinction by site. It was also unable to keep the
same site location every year for colext function. Nevertheless,
criticism about it is very welcome.
K estimated by:
> max(java.y, na.rm = T)+20
[1] 24
> str(java.winter)
Formal class 'unmarkedFramePCount' [package "unmarked"] with 5 slots
 ..@ y       : int [1:91, 1:30] 0 0 0 0 0 0 0 0 0 0 ...
 .. ..- attr(*, "dimnames")=List of 2
 .. .. ..$ : NULL
 .. .. ..$ : chr [1:30] "V1" "V2" "V3" "V4" ...
 ..@ obsCovs : NULL
 ..@ siteCovs:'data.frame':    91 obs. of  8 variables:
 .. ..$ site      : Factor w/ 57 levels "3B1004","3B1005",..: 7 9 10
1 2 3 4 5 6 11 ...
 .. ..$ hist      : int [1:91] 5 5 5 5 5 5 5 5 5 4 ...
 .. ..$ head      : Factor w/ 4 levels "0,02","0,03",..: 4 4 4 4 4 4
4 4 4 1 ...
 .. ..$ ranch     : Factor w/ 2 levels "0,00","0,01": 2 2 2 2 2 2 2 2
2 1 ...
 .. ..$ forest.nat: Factor w/ 48 levels "0,00","0,04",..: 37 2 3 44
42 34 45 19 29 38 ...
 .. ..$ forest.exo: Factor w/ 27 levels "0,00","0,01",..: 1 27 26 1 1
1 1 23 1 1 ...
 .. ..$ crop      : Factor w/ 38 levels "0,00","0,01",..: 16 3 4 9 10
19 7 1 17 14 ...
 .. ..$ year      : int [1:91] 0 0 0 0 0 0 0 0 0 0 ...
 ..@ mapInfo : NULL
 ..@ obsToY  : num [1:30, 1:30] 1 0 0 0 0 0 0 0 0 0 ...


On 2 nov 2010, 11:30, Richard Chandler <richard.chandl...@gmail.com>
wrote:

> > On 11/2/10 8:49 AM, "Richard Chandler" <richard.chandl...@gmail.com>

> > wrote:
>
> > Hi Jorge,
>
> > Can you post the results of
>
> > summary(uftest)
>
> > With a naive occupancy estimate of 0.04, you might be out of luck. But feel
> > free to send me the data.
>
> > Richard
>
> > On Mon, Nov 1, 2010 at 10:37 PM, Jorge A. Ahumada <

> > j.ahum...@conservation.org> wrote:
>
> > No, I tried that; same problem..
>
> > Jorge
>

> > On 11/1/10 10:29 PM, "Ian Fiske" <ianfi...@gmail.com <

[Richard Chandler]

Hi Carlos,

Your continuous covariate "forest.nat" is being treated as a factor by R (ie a categorical variable). Try importing the data using something like:

mydata <- read.csv(blah.csv, dec=",")

Then try standardizing "forest.nat". Starting values should not be necessary.

Hope this works,
Richard


_____________________________________
Richard Chandler, post-doc
USGS Patuxent Wildlife Research Center
301-497-5696

From:	Richard Chandler <richard....@gmail.com>
To:	Unmarked package <unma...@googlegroups.com>
Date:	06/17/2011 08:45 AM
Subject:	[unmarked] Fwd: Hessian is singular
Sent by:	unma...@googlegroups.com

---

[Carlos]

Dear Richard,
You are sure. it was wrongly set as a factor, sorry.
thanks for your comment. It was very helpful.
cheers,
Carlos

On Jun 17, 9:54 am, Richard Chandler <rchand...@usgs.gov> wrote:
> Hi Carlos,
>
> Your continuous covariate "forest.nat" is being treated as a factor by R
> (ie a categorical variable). Try importing the data using something like:
>
> mydata <- read.csv(blah.csv, dec=",")
>
> Then try standardizing "forest.nat". Starting values should not be
> necessary.
>
> Hope this works,
> Richard
>
> _____________________________________
> Richard Chandler, post-doc
> USGS Patuxent Wildlife Research Center
> 301-497-5696
>
> From:
> Richard Chandler <richard.chandl...@gmail.com>
> To:
> Unmarked package <unma...@googlegroups.com>
> Date:
> 06/17/2011 08:45 AM
> Subject:
> [unmarked] Fwd: Hessian is singular
> Sent by:
> unma...@googlegroups.com
>
>
>
>
>
>
>

> ---------- Forwarded message ----------
> From: Carlos <carloshsalva...@gmail.com>
> Date: Thu, Jun 16, 2011 at 6:39 PM
> Subject: Re: Hessian is singular
> To: Richard Chandler <richard.chandl...@gmail.com>
>
> Dear all,

> I´ve gotten the same problem (?Error: Hessian is singular.  Try using
> fewer covariates.?). Nevertheless, the situation is quite different.

> It happened in pcount function and by using continuous covariable.
> The null model worked properly as well as other models. Nevertheless,

> the covarible ?forest? is similar with other ones and it doesn´t work,

> interval. The ?year? was treated as a dummy covariable. So that, all

> > > though ? here is an example of the output from the model fit:

[seanh...@trentu.ca]

"Hessian is singular" can also be caused by having words in a matrix of numbers - for example if you forget to take out an "NA", R will convert the entire matrix to a factor, and produce the aforementioned error