Error in .local(obj, coefficients, ...) : ncol(coefficients) == length(est) is not TRUE

[Josh Twining]

Hello all,

I am trying to use unmarked to produce predictive heat maps for probability occurence of three species. Firstly using single species models and then multi-species models afterwards. But first things first, everything was going well until I was trying to use my top single species occu model to predict psi for the rest of the region and I get the error "Error in .local(obj, coefficients, ...) :    ncol(coefficients) == length(est) is not TRUE". I have played around having the sampled sites both in and excluded from the predicted sites file. Also with and without scaling of my initial occupancy model dataframe. My dataset is from repeated sampling (14 - 33 times) of 207 sites. Covariate data is for extracted from land cover maps at a 1km squared scale. 

From my reading of the conversations in the google group, most people who get this error it is when they have site covariates that are factors that are not set to one level, but all my covariates are continuous, and scaled (tried without scaling also). My code is below.

Any advice or help I could get would be massively appreciated, imagine there is something very simple I haven't considered but its been a few days now of banging my head against the desk :) 

All the best

Josh 

Code starts here:

#Data in

y <- read.csv("pinemarten2020noname.csv")

summary(y)

str(y)

siteCovs <- read.csv("Habitat.covariate.sites.only.2020.noname.grass.csv")

umf <- unmarkedFrameOccu(y = y, siteCovs = siteCovs)

#Scaling covariates

umf@siteCovs$bog <- scale(umf@siteCovs$bog)

umf@siteCovs$broadleaf <- scale(umf@siteCovs$broadleaf)

umf@siteCovs$built <- scale(umf@siteCovs$built)

umf@siteCovs$dwarf <- scale(umf@siteCovs$dwarf)

umf@siteCovs$conifer <- scale(umf@siteCovs$conifer)

umf@siteCovs$rough <- scale(umf@siteCovs$rough)

umf@siteCovs$people_km2 <- scale(umf@siteCovs$people_km2)

umf@siteCovs$dwarf <- scale(umf@siteCovs$dwarf)

umf@siteCovs$Grassland <- scale(umf@siteCovs$Grassland)

umf@siteCovs$River <- scale(umf@siteCovs$River)

umf@siteCovs$lat <- scale(umf@siteCovs$lat)

umf@siteCovs$long <- scale(umf@siteCovs$long)

# psi(null) for detection covariate selection

full <- occu(formula = ~ broadleaf + conifer  + built + rough + people_km2 + Grassland

               ~ 1,

             data = umf)

modelList.p <- dredge(full,

                    rank = "AIC")

#full model selection for psi with p selection constant

full <- occu(formula = ~ rough + people_km2

             ~  broadleaf + conifer  + built + rough + Grassland + lat + long + people_km2,

             data = umf)

modelList.psi.p <- dredge(full,

                      rank = "AIC")

topmodel <- occu(formula = ~ rough + people_km2

             ~  broadleaf + conifer + lat + long,

             data = umf)

#read in covariate data for rest of area for predict (contains 14404 rows of land cover map data at 1km scale)

allsites <- read.csv("habitat info full all 2020 updated.csv")

occuPred <- predict(topmodel,

                    type = "state",

                    newdata = allsites,

                    na.rm = TRUE,

                    inf.rm = TRUE)

returns

Error in .local(obj, coefficients, ...) :    ncol(coefficients) == length(est) is not TRUE

[Ken Kellner]

Hi Josh,

It doesn't look like you are doing anything wrong. Can you post the output of

sessionInfo()

and

str(allsites)

Also if you are willing to email me (off-list) the CSV files used in the script I can take a closer look.

Ken

[Josh Twining]

Dear Ken,

Thank you for your quick and useful response.

So firstly

> sessionInfo()
R version 3.6.3 (2020-02-29)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows 10 x64 (build 19042)

Matrix products: default

locale:
[1] LC_COLLATE=English_United Kingdom.1252  LC_CTYPE=English_United Kingdom.1252  
[3] LC_MONETARY=English_United Kingdom.1252 LC_NUMERIC=C                          
[5] LC_TIME=English_United Kingdom.1252    

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base    

other attached packages:
[1] car_3.0-10      carData_3.0-4   MuMIn_1.43.17   unmarked_1.1.1  lattice_0.20-38

loaded via a namespace (and not attached):
 [1] zip_2.1.1         Rcpp_1.0.6        cellranger_1.1.0  compiler_3.6.3    pillar_1.5.1      nloptr_1.2.2.2  
 [7] plyr_1.8.6        forcats_0.5.1     tools_3.6.3       boot_1.3-24       lme4_1.1-26       statmod_1.4.35  
[13] lifecycle_1.0.0   tibble_3.1.0      nlme_3.1-144      pkgconfig_2.0.3   rlang_0.4.10      openxlsx_4.2.3  
[19] Matrix_1.2-18     curl_4.3          parallel_3.6.3    haven_2.3.1       rio_0.5.26        raster_3.4-5    
[25] vctrs_0.3.6       hms_1.0.0         stats4_3.6.3      grid_3.6.3        data.table_1.14.0 fansi_0.4.2      
[31] readxl_1.3.1      foreign_0.8-75    sp_1.4-5          minqa_1.2.4       magrittr_2.0.1    codetools_0.2-16
[37] ellipsis_0.3.1    MASS_7.3-51.5     splines_3.6.3     abind_1.4-5       utf8_1.2.1        stringi_1.5.3    
[43] crayon_1.4.1    

Then str(allsites) showed me there was of course something very obviously wrong, a number of my continuous covariates were factors not numerics. But have now corrected that using for example "allsites$bog<-as.numeric(as.character(allsites$bog))".

> str(allsites)

'data.frame': 14401 obs. of  11 variables:
 $ lat             : num  55.1 54.8 54.9 54.8 54.9 ...
 $ long            : num  -7.64 -7.47 -7.47 -7.46 -7.46 ...
 $ bog             : num  0 0 0 0.0086 0.0106 0 0 0 0 0 ...
 $ broadleaf       : num  0.0256 0 0 0 0.055 ...
 $ built           : num  0.056 0.0257 0 0.0505 0.043 0 0 0 0.0284 0.0339 ...
 $ conifer         : num  0.028 0.135 0 0 0 ...
 $ dwarf           : num  0.0275 0 0 0.0156 0 0 0 0 0 0 ...
 $ rough           : num  0 0 0 0 0 ...
 $ River.Length..m.: num  705 809 0 3107 2101 ...
 $ people_km2      : num  164.62 23.89 27.94 1296.72 4.74 ...
 $ Grassland       : num  0.863 0.84 1 0.925 0.891 ...

Now I try and run: 

> occuPred <- predict(topmodel,
+                     type = "state",
+                     newdata = allsites,
+                     na.rm = TRUE,
+                     inf.rm = TRUE)
Error: Matrices must have same number of rows in cbind2(.Call(dense_to_Csparse, x), y).

Is unmarked like presence, where my initial occurrence data file (y) and sitecovs data should have the same number of rows as the predict file (allsites, n =14,402) rows just NA'd throughout?

Alternatively. I am thinking the error may be coming from the fact that lat long as in my top model as covariates, but in the predict file, the first two columns are lat long. I have played around having lat, long in once (just first two columns of predict file) and twice (first two columns for location, and in the same position as in original sitecovs file), but did not achieve anything. 

I will send on the data and covariate CSVs and full script privately now, just again a quick thank you for your time and help with this, much appreciated,

All the best

Josh

--
You received this message because you are subscribed to the Google Groups "unmarked" group.
To unsubscribe from this group and stop receiving emails from it, send an email to unmarked+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/unmarked/e7692563-9bc4-43e3-8eeb-865b2a4fa544n%40googlegroups.com.

--

Email: joshuap...@gmail.com / jtwin...@qub.ac.uk

Mob: 07889214732

[Patrick]

Hi all,

Was a solution found for this error? I am finding that it occurs for certain analyses, but not others... I don't think it has anything to do with factor formatting. 

Thanks,

Patrick

[Ken Kellner]

I can't remember if we ended up solving this problem, if Josh sees this perhaps he can remind me. Either way if you can send me (off-list) code and data for a model that gives this error, I am happy to look at it.

Ken

[Josh Twining]

Hi Andreas, Patrick, and Ken

Ken you diagnosed and helped fix my issue which was being caused by NAs in the newdata I wasn't aware of. These were resultant from converting covariates from characters to numeric. Could something similar be happening with the as.factor conversion in Andreas code? 

Something else I can see in Andreas's code which you improved in my approach on this occasion was scaling within the model formula rather than outside it so avoid having to do conversions manually later on when making predictions,

Hope this is of some help! 

Josh

To view this discussion on the web visit https://groups.google.com/d/msgid/unmarked/d9ee0ace-8f62-4d22-81ba-176b30ba7f08n%40googlegroups.com.

--

Postdoctoral Research Scientist,

321 Fernow Hall,

Cornell University,

Ithaca, New York, 

14850

Email: joshuap...@gmail.com / jpt...@cornell.edu 

Mob: (607) 262 2918

[ANDRES MAURICIO DIAZ VALLEJO]

Hi to All,

I tried to convert covariates from characters to numeric, and for something, it was produced NA's... But I have already fixed it!

Thank you for all your help!

You received this message because you are subscribed to a topic in the Google Groups "unmarked" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/unmarked/qIc1P-PKesI/unsubscribe.
To unsubscribe from this group and all its topics, send an email to unmarked+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/unmarked/CA%2Bg%2BShkV_Du%2BpiCwco1Fz5H5oPTL7i8EunTsF-1gYSrhMSeAVg%40mail.gmail.com.

"La información aquí contenida es para uso exclusivo de la persona o entidad de destino. Está estrictamente prohibida su utilización, copia, descarga, distribución, modificación y/o reproducción total o parcial, sin el permiso expreso de Universidad de Antioquia, pues su contenido puede ser de carácter confidencial y/o contener material privilegiado. Si usted recibió esta información por error, por favor contacte en forma inmediata a quien la envió y borre este material de su computador. Universidad de Antioquia no es responsable por la información contenida en esta comunicación, el directo responsable es quien la firma o el autor de la misma."
UdeA

[Patrick Roberts]

Thanks everyone for your input. I seem to have figured it out and it appears to have something to do with having factors levels... Converting the factor to a character and back to a factor seemed to do the trick. 

Thanks again!

To view this discussion on the web visit https://groups.google.com/d/msgid/unmarked/CAH%3DdA%2BZtjEf4m-wMCoyErv1j2Qm1P_YYr2wBsCzy0gt2%2BOE-GQ%40mail.gmail.com.

[Airy González]

I am getting this error and not using characters or factors. It works if I remove the random term from the occupancy model;  if I put it back, the model fits, but predict() won't work. I am happy to send data and the code if someone can help me. 

~Airy

[Airy González]

Sorry, I realized I was too vague. I meant to say  I am not using fixed covariates in the form of characters or factors. I do have a random effect of PlotID. I observed that when I remove the random effect from my psi model, then I am able to use the predict(); however, when I fit my psi model with the random effect, I get the error the others are mentioning when trying to use predict(): Error in .local(obj, coefficients, ...) : ncol(coefficients) == length(est) is not TRUE

~Airy

--
*** Three hierarchical modeling email lists ***
(1) unmarked (this list): for questions specific to the R package unmarked
(2) SCR: for design and Bayesian or non-bayesian analysis of spatial capture-recapture
(3) HMecology: for everything else, especially material covered in the books by Royle & Dorazio (2008), Kéry & Schaub (2012), Kéry & Royle (2016, 2021) and Schaub & Kéry (2022)
---

You received this message because you are subscribed to the Google Groups "unmarked" group.
To unsubscribe from this group and stop receiving emails from it, send an email to unmarked+u...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/unmarked/2f799309-8a53-4591-afb1-927de202b219n%40googlegroups.com.

--

Airy Adriana Peralta

​Ph. D. Student​ & Professional Aide

​Ecology and Evolutionary Biology​

Institute of Arctic and Alpine Research - CU Boulder

​Airy.Gonza...@colorado.edu​

[Ken Kellner]

If you send me the data and code off list I'll take a look.

Ken

> >>>>>> *Error: Matrices must have same number of rows in
> >>>>>> cbind2(.Call(dense_to_Csparse, x), y).*

> >>>>>>> <https://groups.google.com/d/msgid/unmarked/e7692563-9bc4-43e3-8eeb-865b2a4fa544n%40googlegroups.com?utm_medium=email&utm_source=footer>
> >>>>>>> .

> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>> --
> >>>>>> Email: joshuap...@gmail.com / jtwin...@qub.ac.uk
> >>>>>> Mob: 07889214732
> >>>>>>
> >>>>> --
> >>>> You received this message because you are subscribed to the Google
> >>>> Groups "unmarked" group.
> >>>> To unsubscribe from this group and stop receiving emails from it, send
> >>>> an email to unmarked+u...@googlegroups.com.
> >>>> To view this discussion on the web visit
> >>>> https://groups.google.com/d/msgid/unmarked/d9ee0ace-8f62-4d22-81ba-176b30ba7f08n%40googlegroups.com

> >>>> <https://groups.google.com/d/msgid/unmarked/d9ee0ace-8f62-4d22-81ba-176b30ba7f08n%40googlegroups.com?utm_medium=email&utm_source=footer>
> >>>> .

> >>>>
> >>>
> >>>
> >>> --
> >>> Postdoctoral Research Scientist,
> >>> 321 Fernow Hall,
> >>> Cornell University,
> >>> Ithaca, New York,
> >>> 14850
> >>>
> >>> Email: joshuap...@gmail.com / jpt...@cornell.edu

> >>> Mob: (607) 262 2918 <(607)%20262-2918>

> >>>
> >>> --
> >>> You received this message because you are subscribed to a topic in the
> >>> Google Groups "unmarked" group.
> >>> To unsubscribe from this topic, visit
> >>> https://groups.google.com/d/topic/unmarked/qIc1P-PKesI/unsubscribe.
> >>> To unsubscribe from this group and all its topics, send an email to
> >>> unmarked+u...@googlegroups.com.
> >>> To view this discussion on the web visit
> >>> https://groups.google.com/d/msgid/unmarked/CA%2Bg%2BShkV_Du%2BpiCwco1Fz5H5oPTL7i8EunTsF-1gYSrhMSeAVg%40mail.gmail.com

> >>> <https://groups.google.com/d/msgid/unmarked/CA%2Bg%2BShkV_Du%2BpiCwco1Fz5H5oPTL7i8EunTsF-1gYSrhMSeAVg%40mail.gmail.com?utm_medium=email&utm_source=footer>
> >>> .

> >>>
> >>
> >> "La información aquí contenida es para uso exclusivo de la persona o
> >> entidad de destino. Está estrictamente prohibida su utilización, copia,
> >> descarga, distribución, modificación y/o reproducción total o parcial, sin
> >> el permiso expreso de Universidad de Antioquia, pues su contenido puede ser
> >> de carácter confidencial y/o contener material privilegiado. Si usted
> >> recibió esta información por error, por favor contacte en forma inmediata a
> >> quien la envió y borre este material de su computador. Universidad de
> >> Antioquia no es responsable por la información contenida en esta
> >> comunicación, el directo responsable es quien la firma o el autor de la
> >> misma."
> >> UdeA
> >>
> >> --
> >> You received this message because you are subscribed to a topic in the
> >> Google Groups "unmarked" group.
> >> To unsubscribe from this topic, visit
> >> https://groups.google.com/d/topic/unmarked/qIc1P-PKesI/unsubscribe.
> >> To unsubscribe from this group and all its topics, send an email to
> >> unmarked+u...@googlegroups.com.
> >>
> > To view this discussion on the web visit
> >> https://groups.google.com/d/msgid/unmarked/CAH%3DdA%2BZtjEf4m-wMCoyErv1j2Qm1P_YYr2wBsCzy0gt2%2BOE-GQ%40mail.gmail.com

> >> <https://groups.google.com/d/msgid/unmarked/CAH%3DdA%2BZtjEf4m-wMCoyErv1j2Qm1P_YYr2wBsCzy0gt2%2BOE-GQ%40mail.gmail.com?utm_medium=email&utm_source=footer>
> >> .

> >>
> >
>
> --
> *** Three hierarchical modeling email lists ***
> (1) unmarked (this list): for questions specific to the R package unmarked
> (2) SCR: for design and Bayesian or non-bayesian analysis of spatial capture-recapture
> (3) HMecology: for everything else, especially material covered in the books by Royle & Dorazio (2008), Kéry & Schaub (2012), Kéry & Royle (2016, 2021) and Schaub & Kéry (2022)
> ---

> You received this message because you are subscribed to the Google Groups "unmarked" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to unmarked+u...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/unmarked/2f799309-8a53-4591-afb1-927de202b219n%40googlegroups.com.

[Airy González]

Done! Thank you, Ken.

~Airy

To view this discussion on the web visit https://groups.google.com/d/msgid/unmarked/ZSWMURyFHH865CCV%40COYOTE.

[Christopher Dennison]

Evening Airy,

I hope this message finds you well. I am wondering if Ken was able to help fix the issue RE the random effect and predict()? I am having similar trouble with plotEffect().

Thank you so much for your time

- Chris

[Ken Kellner]

I think we solved the problem but I don't remember what it was and I can't find the email. If you can send me the data/code I'll take a look and hopefully remember to record the solution here this time.

[Christopher Dennison]

Afternoon Ken,

Much appreciated! Thank you so much for your time.

I have attached some code and the .csv files I am using to generate the unmarkedFrame, and my (attempt) at plotEffect().

Warm regards,

- Chris 🐦🌱

To view this discussion on the web visit https://groups.google.com/d/msgid/unmarked/0347dfc9-304b-4973-b465-b1c7b65447f7n%40googlegroups.com.

[Ken Kellner]

Hi Chris,

The immediate reason for the problem is the year covariate. It's numeric in the site covariates, but ultimately you are wanting to treat it as a factor. This is figured out automatically by the model fitting code, but not by the predict code that is used by plotEffects. If you manually specify that year should be a factor before creating the unmarkedFrame, e.g. with

INBU_siteCov <- INBU_siteCov %>%
  rename(site = site_name) %>%
  mutate(year = as.factor(year),
         site = as.factor(site))

this solves the error. As a more general comment this is always a good idea with unmarked and probably other modeling packages: if you want to treat something as a factor, set this explicitly yourself, and don't rely on the package to figure it out. Especially if the covariate could otherwise be treated as numeric (like year).

Unrelated to this issue, the estimates for the random effect variances are poor (one is huge and one is zero). I would not want to use the model results as-is. I think you are probably not going to be able to treat both site and year as random effects considering there are only three unique levels of year. I'd keep site as a random effect and leave out year, or treat year as a fixed effect instead. Both of those options seem to work well. Alternatively you could try to fit the model with both random effects in a Bayesian framework with JAGS or ubms, but I doubt that'll help much.

Ken

[Ken Kellner]

Also, I would recommend scaling covariates that have high absolute magnitudes (such as vl_100) to have mean 0 and sd 1. You can do this with scale(vl_100) right in the formula.

Ken

[Christopher Dennison]

This is an excellent insight! Thank you for your time and support, Ken. Much appreciated  - Chris 🐦🌱

To view this discussion on the web visit https://groups.google.com/d/msgid/unmarked/629e5478-6553-401d-b787-44794cc79579n%40googlegroups.com.