Caching Large Objects

[dario....@gmail.com]

Hello,

I am trying to compile a document, and it has an error when the code chunk finishes and it is time to write out the cache.

The error message has

long vectors not supported yet: connections.c:5269

But I am using version 3, which supports long vectors.

R version 3.0.1 (2013-05-16)
Platform: x86_64-w64-mingw32/x64 (64-bit)

locale:
[1] LC_COLLATE=English_Australia.1252 LC_CTYPE=English_Australia.1252
[3] LC_MONETARY=English_Australia.1252 LC_NUMERIC=C
[5] LC_TIME=English_Australia.1252

attached base packages:
[1] stats graphics grDevices utils datasets methods base

other attached packages:
[1] knitr_1.2

loaded via a namespace (and not attached):
[1] digest_0.6.3 evaluate_0.4.4 formatR_0.8 stringr_0.6.2
[5] tools_3.0.1

Is there a limit to the size of objects which can be cached ? There is no statement in the documentation about it.

--------------------------------------
Dario Strbenac
PhD Student
University of Sydney
Camperdown NSW 2050
Australia

[Yihui Xie]

This is the first time I have heard of this. Can you provide a minimal
reproducible example? e.g. you can replace your real data with random
numbers. Thanks!

Regards,
Yihui
--
Yihui Xie <xiey...@gmail.com>
Phone: 206-667-4385 Web: http://yihui.name
Fred Hutchinson Cancer Research Center, Seattle

> --
> You received this message because you are subscribed to the Google Groups
> "knitr" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to knitr+un...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

[dario....@gmail.com]

Hello,

I was able to reproduce this in a small example. Here is the code from minimal.Rnw

\documentclass{article}

\begin{document}

<<makeMatrix, cache=TRUE, cache.vars="bigMatrix">>=

bigMatrix <- matrix(runif(1000000000), ncol = 1000)
print(object.size(bigMatrix), units = "Gb")
sessionInfo()
@

\end{document}

Here is the entire error message

Quitting from lines 5-9 (minimal.Rnw)
Error in lazyLoadDBinsertVariable(vars[i], from, datafile, ascii, compress,  :

  long vectors not supported yet: connections.c:5269

Calls: knit ... <Anonymous> -> <Anonymous> -> lazyLoadDBinsertVariable

Execution halted

[Yihui Xie]

Thanks, but I do not have enough memory on my laptop (the object you
created requires about 8G memory). I will test this when I have access
to our servers. Before that, we can still figure out the root of this
problem. You can try this naive cache method via save() and load():


\documentclass{article}

\begin{document}

<<makeMatrix, cache=TRUE>>=

bigMatrix <- matrix(runif(1000000000), ncol = 1000)
print(object.size(bigMatrix), units = "Gb")

save(bigMatrix, file = 'bigMatrix.RData')
rm(bigMatrix)
@
<<loadMatrix, cache=FALSE>>=
load('bigMatrix.RData')
@

\end{document}


If this works, the problem came from lazyLoad(). Otherwise it is just
because R cannot load large objects under 64-bit Windows in certain
cases, as described in R 3.0.0 NEWS:
http://cran.r-project.org/src/base/NEWS.html

Regards,
Yihui
--
Yihui Xie <xiey...@gmail.com>
Phone: 206-667-4385 Web: http://yihui.name
Fred Hutchinson Cancer Research Center, Seattle

[dario....@gmail.com]

The method of using the functions save and load runs without error.

[Yihui Xie]

Thanks! Notes taken: https://github.com/yihui/knitr/issues/572 I'll
look into it in the future. For now, you can probably just go with the
save()/load() approach.

Regards,
Yihui
--
Yihui Xie <xiey...@gmail.com>
Phone: 206-667-4385 Web: http://yihui.name
Fred Hutchinson Cancer Research Center, Seattle

[Yihui Xie]

This should be fixed in https://github.com/yihui/knitr/issues/572 now.

Regards,
Yihui
--
Yihui Xie <xiey...@gmail.com>

Web: http://yihui.name
Department of Statistics, Iowa State University
2215 Snedecor Hall, Ames, IA