Strategy to use ddo localDiskConn and have a cut and paste, tar.gz, vdb?

[jeremiah rounds]

Hi all,

This touches on some parts of Trelliscope I haven't used so I thought I would ask for a public brainstorm.

1) Some experimentation has revealed to me that vdb's made with multiple calls to makeDisplay (for distinct panel functions) that use kvMemory backed DDOs copy the data in those DDO for each panelFn (eg if a kvMemory DDO is 100 mb than you will be storing 100mb per panelFn).

2) A way around that is to use a DDO backed by localDiskConn, but datadr ddo does not tolerate "relative paths", which implies to me that the vdb that uses a localDiskConn DDO as source data cannot be cut and paste to a collaborator via email (or what not).  You will break the link to the DDO to which the vdb refers by moving it to a new machine even if you stored it within the vdb folder.

Essentially at my current understanding of Trelliscope (which may be flawed) the choice seems to be "use inefficient storage strategy and have the ability to cut and paste/tar.gz/sharedrive vdbs" or "use efficient storage strategy but have broken localDiskConn paths if you move the vdb folder".  Is that correct?

I was wondering do any of those lesser used Trelliscope functions fix this? Because there was one that seemed like it might but it isn't in the tutorial I don't think.

[Ryan Hafen]

You are correct that we try to keep VDB directories as self-contained as possible.  With kvMemory, we have to always write because otherwise we can’t capture the state of the data for future viewing of the display.

However, a less-well-documented feature of local disk DDOs is the syncLocalData function, which is called when you do a webSync, but which you can call any time you would like.  This takes any local disk DDOs that are referenced in any of your displays and copies the data to a data directory in your VDB, making your VDB self-contained and ready to share with someone or push to a shiny server of shinyapps.io.

Ryan

--
You received this message because you are subscribed to the Google Groups "Tessera-Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tessera-user...@googlegroups.com.
To post to this group, send email to tesser...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/tessera-users/45826be2-b3cb-4260-a386-15f648870883%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[jeremiah rounds]

Thats what I thought syncLocalData did but I didn't want to steer myself into this "fix" of using localDiskConn with syncLocalData unless I was sure.  Thanks Ryan.

[jeremiah rounds]

Basically I am asking do you think datadr would run perfectly fine without line 35 (or a flag to control its execution) https://github.com/tesseradata/datadr/blob/5334ea10c86136de2fea7ff19a69841388345fc6/R/conn_localDisk.R#L35 ?

The reason is certain things about Trelliscope appear to be easier to manage in unusual cases if it doesn't execute.

[jeremiah rounds]

Hi Ryan,

I thought up some ideas for "relative" locaDiskConn and I went to test the non-relative version (that is standard Trelliscope use-cases), and I made a piece of code that works and I don't understand why... maybe you can shed some light on it?

It is this:

data(iris)

tmp1 = tempfile(tmpdir="/tmp")

tmp2 = tempfile(tmpdir="/tmp")

conn1 = vdbConn(tmp1, autoYes = TRUE)

ldisk_path = file.path(conn1$path, "data", "local_ddo")

dir.create(ldisk_path, recursive = TRUE)

ldisk = localDiskConn(ldisk_path,autoYes=TRUE)

kvmem = divide(iris, by="Species")

ldmem = convert(kvmem, ldisk)

panelFn = function(v) plot(v$Petal.Width, v$Petal.Length)

makeDisplay(ldmem, panelFn=panelFn, name="test", conn=conn1)

trelliscope::copy_dir(tmp1, tmp2)

unlink(tmp1, recursive=TRUE)

rm("ldmem")

rm("kvmem")

list.files(ldisk_path)

conn2 = vdbConn(tmp2, autoYes=TRUE)  

list.files(conn2$path, recursive = TRUE)

view(conn=conn2)  #<--- how is this working?

At this point I expected view(conn=conn2) to fail to find the ldmem since it is just a copy of conn1, but it doesn't it makes normal looking plots and I am not sure why?   The spirit of the question is there anything in Trelliscope Viewer code that is fixing this up? I need to understand that before I can move on to my idea for relative path DDOs.

I inspected the disolayObj.Rdata and it does this:

> e = new.env()

> load(file.path(conn2$path, "displays", "common", "test", "displayObj.Rdata"), envir = e)

> str(e$displayObj)

List of 21

 $ name           : chr "test"

 $ group          : chr "common"

 $ desc           : chr ""

 $ mdDesc         : NULL

 $ preRender      : logi FALSE

 $ panelFn        :function (v)  

  ..- attr(*, "srcref")=Class 'srcref'  atomic [1:8] 1 11 1 57 11 57 1 1

  .. .. ..- attr(*, "srcfile")=Classes 'srcfilecopy', 'srcfile' <environment: 0x1f9fb36a0> 

  ..- attr(*, "class")= chr [1:2] "function" "rplotFn"

 $ panelFnType    : chr "rplotFn"

 $ panelDataSource:

Error in readChar(con, 5L, useBytes = TRUE) : cannot open the connection

In addition: Warning message:

In readChar(con, 5L, useBytes = TRUE) :

  cannot open compressed file '/private/tmp/file4f2b78db6436/data/local_ddo/4a7a45a288b320d14537ba28cfdb8db5.Rdata', probable reason 'No such file or directory'

Which is exactly what I expect!  The question is why isn't that happening on view(conn=conn2)?

[jeremiah rounds]

Its line 46 here isn't it:

https://github.com/tesseradata/trelliscope/blob/9c0dd358e504e3877ee089fc2ea65fac39596239/R/displayObj.R#L45

Well that helps a lot!  I put a ddo localDiskConn into ./data and zipped it up and sent it to a VM and it ran fine because of that line and it solves the whole issue.

--

You received this message because you are subscribed to the Google Groups "Tessera-Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tessera-user...@googlegroups.com.
To post to this group, send email to tesser...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/tessera-users/90307cc0-09df-4cf1-8dd2-a34d084c8881%40googlegroups.com.

[Ryan Hafen]

Hi Jeremiah,

Just wanted to confirm - are you satisfied with the way it is, or would you still like to discuss the relative paths?

Ryan