datadr example attribute

[jeremiah rounds]

Hi,

Is there a way to set the example attribute to something specific that is not currently one of your key/value pairs?

Why?  Because I have a data division by year that I am more or less happy with for the parsimoniousness of a time series analysis, but they are kind of large.   I would rather not have a 1 gig example key/value pair load when ddf() or ddf() is called and the get attributes line  "res <- setAttributes(res, attrs)" runs.   

 I am finding local storage to be a wonderful way to work with this data btw.  I think local storage was an idea before its time because coupled with a large solid state drive some of the delay in constant disk access is sharply mitigated.

[jeremiah rounds]

Hmm I dug around a bit more and hacked together what I think is the solution?

inconn <- localDiskConn(file.path("ddf"))

byDDF= ddf(inconn, update=TRUE) 

attr = getAttributes(byDDF, "example")

attr$ddo$example$value = attr$ddo$example$value[1:1000,]

setAttributes(byDDF, attr$ddo)

Not sure if that will stick though.

[hafen]

This will work.  The new attribute will be saved as meta data and will be available next time you load the ddf, since at the end of setAttributes, saveAttrs.localDiskConn is called.

I suppose it would be nice to have the option to not need to populate the "example" attribute when updating attributes.

[hafen]

BTW thanks for the feedback on local disk.  An ideal situation for local disk would be an array of solid state drives, although I've never had such a thing to test on.  I think there are probably a lot of optimizations that could be done to the local disk storage and computation.  For example, currently each key-value pair is a file, but it would be nice to use something like berkeleydb or leveldb or some serverless key value store.

On Thursday, August 13, 2015 at 10:58:28 AM UTC-6, jeremiah rounds wrote: