Removing big objects from sim list from inside a function

12 views
Skip to first unread message

Tati Micheletti

unread,
Jul 5, 2018, 4:30:06 PM7/5/18
to SpaDES Users
Dear SpaDES users,

I wanted to know if there is a way to remove objects (i.e. a several Gb one) that were assigned to the sim list, passed to another function, and dealt with at some point in that function. I tried finding the objects using pryr::where("x"), and it returns the current environment I am in (the function environment). I can easily just remove the object from there, but where is the original one? I have the feeling I am only removing a pointer to the object. I say this because when I used:  gc()
rm(x)
gc()

I have almost the same results from gc() the second time I had the first time (just a few bytes less). So my question is: how should I proceed in this case? Can I access somehow the sim list while inside a function and erase or turn into NA an object from the list? I couldn't even find the environment where my "sim" is stored.

Thanks a lot in advance! :)

Tati

ceresv...@hotmail.com

unread,
Jul 5, 2018, 8:01:55 PM7/5/18
to SpaDES Users
Reproducible example?

I mean, I don't see why you can't simply do

sim$obj <- NULL

Tati Micheletti

unread,
Jul 5, 2018, 8:08:25 PM7/5/18
to SpaDES Users
I was trying to do it inside a nested function, for which I am NOT passing sim (as it is contains huge objects). I gave up and decided to call the objects inside the function without assigning them to sim. It will be faster and save memory afterwards. Thanks for answering, though! :)

Eliot McIntire

unread,
Jul 5, 2018, 10:27:32 PM7/5/18
to SpaDES Users
One option is to pass the environment of the simList into the function, not the simList object:

myFun <- function(largeObj, simEnv, objectNameInSimList) {
    print(object.size(largeObj)) # it is big, and you can use it here
    rm(list = objectNameInSimList, envir = simEnv) 
    # don't have to rm the largeObj because it will disappear at the end of this function
    return("I have deleted a large object in the simList environment")
}

sim <- simInit()
sim$small <- rnorm(1)
sim$myLargeObject <- rnorm(1e7)
objSize(sim) # shows large object
myFun(largeObj = sim$myLargeObject, simEnv = envir(sim), objectNameInSimList = "myLargeObject")
objSize(sim) # shows large object is gone
gc() # should cause a change
gc() # should cause a change again
gc() # should cause a change again

Eliot McIntire

unread,
Jul 5, 2018, 11:28:36 PM7/5/18
to SpaDES Users
But really, just rm it after it is needed:

myFun <- function(largeObj) {
    print(object.size(largeObj)) # it is big, and you can use it here
    # don't have to rm the largeObj because it will disappear at the end of this function
    return("I have deleted a large object in the simList environment")
}

sim <- simInit()
sim$small <- rnorm(1)
sim$myLargeObject <- rnorm(1e8)
objSize(sim) # shows large object
myFun(largeObj = sim$myLargeObject)
rm(list = "myLargeObject", envir = envir(sim))

objSize(sim) # shows large object is gone
gc() # should cause a change
gc() # should cause a change again
gc() # should cause a change again

#######################################
if you need to cache it, you could try using omitArg the one:
myFun <- function(largeObjName, mySim) {
    print(object.size(mySim[[largeObjName]])) # it is big, and you can use it here
    # don't have to rm the largeObj because it will disappear at the end of this function
    mySim[[largeObjName]] <- NULL
    # you can or don't need to return the simList, as the line above
    return("I have deleted a large object in the simList environment")
}

sim <- simInit()
sim$small <- rnorm(1)
sim$myLargeObject <- rnorm(1e8)
objSize(sim) # shows large object
# slow because it has to run digest of simList
sim$myLargeObject <- rnorm(1e8)
system.time(Cache(myFun, largeObjName = "myLargeObject", mySim = sim))
#800000048 bytes
#   user  system elapsed 
#   0.24    0.09    0.54 

# Faster with omitArgs -- this difference will get larger as the object gets larger
system.time(Cache(myFun, largeObjName = "myLargeObject", mySim = sim, omitArgs = "mySim"))
#800000048 bytes
#   user  system elapsed 
#   0.04    0.08    0.19 

Reply all
Reply to author
Forward
0 new messages