Out of memory when creating lots of Mapper objects
deadfolk
I'm back to a memory problem I posted about a little while back, but I
thought I had put down to garbage collection problems with some
pointers from Jeppe. I'm pretty sure now that it's not a GC issue, as
I have profiled the app in VisualVM.
Basically, I'm iterating over a file and using Mapper to save a row to
the database for each line. The problem is that the mapper objects I
am creating in the loop don't seem to be getting garbage collected,
and I don't know why, unless Mapper is retaining references to them
somehow. Is there something I have to do explicitly dereference
mapper objects? As they are created in the foreach block, I don't see
how I could be accidentally retaining them otherwise.
I would very much appreciate any help, as I am struggling with this
one...as it's for a live project, if I can't get to the bottom of it
soon, I'm going to have to strip out the Mapper code and do it the old-
fashioned way.
Paraphrased code below. There may be typos in this, but you get the
idea. The 'row' is the object which I seem to be accumulating many
instances of, according to VisualVM.
Thanks for any help,
Matt
parser.foreach{line =>
val row = Row.create
row.generatedTime(generatedTime)
//set other fields...
//Look up some other data...
OtherMappedObject.findAllDb(CoreDB,
By(OtherMappedObject.id,
row.otherObjId)).firstOption.foreach{obj =>
row.objName(obj.name)
}
//A nice calculated field...
val x = BigDecimal(row.amount.is) * row.rate.is
row.total(x)
row.save
}
David Pollak
Hi All,
I'm back to a memory problem I posted about a little while back, but I
thought I had put down to garbage collection problems with some
pointers from Jeppe. I'm pretty sure now that it's not a GC issue, as
I have profiled the app in VisualVM.
Basically, I'm iterating over a file and using Mapper to save a row to
the database for each line. The problem is that the mapper objects I
am creating in the loop don't seem to be getting garbage collected,
and I don't know why, unless Mapper is retaining references to them
somehow. Is there something I have to do explicitly dereference
mapper objects? As they are created in the foreach block, I don't see
how I could be accidentally retaining them otherwise.
I would very much appreciate any help, as I am struggling with this
one...as it's for a live project, if I can't get to the bottom of it
soon, I'm going to have to strip out the Mapper code and do it the old-
fashioned way.
Paraphrased code below. There may be typos in this, but you get the
idea. The 'row' is the object which I seem to be accumulating many
instances of, according to VisualVM.
Thanks for any help,
Matt
parser.foreach{line =>
val row = Row.create
row.generatedTime(generatedTime)
//set other fields...
//Look up some other data...
OtherMappedObject.findAllDb(CoreDB,
By(OtherMappedObject.id,
row.otherObjId)).firstOption.foreach{obj =>
row.objName(obj.name)
}
//A nice calculated field...
val x = BigDecimal(row.amount.is) * row.rate.is
row.total(x)
row.save
}
--
You received this message because you are subscribed to the Google Groups "Lift" group.
To post to this group, send email to lif...@googlegroups.com.
To unsubscribe from this group, send email to liftweb+u...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/liftweb?hl=en.
--
Lift, the simply functional web framework http://liftweb.net
Beginning Scala http://www.apress.com/book/view/1430219890
Follow me: http://twitter.com/dpp
Surf the harmonics
deadfolk
Thanks for taking the time to help with this - and I apologise in
advance for this wordy response.
Please disregard the findAll() call in the sample - I'm new to lift
and hadn't noticed the find() method. In this case the result is
unique, so this isn't related to the issue.
More investigation has pointed me to my use of DB.buildLoanWrapper()
in Boot.Scala, as follows:
S.addAround(DB.buildLoanWrapper(List(CoreDB)))
Seems a pretty textbook example of using this method as far as I can
tell. However, if I remove this line, the problem goes away -
although I obviously lose my transaction capabilities.
Looking further in VisualVM, I see that I can trace all the object
instances to DB$ConnectionHolder. The path to it looks like this:
Row
--\FieldOwner
..\(etc.)
--\inst$3 (type MetaMapper$$anonfun$setupInstanceForPostCommit$1)
----\hd type($colon$colon)
------\tl type($colon$colon)
--------\tl
(these nested tl references continue for hundreds of levels)
----------\postCommit type(DB$ConnectionHolder)
...and from there right up to the web server classes. The list of
nested 'tl' references is longer on older instances of Row, so I
suspect that one level is being created in each iteration of my loop.
To test this on a blank slate, I created a new project from the basic
archetype and added a service that creates and saves objects in a
simple loop like this:
object TestService {
def test(): Node= {
for(i <- 1 to 1000000000) {
val row = Row.create
row.save
}
<Done></Done>
}
}
The Row class used here is a simple LongKeyedMapper with just a
primary key and no other fields.
This exhibits the same behaviour - fine when there's no LoanWrapper,
but keeps references to all created objects when it's there.
Am I doing something really stupid, or is it just that
DB.buildLoanWrapper() is not supposed to be used in this way i.e. when
creating many objects before exiting the request?
If this is the case, is there a way I can force the loan wrapper to
commit to the DB and flush its references every x amount of
iterations?
I hope this all makes sense - thanks again for reading.
Matt
On May 26, 6:40 pm, David Pollak <feeder.of.the.be...@gmail.com>
wrote:
> > .
> >http://groups.google.com/group/liftweb?hl=en.
>
> --
> Lift, the simply functional web frameworkhttp://liftweb.net
David Pollak
David
To unsubscribe from this group, send email to liftweb+u...@googlegroups.com.
--
deadfolk
On May 30, 6:08 pm, David Pollak <feeder.of.the.be...@gmail.com>
wrote: