Caching and memory usage
Andrew Bateman
Transfer.
I am currently running a script which is inserting data from an
existing database into Transfer objects, then using Transfer to insert
the data into a new database. I am running into some issues regarding
memory usage.
Originally I had the Transfer cache set to the default settings [i.e.
no entries in my config file for caching]. After creating around
100-150 objects, I was experiencing java.lang.OutOfMemoryError errors.
So, I thought I would set the default cache as follows:
<defaultcache>
<maxobjects value="10"/>
<maxminutespersisted value="2"/>
<scope type="instance"/>
</defaultcache>
Same story: 100-150 objects in, my server packs up with a
java.lang.OutOfMemoryError.
OK, so to test a theory, I set my defaultcache to use the request
scope. This time, after about 100-150 objects, my ColdFusion server
gradually reached its maximum memory allocation, but continued to
bubble away without throwing an OutOfMemoryError. The same thing
occurs when I set the cache scope to "none".
My understanding is that Transfer works with the JVM to ensure the
cache does not use too much memory. If so, can you think of any
reason why an OutOfMemoryError could occur? The growth in memory
allocation to Jrun is quite linear, and it does not appear that any
individual object or line of my code is the culprit.
FYI I am running ColdFusion 7,0,2,142559 standard edition, on Windows
2003. Java version is 1.4.2_09 . The maximum heap size is 512MB, and I
am using the following JVM arguments:
-server -Dsun.io.useCanonCaches=false -XX:MaxPermSize=128m -
Djava.compiler=NONE -Xnoagent -Xdebug -
Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=8000 -
Dcoldfusion.rootDir={application.home}/../ -
Dcoldfusion.libPath={application.home}/../lib
Following is a trace of one of my OutOfMemoryErrors......
22/04 16:01:53 error Event Handler Exception.
[1]java.lang.OutOfMemoryError
[0]coldfusion.runtime.EventHandlerException: Event Handler Exception.
at coldfusion.runtime.AppEventInvoker.onRequest(AppEventInvoker.java:
186)
at coldfusion.filter.ApplicationFilter.invoke(ApplicationFilter.java:
215)
at
coldfusion.filter.RequestMonitorFilter.invoke(RequestMonitorFilter.java:
51)
at coldfusion.filter.PathFilter.invoke(PathFilter.java:86)
at coldfusion.filter.ExceptionFilter.invoke(ExceptionFilter.java:69)
at
coldfusion.filter.BrowserDebugFilter.invoke(BrowserDebugFilter.java:
52)
at
coldfusion.filter.ClientScopePersistenceFilter.invoke(ClientScopePersistenceFilter.java:
28)
at coldfusion.filter.BrowserFilter.invoke(BrowserFilter.java:38)
at coldfusion.filter.GlobalsFilter.invoke(GlobalsFilter.java:38)
at coldfusion.filter.DatasourceFilter.invoke(DatasourceFilter.java:
22)
at
coldfusion.filter.RequestThrottleFilter.invoke(RequestThrottleFilter.java:
115)
at coldfusion.CfmServlet.service(CfmServlet.java:107)
at
coldfusion.bootstrap.BootstrapServlet.service(BootstrapServlet.java:
78)
at jrun.servlet.ServletInvoker.invoke(ServletInvoker.java:91)
at jrun.servlet.JRunInvokerChain.invokeNext(JRunInvokerChain.java:42)
at
jrun.servlet.JRunRequestDispatcher.invoke(JRunRequestDispatcher.java:
257)
at
jrun.servlet.ServletEngineService.dispatch(ServletEngineService.java:
541)
at
jrun.servlet.jrpp.JRunProxyService.invokeRunnable(JRunProxyService.java:
204)
at jrunx.scheduler.ThreadPool
$DownstreamMetrics.invokeRunnable(ThreadPool.java:318)
at jrunx.scheduler.ThreadPool
$ThreadThrottle.invokeRunnable(ThreadPool.java:426)
at jrunx.scheduler.ThreadPool
$UpstreamMetrics.invokeRunnable(ThreadPool.java:264)
at jrunx.scheduler.WorkerThread.run(WorkerThread.java:66)
I would appreciate any suggestions you may have.
Regards, Andrew
PS. how can I dump or otherwise inspect the cache? The documentation
says that I should be able to find the cache in "[scope].transfer",
but this does not seem to be the case for me!
Mark Mandel
>
> Hi Mark, got an issue for you regarding caching and memory usage in
> Transfer.
>
> I am currently running a script which is inserting data from an
> existing database into Transfer objects, then using Transfer to insert
> the data into a new database. I am running into some issues regarding
> memory usage.
Honestly, I'm not sure if Transfer is the right tool for migrating
data from one database to another.
That being said, I've run applications that have have well over a
thousand objects in cache with no major issues.
>
> Originally I had the Transfer cache set to the default settings [i.e.
> no entries in my config file for caching]. After creating around
> 100-150 objects, I was experiencing java.lang.OutOfMemoryError errors.
> So, I thought I would set the default cache as follows:
>
> <defaultcache>
> <maxobjects value="10"/>
> <maxminutespersisted value="2"/>
> <scope type="instance"/>
> </defaultcache>
>
> Same story: 100-150 objects in, my server packs up with a
> java.lang.OutOfMemoryError.
Unfortunately you can't force the JVM to garbage collect, but that
being said, if it is running out of memory, it should gc
automatically.
>
> OK, so to test a theory, I set my defaultcache to use the request
> scope. This time, after about 100-150 objects, my ColdFusion server
> gradually reached its maximum memory allocation, but continued to
> bubble away without throwing an OutOfMemoryError. The same thing
> occurs when I set the cache scope to "none".
I'm not sure why you would get the difference... the only thing I can
suggest is that in your code, you need to make sure as you loop
around, you make sure there is no reference to the utilised object
after you have used it.
>
> My understanding is that Transfer works with the JVM to ensure the
> cache does not use too much memory. If so, can you think of any
> reason why an OutOfMemoryError could occur? The growth in memory
> allocation to Jrun is quite linear, and it does not appear that any
> individual object or line of my code is the culprit.
The only thing is - if an object is active in the current thread, it
cannot be GC'd.. so you need to make sure there is no reference to any
non active objects.
>
>
> PS. how can I dump or otherwise inspect the cache? The documentation
> says that I should be able to find the cache in "[scope].transfer",
> but this does not seem to be the case for me!
Currently this is planned for a post 1.0 implementation.
Let me know how it goes.
Mark
Andrew Bateman
Mark Mandel
advice without seeing some of the migration code.
Mark
On 4/22/07, Andrew Bateman <andrew....@newgency.com> wrote:
>
> v.0.6.3, though it was also happening before when I was using the previous version.
>
>
>
> >
>
Andrew Bateman
Andrew Bateman
My name is Andrew, and I am an idiot. The problem had nothing at all to do with Transfer and everything to do with me being a monkey.
I was caching my import query, which was over 10,000 rows, which was a good idea. Unfortunately, due to a slip of the fingers, I had set the cache timeout to be #CreateTimespan(0,0,1,0)#. Yes folks, every minute!
Turns out this was causing my OutOfMemoryErrors, as for some reason ColdFusion was not releasing the memory allocated to the cached queries. Setting this to something more sensible has fixed the problem and everything is humming away nicely.
Mark [and everyone on the list], sorry for wasting your time.
Andrew [slinking away quietly to eat a banana and throw faeces at onlookers]
Mark Mandel
I was totally scratching my head, b/c I have unit tests that create
more than 150 objects, so I was totally confused.
Good it was a simple fix!
Regards,
Mark
On 4/22/07, Andrew Bateman <andrew....@newgency.com> wrote:
>
Dan Wilson
"Andrew [slinking away quietly to eat a banana and throw faeces at onlookers]"
dw
--
"Come to the edge, he said. They said: We are afraid. Come to the edge, he said. They came. He pushed them and they flew."
Guillaume Apollinaire quotes
Jaime Metcher
But surely, not withstanding the actual species of the programmer, this would have to qualify as a bug or at least a misfeature in CF itself? Query caching is supposed to be a nice but optional performance boost, not blanket permission to eat the server's memory. Unless you're doing something like rerunning the query every time through a loop, in which case one would have to think there's a better way to do this
Jaime Metcher.