memory problems with mysql
Tobias Rosenbaum
I'm using matlab 2009b on a windows 7 64bit PC. I have a rather large mysql database from which i query data.
My problem is that, after a certain time, matlab starts to throw error messages around and slows down to a point where I have to close it using the task manager.
This is what I do:
query number of datasets
fetch number of datasets
for i = 1:size(datasets)
query dataset(i)
fetch query
process dataset
clear query dataset
end
I would expect this to clear any memory space that the last query used, but for some reason, the java error appears after a certain amount of queries (i can do those in a random order, it doesn't matter). Obviously, data is stacking up in a place I can't reach and clear. Sometimes it's the query command crashing my script, sometimes the fetch command.
I would be very glad If anyone has an idea on how to solve this mystery.
Error message:
??? Java exception occurred:
java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Unknown Source)
at java.util.Vector.ensureCapacityHelper(Unknown Source)
at java.util.Vector.addElement(Unknown Source)
at com.mathworks.toolbox.database.fetchTheData.dataFetch(fetchTheData.java:632)
Error in ==> cursor.fetch at 110
dataFetched = dataFetch(fet,resultSetMetaData,p.NullStringRead,tmpNullNumberRead);
Error in ==> analysis at 66
tmp = fetch(myQuery);
Or (other error message)
??? Java exception occurred:
java.lang.OutOfMemoryError: Java heap space
at com.mysql.jdbc.Buffer.getBytes(Buffer.java:198)
at com.mysql.jdbc.Buffer.readLenByteArray(Buffer.java:318)
at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1375)
at com.mysql.jdbc.MysqlIO.readSingleRowSet(MysqlIO.java:2369)
at com.mysql.jdbc.MysqlIO.getResultSet(MysqlIO.java:451)
at com.mysql.jdbc.MysqlIO.readResultsForQueryOrUpdate(MysqlIO.java:2076)
at com.mysql.jdbc.MysqlIO.readAllResults(MysqlIO.java:1451)
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:1787)
at com.mysql.jdbc.Connection.execSQL(Connection.java:3277)
at com.mysql.jdbc.Connection.execSQL(Connection.java:3206)
at com.mysql.jdbc.Statement.executeQuery(Statement.java:1232)
at com.mathworks.toolbox.database.sqlExec.executeTheSelectStatement(sqlExec.java:170)
Error in ==> cursor.cursor at 150
resultSetVector = executeTheSelectStatement(curs.Cursor,curs.Statement);
Error in ==> database.exec at 34
curs=cursor(connect,sqlQuery);
Error in ==> analysis at 62
myQuery = exec(conn,['select * from spike where FK_Series = ' '''' num2str(series.data{h,1}) '''
AND FK_Unit =' '''' num2str(units.data{k,1}) ''' ']);
Exception in thread "Timer-1" java.lang.NullPointerException
at com.mathworks.mlwidgets.explorer.control.DirectoryListing$5.receive(DirectoryListing.java:242)
at com.mathworks.mlwidgets.explorer.control.DirectoryListing$5.receive(DirectoryListing.java:240)
at com.mathworks.util.NativeJava.listFiles(Native Method)
at com.mathworks.mlwidgets.explorer.control.DirectoryListing.loadAndSendDirectory(DirectoryListing.java:238)
at com.mathworks.mlwidgets.explorer.control.DirectoryListing.list(DirectoryListing.java:171)
at com.mathworks.mlwidgets.explorer.control.DirectoryListing.getChildren(DirectoryListing.java:115)
at com.mathworks.mlwidgets.explorer.control.DirectoryListing.getChildren(DirectoryListing.java:99)
at com.mathworks.mlwidgets.explorer.control.DirectoryDocumentListing.refresh(DirectoryDocumentListing.java:244)
at com.mathworks.mlwidgets.explorer.control.RefreshDaemon$3.run(RefreshDaemon.java:198)
at java.util.TimerThread.mainLoop(Unknown Source)
at java.util.TimerThread.run(Unknown Source)
[ConditionalEventPump] Exception occurred during event dispatching:
java.lang.OutOfMemoryError: Java heap space
at javax.swing.RepaintManager.updateWindows(Unknown Source)
at javax.swing.RepaintManager.paintDirtyRegions(Unknown Source)
at javax.swing.RepaintManager.paintDirtyRegions(Unknown Source)
at javax.swing.RepaintManager.seqPaintDirtyRegions(Unknown Source)
at javax.swing.SystemEventQueueUtilities$ComponentWorkRequest.run(Unknown Source)
at java.awt.event.InvocationEvent.dispatch(Unknown Source)
at java.awt.EventQueue.dispatchEvent(Unknown Source)
at java.awt.EventDispatchThread.pumpOneEventForFilters(Unknown Source)
at java.awt.EventDispatchThread.pumpEventsForFilter(Unknown Source)
at java.awt.EventDispatchThread.pumpEventsForHierarchy(Unknown Source)
at java.awt.EventDispatchThread.pumpEvents(Unknown Source)
at java.awt.EventDispatchThread.pumpEvents(Unknown Source)
at java.awt.EventDispatchThread.run(Unknown Source)
Tristram Scott
> hello everybody,
>
> I'm using matlab 2009b on a windows 7 64bit PC. I have a rather large
mysql database from which i query data.
>
> My problem is that, after a certain time, matlab starts to throw error
messages around and slows down to a point where I have to close it using
the task manager.
>
> This is what I do:
>
> query number of datasets
> fetch number of datasets
> for i = 1:size(datasets)
> query dataset(i)
> fetch query
> process dataset
> clear query dataset
> end
>
Do you maintain and use just the one database connection, or do you open a
new connection for each query in the loop? We have seen java heap errors
across several versions of the database toolbox. At least in the latest
version, the solution seems to have been to ensure that every time we open
a connection we must remember to close it explicitly. It is not sufficient
to clear (or overwrite) the variable the connection is associated with.
There should be some bugs on file for this. I suggest that you ontact TMW
technical support for some more detailed advice.
--
Dr Tristram J. Scott
Energy Consultant