Issue with transactions...

[animus...@gmail.com]

I have noticed that during a process that the memory usage is up to 13 to 14 GB.

I tried using gc.collect() which did nothing. So I tried a "transaction.abort()" and most of the memory was returned after about 2 to 3 minutes.

This process does not change anything within the persistent objects so why is it adding those persistent objects to a transaction? I have added a "transaction.abort()" to the process and the memory stays around 370 MB.

[Jason Madden]

It's not adding them to a transaction.

ZODB Connections cache persistent objects. This is for performance, and in the case of modifications, to ensure that all the objects accessed come from one consistent view of the database as-of the time the transaction began. Until the transaction ends, and the view of the database is updated, persistent objects that the connection accesses remain in the connection's cache.

At transaction boundaries, the Connection automatically performs cache garbage collection, removing unmodified persistent objects from the cache as necessary to keep the cache within configured limits.

By calling `transaction.abort()`, you are creating a transaction boundary: not only are you freeing objects from the cache, you are also updating the view of the database to incorporate whatever recent changes may have been committed, which may be inconsistent with the previous view of the database you were working with.

It's good practice to explicitly delineate transaction boundaries in your application: units of work that need a consistent view of the database. In a server or web application, transaction boundaries usually correspond to client requests. At the beginning of processing the request, call `transaction.begin()` to update the view of the database, and at the end, call `transaction.commit()` (or `abort()`) to free (cached) resources.

If you do nat have natural transaction boundaries in your application, you can manually ask the Connection to remove un-modified objects from its cache at appropriate intervals using `Connection.cacheGC()`[1] (or `cacheMinimize()`, but that negates the point of having a cache entirely). Unlike arbitrarily committing or aborting a transaction during processing, this won't update the view of the database, meaning the objects stay consistent with previously seen objects. Note, however, that on certain storage backends such as RelStorage, long-running transactions can become expensive if there are other transactions writing to the database.

[1] https://github.com/zopefoundation/ZODB/blob/22df1fd1141ab0c9db2a163799b87fe3585f15ef/src/ZODB/interfaces.py#L170

~Jason

[animus...@gmail.com]

 Thanks for the reply. I guess I will need to use the cache methods to control memory usage then.