Memory leak? Or how to unload cefpython complete

[thiemo...@googlemail.com]

Hi,

is there another deconstruction than cefpython.Shutdown()?

I think this is a error in my code but cefpython.Shutdown() don't free the memory as expected. I want to use cefpython as frontend for flask and start it with a button (it can be opened again without closing the main program with other URL an/or other size). The program is pyside based but the I tried it with other frameworks too (wx, win32 same error). I can't find my error and hope you can help if it is not a issue...

the memory from cefpython.MessageLoop()will never get released.

Here is a trace of the memory usage:

######### First start (memory will not be free after cefpython.QuitMessageLoop and cefpython.Shutdown()):

Line # Mem usage Increment Line Contents
================================================
297 @profile
298 63.039 MB 0.000 MB def __init__(self):
299 63.039 MB 0.000 MB QtCore.QThread.__init__(self, parent = None)
300
301 63.039 MB 0.000 MB appSettings = {} # See: http://code.google.com/p/cefpython/wiki/AppSettings
302 63.039 MB 0.000 MB appSettings["multi_threaded_message_loop"] = False
303 69.145 MB 6.105 MB cefpython.Initialize(appSettings)
304
305 69.164 MB 0.020 MB wndproc = {
306 69.172 MB 0.008 MB win32con.WM_CLOSE: self.QuitApplication,
307 69.172 MB 0.000 MB win32con.WM_SIZE: cefpython.WindowUtils.OnSize,
308 69.172 MB 0.000 MB win32con.WM_SETFOCUS: cefpython.WindowUtils.OnSetFocus,
309 69.172 MB 0.000 MB win32con.WM_ERASEBKGND: cefpython.WindowUtils.OnEraseBackground
310 }
311
312 69.602 MB 0.430 MB windowID = cefwindow.CreateWindow("CefAdvanced", "cefadvanced", 800, 600, None, None, "icon.ico", wndproc)
313 69.602 MB 0.000 MB windowInfo = cefpython.WindowInfo()
314 69.602 MB 0.000 MB windowInfo.SetAsChild(windowID)
315
316 69.602 MB 0.000 MB browserSettings = {} # See: http://code.google.com/p/cefpython/wiki/BrowserSettings
317 69.602 MB 0.000 MB browserSettings["history_disabled"] = True
318 69.602 MB 0.000 MB browserSettings["application_cache_disabled"] = True
319 73.473 MB 3.871 MB browser = cefpython.CreateBrowserSync(windowInfo, browserSettings, "http://www.google.de"
320 91.957 MB 18.484 MB cefpython.MessageLoop()
321 94.422 MB 2.465 MB cefpython.Shutdown()
322 94.422 MB 0.000 MB self.exit()

######### Second start cefpython.Shutdown() will free memory much less than expected -> From now on cefpython.Shutdown() always free 0,3MB memory but memory usage of MessageLoop increase and will never be wiped. Every new open takes 1-2 more memory (I've tried this in a loop and have stopped it on 800MB memory usage):

Line # Mem usage Increment Line Contents
================================================
297 @profile
298 95.188 MB 0.000 MB def __init__(self):
299 95.188 MB 0.000 MB QtCore.QThread.__init__(self, parent = None)
300
301 95.188 MB 0.000 MB appSettings = {} # See: http://code.google.com/p/cefpython/wiki/AppSettings
302 95.188 MB 0.000 MB appSettings["multi_threaded_message_loop"] = False
303 95.211 MB 0.023 MB cefpython.Initialize(appSettings)
304
305 95.211 MB 0.000 MB wndproc = {
306 95.211 MB 0.000 MB win32con.WM_CLOSE: self.QuitApplication,
307 95.211 MB 0.000 MB win32con.WM_SIZE: cefpython.WindowUtils.OnSize,
308 95.211 MB 0.000 MB win32con.WM_SETFOCUS: cefpython.WindowUtils.OnSetFocus,
309 95.211 MB 0.000 MB win32con.WM_ERASEBKGND: cefpython.WindowUtils.OnEraseBackground
310 }
311
312 95.211 MB 0.000 MB windowID = cefwindow.CreateWindow("CefAdvanced", "cefadvanced", 800, 600, None, None, "icon.ico", wndproc)
313 95.211 MB 0.000 MB windowInfo = cefpython.WindowInfo()
314 95.211 MB 0.000 MB windowInfo.SetAsChild(windowID)
315
316 95.211 MB 0.000 MB browserSettings = {} # See: http://code.google.com/p/cefpython/wiki/BrowserSettings
317 95.211 MB 0.000 MB browserSettings["history_disabled"] = True
318 95.211 MB 0.000 MB browserSettings["application_cache_disabled"] = True
319 95.211 MB 0.000 MB browser = cefpython.CreateBrowserSync(windowInfo, browserSettings, "http://www.google.de"
320 96.438 MB 1.227 MB cefpython.MessageLoop()
321 96.090 MB -0.348 MB cefpython.Shutdown()
322 96.090 MB 0.000 MB self.exit()

######### Here is my complete class:

class CefAdvanced(QtCore.QThread):
def QuitApplication(parent, windowID, msg, wparam, lparam):
cefpython.QuitMessageLoop()
win32gui.DestroyWindow(windowID)
browser = cefpython.GetBrowserByWindowHandle(windowID)
browser.CloseBrowser()
className = cefwindow.GetWindowClassName(windowID)
win32gui.UnregisterClass(className, None)

@profile
def __init__(self):
QtCore.QThread.__init__(self, parent = None)

appSettings = {} # See: http://code.google.com/p/cefpython/wiki/AppSettings
appSettings["multi_threaded_message_loop"] = False
cefpython.Initialize(appSettings)

wndproc = {
win32con.WM_CLOSE: self.QuitApplication,
win32con.WM_SIZE: cefpython.WindowUtils.OnSize,
win32con.WM_SETFOCUS: cefpython.WindowUtils.OnSetFocus,
win32con.WM_ERASEBKGND: cefpython.WindowUtils.OnEraseBackground
}

windowID = cefwindow.CreateWindow("CefAdvanced", "cefadvanced", 800, 600, None, None, "icon.ico", wndproc)
windowInfo = cefpython.WindowInfo()
windowInfo.SetAsChild(windowID)

browserSettings = {} # See: http://code.google.com/p/cefpython/wiki/BrowserSettings
browserSettings["history_disabled"] = True
browserSettings["application_cache_disabled"] = True
browser = cefpython.CreateBrowserSync(windowInfo, browserSettings, "http://www.google.de"
cefpython.MessageLoop()
cefpython.Shutdown()
self.exit()

[Czarek Tomczak]

Hi Thiemo,

I'm seeing that you call QuitMessageLoop(), you probably shouldn't be doing this,

when the main browser window is destroyed a message loop should automatically

break. Calling CloseBrowser() is probably also redundant, as you already destroy

the window.

CEF is multithreaded, even after cefpython.Shutdown() returns, CEF might still be freeing

some resources, how about giving it some more time, a second or two before checking

the memory?

See this topic "Please Fix Memory Leaks" [1] on CEF C++ forum:

The V8 JavaScript engine uses a garbage collector optimized for performance. 
As a result, memory allocated by V8 is not immediately freed. You should see
memory usage reach a stable level over extended periods of time. Issue 86
describes a good technique for measuring memory usage in CEF.
To facilitate fast shutdown WebKit does not free all memory on exit but instead
leaves that task to the operating system. See issue 15 for additional information.

I've posted a question to Marshall in this topic regarding freeing all the memory.

See also Issue 86 [2] in the CEF C++ Issue Tracker for measuring CEF memory

usage.

Cheers,

Czarek

[1] http://magpcss.org/ceforum/viewtopic.php?f=7&t=188

[2] https://code.google.com/p/chromiumembedded/issues/detail?id=86

[Czarek Tomczak]

See also Issue 15 "Browser leak?" [1] in the CEF C++ Issue Tracker that was 

mentioned in the quote.

Cheers,

Czarek

[1] https://code.google.com/p/chromiumembedded/issues/detail?id=15

[thiemo...@googlemail.com]

Thank you for the fast answer. If I disable QuitMessageLoop()it seems to me that the function is never ending (memory profiler don't returns -> Function is still running...) I've waited a minute on each run. I disabled Shutdown()too and this saves memory on each run (but not enough - Still leaking memory)

I've waited some second on each new open but the problem still remains. I only open the google.com startpage - If I do something (e.g. open links, youtube...) the memory usage is much higher and will also never get released. On some sites the memory leak is 30 MB and more.

I hope there will be a option to clean cef complete from memory after close the window because It's a great addition to show e.g. Help pages or flask/tornado apps :-)

I've added a complete working example if you want to try. I use memory_profiler (https://pypi.python.org/pypi/memory_profiler) to check the memory stats:

#!/usr/bin/python
# -*- coding: utf-8 -*-

import sys
from PySide.QtCore import *
from PySide.QtGui import *

import win32con # pywin32 extension
import win32gui

import cefpython1.cefpython_py27 as cefpython
import cefpython1.cefwindow as cefwindow

from memory_profiler import profile

class Form(QDialog):

def __init__(self, parent=None):
super(Form, self).__init__(parent)
self.button = QPushButton("ShowCEF")
layout = QVBoxLayout()
layout.addWidget(self.button)
self.setLayout(layout)
self.button.clicked.connect(self.CefSimple)

@profile
def CefSimple(self):

appSettings = {} # See: http://code.google.com/p/cefpython/wiki/AppSettings
appSettings["multi_threaded_message_loop"] = False
cefpython.Initialize(appSettings)

wndproc = {
win32con.WM_CLOSE: self.QuitApplication,
win32con.WM_SIZE: cefpython.WindowUtils.OnSize,
win32con.WM_SETFOCUS: cefpython.WindowUtils.OnSetFocus,
win32con.WM_ERASEBKGND: cefpython.WindowUtils.OnEraseBackground
}

windowID = cefwindow.CreateWindow("CefAdvanced", "cefadvanced", 800, 600, None, None, "icon.ico", wndproc)
windowInfo = cefpython.WindowInfo()
windowInfo.SetAsChild(windowID)

browserSettings = {} # See: http://code.google.com/p/cefpython/wiki/BrowserSettings
browserSettings["history_disabled"] = True
browserSettings["application_cache_disabled"] = True

browser = cefpython.CreateBrowserSync(windowInfo, browserSettings, "http://www.google.de")
cefpython.MessageLoop()
#cefpython.Shutdown()
#self.close()

def QuitApplication(parent, windowID, msg, wparam, lparam):
cefpython.QuitMessageLoop()
win32gui.DestroyWindow(windowID)

#browser = cefpython.GetBrowserByWindowHandle(windowID)
#browser.CloseBrowser()

className = cefwindow.GetWindowClassName(windowID)
win32gui.UnregisterClass(className, None)

if __name__ == '__main__':
app = QApplication(sys.argv)
form = Form()
form.show()
sys.exit(app.exec_())

[Czarek Tomczak]

Have you tried opening and closing multiple browser windows and see

how memory behaves?

The cefadvanced.py example doesn't call QuitMessageLoop() and it works

as expected.

Cheers,

Czarek

[thiemo...@googlemail.com]

I did not try multiple windows (I destroy the window before each instance -> it should not be possible to open the window multiple times for my usage)

I've tried this with cefadvanced.py file but have the same problem

I start the file -> 20 MB memory usage

I click on "window.open('cefadvanced.html')" and open "Project's website: http://code.google.com/p/cefpython/" -> 39,3 MB memory usage

I close the popup -> Wait a minute -> Still 37MB memory usage

If I click around on some pages (in the popup) like youtube the memory usage raise up to 150MB and will not be cleaned after I exit the popup.

Thank you :-)

Thiemo

[Czarek Tomczak]

Chromium and webkit have some sophisticated methods of freeing the

memory, you can't see the whole picture with some simple test. Take

this for example, when you open the first 10 browser popup windows then

the memory goes up with each of them, but after 10 or 20 of them opened

and closed you will see a decrease in memory. So this is not a memory

leak, but some way of optimization of not to free the memory instantly.

Remember that in CEF 1 the v8 javascript engine is shared across all 

browser windows and when you close one of them there is no guarantee

that the memory will be freed at this specific moment, because it may be

reused for the next browser window, so freeing it now and assigning again

later may be wasteful.

Cheers,

Czarek

[thiemo...@googlemail.com]

Thats OK if the whole program is used as Browser (e.g. Chrome itself) but for a embedded Version it waste a lot of memory if it can/will not completely wipe the used memory on close.

In my test cases the memory will never get released if I close a popup or the main window - All JS, Images etc. that has been loaded anytime before in a popup will stay in memory. For e.g. long running processes that usually never get closed (deamon, services with CEF as Interface) this is a huge problem. I've a memory usage (without restarting the program) with more than 800MB after 2 Days (open Help pages, links to Github project, tutorials on youtube).

Do you see any option to unload the whole CEF at runtime and release the memory or force CEF to release all used Memory if the window has been destroyed ?

[Czarek Tomczak]

How much memory do you have on that system that used 800MB after 2 days?

Have you also tried testing CEF 3 and see how it behaves in regards to memory

consumption, is it any better?

Do you see any option to unload the whole CEF at runtime and release the memory 
or force CEF to release all used Memory if the window has been destroyed ?

Please go ask about it on the CEF C++ Support Forum [1] and let's see what Marshall

has to say, mention your 800 MB case. Also ask if there is any API to the "purge

memory button" that can be found in Google Chrome Task Manager (via the 

--purge-memory-button flag).

Cheers,

Czarek

[1] http://www.magpcss.org/ceforum/viewforum.php?f=6

[Czarek Tomczak]

You could also ask about the --memory-model=(high|medium|low) flag

if it would be possible to make it available through some settings. It

wouldn't hurt also asking about the "about:memory" page if there are

any plans for it.

Czarek

[thiemo...@googlemail.com]

Thynks for your reply :-)

I've tried this with CEF3 -> Here it works good (little overhead is here too, but no leak -> Don't got it over 90MB usage)

I've uploaded the sample working sample here:

http://pastebin.com/tpCePiaY

One little error: If I set line 125 to cefpython.Shutdown() (proper way - If I use e.g. a video it will not stop if I don't shutdown) cefpython is crashing with the follwing error:

[0501/170111:FATAL:content_main_runner.cc(717)] Check failed: InitializeSandbox(sandbox_info).

Any idea why or what I'm doing wrong ?

[thiemo...@googlemail.com]

Crash is on next opening -> Not instant

[Czarek Tomczak]

I'm not sure what you mean, you cannot call cefpython.Initialize() or

cefpython.Shutdown() more than once in program lifetime.

See line 159 of pyside.py example [1], read the comments that explain

what you're probably experiencing:

    # Need to destroy QApplication(), otherwise Shutdown() fails.
    # Unset main window also just to be safe.
    del mainWindow
    del app
    cefpython.Shutdown()

Do not call sys.exit() ("sys.exit(app.exec_())") otherwise cefpython.Shutdown() 

won't complete, it might be cleaning some system resources even after the

function has returned.

Btw. Flash support in CEF 1 is not perfect, I wouldn't be surprised if

the high memory usage was caused by Flash.

Cheers,

Czarek

[1] https://code.google.com/p/cefpython/source/browse/cefpython/cef1/windows/binaries/pyside.py#159

[thiemo...@googlemail.com]

Thank you very much for your answers :-)

I've now a great working solution with multiprocessing. I simply fork a new process within my main program and close exit the process on close. Multiple Instances are no more problem and any overhead is gone on exit :-)

If you could find some time you should try to completely unload the libcef.dll on shutdown (timer is still running) if possible.

Thank you again for his great lib :-)