Off-screen in cef3 linux

[jeg...@gmail.com]

Hi,

I started to try to use the implemented off-screen rendering of cef3 for linux. (Since it is available now)
Currently i am working with r1297 of cef.

But to be honest I am quite lost. My knowledge with cython and cef is still small.
I started by making the function SetAsOffscreen() available in linux. After editing cef_linux.h and cef_types_linux.h (by adding functions and variables) I got it to the point on which it compiles without any errors.

But as I thought, it can't be that simple.
When I try to "cefpython.CreateBrowserSync()" cef throws an error: ERROR_REPORT:browser_host_impl.cc(286)] Check failed: false. CefRenderHandler implementation is required

Can someone guess how much work it needs that this can be done? (I just don't have a clue)
What have to be done?

Thanks,
Jegger

[Czarek Tomczak]

Hi Jegger,

The CEF 3 OSR on Linux is still in the trunk, so it's probably not yet stable. Also when building

cefpython using CEF trunk you might encounter some troubles, headers might have to be updated

along with some other APIs. The trunk uses a more recent version of Chromium.

CefRenderHandler [1] needs to bo implemented for OSR. It's implemented in CEF Python 1 in

the render_handler.pyx file [2], see also paint_buffer.pyx [3] and cpp_utils/PaintBuffer.h [4].

The example of CEF 1 OSR is in the panda3d_.py example [5]. See also Issue 36 "Add off-screen

rendering support + Panda3D example" [6] that shows how we've managed to implement OSR

in CEF Python 1.

There is an issue [7] in the CEF Python Issue Tracker regarding OSR support in CEF 3.

Regards,

Czarek

[1] https://code.google.com/p/cefpython/source/browse/cefpython/cef3/include/cef_render_handler.h

[2] https://code.google.com/p/cefpython/source/browse/cefpython/render_handler.pyx

[3] https://code.google.com/p/cefpython/source/browse/cefpython/paint_buffer.pyx

[4] https://code.google.com/p/cefpython/source/browse/cefpython/cpp_utils/PaintBuffer.h

[5] https://code.google.com/p/cefpython/source/browse/cefpython/cef1/windows/binaries/panda3d_.py

[6] https://code.google.com/p/cefpython/issues/detail?id=36

[7] https://code.google.com/p/cefpython/issues/detail?id=69

[Czarek Tomczak]

See also client_handler:

https://code.google.com/p/cefpython/source/browse/cefpython/cef1/client_handler/client_handler.h#250

-Czarek

[jeg...@gmail.com]

Thanks for your answers.

My current state is: https://github.com/jegger/cefpython/commit/677bed47c5767e660839a7e08456586b10735940
(or should I use google code? => never used it)

I wrote a short python script (mostly copied from the panda3d example): http://pastebin.com/FpgG9dFH (which uses the kivy framework) to test the changes.
output: http://pastebin.com/y762i3jy

I guess the log looks not bad. But now I don't have a clue how to get to the texture from cef.
Should the function OnPaint() in the ClientHandler class be called? (It doesn't)
Is the displayed ERROR_REPORT from cef important?

Thanks,
Jegger

[Czarek Tomczak]

Jegger,

In the commit I see that you've updated the CEF revision to 1297, but the binaries you're using seem

to be using a different revision. The error you get:

[0714/135755:ERROR_REPORT:render_widget_host_view_osr.cc(271)] Check failed: false. 

Says that the check failed on line 271 in file render_widget_host_view_osr.cc. But in CEF trunk revision

1297 this check seems to be on line 267:

https://code.google.com/p/chromiumembedded/source/browse/trunk/cef3/libcef/browser/render_widget_host_view_osr.cc?r=1297#267

This error in the console occurs as it seems you're not providing the view rect, RenderHandler_GetViewRect()

is missing. But I don't think this is necessary to make it work. But there is something wrong with the binaries

you're using, they are not revision 1297.

Have you set the browser size? See the panda3d example:

https://code.google.com/p/cefpython/source/browse/cefpython/cef1/windows/binaries/panda3d_.py#164

Regards,

Czarek

[Czarek Tomczak]

Regarding the wrong revision, I just saw that the Linux OSR was merged to branch 1453

in revision 1295, so the the error check seems to be on the correct line:

https://code.google.com/p/chromiumembedded/source/browse/branches/1453/cef3/libcef/browser/render_widget_host_view_osr.cc?r=1297#271

You need to set browser size to make it work, OnPaint() should then be called. Otherwise

analyse the panda3d example again whether you're not missing some part.

-Czarek

[jeg...@gmail.com]

Can this even be done with the current cef?

As I see in the browser.pyx, the function SetSize() tries to map the function SetSize in: https://code.google.com/p/chromiumembedded/source/browse/branches/1453/cef3/include/cef_browser.h

But there isn't a function like GetSize in this file. In cef1 it seems to be implemented.
Is there a workaround?

[Czarek Tomczak]

Alternately you can provide the view rectangle in CefRenderHandler::GetViewRect() - fill the CefRect and return true.

That should be enough, otherwise try GetRootScreenRect() / GetScreenInfo(), but according to the documentation

providing the view rect is enough.

Regards,

Czarek

[jeg...@gmail.com]

Sorry to bother you again,

Actually I don't understand what you mean. Maybe the real issue is that I don't really understand how cef handles osr.

I assume of the following:
1. init cef
2. set offscreen rendering flag
3. create browser
4. define the size of the texture (for cef) => via SetSize() or GetViewRect()?
5. receive the texture in the function OnPaint of RenderHandler?
Is this correct? I was not able to find any documentation about that yet.

If I assume the way I described above, then GetViewRect(browser, (0,0,width,height)) should be called from the .py ? But GetViewRect seems something like a callback for me which get called from cef?

Thanks,
Jegger

[Czarek Tomczak]

Before step3 "create browser" you need to implement RenderHandler, while browser is being created

the RenderHandler::GetViewRect() callback will be called to get the size of the browser, then OnPaint()

will be called. RenderHandler methods are all callbacks, they are being called by CEF. As CEF 3 doesn't 

have the SetSize() method, it seems that the only way to provide the size of the browser is through 

RenderHandler callback.

The steps are:

1. Implement RenderHandler::GetViewRect() and OnPaint()

2. Initialize CEF

3. Set off-screen rendering flag for the windowInfo structure

4. Create browser

5. GetViewRect() callback gets called - you have to provide the size of the browser

6. OnPaint() callback gets called

Regards,

Czarek

[jeg...@gmail.com]

I have a hard time calling SetClientHandler() in browser.pyx while the browser is being created.

From the .py file I can't do that cause I don't have the browser object yet. I thought of several approaches but either they failed or are just ugly. I wish it would be somehow possible to pass the handler to the function CreateBrowserSync() directly.

What I don't understand: in cefpython.pyx - CreateBrowserSync() there get cef_browser_static.CreateBrowserSync() called. After that somehow GetPyBrowser() in browser.pyx get called. But from where?

What would you think is an appropriate solution?

Regards,
Jegger

[Czarek Tomczak]

On Tuesday, July 16, 2013 10:11:05 AM UTC+2, jeg...@gmail.com wrote:

I have a hard time calling SetClientHandler() in browser.pyx while the browser is being created.

From the .py file I can't do that cause I don't have the browser object yet. I thought of several approaches but either they failed or are just ugly. I wish it would be somehow possible to pass the handler to the function CreateBrowserSync() directly.

This seems like a problem that the GetViewRect() is being called during browser creation, but the current

implementation in cefpython allows to set the client handler only after the browser is created synchronously.

I don't see an easy fix for that, but I have an idea, try setting the initial navigation when creating browser 

to "about:blank" and navigate to the real url only after the client handler has been set up:

self.browser = cefpython.CreateBrowserSync(

windowInfo, browserSettings,

navigateUrl="about:blank")

self.browser.SetClientHandler(

ClientHandler(self.browser, self.texture))

self.browser.GetMainFrame().LoadUrl(GetApplicationPath("cefsimple.html")))

And let's hope that GetViewRect() is called again.

...

I just remembered that there is the WasResized() function:

///

// Notify the browser that the widget has been resized. The browser will first

// call CefRenderHandler::GetViewRect to get the new size and then call

// CefRenderHandler::OnPaint asynchronously with the updated regions. This

// method is only used when window rendering is disabled.

///

/*--cef()--*/

virtual void WasResized() =0;

After setting client handler call browser.WasResized() and it should work.

Regards,

Czarek 

 

[jeg...@gmail.com]

GetViewRect() doesn't get called after LoadUrl(). But after WasResized()! Even if it's in the doc of WasResized() written, OnPaint() does not get called afterwards. Not even the one in render_handler.pyx.

My current cycle is (line 42 - 70): https://github.com/jegger/cefpython/blob/cef3_linux_osr/cefpython/cef3/linux/binaries_64bit/kivypy.py

Thanks,
Jegger

[jeg...@gmail.com]

But what I saw: GetViewRect() get called two times. (If that may matter somehow)
See log here: http://pastebin.com/au5aqLnt

[Czarek Tomczak]

Try setting the rect in client_handler.cpp, replace that:

return RenderHandler_GetViewRect(browser, rect);

with:

rect.x = 0
rect.y = 0
rect.width = 800
rect.height = 600
return True 

This will tell us whether the first call to GetViewRect() when browser is being created is significant.

Try also providing the required data in GetScreenRect() and GetScreenPoint().

Put a printf("OnPaint()\n"); in client_handler.cpp.

Check for error messages in the console.

Try to pass 0 when calling SetAsOffscreen().

What kind of handle are you passing to the SetAsOffscreen() method? Is this a pointer to the physical

window or to the gtk widget? Try both, though I think you have to pass pointer to the gtk widget.

-CzarekTry 

[jeg...@gmail.com]

On Tuesday, July 16, 2013 3:44:46 PM UTC+2, Czarek Tomczak wrote:
> Try setting the rect in client_handler.cpp, replace that:
>
> return RenderHandler_GetViewRect(browser, rect);
>
>
> with:
>
> rect.x = 0
> rect.y = 0
> rect.width = 800
> rect.height = 600
> return True 
>
>
> This will tell us whether the first call to GetViewRect() when browser is being created is significant.

I did that. It seems it doesn't have any impact.

>
>
> Try also providing the required data in GetScreenRect() and GetScreenPoint().
>

I didn't do that yet. I just added printf()'s on this functions in client_handler.cpp to see if they are getting called. => they never getting called.

>
> Put a printf("OnPaint()\n"); in client_handler.cpp.
>

Also done. Seems OnPaint don't get called either.

>
> Check for error messages in the console.
>
>
> Try to pass 0 when calling SetAsOffscreen().

This raises an error. => http://pastebin.com/e8FzFmGa

>
>
> What kind of handle are you passing to the SetAsOffscreen() method? Is this a pointer to the physical
> window or to the gtk widget? Try both, though I think you have to pass pointer to the gtk widget.
>

This number I wrote was just a dummy number. I thought it doesn't have any impact?
Anyhow, I indicate the windowID now and pass it to SetAsOffscreen(). I use the windowID because in the panda3d example the function getIntHandle() get called which returns the windowID.
(This doesn't rise any error but onPaint() doesn't get called)

As I see just now the datatype seems different on linux. => include/internal/cef_typeds_linux.h => GtkWidget* (as you expected).
I have to admit that I currently have no idea how to get to the pointer of the gtk widget. PyGame is the window provider which kivy is using.

Jegger

[Czarek Tomczak]

On Tuesday, July 16, 2013 4:52:48 PM UTC+2, jeg...@gmail.com wrote:

> Try to pass 0 when calling SetAsOffscreen().
This raises an error. => http://pastebin.com/e8FzFmGa

Try commenting out this code in window_info.pyx so that value of 0 is allowed to be passed to

the function:

if not windowInfo.parentWindowHandle:
        raise Exception("WindowInfo: parentWindowHandle is not set")

... 

> What kind of handle are you passing to the SetAsOffscreen() method? Is this a pointer to the physical
> window or to the gtk widget? Try both, though I think you have to pass pointer to the gtk widget.
>
This number I wrote was just a dummy number. I thought it doesn't have any impact?

Passing a random address of an expected pointer is not a good idea.

Anyhow, I indicate the windowID now and pass it to SetAsOffscreen(). I use the windowID because in the panda3d example the function getIntHandle() get called which returns the windowID.
(This doesn't rise any error but onPaint() doesn't get called)

As I see just now the datatype seems different on linux. => include/internal/cef_typeds_linux.h => GtkWidget* (as you expected).
I have to admit that I currently have no idea how to get to the pointer of the gtk widget. PyGame is the window provider which kivy is using. 

I've only used PyGTK and WxPython on Linux. In wxPython the controls have two methods: GetHandle() to return

handle to the physical window and GetGtkWidget() that returns a pointer to a gtk widget.

-Czarek

[jeg...@gmail.com]

With a value of 0 to SetAsOffscreen() -> no success. (after commenting the if statement)

---

I just tried something: https://gist.github.com/jegger/6009801
There I mixed the wxpython example with os-rendering. OnPaint get called!!

It seems that the gtkwidget is requred. I think my issue in this case is that with PyGame I don't have any gtkwidget at all.

Is OSR still possible without gtk on linux? Is gtk absolutely necessary?

[Czarek Tomczak]

On Tuesday, July 16, 2013 5:40:59 PM UTC+2, jeg...@gmail.com wrote:

Is OSR still possible without gtk on linux? Is gtk absolutely necessary?

Try asking this question on the CEF C++ Forum:

http://www.magpcss.org/ceforum/viewforum.php?f=6

-Czarek

[Czarek Tomczak]

What is your use case? Do your application have any window? You could create a hidden

window and just pass its handle to CEF.

-Czarek

[jeg...@gmail.com]

http://www.magpcss.org/ceforum/viewtopic.php?f=6&t=10851

Passing NULL should work.
Does SetAsOffscreen(0) result to NULL? - if not; I've overwritten ParentWidget and set it to NULL in cef_linux.h to be sure. => But it's still not working.

I think I try to write a script with an own main loop (without another framework - in this case kivy).
Does cefpython need some special things when running in a other loop? Something that has to be called every frame etc...?

[Czarek Tomczak]

On Tuesday, July 16, 2013 8:24:05 PM UTC+2, jeg...@gmail.com wrote:

I think I try to write a script with an own main loop (without another framework - in this case kivy).
Does cefpython need some special things when running in a other loop? Something that has to be called every frame etc...?

Of course! :-) You've missed it in the panda3d example:

def messageLoop(self, task):
    cefpython.MessageLoopWork()
    return Task.cont

Call it in the OnIdle event or similar. -Czarek

[Czarek Tomczak]

You have two options, either run the CEF loop or integrate into existing application loop, see the

cefpython wiki page:

https://code.google.com/p/cefpython/wiki/cefpython

----------------------------------------------------------------------------------------------

void MessageLoop()

Run the CEF message loop. Use this function instead of an application- provided message loop to get the best balance between performance and CPU usage. This function should only be called on the main application thread (UI thread) and only if cefpython.Initialize() is called with a ApplicationSettings.multi_threaded_message_loop value of false. This function will block until a quit message is received by the system.

void MessageLoopWork()

Perform a single iteration of CEF message loop processing. This function is used to integrate the CEF message loop into an existing application message loop. Care must be taken to balance performance against excessive CPU usage. This function should only be called on the main application thread (UI thread) and only if cefpython.Initialize() is called with aApplicationSettings.multi_threaded_message_loop value of false. This function will not block.

Alternatively you could create a periodic timer (with 10 ms interval) that calls cefpython.MessageLoopWork().

--------------------------------------------------------------------------------------------------

-Czarek

[jeg...@gmail.com]

Haha, okay that's a bit embarrassing ;) - this was easy!

Now OnPaint() get called. But... if I do: buffer.GetString() =>
the return value is very strange: it starts with: 511TV-OUT 32MB?Ɂ?d HGladiac 511TV-OUT 64MB@V??e...
As a result of this, the rendered image is just nonsense: http://jegger.ch/datapool/images/cefpy.png (left bottom corner)
That isn't normal?

Thanks again for your help,
Jegger

[Czarek Tomczak]

On Wednesday, July 17, 2013 5:07:51 PM UTC+2, jeg...@gmail.com wrote:

Now OnPaint() get called. But... if I do: buffer.GetString() =>
the return value is very strange: it starts with: 511TV-OUT 32MB?Ɂ?d HGladiac 511TV-OUT 64MB@V??e...
As a result of this, the rendered image is just nonsense: http://jegger.ch/datapool/images/cefpy.png (left bottom corner)

You've commented out important parts in paint_buffer.pyx:

https://github.com/jegger/cefpython/blob/cef3_linux_osr/cefpython/paint_buffer.pyx

Restore it to the original state and then see how to convert PaintBuffer (buffer variable) to an image,

from the panda3d example:

def _saveImage(self):

try:

from PIL import Image

except:

print("PIL library not available, can't save image")

return

(width, height) = self.browser.GetSize(cefpython.PET_VIEW)

buffer = self.browser.GetImage(cefpython.PET_VIEW, width, height)

image = Image.fromstring(

"RGBA", (width,height),

buffer.GetString(mode="rgba", origin="top-left"),

"raw", "RGBA", 0, 1)

image.save("panda3d_image.png", "PNG")

Regards,

Czarek

[jeg...@gmail.com]

You'r right that seems to matter. (paint_buffer.pyx)
But If I just use: buffer.GetString(mode="bgra", origin="top-left") both functions stay untouched. Top-left shouldn't be an issue anyway because I can flip the texture in kivy afterwards.

While trying to implement _saveImgae() I saw that include/cef_browser.h only in cef1 has the function GetImage() which seems needed for this.

Long story short: It is working now!
I want to thank you for your support. Even it looks like you did all the work ;) - I at least learned a lot.
---

In RenderHandler_OnPaint() I need the size of the browser. This is in cef1 done by GetSize() -> But this isn't implemented in cef3 as we know. I call GetViewRect() to get the size. Is this a good idea?
https://github.com/jegger/cefpython/blob/9022b264b3c209c126c93f321825b6e66592d88a/cefpython/render_handler.pyx

Thanks,
Jegger

[Czarek Tomczak]

On Thursday, July 18, 2013 11:48:09 AM UTC+2, jeg...@gmail.com wrote:

While trying to implement _saveImgae() I saw that include/cef_browser.h only in cef1 has the function GetImage() which seems needed for this.

The buffer returned by GetImage() is the same that you get in the OnPaint() callback.

 

In RenderHandler_OnPaint() I need the size of the browser. This is in cef1 done by GetSize() -> But this isn't implemented in cef3 as we know. I call GetViewRect() to get the size. Is this a good idea?
https://github.com/jegger/cefpython/blob/9022b264b3c209c126c93f321825b6e66592d88a/cefpython/render_handler.pyx

In CEF 3 there are new parameters (width and height) in the OnPaint() method:

https://code.google.com/p/chromiumembedded/source/browse/branches/1453/cef3/include/cef_render_handler.h?r=1306#122

You can ask on the CEF Forum to bring back GetSize/SetSize methods to the CefBrowser object,

but seems like this is not really necessary to make it work.

Regards,

Czarek

[Czarek Tomczak]

Hi Jegger,

It would be nice to have a PyGame or Kivy OSR example (or both) in CEF Python 3, would you be interested

in creating a demo example(s)? Or you could just provide me with what you already have and I could base on that.

I see the kivypy.py script in the repository:

https://github.com/jegger/cefpython/blob/9e349b2452e6d3c34e263507f47917b5897d94b1/cefpython/cef3/linux/binaries_64bit/kivypy.py

What does this example do?

You also mentioned PyGame but I haven't found any scripts running PyGame in your repository.

Best regards,

Czarek

[jeg...@gmail.com]

Hi Czarek,

Regarding PyGame: Kivy uses PyGame as their window provider. But I don't have any PyGame experience.

--

I just pushed this minute a newer version with working mouse/touch support. As kivy is mainly built for touch, this means it has touch support. The keyboard implementation is weak. Writing is possible but other keys like backspace or tab aren't working. What the example does (it is a WIP):
- Launches a website in the size of the window.
- Forwards on_touch_down, on_touch_move, on_touch_up (is working with the mouse to)
- Forwarding key inputs

What I need to do next is: Stable keyboard input, Get a callback when a textinput field is activ (to show the onscreen keyboard in kivy) -> Didn't do any research on this. (I hope this is somehow possible), And do some kind of go_back, go_forward UI.

I will work on that the week after next. (Next, I enjoy my holiday :) ). I am sure I can provide in that week a stable kivy example, as I need that anyway.

Best regards,
Jegger

[jeg...@gmail.com]

I had a hard time merging your changes with my current branch. Can't say why - it just run into some wired errors. - Anyway: I just think it would be super cool if you can merge the osr code into your master. Do I have to do anything for that? - I only used pull requests to collaborate (github) before. I have no idea of google code.

The kivy example isn't finished yet:
- GoBack() and GoForward() should already be implemented? (If I call them, they don't do anything)
- Keyboard input: CefKeyEvent =>How do I send key like enter, or backspace etc? (the doc of cef(CefKeyEvent) isn't helping me there)
- To get a callback when a textinput field is active: I think thats the same issue like #57 on the cefpython issue tracker?

It's possible that I get myself the answers to the questions above. But if you have some useful tips i'd be glad.

Regards,
Jegger

[Czarek Tomczak]

Hi Jegger,

On Saturday, August 3, 2013 1:35:46 PM UTC+2, jeg...@gmail.com wrote:

I had a hard time merging your changes with my current branch. Can't say why - it just run into some wired errors. - Anyway: I just think it would be super cool if you can merge the osr code into your master. Do I have to do anything for that? - I only used pull requests to collaborate (github) before. I have no idea of google code.

After you have a complete example create a patch from your changes, submit it to Issue 69.

 

The kivy example isn't finished yet:
- GoBack() and GoForward() should already be implemented? (If I call them, they don't do anything)

Yes, they are implemented and should work. Otherwise ask about this problem on the CEF C++ Forum.

- Keyboard input: CefKeyEvent =>How do I send key like enter, or backspace etc? (the doc of cef(CefKeyEvent) isn't helping me there)

See the functions starting with Send* in CefBrowser: SendKeyEvent, SendMouseClickEvent and others.

https://code.google.com/p/chromiumembedded/source/browse/trunk/cef3/include/cef_browser.h#410

In the panda3d example you will find uses of the Send* functions. Though, they might differ a bit in CEF 3.

 

- To get a callback when a textinput field is active: I think thats the same issue like #57 on the cefpython issue tracker?

If it is Touch events that you need, then yes, issue 57 needs to be resolved.

Best regards,

Czarek