Webkit/Blink

[Erik Boettcher]

What will happen to Berkelium through the Wekbit/Blink separation?  Which way will Berklium go?

iOS comes to mind with regards to this switch as Blink won't be ported to iOS.

[Ewen Cheslack-Postava]

Berkelium is far behind current Chromium, it hasn't been updated to
use anything past Chromium12. So the fork doesn't affect us at all
unless something happens with the repositories that makes getting the
old source code a problem. Until someone finds the time and
inclination to update to a newer version, this isn't an issue.

-Ewen

> --
> You received this message because you are subscribed to the Google Groups
> "Berkelium" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to berkelium+...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

[Dennis Rieks]

Hi,

im working on a new berkelium version (https://github.com/berkelium/berkelium) supporting the latest chrome version. i think it will not be possible to use Webkit when chromium uses blink. blink will be too much integrated into chromium and berkelium uses chromium.

by the way: if someone is intrested in contributing to the new version feel free to contact me :-)

Dennis

2013/4/11 Ewen Cheslack-Postava <ewe...@cs.stanford.edu>

[Matt Clark]

I've been wanting to update berkelium for a while, but wasn't really sure where to start... so if you would like help, I would love to give it^^

[Roland Eckl]

Hi Dennis,

an up-to-date berkelium would be great!

What is the status of your project? On Github you just say that it is not usable yet. ;-) Does it at least build? Where could you need support most?

And the repository's structure looks quite different from the one known from sirakata/berkelium. How much did you actually take over?

Hopefully your plans for Java, C, CPP and .NET support are not too ambitious... better start with CPP in the begin. Once a stable base is reached, add the others.

Cheers,

   Roland

[Dennis Rieks]

Hi,

currently only cpp is implemented. The API for all languages is auto generated. The Other languages should only be wrapper to the berkelium-cpp implementation.

There is an berkelium library project with name "berkelium-cpp", inside this there is the library itself, some demo and a unit test. another executable is "host-fake", this is an simple main() function that sends for every received ipc command the expected response.

the unit test is called with this mock and a second time is is called with the real "berkelium-host" executable.

currently the unit test (with host fake) works well on linux.

the next steps are:

 * implementing IPC for windows and mac

 * updating berkelium-host so that all unit tests works with it

 * removing the boost dependencies

the new version is a complete rewritten version, the chromium sources we used for the old berkelium version are so outdated that i don't see a change to reuse the complete old code.

the old berkelium was linked against the user executable. because of this it is not possible the use berkelium in an windows 64 executable (because there is no win64 chromium version)

this issue is fixed in the new version by using IPC between the user executable (linked with the berkelium library, 32 and 64 bit are supported) and the berkelium host (this is always 32 bit on windows).

chromium internally also uses IPC so this should be no problem.

the only thing we must take care of is that the berkelium IPC must use fixed size arguments (int on 32 and 64 bit systems is not the same, e.g. int32_t is used for this)

At the moment unfortunately I have only little time but i want to work on berkelium again in the next days.

If you develop on Windows or Mac it would be nice if you can try to implement the IPC Classes so that the unit test works. For Mac is can be an solution to rename the *Linux* file to Posix and adapt only some places.

The Windows Implementation needs more work i think, there was an windows version some time ago but i think this version will not compile.

Another thing (also for Linux) is to replace all boost filesystem calls with an one "Berkelium/Impl/Filesystem.hpp" API.

The next steps I plan are creating some tickets in an bug tracker system with instructions on how to implement the next needed features.

Please let me know if you need help digging into to code.

Cheers,

Dennis

2013/4/22 Roland Eckl <eckl....@googlemail.com>

[Guillaume Le Biller]

Hello,

First, it's a good things to start everything over and have an up to date Chromium version with berkelium.

I started to dig into your new code for berkelium in order to start and help the Windows implementation of it.

The Filesystem implementation is quite easy and is working for me, however the IPC implementation is a little bit complex to understand.

I managed to make things working using windows named pipes, but I do not understand your Channels.

Why are you creating 4 pipes with 4 differents names ?

What exactly are the meaning of server, read & reverse ?
The windows named pipes are synchronous methods by default and your current unit test do not fit this architecture.

Moreover, using the asynchronous version would need also more changes on the Pipe & Channel interfaces.

If you are interested I could create a merge request for the Filesystem implementation for the moment.

And if I manage to understand more your architecture, I can work to adapt it to the Windows implementation.

Best Regards,

Guillaume

[Dennis Rieks]

Hi Guillaume,

very nice to hear about your work! :-)

Every Merge request is very welcome. If you are interested in supporting the windows version i can grand you write access to the main repository.

I'm currently reworking the IPC code. The old code was only written to "let it work". 

The Idea behind the class Pipe is to represent a unix pipe. Process A can write and read the pipe, also a second process B. The problem is that a Pipe is generally a unidirectional first in first out channel, not a bidirectional connection.

This means:

A: write(x)

B: read = x

(this is ok)

But also:

A: write(x)

A: read = x

B: read -> blocking until A writes something

Because of this I use to Pipes for every Channel, one Pipe for sending from A to B and another Pipe for sending from B to A.

2013/10/15 Guillaume Le Biller <guillaume...@gmail.com>

--

Dennis Rieks

agido GmbH
Hörder Hafenstraße 3a
44263 Dortmund

Fon: +49-231-399802-70
Fax: +49-231-399802-69
Skype: dennisrieks

Amtsgericht Dortmund: HRB 20179
USt-IdNr.: DE225365236
Geschäftsführer: Thomas Louis

[Dennis Rieks]

(accidentally sent... continue here)

As you can see here: https://github.com/berkelium/berkelium/blob/master/berkelium-cpp/src/lib/IPC/Channel.cpp#L34-35 this two Pipes are pin and pout.

The Server Parameter is used to connect Process A pin (named xxxxxx1 in the Filesystem) to Process B's pout. The second Pipe (named xxxxxx2) is connected to A's pout and B's pin. The creator of the channel is (normaly, expect for unit tests) the server (using createChannel) and the other Process is the client (using getChannel).

The same applies to the reverse channel, named xxxxxx3 and xxxxxx4.

The "normal" or "default" channel is used to send Messages created e.g. by the Library (maybe tab.navigateTo()), the reverse Channel is used for Messages for the same channel (in this case: the tab) but created by berkelium host process. This can be onReady or onPaint. 

A IPC Channel is a communication link between the library and the host process, "default" channel is currently used by Instance, Window and Tab, the reverse Channel is mapped to WindowDelegate and TabDelegate.

I hope you understand me, please ask if not.

Because Windows Named Pipes (I also planned to use them) can read and write (I would have previously read the instructions..) at the same time it makes no sense to Map the Pipe Class to both windows and unix pipes at the same time.

I have rewritten the IPC Code during the last week (the unit tests don't work currently, but i commited everything that you can see the changes)

One Problem is that every Pipe needs a file handle and there are limited to 1024 by default for Linux Users.

I created a new ChannelGroup (using the four pipes like Channel before). But all Channels of this Channel Group shares the same Pipes, because of this there are only four file handels (+two for stdout and stderr) needed for every host instance started by the library. (The old implementation needed [count(window + tab) * 4 + 4] Pipes, this will make it impossible to open much windows or tabs)

I think we should rename Pipe to PosixUniPipe [do you know a better name?] (for Linux and MacOS) and create a new Pipe. The new Pipe can be implemented using WindowsNamedPipe for windows and a new PosixPipe (using two PosixUniPipes)

The ChannelGroup must then be refactored to use the new Pipe, but this should be more or less easy.

Currently both berkelium library and host are single threaded (in the scope of berkelium, there are other chromium threads in the host and there can be other user threads on the library part)

Because of this it is required to call runtime.update() (like the old berkelium version). The will read all available data from all pipes and pass the decoded messages to the matching Channels.

Examples for this in the library code:

For example here https://github.com/berkelium/berkelium/blob/master/berkelium-cpp/src/lib/API/Window.cpp#L64-72 , but currently commented because there are no Window Delegate Functions implemented.

Another implementation (for Tab) is here:

https://github.com/berkelium/berkelium/blob/master/berkelium-cpp/src/lib/API/Tab.cpp#L51-67

[It makes sense to pass the Message directly to internalUpdate(), i already planned to change this]

More interesting is the same implementation on the host side:

https://github.com/berkelium/berkelium/blob/master/berkelium-cpp/src/host/shared/BerkeliumHostInstance.cpp#L58-L74

Here (already onDataReady with Message, not internalUpdate) the decoded Message is passed to onCreateWindow or exitHost.

[other example with onCreateTab here https://github.com/berkelium/berkelium/blob/master/berkelium-cpp/src/host/shared/BerkeliumHostWindow.cpp#L70-L82]

In onCreateWindow the Channel ID of the new Window is send using IPC https://github.com/berkelium/berkelium/blob/master/berkelium-cpp/src/host/shared/BerkeliumHostInstance.cpp#L82-L84 to the Library, which is receved here https://github.com/berkelium/berkelium/blob/master/berkelium-cpp/src/lib/API/Instance.cpp#L163-L165

This are examples for the "default" Channel, the reverse channel is not really used yet. An (not realy good) example is here:

https://github.com/berkelium/berkelium/blob/master/berkelium-cpp/src/host/fake/fake.cpp#L217-L218

and 

https://github.com/berkelium/berkelium/blob/master/berkelium-cpp/src/host/fake/fake.cpp#L255 which sends "onReady" the the Tab Delegate. Note that the Channel is "tab2" created using "ChannelRef tab2(tab->getReverseChannel());"

I hope this helps to understand my code. My first priority currently is to create a first usable Version. Because I don't have too much time there is not good documentation yet :-(

I wrote some examples for the client side API, you can find them here: http://berkelium.github.io/berkelium/

If you have any questions or if you need help please contact me.

Best regards

Dennis

[Guillaume Le Biller]

Hello,

thanks for you great response. I still need to walk trough more code to completely understand it, but at least the Pipe system is more clear.

You are right, I think we can use bi-directional named pipes on windows instead of two uni-directional.

Does the Unix pipes are synchronous or asynchronous ? Is it usefull trying to implement the asynchrnous version of windows named pipe ?

Or since there will be two separate process, we can stick to the blocking api calls (which is easier) ?

I will first clean the Filesystem Windows implementation and commit it.

I think it could be nice to have some unit test for the Unix pipes that I can run on windows to test the windows implementation.

Does the IpcPipeTest.cpp (sendRecv) test is currently working in linux ? The current github version is not working because it's trying 

to use pipes in the same thread (and I am currently using the blocking version of the API).

I will probably have to write a new one from scratch since even the ASSERT on the file's existence is not valid for Windows.

Thanks for your explanation :)

Best regards,

Guillaume

[Dennis Rieks]

Hi Guillaume,

the PipeGroup uses "select" to wait for incoming Data on any of the Pipes added to the PipeGroup. Berkelium Host stdout and stderr are also added to the PipeGroup (one per Runtime).

There is a PipeCallback and ChannelCallback class to process incoming Data.

If the library code calls runtime.update(timeout) will be passed to PipeGroup.recv(timeout) here https://github.com/berkelium/berkelium/blob/master/berkelium-cpp/src/lib/IPC/PipeGroup.cpp#L109

This function will add all file descriptor IDs added to the PipeGroup to "fd_set fds;" (first for loop) and pass them to the select system call.

select will wait forever (timeout = -1) or the timeout.

If any data is available select will return without waiting for more data.

select will return 0 if there is no data. If some data is ready the second for loop will test all Pipes if there file descriptor is marked in "fds". If so, "onDataReady" is invoked for this Pipe Callback (every Pipe can have a callback, see registerCallback)

[if the pipe is a member of a channel, this PipeCallback will invoke the matching ChannelCallback]

The IPC would block if there is a try to read from a pipe if there is no data ready, but because select is used to "wait for" data this should not happen. [Message has a length field of type short, if you write only this short into the pipe (and no data) the berkelium library will hang and block the thread called update() until enough data is ready. This should be addressed later but is currently no problem i think.

I don't know how the asynchronous windows implementation is working exactly, but I think we should not use it if another thread is needed for them. I think that there is something similar to select in the windows API.

BTW: The reason why I don't used the chromium ipc code is that i think we should provide a binary safe api. This way we can later just replace berkelium.exe to "install" a newer berkelium/chromium version without the need to relink the application.

If there are changes in chromium ipc code (and this not a public api...) a newer berkelium.exe maybe don't work.

Best regards,

Dennis

2013/10/16 Guillaume Le Biller <guillaume...@gmail.com>

[Guillaume Le Biller]

Hello Dennis,

Thanks for the clarification of making a new API in berkelium-host and not use Cromium IPC code directly ;) I was about to ask the question !

It's more clear now how the pipe system is being used in the library. Thanks.

I have some commits that are ready for the UtilWin32 & FilesystemWindows classes.

I also added some required cases in the CMakeLists files.

There are few things that I have noticed although :

* There are lot of WINDOWS defines : OS_WINDOWS, WIN32, WINDOWS. I think we should make a clean unique define for it.

* It's the same with the OS specific implementation : PipeWindows, FileSystemWindows, ProcessWin32, UtilWin32. Maybe choose one format and apply to the two others ;)

* The PipeGroup class need to have OS specific implementation since select does not work on Windows Named Pipes. We will probably have to use the asynchronous mode of windows pipes. It's using a callback system and does not require threads. Therefor we should be able to call the correct onDataReady pipe callback.

Best regards,

Guillaume

[Dennis Rieks]

Hi all,

I started a small overview for the new version:

https://docs.google.com/presentation/d/1XvlWYgO1_rPtIb65cY1gXfSawaZDrYLrK3xxGKJfWzU/edit

Please feel free to add questions or improvements ;-)

Regards,

Dennis

2013/10/20 Guillaume Le Biller <guillaume...@gmail.com>

Hello,

thanks for fixing the unit test, I will  be able to try it with the windows version :)

I think that It's better to let you finish the first version of the PipeLinux class

before working on the Windows one since it may change a lot :) As soon as you have a good version of it just tell me and I will look 

into creating the PipeWindows :)

I don't have time until middle of next week to work on it, I will keep you informed.

I will create a skype account, thanks !

Regards,

Guillaume

On Sat, Oct 19, 2013 at 9:24 PM, Dennis Rieks <denni...@googlemail.com> wrote:

Hi Guillaume,

I fixed some bugs introduced with commit "ChannelGroup". Now all tests are working again on linux, at least with host-fake.

The next step I want to start now is to create a PipeLinux class with send/recv support using two "UnixPipes". I will also remove all Linux Code from common classes and move it to Linux classes.

If you have any questions you can also chat with me via skype, my account is dennisrieks.

Regards,

Dennis