Announcement: Status, development, concept, ideas, etc.

852 views
Skip to first unread message

Kevin Ingwersen

unread,
Dec 22, 2013, 8:29:10 PM12/22/13
to appj...@googlegroups.com
Hey everyone!

Phew, it's been a while, isn't it? :§
Well incase you wonder how I am...well, I certainly feel like the wobble-bass of Dubstep. XD From one side to another, forth and back...all around and over and upside down.


Why?
Because, development. In LESS THAN A MONTH i taught myself C. I then added some basic ObjC coding - and extended that all with my PHP knowledge and made it C++ - and finally Obj-C++. So things like

[instance Operation:@"default" withData:classInstance->getInfo()];

is not even unnormal, rather trivial. :) Kinda proud of myself right here!

But for what all the learning? Because Deskshell.


Evolution of Deskshell.
I have taken over appJS, and have looked into it and decided: I can not redo it. The reason is simple: the building just won't work! Making it CEF3 ready would require a whole re-write - so I just declared it as obselete and done. But, it's spirit is not gonna stay as it is. In fact, I have planned a relaunch as "AppJS Remix". I am a DJ - so for me, a remix is taking a song, and making something new of it. That is what I am doing with appjs here.
The Deskshell I am creating now is a structure based off nodejs, the thing we all know and - one or another ;) - may love. So, let me show you the structure first:

nodeJS -> deskshell-core -> |
-> nodeJS \
-> ph7 | -> libcef -> WebView
-> html5 /

I am not mad at you if you are like "WTH is that?!" - let me explain.

The new Deskshell version is based off a nodejs binary, that has a modified "entrence point" - that part, that is run when you call a binary. But nodejs' entrence point is actually a pure javascript file, and called by main(). The modified entrence script will read in the .desk file and initialize a module: Deskshell-core. The settings/configuration stored in .desk are then pushed to the "core" module. That module now decides on hwot o proceed:

- If the user specified to use a php script to bootstrap the resutlting app, lets use ph7.
- If the user wants to use nodejs, use nodejs in a new process and let the current just die away.
- Otherwise, create a webbrowser and display the content of the htdocs folder. That however is CGI enabled as well.

So, you can either bootstrap a programm (setting up custom listeners, functions, prepairing files, etc) or use Deskshell as a frontent for a plain HTML5 app. This all happens thru Deskshell-core, which is capable of calling in another engine, or relaunch itself. As you see, the plain binary will only really prepair for a new process. That said - you get two scripting languages in one. ph7 is a "fake" php, basically an embedable php for smaller projects based off the 5.3 definitions (no traits, no short-hand syntax for arrays/objects). However, its extendable thru a C-API. I have created a C++ wrapper (ph7++) that will hopefuly soonb e released by the dev team, once I helped them to fix the socket connections (socket_read/write/connect/...). I also ported some actual PHP extensions - currently only curl - to this engine. So in that term, you'd get a full PHP if you want to bootstrap your app using PHP, or even use it within the pages of your project that are subject to be displayed int he browser. .php files will be treatened as such - whilst .njs files will be processed by nodejs, giving you the ability to use nodejs templating engines for a Deskshell app.

The structure is ment to make it possible for javascript and PHP users to create a bootstrap if they want to. Both have their advantages, and disadvantages. I am a very engaged PHP developer, but I can do little JS. And drag0n, about which I will talk later, is a pure PHP project - so it will barely use any javascript at all.

Scripting the application launch also means that you can add custom features ont he go: Like you can implement a cgi for perl, python - or eve lua. Or you can check for updates. Oh, yes, we even have a gui.


The "new" GUI concept
I have been searching and doing heavy research on using a GUI framework, since CEF - chromium embedded framework - requires you to prepair a GUI for it, as it itself will only draw a bitmask into the window, and nothing else. That is also why their loading icons and such are png files - CEf displays a "picture". Of course, users with handycaps can use it too because its of course more than just a picture. But in all generall, CEF only really draws a picture, and according to the code that calls the library, updates the code.

My solution here was to use wxWidgets. Mainly because:
wxNode: https://github.com/joeferner/wxNode
and
wxWebViewChromium: https://github.com/steve-lamerton/wxWebViewChromium

As you see: The first makes wxWidgets available to nodejs - and the second acts as a simple wrapper for wxWidgets. With that in mind, we can now use a CEF instance according to the wx coding style! That said: We only need to compile one library (wxwidgets) with only the basic requirements in order to kick off a browser - with menu and whatever. And yes, a transparent window is possible too, from what I have seen in wxWidgets! So, you can still creare cromeless windows liek you did in AppJS.
Oh yeah, AppJS... ^^


AppJS Remix
Now that I told you how I am making the GUI, and how I am going to extend the GUI part to its underneath scripting engines, I can tell you that the extension - that will only be seen internally - will be called "appjs-remix". That means, that the combination of wxWebviewChromium and ndoejs actually represents the spirit of the old yet obselete appjs! I have thought - but not concepted - about re-implementing some of the old API in order to support some of the old features. However - once you are done, the actual event listeners you used to use are invaild. The scope then moves into a seperate process - with rendering and v8 engine - and I can not really port an IPC kind of communication - i am not skilled enough :I. But you'd at least get some of the old API back, that'd help you kick off a basic app again! ^^


Schedule?
First, chrismas. Then, I have to finish a PHP based intranet project for a company that I have worked for a while ago. They want to give me a job later, and help me a bit with my financials, it seems :p. Well, I have to make their project first, as I have been delaying for a while now and promised to do it in my winterbreak - which is now. After that, I will start pulling together a nodejs binary with the new libraries and programms integrated - basically, integrating the modules from above into the binary and seeing how they work, against which version, and how they work across plattforms. After that, I should have a nice pile with appjs-remix, ph7-node, wxnode and a base for deskshell-core. A nice pile to play with... then its just a matter of time to make Deskshell-core the "default" entry point. Once that is done, I actually oculd say that the binary itself is done. After it has been confirmed to worka cross platforms and that already created deskshell apps are still compatible, I can start to extend it further.
To make you think about something interesting:

- libcef is ~40MB big.
- ph7 - with my additions - is maybe 1MB
- wxwidgets full set of libraries on OS X are 42MB. Consider that we will only use maybe 1/10 of that :)

So, the future Deskshell runtime will be about 50-60 MB. But I am not done here. After I have this done, I can go and remove unneeded features out of CEF and start creating a new libcef - libcef-deskshell. That, if we dream a little, may only be 20MB. Calculating again, gives us a 40MB build. Now running UPX over it, by loosing a bit of speed in launching - we get down to 20MB. Now tell me if that isn't a small but powerful runtime?? =D


drag0n
Drag0n, yes, it still exists. Just recently I published a WIP repo of d0p - a type of archiving that allows adding a header description in YAML format to the archive. Then running 'd0p explain archive.d0p' can output useful information - such as a description, website, or whatever. In the sense of drag0n, it saves adding an informative folder with all the info files and alike. Instead, we can just query each header in the folder and store it in memory for a while and examine it, use it, or whatever. We can even use this method to read from a HTTP stream! So we can litterally extract the header out of the internet. Then we can display that in the drag0n interface and let the user confirm the installation - and download the rest, and stream the archive down to disk.
But drag0n has another, more important, factor. Drag0n consists of many different parts:

- "appstore" like interface to install and manage packages on a local system.
- Distribution platform for developers.
- Build tool for developers.

A subproject dubbed "d0.build" is just a little script that relaunches your shell and lets you use a portable toolchain, for building whatever application you may want to build! That said: its powerful. It can build packages from their source code right from a GUI and install them.
BUT THAT IS NOT ALL.
It is an appstore like thing. So, Deskshell apps can be distributed thru it. and since it is a deskshell app itself, you don't need to supply the whole runtime! Instead, it just "links" against the runtime it has within itself. So your previously 30MB big Deskshell app, due to inclusion of the runtime, may turn into a 5MB html5 based app. That does not just make downloading, but also installing a hell load faster!
Drag0n however does not just distribute Deskshell apps - but also GNUstep and gives a re-birth to a long-forgotten installer.


GNUstep and Fink
Eversince brew and MacPorts have launched, Fink has been pushed into the back of the scene, and I see a lot of people that use brew. But to be very honest, the reason why so rare people are using an installer is: No GUI. drag0n therefore provides a GUI and lets developers distribute apps for platforms that nobody has probably even seen xD. GNUstep is a platform to develop Objective-C/++ based applications across Mac, Windows and Linux. So drag0n installs already two platforms: Deskshell and GNUstep. And unterneath itself, it will use Fink with its very nice output format, and parses that in order to make manythings more understandable to the software itself. So in theory, alongside the drag0n structured repos, you can use an existing APT repo and just distribute packages thru that into drag0n. Why this works? Because Fink is APT for Mac. But drag0n will use the according package managers that are right for the current platform. I saw something called "win-get" for Windows, that is an APT clone for Windows. On anything Linux, I can just use the current package manager - zypper, apt, whatever.


Conclusion.
Due to my many projects within Fink, GNUstep and Deskshell, I am creating an SDK that makes creating HTML/JS/CSS based apps easier, but also adds support for server-side scripting with PHP and nodejs itself - making it a fully round-up solution to quickly create applications. And since you have actual wxWidgets bindings in there, you could even tell Deskshell to be fully pale, and use wxWidgets to create your app! That is, if you want to... ^^
But I am not done here, not closely. Due to my find-out about nodejs for ios, and the current CEF goal to port to Android and iOS, we might see Deskshell on mobile platforms too! And hopefuly I can find a way to devleop a port for win8 metro UI.


Getting involved.
The appjs project may be deprecated/obselete/whatever - but it is the perfect place to talk to me. It is where everything started. Alternatively, feel free to browse my site http://ingwie.me (still under construction) and contact me directly - or just mention Simon and ask him for his email. Simon knows - i think ^^; - everything about the plans that I have made - and ing eneral he is a very nice person to talk to and make concepts with. ^^



Kind regards, have nice holidays and all the like!
- Ingwie Phoenix, alias Kevin Ingwersen

PS: In some time soon, Ill make a video and hold a short speech about a lot more things than what I mentioned here. Yes...I ish bussy burdeh @.@ ... err, I mean, phoenix. o.o
Oh and if you find typos, you may keep them =D

My Other projects (skip if not curios)
- GNUstep for Mac OS: Create an alternative toolchain to Apple's Xcode and make OS X development easier - but also cross-platform development. My build will feature a static libobjc2 build, in order to make sure you dont depend on anything out of my build.
- Experimenting with wxWidgets and CEF: Making sure that the appjs-remix and wxWebViewChromium things work just as expected.
- Create the full drag0n structure and interface: I have a PHP layout for drag0n (d0) that will make it possible to use it anywhere where you want. And another idea is to create a single, little binary. So even if you dont have PHP or anything, but need a short bootstrap thing to download libraries or dependencies, that will help you. Just compile d0.cpp and d0p.cpp into a d0 binary, and you can issue commands such as "d0 install libicu" to get the latest version of icu into your current working path. As it will just be a bootstrap helper, it will not write any information files or alike - it will read everything off thru the internet.
- Create a Deskshell "compiler" for the current version to make proper Mac OS X bundles.
- Finish other stuff such as WingStyle, my own site, and my other stuff you can see on my small site.

Nicolas Embleton

unread,
Dec 23, 2013, 5:47:29 PM12/23/13
to appj...@googlegroups.com, ingwi...@googlemail.com
That sounds like a lot of awesome hard work... Congrats and kudos for the learning curve :)

I will be (passively, sorry) looking forward to seeing what comes out soon :)

Wish you luck.

Kevin Ingwersen

unread,
Dec 26, 2013, 6:43:54 PM12/26/13
to Nicolas Embleton, appj...@googlegroups.com
@Nicolas: Hehe, thank you :)

I have some updates! ^_^

So, first I found this ultra-brilliant idea of a developer. This however is only useful if you have some C++ base: https://github.com/rvagg/node-addon-examples
It is, litterally, a point-blank example page for nodejs addons. that is how i teach myself the best o.o

I tried out a lot of different GUI tools - wxWidgets, CEGUI, and even Qt xD - but then came across a few issues.

- The wxNode plugin does not compile with wxWidgets 3.0. The suggested 2.9.3 release does not build on OS X. So, that one falls off.
- wxWebViewChromium wont compile, because it was PLAINLY written for Windows. So unless I give it a whole rewrite, its useless. x.x
- CEGUI has so many dependencies, that I now have a full bloat of Boost, lua, python, and whatnot. x.x… no thanks, too many dependencies.

So I searche dfurther…and it was my DJing that brought me a solution. To stream the output from my mixer as MP3 to my server, i use a tool called „Butt“ (broadcast using this tool). And when I read on its sourceforge page again, it revelaed that it used „FLTK“. Another browser I once saw, called Dillo, does too! But it looked weird when I tried… So I went, and got the actual, full, source: FLTK 1.3.

Playing around with it actually made me belive one thing: The developers that used this library obviously gave a s*** on the actual GUI. FLTK allows yout of ully colorize it, replace buttons with actual images - so on and so forth. I am currently even poking the developers to add controll over the titlebar as well. Yes, I talked to the devs. And I want to tell you why ;)

As I digged further into it, I created a little, simple test app, and also examined the examples. Quickly I learned that the classes it uses - its c++ - are extendable, and allow you to override anything you’d like to have overriden. But there is one special function that is overkill. You see, to spawn the base window, it uses the native toolchain (HWND, NSWindow, GtkWidget, …). But due to its brilliant way of building, the whole controll falls under Fl_Window, Fl_Button, etc. Effectively making it possible to host the same API on each plattform. This also counts for menubars. With an actual #ifdef, I can either use an in-window menu bar for windows and linux, or the actual OS X toolbar:

#ifdef __APPLE__
Fl_Menu men = fl_sys_menu(…);
#else
Fl_Menu men = fl_menu(…);
#endif

See what i mean? =)

But, the overkill function - or rather „method“ - is fl_xid(Fl_Window *). What it does is, it spills out the actual, native, window object! So that way, I can use this framework with CEF. Why? Because Cef::createBrowser() requires a native window object - NSWindow, HWND, GtkWindow….etc. So I can just pass fl_xid(myWindow) as an actual parameter, and the problem is fully solved - we create our window over FLTK, and let CEF take the inside with a native API.

- That problem is solved - the GUI i am using is FLTK :)!

Next thing? Nodejs bindings.
I want to create basic bindings for FLTK. The great thing is, that the actual library is…

Ingwie@ingwies-air ~/Downloads/fltk-1.3.2/lib $ du -h *.a
1,6M libfltk.a
32K libfltk_forms.a
156K libfltk_gl.a
72K libfltk_images.a
228K libfltk_png.a

tiny. And since we are already using the library for CEF - we can reuse the implied symbols for some nodejs binding. So one more nodejs module I will actually make is fltk-basic. So we can spawn small alerts and stuff - in case something goes horribly wrong. The same bindings will be available within the php scope too - so you can spawn windows with each and every method - using alert() or fltk.dialog() :)

- Next issue tackled. a nodejs mini-gui will be made!

So thats for now from my side, Im going back to developing. This is just getting started man! ^-^

Kind regards, Ingwie

PS: If you are doing small apps in C++ and want them cross plattform, I’d really hint you towards FLTK, its simple and fast. and ultra tiny. ^^

Simon Horton

unread,
Dec 26, 2013, 8:24:37 PM12/26/13
to appj...@googlegroups.com, Nicolas Embleton

Wow great Kevin. Really interesting to see the thought process and how the ideas and solutions are getting better and better. Interesting Cef takes a window object to be added to, I had not thought about it exactly like that before. Excited to see the end result of these ideas :-)

Simon

--
You received this message because you are subscribed to the Google Groups "appjs-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to appjs-dev+...@googlegroups.com.
To post to this group, send email to appj...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Kevin Ingwersen

unread,
Dec 26, 2013, 8:27:25 PM12/26/13
to appj...@googlegroups.com, Nicolas Embleton
Well I’ll continue to post about my progress, because I think its motivating! ^_^
Currently I am making hello-world apps with FLTK to practice it. Tomorrow, I will use fl_xid and actually see how CEF behaves underneath it. FLTK creates a subclass to NSWindow called FLWindow - notice, no underscore. But it should work still. Once i have a test with a basic sample done, I’ll put online the basic example so you can try to compile it on your platform - FLTK is fully cross-platform. I was also just taught how to speed compile it - so i need to colect some config.h’s, the configurations generated from ./configure. I need one from Windows and some linux’es…but ill do that stuff later. :)

Kevin Ingwersen

unread,
Dec 27, 2013, 1:59:53 AM12/27/13
to appj...@googlegroups.com, Nicolas Embleton
Fingers crossed! I just hand-build my simplecef. Now to substract the platform-specific code and replace it one by one with FLTK! :)
Am 27.12.2013 um 02:24 schrieb Simon Horton <siho...@gmail.com>:

Kevin Ingwersen

unread,
Dec 27, 2013, 3:13:12 AM12/27/13
to Nicolas Embleton, appj...@googlegroups.com
Ok… its 9am, time to sleep.
Meanwhile Ill wait for the devs to mail me a fix for an override I just cant get right. Unless that, everything else compiled just fine. But the file where the errors are is actually the app bootstrapper. So I can’t get it off x).
Anyway, work for today~

Simon Horton

unread,
Dec 27, 2013, 3:38:36 AM12/27/13
to appj...@googlegroups.com, Nicolas Embleton

Wow that was fast, assumed it would be more challenging!

Kevin Ingwersen

unread,
Dec 27, 2013, 4:32:28 AM12/27/13
to appj...@googlegroups.com, Nicolas Embleton
Very flickery, and funnily upside down, I got CEF to render within an FLTK window :)
Now to fix the damn flicker… @.@ it felt like eye cancer...

Kevin Ingwersen

unread,
Jan 1, 2014, 7:43:56 PM1/1/14
to appj...@googlegroups.com, Nicolas Embleton
That should explain itself. :)

Kevin Ingwersen

unread,
Jan 1, 2014, 10:59:09 PM1/1/14
to appj...@googlegroups.com, Nicolas Embleton
A follow up to my last puush link: https://github.com/IngwiePhoenix/fltk-cef

Kevin Ingwersen

unread,
Jan 3, 2014, 5:15:53 PM1/3/14
to appj...@googlegroups.com, Nicolas Embleton
While letting the developers from FLTK work on some fixes, I have taken the time and looked into another requirement.

When using CEF, the actual libcef.dylib/a/dll only exports a C-API. To use it thru c++, one needs to build the libcef_dll_wrapper, a static library that converts the C api into C++.

Fortunately, the distribution lacks two features:
- Building a proper .framework for Mac OS without xcode, and cefclient
- Building just the static library.

I got around the first problem easily, and made a shell script - that simple it is.
The second was a bit harder to get around… In fact, I had to studdy GYP some.

In order to just build the wrapper library, I read up the GYP file (cefclient.gyp) and determined what I needed, then went and tried out to compile by hand. After several tries, I figured that I needed two macros, two include paths, and a perma-include. But then, I compiled it pretty nicely, and even reproduced the static library build thru xcode! woohoo~ :3 But, how to build without xCode or gyp? … Ninja.

In the past, I had started a set of php-based development tools: pcc and autoNinja. PCC compiles a php source file into the ph7 engine and makes it executable. autoNinja takes a php script, with pre-exported functions, and allows it to define some build rules and the like. That way, I can speedfuly and automatically build ninja buildfiles. Now, I have created a 10-liner of PHP code, and made a full build file for Ninja: http://puu.sh/68syn.ninja

With further adjustion, the autoNinja file i used (ninja.an) will soon detect the OS and set the declarations right. Currently, it’ll only build on OS X, because clang defines OS_MACOSX. You can manually change that to OS_WIN32 or OS_LINUX. But I didn’t try out the other platforms yet ovo’

So, here is a current list of dependencies, and how far I got with them.

- FLTK: Working.
- PH7: Working.
- CEF: Working.
- libcef_dll_wrapper: Working, buildable.
- FLTK-CEF bridge: Half-done
- deskshell-core: Not started.
- node-fltk: Not started.
- node-ph7: not started.
- node-simple-gui: Not started

To explain the node extensions:
node-fltk will offer a soft binding to FLTK. We already use it in fltk-cef, so we can re-use the library if we’re already compiling almost the whole thing into the binary.
node-ph7 will launch ph7 and give it the actual script, register exports…so on and so forth.
node-simple-gui will offer dialogs, file picker and stuff liek that thru FLTK.

So? :3 I am happy!

sihorton

unread,
Jan 7, 2014, 5:15:06 AM1/7/14
to appj...@googlegroups.com, Nicolas Embleton, ingwi...@googlemail.com
Very fun to see how far you have come and everything that has been going on. Looking forward to seeing the new possibilities in deskshell and am itching to write some applications that run inside these programs!

/Simon
 

Kevin Ingwersen

unread,
Jan 13, 2014, 5:41:35 AM1/13/14
to sihorton, appj...@googlegroups.com, Nicolas Embleton
Amazing news! CEF and FLTK now behave perfectly together!! =D
The person responsible for the Mac port helped me fix errors - and now it works amazingly well.
To make the platform dependent changes, I am using an aproach that lets me include one heater, which includes the one needed for the platform. So I have mac.h and mac.mm to implement 2 functions to handle small additions. Same will be done for win.h/cpp and linux.h/.cpp.
Now i am very happy, waiting for the changes we aproached to be merged into the svn repo, so everyone can use them.
So, in other news:

- FLTK and CEF: Working!

Time to modify and clean the code and test it on other platforms. :))) Once that works, it’s time for node-fltk.
ph7 is currently also learning a new function to import scripted functions into C scope - so we can do crazy stuff with callbacks from the bootstrap code.
After that works, node-ph7 should be no biggie to be made.
That’s my plan for now :)

Simon Horton

unread,
Jan 13, 2014, 5:51:30 AM1/13/14
to Ingwie Phoenix, Nicolas Embleton, appj...@googlegroups.com

Wow well done again Ingwie! Amazed at the pace of these developments - keep up the great work.

Simon

Kevin Ingwersen

unread,
Jan 16, 2014, 5:38:22 AM1/16/14
to sihorton, Nicolas Embleton, appj...@googlegroups.com
Good news! After my success with FLTK+CEF, I went and created a new github organization: http://github.com/Deskshell-Core
Why the caps?.. Looked friendly-er ^^;

Anyway. I uploaded my first nodejs extension too! :)
And this is one more part of the new Deskshell-core project, which will use CEF and other things that I explained earlier. Well, once this works, I have one more part done. In the next step, I will be wrapping the FLTK+CEF stuff into a suitable little piece of nodejs code. It is only really ment to be fired off as a window. It will have however a basic API for configurating it with things such as custom schemes and so on and so forth. Thats neccessary to allow CEF configuration thru .desk files.
Then, I am going to create a rather bulky wrapper around fltk itself: FL(N)TK, Fast Light (NodeJS) ToolKit. With that, we get things like alerts, native fileschooser and even more.

That done, I have:
- An in-process PHP engine for nodejs.
- A usable CEF from nodejs, implemented via FLTK.
- A GUI toolkit to display things like alerts, if a programm cant do its thing for example.
- And for the best: The three very basics for deskshell-core!

Next I will have to deal with combining it all and creating the core functions. So that we can do things like that:

return require(„deskshell-core“).runWithObject( require(/*.desk file path here, probably argv[1].*/) );

This one-liner will kick off the actual core, which in itself will require fltk, cef and ph7, trigger the creation of a custom cef scheme, add a nice callback to read from the files and so on and so forth.
The custom cef scheme is especially useful as it may switch between reading flat files from disk - OR utilize simon’s vfs and read from a blob! :)

So, I am very, very close. Once I have the modules done, I will be able to invite you scripters. Because then I need a break, but your help! As deskshell-core is nothing but a nodejs extension - flat javascript - I think you can help me here creating the core, that will set everything up.

Alright…back to hacking. I just thought i’d share my current process.

Oh yeah, if you are a scripter or low level dev - or know some - you can pass them my email and invite them to Deskshell. Because there are several things I still need:

- FLTK needs to be gyp-yfied.
- Deskshell-Core needs an API concept.
- The in-app API (from within cgi scripts and such) needs to be designed too.
- Anything that will be helpful to enhance the usage of the whole things.

Happy hacking, and see you all on my next update! :)

Kind regards, Ingwie

Kevin Ingwersen

unread,
Jan 16, 2014, 1:55:07 PM1/16/14
to sihorton, Nicolas Embleton, appj...@googlegroups.com
One more thing to report: I now know where and how to patch the nodejs source tree to bring native modules into the binary, and use it from there - including JS files! Here is a small glimpse from my terminal, loading ph7 from inside the same nodejs executable that the script itself is inside - a fresh build as well.

Ingwie@Ingwies-Air ~/Downloads/node-v0.10.24 $ ./out/Release/node
--> We're using the inoffical process.binding method to obtain ph7 - from THIS executable!
--> No sh*t, this is all inside the same binary. This is going straight from the node binary... :)
> ph7: { create: [Function] }
--> Let's create a ph7_vm and have a peek at what we get...
> pVM: { include_path: [],
  '$argv': [],
  '$SGLOBALS': [],
  '$GLOBALS': [],
  '$_HEADERS': [],
  '$_SESSION': [],
  '$_COOKIE': [],
  '$_POST': [],
  '$_GET': [],
  '$_ENV': [],
  '$_SERVER': [],
  config: [Function],
  compile: [Function],
  compileFile: [Function] }

I am so excited! ^_^
Am Mo. Jan. 13 2014 11:51:30 schrieb Simon Horton:

Kevin Ingwersen

unread,
Jan 18, 2014, 6:45:21 AM1/18/14
to sihorton, Nicolas Embleton, appj...@googlegroups.com
And here it comes - release numero uno.
Last night, I created a working nodejs binding to ph7 - so now we can run php code from within nodejs! that way, we’re able to run php apps inside deskshell without an external binary.
The (currently rc1) version can be fond on Deskshell-Core: http://github.com/Deskshell-Core/node-ph7
.node binaries for the varous platforms are welcomed ;)
Currently, I am looking for a way to make npm install the module, without trying to build it, since not every OS has pre-installed compilers, like Windows. So if you now some npm stuff with native addons, please let me know.

The module also has been prepaired for an in-binary compilation to nodejs. I have released a tutorial on the nodejs mailing list contianing a way to staticaly compile javascript and addon files into the binary.

Next is the FLTK binding! :)

Kind regards, Ingwie

Am Mo. Jan. 13 2014 11:51:30 schrieb Simon Horton:

Kevin Ingwersen

unread,
Jan 18, 2014, 6:51:02 AM1/18/14
to sihorton, Nicolas Embleton, appj...@googlegroups.com
Now available on npm too: https://npmjs.org/package/ph7
Am Mo. Jan. 13 2014 11:51:30 schrieb Simon Horton:

sihorton

unread,
Jan 18, 2014, 7:55:15 AM1/18/14
to appj...@googlegroups.com, sihorton, Nicolas Embleton, ingwi...@googlemail.com
This is really cool, I think the ph7 module can become very popular, will be interesting to see what people make of it and how it is used!

For a binary package install then you can look at the sqlite3 module (https://npmjs.org/package/sqlite3), I have noticed that now it just drops a binary into the folder on windows, before it used to compile the module instead. I don't think there is any clever magic involved in achieving this. It is possible to run an install script, this can see what the os and version / type are and then download / extract a binary instead of compiling the module.

/Simon

Kevin Ingwersen

unread,
Jan 18, 2014, 11:27:16 AM1/18/14
to sihorton, appj...@googlegroups.com, Nicolas Embleton
I found a way :) I am going to use npm inside npm! To be clear: I can use the package.json’s „scripts“ section to fire off a „configure“ script. So that way, I can do custom stuff in js. Not just that, but I can also add urls to .tar files as a devDependency. Tha tway, if somebody wants to build the package, they run „npm install“ INSIDE the package. Then, the DevDependencies are installed - which in this case would be the bindings.gyp file, which allows building :).
Using npm, I can also download binaries via npm. I looked into how appjs did it, and it was quite easy. There was more than one appjs package:
- appjs-darwin
- appjs-win32
- appjs-linux-x86
- and more.
So I can do the same:
- ph7-win32
- ph7-darwin
- …
That way, I can download the package at need. Speaking of which; to use npm, it has been added as a dependency to the file. So i get the small bit of npm that I can call from node:

var npm = require(„npm“);
npm.load(); // Let it initialize itself
npm.install(„ph7-darwin“);

Voila. :)

I have to work on the architecture some more, but I’ll be able to drop that on. But that way, people can do the following:

$ npm install ph7
$ cd node_modules/ph7
$ npm install
$ npde-gyp rebuild

The last two commands install the dev dependencies - and the last builds the module! So, the resulting build/Release/ph7.node can be published.

I will need various builds of the project anyway. But just look into appjs/package.json and see how it calls a script called preinstall.js . Open that, and lookw hat Milani did. Well he was depending on the npm binary to be in PATH, but my aproach works without that too. Thus, I’d rather use actual js than child_process.exec for that task :)

Kevin Ingwersen

unread,
Jan 31, 2014, 3:48:39 PM1/31/14
to sihorton, appj...@googlegroups.com, Nicolas Embleton
Okey! Some new news from what I have here…and I have a near-alpha O.O!

So, the process is now going in a way that is different from before. I am patching the node.gyp and node_natives.h file in order to enable native modules to be compiled into the nodejs binary - same with javascript modules. That way, I have implemented ph7, native addon, and optimist (with its dependencies wordwrap and minimist) into the nodejs binary…but I am not done. So, here is a list of the status:

- FLTK+CEF PoC: It works. I can launch CEF thru FLTK, I just need to add minimal, platform specific, code to enable multiple windows and such…its really small. :o
- FLTK+CEF for NodeJS: Bindings in process. I am currently having some issues with GYP again x.x; And, well, with splitting the different calls. Its not easy, but its the biggest challange. But once this monster is done, then we have the browser we want. :)
- ph7: Its buggy currently, and I have a lot of redundant code. I have to work thru it, and fix it. But the base by itself actually works - but not all the things get converted correctly into ph7_values. o-o
- Deskshell-Core: The core code is already half-done. This is the code, that runs instead of nodejs’ actual main() function. It can differenciate on the different file extensions, and such. So, its going somewhere.
- Deskshell: The public API is not done - yet.
- ttvfs: This is a new piece of C++ code i found. It enables AMAZING! features. I’ll explain that in a bit.

So, yeah, ttvfs… ^^

A while ago, I was looking for some really exotic kinds of software; virtual file systems, reading RAM as a block device (to interpret it as FAT32) and things like that. Then, very randomly, I came across this small project called ttvfs: https://github.com/fgenesis/ttvfs
This VFS implementation does not just offer a cross-platform way to manipulate and handle files…no, it offers a special way of handling internal buffers. In fact, I can turn a char buffer into memory file. Here is a short bit of code to demonstrate what I ment:

// Now, lets create a memfile.
VFSFileMem* memZD = new VFSFileMem("data.zip", ZipData, ZipData_length, VFSFileMem::REUSE);

vfs.GetDirRoot()->add(memZD, true, VFSDir::NONE);

// Mount the memory-virtual data.zip as a folder called data.
vfs.AddArchive("data.zip", false, "data");

After the .AddArchive call, we now can access the contents of data.zip like it was a folder - AND data.zip is actually a char buffer residing in memory. That is cool, is it not? Okay, maybe you didnt understand what it does yet… So, let me word this differently.

Imagine you have a Deskshell app, but want to bundle the source code into the binary, to make extraction/modification harder. You can put the entire source into a zip file, run that zip file thru a tool I created today and turn it into a C source file - which is just a char array - one, that you can then mount using the VFS and access your files again, just like they were local.

But how is this useful to the end-user? Well, I am working on nodejs bindings for this nice thing! Because, besides from giving us another FS implementation, it can make it possible to treat variables as they were actual files - so we can serve a whole app from within a zip file, that is even burried inside the nodejs binary. Cool, huh? :)

If you are curios about the tool I mentioned: http://github.com/IngwiePhoenix/file2c

Here is an example output:

Ingwie@Ingwies-Air /tmp $ cat foo.txt
Hello, world\!
Ingwie@Ingwies-Air /tmp $ ./file2c foo.txt fooFile > foo.c
Ingwie@Ingwies-Air /tmp $ cat foo.c
char fooFile[] = {
0x48, 0x65, 0x6C, 0x6C, 0x6F, 0x2C, 0x20, 0x77, 0x6F, 0x72, 0x6C, 0x64, 0x5C, 0x21, 0x0A
};
int fooFile_length=15;

So you see, we produced a small C file, that contains the file, stored in its actual bytes. From now on, we can use this file in our own C++ project and use ttvfs to mount that file in memory, and treat it just like it was a real file. That, being just one of thousand possibilities.

But there is more.

I am currently working on an alpha-version of the CEF+FLTK binding for NodeJS, so we can run at least a basic browser off a specific path. In the next version, I will first make it possible to patch the router process - so that we can modify the router to read files using the VFS implementation, or a custom one (like if you have made your own method to read files from somewhere).

I will post a few more details when I have a working preview version. :)

Kind regards, Ingwie

Simon Horton

unread,
Jan 31, 2014, 5:11:41 PM1/31/14
to appj...@googlegroups.com, Nicolas Embleton
Wow I have to say I continue to be amazed at the progress, new ideas and parallel development of multiple cool things all at the same time. This is build and building into one big massive new generation of deskshell that is going to land. As always I cannot wait to begin running my apps in this thing!

/Simon 


On Fri, Jan 31, 2014 at 9:48 PM, Kevin Ingwersen <ingwi...@googlemail.com> wrote:
Okey! Some new news from what I have here...and I have a near-alpha O.O!

So, the process is now going in a way that is different from before. I am patching the node.gyp and node_natives.h file in order to enable native modules to be compiled into the nodejs binary - same with javascript modules. That way, I have implemented ph7, native addon, and optimist (with its dependencies wordwrap and minimist)  into the nodejs binary...but I am not done. So, here is a list of the status:

        - FLTK+CEF PoC: It works. I can launch CEF thru FLTK, I just need to add minimal, platform specific, code to enable multiple windows and such...its really small. :o

        - FLTK+CEF for NodeJS: Bindings in process. I am currently having some issues with GYP again x.x; And, well, with splitting the different calls. Its not easy, but its the biggest challange. But once this monster is done, then we have the browser we want. :)
        - ph7: Its buggy currently, and I have a lot of redundant code. I have to work thru it, and fix it. But the base by itself actually works - but not all the things get converted correctly into ph7_values. o-o
        - Deskshell-Core: The core code is already half-done. This is the code, that runs instead of nodejs' actual main() function. It can differenciate on the different file extensions, and such. So, its going somewhere.
        - Deskshell: The public API is not done - yet.
        - ttvfs: This is a new piece of C++ code i found. It enables AMAZING! features. I'll explain that in a bit.

So, yeah, ttvfs... ^^


A while ago, I was looking for some really exotic kinds of software; virtual file systems, reading RAM as a block device (to interpret it as FAT32) and things like that. Then, very randomly, I came across this small project called ttvfs: https://github.com/fgenesis/ttvfs
This VFS implementation does not just offer a cross-platform way to manipulate and handle files...no, it offers a special way of handling internal buffers. In fact, I can turn a char buffer into memory file. Here is a short bit of code to demonstrate what I ment:


    // Now, lets create a memfile.
    VFSFileMem* memZD = new VFSFileMem("data.zip", ZipData, ZipData_length, VFSFileMem::REUSE);

    vfs.GetDirRoot()->add(memZD, true, VFSDir::NONE);

    // Mount the memory-virtual data.zip as a folder called data.
    vfs.AddArchive("data.zip", false, "data");

After the .AddArchive call, we now can access the contents of data.zip like it was a folder - AND data.zip is actually a char buffer residing in memory. That is cool, is it not? Okay, maybe you didnt understand what it does yet... So, let me word this differently.
> - ...

Kevin Ingwersen

unread,
Jan 31, 2014, 5:18:12 PM1/31/14
to appj...@googlegroups.com, Nicolas Embleton
I am stuck with compiling some objective-c thru GYP. Have to wait for the GYP devs to help me out; because that is blocking the way for a first alpha. xD I need to get the OBJC working, otherwise I can not compile on mac… o-o

Kevin Ingwersen

unread,
Jan 31, 2014, 6:48:27 PM1/31/14
to appj...@googlegroups.com, Nicolas Embleton
Looks like it was drag0n’s toolchain that caused the error - its ObjC compiler needs upgrading. o-o
Now, I have a very tiny binding to CEF…testing that now :)
Am Fr. Jan. 31 2014 23:11:41 schrieb Simon Horton:

Kevin Ingwersen

unread,
Jan 31, 2014, 6:58:56 PM1/31/14
to appj...@googlegroups.com, Nicolas Embleton
My first attempt just crashed straight away. :D

Ingwie@Ingwies-Air ~/Work/node-dev/fltk-cef/build/Release $ node
> o=require("./fltk_cef.node");
{ run: [Function] }
> o.run();
Illegal instruction: 4

Whatever! you see that there is the .run method? :) So, I am likely on the right road here. I just have to check why I am getting THAT error O.o
Am Fr. Jan. 31 2014 23:11:41 schrieb Simon Horton:

Kevin Ingwersen

unread,
Jan 31, 2014, 7:05:44 PM1/31/14
to appj...@googlegroups.com, Nicolas Embleton
Test 2 is a success. Although i DID get a crash, I did also get what i wanted!

Ingwie@Ingwies-Air ~/Work/node-dev/fltk-cef/build/Release $ node
> o=require("./fltk_cef.node");
{ run: [Function] }
> o.run();
2014-02-01 01:02:28.732 node[99268:507] Internals of CFAllocator not known; out-of-memory failures via CFAllocator will not result in termination. http://crbug.com/45650
[0201/010228:WARNING:resource_bundle.cc(251)] locale_file_path.empty()
[0201/010228:FATAL:main_delegate.cc(448)] Check failed: !loaded_locale.empty(). Locale could not be found for en-US
Trace/BPT trap: 5

This error there is just telling me that it can not find its resources. So in fact, I need to now mimic the OS X structure around this library. Once I did, I should be able to run this right away. But this also means I now need to change from working on a .node file to working on an actual nodejs source tree. Why? Because its easier to handle linking a binary to a dynamic library rather than linking a binary to a dynamic library,which links to a dynamic library. o-o But, I have an idea how to solve this all together! :)

So, from now on, I am working on the FLTK+CEF bindings, on the real nodejs source tree. I may find a way to get that working with two linked dynamic libs. But hey! It nearly started up after all :)
Am Fr. Jan. 31 2014 23:11:41 schrieb Simon Horton:

Nicolas Embleton

unread,
Jan 31, 2014, 10:23:03 PM1/31/14
to Kevin Ingwersen, appj...@googlegroups.com

Pretty cool... That's an awesome step to a working POC... Like Simon said, can't wait to see it happening...

Kevin Ingwersen

unread,
Jan 31, 2014, 10:25:49 PM1/31/14
to Nicolas Embleton, appj...@googlegroups.com
If GYP had been behaving since an hour or two, I could have even send in a screenshot already.

The problem is, that I can not build the shared library in the way that I want to. So defacto, I am a bit stuck… XD Anyway, I hope that I can find my fix soon. Gah, once I am done with GYP, i am _DONE_ with gyp. most frustrating piece of tool ever….

Kevin Ingwersen

unread,
Feb 1, 2014, 1:03:51 AM2/1/14
to Nicolas Embleton, appj...@googlegroups.com
Okay, so I get the browser to open, but the helper process crashes straight away. That makes me wonder how CEF tries to interact in its multi-process architecture. The sub-process probably tries to read memory from some nodejs chunk it can’t work out.
But I swear, for a short second, it does indeed try to render google.com :) Very, very close~!

Simon Horton

unread,
Feb 1, 2014, 3:42:05 AM2/1/14
to appj...@googlegroups.com, Nicolas Embleton

Take your time and play around with it. This is a really good direction and it is worth getting it to work really well. I am sure many similar problems will popup and you will get around them as you always do. Don't be stressed by us looking forward to the end result we all understand this will take its time, but is well worth investing in.

Simon

Kevin Ingwersen

unread,
Feb 3, 2014, 7:24:25 PM2/3/14
to appj...@googlegroups.com, Nicolas Embleton
I just gonna say this: If libcef wasnt „broken“, i had a demo.

I am talking on the cef forum with that magreen person - and he is helping me to tyke down errors. One has to do with - i belive - GL rendering. The errors came up while using my node-fltk-cef build. I already though that this was my code’s fault - TURNS OUT ITS NOT! :D

After Magreen suggested me to build a new cefclient, I build a debug version of it. Guess what,it threw the EXACT same errors O.O! That said, I belive that I basically finished a miniature binding for CEF to nodejs - i just have to fight the GL issues :)

Tonight is awesome, period. ^.^
Am Sa. Feb. 01 2014 09:42:05 schrieb Simon Horton:

Take your time and play around with it. This is a really good direction and it is worth getting it to work really well. I am sure many similar problems will popup and you will get around them as you always do. Don't be stressed by us looking forward to the end result we all understand this will take its time, but is well worth investing in.

Simon


Kevin Ingwersen

unread,
Feb 7, 2014, 6:42:06 PM2/7/14
to appj...@googlegroups.com, Nicolas Embleton
With an earlier version of CEF - as the current one is obviously partialyl broken on OS X - and a lot of learning in GYP, i made … this!!! :D


The current error has to do with the CEF context. I guess it has to do with the way I quit CEF after all… o.o But hey! This is a CSS 3D demo running, from a CEF, spawned via nodejs :D

The root for Deskshell is now here. Time to code mor eon this.

Kevin Ingwersen

unread,
Feb 9, 2014, 9:58:36 AM2/9/14
to appj...@googlegroups.com, Nicolas Embleton
And it just became even better =D.
I figured out, that I do not neccessarily need to use CEF’s crapped v8 „interface“, but I can stick to the real interface instead! So now, I am using actual v8 code within CEF.
That means, I can indeed copy a nodejs v8 object - such as the global one? - and put it into CEF’s global scope. I can not intercept any of the browser frames directly, as ndoejs only has access to the main browser, and non-ui, thread. Therefore, all must go thru that global object. Which in return could do great things like this:


// CEF….
<script type="text/javascript">
// Maybe, the require object is copied into CEF's global later, but this is an idea for now.
var fs = deskshell.require("fs"),
var ct = fs.readFileSync("./blabla.txt");
deskshell.$ = jQuery;
</script>

// In node...
var $ = deskshell.$;
$("#header").html("Beep o.o");


Another example would be:
// nodejs
deskshell.myFunc = function(a, b){
console.log("a + b =", a*1+b*1);
};

// CEF
<script type="text/javascript">
deskshell.myFunc(1,2);
</script>


Please note, that in this case, the nodejs console will be used instead, as the object was created there, and not in v8. That doesnt mean though that we couldnt...

// nodejs
deskshell.njs_console = console;
// CEF
<script type="text/javascript">
deskshell.cef_console = console;
deskshell.log = function(msg) {
deskshell.cef_console.log(msg);
deskshell.njs_console.log(msg);
}
</script>


Interesting, no? That all is possible because CEF and Nodejs share the same javascript engine. But there is one thing I am already concerned about…and that is something I will have to work hard on: keeping the code so clean, that reworking parts of it wont be impossible. Therefore, the actual compiled layout will be quite a weird one x3.
Also, managing to keep a difference between the two v8’s is not easy, and I dont know yet how this will work out when I decide to compile it into a single binary. I can imagine though, that THIS move will cause me to learn more about GDB and LLDB… o-o

Reason? Because…see. In libcef, and in nodejs’ binary, we have the same symbols defined. Therefore, I need to get the v8 engine matched. And I mean, very matched - same revision, same release. Then, I am quite curios about what will happen in memory. Say the dynamic symbol lookup („dyld" on mac) finds the symbol for v8::Initialize in node and CEF - which one will it choose, which one will allocate its memory, and where? I heared that dynamic libraries have their very own way of being allocated in memory. Its confusing to me, but I’ll know…at some point o.o

For now, dream of a wonderful global object thru which you can at least interact with the browser process itself. X3

sihorton

unread,
Feb 9, 2014, 10:21:16 AM2/9/14
to appj...@googlegroups.com, Nicolas Embleton, ingwi...@googlemail.com
Bit quiet on my end for a while. New people came in at work and totally messed everything up. So I left and got a new better job in two days :-) Anyway a lot of things keeping me busy but will resurface at some time.

This looks like magic to me! Amazing if it would work but I think we will have to be careful about crashes here. If we have two separate v8 engines running and try to pass objects back and forwards between them the garbage collector can delete an object from the other v8 and we will get a crash at some point. Likewise we can get problems with two different threads accessing and changing values at the same time, leading to strange randomness and the like. Not had time to understand the design or the code so don't take this as criticism. Just that we would need careful planning to achieve this. Basically AppJS had problems caused by using a single event thread for everything. Hopefully we can have two "processes", something for the browser and something for nodejs and then some form of boundary between them. I would suggest something like a web socket, web worker or similar interface. Obviously though it would be wonderful to just be able to access everything from everywhere (or have a global object available in both scopes). 

If we had a command "injectScript" then it would be possible for nodejs script to add code (i.e. functions) to a running window object, likewise maybe the browser could do something similar. Then you could call a function and pass in a json object as a parameter and get a json object as the response. I think though if we try passing C++ objects available to v8 from one engine to the other we could get some tricky stuff going on, very difficult to debug.

Once again though please don't think I am being negative here. It sounds wonderful, all I am saying is to have a play around and see how far you can go, have a little think through these complex issues and if they are not able to be solved we can fall back to the message passing idea.

As stated I am kind of swamped at the moment so if I take a while to reply give me some time and I will reply in the end! Once again though great work ingwie you are really out there ahead of the game again and it is great to see! :-)

/Simon

Kevin Ingwersen

unread,
Feb 9, 2014, 10:44:19 AM2/9/14
to sihorton, appj...@googlegroups.com, Nicolas Embleton
Two days?! That soudns like magic to me o.o…

Yeeeep, you understood exactly what issue i am about to run into xD. That is why I am trying to currently match the versions of v8. Good thing is, that v8 has a wonderful type that disassociates it from the GC: v8::Persistent. As the name says, those are always available, and will only disapepar if the programme xits, or is expicitely ::Destroy()’d :)

For the event loops…yep. That is the next issue I have. Well, I know that I can seperate them. Here is a bit of a threading hieracy that I know would work:

nodejs
|- fltk gui event loop. This thing is so pseudo… xD. It actually only idles about and around, and it is only used once to create a window for CEF.
|- CEF message loop. This spawns multiple threads or processes. I belive that I if I kick that off in a seperate thread, it will become non-blocking and hence will run totally async. Therefore, it shouldnt block nodejs’, as its in a separate thread…methinks.
|- Deskshell boundary. This is just a little thread with a few functions that keeps a v8::Persistent, which correspondends to the Deskshell object, available in both other threads. That however comes back to what oyu mentioned. I need to implement some kind of locking (Mutexes o-o) so the data isnt randomized. Howeve,r if I get that one fixed, then this object exists between the two v8 engines. Both use the same api - and hopefuly the same code - to access that Persistent. That way, it should be ensured that the data doesnt get corrupted. At least, thats my theory.

I am glad to hear from you again anyway :). I will keep posting all my updates right here. Have you seen the screenshot i sent in an earlier email? I went and picked up my bacardi for that night, because being able to run cEF from nodejs was the first big burden to go over. Now, this burden is done for :)

sihorton

unread,
Feb 9, 2014, 12:36:38 PM2/9/14
to appj...@googlegroups.com, sihorton, Nicolas Embleton, ingwi...@googlemail.com
Well you know how it is in IT, if you have someone that is good and hard working then you really want them in your team so your life is easy ;-) So lets just say there are many places that would love to have me :-)

What I suggest is lets do both approaches. Basically my "conservative" approach won't take much work at all. The current deskshell already uses websockets to connect nodejs and the browser. That code at the moment is "demo status" so you can create an api however is best and convenient to you. Lets just make it so that if running deskshell in nodejs + separate browser that the api works, and then we can have the same api if running in the new deskshell. Since we already have this code, and we have the appjs bridge this should not take too much effort.

Then we can have the "ultimate turbo" approach where you can do as much as you find fun.  Great thing is it means there is no pressure on you if we do find some weird bug or problem since we can just fall back to the "conservative" approach again. This gives you the creative freedom which you excel in!

Your approach sounds good so I encourage you to follow what you are doing and see where it leads. For me personally if I just got a "browser" CEF that was cross platform then I could swap it in for chrome which is annoying me on windows now. Anything more than that is a bonus. We also have the CEF that I hacked together which should also be cross platform but is only temporary until your version lands. Currently I don't have so much spare hacking time but I am sure that will change.

/Simon

Ashish Negi

unread,
Feb 10, 2014, 12:19:53 PM2/10/14
to appj...@googlegroups.com, Nicolas Embleton, ingwi...@googlemail.com
It is great to see somebody so much dedicated :)


On Sunday, 9 February 2014 20:28:36 UTC+5:30, Kevin Ingwersen wrote:

>Reason? Because…see. In libcef, and in nodejs’ binary, we have the same symbols defined. Therefore, I need to get the v8 engine matched. And I mean, very matched - same revision, same release. Then, I am quite curios about what will happen in memory. Say the dynamic symbol lookup („dyld" on mac) finds the symbol for v8::Initialize in node and CEF - which one will it choose, which one will allocate its memory, and where? I heared that dynamic libraries have their very own way of being allocated in memory. Its confusing to me, but I’ll know…at some point o.o

I think that  V8 is static library . Have you compiled it as dynamic library ?
Even then think about it like that if somebody created a library like you know printf or cout functions of c and c++ are also in library, then if multiple programs use them, they should be given different memory, otherwise output of one program would garble into others :)
Libraries are just code. Whichever process uses that code is given separate memory. So i think that if CEF and nodejs are separate process, they would be having different run-time v8 engines instances.

if we put the nodejs and CEF in one process such that they both share same v8 instance  then the thing would work out of box. ( original appjs was one process but had two v8's, so it required that js-bridge that mirrored everything in both lands ) it was like desksheel minus 'websockets'.

Also, if we study CEF3 architecture, i think ( i am not 100% sure ) that like when we open new tabs in chrome, it creates new renderers and new v8 engines in the renderer. So v8 is not designed to be like one global thing, from which both CEF or nodejs could talk. 

 I just have a random thought that if we can let CEF create its v8 and then we just expose the v8 - handler and where nodejs initiates v8, pass that handler there, may be furthur nodejs internal-objects would be created in the same v8.

Simon Horton

unread,
Feb 10, 2014, 12:24:50 PM2/10/14
to appj...@googlegroups.com, Ingwie Phoenix, Nicolas Embleton

Good points Ashish, you are very welcome to the discussion,  I know you have previously made excellent technical contributions. Maybe you and ingwie can get connected on chat and discuss ideas. If you do that post the conclusions to the mailing list so everyone is informed.

Simon

--

Kevin Ingwersen

unread,
Feb 10, 2014, 3:19:30 PM2/10/14
to Ashish Negi, appj...@googlegroups.com, Nicolas Embleton
Hey Ashish! 


It is great to see somebody so much dedicated :)
Great to hear youre happy about the fact! :)


I think that  V8 is static library . Have you compiled it as dynamic library ?
Even then think about it like that if somebody created a library like you know printf or cout functions of c and c++ are also in library, then if multiple programs use them, they should be given different memory, otherwise output of one program would garble into others :)
Libraries are just code. Whichever process uses that code is given separate memory. So i think that if CEF and nodejs are separate process, they would be having different run-time v8 engines instances.

if we put the nodejs and CEF in one process such that they both share same v8 instance  then the thing would work out of box. ( original appjs was one process but had two v8's, so it required that js-bridge that mirrored everything in both lands ) it was like desksheel minus 'websockets'.
Nodejs compiles v8 staticaly into itself - and libcef compiles v8 first as a static lib, then combines several static libs (like WebCore) into one big library: libcef. So, I have one v8 engine in the nodejs binary, another in CEF itself - they dotn share the same via a static lbirary, rather have their own.

To avoid a blocking call to CefDoMessageLoop(), I have two options: Either kicking off the whole thing in a seperate thread, or integrating CefDoMessageLoopOnce() into nodejs’ call routine, so its always added to the stack of execution. That way, the call wouldnt be blocking, and I would have both processes running fine alongside. However, I will most probably go with the multi-threaded aproach. That way I can let CEF do everything on its own, independent from nodejs.

In the process of creating the CEF instance, I am creating a pointer to a Cef instance - this pointer, I can reuse to do JS integration, accessing various things, and the like. So, I have the global window object, and can register persistent objects and the like.

I have done a few tests, and used CEF’s v8 independent form Nodejs. I now just have to find a matching Nodejs version - and, to ensure that specific features are different. I currently have to mingle my way thru with two different v8’s - 3.17 and 3.21. The current nodejs 0.11-pre branch uses 3.22 - which means i have a matching version, and that is what I am compiling against now. According to the mailing list, 0.12 will be the next stable nodejs - and that following one might be 1.0, finally.

Its just the issue of finding two API-compatible v8 engines - in nodejs, and CEF. The problem isnt made easier with the fact that currently latest CEF has strong issues on OS X when running from the command line - it throws GL errors all over the place o_o. So I have to stick with the versionb efore that.

According to v8 docs, I can expect that I can use a roughly x.x3 difference. So, if a feature is to be removed in 3.30, for example, it is announced to be deprecated in 3.28 - then really marked deprecated in 3.29 and most likely removed in 3.30. Same counts for API changes.

So, although I would be able to „easily“ upgrade the API itself, i have to always have two matching v8 versions alongside x3.


Also, if we study CEF3 architecture, i think ( i am not 100% sure ) that like when we open new tabs in chrome, it creates new renderers and new v8 engines in the renderer. So v8 is not designed to be like one global thing, from which both CEF or nodejs could talk. 
Almost right. CEF uses shared memory and IPC to keep the memory shared. The renderer connects up to the existing memory, which is why I can reuse certain things i created int he main program. So, if i created a sub-object on window, i can reuse it in anything else, as it loads that object.


 I just have a random thought that if we can let CEF create its v8 and then we just expose the v8 - handler and where nodejs initiates v8, pass that handler there, may be furthur nodejs internal-objects would be created in the same v8.
That would mean I had to recompile nodejs, rewrite the whoel build step, but it woudl actually work. It just would mean that nodejs would loose a lot of its speed. from what I heared, its not very recommended to use v8 as a static library for nodejs :p. But, it would be one try if nothing would work any more - and actually write a CEF based app that simply started off nodejs within a browser window. Funny idea, but ultra hard to archieve, especially for osmeone as me, who is not „that“ far on low-level coding XD.

… to be hoenst, the only reason why I understand most C++ is, my knowledge about PHP … ;)

Ashish Negi

unread,
Feb 10, 2014, 11:45:44 PM2/10/14
to appj...@googlegroups.com, Ashish Negi, Nicolas Embleton, ingwi...@googlemail.com


On Tuesday, 11 February 2014 01:49:30 UTC+5:30, Kevin Ingwersen wrote:

> Its just the issue of finding two API-compatible v8 engines - in nodejs, and CEF. The problem isnt made easier with the fact that currently latest CEF has strong issues on OS X when running from the command line - it throws GL errors all over the place o_o. So I have to stick with the versionb efore that.


If we are finally going to have two v8 instances, why do we need to have compatible v8's ?
I think that we are going to pass javascript objects in-between, two different v8 will do. Its like server and client.

Are you planning something else ?
 

Kevin Ingwersen

unread,
Feb 11, 2014, 4:33:36 AM2/11/14
to appj...@googlegroups.com, Ashish Negi, Nicolas Embleton
Only if the two v8’s matches, then I can be sure that moving objects between them will work as expected - and that I can use the same api across the entire app. Thats important, for memory management and for API maintenance. o.o

Simon Horton

unread,
Feb 11, 2014, 4:53:07 AM2/11/14
to appj...@googlegroups.com, Nicolas Embleton, ashish negi

Quick note here: problem will be javascript is designed to be single threaded in its execution but with 2 v8 and two threads able to access same memory and objects you will get strange bugs and errors.

Very short reason: one command in JS may have many actual steps that are carried out in sequence, whoever in the middle of that sequence second v8 can be context switched in to the processor and execute on the same object / data. This is unexpected for the original programmer and undefined what will happen, e.g. if one v8 creates overwrites a text file with new text and another v8 reads the file maybe you get part of the text from first version of the file then context switch and you get the rest of the content from the second version of the file. Massively weird bug almost impossible to debug. This is why parallel programming is really tricky, this is one reason nodejs is way cool because event oriented means you can write complex applications but can ignore the hard parallel stuff.

Not sure that was well written or easy to understand but this area is tricky. Having a "web socket" interface between the two v8 (a pretend one with same properties but messages copied from one to the other instead of going over the network) smartly avoids all of these problems and makes solution easy to use and no weird bugs. This is Simons "conservative" approach.

Having said that direct access to jquery and the dom from nodejs is beautiful and may be worth doing any way (with a disclaimer ok this could be dodgy).

Another point though is I am coming with the traditional accepted approach to these problems so I don’t want to shut down the creativity, a fresh pair of eyes looking at the problem without preconceived ideas can come up with innovative solutions that are very smart. So play around and see if we can get something that is safe but cool.

Simon

Kevin Ingwersen

unread,
Feb 11, 2014, 5:10:43 AM2/11/14
to appj...@googlegroups.com, Nicolas Embleton, ashish negi
Hey Simon.

Someone recently asked on the v8 users mailing list, if v8 is threadsafe. Answer: Yep!

The way it does is using isolates. As the name suggests, it isolates the v8 process from anything else. Node 0.11 uses them, and CEF does too (v8 3.21 and 3.22). Therefore, they will run independent form each other. The only thing I will be doing, is giving both access to one, shared, object. That might become rather funky, but as it is only one tiny piece, it would be easier to find out what is going wrong :)

But I have also thought about a sockets aproach too - actually, I thought about tricking socket.io into beliving it sent data thru the network - but did instead send it into a memory buffer. The same buffer is read by the client (and thus cleared) and the client thinks its a regular socket message. It’d be a nother, interesting aproach. So, it’s going to be nteresting which aproach works. For now, I am just trying to implement various APIs o.o

Simon Horton

unread,
Feb 11, 2014, 7:00:09 AM2/11/14
to appj...@googlegroups.com, Nicolas Embleton, ashish negi

Great sounds good, give it a try. As I tried to say go for it I am not putting the brakes on just flagging up relavent issues as they can be helpful to think through. Sounds like a good solution. If that global object had a function to register an event listener for a named event and a function to trigger a named event then you basically have a socket like api (json object as parameter in).

Simon

Simon

Kevin Ingwersen

unread,
Feb 11, 2014, 4:16:02 PM2/11/14
to appj...@googlegroups.com, Nicolas Embleton, ashish negi
Oh thats alright! :)
Its good to keep me reminded to rethink things - i tend to be too fast at some time o.o

A bit ago, I discovered something that made me look quite baffled - look! http://www.cappuccino-project.org/learn/demos.html

Cappuccino is like Objective-C but for javascript. Not only is ObjC one of my favorite language syntaxes - but Cappu says it can make „desktop applications in the web“ - now, combine that with Deskshell, and a possible full functional file i/o and things alike? Then Cappu might be a great framework to use with Deskshell :)! Thought i’d share so me bits of what I am finding here.

Currently, I am trying to fix a bug that makes CEF crashw hen I close it here on my end. Once that is fixed, I can move on and implement more of the API. First stop is to implement CefSettings - the thing that offers all the various browser settings, like remote debugging and the like. I am looking forward to implement some api to give basic control of CEf, since that is really the only kind of thing we really want.

Thanks to FLTK, we might also get back the „chromeless“ windows - i saw a switch in fl_Window, that would allow me to turn off the window borders - so we can design cool things without the OS chrome around. Some peeps might like this. ^.^~

Oh well, so much going on ^^. Well, i am going to prepair a proper building experience, then I am going to upload my code to github, so everybody can have a taste of my current work, and for possible contributors.

Ashish, if you wish to be added to the Deskshell-Core github group, just give me your handle :3

Ashish Negi

unread,
Feb 11, 2014, 11:24:04 PM2/11/14
to appj...@googlegroups.com, Ashish Negi, Nicolas Embleton, ingwi...@googlemail.com


On Tuesday, 11 February 2014 15:03:36 UTC+5:30, Kevin Ingwersen wrote:
Only if the two v8’s matches, then I can be sure that moving objects between them will work as expected - and that I can use the same api across the entire app. Thats important, for memory management and for API maintenance. o.o


If you are going this way, this means that there would be only one memory that is being used by 2 V8's. This simply mean that there is one v8 which is being used by two different processes. Each process has its own v8-isolate. and You want one object of each isolate to be sharable. Is it what you are working on ?

The problem here would be that when nodejs is working in its isolate, if CEF wants nodejs-that-shared-object, it has to move in nodes-isolate. But only one can be present in the isolate at one time. It would not be allowed. Have you thought about this ?


Ashish Negi

unread,
Feb 11, 2014, 11:27:30 PM2/11/14
to appj...@googlegroups.com, Nicolas Embleton, ashish negi, ingwi...@googlemail.com


On Wednesday, 12 February 2014 02:46:02 UTC+5:30, Kevin Ingwersen wrote:

Ashish, if you wish to be added to the Deskshell-Core github group, just give me your handle :3

I have not done any work to deserve this as of now. Just trespassing .
But this is a beautiful project.

Kevin Ingwersen

unread,
Feb 12, 2014, 8:45:14 AM2/12/14
to Ashish Negi, appj...@googlegroups.com, Nicolas Embleton
Yes, youre pretty close to what  am about to be doing.

I sent an email to the v8-users mailing list, and got an interesting response: „You may create a Persistent<Object> in each engine, but let them both take a refference to the same C++ class“.

This would mean, that in nodejs, and cef, i create exactly one persistant, scope-independent, object - but it has a reference to the same C++ class, which can store values and the like and act as a bridge.

The goal is simply to mirror an object in CEF and nodejs at the same time. So, consider the following API:

var fltk_cef = require("fltk_cef");
var browser = new fltk_cef.Browser();
var MirrorObject = browser.mirror;
MirrorObject.myfunc = function(){ console.log("Meep"); };

Now, MirrorObject is a refference to an object, that also exists in CEF - under window.mirror. I choose the name mirror, just to showcase what i am basically doing.

Its very tricky - but if ithis works out, I am gonna be damn happy. :)

Ashish Negi

unread,
Feb 12, 2014, 1:00:39 PM2/12/14
to appj...@googlegroups.com, Ashish Negi, Nicolas Embleton, ingwi...@googlemail.com


On Wednesday, 12 February 2014 19:15:14 UTC+5:30, Kevin Ingwersen wrote:

var fltk_cef = require("fltk_cef");
var browser = new fltk_cef.Browser();
var MirrorObject = browser.mirror;
MirrorObject.myfunc = function(){ console.log("Meep"); };


so if we do mirrorObject.require = require.
and require would be available on cef side ?

this is great. :)
we would have to think about this multithreaded javascript approach though.

Kevin Ingwersen

unread,
Feb 12, 2014, 3:06:33 PM2/12/14
to Ashish Negi, appj...@googlegroups.com, Nicolas Embleton

Am Mi. Feb. 12 2014 19:00:39 schrieb Ashish Negi:
> we would have to think about this multithreaded javascript approach though.

Well, multiple threads can access the same data. I just have to implement things like Mutexes. In v8, that is a v8::Locker :)

Ashish Negi

unread,
Feb 12, 2014, 11:10:34 PM2/12/14
to appj...@googlegroups.com, Ashish Negi, Nicolas Embleton, ingwi...@googlemail.com


On Thursday, 13 February 2014 01:36:33 UTC+5:30, Kevin Ingwersen wrote:

Well, multiple threads can access the same data. I just have to implement things like Mutexes. In v8, that is a v8::Locker :)

lets say we have mirror.nodejs

now this nodejs variable would also be  available on other side. if and only if they use it through the mirror only then, the locker would work. if they access it through their own, they can change the variables.

this means that if we have shared any object, then it should be used only through that shared object in both sides.
like we can do
mirror.require = require
require = mirror.require

now anybody who would be using require in nodejs world, would be using mirror.require and cef they have to use mirror.require.
then the locker thing would work. :)

great.

Kevin Ingwersen

unread,
Feb 13, 2014, 3:00:39 AM2/13/14
to appj...@googlegroups.com, Ashish Negi, Nicolas Embleton
Now you got the whole plan! :D
That is exactly what I am to implement. ^^

Kevin Ingwersen

unread,
Feb 13, 2014, 4:45:34 PM2/13/14
to appj...@googlegroups.com, Ashish Negi, Nicolas Embleton
For anybody who is curios, I have published my current work. Thing is, I am kind of burnt out X3. Not been able to really do a lot of coding…my mind went kind of dizzy, as I was getting back into doing simpler things. For one, I am trying to extend a build tool called … „build“. I am adding functions and the like to check for headers, tools, and the like. so it can be like a minimal CMake. It is so cool, that it got a minimal scripting language too - which is very, very clean. Like, we can do things like this:


A build config:
# This is how a target looks like.
target "test" {
# We build an EXEcutable.
rule "exe";
# We define inputs...
input [files("*.cpp")];
}

# And this is an action.
action "all“ {
# We tell the tool that we want to build the target Test.
build: "Test";
}


Output:
$ build
--- build test ---
[ 0%] deps: functions.o
[ 33%] cpp: functions.o
[ 33%] deps: main.o
[ 66%] cpp: main.o
[ 66%] deps: test
[100%] exe: test


So, I am planning to utilize this more often now, and it will also run the overall building process of deskshell. Its being made Windows compatible atm…Well - that is what I am doing when my head simply cant compete X3.

So, feel free to roam around in the soruce code.
The code can be found here!: https://github.com/Deskshell-Core/node-fltk_cef

Kind regards, Ingwie

Paco Zarate

unread,
Feb 13, 2014, 5:49:19 PM2/13/14
to appj...@googlegroups.com, Ashish Negi, Nicolas Embleton
Hey guys
this thread is awesome, thanks for sharing it publicly!

Ing: Your work is really interesting, just don't burn up in the process jajaja



Kind regards!

On Thu, Feb 13, 2014 at 2:45 PM, Kevin Ingwersen <ingwi...@googlemail.com> wrote:
For anybody who is curios, I have published my current work. Thing is, I am kind of burnt out X3. Not been able to really do a lot of coding...my mind went kind of dizzy, as I was getting back into doing simpler things. For one, I am trying to extend a build tool called ... "build". I am adding functions and the like to check for headers, tools, and the like. so it can be like a minimal CMake. It is so cool, that it got a minimal scripting language too - which is very, very clean. Like, we can do things like this:



A build config:
# This is how a target looks like.
target "test" {
        # We build an EXEcutable.
        rule "exe";
        # We define inputs...
        input [files("*.cpp")];
}

# And this is an action.
action "all" {
        # We tell the tool that we want to build the target Test.
        build: "Test";
}


Output:
$ build
 --- build test ---
[  0%]       deps: functions.o
[ 33%]        cpp: functions.o
[ 33%]       deps: main.o
[ 66%]        cpp: main.o
[ 66%]       deps: test
[100%]        exe: test


So, I am planning to utilize this more often now, and it will also run the overall building process of deskshell. Its being made Windows compatible atm...Well - that is what I am doing when my head simply cant compete X3.

Kevin Ingwersen

unread,
Feb 13, 2014, 7:07:09 PM2/13/14
to appj...@googlegroups.com, Ashish Negi, Nicolas Embleton
Not burning up? Thats why I am taking a little break, going down to easier things, so I can calm a bit. :3 And, that is why i shared the code I was working with - so people could go and lurk it o.o
But once I have gathered new energy, I will work on it again. But for ow, I am just…well, there is a yugioh turnament on saturday, cinema on sunday, a pucka-punch-packed Monday and a „im so gonna skip school“-thuesday XD. So for a bit, I really just need to live life for a bit ^.^;