Announcement: Status, development, concept, ideas, etc.

857 views
Skip to first unread message

Kevin Ingwersen

unread,
Dec 22, 2013, 8:29:10 PM12/22/13
to appj...@googlegroups.com
Hey everyone!

Phew, it's been a while, isn't it? :§
Well incase you wonder how I am...well, I certainly feel like the wobble-bass of Dubstep. XD From one side to another, forth and back...all around and over and upside down.


Why?
Because, development. In LESS THAN A MONTH i taught myself C. I then added some basic ObjC coding - and extended that all with my PHP knowledge and made it C++ - and finally Obj-C++. So things like

[instance Operation:@"default" withData:classInstance->getInfo()];

is not even unnormal, rather trivial. :) Kinda proud of myself right here!

But for what all the learning? Because Deskshell.


Evolution of Deskshell.
I have taken over appJS, and have looked into it and decided: I can not redo it. The reason is simple: the building just won't work! Making it CEF3 ready would require a whole re-write - so I just declared it as obselete and done. But, it's spirit is not gonna stay as it is. In fact, I have planned a relaunch as "AppJS Remix". I am a DJ - so for me, a remix is taking a song, and making something new of it. That is what I am doing with appjs here.
The Deskshell I am creating now is a structure based off nodejs, the thing we all know and - one or another ;) - may love. So, let me show you the structure first:

nodeJS -> deskshell-core -> |
-> nodeJS \
-> ph7 | -> libcef -> WebView
-> html5 /

I am not mad at you if you are like "WTH is that?!" - let me explain.

The new Deskshell version is based off a nodejs binary, that has a modified "entrence point" - that part, that is run when you call a binary. But nodejs' entrence point is actually a pure javascript file, and called by main(). The modified entrence script will read in the .desk file and initialize a module: Deskshell-core. The settings/configuration stored in .desk are then pushed to the "core" module. That module now decides on hwot o proceed:

- If the user specified to use a php script to bootstrap the resutlting app, lets use ph7.
- If the user wants to use nodejs, use nodejs in a new process and let the current just die away.
- Otherwise, create a webbrowser and display the content of the htdocs folder. That however is CGI enabled as well.

So, you can either bootstrap a programm (setting up custom listeners, functions, prepairing files, etc) or use Deskshell as a frontent for a plain HTML5 app. This all happens thru Deskshell-core, which is capable of calling in another engine, or relaunch itself. As you see, the plain binary will only really prepair for a new process. That said - you get two scripting languages in one. ph7 is a "fake" php, basically an embedable php for smaller projects based off the 5.3 definitions (no traits, no short-hand syntax for arrays/objects). However, its extendable thru a C-API. I have created a C++ wrapper (ph7++) that will hopefuly soonb e released by the dev team, once I helped them to fix the socket connections (socket_read/write/connect/...). I also ported some actual PHP extensions - currently only curl - to this engine. So in that term, you'd get a full PHP if you want to bootstrap your app using PHP, or even use it within the pages of your project that are subject to be displayed int he browser. .php files will be treatened as such - whilst .njs files will be processed by nodejs, giving you the ability to use nodejs templating engines for a Deskshell app.

The structure is ment to make it possible for javascript and PHP users to create a bootstrap if they want to. Both have their advantages, and disadvantages. I am a very engaged PHP developer, but I can do little JS. And drag0n, about which I will talk later, is a pure PHP project - so it will barely use any javascript at all.

Scripting the application launch also means that you can add custom features ont he go: Like you can implement a cgi for perl, python - or eve lua. Or you can check for updates. Oh, yes, we even have a gui.


The "new" GUI concept
I have been searching and doing heavy research on using a GUI framework, since CEF - chromium embedded framework - requires you to prepair a GUI for it, as it itself will only draw a bitmask into the window, and nothing else. That is also why their loading icons and such are png files - CEf displays a "picture". Of course, users with handycaps can use it too because its of course more than just a picture. But in all generall, CEF only really draws a picture, and according to the code that calls the library, updates the code.

My solution here was to use wxWidgets. Mainly because:
wxNode: https://github.com/joeferner/wxNode
and
wxWebViewChromium: https://github.com/steve-lamerton/wxWebViewChromium

As you see: The first makes wxWidgets available to nodejs - and the second acts as a simple wrapper for wxWidgets. With that in mind, we can now use a CEF instance according to the wx coding style! That said: We only need to compile one library (wxwidgets) with only the basic requirements in order to kick off a browser - with menu and whatever. And yes, a transparent window is possible too, from what I have seen in wxWidgets! So, you can still creare cromeless windows liek you did in AppJS.
Oh yeah, AppJS... ^^


AppJS Remix
Now that I told you how I am making the GUI, and how I am going to extend the GUI part to its underneath scripting engines, I can tell you that the extension - that will only be seen internally - will be called "appjs-remix". That means, that the combination of wxWebviewChromium and ndoejs actually represents the spirit of the old yet obselete appjs! I have thought - but not concepted - about re-implementing some of the old API in order to support some of the old features. However - once you are done, the actual event listeners you used to use are invaild. The scope then moves into a seperate process - with rendering and v8 engine - and I can not really port an IPC kind of communication - i am not skilled enough :I. But you'd at least get some of the old API back, that'd help you kick off a basic app again! ^^


Schedule?
First, chrismas. Then, I have to finish a PHP based intranet project for a company that I have worked for a while ago. They want to give me a job later, and help me a bit with my financials, it seems :p. Well, I have to make their project first, as I have been delaying for a while now and promised to do it in my winterbreak - which is now. After that, I will start pulling together a nodejs binary with the new libraries and programms integrated - basically, integrating the modules from above into the binary and seeing how they work, against which version, and how they work across plattforms. After that, I should have a nice pile with appjs-remix, ph7-node, wxnode and a base for deskshell-core. A nice pile to play with... then its just a matter of time to make Deskshell-core the "default" entry point. Once that is done, I actually oculd say that the binary itself is done. After it has been confirmed to worka cross platforms and that already created deskshell apps are still compatible, I can start to extend it further.
To make you think about something interesting:

- libcef is ~40MB big.
- ph7 - with my additions - is maybe 1MB
- wxwidgets full set of libraries on OS X are 42MB. Consider that we will only use maybe 1/10 of that :)

So, the future Deskshell runtime will be about 50-60 MB. But I am not done here. After I have this done, I can go and remove unneeded features out of CEF and start creating a new libcef - libcef-deskshell. That, if we dream a little, may only be 20MB. Calculating again, gives us a 40MB build. Now running UPX over it, by loosing a bit of speed in launching - we get down to 20MB. Now tell me if that isn't a small but powerful runtime?? =D


drag0n
Drag0n, yes, it still exists. Just recently I published a WIP repo of d0p - a type of archiving that allows adding a header description in YAML format to the archive. Then running 'd0p explain archive.d0p' can output useful information - such as a description, website, or whatever. In the sense of drag0n, it saves adding an informative folder with all the info files and alike. Instead, we can just query each header in the folder and store it in memory for a while and examine it, use it, or whatever. We can even use this method to read from a HTTP stream! So we can litterally extract the header out of the internet. Then we can display that in the drag0n interface and let the user confirm the installation - and download the rest, and stream the archive down to disk.
But drag0n has another, more important, factor. Drag0n consists of many different parts:

- "appstore" like interface to install and manage packages on a local system.
- Distribution platform for developers.
- Build tool for developers.

A subproject dubbed "d0.build" is just a little script that relaunches your shell and lets you use a portable toolchain, for building whatever application you may want to build! That said: its powerful. It can build packages from their source code right from a GUI and install them.
BUT THAT IS NOT ALL.
It is an appstore like thing. So, Deskshell apps can be distributed thru it. and since it is a deskshell app itself, you don't need to supply the whole runtime! Instead, it just "links" against the runtime it has within itself. So your previously 30MB big Deskshell app, due to inclusion of the runtime, may turn into a 5MB html5 based app. That does not just make downloading, but also installing a hell load faster!
Drag0n however does not just distribute Deskshell apps - but also GNUstep and gives a re-birth to a long-forgotten installer.


GNUstep and Fink
Eversince brew and MacPorts have launched, Fink has been pushed into the back of the scene, and I see a lot of people that use brew. But to be very honest, the reason why so rare people are using an installer is: No GUI. drag0n therefore provides a GUI and lets developers distribute apps for platforms that nobody has probably even seen xD. GNUstep is a platform to develop Objective-C/++ based applications across Mac, Windows and Linux. So drag0n installs already two platforms: Deskshell and GNUstep. And unterneath itself, it will use Fink with its very nice output format, and parses that in order to make manythings more understandable to the software itself. So in theory, alongside the drag0n structured repos, you can use an existing APT repo and just distribute packages thru that into drag0n. Why this works? Because Fink is APT for Mac. But drag0n will use the according package managers that are right for the current platform. I saw something called "win-get" for Windows, that is an APT clone for Windows. On anything Linux, I can just use the current package manager - zypper, apt, whatever.


Conclusion.
Due to my many projects within Fink, GNUstep and Deskshell, I am creating an SDK that makes creating HTML/JS/CSS based apps easier, but also adds support for server-side scripting with PHP and nodejs itself - making it a fully round-up solution to quickly create applications. And since you have actual wxWidgets bindings in there, you could even tell Deskshell to be fully pale, and use wxWidgets to create your app! That is, if you want to... ^^
But I am not done here, not closely. Due to my find-out about nodejs for ios, and the current CEF goal to port to Android and iOS, we might see Deskshell on mobile platforms too! And hopefuly I can find a way to devleop a port for win8 metro UI.


Getting involved.
The appjs project may be deprecated/obselete/whatever - but it is the perfect place to talk to me. It is where everything started. Alternatively, feel free to browse my site http://ingwie.me (still under construction) and contact me directly - or just mention Simon and ask him for his email. Simon knows - i think ^^; - everything about the plans that I have made - and ing eneral he is a very nice person to talk to and make concepts with. ^^



Kind regards, have nice holidays and all the like!
- Ingwie Phoenix, alias Kevin Ingwersen

PS: In some time soon, Ill make a video and hold a short speech about a lot more things than what I mentioned here. Yes...I ish bussy burdeh @.@ ... err, I mean, phoenix. o.o
Oh and if you find typos, you may keep them =D

My Other projects (skip if not curios)
- GNUstep for Mac OS: Create an alternative toolchain to Apple's Xcode and make OS X development easier - but also cross-platform development. My build will feature a static libobjc2 build, in order to make sure you dont depend on anything out of my build.
- Experimenting with wxWidgets and CEF: Making sure that the appjs-remix and wxWebViewChromium things work just as expected.
- Create the full drag0n structure and interface: I have a PHP layout for drag0n (d0) that will make it possible to use it anywhere where you want. And another idea is to create a single, little binary. So even if you dont have PHP or anything, but need a short bootstrap thing to download libraries or dependencies, that will help you. Just compile d0.cpp and d0p.cpp into a d0 binary, and you can issue commands such as "d0 install libicu" to get the latest version of icu into your current working path. As it will just be a bootstrap helper, it will not write any information files or alike - it will read everything off thru the internet.
- Create a Deskshell "compiler" for the current version to make proper Mac OS X bundles.
- Finish other stuff such as WingStyle, my own site, and my other stuff you can see on my small site.

Nicolas Embleton

unread,
Dec 23, 2013, 5:47:29 PM12/23/13
to appj...@googlegroups.com, ingwi...@googlemail.com
That sounds like a lot of awesome hard work... Congrats and kudos for the learning curve :)

I will be (passively, sorry) looking forward to seeing what comes out soon :)

Wish you luck.

Kevin Ingwersen

unread,
Dec 26, 2013, 6:43:54 PM12/26/13
to Nicolas Embleton, appj...@googlegroups.com
@Nicolas: Hehe, thank you :)

I have some updates! ^_^

So, first I found this ultra-brilliant idea of a developer. This however is only useful if you have some C++ base: https://github.com/rvagg/node-addon-examples
It is, litterally, a point-blank example page for nodejs addons. that is how i teach myself the best o.o

I tried out a lot of different GUI tools - wxWidgets, CEGUI, and even Qt xD - but then came across a few issues.

- The wxNode plugin does not compile with wxWidgets 3.0. The suggested 2.9.3 release does not build on OS X. So, that one falls off.
- wxWebViewChromium wont compile, because it was PLAINLY written for Windows. So unless I give it a whole rewrite, its useless. x.x
- CEGUI has so many dependencies, that I now have a full bloat of Boost, lua, python, and whatnot. x.x… no thanks, too many dependencies.

So I searche dfurther…and it was my DJing that brought me a solution. To stream the output from my mixer as MP3 to my server, i use a tool called „Butt“ (broadcast using this tool). And when I read on its sourceforge page again, it revelaed that it used „FLTK“. Another browser I once saw, called Dillo, does too! But it looked weird when I tried… So I went, and got the actual, full, source: FLTK 1.3.

Playing around with it actually made me belive one thing: The developers that used this library obviously gave a s*** on the actual GUI. FLTK allows yout of ully colorize it, replace buttons with actual images - so on and so forth. I am currently even poking the developers to add controll over the titlebar as well. Yes, I talked to the devs. And I want to tell you why ;)

As I digged further into it, I created a little, simple test app, and also examined the examples. Quickly I learned that the classes it uses - its c++ - are extendable, and allow you to override anything you’d like to have overriden. But there is one special function that is overkill. You see, to spawn the base window, it uses the native toolchain (HWND, NSWindow, GtkWidget, …). But due to its brilliant way of building, the whole controll falls under Fl_Window, Fl_Button, etc. Effectively making it possible to host the same API on each plattform. This also counts for menubars. With an actual #ifdef, I can either use an in-window menu bar for windows and linux, or the actual OS X toolbar:

#ifdef __APPLE__
Fl_Menu men = fl_sys_menu(…);
#else
Fl_Menu men = fl_menu(…);
#endif

See what i mean? =)

But, the overkill function - or rather „method“ - is fl_xid(Fl_Window *). What it does is, it spills out the actual, native, window object! So that way, I can use this framework with CEF. Why? Because Cef::createBrowser() requires a native window object - NSWindow, HWND, GtkWindow….etc. So I can just pass fl_xid(myWindow) as an actual parameter, and the problem is fully solved - we create our window over FLTK, and let CEF take the inside with a native API.

- That problem is solved - the GUI i am using is FLTK :)!

Next thing? Nodejs bindings.
I want to create basic bindings for FLTK. The great thing is, that the actual library is…

Ingwie@ingwies-air ~/Downloads/fltk-1.3.2/lib $ du -h *.a
1,6M libfltk.a
32K libfltk_forms.a
156K libfltk_gl.a
72K libfltk_images.a
228K libfltk_png.a

tiny. And since we are already using the library for CEF - we can reuse the implied symbols for some nodejs binding. So one more nodejs module I will actually make is fltk-basic. So we can spawn small alerts and stuff - in case something goes horribly wrong. The same bindings will be available within the php scope too - so you can spawn windows with each and every method - using alert() or fltk.dialog() :)

- Next issue tackled. a nodejs mini-gui will be made!

So thats for now from my side, Im going back to developing. This is just getting started man! ^-^

Kind regards, Ingwie

PS: If you are doing small apps in C++ and want them cross plattform, I’d really hint you towards FLTK, its simple and fast. and ultra tiny. ^^

Simon Horton

unread,
Dec 26, 2013, 8:24:37 PM12/26/13
to appj...@googlegroups.com, Nicolas Embleton

Wow great Kevin. Really interesting to see the thought process and how the ideas and solutions are getting better and better. Interesting Cef takes a window object to be added to, I had not thought about it exactly like that before. Excited to see the end result of these ideas :-)

Simon

--
You received this message because you are subscribed to the Google Groups "appjs-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to appjs-dev+...@googlegroups.com.
To post to this group, send email to appj...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Kevin Ingwersen

unread,
Dec 26, 2013, 8:27:25 PM12/26/13
to appj...@googlegroups.com, Nicolas Embleton
Well I’ll continue to post about my progress, because I think its motivating! ^_^
Currently I am making hello-world apps with FLTK to practice it. Tomorrow, I will use fl_xid and actually see how CEF behaves underneath it. FLTK creates a subclass to NSWindow called FLWindow - notice, no underscore. But it should work still. Once i have a test with a basic sample done, I’ll put online the basic example so you can try to compile it on your platform - FLTK is fully cross-platform. I was also just taught how to speed compile it - so i need to colect some config.h’s, the configurations generated from ./configure. I need one from Windows and some linux’es…but ill do that stuff later. :)

Kevin Ingwersen

unread,
Dec 27, 2013, 1:59:53 AM12/27/13
to appj...@googlegroups.com, Nicolas Embleton
Fingers crossed! I just hand-build my simplecef. Now to substract the platform-specific code and replace it one by one with FLTK! :)
Am 27.12.2013 um 02:24 schrieb Simon Horton <siho...@gmail.com>:

Kevin Ingwersen

unread,
Dec 27, 2013, 3:13:12 AM12/27/13
to Nicolas Embleton, appj...@googlegroups.com
Ok… its 9am, time to sleep.
Meanwhile Ill wait for the devs to mail me a fix for an override I just cant get right. Unless that, everything else compiled just fine. But the file where the errors are is actually the app bootstrapper. So I can’t get it off x).
Anyway, work for today~

Simon Horton

unread,
Dec 27, 2013, 3:38:36 AM12/27/13
to appj...@googlegroups.com, Nicolas Embleton

Wow that was fast, assumed it would be more challenging!

Kevin Ingwersen

unread,
Dec 27, 2013, 4:32:28 AM12/27/13
to appj...@googlegroups.com, Nicolas Embleton
Very flickery, and funnily upside down, I got CEF to render within an FLTK window :)
Now to fix the damn flicker… @.@ it felt like eye cancer...

Kevin Ingwersen

unread,
Jan 1, 2014, 7:43:56 PM1/1/14
to appj...@googlegroups.com, Nicolas Embleton
That should explain itself. :)

Kevin Ingwersen

unread,
Jan 1, 2014, 10:59:09 PM1/1/14
to appj...@googlegroups.com, Nicolas Embleton
A follow up to my last puush link: https://github.com/IngwiePhoenix/fltk-cef

Kevin Ingwersen

unread,
Jan 3, 2014, 5:15:53 PM1/3/14
to appj...@googlegroups.com, Nicolas Embleton
While letting the developers from FLTK work on some fixes, I have taken the time and looked into another requirement.

When using CEF, the actual libcef.dylib/a/dll only exports a C-API. To use it thru c++, one needs to build the libcef_dll_wrapper, a static library that converts the C api into C++.

Fortunately, the distribution lacks two features:
- Building a proper .framework for Mac OS without xcode, and cefclient
- Building just the static library.

I got around the first problem easily, and made a shell script - that simple it is.
The second was a bit harder to get around… In fact, I had to studdy GYP some.

In order to just build the wrapper library, I read up the GYP file (cefclient.gyp) and determined what I needed, then went and tried out to compile by hand. After several tries, I figured that I needed two macros, two include paths, and a perma-include. But then, I compiled it pretty nicely, and even reproduced the static library build thru xcode! woohoo~ :3 But, how to build without xCode or gyp? … Ninja.

In the past, I had started a set of php-based development tools: pcc and autoNinja. PCC compiles a php source file into the ph7 engine and makes it executable. autoNinja takes a php script, with pre-exported functions, and allows it to define some build rules and the like. That way, I can speedfuly and automatically build ninja buildfiles. Now, I have created a 10-liner of PHP code, and made a full build file for Ninja: http://puu.sh/68syn.ninja

With further adjustion, the autoNinja file i used (ninja.an) will soon detect the OS and set the declarations right. Currently, it’ll only build on OS X, because clang defines OS_MACOSX. You can manually change that to OS_WIN32 or OS_LINUX. But I didn’t try out the other platforms yet ovo’

So, here is a current list of dependencies, and how far I got with them.

- FLTK: Working.
- PH7: Working.
- CEF: Working.
- libcef_dll_wrapper: Working, buildable.
- FLTK-CEF bridge: Half-done
- deskshell-core: Not started.
- node-fltk: Not started.
- node-ph7: not started.
- node-simple-gui: Not started

To explain the node extensions:
node-fltk will offer a soft binding to FLTK. We already use it in fltk-cef, so we can re-use the library if we’re already compiling almost the whole thing into the binary.
node-ph7 will launch ph7 and give it the actual script, register exports…so on and so forth.
node-simple-gui will offer dialogs, file picker and stuff liek that thru FLTK.

So? :3 I am happy!

sihorton

unread,
Jan 7, 2014, 5:15:06 AM1/7/14
to appj...@googlegroups.com, Nicolas Embleton, ingwi...@googlemail.com
Very fun to see how far you have come and everything that has been going on. Looking forward to seeing the new possibilities in deskshell and am itching to write some applications that run inside these programs!

/Simon
 

Kevin Ingwersen

unread,
Jan 13, 2014, 5:41:35 AM1/13/14
to sihorton, appj...@googlegroups.com, Nicolas Embleton
Amazing news! CEF and FLTK now behave perfectly together!! =D
The person responsible for the Mac port helped me fix errors - and now it works amazingly well.
To make the platform dependent changes, I am using an aproach that lets me include one heater, which includes the one needed for the platform. So I have mac.h and mac.mm to implement 2 functions to handle small additions. Same will be done for win.h/cpp and linux.h/.cpp.
Now i am very happy, waiting for the changes we aproached to be merged into the svn repo, so everyone can use them.
So, in other news:

- FLTK and CEF: Working!

Time to modify and clean the code and test it on other platforms. :))) Once that works, it’s time for node-fltk.
ph7 is currently also learning a new function to import scripted functions into C scope - so we can do crazy stuff with callbacks from the bootstrap code.
After that works, node-ph7 should be no biggie to be made.
That’s my plan for now :)

Simon Horton

unread,
Jan 13, 2014, 5:51:30 AM1/13/14
to Ingwie Phoenix, Nicolas Embleton, appj...@googlegroups.com

Wow well done again Ingwie! Amazed at the pace of these developments - keep up the great work.

Simon

Kevin Ingwersen

unread,
Jan 16, 2014, 5:38:22 AM1/16/14
to sihorton, Nicolas Embleton, appj...@googlegroups.com
Good news! After my success with FLTK+CEF, I went and created a new github organization: http://github.com/Deskshell-Core
Why the caps?.. Looked friendly-er ^^;

Anyway. I uploaded my first nodejs extension too! :)
And this is one more part of the new Deskshell-core project, which will use CEF and other things that I explained earlier. Well, once this works, I have one more part done. In the next step, I will be wrapping the FLTK+CEF stuff into a suitable little piece of nodejs code. It is only really ment to be fired off as a window. It will have however a basic API for configurating it with things such as custom schemes and so on and so forth. Thats neccessary to allow CEF configuration thru .desk files.
Then, I am going to create a rather bulky wrapper around fltk itself: FL(N)TK, Fast Light (NodeJS) ToolKit. With that, we get things like alerts, native fileschooser and even more.

That done, I have:
- An in-process PHP engine for nodejs.
- A usable CEF from nodejs, implemented via FLTK.
- A GUI toolkit to display things like alerts, if a programm cant do its thing for example.
- And for the best: The three very basics for deskshell-core!

Next I will have to deal with combining it all and creating the core functions. So that we can do things like that:

return require(„deskshell-core“).runWithObject( require(/*.desk file path here, probably argv[1].*/) );

This one-liner will kick off the actual core, which in itself will require fltk, cef and ph7, trigger the creation of a custom cef scheme, add a nice callback to read from the files and so on and so forth.
The custom cef scheme is especially useful as it may switch between reading flat files from disk - OR utilize simon’s vfs and read from a blob! :)

So, I am very, very close. Once I have the modules done, I will be able to invite you scripters. Because then I need a break, but your help! As deskshell-core is nothing but a nodejs extension - flat javascript - I think you can help me here creating the core, that will set everything up.

Alright…back to hacking. I just thought i’d share my current process.

Oh yeah, if you are a scripter or low level dev - or know some - you can pass them my email and invite them to Deskshell. Because there are several things I still need:

- FLTK needs to be gyp-yfied.
- Deskshell-Core needs an API concept.
- The in-app API (from within cgi scripts and such) needs to be designed too.
- Anything that will be helpful to enhance the usage of the whole things.

Happy hacking, and see you all on my next update! :)

Kind regards, Ingwie

Kevin Ingwersen

unread,
Jan 16, 2014, 1:55:07 PM1/16/14
to sihorton, Nicolas Embleton, appj...@googlegroups.com
One more thing to report: I now know where and how to patch the nodejs source tree to bring native modules into the binary, and use it from there - including JS files! Here is a small glimpse from my terminal, loading ph7 from inside the same nodejs executable that the script itself is inside - a fresh build as well.

Ingwie@Ingwies-Air ~/Downloads/node-v0.10.24 $ ./out/Release/node
--> We're using the inoffical process.binding method to obtain ph7 - from THIS executable!
--> No sh*t, this is all inside the same binary. This is going straight from the node binary... :)
> ph7: { create: [Function] }
--> Let's create a ph7_vm and have a peek at what we get...
> pVM: { include_path: [],
  '$argv': [],
  '$SGLOBALS': [],
  '$GLOBALS': [],
  '$_HEADERS': [],
  '$_SESSION': [],
  '$_COOKIE': [],
  '$_POST': [],
  '$_GET': [],
  '$_ENV': [],
  '$_SERVER': [],
  config: [Function],
  compile: [Function],
  compileFile: [Function] }

I am so excited! ^_^
Am Mo. Jan. 13 2014 11:51:30 schrieb Simon Horton:

Kevin Ingwersen

unread,
Jan 18, 2014, 6:45:21 AM1/18/14
to sihorton, Nicolas Embleton, appj...@googlegroups.com
And here it comes - release numero uno.
Last night, I created a working nodejs binding to ph7 - so now we can run php code from within nodejs! that way, we’re able to run php apps inside deskshell without an external binary.
The (currently rc1) version can be fond on Deskshell-Core: http://github.com/Deskshell-Core/node-ph7
.node binaries for the varous platforms are welcomed ;)
Currently, I am looking for a way to make npm install the module, without trying to build it, since not every OS has pre-installed compilers, like Windows. So if you now some npm stuff with native addons, please let me know.

The module also has been prepaired for an in-binary compilation to nodejs. I have released a tutorial on the nodejs mailing list contianing a way to staticaly compile javascript and addon files into the binary.

Next is the FLTK binding! :)

Kind regards, Ingwie

Am Mo. Jan. 13 2014 11:51:30 schrieb Simon Horton:

Kevin Ingwersen

unread,
Jan 18, 2014, 6:51:02 AM1/18/14
to sihorton, Nicolas Embleton, appj...@googlegroups.com
Now available on npm too: https://npmjs.org/package/ph7
Am Mo. Jan. 13 2014 11:51:30 schrieb Simon Horton:

sihorton

unread,
Jan 18, 2014, 7:55:15 AM1/18/14
to appj...@googlegroups.com, sihorton, Nicolas Embleton, ingwi...@googlemail.com
This is really cool, I think the ph7 module can become very popular, will be interesting to see what people make of it and how it is used!

For a binary package install then you can look at the sqlite3 module (https://npmjs.org/package/sqlite3), I have noticed that now it just drops a binary into the folder on windows, before it used to compile the module instead. I don't think there is any clever magic involved in achieving this. It is possible to run an install script, this can see what the os and version / type are and then download / extract a binary instead of compiling the module.

/Simon

Kevin Ingwersen

unread,
Jan 18, 2014, 11:27:16 AM1/18/14
to sihorton, appj...@googlegroups.com, Nicolas Embleton
I found a way :) I am going to use npm inside npm! To be clear: I can use the package.json’s „scripts“ section to fire off a „configure“ script. So that way, I can do custom stuff in js. Not just that, but I can also add urls to .tar files as a devDependency. Tha tway, if somebody wants to build the package, they run „npm install“ INSIDE the package. Then, the DevDependencies are installed - which in this case would be the bindings.gyp file, which allows building :).
Using npm, I can also download binaries via npm. I looked into how appjs did it, and it was quite easy. There was more than one appjs package:
- appjs-darwin
- appjs-win32
- appjs-linux-x86
- and more.
So I can do the same:
- ph7-win32
- ph7-darwin
- …
That way, I can download the package at need. Speaking of which; to use npm, it has been added as a dependency to the file. So i get the small bit of npm that I can call from node:

var npm = require(„npm“);
npm.load(); // Let it initialize itself
npm.install(„ph7-darwin“);

Voila. :)

I have to work on the architecture some more, but I’ll be able to drop that on. But that way, people can do the following:

$ npm install ph7
$ cd node_modules/ph7
$ npm install
$ npde-gyp rebuild

The last two commands install the dev dependencies - and the last builds the module! So, the resulting build/Release/ph7.node can be published.

I will need various builds of the project anyway. But just look into appjs/package.json and see how it calls a script called preinstall.js . Open that, and lookw hat Milani did. Well he was depending on the npm binary to be in PATH, but my aproach works without that too. Thus, I’d rather use actual js than child_process.exec for that task :)

Kevin Ingwersen

unread,
Jan 31, 2014, 3:48:39 PM1/31/14
to sihorton, appj...@googlegroups.com, Nicolas Embleton
Okey! Some new news from what I have here…and I have a near-alpha O.O!

So, the process is now going in a way that is different from before. I am patching the node.gyp and node_natives.h file in order to enable native modules to be compiled into the nodejs binary - same with javascript modules. That way, I have implemented ph7, native addon, and optimist (with its dependencies wordwrap and minimist) into the nodejs binary…but I am not done. So, here is a list of the status:

- FLTK+CEF PoC: It works. I can launch CEF thru FLTK, I just need to add minimal, platform specific, code to enable multiple windows and such…its really small. :o
- FLTK+CEF for NodeJS: Bindings in process. I am currently having some issues with GYP again x.x; And, well, with splitting the different calls. Its not easy, but its the biggest challange. But once this monster is done, then we have the browser we want. :)
- ph7: Its buggy currently, and I have a lot of redundant code. I have to work thru it, and fix it. But the base by itself actually works - but not all the things get converted correctly into ph7_values. o-o
- Deskshell-Core: The core code is already half-done. This is the code, that runs instead of nodejs’ actual main() function. It can differenciate on the different file extensions, and such. So, its going somewhere.
- Deskshell: The public API is not done - yet.
- ttvfs: This is a new piece of C++ code i found. It enables AMAZING! features. I’ll explain that in a bit.

So, yeah, ttvfs… ^^

A while ago, I was looking for some really exotic kinds of software; virtual file systems, reading RAM as a block device (to interpret it as FAT32) and things like that. Then, very randomly, I came across this small project called ttvfs: https://github.com/fgenesis/ttvfs
This VFS implementation does not just offer a cross-platform way to manipulate and handle files…no, it offers a special way of handling internal buffers. In fact, I can turn a char buffer into memory file. Here is a short bit of code to demonstrate what I ment:

// Now, lets create a memfile.
VFSFileMem* memZD = new VFSFileMem("data.zip", ZipData, ZipData_length, VFSFileMem::REUSE);

vfs.GetDirRoot()->add(memZD, true, VFSDir::NONE);

// Mount the memory-virtual data.zip as a folder called data.
vfs.AddArchive("data.zip", false, "data");

After the .AddArchive call, we now can access the contents of data.zip like it was a folder - AND data.zip is actually a char buffer residing in memory. That is cool, is it not? Okay, maybe you didnt understand what it does yet… So, let me word this differently.

Imagine you have a Deskshell app, but want to bundle the source code into the binary, to make extraction/modification harder. You can put the entire source into a zip file, run that zip file thru a tool I created today and turn it into a C source file - which is just a char array - one, that you can then mount using the VFS and access your files again, just like they were local.

But how is this useful to the end-user? Well, I am working on nodejs bindings for this nice thing! Because, besides from giving us another FS implementation, it can make it possible to treat variables as they were actual files - so we can serve a whole app from within a zip file, that is even burried inside the nodejs binary. Cool, huh? :)

If you are curios about the tool I mentioned: http://github.com/IngwiePhoenix/file2c

Here is an example output:

Ingwie@Ingwies-Air /tmp $ cat foo.txt
Hello, world\!
Ingwie@Ingwies-Air /tmp $ ./file2c foo.txt fooFile > foo.c
Ingwie@Ingwies-Air /tmp $ cat foo.c
char fooFile[] = {
0x48, 0x65, 0x6C, 0x6C, 0x6F, 0x2C, 0x20, 0x77, 0x6F, 0x72, 0x6C, 0x64, 0x5C, 0x21, 0x0A
};
int fooFile_length=15;

So you see, we produced a small C file, that contains the file, stored in its actual bytes. From now on, we can use this file in our own C++ project and use ttvfs to mount that file in memory, and treat it just like it was a real file. That, being just one of thousand possibilities.

But there is more.

I am currently working on an alpha-version of the CEF+FLTK binding for NodeJS, so we can run at least a basic browser off a specific path. In the next version, I will first make it possible to patch the router process - so that we can modify the router to read files using the VFS implementation, or a custom one (like if you have made your own method to read files from somewhere).

I will post a few more details when I have a working preview version. :)

Kind regards, Ingwie

Simon Horton

unread,
Jan 31, 2014, 5:11:41 PM1/31/14
to appj...@googlegroups.com, Nicolas Embleton
Wow I have to say I continue to be amazed at the progress, new ideas and parallel development of multiple cool things all at the same time. This is build and building into one big massive new generation of deskshell that is going to land. As always I cannot wait to begin running my apps in this thing!

/Simon 


On Fri, Jan 31, 2014 at 9:48 PM, Kevin Ingwersen <ingwi...@googlemail.com> wrote:
Okey! Some new news from what I have here...and I have a near-alpha O.O!

So, the process is now going in a way that is different from before. I am patching the node.gyp and node_natives.h file in order to enable native modules to be compiled into the nodejs binary - same with javascript modules. That way, I have implemented ph7, native addon, and optimist (with its dependencies wordwrap and minimist)  into the nodejs binary...but I am not done. So, here is a list of the status:

        - FLTK+CEF PoC: It works. I can launch CEF thru FLTK, I just need to add minimal, platform specific, code to enable multiple windows and such...its really small. :o

        - FLTK+CEF for NodeJS: Bindings in process. I am currently having some issues with GYP again x.x; And, well, with splitting the different calls. Its not easy, but its the biggest challange. But once this monster is done, then we have the browser we want. :)
        - ph7: Its buggy currently, and I have a lot of redundant code. I have to work thru it, and fix it. But the base by itself actually works - but not all the things get converted correctly into ph7_values. o-o
        - Deskshell-Core: The core code is already half-done. This is the code, that runs instead of nodejs' actual main() function. It can differenciate on the different file extensions, and such. So, its going somewhere.
        - Deskshell: The public API is not done - yet.
        - ttvfs: This is a new piece of C++ code i found. It enables AMAZING! features. I'll explain that in a bit.

So, yeah, ttvfs... ^^


A while ago, I was looking for some really exotic kinds of software; virtual file systems, reading RAM as a block device (to interpret it as FAT32) and things like that. Then, very randomly, I came across this small project called ttvfs: https://github.com/fgenesis/ttvfs
This VFS implementation does not just offer a cross-platform way to manipulate and handle files...no, it offers a special way of handling internal buffers. In fact, I can turn a char buffer into memory file. Here is a short bit of code to demonstrate what I ment:


    // Now, lets create a memfile.
    VFSFileMem* memZD = new VFSFileMem("data.zip", ZipData, ZipData_length, VFSFileMem::REUSE);

    vfs.GetDirRoot()->add(memZD, true, VFSDir::NONE);

    // Mount the memory-virtual data.zip as a folder called data.
    vfs.AddArchive("data.zip", false, "data");

After the .AddArchive call, we now can access the contents of data.zip like it was a folder - AND data.zip is actually a char buffer residing in memory. That is cool, is it not? Okay, maybe you didnt understand what it does yet... So, let me word this differently.
> - ...

Kevin Ingwersen

unread,
Jan 31, 2014, 5:18:12 PM1/31/14
to appj...@googlegroups.com, Nicolas Embleton
I am stuck with compiling some objective-c thru GYP. Have to wait for the GYP devs to help me out; because that is blocking the way for a first alpha. xD I need to get the OBJC working, otherwise I can not compile on mac… o-o

Kevin Ingwersen

unread,
Jan 31, 2014, 6:48:27 PM1/31/14
to appj...@googlegroups.com, Nicolas Embleton
Looks like it was drag0n’s toolchain that caused the error - its ObjC compiler needs upgrading. o-o
Now, I have a very tiny binding to CEF…testing that now :)
Am Fr. Jan. 31 2014 23:11:41 schrieb Simon Horton:

Kevin Ingwersen

unread,
Jan 31, 2014, 6:58:56 PM1/31/14
to appj...@googlegroups.com, Nicolas Embleton
My first attempt just crashed straight away. :D

Ingwie@Ingwies-Air ~/Work/node-dev/fltk-cef/build/Release $ node
> o=require("./fltk_cef.node");
{ run: [Function] }
> o.run();
Illegal instruction: 4

Whatever! you see that there is the .run method? :) So, I am likely on the right road here. I just have to check why I am getting THAT error O.o
Am Fr. Jan. 31 2014 23:11:41 schrieb Simon Horton:

Kevin Ingwersen

unread,
Jan 31, 2014, 7:05:44 PM1/31/14
to appj...@googlegroups.com, Nicolas Embleton
Test 2 is a success. Although i DID get a crash, I did also get what i wanted!

Ingwie@Ingwies-Air ~/Work/node-dev/fltk-cef/build/Release $ node
> o=require("./fltk_cef.node");
{ run: [Function] }
> o.run();
2014-02-01 01:02:28.732 node[99268:507] Internals of CFAllocator not known; out-of-memory failures via CFAllocator will not result in termination. http://crbug.com/45650
[0201/010228:WARNING:resource_bundle.cc(251)] locale_file_path.empty()
[0201/010228:FATAL:main_delegate.cc(448)] Check failed: !loaded_locale.empty(). Locale could not be found for en-US
Trace/BPT trap: 5

This error there is just telling me that it can not find its resources. So in fact, I need to now mimic the OS X structure around this library. Once I did, I should be able to run this right away. But this also means I now need to change from working on a .node file to working on an actual nodejs source tree. Why? Because its easier to handle linking a binary to a dynamic library rather than linking a binary to a dynamic library,which links to a dynamic library. o-o But, I have an idea how to solve this all together! :)

So, from now on, I am working on the FLTK+CEF bindings, on the real nodejs source tree. I may find a way to get that working with two linked dynamic libs. But hey! It nearly started up after all :)
Am Fr. Jan. 31 2014 23:11:41 schrieb Simon Horton:

Nicolas Embleton

unread,
Jan 31, 2014, 10:23:03 PM1/31/14
to Kevin Ingwersen, appj...@googlegroups.com

Pretty cool... That's an awesome step to a working POC... Like Simon said, can't wait to see it happening...

Kevin Ingwersen

unread,
Jan 31, 2014, 10:25:49 PM1/31/14
to Nicolas Embleton, appj...@googlegroups.com
If GYP had been behaving since an hour or two, I could have even send in a screenshot already.

The problem is, that I can not build the shared library in the way that I want to. So defacto, I am a bit stuck… XD Anyway, I hope that I can find my fix soon. Gah, once I am done with GYP, i am _DONE_ with gyp. most frustrating piece of tool ever….

Kevin Ingwersen

unread,
Feb 1, 2014, 1:03:51 AM2/1/14
to Nicolas Embleton, appj...@googlegroups.com
Okay, so I get the browser to open, but the helper process crashes straight away. That makes me wonder how CEF tries to interact in its multi-process architecture. The sub-process probably tries to read memory from some nodejs chunk it can’t work out.
But I swear, for a short second, it does indeed try to render google.com :) Very, very close~!

Simon Horton

unread,
Feb 1, 2014, 3:42:05 AM2/1/14
to appj...@googlegroups.com, Nicolas Embleton

Take your time and play around with it. This is a really good direction and it is worth getting it to work really well. I am sure many similar problems will popup and you will get around them as you always do. Don't be stressed by us looking forward to the end result we all understand this will take its time, but is well worth investing in.

Simon

Kevin Ingwersen

unread,
Feb 3, 2014, 7:24:25 PM2/3/14
to appj...@googlegroups.com, Nicolas Embleton
I just gonna say this: If libcef wasnt „broken“, i had a demo.

I am talking on the cef forum with that magreen person - and he is helping me to tyke down errors. One has to do with - i belive - GL rendering. The errors came up while using my node-fltk-cef build. I already though that this was my code’s fault - TURNS OUT ITS NOT! :D

After Magreen suggested me to build a new cefclient, I build a debug version of it. Guess what,it threw the EXACT same errors O.O! That said, I belive that I basically finished a miniature binding for CEF to nodejs - i just have to fight the GL issues :)

Tonight is awesome, period. ^.^
Am Sa. Feb. 01 2014 09:42:05 schrieb Simon Horton:

Take your time and play around with it. This is a really good direction and it is worth getting it to work really well. I am sure many similar problems will popup and you will get around them as you always do. Don't be stressed by us looking forward to the end result we all understand this will take its time, but is well worth investing in.

Simon


Kevin Ingwersen

unread,
Feb 7, 2014, 6:42:06 PM2/7/14
to appj...@googlegroups.com, Nicolas Embleton
With an earlier version of CEF - as the current one is obviously partialyl broken on OS X - and a lot of learning in GYP, i made … this!!! :D


The current error has to do with the CEF context. I guess it has to do with the way I quit CEF after all… o.o But hey! This is a CSS 3D demo running, from a CEF, spawned via nodejs :D

The root for Deskshell is now here. Time to code mor eon this.

Kevin Ingwersen

unread,
Feb 9, 2014, 9:58:36 AM2/9/14
to appj...@googlegroups.com, Nicolas Embleton
And it just became even better =D.
I figured out, that I do not neccessarily need to use CEF’s crapped v8 „interface“, but I can stick to the real interface instead! So now, I am using actual v8 code within CEF.
That means, I can indeed copy a nodejs v8 object - such as the global one? - and put it into CEF’s global scope. I can not intercept any of the browser frames directly, as ndoejs only has access to the main browser, and non-ui, thread. Therefore, all must go thru that global object. Which in return could do great things like this:


// CEF….
<script type="text/javascript">
// Maybe, the require object is copied into CEF's global later, but this is an idea for now.
var fs = deskshell.require("fs"),
var ct = fs.readFileSync("./blabla.txt");
deskshell.$ = jQuery;
</script>

// In node...
var $ = deskshell.$;
$("#header").html("Beep o.o");


Another example would be:
// nodejs
deskshell.myFunc = function(a, b){
console.log("a + b =", a*1+b*1);
};

// CEF
<script type="text/javascript">
deskshell.myFunc(1,2);
</script>


Please note, that in this case, the nodejs console will be used instead, as the object was created there, and not in v8. That doesnt mean though that we couldnt...

// nodejs
deskshell.njs_console = console;
// CEF
<script type="text/javascript">
deskshell.cef_console = console;
deskshell.log = function(msg) {
deskshell.cef_console.log(msg);
deskshell.njs_console.log(msg);
}
</script>


Interesting, no? That all is possible because CEF and Nodejs share the same javascript engine. But there is one thing I am already concerned about…and that is something I will have to work hard on: keeping the code so clean, that reworking parts of it wont be impossible. Therefore, the actual compiled layout will be quite a weird one x3.
Also, managing to keep a difference between the two v8’s is not easy, and I dont know yet how this will work out when I decide to compile it into a single binary. I can imagine though, that THIS move will cause me to learn more about GDB and LLDB… o-o

Reason? Because…see. In libcef, and in nodejs’ binary, we have the same symbols defined. Therefore, I need to get the v8 engine matched. And I mean, very matched - same revision, same release. Then, I am quite curios about what will happen in memory. Say the dynamic symbol lookup („dyld" on mac) finds the symbol for v8::Initialize in node and CEF - which one will it choose, which one will allocate its memory, and where? I heared that dynamic libraries have their very own way of being allocated in memory. Its confusing to me, but I’ll know…at some point o.o

For now, dream of a wonderful global object thru which you can at least interact with the browser process itself. X3

sihorton

unread,
Feb 9, 2014, 10:21:16 AM2/9/14
to appj...@googlegroups.com, Nicolas Embleton, ingwi...@googlemail.com
Bit quiet on my end for a while. New people came in at work and totally messed everything up. So I left and got a new better job in two days :-) Anyway a lot of things keeping me busy but will resurface at some time.

This looks like magic to me! Amazing if it would work but I think we will have to be careful about crashes here. If we have two separate v8 engines running and try to pass objects back and forwards between them the garbage collector can delete an object from the other v8 and we will get a crash at some point. Likewise we can get problems with two different threads accessing and changing values at the same time, leading to strange randomness and the like. Not had time to understand the design or the code so don't take this as criticism. Just that we would need careful planning to achieve this. Basically AppJS had problems caused by using a single event thread for everything. Hopefully we can have two "processes", something for the browser and something for nodejs and then some form of boundary between them. I would suggest something like a web socket, web worker or similar interface. Obviously though it would be wonderful to just be able to access everything from everywhere (or have a global object available in both scopes). 

If we had a command "injectScript" then it would be possible for nodejs script to add code (i.e. functions) to a running window object, likewise maybe the browser could do something similar. Then you could call a function and pass in a json object as a parameter and get a json object as the response. I think though if we try passing C++ objects available to v8 from one engine to the other we could get some tricky stuff going on, very difficult to debug.

Once again though please don't think I am being negative here. It sounds wonderful, all I am saying is to have a play around and see how far you can go, have a little think through these complex issues and if they are not able to be solved we can fall back to the message passing idea.

As stated I am kind of swamped at the moment so if I take a while to reply give me some time and I will reply in the end! Once again though great work ingwie you are really out there ahead of the game again and it is great to see! :-)

/Simon

Kevin Ingwersen

unread,
Feb 9, 2014, 10:44:19 AM2/9/14
to sihorton, appj...@googlegroups.com, Nicolas Embleton
Two days?! That soudns like magic to me o.o…

Yeeeep, you understood exactly what issue i am about to run into xD. That is why I am trying to currently match the versions of v8. Good thing is, that v8 has a wonderful type that disassociates it from the GC: v8::Persistent. As the name says, those are always available, and will only disapepar if the programme xits, or is expicitely ::Destroy()’d :)

For the event loops…yep. That is the next issue I have. Well, I know that I can seperate them. Here is a bit of a threading hieracy that I know would work:

nodejs
|- fltk gui event loop. This thing is so pseudo… xD. It actually only idles about and around, and it is only used once to create a window for CEF.
|- CEF message loop. This spawns multiple threads or processes. I belive that I if I kick that off in a seperate thread, it will become non-blocking and hence will run totally async. Therefore, it shouldnt block nodejs’, as its in a separate thread…methinks.
|- Deskshell boundary. This is just a little thread with a few functions that keeps a v8::Persistent, which correspondends to the Deskshell object, available in both other threads. That however comes back to what oyu mentioned. I need to implement some kind of locking (Mutexes o-o) so the data isnt randomized. Howeve,r if I get that one fixed, then this object exists between the two v8 engines. Both use the same api - and hopefuly the same code - to access that Persistent. That way, it should be ensured that the data doesnt get corrupted. At least, thats my theory.

I am glad to hear from you again anyway :). I will keep posting all my updates right here. Have you seen the screenshot i sent in an earlier email? I went and picked up my bacardi for that night, because being able to run cEF from nodejs was the first big burden to go over. Now, this burden is done for :)

sihorton

unread,
Feb 9, 2014, 12:36:38 PM2/9/14
to appj...@googlegroups.com, sihorton, Nicolas Embleton, ingwi...@googlemail.com
Well you know how it is in IT, if you have someone that is good and hard working then you really want them in your team so your life is easy ;-) So lets just say there are many places that would love to have me :-)

What I suggest is lets do both approaches. Basically my "conservative" approach won't take much work at all. The current deskshell already uses websockets to connect nodejs and the browser. That code at the moment is "demo status" so you can create an api however is best and convenient to you. Lets just make it so that if running deskshell in nodejs + separate browser that the api works, and then we can have the same api if running in the new deskshell. Since we already have this code, and we have the appjs bridge this should not take too much effort.

Then we can have the "ultimate turbo" approach where you can do as much as you find fun.  Great thing is it means there is no pressure on you if we do find some weird bug or problem since we can just fall back to the "conservative" approach again. This gives you the creative freedom which you excel in!

Your approach sounds good so I encourage you to follow what you are doing and see where it leads. For me personally if I just got a "browser" CEF that was cross platform then I could swap it in for chrome which is annoying me on windows now. Anything more than that is a bonus. We also have the CEF that I hacked together which should also be cross platform but is only temporary until your version lands. Currently I don't have so much spare hacking time but I am sure that will change.

/Simon

Ashish Negi

unread,
Feb 10, 2014, 12:19:53 PM2/10/14
to appj...@googlegroups.com, Nicolas Embleton, ingwi...@googlemail.com
It is great to see somebody so much dedicated :)


On Sunday, 9 February 2014 20:28:36 UTC+5:30, Kevin Ingwersen wrote:

>Reason? Because…see. In libcef, and in nodejs’ binary, we have the same symbols defined. Therefore, I need to get the v8 engine matched. And I mean, very matched - same revision, same release. Then, I am quite curios about what will happen in memory. Say the dynamic symbol lookup („dyld" on mac) finds the symbol for v8::Initialize in node and CEF - which one will it choose, which one will allocate its memory, and where? I heared that dynamic libraries have their very own way of being allocated in memory. Its confusing to me, but I’ll know…at some point o.o

I think that  V8 is static library . Have you compiled it as dynamic library ?
Even then think about it like that if somebody created a library like you know printf or cout functions of c and c++ are also in library, then if multiple programs use them, they should be given different memory, otherwise output of one program would garble into others :)
Libraries are just code. Whichever process uses that code is given separate memory. So i think that if CEF and nodejs are separate process, they would be having different run-time v8 engines instances.

if we put the nodejs and CEF in one process such that they both share same v8 instance  then the thing would work out of box. ( original appjs was one process but had two v8's, so it required that js-bridge that mirrored everything in both lands ) it was like desksheel minus 'websockets'.

Also, if we study CEF3 architecture, i think ( i am not 100% sure ) that like when we open new tabs in chrome, it creates new renderers and new v8 engines in the renderer. So v8 is not designed to be like one global thing, from which both CEF or nodejs could talk. 

 I just have a random thought that if we can let CEF create its v8 and then we just expose the v8 - handler and where nodejs initiates v8, pass that handler there, may be furthur nodejs internal-objects would be created in the same v8.

Simon Horton

unread,
Feb 10, 2014, 12:24:50 PM2/10/14
to appj...@googlegroups.com, Ingwie Phoenix, Nicolas Embleton

Good points Ashish, you are very welcome to the discussion,  I know you have previously made excellent technical contributions. Maybe you and ingwie can get connected on chat and discuss ideas. If you do that post the conclusions to the mailing list so everyone is informed.

Simon

--

Kevin Ingwersen

unread,
Feb 10, 2014, 3:19:30 PM2/10/14
to Ashish Negi, appj...@googlegroups.com, Nicolas Embleton
Hey Ashish! 


It is great to see somebody so much dedicated :)
Great to hear youre happy about the fact! :)


I think that  V8 is static library . Have you compiled it as dynamic library ?
Even then think about it like that if somebody created a library like you know printf or cout functions of c and c++ are also in library, then if multiple programs use them, they should be given different memory, otherwise output of one program would garble into others :)
Libraries are just code. Whichever process uses that code is given separate memory. So i think that if CEF and nodejs are separate process, they would be having different run-time v8 engines instances.

if we put the nodejs and CEF in one process such that they both share same v8 instance  then the thing would work out of box. ( original appjs was one process but had two v8's, so it required that js-bridge that mirrored everything in both lands ) it was like desksheel minus 'websockets'.
Nodejs compiles v8 staticaly into itself - and libcef compiles v8 first as a static lib, then combines several static libs (like WebCore) into one big library: libcef. So, I have one v8 engine in the nodejs binary, another in CEF itself - they dotn share the same via a static lbirary, rather have their own.

To avoid a blocking call to CefDoMessageLoop(), I have two options: Either kicking off the whole thing in a seperate thread, or integrating CefDoMessageLoopOnce() into nodejs’ call routine, so its always added to the stack of execution. That way, the call wouldnt be blocking, and I would have both processes running fine alongside. However, I will most probably go with the multi-threaded aproach. That way I can let CEF do everything on its own, independent from nodejs.

In the process of creating the CEF instance, I am creating a pointer to a Cef instance - this pointer, I can reuse to do JS integration, accessing various things, and the like. So, I have the global window object, and can register persistent objects and the like.

I have done a few tests, and used CEF’s v8 independent form Nodejs. I now just have to find a matching Nodejs version - and, to ensure that specific features are different. I currently have to mingle my way thru with two different v8’s - 3.17 and 3.21. The current nodejs 0.11-pre branch uses 3.22 - which means i have a matching version, and that is what I am compiling against now. According to the mailing list, 0.12 will be the next stable nodejs - and that following one might be 1.0, finally.

Its just the issue of finding two API-compatible v8 engines - in nodejs, and CEF. The problem isnt made easier with the fact that currently latest CEF has strong issues on OS X when running from the command line - it throws GL errors all over the place o_o. So I have to stick with the versionb efore that.

According to v8 docs, I can expect that I can use a roughly x.x3 difference. So, if a feature is to be removed in 3.30, for example, it is announced to be deprecated in 3.28 - then really marked deprecated in 3.29 and most likely removed in 3.30. Same counts for API changes.

So, although I would be able to „easily“ upgrade the API itself, i have to always have two matching v8 versions alongside x3.


Also, if we study CEF3 architecture, i think ( i am not 100% sure ) that like when we open new tabs in chrome, it creates new renderers and new v8 engines in the renderer. So v8 is not designed to be like one global thing, from which both CEF or nodejs could talk. 
Almost right. CEF uses shared memory and IPC to keep the memory shared. The renderer connects up to the existing memory, which is why I can reuse certain things i created int he main program. So, if i created a sub-object on window, i can reuse it in anything else, as it loads that object.


 I just have a random thought that if we can let CEF create its v8 and then we just expose the v8 - handler and where nodejs initiates v8, pass that handler there, may be furthur nodejs internal-objects would be created in the same v8.
That would mean I had to recompile nodejs, rewrite the whoel build step, but it woudl actually work. It just would mean that nodejs would loose a lot of its speed. from what I heared, its not very recommended to use v8 as a static library for nodejs :p. But, it would be one try if nothing would work any more - and actually write a CEF based app that simply started off nodejs within a browser window. Funny idea, but ultra hard to archieve, especially for osmeone as me, who is not „that“ far on low-level coding XD.

… to be hoenst, the only reason why I understand most C++ is, my knowledge about PHP … ;)

Ashish Negi

unread,
Feb 10, 2014, 11:45:44 PM2/10/14
to appj...@googlegroups.com, Ashish Negi, Nicolas Embleton, ingwi...@googlemail.com


On Tuesday, 11 February 2014 01:49:30 UTC+5:30, Kevin Ingwersen wrote:

> Its just the issue of finding two API-compatible v8 engines - in nodejs, and CEF. The problem isnt made easier with the fact that currently latest CEF has strong issues on OS X when running from the command line - it throws GL errors all over the place o_o. So I have to stick with the versionb efore that.


If we are finally going to have two v8 instances, why do we need to have compatible v8's ?
I think that we are going to pass javascript objects in-between, two different v8 will do. Its like server and client.

Are you planning something else ?
 

Kevin Ingwersen

unread,
Feb 11, 2014, 4:33:36 AM2/11/14
to appj...@googlegroups.com, Ashish Negi, Nicolas Embleton
Only if the two v8’s matches, then I can be sure that moving objects between them will work as expected - and that I can use the same api across the entire app. Thats important, for memory management and for API maintenance. o.o

Simon Horton

unread,
Feb 11, 2014, 4:53:07 AM2/11/14
to appj...@googlegroups.com, Nicolas Embleton, ashish negi

Quick note here: problem will be javascript is designed to be single threaded in its execution but with 2 v8 and two threads able to access same memory and objects you will get strange bugs and errors.

Very short reason: one command in JS may have many actual steps that are carried out in sequence, whoever in the middle of that sequence second v8 can be context switched in to the processor and execute on the same object / data. This is unexpected for the original programmer and undefined what will happen, e.g. if one v8 creates overwrites a text file with new text and another v8 reads the file maybe you get part of the text from first version of the file then context switch and you get the rest of the content from the second version of the file. Massively weird bug almost impossible to debug. This is why parallel programming is really tricky, this is one reason nodejs is way cool because event oriented means you can write complex applications but can ignore the hard parallel stuff.

Not sure that was well written or easy to understand but this area is tricky. Having a "web socket" interface between the two v8 (a pretend one with same properties but messages copied from one to the other instead of going over the network) smartly avoids all of these problems and makes solution easy to use and no weird bugs. This is Simons "conservative" approach.

Having said that direct access to jquery and the dom from nodejs is beautiful and may be worth doing any way (with a disclaimer ok this could be dodgy).

Another point though is I am coming with the traditional accepted approach to these problems so I don’t want to shut down the creativity, a fresh pair of eyes looking at the problem without preconceived ideas can come up with innovative solutions that are very smart. So play around and see if we can get something that is safe but cool.

Simon

Kevin Ingwersen

unread,
Feb 11, 2014, 5:10:43 AM2/11/14
to appj...@googlegroups.com, Nicolas Embleton, ashish negi
Hey Simon.

Someone recently asked on the v8 users mailing list, if v8 is threadsafe. Answer: Yep!

The way it does is using isolates. As the name suggests, it isolates the v8 process from anything else. Node 0.11 uses them, and CEF does too (v8 3.21 and 3.22). Therefore, they will run independent form each other. The only thing I will be doing, is giving both access to one, shared, object. That might become rather funky, but as it is only one tiny piece, it would be easier to find out what is going wrong :)

But I have also thought about a sockets aproach too - actually, I thought about tricking socket.io into beliving it sent data thru the network - but did instead send it into a memory buffer. The same buffer is read by the client (and thus cleared) and the client thinks its a regular socket message. It’d be a nother, interesting aproach. So, it’s going to be nteresting which aproach works. For now, I am just trying to implement various APIs o.o

Simon Horton

unread,
Feb 11, 2014, 7:00:09 AM2/11/14
to appj...@googlegroups.com, Nicolas Embleton, ashish negi

Great sounds good, give it a try. As I tried to say go for it I am not putting the brakes on just flagging up relavent issues as they can be helpful to think through. Sounds like a good solution. If that global object had a function to register an event listener for a named event and a function to trigger a named event then you basically have a socket like api (json object as parameter in).

Simon

Simon

Kevin Ingwersen

unread,
Feb 11, 2014, 4:16:02 PM2/11/14
to appj...@googlegroups.com, Nicolas Embleton, ashish negi
Oh thats alright! :)
Its good to keep me reminded to rethink things - i tend to be too fast at some time o.o

A bit ago, I discovered something that made me look quite baffled - look! http://www.cappuccino-project.org/learn/demos.html

Cappuccino is like Objective-C but for javascript. Not only is ObjC one of my favorite language syntaxes - but Cappu says it can make „desktop applications in the web“ - now, combine that with Deskshell, and a possible full functional file i/o and things alike? Then Cappu might be a great framework to use with Deskshell :)! Thought i’d share so me bits of what I am finding here.

Currently, I am trying to fix a bug that makes CEF crashw hen I close it here on my end. Once that is fixed, I can move on and implement more of the API. First stop is to implement CefSettings - the thing that offers all the various browser settings, like remote debugging and the like. I am looking forward to implement some api to give basic control of CEf, since that is really the only kind of thing we really want.

Thanks to FLTK, we might also get back the „chromeless“ windows - i saw a switch in fl_Window, that would allow me to turn off the window borders - so we can design cool things without the OS chrome around. Some peeps might like this. ^.^~

Oh well, so much going on ^^. Well, i am going to prepair a proper building experience, then I am going to upload my code to github, so everybody can have a taste of my current work, and for possible contributors.

Ashish, if you wish to be added to the Deskshell-Core github group, just give me your handle :3

Ashish Negi

unread,
Feb 11, 2014, 11:24:04 PM2/11/14
to appj...@googlegroups.com, Ashish Negi, Nicolas Embleton, ingwi...@googlemail.com


On Tuesday, 11 February 2014 15:03:36 UTC+5:30, Kevin Ingwersen wrote:
Only if the two v8’s matches, then I can be sure that moving objects between them will work as expected - and that I can use the same api across the entire app. Thats important, for memory management and for API maintenance. o.o


If you are going this way, this means that there would be only one memory that is being used by 2 V8's. This simply mean that there is one v8 which is being used by two different processes. Each process has its own v8-isolate. and You want one object of each isolate to be sharable. Is it what you are working on ?

The problem here would be that when nodejs is working in its isolate, if CEF wants nodejs-that-shared-object, it has to move in nodes-isolate. But only one can be present in the isolate at one time. It would not be allowed. Have you thought about this ?


Ashish Negi

unread,
Feb 11, 2014, 11:27:30 PM2/11/14
to appj...@googlegroups.com, Nicolas Embleton, ashish negi, ingwi...@googlemail.com


On Wednesday, 12 February 2014 02:46:02 UTC+5:30, Kevin Ingwersen wrote:

Ashish, if you wish to be added to the Deskshell-Core github group, just give me your handle :3

I have not done any work to deserve this as of now. Just trespassing .
But this is a beautiful project.

Kevin Ingwersen

unread,
Feb 12, 2014, 8:45:14 AM2/12/14
to Ashish Negi, appj...@googlegroups.com, Nicolas Embleton
Yes, youre pretty close to what  am about to be doing.

I sent an email to the v8-users mailing list, and got an interesting response: „You may create a Persistent<Object> in each engine, but let them both take a refference to the same C++ class“.

This would mean, that in nodejs, and cef, i create exactly one persistant, scope-independent, object - but it has a reference to the same C++ class, which can store values and the like and act as a bridge.

The goal is simply to mirror an object in CEF and nodejs at the same time. So, consider the following API:

var fltk_cef = require("fltk_cef");
var browser = new fltk_cef.Browser();
var MirrorObject = browser.mirror;
MirrorObject.myfunc = function(){ console.log("Meep"); };

Now, MirrorObject is a refference to an object, that also exists in CEF - under window.mirror. I choose the name mirror, just to showcase what i am basically doing.

Its very tricky - but if ithis works out, I am gonna be damn happy. :)

Ashish Negi

unread,
Feb 12, 2014, 1:00:39 PM2/12/14
to appj...@googlegroups.com, Ashish Negi, Nicolas Embleton, ingwi...@googlemail.com


On Wednesday, 12 February 2014 19:15:14 UTC+5:30, Kevin Ingwersen wrote:

var fltk_cef = require("fltk_cef");
var browser = new fltk_cef.Browser();
var MirrorObject = browser.mirror;
MirrorObject.myfunc = function(){ console.log("Meep"); };


so if we do mirrorObject.require = require.
and require would be available on cef side ?

this is great. :)
we would have to think about this multithreaded javascript approach though.

Kevin Ingwersen

unread,
Feb 12, 2014, 3:06:33 PM2/12/14
to Ashish Negi, appj...@googlegroups.com, Nicolas Embleton

Am Mi. Feb. 12 2014 19:00:39 schrieb Ashish Negi:
> we would have to think about this multithreaded javascript approach though.

Well, multiple threads can access the same data. I just have to implement things like Mutexes. In v8, that is a v8::Locker :)

Ashish Negi

unread,
Feb 12, 2014, 11:10:34 PM2/12/14
to appj...@googlegroups.com, Ashish Negi, Nicolas Embleton, ingwi...@googlemail.com


On Thursday, 13 February 2014 01:36:33 UTC+5:30, Kevin Ingwersen wrote:

Well, multiple threads can access the same data. I just have to implement things like Mutexes. In v8, that is a v8::Locker :)

lets say we have mirror.nodejs

now this nodejs variable would also be  available on other side. if and only if they use it through the mirror only then, the locker would work. if they access it through their own, they can change the variables.

this means that if we have shared any object, then it should be used only through that shared object in both sides.
like we can do
mirror.require = require
require = mirror.require

now anybody who would be using require in nodejs world, would be using mirror.require and cef they have to use mirror.require.
then the locker thing would work. :)

great.

Kevin Ingwersen

unread,
Feb 13, 2014, 3:00:39 AM2/13/14
to appj...@googlegroups.com, Ashish Negi, Nicolas Embleton
Now you got the whole plan! :D
That is exactly what I am to implement. ^^

Kevin Ingwersen

unread,
Feb 13, 2014, 4:45:34 PM2/13/14
to appj...@googlegroups.com, Ashish Negi, Nicolas Embleton
For anybody who is curios, I have published my current work. Thing is, I am kind of burnt out X3. Not been able to really do a lot of coding…my mind went kind of dizzy, as I was getting back into doing simpler things. For one, I am trying to extend a build tool called … „build“. I am adding functions and the like to check for headers, tools, and the like. so it can be like a minimal CMake. It is so cool, that it got a minimal scripting language too - which is very, very clean. Like, we can do things like this:


A build config:
# This is how a target looks like.
target "test" {
# We build an EXEcutable.
rule "exe";
# We define inputs...
input [files("*.cpp")];
}

# And this is an action.
action "all“ {
# We tell the tool that we want to build the target Test.
build: "Test";
}


Output:
$ build
--- build test ---
[ 0%] deps: functions.o
[ 33%] cpp: functions.o
[ 33%] deps: main.o
[ 66%] cpp: main.o
[ 66%] deps: test
[100%] exe: test


So, I am planning to utilize this more often now, and it will also run the overall building process of deskshell. Its being made Windows compatible atm…Well - that is what I am doing when my head simply cant compete X3.

So, feel free to roam around in the soruce code.
The code can be found here!: https://github.com/Deskshell-Core/node-fltk_cef

Kind regards, Ingwie

Paco Zarate

unread,
Feb 13, 2014, 5:49:19 PM2/13/14
to appj...@googlegroups.com, Ashish Negi, Nicolas Embleton
Hey guys
this thread is awesome, thanks for sharing it publicly!

Ing: Your work is really interesting, just don't burn up in the process jajaja



Kind regards!

On Thu, Feb 13, 2014 at 2:45 PM, Kevin Ingwersen <ingwi...@googlemail.com> wrote:
For anybody who is curios, I have published my current work. Thing is, I am kind of burnt out X3. Not been able to really do a lot of coding...my mind went kind of dizzy, as I was getting back into doing simpler things. For one, I am trying to extend a build tool called ... "build". I am adding functions and the like to check for headers, tools, and the like. so it can be like a minimal CMake. It is so cool, that it got a minimal scripting language too - which is very, very clean. Like, we can do things like this:



A build config:
# This is how a target looks like.
target "test" {
        # We build an EXEcutable.
        rule "exe";
        # We define inputs...
        input [files("*.cpp")];
}

# And this is an action.
action "all" {
        # We tell the tool that we want to build the target Test.
        build: "Test";
}


Output:
$ build
 --- build test ---
[  0%]       deps: functions.o
[ 33%]        cpp: functions.o
[ 33%]       deps: main.o
[ 66%]        cpp: main.o
[ 66%]       deps: test
[100%]        exe: test


So, I am planning to utilize this more often now, and it will also run the overall building process of deskshell. Its being made Windows compatible atm...Well - that is what I am doing when my head simply cant compete X3.

Kevin Ingwersen

unread,
Feb 13, 2014, 7:07:09 PM2/13/14
to appj...@googlegroups.com, Ashish Negi, Nicolas Embleton
Not burning up? Thats why I am taking a little break, going down to easier things, so I can calm a bit. :3 And, that is why i shared the code I was working with - so people could go and lurk it o.o
But once I have gathered new energy, I will work on it again. But for ow, I am just…well, there is a yugioh turnament on saturday, cinema on sunday, a pucka-punch-packed Monday and a „im so gonna skip school“-thuesday XD. So for a bit, I really just need to live life for a bit ^.^;

Paco Zarate

unread,
Feb 14, 2014, 10:47:18 AM2/14/14
to appjs-dev, Ashish Negi, Nicolas Embleton
It looks like a really great plan! Have a nice weekend!!


Kevin Ingwersen

unread,
Feb 21, 2014, 7:55:24 AM2/21/14
to appj...@googlegroups.com, Ashish Negi, Nicolas Embleton
Hey everyone! Long time no message, hm? ^^ Well, today, I have some news for you.

First on, I survived my pucka-punch of a weekend - I ended up as first in the turnament and that motivated me lots, so I went and did some rather unique researching that I pushed away for months: The conjunction of threads.
…what a blending idea to do that. It changed the way I thought about things such as Pthreads and the STL threads. Lots.

With that knowledge, I went back and sat infront of my todo list. I was still getting a weird „Invalid context“ error from CEF, and so I was taking research on that. Say… ever browsed a 20GB source tree, trying to find a specific line? o-o It was horrible. And, I didnt find my answer either. The answer came to me on a rather random way.

While I am strongly coding on Deskshell-Core, I am also taking a peek at the alternative build system, and the library it uses „libbu++“. And that, also brings me to my packaging idea d0p. I did a long research untill I could find a nice and usable, embedable, archiving solution. I wanted it to be C++, because C++ API is mostly much cleaner. And in the end, I did find libcanister - an archiving library that is pointed towards speed. But it was incomplete.
While I extended Build, learned how to use Libbu++ and tested libcanister, I was forced to learn to do the basics of LLDB - thats like GDB, just for LLVM, the system that the Clang compiler comes from.

And with LLDB, I went back and poked my node-fltk_cef port. And wow, I did find the reason for the crash.

According to CEF’s docs, it does not like threading on OS X. And especially, it does not like to run across threads. And what I learned now is: It does not like to run on any other thread than t#0!

While NodeJS puts its event loop (uv_default_loop()) into t#0, the CEF extension happens to get booted in t#1! So THAT was the issue.

When CefShutDown is called, it looks into t#0 and when it finds out that it is invoked in t#1, it creeps and fails. Well, sh… x.x That means, that CEF is unusable in t#1 and upwards. Solution: Looking at Brackets.

Brackets uses nodejs and CEF in different processes entirely. The result is rather obvious: CEF runs fine in t#0 now. So I came to the solution that I need to learn the concept of shared memory, inter process communication and such.
For the first, I found a cool SO answer: http://stackoverflow.com/a/4353041/2423150
IPC is yet too far away from me  for now. Yet.

But thats not the only place I turned to. There is also Awesomium. Awesomium is a well hacked CEF. So instead of the CefBrowser class, you have Awesomium. And instead of CefV8Value, you have JSValue - and so on, and so forth.

Ironicaly, Awesomium must have stripped its resulting library. I can NOT find exported v8 symbols…I should turn to a Debug library and see what is going on here. Oh, did I mention that Awesomium seems to be closed source? :/ I only am able to download binary releases for Windows, linux and OS X.

But Awesomium has not just a much easier api, and obviously has its GUI code embedded already… It obviously works fairly different to CEF. I said, well hacked.

Currently, I am exploring the API and I realized, that I could theoreticaly convert forth and back between Awesomium and V8 - because both offer functions like .sInteger, .sString and so on and so forth. So maybe this is another solution: emebeding Awesomium…? That however changes some thoughts:
- One part of the Deskshell project is closed source.
- I have to write a v8<->Awesomium bridge, mainly creating proxy objects so I can still use all of nodejs’ options thru Awesomium’s browser, keeping the „global mirror object“ idea alive.
- If Awesomium is not being updated, I am stuck too.
So, that is the problem here for now.

But thats not all. I now realized what made node-webkit different. They didnt just embed bare WebKit, but they also changed the backing JS library, that is why native nodejs addons dont work as expected - and why node-webkit has compatibility issues. Aaaaha…. So, I can’t embed WebKit myself - can i?… I am currently looking into WebKit, Gecko and Blink (a Webkit fork, lol) and looking what I can do.

To summarize it all: I have to poke the CEF devs to implement more friendly threading into CEF, so I can run the whole thing in one external thread. I also have to learn more about libuv, in order to add functions into the event loop. Its not easy, and allt hat is going far deeper than I thought I would go after barely a half year of C++ experience O.O

But I am still on it, and my goal is clear. I want:

- A simple and reusable API.
- To be able to really use Node within a browser, without ultra-modding Node into a browser scope.
- An API in javascript that is easily understood and usable.


Well, I thought i’d send in an update. You can look at http://magpcss.org/ceforum/  to see me on CEF, http://answers.awesomium.com/ to look into Awesomium and my discussion, the nodejs google group to see me posting and asking, the gyp-developer mailing list to casually ask for help, my github IngwiePhoenix to see recent uploads and commits and activity.

I wish you a great day, and wel, let’s hope I can make it thru =D

Kind regards, Ingwie.

sihorton

unread,
Feb 22, 2014, 2:49:13 AM2/22/14
to appj...@googlegroups.com
Found an interesting blog post, the latest version of node contains support for running with multiple contexts.

http://strongloop.com/strongblog/whats-new-node-js-v0-12-multiple-context-execution/

Not sure if this is a useful solution or if it will trigger an idea for a solution in someone. Read it and add it into the mix of ideas currently being discussed.

Simon

Paco Zarate

unread,
Feb 22, 2014, 4:11:25 AM2/22/14
to appjs-dev
Hey guys,
this is a fork of aweomium when it was LGPL
http://code.google.com/p/chromiumoffscreenrenderer/
The wiki has a couple of explanation pages
They also tried to add some python bindings
Some additional explanation:
http://ajeanius.wordpress.com/page/2/ from the author to see what is he doing

Another project (now not maintained): http://berkelium.org/




Nazca Sistemas
~~~~~~~~~~~~~
C. Coruña 453
Col Ampliación La Rosita
Torreón, Coahuila
C.P. 27250
(871) 711.25.23
(871) 711.25.10


Shizza M

unread,
Feb 28, 2014, 10:12:10 PM2/28/14
to appj...@googlegroups.com

Kevin Ingwersen

unread,
Mar 1, 2014, 9:14:19 AM3/1/14
to appj...@googlegroups.com
I tried node-webkit. but when I read the build instructions…i more or less sunk thru the ground.
To build node-webkit, you have to download the entire Google Chrome source, then add node-webkit into the build process…and then build. The result is 10GB of junk you’ll most probably never need again. Ugh.

I also looked at atom.io - and i belive I know what they used. From what I can read, I belive they used Brackets Shell. All the listed features are 1:1 what the brackets-shell offers.

Currently, I am having some experiments. OS X development process is making it a bit troubblesome to do things as I want. Cocoa wants to run in the main thread, so I have to find a way to embed CEF’s message loop into nodejs’. Nothing impossible, but certainly not the nicest thing to do x).

I belive I can post you guys some more detailed description soon, once I have come a lil further! Thanks for th elinks also, I am reading them too, taking them into account.

Kind regards, Ingwie.

Paco Zarate

unread,
Mar 1, 2014, 3:30:48 PM3/1/14
to appjs-dev
Hey guys,
have you seen nowjs? it is a non-maintained project but the concept is interesting, they call functions from browser to nodejs but instead of mixing the v8 loops it is using socket.io but the code seems to be transparent using the now object:
https://github.com/Flotype/now

i.e.
<script type="text/javascript" src="http://localhost:8080/nowjs/now.js"></script>

<script type="text/javascript">
  now.ready(function(){
    // "Hello World!" will print on server
    now.logStuff("Hello World!");
  });
</script>

the logStuff is called on the server side.

This same concept can apply to php or other languages:
http://bergie.iki.fi/blog/dnode-make_php_and_node-js_talk_to_each_other/


Nazca Sistemas
~~~~~~~~~~~~~
C. Coruña 453
Col Ampliación La Rosita
Torreón, Coahuila
C.P. 27250
(871) 711.25.23
(871) 711.25.10


Shizza M

unread,
Mar 10, 2014, 4:13:37 AM3/10/14
to appj...@googlegroups.com, ingwi...@googlemail.com
Deskshell was given an article in the web designer magazine! How did I miss that?

https://www.imagineshop.co.uk/tag/product/list/tagId/21955/

On Monday, December 23, 2013 11:29:10 AM UTC+10, Kevin Ingwersen wrote:
Hey everyone!

Phew, it's been a while, isn't it? :§
Well incase you wonder how I am...well, I certainly feel like the wobble-bass of Dubstep. XD From one side to another, forth and back...all around and over and upside down.


Why?
Because, development. In LESS THAN A MONTH i taught myself C. I then added some basic ObjC coding - and extended that all with my PHP knowledge and made it C++ - and finally Obj-C++. So things like

    [instance Operation:@"default" withData:classInstance->getInfo()];

is not even unnormal, rather trivial. :) Kinda proud of myself right here!

But for what all the learning? Because Deskshell.


Evolution of Deskshell.
I have taken over appJS, and have looked into it and decided: I can not redo it. The reason is simple: the building just won't work! Making it CEF3 ready would require a whole re-write - so I just declared it as obselete and done. But, it's spirit is not gonna stay as it is. In fact, I have planned a relaunch as "AppJS Remix". I am a DJ - so for me, a remix is taking a song, and making something new of it. That is what I am doing with appjs here.
The Deskshell I am creating now is a structure based off nodejs, the thing we all know and - one or another ;) - may love. So, let me show you the structure first:

    nodeJS -> deskshell-core -> |
                                -> nodeJS  \
                                -> ph7     | -> libcef -> WebView
                                -> html5   /
                               
I am not mad at you if you are like "WTH is that?!" - let me explain.

The new Deskshell version is based off a nodejs binary, that has a modified "entrence point" - that part, that is run when you call a binary. But nodejs' entrence point is actually a pure javascript file, and called by main(). The modified entrence script will read in the .desk file and initialize a module: Deskshell-core. The settings/configuration stored in .desk are then pushed to the "core" module. That module now decides on hwot o proceed:

    - If the user specified to use a php script to bootstrap the resutlting app, lets use ph7.
    - If the user wants to use nodejs, use nodejs in a new process and let the current just die away.
    - Otherwise, create a webbrowser and display the content of the htdocs folder. That however is CGI enabled as well.
   
So, you can either bootstrap a programm (setting up custom listeners, functions, prepairing files, etc) or use Deskshell as a frontent for a plain HTML5 app. This all happens thru Deskshell-core, which is capable of calling in another engine, or relaunch itself. As you see, the plain binary will only really prepair for a new process. That said - you get two scripting languages in one. ph7 is a "fake" php, basically an embedable php for smaller projects based off the 5.3 definitions (no traits, no short-hand syntax for arrays/objects). However, its extendable thru a C-API. I have created a C++ wrapper (ph7++) that will hopefuly soonb e released by the dev team, once I helped them to fix the socket connections (socket_read/write/connect/...). I also ported some actual PHP extensions - currently only curl - to this engine. So in that term, you'd get a full PHP if you want to bootstrap your app using PHP, or even use it within the pages of your project that are subject to be displayed int he browser. .php files will be treatened as such - whilst .njs files will be processed by nodejs, giving you the ability to use nodejs templating engines for a Deskshell app.

The structure is ment to make it possible for javascript and PHP users to create a bootstrap if they want to. Both have their advantages, and disadvantages. I am a very engaged PHP developer, but I can do little JS. And drag0n, about which I will talk later, is a pure PHP project - so it will barely use any javascript at all.

Scripting the application launch also means that you can add custom features ont he go: Like you can implement a cgi for perl, python - or eve lua. Or you can check for updates. Oh, yes, we even have a gui.


The "new" GUI concept
I have been searching and doing heavy research on using a GUI framework, since CEF - chromium embedded framework - requires you to prepair a GUI for it, as it itself will only draw a bitmask into the window, and nothing else. That is also why their loading icons and such are png files - CEf displays a "picture". Of course, users with handycaps can use it too because its of course more than just a picture. But in all generall, CEF only really draws a picture, and according to the code that calls the library, updates the code.

My solution here was to use wxWidgets. Mainly because:
wxNode: https://github.com/joeferner/wxNode
and
wxWebViewChromium: https://github.com/steve-lamerton/wxWebViewChromium

As you see: The first makes wxWidgets available to nodejs - and the second acts as a simple wrapper for wxWidgets. With that in mind, we can now use a CEF instance according to the wx coding style! That said: We only need to compile one library (wxwidgets) with only the basic requirements in order to kick off a browser - with menu and whatever. And yes, a transparent window is possible too, from what I have seen in wxWidgets! So, you can still creare cromeless windows liek you did in AppJS.
Oh yeah, AppJS... ^^


AppJS Remix
Now that I told you how I am making the GUI, and how I am going to extend the GUI part to its underneath scripting engines, I can tell you that the extension - that will only be seen internally - will be called "appjs-remix". That means, that the combination of wxWebviewChromium and ndoejs actually represents the spirit of the old yet obselete appjs! I have thought - but not concepted - about re-implementing some of the old API in order to support some of the old features. However - once you are done, the actual event listeners you used to use are invaild. The scope then moves into a seperate process - with rendering and v8 engine - and I can not really port an IPC kind of communication - i am not skilled enough :I. But you'd at least get some of the old API back, that'd help you kick off a basic app again! ^^


Schedule?
First, chrismas. Then, I have to finish a PHP based intranet project for a company that I have worked for a while ago. They want to give me a job later, and help me a bit with my financials, it seems :p. Well, I have to make their project first, as I have been delaying for a while now and promised to do it in my winterbreak - which is now. After that, I will start pulling together a nodejs binary with the new libraries and programms integrated - basically, integrating the modules from above into the binary and seeing how they work, against which version, and how they work across plattforms. After that, I should have a nice pile with appjs-remix, ph7-node, wxnode and a base for deskshell-core. A nice pile to play with... then its just a matter of time to make Deskshell-core the "default" entry point. Once that is done, I actually oculd say that the binary itself is done. After it has been confirmed to worka cross platforms and that already created deskshell apps are still compatible, I can start to extend it further.
To make you think about something interesting:

    - libcef is ~40MB big.
    - ph7 - with my additions - is maybe 1MB
    - wxwidgets full set of libraries on OS X are 42MB. Consider that we will only use maybe 1/10 of that :)

So, the future Deskshell runtime will be about 50-60 MB. But I am not done here. After I have this done, I can go and remove unneeded features out of CEF and start creating a new libcef - libcef-deskshell. That, if we dream a little, may only be 20MB. Calculating again, gives us a 40MB build. Now running UPX over it, by loosing a bit of speed in launching - we get down to 20MB. Now tell me if that isn't a small but powerful runtime?? =D


drag0n
Drag0n, yes, it still exists. Just recently I published a WIP repo of d0p - a type of archiving that allows adding a header description in YAML format to the archive. Then running 'd0p explain archive.d0p' can output useful information - such as a description, website, or whatever. In the sense of drag0n, it saves adding an informative folder with all the info files and alike. Instead, we can just query each header in the folder and store it in memory for a while and examine it, use it, or whatever. We can even use this method to read from a HTTP stream! So we can litterally extract the header out of the internet. Then we can display that in the drag0n interface and let the user confirm the installation - and download the rest, and stream the archive down to disk.
But drag0n has another, more important, factor. Drag0n consists of many different parts:

    - "appstore" like interface to install and manage packages on a local system.
    - Distribution platform for developers.
    - Build tool for developers.
   
A subproject dubbed "d0.build" is just a little script that relaunches your shell and lets you use a portable toolchain, for building whatever application you may want to build! That said: its powerful. It can build packages from their source code right from a GUI and install them.
BUT THAT IS NOT ALL.
It is an appstore like thing. So, Deskshell apps can be distributed thru it. and since it is a deskshell app itself, you don't need to supply the whole runtime! Instead, it just "links" against the runtime it has within itself. So your previously 30MB big Deskshell app, due to inclusion of the runtime, may turn into a 5MB html5 based app. That does not just make downloading, but also installing a hell load faster!
Drag0n however does not just distribute Deskshell apps - but also GNUstep and gives a re-birth to a long-forgotten installer.


GNUstep and Fink
Eversince brew and MacPorts have launched, Fink has been pushed into the back of the scene, and I see a lot of people that use brew. But to be very honest, the reason why so rare people are using an installer is: No GUI. drag0n therefore provides a GUI and lets developers distribute apps for platforms that nobody has probably even seen xD. GNUstep is a platform to develop Objective-C/++ based applications across Mac, Windows and Linux. So drag0n installs already two platforms: Deskshell and GNUstep. And unterneath itself, it will use Fink with its very nice output format, and parses that in order to make manythings more understandable to the software itself. So in theory, alongside the drag0n structured repos, you can use an existing APT repo and just distribute packages thru that into drag0n. Why this works? Because Fink is APT for Mac. But drag0n will use the according package managers that are right for the current platform. I saw something called "win-get" for Windows, that is an APT clone for Windows. On anything Linux, I can just use the current package manager - zypper, apt, whatever.


Conclusion.
Due to my many projects within Fink, GNUstep and Deskshell, I am creating an SDK that makes creating HTML/JS/CSS based apps easier, but also adds support for server-side scripting with PHP and nodejs itself - making it a fully round-up solution to quickly create applications. And since you have actual wxWidgets bindings in there, you could even tell Deskshell to be fully pale, and use wxWidgets to create your app! That is, if you want to... ^^
But I am not done here, not closely. Due to my find-out about nodejs for ios, and the current CEF goal to port to Android and iOS, we might see Deskshell on mobile platforms too! And hopefuly I can find a way to devleop a port for win8 metro UI.


Getting involved.
The appjs project may be deprecated/obselete/whatever - but it is the perfect place to talk to me. It is where everything started. Alternatively, feel free to browse my site http://ingwie.me (still under construction) and contact me directly - or just mention Simon and ask him for his email. Simon knows - i think ^^; - everything about the plans that I have made - and ing eneral he is a very nice person to talk to and make concepts with. ^^



Kind regards, have nice holidays and all the like!
- Ingwie Phoenix, alias Kevin Ingwersen

PS: In some time soon, Ill make a video and hold a short speech about a lot more things than what I mentioned here. Yes...I ish bussy burdeh @.@ ... err, I mean, phoenix. o.o
Oh and if you find typos, you may keep them =D

My Other projects (skip if not curios)
- GNUstep for Mac OS: Create an alternative toolchain to Apple's Xcode and make OS X development easier - but also cross-platform development. My build will feature a static libobjc2 build, in order to make sure you dont depend on anything out of my build.
- Experimenting with wxWidgets and CEF: Making sure that the appjs-remix and wxWebViewChromium things work just as expected.
- Create the full drag0n structure and interface: I have a PHP layout for drag0n (d0) that will make it possible to use it anywhere where you want. And another idea is to create a single, little binary. So even if you dont have PHP or anything, but need a short bootstrap thing to download libraries or dependencies, that will help you. Just compile d0.cpp and d0p.cpp into a d0 binary, and you can issue commands such as "d0 install libicu" to get the latest version of icu into your current working path. As it will just be a bootstrap helper, it will not write any information files or alike - it will read everything off thru the internet.
- Create a Deskshell "compiler" for the current version to make proper Mac OS X bundles.
- Finish other stuff such as WingStyle, my own site, and my other stuff you can see on my small site.

Kevin Ingwersen

unread,
Mar 10, 2014, 4:21:44 AM3/10/14
to Shizza M, appj...@googlegroups.com
Whaaaaaaaaaaaaat :O? Must read!
Message has been deleted
Message has been deleted

sihorton

unread,
Mar 10, 2014, 6:01:52 AM3/10/14
to appj...@googlegroups.com, Shizza M, ingwi...@googlemail.com
Ignore the two deleted posts from me for now. However I have an awesome post for http://thedailywtf.com/ once everything has been sorted out, until then I should wait a bit before discussing it all.

Kevin Ingwersen

unread,
Mar 11, 2014, 3:51:16 PM3/11/14
to Shizza M, appj...@googlegroups.com
Hey everyone!

Okay…time to explain whats going on once again. ^.^

I have to say first, thanks to all that have been waiting patiently for all the updates I post, which are more like entries in a blog. and I am happy to see free time coming up, which means I can finalize http://ingwie.me and have more reports on there as well. But, what has been going on, Deskshell related?

First, to differenciate the current deskshell from the one that I am working with, I am speaking of the „Deskshell WebDev SDK“ - in short, DW. It is just a dynonym I use in my workspace, to differenciate between the various copies of things that I have.
But thats not all I did. I did a lot of reasearch and learning. Like, I learned to use LLDB, an equivalent of GDB, just for the LLVM builds (Clang ’n stuff). And that helped me learn a bunch of things:

- Cocoa, OS X’ GUI stuff, _must_ run on the main thread.
- CEF’s message loop must start and end on the main thread.
- Nodejs can be put into a different thread.
- CEF’s routine functions must be added via the native GUI’s functions. So I have to create an AppDelegate on OS X to properly shut down CEF…sweet.
- FLTK can be used alongside native code - its tricky, but posible. I want to minimize the native GUI code as far as possible.
- …I really must work on drag0n.

The biggest dependency needs to takeover the main thread.
As CEF and Cocoa want to run on the main thread, and its not very adviced - yet possible - to run nodejs on a different threat, I have come up to a whole new idea and concept: If nodejs cant embed CEF - can CEF embed nodejs? And I’m like … O.O!
Imagine the following: You are writing a usual webapp, but when you run it thru the DW SDK, you dont just get window, console and such - but you also get require, process and even more! Yes, I am speaking about importing the nodejs godness into an actual browser window. But that is not all. As we can run a separate ndoejs instance in a different threat or process, we can create an inner-process/thread communication using memory-based sockets. That sounds ultra confusing, but its easy: We allocate a piece of shared memory, and give it a name, like: „/DeskshellWSSDK/socket“. Now, this one can be used from everywhere that shares the same memory, or knows the name. And because we do know the name, both threads can individualy access this data and treat it just like a socket - so we can read and write to it. With a continious listener process, we can then listen on that socket, so we can archieve background tasks.
The best part: No colliding event loops!
So if I get this aproach done, we would gain the following features (so far):

- Background process/thread that can do long-term and blocking stuff in a way that doesnt break the browser.
- The background script can send messages to the main program to tell it what to do; to launch, to minimize or to execute javascript - bla, bla, bla :)
- We would get all the nodejs goodness in a regular <script> tag, so it would truely feel like „not just a browser“.

And more, of course. But most notably, the process of running the application would be different. The application would not show untill all the gui preparation was done. This, is actually a problem that I have to figure out. But I belive that it is possible to keep a NSWindow unrendered and such, and render it later during aplication launch. So, that we could have a code like this in the background:


var gui = require("fltk");
var Menu = new gui.menu(); // On non-osx OSes, you may set additional info such as height and such.
Menu.addDefaults();
// Creating a cool menu entry.
var coolMenu = [
{
name: "Beep", 
callback: function(e){/*...*/}
}
];
Menu.add("My cool menu", coolMenu);
// app is already defined, to control the "foreground" app.
// This call is not blocking, but...
app.start();
app.on("launched",...);


As you can see, this is already slightly modified nodejs. And indeed, we’d be using a modified nodejs library, in order to support some extremely cool stuff. Some integrated modules, that we even are gonna use for the internal stuff is:

- async
- JASON
- SQLite ( most probably. alternatively, I may make bindings for SQLCipher. )

Async is cool to kick things off in different calls, and to get disturbing stuff out of the way.
JASON is going to be used to transfer complete JS objects. It does not just serialize values - but also functions, and basically everything. This is essential when sending code from the backend to the frontend.

The configuration will change
Currently, the .desk files are JSON, but as I just discovered, I will use JSON5: https://github.com/aseemk/json5
This looks like a very cool way for configuration. Alternatively, soemthing like js-yaml or such will do. And in the worst-worst case, I will code bindings to libucl - which is a hybrid config of yaml and JSON. The configuration might - just maybe - look like this then:

{
name: "drag0n",
version: "0.0.1",
main: "lib/drag0n-daemon.js",
htdocs: "www",
// This should be useful for debugging! :)
remote_debug: 9999
}

It looks simple, and it is.


CGI in Deskshell
Due to my nature in PHP, I of course want to keep using that. I found Cappuchino - an objective javascript library that makes developing webapps look like Cocoa - very useful, but I wish to stick with Yii - especially as Yii2 is supposed to bring lots of goodies. Therefore, I am working on embedding either ph7 - or the real PHP. That means, that if you request a PHP file, it will be run as such. So you can keep using modern web technology.


And: the coolest extension ever.
I found a project called ttvfs. A c++ library, that lets me treat things like zip archives as they were actual files, so I can do things like that:

std::string Content = ttvfs::ReadFile(„./data.zip/foo.txt“);

And, then, there is also the library libcanister - a very, very tiny archiving library that is looking to be one of the fastest. Combining both together: We would get container files. So we may not specify htdocs - but we could specify htcontainer instead for the config, and even in URLs, we might use things like:


That, makes things ultra small, and fast.


As you see, I made my mind on proper and new ideas. I am trying to make one of the best WebDev SDK’s available - because in the meantie, I got to paly with some others. most especially node-webkit made me shiver; to build it, you gotta download all of Chrome’s source code. Yikes!


Speaking of others.

How about deskehell on other platforms?
Like, I followed the innovation of a group on github called node-app. And they released an ObjC framework called Nodelike. That allows OS X and IOS developers to embed a „node-like“ javascript engine into their apps. Furthermore, it has been coded so, that you can embed that engine into a webview. In return, this means that you have a nodejs interpreter inside your webview. But not jsut there you have that. I recently found out that people compiled SDL with an actual webview for the PS3, as well as some people pointed me to a webview on android.
What do I want to say with that?
If I get the original, this one, Deskshell working, I will see if I can make more devices easier to be develoepd with!


And what aobut drag0n?
That is kind of a side note.When I recieved my raspberry pi, i could just use apt to get everything installed. For the deskshell build, I will have a minimal binary ready, which will let you use libd0 from the commandline, in order to install dependencies if needed. For that time, I will have to provide the said things on my own. I am working on libd0 and libd0p. The first being an installer and package manager, whilst the second is the package format to be used by default. That will help when building Deskshell - a lot. :)

And there ends my long post again. Hope I didnt confuse you, but I hope you get what I was talking about. Let’s hope that things work out as I want, because then we could have an alpha quite soon! ^_^


Kind regards, Ingwie

Simon Horton

unread,
Mar 11, 2014, 4:51:52 PM3/11/14
to appj...@googlegroups.com, Shizza M

Hi Ingwie :-)

Great ideas and creativity as usual. You are making a lot of sense and good research and experimentation. I will make the following points, see what you think of them:

A) I think CEF should embed nodejs as you have decided. Reason for me is startup time, shortest time from icon click to window display is the goal. We can then fire js events on the window -- onNodeJSAvailable() and so on. This is really important as it lets developers put a splash screen, loader or whatever they want and very simple apps load faster. Difference between 2 seconds to open and 30 seconds is a massive boost in quality feeling.

B) This is not a major point but remember we can open multiple windows so we will have multiple browser threads and stuff. This means multiple threads accessing same socket. Implemented in the standard event oriented design where you can fire a named event and pass json and then register a callback to receive all or a particular named event gets around all nasty issues, and is already familiar to web programmers.

C) Sorry to be boring but I suggest standard json for config. Reason is you can load and parse in both browser and nodejs without any library and in few lines of code. This makes it easy to write code to allow easy editing. I would suggest we have a little deskshell app for editing the config this can support yml and anything else developers want, but then saves to disk as json.

D) There are so many cool things you are investigating and great ideas -- very cool. What I would suggest is lets have an ultra minimal core that can load super fast. We can then have plugins / extensions that add capability to the platform. Again this will allow super simple little apps to be tiny, but kitchen sink massive apps to also be supported. Also makes cross platform development easier since certain modules can be available on certain platforms only initially then we can standardise later on.

E) Container apps / packaging is cool great to see you getting better and better solutions. Doing it in nodejs gives you ability to stream the files out of the box, but pure c++ is great. Lets implement both and let developers decide what is best for their app!

All this talk and excitement is making me motivated to release that hack I did with cef + nodejs, it is not as grand as your work but gives us something to experiment with in the mean time.

/Simon

--
You received this message because you are subscribed to the Google Groups "appjs-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to appjs-dev+...@googlegroups.com.
To post to this group, send email to appj...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Paco Zarate

unread,
Mar 12, 2014, 1:45:20 AM3/12/14
to appjs-dev, Shizza M
Great guys! I will read the notes and try to give some feedback.

Simon Horton

unread,
Mar 12, 2014, 1:51:36 AM3/12/14
to appj...@googlegroups.com, Shizza M

Looking forward to your input Paco :-)

Paco Zarate

unread,
Mar 12, 2014, 3:27:55 AM3/12/14
to appjs-dev, Shizza M
A lot of info to analyze! Well no complaints because I was expecting this email since the weekend jaja

The strategy for using cef to load nodejs I think is a good one. It is similar to the tidesdk (but it used webkit and then load the other stuff). So we know there is a project that could use that approach. I don't recommend looking at that code because it simply overwhelming.
But that remembers me something that actually can be interesting. Appcelerator developed a little monster called kroll ( https://github.com/appcelerator/kroll ) . This is a bridge that allowed to interpret the <script> sections on the browser side and use php, python and ruby on those sections. It was using the full versions of those languages and using a proxy to bridge the javascript objects to the other languages contexts. So it was transparent for the user to use php in the client side.
Something like this is possible using kroll:
<script>
    var globalNumber = 23;
</script>
<script type="text/php">
    $globalNumber = 13; // globalNumber will change
 
    function changeNumber($number)
    {
        global $globalNumber;
        $number = 10; // globalNumber will not change
        $globalNumber = 10; // globalNumber will change
    }
    changeNumber($globalNumber)
</script>
or this code:
<script type="text/php">
    function my_php_function($item)
    {
        $result = array();
        for ($index = 0; $index < $item->count(); $index++)
        {
            $result[] = $item[$index] + 2;
        }
        return $result;
    }
</script>
 
<script>
    alert(my_php_function([1, 2, 3, 4]));
</script>

or this (to include an external php file):
<script type="text/php">
    include("my_other_file.php");
</script>

I should warn that there are some reports on memory leaking when accessing files using php but it was a really interesting way to use php on javascript.

Also what Simon says about keeping the resulting app as small as possible is very important. So it would be important to use only one php engine. Simon, can you explain me please in general how deskshell packages the files? My guess is that it installs cef engine and nodejs only once if needed and that the setup app downloads them if it is needed.

Thanks in advance!
Paco.

Kevin Ingwersen

unread,
Mar 12, 2014, 4:46:21 AM3/12/14
to appj...@googlegroups.com, Shizza M
Hey.

I just looked into Kroll.

I had once wanted to port the drag0n GUI to TideSDK, but the fact that it didnt work on OS X stopped me from doing so - and as they furthermore started to take this stuff to closed source and started to make money from it, I even distanced myself from it by far.
I saw Kroll before, and I was fascinated, because you could litterally do:


<script type="text/php">
$arr = jQuery(".myclass")->findall();
foreach($arr as $node) {
$node->html($node->html()." - BEEP.");
}
</script>


I wanted to find Kroll before, but gave up my search because I thought „It may only work with tidesdk anyway“. Now that you linked me to it, I tried to build it, even installed scons (wow, its very slow…). And I got a build error right away. Scons tried to build with -isysroot /Developer…. thing is, that part doesnt even exist. On older OS X, it does…but not on anything newer than 10,.8 - and I am on 10.9. So, I cant build Kroll for testing, so I might have to poke the devs and modify the build so it works.

The sad thing is, that in order to add additional <script type=…> stuff to CEF, you have to modify its source, which would then again require downlaoding all the gigabytes of code. Maybe I can find another use for Kroll, but by far, I like that microkernel :)

For now though, I am waiting for a reply on my github issue I put at th enodejs page on github, waiting for them to reply to a question I had. If they dont, then I will try to hack the nodejs source myself, and create libnodejs, so its embedable. x)

But thanks for the link to Kroll, very useful!


@simon: You are right, I was too fascinated by JASON and JSON5. However, the internal protocol will definitively use JASON, because we can transfer full javascript objects too.
About the socket: Just think of it as a socket.io process. It works similar, just that the background script - who has the app object - is the server, whilst all windows are clients. In fact, I am pondering to find a way to „trick“ socket.io in working over a memory buffer. That would make things ultra-simple. Alternatively, I could see if I can run a websocket server thru memory. But the important thing is: it MUST go thru memory. I can’t afford to open any userland ports for inner process communication.
For the bootup time: I am going to implement a rather simple set of functions to the app object. In fact, one will be

app.start();

If given no parameters, it will kick off a basic window, with basic functions, and CEF within! So the startup time should be minimal. So you could do something like this:


app.start();

var dl = require("downloader");
dl.get(/* Do something like a heavy blocking task here, it will not interfere the browser. */);

dl.on("complete",function(e){
// Trigger an event, that has been set in browser.
// e will be copied using JASON, so it will be there 1:1!
app.trigger("download_complete",e);
});


We’ll see how it’s gonna be going! ^^

Paco Zarate

unread,
Mar 12, 2014, 4:59:22 AM3/12/14
to appjs-dev, Shizza M
Ing
On the kroll issue, maybe this patch can give some light
https://github.com/meeech/kroll/commit/8e3b031f997d2290bee3f15b57a60862e86be7bb
it is for 10.6 so it would require you to check if the equivalent 10.9 path exists:
something like:
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.9.sdk


Sorry I don't have a mac with me right now.

Kevin Ingwersen

unread,
Mar 12, 2014, 5:01:35 AM3/12/14
to appj...@googlegroups.com, Shizza M
You nailed it, thats the path I would need. X)
Looking into it~ but its english class now o.o

Simon Horton

unread,
Mar 12, 2014, 5:56:29 AM3/12/14
to appj...@googlegroups.com

Great discussion, excellent points.

The kroll thing is cool but we can do that in pure JS. Just include some deskshellMagic.js script tag upon loading it then scans for script tags with type="text/php" then it can take the text and call a function:

Deskshell.ext.php.eval (scriptText);

Deskshell .app file can list the api's to make available on startup. If php is to be loaded you can then have a persistent php environment that you can tell to eval bits of php script. Same would work for other languages.

As to passing actual objects around I dont like the idea since we don't want memory leaks or random crashes. However as Ingwie says the "socket.io" "design pattern" solves all of those issues, is well understood, no pointer or memory issues. If anyone listens to me then we will support that interface pattern. Data passed will be json string because it is good enough.

Now you all are brilliant so you are welcome to add the following api:

Deskshell.unsafeButFast.

And everyone can use that if they want. You can try passing DOM objects direct to php and fix it so it even works. It is possible to use messagePack to send binary data rather than json and so on. If we have a nice plugin architecture it is awesome since anyone can do an experiment and if it is better everyone switches to it :-)

As I understand it script tags with type attributes that are not something like text/javascript or similar are ignored by the browser and that works cross browser and platform, I remember using that trick some years back on a project I built.

/Simon

Simon Horton

unread,
Mar 12, 2014, 6:08:09 AM3/12/14
to appj...@googlegroups.com

For startup time the current design starts nodejs, initializes, reads the config, runs user script that boots off CEF that then gives the user a window they can look at.

I like that design since it ia nodejs centric.

However for performance if the first thing that starts is CEF then internally it is multi-threaded and starts all the initialization in parallel, then use C library to read in json config and open a loading page/ splash screen url from that -- my horrible hack that does that is extremely fast to then open.

Then we can also boot off nodejs, php, whatever and when they are ready and willing fire js load events on the main page. This allows intelligent app designers to have really great load UI and deskshell appears to be blazingly fast.

As to the packaging, all I do is append all of the application files into a single file, at the end of that file is a json string with a table of contents. I then wrote streaming code so when you request a "file" it starts from the start offset and streams till it gets to the end. No compression since that will slow it down and is unnecessary: you can gzip or whatever the whole file afterwards if distributing over the network.

/Simon

Kevin Ingwersen

unread,
Mar 12, 2014, 6:38:00 AM3/12/14
to appj...@googlegroups.com
If the concept that I am working on works, then the order would be:

1. Pre-init with reading the config
2. The background nodejs process is started
2a. While on the main thread, CEF is waiting to recieve its „show up“ event (i.e.: app.start(); )
3. CEF loads the next usable resource from the config, and while it does a ndoejs environment is spawned int he window scope, thus a global object is added to emit and trigger events all around.

That means that we also can use native nodejs extensions - like:


<script>
var threads_a_gogo = require("./threads_a_gogo.node");
// ...
</script>


That would archieve a very fast startup time, because if the first function that the background nodejs process is to start a default brwoser, then we would click the icon and almost instantanious, would get a browser window. We can then wait for the ready event to request a new page.

I am currently experimenting with taking Zend PHP and writing my own build system for it. I tried to convince the devs to use CMake, but they dont even want to create a new-ish layer for their 1990’s API XD…so I am going to take this to my own task.

Simon Horton

unread,
Mar 12, 2014, 6:39:34 AM3/12/14
to appj...@googlegroups.com

Sounds great, a very good solution I think.

/Simon

Paco Zarate

unread,
Mar 12, 2014, 7:27:44 PM3/12/14
to appjs-dev
Simon is the socket.io already implemented in deskshell?

If not maybe we should take a look at dnode, there are already some nodejs + php (and other languages too) + dnode in the wild.
My guess is that dnode is basically using the same approach than socket.io but with some nodejs integration already done.

https://github.com/substack/dnode
http://bergie.iki.fi/blog/dnode-make_php_and_node-js_talk_to_each_other/
https://github.com/bergie/dnode-php

I agree with you that this type of IPC will be much more stable than other more low level options.

I have read some interesting things on Boost.Interprocess but I have never done anything with that library.

Paco.

Simon Horton

unread,
Mar 13, 2014, 1:43:02 AM3/13/14
to appj...@googlegroups.com

Hi Paco,

When I talk about socket.io I just mean the "pattern" rather than actual implementation. If you view their website then they have nice docs about how to talk from "web page" to "server". My suggestion all along has been implement the same "interface" but you can have anything you want behind.

So for example nodejs can receive a pointer to the js object from the browser and then run code on it. However it is also easy (but not as fast) to convert js object to text string, copy that to php or nodejs or anything, convert string to php / js object and continue. This is 100% safe with no memory leaks, random crashes or risks. Also most modern web programmer understands the interface instantly and if not it is very quick and simple to use.

To answer your exact question, yes the current implementation gets nodejs to exec chrome and then opens debug websocket connection and also if needed a websocket connection between nodejs and browser.

Additional advantage of websocket interface pattern is that it will be easy to have same code work in deskshell as will work on application with remote server backend over websocket.

Yes I looked at dnode before we could use that. This is open source so feel free to create github repo and hack together a (bad) demo illustrating what you want to do, or just some gists with pseudo code as examples. Don't worry about it being beautiful we are all programmers so understand the development process. Then we can discuss the best way to add it to deskshell and any further ideas.

I am very happy to let deskshell develop in many directions and let the app developers decide which technologies they want to pick. I also see my role as just helping people get involved, supporting them and getting people working together rather than doing all the work myself while everyone watches me.

We have a lot of potential in this project and a lot of enthusiasm and rapid progress so you are very welcome to help and hack on whatever you feel is fun.

/Simon

Kevin Ingwersen

unread,
Mar 13, 2014, 4:13:05 PM3/13/14
to appj...@googlegroups.com
Guuuuuuys :)!

With some search and some googly hands, I found this:
https://github.com/rynlbrwn/channel

I just tested it with one thread, but appearently its supposed to run with multiple threads too. For my purpose, it will only really need to spawn over two - main and secondary - anyway. But this is EXACTLY what I needed! Just look at my example program, it should give you an idea.


#include <iostream>
#include <pthread.h>
#include <unistd.h>
#include "channel.h"

using namespace std;

// Create public channel - only one.
Channel<string*> c(15);

void* thread1(void* data){
sleep(5);
//string* sn = new string("Yo guys, sup??");
c.Send((string*)new string("Yo guys, sup??"));
//delete sn;
return NULL;
}

int main(void) {
pthread_t th1;
int rt1;
// Create threads
cout << "Creating 1 thread" << endl;
rt1 = pthread_create(&th1, NULL, thread1, NULL);
cout << "Detach" << endl;
pthread_detach(th1);
// Nooooow.... We expect to get a string value.
string* ot;
// And, wait.
while(!c.Receive(&ot)) ;
cout << "Result: " << ot->c_str() << endl;
return 0;
}

Kevin Ingwersen

unread,
Mar 13, 2014, 4:47:48 PM3/13/14
to appj...@googlegroups.com
Did a multi thread test with 3 sub threads and the main thread.

Result: WORKS!

Now we have an inter thread comunication! That is, basicaly, what I needed. Now I just need to implement v8 functions and add that to the global stuff, and test if I can actually push v8 stuff thru it, or if I have to use native/own data types. But in fact, Channel is exactly what I needed. :)

#include <iostream>
#include <pthread.h>
#include <unistd.h>
#include "channel.h"

using namespace std;

// Create public channel - only one.
Channel<string*> c(15);

void* thread1(void* arg){
int id=*((int*)(&arg));
// Lets play "Catch the message"!
string* ot;
while(!c.Receive(&ot)) ;
cout << ot->c_str();
delete ot;
return NULL;
}

int main(void) {
pthread_t th1, th2, th3;
// Prepair the game
c.Send((string*)new string("1"));
c.Send((string*)new string("2"));
c.Send((string*)new string("3"));
c.Send((string*)new string("4"));
c.Send((string*)new string("5"));
sleep(1);

// Create threads
pthread_create(&th1, NULL, thread1, (void*)1);
pthread_create(&th2, NULL, thread1, (void*)2);
pthread_create(&th3, NULL, thread1, (void*)3);
pthread_detach(th1);
pthread_detach(th2);
pthread_detach(th3);
// Nooooow.... We expect to get a string value.
string* ot2;
// And, wait.
while(!c.Receive(&ot2)) ;
cout << ot2->c_str();
delete ot2;
cout << endl;
return 0;

Simon Horton

unread,
Mar 13, 2014, 5:07:47 PM3/13/14
to appj...@googlegroups.com

Wow the pace of development and lateral thinking here is amazing :-)

Paco Zarate

unread,
Mar 15, 2014, 2:48:11 AM3/15/14
to appjs-dev
Nice sample Kevin! jaja seems like it will allow to continue in the integration you are doing!


Simon: Thanks for your explanation.
Have you read about chrome apps? seems something we can use and google is releasing a toolchain for developing them on mobile. If we integrate the communication to a sockets.io from the chrome apps this can use the node.js libraries in addition to the chrome ones.

Another project I have read about is Apache Thrift http://thrift.apache.org/ that allows to do an IPC between manu languages.

Paco.
Ps. Guys, where can i find the deskshellMagic.js to eval the php code?

Paco Zarate

unread,
Mar 15, 2014, 2:52:22 AM3/15/14
to appjs-dev
Hope you enjoy it

Simon Horton

unread,
Mar 15, 2014, 8:20:10 AM3/15/14
to appj...@googlegroups.com
Hi Paco,

I have investigated using chrome apps, currently we actually use a command line argument to a portable version of google chrome and that is what is displayed. This is how we get rid of navigation bar etc. There are basically always going to be security restrictions with chrome apps and those restrictions are going to be constantly updated and changed -- not good if you are developing a business app that you want to run for 5 years for example. Basically whatever google gives us in terms of apps we can use in deskshell and then add nodejs for doing the security restricted stuff like reading local databases, writing direct to disk and so on.

There is no deskshellMagic.js yet -- but I like the idea of doing it like I described. We currently can run php scripts via a handler -- if you request a url that ends in .php then pass it to php script interpreter via the command line and deal with post and get etc. This lets it work a little like apache or similar webserver and is useful if you want to use a php library for generating pdf or sending email or similar. Let me know if you are interested and I can send you an example deskshell application with php setup for windows.

/Simon

Paco Zarate

unread,
Mar 15, 2014, 12:55:27 PM3/15/14
to appjs-dev

Yeah it would be awesome!

I have downloaded  the appjs-deskshell project to know more about the current way of doing things and start helping
Is there any other repository I should be aware of ?

Simon Horton

unread,
Mar 15, 2014, 1:57:20 PM3/15/14
to appj...@googlegroups.com
There is the showcase repository also that has various extensions like the file picker and so on.

Kevin Ingwersen

unread,
Mar 20, 2014, 3:38:44 AM3/20/14
to appj...@googlegroups.com
Bingo! :D

Ingwie@Ingwies-Air ~/Work/deskshell/mod_nodejs $ ls -1 out/Debug/
./
../
.ninja_deps
.ninja_log
build.ninja
gen/
gyp-mac-tool
libcares.a
libchrome_zlib.a
libhttp_parser.a
libnodejs.a <----
libopenssl.a
libuv.a
libv8_base.x64.a
libv8_nosnapshot.x64.a
libv8_snapshot.a
mksnapshot.x64
node
node.d
obj/
openssl-cli

There you have it, I now have a fully working nodejs build, that was created off a static library. Now, I can go and extend onto nodejs’ core and rewrite main parts, so that I have an embed API. I am especially looking to finalize my finding with a simple API call as such:
// We do some trickery here to escape CefV8 and go into real v8...
Local<Object> CEFglobal = Context->Global();
node::installEnvironment( CEFglobal );

Now, CEFglobal contains:

CEFglobal
|- process
|- require
|- document
|- …

As  you see, all the node environment functions, libuv bindings and more is now available thru a browser global object. Combined with the existing document object and such, we now have an AMAZING browser environment. Well, we dont really have that now, but will be when I am done with hardcore modding nodejs.

Oh yeah, I should mention my nodejs version: 0.11.13! I am working with the bleeding-edge technology for you guys. :) Nodejs devs also know about my modding work, i have a chance of merging my embed API into master, if I do it good! ^^

So yeah, that’s my current state. I now can savely test multi-threading the nodejs-core, which is mandatory for my ideas.
Hope you like the surprise. :)

Simon Horton

unread,
Mar 22, 2014, 2:50:53 AM3/22/14
to appj...@googlegroups.com

Great! Go for it!!

Kevin Ingwersen

unread,
Mar 25, 2014, 7:26:12 PM3/25/14
to appj...@googlegroups.com
And here is the breakthru i was looking for!!

Ingwie@Ingwies-Air ~/Work/deskshell/mod_nodejs/out/Debug $ ninja
[1/2] CXX obj/src/node.node_main.o
FAILED: c++ -MMD -MF obj/src/node.node_main.o.d -DOPENSSL_NO_SSL2=1 -D_DARWIN_USE_64_BIT_INODE=1 -DNODE_WANT_INTERNALS=1 '-DARCH="x64"' '-DPLATFORM="mac"' '-DNODE_TAG=""' -DDEBUG -D_DEBUG -I../../src -I../../tools/msvs/genfiles -I../../deps/uv/src/ares -I../../deps/v8/include -I../../deps/uv/include -I../../deps/cares/include -I../../deps/zlib -Igen -O0 -gdwarf-2 -mmacosx-version-min=10.5 -arch x86_64 -Wall -Wendif-labels -W -Wno-unused-parameter -fno-rtti -fno-exceptions -fno-threadsafe-statics -fno-strict-aliasing -c ../../src/node_main.cc -o obj/src/node.node_main.o
../../src/node_main.cc:77:14: error: no viable conversion from 'v8::Isolate *' to 'v8::Isolate'
v8::Isolate isl = v8::Isolate::New();
^ ~~~~~~~~~~~~~~~~~~
../../deps/v8/include/v8.h:4194:3: note: candidate constructor not viable: no known conversion from 'v8::Isolate *' to 'const v8::Isolate &' for 1st argument; dereference the argument with *
Isolate(const Isolate&);
^
1 error generated.
ninja: build stopped: subcommand failed.
Ingwie@Ingwies-Air ~/Work/deskshell/mod_nodejs/out/Debug $ ninja
[2/2] LINK node, POSTBUILDS
Ingwie@Ingwies-Air ~/Work/deskshell/mod_nodejs/out/Debug $ ./node
...
Args: ./node
Meep from contextify.
> Ingwie@Ingwies-Air ~/Work/deskshell/mod_nodejs/out/Debug $

This bit of amazingness off my terminal says the following:
I was having an issue with an Isolate…but why did I need to create a new one?

So, I spend the last weeks poking people like crazy. Like, I am almost done with my mod-bomb on the buildsystem that I am contributing to. My fork is now inoffically known as IceTea. As a test, I began creating functions and stuff so I could soon port the whole of PHP5 into this. The great part is, there is also a PHP CPP api - some people wrote a wrapper. Now, this looks ultra-similar to v8:

PHP::Value hello_world(const PHP::Arguments args) {
cout << „Hello world!“ << endl;
}

Well, I excluded the „make it into a extension“ part, but you see how simple this is? Similar is archieved in v8. Therefore, I am very close to enter a new part for Deskshell: We might have the actual and real PHP integrated, not just the minimal ph7!

But, the most important part was to get NodeJS working. And that I did now get.

When Nodejs initializes itself, v8 creates a few things - like the Isolate, something that wraps around the thread-local instance. It was funny that an ex-maintainer of Nodejs was the one replying to me on v8-users, who very precisely told me how to get aorund the problem that I had. Result: nodejs now threads in a non-main thread. This means, that I can now let it idle int he background and let the browser go into the frontend, so that the GUI stuff is satisfied with it’s need to be main’ed.

Whats now missing is my other libnodejs patches, so that I can modify a v8 object and add all the nodejs bindings into it. But thats kind of a peanut now :) So close to have that done.

The current status/todo list looks as follows:

- nodejs library-zation: Half done, need proper bindings
- Porting PHP5 (+PHP-CPP) to IceTea: Half done, need to finalize generate() in order to create config.h
- Creating CEF-based frontend: Not startet, yet. Need to redo some parts, mainly to do with FLTK and native code. Will spawn native windows, but use the rest thru FLTK…
- FLTK bindings to nodejs: Not startet. This will, however, only work in the front end! That is due to the GUI restriction. I need a proper thread-detection.
- Deskshell „WebDev“ specifications: Not started
- Inter-thread communcation module („ChannelDesk“): Not startet, yet. Its very very simple, it will be the last part to be done.
- Creating source tree: Half done, I need to test IceTea on Windows first. :)
- libcanister integration for packaging stuff: Not startet

So far, so good. :) Gonna see how this is coming along later on! But for real, this is an awesome step forward. Phew… ^^

Kevin Ingwersen

unread,
Mar 30, 2014, 7:32:33 PM3/30/14
to appj...@googlegroups.com
I reached a limit, one I can not overcome. The limit of standing inbetween two different dev teams.

I have to drop nodejs - sounds weird? let me elaborate ^^.

So - for the last time, I have tried to port nodejs to work inside and around CEF. But it turns out that this task is harder than it looks. I learned SO MUCH. Friends ask me about a behavior of a browser, and I can answer them off the caff. xD But this knowledge is leading me to a point where I have realized, thats impossible to bring together nodejs and CEF, due to their inner v8 engines evolving over and over and over. So, when CEF updates to, like, 3.28, nodejs will stay at 3.24 for years. Currently, node-v0.10.x uses v8 3.19 - very, very outdated. :p You can check what version you realyl have by doing this:

console.log(process.versions);

But over the time, I have learned a lot about CEF and its api. As much as I do not like how they laid out the wrapper around v8, I have started to understand why they have made it so! In fact, I am now rather glad they did o.o;

Due to them abstracting the API, one does not have to care for any v8::Isolate, v8::HandleScope or even v8::Context. So that leads me to a point where I have decided:

- I will drop nodejs.
- I will port „nodejs“ to CEF.

That means:

- Re-doing the libuv bindings for CEFV8 (further „cV8“).
- Re-implementing the other deps (http_parser, c-ares, zlib(probably replacing with minizip - smaller.)) into cV8.

libuv is what drove nodejs’ event loop, but also gave it some great functions. Like, did you know that you can only use ANSI escape sequences in anodejs app even on windows, because libuv parses that into windows.compatible output? Look for uv_tty_*() :)

But really, that is the only way left that I have to create that kind of binding. So my future plan will drop nodejs, but reimplement its nature into plain cV8. That means we can stay  all the way with the latest CEF - or even bleedingedge builds! But that changes my roadmap and the list of dependencies again. You will notice that it looks snigificantly different - more CEF oriented.

- CEF
- libuv
- http_parser
- c-ares (I might even drop that! Name me an app that needs to async-ly check for DNS stuff?)
- node-cV8 (very small bit that helps re-create the nodejs environment)
- libd0p (for app containers)
- libcanister
- FLTK
- fltk-cV8

This looks tiny for now, but later I am going to add PHP5, so we have a full „like a server“ environment, in a beautyful browser. Sorry nodejs, but you will be living again in cV8 :). Because I can actually re-use some of the original nodejs source in this, as I have learned how to utilize raw v8 in CEF. So in fact, I can escape cV8 completely.

I thought I’d just tell you guys about my current process! :)

Kind regards - and, good night….its 1.32am here, been doing too much research again o.o;

Kevin Ingwersen

unread,
Mar 31, 2014, 5:21:11 PM3/31/14
to appj...@googlegroups.com
Quick message: http://puu.sh/7Re6B.md

Here is my current process! I have done a lot of tests…and this is very much what I am gonna be doing. This will result in so many CEF based resources, that CEF itself might become famous „again“ XD.

tbuers

unread,
Jun 1, 2014, 10:32:56 AM6/1/14
to appj...@googlegroups.com, ingwi...@googlemail.com
Hi,

Just found out about this project since yesterday but the progress sound nice! Looking forward to seeing a running proto!

I might be nitpicking be but it sounds more like a new cefphp project to me.

Also ( I hope you could answer this, since I'm not getting anywhere with limited c knowledge ) I've been reading up something on cef and wondering though if its possible to export existing c/cpp functions to the Window/DOM and also if it's possible to do some sort of thread interaction. If that is possible we ( I'll work on that limited knowledge :D ) could just define a API in which you can manage an CEF window and DOM. That way you can still switch to whatever language you prefer for the "host" environment. Or am I just paraphrasing here?

The package system sounds like a definite must for any sort of application and I like the idea of just a single "tarball"-like application

About the app protocol. Why does the app name need to be the domain name wouldn't just using the app:// suffice?

Op maandag 31 maart 2014 23:21:11 UTC+2 schreef Kevin Ingwersen:

Kevin Ingwersen

unread,
Jun 1, 2014, 1:07:03 PM6/1/14
to tbuers, appj...@googlegroups.com
Hey tbuers!

Thanks for taking interest in the project that I have been working on. But, I must admit, I have not posted a single update in so long here…

Over the time, I have been working on this and I have opened a public github repo, changed and fixed up my backend decisions, etc etc.

As you may have heared, Atom Shell was released; the backend that the Atom.io editor is based upon. However, I was already about to rebuild this, before the public release, before github even decided to deliver it fully as open source; I had reverse engeneered the software and figured out how it worked by far before the release. XD
So I went, and began to rebuild it…and found libchromiumcontent, a project that was quietly started, and is a dynamic library that provides bare access to Google’s components and the bare Content API; the bare-bones of Chrome/Chromium! That, however, provides more low-level knowledge if one wants to use it. And I must admit, I am still learning C++ myself o.o

The „Phoenix Engine“ project however, features some things that Atom shell does not, and can also just be used as a C++ library in general. The estimated size is 40MB…especially when I poked and prodded a copy of libchromiumcontent in order to build a smaller library (WebRTC is so large…).

Currently, I am working on a build system. I need to be able to build this all cross-platform, so I am working on IceTea, the concept of a configurable and scriptable buildsystem across platforms.

The progress I have made so far is, that FLTK, the GUI toolkit I wish to use in the project to render WebKit views, can be built without autotools now. As well as 2/3 of PHP 5.5.11. Some other libraries are just consiting of source files. The primary interface will be a C++ like scripting language called AngelScript. It needs to be fast and reliable, and AS has been used in game engines before - why not use it, to experiment with Phoenix Engine without compiling code?

This, as you realized, is just a kind-of Framework so far. Deskshell is then going to inherit this, and provide a more user-friendly method of using packaged apps and using JS instead of AngelScript to bootstrap things. I still have to concept this out a little. Luckily, V8 uses JIT, so I can count on it’s speed also.

About your CEF related questions. They have an article about hwo to write JS extensions for CEF in C++. The way they do it, is to register handler classes. So each JS function/object must come from a C++ class that derives some Cef-class that I forgot the name of. Then you can register it as either just a function, or an extension that keeps its state across page reloads.

The app:// protocoll was only a mockup and idea to get away from abusing HTTP for this kind of thing, which means that we can use our own handlers. For example, I could use libuv - the I/O library in nodejs - to asynchronously load and display pages. Depending on the file, I could then also just use an archieve reader (libarchive or libcanister thru libttvfs).

Here are some links that might be of interest for you:

Other links you can find as submodules in the Phoenix Engine project.

A small note about the IceTea build tool: It is still in development. I will be writing this over config4cpp, using ph7 as the scripting language. So it will be a „config4IceTea+ph7“ thing…
and for colours, rlutil: https://github.com/tapio/rlutil


This hopefuly answers all your questions and gives some insight in the current status! ^^

Kind regards,
Ingwie!

tbuers

unread,
Jun 1, 2014, 6:03:44 PM6/1/14
to appj...@googlegroups.com, t.b...@gmail.com, ingwi...@googlemail.com
Hey Ingwie,

Yes and it raises a bunch of others. Mainly the reason you choose to start creating icetea instead of using some conventional alternative ( or at least for now to get to a rough proto up. ).

I'll experiment a bit with the libchromiumcontent and the brightray shim libraries. To see if i can somehow contribute.

*PS It seems that the icetea and engine overlap in your repo ( maybe separate the 2? )*




Op zondag 1 juni 2014 19:07:03 UTC+2 schreef Kevin Ingwersen:

IngwiePhoenix

unread,
Jan 31, 2015, 8:42:50 AM1/31/15
to appj...@googlegroups.com, ingwi...@googlemail.com
Oh my god. I havent posted here in so long......i should be ashamed.

Okay, so, what have I been doing, and why was I pretty much dead for half a year as it felt? Easy. I have been learning, studdying and throwing my brain out. :)

First, I have advanced in school, and in the meantime, learned a lot of C++ and cool new PHP stuff. In fact, I have even gone as far aso to studdy NodeJS further and get to the point where I know most of its API from the cuff, and thus using it to build a blazing fast web app. When the app has loaded once, it saves 95% of its data into the user's cache and on further loading, only requires ~10KB of data to be transmitted. Hardcore caching, man. 

But I also continued to work in the C++ area - as such, I have continued work on the PhoenixEngine, the follow-up to AppJS and hopefuly the thing that rounds up Deskshell. After all, Phoenix Engine is hosted int he Deskshell-Core github organization instead of my own account.

But what exactly did I do there? Why did I become so silent? Because I had to solve a main problem.

You see, when you have many modules and libraries and alike, you _will_ end up finding that every of these builds custom-ish. None of them builds on a real standard way. Those that build on CMake suffer from poor code readability in their CMakeLists.txt file, and those using autotools suffer cross-platform abilities - since you first need MinGW, and then the checks may throw false warnings or alike. So I had started my own build tool, forking off an existing tool called "build". The further I went on coding ontop of it, the more I realized how it didn't suit my needs. So I went trough a lot of prototypes - example: I used config4cpp and modified the lexer and parser to execute scripts on the fly - and finally found a soltuion.

The solution's name is ObjectScript. Its a 2 .cpp and about 4 .h files large scripting language with a full JavaScript/Lua-esque syntax, syntactic shuggar from PHP and Python as well as some really neat and unique features. Maybe the C++ API is a bit funny to work with, but it turns out to be worth the troubble. And now, I have an almost complete build tool. A few features are missing and some bugs still exist. Dependency tracking is still a thing, as well as the build executor still being a bit off. For this matter, I am talking to exactly two people - one being the developer of Ninja - and another being from a company - to get this thing rolling.

It already configures and builds a few parts of the engine correctly and works super nice on my system. It works on Windows, Mac and Linux, and has a total binary size of roughly 1MB. Its tiny, fast, awesome and smart.

For instance. You can extend it by dropping scripts into the .IceTea folder within your project. All files with the .it ending are automatically required before the main script runs. Thus, this folder is added to your require paths. So if you wanted to make a set of utilities, you could make a subfolder there called "utils" and name your file "net.os" (.os = ObjectScript), then you could do this:

   require "utils/net"

Awesome, right? :)

But there is more! IceTea is the only build tool that combines a "static" build tool (which means, you have to manually type in your files that should be worked with) and a "dynamic" one. So you can use functions like glob() for a quick testing, and then write it all out properly. It actually goes so far, that you can generate an array as it parses:

var myArr = @{
   
var a=[];
   
for(var k,v in pfs.glob("mydir","*.c"))
        a
[] = v;
   
for(var k,v in pfs.glob("datotherdir","*.cxx"))
        a
[] = v;
    return a;
};

the myArr variable will now have a generated content! Now don't tell me this isn't useful at some places. For instance: The Engine will use this method to decide if it should include Objective-C files or not.

So, that is a short blink into IceTea. You can learn more about it here: https://github.com/IngwiePhoenix/IceTea

Now, IceTea builds me quite a few libraries now. Here is a listing of the output folder after a build I just ran:

$ ls -1 out
FLTK/
libAngelScript
.a
libFLTK
.a
libObjectScript
.a
libTrololo
.a
libcanister
.a
libhttp_parser
.a
libsqlite3
.a
libttvfs
.a

So what do we have here? Those are just a few of the dependencies you will see in the future. Let me go over the ones you see here, as a short reminder. Because some I changed, some I removed, but some I already mentioned in this threads first post.

libAngelScript.a: AngelScript is a _very_ C++ like scripting language. It's aim is to work as best as possible with an existing program as well as new ones, and to be really, really fast. It's the most advanced scripting language I have ever seen. For instance: When doing the native calling convention - when a scripting language calls a native function - it uses actual assembly. Just think about it. The guy behind this language doesnt even work at a company like Google, whose JS engine converts stuff to ASM...and this scripting language supports a wide variety of platforms. From your traditional Windows over to PS3, Xbox 360, Wii, PSVita, iPhone, Android, various flavours of Linux and some very exotic processor architectures. And if the hardware is still not supported, it will automatically fall back into using the AS_MAX_PORTABILITY macro, which runs the native calling convention using a C++ backend, which requires specially signatured functions, just like you'd expect in v8, PHP or anywhere else.

libFLTK.a: Fast Light Toolkit. This thing will save me almost all of my GUI work, since it will do it for me. It runs on X11, GTK, Cocoa and Windows. In other words: Almost everywhere where you'd speak of a "computer with a common desktop environment". Such as most famous Linux distros, OS X and Windows. It also runs on the Raspberry Pi.

libObjectScript.a: This is the scripting language that IceTea uses and which is so tiny, that I'll just include it. You can use it as a tool language since its really small. It also is ment to be used for website development. So if you need a light alternative to PHP, OS can serve you too. Its mainly included because I have a MVC framework for it in the works and because the language has some really neat features.

libTrololo.a: (Nope, I am totally not oding a reference here...) This is a C++ wrapper about SQLite3. I might replace it with SoCi.

libcanister.a: A custom file archive structure. It's uncompressed but does some great archiving. Since I am going to use the ZPAQ algorythm for compression and other things, I might have just found a really smooth combination of libraries. :) Because: libcanister will allow me to attach ZPAQ as a compressor. So you get a cool C++ API plus maximum compression. useful, when your app will download huge data. A tool will be created to use this mechanism outside of the Phoenix Engine.

libhttp_parser.a: First part of the NodeJS core parts that I will be using to rebuild the NodeJS API inside of Phoenix Engine. Missing are C-Ares and libuv.

libsqlite3.a: Should be obvious. I am not sure, but I think Chromium has that already inside, which would make this copy redundant. I am looking for an encryptable DB anyway, for when I am allowing source code for Phoenix Engine apps to be embedded into a binary, so that you can encrypt pretty much everything.

libttvfs.a: A file system abstraction, that allows me to treat archive files as if they were regular folders. This also means, that you will be able to do the same under a special ttvfs object. This will also allow you to issue requests INTO archives. Imagine you're downloading game data for your app as a zip. You don't need to extract the zip, you can just access it. This also will go for the canister/ZPAQ mechanism.

Now there are a few dependencies missing - such as PHP, some special PHP extensions, some special AngelScript extensions, and of course for ObjectScript some enhancing extensions. Then there is libuv, c-ares, curl and re2c, which is required for PHP's source builds. Although I dont want to change any .re file, which may mean that I can drop this utility.

What I am currently not adding is the ZPAQ part, as well as libchromiumcontents and Brightray, on which most of my work will be based off. Why Brightray? It is what Atom Shell offers to use, and it is simply a shim over chromium contents and will make working with a few things much easier.

So yeah. I am working on making all the relevant parts build using IceTea, while improving it int he process to become a really awesome,s tand alone build tool for everyone. I am also looking into features such as:

- Encrypting and embedding code into a binary to create commercial apps based off Phoenix Engine.
- Encrypted Database support
- Native extensions to the engine itself (i.e.: Add Python scripting support)
- Seeing if I can use Appcelerators old Kroll micro kernel to combine the power of script languages
- Creating a general IPC based protocol so that all the different processes and script languages can talk to one another
- Adding first-class debug support.

That are the current updates. you can look at https://github.com/Deskshell-Core/PhoenixEngine and lurk into the .IceTea folder to see what I am working on.

You can also download and compile IceTea and test it out yourself. Or contribute by forking it and issuing a pull request. Here is a short rundown of all the features that are currently broken or missing:

- Select various targets using a tag() function.
- Split the build executor into a script runner on the main thread, and command runner in the background on multiple threads. That way, the scripting language can generate more commands while the others are being processed - and it only needs to run on a single thread with no inter-thread interruption. Should also be faster. I'd like to implement this based off the stlplus::async_subprocess class and my tinythread based threadpool implementation (threading.h)
- Proper task sorting so that we have a proper DAG and topological order of the targets. This includes dependency tracking.
- Make rules depend on a target. (I.e.: the rule to process .re files first needs the target "re2c" built before it can run)
- Dry-run mode
- Better status display.

About the build executor. I am thinking of actually first letting the script language generate all the tasks that should be run, and THEN running the command queue, so that we have a "n done/n total" display like Ninja. Its complicated...since IceTea is different than Ninja, but wants some of its features.

Again, if you want to help, let me know, fork IceTea and send back a fix or two or whatever yout hink will be helpful!


Kind regards, Ingwie

Paco Zarate

unread,
Feb 1, 2015, 9:55:55 PM2/1/15
to appjs-dev
Hey Ingwie,
Great to see you publishing news on the project!

Paco
It is loading more messages.
0 new messages