Is it possible to run FLTK headless?

[ed...@tantec.de]

I wonder about the effort to run FLTK on a system without graphic card,
but I really don't have a clue, how this could be realised.

I continued to think about peoples demand on web interface and would
like to make all FLTK software accessible by network client.

Concept:

The FLTK software runs on a server and when there is a connection, the
client opens windows of the same size as displayed on the server and
copies the windows content from the server to the client. The client
input (e.g. keyboard, mouse) to the remote windows will be sent back to
the server and be injected to the event loop.

It seems easy to me, to create a proof of concept, using e.g. Fl_Surface
from draw(), Fl_Event_Dispatch for copy of events and web stuff out of
the box, but does it make sense to generate just another vnc?

I think, the project would make sense, when there is the benefit to
connect to a FLTK software on a headless unit. The server should start
the FLTK software with a virtual graphic context, when there is an
incoming network connection and then commit the connection to the software.

The client would be a thin client and doesn't need to be "installed",
but can be executed from the desktop. Okay, it will stay to be system
dependend, but also a browser is system dependend software. But in
comparison to a vnc solution there is no system integration necessary,
it is just a software to software communication, so it should work fast
and reliable.

What do you think?

[Greg Ercolano]

On 12/09/16 10:45, ed...@tantec.de wrote:
> I wonder about the effort to run FLTK on a system without graphic card,
> but I really don't have a clue, how this could be realised.

What you're describing sounds a lot like X windows?

That's provided this since the 1980's.

The whole idea of X windows is that the program can run on a different
machine from the one with the display/keyboard/mouse.

The geometry/images are written over the network from
server -> client via the X windows protocol, and keyboard/mouse events
go the other direction from client -> server.

This feature is still there in all linux and even OSX's X server,
but it's used kinda rarely in my experience, because remote desktop
programs (like TeamViewer) have filled the void.

There was a time when you could purchase X windows "terminals",
diskless machines that boot straight into an X server. The "server"
would let several of these terminals be active, providing separate
desktops to each one. All programs/desktop tools would run on the
server. The workstation would only be used as a display.

[ed...@tantec.de]

Greg Ercolano schrieb:

> On 12/09/16 10:45, ed...@tantec.de wrote:

>> I wonder about the effort to run FLTK on a system without graphic card,
>> but I really don't have a clue, how this could be realised.
>

> What you're describing sounds a lot like X windows?

Yes, I agree. I would like to make all my FLTK software able to work
like X windows, Fast Light Web Interface. ;o)

It seems to me, a lot of work was spent to FLTK system interfaces,
classes like Fl_Surface and Fl_Device.

When I look to "31.22 Fl Device Class Reference" it seems to me, that
all FLTK output goes to Fl_Surface and is connected by Fl_Device to the
Fl_Graphics_Driver. If I write a Fl_Network_Graphics_Driver, that
writes the drawing orders to the FLTK X server on remote machine (which
executes the drawing orders on local hardware), what more would I need
to start a FLTK software on a headless machine? Some setup of the main
loop, to insert the remote inputs received from network?

Also Fl_Device misses functions to set up the window positions and so I
may need to balance the positions of the remote windows by a local
window mangager. I think, we had issues like this before, it is a kind
of porting to a special machine. Maybe someone remembers articles or
threads regarding to this? Any hints?

When not too hard to realise, this would be a very interesting project,
because a lot of my software works on old XP machines in the basement,
e.g. burning in lamps. One client for all FLTK software in the web would
be a powerful thing, even when it does not work with a web browser.

[Albrecht Schlosser]

On 11.12.2016 at 17:46 ed_09@... wrote:
> Greg Ercolano schrieb:

>> On 12/09/16 10:45, ed_09@... wrote:
>>> I wonder about the effort to run FLTK on a system without graphic card,
>>> but I really don't have a clue, how this could be realised.
>>
>> What you're describing sounds a lot like X windows?
>
> Yes, I agree. I would like to make all my FLTK software able to work
> like X windows, Fast Light Web Interface. ;o)
>
> It seems to me, a lot of work was spent to FLTK system interfaces,
> classes like Fl_Surface and Fl_Device.
>
> When I look to "31.22 Fl Device Class Reference" it seems to me, that
> all FLTK output goes to Fl_Surface and is connected by Fl_Device to the
> Fl_Graphics_Driver. If I write a Fl_Network_Graphics_Driver, that
> writes the drawing orders to the FLTK X server on remote machine (which
> executes the drawing orders on local hardware), what more would I need
> to start a FLTK software on a headless machine? Some setup of the main
> loop, to insert the remote inputs received from network?

It's not only remote "inputs" if I take this to mean keyboard and mouse.
It's also remote drawing requests (resize, occluding and exposing areas
of the display), and maybe a lot more.

> Also Fl_Device misses functions to set up the window positions and so I
> may need to balance the positions of the remote windows by a local
> window mangager. I think, we had issues like this before, it is a kind
> of porting to a special machine. Maybe someone remembers articles or
> threads regarding to this? Any hints?

I don't remember such articles, and I don't really understand what
exactly you mean. A window manager on the client side should always be
available, and the server would certainly need some kind of "virtual
display", maybe all in one Fl_Surface_* device or something like that.
However, that would also mean (AFAICT) to rewrite the server side to
display everything in the "Fl_Network_Surface (?)" device and so on. I
don't know, it's just too complicated to think about it...

> When not too hard to realise, this would be a very interesting project,
> because a lot of my software works on old XP machines in the basement,
> e.g. burning in lamps. One client for all FLTK software in the web would
> be a powerful thing, even when it does not work with a web browser.

I have implemented a similar project, where the server side of the
project is NOT implemented with FLTK. It does not even run on a server
that would be able to run FLTK (the server currently uses OpenVMS,
porting to Linux is in progress). However, it's a commercial project, so
I can't provide any code. The project was started with the intent to
have a light client that runs on popular hardware and OS's (Windows,
Linux, MacOS), so FLTK was chosen to implement the client side. There
are servers that support several hundred clients at the same time.

To give you a clue how it works: The server has some kind of forms
generator, something you could compare with fluid. The current version
doesn't have a GUI yet, but this may be a future extension (implemented
with FLTK, of course). The forms generator can define windows, buttons,
groups and many more special widgets. The server communicates with the
client over TCP, optional with SSL support. The communication protocol
(over TCP/SSL) is, of course, proprietary, but you can imagine that the
server sends some code like a fluid (.fl) file (forms data), together
with the user data to "fill" the widgets, to the client. The user data
is maintained in a database on the server.

The client is implemented in FLTK. It interprets the "forms" data,
generates FLTK windows and all widgets included in these windows on the
fly (!), and lets the user interact with the windows and widgets. There
are several actions that let the client send data back to the server,
for instance button click, window close and some more events (e.g.
double-click on Fl_Browser items). Everything else like text input,
mouse movement, window resizing etc. is handled locally in the client,
hence it is really fast because no server interaction is needed (no
round trip delay).

Besides the flexible and fast client the advantage is that software
modifications can mostly be done on the server side. Only new (client)
features need a client software update, which is rare.

Well, I don't know if this is helpful for your thoughts. It's really
different than writing a FLTK application on the server and "mirroring"
everything to a client, but maybe you got some hints. Or maybe not ;-)

[Ian MacArthur]

I don't remember such articles, and I don't really understand what exactly you mean. A window manager on the client side should always be available, and the server would certainly need some kind of "virtual display", maybe all in one Fl_Surface_* device or something like that. However, that would also mean (AFAICT) to rewrite the server side to display everything in the "Fl_Network_Surface (?)" device and so on. I don't know, it's just too complicated to think about it...

I wonder if Ed is thinking of a platform that doesn’t have X11 or a WM though?

I know I had fltk running on vxWorks way back when, with no WM and a very basic shim to simulate X11, but we ran it as a “kiosk” fullscreen, so that was fine. I did hack up a version of aewm (and of flwm) to run as WM’s on that system too, just so we could get floating windows for some hack tests, but we never deployed it like that.

FWIW, wherever I’ve had X11, it has Just Worked.

For a long time I ran one of my R.Pi’s headless, using the X-server support in OSX as the display, and it always just worked. Indeed, there were some odd corner cases where it worked “better” than the Pi - XGL support in early Pi’s was poor (still not great) and depended heavily on Mesa to work at all (so very sloooow).

The same code running on the Pi but displaying on the OSX server was Much Better, as the XGL support on OSX was way better and faster, even when being served over the wire.

FWIW: Also used this to run the a Pi from Win7 using Xming as the X-server; that also worked pretty well. Xming is a pretty decent Server these days!

Dunno if any of this is useful to Ed though!

[Greg Ercolano]

On 12/11/16 14:21, Ian MacArthur wrote:
> For a long time I ran one of my R.Pi’s headless, using the X-server support
> in OSX as the display, and it always just worked. Indeed, there were some
> odd corner cases where it worked “better” than the Pi - XGL support in
> early Pi’s was poor (still not great) and depended heavily on Mesa
> to work at all (so very sloooow).
>
> The same code running on the Pi but displaying on the OSX server was
> Much Better, as the XGL support on OSX was way better and faster,
> even when being served over the wire.
>
> FWIW: Also used this to run the a Pi from Win7 using Xming as the
> X-server; that also worked pretty well. Xming is a pretty decent
> Server these days!

Ya, I've had good luck with the X server that comes with Mac OSX
for my fltk apps.

X windows seems like a good way to do what Ed is talking about,
rather than having to reinvent that wheel.. but it depends on the
application I suppose.

I have nothing to add regarding using the Fl_Device_Xxx to implement
graphics drawing and event flow.

Ed, you might want to look at the X protocol to see what you're in for;
basically you'd want bidrectional TCP to move events in one direction,
and send shapes/pixels in the other direction.

[Ian MacArthur]

On Sun Dec 11 2016 23:34:57, Greg Ercolano wrote:
>
> Ya, I've had good luck with the X server that comes with Mac OSX
> for my fltk apps.
>
> X windows seems like a good way to do what Ed is talking about,
> rather than having to reinvent that wheel.. but it depends on the
> application I suppose.
>
> I have nothing to add regarding using the Fl_Device_Xxx to implement
> graphics drawing and event flow.
>
> Ed, you might want to look at the X protocol to see what you're in for;
> basically you'd want bidrectional TCP to move events in one direction,
> and send shapes/pixels in the other direction.

Yes - I’m not clear what Ed’s use case is here, but if the display target is a “standard” target, i.e. a PC running WinXX, OSX or Linux, then setting up an X-server might be the easy way to go.
Both Linux and OSX provide an X-server built-in, in effect, and Xming on Windows is easy to set up and very effective.

This does rather assume that the headless source of the app also knows the X protocol of course, but I’m envisaging some kind of simple headless “embedded” target (since that’s what I often have, I guess it’s what I assume everyone else has; I acknowledge that may not be a valid assumption!)

If the client end is running embedded Linux (say) then the X protocol may be readily available.

Beyond that, I have had good results with vnc in the past, too, which is also widely supported.

[chris]

Am 11.12.2016 um 18:50 schrieb Albrecht Schlosser:
> To give you a clue how it works: The server has some kind of forms
> generator, something you could compare with fluid. The current version
> doesn't have a GUI yet, but this may be a future extension (implemented
> with FLTK, of course). The forms generator can define windows, buttons,
> groups and many more special widgets. The server communicates with the
> client over TCP, optional with SSL support. The communication protocol
> (over TCP/SSL) is, of course, proprietary, but you can imagine that the
> server sends some code like a fluid (.fl) file (forms data), together
> with the user data to "fill" the widgets, to the client. The user data
> is maintained in a database on the server.

Interesting concept..

Just out of curiosity, may I ask if there is caching (of forms or data)
needed on the client side or is the payload small enough to be (re-)sent
every time? What about images?

[Edzard Egberts]

Albrecht Schlosser wrote:
> To give you a clue how it works:

Yes, this is a standard concept of web interface.

> The server has some kind of forms
> generator, something you could compare with fluid.

Yes, with fluid you define "forms", the RAM representation of the GUI
you need on the headless machine to interface with the GUI. At first
side a FLTK window is a structure in RAM, which defines a rectangle with
a position and some drawing orders to the rectangle. On the other side
there is the system window which draws the RAM structure to the GUI,
using orders like draw(char*, x, y), line(x1,y1,x2,y2) and so on (same
orders as delivered to FL_Graphic_Device). The protocol would just pack
the FLTK orders to a frame and send them.

So I understand the FLTK program as a RAM structure, that writes drawing
orders to the GUI and this is the bottleneck, to cut the software into a
RAM- and a GUI-piece. The FLTK RAM representation sends it's drawing
orders to the client, that creates and manages the GUI windows, where
the drawing orders are applied to.

But I'm missing a class of orders, because FLTK also manages position
and number of windows. I think for headless usage there also must be a
RAM representation of window positions, something to replace the system
window representation of Fl_Window. Should take a more closer look to
Fl_Window, I think...

> The client is implemented in FLTK. It interprets the "forms" data,
> generates FLTK windows and all widgets included in these windows on the
> fly (!), and lets the user interact with the windows and widgets. There
> are several actions that let the client send data back to the server,
> for instance button click, window close and some more events (e.g.
> double-click on Fl_Browser items). Everything else like text input,
> mouse movement, window resizing etc. is handled locally in the client,
> hence it is really fast because no server interaction is needed (no
> round trip delay).

Yes, the round trip delay is a disadvantage of my concept, but mouse
movement and windows resizing also would be handled without round trip.

> Besides the flexible and fast client the advantage is that software
> modifications can mostly be done on the server side.

Yes, this is the main thought - the client can stay simple, like a vnc,
that doesn't need to be adapted to software it shows.

> It's really
> different than writing a FLTK application on the server and "mirroring"
> everything to a client

It is not that different. My concept would use more ressources on server
side (but I think the footprint of FLTK classes is not so much) and
connection bandwidth. But it also would go deeper into FLTK than a VNC
copy of local drawing, because just the drawing orders would be
generated on server side, the drawing itself would happen on client
side. The real advantage of the concept would be, that it might enable a
lot of existing FLTK applications to be operated by web client.

What would I need beside of writing a Fl_Network_Graphics_Driver, trying
to patch the system window information of Fl_Window and accessing the
event loop by Fl_Event_Dispatch? I would like to know, which parts of
FLTK connect the RAM representation to the system hardware and how to
disconnect.

Anyway, I'm not sure, if it makes sense to realise this concept, but I
would like to understand, how much the effort of a general solution
would be.

[Albrecht Schlosser]

Yes, no problem, here are the answers:

We thought about caching forms data, but this is somewhat difficult to
do because the forms are far from static. The server program can and
does change many widget attributes like colors, activate/deactivate
widgets (buttons and more) after loading the forms data from the
database. Obviously we could send these changes to the client, but it's
not worth the effort. Forms data for one transmission is never more than
40-50 KB and the protocol uses (zip) compression. Since forms data
includes much text it can be compressed very well.

User data is never cached, but there are some other attributes like the
choices of something like Fl_Input_Choice widgets that are downloaded
only when requested and then cached.

The total payload of each transmission can be max. about 1 MB (may be
extended), but compression reduces this to usually 10-20% of the
original size. Most clients use local networks where this transfer size
is absolutely neglectible, others that use the internet are still quick
enough. Remember that most of the time a user needs to complete his/her
task is done locally on the client side. Only after filling a "form"
(window) the user clicks a button, sends his input back and gets a new
window full of data. Of course it's not always only one window, there
can be several (modal) windows as well.

The largest data size is when the user generates a report and downloads
it to the client. If the data exceeds the transfer limits, then it is
sent in several chunks, so this can take some time. Such a report can be
text-only or for instance a pdf file and can be viewed, printed, and
stored on the client computer by the user. Text data can be viewed in
the client in Fl_Text_Display widgets, free text input is done in
Fl_Input and Fl_Text_Editor widgets.

We don't use many images currently, although the application can
download Tiff images for viewing. Besides that there are also some
drawing primitives that can draw a graph in a client widget. Many of
FLTK's fl_(draw_)something() methods (lines, rectangles etc.) are
implemented for special purpose graphics.

We also use icons, primarily for buttons, but these icons are predefined
in (i.e. compiled into) the client. The server sends only their names so
this is never a size problem. A future extension would be to send icons
(i.e. small, supposedly png images) to the client so that these icons
can be generated on the server side, but that's not yet implemented. If
it were implemented, these icons would also be cached by the client.

[chris]

Thanks for this detailed answer.
It gives me the impression of a very well designed "system" - wow!

I can envision the big advantages of keeping the business logic and even
the screen layouts at the server side, keeping the client software
simple, small and maintainable. And all this without complex scripting
and browser access (~HTML5). And the client software still has native
access to the hardware e.g. printer.

<philosopher mode>
So this may be another data point to the question asked in the thread
over there about the future of desktop applications: well designed
special purpose systems will still have a future and FLTK has the
potential to be part in such systems because of its platform independency.
</philosopher mode>

[Ian MacArthur]

All,

Can I just say that I was reading this, and became somewhat “disoriented” - it seems to be using the terms “client” and “server” the opposite way around to what I expect, at least thinking about the problem in terms of X11.

Or maybe I just didn’t understand. (Which is equally likely, of course!)


Under X, the “common” program that handles the actual display is the “server", the program that does the app logic, but does not have a display of its own, is the “client”.

If I understand Albrecht’s description, that appears to be the opposite sense? Maybe...

[Albrecht Schlosser]

Yes, it was, if you were thinking "X".

The point is that we discussed (AFAICT and as far as I'm concerned) on a
more theoretic base, and particularly my description is about a
client/server structure. Especially my term 'client' referred to the
client program like a browser (as opposed to a web server).

PS: I always found the X speech confusing and I like to emphasize
"_display_ server" instead of only "server". However, in my case the
client program is the remote access tool and the server runs the
application program. It just happens by design (not by accident) that
this "thin client" also does the display and input task...

[Greg Ercolano]

On 12/12/16 10:36, Ian MacArthur wrote:
> it seems to be using the terms “client” and “server” the opposite way around
> to what I expect, at least thinking about the problem in terms of X11.
>
> Or maybe I just didn’t understand. (Which is equally likely, of course!)
>
> Under X, the “common” program that handles the actual display is the “server",
> the program that does the app logic, but does not have a display of its own, is the “client”.

Yeah, that might be me.. the OP used "thin client" (the X display/kvm),
so framed in that context, the display is the client and the machine
running the app is the server.

So I was using his terminology when I said this in my 12/09 email:

"The geometry/images are written over the network from

[app] server -> [thin] client via the X windows protocol, and

keyboard/mouse events go the other direction from

[thin] client -> [app] server.

I know when I'm talking broadly about client/server,
I use "server" and "client" in the context of a file server
running the apps with multiple clients connecting to it.
It's the common context most people think of.

But yes, when talking about the actual X protocol, the
display/keyboard/mouse is the X server (the network listener)
and the app is running on the file server, but is actually
the network client running the Xlib client library.

Regarding thin clients, diskless X terminals of the 1990's
never seemed to catch on. I saw one once and used it briefly,
but it was slow because the server was under load, probably
a common problem. Few servers of the 1990's could handle
running multiple apps very efficiently because servers
had slow cpus, limited memory/disk, and were easily overloaded
just running the C compiler, let alone GUI apps..

[Brian Davis]

NVIDIA GRID http://www.nvidia.com/object/grid-technology.html can be run from a web browser form Mac, PC, Android, iOS, etc but the server arch backend is $$$.  Allows Virtual Desktops to be graphically accelerated and run through a web browser or remote desktop (Win7/10 Enterprise + RemoteFX + Windows Server 2008/2010 R2 flavors for the windows minded using K10/20 cards).  This basically allows rendering to a render buffer on the GPU and MPEG / Video streaming to client with feedback for mouse/keyboard events as I understand it.  You might be reinventing the wheel here, but as I see it there is no democratization of GRID as yet.  Maybe ATI will come out with something to compete.

It would be great if docker would allow access to the gpu... Oh hey look it does:

https://devblogs.nvidia.com/parallelforall/nvidia-docker-gpu-server-application-deployment-made-easy/


Direct rendering Infrastructure: https://en.wikipedia.org/wiki/Direct_Rendering_Infrastructure
Where it seems docker containers can be run in privileged mode allwoign access to DRI.

Can this be done? GRID on the cheep with DRI and docker containers? Who knows... 

My 2cents.

[ed...@tantec.de]

Merry Christmas and a Happy New Year! :o)

Okay, I'm a little late, but here comes the sequel of "Is it possible to
run FLTK headless?". I wanted to provide some code "after Christmas",
but it got some more code and a little later.

http://www.edzeg.net/Fl_Remote