WebSockets for Lua (open discussion)

[bel]

WebSockets are a very useful feature in civetweb. You can use it to establish a connection to the server and let a the browser continuously exchange data with the server. The server can send data to the browser, the client does not need to poll repeatedly.

The Lua integration is yet another very useful feature in civetweb. A powerful, fast and extendible extendable scripting language with a small footprint available as in-server module, faster than PHP trough CGI, suitable as a prototyping language (for C/C++ programmers) as well as as a main language.

Too bad that these two features can not be combined at the moment: Websockets are currently only available for C/C++ programmers, not for Lua. Even if you finally want to use websockets from the C application, a quick Lua prototype can be very useful.
I think I should make this possible.

However, there are a view design considerations that will directly influence the way this feature can be used, so I will post them here in the next days. Maybe we can get a discussion similar to the one of the threading model, however I will meanwhile implement and study some prototypes.
I hope there are no objections in principle to providing websockets for Lua?

[Thomas Davis]

Sounds like a natural extension. :)

--
Sourceforge
https://sourceforge.net/projects/civetweb/
---
You received this message because you are subscribed to the Google Groups "civetweb" group.
To unsubscribe from this group and stop receiving emails from it, send an email to civetweb+u...@googlegroups.com.
To post to this group, send email to cive...@googlegroups.com.
Visit this group at http://groups.google.com/group/civetweb.
To view this discussion on the web visit https://groups.google.com/d/msgid/civetweb/1d497b0b-6327-411d-ba1a-a26f51aba7d4%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[bel]

Design consideration 1:
How to translate the script path?
-----------------------------------------------

Websocket actually has its own root hierarchy. A websocket address is typically opened in a way like (JavaScript code):

websocket_connection = new WebSocket("ws://" + window.location.host + "/MyPath/ToSomeWhere");

So you replace http:// (or https://) by ws://, add the IP and have your resource path.


Alternatives:
1a) You could use the same document root path on the server side for http:// and ws://
In this case Websocket Lua scripts must have a specific file extention, and http and https requests should not display these files (you dont want to show the server side Lua source to the clients in the browser).

1b) You could use a different document root path on the server side, which means a new config of an additional path.

1c) You could use just one Lua script for all websocket requests. This means a new config of an additional file.
The actual websock resource path is passed on to the Lua script file. Then you can still decide in Lua to split to load and invoce an additional script.
This is both, the solution that is most simple to implement and that offers the highest flexibility.


So I would go for "1c".
Any objections?

[bel]

Design consideration 2:
C callbacks and Lua callbacks
---------------------------------------------

Currently there are 4 callbacks in C that you can use to implement websocket support:
- websocket_connect        ... called when the client opens the ws://... address before the websocket handshake, can be used to reject the connection
- websocket_ready          ... called after the websocket handshake
- websocket_data            ... called if the client (= browser) sends data
- close_connection         ... called if the client or server closes the websocket

Of course we should not break existing C code, so these callbacks must work also after this extension - at least if there is no Lua page available that will handle the websocket request.

I suggest the following strategy:

Step 1) allways call the C websocket_connect, independent of Lua. If it rejects the connection it is rejected. Lua will never be informed.

Step 2) if there is a Lua page, load it - notify it by a callback equivalent to the C connect, so it might reject the connection

Step 3) do the websocket handshake

Step 4) if there is Lua a page call the "ready" function of Lua, but not the one of C
if there is no Lua page call websocket_ready from C (not Lua)

Step 5+ .. always when the client sends data)
if there is Lua page call the "data" function of Lua, but not the one of C
if there is no Lua page call websocket_data from C (not Lua)

Final Step -1) if there is a Lua page, call the "close" callback of Lua

Final Step) always call the "close" function of C

[bel]

Design consideration 3:
Relation between Lua state and webserver connection
------------------------------------------------------------------------------

After two topics for warming up, now the first tricky ones:
When should the Lua script be loaded and unloaded? How should two webserver connections communicate with each other? Should connections share one Lua state?

3a) The "naive" approach is to handle a websocket connection similar to the normal web page:
Load it when at "websocket_connect", call the "ready" callback, then call the "data" callback whenever the browser sends data.
Finally call the "close" callback and unload the page (i.e., close the Lua state).

This means, that two Lua websocket connections need to use the database to communicate which each other. They have to open the same database and use it simultaneously.
We are using SQLite in the "serialized" threding model, so you can use multiple independent connections to the same database, but you need to handle the "Database is busy" return code for every call to SQLite, so your Lua code will look something like this:

repeat
  r = db:exec(mySQLStatement)
until r ~= 5

What I am doing here is to retry to execute mySQLStatement until the database returns something else than "I am currently busy from another thread", aka 5.
Not really nice - it's an extra programming complication and it yields in busy waits.

One could add a serializing mutex in the lsqlite library in C, but I think this should not be done, because (a) one should not change a third party library to maintain maintainability, and (b) this serialization is only required if you are working with two different database connections to the same database from two different threads, so you will slow down connections to different databases, although there is absolutely no need to do so.
One could implement a mutex library for Lua and let the Lua user lock the database. Certainly doable, but again it makes the programming model in Lua more complicated - a mutex is nothing really natural to a Lua programmer.

It becomes even more complicated, if you want to send unasked data from the server to the browser. Consider a simple chat between two websocket connections. Lua-Websocket 1 receives a message from browser 1 and wants to send it via websocket connection 2 to browser 2 - this requires yet an additional set of Lua functions ... still doable.
However, if you consider a server which sends the current time (or any other server local data) to all connected websockets, you cannot realize it reasonably with this model.

So Model 3a - "one Lua state per websock connection" has several drawbacks.
The other extreme alternative 3b would be "one Lua state for all websocket connections", and 3c is a hybrid solution.

[bel]

So Model 3a - "one Lua state per websock connection" has several drawbacks.
The other extreme alternative 3b would be "one Lua state for all websocket connections", and 3c is a hybrid solution.

3b - one single Lua state for the entire webserver:

The implementation is rather simple in this case: Create a Lua state with a given script when the webserver is started, forward all callbacks to Lua (instead of C), destroy the Lua state when the webserver is closed.
Since a Lua state can only be used by one operating system thread at a time (don't get this mixed up with Lua threads which are not operating system threads but multiple coroutines within one operating system thread). So all websocket worker tasks that want to perform Lua websocket operations need to be serialized with a mutex. This introduces a bottleneck if a server has multiple types of unrelated websockets that could easily run in parallel. Furthermore one needs one extra thread which periodically calls a callback function within the Lua state. This is required to give the server a chance to send unasked data to the clients, like the current time server already mentioned in 3a.

[bel]

As a variation of 3b one could also let Lua wait for requests instead of using callbacks and a cyclic call. Still you will create the Lua state when the server starts, and create an extra thread that calls it immediately.
The Lua programm will contain the wait function (this is possible since Lua 5.2):

repeat
   request = WaitForNewWebSockRequestOrTimeout(100)
   -- request is a table that will contain all relevant information
   if request.type == "ready" then  
      HandleReadyRequest(request)
   end
   ...
   SendCurrentTimeToAllWebsockConnections()
until request.type == "exit"

the C workers will communicate via mess queues with Lua.

Significantly more work to implement, more complex to understand and maintain in C. The cyclic call from outside will not be required, since Lua can handle this with the timeout.
For the Lua application probably not really more convenient as compared to just declaring global functions that will be called by the framework like in the above cases.

[bel]

So Model 3a - "one Lua state per websock connection" has several drawbacks.
The other extreme alternative 3b would be "one Lua state for all websocket connections", and 3c is a hybrid solution.

3c) the hybrid solution means that there are some independent Lua states.

While 3a has one state per connection, but a complex communication between them, 3b has just one state (so no need for communication) but it does not allow to have two really independent websocket implementations at the same webserver.
So solution 3c would be to use multiple Lua states and use the websocket address (see design consideration 1) to distinguish what state should be used. The single file approach (1c) needs to be extended for that.

[Ed Swift]

On Thursday, October 31, 2013 6:12:57 PM UTC-4, bel wrote:

Design consideration 1:
How to translate the script path?
-----------------------------------------------

Websocket actually has its own root hierarchy. A websocket address is typically opened in a way like (JavaScript code):

websocket_connection = new WebSocket("ws://" + window.location.host + "/MyPath/ToSomeWhere");

So you replace http:// (or https://) by ws://, add the IP and have your resource path.


Alternatives:
1a) You could use the same document root path on the server side for http:// and ws://
In this case Websocket Lua scripts must have a specific file extention,

Would they need a specific extension? One minor gripe I have with mongoose (and CW, I assume; haven't tried it yet) is that you can't have a .lua file handle a request directly, you have to run it though a .lp file. Couldn't .lua files be served up through Lua instead of as plain text, and then we could use the same -url_rewrite_patterns to point websocket requests at the appropriate file (inside or outside of the document root)?

 

and http and https requests should not display these files (you dont want to show the server side Lua source to the clients in the browser). 

If they're inside the document root, -hide_files_patterns should cover that, shouldn't it?

It seems like this would offer a lot of flexibility without needing any new config, but I may be missing something. I guess you'd want to provide the URI scheme to Lua, so the script can check that it's actually a websocket request, but I feel like it should be there anyway. Right now there is no way to distinguish between http and https on regular Lua pages as far as I know.

 

1b) You could use a different document root path on the server side, which means a new config of an additional path.

How would you map the resource path to the Lua files, though? People probably don't want their resource paths ending with ".lua".

 

1c) You could use just one Lua script for all websocket requests. This means a new config of an additional file.
The actual websock resource path is passed on to the Lua script file. Then you can still decide in Lua to split to load and invoce an additional script.
This is both, the solution that is most simple to implement and that offers the highest flexibility.


So I would go for "1c".
Any objections?

 

That should work, but it would mean you'd always need a sort of "router script" if you're using multiple resource paths, and it seems like that routing could be handled by -url_rewrite_patterns and organization of scripts within the filesystem instead. Something feels off to me about having a single entrypoint for all websocket requests, but I can't put my finger on it.

Maybe using the rewrite patterns for both http and websocket requests is too weird, though? It complicates cases where you need the websocket resource to have the same path as a document resource for some reason, especially requests with no path.

I'm sure you'll figure out the best way to handle it, just throwing this out there as food for thought.

[bel]

Ed Swift:

Would they need a specific extension?

If we do not give them a separate document root (1a), they will need a specific extention - I would use *.lws (Lua WebSocket).
If we give them a separate root (1b) or just use one file (1c), we are free.


Ed Swift:

One minor gripe I have with mongoose (and CW, I assume; haven't tried it yet) is that you can't have a .lua file handle a request directly, you have to run it though a .lp file. Couldn't .lua files be served up through Lua instead of as plain text, ...

Well, it would be simple to add support for using lua files directly instead through the .lp files. It is also possible to use Lua through CGI by using the Lua standalone interpreter as CGI interpreter. An example how lp and CGI are related is currently here
https://github.com/sunsetbrew/civetweb/tree/master/test/ajax
An additional possibility to use Lua directly, similar to the CGI but as in-server module could easily be added. In fact I thought about that, since it is certainly simpler to program in Lua.
Its not a big deal to implement in the webserver and not a real complication to maintain, yet it would be a third way to invoke Lua - still I do see some advantages .. where advantages would it have for you?

... this was all about "normal" web pages here, not about websockets.


Ed Swift:

... and then we could use the same -url_rewrite_patterns to point websocket requests at the appropriate file (inside or outside of the document root)?

I haven't used url_rewrite_patterns myself, but I think it will still not serve the purpose here:
It would have to distinguish between http(s)://server.xy/file.lws and ws://server.xy/file.lws - the first one must not find the websocket script "file.lws" at all, the second one must do the websocket operation.


Ed Swift:

If they're inside the document root, -hide_files_patterns should cover that, shouldn't it?

Well, one should not have to configure it. But in principle you are right, one would use the same mechanism as for hiding password files (you also do not need to add the ".htpasswd" manually to the hide_file_patterns config).

Ed Swift:

It seems like this would offer a lot of flexibility without needing any new config, but I may be missing something. I guess you'd want to provide the URI scheme to Lua, so the script can check that it's actually a websocket request, but I feel like it should be there anyway. Right now there is no way to distinguish between http and https on regular Lua pages as far as I know.

In Lua there is a string in the variable mg.request_info.http_headers.Referer, e.g. "http://localhost/". I guess this would be "https://".. for a https connection.

But no, I dont want to let Lua decide if it is a websocket request or not. The server already knows that. A websocket connection is also not really a page like the "*.lp" pages are. Its a script that exports a set of functions that are called on websocket operations. The <? some_lua_code ?> syntax of *.lp pages is unsuitable for websockets - the plain Lua script (your *.lua = my *.lws discussed above) would work.

 Ed Swift:

How would you map the resource path to the Lua files, though? People probably don't want their resource paths ending with ".lua".

In the case of using a separate resource path I don't need any file endings. One can either use files that do not have any endings, or append the *.lws or *.lua in the server.
 

Ed Swift:

That should work, but it would mean you'd always need a sort of "router script" if you're using multiple resource paths, and it seems like that routing could be handled by -url_rewrite_patterns and organization of scripts within the filesystem instead. Something feels off to me about having a single entrypoint for all websocket requests, but I can't put my finger on it.

One bad thing clearly is that two completely independent projects, both using websockets, cannot be served by the same webserver without having to write a specific script.
This is actually one of the problem I saw in 3b.
 

Ed Swift:

I'm sure you'll figure out the best way to handle it, just throwing this out there as food for thought.

Thanks for contributing to this discussion. I'm just trying to make it as simple as possible for the Lua users and yet flexible enough.

[Ed Swift]

On Friday, November 1, 2013 6:08:45 PM UTC-4, bel wrote:

[...]

Ed Swift:

One minor gripe I have with mongoose (and CW, I assume; haven't tried it yet) is that you can't have a .lua file handle a request directly, you have to run it though a .lp file. Couldn't .lua files be served up through Lua instead of as plain text, ...

Well, it would be simple to add support for using lua files directly instead through the .lp files. It is also possible to use Lua through CGI by using the Lua standalone interpreter as CGI interpreter. An example how lp and CGI are related is currently here
https://github.com/sunsetbrew/civetweb/tree/master/test/ajax
An additional possibility to use Lua directly, similar to the CGI but as in-server module could easily be added. In fact I thought about that, since it is certainly simpler to program in Lua.
Its not a big deal to implement in the webserver and not a real complication to maintain, yet it would be a third way to invoke Lua - still I do see some advantages .. where advantages would it have for you?

A good example I ran into recently is a REST server I'm working on that only sends JSON, never HTML. It works under CGI, but is designed to work under other environments as well. Mongoose support is there and working (with some hacks, MG doesn't expose all the info I need in request_info, but that's going OT). For the Mongoose support to work, I need to add .lp files that essentially just contain <? require 'somefile.lua' ?>. Since things like CGILua also support Lua pages (same extension but with a different syntax), I imagine this is confusing to someone looking at the project -- those files are only useful for MG/CW. I'd rather not have to have those files there, since I don't need the special .lp functionality to serve JSON.

 

... this was all about "normal" web pages here, not about websockets.


Ed Swift:

... and then we could use the same -url_rewrite_patterns to point websocket requests at the appropriate file (inside or outside of the document root)?

I haven't used url_rewrite_patterns myself, but I think it will still not serve the purpose here:
It would have to distinguish between http(s)://server.xy/file.lws and ws://server.xy/file.lws - the first one must not find the websocket script "file.lws" at all, the second one must do the websocket operation.

Right, if you don't want the Lua script deciding whether it was a websocket request or not (as below), this is no good.

 

Ed Swift:

If they're inside the document root, -hide_files_patterns should cover that, shouldn't it?

Well, one should not have to configure it. But in principle you are right, one would use the same mechanism as for hiding password files (you also do not need to add the ".htpasswd" manually to the hide_file_patterns config).

We probably shouldn't have to, but do we already have to configure it hide .lua files, since they'll get served up as plain text. Serving .lua files through Lua would fix that, though.

 

Ed Swift:

It seems like this would offer a lot of flexibility without needing any new config, but I may be missing something. I guess you'd want to provide the URI scheme to Lua, so the script can check that it's actually a websocket request, but I feel like it should be there anyway. Right now there is no way to distinguish between http and https on regular Lua pages as far as I know.

In Lua there is a string in the variable mg.request_info.http_headers.Referer, e.g. "http://localhost/". I guess this would be "https://".. for a https connection.

Well, the usual mantra is "never trust Referer," because it's sent from the client, so it can be spoofed or mangled by proxies or erased off by privacy things. I think it would be good to have the scheme in that request_info table, but that's probably OT for this thread if the Lua script won't be deciding whether it was a websocket request anyway.

 

But no, I dont want to let Lua decide if it is a websocket request or not. The server already knows that. A websocket connection is also not really a page like the "*.lp" pages are. Its a script that exports a set of functions that are called on websocket operations. The <? some_lua_code ?> syntax of *.lp pages is unsuitable for websockets - the plain Lua script (your *.lua = my *.lws discussed above) would work.

That makes sense to me. The .lp thing certainly isn't needed for websockets, and letting the Lua script decide is probably too weird.

 

 Ed Swift:

How would you map the resource path to the Lua files, though? People probably don't want their resource paths ending with ".lua".

In the case of using a separate resource path I don't need any file endings. One can either use files that do not have any endings, or append the *.lws or *.lua in the server.
 

Files with no endings would work, but I think .lua would feel the nicest. There's really no reason to name it .lws, is there? It's just a regular Lua script, after all, with access to some special table of websocket stuff. With regular .lua extensions we don't have to jump through hoops to get our text editors to realize we're editing Lua scripts.

 

Ed Swift:

That should work, but it would mean you'd always need a sort of "router script" if you're using multiple resource paths, and it seems like that routing could be handled by -url_rewrite_patterns and organization of scripts within the filesystem instead. Something feels off to me about having a single entrypoint for all websocket requests, but I can't put my finger on it.

One bad thing clearly is that two completely independent projects, both using websockets, cannot be served by the same webserver without having to write a specific script.
This is actually one of the problem I saw in 3b.
 

Yeah, that's it, I think. No easy way to drop in a separate project without editing another script. Maybe this isn't a big deal, though. If you didn't have to edit that script, I guess you'd have to edit the rewrite patterns or do something similar. I'm definitely leaning toward 3b now.

 

Ed Swift:

I'm sure you'll figure out the best way to handle it, just throwing this out there as food for thought.

Thanks for contributing to this discussion. I'm just trying to make it as simple as possible for the Lua users and yet flexible enough.

I am fully confident that you will succeed. Thanks for your excellent work on this.   

[Ed Swift]

On Friday, November 1, 2013 8:25:59 PM UTC-4, Ed Swift wrote:

 I'm definitely leaning toward 3b now.

 

Whoops, I meant 1c.

[bel]

A good example I ran into recently is a REST server I'm working on that only sends JSON, never HTML. It works under CGI, but is designed to work under other environments as well. Mongoose support is there and working (with some hacks, MG doesn't expose all the info I need in request_info, but that's going OT). 

The websocket application I've got in mind serves text that is interpreted as operations with SVG objects, so that's a quite similar problem. What did you miss in request_info?

 

For the Mongoose support to work, I need to add .lp files that essentially just contain <? require 'somefile.lua' ?>. Since things like CGILua also support Lua pages (same extension but with a different syntax), I imagine this is confusing to someone looking at the project.

I know ;-)
But changing this will break compatibility here.

 

We probably shouldn't have to, but do we already have to configure it hide .lua files, since they'll get served up as plain text. Serving .lua files through Lua would fix that, though.

One reason why I did not want to use the extension *.lua: If you use 'require something' in Lua, it opens the file 'something.lua', so I can distinguish between pages that are accessible from the webserver, and libraries that can be required but do not form pages by themselves and are not served directly.
Well if you compare to CGILua again, they do use the .lua extension (for those who do not know what we mean - see here: http://keplerproject.github.io/cgilua/manual.html), and hiding *.lua libraries can certainly be done differently than by putting them into the document root and hiding them.
 

 Well, the usual mantra is "never trust Referer," because it's sent from the client, so it can be spoofed or mangled by proxies or erased off by privacy things. I think it would be good to have the scheme in that request_info table, but that's probably OT for this thread if the Lua script won't be deciding whether it was a websocket request anyway.

I agree, one should at least have the server port accessible in Lua.

 

Files with no endings would work, but I think .lua would feel the nicest. There's really no reason to name it .lws, is there? It's just a regular Lua script, after all, with access to some special table of websocket stuff. With regular .lua extensions we don't have to jump through hoops to get our text editors to realize we're editing Lua scripts.

 Well, again I would distinguish between serve-able scripts (lws) and pure libraries (lua), but this can certainly be done differently - see above.

[Ed Swift]

On Sunday, November 3, 2013 4:55:15 PM UTC-5, bel wrote:

A good example I ran into recently is a REST server I'm working on that only sends JSON, never HTML. It works under CGI, but is designed to work under other environments as well. Mongoose support is there and working (with some hacks, MG doesn't expose all the info I need in request_info, but that's going OT). 

The websocket application I've got in mind serves text that is interpreted as operations with SVG objects, so that's a quite similar problem. What did you miss in request_info?

The URI scheme (SERVER_PROTOCOL in CGI terms; Referer is really no good as we discussed) and the current script (SCRIPT_NAME in CGI terms). The "uri" field doesn't cover the current script when using redirects. CGI splits this into SCRIPT_NAME and PATH_INFO.

 

 

For the Mongoose support to work, I need to add .lp files that essentially just contain <? require 'somefile.lua' ?>. Since things like CGILua also support Lua pages (same extension but with a different syntax), I imagine this is confusing to someone looking at the project.

I know ;-)
But changing this will break compatibility here.

Yeah, I didn't mean to suggest changing the syntax of .lp files, I'd just like to be able to avoid using them at all when not serving HTML. That's not really a big problem for me though, just a minor inconvenience.

 

 

We probably shouldn't have to, but do we already have to configure it hide .lua files, since they'll get served up as plain text. Serving .lua files through Lua would fix that, though.

One reason why I did not want to use the extension *.lua: If you use 'require something' in Lua, it opens the file 'something.lua', so I can distinguish between pages that are accessible from the webserver, and libraries that can be required but do not form pages by themselves and are not served directly.
Well if you compare to CGILua again, they do use the .lua extension (for those who do not know what we mean - see here: http://keplerproject.github.io/cgilua/manual.html), and hiding *.lua libraries can certainly be done differently than by putting them into the document root and hiding them.
 

The only issue with putting the .lua files somewhere else is that Lua won't let you do something like:

    require '../somewhere/somefile'

So instead you have to do something like:

    package.path = '../?.lua;' .. package.path

    require 'somewhere/somefile'

    

...which can get ugly, but is certainly feasible. 

 

 Well, the usual mantra is "never trust Referer," because it's sent from the client, so it can be spoofed or mangled by proxies or erased off by privacy things. I think it would be good to have the scheme in that request_info table, but that's probably OT for this thread if the Lua script won't be deciding whether it was a websocket request anyway.

I agree, one should at least have the server port accessible in Lua.

Yeah, I believe MG uses something like "is_ssl," which I thought I exposed to mg.request_info a while back in a patch I made for some unrelated stuff that Sergey accepted, but maybe he took that part out, or maybe I'm just remembering wrong. Anyway, I think port and scheme/protocol fields would make more sense than is_ssl, or whatever it was called.

 

 

Files with no endings would work, but I think .lua would feel the nicest. There's really no reason to name it .lws, is there? It's just a regular Lua script, after all, with access to some special table of websocket stuff. With regular .lua extensions we don't have to jump through hoops to get our text editors to realize we're editing Lua scripts.

 Well, again I would distinguish between serve-able scripts (lws) and pure libraries (lua), but this can certainly be done differently - see above.

Well, .lws does have a nice ring to it, I have to admit. ;) 

[bel]

For CGI, the protocol (http vs. https) is not available in SERVER_PROTOCOL but in a variable HTTPS. This the case for the current CGI implementation here and it's common (see e.g., http://stackoverflow.com/questions/16825243/why-is-phps-server-protocol-showing-http-1-1-even-when-using-https). Currently this variable does not exist in Lua.
I found quite some variables that are available to the CGI environment but not to the Lua environment. I would make all of them available to Lua as well.

Thus, the content of the mg library will look like this (for a normal request):

• redirect = function
  
• write = function
• include = function
  
• version = 1.6
• onerror = function
  
• cry = function
  
• auth_domain = SomeWhere.net
• document_root = r:\
• read = function
  
• request_info:
• https = false
• script_name = r:\/page2.lp
• uri = /page2.lp
• server_port = 80
• remote_port = 2048
• http_version = 1.1
• remote_ip = 2130706433
• num_headers = 8
• http_headers:
• Cache-Control = max-age=0
• Accept-Encoding = gzip, deflate
• Accept-Language = en-us,en;q=0.5
• Referer = http://localhost/
• Host = localhost
• Connection = keep-alive
• Accept = text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
• User-Agent = Mozilla/5.0 (Windows NT 5.1; rv:15.0) Gecko/20100101 Firefox/15.0.1
• request_method = GET

I added in mg: document_root (already in the last version) and auth_domain
and in mg.request_info: https, server_port, script_name and remote_user = "username" + auth_type = "Digest" in case there is a http password for the page.
Now Lua should contain all the information available for CGI, although the names are different (I just followed the current naming convention for Lua).

Still missing anything?

Furthermore, there is already an almost complete implementation for plain Lua pages in addition to the *.lp files, but it was not bound to a file extension yet.
For the moment I assigned *.lsp to test it. All additional variables mentioned above are also available, and they will be available for websockets.

[bel]

• remote_ip = 2130706433

I also wonder if this form is really useful for anyone.
2130706433 is 7F000001 in hex, which means 127.0.0.1 - it is correct for the test above, but isn't "127.0.0.1" as a text a much more useful information?
Of course changing it will force everyone that is actually using it in Lua to change the code - is there anyone?

[Ed Swift]

On Tuesday, November 5, 2013 5:10:59 PM UTC-5, bel wrote:

For CGI, the protocol (http vs. https) is not available in SERVER_PROTOCOL but in a variable HTTPS. This the case for the current CGI implementation here and it's common (see e.g., http://stackoverflow.com/questions/16825243/why-is-phps-server-protocol-showing-http-1-1-even-when-using-https).

Thanks for catching that, I was looking at some code I had only tested over http.

The remote ip would be much nicer as a string. Maybe add remote_ipv4 / remote_ipv6 and leave remote_ip alone in case anyone was using it?

 

[bel]

The remote ip would be much nicer as a string. Maybe add remote_ipv4 / remote_ipv6 and leave remote_ip alone in case anyone was using it?

 

I would call it remote_addr, just like the corresponding CGI variable (see e.g. http://www.cgi101.com/book/ch3/text.html) ... and I found out that ipv6 currently does not work anyway, so -again- something that needs to be fixed first, in order to get a complete solution for the whole topic.

[Ed Swift]

Yes, probably best to stick with CGI names. RFC 3875 also has a nice list on page 2, but doesn't include non-standard (but common) ones like PATH. 

As long as we're going a bit off-topic here, how would you guys feel about letting Lua and CGI handle PUT/DELETE requests?

I need to be able to handle PUT and DELETE requests, and PATCH would be nice too. Looking at handle_request, I see that there is some checking for "is_put_or_delete_request" and calls to "send_authorization_request" before control gets passed off to the Lua or CGI script. 

Could we add an option to allow CGI and Lua to handle requests with any HTTP method, or just move those bits of code up a few lines and hope nobody was using built-in PUT/DELETE handling and CGI/Lua scripts together in some weird way that would cause this to break their app?

[bel]

Am Samstag, 9. November 2013 05:18:55 UTC+1 schrieb Ed Swift:

Yes, probably best to stick with CGI names. RFC 3875 also has a nice list on page 2, but doesn't include non-standard (but common) ones like PATH. 

As long as we're going a bit off-topic here, how would you guys feel about letting Lua and CGI handle PUT/DELETE requests?

I need to be able to handle PUT and DELETE requests, and PATCH would be nice too. Looking at handle_request, I see that there is some checking for "is_put_or_delete_request" and calls to "send_authorization_request" before control gets passed off to the Lua or CGI script. 

Could we add an option to allow CGI and Lua to handle requests with any HTTP method, or just move those bits of code up a few lines and hope nobody was using built-in PUT/DELETE handling and CGI/Lua scripts together in some weird way that would cause this to break their app?

off-topic ?
For me, in the beginning Websockets + Lua was just a combination of two existing features. If there are issues in one of the base features, I think they need to be discussed and solved if possible. It's not a good idea to build a new feature on top of something that should have been looked at in more detail before.
Technically both features work well, but I agree, the Lua programming interface chosen for the Lua generated pages is somewhat ad hoc.
For me, in addition to the advantages of Lua already mentioned ("powerful, fast, extendible, small footprint", http://www.lua.org/about.html#what), there is one more advantage: It is a designed language, not a grown one.

What we do here is providing a programming interface from the C part for our users of the Lua part. The Lua standard libraries already do have a well designed and well documented interface. Also we should provide a well designed interface here (and if the server is embedded in another application, allow the host application to add a well designed interface - this is already possible). This interface should be the same for 'normal' Lua pages, and for websockets, maybe apart from minimal differences that are directly caused by the somewhat different mode of operation. It should be at least possible to do the same in Lua as in CGI.


PUT/DELETE requests
While a GET and a POST request are directed to existing resources, a PUT request is not. A PUT request is meant to add a new resource that does not exist. So "PUT /somewhere/something.lua HTTP/1.1" means create a file /somewhere/something.lua with the content of the request - it does not mean call the Lua script /somewhere/something.lua and give it the data of the request - that would be a POST request. In analogy "DELETE /somewhere/something.lua HTTP/1.1" does not mean call the file /somewhere/something.lua but just remove this file from the server (without calling it first). In other words: A "PUT"/"DELETE" request is not supposed to be handled by the resource, but it adds/removes the resource itself.

One could thing about adding one particular "PUT" and "DELETE" Lua script. This script would be called for all "PUT" and/or "DELETE" requests. It will have access to the resource name and decide what to do. One restriction is, that it will only be possible to have one such PUT/DELETE handling script per server, so two independent contents on the same server cannot have two independent PUT/DELETE handling scripts. The alternative is to handle file creation and deletion at the server by using a POST request. This is already possible and there can be as many independent scripts as you like.

Where do you see the advantage of PUT/DELETE vs. POST?
I have to think about PATCH - on the one hand it is similar to PUT/DELETE since it is supposed to change the resource instead of calling it, on the other hand, in contrast to PUT/DELETE a server will always require a script for doing a PATCH request.

[Ed Swift]

On Saturday, November 9, 2013 7:05:55 AM UTC-5, bel wrote:

Am Samstag, 9. November 2013 05:18:55 UTC+1 schrieb Ed Swift:

Yes, probably best to stick with CGI names. RFC 3875 also has a nice list on page 2, but doesn't include non-standard (but common) ones like PATH. 

As long as we're going a bit off-topic here, how would you guys feel about letting Lua and CGI handle PUT/DELETE requests?

I need to be able to handle PUT and DELETE requests, and PATCH would be nice too. Looking at handle_request, I see that there is some checking for "is_put_or_delete_request" and calls to "send_authorization_request" before control gets passed off to the Lua or CGI script. 

Could we add an option to allow CGI and Lua to handle requests with any HTTP method, or just move those bits of code up a few lines and hope nobody was using built-in PUT/DELETE handling and CGI/Lua scripts together in some weird way that would cause this to break their app?

off-topic ?
For me, in the beginning Websockets + Lua was just a combination of two existing features. If there are issues in one of the base features, I think they need to be discussed and solved if possible. It's not a good idea to build a new feature on top of something that should have been looked at in more detail before.
Technically both features work well, but I agree, the Lua programming interface chosen for the Lua generated pages is somewhat ad hoc.
For me, in addition to the advantages of Lua already mentioned ("powerful, fast, extendible, small footprint", http://www.lua.org/about.html#what), there is one more advantage: It is a designed language, not a grown one.

What we do here is providing a programming interface from the C part for our users of the Lua part. The Lua standard libraries already do have a well designed and well documented interface. Also we should provide a well designed interface here (and if the server is embedded in another application, allow the host application to add a well designed interface - this is already possible). This interface should be the same for 'normal' Lua pages, and for websockets, maybe apart from minimal differences that are directly caused by the somewhat different mode of operation. It should be at least possible to do the same in Lua as in CGI.


PUT/DELETE requests
While a GET and a POST request are directed to existing resources, a PUT request is not. A PUT request is meant to add a new resource that does not exist.

Agreed.

 

So "PUT /somewhere/something.lua HTTP/1.1" means create a file /somewhere/something.lua with the content of the request - it does not mean call the Lua script /somewhere/something.lua and give it the data of the request - that would be a POST request. In analogy "DELETE /somewhere/something.lua HTTP/1.1" does not mean call the file /somewhere/something.lua but just remove this file from the server (without calling it first). In other words: A "PUT"/"DELETE" request is not supposed to be handled by the resource, but it adds/removes the resource itself.

I don't exactly agree with this. I think the mistake here is equating "resource" with "file."

 "DELETE /somewhere/something.lua" should certainly mean "delete that Lua script," but what about something like:

     DELETE /somewhere/user/12 HTTP/1.1

 

This probably means "delete the user with an id of 12," and the resource probably represents something like a row in a database table, not a file.

Now suppose all requests to "/somewhere/..." are redirected to "/controller.lua" and it's up to controller.lua to figure out what to do based on the http method and resource path. So, controller.lua sees that it's a DELETE request for user/12, and it deletes user 12 from the database (after checking appropriate permissions and so on).

The is the way a normal RESTful server would work. You wouldn't want to use something like...

    GET /somewhere/user/12?action=delete HTTP/1.1

...because it doesn't really jive with the way REST is supposed to work. To create a true RESTful server, we need to be able to handle PUT and DELETE the same way we can handle GET or POST.

One could thing about adding one particular "PUT" and "DELETE" Lua script. This script would be called for all "PUT" and/or "DELETE" requests. It will have access to the resource name and decide what to do. One restriction is, that it will only be possible to have one such PUT/DELETE handling script per server, so two independent contents on the same server cannot have two independent PUT/DELETE handling scripts. The alternative is to handle file creation and deletion at the server by using a POST request. This is already possible and there can be as many independent scripts as you like.

Where do you see the advantage of PUT/DELETE vs. POST?
I have to think about PATCH - on the one hand it is similar to PUT/DELETE since it is supposed to change the resource instead of calling it, on the other hand, in contrast to PUT/DELETE a server will always require a script for doing a PATCH request.

See above. :)

At the end of the day, I think it's just a matter of how much control Lua should have, and how much should be "built in" to the server. I would think that if someone is using Lua/CGI, they want to have as much control as reasonably possible, and do not care as much about the built-in functionality, so a good default behavior would be to pass everything off to the script, maybe with some override option that lets the server intercept certain things (like DELETE/PUT) and do stuff without passing control to the script. 

[Ed Swift]

On Saturday, November 9, 2013 12:35:39 PM UTC-5, Ed Swift wrote:

Now suppose all requests to "/somewhere/..." are redirected to "/controller.lua" 

Rewritten, not redirected (controller.lua handles the request, but the resource is still "/somewhere/user/12"). 

[bel]

I don't exactly agree with this. I think the mistake here is equating "resource" with "file."

 "DELETE /somewhere/something.lua" should certainly mean "delete that Lua script," but what about something like:

     DELETE /somewhere/user/12 HTTP/1.1

 

This probably means "delete the user with an id of 12," and the resource probably represents something like a row in a database table, not a file.

Now suppose all requests to "/somewhere/..." are rewritten to "/controller.lua" and it's up to controller.lua to figure out what to do based on the http method and resource path. So, controller.lua sees that it's a DELETE request for user/12, and it deletes user 12 from the database (after checking appropriate permissions and so on).

The is the way a normal RESTful server would work. You wouldn't want to use something like...

    GET /somewhere/user/12?action=delete HTTP/1.1

...because it doesn't really jive with the way REST is supposed to work. To create a true RESTful server, we need to be able to handle PUT and DELETE the same way we can handle GET or POST.

Well, as stated above, I would use a POST request, not a GET request. While GET requests are specified to 'not change anything' at the server, POST requests may do.
Up to now I always got along with GETs and POSTs - this combination always allowed me to do whatever was required.

But, anyway, I took some time and had a look at the REST specification, then read your post again, and now I think I got the point.

The current implementation allows you to use CGI requests in the following way (e.g., to query user information and set a new password):
GET something.cgi/user/12?format=json HTTP/1.1

POST something.cgi/user/12?format=json
{'password':'9e107d9d372bb6826bd81d3542a419d6',
 'newpassword': 'e4d909c290d0fb1ca068ffaddf22cbd0'}

Now bad news:
a) You probably do not want to see the "cgi" in the page. For /something.cgi one could hide the .cgi and access it just by using /something/ by moving the script to /something/index.cgi, but then the additional resource path does not work anymore.
b) What works for cgi does not work for Lua - as already mentioned, I think Lua should be at least as good as cgi.
c) You probably do not want to use a POST request to change the password but PATCH. Currently the server does not know PATCH at all.
Apart from the cgi exception above, the 'resource = file' approach is manifested in several places in the server - what at least explains the current PUT/DELETE handling.

I did not check if rewrite patterns could solve any of the problems above.


If you just want to decide everything in Lua, you could just call mg_exec_lua_script in the begin_request callback - mg_exec_lua_script is a global function that is already providing support *.lua (not *.lp) files.
Do you compile the server yourself, or are you using a prebuilt binary?

[Ed Swift]

A and B can be solved with rewrites, kind of. The script name was missing before, so you could only get the whole URI, there was no way to get "just the redirected part." With the additions we discussed earlier, this won't be a problem anymore (if the whole resource was /somewhere/user/12/ and the script was at /somewhere/index.lua then we can infer that the part of the resource we care about is /user/12/).

As for CGI, I don't really understand the need to kill the request if it doesn't match certain HTTP methods (ignoring the fact that DELETE and PUT have already been intercepted at this point). If I want my CGI script to understand PATCH requests, or even make up some new method just for my application (which seems to be ok according to RFC 2616), is there any reason I shouldn't be able to? 

If you just want to decide everything in Lua, you could just call mg_exec_lua_script in the begin_request callback - mg_exec_lua_script is a global function that is already providing support *.lua (not *.lp) files.
Do you compile the server yourself, or are you using a prebuilt binary?

I'm building it myself. The thing is, I'm working on a generic REST API server, not an API itself, but a tool that allows you to create RESTful APIs quickly. It was initially intended to work with Mongoose, but as I was working on it I realized there were some show-stoppers (like no DELETE/PUT), so I built it to run under CGI (via Apache) first, and then added LuaNode (like node.js) support, then added support for a simple testing server that just uses LuaSockets. Mongoose/Civetweb support is still there, but it's hard to recommend MG/CW as a legitimate host for this tool, since it can't handle PUT/DELETE. 

So, I don't need to use PUT/DELETE directly (right now), but I do want support for it in my project, and without support in Civetweb, I can't list it as a compatible server. Looking at it from this angle, maybe you can see why it feels weird to have PUT/DELETE requests intercepted. No other server does that, and if one did, I would be looking for a way to turn it off so that my project can work the way it's supposed to in that environment.

I guess the question is whether we care if Civetweb is useful for tools like this, or if we'd rather let the built-in handling keep priority over scripts (with no way to override it).

Here's what I wonder, though. Maybe people are using the built in PUT/DELETE handling to store documents and static stuff, but how many people are using them to upload/remove executable CGI scripts or Lua pages? If people are doing this, *should* they be? To me that just sounds like a problem waiting to happen; surely there is a better way to deploy. How much harm would really be done by just letting the scripts handle those requests? Is it possible that the order things are handled in was not carefully planned and the PUT/DELETE check just happened to end up before the CGI / Lua page checks? Maybe the intent was never to prevent scripts from handling those requests, but nobody ran into the problem before now?

The tool I'm working on is here if you're curious: https://github.com/abadc0de/moonwalk

It is more or less a Lua version of this: http://luracast.com/products/restler

[bel]

 

I guess the question is whether we care if Civetweb is useful for tools like this, or if we'd rather let the built-in handling keep priority over scripts (with no way to override it).

Here's what I wonder, though. Maybe people are using the built in PUT/DELETE handling to store documents and static stuff, but how many people are using them to upload/remove executable CGI scripts or Lua pages? If people are doing this, *should* they be? To me that just sounds like a problem waiting to happen; surely there is a better way to deploy. How much harm would really be done by just letting the scripts handle those requests? Is it possible that the order things are handled in was not carefully planned and the PUT/DELETE check just happened to end up before the CGI / Lua page checks? Maybe the intent was never to prevent scripts from handling those requests, but nobody ran into the problem before now?

I thought about the problem, and I think the best behavior of the server would be the following:
1) if there is a request, check if the uri is a file (like somewhere/something.html or somewhere/something.cgi), or another resource handled by a script (like somewhere/something.cgi/someone or users.lsp/user123), or does not exist
2) it a put/delete request is directed to a file, ask for the put/delete permission (i.e., current behavior) - the same for a PUT request for a new file
3) if ANY request is addressing a resource handled by the script, forward it to the script. A *.cgi file is a script, a plain lua file (*.lsp) is a script, but I would not see a *.lp file as a script (it's a text file that may contain script elements within special tags).

This would be compatible with the previous behavior. PUT requests to resources handled by a script did not work up to now anyway, so it's just an extension.
What do you think?

[Ed Swift]

On Wednesday, November 13, 2013 4:50:54 PM UTC-5, bel wrote:

 

I thought about the problem, and I think the best behavior of the server would be the following:
1) if there is a request, check if the uri is a file (like somewhere/something.html or somewhere/something.cgi), or another resource handled by a script (like somewhere/something.cgi/someone or users.lsp/user123), or does not exist
2) it a put/delete request is directed to a file, ask for the put/delete permission (i.e., current behavior) - the same for a PUT request for a new file
3) if ANY request is addressing a resource handled by the script, forward it to the script. A *.cgi file is a script, a plain lua file (*.lsp) is a script, but I would not see a *.lp file as a script (it's a text file that may contain script elements within special tags).

This would be compatible with the previous behavior. PUT requests to resources handled by a script did not work up to now anyway, so it's just an extension.
What do you think?

So, just to be sure we are on the same page, here's my (probably oversimplified) summary:

- Civetweb has built-in support for "static resources," i.e. files. If the client tries to GET a static resource, Civetweb serves it up raw.  If they try to PUT it there, Civetweb stores it, and if they try to DELETE it, Civetweb removes it. This is all fine.

- Civetweb also has support for "dynamic resources," handled by CGI scripts and Lua pages. When the client tries to GET these (or POST to them), Civetweb doesn't serve them up raw like static resources, it processes them first. But, currently Civetweb tries to PUT and DELETE dynamic resources as if they were static, which is weird.

- We can try to fix the weirdness by checking whether the requested resource is static or dynamic. If it's dynamic, pass the request on to the appropriate handler (the shell, or Lua, or whatever), no matter what the HTTP method is. If it's static, use the existing behavior.

I'm not sure .lp shouldn't count as a script. While it might technically not be a script, I think the important distinction to make might be "static resources" vs. "dynamic resources." Unless you think people might have some good reason to want to delete Lua pages from their server in this way, or there is some technical reason it can't be done, I think it might be better just to pass the request on to Lua.

Consider a case where someone runs the standalone server with "-url_rewrite_patterns /api/**=/controller.lp"

Suppose the client sends `DELETE /api/user/12`. The requested resource was `/api/user/12`, which is not a file that exists; it's a "dynamic resource." Civetweb looks at the rewrite patterns and sees that "controller.lp" wants the request. Doesn't it make the most sense to pass the request on to controller.lp? You can't delete /api/user/12 since it's not a file, and deleting controller.lp would be silly. There's really no other sensible option, is there?

I realize the Lua pages aren't ideal for handling something like a DELETE, since it may not even want to send a body, but I'm also worried about Civetweb and Mongoose needlessly diverging. For example, I don't want to end up needing both a "foo.lp" and a "foo.lsp" file containing nothing but `require "foo"` if I want to support both Mongoose and Civetweb. If we treat .lp files as scripts, it might make it easier for Moongoose to follow suit (because they could do it without having to support .lsp).

[Ed Swift]

By the way, Mongoose actually used ".lsp" for Lua pages originally, until someone on the mailing list pointed out that .lsp is usually associated with Lisp. It might be a good idea to come up with a different name for those. I'd still be in favor of just using .lua, really. If it's a security concern, maybe it could be configurable and turned off by default, e.g. add something like "lua_file_extension" to config_options and have it default to NULL, and people who want to use regular Lua scripts instead of pages can set it to ".lua" or whatever else they like.

[bel]

I'm not sure .lp shouldn't count as a script. While it might technically not be a script, I think the important distinction to make might be "static resources" vs. "dynamic resources." Unless you think people might have some good reason to want to delete Lua pages from their server in this way, or there is some technical reason it can't be done, I think it might be better just to pass the request on to Lua.

There are actually three different resources (plus not existing resources as case 4):

1) pure static content like *.html, *.css, *.png, ... in this case resource=file
The http methods would work like this:

GET: deliver the page
POST: not supported
PUT: replace the resource file
DELETE: delete the resource file

2) static data with some elements that are determined on request like *.shtml (server side includes), and *.lp in the present form
The http methods would work basically identical to the first case:

GET: deliver the page, fill in the missing elements
POST: not supported
PUT: replace the resource file
DELETE: delete the resource file

3) pure dynamic data like *.cgi and *.lsp, ... in this case resource != file
The http methods would work differently:

GET: forward request to the script
POST: forward request to the script
PUT: forward request to the script
DELETE: forward request to the script

4) a valid url could also address a non-existing resource not managed by a script

GET: error 404
POST: error 404
PUT: put it there as a file
DELETE: error 404

In short, I would see a *.lp equivalent to a server-side include (ssi), which is also not a fully featured script engine, but only meant to add some dynamic elements to an otherwise static page.
Yes, it is possible to create a *.shtml ssi page with just one line like <!--#exec "ls -l" -->, similar to a *.lp page with one line <? os.execute('ls - l') ?>.
One can also create a page with 97% static content and three such <!-- --> or <? ?> tags. Which one of the three tags should handle a POST/PUT/DELETE request, and what about the static content in this case?
That's why I would use these requests only with real scripts, that do not one body and no static parts.

 

Consider a case where someone runs the standalone server with "-url_rewrite_patterns /api/**=/controller.lp"

You could either use "-url_rewrite_patterns /api/**=/controller.lsp" ... *.lsp would be real scripts, *.lp not (see above).

You could also use an URL like
   /api.lsp/somepath/someobject
without any rewrite patterns. If you do not care about the .lsp in the path, we are done here.
Otherwise you can use a rewrite pattern to replace it. I successfully tested url_rewrite_patterns /api/=r:\page2.lsp/ in Windows, so you can even remove the script completely from the document root.


 

Suppose the client sends `DELETE /api/user/12`. The requested resource was `/api/user/12`, which is not a file that exists; it's a "dynamic resource." Civetweb looks at the rewrite patterns and sees that "controller.lp" wants the request. Doesn't it make the most sense to pass the request on to controller.lp? You can't delete /api/user/12 since it's not a file, and deleting controller.lp would be silly. There's really no other sensible option, is there?

I would use *.lsp instead of *.lp, see above
 

I realize the Lua pages aren't ideal for handling something like a DELETE, since it may not even want to send a body, but I'm also worried about Civetweb and Mongoose needlessly diverging. For example, I don't want to end up needing both a "foo.lp" and a "foo.lsp" file containing nothing but `require "foo"` if I want to support both Mongoose and Civetweb. If we treat .lp files as scripts, it might make it easier for Moongoose to follow suit (because they could do it without having to support .lsp)

I haven't read the mongoose mailing list since about one and a half year. Mongoose was still MIT licensed that time. I will not read that list anymore, so civetweb cannot get infected by GPL pieces this way.
The only thing we can do here is try to make civetweb an inherently conclusive piece of software with a well designed programming interface for server side pages.
I don't know about the mongoose, and essentially I do not care anymore since I cannot use it anymore anyway, due to the licence change and due to fixes I made here that are mandatory for me.

 

By the way, Mongoose actually used ".lsp" for Lua pages originally, until someone on the mailing list pointed out that .lsp is usually associated with Lisp. It might be a good idea to come up with a different name for those. I'd still be in favor of just using .lua, really. If it's a security concern, maybe it could be configurable and turned off by default, e.g. add something like "lua_file_extension" to config_options and have it default to NULL, and people who want to use regular Lua scripts instead of pages can set it to ".lua" or whatever else they like.

 
That might be the explanation why there was already an almost working implementation for plain Lua scripts, that I just connected to a file extension (apparently the same again).

I guess there is no obvious two or three letter abbreviation that does not collide with some other possible meaning. I just looked for *.lp and found the following things: LaTeX3 Drawing File, FlexRip File, Lightscape Preparation File, LP88 Linear Programming File, Voice Manager Audio Recording, NASCAR Racing 2002 Track Configuration Options, Star Wars The Force Unleashed levelpack file  ... so what?
The most obvious name certainly is *.lsp (Lua Server Page), in analogy to *.jsp (Java Server Page), or *.asp, *.csp, *.psp, and one could rename lisp files to *.lisp as well.

Currently there is a config for "cgi_pattern" (default *.cgi, *.pl, *.php) and "ssi_pattern" (default *.shtm, *.shtml) so one could make one for "lsp_pattern" (I think *.lsp would be a good default). There is no option configure *.lp right now.

[Ed Swift]

On Thursday, November 14, 2013 5:27:49 PM UTC-5, bel wrote:

There are actually three different resources (plus not existing resources as case 4):

Hmm, I was counting your type 2 as simply being dynamic, but I guess it makes sense to think of it as mixed. I wasn't even thinking about SSI; you're right, it is hard to think of mostly static documents with an SSI instruction or two as being truly dynamic (although I guess you could).

So for static and mixed, we keep the current behavior, and for dynamic, we pass the request to the script. That sounds good to me as long as we support plain Lua scripts and don't care about diverging from Mongoose.

I think .lp files should still be able to handle POSTs, though. Removing that functionality would break simple pages with forms that post to themselves.

 

Otherwise you can use a rewrite pattern to replace it. I successfully tested url_rewrite_patterns /api/=r:\page2.lsp/ in Windows, so you can even remove the script completely from the document root.

Wouldn't this make a good justification for just using .lua instead of .lsp? Recommended usage can be putting the scripts outside the web root, so there's no security risk.

 

That might be the explanation why there was already an almost working implementation for plain Lua scripts, that I just connected to a file extension (apparently the same again).

I mean they used the .lsp for the Lua pages -- .lsp became .lp.

The most obvious name certainly is *.lsp (Lua Server Page), in analogy to *.jsp (Java Server Page), or *.asp, *.csp, *.psp, and one could rename lisp files to *.lisp as well. 

Actually this is the problem I see with the .lsp name. It is the most obvious name for a server page which is probably why Sergey used it instead of .lp at first. But we're not using it for a server page, we're using it for plain Lua. JSP, ASP, and PSP are a mix of static and dynamic content, just like our .lp files (I haven't heard of CSP). Naming plain Lua scripts ".lsp" does imply that they are like these other "mixed" formats, when really they aren't. Unfortunately I don't have a better suggestion, except maybe ".ls" -- a Lua script as opposed to a Lua "page" (or... .lua).

 

Currently there is a config for "cgi_pattern" (default *.cgi, *.pl, *.php) and "ssi_pattern" (default *.shtm, *.shtml) so one could make one for "lsp_pattern" (I think *.lsp would be a good default). There is no option configure *.lp right now.

This sounds good. If I can't talk you into using .lua instead of .lsp, I at least want to be able to use it myself. ;)

A simpler approach to this might be including an option to turn off the default PUT/DELETE handling. Then we don't have to worry about things like whether .lp files should or shouldn't get certain requests. If the built-in handling is enabled, it will take priority, and if it's disabled, the request will get passed to whatever is handling it. I'm assuming here that people aren't likely to be using both the built-in handling and Lua/CGI, and need support for other methods in their scripts.

[bel]

Fr., 15. November 2013 03:25:52 UTC+1, Ed Swift:

So for static and mixed, we keep the current behavior, and for dynamic, we pass the request to the script. That sounds good to me as long as we support plain Lua scripts and don't care about diverging from Mongoose.

I think it is unavoidable, that civetweb and mongoose diverge sooner or later. Civetweb can not take anything from mongoose, because the GPL license of mongoose would force civetweb to become GPL as well. Mongoose could take something from civetweb, but they would have to put it in an extra file that sustains the MIT license statement - one cannot just take the code and drop the license statement.
Independent from license issues: Unless both projects stand stock-still, inevitably there will be increasingly more differences.
So on the long term, no one should expect compatibility between these two projects.

Everyone: Tell me if you think differently.

 

I think .lp files should still be able to handle POSTs, though. Removing that functionality would break simple pages with forms that post to themselves.

Well, I would not just remove it, and leave everyone in the rain. But if we do have real dynamic resources (type 3), using static ones with dynamic elements (type 2) for post requests should be considered as deprecated.

This actually raises the question where we should put information for the users, like "this should be done that way" ... we are missing a user manual.

I mean they used the .lsp for the Lua pages -- .lsp became .lp.

...

Actually this is the problem I see with the .lsp name. It is the most obvious name for a server page which is probably why Sergey used it instead of .lp at first

 
I did not know that. Actually I do not know if mongoose ever supported plain Lua.
I once made a different Lua integration before mongoose had it - with plain Lua scripts. The name *.lp is used in CGILua as well (I think you know), but CGILua works differently anyway.

But we're not using it for a server page, we're using it for plain Lua. JSP, ASP, and PSP are a mix of static and dynamic content, just like our .lp files (I haven't heard of CSP). Naming plain Lua scripts ".lsp" does imply that they are like these other "mixed" formats, when really they aren't.

Maybe using *.lsp instead of *.lp again would also be an option. It seems like mongoose did not take it that strict with the compatibility of file names before either.

 

This sounds good. If I can't talk you into using .lua instead of .lsp, I at least want to be able to use it myself. ;)

I would not consider file extensions as the main issue - I'm just fine if all of them can be configured. I'm only concerned about comprehensibility of the default.

A simpler approach to this might be including an option to turn off the default PUT/DELETE handling.

It's actually not really simpler. I made a test implementation before my last post anyway, to see if what I'm suggesting does really work.

[bel]

After studying and thinking for a while, the following suggestion:

- add a config parameter "lua_server_page_pattern", the default is *.lp and *.lsp (a difference between lp and lsp might indeed be somewhat difficult to understand, in particular since it apparently has changed in the past)
- add a config parameter "lua_script_pattern", the default is *.lua (*.lp and *.lsp are used above, and others used *.lua for this purpose as well)
- all options do only exist if you build with USE_LUA
- POST to *.lp should be deprecated - I do not know if anyone used it, since it does not even have the CGI equivalent field CONTENT_LENGTH yet. The previous categorization would still be valid (I just adapted the file extensions):

On Thursday, November 14, 2013 11:27:49 PM UTC+1, bel wrote:

There are actually three different resources (plus not existing resources as case 4):

1) pure static content like *.html, *.css, *.png, ... in this case resource=file
The http methods would work like this:

GET: deliver the page
POST: not supported
PUT: replace the resource file
DELETE: delete the resource file

2) static data with some elements that are determined on request like *.shtml (server side includes), and *.lp in the present form + *.lsp

The http methods would work basically identical to the first case:

GET: deliver the page, fill in the missing elements
POST: not supported
PUT: replace the resource file
DELETE: delete the resource file

3) pure dynamic data like *.cgi and *.lua, ... in this case resource != file

The http methods would work differently:

GET: forward request to the script
POST: forward request to the script
PUT: forward request to the script
DELETE: forward request to the script

4) a valid url could also address a non-existing resource not managed by a script

GET: error 404
POST: error 404
PUT: put it there as a file
DELETE: error 404

All these considerations and the previously mentioned additions are meant for Lua scripts triggered due to standard http or https requests, so we are not yet at the "websocket" part.
Still I think it was necessary to solve this before going on with the "Websocket for Lua" implementation, which was and still is the original intention of this thread.
If there are no objections, I would add all the patches mentioned above together with Websockets for Lua - they should be beneficial there as well. The changes should be compatible.

In the following I would focus again on WebSockets.

[Ed Swift]

On Monday, November 18, 2013 4:54:59 PM UTC-5, bel wrote:

After studying and thinking for a while, the following suggestion:

- add a config parameter "lua_server_page_pattern", the default is *.lp and *.lsp (a difference between lp and lsp might indeed be somewhat difficult to understand, in particular since it apparently has changed in the past)
- add a config parameter "lua_script_pattern", the default is *.lua (*.lp and *.lsp are used above, and others used *.lua for this purpose as well)
- all options do only exist if you build with USE_LUA
- POST to *.lp should be deprecated - I do not know if anyone used it, since it does not even have the CGI equivalent field CONTENT_LENGTH yet. The previous categorization would still be valid (I just adapted the file extensions):

That sounds very reasonable.

I used POST before, not in production, but just experimenting. You don't really need Content-Length because you can just keep calling mg.read until it returns nil (it just returns 1024 bytes or less at a time). I don't remember if Content-Length appeared in the table with the rest of the headers, but I don't see why it wouldn't. Are you sure it isn't there? Content-Type was definitely there (maybe worth noting since neither of those headers appear with the rest of the headers under Apache/CGI, since they have their own variables, I guess).

 

All these considerations and the previously mentioned additions are meant for Lua scripts triggered due to standard http or https requests, so we are not yet at the "websocket" part.
Still I think it was necessary to solve this before going on with the "Websocket for Lua" implementation, which was and still is the original intention of this thread.
If there are no objections, I would add all the patches mentioned above together with Websockets for Lua - they should be beneficial there as well. The changes should be compatible.

In the following I would focus again on WebSockets.

I'll take full blame for any sidetracking which may or may not have occurred. ;)

There are still some other Lua problems (Civetweb forked off in the middle of some refactoring that broke stuff that is now fixed again in MG), but we should probably discuss them separately; they shouldn't slow down WebSockets. I know there are some people on this list who would probably prefer not to hear about Lua at all, so maybe we can consolidate other concerns into one thread.

[bel]

On Thursday, November 21, 2013 11:52:43 AM UTC+1, Ed Swift wrote:

I used POST before, not in production, but just experimenting. You don't really need Content-Length because you can just keep calling mg.read until it returns nil (it just returns 1024 bytes or less at a time). I don't remember if Content-Length appeared in the table with the rest of the headers, but I don't see why it wouldn't. Are you sure it isn't there? Content-Type was definitely there (maybe worth noting since neither of those headers appear with the rest of the headers under Apache/CGI, since they have their own variables, I guess).

It is indeed available as a member of the table mg.request_info.header_info named "Content-Length". Since Content-Length is not a valid Lua qualifier (due to the '-' character), you have to use a somewhat different syntax than for other variables. All other CGI like variables are available as members of mg.request_info with valid names (e.g., mg.request_info.request_method).
Anyway, I would just stick to my previous recommendation: Do not use *.lp for POST requests, use full scripts instead.

On Thursday, November 21, 2013 11:52:43 AM UTC+1, Ed Swift wrote:

I'll take full blame for any sidetracking which may or may not have occurred. ;)

There are still some other Lua problems (Civetweb forked off in the middle of some refactoring that broke stuff that is now fixed again in MG), but we should probably discuss them separately; they shouldn't slow down WebSockets. I know there are some people on this list who would probably prefer not to hear about Lua at all, so maybe we can consolidate other concerns into one thread.

Nothing we did here was unnecessary. Fixing the Lua base implementation is required anyway, and now gives me more options in the implementation of the websockets, which now also get a programming model closer to the one of standard Lua script used for normal pages. If you know any more bugs just let me know.
Those who do not want to hear of Lua (what a pity) still have the option of just not setting the USE_LUA compiler define. It might be even possible that they still profit from these changes, since it might be possible to extend the implementation later on to use websockets also via CGI.

[bel]

I made prototype implementations to check the consequences of different designs.
It turns out, that the challenge is actually to define a proper execution model. It is rather simple to just forward the C callbacks (e.g., "websocket ready", "websocket received data", "websocket close") to a Lua script, but this would not allow us to play off one important strengths of the websocket communication model: the server cannot notify the client of any events. If we only answer one "websocket received data" callback and then let the connection poll again for yet another "websocket received data", we are basically in a situation equivalent to sending a sequence of GET or POST requests to Lua scripts. That did not work either before I fixed it (see posts above) - still this is not what websockets are meant to be (maybe we should still include the other fixes into the main stream soon).

The key is to let the websocket connection NOT block within the "poll" function call to wait for "websocket received data", but to let it wait for events from within the server as well. Thus, a websocket programming model is interacting with a general threading model.

The simplest possible implementation is to use a timeout when waiting for data from the client. So the Lua code waits either for data, or for a certain period of time. I used the Lua built-in function "coroutine.yield" to wait for data with a certain timeout. The arguments are BOOLEAN: keepWebSocketOpen and NUMBER: timeout. I have a test in my prototype implementation: it displays a clock in the browser (see https://github.com/bel2125/civetweb/blob/master/test/websocket.xhtml) which is updated by the server websocket script (see https://github.com/bel2125/civetweb/blob/master/test/websocket.lua).

Still, the luxury solution would be to allow the websocket to wait for all kind of events, client data, client disconnects, timeouts, cross communication between websockets (like a chat client), events sent from a C callback, etc. - breaking the 1:1 relation between threads and connections would also be quite beneficial here ...

[bel]

The changes I made for "Websockets for Lua" work if both USE_LUA and USE_WEBSOCKET are defined. Both, Lua and Websocket remain optional features within civetweb.
In case anyone would consider this an advantage, I could add a new define USE_LUA_WEBSOCKET, so one could use Lua and Websocket without using "Websockets for Lua".

We could also first only merge the fixes I made for Lua alone (prepared in a way that one can add "Websockets for Lua" later).
They are fully compatible, and I think useful for everyone who already uses Lua.
The "Websockets for Lua" code would remain in my repository, for those who want to try it. In case we change the threading model (see the corresponding thread in this mailing list), there are opportunities that can also improve "Websockets for Lua", but maybe they impose an interface change for the websocket scripts. If we do not change the threading model, the current state of "Websockets for Lua" can still be useful.

I am fine if we choose to only merge the Lua fixes now (better: middle of the week), and nothing that may change the interface later. However, it might be a little extra work to split the change set into fixes for Lua and features made for "Websockets for Lua" (@Thomas: there is certainly a way, but I might need some assistance in how to handle that with GitHub).

[Thomas Davis]

Happy to help with GitHub as always.  And thank you very much for the design drawings.

Thomas

[bel]

It's been a while since I started with the implementation of websockets for Lua, and it happened that a couple of bug fixes and small features where just along the way .. so I could not do other than pick them up as well ;-)
These fixes and features include:

• Fixes for Lua Server Pages, as described within this thread. Added an example (test/page2.lp).
  

• Added support for plain Lua Scripts, as described within this thread. An example script has been added (test/page2.lua).
  

These two examples also document the difference between *.lp (Lua Server Pages) and *.lua (Plain Lua Scripts).

• A completely new, and more illustrative websocket example for C.

IMO the previous C programming example in examples/websocket/ did not show the essence of websocket communication, so I dropped it and made a completely new one. The build "ex_websocket" now uses the new example: Multiple clients can connect to the websocket by opening websock.htm. This page shows 4 buttons (A, B, C, D) and whenever one client pushes one button, all clients can see it. One can also type a letter within the server console window which is then transferred to all attached clients as well. Furthermore, the server periodically sends the server time to all clients. They display it within the title bar of the window. So this example is primarily on "push" communication from the server to the clients (instead of polling by the clients with periodic GET requests), and on client to client cross communication - so I think this kind of test application is showing the essentials of of a communication situation that may actually profit from websockets.

• Update of SQLite3 to Version 3.8.1.

• Add "date" header field to replies, according to the requirements of RFC 2616 (the HTTP standard), Section 14.18

Actually replies to directory list requests, put/delete requests, error replies, ... did not contain the "date" field, but the RFC says they should. This is a minor http protocol violation, reported by a web debugging tool.

• The option to use an additional "websocket_root" configuration for ws:// urls (optional - otherwise document_root is used for ws:// as well as for http:// urls). One may use URL rewriting on both.

• Some minor stuff like fixing comments.
  

• Finally: An implementation of "Websocket for Lua". Again I added an example (test/websocket.xhtml and test/websocket.lua).
  Note that the Lua programming interface may change, if the threading model changes.
  A change in the threading model is not actually required for "WebSockets for Lua", but if we choose to do a new threading model which may allow us an even better "Websocket for Lua" binding, I will adapt the Lua binding accordingly (if required or beneficial).
  

Everything should be compatible (unless someone exploited a bug).

Both LUA and WEBSOCKET remain optional features to civetweb. You have to enable both: USE_WEBSOCKET and USE_LUA to use Websockets for Lua.

Everything was tested on Windows + stress tested.

Still, if I accidentally broke something, tell me and I will have a look at it.

[bel]

Finally tested for Linux as well.

In case someone likes to try it on his own:
- compile the server with USE_LUA + USE_WEBSOCKETS (Could we actually make this the default for the pre-built Windows binaries?)
- put websocket.xhtml and websocket.lua (from the test/ directory) into your websocket root.
- open websocket.xhtml - you should see a (digital) clock in a circle - (maybe I'm going to give it some watch hands, when I fell like doing a little bit JavaScript+SVG programming)

[bel]

The docu (UserManual.md) states:

# Lua Server Pages
Pre-built Windows and Mac civetweb binaries have built-in Lua Server Pages
support.

Do we actually have pre-built binaries for Mac? Or is this a remnant in the docu?

Now we support at least plain Lua scripts as well. I will add this to UserManual.md. Or should this be done only immediately before a new version of the binaries is put to sourceforge?

I will document the WebSockets for Lua a little later, when we know if a potential change in the thread handling model will influence the websocket programming model.

[Thomas Davis]

We do have a pre-built package for mac.  It is built using Makefile.osx

--
Sourceforge
https://sourceforge.net/projects/civetweb/
---
You received this message because you are subscribed to the Google Groups "civetweb" group.
To unsubscribe from this group and stop receiving emails from it, send an email to civetweb+u...@googlegroups.com.
To post to this group, send email to cive...@googlegroups.com.
Visit this group at http://groups.google.com/group/civetweb.

To view this discussion on the web visit https://groups.google.com/d/msgid/civetweb/339ec1fc-fc73-4d36-91b1-b9140d602f47%40googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

[bel]

On Tuesday, December 17, 2013 7:05:33 PM UTC+1, No Face Press wrote:

We do have a pre-built package for mac.  It is built using Makefile.osx

Indeed, we have that.

What we do not have yet is the <?=variable?> syntax for Lua Server Pages.
PHP has this as well (http://en.wikipedia.org/wiki/PHP_syntax_and_semantics#Variables_and_comments).
I think we (I) should add that.

[bel]

On Tuesday, December 17, 2013 10:28:01 PM UTC+1, bel wrote:

What we do not have yet is the <?=variable?> syntax for Lua Server Pages.
PHP has this as well (http://en.wikipedia.org/wiki/PHP_syntax_and_semantics#Variables_and_comments).
I think we (I) should add that.

Meanwhile I added support for the <?=  var ?> syntax in Lua Server Pages and tested it in my branch for Linux and Windows.
As a test, I added the two equivalent lines into the file page.lp in the test directory

<p> URI is <? mg.write(mg.request_info.uri) ?></p>
<p> URI is <?=mg.request_info.uri?></p>

The basic idea of <?=xy?> is to print the content of a variable xy. The first implementation is basically equivalent to replacing <=? by <?mg.write( and the following ?> by )?>, so it is possible to enter more than just a variable name. Something like
<?= tostring(raw_value * conversion_factor) .. " " .. unit_name ?>
might be useful as well, so it should be allowed. A potential abuse would be an expression like
<?= ""); something_completely_different; io.write("" ?>
Still I think there is no need to add code to prevent this. So I would just document that this is not the intended use (<? something_completely_different ?> would be correct and do the same).

<?= ?> is meant for an expression that evaluates to a string. Everything else is UNDEFINED and may change.


Along with the <?= ?> implementation, I made some functions that are already available in C accessible to Lua.
get_var, get_mime_type, get_cookie, md5, url_encode, url_decode .. they are basically the same functions as in civetweb.h

All this is tested in Windows and Linux (x86 only), but not documented yet.

Lua also could not support html keep_alive. Currently all LSP and plain Lua Scripts need to close the connection. I would allow a Lua script or page to specify mg.keep_alive(true) if it supports keep-alive - it must set the content-length properly or use chunked transport. This is neither tested nor documented yet.

All implementations are currently available only at my github-branch - I would merge it once test and docu are available.
I can only test Windows and Linux. I don't have MacOS-X, and I don't have big endian (Motorola) CPUs.

[Tony Wilk]

On Thursday, October 31, 2013 10:03:13 PM UTC, bel wrote:

WebSockets are a very useful feature in civetweb. You can use it to establish a connection to the server and let a the browser continuously exchange data with the server. The server can send data to the browser, the client does not need to poll repeatedly.

*snip*

Apologies in advance if this has been covered before, couldn't see a reference to it tho...

WebSocket and use of Sec-WebSocket-Protocol (not implemented in civetweb 1v5)

I've written a WebSocket server in PHP for my own use (test app at http://tonywilk.no-ip.org/angularjs/websocket.html , PHP source linked at the bottom)
which uses Sec-WebSocket-Protocol to provide different services in one websocket server. The above-linked example demonstrates some really simple protocols for 'echo', 'chat' and 'command'.

Simply: the client sends "Sec-WebSocket-Protocol: chat" and the server checks if "chat" is something it supports, if so it completes the handshake. Once a connection is running you can simply direct frames to a function to handle that protocol.

If your early 'direct websockets to a lua script idea' still holds - I think it would be a good idea to implement protocol so you can switch traffic based on the protocol in use rather than implementing some custom in-data method.

If you have a look at the PHP you'll get the idea - there's $protocolHandlers which is an array of protocols to handler functions
 (don't look too hard at the html/javascript... that's a right mess :)

Yours,
  TonyWilk

[bel]

On Sunday, January 26, 2014 4:38:12 AM UTC+1, Tony Wilk wrote:

I've written a WebSocket server in PHP for my own use (test app at http://tonywilk.no-ip.org/angularjs/websocket.html , PHP source linked at the bottom)
which uses Sec-WebSocket-Protocol to provide different services in one websocket server. The above-linked example demonstrates some really simple protocols for 'echo', 'chat' and 'command'.

Simply: the client sends "Sec-WebSocket-Protocol: chat" and the server checks if "chat" is something it supports, if so it completes the handshake. Once a connection is running you can simply direct frames to a function to handle that protocol.

I tested your chat server, and this seems to do pretty much what I intended as a first test case for Websockets for Lua.
I have to study your PHP script in more detail. Your websocket-server loop is directly programmed in PHP?

 

If your early 'direct websockets to a lua script idea' still holds - I think it would be a good idea to implement protocol so you can switch traffic based on the protocol in use rather than implementing some custom in-data method.

The idea still holds, I just had to add some functions for "normal" Lua scripts and Lua server pages get before.
You are using host and protocol for opening a new websocket.
 
Thank you for the scripts, I will study them more closely.

[Tony Wilk]

On Sunday, January 26, 2014 9:03:33 PM UTC, bel wrote:
*snip*

I tested your chat server, and this seems to do pretty much what I intended as a first test case for Websockets for Lua.

I have to study your PHP script in more detail. Your websocket-server loop is directly programmed in PHP?

*snip*

The websocket.php is just php (I'm running PHP version 5.3.15)
you can run it as CLI on a server or from a browser (see top of php source) - just pick an appropriate port number

note that websocket.html can start the server by inserting an iframe:

src="http://tonywilk.no-ip.org/angularjs/websocket.php"

My apache server allows this to run forever - others may have an execution time limit.

You are using host and protocol for opening a new websocket.

Yes; note that the server is only allowed to send Sec-WebSocket-Protocol in the handshake if a client requests it.
In my case I make it mandatory.

Yours,
  TonyWilk

[bel]

On Sunday, January 26, 2014 11:22:09 PM UTC+1, Tony Wilk wrote:

The websocket.php is just php (I'm running PHP version 5.3.15)
you can run it as CLI on a server or from a browser (see top of php source) - just pick an appropriate port number

It seems, I cannot run it in from the command line in PHP5.4.24:

php.exe websocket.php -d -p12345
Starting WebSocket server
Server started: 27/01/2014 22:24:42 socket: 'Resource id #4' listening on: 0.0.0.0 : 12345

Notice: Array to string conversion in websocket.php on line 83
<b>Error:</b> [2048] Only variables should be passed by reference in websocket.php at line 145: Array<br>

Maybe my php.ini file is not ideal, I basically only set max_execution_time=3600, date.timezone, extension=php_sockets.dll and extension_dir.
Anyway, my intention was to do this in Lua instead of PHP, so I am just studying the script and the concepts used.

 

You are using host and protocol for opening a new websocket.

Yes; note that the server is only allowed to send Sec-WebSocket-Protocol in the handshake if a client requests it.
In my case I make it mandatory.

While I had nor problems with Firefox on one Computer, Internet Explorer 10 on another computer did not work - The "Messages:" box said:   ReferenceError: "WebSocket" is undefined
I do not know yet if this is because IE10 does not like the protocol specification, or because some proxy/firewall dislikes port 12350.
Furthermore some browsers (Google Chrome, Arora/Webkit) asked for a login, while the others did not.

[bel]

It's been a while, since I started with "Websockets for Lua". First of all, I had to add numerous Lua functions and fix issues that came along in the existing Lua binding. Then I made a first version, but after some tests in real situations I decided to completely rewrite it. The most important insight was that one Lua state has to be shared between several websocket connections. It is not efficient to let them communicate through a database. A Lua script that is suitable for a websocket should define a few non-local functions as callbacks:

open ... a connection is about to be opened
ready ... a connection is ready and may receive data
data ... data has been received from a connection
close ... a connection has been closed

It is easy for the Lua script to create a list of currently open connections. Data received from one websocket client may be sent directly to another client. I still have to add a timer, to get server-side events into the websocket Lua state.