parsing POST multipart/form-data file uploads with WSAPI's request.lua?

[James Salsman]

tl;dr: How do I parse file uploads in Civetweb with Lua?

WSAPI's request.lua has a multipart/form-data parser for POST file uploads:

http://github.com/keplerproject/wsapi/blob/master/src/wsapi/request.lua#L123

However, I see that WSAPI is designed for CGI:

http://keplerproject.github.io/wsapi/libraries.html

This message suggests Civetweb's .lp Lua pages don't support POST requests (why?):

https://groups.google.com/d/msg/civetweb/QUM-MdHRZIQ/wgxquoWpV8IJ

But the UserManual.md documentation for mg.read() and mg.request_info.request_method suggests that POSTs are supported. Sadly though, when I tried to access a .lp Lua page with a POST, the connection was reset.

I wasn't able to get .lsp Lua  to run on Civetweb v.1.5 for Win32. What are the civetweb.conf lines to get that to work? Do I need to do any of the things described at https://groups.google.com/d/msg/civetweb/0WZZfBY7uFI/yxIfnlJNKVQJ
?

I suspect I can make the cgi_interpreter an external copy of Lua, but the whole reason I love Civetweb is to avoid that kind of bloat. But if that is the best way, that is okay with me because you are all awesome. (Fun fact: Civetweb is less than 2% of the size of the download which enables CGI on IIS.)

Best regards,
James Salsman

[bel]

POST requests for Lua do work, I use them in a couple of real projects where they work very reliable.

> WSAPI's request.lua has a multipart/form-data parser for POST file uploads:
> http://github.com/keplerproject/wsapi/blob/master/src/wsapi/request.lua#L123

Civetweb does not strictly support the WSAPI - some parts are similar, and some libraries have been taken from the Kepler Project. They do have several very good libraries, and thanks to their license, it is permitted to use them here.
However, civetweb does not fulfill the WSAPI - it has its own api, it already had the “mg” API when it was created (forked) in summer 2013.

It still uses the "mg" API. I thought about redefining the interface, and had a closer look to WSAPI some time ago, but this will break compatibility (which can be reached again by a Lua preload file). Still for me this API change is not the most important point in the moment.

> However, I see that WSAPI is designed for CGI:
> http://keplerproject.github.io/wsapi/libraries.html

Civetweb does not use CGI for the .lp and .lua pages - CGI spawns a new executable for each and every request. Instead civetweb uses the Lua interpreter in the same process as the webserver, like an in-server module, which is significantly more efficient.
It is still possible to use Lua as a CGI process with civetweb, I just do not see the advantage.
Here is an example of both (Lua as in server page and as a CGI)
https://github.com/bel2125/civetweb/tree/master/test/ajax

> This message suggests Civetweb's .lp Lua pages don't support POST requests (why?):
> https://groups.google.com/d/msg/civetweb/QUM-MdHRZIQ/wgxquoWpV8IJ

My message in the "Websocket for Lua" discussion was not, that *.lp does not support POST requests.
My intention was, that *.lp pages should be considered as similar to *shtml which can not handle POST, and to use full Lua scripts for POST, PUT, DELETE, ...
Still *.lp pages are able to handle POST requests - still I recommend *.lua pages for this.
I just did not extend *.lp pages to handle PUT, DELETE, ... requests, like I did for full Lua pages.

> But the UserManual.md documentation for mg.read() and mg.request_info.request_method suggests
> that POSTs are supported. Sadly though, when I tried to access a .lp Lua page with a POST, the
> connection was reset.

This is an example of a page that handles POST and GET to filename.lua?key1=1&key2=whatever
The POST request will store the data into a database, the GET request will read it from there

require 'luacommon'

qs = mg.request_info.query_string
if not qs then
    mg.write("HTTP/1.0 500 Invalid\r\nContent-Type: text/plain\r\n\r\n")
    mg.write("No query string")
    return
end

local db = sqlite3.open('data.db')


key1 = tonumber(mg.get_var(qs, "key1"));
key2 = tostring(mg.get_var(qs, "key2"));
local data = ""

if type(key1)~="number" or key1<1 then
    mg.write("HTTP/1.0 500 Invalid\r\nContent-Type: text/plain\r\n\r\n")
    mg.write("ERROR")
    return
end

mg.write("HTTP/1.0 200 OK\r\nContent-Type: text/plain\r\nCache-Control: no-cache\r\n Cache-Control: max-age=0\r\n\r\n")

if mg.request_info.request_method == "POST" then
  data = mg.read();
  str = [[INSERT OR REPLACE INTO tabX VALUES("]] .. key1 .. [[", "]] .. key2 .. [[", datetime("now"), "]] .. data .. [[");]]
  repeat
    r = db:exec(str);
  until r~=5;
  return
end

str = [[SELECT data FROM tabX WHERE key1 = "]] .. key1 .. [[" AND key2 = "]].. key2 .. [[";]]

stmt = db:prepare(str)
if not stmt then return end
while stmt:step() == sqlite3.ROW do
  local v = stmt:get_values()
  data = v[1]
end
db:close()

mg.write(data)

You will have to modify this *.lua script to fit it to your needs.
In fact, this will not only work for "POST" requests, but for "PUT" as well. You could use a lua script as a "resource handling script" like this demo:
https://github.com/bel2125/civetweb/blob/master/test/resource_script_demo.lua
You can GET/PUT/POST/DELETE/... to resource_script_demo.lua/mypath/myfile.xyz - the script curently only handles GET (line 47) and falls into a default "Method not allowed" (line 68), but here you can define any other method, just like the "POST" request is handled in the example above.

> I wasn't able to get .lsp Lua  to run on Civetweb v.1.5 for Win32.

Most of the Lua related enhancements are available with v1.6:
https://github.com/bel2125/civetweb/blob/master/RELEASE_NOTES.md

Did you take the precompiled v1.5 from SourceForge?
I know, it's been a while since the v1.5 release, and I intended to make an 1.6 before easter, but then (for several reasons) I did not find the time.
The current source at https://github.com/bel2125/civetweb will support all these Lua features, if you build it on your own with Visual Studio (Express).
If you don't want to compile it yourself (what I would fully understand), I promise to make a 1.6 release soon. I just wanted to complete "Websockets for Lua" before a release, but I found that it is better to rewrite my original implementation. The new "Websocket for Lua" implementation is not compatible with the old one, so I did not want to create a release where the interface will change completely a few months later - since the old implementation is in no release yet, this should not be a problem.
This interface change is only relevant for "Websockets for Lua", not for normal *.lp pages or Lua scripts.

> What are the civetweb.conf lines to get that to work?

You don't need any config lines with the current version 1.6.

> Do I need to do any of the things described at https://groups.google.com/d/msg/civetweb/0WZZfBY7uFI/yxIfnlJNKVQJ?

This has already been implemented in the current version.

> I suspect I can make the cgi_interpreter an external copy of Lua, but the whole reason I love Civetweb is to avoid that kind of bloat.
> But if that is the best way, that is okay with me because you are all awesome.

As already mentioned above: No, don't use the external cgi interpreter. Please take the new version.

> (Fun fact: Civetweb is less than 2% of the size of the download which enables CGI on IIS.)

;-)