Minimalistic FastCGI Wrapper
1,029 views
Skip to first unread message
murodese
Nov 16, 2009, 1:45:46 PM11/16/09
to golang-nuts
Why use FastCGI rather than Go's inbuilt http package? So it can co-
exist on shared hosting, of course.
Below is a minimalist FCGI wrapper and an example of how to use it.
// fastcgi.go
package fastcgi
/*
#include <stdio.h>
#include <stdlib.h>
#include <fcgi_stdio.h>
*/
import "C"
import "fmt"
func Accept() (C.int)
{
return C.FCGI_Accept();
}
func Finish()
{
C.FCGI_Finish();
}
func StartFilterData() (int)
{
return int(C.FCGI_StartFilterData());
}
func SetExitStatus(status int)
{
C.FCGI_SetExitStatus(C.int(status));
}
func Printf(format string, v ...) (int)
{
str := fmt.Sprintf(format, v);
return int(C.FCGI_puts(C.CString(str)));
}
func Puts(str string) (int)
{
return int(C.FCGI_puts(C.CString(str)));
}
// main.go
package main
import "fastcgi"
func main()
{
count := 0;
for fastcgi.Accept() >= 0
{
// fastcgi.Puts() will be enough in most cases
fastcgi.Printf("Content-type: text/html\r\n\r\n<title>CGI Hello!</
title><h1>CGI Hello!</h1><body>%v</body>\n", count);
count++;
// use the env package to get cgi vars
}
}
exist on shared hosting, of course.
Below is a minimalist FCGI wrapper and an example of how to use it.
// fastcgi.go
package fastcgi
/*
#include <stdio.h>
#include <stdlib.h>
#include <fcgi_stdio.h>
*/
import "C"
import "fmt"
func Accept() (C.int)
{
return C.FCGI_Accept();
}
func Finish()
{
C.FCGI_Finish();
}
func StartFilterData() (int)
{
return int(C.FCGI_StartFilterData());
}
func SetExitStatus(status int)
{
C.FCGI_SetExitStatus(C.int(status));
}
func Printf(format string, v ...) (int)
{
str := fmt.Sprintf(format, v);
return int(C.FCGI_puts(C.CString(str)));
}
func Puts(str string) (int)
{
return int(C.FCGI_puts(C.CString(str)));
}
// main.go
package main
import "fastcgi"
func main()
{
count := 0;
for fastcgi.Accept() >= 0
{
// fastcgi.Puts() will be enough in most cases
fastcgi.Printf("Content-type: text/html\r\n\r\n<title>CGI Hello!</
title><h1>CGI Hello!</h1><body>%v</body>\n", count);
count++;
// use the env package to get cgi vars
}
}
jesse.dailey
Nov 16, 2009, 4:46:20 PM11/16/09
to golang-nuts
Nice little program... I like that you used the existing fcgi library.
But, the reason to use FastCGI is because HTTP is a ridiculous
protocol... and no one serves it "properly" except the real web
servers. Also, FastCGI is "Fast", people say.
But only if you use it right: thou shalt multiplex!
FastCGI is actually a _dog_ of a protocol; if you don't multiplex, you
are just wasting time on all it's ridiculous overhead. (What a cleaner
protocol could do in 1 read + 1 write, FastCGI splits into 5+ reads,
3+ writes... and that pair encoding, blergh).
So, there is also this: http://github.com/jldailey/fcgigo/
It's a working FastCGI implementation in pure Go.
It only supports running in TCP Responder mode (it doesn't support
dynamic spawning IOW)... but this allows for distributed load-
balancing (one of the points of FastCGI).
There is still an outstanding bug with net.Conn.Close(), so fcgigo
can't support multiplexing yet, but once that race condition bug is
fixed, one small code change will re-enable it...
(Also, you use FastCGI because HTTP is a ridiculous protocol to
implement 100% correctly, so one should just let web servers implement
it, instead of having 9000 http server libraries in every language
that all do the caching edge cases differently... the built-in http
server is just for bare bones local testing purposes only... )
What fcgigo will eventually (very soon) be is an HTTP->FastCGI->WSGI
bridge.
Fast and highly concurrent WSGI apps in Go, yay! :)
For some reason, FastCGI development has been hijacked by PHP's
retarded implementation of it... and all of the web servers just
follow suit... no one uses GET_VALUES, no one multiplexes, no one
honors keep-alive or FCGI_KEEPCONN. All because PHP doesn't, so, "why
bother?". Lighttpd is THE WORST about this. They have no intention
of ever writing a real FastCGI protocol... they just have their
"FastPHP" protocol instead... which is just a total waste of time for
everyone involved...
But, the reason to use FastCGI is because HTTP is a ridiculous
protocol... and no one serves it "properly" except the real web
servers. Also, FastCGI is "Fast", people say.
But only if you use it right: thou shalt multiplex!
FastCGI is actually a _dog_ of a protocol; if you don't multiplex, you
are just wasting time on all it's ridiculous overhead. (What a cleaner
protocol could do in 1 read + 1 write, FastCGI splits into 5+ reads,
3+ writes... and that pair encoding, blergh).
So, there is also this: http://github.com/jldailey/fcgigo/
It's a working FastCGI implementation in pure Go.
It only supports running in TCP Responder mode (it doesn't support
dynamic spawning IOW)... but this allows for distributed load-
balancing (one of the points of FastCGI).
There is still an outstanding bug with net.Conn.Close(), so fcgigo
can't support multiplexing yet, but once that race condition bug is
fixed, one small code change will re-enable it...
(Also, you use FastCGI because HTTP is a ridiculous protocol to
implement 100% correctly, so one should just let web servers implement
it, instead of having 9000 http server libraries in every language
that all do the caching edge cases differently... the built-in http
server is just for bare bones local testing purposes only... )
What fcgigo will eventually (very soon) be is an HTTP->FastCGI->WSGI
bridge.
Fast and highly concurrent WSGI apps in Go, yay! :)
For some reason, FastCGI development has been hijacked by PHP's
retarded implementation of it... and all of the web servers just
follow suit... no one uses GET_VALUES, no one multiplexes, no one
honors keep-alive or FCGI_KEEPCONN. All because PHP doesn't, so, "why
bother?". Lighttpd is THE WORST about this. They have no intention
of ever writing a real FastCGI protocol... they just have their
"FastPHP" protocol instead... which is just a total waste of time for
everyone involved...
Eric
Nov 16, 2009, 6:59:35 PM11/16/09
to golang-nuts
Ha, I did something like that, http://launchpad.net/~ericmoritz/+junk/gofastcgi
It was really just a proof of concept more than anything... It's
definitely not complete.
It was really just a proof of concept more than anything... It's
definitely not complete.
murodese
Nov 16, 2009, 11:11:01 PM11/16/09
to golang-nuts
Great work!
The reason I hooked off existing http servers is purely for usage on
shared hosting - unfortunately, Apache's design states that
multiplexing will never be allowed.
Of course, now I'm having fun trying to work out why go's cgi
environment variables aren't getting set (at all)...
The reason I hooked off existing http servers is purely for usage on
shared hosting - unfortunately, Apache's design states that
multiplexing will never be allowed.
Of course, now I'm having fun trying to work out why go's cgi
environment variables aren't getting set (at all)...
eugene
Nov 17, 2009, 5:16:36 AM11/17/09
to golang-nuts
Great work. I'm so excited to see a fastcgi implementation for go. I
think it's a big step in the right direction for building scalable web
apps in go (particularly in the shared server environments as you've
said).
I had a crack at trying to get the github package working with
lighttpd without much success.
Will this work with lighttpd?
I used the following configuration:
http://pastie.org/702310
I could see lighttpd connection to the fcgi.out process, but it would
just keep printing "Connection Accepted"
And lighttpd kept reporting:
2009-11-17 21:05:44: (mod_fastcgi.c.1336) [ERROR]: spawning fcgi
failed.
2009-11-17 21:05:44: (server.c.895) Configuration of plugins failed.
Going down.
I added some debug statements and it seems to be dropping out on the
binary.Read( ... ) line getting an EOF
func newFCGIHeader(r io.Reader) (FCGIHeader, os.Error) {
var h FCGIHeader;
err := binary.Read(r, binary.BigEndian, &h);
return h, err;
}
I don't know a lot about lighttpd configurations for non-php servers,
so forgive me if I'm doing something stupid!
Keep up the great work!
Cheers,
Eugene
think it's a big step in the right direction for building scalable web
apps in go (particularly in the shared server environments as you've
said).
I had a crack at trying to get the github package working with
lighttpd without much success.
Will this work with lighttpd?
I used the following configuration:
http://pastie.org/702310
I could see lighttpd connection to the fcgi.out process, but it would
just keep printing "Connection Accepted"
And lighttpd kept reporting:
2009-11-17 21:05:44: (mod_fastcgi.c.1336) [ERROR]: spawning fcgi
failed.
2009-11-17 21:05:44: (server.c.895) Configuration of plugins failed.
Going down.
I added some debug statements and it seems to be dropping out on the
binary.Read( ... ) line getting an EOF
func newFCGIHeader(r io.Reader) (FCGIHeader, os.Error) {
var h FCGIHeader;
err := binary.Read(r, binary.BigEndian, &h);
return h, err;
}
I don't know a lot about lighttpd configurations for non-php servers,
so forgive me if I'm doing something stupid!
Keep up the great work!
Cheers,
Eugene
jesse.dailey
Nov 17, 2009, 9:20:17 AM11/17/09
to golang-nuts
I just checked my git diff on that and I hadn't checked in the work-
around for the Conn.Close() race condition, and a few other tweaks...
its all in now.
> Will this work with lighttpd?
far (I had hopes it would have an actual "fast" FastCGI
implementation, but they also don't multiplex, and their FCGI_PARAMS
are done slower than they should be).
I never use the bin-path. Lighttpd has always been very inconsistent
between versions on how this process works, so I gave up tracking it
long ago (for instance, in the past it would _never_ talk over a tcp
port to a dynamically spawned process [only over pipes to the
process's stdin/out], then at one point they changed that, but no
matter how many processes it spawned, you could only put one "port" in
the config, so now maybe they support incrementing the port per
process spawned, I dont know... but then each process has to implement
the same logic [try to bind 7143, if in use, increment and repeat]...
blah... not worth it.)
You should just set "host" and "port", then run the fcgi.out as a
separate process yourself.
Here is the sample output I just got by hitting F5 on a browser
pointed at my lighttpd instance:
Listening
Connection accepted
Request complete: / in 4.97 ms.
Connection accepted
Request complete: /favicon.ico in 25.45 ms
Here is my lighttpd config (I include this file at the end of the main
config):
server.modules += ( "mod_fastcgi" )
fastcgi.debug = 0
fastcgi.server =(""
=>
("" =>
(
"host" => "127.0.0.1",
"port" => 7143,
"check-local" => "disable",
),
)
)
I wrote this as my second-ever Go program, so I'm sure it will
improve...
jesse.dailey
Nov 17, 2009, 9:49:08 AM11/17/09
to golang-nuts
@murodese:
FCGI sends the environment variables 'over the wire' to your process,
as a part of the protocol, so there should/might be something in that
fcgi library for explicitly reading them from the connection, then
patching them into the process's environment.
FCGI sends the environment variables 'over the wire' to your process,
as a part of the protocol, so there should/might be something in that
fcgi library for explicitly reading them from the connection, then
patching them into the process's environment.
jesse.dailey
Nov 17, 2009, 9:54:00 AM11/17/09
to golang-nuts
Also, I think lighttpd has an option that forces it to copy the
environment directly, avoiding the protocol... again I will just blame
PHP for this... ;)
environment directly, avoiding the protocol... again I will just blame
PHP for this... ;)
eugene
Nov 17, 2009, 7:06:33 PM11/17/09
to golang-nuts
Thanks heaps. Was definitely the lighttpd conf files. I used your
settings and now it works like a dream.
Next step: Word domination :-)
Cheers,
Eugene
settings and now it works like a dream.
Next step: Word domination :-)
Cheers,
Eugene
murodese
Nov 18, 2009, 4:19:05 AM11/18/09
to golang-nuts
Pretty much. The issue was that go only ever reads environment
variables and populates os.Environ() once - at program start. Changed
the fcgi wrapper to pull the envvars from **environ and put them into
the Go environment.
// fastcgi.go
package fastcgi
/*
#include <stdio.h>
#include <stdlib.h>
#include <fcgi_stdio.h>
extern char **environ;
char *envvar(int index)
{
return environ[index];
}
*/
import "C"
import "os"
import "fmt"
import "strings"
func Accept() (int)
{
retVal := int(C.FCGI_Accept());
for i, next := 0, true; next; i++
{
s := C.GoString(C.envvar(C.int(i)));
if (len(s) > 0)
{
keyVal := strings.Split(s, "=", 2);
os.Setenv(keyVal[0], keyVal[1]);
}
else
{
next = false;
}
}
return retVal;
> > > But, the reason to useFastCGIis because HTTP is a ridiculous
> > > > //fastcgi.Puts() will be enough in most cases
variables and populates os.Environ() once - at program start. Changed
the fcgi wrapper to pull the envvars from **environ and put them into
the Go environment.
// fastcgi.go
package fastcgi
/*
#include <stdio.h>
#include <stdlib.h>
#include <fcgi_stdio.h>
char *envvar(int index)
{
return environ[index];
}
*/
import "C"
import "os"
import "fmt"
import "strings"
func Accept() (int)
{
retVal := int(C.FCGI_Accept());
for i, next := 0, true; next; i++
{
s := C.GoString(C.envvar(C.int(i)));
if (len(s) > 0)
{
keyVal := strings.Split(s, "=", 2);
os.Setenv(keyVal[0], keyVal[1]);
}
else
{
next = false;
}
}
return retVal;
> > > protocol... and no one serves it "properly" except the real web
> > > servers. Also,FastCGIis "Fast", people say.
>
> > > But only if you use it right: thou shalt multiplex!
>
> > >FastCGIis actually a _dog_ of a protocol; if you don't multiplex, you
> > > But only if you use it right: thou shalt multiplex!
>
> > > are just wasting time on all it's ridiculous overhead. (What a cleaner
> > > protocol could do in 1 read + 1 write,FastCGIsplits into 5+ reads,
>
> > > 3+ writes... and that pair encoding, blergh).
>
> > > So, there is also this:http://github.com/jldailey/fcgigo/
>
> > > It's a workingFastCGIimplementation in pure Go.
> > > 3+ writes... and that pair encoding, blergh).
>
> > > So, there is also this:http://github.com/jldailey/fcgigo/
>
>
> > > It only supports running in TCP Responder mode (it doesn't support
> > > dynamic spawning IOW)... but this allows for distributed load-
> > > balancing (one of the points ofFastCGI).
>
> > > There is still an outstanding bug with net.Conn.Close(), so fcgigo
> > > can't support multiplexing yet, but once that race condition bug is
> > > fixed, one small code change will re-enable it...
>
> > > (Also, you useFastCGIbecause HTTP is a ridiculous protocol to
> > > It only supports running in TCP Responder mode (it doesn't support
> > > dynamic spawning IOW)... but this allows for distributed load-
> > > balancing (one of the points ofFastCGI).
>
> > > There is still an outstanding bug with net.Conn.Close(), so fcgigo
> > > can't support multiplexing yet, but once that race condition bug is
> > > fixed, one small code change will re-enable it...
>
> > > implement 100% correctly, so one should just let web servers implement
> > > it, instead of having 9000 http server libraries in every language
> > > that all do the caching edge cases differently... the built-in http
> > > server is just for bare bones local testing purposes only... )
>
> > > What fcgigo will eventually (very soon) be is an HTTP->FastCGI->WSGI
> > > bridge.
>
> > > Fast and highly concurrent WSGI apps in Go, yay! :)
>
> > > For some reason,FastCGIdevelopment has been hijacked by PHP's
> > > it, instead of having 9000 http server libraries in every language
> > > that all do the caching edge cases differently... the built-in http
> > > server is just for bare bones local testing purposes only... )
>
> > > What fcgigo will eventually (very soon) be is an HTTP->FastCGI->WSGI
> > > bridge.
>
> > > Fast and highly concurrent WSGI apps in Go, yay! :)
>
> > > retarded implementation of it... and all of the web servers just
> > > follow suit... no one uses GET_VALUES, no one multiplexes, no one
> > > honors keep-alive or FCGI_KEEPCONN. All because PHP doesn't, so, "why
> > > bother?". Lighttpd is THE WORST about this. They have no intention
> > > of ever writing a realFastCGIprotocol... they just have their
> > > follow suit... no one uses GET_VALUES, no one multiplexes, no one
> > > honors keep-alive or FCGI_KEEPCONN. All because PHP doesn't, so, "why
> > > bother?". Lighttpd is THE WORST about this. They have no intention
> > > "FastPHP" protocol instead... which is just a total waste of time for
> > > everyone involved...
>
> > > On Nov 16, 1:45 pm, murodese <rawr...@gmail.com> wrote:
>
> > > > Why useFastCGIrather than Go's inbuilt http package? So it can co-
> > > everyone involved...
>
> > > On Nov 16, 1:45 pm, murodese <rawr...@gmail.com> wrote:
>
Petri Lehtinen
Nov 19, 2009, 4:54:07 AM11/19/09
to golang-nuts
On Nov 17, 4:20 pm, "jesse.dailey" <jesse.dai...@gmail.com> wrote:
> I never use the bin-path. Lighttpd has always been very inconsistent
> between versions on how this process works, so I gave up tracking it
> long ago (for instance, in the past it would _never_ talk over a tcp
> port to a dynamically spawned process [only over pipes to the
> process's stdin/out], then at one point they changed that, but no
> matter how many processes it spawned, you could only put one "port" in
> the config, so now maybe they support incrementing the port per
> process spawned, I dont know... but then each process has to implement
> the same logic [try to bind 7143, if in use, increment and repeat]...
> blah... not worth it.)
The FastCGI specification says that the parent process (lighttpd)
> I never use the bin-path. Lighttpd has always been very inconsistent
> between versions on how this process works, so I gave up tracking it
> long ago (for instance, in the past it would _never_ talk over a tcp
> port to a dynamically spawned process [only over pipes to the
> process's stdin/out], then at one point they changed that, but no
> matter how many processes it spawned, you could only put one "port" in
> the config, so now maybe they support incrementing the port per
> process spawned, I dont know... but then each process has to implement
> the same logic [try to bind 7143, if in use, increment and repeat]...
> blah... not worth it.)
leaves file descriptor 0 open before exec()ing, and the child process
(fastcgi.go) should, as its first job, call accept() on that socket,
be it a bound TCP socket or unix domain socket. I believe that
lighttpd conforms to this when you use bin-path, but am I wrong?
jesse.dailey
Nov 19, 2009, 9:09:21 AM11/19/09
to golang-nuts
That's interesting, I hadn't noticed that bit of the spec... Like I
say, I haven't tried to use bin-path in more than a year, and before
then it had already changed a few times. When I get some time again
after the holiday (pesky work project)... I'll start poking at it
again.
*time passes*
Actually, I couldn't resist... So I read mod_fastcgi.c over quickly...
and I think I see how it works.
First, before it forks, it binds a listening TCP socket inside the
parent process, on some fcgi_fd != 0.
Then, after the fork:
child:
1012 if(fcgi_fd != FCGI_LISTENSOCK_FILENO) {
1013 close(FCGI_LISTENSOCK_FILENO);
1014 dup2(fcgi_fd, FCGI_LISTENSOCK_FILENO);
1015 close(fcgi_fd);
1016 }
So if I understand this correctly, lighttpd 'pre-binds' your listen
socket for you (on some non-0 fd), and then dups it onto fd 0 for you
after the fork, but before the exec()?
So, yeah, if you are dynamically spawned, don't specify "port" in the
config, and just open a socket right on fd = 0.
say, I haven't tried to use bin-path in more than a year, and before
then it had already changed a few times. When I get some time again
after the holiday (pesky work project)... I'll start poking at it
again.
*time passes*
Actually, I couldn't resist... So I read mod_fastcgi.c over quickly...
and I think I see how it works.
First, before it forks, it binds a listening TCP socket inside the
parent process, on some fcgi_fd != 0.
Then, after the fork:
child:
1012 if(fcgi_fd != FCGI_LISTENSOCK_FILENO) {
1013 close(FCGI_LISTENSOCK_FILENO);
1014 dup2(fcgi_fd, FCGI_LISTENSOCK_FILENO);
1015 close(fcgi_fd);
1016 }
So if I understand this correctly, lighttpd 'pre-binds' your listen
socket for you (on some non-0 fd), and then dups it onto fd 0 for you
after the fork, but before the exec()?
So, yeah, if you are dynamically spawned, don't specify "port" in the
config, and just open a socket right on fd = 0.
Petri Lehtinen
Nov 22, 2009, 2:12:41 PM11/22/09
to golang-nuts
On Nov 19, 4:09 pm, "jesse.dailey" <jesse.dai...@gmail.com> wrote:
> First, before it forks, it binds a listening TCP socket inside the
> parent process, on some fcgi_fd != 0.
>
> Then, after the fork:
>
> child:
> 1012 if(fcgi_fd != FCGI_LISTENSOCK_FILENO) {
> 1013 close(FCGI_LISTENSOCK_FILENO);
> 1014 dup2(fcgi_fd, FCGI_LISTENSOCK_FILENO);
> 1015 close(fcgi_fd);
> 1016 }
>
> So if I understand this correctly, lighttpd 'pre-binds' your listen
> socket for you (on some non-0 fd), and then dups it onto fd 0 for you
> after the fork, but before the exec()?
Correct. Actually, IIRC this is the only way to start a fastcgi
> First, before it forks, it binds a listening TCP socket inside the
> parent process, on some fcgi_fd != 0.
>
> Then, after the fork:
>
> child:
> 1012 if(fcgi_fd != FCGI_LISTENSOCK_FILENO) {
> 1013 close(FCGI_LISTENSOCK_FILENO);
> 1014 dup2(fcgi_fd, FCGI_LISTENSOCK_FILENO);
> 1015 close(fcgi_fd);
> 1016 }
>
> So if I understand this correctly, lighttpd 'pre-binds' your listen
> socket for you (on some non-0 fd), and then dups it onto fd 0 for you
> after the fork, but before the exec()?
process that is mentioned in the fastcgi specification. The case where
the fastcgi process is started outside the web server and the web
server is configured to connect to it is "non-standard", although all
major web servers seem to support it.
Reply all
Reply to author
Forward
0 new messages