Uniform Request Response api

[Mikael Karlsson]

Hi,

this proposal started as a question to Filippo of why a StartResponse fun was used instead of just returning Status and Headers together with the Body. So I'll just quote it and Filippos answer to give a background

My question:
>>
Why are you using a StartResponse callback instead of just returning a tuple {Status,Headers,Body} at the end of an application fun? When looking into your ewgi_mochiweb module you need to start a buffer just to collect Status and Headers and keep them on hold until the function returns, isn't that so? Chunked encoding would be {Status, Headers, fun F/0} I  guess.
 
I guess I am missing something fundamentally, but having a "clean" function is more,  well,  functional  programming style :-)

Taking it further, if
Response = {Status,Headers, Body | fun F/0}
Request = Env

you could then have an application fun that takes both as Input and also return both
{Request2, Response2}  =  handle_req({Request1, Response1})

Meaning that handle_req is either client/gateway/application and that it would be quite easy to use higher order functional programming and chain filter or dispatch funs or whatever, like lists:foldl(fun(A, Accu) -> A(Accu) end, {Req0,Resp0}, ListOfFuns).
<<

Filippos answer:
>>
For ewgi the only reason it has StartResponse is that I made a simple port of python's wsgi. I wanted something to play with quickly so I reimplemented the python's specs without much thought.
Then, playing with it, I realized the approach is not very "functional" and doesn't fit very well into erlang.

I had in mind to try an approach like the one you propose sooner or later.
The only problem might be with chunked request and responses, but using a fun as you said should work.
I was thinking to use a fun/2 with Request and Response as arguments so if you e.g. start uploading a file and fill up the disk you can still change the headers.
<<

So the implementation proposal is to have an, even more than now, uniform (agnostic?) interface for the application programmer that makes it easy to write clients, applications or gateways/dispatchers and to connect them together.

One start is to have the Reqest and Response in the application call but also in the return value as proposed above.
{Request, Response} = handle({Request, Response})
Now any application fun can (re)write the Request or the Reponse before passing it on to other funs or returning to the caller.

This may be extended maybe to take (parameterized) Erlang modules with a behaviour to have a handle/1 function. These could then be initiated and connected to other modules with the same interface (behaviour).

-module(simplechainingapp,[AuthModule, RouteModule, VdirModule]).
...
handle(ReqResponse) ->
 handle(ReqResponse,[AuthModule, RouteModule, VdirModule]).

handle(ReqResponse, []) -> ReqResponse;
handle(ReqResponse,[M|T]) -> handle(M:handle(ReqResponse),T).

I was inspired by the Restlet (a Java based REST framework, http://www.restlet.org/) that has something of this approach using a uniform interface for all type of "restlets". For parameterized Erlang modules that could be one way to do a similar "wiring", connecting modules together, check out: http://planeterlang.org/story.php?title=The_black_art_of_Erlangs_parameterized_modules

I guess chunked requests and responses need some further thinking, but could this be a good way to go? Or maybe there is a good reason to keep ewgi as it is now, even though it seems to complicate things for application programmers that, as I understand it, have to maintain buffer processes to keep Status and Headers in their state?

Best Regards
Mikael

[Hunter Morris]

On Oct 8, 4:20 pm, "Mikael Karlsson" <karlsson...@gmail.com> wrote:
> One start is to have the Reqest and Response in the application call but
> also in the return value as proposed above.
> {Request, Response} = handle({Request, Response})
> Now any application fun can (re)write the Request or the Reponse before
> passing it on to other funs or returning to the caller.

I have started work on request/response modules in my smak library
which achieves much the same goal. I realised far too late that the
concept should really be pushed up to EWGI where the StartResponse
function has become unnecessary. I think Fillipo's original motivation
was just to have something up and running quickly, and Python's WSGI
was the simplest thing to copy.

With a more Erlang-centric approach, we can make EWGI more functional
without the need for buffer processes. I like the idea of having a
tuple made up of request/response structures. We could make them well-
defined tuples (perhaps records? although they might just complicate
build processes) with an element for additional data (perhaps a dict
or gb_tree?).

Scott Parish made a good suggestion that Filippo forwarded to the list
a few months ago about header case insensitivity. A well-defined tuple
(where element 1 would correspond to HTTP_REQUEST_METHOD, etc)
probably makes the most sense for performance and the case sensitivity
problem. Does that sound reasonable?

> This may be extended maybe to take (parameterized) Erlang modules with a
> behaviour to have a handle/1 function. These could then be initiated and
> connected to other modules with the same interface (behaviour).
>
> -module(simplechainingapp,[AuthModule, RouteModule, VdirModule]).
> ...
> handle(ReqResponse) ->
>  handle(ReqResponse,[AuthModule, RouteModule, VdirModule]).
>
> handle(ReqResponse, []) -> ReqResponse;
> handle(ReqResponse,[M|T]) -> handle(M:handle(ReqResponse),T).
>

> I was inspired by the Restlet (a Java based REST framework,http://www.restlet.org/) that has something of this approach using a uniform

> interface for all type of "restlets". For parameterized Erlang modules that
> could be one way to do a similar "wiring", connecting modules together,

> check out:http://planeterlang.org/story.php?title=The_black_art_of_Erlangs_para...

That's an excellent suggestion. I think parameterized modules could
make implementing one of these "restlet" modules fairly flexible.

> I guess chunked requests and responses need some further thinking, but could
> this be a good way to go? Or maybe there is a good reason to keep ewgi as it
> is now, even though it seems to complicate things for application
> programmers that, as I understand it, have to maintain buffer processes to
> keep Status and Headers in their state?

In order to handle chunked requests/responses, I think a simple
'stream' library should do the trick. I have one in the smak library
here:

http://github.com/skarab/smak/tree/master/src/smak_streams.erl

The stream data structure is effectively a fun/0 which returns {Data,
fun/0} or {}.

Best,
Hunter

[Mikael Karlsson]

Hi,

Great response, thanks.

Having a well defined tuple in the Request that is tuned to make use of Erlangs pattern matching mechanism is probably a good solution. The first elements could be the mandatory ones that  implementors make use of most and on a format that suites pattern matching. Of course the HTTP_REQUEST_METHOD as you mention is the first. Another one I come to think of may be the path, tokenized into a list:  "/xxx/yyyy/zzzz"  ->  ["xxx","yyyy","zzzz"], which could make life easier for implementors of REST-based applications as you could easily apply pattern matching to trigger on the interesting paths. Check out Steve Vinoskis presentation:
http://jaoo.com.au/brisbane-2008/file?path=/jaoo_aus2008/slides/vinoski_rest-erlang-yaws.pdf
for instance.
One thing to handle properly is trailing slashes though. Having the rest of elements as a key-value list, dict or any other structure with an api to access elements is probably good.

I do not think that there will be so many elements in the header that performance would be an issue even if using a simple key-value list ( if you also have direct access to the primary ones ) but of course having an own api, so that it is possible to change the implementation, is a good thing in case you need that.
Also some parts maybe more efficient if atoms instead of strings are used? {'GET',...} instead of {"GET", ..} ?

Using records? Maybe, but you will get compile dependencies to .hrl files. If the number of elements in the tuple are kept small and most of the Header goes into the key-value bag it may not be necessary.

Your streaming solution looks very elegant and functional :-). If I understood Filippos comment I beleive one argument (fun/1) could be added to use the same RequestResponse tuple in the call to that if for instance the disc got full during upload and transmission needs to be interrupted, right - or was it the Status in the Response that should change? Should the streaming fun also return a value on RequestResponse format? This could open for some interesting solutions, same interface everywhere...

Best regards
Mikael

2008/10/8 Hunter Morris <hunter...@gmail.com>

[Hunter Morris]

On Oct 9, 1:25 pm, "Mikael Karlsson" <karlsson...@gmail.com> wrote:
> Having a well defined tuple in the Request that is tuned to make use of
> Erlangs pattern matching mechanism is probably a good solution. The first
> elements could be the mandatory ones that  implementors make use of most and
> on a format that suites pattern matching. Of course the HTTP_REQUEST_METHOD
> as you mention is the first. Another one I come to think of may be the path,
> tokenized into a list:  "/xxx/yyyy/zzzz"  ->  ["xxx","yyyy","zzzz"], which
> could make life easier for implementors of REST-based applications as you
> could easily apply pattern matching to trigger on the interesting paths.

> Check out Steve Vinoskis presentation:http://jaoo.com.au/brisbane-2008/file?path=/jaoo_aus2008/slides/vinos...

> for instance.
> One thing to handle properly is trailing slashes though. Having the rest of
> elements as a key-value list, dict or any other structure with an api to
> access elements is probably good.

Take a look at the EWGI specification wiki document in the google
project (http://code.google.com/p/ewgi/wiki/EWGISpecification). I
modified the specification to include some of the things we've been
talking about. I'd like feedback before I start re-working the
reference implementations. Meanwhile, I'll check out Steve Vinoski's
presentation.

> I do not think that there will be so many elements in the header that
> performance would be an issue even if using a simple key-value list ( if you
> also have direct access to the primary ones ) but of course having an own
> api, so that it is possible to change the implementation, is a good thing in
> case you need that.
> Also some parts maybe more efficient if atoms instead of strings are used?
> {'GET',...} instead of {"GET", ..} ?

Yes, I agree that atoms may be more useful in certain situations.
Let's examine common requests and modify the spec as necessary.

> Using records? Maybe, but you will get compile dependencies to .hrl files.
> If the number of elements in the tuple are kept small and most of the Header
> goes into the key-value bag it may not be necessary.

My first crack at changing the spec has some heavyweight tuples.
Perhaps your suggestion for reducing the size could make records less

necessary.

> Your streaming solution looks very elegant and functional :-). If I
> understood Filippos comment I beleive one argument (fun/1) could be added to
> use the same RequestResponse tuple in the call to that if for instance the
> disc got full during upload and transmission needs to be interrupted, right
> - or was it the Status in the Response that should change? Should the
> streaming fun also return a value on RequestResponse format? This could open
> for some interesting solutions, same interface everywhere...

That's an interesting idea. Take a look at the 'read_input' EWGI
variable in the request. I made use of a callback mechanism which is
similar to continuation-passing style. I haven't really treated error
conditions fully in the spec yet, but I would like to as that's where
Erlang really soars. ;)

Thanks again for your feedback!

Best,
Hunter

[Filippo Pacini]

Hunter Morris wrote:
> On Oct 9, 1:25 pm, "Mikael Karlsson" <karlsson...@gmail.com> wrote:
>> Having a well defined tuple in the Request that is tuned to make use of
>> Erlangs pattern matching mechanism is probably a good solution. The first
>> elements could be the mandatory ones that implementors make use of most and
>> on a format that suites pattern matching. Of course the HTTP_REQUEST_METHOD
>> as you mention is the first. Another one I come to think of may be the path,
>> tokenized into a list: "/xxx/yyyy/zzzz" -> ["xxx","yyyy","zzzz"], which
>> could make life easier for implementors of REST-based applications as you
>> could easily apply pattern matching to trigger on the interesting paths.
>> Check out Steve Vinoskis presentation:http://jaoo.com.au/brisbane-2008/file?path=/jaoo_aus2008/slides/vinos...
>> for instance.
>> One thing to handle properly is trailing slashes though. Having the rest of
>> elements as a key-value list, dict or any other structure with an api to
>> access elements is probably good.
>
> Take a look at the EWGI specification wiki document in the google
> project (http://code.google.com/p/ewgi/wiki/EWGISpecification). I
> modified the specification to include some of the things we've been
> talking about. I'd like feedback before I start re-working the
> reference implementations. Meanwhile, I'll check out Steve Vinoski's
> presentation.

I gave a look at the specification. I think this is the right way to do it.

A few notes:
- Callables: IMHO here we should find a clean way to specify modules to
be called on remote nodes.
Might a tuple {Node, Mod, Handler} work?
- Why use {ewgi_context, Request, _Response} instead of simply
{Request, Response}

>
>> I do not think that there will be so many elements in the header that
>> performance would be an issue even if using a simple key-value list ( if you
>> also have direct access to the primary ones ) but of course having an own
>> api, so that it is possible to change the implementation, is a good thing in
>> case you need that.
>> Also some parts maybe more efficient if atoms instead of strings are used?
>> {'GET',...} instead of {"GET", ..} ?
>
> Yes, I agree that atoms may be more useful in certain situations.
> Let's examine common requests and modify the spec as necessary.

I agree too.

>> Using records? Maybe, but you will get compile dependencies to .hrl files.
>> If the number of elements in the tuple are kept small and most of the Header
>> goes into the key-value bag it may not be necessary.
>
> My first crack at changing the spec has some heavyweight tuples.
> Perhaps your suggestion for reducing the size could make records less
> necessary.
>

IMHO we coud move the remote_* and the server_*

>> Your streaming solution looks very elegant and functional :-). If I
>> understood Filippos comment I beleive one argument (fun/1) could be added to
>> use the same RequestResponse tuple in the call to that if for instance the
>> disc got full during upload and transmission needs to be interrupted, right
>> - or was it the Status in the Response that should change? Should the
>> streaming fun also return a value on RequestResponse format? This could open
>> for some interesting solutions, same interface everywhere...
>
> That's an interesting idea. Take a look at the 'read_input' EWGI
> variable in the request. I made use of a callback mechanism which is
> similar to continuation-passing style. I haven't really treated error
> conditions fully in the spec yet, but I would like to as that's where
> Erlang really soars. ;)
>

Yes I was thinking to the same interface also for streams: they get
ReqResponse and returns the same.
For errors my idea was to simply manage it through the status code, but
the solution as is in the specs is much better.

cheers,
filippo

[Hunter Morris]

On Oct 9, 5:13 pm, Filippo Pacini <filippo.pac...@gmail.com> wrote:
> I gave a look at the specification. I think this is the right way to do it.
>
> A few notes:
> - Callables: IMHO here we should find a clean way to specify modules to
> be called on remote nodes.
> Might a tuple {Node, Mod, Handler} work?

Do you mean calling them using the rpc module?

> - Why use {ewgi_context, Request, _Response} instead of simply
> {Request, Response}

It maintains compatibility with the records (for convenience).

> IMHO we coud move the remote_* and the server_*

Should we put them into a separate part of the record or put them into
the key-value bag?

Some additional things I want to work out are:

* What data structure should the HTTP header and EWGI data bags be?
I initially started by implementing a dict, but since the number
of pairs in each is likely to be quite small, a property list might
perform better?

* What data structure should the "error" property in the Response be?
Python uses output from sys.exc_info(). Perhaps we should use the
form of the output from erlang:get_stacktrace/0, which is [{Module,
Function, Arity | Args}].

* How should we re-order the top-level Request tuple? Perhaps we could
break it up to make use without records slightly easier. Something
like this might be more useful:

-record(ewgi_request, {
request_method, %% atom: 'GET', 'POST', etc
script_name,
path_info,
other=#ewgi_request_other{}
}).

-record(ewgi_request_other, {
auth_type,
content_length,
content_type,
ewgi=#ewgi_spec{},
gateway_interface,
http_headers=#ewgi_http_headers{},
path_translated,
query_string,
remote_addr,
remote_host,
remote_ident,
remote_user,
remote_user_data,
server_name,
server_port,
server_protocol,
server_software
}).

Best,
Hunter

[Filippo Pacini]

Hunter Morris wrote:
> On Oct 9, 5:13 pm, Filippo Pacini <filippo.pac...@gmail.com> wrote:
>> I gave a look at the specification. I think this is the right way to do it.
>>
>> A few notes:
>> - Callables: IMHO here we should find a clean way to specify modules to
>> be called on remote nodes.
>> Might a tuple {Node, Mod, Handler} work?
>
> Do you mean calling them using the rpc module?

Yes!

>
>> - Why use {ewgi_context, Request, _Response} instead of simply
>> {Request, Response}
>
> It maintains compatibility with the records (for convenience).

Ok.

>
>> IMHO we coud move the remote_* and the server_*
>
> Should we put them into a separate part of the record or put them into
> the key-value bag?

Yes I was thinking to put them in the bag. Just to have a smaller
request tuple.

>
> Some additional things I want to work out are:
>
> * What data structure should the HTTP header and EWGI data bags be?
> I initially started by implementing a dict, but since the number
> of pairs in each is likely to be quite small, a property list might
> perform better?

The interface of proplists IMHO is cleaner, but this is a matter of
taste and if you already have a dict in place that's fine for me :-).
As long as we have an api to look into it we can also change the
implementation later.
Another thing to consider is if we should use atoms or strings as keys
in the bag.
We changed from atoms to strings in the past because, as you pointed
out, and if I remember well :-), atoms might be exposed to an attack
where a malicious user sending requests with different http headers
might exhaust the atoms.

>
> * What data structure should the "error" property in the Response be?
> Python uses output from sys.exc_info(). Perhaps we should use the
> form of the output from erlang:get_stacktrace/0, which is [{Module,
> Function, Arity | Args}].

Yes I agree. We should do like in python: suggest to use a try/catch in
the application and in case of error return erlang:get_stacktrace/0
What do we do with the stack trace in the server implementation?
We call error_logger:error_report/1

We might also consider some options to specify a custom error logger.

What do you think?

I'd use the bag concept for the less used fields.
What about:

-record(ewgi_request, {
request_method, %% atom: 'GET', 'POST', etc
script_name,
path_info,

ewgi=#ewgi_spec{},
http_headers=#ewgi_http_headers{},
path_translated,
query_string,
others %bag
}).

The content-type / lenght might go in the headers the rest in the bag.
Would this be too slow?

[Mikael Karlsson]

2008/10/10 Filippo Pacini <filippo...@gmail.com>

Hunter Morris wrote:
> On Oct 9, 5:13 pm, Filippo Pacini <filippo.pac...@gmail.com> wrote:
>> I gave a look at the specification. I think this is the right way to do it.
>>
>> A few notes:
>> - Callables: IMHO here we should find a clean way to specify modules to
>> be called on remote nodes.
>> Might a tuple {Node, Mod, Handler} work?
>
> Do you mean calling them using the rpc module?

Yes!

While you are at it, I think this needs a proper definition. I think I might have mixed things up with my parameterized module proposal. I good idea, but not for the specification maybe?
A callable is a function in the same module (as the caller) or in another module, on the same node or on a remote node. Or should one (also) think about processes on (and messages to ) remote nodes?
A callable is not an Erlang module? Using (parameterized) modules the way we talked about would be an implementation that is compliant with the specification, but not part of it? One thing to keep in mind concerning parameterized modules is that they are not part of "officially" supported erlang.

>
>> - Why use {ewgi_context, Request, _Response} instead of simply
>> {Request, Response}
>
> It maintains compatibility with the records (for convenience).

Ok.

 

A small note here, if you haven't noticed already:
Parameterized modules may show up as records :-)
Try to make and compile a module like:
-module(ewgi_context,[Request, Response])
do ewgi_context:new(3,4) on it and you get:
{ewgi_context,3,4}
Not  to say that the new function will always do this in future implementations of Erlang.

>
>> IMHO we coud move the remote_* and the server_*
>
> Should we put them into a separate part of the record or put them into
> the key-value bag?

Yes I was thinking to put them in the bag. Just to have a smaller
request tuple.

>
> Some additional things I want to work out are:
>
> * What data structure should the HTTP header and EWGI data bags be?
>     I initially started by implementing a dict, but since the number
> of pairs in each is likely to be quite small, a property list might
> perform better?

The interface of proplists IMHO is cleaner, but this is a matter of
taste and if you already have a dict in place that's fine for me :-).
As long as we have an api to look into it we can also change the
implementation later.
Another thing to consider is if we should use atoms or strings as keys
in the bag.
We changed from atoms to strings in the past because, as you pointed
out, and if I remember well :-), atoms might be exposed to an attack
where a malicious user sending requests with different http headers
might exhaust the atoms.

Good point. Any application would have to be careful and convert only from a set of matching strings  to predefined atoms if they are to be used. How is atoms garbage collected today, there has been some improvements here lately if I remember right.

>
> * What data structure should the "error" property in the Response be?
>     Python uses output from sys.exc_info(). Perhaps we should use the
> form of the output from erlang:get_stacktrace/0, which is [{Module,
> Function, Arity | Args}].

Yes I agree. We should do like in python: suggest to use a try/catch in
the application and in case of error return erlang:get_stacktrace/0
What do we do with the stack trace in the server implementation?
We call error_logger:error_report/1

We might also consider some options to specify a custom error logger.

What do you think?

I am not sure how the error property is going to be used. Can't errors be handled with try catch only - or that is maybe your suggestion?

I also think that the Status part of the response should be grouped into one tuple as before - this makes it possible to use the predefined macros. It is not so easy to do it if they are the first and second element in a longer tuple.

Concerning streaming - if the MessageBody part of the response always would be on the form {iolist(), Extra }, were the second element could be a stream (fun/1) or undefined (or any future addons to the spec...) would this make the API simpler? Or more complicated? It would anyway mean that a callable and streaming fun could both work with the same Response structure.

Not for me :-)
 
Best regards
Mikael

[filippo]

On Oct 12, 9:56 pm, "Mikael Karlsson" <karlsson...@gmail.com> wrote:
> 2008/10/10 Filippo Pacini <filippo.pac...@gmail.com>

> > Hunter Morris wrote:
> > > On Oct 9, 5:13 pm, Filippo Pacini <filippo.pac...@gmail.com> wrote:
>
> While you are at it, I think this needs a proper definition. I think I might
> have mixed things up with my parameterized module proposal. I good idea, but
> not for the specification maybe?
> A callable is a function in the same module (as the caller) or in another
> module, on the same node or on a remote node. Or should one (also) think
> about processes on (and messages to ) remote nodes?

We might define a callable as one of:
- a function in the same module
- a function in another module
- a parametrized module M (in this case we call M:handle)
- a Module, and Function on a remote node. How do we specify this
case?
{Node, Mod, Fun} as you say might conflict with the above.

The easiest solution might be using a fun() to wrap the parametrized
module and remote calls, but this has some problems: I paste below
some considerations scott parish made to the list

Scotts consideration:
>>
From what i understand, erlang makes no guarantees that lambdas are
compatible across releases. As i understand it, by using lambdas for
the
Application and StartResponse (as well as the one in the header if
set),
this will make it unsafe for users to upgrade erlang releases in a hot
environment if the lambdas are passed across VMs. this may not be a
problem, but should be considered. the spec however did suggest that
there
may be applications that pass the request to a separate VM; if lambda
are
still used by ewgi, it might be prudent to add a warning not to pass
the
lambdas across VMs, as it would be very tempting to do this.

<<

> A callable is not an Erlang module? Using (parameterized) modules the way we
> talked about would be an implementation that is compliant with the
> specification, but not part of it? One thing to keep in mind concerning
> parameterized modules is that they are not part of "officially" supported
> erlang.

Parameterized modules gives a lot of flexibility so HIMO we should
find a way to fit them in the specs.

>
> > >> - Why use {ewgi_context, Request, _Response} instead of simply
> > >> {Request, Response}
>

> > > Some additional things I want to work out are:
>
> > > * What data structure should the HTTP header and EWGI data bags be?
> > >     I initially started by implementing a dict, but since the number
> > > of pairs in each is likely to be quite small, a property list might
> > > perform better?
> > The interface of proplists IMHO is cleaner, but this is a matter of
> > taste and if you already have a dict in place that's fine for me :-).
> > As long as we have an api to look into it we can also change the
> > implementation later.
> > Another thing to consider is if we should use atoms or strings as keys
> > in the bag.
> > We changed from atoms to strings in the past because, as you pointed
> > out, and if I remember well :-), atoms might be exposed to an attack
> > where a malicious user sending requests with different http headers
> > might exhaust the atoms.
>
> Good point. Any application would have to be careful and convert only from a
> set of matching strings  to predefined atoms if they are to be used. How is
> atoms garbage collected today, there has been some improvements here lately
> if I remember right.

Googling quickly I didn't find much. Maybe we should ask the erlang-
questions mailing list.

>
> > > * What data structure should the "error" property in the Response be?
> > >     Python uses output from sys.exc_info(). Perhaps we should use the
> > > form of the output from erlang:get_stacktrace/0, which is [{Module,
> > > Function, Arity | Args}].
> > Yes I agree. We should do like in python: suggest to use a try/catch in
> > the application and in case of error return erlang:get_stacktrace/0
> > What do we do with the stack trace in the server implementation?
> > We call error_logger:error_report/1
>
> > We might also consider some options to specify a custom error logger.
>
> > What do you think?
>
> I am not sure how the error property is going to be used. Can't errors be
> handled with try catch only - or that is maybe your suggestion?

Sometimes handling the error where it occurs is easier for debugging.
So my suggestion (Hunter actually :-)) was to use of the try/catch in
the application handle.

>
> I also think that the Status part of the response should be grouped into one
> tuple as before - this makes it possible to use the predefined macros. It is
> not so easy to do it if they are the first and second element in a longer
> tuple.

I agree with this.

>
> Concerning streaming - if the MessageBody part of the response always would
> be on the form {iolist(), Extra }, were the second element could be a stream
> (fun/1) or undefined (or any future addons to the spec...) would this make
> the API simpler? Or more complicated? It would anyway mean that a callable
> and streaming fun could both work with the same Response structure.
>

IMVHO this would complicate the build up of the response without many
advantages.
I don't see much space for changes in the body part in the future.
Unless HTTP changes significantly :-).

best regards,
filippo

[Mikael Karlsson]

2008/10/13 filippo <filippo...@gmail.com>

We might define a callable as one of:

- a function in the same module

- a function in another module
- a parametrized module M (in this case we call M:handle)
- a Module, and Function on a remote node. How do we specify this
case?
{Node, Mod, Fun} as you say might conflict with the above.

In the case of a module I think It may not need to be  parameterized (but of course it can be)?
{Node, Mod, Fun} I interpret as a Module and Function on a remote node, or do you also think of rpc:call(Node, Mod, handle,[ReqResp])? Maybe it could be solved by just saying 'handle' in the Fun part: {Node, Mod, handle} ?

I guess an API fun like ewgi_application:run/3 would be good in the specification too, to clarify.
run(F,RR) when is_function(F,1) -> F(RR);
%% run(P,RR) when is_pid(P) -> ..??; 
run(M,RR)            -> M:handle(RR);
run({M,F}, RR)      -> M:F(RR);
run({N,M,F},RR)    ->  rpc:call(N, M, F, [RR]);
run(???,RR) ->  ??.

About rpc calls, there is no conflict with the intention of using HTTP when remoting elsewhere with ewgi?

Parameterized modules gives a lot of flexibility so HIMO we should
find a way to fit them in the specs.

OK I am convinced too.
 

> Good point. Any application would have to be careful and convert only from a
> set of matching strings  to predefined atoms if they are to be used. How is
> atoms garbage collected today, there has been some improvements here lately
> if I remember right.

Googling quickly I didn't find much. Maybe we should ask the erlang-
questions mailing list.

 

Correct, I did not find much either - guess it was wishful thinking. So there is still an open issue to use strings or that ewgi implementations should carefully convert to a predefined set of atoms.

> I am not sure how the error property is going to be used. Can't errors be
> handled with try catch only - or that is maybe your suggestion?

Sometimes handling the error where it occurs is easier for debugging.
So my suggestion (Hunter actually :-)) was to use of the try/catch in
the application handle.

I think I understand. I look forward to see this on the wiki-spec.

>
> I also think that the Status part of the response should be grouped into one
> tuple as before - this makes it possible to use the predefined macros. It is
> not so easy to do it if they are the first and second element in a longer
> tuple.

I agree with this.

 

It would be easy to add that if you only return a number, an ewgi compliant application would fix the string part and add utility funs for this (the same way as the macros) in ewgi.

>

> Concerning streaming - if the MessageBody part of the response always would
> be on the form {iolist(), Extra }, were the second element could be a stream
> (fun/1) or undefined (or any future addons to the spec...) would this make
> the API simpler? Or more complicated? It would anyway mean that a callable
> and streaming fun could both work with the same Response structure.
>

IMVHO this would complicate the build up of the response without many
advantages.
I don't see much space for changes in the body part in the future.
Unless HTTP changes significantly :-).

best regards,
filippo

Yes I think you are right.

Best regards
Mikael

[filippo]

On Oct 13, 10:55 pm, "Mikael Karlsson" <karlsson...@gmail.com> wrote:
> 2008/10/13 filippo <filippo.pac...@gmail.com>

>
> I guess an API fun like ewgi_application:run/3 would be good in the
> specification too, to clarify.
> run(F,RR) when is_function(F,1) -> F(RR);
> %% run(P,RR) when is_pid(P) -> ..??;
> run(M,RR)            -> M:handle(RR);
> run({M,F}, RR)      -> M:F(RR);
> run({N,M,F},RR)    ->  rpc:call(N, M, F, [RR]);
> run(???,RR) ->  ??.

Yes that's exactly what I mean.

>
> About rpc calls, there is no conflict with the intention of using HTTP when
> remoting elsewhere with ewgi?

Do you mean a conflict between the rpc call to N and an http call on
N?

rpc:call calls a ewgi application on a remote node using the erlang VM
features.
If the remote node has an HTTP server with ewgi the same module can be
used locally with
run({M, F}, RR)

This should work.

> > > I also think that the Status part of the response should be grouped into
> > one
> > > tuple as before - this makes it possible to use the predefined macros. It
> > is
> > > not so easy to do it if they are the first and second element in a longer
> > > tuple.
> > I agree with this.
>
> It would be easy to add that if you only return a number, an ewgi compliant
> application would fix the string part and add utility funs for this (the
> same way as the macros) in ewgi.
>

Yes fixing this would be easy, but IMHO it's better having only one
way to do it.
The specs should as precise as possible. This should also make
implementation easier.

cheers,
filippo

[Mikael Karlsson]

I wasn't very clear about my rpc question and it actually may be a bit off topic but I read a couple of articles by Steve Vinoski (again)  where he discusses RPC vs. REST, see: http://steve.vinoski.net/blog/internet-computing-columns/
I guess that was what triggered my question.

Only one way to do it for the Status header and a precise spec sounds good.
(... You have got at least four ways to interpret a callable though :-).

Best Regards
Mikael

2008/10/14 filippo <filippo...@gmail.com>

[Hunter Morris]

On Oct 16, 9:17 am, "Mikael Karlsson" <karlsson...@gmail.com> wrote:
> I wasn't very clear about my rpc question and it actually may be a bit off
> topic but I read a couple of articles by Steve Vinoski (again)  where he
> discusses RPC vs. REST, see:http://steve.vinoski.net/blog/internet-computing-columns/
> I guess that was what triggered my question.

Steve's arguments against the use of RPC are very strong. I'm of the
opinion that we should leave RPC out of the EWGI spec because it is so
easily implemented by a middleware application.

The main reason that I thought it would be useful to allow for
multiple definitions of a callable than the function with arity 1 was
hot code reloading. However, because hot code reloading is really the
responsibility of the released application, it will be better handled
by the server implementation and/or the application implementation.
Therefore, it might make sense to make the callable spec simpler (i.e.
by only allowing a "callable" to be defined as a fun/1). The extra
cases (RPC, parameterised modules, modules that export the
ewgi_application behaviour, etc) can be implemented either by the
server in the mechanism used at the top-level or the application as
middleware components. I have included example middleware components
in ewgi_application.

> Only one way to do it for the Status header and a precise spec sounds good.
> (... You have got at least four ways to interpret a callable though :-).

With my above suggestion for simplifying the definition of the
callable, I think we have a much more concise spec.

> > > Correct, I did not find much either - guess it was wishful thinking. So
> > > there is still an open issue to use strings or that ewgi implementations
> > > should carefully convert to a predefined set of atoms.

I think the predefined set of atoms should include request elements
whose possibilities can be known in advance (e.g. request_method =
'OPTIONS' | 'GET' | 'HEAD' | 'POST' | 'PUT' | 'DELETE' | 'TRACE' |
'CONNECT' | string()). The HTTP 1.1 spec does allow for additional
request methods which can be represented as lists as they will be
quite rare. I can't think of any other obvious candidates besides
request_method.

After looking again at the Python WSGI spec and the popular web
application frameworks that make use of it (in particular, Django,
Pylons, and TurboGears), many web application will likely not be
dealing with the EWGI types directly. My nascent Erlang web
framework, smak, has its own set of modules which help in dealing with
the various HTTP particulars (parsing URLs, dealing with status codes,
rendering appropriate content types, etc). For example, in order to
provide URL routing that leverages Erlang pattern matching, a simpler
tuple containing only path information may be used:

route_request({"app"=ScriptName, ["path", "to", "url"]=Path,
_Request}) ->
%% ...

I am going to finish adding to the spec in its current form, but
consider the above callable simplification.

Best,
~Hunter

[Mikael Karlsson]

I think Joe Armstrong made a nice comment to Steves articles:

http://armstrongonsoftware.blogspot.com/2008/05/road-we-didnt-go-down.html

where he also discusses how we do things in Erlang (Sending messages to processes). There could be room for handling process ids in the spec. but I think that this could be wrapped into the fun/1. If it turns out that this should be treated explicitly I it is probably easier to add this to the specification later - than to remove anything that turned out to be unnecessary in the end.

As I understand it we may expect the "Context" (Request/Response) primarily to be passed around through different fun/1 funs in the same process, the worker process that is spawned to handle the incoming http request. But it may also be passed to other processes (services), on the same node or on remote nodes, which in turn may spawn new worker processes. For instance database server connectors handling pooled connections, clients to LDAP servers for authentication, other http clients that maintain local caches etc.

> I am going to finish adding to the spec in its current form, but
> consider the above callable simplification.

Seems that the specification is converging nicely.

Best regards
Mikael

[Filippo Pacini]

Ok I agree.
We can leave RPC and parameterized modules out of the specs and simply
go with fun/1 as Hunter proposed.

If later someone asks support for these features we'll look for a way to
fit them nicely in the specs.

Best regards,
filippo