Multipart/form-data parser

[tomás]

Hello everyone!

A couple of days ago I installed the wonderful Ebb server for an
application I'm building, only to find out that it lacks multipart/
form-data parsing! Since I really need to handle image uploads and
stuff, I was wondering if there are any short-term plans to have that
feature included in the app. :)

That's all. Thanks Ryan and anyone else for this excellent piece of
software!

Regards,
Tomas

[ry]

Hi Tomás,

Multipart parsing is something that is being handled at the framework
layer. That is, Merb or Rails will do the parsing. Ezra has his own
implementation in Merb. Rails uses cgi.rb, I think.

I would like a C implementation of the multipart parser to take this
weight off the back of the frameworks. Since it's a computationally
heavy operation, one would expect great performance increases. The
frameworks would need to cooperate and release this control to Ebb, of
course.

So - you should be able to do uploads currently.

ry

[carmen]

On Sun Apr 06, 2008 at 04:35:38PM -0700, ry wrote:
>
> Hi Tom?s,

>
> Multipart parsing is something that is being handled at the framework
> layer. That is, Merb or Rails will do the parsing. Ezra has his own
> implementation in Merb. Rails uses cgi.rb, I think.

theres also a decent one in Mongrel's Ruby code, i was reading it yesterday.

well, its all cracked out actually. but it claims its not posisble to get it faster (w/o going to C i guess)

[cdr]

> theres also a decent one in Mongrel's Ruby code, i was reading it yesterday.

ok now i can't find it

mabe it was Rack, or Merb, or Ramaze

in any case, as Ry said, clear library candidate. with Ruby and C versions.. any takers?

[ry]

> in any case, as Ry said, clear library candidate. with Ruby and C versions.. any takers?

I plan to write the parser in c at some point in the next month - I
haven't been able to find a suitable existing c library. If anyone
else wants to take a crack at it, extracting some of Zed and Ezra's
string searching code might be useful:
http://github.com/ezmobius/bmhsearch/tree/master

The algorithm is roughly

1. get the boundary from Mongrel parser
CONTENT_TYPE: multipart/form-data.*boundary=\"?([^\";,]+)

2. use zed's string search thing to split input - put (beginning,
end)
tuples onto stack

3. rewind input

while( part = part_stack.pop )

4a. Parse headers
filename = head[/Content-Disposition:.* filename="?([^\";]*)"?/
ni, 1]
content_type = head[/Content-Type: (.*)\r\n/ni, 1]
name = head[/Content-Disposition:.* name="?([^\";]*)"?/ni, 1]

4b. Read the body of the part either into tempfile or into memory

4c. push (body, filename, content_type, name) onto output_stack

end

ry

[tomás]

>
> Multipart parsing is something that is being handled at the framework
> layer. That is, Merb or Rails will do the parsing. Ezra has his own
> implementation in Merb. Rails uses cgi.rb, I think.
>

> So - you should be able to do uploads currently.
>

Wonderful! I'll definitely give it a shot.

Tomás

[ry]

Actually I just discovered a bug in the tempfile handling, so large
uploads might not be working after all.
I'll fix this for the 0.2 release (which hopefully is soon)

ry

[ry]

Just a note - I've removed tempfile handling in ebb:

http://github.com/ry/ebb/commit/9bc8da2c4cab2643f399f26dffdef1f48cdf6d7a

Previously ebb would handle uploads - now it only grabs enough from
the socket to parse the headers, then passes control off to ruby (or
python, or whatever). Ruby can choose to pull down the rest of the
request or drop the connection - and it can do that into memory or
into tmpfile.

I like this a lot because it simplifies my code greatly:

lakshmi:~/projects/ebb% wc -l src/ebb* src/parser.* ruby_lib/
ebb.rb
613 src/ebb.c
102 src/ebb.h
345 src/ebb_python.c
304 src/ebb_ruby.c
53 src/parser.h
239 src/parser.rl
233 ruby_lib/ebb.rb
1889 total


ry