Rack throws fatal error on multi-part upload

[Nate Wiger]

We have found a specific situation with binary multi-part uploads
causing Rack to throw a fatal error.

The specific files we have tested shows that >= 112kb works, <= 110k
fails. The error thrown up from Rack is:

/!\ FAILSAFE /!\ 2009-04-14 17:23:32 -0400
Status: 500 Internal Server Error
invalid byte sequence in US-ASCII
/usr/playerconnect-stack-2009c/lib/ruby/gems/1.9.1/gems/
actionpack-2.3.2/lib/action_controller/vendor/rack-1.0/rack/utils.rb:
320:in `=~'
/usr/playerconnect-stack-2009c/lib/ruby/gems/1.9.1/gems/
actionpack-2.3.2/lib/action_controller/vendor/rack-1.0/rack/utils.rb:
320:in `block in parse_multipart'
/usr/playerconnect-stack-2009c/lib/ruby/gems/1.9.1/gems/
actionpack-2.3.2/lib/action_controller/vendor/rack-1.0/rack/utils.rb:
315:in `loop'
/usr/playerconnect-stack-2009c/lib/ruby/gems/1.9.1/gems/
actionpack-2.3.2/lib/action_controller/vendor/rack-1.0/rack/utils.rb:
315:in `parse_multipart'
/usr/playerconnect-stack-2009c/lib/ruby/gems/1.9.1/gems/
actionpack-2.3.2/lib/action_controller/vendor/rack-1.0/rack/request.rb:
125:in `POST'
/usr/playerconnect-stack-2009c/lib/ruby/gems/1.9.1/gems/
actionpack-2.3.2/lib/action_controller/vendor/rack-1.0/rack/
methodoverride.rb:15:in `call'
/usr/playerconnect-stack-2009c/lib/ruby/gems/1.9.1/gems/
actionpack-2.3.2/lib/action_controller/params_parser.rb:15:in `call'
/usr/playerconnect-stack-2009c/lib/ruby/gems/1.9.1/gems/
actionpack-2.3.2/lib/action_controller/rewindable_input.rb:25:in
`call'
/usr/playerconnect-stack-2009c/lib/ruby/gems/1.9.1/gems/
actionpack-2.3.2/lib/action_controller/session/abstract_store.rb:
122:in `call'
/usr/playerconnect-stack-2009c/lib/ruby/gems/1.9.1/gems/
actionpack-2.3.2/lib/action_controller/failsafe.rb:11:in `call'
/usr/playerconnect-stack-2009c/lib/ruby/gems/1.9.1/gems/
actionpack-2.3.2/lib/action_controller/vendor/rack-1.0/rack/lock.rb:
11:in `block in call'
<internal:prelude>:8:in `synchronize'
/usr/playerconnect-stack-2009c/lib/ruby/gems/1.9.1/gems/
actionpack-2.3.2/lib/action_controller/vendor/rack-1.0/rack/lock.rb:
11:in `call'
/usr/playerconnect-stack-2009c/lib/ruby/gems/1.9.1/gems/
actionpack-2.3.2/lib/action_controller/dispatcher.rb:106:in `call'
/usr/playerconnect-stack-2009c/lib/ruby/gems/1.9.1/gems/
actionpack-2.3.2/lib/action_controller/cgi_process.rb:44:in
`dispatch_cgi'
/usr/playerconnect-stack-2009c/lib/ruby/gems/1.9.1/gems/
actionpack-2.3.2/lib/action_controller/dispatcher.rb:102:in
`dispatch_cgi'
/usr/playerconnect-stack-2009c/lib/ruby/gems/1.9.1/gems/
actionpack-2.3.2/lib/action_controller/dispatcher.rb:28:in `dispatch'
/usr/playerconnect-stack-2009c/lib/ruby/gems/1.9.1/gems/thin-1.0.0/
lib/rack/adapter/rails.rb:54:in `serve_rails'
/usr/playerconnect-stack-2009c/lib/ruby/gems/1.9.1/gems/thin-1.0.0/
lib/rack/adapter/rails.rb:74:in `call'
/usr/playerconnect-stack-2009c/lib/ruby/gems/1.9.1/gems/thin-1.0.0/
lib/thin/connection.rb:63:in `pre_process'
/usr/playerconnect-stack-2009c/lib/ruby/gems/1.9.1/gems/thin-1.0.0/
lib/thin/connection.rb:54:in `process'
/usr/playerconnect-stack-2009c/lib/ruby/gems/1.9.1/gems/thin-1.0.0/
lib/thin/connection.rb:39:in `receive_data'
/usr/playerconnect-stack-2009c/lib/ruby/gems/1.9.1/gems/
eventmachine-0.12.6/lib/eventmachine.rb:240:in `run_machine'
/usr/playerconnect-stack-2009c/lib/ruby/gems/1.9.1/gems/
eventmachine-0.12.6/lib/eventmachine.rb:240:in `run'
/usr/playerconnect-stack-2009c/lib/ruby/gems/1.9.1/gems/thin-1.0.0/
lib/thin/backends/base.rb:57:in `start'
/usr/playerconnect-stack-2009c/lib/ruby/gems/1.9.1/gems/thin-1.0.0/
lib/thin/server.rb:150:in `start'
/usr/playerconnect-stack-2009c/lib/ruby/gems/1.9.1/gems/thin-1.0.0/
lib/thin/controllers/controller.rb:80:in `start'
/usr/playerconnect-stack-2009c/lib/ruby/gems/1.9.1/gems/thin-1.0.0/
lib/thin/runner.rb:173:in `run_command'
/usr/playerconnect-stack-2009c/lib/ruby/gems/1.9.1/gems/thin-1.0.0/
lib/thin/runner.rb:139:in `run!'
/usr/playerconnect-stack-2009c/lib/ruby/gems/1.9.1/gems/thin-1.0.0/
bin/thin:6:in `<top (required)>'
/usr/playerconnect-stack-2009c/bin/thin:19:in `load'
/usr/playerconnect-stack-2009c/bin/thin:19:in `<main>'

Any help is MUCH appreciated.

Thanks,
Nate

[Michael Fellinger]

On Tue, 14 Apr 2009 16:19:56 -0700 (PDT)
Nate Wiger <nwi...@gmail.com> wrote:

>
> We have found a specific situation with binary multi-part uploads
> causing Rack to throw a fatal error.
>
> The specific files we have tested shows that >= 112kb works, <= 110k
> fails. The error thrown up from Rack is:
>
> /!\ FAILSAFE /!\ 2009-04-14 17:23:32 -0400
> Status: 500 Internal Server Error
> invalid byte sequence in US-ASCII

[snip]

That looks like an exception from Ruby 1.9, which version do you use?

--
^ manveru

[Michael Fellinger]

On Tue, 14 Apr 2009 16:19:56 -0700 (PDT)
Nate Wiger <nwi...@gmail.com> wrote:

>
> We have found a specific situation with binary multi-part uploads
> causing Rack to throw a fatal error.
>
> The specific files we have tested shows that >= 112kb works, <= 110k
> fails. The error thrown up from Rack is:
>
> /!\ FAILSAFE /!\ 2009-04-14 17:23:32 -0400
> Status: 500 Internal Server Error
> invalid byte sequence in US-ASCII

I've narrowed the cause down. It seems like we need to add some additional
handling for Ruby 1.9.
I assume that your current locale is set to C, which would cause Ruby to expect
input from outside to be in US-ASCII.
I tried following simple experiment:

sigma ~ % LC_ALL=C irb19
Encoding.default_external
# #<Encoding:US-ASCII>
file = open('MediumFile.jpg').read; file.encoding
# #<Encoding:US-ASCII>
file =~ /foo/
ArgumentError: invalid byte sequence in US-ASCII
from (irb):3
from /usr/bin/irb19:12:in `<main>'
file.force_encoding('ASCII-8BIT'); file.encoding
# #<Encoding:ASCII-8BIT>
file =~ /foo/
# nil

This means we have to change the encoding of the body to ASCII-8BIT for binary
data.
Where this change of encoding should take place isn't easy to say, I think that
it should happen in the Handler if possible...

--
^ manveru

[Hongli Lai]

On Apr 16, 8:09 am, Michael Fellinger <m.fellin...@gmail.com> wrote:
> I've narrowed the cause down. It seems like we need to add some additional
> handling for Ruby 1.9.
> I assume that your current locale is set to C, which would cause Ruby to expect
> input from outside to be in US-ASCII.
> I tried following simple experiment:
>
> sigma ~ % LC_ALL=C irb19
> Encoding.default_external
> # #<Encoding:US-ASCII>
> file = open('MediumFile.jpg').read; file.encoding
> # #<Encoding:US-ASCII>
> file =~ /foo/
> ArgumentError: invalid byte sequence in US-ASCII
>         from (irb):3
>         from /usr/bin/irb19:12:in `<main>'
> file.force_encoding('ASCII-8BIT'); file.encoding
> # #<Encoding:ASCII-8BIT>
> file =~ /foo/
> # nil
>
> This means we have to change the encoding of the body to ASCII-8BIT for binary
> data.
> Where this change of encoding should take place isn't easy to say, I think that
> it should happen in the Handler if possible...

This shouldn't be a problem if rack.input is opened in binary mode.

In your case, you need to call 'f.binmode'. In Thin's case, it
probably needs to do this:
- If the input stream is a temp file, then it needs to call 'binmode'
on it.
- If the input stream is a StringIO, then it needs to call set_encoding
("ASCII-8BIT") on it.

I think the specification should be updated with this information.

[candlerb]

> I think the specification should be updated with this information.

I think it should also be made clear, every time a String is passed to
or from any Rack module, whether its encoding must be set to a
particular value, or whether the encoding is undefined.

In particular:

rack.input#each - yields strings
response#each - yields strings
headers - hash of strings

Is the receiver of these strings entitled to rely on the encoding
being set to anything in particular(*), or must the receiver always
call #force_encoding before calling any method which might be affected
by the encoding?

Regards,

Brian.

(*) It is not clear what would be the correct encoding to require.
ASCII-8BIT at least defeats the sort of nonsense causing this crash in
parse_multipart. However it's possible to argue for something else,
e.g. the encoding should be that given in the Content-Type: header. I
don't think that's a good idea, especially if it's possible that a
multi-byte character could be split across two strings.

[Hongli Lai]

Absolutely.

I think ASCII-8BIT is the correct encoding for binary data because
that's what File#binmode changes the encoding to, but I'm not sure.
Ruby-core probably knows better.