De-chunking Middleware

[Bill Moseley]

This was discussed here recently: https://github.com/plack/Plack/issues/404

I have a Node.js server making API calls to a Catalyst REST app and the requests are chunked.   Catalyst doesn't currently handle this so I've written a quick Middleware class to de-chunk.

I've only done minimal testing so far but seems to work.  I'll test much more Monday, but wanted to check on my implementation here.   

Does anyone see anything below that will break?  One thing I'm curious about is if I need to worry about timeouts and if so how best to handle.

I'm completely ignoring chunk-extensions and trailing entity-headers for now.

sub _prepare_env {

    my ( $self, $env ) = @_;

    return unless do { no warnings; lc delete $env->{HTTP_TRANSFER_ENCODING} eq 'chunked' };    ## no critic

    my $input = $env->{'psgix.io'};

    my $read_line = sub {

        my $line = '';

        while ( 1 ) {

            die 'Failed to read from socket'

                unless sysread( $input, my $char, 1 ) == 1;

            $line .= $char;

            return $line if $line =~ s/\015\012$//;

        }

    };

    my $content_length = 0;

    my $buf            = Plack::TempBuffer->new;

    while ( 1 ) {

        # Read chunked header

        my $line = $read_line->();

        die 'Failed to read chunked header' unless length $line;

        # Grab our size and trailer

        my ( $hex_size, $trailer ) = $line =~ /^([0-9a-fA-F]+)(.*)$/;

        die "Failed to find hex size in chunked header [$line]" unless defined $hex_size;

        my $size = hex $hex_size;

        # End of chunks?  Then read entity-headers

        if ( $size == 0 ) {

            # read to empty CRLF

            1 while length $read_line->();

            last;

        }

        my $bytes = sysread( $input, my $buffer, $size );

        die "Tried to read chunksize of $size bytes but read $bytes"

            unless $bytes == $size;

        $content_length += $size;

        $buf->print( $buffer );

        die 'Failed to read CRLF after reading chunk'

            if $read_line->();

    }

    $env->{CONTENT_LENGTH} = $content_length;

    $env->{'psgi.input'} = $buf->rewind;

    return;

}

--
Bill Moseley
mos...@hank.org

[Bill Moseley]

On Sun, Jun 30, 2013 at 12:16 PM, Bill Moseley <mos...@hank.org> wrote:

This was discussed here recently: https://github.com/plack/Plack/issues/404

Well, I'm having problems with this under mod_perl.   I guess I'm not understanding where the data is.

$env has this:

   'psgi.input' => bless( do{\(my $o = '140312956905104')}, 'Apache2::RequestRec' )

Should't I be able to call ->read on psgi.input?   Has something already read the input stream?

If I run the Catalyst built-in server the de-chunking code works, but under mod_perl I get this:

Sun Jun 30 14:37:12 2013] [error] [client 127.0.0.1] Failed to read from socket [Apache2::RequestRec=SCALAR(0x7f9d220ab7c0)] [0] at /home/bill/perl5/perlbrew/perls/perl-5.14.2/lib/site_perl/5.14.2/Plack/Middleware/DeChunked.pm line 40.\n

 34     my $input = $env->{'psgix.io'} || $env->{'psgi.input'};

 35

 36     my $read_line = sub {

 37         my $line = '';

 38         while ( 1 ) {

 39             my $bytes = $input->read( my $char, 1 );

 40             die "Failed to read from socket [$input] [$bytes]"

 41                 unless $bytes == 1;

 42             $line .= $char;

 43             return $line if $line =~ s/\015\012$//;

 44         }

 45     };

 46

So, I could use some pointers.

Thanks,

--
Bill Moseley
mos...@hank.org

[Bill Moseley]

On Sun, Jun 30, 2013 at 2:43 PM, Bill Moseley <mos...@hank.org> wrote:

Sun Jun 30 14:37:12 2013] [error] [client 127.0.0.1] Failed to read from socket [Apache2::RequestRec=SCALAR(0x7f9d220ab7c0)] [0] at /home/bill/perl5/perlbrew/perls/perl-5.14.2/lib/site_perl/5.14.2/Plack/Middleware/DeChunked.pm line 40.\n

Well, never mind.

$env->{'psgi.input'}->read works fine, obviously.   The problem is that $c->read reads unchunked data.  But $env->{HTTP_TRANSFER_ENCODING} still  says 'chunked' and there's no CONTENT_LENGTH set.

Is that expected?

--
Bill Moseley
mos...@hank.org