Proxying a large response.

[rasta]

Hi!

I'm using mojolicious 6.11. I have a proxy environment like this:

apache -> hypnotoad -> tomcat

and the proxy code looks like this:

sub proxy {

...

  if (Mojo::IOLoop->is_running) {

    $self->render_later;

    $self->ua->get(

      $url,

      sub {

        my ($c, $tx) = @_;

        _proxy_tx($self, $tx);

      }

    );

  }else {

    my $tx = $self->ua->get($url);

    _proxy_tx($self, $tx);

  }

}

sub _proxy_tx {

  my ($c, $tx) = @_;

  if (my $res = $tx->success) {

    $c->tx->res($res);

    $c->rendered;

  }

  else {

    my $error = $tx->error;

    $c->tx->res->headers->add('X-Remote-Status', $error->{code} . ': ' . $error->{message});

    $c->render(status => 500, text => 'Failed to fetch data: '.$c->app->dumper($tx->error));

  }

}

Am I doing this right at all? I thought it's going to stream the stuff through but that's not happening, mojo writes the data down to it's mojo.tmp file and it seems that nothing comes out unless it fetched the whole file from tomcat. The problem is it works for smaller files but when I'm proxying 2.7GB file after the mojo.tmp file reaches ~ 2.5GB I'll get 

The proxy server received an invalid response from an upstream server.
The proxy server could not handle the request GET /api/object/o:292171/diss/Content/download.

Reason: Error reading from remote server

from Apache. In tomcat log it seems the file was sent OK (the bytesize matches and the response is 200). Mojo will log

"Nothing has been rendered, expecting delayed response"

before it starts proxying but nothing afterwards.

Any idea what could be wrong here or how to do this better? Otherwise I'd have to send a redirect but I'd like to avoid that.

Thanks!

Rasta

[rasta]

I found this code from Mojolicious::Plugin::Proxy works like a charm:

$self->client->async->get($url, sub {

    my ($self, $tx) = @_;

            

    if (my $res=$tx->success) {

      $self->tx->res($res);

      $self->rendered;

    }else {

      my ($msg,$error) = $tx->error;

      $self->tx->res->headers->add('X-Remote-Status',$error.': '.$msg); 

      $self->render(

        status => 500,

        text => 'Failed to fetch data from backend'

      );

    }

  })->start;

[Rastislav Hudak]

Meh, it doesn't, I was testing it wrong :-/

--
You received this message because you are subscribed to a topic in the Google Groups "Mojolicious" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/mojolicious/QaVw_wk1JnY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to mojolicious+unsubscribe@googlegroups.com.
To post to this group, send email to mojol...@googlegroups.com.
Visit this group at https://groups.google.com/group/mojolicious.
For more options, visit https://groups.google.com/d/optout.

[sri]

There are ways to do this efficiently, but i don't think anyone has ever written a full implementation before. It

requires a few things to be combined.

    https://github.com/kraih/mojo/blob/0c79e9a8b50d23d04f6edca6e52d1c179305f3c9/t/mojo/ioloop.t#L219

    http://mojolicious.org/perldoc/Mojolicious/Guides/Cookbook#Streaming-response

    http://mojolicious.org/perldoc/Mojolicious/Guides/Cookbook#Streaming-request

    http://mojolicious.org/perldoc/Mojolicious/Guides/Rendering#Streaming

It's really a bit tricky, but you can do this right and stream requests and responses in both directions. The

biggest problem is that you have to mess around with the streams to be able to handle backpressure and

throttle your reads.

It would even take me like a day to implement this, so i guess it would be nice if someone made a

Mojolicious::Plugin::StreamingProxy.

--

sebastian

[rasta]

Thanks for the pointers Sebastian. I personally miss the xp to develop this but at least I know the problem is not the code I'm using or wrong settings.