Fetch gzip-compressed content [Net::HTTP::Spore 0.06]

[Adam Perlman]

I would like to fetch gzip'd API content and there seems to be plenty of hooks built into Net::HTTP::Spore but I'm stumped as to where to start.

According to:

http://stackoverflow.com/questions/1285305/how-can-i-accept-gzip-compressed-content-using-lwpuseragent

my $response = $ua->get('http://stackoverflow.com/feeds', 
    'Accept-Encoding' => HTTP::Message::decodable);

print $response->decoded_content;

I don't need to change the UserAgent, but I do need to change the response call and decode the results.

I crawled through the documentation but I couldn't find an example of changing the response call.  Furthermore, how would I change the result from 'content' to 'decoded_content'?

TIA

[Stéphane Raux]

Hi Adam,

I think the best solution would be to create a middleware wich would
handle the decoding of the results.
If you are using the perl implementation, you can take a look to
Net::HTTP::Spore::Middleware::UserAgent in order to get a simple
example of how to change the request header.

Regards,

Stéphane

> --
> You received this message because you are subscribed to the Google Groups
> "SPORE" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to spore-rest+...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Stéphane Raux]

And if you need to change the result, you can also look at
Net::HTTP::Spore::Middleware::Format::*

[Adam Perlman]

Thanks for the quick reply - I took your advice and created a new Middleware package with the following:

-----

use Compress::Zlib;

use Moose;

extends 'Net::HTTP::Spore::Middleware::Format';

sub encode       { compress($_[1])   }

sub decode       { uncompress($_[1]) }

sub accept_type  { ( 'Accept-Encoding' => 'gzip' ) }

-----

I can see the header is sending correctly and the web server is returning a gzip'ed response.  However, it's not uncompressing.  I want it to return an uncompress'ed XML file.

It works without the Format::Gzip line.

 

-----

$spore->enable('+Format::Gzip');

$spore->enable('Format::XML');

-----

I feel like I'm very close to a solution but I'm missing something... Thanks!

[Adam Perlman]

After mucking around a bit it's able to decode the gzip'd data:

------

use IO::Uncompress::Gunzip qw/gunzip/;

use Moose;

extends 'Net::HTTP::Spore::Middleware::Format';

sub decode       { my $out; gunzip(\$_[1], \$out); $out }

sub accept_type  { ( 'Accept-Encoding' => 'gzip' ) }

-------

However, even though I'm enabling:

---------

$spore->enable('+Format::Gzip');

$spore->enable('Format::XML');

---------

It's returning the raw XML feed.  Commenting out the Gzip package doesn't send the Accept-Encoding so it returns the parsed XML.  My thinking was that if I enabled multiple formats they would be processed in the order they were enabled.  Does one override the other?  Should I be calling this in some other way?

TIA, 

--
You received this message because you are subscribed to a topic in the Google Groups "SPORE" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/spore-rest/r2eDe64EJTo/unsubscribe.
To unsubscribe from this group and all its topics, send an email to spore-rest+...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
~~ adam ~~

[Adam Perlman]

[Maybe this will help someone else]

It looks like you can't combine 'enables' so my solution for adding 'gzip' is messy but in the end this worked.

Combine the two functions together into one enable and comment out the base Format::XML package.

-----

$spore->enable('+Format::GzipXML');

#  $spore->enable('Format::XML');

-----

Then extend the Format::XML Middleware in your custom Gzip package

-----

use IO::Uncompress::Gunzip qw/gunzip/;

use Moose;

extends 'Net::HTTP::Spore::Middleware::Format::XML';

sub decode       {

   my $out; 

   gunzip(\$_[1], \$out);

   $_[0]->SUPER::decode($out)

}

sub accept_type { ( 'Accept-Encoding' => 'gzip', 'Accept' => 'text/xml' ) }

-----

This isn't great because it means that you can't just enable a universal Gzip format.  If there's a better way, I'd really like to hear it.

[Stéphane Raux]

Hi Adam,

You can combine the middlewares, but you must pay attention to their
order of execution, as they work like a stack :

If you do the following :

$spore->enable('+Format::Gzip');

$spore->enable('Format::XML');

Then the Gzip Middleware encode() function will be called first then
the XML encode() function, and the XML decode() function will be
called before the Gzip decode.

In other words, the encode() functions of your middlewares will be
called in the same order of their declaration, and the decode()
functions will be called in the reverse order.

I think just declaring the XML middleware before the Gzip would be
fine in your case.

Hope that helps,

Stéphane

[Adam Perlman]

Somewhere I'm misunderstanding what you're saying.

I changed the Gzip package to remove the XML call:

-----

use IO::Uncompress::Gunzip qw/gunzip/;

use Moose;

extends 'Net::HTTP::Spore::Middleware::Format';

sub decode {

   my $out;

   gunzip(\$_[1], \$out);

   $out

}

sub accept_type { ( 'Accept-Encoding' => 'gzip' ) }
-----

...then reversed the order of the calls as you suggested:

-----

$spore->enable('Format::XML');

$spore->enable('+Format::Gzip');

-----

When I run it, the sending headers now include:

> Accept: text/xml

but no longer contain:

> Accept-Encoding: gzip

The return headers no longer contain:

< Content-Encoding: gzip

Even though I can see that it's loading the Gzip package:

== enabling middleware Format::Gzip

Reviewing the Content-Length header for each call reveals that the return values are definitely not gzip'd in this scenario.

On Tuesday, May 13, 2014 9:24:05 AM UTC-7, Adam Perlman wrote: