posting photoset using API v2 results in a memory limit error (data to be posted is too large if urlencoded)

[matteo bononi]

Good day all.

I'm developing a basic script that posts images on tumblr, everything is working fine since I've developed the photoset posting.

actually, I'm using these 2 function in order to retrieve the oAuth:

function oauth_gen($method, $url, $iparams, &$headers) {    

    $iparams['oauth_consumer_key'] = CONSUMER_KEY;

    $iparams['oauth_nonce'] = strval(time());

    $iparams['oauth_signature_method'] = 'HMAC-SHA1';

    $iparams['oauth_timestamp'] = strval(time());

    $iparams['oauth_token'] = OAUTH_TOKEN;

    $iparams['oauth_version'] = '1.0';

    $iparams['oauth_signature'] = oauth_sig($method, $url, $iparams);

    

    $oauth_header = array();

    foreach($iparams as $key => $value) {

        if (strpos($key, "oauth") !== false) { 

           $oauth_header []= $key ."=".$value;

        }

    }

    $oauth_header = "OAuth ". implode(",", $oauth_header);

    $headers["Authorization"] = $oauth_header;

}

function oauth_sig($method, $uri, $params) {

    

    $parts []= $method;

    $parts []= rawurlencode($uri);

   

    $iparams = array();

    ksort($params);

    foreach($params as $key => $data) {

            if(is_array($data)) {

                $count = 0;

                foreach($data as $val) {

                    $n = $key . "[". $count . "]";

                    $iparams []= $n . "=" . rawurlencode($val);

//$iparams []= $n . "=" . $val;

                    $count++;

                }

            } else {

                $iparams[]= rawurlencode($key) . "=" .rawurlencode($data);

            }

    }

    $parts []= rawurlencode(implode("&", $iparams));

//debug($parts,"parts");

//die();

    $sig = implode("&", $parts);

    return base64_encode(hash_hmac('sha1', $sig, CONSUMER_SECRET."&". OAUTH_SECRET, true));

}

the bold line is the one that gives the out of memory error on the server, I've actually 68Mb of memory that I cannot modify, I've noticed this:

if I want to post let's say 4 images, I pass an array with the binary content of the images, which could be about 4Mb (let's say about 1Mb for each image), but once the data is beeing urlencoded, this data become more than 12Mb.

this is an example, just to understand:
this is the first bytes of an image:

?? JFIF    ??

 

 ?? ?? k ??            

??

the urlencoded string is:

%EF%BF%BD%EF%BF%BD+%10JFIF+%01%01++%01+%01++%EF%BF%BD%EF%BF%BD+%03%02%02%02%02%02%03%02%02%02%03%03%03%03%04%06%04%04%04%04%04%08%06%06%05%06%09%08%0D%0A%0D%0A%09%08%09%09%0D%0A%0C%0F%0C%0D%0A%0B%0E%0B%09%09%0D%0A%11%0D%0A%0E%0F%10%10%11%10%0D%0A%0C%12%13%12%10%13%0F%10%10%10%EF%BF%BD%EF%BF%BD%01%03%03%03%04%03%04%08%04%04%08%10%0B%09%0B%10%10%10%10%10%10%10%10%10%10%10%10%10%10%10%10%10%10%10%10%10%10%10%10%10%10%10%10%10%10%10%10%10%10%10%10%10%10%10%10%10%10%10%10%10%10%10%10%10%10%EF%BF%BD%EF%BF%BD%08%02k%02%1C%03%01%11+%02%11%01%03%11%01%EF%BF%BD%EF%BF%BD+++%07%01%01%01%01++++++++++%02%03%04%05%06%07%01%08%09%0D%0A%EF%BF%BD%EF%BF%BD%10+%02%01%03%02

this is very very long, for a bunch of bytes...

does anyone has faced this problem? does anyone has some clues on how solve it? is it possible to solve it?

[matteo bononi]

I've tried to base64_encode the gif data before making the call, with no luck, tumblr answer with a bad request error.

[matteo bononi]

I'm push the conversation forward here:

http://stackoverflow.com/questions/36747697/oauth-signature-creation-issue-with-php

[Michael Ceyko]

Hey Matteo,

Unfortunately, I don't think there's any way to make the url-encoded string any smaller - aside from encoding a smaller image :) You're probably going to need to up the amount of memory allocated per php request. Check out the memory_limit option in php.ini.

Alternatively, there are other ways you can upload photos too: base64-encoded files, and multipart/form-data encoded requests.

However, keep in mind that for multipart form OAuth signatures, the tumblr api goes slightly off-spec. The RFC specifies that only x-www-form-urlencoded requests should include the request body in the signature. For tumblr api requests, a multipart form request's body is also included in the signature. All non-file parameters in the body must also be included in calculating the signature. The logic to do so can be a bit complex, so we recommend using one of our official api clients instead. You can find them here: http://developers.tumblr.com/clients

Regards,

Mike Ceyko

[matteo bononi]

Hi Mike! thanks for your answer,

the problem is only when I try to post a photoset, not a "normal" photo, also because having a hard limit to 2Mb for gif files, normal photos (also encoded) will never rise too much the memory usage.

The official Tumblr client is not an option, I cannot install other dependencies it needs.

The main logical problem is the oAuth flow, wich seems to be weird, I mean, on the documentation it says that there are a number of field you have to send in order to have a token:

$iparams['oauth_consumer_key'] = CONSUMER_KEY;

    $iparams['oauth_nonce'] = strval(time());

    $iparams['oauth_signature_method'] = 'HMAC-SHA1';

    $iparams['oauth_timestamp'] = strval(time());

    $iparams['oauth_token'] = OAUTH_TOKEN;

    $iparams['oauth_version'] = '1.0';

    $iparams['oauth_signature'] = oauth_sig($method, $url, $iparams);

which sounds legit. The weird part is that if you don't send the entire array of parameters of the real call, the authorization header will be rejected once you will post your contents using that signature.

this is also been pointed by the offical helpdesk which answered me in this way:

You shouldn't need to include the files in the parameters used for generating the oauth signature. 

from all the test I've done, this assertion is false, because I actually DO NEED to send the entire post request to be included in the signature in order to be able to post on Tumblr, this lead exactly yo the problem I have, so, the limits that Tumblr has given in the documentation refer to the URLENCODED files, not the plain files.

I cannot use another way to attach the fiiles due to the fact that photoset are an array of urlencoded binary contents.

[Michael Ceyko]

If you are sending the request as x-www-form-urlencoded, as it seems like you are from your StackOverflow post, then, yes, all parameters must be included in the body.

The answer you received regarding leaving some parameters out of the OAuth signature is specifically for multipart/form-data requests. For these, you must not include any file parameters in the OAuth signature.

In your case, it seems like you should switch to using multipart/form-data requests, so you don't need to include the image data in the signature calculation

As an alternative way of doing this, you say that you cannot install the dependencies of the tumblr.php api client, correct? However, have you tried a manual "install" of the php client's files instead? Just install the client on your local machine via composer require tumblr/tumblr. Then copy the files that it has downloaded into the repo for your project, and upload them onto the remote host. It's definitely not a recommended way of installing a package, but it may work around your issues.

[matteo bononi]

Mike... OMG. you simply save my day. Really, it is something like 2 months since I first saw this problem and was really an headache!

I really didn't notice the difference of the request. So simple!!

Thanks a lot, that change totally everything, by the way, if you like to answer on stackoverflow, then I'll accept your answer (if you have a user there) otherwise I'll credit you on the answer I'll post there.

also the "manual install" is a very good idea, I'll surely try that also for take a look at the standard client, but my problem is not only the lack of composer on the server, is also because I've a very limited scope there and I must keep everything very very "simple" without files and libraries that aren't totally handwritten (or at least controlled).