HTTP Processor Results Cut-off

[Mathew Tasalloti]

Hi Christoph,

We are using our Fusio server to forward a large volume of data (over 8K) to an external API customer.   At around 5K, it seems that our results are getting cut off and we can't make sense of the pattern.   The max size that we can get working seems to be 17 records passed back at around 8K of data.

Per the shots below, if we request 17 rows to be returned, we get the correct results.   However, if we request 18 rows to be returned, then the content is cut-off and the content length is presented to us as 3863 bytes - about half of what we got when we requested 17 rows.

Here are the results working with 17 records passed back.

Here are the snapshots of when we ask for 18 rows (just one more) - you can see the records cut off and count gone down by half:

Any idea what is happening here as we are not sure if there is somewhere in the settings where we can set the maximum or desired length of these buffers, if there is a hard max result set count or an issue to report.

Thanks in advance for your help !!!

Mathew

[Christoph Kappestein]

Hi Mathew,

Iam not sure whether this is the same case but another customer had a similar issue where the response of the HTTP processor was cut off, the reason here was that the remote server has send the response compressed with a Content-Length header of the compressed body, Fusio has then redirect this Content-Length header to the client but the body was not compressed since Fusio does not compress the response, then the body was cut-off because of the mismatching Content-Length, we have then commited a fix to always remove the Content-Length header from the remote response s. https://github.com/apioo/fusio-adapter-http/commit/51f53e3310a5307539aedcd454334194ac73102b this fix will then be available with the next release.

So could you check whether your remote server sends the response compressed or maybe deactivate the compression for testing? Otherwise Fusio simply forwards the response so there is no logic to cut off the response.

best regards
Christoph

[Mathew Tasalloti]

Hi Christoph,

First off, thanks so much for the very timely response and direction.   We've been looking into this using your insights and we "think" we see where the problem is.   The results sent by the source server are not beautified, however, when we get the results back through fusio, they seem to be beautified.   If this is correct, then the beautification spaces are what is causing the cut-off in the contents.   We've ensured that the Content-Length does in-fact match the data parcel being returned and not compression is being applied.

Here is what we are expecting to get back from our API call:

However, this is what is being forwarded to us (beautifed):

Here are some more insights:

- The Action seems to have all the data when we review it in the Designer

- The Route is set to Passthru

- We've also tried setting the Route to "No Schema"

Any thoughts on how we could turn off beautification perhaps (assuming this is the issue).

Thanks again for your help with this,

Mathew

[Mathew Tasalloti]

Hi Christoph,

I figured out a way of getting the data from our upstream server without having it cut off.   This might solve our problem until we can address the normal HttpProcessor method.

Here is the workaround:

1) I created a CLI curl action to send a query to the upstream server:    "/usr/bin/curl --request GET --url http://10.6.103:3000/function  --header ''username: someuser''"

2) I created a route and tied it to the action

Calling the route this way doesn't cut anything off ! we get the full results.

Question regarding this method:  is there any way to add the parameters that the user specifies into our CLI command?  Is there a way of getting the user id into the CLI for backend processing ?

Thanks!

Mathew

[Mathew Tasalloti]

Just sharing our finding here in case it helps others as well.

Comparing the latest Fusio code on our server with the Adapter Github repository, the following highlighted line in HttpEngine.php is missing in our server (line 155):

        $response    = $this->client->request($request->getMethod(), $url, $options);
        $contentType = $response->getHeaderLine('Content-Type');
        $response    = $response->withoutHeader('Content-Type');
        $response    = $response->withoutHeader('Content-Length');
        $body        = (string) $response->getBody();

I added it and we now seem to have no more cut-off of the JSON data.

Cheers,

Mathew

[Christoph Kappestein]

Hi Mathew,

just also for the public, so this is indeed the correct fix which can be used until we release a new version, the next release will then include this fix.

best regards
Christoph