Truncated json response

[mknag...@gmail.com]

Hi,

I have implemented an API to retrieve data from database, I am able to retrieve complete data from database to API but from the ngrest getting truncate output.

I checked the bug #58: https://github.com/loentar/ngrest/issues/58 but the ngrest which i have is updated one and MemPool.* has fixed code but still the symptoms is same as bug #58  when the response body size exceeds 64k it gets truncated.

Output from the API:

[{

"r_index":1535,

"bandwidth":"30000000 Kbit",

"description":"** ad00cbr65 VOD STREAMING B-Side **",

"encapsulation":"802",

"hardware_type":"VLAN sub-interface(s)",

"input_bit_rate":0,

"input_broadcast_packets":451207,

"input_bytes":1102158834,

"input_multicast_packets":3272133,

"input_packet_rate":0,

"input_packets":3817234,

"input_rate_time":"5 minute",

"link_status":"up",

"mtu":"9196",

"output_bit_rate":113488000,

"timeupdated":"2018-10-09 02:34:00",

"total_input_drops":22,

"total_output_drops":24,

"unrecognized_upper_level_protocol_drops":0

},

{

"r_index":1535,

"bandwidth":"30000000 Kbit",

"description":"** ad00cbr65 VOD STREAMING B-Side **",

"encapsulation":"802",

"hardware_type":"VLAN sub-interface(s)",

"input_bit_rate":0,

"input_broadcast_packets":451207,

"input_bytes":1102158834,

"input_multicast_packets":3272133,

"input_packet_rate":0,

"input_packets":

Could you please help me to fixing this issue.Thank you.

[Dmitry Utkin]

Hello,

do you use web server deployed ngrest server or simple ngrest server ?

среда, 21 ноября 2018 г., 15:20:01 UTC+3 пользователь m***@gmail.com написал:

[devus...@gmail.com]

Hi,

I work in the same project. We are using nginx as our webserver.

Thanks

[mknag...@gmail.com]

HI Dmitry,

we are using Nginx (nginx version: nginx/1.14.0) as a webserver deployed above ngrest on Ubuntu 16.04 Virtual Machine.   

[Dmitry Utkin]

Hi,

I tried both ngrest simple server and nginx module. ngrest simple server works just well even on large responses > 100MB. 

$ wget -S http://localhost:9098/ngrest/test/get1

--2018-11-24 12:51:03--  http://localhost:9098/ngrest/test/get1

Resolving localhost (localhost)... 127.0.0.1

Connecting to localhost (localhost)|127.0.0.1|:9098... connected.

HTTP request sent, awaiting response... 

  HTTP/1.1 200 OK

  Content-Type: application/json

  Server: ngrest

  Content-Length: 108277792

  Date: Sat, Nov 24 2018 09:51:05 GMT

  Connection: Keep-Alive

Length: 108277792 (103M) [application/json]

Saving to: ‘get1’

get1                                     100%[=================================================================================>] 103,26M   449MB/s    in 0,2s    

2018-11-24 12:51:05 (449 MB/s) - ‘get1’ saved [108277792/108277792]

nginx module works when response is less than about 2 MB.

$ wget -S http://localhost/ngrest/test/get1

--2018-11-24 12:46:26--  http://localhost/ngrest/test/get1

Resolving localhost (localhost)... 127.0.0.1

Connecting to localhost (localhost)|127.0.0.1|:80... connected.

HTTP request sent, awaiting response... 

  HTTP/1.1 200 OK

  Server: nginx/1.10.1

  Date: Sat, 24 Nov 2018 09:46:26 GMT

  Content-Type: application/json

  Content-Length: 2107792

  Connection: close

Length: 2107792 (2,0M) [application/json]

Saving to: ‘get1’

get1                                     100%[=================================================================================>]   2,01M  --.-KB/s    in 0,004s  

2018-11-24 12:46:26 (568 MB/s) - ‘get1’ saved [2107792/2107792]

but when response is larger than about 2,5 MB it may stuck or close connection.

$ wget -S http://localhost/ngrest/test/get1

--2018-11-24 12:49:08--  http://localhost/ngrest/test/get1

Resolving localhost (localhost)... 127.0.0.1

Connecting to localhost (localhost)|127.0.0.1|:80... connected.

HTTP request sent, awaiting response... 

  HTTP/1.1 200 OK

  Server: nginx/1.10.1

  Date: Sat, 24 Nov 2018 09:49:08 GMT

  Content-Type: application/json

  Content-Length: 21477792

  Connection: close

Length: 21477792 (20M) [application/json]

Saving to: ‘get1.1’

get1.1                                    98%[===============================================================================>  ]  20,10M  --.-KB/s    eta 0s     

this may be related to the problem you described. 

By the module logs all memory blocks are written to the chain, but ngx_http_output_filter returns NGX_AGAIN error upon request finalization. I'm not sure how to handle it correctly. This may be the error. Because when response is smaller, and it works correctly ngx_http_output_filter returns no error.

I'm not expert on creating nginx modules, so any help of fixing this is appreciated.

четверг, 22 ноября 2018 г., 8:34:46 UTC+3 пользователь m*****4...@gmail.com написал: