Problem with chunked transfer

[Thierry Florac]

Hi,

My Pyramid application is supporting partial contents transfer for requests using "range" headers, actually to serve PDF and videos files.

The application is deployed behind an HAProxy with Apache and mod_wsgi; after using this feature without any problem for years (with Apache 2.4.25, mod_wsgi 4.5.11 and Python 3.5), I add to upgrade my configuration recently (with Apache 2.4.62, mod_wsgi 4.9.4 and Python 3.11) and I now get errors in Apache logs:

>>>  Received request requiring chunked transfer encoding, but optional support for chunked transfer encoding has not been enabled.

I found references to topics saying that WSGI protocol doesn't support chunked transfer encoding natively, and to a configuration directive (WSGIChunkedRequest) that I added to my Apache configuration. Now the error disappeared, but files download is still really very very long!

Would you have any idea about the origin of this issue?

Best regards,

Thierry

-- 

  https://www.ulthar.net -- http://pyams.readthedocs.io

[Jonathan Vanasco]

Have you seen these?

https://github.com/GrahamDumpleton/mod_wsgi/issues/312

and 

https://modwsgi.readthedocs.io/en/develop/configuration-directives/WSGIChunkedRequest.html

[Thierry Florac]

Hi Jonathan,

Thank you for the links, but I already read them (and that's why I added the WSGIChunkedRequest parameter to mod_wsgi configuration).

After searching a while and making many tests, the problem doesn't seem to be at the application level, but at the system level!

I use an NFS share with cachefilesd on my web frontends to store external files (I use ZODB blobs with Relstorage), and it's probably this component which doesn't work as expected; if I get access to these resources through the server where files are stored (on which the web application is also deployed), it works and performances are as expected.

I'm now trying to find the origin of this NFS performance problem; if I can't make it work correctly, maybe I'll switch to Relstorage blobs stored in the PostgreSQL ZODB backend (with a RelStorage client cache), but this configuration generates very big PostgreSQL databases (I have 800 GB of data files) which are quite hard to handle...

Best regards,

Thierry

-- 

  https://www.ulthar.net -- http://pyams.readthedocs.io

--
You received this message because you are subscribed to the Google Groups "pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pylons-discus...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/pylons-discuss/b15cc5bf-ab90-4875-9fd0-b7b987c56478n%40googlegroups.com.

[Jonathan Vanasco]

Sorry those weren't helpful to you.  I had remembered that accidental lack of documentation from years ago.

In the past, I've offloaded this stuff to Apache/Nginx and used time-sensitive signed URLs.  That might be easier to implement if it's compatible with your setup.

[Thierry Florac]

Hi,

I investigated a bit more and after ruling out the possibility of network issues, it seems that the origin of this problem comes from the "cachefilesd" agent!

I disabled the service and files are then downloaded without any problem from the NFS server, so I'm going to look at Debian NFS support to get any information...

Best regards,

Thierry

-- 

  https://www.ulthar.net -- http://pyams.readthedocs.io

To view this discussion visit https://groups.google.com/d/msgid/pylons-discuss/f68759e6-1159-4881-bc28-2d6b7cf612b3n%40googlegroups.com.

[Florian Schulze]

> I'm now trying to find the origin of this NFS performance problem; if I
> can't make it work correctly, maybe I'll switch to Relstorage blobs stored
> in the PostgreSQL ZODB backend (with a RelStorage client cache), but this
> configuration generates very big PostgreSQL databases (I have 800 GB of
> data files) which are quite hard to handle...

Be aware that PostgreSQL has two different ways to store blobs and one is limited to 1 GB per blob.

Regards,
Florian Schulze