davrods + mod_cache_disk, cached file refresh problem

[Tony Edgin]

Hi everyone.

I've discovered that the cache for our davrods-based WebDAV service doesn't always refresh a stale entry. When a client makes a GET request for a file that has an expired, valid, cached copy, the client receives a 304 (NOT MODIFIED) response. By "expired, valid, cached copy", I mean that the copy has been in the cache longer than its expiration time, but it is still a valid copy of the data object in iRODS. What should happen is that the expiration time of the cached copy is updated, and the client receives a 200 (OK) with the cached copy being the body of the response.

I'm hoping someone can help me figure out why this is happening. We are using mod_cache and its storage module, mod_cache_disk, to manage a file cache local to the hosting server. It runs in the same vhost as davrods. Here's a minimal configuration that demonstrates the issue.

[apache]# cat /etc/httpd/conf.d/davrods-anonymous-vhost.conf

<VirtualHost *:80>
  ### CACHING CONFIGURATION
  CacheDetailHeader  On
  CacheEnable        disk /
  CacheRoot          /var/cache/httpd/proxy
 
  # Have cached files expire quickly
  CacheMaxExpire 1      
  ###

  ServerName 128.196.65.156

  <Location />
    DirectoryIndex disabled

    AuthType  None
    Require   all granted

    Dav davrods-locallock

    DavRodsEnvFile         /etc/httpd/irods/irods_environment.json
    DavRodsServer          data-dev.cyverse.rocks 1247
    DavRodsZone            cyverse.dev
    DavRodsAnonymousMode   On
    DavRodsAnonymousLogin  "anonymous" ""
    DavRodsExposedRoot     /cyverse.dev/home/shared
    DavRodsLockDB          /var/lib/davrods/lockdb_locallock
  </Location>
</VirtualHost>

[client]# curl -v http://128.196.65.156/MOTD
> GET /MOTD HTTP/1.1
> User-Agent: curl/7.29.0
> Host: 128.196.65.156
> Accept: */*
>
< HTTP/1.1 200 OK
< Date: Thu, 13 Aug 2020 20:22:07 GMT
< Server: Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips
< Last-Modified: Thu, 13 Aug 2020 19:12:00 GMT
< ETag: "14-01597345920"
< Accept-Ranges: bytes
< Content-Length: 20
< X-Cache-Detail: "cache miss: attempting entity save" from 128.196.65.156
<
Thanks for helping!

[client]# curl -v http://128.196.65.156/MOTD

> GET /MOTD HTTP/1.1
> User-Agent: curl/7.29.0
> Host: 128.196.65.156
> Accept: */*
>
< HTTP/1.1 304 Not Modified
< Date: Thu, 13 Aug 2020 20:22:10 GMT
< Server: Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips
< ETag: "14-01597345920"
< 

I've found a workaround. If the davrods configuration is moved into a private vhost, and mod_proxy is used to convert the publicl vhost into a reverse proxy for the private one, caching works as expected. Here's the modified configuration.

[apache]# cat /etc/httpd/conf.d/davrods-anonymous-vhost.conf

<VirtualHost *:80>
  ### CACHING CONFIGURATION
  CacheDetailHeader  On
  CacheEnable        disk /
  CacheRoot          /var/cache/httpd/proxy
 
  # Have cached files expire quickly
  CacheMaxExpire 1
  ###

  ServerName 128.196.65.156

  ProxyPass / http://127.0.0.1/
</VirtualHost>

<VirtualHost 127.0.0.1:80>
  ServerName 128.196.65.156

  <Location />
    DirectoryIndex disabled

    AuthType  None
    Require   all granted

    Dav davrods-locallock

    DavRodsEnvFile         /etc/httpd/irods/irods_environment.json
    DavRodsServer          data-dev.cyverse.rocks 1247
    DavRodsZone            cyverse.dev
    DavRodsAnonymousMode   On
    DavRodsAnonymousLogin  "anonymous" ""
    DavRodsExposedRoot     /cyverse.dev/home/shared
    DavRodsLockDB          /var/lib/davrods/lockdb_locallock
  </Location>
</VirtualHost>

[client]# curl -v http://128.196.65.156/MOTD

> GET /MOTD HTTP/1.1
> User-Agent: curl/7.29.0
> Host: 128.196.65.156
> Accept: */*
>
< HTTP/1.1 200 OK
< Date: Thu, 13 Aug 2020 20:47:02 GMT
< Server: Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips
< Last-Modified: Thu, 13 Aug 2020 19:12:00 GMT
< ETag: "14-01597345920"
< Accept-Ranges: bytes
< Content-Length: 20
< X-Cache-Detail: "cache miss: attempting entity save" from 128.196.65.156
<
Thanks for helping!

[client]# curl -v http://128.196.65.156/MOTD

> GET /MOTD HTTP/1.1
> User-Agent: curl/7.29.0
> Host: 128.196.65.156
> Accept: */*
>
< HTTP/1.1 200 OK
< Date: Thu, 13 Aug 2020 20:47:06 GMT
< Server: Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips
< Last-Modified: Thu, 13 Aug 2020 19:12:00 GMT
< Accept-Ranges: bytes
< Content-Length: 20
< ETag: "14-01597345920"
< X-Cache-Detail: "conditional cache hit: entity refreshed" from 128.196.65.156
<
Thanks for helping!

According to the the documentation for mod_cache (http://httpd.apache.org/docs/2.4/mod/mod_cache.html), it is implemented as a filter and can be placed in front of any content generator. See http://httpd.apache.org/docs/current/filter.html for more details on filters and content generators. Both davrods and mod_proxy are content generators, so the indirection of having davrods accessed through mod_proxy shouldn't be needed. It feels like I'm working around a bug.

Does anyone have any idea about what I'm doing wrong in my original configuration?

Thanks in advance!

Tony Edgin

[tedgin...@gmail.com]

Here's a follow up.

I did some more testing.  I think this is a bug in davrods. I've submitted an issue on the davrods repository. See https://github.com/UtrechtUniversity/davrods/issues/22.

Cheers,

Tony