Re: [Django] #9249: Google Analytics' Cookies break CacheMiddleware when SessionMiddleware turns on Vary: Cookie
Django
turns on Vary: Cookie
---------------------------------------+-------------------------------
Reporter: pixelcort | Owner: vvd
Type: New feature | Status: assigned
Milestone: | Component: HTTP handling
Version: 1.0 | Severity: Normal
Resolution: | Keywords: cache cookies
Triage Stage: Accepted | Has patch: 1
Needs documentation: 0 | Needs tests: 0
Patch needs improvement: 1 | Easy pickings: 0
UI/UX: 0 |
---------------------------------------+-------------------------------
Changes (by lukeplant):
* ui_ux: => 0
* easy: => 0
Comment:
Regarding regexes - I would favour using compiled regexes - this is
consistent with other settings that are regexes. (Only the URL conf
appears to be different here).
--
Ticket URL: <https://code.djangoproject.com/ticket/9249#comment:6>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
Django
turns on Vary: Cookie
Type: New feature | Status: assigned
Severity: Normal | Resolution:
Keywords: cache cookies | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
Comment (by anonymous):
Instead of listing individual cookies explicitly, I feel it would be
better to have Django keep a record on whether or not a cookie was been
accessed. This can be done in similar fashion as to how Django currently
checks whether or not the session was accessed.
Benefits:
- This would work without any configuration. Any additional cookies set by
whatever frontend Javascript code that are not used by Django views would
automatically be ignored.
- No new setting & accompanying documentation
--
Ticket URL: <https://code.djangoproject.com/ticket/9249#comment:7>
Django
turns on Vary: Cookie
-------------------------------+------------------------------------
Reporter: pixelcort | Owner: vvd
Type: New feature | Status: assigned
Component: HTTP handling | Version: 1.0
Severity: Normal | Resolution:
Keywords: cache cookies | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
-------------------------------+------------------------------------
* cc: raymond.penners@… (added)
--
Ticket URL: <https://code.djangoproject.com/ticket/9249#comment:8>
Django
turns on Vary: Cookie
-------------------------------+------------------------------------
Reporter: pixelcort | Owner: vvd
Type: New feature | Status: assigned
Component: HTTP handling | Version: 1.0
Severity: Normal | Resolution:
Keywords: cache cookies | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
-------------------------------+------------------------------------
* cc: django@… (added)
--
Ticket URL: <https://code.djangoproject.com/ticket/9249#comment:9>
Django
turns on Vary: Cookie
-------------------------------+------------------------------------
Reporter: pixelcort | Owner: vvd
Type: New feature | Status: assigned
Component: HTTP handling | Version: 1.0
Severity: Normal | Resolution:
Keywords: cache cookies | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
-------------------------------+------------------------------------
* cc: harm.verhagen@… (added)
--
Ticket URL: <https://code.djangoproject.com/ticket/9249#comment:10>
Django
turns on Vary: Cookie
-------------------------------+------------------------------------
Reporter: pixelcort | Owner: vvd
Type: New feature | Status: assigned
Component: HTTP handling | Version: 1.0
Severity: Normal | Resolution:
Keywords: cache cookies | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
-------------------------------+------------------------------------
Comment (by harm):
The suggestion mentioned in comment:6: "automatically only taking into
account the actual used cookies in the cache key" that would work also in
the following case:
* csrf middleware is enabled (so user sends csrftoken cookie every
request.
* Some view A depends on cookie foobar_enabled (2 values), but that
specific view does NOT use any csrf token.
In the current situation caching view A does not work between clients
(as different csrf tokens cooies, cause different cache keys in the view
that doesn't use this cookie)
With suggestion comment:6 this would automatically work.
--
Ticket URL: <https://code.djangoproject.com/ticket/9249#comment:11>
Django
turns on Vary: Cookie
-------------------------------+------------------------------------
Reporter: pixelcort | Owner: vvd
Type: New feature | Status: assigned
Component: HTTP handling | Version: 1.0
Severity: Normal | Resolution:
Keywords: cache cookies | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
-------------------------------+------------------------------------
Comment (by KyleMac):
It was easy to monkey patch `django.http.parse_cookie` to use a custom
dictionary that logs gets and sets but I had to take into account some
things.
1. This CSRF middleware accesses `CSRF_COOKIE_NAME` on every request so
ignore that and check `request.META.get['CSRF_COOKIE_USED']` instead.
2. The session middleware accesses `SESSION_COOKIE_NAME` on every request
so ignore that and check `request.session.accessed` instead.
3. There may be other contrib middleware I'm not using and the best
solution would be for everything to not access their cookies until needed.
However, it turns out that IE and Firefox have the same issue but client
side. The Google Analytics cookies will cause them to completely
invalidate their cache and will not even send `If-Modified-Since` and so
any site that uses `Vary: Cookie` and Google Analytics effectively has no
client side caching for the majority of their users.
In Chrome requests go something like the following (Opera and Safari seem
similar but I've tested them less).
1. Chrome requests a page and gets the following:
{{{
Cache-Control: public, max-age=600
Vary: Cookie
}}}
2. If you navigate away from the page and come back to it (the actual
reload button sometimes behaves differently in Chrome to other browsers)
and the page hasn't expired (i.e. `max-age`) then SOMETIMES it will just
serve that from the cache. I don't really understand this and it's
probably some kind of heuristics.
3. Otherwise request a new page with the `If-Modified-Since` and the
contents of our cookies. Due to my monkey patch and custom decorator it
gets a 304 response and all is good.
In IE (I tested 9 and 10) and Firefox (currently 14) it's more like the
following.
1. Receive a page with the same headers as above.
2. Once the page is loaded, Google Analytics updates it cookies and now
the page is immediately completely invalidated.
3. Nothing is ever served from the local cache and so a new request is
sent. However since the page was invalidated in step 2 not even an `If-
Modified-Since` header is sent and so you get a full 200 response every
single time.
I might now go in the opposite direction and strip out `Vary: Cookie` on
every response and raise an exception if any cached view tries to access
cookies it wasn't meant to.
--
Ticket URL: <https://code.djangoproject.com/ticket/9249#comment:12>
Django
turns on Vary: Cookie
-------------------------------+------------------------------------
Reporter: pixelcort | Owner: vvd
Type: New feature | Status: assigned
Component: HTTP handling | Version: 1.0
Severity: Normal | Resolution:
Keywords: cache cookies | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
-------------------------------+------------------------------------
Comment (by KyleMac):
I've now managed to get this to work in Internet Explorer and Firefox.
The fix for IE is quite simple and is due to it sending a non standard
`If-Modified-Since` header that ConditionalGetMiddleware fails to parse.
I've opened ticket #18648 for that.
While Firefox won't send `If-Modified-Since` with `Vary: Cookie` it will
send back the Etag. So to get Firefox to work as expected all you need to
do is use a hash of `Last-Modified` as the Etag (assuming there isn't
already a proper Etag for the response).
--
Ticket URL: <https://code.djangoproject.com/ticket/9249#comment:13>
Django
turns on Vary: Cookie
-------------------------------+------------------------------------
Reporter: pixelcort | Owner: vvd
Type: New feature | Status: assigned
Component: HTTP handling | Version: 1.0
Severity: Normal | Resolution:
Keywords: cache cookies | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
-------------------------------+------------------------------------
* cc: trbs@… (added)
--
Ticket URL: <https://code.djangoproject.com/ticket/9249#comment:14>
Django
turns on Vary: Cookie
-------------------------------+------------------------------------
Reporter: pixelcort | Owner: vvd
Component: HTTP handling | Version: 1.4-rc-2
Severity: Normal | Resolution:
Keywords: cache cookies | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
-------------------------------+------------------------------------
* needs_docs: 0 => 1
* version: 1.0 => 1.4-rc-2
* type: New feature => Bug
--
Ticket URL: <https://code.djangoproject.com/ticket/9249#comment:15>
Django
turns on Vary: Cookie
-------------------------------+------------------------------------
Reporter: pixelcort | Owner: vvd
Component: HTTP handling | Version: 1.0
Severity: Normal | Resolution:
Keywords: cache cookies | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 1
Needs tests: 0 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
-------------------------------+------------------------------------
* version: 1.4-rc-2 => 1.0
* type: Bug => New feature
Old description:
> When using Google Analytics on a Django project with CacheMiddleware and
> SessionMiddleware turned on, the Cookies that Google Analytics apparently
> change on each reload, invalidating the Vary: Cookie parameter that
> SessionMiddleware is setting.
>
> There should be a way to define cookie prefixes, such as '__utm', to
> ignore for cookie variation for caching.
New description:
When using Google Analytics on a Django project with CacheMiddleware and
SessionMiddleware turned on, the Cookies that Google Analytics apparently
change on each reload, invalidating the Vary: Cookie parameter that
SessionMiddleware is setting.
There should be a way to define cookie prefixes, such as ``'__utm'``, to
ignore for cookie variation for caching.
--
Comment:
If you look at the discussion, this isn't a bug; it's really a new
feature.
The version field tracks the version the bug was reported in.
Related: #15201. Caching is hard.
--
Ticket URL: <https://code.djangoproject.com/ticket/9249#comment:16>
Django
turns on Vary: Cookie
Reporter: pixelcort | Owner: anonymous
Type: New feature | Status: assigned
Component: HTTP handling | Version: 1.0
Severity: Normal | Resolution:
Keywords: cache cookies | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 1
Needs tests: 0 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
Changes (by anonymous):
* owner: vvd => anonymous
--
Ticket URL: <https://code.djangoproject.com/ticket/9249#comment:17>
Django
turns on Vary: Cookie
-------------------------------+------------------------------------
Type: New feature | Status: assigned
Component: HTTP handling | Version: 1.0
Severity: Normal | Resolution:
Keywords: cache cookies | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 1
Needs tests: 0 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
-------------------------------+------------------------------------
* owner: anonymous => ambv
--
Ticket URL: <https://code.djangoproject.com/ticket/9249#comment:18>
Django
turns on Vary: Cookie
-------------------------------+------------------------------------
Reporter: pixelcort | Owner: ambv
Type: New feature | Status: assigned
Component: HTTP handling | Version: 1.0
Severity: Normal | Resolution:
Keywords: cache cookies | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 1
Needs tests: 0 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
-------------------------------+------------------------------------
Comment (by collinanderson):
A work around is to create a middleware that deletes all the cookies in
request.COOKIE that django should ignore. (Just don't delete the important
cookies. :) Example from DjangoCon.eu 2016: https://youtu.be/AZ4ISa1u-
HE?t=12548
--
Ticket URL: <https://code.djangoproject.com/ticket/9249#comment:19>
Django
turns on Vary: Cookie
-------------------------------+------------------------------------
Type: New feature | Status: new
Component: HTTP handling | Version: 1.0
Severity: Normal | Resolution:
Keywords: cache cookies | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 1
Needs tests: 0 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
-------------------------------+------------------------------------
* owner: Łukasz Langa => (none)
* status: assigned => new
--
Ticket URL: <https://code.djangoproject.com/ticket/9249#comment:20>
Django
turns on Vary: Cookie
Reporter: pixelcort | Owner: (none)
Type: New feature | Status: closed
Component: HTTP handling | Version: 1.0
Keywords: cache cookies | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Mariusz Felisiak):
* status: new => closed
* needs_better_patch: 1 => 0
* needs_docs: 1 => 0
* has_patch: 1 => 0
* resolution: => wontfix
* stage: Accepted => Unreviewed
Comment:
This issue is rather niche, moreover, the
[https://code.djangoproject.com/ticket/9249#comment:19 Collin's
workaround] is straightforward and can be implemented by any app on its
own. As far as I'm aware it's not something that Django itself has to
provide.
--
Ticket URL: <https://code.djangoproject.com/ticket/9249#comment:21>