cookiejar changes?

[tandalat]

Hello,

The current implementation of cookiejar.Cookies(url.URL) returns cookies with only attributes Name and Value being set. It would be more beneficial if they contain most attributes (with the exception of max-age) set in a response by the server as this enables code extracting cookies to extend further processing such as adding persistence storage. I'd prepare a patch if the request makes sense.

Thanks,

tan

[Nigel Tao]

Re persistent storage, there was a lot of discussion with Dr. Volker
Dobler (CC'ed) and I a couple of years ago about how to do a cookie
jar with pluggable storage, but we never landed on a satisfactory
design, let alone a perfect one, so what's in the standard library is
not persistent.

If you come to us with an existence proof of a good design for a
persistent cookie jar, I'm happy to look at it, but in the meantime,
if you want persistence, it''s probably best to fork
net/http/cookiejar for now. Once a change lands in the standard
library and is released, we can no longer change its API, and I don't
know what the right API is yet.

https://groups.google.com/forum/#!topicsearchin/golang-dev/cookiejar$20-%22broken$20by%22
and
https://groups.google.com/d/topic/golang-dev/7Jcbgrbah2s/discussion
might be places to start looking for previous discussions.

[Volker Dobler]

On Thu, Jul 30, 2015 at 3:49 AM, Nigel Tao <nige...@golang.org> wrote:
> On Thu, Jul 30, 2015 at 9:50 AM, tandalat <tan....@gmail.com> wrote:
>> The current implementation of cookiejar.Cookies(url.URL) returns cookies
>> with only attributes Name and Value being set. It would be more beneficial
>> if they contain most attributes (with the exception of max-age) set in a
>> response by the server as this enables code extracting cookies to extend
>> further processing such as adding persistence storage. I'd prepare a patch
>> if the request makes sense.
>
> Re persistent storage, there was a lot of discussion with Dr. Volker
> Dobler (CC'ed) and I a couple of years ago about how to do a cookie
> jar with pluggable storage, but we never landed on a satisfactory
> design, let alone a perfect one, so what's in the standard library is
> not persistent.
>
> If you come to us with an existence proof of a good design for a
> persistent cookie jar, I'm happy to look at it, but in the meantime,
> if you want persistence, it''s probably best to fork
> net/http/cookiejar for now. Once a change lands in the standard
> library and is released, we can no longer change its API, and I don't
> know what the right API is yet.
>

Yes, a persistent cookiejar with proper error handling will be very hard.

But If I understand Tan correctly he is suggesting to just add some
API to retrieve all cookie entries from the jar. Maybe something like

// UsedETLDs returns all effective top level domains for which
// the Jar contains cookies.
func (j *Jar) UsedETLDs() []string

// CookiesFor returns all stored (and not expired) cookies for
// the given effective top level domain etld.
func (j *Jar) CookiesFor(etld string) []Entry

with Entry some exported form of cookiejar.entry.
(While the two stage retrieval might look strange it has some
benefits.)

Based on this API a user of package cookiejar could extract all
stored cookies and persist them herself. Probably loading them
back into a Jar would require some help to safeguard against subtle
errors and PSL changes.

Coming up with the "right" API here will be difficult, but might be
simpler than a real persistent cookiejar.

V.

[tandalat]

On Wednesday, July 29, 2015 at 6:49:30 PM UTC-7, Nigel Tao wrote:

Re persistent storage, there was a lot of discussion with Dr. Volker
Dobler (CC'ed) and I a couple of years ago about how to do a cookie
jar with pluggable storage, but we never landed on a satisfactory
design, let alone a perfect one, so what's in the standard library is
not persistent.

If you come to us with an existence proof of a good design for a
persistent cookie jar, I'm happy to look at it, but in the meantime,
if you want persistence, it''s probably best to fork
net/http/cookiejar for now. Once a change lands in the standard
library and is released, we can no longer change its API, and I don't
know what the right API is yet.

https://groups.google.com/forum/#!topicsearchin/golang-dev/cookiejar$20-%22broken$20by%22
and
https://groups.google.com/d/topic/golang-dev/7Jcbgrbah2s/discussion
might be places to start looking for previous discussions.

I'd read these threads before posting the change request for http.cookiejar and was aware of the difficulty to come up with a satisfactory design. Hence, my proposal isn't about a persistent cookie jar per se but an ability for other cookie processors to implement any extra handling required by their respective applications, and a consistency with original cookie information set in the response's header. A reasonable expectation is that cookie attributes (domain, path, expires, httponly etc.) sent by an application from the server side should be readily obtained via a cookie object returned by http.cookiejar.

I was caught by surprise when all I could get out of a cookie object returned by cookiejar.Cookies() ([] http.Cookie) was just the name and value of the cookie while http.Cookie implements all fields defined for a Set-Cookie in RFC 6265. This pretty much limits the use of http.cookiejar to http.Client and provides little benefits for other cookie processors to perform any extra handling such as using domain, path, expires etc. for automated services (single sign-on, for instance) or persisting any non-session cookies etc.

 

[tandalat]

On Thursday, July 30, 2015 at 1:04:24 AM UTC-7, Volker Dobler wrote:

But If I understand Tan  correctly he is suggesting to just add some
API to retrieve all cookie entries from the jar. Maybe something like

    // UsedETLDs returns all effective top level domains for which
    // the Jar contains cookies.
    func (j *Jar) UsedETLDs() []string

    // CookiesFor returns all stored (and not expired) cookies for
    // the given effective top level domain etld.
    func (j *Jar) CookiesFor(etld string) []Entry

with Entry some exported form of cookiejar.entry.
(While the two stage retrieval might look strange it has some
benefits.)

Yes, indeed. Is there a reason why cookie objects returned by cookiejar.Cookies() do not have other fields set other than Name and Value? In the source code for cookiejar.cookies(...):

225		for _, e := range selected {
226			cookies = append(cookies, &http.Cookie{Name: e.Name, Value: e.Value})
227		}

My proposal is simply filling up other fields like so,

 for _, e := range selected {
        c := &http.Cookie{
                 Name:     e.Name,
                 Value:    e.Value,
                 Path:     e.Path,
                 Domain:   e.Domain,
                 Expires:  e.Expires,
                 Secure:   e.Secure,
                 HttpOnly: e.HttpOnly,
              } 
       cookies = append(cookies, c)
}

What's the catch here?

 

Based on this API a user of package cookiejar could extract all
stored cookies and persist them herself. Probably loading them
back into a Jar would require some help to safeguard against subtle
errors and PSL changes.

Won't keying the urls (used to retrieve cookies) for serialized cookies on disk or database and later using them to load the cookies back into the jar keep things consistent? For example,

        store[u.String()] = jar.Cookies(u)

        ... serialize to some storage ...

        ... load from storage

        for k, v := range store {

                 jar.SetCookies(url.MustParse(k), v)

        }

        ....

Thanks

tan

[Volker Dobler]

On Fri, Jul 31, 2015 at 10:36 AM, tandalat <tan....@gmail.com> wrote:
[...]

> Yes, indeed. Is there a reason why cookie objects returned by
> cookiejar.Cookies() do not have other fields set other than Name and Value?
> In the source code for cookiejar.cookies(...):
>
> 225 for _, e := range selected {
> 226 cookies = append(cookies, &http.Cookie{Name: e.Name, Value: e.Value})
> 227 }
>
>
> My proposal is simply filling up other fields like so,
>
> for _, e := range selected {
> c := &http.Cookie{
> Name: e.Name,
> Value: e.Value,
> Path: e.Path,
> Domain: e.Domain,
> Expires: e.Expires,
> Secure: e.Secure,
> HttpOnly: e.HttpOnly,
> }
> cookies = append(cookies, c)
> }
>
>
> What's the catch here?
>

1. This would break the Go API compatibility guaranty.

2. Cookies() returns http.Cookies suitable for generating
a Cookie header. If all fields would be filled up you would
get a cookie which would serialize to a Set-Cookie header.
(package http could clear the other fields).

3. How would you differentiate between a domain and a
host cookie?

4. You lose the creation and last access time, so you
cannot use this for properly persisting and re-loading
the cookies.

>
>>
>> Based on this API a user of package cookiejar could extract all
>> stored cookies and persist them herself. Probably loading them
>> back into a Jar would require some help to safeguard against subtle
>> errors and PSL changes.
>
>
> Won't keying the urls (used to retrieve cookies) for serialized cookies on
> disk or database and later using them to load the cookies back into the jar
> keep things consistent? For example,
>
> store[u.String()] = jar.Cookies(u)
> ... serialize to some storage ...
> ... load from storage
> for k, v := range store {
> jar.SetCookies(url.MustParse(k), v)
> }
> ....
>

Basically yes, but there might be some ugly corner cases like
domain cookies which are no longer allowed because the
domain is now considered a ETLD (because the PSL used
to re-load the cookies differs from the one used during storing).
And possible more if I thought about it longer.

The main problem with keying by _URL_ is the Path field:
You cannot know which URLs (i.e. which paths) you have
to request as Cookies() returns only cookies which match
the path of the URL. How would you know that the jar
contains a cookie with Path="/some/obscure/deep/path"?

V.

[...]

[roger peppe]

FWIW I would love to see a decent disk-persistent cookiejar implementation. I forked the stdlib cookiejar package to make github.com/juju/persistentcookiejar as a very quick hack but it's demonstrably wrong in the face of concurrent users, and I'm sure for other reasons too.

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.