webob SignedSerializer questions

[Jonathan Vanasco]

I've been updating my session library to utilize typing/mypy and this implementation detail surfaced between mypy integration and some new tests.

I hope someone can enlighten me on this and some best testing practices...

Webob's SignedSerializer expects to dump to, and load from, `bytes`.  The default behavior is handled by taking json.dumps/loads - which operates on strings - and transcoding it between bytes and string. (See https://github.com/Pylons/webob/blob/main/src/webob/cookies.py)

Webob's set_cookie is designed to accept strings, and transcodes it to bytes with utf8 encoding.

(https://github.com/Pylons/webob/blob/main/src/webob/response.py)

Webob's request does a lot of transcoding between bytes and str for the cookies, and both utf8 and latin1 charsets are used.

Can anyone share why these design decisions were made?  There just seems to be a lot of needless transcoding going around, and the design decisions in webob/pyramid tend to be very deliberate and thought out - so I must be missing something.  

Can anyone also share exactly why there are some latin-1 charsets in the webob cookie logic?

Thank you in advance.

[Bert JW Regeer]

I can’t directly comment on what part of the code you are looking at, but it is likely because Cookies are set using HTTP headers, and HTTP headers are limited in subset to latin-1 only (minus a bunch of other characters/control characters/whatnot). The reality is that they tend to get mangled and eaten alive, and or modified every which way. The only safe set is ASCII characters.

The same is true for other parts of the HTTP request and response.

Thank you in advance.

--
You received this message because you are subscribed to the Google Groups "pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pylons-discus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/pylons-discuss/c2c36996-39c3-434a-91a9-a77bb2343027n%40googlegroups.com.

[Jonathan Vanasco]

Thanks for the quick response.

After digging through this for a bit, I think there just may be a lot of artifacts in here from years of code updates and specs changing.

In webob's response.Response, set_cookie accepts a string (https://github.com/Pylons/webob/blob/main/src/webob/response.py#L1023-L1028),

then encodes it to utf-8 bytes (

https://github.com/Pylons/webob/blob/main/src/webob/response.py#L1086 / https://github.com/Pylons/webob/blob/main/src/webob/util.py#L47-L51 )

and invokes webob.cookies's make_cookie  (https://github.com/Pylons/webob/blob/main/src/webob/cookies.py#L492 )

which does a lot of work with bytes.. but then serializes it to text (https://github.com/Pylons/webob/blob/main/src/webob/cookies.py#L286)

_parse_cookie is dealing with latin-1 (https://github.com/Pylons/webob/blob/main/src/webob/cookies.py#L201-L202)

mutate_header is dealing with ascii, utf-8, latin-1 (https://github.com/Pylons/webob/blob/main/src/webob/cookies.py#L56) for different bits of the spec

It's just really interesting and seems to be all over.  I'm familiar with most of the specs, so understand the ascii requirements for names, but it's just hard to decipher from scratch.

The thing that triggered me on this path, is this behavior:

SignedSerializer expects strings (https://github.com/Pylons/webob/blob/main/src/webob/cookies.py#L663-L688) and returns bytes.

But the default serializer converts strings to bytes (https://github.com/Pylons/webob/blob/main/src/webob/cookies.py#L620) 

The output of that is commonly used as cookie.  

While set_cookie deals expects strings (https://github.com/Pylons/webob/blob/main/src/webob/response.py#L1001-L1029).  

It immediately turns it into bytes (https://github.com/Pylons/webob/blob/main/src/webob/response.py#L1086), and sends it to make_cookie

make_cookie does a lot of work on bytes (https://github.com/Pylons/webob/blob/main/src/webob/cookies.py#L492)

and then invokes a serialization  (https://github.com/Pylons/webob/blob/main/src/webob/cookies.py#L613)

said serialization works on bytes, and returns strings ( https://github.com/Pylons/webob/blob/main/src/webob/cookies.py#L319)

At this point I lose track of the next few steps.

It just seems very odd to me that there are so many conversions going on, without detailed documentation disclosing why.