[racket] HTTP-POST byte string gets truncated

[Mikko Tiihonen]

Hi, again!

I'm continuing to build a small HTTP-client. The problem is now that the POST parameter/value byte strings sent by put-pure-port and post-pure-port seem to get truncated somewhere. The request-post-data/raw shows that the byte string gets prepended with "\r\n\r\n" and truncated by four bytes. E.g.

(post-pure-port uri #"param1=hello&param2=world" header)

is received as

#"\r\n\r\nparam1=hello&param2=w"

Is there something I'm missing or this really a bug?

Best,

-Mikko
____________________
Racket Users list:
http://lists.racket-lang.org/users

[Gregory Woodhouse]

I'm not sure what's going on, but the HTTP protocol specifies that the header and message body be separated by \r\n\r\n.

Sent from my iPhone

[Danny Yoo]

On Sat, Oct 6, 2012 at 4:50 AM, Mikko Tiihonen
<mikko.t...@tmtiihonen.fi> wrote:
> Hi, again!
>
> I'm continuing to build a small HTTP-client. The problem is now that the POST parameter/value byte strings sent by put-pure-port and post-pure-port seem to get truncated somewhere. The request-post-data/raw shows that the byte string gets prepended with "\r\n\r\n" and truncated by four bytes. E.g.
>
> (post-pure-port uri #"param1=hello&param2=world" header)
>
> is received as
>
> #"\r\n\r\nparam1=hello&param2=w"

Odd. What's the content of 'header' here? It's the only free
variable I see whose value I don't quite understand yet.

[Mikko Tiihonen]

Hi, Danny,

the header is here:

(define header (list (insert-field #"Host" #"localhost:8080"
(insert-field #"Connection" #"keep-alive"
(insert-field #"User-Agent" #"Mozilla/5.0 (testClient.rkt 0.1)"
(insert-field #"Accept-Encoding" #"gzip"
(insert-field #"Accept" #"text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"
(insert-field #"Accept-Language" #"en-us"
(insert-field #"Accept-Charset" #"ISO-8859-1,UTF-8;q=0.7,*;q=0.7"
(insert-field #"Cache-control" #"no-cache" empty-header))))))))))

For some reason the request struct on the server side seems to get the POST data four bytes off... I also noticed that empty-header does not evaluate to "\r\n\r\n" as specified in net/head documentation:
> empty-header
"\r\n"

validate-header validates the header as one string, but post-pure-port requires that the header is contained in a list of strings ("optional list of strings can be used to send header lines to the server"). Is there some kind of a mismatch between net/url and net/head?

Cheers,

-Mikko

[Greg Hendershott]

> validate-header validates the header as one string, but post-pure-port requires that the header is contained in a list of strings ("optional list of strings can be used to send header lines to the server"). Is there some kind of a mismatch between net/url and net/head?

Yes, unfortunately they represent headers differently. net/head
represents headers as a single "string" or #"bytes string" consisting
of \r\n separated elements. This is closest to the headers in real
life. However net/url uses a list of elements (and they're "string"
elements only; not #"byte string").

Since you're using post-pure-port from net/url, I think you want:

(define header
(list "Host: localhost:8080"
"Connection: keep-alive"
"User-Agent: Mozilla/5.0 (testClient.rkt 0.1)"
"Accept-Encoding: gzip"
"Accept:
text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"
"Accept-Language: en-us"
"Accept-Charset: ISO-8859-1,UTF-8;q=0.7,*;q=0.7"
"Cache-control: no-cache"))
(post-pure-port my-url my-data header)


On Mon, Oct 8, 2012 at 3:07 AM, Mikko Tiihonen

[Mikko Tiihonen]

Thanks, Greg and Danny!

Problem solved - now the POST parameters are also correct! Seems that the different representation of headers in net/head caused the POST parameters following the header being parsed incorrectly.

Cheers,

-Mikko

[Greg Hendershott]

The different representations can be inconvenient. Plus sometimes it's
easiest to deal with headers as what they really mean to you: A
dictionary of keys and values. If you do this a lot, you may end up
creating some helpers to convert among representations like the
following.

(define/contract (heads-list->string xs)
((listof string?) . -> . string?)
(string-append (string-join xs "\r\n") "\r\n\r\n"))

(define/contract (heads-string->list s)
(string? . -> . (listof string?))
(filter (lambda (s) (not (equal? s "")))
(regexp-split "\r\n" s)))

(define/contract (heads-list->dict xs)
((listof string?) . -> . dict?)
(for/list ([x (in-list xs)])
(match x
[(pregexp "^(\\S+)\\s*:\\s*(.*?)$" (list _ k v)) (cons k v)]
[else (error 'heads-list->dict "bad header: ~a" x)])))

(define/contract (heads-dict->list d)
(dict? . -> . (listof string?))
(for/list ([(k v) (in-dict d)])
(format "~a: ~a" k v)))

(module+ test
(require rackunit)
;; Round-trip?
(define header (list "a: 1" "b: 2" "c: 3"))
(check-equal? (heads-string->list (heads-list->string header)) header)
(check-equal? (heads-dict->list (heads-list->dict header)) header)
;; Edge cases?
(check-equal? (heads-list->string '()) "\r\n\r\n")
(check-equal? (heads-string->list "\r\n\r\n") '())
(check-equal? (heads-list->dict '()) '())
(check-equal? (heads-dict->list '()) '()))

I just wrote this now. There are some oversimplifications like the
fact that HTTP headers may duplicate keys. But you get the idea.

On Mon, Oct 8, 2012 at 8:04 AM, Mikko Tiihonen