Practical HTTP requests?

[fixpoint]

I'm on the search for a new programming language to learn, so I thought I'd check out Racket, but I'm having a hard time trying to do a very common task that would make Racket practical for my use at work:

• Make a series of HTTP requests to an API that returns JSON responses
• Reuse the HTTP connection to avoid creating new TCP connections for each request
• Honors the `http_proxy`, `https_proxy`, and `no_proxy` environment variables
• Maintains a "session" where cookies set by the server are sent back in subsequent requests

My daily driver is Python, so I'm used to its `requests` library. Go and Rust have similar libraries. While I don't mind piecing some of this together, I'm struggling as a new user to figure out what libraries are recommended and, more importantly, how to put them together to accomplish the above points.

Thank you.

p.s. The Racket documentation is by far the best looking documentation I've read in any language. It's quite amazing!

[evdubs]

Have you taken a look at net/url? From there, I see:

• make-http-connection, which can allow calls to get-pure-port/headers to stay connected if the server allows it.
• current-proxy-servers and current-no-proxy-servers, which show they respect http_proxy, https_proxy, and no_proxy.

There are also get-pure-port related functions that allow setting headers, which I think should let you set your cookies? For JSON, you can take look at the json docs if you're interested in converting the HTTP response bodies into a JSON expression. These expressions can then be treated like any other expression where you can map/filter/fold over lists and access map elements with hash table functions.

To find this stuff, I just searched for "http" and "json" using the ". . . search manuals . . ." box at the top right of the docs. The search looks through all of the Racket API as well as many user-contributed packages. It makes it very easy to find whatever you might be interested in.

Evan

[fixpoint]

I was under the incorrect impression that net/url did not support HTTP 1.1. I suspect I stumbled upon some out of date information on stackoverflow. Thanks for pointing me in the right direction. Using net/url, net/cookies, and json, I hacked the following code (please be gentle, it is literally the first lines of Racket I've ever written). There are two key problems with this code:

1. Because cookies handling is not part of net/url, my use of get-pure-port/headers is not able to see cookies, and thus save them, in any of the 3xx redirect responses.
2. I need a post-pure-port/headers equivalent, so I can post data and reuse the connection. It looks like you can only reuse connections in net/url with only get-pure-port/headers.

Am I missing any other libraries out there?

#lang racket

(require json

         net/head

         net/url

         net/cookies/user-agent)

(define (get-json-request url #:connection conn)

  (define req-headers

    (let ([cookies (cookie-header url)])

      (cond [(false? cookies) '()]

            [else (list (format "Cookie: ~a" cookies))])))

  (display (format "Headers sent were:\n ~a\n" req-headers))

  (define-values

    (in-port resp-headers)

    (get-pure-port/headers url req-headers #:connection conn #:redirections 1))

  (extract-and-save-cookies! (map string->bytes/utf-8 (string-split resp-headers "\r\n")) url)

  (display (format "\nReceived headers were:\n ~a" resp-headers))

  (read-json in-port))

;; Start a session

(define conn (make-http-connection))

;; Make a request that will force server to set cookies

(displayln (jsexpr->string

            (get-json-request

             (string->url "https://postman-echo.com/cookies/set?foo1=bar1&foo2=bar2")

             #:connection conn)))

;; Make another request to see what cookies we sent server

(displayln "\n--------------------------------------\n")

(displayln (jsexpr->string

            (get-json-request

             (string->url "https://postman-echo.com/cookies")

             #:connection conn)))

;; Close session

(http-connection-close conn)

[Jens Axel Søgaard]

Take a look at the `http` package.

    https://docs.racket-lang.org/http/index.html

I believe it can be used to receive and send cookies (and other headers), but 
I don't know how much is automatic.

/Jens Axel

Racket Stories
https://racket-stories.com

--
You received this message because you are subscribed to the Google Groups "Racket Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to racket-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/racket-users/f71ad314-677f-4f42-ae7c-9e738d88eded%40googlegroups.com.

--

--
Jens Axel Søgaard

[fixpoint]

Thanks, but that library does not support proxies. Oh well, my Racket adventures have come to an end as I’ll be unable to convince others at work to try it without a practical, batteries included http library. 

To unsubscribe from this group and stop receiving emails from it, send an email to racket...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/racket-users/f71ad314-677f-4f42-ae7c-9e738d88eded%40googlegroups.com.

[Sam Tobin-Hochstadt]

I think you may want to use the lower-level 'net/http-client' library, which supports connection reuse and gives more control over the details of redirects. 

Sam

--
You received this message because you are subscribed to the Google Groups "Racket Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to racket-users...@googlegroups.com.

[Brian Adkins]

I spent many years developing in Ruby before switching to Racket, so I understand the appeal of a "batteries included" language. Python excels in this area. If you're weighting the "batteries included" aspect very high, then Racket may not be suitable for you at this time, but in that case, I expect you'd just stick with Python.

If you're patient, I think you may find many compelling reasons to consider Racket. I won't list them here (other than the fact that the community alone is compelling enough to invest more time learning the language), or advocate for the language in general, but if you decide to check it out more thoroughly later, another library that may be of interest is:

https://docs.racket-lang.org/cookies/index.html

I've been developing web applications in Racket for the last year and a half, or so, but I've been focused on server-side functionality, and I've either found a Racket package that does what I need directly, or I've found a set of them that allow me to easily layer the extra code on top that I need. I was used to the Rails framework, so I thought it would be tough switching to Racket, but in hindsight, it was trivial.

I will be doing more web scraping soon for some data science applications, and given what Racket provides currently, it will be easy to write something that fulfills the requirements you listed in your first post, although I have no need for proxy servers personally.

Brian

[Brian Adkins]

Sorry - missed the fact that you already found the cookie library :)

[Sorawee Porncharoenwase]

I did a web scraping in Racket a couple of months ago, and I agree that it is painful. On the bright side, Bogdan seems to start working on a practical requests-like library due to this thread, so we should have a good practical HTTP request library soon :)

To unsubscribe from this group and stop receiving emails from it, send an email to racket-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/racket-users/21032250-5dc7-47eb-8e81-f4c2f6b9e52d%40googlegroups.com.