Get request send me hmtl content but works fine from python and curl

[ryan embgrets]

I am trying to call a simple api by using golang. But, each time it sends me html content of login page instead of actual data. But same get request works from python and curl. 

package main

import (
"fmt"
"io/ioutil"
"net/http"
"os"
)

func main() {

client := &http.Client{}
req, err := http.NewRequest("GET", "https://www.lrn.com", nil)
if err != nil {
os.Exit(1)
}

q := req.URL.Query()
q.Add("phoneList", "XXXXXX")
q.Add("output", "json")
q.Add("version", "5")
//req.URL.RawQuery = q.Encode()
req.Header.Set("loginId", "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX")

fmt.Println(req.URL.String())

resp, err := client.Do(req)

if err != nil {
fmt.Println("Errored when sending request to the server")
return
}

defer resp.Body.Close()
resp_body, _ := ioutil.ReadAll(resp.Body)

fmt.Println(resp.Status)
fmt.Println(string(resp_body))
}

Above script gives me html content of login page. But if i use python, it works just fine. 

import requests

r=requests.get("https://www.lrn.com", params = {'version':'5', "phoneList":"XXXXXX", "output":"json"}, headers={"loginId":"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX  "})

print r.text

Could someone please explain me what might be wrong in my golang script. 

Ryan. 

[Tamás Gulácsi]

I bet requests (and curl) encodes the params as multipart/form-data, NOT query string.

Do the same: https://play.golang.org/p/L4YryKNjju4

[burak serdar]

On Sat, Sep 19, 2020 at 11:47 PM Tamás Gulácsi <tgula...@gmail.com> wrote:
>
> I bet requests (and curl) encodes the params as multipart/form-data, NOT query string.
> Do the same: https://play.golang.org/p/L4YryKNjju4

A GET request does not have a body.

I suggest you print out the request struct completely before
submitting it, with the headers and URL and everything, and then maybe
you can pinpoint the problem.

> --
> You received this message because you are subscribed to the Google Groups "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/95c62bc9-7772-4309-907d-09590dae9a25n%40googlegroups.com.

[ryan embgrets]

Thanks for taking time to answer my question. I have put the actual data except the loginId header..

https://play.golang.org/p/KrnxWLVj8s2

But i still get the html content in golang but when i use python i get the invalid api credentials. Due to incorrect loginId header. 

import requests

r = requests.get("https://www.dncscrub.com/app/main/rpc/scrub", params={'version': '5', "phoneList": "2123727200"},

                 headers={"loginId": "0610A62F"})

print r.text

So, responses are totally different from two languages. Python looks correct. 

Ryan

To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/CAMV2RqrvsdZvTfBax1Z2Ke0YL%3DzVtp20WBUMfnyB%2BLNwOVQLiw%40mail.gmail.com.

[Tamás Gulácsi]

Log the Python request, to see what it really sends - or recreate it with curl, which is easier to debug (--trace).

[Brian Candler]

My suggestion is to change the URL to something you control, with http not https, and see how the requests differ using tcpdump.  Here's what I get:

---- python3 ----

import requests

r=requests.get("http://localhost/foo", params = {'version':'5', "phoneList":"XXXXXX", "output":"json"}, headers={"loginId":"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX  "})

print(r.status_code)

print(r.text)

-->

GET /foo?version=5&phoneList=XXXXXX&output=json HTTP/1.1

Host: localhost

User-Agent: python-requests/2.24.0

Accept-Encoding: gzip, deflate

Accept: */*

Connection: keep-alive

loginId: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

---- go 1.14.6 ----

Code as per original posting, but with URL changed to http://localhost/foo

-->

GET /foo HTTP/1.1

Host: localhost

User-Agent: Go-http-client/1.1

Loginid: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

Accept-Encoding: gzip

So the first problem is that the URL query parameters are not getting encoded.

Uncommenting the line "req.URL.RawQuery = q.Encode()", I get:

-->

GET /foo?output=json&phoneList=XXXXXX&version=5 HTTP/1.1

Host: localhost

User-Agent: Go-http-client/1.1

Loginid: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

Accept-Encoding: gzip

That looks reasonable to me.  Now change the URL to "https://www.dncscrub.com/app/main/rpc/scrub"  (at this point I cannot tcpdump the packets of course, but just show the application response)

- with go, the response is a 200 OK with HTML containing login information

- with python, I get exactly the same

Now changing to https://www.lrn.com 

- with go the response is a 200 and a huge HTML/CSS page

- with python I get exactly the same

$ python3 test3.py | wc

   1288   12333  241224

$ go run . | wc

    1289   12336  241286

So I'm afraid I cannot see any difference between go and python, whereas you said "responses are totally different from two languages".  If they are still different for you, this is weird.

For completeness, here are the last two programs I ran.

---- python3 ----

import requests

r=requests.get("https://www.lrn.com", params = {'version':'5', "phoneList":"XXXXXX", "output":"json"}, headers={"loginId":"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX  "})

print(r.status_code)

print(r.text)

---- go ----

package main

import (

        "fmt"

        "io/ioutil"

        "net/http"

        "os"

)

func main() {

        client := &http.Client{}

        //req, err := http.NewRequest("GET", "http://localhost/foo", nil)

        req, err := http.NewRequest("GET", "https://www.lrn.com", nil)

        if err != nil {

                os.Exit(1)

        }

        q := req.URL.Query()

        q.Add("phoneList", "XXXXXX")

        q.Add("output", "json")

        q.Add("version", "5")

[Jesper Louis Andersen]

On Sun, Sep 20, 2020 at 8:10 AM burak serdar <bse...@computer.org> wrote:

A GET request does not have a body.

Murky waters ahead.

RFC 2616 explicitly states that a supplied body SHOULD be forwarded by the server on any request type. This has led some people to use bodies on GET requests; ElasticSearch I'm looking at you. However, the newer RFC 7231 explicitly states that sending a body on a GET has no determined semantics and that "sound" clients ought to reject such bodies. But since RFC 2616 existed at some point, you might still find certain APIs which do expect GET requests with bodies, however wrong that may seem.

Getting rid of mistakes is really hard once they're been used in anger. And HTTP is riddled with design mistakes which shouldn't have been put in, in hindsight. Hopefully the new section of HTTP RFCs mean we can clean up over time as we proceed.

 

[ryan embgrets]

Thanks everyone for your valuable feedback. 

I was able to figure out the issue that was happening due to the case-sensitive nature of the loginId header, which was being normalized in golang. 

So, after adding the below snippet i was able to make it work. 

req.Header["loginId"] = []string{"0610A62F"}




Ryan. 

--

You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/CAGrdgiVfaVwmX5P%3DNd-ZPOcFOZP-XoETwAy5%2BBTVBzsRUi1F3A%40mail.gmail.com.

[Tamás Gulácsi]

Which is absolutely non-standard :)

Headers are case-insensitive, so if the server relies on case, it's against the HTTP standard :)

[Brian Candler]

On Sunday, 20 September 2020 12:18:09 UTC+1, ryan embgrets wrote:

Thanks everyone for your valuable feedback. 

I was able to figure out the issue that was happening due to the case-sensitive nature of the loginId header, which was being normalized in golang. 

So, after adding the below snippet i was able to make it work. 

req.Header["loginId"] = []string{"0610A62F"}




Good catch.  I had not noticed it was "Loginid" in the tcpdump output.