Trouble querying REST API with net/http (api verified working with curl)

mark mellar

未讀,

2016年6月23日晚上9:12:072016/6/23

收件者：golang-nuts

Hi folks,

We have a REST API in our organisation which I'd like my Go program to query.

I'm able to get the data using curl

curl -c ./cookies -k -H "Content-Type: application/json" -X POST -d '{"user":"admin","password":"admin"}' "https://hostmane.here.com:443/rest/json/login"

curl -k -s -b ./cookies -H "Content-Type: application/json" -X GET  "https://hostmane.here.com:443/rest/json/flows"

The first command gets an authentication cookie, the 2nd uses it to query the API, resulting in a bunch of JSON being printed to STDOUT, which is great!

My problem occurs when I try reproducing this behavior in Go...

package main

import (
	"crypto/tls"
	"errors"
	"fmt"
	"io/ioutil"
	"net/http"
	"strings"
	"time"
)

func main() {
	authCookie, err := authenticate("hostname", "admin", "admin")
	if err != nil {
		fmt.Println("ERROR " + err.Error())
		return
	}
	fmt.Printf("AUTH SUCCESS:\n%+v\n", authCookie)
	result, err := queryFlows("hostname", authCookie)
	if err != nil {
		fmt.Println("ERROR " + err.Error())
		return
	}
	fmt.Printf("QUERY SUCCESS:\n%+s\n", result)

}

func authenticate(host, username, password string) (*http.Cookie, error) {
	tr := &http.Transport{
		TLSClientConfig: &tls.Config{InsecureSkipVerify: true},
	}
	client := &http.Client{Transport: tr}
	body := strings.NewReader(`{"user":"` + username + `","password":"` + password + `"}`)
	req, err := http.NewRequest("POST", "https://"+host+".here.com:443/rest/json/login", body)
	if err != nil {
		return nil, err
	}
	req.Header.Set("Content-Type", "application/json")

	resp, err := client.Do(req)
	if err != nil {
		return nil, err
	}
	defer resp.Body.Close()
	respBody, err := ioutil.ReadAll(resp.Body)
	if err != nil {
		return nil, err
	}
	if resp.StatusCode != 200 {
		return nil, errors.New(string(respBody))
	}
	cookies := resp.Cookies()
	authCoookie := cookies[0]
	return authCoookie, nil
}

func queryFlows(host string, authCookie *http.Cookie) (string, error) {
	tr := &http.Transport{
		TLSClientConfig: &tls.Config{InsecureSkipVerify: true},
	}
	client := &http.Client{Transport: tr, Timeout: 1 * time.Minute}
	req, err := http.NewRequest("Get", "https://"+host+".here.com:443/rest/json/flows", nil)
	if err != nil {
		return "", err
	}
	req.Header.Set("Content-Type", "application/json")
	req.AddCookie(authCookie)

	resp, err := client.Do(req)
	if err != nil {
		return "", err
	}
	defer resp.Body.Close()
	respBody, err := ioutil.ReadAll(resp.Body)
	if err != nil {
		return "", err
	}
	if resp.StatusCode != 200 {
		return "", errors.New(string(respBody))
	}
	return string(respBody), nil
}

Upon running this I See the following...









AUTH SUCCESS:
connect.sid=s%3AEaciXI7sKXR3Q1Wv6GU2Yt%2BN.MdlA9oaJuAc1C4MnP%2F997sn1vKvyXyow9bw69fr5ls8; Path=/; HttpOnly

ERROR <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>502 Proxy Error</title>
</head><body>
<h1>Proxy Error</h1>
<p>The proxy server received an invalid
response from an upstream server.<br />
The proxy server could not handle the request <em><a href="/rest/json/flows">Get&nbsp;/rest/json/flows</a></em>.<p>
Reason: <strong>Error reading from remote server</strong></p></p>
</body></html>

It looks like there's something not quite configured correctly in my queryFlows function, but I haven't managed to figure it out. Googling pointed me towards increasing the value of client.Timeout (the curl query takes around 2 secs). But setting the value to 1 minute had no impact, giving the same error in significantly less than 1 minute.

Any ideas as to what curl might be doing that I'm not would be greatly appreciated!

Cheers,

Mark.

Tamás Gulácsi

未讀,

2016年6月24日凌晨12:46:062016/6/24

收件者：golang-nuts

Just a blind shoot into the darkness: try to use one (global) http.Client! Also, you can ask curl to dump the request, and net/http/httputil also to compare them.

mark mellar

未讀,

2016年6月24日凌晨2:32:362016/6/24

收件者：golang-nuts

Good tip on using httputil to inspect the request! Unfortunately I'm not seeing anything obviously wrong...

Using curl -v

* Trying <an-ip>...
* Connected to host.here.com (<an-ip>) port 443 (#0)
* TLS 1.0 connection using TLS_DHE_RSA_WITH_AES_256_CBC_SHA
* Server certificate: host
> GET /rest/json/flows HTTP/1.1
> Host: host.here.com
> User-Agent: curl/7.43.0
> Accept: */*
> Cookie: connect.sid=s%3AeQ30DeGtFyrhrntWofMWpGXo.%2F745zwxR0ErYrpnoVDklkt%2F7H9FX5GfcroBFXg5M6Ag
> Content-Type: application/json

using httputil.DumpRequestOut

Get /rest/json/flows HTTP/1.1

Host: host.here.com:443

User-Agent: Go-http-client/1.1

Content-Type: application/json

Cookie: connect.sid=s%3A%2FWuxV7HdAnHDweXFvs4N8%2BaB.DuNDxfVN6sLhLO%2Flu3hIY7PfUnfMquyRIfXSllGtZpM

Accept-Encoding: gzip

I'm suspicious about the Accept-Encoding field. I tried twiddling DisableCompression in my transport but nothing changed...

Cheers,

Mark

mark mellar

未讀,

2016年6月26日下午6:22:302016/6/26

收件者：golang-nuts

For anyone else having similar issues, I found I was able to work around this by using client.Get() rather than client.Do().

Still not sure what the root cause is though...

Cheers.

Kiki Sugiaman

未讀,

2016年6月27日凌晨2:12:592016/6/27

收件者：golan...@googlegroups.com

I know this is a bit late, so perhaps you don't care anymore...

Anyway, the server may not understand the "Get" method in this line:

req, err :=
        http.NewRequest("Get",
        "https://"+host+".here.com:443/rest/json/flows", nil)

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

mark mellar

未讀,

2016年6月27日中午12:57:502016/6/27

收件者：golang-nuts

ksug, you were right... I was able to replicate the error using curl by changing the -x parameter from 'GET' to 'Get' .

How embarrassing, I'll go sit in the corner and think about what I've done...

Thanks for your help guys!

Mark.

Kiki Sugiaman

未讀,

2016年6月27日晚上8:05:512016/6/27

收件者：golan...@googlegroups.com

Haha! Happens to the best of us every once in a while.

回覆所有人

回覆作者

轉寄