how to get a special URL by golang ”net/http"

[benjiam shen]

HI

   I want to use "net/http" to get something from a specail web server.

following is the url that's host needed.

http://xxxx/namedObjects/164861/%2Ftmp%2Ftritonqa%2Ftest_121748_part_files_dirs%2F20140718223538%2Ftest_file_1405694198_19160.dat?versionId=1405694274&exactVersion=0

it is so special , that we need QueryEscape party of path , but not all.

<pre>

u, err := url.Parse("http://" + req.Host)

if err != nil {

//log.Fatal(err)

}

u.Scheme = "http"

u.Host = req.Host

u.Path = "/namedObjects/" + strconv.FormatInt(req.Machineid, 10) + "/" + requests[i].Filename

                fmt.Println("Path = ", u.Path)

q := u.Query()

q.Set("versionId", strconv.FormatInt(requests[i].Restore_time, 10))

q.Set("exactVersion", "0")

u.RawQuery = q.Encode()

reqest.URL = u

fmt.Println("%v", reqest.URL)

</pre>

the log output.

Path =  /namedObjects/164861//tmp/tritonqa/test_121748_part_files_dirs/20140718223538/test_file_1405694198_19160.dat

Request:

%v http://xxxx/namedObjects/164861//tmp/tritonqa/test_121748_part_files_dirs/20140718223538/test_file_1405694198_19160.dat?versionId=1405694274&exactVersion=0

after investigation, in server side, we found the client send the wrong GET command

"/namedObjects/164861//tmp/tritonqa/test_121748_part_files_dirs/20140718223538/test_file_1405694198_19160.dat?versionId=1405694274&exactVersion=0"

but I propose following request.

/namedObjects/164861/%2Ftmp%2Ftritonqa%2Ftest_121748_part_files_dirs%2F20140718223538%2Ftest_file_1405694198_19160.dat?versionId=1405694274&exactVersion=0

I can't change any of server side. so we must solve this issue in client side. do you know how to do it ?

[Tamás Gulácsi]

Why not escape the "/"+requests[i].Filename part before adding it to the url?

[benjiam shen]

it can't work. i think it was caused by our special prtocol of production

what is we want

case 1
if the raw path is ""/namedobject/1111/C:\\tmp\\tritonqa\\test_4662_fullPercentage_and_verify_it_works\\20140516064843\\test_file_1400222925_33363.dat""

then send query should like this

GET /namedobject/1111/C:%5Ctmp%5Ctritonqa%5Ctest_4662_fullPercentage_and_verify_it_works%5C20140516064843%5Ctest_file_1400222925_33363.dat?exactVersion=0&versionId=1397227272 HTTP/1.1

if  i use following code in go lang, it can work.

<pre>

func fun1() {
    client := &http.Client{}

    myurl := "http://192.168.228.10/namedObjects/164861/%2Ftmp%2Ftritonqa%2Ftest_121748_part_files_dirs%2F20140718223538%2Ftest_file_1405694201_33172.dat?versionId=1405694277&exactVersion=0"
    fmt.Println("url = ", myurl)
   
   
    reqest, err := http.NewRequest("GET", myurl, nil)
    if err != nil {
        fmt.Println("err != nil")
        fmt.Println(err)
    }

   
    u, err := url.Parse("http://192.168.228.10")
    if err != nil {
        log.Fatal(err)
    }
    u.Scheme = "http"
    u.Host = "192.168.228.10"
    u.Path = "/namedobject/1111/C:\\tmp\\tritonqa\\test_4662_fullPercentage_and_verify_it_works\\20140516064843\\test_file_1400222925_33363.dat"
    fmt.Println("PATH = ", u.Path)
     q := u.Query()

    q.Set("versionId", "1397227272")

    q.Set("exactVersion", "0")

    u.RawQuery = q.Encode()

    reqest.URL = u
    
    

    fmt.Println("reqest %v", reqest.URL)
    response, err := client.Do(reqest)
    fmt.Println("done")
    fmt.Println("response %v", response," ", err)
}

</pre>


case 2.  if the  rawpath look like this
"u.Path = "/namedobject/1111//tmp/tritonqa/test_121748_part_files_dirs/20140718223538/test_file_1405694201_33172.dat"
we wish the query like this

"/namedObjects/164861/%2Ftmp%2Ftritonqa%2Ftest_121748_part_files_dirs%2F20140718223538%2Ftest_file_1405694201_33172.dat"
we need escape "/" and other unicode . if i use the code like before, in server side, we will got some query like this

"GET /namedobject/1111//tmp/tritonqa/test_121748_part_files_dirs/20140718223538/test_file_1405694201_33172.dat?exactVersion=0&versionId=1397227272 HTTP/1.1
"
it seem "net/http" don't work as i wish

as you told. I have test your code.

the code looks like this

<pre>

func fun1() {
    client := &http.Client{}

    myurl := "http://192.168.228.10/namedObjects/164861/%2Ftmp%2Ftritonqa%2Ftest_121748_part_files_dirs%2F20140718223538%2Ftest_file_1405694201_33172.dat?versionId=1405694277&exactVersion=0"
    fmt.Println("url = ", myurl)

    reqest, err := http.NewRequest("GET", myurl, nil)
    if err != nil {
        fmt.Println("err != nil")
        fmt.Println(err)
    }

    u, err := url.Parse("http://192.168.228.10")
    if err != nil {
        log.Fatal(err)
    }
    u.Scheme = "http"
    u.Host = "192.168.228.10"
    u.Path = "/namedobject/1111/" + url.QueryEscape("/tmp/tritonqa/test_121748_part_files_dirs/20140718223538/test_file_1405694201_33172.dat")
    fmt.Println("PATH = ", u.Path)
    q := u.Query()

    q.Set("versionId", "1397227272")

    q.Set("exactVersion", "0")

    u.RawQuery = q.Encode()

    reqest.URL = u

    fmt.Println("reqest %v", reqest.URL)
    response, err := client.Do(reqest)
    fmt.Println("done")
    fmt.Println("response %v", response, " ", err)
</pre>

the send query looks like this

GET /namedobject/1111/%252Ftmp%252Ftritonqa%252Ftest_121748_part_files_dirs%252F20140718223538%252Ftest_file_1405694201_33172.dat

what's i want is  %2F , but is is " %252F" , it can't work.

[Tamás Gulácsi]

I see. url.Path escapes its input, so adding it escaped input doubly escapes it (%2f becomes%252f).
A. Don't use u.Path but RawQury
B.don't use url.URL, just assemble the url manually.

[benjiam shen]

I don't know how to assemble the url manually. would you please give some code?

func (c *Client) Do(req *Request) (resp *Response, err error)

type Request struct {
        // Method specifies the HTTP method (GET, POST, PUT, etc.).
        // For client requests an empty string means GET.
        Method string

        // URL specifies either the URI being requested (for server
        // requests) or the URL to access (for client requests).
        //
        // For server requests the URL is parsed from the URI
        // supplied on the Request-Line as stored in RequestURI.  For
        // most requests, fields other than Path and RawQuery will be
        // empty. (See RFC 2616, Section 5.1.2)
        //
        // For client requests, the URL's Host specifies the server to
        // connect to, while the Request's Host field optionally
        // specifies the Host header value to send in the HTTP
        // request.
        URL *url.URL

        // The protocol version for incoming requests.
        // Client requests always use HTTP/1.1.
        Proto      string // "HTTP/1.0"
        ProtoMajor int    // 1
        ProtoMinor int    // 0

        // A header maps request lines to their values.
        // If the header says
        //
        //	accept-encoding: gzip, deflate
        //	Accept-Language: en-us
        //	Connection: keep-alive
        //
        // then
        //
        //	Header = map[string][]string{
        //		"Accept-Encoding": {"gzip, deflate"},
        //		"Accept-Language": {"en-us"},
        //		"Connection": {"keep-alive"},
        //	}
        //
        // HTTP defines that header names are case-insensitive.
        // The request parser implements this by canonicalizing the
        // name, making the first character and any characters
        // following a hyphen uppercase and the rest lowercase.
        //
        // For client requests certain headers are automatically
        // added and may override values in Header.
        //
        // See the documentation for the Request.Write method.
        Header Header

        // Body is the request's body.
        //
        // For client requests a nil body means the request has no
        // body, such as a GET request. The HTTP Client's Transport
        // is responsible for calling the Close method.
        //
        // For server requests the Request Body is always non-nil
        // but will return EOF immediately when no body is present.
        // The Server will close the request body. The ServeHTTP
        // Handler does not need to.
        Body io.ReadCloser

        // ContentLength records the length of the associated content.
        // The value -1 indicates that the length is unknown.
        // Values >= 0 indicate that the given number of bytes may
        // be read from Body.
        // For client requests, a value of 0 means unknown if Body is not nil.
        ContentLength int64

        // TransferEncoding lists the transfer encodings from outermost to
        // innermost. An empty list denotes the "identity" encoding.
        // TransferEncoding can usually be ignored; chunked encoding is
        // automatically added and removed as necessary when sending and
        // receiving requests.
        TransferEncoding []string

        // Close indicates whether to close the connection after
        // replying to this request (for servers) or after sending
        // the request (for clients).
        Close bool

        // For server requests Host specifies the host on which the
        // URL is sought. Per RFC 2616, this is either the value of
        // the "Host" header or the host name given in the URL itself.
        // It may be of the form "host:port".
        //
        // For client requests Host optionally overrides the Host
        // header to send. If empty, the Request.Write method uses
        // the value of URL.Host.
        Host string

        // Form contains the parsed form data, including both the URL
        // field's query parameters and the POST or PUT form data.
        // This field is only available after ParseForm is called.
        // The HTTP client ignores Form and uses Body instead.
        Form url.Values

        // PostForm contains the parsed form data from POST or PUT
        // body parameters.
        // This field is only available after ParseForm is called.
        // The HTTP client ignores PostForm and uses Body instead.
        PostForm url.Values

        // MultipartForm is the parsed multipart form, including file uploads.
        // This field is only available after ParseMultipartForm is called.
        // The HTTP client ignores MultipartForm and uses Body instead.
        MultipartForm *multipart.Form

        // Trailer specifies additional headers that are sent after the request
        // body.
        //
        // For server requests the Trailer map initially contains only the
        // trailer keys, with nil values. (The client declares which trailers it
        // will later send.)  While the handler is reading from Body, it must
        // not reference Trailer. After reading from Body returns EOF, Trailer
        // can be read again and will contain non-nil values, if they were sent
        // by the client.
        //
        // For client requests Trailer must be initialized to a map containing
        // the trailer keys to later send. The values may be nil or their final
        // values. The ContentLength must be 0 or -1, to send a chunked request.
        // After the HTTP request is sent the map values can be updated while
        // the request body is read. Once the body returns EOF, the caller must
        // not mutate Trailer.
        //
        // Few HTTP clients, servers, or proxies support HTTP trailers.
        Trailer Header

        // RemoteAddr allows HTTP servers and other software to record
        // the network address that sent the request, usually for
        // logging. This field is not filled in by ReadRequest and
        // has no defined format. The HTTP server in this package
        // sets RemoteAddr to an "IP:port" address before invoking a
        // handler.
        // This field is ignored by the HTTP client.
        RemoteAddr string

        // RequestURI is the unmodified Request-URI of the
        // Request-Line (RFC 2616, Section 5.1) as sent by the client
        // to a server. Usually the URL field should be used instead.
        // It is an error to set this field in an HTTP client request.
        RequestURI string

        // TLS allows HTTP servers and other software to record
        // information about the TLS connection on which the request
        // was received. This field is not filled in by ReadRequest.
        // The HTTP server in this package sets the field for
        // TLS-enabled connections before invoking a handler;
        // otherwise it leaves the field nil.
        // This field is ignored by the HTTP client.
        TLS *tls.ConnectionState
}


type URL struct {
        Scheme   string
        Opaque   string    // encoded opaque data
        User     *Userinfo // username and password information
        Host     string    // host or host:port
        Path     string
        RawQuery string // encoded query values, without '?'
        Fragment string // fragment for references, without '#'
}

I don't know how to set the url without the URL struct.

[benjiam shen]

thanks, I think i solved this problem.  It should use

u.Opaque  not u.RawQury  

On Sunday, 20 July 2014 18:47:30 UTC+8, Tamás Gulácsi wrote: