How to return a file pointer from data loaded from the web?

[agivney]

I been having trouble solving this problem, I am a novice Golang programmer.

The following code loads a file from the web, and returns the contents of the file, this is fine. However I wish to return a file handle/pointer instead, how would I do this? The reason I do not wish to create a physical l file is that this software scans potentially 100,000 of files from web (from web address logs) and there is no need to write these files to the disk.

I tried using the mmap-go package, but it does not seem to have ability to create a file from a []byte sequence. It only seems to create a file from loading a real file, but in my situation, I am loading a file into memory from the web, and this data structure does not fit into the rest of my code which works with files.


func GetFileFromWeb(webAddress string) []byte {
  
    //debug code
    webAddress = "https://d1ohg4ss876yi2.cloudfront.net/preview/golang.png"
    rawURL := webAddress

    //I do not wish to create a physical file, but one in memory, this code is in preparation for the io.Copy(file.resp.Body) below which a creates a real file.
    //fileURL, err := url.Parse(rawURL)
    //file, err := os.Create("test2.jpg")
    //defer file.Close()

    check := http.Client{
        CheckRedirect: func(r *http.Request, via []*http.Request) error {
            r.URL.Opaque = r.URL.Path
            return nil
        },
    }
    resp, err := check.Get(rawURL) // add a filter to check redirect
//this code simply copies the loaded from memory into the filehandle created above, but I do not want this
    //pictureData, _ := ioutil.ReadAll(resp.Body)

    defer resp.Body.Close()
    // fmt.Println(resp.Status)

    // size, err := io.Copy(file, resp.Body)

    if err != nil {
        panic(err)
    }

    // fmt.Printf("%s with %v bytes downloaded", "test2.jpg", size)
// This returns an array of bytes, but I want to return a file handle pointing to the data instead.
    return pictureData
}

Thanks in advance.

[Dave Cheney]

resp.Body is an io.Reader, you can just return that from your function. Please remember to close it.

Thanks

Dave

[agivney]

Thanks Dave. I think the root of my problem was not understanding how io.Reader works and not taking into account that the buffer empties once it is read.

For the sake of helping others I will stick up my working function, of course I am sure it can be improved significantly.

func GetFileFromWeb(webAddress string) (io.Reader, int) {
    //debug code
    webAddress = "http://www.personal.psu.edu/jul229/mini.jpg"

  

 

    check := http.Client{
        CheckRedirect: func(r *http.Request, via []*http.Request) error {
            r.URL.Opaque = r.URL.Path
            return nil
        },
    }

    resp, err := check.Get(webAddress) // add a filter to check redirect
  
//This creates a copy of the buffer so I can obtain the length of the buffer, since once the buffer is read it is empty
    buf, _ := ioutil.ReadAll(resp.Body)
    rdr2 := myReader{bytes.NewBuffer(buf)}
    defer resp.Body.Close()

 
    if err != nil {
        panic(err)
    }

    return rdr2, len(buf)

}

Thank Arthur.

[Mat Evans]

Hi,

I think you should have another look at io.Reader - in the function below you are basically reading the contents of the http.Response.Buffer into another Buffer. Instead of doing that, may be it would be good to use the initial Body buffer as the source of streaming data as Dave suggested..