net/http "Network Error" on file serving

[569...@gmail.com]

I'm attempting to serve PDF files from a Go app, however I am getting an unknown "network error" response half way through the download. This happens when attempting to download on both Chrome & Firefox but not if I use wget from a remote server to download the file.

It must be some way that net/http is attempting to serve the file. They're PDFs, all cached, so the PDF itself is a slice of bytes currently in memory. The problem is not the slice of bytes that is being sent, those are all valid and complete PDF files.

For some downloads it works correctly, and for others the download cuts off with a "Network Error" response from Chrome. You can see an example of it doing this here:

https://www.forgottenbooks.com/en/download/Kashmir_10291127.pdf

Following is the code for the download:

func DownloadPDF(rw http.ResponseWriter, r *http.Request, fid int) {
 
 book := engine.BookInfo[fid]
 data, err := bookcache.PDF(int(fid))
 if err != nil {
 panic(err)
 }
 
 rw.Header().Set(`Content-Type`, `application/pdf`)
 rw.Header().Set(`Filename`, book.LinkTitle + `.pdf`)
 rw.Header().Set(`Content-Disposition`, `attachment`)
 rw.Header().Set(`Content-Transfer-Encoding`, `binary`)
 rw.Header().Set(`Accept-Ranges`, `bytes`)
 rw.Header().Set(`Connection`, `close`)
 rw.Header().Set(`Content-Length`, conv.String(len(data)))
 
 // Send
 var i int
 to := len(data) - 65536
 for ; i<to; i+=65536 {
 rw.Write(data[i:i+65536])
 }
 rw.Write(data[i:])
}

Does anyone have an idea as to why this could be happening and how to fix it?'

Thanks,

Alasdair

[Dave Cheney]

I would handle the error properly rather than panicking, instread return a 500 status code. You don't need to do your own chunking of the write, just write the whole file to the client in one call.

[569...@gmail.com]

Hi Dave,

Thanks for your advice. Do you have anything to say about the file only partially downloading?

BTW, I'm using Go 1.6.2 and this is an SSL site (which I think is related to the issue.)

Thanks!

[Dave Cheney]

Both my suggestions relate to the problem you described.

[569...@gmail.com]

Hi Dave, neither one of those is the issue:

The download is not failing, it's half-downloading and then stopping with network error. So the panic is unrelated entirely.

Regarding the chunking, I will remove this as you suggested. However, I actually put that in in an attempt to see if it solved the network error issue. Previously I had just written it all in one write, the same as what you are suggesting. The network error appears regardless of the chunking or single write.

[Axel Wagner]

I can not reproduce your problem. I downloaded the linked file three times and every time it completed correctly. Using Chrome 49 on a Fedora Linux on a very fast internet connection.

One thing that jumps out to me, is that you set 'Accept-ranges', but don't seem to actually accept ranges.

But it's also extremely difficult to troubleshoot this, if a) we don't have all the code, b) we don't have the data, c) we don't even have the logs and d) we can't even reproduce the issue. Try adding logging and looking at the output of the server binary for a failed download. You could also try to inspect the network traffic of a failed download, either with chrome developer tools or using wireshark or the like to find out what the actual requests including headers are, that are send and find differences between what chrome does and what wget/curl do.

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Dave Cheney]

Ok, next step would be to log the results of the write call, if there is failing that will give you a clue which party is at fault.

[569...@gmail.com]

Hi Axel,

The accept-ranges was another attempt of mine to try various headers to solve the issue. It makes no difference so I can remove that.

I suspect that the SSL has something to do with the "network error".

Also Internet speed does seem relevant. The download seems to stop after a specific amount of time rather than after a certain number of KB, so fast connections can download it before it stops and don't have the issue. I have ReadTimeout set to 10 minutes and WriteTimeout set to 1 hour, so that variable I doubt is the issue, unless there is another timeout variable hidden away in net/http or tls?

[Axel Wagner]

I read your assumption about TLS being the issue the first two times, I just tend to disagree, until proven otherwise. The go TLS implementation is widely used without these problems.

Again, without further information, it's very hard to assist you in debugging this remotely over E-Mail. I gave you a couple of general hints about how to find the problem. Try them, or, if you want other people to try them for you, provide the full source code, data and steps to reproduce (ideally locally, to rule out effects that depend on connection latency, throughput or the like).

--

[569...@gmail.com]

Hi Dave,

OK, I changed the code to:

  rw.Header().Set(`Content-Type`, `application/pdf`)
 rw.Header().Set(`Filename`, book.LinkTitle + `.pdf`)
 rw.Header().Set(`Content-Disposition`, `attachment`)
 rw.Header().Set(`Content-Transfer-Encoding`, `binary`)

 rw.Header().Set(`Content-Length`, conv.String(len(data)))
 rw.Header().Set(`Cache-Control`, `must-revalidate, post-check=0, pre-check=0`)
 rw.Header().Set(`Pragma`, `public`)
 if i, err := rw.Write(data); err != nil {
 log.Println(err, ` : Wrote`, i, `of`, len(data))
 }

And got this:

2016/06/01 07:26:24 http2: stream closed  : Wrote 0 of 54417789

Does that tell us anything that can be used to solve the issue?

Thanks!

[Dave Cheney]

The other side didn't like what you sent it. Try with curl. Also, drop the content length header, it's not necessary.

[569...@gmail.com]

Weirdly it says it transferred 0 bytes but that's not true, it transferred several MB before failing. And where it fails is always a binary part of the PDF file after "stream" so I wonder if it's trying to interpret that as something rather than part of the payload?

What do you mean to try with CURL? Try sending it with CURL? How can I do that, it's all part of a much more complicated webserver app?

[Dave Cheney]

On Wednesday, 1 June 2016 16:12:45 UTC+10, 569...@gmail.com wrote:

Weirdly it says it transferred 0 bytes but that's not true, it transferred several MB before failing. And where it fails is always a binary part of the PDF file after "stream" so I wonder if it's trying to interpret that as something rather than part of the payload?

How do you know it transmitted several mb ? Is there a proxy between your program and a client ?

 

What do you mean to try with CURL? Try sending it with CURL? How can I do that, it's all part of a much more complicated webserver app?

I don't understand, replace chrome with cURL, it's just a get request, right ? 

[Dave Cheney]

curl -v https://www.forgottenbooks.com/en/download/Kashmir_10291127.pdf > /dev/null

works fine, but I'm 99% sure this version of curl doesn't speak http2. If your go program is handling the ssl termination, i'd try disabling http2 and upgrading to Go 1.6.2 if you haven't already.