Download large Zip file via sftp

[Phillip Siessl]

Hi 

i have some issues when i try to download a larger zip file (5GB) via the sftp package in go. 

about 20% of the time it runs through smoothly but the other 80% i get a panic with

"fatal error: concurrent map writes writing to file" or "fatal error: concurrent map read and map write".

this is the code for the download function: 

func DownloadZip(sc sftp.Client, remoteFile, localFile string) (err error) {

fmt.Fprintf(os.Stdout, "Downloading [%s] to [%s] ...\n", remoteFile, localFile)

srcFile, err := sc.OpenFile(remoteFile, (os.O_RDONLY))

if err != nil {

fmt.Fprintf(os.Stderr, "Unable to open remote file: %v\n", err)

return

}

defer srcFile.Close()

fmt.Println("Create locale file")

dstFile, err := os.Create(localFile)

if err != nil {

fmt.Fprintf(os.Stderr, "Unable to open local file: %v\n", err)

return

}

defer dstFile.Close()

fmt.Println("Copy remote file")

bytes, err := io.Copy(dstFile, srcFile)

if err != nil {

fmt.Fprintf(os.Stderr, "Unable to download remote file: %v\n", err)

os.Exit(1)

}

fmt.Println("Save local file")

fmt.Fprintf(os.Stdout, "%d bytes copied\n", bytes)

return

}

If anybody has some ideas i would be very happy. thx

[Tamás Gulácsi]

Possibly it's a concurrency error in the sftp library you're using.

Compile with "-race" flag and test it - it should print out the offending code.

[burak serdar]

On Wed, Aug 23, 2023 at 6:37 AM Phillip Siessl <phillip...@gmx.at> wrote:

Hi 

i have some issues when i try to download a larger zip file (5GB) via the sftp package in go. 

about 20% of the time it runs through smoothly but the other 80% i get a panic with

"fatal error: concurrent map writes writing to file" or "fatal error: concurrent map read and map write".

There are no map operations in the code you included. The error is happening somewhere else. The panic should include enough information to tell where it is coming from.

 

this is the code for the download function: 

func DownloadZip(sc sftp.Client, remoteFile, localFile string) (err error) {

fmt.Fprintf(os.Stdout, "Downloading [%s] to [%s] ...\n", remoteFile, localFile)

srcFile, err := sc.OpenFile(remoteFile, (os.O_RDONLY))

if err != nil {

fmt.Fprintf(os.Stderr, "Unable to open remote file: %v\n", err)

return

}

defer srcFile.Close()

fmt.Println("Create locale file")

dstFile, err := os.Create(localFile)

if err != nil {

fmt.Fprintf(os.Stderr, "Unable to open local file: %v\n", err)

return

}

defer dstFile.Close()

fmt.Println("Copy remote file")

bytes, err := io.Copy(dstFile, srcFile)

if err != nil {

fmt.Fprintf(os.Stderr, "Unable to download remote file: %v\n", err)

os.Exit(1)

}

fmt.Println("Save local file")

fmt.Fprintf(os.Stdout, "%d bytes copied\n", bytes)

return

}

If anybody has some ideas i would be very happy. thx

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/7afc7aed-a362-4e93-aeec-d5a42005c10cn%40googlegroups.com.

[Ozan Hacıbekiroğlu]

Hi Phillip,

Can you please try to use *sftp.Client as a pointer instead of sftp.Client? In addition, concurrent read configuration provided to sftp client can be omitted to reproduce.

23 Ağustos 2023 Çarşamba tarihinde saat 15:37:10 UTC+3 itibarıyla Phillip Siessl şunları yazdı:

[Howard C. Shaw III]

I want to second what Ozan has said, use *sftp.Client not sftp.Client, but I want to throw out a couple of words about that.

First, the documentation specifically warns against copying Mutex. So you need to be fairly certain that no mutexes are used in a  struct before using value semantics. Anything that mentions it is using concurrency or is safe to use under concurrency is likely to be using mutexes, so probably needs pointer semantics. 

Second, I want to recommend that in looking at the documentation, you pay particular attention to the method signatures, and especially the receiver. If in the documentation of a struct all the methods have a pointer receiver; i.e. they look like:

func (c *Client) Close() error

             ^^^

having the * in front of the class name, this is a good indication that you need to pass it around yourself in that way. If they all look like:

func (c Client) Close() error

without the *, then you are pretty safe to pass it around as a value, since all the method calls are going to be doing that anyway.

Finally, if the documentation includes a New{Struct} function, pay attention to the return type - if it returns a pointer, pass it around as a pointer. If it returns a value, then you can probably leave it as a value.

Good luck!

Howard