monitoring io.Copy progress

2,828 views
Skip to first unread message

gocss

unread,
Jan 7, 2016, 9:56:29 AM1/7/16
to golang-nuts

need to monitor progress of io.Copy(dest,src)

anybody have an example doing this ?

seems need to add filter type func i.e.
io.Copy(dest, filter(src))

(sorry if a later post appears, posted with wrong email account,
was scratching head why this post had not yet appeared)

Caleb Doxsey

unread,
Jan 7, 2016, 10:12:12 AM1/7/16
to golang-nuts
You could use a TeeReader and implement a Writer which tracks the progress: https://play.golang.org/p/N6xL8_fnV2.

For filtering you could use the golang.org/x/text/transform package. Create a type which implements Transformer and then use NewReader / NewWriter accordingly.

paulto...@gmail.com

unread,
Jan 7, 2016, 11:57:32 AM1/7/16
to golang-nuts

I want to monitor the progress of io.Copy(dest, src)
Any example of this done yet ?

Seems need to replace src with func i.e.
io.Copy(dest, filter(src))

anybody have example of func filter ?


gocss

unread,
Jan 7, 2016, 12:13:18 PM1/7/16
to golang-nuts
Caleb,

the one in the playground is what I was looking for :) thanks,

Is there  a way within Write function of LogProgressWriter to detect when EOF occurs ?

the reference to x/text/transform  I get
Internal server error.

Caleb Doxsey

unread,
Jan 7, 2016, 2:06:51 PM1/7/16
to golang-nuts
Another way of doing the same thing would be to wrap the reader like this: https://play.golang.org/p/6nZCwXLlgT .

godoc.org is sadly broken right now. You can get docs on that package from the terminal. Run:


- Caleb

Chris Kastorff

unread,
Jan 8, 2016, 1:22:57 PM1/8/16
to paulto...@gmail.com, golang-nuts
Create a type that wraps the Reader and whatever you need to show progress (maybe nothing), then you can count the number of bytes read from each inner Read call.

For example, this code does that and limits the progress printing rate to once per 256KiB: https://github.com/encryptio/go-git-annex-external/blob/master/external/progress.go

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Andy Song

unread,
Jan 8, 2016, 8:19:52 PM1/8/16
to Chris Kastorff, paulto...@gmail.com, golang-nuts
Or maybe run your io.Copy() in a separate goroutine and loop&check your dst length in your main goroutine.

Val

unread,
Jan 9, 2016, 12:05:54 PM1/9/16
to golang-nuts, encr...@gmail.com, paulto...@gmail.com
Hi Andy, I'm not sure what you mean by "dst length" but this approach looks very data-race-prone.
Cheers
Val

contact....@gmail.com

unread,
Dec 31, 2016, 12:58:47 AM12/31/16
to golang-nuts
Hi, I saw your very interesting answer.

Does that mean that io.copy will Always call the underlying io.Read method ? Can you then update a counter rather than just print stuff in the Read function ? Thanks for your answer.
Reply all
Reply to author
Forward
0 new messages