hashstream: reader / writer hash

[Glen Newton]

I wanted to do a sha512 on a really large file (too large for memory) and went looking for something like Java's https://docs.oracle.com/javase/7/docs/api/java/security/DigestInputStream.html but didn't find anything.

So I wrote a small library for this:
https://github.com/gnewton/hashstream
 godocs: https://godoc.org/github.com/gnewton/hashstream

 reader and writer implemented
General constructors taking a reader or writer and a hash, as well as convenience constructors taking only a reader or writer for the common hashes (MD5, SHA1, SHA224, SHA256, SHA384,SHA512).

Feedback on the code appreciated.

Oh, I return an error when a zero length []byte or nil reader is passed in. Convention is to panic? Please advise.

Also if this already exists in the community, please let me know.

Thanks,
Glen

[Dustin]

Just wondering, why couldn't you use: https://golang.org/pkg/crypto/sha512/

[Glen Newton]

I am reading from an incoming stream, transforming it,  and writing to a file. Once the file is written i want the SHA1 of the file. The resulting file might be very large and I do not want to keep it in memory.
This library allows me a convenient way of doing this. [Internally it uses https://golang.org/pkg/crypto/sha512/ among others]

I could handle things separately but this is easier.

[Dustin]

Why not just use io.MultiWriter(hasher, file)? http://golang.org/pkg/io/#MultiWriter

Each write will be stream written to both the hash and the file, then just get the sum when you're done writing?

[Brad Fitzpatrick]

Exactly. io.MultiWriter was actually added to the standard library for exactly this purpose originally, writing to one place while also taking the digest of what was written. (of course, it ends up having lots of uses)

On Sun, Aug 16, 2015 at 6:44 AM, Dustin <djhe...@gmail.com> wrote:

Why not just use io.MultiWriter(hasher, file)? http://golang.org/pkg/io/#MultiWriter

Each write will be stream written to both the hash and the file, then just get the sum when you're done writing?

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Glen Newton]

Right. Yes, I hadn't looked at that or thought of it in that way.  That is better!  :-)

Thanks,
Glen

[Glen Newton]

I'll leave the repo but replace the docs with an example of my particular use case (with writer and reader) using MultiWriter...

[Glen Newton]

Done. md5 hash writer MultiWriter example in repo docs.

thanks,
Glen

[Giulio Iotti]

On Sunday, August 16, 2015 at 6:54:50 AM UTC+3, Glen Newton wrote:

Oh, I return an error when a zero length []byte or nil reader is passed in. Convention is to panic? Please advise.

No, please don't panic. (Pun intended.)

If you are in main (it's an executable), just use "log.Fatal(err)" instead.

-- 

Giulio Iotti

https://github.com/dullgiulio