Converting sha1 hash to integer?

[Victor Hooi]

I'm trying to convert a sha1 hash into an integer, so that it can use it to index an array of words (i.e. to pick a word from the list). s below is a string:

var myint int
buf := bytes.NewBuffer(sha1.Sum([]byte(s)))
binary.Read(buf, binary.LittleEndian, &myint)

However, I get:

cannot use sha1.Sum(([]byte)(s)) (type [20]byte) as type []byte in argument to bytes.NewBuffer

Is it somehow not able to treat a 20-byte array as a []byte?

Also, if there's an better way to do this, please let me know.

[aro...@gmail.com]

sha1.Sum returns an array ([20]byte), not a slice, but bytes.NewBuffer takes a slice, not an array.

hashBytes := sha1.Sum([]byte(s))

buf := bytes.NewBuffer(hashBytes[:])

should work.

[Victor Hooi]

Aha, ok, so that explains it - I should convert the array to a slice before passing it to binary.Read, which is what bytes.NewBuffer() does.

input_string := "The quick brown fox jumped over the lazy dog"
var myint int
hashBytes := sha1.Sum([]byte(input_string))
buf := bytes.NewBuffer(hashBytes[:])
binary.Read(buf, binary.LittleEndian, &myint)
fmt.Println(hashBytes)
fmt.Println(myint)

However, for some odd reason, I'm now getting 0 (zero) as the value of myint?

[Rob Pike]

A SHA hash won't fit in an int, so you'll only get the part of the
hash you select by asking for LittleEndian. I'm sure you know that but
I want to be clear.

I'd do something different and just use the [20]byte as a key for a
map. No allocation required or information ignored.

-rob

> --
> You received this message because you are subscribed to the Google Groups
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-nuts...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Val]

Hi Victor,
the first problem in this code is ignoring the error returned by binary.Read .
The second problem is told inside the aforementioned error message :

  binary.Read: invalid type *int

Cheers

[roger peppe]

On 22 July 2015 at 23:01, Victor Hooi <victo...@gmail.com> wrote:
> Aha, ok, so that explains it - I should convert the array to a slice before
> passing it to binary.Read, which is what bytes.NewBuffer() does.
>
>> input_string := "The quick brown fox jumped over the lazy dog"
>> var myint int
>> hashBytes := sha1.Sum([]byte(input_string))
>> buf := bytes.NewBuffer(hashBytes[:])
>> binary.Read(buf, binary.LittleEndian, &myint)
>> fmt.Println(hashBytes)
>> fmt.Println(myint)
>
>
> However, for some odd reason, I'm now getting 0 (zero) as the value of
> myint?

+1 to rob's remarks. But I'll just say that the above code can
be quite a lot simpler:

hashBytes := sha1.Sum([]byte("The quick brown fox jumped over the
lazy dog"))
myint := binary.BigEndian.Uint64(hashBytes[:])

http://play.golang.org/p/jK_v5Y_0tX

BTW note that you can't use binary.Read with int because
it can vary in size (that's why your code doesn't work).
If you changed int to (say) int64, it could work.

cheers,
rog.

>
> On Thursday, 23 July 2015 07:48:16 UTC+10, aro...@gmail.com wrote:
>>
>> sha1.Sum returns an array ([20]byte), not a slice, but bytes.NewBuffer
>> takes a slice, not an array.
>>
>> hashBytes := sha1.Sum([]byte(s))
>> buf := bytes.NewBuffer(hashBytes[:])
>>
>> should work.
>>
>> On Wednesday, July 22, 2015 at 2:37:10 PM UTC-7, Victor Hooi wrote:
>>>
>>> I'm trying to convert a sha1 hash into an integer, so that it can use it
>>> to index an array of words (i.e. to pick a word from the list). s below is a
>>> string:
>>>
>>> var myint int
>>> buf := bytes.NewBuffer(sha1.Sum([]byte(s)))
>>> binary.Read(buf, binary.LittleEndian, &myint)
>>>
>>> However, I get:
>>>
>>>> cannot use sha1.Sum(([]byte)(s)) (type [20]byte) as type []byte in
>>>> argument to bytes.NewBuffer
>>>
>>>
>>> Is it somehow not able to treat a 20-byte array as a []byte?
>>>
>>> Also, if there's an better way to do this, please let me know.
>

[Victor Hooi]

Hi,

Yeah, I was planning to just take the low-order bits - roughly enough to cover the entries in my wordlist, and use that to index in my wordlist. However, I figured I first needed to understand how to convert a byte array into an int.

I'm still a bit confused about the zero part - I had thought binary.Read would still read in some bytes, so that the output is not zero?

You mention using the [20]byte directly as the key for a map - sorry if this is a stupid question, but are you perhaps able to provide any more details at all about that?

For arguments sake, my wordlist is something like this:

var cipher_words = []string{"aardvark", "buffalo", "camel", "etc" }

How exactly would I convert that into a map that could be keyed by the entire space of [20]byte values? 

Thanks,

Victor

[Val]

Hi Victor
I'll give a try on these two questions:

[20]byte is a fixed-sized type, and is suitable to be the key type of a map  (unlike, say, []byte which cannot be used as key type).
Here is a demo : http://play.golang.org/p/4R5eg_qKfX

You may also rename the custom key type if it makes your program clearer : http://play.golang.org/p/8-CRurqv5v

Hopes this helps
Best regards

On Thursday, July 23, 2015 at 12:43:36 AM UTC+2, Victor Hooi wrote:

You mention using the [20]byte directly as the key for a map - sorry if this is a stupid question, but are you perhaps able to provide any more details at all about that?

[Victor Hooi]

Thanks for the Playground link, particularly http://play.golang.org/p/4R5eg_qKfX. I'm still trying to wrap my head around what Rob wrote earlier, but it's certainly helping.

If I'm reading this right, we just iterate over our wordlist, sha-1 hash the actual word, and use the hash as the key back to the word.

However, this means there's no guarantee that each hash we encounter will actually map to a word, correct?

To clarify, this was my original aim:

var cipher_words = []string{"aardvark", "buffalo", "camel", "dugong", "elephant"} // In reality, this would probably be a wordlist of about 1 million words
input_word := "transactions"
hashBytes := sha1.Sum([]byte(input_word))
/* Do some stuff with hashBytes to get an integer between 0 and 1 million (or thereabouts) */ 

output_word := cipher_words[int_from_hash % len(cipher_words)]

In this way, we're always guaranteed to pick a word from the wordlist.

Is there a way to make the map approach Rob suggested still work with that guarantee?

[Val]

Fair enough, now I understand better the use case.
Here is a simple demo with hash/fnv which returns directly a uint32, so we can use a slice instead of a map : http://play.golang.org/p/k8bws03uid

Warning: hash/fnv is explicitly a non-cryptographic hash function, so it may or may not fit your needs, depending on how your program will be used and how bad it would be if someone managed to guess input when knowing only the output. With only 1M words, you are prepared to have a lot of collisions (familiar with the birthday problem?).
You may also combine two hashes, for example sha1 which is strong and returns bytes, then fnv which is weak and returns uint32 : http://play.golang.org/p/YfjQmJwULi

Cheers