The docs for secretbox seem very wrong

[Dean Schulze]

The docs for secretbox.Seal say:

func Seal(out, message []byte, nonce *[24]byte, key *[32]byte) []byte

Seal appends an encrypted and authenticated copy of message to out, which must not overlap message. The key and nonce pair must be unique for each distinct message and the output will be Overhead bytes longer than message.

Using their example code below I find that

the first argument out is unchanged
the return value is len(nonce) + Overhead longer than the input message, not Overhead longer

It's not clear what the first argument to Seal does.  Nothing is appended to it and it is not changed, but it has to be there.  Does it have to be the nonce[:]?


Likewise the docs for Open say

func Open(out, box []byte, nonce *[24]byte, key *[32]byte) ([]byte, bool)

Open authenticates and decrypts a box produced by Seal and appends the message to out, which must not overlap box. The output will be Overhead bytes smaller than box.

In their example code the out parameter is nil.  So what does it do?  The second argument is encrypted[len(nonce):] which includes the Overhead at the start of the []byte.  Apparently that Overhead is important.

The docs seem wildly wrong.

Here's code based on their example:

func main() {

secretKeyBytes, err := hex.DecodeString("6368616e676520746869732070617373776f726420746f206120736563726574")
if err != nil {
panic(err)
}

var secretKey [32]byte
copy(secretKey[:], secretKeyBytes)

var nonce [24]byte
if _, err := io.ReadFull(rand.Reader, nonce[:]); err != nil {
panic(err)
}

nonceOrig := nonce
fmt.Printf("len(nonce):  %v\n", len(nonce))
s := "hello world"
encrypted := secretbox.Seal(nonce[:], []byte(s), &nonce, &secretKey)
fmt.Printf("len(nonce):  %v\n", len(nonce))
fmt.Printf("len(s): %v\n", len(s))
fmt.Printf("len(encrypted): %v\n", len(encrypted))
if !reflect.DeepEqual(nonceOrig, nonce) {
fmt.Println("nonce changed")
}

var decryptNonce [24]byte
copy(decryptNonce[:], encrypted[:24])

if !reflect.DeepEqual(decryptNonce, nonce) {
fmt.Println("decryptNonce, nonce differ")
}

decrypted, ok := secretbox.Open(nil, encrypted[24:], &decryptNonce, &secretKey)
if !ok {
panic("decryption error")
}

fmt.Println(string(decrypted))

encrypted2 := secretbox.Seal([]byte{}, []byte(s), &nonce, &secretKey)
copy(decryptNonce[:], encrypted2[:24])
decrypted2, ok := secretbox.Open(nil, encrypted2[24:], &decryptNonce, &secretKey)
if !ok {
fmt.Printf("decryption error 2: %v\n", decrypted2)
//panic("decryption error 2")
}
fmt.Printf("decrypted2: %v\n", decrypted2)

encrypted3 := secretbox.Seal(nil, []byte(s), &nonce, &secretKey)
decrypted3, ok := secretbox.Open(nil, encrypted3[24:], &nonce, &secretKey)
if !ok {
fmt.Printf("decryption error 3: %v\n", decrypted3)
}
fmt.Printf("decrypted3: %v\n", decrypted3)
}

[Axel Wagner]

I don't really understand your issue. You call

encrypted := secretbox.Seal(nonce[:], []byte(s), &nonce, &secretKey)

That means you pass `nonce[:]` as the `out` argument, `s` as the `message` argument, and the nonce and key and assign the result to `encrypted`.

According to the docs of `secretbox`, `Seal` will `append` the encrypted message to `nonce[:]` and that encrypted message will be 16 bytes longer than the message, which is 11 bytes long. Appending 37 (16+11) bytes to a 24 byte nonce gives you 51 bytes, which is what you observe as the length of `encrypted`.

The length of `nonce` doesn't change (it's an array, after all) - but passing `append` to a slice does not change the length of the slice, it just returns a new slice, so that seems expected.

So, from what I can tell, the code does exactly what the docs say it should do.

> In their example code the out parameter is nil.  So what does it do?

Appending to `nil` allocates a new slice. The point of still accepting an `out` parameter is that you can potentially prevent an allocation by passing in a slice with 0 length and extra capacity (which can then be allocated on the stack, or which is from a `sync.Pool`). If you don't need that, passing in `nil` seems fine.

> The second argument is encrypted[len(nonce):] which includes the Overhead at the start of the []byte. Apparently that Overhead is important.

Yes, the Overhead is important. It is used to authenticate the message. You can imagine the process of `Seal` as "encrypt the message and attach a hash". The hash is the Overhead. The process also needs a random `nonce`, that both the sender and the receiver need to know. That's why the example code sends it along with the message (it doesn't have to be secret). So that `Seal` call does, effectively (again, for illustrative purposes):

encrypted := append(append(nonce, <encryptedMessage>), <hash>)

As `nonce` is an array, this allocates a new backing array for the returned slice, which ends up filled with

<nonce><encryptedMessage><hash>

The `Open` call then retrieves the `nonce` from the first 24 bytes (by copying it into `decryptNonce`) and passes the `<encryptedMessage><hash>` slice as the `box` argument. Which decrypts the message, authenticates the hash and appends the decrypted message to `out` (which is `nil` in the example code).

So, the docs are correct. And it seems to me, the code works as expected. I'm not sure where the misunderstanding is.

[Axel Wagner]

oh I forgot to emphasize: I don't believe the output is *really* `<encryptedMessage><hash>`. That is, I don't believe you can really treat the first N bytes as the encrypted text and decrypt it (say, if you didn't care about the authentication). It's just that you ultimately need to add 16 bytes of extra information to carry that authentication tag, which is why the box needs to be 16 bytes longer than the message. In reality, the two are probably cleverly mixed - I'm not a cryptographer.

I just wanted to demonstrate where all the information ultimately goes.

[Axel Wagner]

For what it's worth, here is an example that demonstrates a typical encryption/decryption roundtrip, perhaps more clearly:

https://go.dev/play/p/ZZry8IgTJQ_-

The `out` parameter can be used to make this more efficient by using pre-allocated buffers (depending on use case) and there are cases where you don't have to send the nonce, because you can derive them from common data, which is why both of these parameters are there. But in the most common usage, you'd do what this example does.

The example code from the docs tries to be a little bit more efficient and packs the `Box` struct into a single byte, perhaps at the cost of understandability.

[Dean Schulze]

If the docs are correct, how do you append to nil?  That's what the docs say.  Take a look at them.

Your code example shows the first parameter to Seal() can be nil.  So what does that parameter do?  How do you append to it?

[Bruno Albuquerque]

Now including the list and hopefully with less typos. :)

append returns a new slice. Appending to nil just means that you are guaranteed that the returned slice will be allocated inside the append function. The same happens if you try to append to a slice that does not have enough capacity to hold the appended items (i.e. append will create a new larger one, copy the existing data, add the data to be appended and return the new slice).

-Bruno

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/3d7d75e5-d79a-4f81-978d-cb6feb7b12efn%40googlegroups.com.

[Jan Mercl]

On Mon, Oct 9, 2023 at 3:46 PM Dean Schulze <dean.w....@gmail.com> wrote:

> If the docs are correct, how do you append to nil?

https://go.dev/play/p/WY0Bycj-_Tn

[Axel Wagner]

I get the impression that the thing you are missing is that appending to a slice does not modify it. That is, `append(s, x...)` modifies neither the length, nor the content of `s`, you have to type `x = append(s, x...)`.

`Seal` (and `Open`) work exactly the same. They append to the `out` parameter - and to enable you to get at the slice, they return it.

The docs could be slightly more clear, by explicitly stating that they return the appended-to slice.0

On Mon, Oct 9, 2023 at 3:46 PM Dean Schulze <dean.w....@gmail.com> wrote:

--

[Dean Schulze]

So why even bother with the first parameter then?  That looks like a badly designed method signature.

[wagner riffel]

On 09/10/2023 12:43, Dean Schulze wrote:
> So why even bother with the first parameter then?  That looks like a
> badly designed method signature.
>

That is not a badly designed method signature, that is very common in
other Go standard libraries too, if you don't bother with the first
parameter it means that you will allocate a new []byte every time, and
you can't reuse a buffer you possible got from a previous call.

A few places I remember where this design appears in shipped in Go
libraries:

https://pkg.go.dev/strconv#AppendInt
https://pkg.go.dev/unicode/utf8#AppendRune
https://pkg.go.dev/fmt#Append
https://pkg.go.dev/time#Time.AppendFormat

-w

[Axel Wagner]

I explained that: Because it allows you to avoid an allocation, in some cases. If you pass `nil`, then `Seal` must allocate a slice to return it. But in some instances, that allocation can be avoided by re-using a buffer, or having the buffer stack allocated. For example, you can also do this

func EncryptToString(message []byte, nonce *[24]byte, key [32]byte) string {

    b := make([]byte, 0, len(message)+secretbox.Overhead)

    b = secretbox.Seal(b, message, &nonce, &key)

    return hex.EncodeEncodeToString(b)

}

Depending on how good the compiler is, this might well serve an allocation, because `b` can live on the stack, instead of having to be heap-allocated by `Seal`. Or you could do

var pool = sync.Pool{ New: func() any { return []byte(nil) } }

func WriteEncrypted(w io.Writer, message []byte, nonce *[24]byte, key [32]byte) (int, error) {

    b := pool.Get().([]byte)

    b = slices.Grow(b, len(message)+secretbox.Overhead))

    defer pool.Put(b[:0])

    return w.Write(b)

}

Or a dozen other patterns, depending on what you want to do with the encrypted or decrypted message.

To be clear: I don't necessarily agree with all the design decisions the `secretbox` API made. But there are reasons for it and those reasons can be understood.

To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/0fc63e62-0911-4ad1-b209-570c063a61c3n%40googlegroups.com.

[Axel Wagner]

s/serve/save/

[David Anderson]

The first parameter lets you avoid an allocation. If you're constructing a packet that consists of ciphertext surrounded by some framing, you can make([]byte, 0, lengthOfPacket) to preallocate the necessary amount of storage upfront. Then, calls to secretbox don't allocate.

If you don't need that behavior, you can pass in a nil as the first parameter to get a fresh slice allocated for you by the append(), but in hot codepaths involving cryptography, being able to preallocate and reuse storage (e.g. via sync.Pool) can give you significant reduction in GC pressure.

- Dave

To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/0fc63e62-0911-4ad1-b209-570c063a61c3n%40googlegroups.com.

[Dean Schulze]

That is mixing concerns (low cohesion), though.  At the very least they need to explain that in the docs.  Otherwise that first parameter makes no sense.

[Axel Wagner]

Feel free to send in a CL to clarify the docs.

To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/7542b613-d9bf-43aa-ad98-877434350de8n%40googlegroups.com.

[Ian Lance Taylor]

On Mon, Oct 9, 2023, 1:59 PM Dean Schulze <dean.w....@gmail.com> wrote:

That is mixing concerns (low cohesion), though.  At the very least they need to explain that in the docs.  Otherwise that first parameter makes no sense.

This is a common design pattern in Go.  We shouldn't expect that every function that uses it has to explain the idea.  We should figure out a way for people to discover and understand it.

Ian

To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/7542b613-d9bf-43aa-ad98-877434350de8n%40googlegroups.com.

[Christian Stewart]

I use this parameter, as others mentioned earlier in this thread, to
avoid re-allocating out.

Please re-review the docs and I think you'll find that with your
updated knowledge of how append(nil, ...) works, it makes a lot of
sense.

When reading those docs, I personally never felt any confusion as to
what the parameters were for, especially if you look at the examples.

best regards,
Christian Stewart

> To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/0fc63e62-0911-4ad1-b209-570c063a61c3n%40googlegroups.com.