Adding methods to non local types that use more state than available in the non local type

[Alexander Shopov]

Hi all,

I need some guidance whether what I want to do is somehow possible in Go. I've already searched answers for two days. My problem boils down to how to sneak more in a type without changing the type.

Lets say I have to implement the following method:

func (s *server) Get(ctx context.Context, request *generated.Request) (*generated.Response, error) {
    var response generated.Request
    return &response, nil

}

In this case it is a zero value (nil) that will be returned for response but that is not the problem - I am just marking the types

I want to return a response that somehow implements a wider interface than generated.Response

Since that object is generated source, I cannot change it or add methods to it, however I can do the following:

// Declare a new type that embeds the generated.Response
type EnrichedResponse generated.Response

// Add a method that depends only on r
func (r *EnrichedResponse) Enriched() int {
    return r.foo + 42
}

// Return the enriched response
func (h *handler) Get(ctx context.Context, request *generated.Request) (*generated.Response, error) {
    var enriched EnrichedResponse
    response := generated.Response(enriched)
    return &response, nil
}

My problem is the Enriched() method - it can only use the state available in the initial generated.Response and I need more in order to implement the functionality.

I cannot just make EnrichedResponse a struct embedding generated.Response and add more state because then I cannot do the conversion from normal response to enriched.

So here is my question - is there any way to have this state somewhere so that the Enriched() method does not take more arguments but can be called directly.
A method declaration in Go is quite static. Additional state can be kept in some kind of closure but then I have no idea how to glue that closure to the method.

Is there any trick I am missing?
-----
Why would I want to do this? What am I trying to achieve?

Basically there is a lot of generated code and I want to keep compatibility with it.
Similar to the way Go embeds wider interfaces into narrower ones I want to be able to add methods to the generated code without having to change it.

Then - whenever some code calls the Get method on the server - based on the whether the returned value implements the Enriched interface or not and the value it returns - I can dispatch behavior.

Kind regards:

al_shopov

[Robert Engels]

Just have the parameters be interfaces and do reflection/type casting. 

On May 11, 2025, at 5:25 PM, Alexander Shopov <a...@kambanaria.org> wrote:



--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/golang-nuts/bbe6bcd8-e33c-41bf-868a-e498561c3e72n%40googlegroups.com.

[Jason E. Aten]

I think you have just misunderstood how to embed types in Go, because you say

> // Declare a new type that embeds the generated.Response
> type EnrichedResponse generated.Response

and this is not type embedding, it is just a type definition, giving a new name and new type

for the existing structure that is generated.Response. Here is an example of embedding:

type EnrichedResponse struct {

   generated.Response // notice there is no field name here, but in literals refer to it by generated.Response

   otherNewState int

}

Now you can call any Reponse methods on EnrichedResponse, and also

call new methods that you add to EntrichedResponse.

[Robert Engels]

If you look at the example given the function accepts the base type and then tries to call the function needing the extended type - pretty sure you can’t do this with embedding as the type is erased when the outer function is called. You need an interface afaik. 

On May 11, 2025, at 8:43 PM, Jason E. Aten <j.e....@gmail.com> wrote:

I think you have just misunderstood how to embed types in Go, because you say

To view this discussion visit https://groups.google.com/d/msgid/golang-nuts/96f2f60e-61c6-4df3-984b-d14583013354n%40googlegroups.com.

[Axel Wagner]

On Mon, 12 May 2025 at 00:25, Alexander Shopov <a...@kambanaria.org> wrote:

I cannot just make EnrichedResponse a struct embedding generated.Response and add more state because then I cannot do the conversion from normal response to enriched.

Why does this have to be a conversion? What's the downside of having to type `r.Response` instead of `generated.Response(r)`? In fact, the former seems preferable?

So here is my question - is there any way to have this state somewhere so that the Enriched() method does not take more arguments but can be called directly.
A method declaration in Go is quite static. Additional state can be kept in some kind of closure but then I have no idea how to glue that closure to the method.

Is there any trick I am missing?

If you want your coworkers sad, and the Engineering gods angry, you can use unsafe: https://go.dev/play/p/tEJOLav360c

A less evil, but still bad way, would be to store it in a global map: https://go.dev/play/p/ZHBJVCduf25 (note: I'm not sure this use of weak pointers is actually correct, I haven't used them yet).

Otherwise, no. The state has to live somewhere. And really, both of these tricks are still really bad and you shouldn't do them. The best way is to respect the type system and if you want extra methods, you need a new type and if that needs extra state, that should be stored in the type.

-----
Why would I want to do this? What am I trying to achieve?

Basically there is a lot of generated code and I want to keep compatibility with it.
Similar to the way Go embeds wider interfaces into narrower ones I want to be able to add methods to the generated code without having to change it.

Change the code generator, and/or add additional code in an extra file. Hacking the language will cause you more pain, in the long run.

 

Then - whenever some code calls the Get method on the server - based on the whether the returned value implements the Enriched interface or not and the value it returns - I can dispatch behavior.

Kind regards:

al_shopov

--

[Axel Wagner]

On Mon, 12 May 2025 at 07:05, Axel Wagner <axel.wa...@googlemail.com> wrote:

A less evil, but still bad way, would be to store it in a global map: https://go.dev/play/p/ZHBJVCduf25 (note: I'm not sure this use of weak pointers is actually correct, I haven't used them yet).

Okay, I'm pretty sure it's not correct. It should be possible to make this work somehow, but I'm a bit too lazy to try right now, especially as you shouldn't do it anyways.

[Alexander Shopov]

Hello all and thanx for the answers. Here I provide more information:

@Robert Engels

| Just have the parameters be interfaces and do reflection/type casting. 

I sadly cannot do this (at least right away). That is why I asked for help. I cannot guarantee the parameters are interfaces - they are what gets generated and that includes structs.

@Jason E. Aten

> // Declare a new type that embeds the generated.Respons

> type EnrichedResponse generated.Response

and this is not type embedding, it is just a type definition

You are correct. Sorry for the confusion the doc comment caused. This is indeed a type definition. This was meant as a connection to the alternative I pointed out:

Make EnrichedResponse a struct embedding generated.Response and add more state like this:

type EnrichedResponse struct {

    generated.Response

    additionalState int

}

Then Enriched is trivial:

func (r *EnrichedResponse) Enriched() int {

    return r.additionalState
}

And it is trivial to add the new state but returning it is impossible:

func (h *handler) Get(ctx context.Context, request *generated.Request) (*generated.Response, error) {

    enriched := EnrichedResponse{

        Response: ....,

        additionalState: 42,

    }

    response := generated.Response(enriched)

    // ^^^^^ WILL NOT WORK

    return &response, nil
}

But then I cannot do the conversion as the types have different memory representations. Wagner's first suggestion points to ways around this.

@Axel Wagner

The global map is something I also thought about but it introduces contention and is generally ugly. The usage of weak.Pointer-s also guards against cases when Response is not comparable. But this would be a bad solution.

The unsafe.Pointer is a cuter and enticingly evil way. But in the end - it is explicitly opting out of type safety. The return of the unnamed struct is nice! 

Is the equality of pointer to struct and pointer to its first member actually always guaranteed by Go lang? https://go.dev/ref/spec#Package_unsafe says The effect of converting between Pointer and uintptr is implementation-defined but I guess it is not the same thing. The sections on Struct types and Alignment do not give such guarantees.

In the end it is not wise to use such a technique on a large code base shared by many developers. I will not be going this way but many thanx for the example it taught me something I did not know.

Axel, you may use it for a 2nd part of Advanced Generics Patterns (nice presentation! I liked it) - Advanced Type Patterns (use types wrongly so we can learn how to make the wrong way the right way).

I was hoping that there is a trick that I am missing. Like - a returned function can access state in a closure but I see no way to do it for methods. I definitely do not want to go the direction - since I want to add a method I should change the code generation, types (struct vs interface-s)  to suit that small change.

Kind regards:

al_shopov

[Axel Wagner]

On Mon, 12 May 2025 at 11:12, Alexander Shopov <a...@kambanaria.org> wrote:

Is the equality of pointer to struct and pointer to its first member actually always guaranteed by Go lang? https://go.dev/ref/spec#Package_unsafe says The effect of converting between Pointer and uintptr is implementation-defined but I guess it is not the same thing. The sections on Struct types and Alignment do not give such guarantees.

Perhaps not, though it seems a very common assumption and if an implementation violates it, I would expect that to break a bunch of code.

You can always make sure, by accounting for a potential offset: https://go.dev/play/p/whdFALRG20P

I guess that still makes the assumption, that a pointer to the entire value stays alive while any pointer to one of its field is alive. I'm not sure *that* is guaranteed either, but at least it's a weaker assumption.

[Alexander Shopov]

seems a very common assumption

I would have guessed it is due to compatibility with C but https://pkg.go.dev/cmd/cgo#hdr-C_references_to_Go explicitly states:

Go struct types are not supported; use a C struct type. 

So it may be completely at the decision of the implementer.

The caveats in https://go.dev/wiki/cgo#common-pitfalls also sound too tricky.

The pointer to the entire value stays alive while any pointer to one of its fields is alive 

That requires some cooperation from the GC. In the case where structs are reasonably small and they contain values rather than pointers this would probably hold.

Thanx for even further details (I am writing down the offset trick) but I'd prefer to not stray too far away from usual conventions.

Kind regards:

al_shopov

[Jason E. Aten]

Forgive me if this off base, as I'm still a little fuzzy on the exact constraints

of you problem... but, as stated, if you want

to associate additional optional behavior and state with any given response

that is constrained to the generated, just use

the responses's pointer (to its struct)  (if local an ephemeral is all you need, or a cryptographic hash of 

its contents, if you need to survive reboots) as the key into a separate map/sync.Map/

database. Standard Go does not move pointers; the GC is a non-moving collector

by intent to enable sane interfacing with C code, and does very well at avoiding

fragmentation (the motivation for compaction/copying GC) by using

size classes instead. Since you can rely on the pointer not moving, just

use it as a key to a map[*generated.Response]*EnrichedResponse to

lookup your additional state on either the caller or responder side.

[Alexander Shopov]

Now that I think about it again - I guess your suggestion could work for my case with some caveats:

1. There can be contention on the map - but as you point out there is sync.Map as well

2. The state should be removed from the map but that can also be arranged by deferring a delete operation

3. If contention is indeed a problem I can mitigate by sharding the map

4. If contention continues to be a problem perhaps weak pointers with this recipe https://github.com/golang/go/issues/67552#issuecomment-2195479919 may help.

It won't be pretty but hey - help beggars can't be choosers!

Kind regards:

al_shopov

You received this message because you are subscribed to a topic in the Google Groups "golang-nuts" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/golang-nuts/5gx3CCMfyJg/unsubscribe.
To unsubscribe from this group and all its topics, send an email to golang-nuts...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/golang-nuts/e823a08d-11c3-49d0-bd24-89eef57efc0an%40googlegroups.com.

[Robert Engels]

Why not just modify the code generator to add a user data pointer to the structure?

I can’t imagine something like that isn’t already there? Either the code generator should be easy to enhance or it should have a side car data field - anything else is hard to rationalize as even rudimentary design. 

On May 12, 2025, at 11:13 AM, Alexander Shopov <a...@kambanaria.org> wrote:



To view this discussion visit https://groups.google.com/d/msgid/golang-nuts/CAP6f5MmNXt-AuUPA4XGc1ZbbYqaMibtdzcxikx04KWNUpy21nA%40mail.gmail.com.

[Jason E. Aten]

Robert's suggestion is obviously the most direct and correct route if you can change the IDL of generated type. After all, code generation is there to simplify generating alot of code from a little definition.

I was assuming that one of constraints was not being able to modify the struct received. You should take that simpler route, if at all possible.

> If contention is indeed a problem I can mitigate by sharding the map

You are in luck. sync.Map already shards for you. It is very well engineered, and in my benchmarks, you will be very hard pressed to do better. Less than 50ns for 10% writes, and 12ns for 100% reads. Its documentation does it a bit of a disservice by discouraging its wider use, but it is awesome; one of the reasons that net/rpc itself was so performant. There are some studies here (with benchmark code you can run yourself to reproduce it: https://github.com/glycerine/uart), the most salient part is this benchmark, going from 100% writes at the top, to 100% reads at the bottom, in steps of 10%.

BenchmarkReadWriteSyncMap
BenchmarkReadWriteSyncMap/frac_0
BenchmarkReadWriteSyncMap/frac_0-8                       11585511        128.9 ns/op      111 B/op        5 allocs/op
BenchmarkReadWriteSyncMap/frac_1
BenchmarkReadWriteSyncMap/frac_1-8                       12977352        131.1 ns/op      101 B/op        4 allocs/op
BenchmarkReadWriteSyncMap/frac_2
BenchmarkReadWriteSyncMap/frac_2-8                       14838945        122.2 ns/op       90 B/op        4 allocs/op
BenchmarkReadWriteSyncMap/frac_3
BenchmarkReadWriteSyncMap/frac_3-8                       14907528        114.5 ns/op       80 B/op        3 allocs/op
BenchmarkReadWriteSyncMap/frac_4
BenchmarkReadWriteSyncMap/frac_4-8                       18078240        100.8 ns/op       70 B/op        3 allocs/op
BenchmarkReadWriteSyncMap/frac_5
BenchmarkReadWriteSyncMap/frac_5-8                       18791480         89.52 ns/op       59 B/op        3 allocs/op
BenchmarkReadWriteSyncMap/frac_6
BenchmarkReadWriteSyncMap/frac_6-8                       21172922         79.55 ns/op       49 B/op        2 allocs/op
BenchmarkReadWriteSyncMap/frac_7
BenchmarkReadWriteSyncMap/frac_7-8                       25711459         73.04 ns/op       38 B/op        2 allocs/op
BenchmarkReadWriteSyncMap/frac_8
BenchmarkReadWriteSyncMap/frac_8-8                       29925382         60.51 ns/op       28 B/op        1 allocs/op
BenchmarkReadWriteSyncMap/frac_9
BenchmarkReadWriteSyncMap/frac_9-8                       41599728         46.34 ns/op       18 B/op        1 allocs/op
BenchmarkReadWriteSyncMap/frac_10
BenchmarkReadWriteSyncMap/frac_10-8                     100000000         11.96 ns/op        8 B/op        1 allocs/op

[Robert Engels]

That’s interesting on the performance of sync.Map as the issues cited here are still open https://github.com/golang/go/issues/28938#issuecomment-441681208

On May 12, 2025, at 12:48 PM, Jason E. Aten <j.e....@gmail.com> wrote:

Robert's suggestion is obviously the most direct and correct route if you can change the IDL of generated type. After all, code generation is there to simplify generating alot of code from a little definition.

To view this discussion visit https://groups.google.com/d/msgid/golang-nuts/eeb31b6d-6253-4633-9c9b-006ef2df44e1n%40googlegroups.com.

[Jason E. Aten]

Interesting. Thanks, Robert, for the pointer to your tests in https://github.com/robaho/go-concurrency-test ; I've not tried to run them.  It seems there have been a dozen or so commits to sync.Map since 2018, and there is even an experimental hash-trie version mentioned below in the history. If you revisit it and find your concerns still hold, I would be quite interested to see how my simple benchmarks were fooled.

https://cs.opensource.google/go/go/+/refs/tags/go1.24.3:src/sync/map.go;l=40;bpv=1;bpt=0;drc=aecb8faa91e2b40e3651ee93c8e70635ed367677;dlc=53b2b64b649b26c7bb3397bec5d86d3b203eb015

[Jason E. Aten]

possibly very relevant --

https://github.com/golang/go/issues/70683

[Michael Knyszek]

On Monday, May 12, 2025 at 3:58:54 PM UTC-4 Jason E. Aten wrote:

possibly very relevant --

https://github.com/golang/go/issues/70683

On Monday, May 12, 2025 at 8:50:33 PM UTC+1 Jason E. Aten wrote:

Interesting. Thanks, Robert, for the pointer to your tests in https://github.com/robaho/go-concurrency-test ; I've not tried to run them.  It seems there have been a dozen or so commits to sync.Map since 2018, and there is even an experimental hash-trie version mentioned below in the history. If you revisit it and find your concerns still hold, I would be quite interested to see how my simple benchmarks were fooled.

https://cs.opensource.google/go/go/+/refs/tags/go1.24.3:src/sync/map.go;l=40;bpv=1;bpt=0;drc=aecb8faa91e2b40e3651ee93c8e70635ed367677;dlc=53b2b64b649b26c7bb3397bec5d86d3b203eb015

On Monday, May 12, 2025 at 8:03:49 PM UTC+1 Robert Engels wrote:

That’s interesting on the performance of sync.Map as the issues cited here are still open https://github.com/golang/go/issues/28938#issuecomment-441681208

Those issues should have been closed with https://github.com/golang/go/issues/70683. I just went ahead and did that with a final comment on each. sync.Map still isn't perfect but might be worth a second look now.