Can generics help me make my library safer?

362 views
Skip to first unread message

mi...@ubo.ro

unread,
Oct 3, 2021, 11:41:04 AM10/3/21
to golang-nuts
I have developed a library that depends very much on reflect package. It caches a specific type and return a function that encodes(does something with) with that kind /type of data. Think of defining database schema using types and generating functions to validate/update/insert data.

I reduced it to a basic example below. Can the generics feature from Go help me make it any safer?  Ideally I would like to make the program below not to compile due the errors of invalid params passed to PrintT instead  to  throw dynamically at runtime.
package main

import (
"errors"
"fmt"
"reflect"
)

type T struct {
F1 string
F2 int
}

type T2 struct {
F1 float32
F2 bool
}

var PrintT = Function(T{})

func main() {

if err := PrintT("one", 1); err != nil {
fmt.Printf("err %v", err)
}
if err := PrintT("one", 1, "another"); err != nil {
fmt.Printf("err %v", err)
}
if err := PrintT("one", "one"); err != nil {
fmt.Printf("err %v", err)
}

}

type ReturnFunc func(params ...interface{}) error

func Function(v interface{}) ReturnFunc {

var paramTypes []reflect.Type
tv := reflect.TypeOf(v)
for i := 0; i < tv.NumField(); i++ {
paramTypes = append(paramTypes, tv.Field(i).Type)
}
fn := func(param ...interface{}) error {
// validate input
if len(param) != len(paramTypes) {
return errors.New("invalid number of params passed")
}
for k, v := range param {
if reflect.TypeOf(v) != paramTypes[k] {
return errors.New("invalid type passed")
}
}
// do something with the params
fmt.Println(param...)
return nil
}
return fn
}


Axel Wagner

unread,
Oct 3, 2021, 12:27:20 PM10/3/21
to mi...@ubo.ro, golang-nuts
I don't think so. There is no way to decompose structs as part of constraints, so you can't generalize over them easily. There is also no way to generalize over an arbitrary number of types, which would be necessary to write a signature of `ReturnFunc`.

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/2bc1a657-c4e0-42a1-82b0-8bc605c429a2n%40googlegroups.com.

Ian Lance Taylor

unread,
Oct 5, 2021, 7:14:20 PM10/5/21
to mi...@ubo.ro, golang-nuts
On Sun, Oct 3, 2021 at 8:41 AM mi...@ubo.ro <mi...@ubo.ro> wrote:
>
> I have developed a library that depends very much on reflect package. It caches a specific type and return a function that encodes(does something with) with that kind /type of data. Think of defining database schema using types and generating functions to validate/update/insert data.
>
> I reduced it to a basic example below. Can the generics feature from Go help me make it any safer? Ideally I would like to make the program below not to compile due the errors of invalid params passed to PrintT instead to throw dynamically at runtime.

I don't understand what your program is trying to do, but the current
generics proposal does not support variadic generic parameters, so I
don't think it's going to help you.

Ian

mi...@ubo.ro

unread,
Oct 6, 2021, 4:21:06 AM10/6/21
to golang-nuts
Hi Ian ,

I've modified the example towards a more specific use case. The main idea in the example below  is to make code related to database operations(i.e SELECT queries) safer and  easier to read. A  kind of json.Unmarshal/Marshal for databases, with validation (type checking, param numbers etc) to avoid a class of bugs/errors such invalid param types/numbers passed, invalid queries, invalid resource type to scan into etc. 
 
Currently the function returned by Select  throws the validation errors at runtime (i.e. invalid param type passed etc). It would be great to have that class of errors checked at compile type.
 
The only way I could achieve that kind of  type checking was  through code generation. I already built a tool to generate functions with the proper param types but it seems that code generation introduces a lot of friction to the point that I stopped using it.

My hope is that one day a Go feature (i.e. a version of Generics) could help the function returned by func Select be type checked at compile time.


package main

import (
"database/sql"
"errors"
"fmt"
"reflect"
)

var sqlDB *sql.DB

type Flower struct {
Color  string
Size   int
Weight int
}

type T struct{}

var FlowerByColor = Select(" * FROM tablex WHERE Color=$ LIMIT 1", reflect.TypeOf(Flower{}))

func main() {
// Select a flower based on its color from database

// invalid resource type; resource of type Flower is expected
       // I would like this to not compile as I pass an invalid resource type T instead of type Flower
var color string
if err := FlowerByColor(T{}, &color); err != nil {
fmt.Printf("err %v\n", err)
}
// invalid param; type string (Folower.Color) type is expected
var colorInvalid int
// I would like this to not compile as I pass an invalid color type
if err := FlowerByColor(Flower{}, colorInvalid); err != nil {
fmt.Printf("err %v\n", err)
}

// correct query
color = "red"
if err := FlowerByColor(Flower{}, color); err != nil {
fmt.Printf("err %v\n", err)
}
// Note: a proper SelectFunc would actually accept only pointers to Flower so that
// it can unmarshal data into the resource as below. For brevity I omitted handling
// pointers in SelectFunc
resource := new(Flower)
if err := FlowerByColor(resource, color); err != nil {
fmt.Printf("err %v\n", err)
}
// so something with the data from realVal
fmt.Printf("our  flower of color %v has a size of %v", color, realVal.Size)

}

// SelectFunc receives a resource and the query params
type SelectFunc func(resource interface{}, params ...interface{}) error

// select receives a sql query and the type that represents
//the sql table from database.
// Returns a function that executes the sql query with the matching params from tv.
func Select(q string, tv reflect.Type) SelectFunc {
paramTypes, err := parseQuery(q, tv)
if err != nil {
panic("invalid query")
}
return  func(resource interface{}, param ...interface{}) error {
// validate input
// resource must match the resource type
if reflect.TypeOf(resource) != tv {
return errors.New("invalid resource type")
}
if len(param) != len(paramTypes) {
return errors.New("invalid number of params passed")
}
for k, v := range param {
if reflect.TypeOf(v) != paramTypes[k] {
return errors.New("invalid argv type passed")
}
}
// do a select database query
resourceFields := fieldsFromResource(reflect.ValueOf(resource))
if err := sqlDB.QueryRow("SELECT "+q, param...).Scan(resourceFields...); err != nil {
return err
}
return nil
}
        
 
}

// parseQuery parses query and the resource t.
// returns the types selected in the query
func parseQuery(query string, t reflect.Type) ([]reflect.Type, error) {
// skip parsing for brevity
return []reflect.Type{t.Field(0).Type}, nil

}

func fieldsFromResource(v reflect.Value) []interface{} {
// skip type fields looping for brevity
return []interface{}{
v.Field(0).Addr().Interface(),
v.Field(1).Addr().Interface(),
v.Field(2).Addr().Interface(),
}
}


roger peppe

unread,
Oct 6, 2021, 7:35:58 AM10/6/21
to mi...@ubo.ro, golang-nuts
On Wed, 6 Oct 2021 at 09:21, mi...@ubo.ro <mi...@ubo.ro> wrote:
Hi Ian ,

I've modified the example towards a more specific use case. The main idea in the example below  is to make code related to database operations(i.e SELECT queries) safer and  easier to read. A  kind of json.Unmarshal/Marshal for databases, with validation (type checking, param numbers etc) to avoid a class of bugs/errors such invalid param types/numbers passed, invalid queries, invalid resource type to scan into etc. 
 
Currently the function returned by Select  throws the validation errors at runtime (i.e. invalid param type passed etc). It would be great to have that class of errors checked at compile type.
 
The only way I could achieve that kind of  type checking was  through code generation. I already built a tool to generate functions with the proper param types but it seems that code generation introduces a lot of friction to the point that I stopped using it.

My hope is that one day a Go feature (i.e. a version of Generics) could help the function returned by func Select be type checked at compile time.

 
If you change your API slightly to use a single selector argument of struct type, you could do something like this:

In summary:

type Flower struct {
    Color  string
    Size   int
    Weight int
}

type ByColor struct {
    Color string
}

var FlowerByColor = Select[Flower, ByColor]("* FROM tablex WHERE Color=$ LIMIT 1")

type SelectFunc[Resource, Args any] func(args Args) (Resource, error)

// Select returns a function that executes the given SQL query,
// expecting results to contain fields matching Resource and
// using fields in Args to select rows.
//
// Both Resource and Args must be struct types; All the fields
// in Args must have matching fields in Resource.
func Select[Resource, Args any](q string) SelectFunc[Resource, Args] {


That is, we can use a combination of reflection and generics. The generics keep the code type-safe. The reflection part does the more specific type checking once only at init time, something I like to think of as "almost statically typed". It's a powerful pattern in my view.

  cheers,
    rog.

Robert Engels

unread,
Oct 6, 2021, 7:45:01 AM10/6/21
to roger peppe, mi...@ubo.ro, golang-nuts
Personally, I think this is overkill (the entire concept not the rog solution)

Even with static checking there is no way to ensure that tablex has the needed fields. Even if you could check this at compile time - it might be different at runtime. 

I don’t think the juice is worth the squeeze. 

Either use an ORM that generates the sql, or create DAOs and rely on test cases to ensure the code is correct.

Far simpler and more robust in my opinion. 

On Oct 6, 2021, at 6:35 AM, roger peppe <rogp...@gmail.com> wrote:


--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.

roger peppe

unread,
Oct 6, 2021, 8:04:23 AM10/6/21
to Robert Engels, mi...@ubo.ro, golang-nuts
On Wed, 6 Oct 2021 at 12:44, Robert Engels <ren...@ix.netcom.com> wrote:
Personally, I think this is overkill (the entire concept not the rog solution)

I tend to agree, but the bait was too irresistible :)

I do think that using reflection in combination with generics the way I showed can be really useful. This is a nice example of how the technique can be applied, even if the problem isn't actually a good one to solve.

Brian Candler

unread,
Oct 6, 2021, 9:06:39 AM10/6/21
to golang-nuts
FWIW, I like the idea of being able to write direct SQL and still have some static type checking.  ORMs are OK for simple "get" and "put", but I have been bitten so many times where I *know* the exact SQL I want for a particular query, but the ORM makes it so damned hard to construct it their way.

Robert Engels

unread,
Oct 6, 2021, 9:17:43 AM10/6/21
to Brian Candler, golang-nuts
I think you can only do that if you make sql parsing a first class language feature - or you need to construct the query using a syntax tree of clauses which is a PITA. Sometimes a hybrid approach - not a full orm - but an sql helper works best so 

sql.Query(table name, field list, where clauses, order by clauses) can work but for complex sql like joins it becomes unwieldy. 

After years of doing both, I’ve settled on that using DAOs creates the simplest and highest performing code. 

On Oct 6, 2021, at 8:07 AM, Brian Candler <b.ca...@pobox.com> wrote:

FWIW, I like the idea of being able to write direct SQL and still have some static type checking.  ORMs are OK for simple "get" and "put", but I have been bitten so many times where I *know* the exact SQL I want for a particular query, but the ORM makes it so damned hard to construct it their way.

mi...@ubo.ro

unread,
Oct 8, 2021, 10:44:15 AM10/8/21
to golang-nuts
 I'm using the library with a nosql. I provided the sql example b/c it's part of the std library.
I like rog's solution(especially on the generic Resource type) but to reduce friction and make it more idiomatic I would need to be able to define the select arguments in the return func (after the query string is parsed) instead to define them manually in a struct. 
  Defining the filters/params in a struct requires extra effort and makes the code more verbose instead to rely on the query parser. Also a query has just 1-2 filters and I don't find structs with 1-2 fields(i.e like `ByColor` struct) very idiomatic/nice to use instead of function parameters.
  I assume that as Ian said that kind of "variadic generic types" is not possible in Go/2 generics. 

roger peppe

unread,
Oct 9, 2021, 4:51:59 AM10/9/21
to mi...@ubo.ro, golang-nuts
On Fri, 8 Oct 2021 at 15:44, mi...@ubo.ro <mi...@ubo.ro> wrote:
 I'm using the library with a nosql. I provided the sql example b/c it's part of the std library.
I like rog's solution(especially on the generic Resource type) but to reduce friction and make it more idiomatic I would need to be able to define the select arguments in the return func (after the query string is parsed) instead to define them manually in a struct. 
  Defining the filters/params in a struct requires extra effort and makes the code more verbose instead to rely on the query parser. Also a query has just 1-2 filters and I don't find structs with 1-2 fields(i.e like `ByColor` struct) very idiomatic/nice to use instead of function parameters.

The only way to do that kind of thing currently AFAIK is to explicitly enumerate variants for different numbers of arguments.
You could layer on top of the existing more generic struct-based code to do that.
For example:

type SelectFunc[Resource, Args any] func(args Args) (Resource, error)
type Select1Func[Resource, Arg0 any] func(arg0 Arg0) (Resource, error)
type Select2Func[Resource, Arg0, Arg1 any] func(arg0 Arg0, arg1 Arg1) (Resource, error)
// etc

func Select1[Resource, Arg0 any](q string) Select1Func[Resource, Arg0] {
        f := Select[Resource, struct{ A Arg0 }](q)
        return func(arg0 Arg0) (Resource, error) {
                return f(struct{ A Arg0 }{arg0})
        }
}

func Select2[Resource, Arg0, Arg1 any](q string) Select2Func[Resource, Arg0, Arg1] {
        f := Select[Resource, struct {
                A0 Arg0
                A1 Arg1
        }](q)
        return func(a0 Arg0, a1 Arg1) (Resource, error) {
                return f(struct {
                        A0 Arg0
                        A1 Arg1
                }{a0, a1})
        }
}


Reply all
Reply to author
Forward
0 new messages