create new struct object from object passed as interface{}

[Sylvain]

Hi,
I defined a function to load all of an sqlite table into a struct.
I have working  functions that,
- given a struct, creates the table
- given a slice of struct objects, fills the data into the db
In order to load the data into a struct, I wrote the LoadDataFromDb function (given at the end of this message).
I pass as input a struct object as interface{}, and a pointer to an interface{} slice
I manage to get the data for each row using the reflect package, but cannot find a way to put the real data into the result slice instead of a pointer. So each time I iterate over a new row, all previous data in the slice gets the value of the new row.

I tried the following
- get the type of s and create a new object using clone:=new(reflect.TypeOf(s).Elem()) but get the following error:
reflect.TypeOf(s).Elem() is not a type
- define (actually, copypaste from another post in this mailing list) a clone function and try to get a new object for each iteration:
func Clone(source,dest interface{}) {
    buff := new(bytes.Buffer)
    enc := gob.NewEncoder(buff)
    dec := gob.NewDecoder(buff)
    enc.Encode(source)
    dec.Decode(dest)
}
But I have to define dest as an interface{}, so I cannot reflect on it after.

Is there a way to fill my results array with s's value instead of its memory address ?

Here is the content of my Sqlite Person table
"4654","1","Sylvain","32"
"42357","2","Vincent","31"

And here is the content of results after loading:
N° 0 :  2 Vincent 31 42357
N° 1 :  2 Vincent 31 42357

Here is my function:

func LoadDataFromDb(db *sqlite3.Database, s interface{}, results *[]interface{}) (err os.Error) {
    sql := fmt.Sprintf("SELECT * FROM %v;",
        StructName(s))
//get Records
    enr, err := db.Prepare(sql)
    if err != nil {
        fmt.Println("db Prepare error: %s", err)
        return
    }
    //iterate over statements from db
    cpt := 0
    for rs := enr.Step(); rs != nil; rs = enr.Step() {
        structure := reflect.Indirect(reflect.ValueOf(s))
        cpt++
        for a, b := range enr.Row() {
            champ := structure.FieldByName(enr.ColumnName(a))
            switch champ.Type().Kind() {
            case reflect.String:
                champ.SetString(b.(string))
            case reflect.Int, reflect.Int16, reflect.Int32, reflect.Int64:
                champ.SetInt(b.(int64))
            case reflect.Uint, reflect.Uint16, reflect.Uint32, reflect.Uint64:
                champ.SetUint(b.(uint64))
            case reflect.Float32, reflect.Float64:
                champ.SetFloat(b.(float64))
            default:
                fmt.Println(champ, "not dealt with yet")
            }
        }
        //DOES NOT WORK, APPENDS A POINTER SO AT THE END EACH ELEM OF THE ARRAY CONTAINS THE LAST ROW
        *results = append(*results, s)
        }
    }
    return
}

[Sylvain]

ok got it
Just in case someone is interrested in loading data into a "generic" struct from sqlite

I used  reflect.New() to create a pointer to a Value of the same type as the provided struct object
then reflect.Indirect() to get the value so thet the type switch would work
at the end, I juste append the reflect.Interface() to the new value. And now my []interface{}  pointer contains objects of the type I sent.

func LoadDataFromDb(db *sqlite3.Database, s interface{}, results *[]interface{}) (err os.Error) {
    sql := fmt.Sprintf("SELECT * FROM %v;",
        StructName(s))
//get Records
    enr, err := db.Prepare(sql)
    if err != nil {
        fmt.Println("db Prepare error: %s", err)
        return
    }
    //iterate over statements from db
    cpt := 0
    for rs := enr.Step(); rs != nil; rs = enr.Step() {

        newObjPtr := reflect.New(reflect.TypeOf(s).Elem())
        newObj := reflect.Indirect(newObjPtr)

        cpt++
        for a, b := range enr.Row() {
            champ := structure.FieldByName(enr.ColumnName(a))
            switch champ.Type().Kind() {
            case reflect.String:
                champ.SetString(b.(string))
            case reflect.Int, reflect.Int16, reflect.Int32, reflect.Int64:
                champ.SetInt(b.(int64))
            case reflect.Uint, reflect.Uint16, reflect.Uint32, reflect.Uint64:
                champ.SetUint(b.(uint64))
            case reflect.Float32, reflect.Float64:
                champ.SetFloat(b.(float64))
            default:
                fmt.Println(champ, "not dealt with yet")
            }
        }

        //works now - newObjPtr points to a new memory adress for each iteration
        *resultats = append(*resultats, newObjPtr.Interface())

        }
    }
    return
}

regards,

[Eleanor McHugh]

On 29 Jul 2011, at 10:53, Sylvain wrote:
> ok got it
> Just in case someone is interrested in loading data into a "generic" struct from sqlite
>
> I used reflect.New() to create a pointer to a Value of the same type as the provided struct object
> then reflect.Indirect() to get the value so thet the type switch would work
> at the end, I juste append the reflect.Interface() to the new value. And now my []interface{} pointer contains objects of the type I sent.

Nice. I've some maintenance to do on the gosqlite3 library over the next few days so I think I might try something similar to give us primitive ORM support. Who knows, perhaps another go-nut will build on that to give us an ActiveRecord equivalent...

Ellie

Eleanor McHugh
Games With Brains
http://feyeleanor.tel
----
if _, ok := reality.(Reasonable); !ok { panic(reality) }

[Sylvain]

I think you are refering to the github.com/kuroneko/sqlite3" library which I am using

I played with the gorm package first, but it had limitations like
- having to create the table manually from a struct
- using an INSERT for each struct object to save. So saving a struct slice of 1000 objects would require 1000 COMMIT

The second point is very important, from my tests every atomic INSERT takes at least 100ms because of the time taken the COMMIT.
By writting a function to do several INSERT inside a transaction, writing 50k+ records takes less than a few seconds.

[Eleanor McHugh]

Which is why I added transaction support into the package - I've been stung by that on projects in the past and didn't want to force the cost onto anyone else. Another trick that might work well depending on your needs is to do the writes to an in-memory database and then use the SQLite backup mechanism to bulk copy it to you database. I've not tested that out with a large workload so I'd love to have some feedback on that.