Creating a generic query function

[dpapathanasiou]

Hello,

I'm new to Go, but experienced in using mongodb with both python and
node.js.

I tried creating a generic query function where I don't know the type
of struct representing the query result in advance:

func DoQuery(databaseName string, collection string, query
interface{}) (result []interface{}) {
c, err := mgo.Dial("localhost")
if err != nil {
panic(err)
}
defer c.Close()
err := c.DB(databaseName).C(collection).Find(query).All(&result)
if err != nil {
panic(err)
}
return result
}

This compiles, and when I run it like this:

for _, person := range DoQuery(bson.M{"lastName": "Jones"}) {
fmt.Println(person)
}

I don't get back a Person struct as I expect (Person is a simple
struct, consisting of two strings, FirstName and LastName), but a map
object, like this, for each person in the result set:

map[_id:ObjectIdHex("502fbbd6fec1300be858767e") FirstName:Alex
LastName:Jones]

If I try to access person.LastName or person.FirstName within the loop
above, the program won't compile, and I get this kind of error:

person.FirstName undefined (type interface {} has no field or method
FirstName)

After reading the section on reflection (http://golang.org/doc/
articles/laws_of_reflection.html), I tried rewriting the loop this
way:

for _, person := range DoQuery(bson.M{"lastName": "Jones"}) {
p := reflect.ValueOf(&person).Elem()
for i := 0; i < p.NumField(); i++ {
fmt.Print(p.Field(i))
}
fmt.Println()
}

While that compiled, it gave me a runtime panic:

panic: result argument must be a slice address

And when I tried to take a slice, I started getting compiler errors:

invalid operation: person[x] (index of type interface {})

So what am I not getting about handling arrays of interface{}?

[Gustavo Niemeyer]

Hi,

> I tried creating a generic query function where I don't know the type
> of struct representing the query result in advance:

(...)

> func DoQuery(databaseName string, collection string, query
> interface{}) (result []interface{}) {

(...)
> err := c.DB(databaseName).C(collection).Find(query).All(&result)
(...)

> I don't get back a Person struct as I expect (Person is a simple
> struct, consisting of two strings, FirstName and LastName), but a map
> object, like this, for each person in the result set:

When you write a struct onto the database, its exported fields are
serialized by the rules defined by the mgo/bson package and become a
normal document into the database. When you load a document back from
the database, the inverse procedure happens, and the fields from the
document are set back onto the fields of the struct value, or the keys
and values of a map.

There's no magic that would turn an interface{} into a Person. If you
want a Person, the value you pass to the mgo loading functions must be
a Person value.

Have a look at the example in the web page for a glimpse on how it works:

http://labix.org/mgo

gustavo @ http://niemeyer.net

[dpapathanasiou]

Hi Gustavo,

On Sunday, August 26, 2012 8:57:50 PM UTC-4, Gustavo Niemeyer wrote:

When you write a struct onto the database, its exported fields are
serialized by the rules defined by the mgo/bson package and become a
normal document into the database. When you load a document back from
the database, the inverse procedure happens, and the fields from the
document are set back onto the fields of the struct value, or the keys
and values of a map.

Thank you for clarifying that. 

There's no magic that would turn an interface{} into a Person. If you
want a Person, the value you pass to the mgo loading functions must be
a Person value.

Ok, I've accepted that as a constraint I have to deal with.

I've come up with the following, which is *almost* want I want:

var ( 

      mgoSession     *mgo.Session 

      databaseName = "mydb" 

) 

func getSession () *mgo.Session {

    if mgoSession == nil { 

        var err error 

        mgoSession, err = mgo.Dial("localhost") 

        if err != nil { 

             panic(err)

        } 

      } 

      return mgoSession.Clone() 

} 

func SearchPeople(query interface{}) (result []Person) { 

         s := getSession() 

         defer s.Close() 

         err := s.DB(databaseName).C("person").Find(query).All(&result)

         if err != nil { 

              panic(err) 

         }

         return result

}

So it's reasonably flexible in that I can pass any query for the person collection to SearchPeople(), and it'll give me back a list of Person objects (the getSession() function was inspired by a post by Alex Luya: https://groups.google.com/d/msg/mgo-users/oULcrwOB8LE/KWcJAx6fHv4J which is helpful, since I don't want to repeat the call to mgo.Dial in every collection query function).

However, I have to have one of these functions for every different collection that I want to query, and except for the return type and line which executes the query and puts the results into &result, I am repeating the same code block over and over again.

I'm not sure how to do this in Go, but in Python, I could just pass a function into a wrapper class which takes care of establishing the connection to the db, and all that passed function has to do is structure the query (do a Find(), Save(), Remove(), etc.):

def with_collection (collection_name, fn):

    """Apply this fn for this collection and return the result"""

    result = None

    try:

        connection = pymongo.Connection('localhost', 27017)

        db = connection['mydb']

        collection = db[collection_name]

        result = fn(collection)

    except pymongo.errors.OperationFailure:

        pass

    return result

The nice thing about this is that each function passed to with_connection() is pretty simple, a one-liner usually, that does the exact thing I need on a specific collection, w/o repeating the try ... except block around establishing the connection.

It seems that in using mgo there's no way around that repetition, but perhaps I do not understand Go well wnough to make the transition in my code.

[Gustavo Niemeyer]

> It seems that in using mgo there's no way around that repetition, but
> perhaps I do not understand Go well wnough to make the transition in my
> code.

It's not clear to me what you're trying to avoid repeating. It's a
single line of code:

err := collection.Find(query).All(&result)

Besides that, what your wrapper does is to establish a new connection
every time it is called, and drop errors on the floor. Both of these
sound like unwise design choices.


gustavo @ http://niemeyer.net

[dpapathanasiou]

On Tuesday, August 28, 2012 1:02:50 PM UTC-4, Gustavo Niemeyer wrote:

> It seems that in using mgo there's no way around that repetition, but
> perhaps I do not understand Go well wnough to make the transition in my
> code.

It's not clear to me what you're trying to avoid repeating. It's a
single line of code:

    err := collection.Find(query).All(&result)

Right, it's a single line of code, but b/c the return type is different for each collection, I'll have a block like this for every collection I'd want to query, right?

func SearchCollection (query interface{}) (result []CollectionStruct) {    

        session, err := mgo.Dial("localhost")

        if err != nil {

                panic(err)

        }

        defer session.Close()

        err := session.DB(databaseName).C(collection).Find(query).All(&result)

        if err != nil { 

             panic(err) 

        }

        return result

}

Worse, if I need to do a count or add limit/offset parameters to the query, then I'd need different versions of this function, even for the same collection.

 

Besides that, what your wrapper does is to establish a new connection
every time it is called, and drop errors on the floor. Both of these
sound like unwise design choices.

The python example could be better, I agree, but the important thing about it is the concept of passing a function that does just the Find()/Save()/Remove() etc. on the given collection, w/o having to repeat all the overhead needed.

I'm studying how higher order functions work in Go, and so I think there is a better solution to this, I just need to improve my ability to write Go code. 

gustavo @ http://niemeyer.net

[Gustavo Niemeyer]

On Tue, Aug 28, 2012 at 2:20 PM, dpapathanasiou
<denis.pap...@gmail.com> wrote:
>>
>> err := collection.Find(query).All(&result)
>
> Right, it's a single line of code, but b/c the return type is different for
> each collection, I'll have a block like this for every collection I'd want
> to query, right?

No.. just pass the expected result as a parameter, like the methods in
mgo themselves do.

> if err != nil {
> panic(err)
> }

I know this is just an example, but as a hopefully useful advice,
don't do that in the real program. Ignoring the error is bad, and so
is blowing the whole application up because an error was seen. Network
issues are everyday issues rather than "panic" situations.


gustavo @ http://niemeyer.net

[dpapathanasiou]

On Tuesday, August 28, 2012 1:47:00 PM UTC-4, Gustavo Niemeyer wrote:

No.. just pass the expected result as a parameter, like the methods in
mgo themselves do.

Ok, this is where I'm getting stuck.

Suppose I have three collections (person, place, thing) and in my Go code, I've defined a struct for each (Person, Place, Thing).

To query those collections, I need logic like this (here I'm just showing the core statement, w/o all the calls to mgo.Dial() and close, etc.):

err := s.DB(databaseName).C("person").Find(query).All(&result) // result is []Person

err := s.DB(databaseName).C("place").Find(query).All(&result)   // result is []Place

err := s.DB(databaseName).C("thing").Find(query).All(&result)    // result is []Thing 

 

I suppose that like the query variable, I can replace explicitly stating the collection with a variable (collection string), so the core statement reduces to just:

err := s.DB(databaseName).C(collection).Find(query).All(&result)

but since the result type is different in each statement, I'm not sure how I can do it w/o repeating each of the func blocks posted earlier, with the only difference being the return type definition.

I realize this is more a question about Go in general than mgo in particular, but even after reading the article on higher-order functions on the wiki, I'm not sure how it would apply in this case (none of the variations I tried compiled).

>        if err != nil {
>             panic(err)
>        }

I know this is just an example, but as a hopefully useful advice,
don't do that in the real program. Ignoring the error is bad, and so
is blowing the whole application up because an error was seen. Network
issues are everyday issues rather than "panic" situations.

Right, I was using panic(err) in the same way you did in your example, i.e., as a placeholder to do something else here (though, of course, if we really want to stamp out this practice, we should stop using it in our examples ;) this is not only true of what I posted here but also of several articles I found on the wiki).

 

gustavo @ http://niemeyer.net

[Gustavo Niemeyer]

On Wed, Aug 29, 2012 at 3:47 PM, dpapathanasiou
<denis.pap...@gmail.com> wrote:
> but since the result type is different in each statement, I'm not sure how I can do it w/o repeating

func yourQuery(c *mgo.Collection, result interface{}) error


gustavo @ http://niemeyer.net

[dpapathanasiou]

Thanks for all the responses.

Perhaps I'm doing something wrong, or haven't quite understood it yet, but I simply cannot get this to work.

On Wednesday, August 29, 2012 2:55:25 PM UTC-4, Gustavo Niemeyer wrote:

func yourQuery(c *mgo.Collection, result interface{}) error

The first time I tried this, I got this runtime error:

panic: result argument must be a slice address

So I thought you must have meant this:

func yourQuery(c *mgo.Collection, result []interface{}) error

But then the code which calls yourQuery() doesn't compile:

./mgo-test.go:24: cannot use result (type []Person) as type []interface {} in function argument

Which means if I have to call yourQuery() with (result []interface{}) instead of (result []Person) then I've come full circle back to my original question.

gustavo @ http://niemeyer.net

[Gustavo Niemeyer]

On Wed, Aug 29, 2012 at 7:01 PM, dpapathanasiou
<denis.pap...@gmail.com> wrote:
>> func yourQuery(c *mgo.Collection, result interface{}) error
>
> The first time I tried this, I got this runtime error:
>
> panic: result argument must be a slice address

Yep, it must be an address.

You can find introductory documentation about Go here:

http://golang.org/doc/


gustavo @ http://niemeyer.net

[dpapathanasiou]

No need to be sarcastic

On Aug 29, 6:16 pm, Gustavo Niemeyer <gust...@niemeyer.net> wrote:
> On Wed, Aug 29, 2012 at 7:01 PM, dpapathanasiou
>

[Patrick-Ranjit D. Madsen]

I Think hus point is that you are making some very basic mistakes that
could easily be caught if you sat down and read the language
documentation. This is not an mgo-driver issue, you are having Go
issues.

Hilsen

Patrick-Ranjit D. Madsen

Sent from my mobile. Please excuse any and all typos, errors,
formatting, lack of punctuation or short responses.

[Gustavo Niemeyer]

On Wed, Aug 29, 2012 at 7:22 PM, dpapathanasiou
<denis.pap...@gmail.com> wrote:
> No need to be sarcastic

I haven't been sarcastic at all. I've explained the exact solution for
the problem you're having multiple times, and the errors messages
you're getting are also properly telling you what to do, but you're
lacking some basic knowledge that the stock documentation will help
you with. Feel free to ask other mgo-related questions once you get
past that point.


gustavo @ http://niemeyer.net

[dpapathanasiou]

Well, my mistake in that I copied the func he provided w/o noticing the typo.

It still doesn't answer my original question, though, which may indeed be more about Go than mgo.

[dpapathanasiou]

Hi again,

On Aug 29, 2:55 pm, Gustavo Niemeyer <gust...@niemeyer.net> wrote:
> On Wed, Aug 29, 2012 at 3:47 PM, dpapathanasiou
>

> <denis.papathanas...@gmail.com> wrote:
> > but since the result type is different in each statement, I'm not sure how I can do it w/o repeating
>
> func yourQuery(c *mgo.Collection, result interface{}) error

I should thank you for this; it wasn't quite what I needed, but it got
me thinking about the problem the right way.

For the record/FWIW, here's how I answered my original question (I see
that Alex Luya has been asking for something similar --
http://groups.google.com/group/mgo-users/msg/7e1f8488c5188c77?hl=en --
so perhaps this will be of use to him, or other people who search this
mailing list later).

I started by defining an all-purpose query function, using a higher-
order function:

func Search(database string, collection string, s
func(*mgo.Collection) error) error {

session, err := mgo.Dial("localhost")

defer session.Close()
if err != nil {
panic(err) // no, not really
}

c := session.DB(database).C(collection)
return s(c)
}

With this in place, I can create simpler functions for the specific
collection struct, and always be sure that I get a slice of the right
type of struct in return, e.g.:

func SearchPeople(query interface{}, result *[]Person) func(c
*mgo.Collection) error {
return func(c *mgo.Collection) error {
return c.Find(query).All(result)
}
}

and calling this from main() or wherever is just:

var result []Person
err := Search("mydb", "person", SearchPeople(bson.M{"lastName":
"Jones"}, &result))
if err != nil {
panic(err) // don't get too excited
}

The critical thing being I can iterate through the result slice and
each one is a Person struct that I can manipulate directly:

for _, p := range result {
fmt.Println(p.FirstName) // or do whatever else
}

So anyway, this *does* answer my original question, and I'm glad for
all your responses.

I'm sorry we ended the exchange the way we did yesterday; I don't know
if I just wasn't making myself clear before, or why we wound up
talking past each other at some point.