Filter the fields to return with find()

[EricBKK]

Hi,

We are using Mongodb from python with pymongo. Our database has about 50 millions records, so we need to optimize our queries to get fast response.

We can't find out how to make a query that will not return fields starting by "_".

Example:

"HTTrack" : {

"URL" : "http://myURL.html",

"Date" : "Mon, 25 Jun 2012 22:31:37 GMT",

"Archive" : "/home/frontware/websites/mypage.html",

                "_EE" : 2

},

"Imported" : {

"On" : ISODate("2012-06-26T07:55:40.820Z")

},

"Related to" : [

{

"Created on" : ISODate("2012-06-26T09:11:06.614Z"),

"_id" : NumberLong("-615146955922640935"),

"Created by" : "Phone Consolidation"

},

{

"Created on" : ISODate("2012-06-26T13:39:48.906Z"),

"_id" : NumberLong("-8580834954651581190"),

"Created by" : "Name Consolidation"

},

{

"Created on" : ISODate("2012-06-26T13:39:48.969Z"),

"_id" : NumberLong("-5378604234799856756"),

"Created by" : "Name Consolidation"

}

],

"_Source" : "test",

"Tel" : [

{

"Type" : "Fax",

"Number" : "+3355555555"

}

],

"_id" : NumberLong("-1365906524383645057")

}

You can see we have several fields name starting by "_".

We would like to execute a find that will return all fields of a record without all fields starting by "_"

Thanks for your suggestions.

Eric

[Robert Citek]

On Tue, Nov 13, 2012 at 12:49 AM, EricBKK <eric....@gmail.com> wrote:
> You can see we have several fields name starting by "_".
> We would like to execute a find that will return all fields of a record
> without all fields starting by "_"

Assuming the collection is named foo, is this what you are looking for?

> db.foo.find({},{_id:0, "HTTrack._EE":0, "Related to._id": 0, _Source:0}).pretty();
{

"HTTrack" : {
"URL" : "http://myURL.html",
"Date" : "Mon, 25 Jun 2012 22:31:37 GMT",
"Archive" : "/home/frontware/websites/mypage.html"
},

"Imported" : {
"On" : ISODate("2012-06-26T07:55:40.820Z")
},
"Related to" : [
{
"Created on" : ISODate("2012-06-26T09:11:06.614Z"),

"Created by" : "Phone Consolidation"
},
{
"Created on" : ISODate("2012-06-26T13:39:48.906Z"),

"Created by" : "Name Consolidation"
},
{
"Created on" : ISODate("2012-06-26T13:39:48.969Z"),
"Created by" : "Name Consolidation"
}
],

"Tel" : [
{
"Type" : "Fax",
"Number" : "+3355555555"
}
]
}

Regards,
- Robert

[EricBKK]

Hi,

Yes we know that. But assume we don't know the name of the fields before we retrieve.

What we try is to return all fields that do not start by "_".

Every record may have or not have fields name starting by "_". The programmer doesn't know the field name, he just need to retrieve all data for every field not starting by "_".

Thanks,

Eric

[Sam Helman]

Unfortunately this cannot be done at the moment.  Projecting on arbitrary field names is not currently a feature.  You may have to narrow it down as much as you can based on what fields you do know, and then do the rest of the processing client-side.

[Ronald Stalder]

you could, however, do the following:

(in javascript for mongo shell, port this to the language you're using)

var lim = { '_id': false };    // takes out the global "_id'

var struct = db.foo.findOne();

for ( var key in struct ) {

   // decide what to do with key, dive into Arrays

   // add to lim Obejct to filter

}

in order to get the lim p.ex. as follows { '_id':false, '_source':false, 'related to._id':false, .... }

then:

db.foo.find( {}, lim ).....

[Eric Fairon]

Ok it confirms what we found.

We now filter on client side, but it means we transfer a lot of data from server to client we have to remove on client.

It would be nice to be able to do something like:

db.foo.find({},{"$regex":{"^_":0}})

Anyway, mongodb is a great product.

Thanks.

On Wed, Nov 14, 2012 at 3:17 AM, Sam Helman <sam.h...@10gen.com> wrote:

Unfortunately this cannot be done at the moment.  Projecting on arbitrary field names is not currently a feature.  You may have to narrow it down as much as you can based on what fields you do know, and then do the rest of the processing client-side.

--
You received this message because you are subscribed to the Google
Groups "mongodb-user" group.
To post to this group, send email to mongod...@googlegroups.com
To unsubscribe from this group, send email to
mongodb-user...@googlegroups.com
See also the IRC channel -- freenode.net#mongodb

[Sam Helman]

Glad you found something that worked.  Making projections more powerful is definitely something that is on the MongoDB roadmap.