Received: by 10.58.2.40 with SMTP id 8mr883433ver.14.1346285289876;
Wed, 29 Aug 2012 17:08:09 -0700 (PDT)
X-BeenThere: mongodb-csharp@googlegroups.com
Received: by 10.220.223.69 with SMTP id ij5ls1947804vcb.6.gmail; Wed, 29 Aug
2012 17:08:09 -0700 (PDT)
Received: by 10.52.34.115 with SMTP id y19mr647736vdi.1.1346285289229;
Wed, 29 Aug 2012 17:08:09 -0700 (PDT)
Date: Wed, 29 Aug 2012 17:08:08 -0700 (PDT)
From: craiggwilson <craiggwil...@gmail.com>
To: mongodb-csharp@googlegroups.com
Message-Id: <70378282-dc4f-4e6f-b6af-a98cc6b2df02@googlegroups.com>
In-Reply-To: <bb4a88de-e4e8-46d6-8887-58a460aa986d@googlegroups.com>
References: <bb4a88de-e4e8-46d6-8887-58a460aa986d@googlegroups.com>
Subject: Re: Count is terribly slow!
MIME-Version: 1.0
Content-Type: multipart/mixed;
boundary="----=_Part_200_1665643.1346285288642"
------=_Part_200_1665643.1346285288642
Content-Type: multipart/alternative;
boundary="----=_Part_201_10066151.1346285288642"
------=_Part_201_10066151.1346285288642
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
Can you do an explain on your query? (instead of running it). I'd image
that you are missing an index or your query is not hitting the index you
think it is.
Second of all: Your Find(...).AsQueryable() isn't doing what you think it
is. Those skips, selects, and orderbys are not getting run on the server,
but rather client side. It feels immediate because the server immediately
start returning values and then your client side starts processing them.
I'm surprised you are seeing results so fast because to perform the
OrderBy(...) it has to load all 12 million documents into memory.
Also, as an FYI, doing paging the way you are doing it is not recommended
for large data sets. Skip will run the query and the move the specified
number of records (the skip count) and then start returning results. It is
much better to keep track of the last document's unique sort key and then
send in a $gt query with it. It enables the query engine to jump
immediately to the proper record.
On Wednesday, August 29, 2012 12:05:19 PM UTC-5, Dustin Blair wrote:
>
>
>
> Using C# driver 1.5 with MongoDB v 2.2
> ~12 million rows of data in Mongo
>
> the mongo query:
> IMongoQuery query = new QueryDocument();
>
> query = Query.GTE("InsertUTC", msgReceivedDate);
> query = Query.And(query, Query.NE("Header.AssetSerialNumber",
> BsonNull.Value));
> query = Query.And(query, Query.EQ("Header.Dealer",
> (int)DealerNetworkEnum.SomeDealer));
>
>
> *When trying to get a simple COUNT using this query*:
> var result = MongoCollection.Count(query);
> return (int)result;
>
> *it takes almost 3 minutes!*
>
> *by itself, the actual query returns data instantly*!
> var result =
> _mongoStorage.MongoCollection.Find(GetMongoQuery(requestDateTime)).AsQueryable()
> .Skip((page - 1) * pageSize).Take(pageSize)
> .Select(t => new DeviceMessage { Header = t.Header, Message =
> t.DevicePacket })
> .OrderBy(f => f.Header.InsertUTC).ToList();
>
> What black magic is needed to return a count quickly??
>
> THANK YOU!
> Dustin
>
>
>
------=_Part_201_10066151.1346285288642
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: quoted-printable
Can you do an explain on your query? (instead of running it). I=
'd image that you are missing an index or your query is not hitting the ind=
ex you think it is.<div><br></div><div>Second of all: Your Find(...).AsQuer=
yable() isn't doing what you think it is. Those skips, selects, and o=
rderbys are not getting run on the server, but rather client side. It=
feels immediate because the server immediately start returning values and =
then your client side starts processing them. I'm surprised you are s=
eeing results so fast because to perform the OrderBy(...) it has to load al=
l 12 million documents into memory.</div><div><br></div><div>Also, as an FY=
I, doing paging the way you are doing it is not recommended for large data =
sets. Skip will run the query and the move the specified number of re=
cords (the skip count) and then start returning results. It is much b=
etter to keep track of the last document's unique sort key and then send in=
a $gt query with it. It enables the query engine to jump immediately=
to the proper record.<br><br>On Wednesday, August 29, 2012 12:05:19 PM UTC=
-5, Dustin Blair wrote:<blockquote class=3D"gmail_quote" style=3D"margin: 0=
;margin-left: 0.8ex;border-left: 1px #ccc solid;padding-left: 1ex;"><br><di=
v><br></div><div>Using C# driver 1.5 with MongoDB v 2.2</div><div>~12 milli=
on rows of data in Mongo</div><div><br></div><div>the mongo query:</div><di=
v><div> IMongoQuery query =3D new QueryDocument(=
);</div><div><br></div><div> query =3D Query.GTE=
("InsertUTC", msgReceivedDate);</div><div> query=
=3D Query.And(query, Query.NE("Header.<wbr>AssetSerialNumber", BsonNull.Va=
lue));</div><div> query =3D Query.And(query, Que=
ry.EQ("Header.Dealer", (int)DealerNetworkEnum.<wbr>SomeDealer));</div></div=
><div><br></div><div><br></div><div><b>When trying to get a simple COUNT us=
ing this query</b>:</div><div><div>var result =3D MongoCollection.Count(que=
ry);</div><div>return (int)result;</div></div><div><br></div><div><b><i><u>=
it takes almost 3 minutes!</u></i></b></div><div><br></div><div><b>by itsel=
f, the actual query returns data instantly</b>!</div><div><div>  =
; var result =3D _mongoStorage.MongoCollection.<wbr>Find(GetM=
ongoQuery(<wbr>requestDateTime)).AsQueryable(<wbr>)</div><div> =
.Skip((page - 1) * pageSize).Take(pageSize)</div><div=
> .Select(t =3D> new DeviceMessage { H=
eader =3D t.Header, Message =3D t.DevicePacket })</div><div> &=
nbsp; .OrderBy(f =3D> f.Header.InsertUTC).ToList();</div><=
/div><div><br></div><div>What black magic is needed to return a count quick=
ly??</div><div><br></div><div>THANK YOU!</div><div>Dustin</div><div><br></d=
iv><div><br></div></blockquote></div>
------=_Part_201_10066151.1346285288642--
------=_Part_200_1665643.1346285288642--
Message from discussion Count is terribly slow!
| Create a group - Google Groups - Google Home - Terms of Service - Privacy Policy |
| ©2013 Google |