Received: by 10.224.189.16 with SMTP id dc16mr10091809qab.0.1352908826459;
Wed, 14 Nov 2012 08:00:26 -0800 (PST)
X-BeenThere: mongodb-user@googlegroups.com
Received: by 10.224.52.141 with SMTP id i13ls1153622qag.7.gmail; Wed, 14 Nov
2012 07:59:58 -0800 (PST)
Received: by 10.224.207.66 with SMTP id fx2mr10077260qab.7.1352908798884;
Wed, 14 Nov 2012 07:59:58 -0800 (PST)
Received: by 10.224.173.148 with SMTP id p20msqaz;
Wed, 14 Nov 2012 07:58:28 -0800 (PST)
Received: by 10.52.76.136 with SMTP id k8mr334104vdw.13.1352908708600;
Wed, 14 Nov 2012 07:58:28 -0800 (PST)
Date: Wed, 14 Nov 2012 07:58:28 -0800 (PST)
From: Alexander Peiniger <alex1...@gmail.com>
To: mongodb-user@googlegroups.com
Message-Id: <ec550dc5-2b58-4a45-bcbc-f2501cedd7ff@googlegroups.com>
In-Reply-To: <AANLkTikRNc9=ZKvUYs9ibF7jS-pJL86nZmnYFSHRdvkk@mail.gmail.com>
References: <711d031a-468d-40af-a871-a35d1dd2631e@v6g2000prd.googlegroups.com>
<AANLkTin2EER5d+d8amTBxUX12yu9L+nOEhHP8v_=cvwo@mail.gmail.com>
<c6243043-bae3-4391-947d-44d23741a91d@j8g2000yqd.googlegroups.com>
<AANLkTikRNc9=ZKvUYs9ibF7jS-pJL86nZmnYFSHRdvkk@mail.gmail.com>
Subject: Re: [mongodb-user] Re: priority queue with mongodb
MIME-Version: 1.0
Content-Type: multipart/mixed;
boundary="----=_Part_357_13748121.1352908708212"
------=_Part_357_13748121.1352908708212
Content-Type: multipart/alternative;
boundary="----=_Part_358_8861506.1352908708212"
------=_Part_358_8861506.1352908708212
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
Sounds great. Could you give a short explanation of your schema and indices
for this?
Alex
On Tuesday, August 3, 2010 7:43:37 PM UTC+2, dustismo wrote:
>
> findAndModify is what we use for a priority queue. We are able to
> process > 1000 tasks per second on a single mongo instance using this
> approach. Just make sure your indexing and querying strategy is
> optimal.
>
> On Tue, Aug 3, 2010 at 12:12 AM, bizz <her...@gmail.com <javascript:>>
> wrote:
> > I tried that but the performance is very bad, comparing to other
> > options.
> >
> > On Aug 2, 6:29 pm, Michael Dirolf <m...@10gen.com> wrote:
> >> I think the best bet will probably be to use findAndModify. The
> >> update/limit that you suggest would be subject to race conditions,
> >> etc. Some sort of bulk findAndModify would be interesting but for now
> >> I would just do multiple calls.
> >>
> >> On Mon, Aug 2, 2010 at 10:07 AM, bizz <hers...@gmail.com> wrote:
> >> > Hi,
> >>
> >> > I want to use mongodb to create jobs priority queue.
> >> > I created a collection containing all the jobs, each job has priority
> >> > and minimum time for the next execution.
> >> > Now, I want to query for bulk of jobs to execute. (I use the java
> >> > client to do so)
> >> > I use the find method to query, sort and limit the jobs I want, and in
> >> > return I get a cursor.
> >>
> >> > My problem:
> >> > After I get the results I would like to mark the jobs as being
> >> > processed. I could come up with the next options:
> >> > 1. go over all the results (one by one with the cursor) and collect
> >> > all the ids, then use the update method to mark the jobs.
> >> > 2. move all the jobs to a new collection dedicated to the processing
> >> > client
> >> > 3. Not use the find method, and instead use FindAndModify for the
> >> > number of jobs I want, one by one
> >>
> >> > The options listed above have bad performance, because I have to go
> >> > over the jobs one by one to collect their ids so I could mark them as
> >> > being executed.
> >>
> >> > If I had one of the following features I would have been able to
> >> > perform this task:
> >> > 1. FindAndModify on bulk of entries
> >> > 2. Update with limit (so I could update using the original query)
> >> > 3. An option to get all the results from the find and not a cursor
> >>
> >> > I couldn't think of any other way that I could do this task with
> >> > better performance,
> >> > Anyone have an Idea?
> >>
> >> > Thanks
> >>
> >> > --
> >> > You received this message because you are subscribed to the Google
> Groups "mongodb-user" group.
> >> > To post to this group, send email to mongod...@googlegroups.com<javascript:>
> .
> >> > To unsubscribe from this group, send email to
> mongodb-user...@googlegroups.com <javascript:>.
> >> > For more options, visit this group athttp://
> groups.google.com/group/mongodb-user?hl=en.
> >
> > --
> > You received this message because you are subscribed to the Google
> Groups "mongodb-user" group.
> > To post to this group, send email to mongod...@googlegroups.com<javascript:>
> .
> > To unsubscribe from this group, send email to
> mongodb-user...@googlegroups.com <javascript:>.
> > For more options, visit this group at
> http://groups.google.com/group/mongodb-user?hl=en.
> >
> >
>
>
------=_Part_358_8861506.1352908708212
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: quoted-printable
Sounds great. Could you give a short explanation of your schema and indices=
for this?<div><br></div><div>Alex<br><br>On Tuesday, August 3, 2010 7:43:3=
7 PM UTC+2, dustismo wrote:<blockquote class=3D"gmail_quote" style=3D"margi=
n: 0;margin-left: 0.8ex;border-left: 1px #ccc solid;padding-left: 1ex;">fin=
dAndModify is what we use for a priority queue. We are able to<br>pro=
cess > 1000 tasks per second on a single mongo instance using this<br>ap=
proach. Just make sure your indexing and querying strategy is<br>opt=
imal.<p></p><p>On Tue, Aug 3, 2010 at 12:12 AM, bizz <<a href=3D"javascr=
ipt:" target=3D"_blank" gdf-obfuscated-mailto=3D"AvKC-L9Zkw4J">her...@gmail=
.com</a>> wrote:<br>> I tried that but the performance is very bad, c=
omparing to other<br>> options.<br>><br>> On Aug 2, 6:29 pm, =
Michael Dirolf <<a>m...@10gen.com</a>> wrote:<br>>> I think the=
best bet will probably be to use findAndModify. The<br>>> update/lim=
it that you suggest would be subject to race conditions,<br>>> etc. S=
ome sort of bulk findAndModify would be interesting but for now<br>>>=
I would just do multiple calls.<br>>><br>>> On Mon, Aug 2, 201=
0 at 10:07 AM, bizz <<a>hers...@gmail.com</a>> wrote:<br>>> >=
; Hi,<br>>><br>>> > I want to use mongodb to create jobs pri=
ority queue.<br>>> > I created a collection containing all the job=
s, each job has priority<br>>> > and minimum time for the next exe=
cution.<br>>> > Now, I want to query for bulk of jobs to execute. =
(I use the java<br>>> > client to do so)<br>>> > I use th=
e find method to query, sort and limit the jobs I want, and in<br>>> =
> return I get a cursor.<br>>><br>>> > My problem:<br>>=
;> > After I get the results I would like to mark the jobs as being<b=
r>>> > processed. I could come up with the next options:<br>>&g=
t; > 1. go over all the results (one by one with the cursor) and collect=
<br>>> > all the ids, then use the update method to mark the jobs.=
<br>>> > 2. move all the jobs to a new collection dedicated to the=
processing<br>>> > client<br>>> > 3. Not use the find me=
thod, and instead use FindAndModify for the<br>>> > number of jobs=
I want, one by one<br>>><br>>> > The options listed above h=
ave bad performance, because I have to go<br>>> > over the jobs on=
e by one to collect their ids so I could mark them as<br>>> > bein=
g executed.<br>>><br>>> > If I had one of the following feat=
ures I would have been able to<br>>> > perform this task:<br>>&=
gt; > 1. FindAndModify on bulk of entries<br>>> > 2. Update wit=
h limit (so I could update using the original query)<br>>> > 3. An=
option to get all the results from the find and not a cursor<br>>><b=
r>>> > I couldn't think of any other way that I could do this task=
with<br>>> > better performance,<br>>> > Anyone have an =
Idea?<br>>><br>>> > Thanks<br>>><br>>> > --<b=
r>>> > You received this message because you are subscribed to the=
Google Groups "mongodb-user" group.<br>>> > To post to this group=
, send email to <a href=3D"javascript:" target=3D"_blank" gdf-obfuscated-ma=
ilto=3D"AvKC-L9Zkw4J">mongod...@googlegroups.com</a>.<br>>> > To u=
nsubscribe from this group, send email to <a href=3D"javascript:" target=3D=
"_blank" gdf-obfuscated-mailto=3D"AvKC-L9Zkw4J">mongodb-user...@<wbr>google=
groups.com</a>.<br>>> > For more options, visit this group athttp:=
//<a href=3D"http://groups.google.com/group/mongodb-user?hl=3Den" target=3D=
"_blank">groups.google.com/<wbr>group/mongodb-user?hl=3Den</a>.<br>><br>=
> --<br>> You received this message because you are subscribed to the=
Google Groups "mongodb-user" group.<br>> To post to this group, send em=
ail to <a href=3D"javascript:" target=3D"_blank" gdf-obfuscated-mailto=3D"A=
vKC-L9Zkw4J">mongod...@googlegroups.com</a>.<br>> To unsubscribe from th=
is group, send email to <a href=3D"javascript:" target=3D"_blank" gdf-obfus=
cated-mailto=3D"AvKC-L9Zkw4J">mongodb-user...@<wbr>googlegroups.com</a>.<br=
>> For more options, visit this group at <a href=3D"http://groups.google=
.com/group/mongodb-user?hl=3Den" target=3D"_blank">http://groups.google.com=
/<wbr>group/mongodb-user?hl=3Den</a>.<br>><br>><br></p><p></p></block=
quote></div>
------=_Part_358_8861506.1352908708212--
------=_Part_357_13748121.1352908708212--
Message from discussion priority queue with mongodb
| Create a group - Google Groups - Google Home - Terms of Service - Privacy Policy |
| ©2013 Google |