Received: by 10.66.85.166 with SMTP id i6mr214017paz.13.1349693689160;
        Mon, 08 Oct 2012 03:54:49 -0700 (PDT)
X-BeenThere: mongodb-user@googlegroups.com
Received: by 10.68.141.133 with SMTP id ro5ls22153084pbb.1.gmail; Mon, 08 Oct
 2012 03:54:34 -0700 (PDT)
Received: by 10.68.233.130 with SMTP id tw2mr4352265pbc.12.1349693674755;
        Mon, 08 Oct 2012 03:54:34 -0700 (PDT)
Date: Mon, 8 Oct 2012 03:54:34 -0700 (PDT)
From: viC <vicky.it....@gmail.com>
To: mongodb-user@googlegroups.com
Message-Id: <d61ad9c0-5d29-453d-8582-a7fd2a937424@googlegroups.com>
In-Reply-To: <30293d8f-59c5-46e3-b766-e5688116fc41@googlegroups.com>
References: <30293d8f-59c5-46e3-b766-e5688116fc41@googlegroups.com>
Subject: Re: Deleting Data That Has Been Processed
MIME-Version: 1.0
Content-Type: multipart/mixed; 
	boundary="----=_Part_554_22431434.1349693674241"

------=_Part_554_22431434.1349693674241
Content-Type: multipart/alternative; 
	boundary="----=_Part_555_21068943.1349693674241"

------=_Part_555_21068943.1349693674241
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit

In case you don't have a criteria to separate them, try this:

    1> rename your collection
    
    2> Now do all your processing on the new collection.

*Warning: It doesn't work well in Sharded environments*

On Thursday, 4 October 2012 18:52:37 UTC+5:30, Toby Ho wrote:
>
> Hello all,
>
> I am writing a batch job to aggregate some data using the aggregation 
> framework. Since the data output is potentially larger than the document 
> limit size, I am using a $limit in the top of my pipeline to reduce the 
> number of objects that are processed at a time. After the aggregation is 
> complete, I save off the result in another collection. Now, I would like to 
> remove all the records I have processed so far. how would I reliably do 
> this without having to worry about race conditions? Or is there a better 
> way to go about what I am trying to do?
>
> Thanks,
> Toby
>

------=_Part_555_21068943.1349693674241
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: quoted-printable

<div>In case you don't have a criteria to&nbsp;separate&nbsp;them, try this=
:</div><div><br></div><div>&nbsp; &nbsp; 1&gt; rename your collection</div>=
<div>&nbsp; &nbsp;&nbsp;</div><div>&nbsp; &nbsp; 2&gt; Now do all your proc=
essing on the new collection.</div><div><br></div><div>*Warning: It doesn't=
 work well in Sharded environments*</div><br>On Thursday, 4 October 2012 18=
:52:37 UTC+5:30, Toby Ho  wrote:<blockquote class=3D"gmail_quote" style=3D"=
margin: 0;margin-left: 0.8ex;border-left: 1px #ccc solid;padding-left: 1ex;=
"><div>Hello all,</div><div><br></div><div>I am writing a batch job to aggr=
egate some data using the aggregation framework. Since the data output is p=
otentially larger than the document limit size, I am using a $limit in the =
top of my pipeline to reduce the number of objects that are processed at a =
time. After the aggregation is complete, I save off the result in another c=
ollection. Now, I would like to remove all the records I have processed so =
far. how would I reliably do this without having to worry about race condit=
ions? Or is there a better way to go about what I am trying to do?</div><di=
v><br></div><div>Thanks,</div><div>Toby</div></blockquote>
------=_Part_555_21068943.1349693674241--

------=_Part_554_22431434.1349693674241--