MongodB can handle up 300 milion record?

[Ketul]

0down votefavorite

I have 300 milion records so can mongodb is suitable for that. ? I already Mongodb database which contains milions of data but when i query then it will takes more time .so can is it possible any feasible to these data in mongodb?

[Tim Hawkins]

It depends on the size of the records, the speed of your disks and how much memory is in your server, and how much of the data is concurrent and in the working set.

On Sat, Mar 21, 2015, 16:57 Ketul <k2ls...@gmail.com> wrote:

0down votefavorite

I have 300 milion records so can mongodb is suitable for that. ? I already Mongodb database which contains milions of data but when i query then it will takes more time .so can is it possible any feasible to these data in mongodb?

--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.
 
For other MongoDB technical support options, see: http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user...@googlegroups.com.
To post to this group, send email to mongod...@googlegroups.com.
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/dd0d94ef-4b86-4a1d-b24d-d61927ade564%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[s.molinari]

If it depends on all that, what then could handle it, if MongoDB can't?

Scott

[Ketul]

hi Tim,

if i used sharding concept then is it possible ?or i have to move  on another technology like hadoop + mongodb.. ?

[Tim Hawkins]

You still have not given us any information about your data,  how big is the documents, how much memory do you have in your server, are you using slow disks, you would be supprised how many people do benchmarking on a 2Gb laptop with a 5400 rpm drive.

On Mon, Mar 23, 2015, 16:28 Ketul <k2ls...@gmail.com> wrote:

hi Tim,

if i used sharding concept then is it possible ?or i have to move  on another technology like hadoop + mongodb.. ?

--

You received this message because you are subscribed to the Google Groups "mongodb-user"
group.
 
For other MongoDB technical support options, see: http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user...@googlegroups.com.
To post to this group, send email to mongod...@googlegroups.com.
Visit this group at http://groups.google.com/group/mongodb-user.

To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/f1a271e1-dd3a-427a-90e3-a4080d1baea8%40googlegroups.com.

[Ketul]

Hi Tim,

i have 50GB data, these are already in my mongodb database so when i try to retrieve any data from that or do any aggregation on that data, it will takes more time so for that i have  asked question, should  if i used sharding  then  got any improvement on that? 

[Tim Hawkins]

How much memory does the machine have?, what type of disks, 5400?, 7200?, raid? Ssd?

On Mon, Mar 23, 2015, 19:48 Ketul <k2ls...@gmail.com> wrote:

Hi Tim,

i have 50GB data, these are already in my mongodb database so when i try to retrieve any data from that or do any aggregation on that data, it will takes more time so for that i have  asked question, should  if i used sharding  then  got any improvement on that? 

--

You received this message because you are subscribed to the Google Groups "mongodb-user"
group.
 
For other MongoDB technical support options, see: http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user...@googlegroups.com.
To post to this group, send email to mongod...@googlegroups.com.
Visit this group at http://groups.google.com/group/mongodb-user.

To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/1bebe1cc-a9fa-47cf-a7e7-63526174e6a7%40googlegroups.com.

[Ketul]

HI Tim

 

4GB RAM and disk type is 7200.

[Tim Hawkins]

Then no 50gb of data and only 3gb of ram. Then unless most of your data is not going to be read, then its not going to be fast and will spend most of its time swapping data in from disk. Rather than sharding you are better off putting the RAM you would have had in the other servers into your first machine. Until you have a reasonable amount of coverage of the dataset.

On Tue, Mar 24, 2015, 13:07 Ketul <k2ls...@gmail.com> wrote:

HI Tim

 

4GB RAM and disk type is 7200.

--

You received this message because you are subscribed to the Google Groups "mongodb-user"
group.
 
For other MongoDB technical support options, see: http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user...@googlegroups.com.
To post to this group, send email to mongod...@googlegroups.com.
Visit this group at http://groups.google.com/group/mongodb-user.

To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/2ae5872c-512f-418d-ada3-48cc24801f12%40googlegroups.com.

[Ketul]

HI kim

 But if i am used sharding concept and divide data in different size then is it not possible? 

or if i have 1 TB data then what is requirements of RAM and other configuration? 

[s.molinari]

It all depends on what data is your "hot" data. But in general, if you have 1TB of data, then 4GB of RAM is most definitely not enough. For 50GB, it is pushing it.

I am not sure why you are asking about sharding. Do you actually have more than one 4GB machine to shard on? 

I've been working on the question of "how much RAM is necessary" for a long time now and there simply isn't a general answer. If you can estimate what kind of data you'll need at any given point in time out of the whole data set, then you can estimate what your working set is going to be like and then plan for some RAM headroom above that. That is the general rule for capacity planning and there is no general rule for the headroom part either. It is basically based on the risk you are willing to take and the amount of peaks in operations you want to properly cover. If you have SSDs, you can take more risk with less RAM headroom. With 7200 SATAs, I'd not push the risk too much and stay conservative.

That all being said, if you need, say, only 10GB of data (including indexes) at any given time from the 50GB you mentioned earlier, then you need probably at least 12-15GB of RAM as a minimum. Probably 15 would be the minimum, maybe even 20. That would mean, you'd need a shard of around 4-6 primary machines, depending on how much RAM the OS takes on each machine and how much headroom you want to give yourself. (oh, and of course, if you are using Mongo in production, you'll need 8-12 more machines for replicas, as a minimum, plus the 3 config servers.) 

Scott

[Ketul]

okay i got it    but my problem is, i have 300 million  records now i want to try fetch last 20 record then its take more time to fetching those data so for that what i have to do?

[Tim Hawkins]

Did you create an index on a suitable field? To support the sorting of the results

Ie

db.mycollection.find().sort({"added":-1}).limit(20)

Where "added" is a timestamp field set to the time the document was added,  and you have an index on the added element

db.mycollection.ensureIndex({"added":-1})

On Tue, Mar 24, 2015, 17:20 Ketul <k2ls...@gmail.com> wrote:

okay i got it    but my problem is, i have 300 million  records now i want to try fetch last 20 record then its take more time to fetching those data so for that what i have to do?

--

You received this message because you are subscribed to the Google Groups "mongodb-user"
group.
 
For other MongoDB technical support options, see: http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user...@googlegroups.com.
To post to this group, send email to mongod...@googlegroups.com.
Visit this group at http://groups.google.com/group/mongodb-user.

To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/84461582-81a4-4f57-906d-a3f10d103fc0%40googlegroups.com.

[Ketul]

Hi tim,

mongodb _id field bydefault contains timestamp then is it necessary to took another time stamp?

[s.molinari]

You can use the objectid (and its index), however you'll have to do some conversion first to be able to use objectids for time range queries. 

Scott