[opinion] for storage size

[PaulKE]

Hello, 

I would like to ask 'export' opinion on running mongo db on Amazon EC2. 

Recently, 10Gen released new AMI that has great packages MongoDB with 4000 Provisioned IOPS.

It has all the manual setup I did like ulimit settings, read aheads, and, three EBS with data, log, and journal. 

It's all good. 

How, based on optimized ratio (10:1) for PIOPS, I set up 400 GB storage for my instance ( i have one master, one slave and a single arbiter in a shard ).

now, here is a question. 

we are writing lots of metric data. you know, the 400 GB can be filled up rather quickly than later. 

Of course, we can keep on adding more shards. But, you know, a single shard is expensive since it has three EC2 instances. 

But, compared to these CPU & Memory cost, storage is relatively cheap. 1 GB costs like 10 cents per month. 

So, if I would like to create 1 TB instead of 400 GB with 4000 IOPS, does this make sense? 

I know that would be more than 20:1 ratio for storage to IOPS. If not, what would be a good method of setting up 1 TB with these new AMIs? 

cost effectiveness matters. In addition, since it's metric data, it's write-heavy. reading will happen maybe few times to export data of off-line analysis on secondary. 

Give me some expert opinions on this situation. 

Thanks in advance. 

[Sam Millman]

"So, if I would like to create 1 TB instead of 400 GB with 4000 IOPS, does this make sense? "

If that is what works then totally. I mean IOPS isn't totally key, tbh if you really cared for performance you would ensure to get a memory bloated instance, they are quite cheap, to ensure you have what you need for your working set. IOPS only helps when you hit the disk and retrieving pages from disk, however, there are ways around the IOPS need.

"cost effectiveness matters. In addition, since it's metric data, it's write-heavy."

If it is write heavy then you will need to think of what your write concern will demand of mongodb. If you require only an ack then MongoDB does not need to write to disk on the spot and can do it gracefully, bit by bit through an fsync queue, however, if you are demanding Mongodb to write to jounal then you could find your IO easily saturated if you are really getting that many writes.

--
--
You received this message because you are subscribed to the Google
Groups "mongodb-user" group.
To post to this group, send email to mongod...@googlegroups.com
To unsubscribe from this group, send email to
mongodb-user...@googlegroups.com
See also the IRC channel -- freenode.net#mongodb
 
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[PaulKE]

Thanks for suggestion. 

For write-concern, I am on default write concern which is 'ack', I think (driver sets getLastError:1 without having other arguments). So, I think it will be OK for demands. 

But, as you suggested, I tried to create 1 TB instead of 400 GB when I create an EC2 instance with Provisioned IOPS 4000 AMI. 

So, at the EBS volume sets, I edited, and had to remove current pre-set 400 GB, and insert new one with 1TB with 4000 IOPS with same driver name 'sdf' (so, it 'overrides').

There is no 'update/change' on EBS volume in the EC2 creation wizard on using this AMI so I had to remove/insert as above step. 

Then, it launched. But, I noticed that it still only uses 400 GB for 'data' volume (dev/sdf). I confirmed with 'df -h' command. 

Now, when I do 'fdisk', I can see that physical volume has 1 TB for sdf. But, only 400 GB were set to be used for 'data'. 

Is there anything I can do to make this to be increased to full capacity of 1 TB? 

I checked volume group (vgdisplay), but, it dose not have one on this instance. 

So, I am confused now. 

[Sam Millman]

Ah yea thats the horrors of templates.

 I think the only way to rebundle the AMI which is stupid. You would proably be better off just doing your own custom image from scratch, tbh the only difference is that the AMI would have MongoDb preinstalled.