Mongo in containers: how to manage memory

[bibelo]

Hello everyone

We have an apache/mongo architecture, organized around Docker. So far everything was fine, but with the bases growing, we now face new challenges.

Right now, we have 3 projects in production, with a total of 6 containers (1 apache container and 1 mongo container per project).

The problem we face is that each mongo container think they have 21GB of memory, so they try to use 80% of it

So the solution for me was to restart one of the container with a memory limit given to the mongo process :

--wiredTigerCacheSizeGB 7

The problem with that solution is that the limit is fixed for each mongo container. If one base needs more memory, I would have to restart the container with another value.

So please I have some questions. I think these questions might interest others in general, since I didn't find anything about that precisely on the Internet :

1) Is there a global solution to balance memory between mongo container?

That is, automatically? (without manual supervision, where I've got to check the I/O on disks)

And also, without restarting the container?

2) Are Cache limitations really useful? Since the OS limits the container anyway, should I remove the limit and disable the alarms I get? (Not doing that anyway)

3) Is it really useful to have one container per base? 

Would it be more practical, under the aspect of memory, to have one big mongo container for all three bases?

What would be the pros and cons for both solutions?

One mongo container for all projects bases

Pros:

Easier to manage

Easier to balance memory

Cons:

Less secure

One mongo container per base:

Pros:

More secure

Easier to backup? (not really)

Cons:

Difficult to handle memory

4) We are having a more general debate how memory should relate to disk with mongo.

Personally, I don't think it's a problem if the base on disk is bigger than its footprint in memory, ie it's OK to have parts of database on disk that are rarely touched and thus won't find themselves in memory.

My devs think that size on disk should be the size in memory. That is, if I have a 40GB databases, I should have 40GB of memory, which appear unrealistic to me.

What is your opinion on that?

Thank you everyone for their answers!

[Robert Cochran]

Hi!

I think one deciding factor in scaling a database how critical it is to the enterprise. If you have a lot of traffic on the database and would suffer financial loss without it, then you need to support the growth in database use. I can say that in general, the more memory you can give to a database server and its engine, the better it is for you. If you run containers you need to give them a lot of memory. 

I realize I'm speaking in generalities -- which is the way I want to. If it were me, I would install as much real memory as I can for my containers, and give each of them the memory they need to operate with extra memory to tap when needed, and configure things such that I do not need to spend all my time on memory tunables. I'd rather go on to other things that have a greater payback for me. 

Thanks

Bob

[Rodrigo Broggi]

Hi,

May I ask you if you're using any container orchestration technologies for your environment (openshift/kubernetes/swarm)?

It sounds like what you are searching for is  to have a more fine grained container runtime configurability, which is well tackled by the above mentioned technologies. 

Thank you and best regards,

Rodrigo

--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.
 
For other MongoDB technical support options, see: https://docs.mongodb.com/manual/support/
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user...@googlegroups.com.
To post to this group, send email to mongod...@googlegroups.com.
Visit this group at https://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/ea9b25de-8bf2-4ff8-9276-b05aec02eb69%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[bibelo]

Hello Rodrigo thank you for your answer,

No we are not using any other tool than the command line so far.

I have in mind the need to use of these tools. We need to streamline the use of your containers, we won't last long with command line :-)

[bibelo]

Hello Robert thank you for your answer!

I agree with your idea.

Unfortunately, we have 3 databases on the same servers with limited resources, so I have to decide how to distribute resources between them.

Then my other question would be about removing the need to distribute resource, and gather all 3 databases under one engine (one container)

[Rodrigo Broggi]

Hi,

Notice that all of the orchestration solutions have also a command line interface (which is the preferred way of most power users). I would suggest for you to check them out.

Br,

Rodrigo

To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/8599bfaa-137a-4eb6-a28d-d5ab943f8be3%40googlegroups.com.

[Robert Cochran]

I think Rodrigo brings up a good point, to use software in order to enable some control. 

I have the impression that you have just one physical server and you want to apportion the memory resources for that physical server to the 3 containers you mention. Also you do not want to add physical memory to the physical server. You don't state how much memory your physical server has, but give the impression that it does not have very much memory. Also, I get the impression that you are unwilling to scale out by adding physical servers. You are not just providing memory to 3 databases: you are also providing memory to operating systems running the containers, and to the supervisor that launches and runs the containers as well. 

It comes down -- again -- to what financial value you are getting from your databases. If the return on investment is minimal then perhaps throttling the existing servers and limiting their resource usage is an option.

If there is financial value to your databases, then perhaps you can consider hiring a specialist with experience in running containerized databases. That person, or perhaps that company, will have a better picture of how to provision each database, how to scale the hardware, and later on, in how to consolidate databases (although consolidation is a pretty major project.) Also as Rodrigo points out, there is control software available, and a specialist would likely have plenty of experience with such software.

I do use Vagrant containers but only on an educational basis. I have zero experience in managing simultaneously running containers and database instances. 

Good luck!