How much disk space require to build mongo from source? 100 GB is not enough.

[Amit Chotaliya]

I am trying to build mongo from source. The server ran out of disk space while compilation was in progress. Here's the details.

I am following steps in https://github.com/mongodb/mongo/wiki/Build-Mongodb-From-Source

- Ubuntu : 18.4 LTS

- Mongo : v3.6.10

Here's the space usage info.

12K     APACHE-2.0.txt

4.0K    CONTRIBUTING.rst

32K     LICENSE-Community.txt

4.0K    README

124K    SConstruct

95G     build

3.7M    buildscripts

1.1G    dbtest

432K    debian

120K    distsrc

40K     docs

744K    etc

15M     jstests

374M    mongo

223M    mongobridge

938M    mongod

921M    mongoperf

463M    mongos

116K    pytests

204K    rpm

288K    site_scons

227M    src

[Bob Cochran]

Hi,

That is definitely not “100 Gb” if you mean gigabytes. It’s less than 11 gigabytes. Try ‘df -h [your build directory]’ to get a pretty exact size estimate. 

Thanks so much 

Bob

--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.
 
For other MongoDB technical support options, see: https://docs.mongodb.com/manual/support/
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user...@googlegroups.com.
To post to this group, send email to mongod...@googlegroups.com.
Visit this group at https://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/3885a544-ed74-4a97-ab59-46b231b6251a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Andrew Morrow]

Hi -

It looks to me like you have built the 'all target. Are you building this for the purpose of development, or for building binaries that you will run in production? If you are doing the former, I can provide some guidance on some build options that will result in a much smaller footprint, but which are not suitable for production. If you are doing the latter, then you don't necessarily need to build the 'all' target, which includes a very large number of statically linked unit tests.

Thanks,

Andrew

[Bob Cochran]

Oops— my bad — I didn’t notice that “95 G” until now. Sorry!

Thanks so much

Bob

[Amit Chotaliya]

Thanks Andrew.

I am building binaries for production. I have made code changes to increase document size.

I was able to compile and run mongod and mongo binaries by specifying these scons targets.

I am not able to find how to compile mongodump and mongorestore. These targets are not available. The documentation does not mention it anywhere.

I was using 2.6 before, it compiled all the targets including dump and restore. Can you please help me with that?

[Andrew Morrow]

Hi -

On Mon, Feb 18, 2019 at 8:26 AM Amit Chotaliya <amit.m.c...@gmail.com> wrote:

Thanks Andrew.

I am building binaries for production. I have made code changes to increase document size.

That sounds like a pretty dangerous change!

 

I was able to compile and run mongod and mongo binaries by specifying these scons targets.

Glad to hear you have that working.

 

I am not able to find how to compile mongodump and mongorestore. These targets are not available. The documentation does not mention it anywhere.

The tools were re-written in Go and are now in a different repository. You can access then and find build instructions here: https://github.com/mongodb/mongo-tools

Thanks,

Andrew

[Amit Chotaliya]

Hi @Andrew, 

Thank you for the help. I was able to compile and run the mongo tools.

I am facing an issue right now. Here's the case.

- I use mongo only as cache and dump data from mysql to mongodb everyday using AWS EMR. That is why I have disabled journaling because I don't need the reliability and consistency.

- In mongo version 2.6 with custom bson/doc size of 64 MB. I was able to dump all the 15 GB data in 1 hour from mysql to mongodb using EMR. The machine config was 8 GB and 2 vCPUs.

- In the new Mongo 3.6 for the same RAM and same data size and everything else same. The mongo is using too much RAM and is getting killed by OOM killer? Can you point if anything that has changed in 3.6 which could explain this behaviour? Also I see the mongo is using 56% of the RAM, as per my understanding it should be using 50% - 1 GB of RAM.

[Andrew Morrow]

Hi Amit -

I'm happy to know you were able to get it built.

Unfortunately, I'm not really the best person to answer your subsequent questions. I've flagged this thread to another engineer who I hope will be able to help you further. I will though note that the differences between MongoDB 2.6 and 3.6 are extensive, so there may not be a simple answer to that question.

Thanks,

Andrew

--

You received this message because you are subscribed to the Google Groups "mongodb-user"
group.
 
For other MongoDB technical support options, see: https://docs.mongodb.com/manual/support/
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user...@googlegroups.com.
To post to this group, send email to mongod...@googlegroups.com.
Visit this group at https://groups.google.com/group/mongodb-user.

To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/1874394d-b8d6-4c6d-9157-e53420097abc%40googlegroups.com.

[Amit Chotaliya]

@Andrew

FYI, Found workaround for it. I changed the storage engine to mmapv1 from WireTiger. Now the memory is in control and it is behaving as before.

[Wan Bachtiar]

Can you point if anything that has changed in 3.6 which could explain this behaviour?

Hi Amit,

As pointed out by Andrew, the differences between v2.6 and v3.6 are extensive. You can browse through from the list of releases between versions below:

• Release Notes v3.0
• Release Notes v3.2
• Release Notes v3.4
• Release Notes v3.6

One of the changes in the release notes above, is starting from v3.2 MongoDB uses WiredTiger as the default storage engine.

as per my understanding it should be using 50% - 1 GB of RAM.

With WiredTiger, MongoDB utilises both the WiredTiger internal cache AND the filesystem cache.
The default 50% of RAM - 1GB ( If you have 8GB, that’s 50% of 7GB = 3.5GB ) is only referring to the WiredTiger internal cache. Please also review:

• WiredTiger: Memory Use
• WiredTiger: FAQs

Via the filesystem cache, MongoDB automatically uses all free memory that is not used by the WiredTiger cache or by other processes.

In mongo version 2.6 with custom bson/doc size of 64 MB. I was able to dump all the 15 GB data in 1 hour from mysql to mongodb using EMR

I’m still unclear why would you increase the BSON document size. Unless a single document size in your collection requires to be greater than 16MB. If a single document reached the BSON document size, instead of increasing the BSON document size, I would suggest to consider your data modelling by making the documents much smaller to improve performance.

Regards,
Wan.

​