Size 43002467 is larger than MaxDocumentSize 16793600.

sunil sharma

unread,

Dec 23, 2015, 7:42:12 AM12/23/15

to mongodb-user

Hi,

I am creating a C#, MVC application with Mongodb. But i am facing a problem when i am trying to save 14 Mb data in data in mongo db in form of List object. I have checked with less amount of data near to 1 mb then it working fine .

Below id the error detail comes when try to save near to 14 mb data in mongo db in C#.net:-

{"Size 43002467 is larger than MaxDocumentSize 16793600."}

at MongoDB.Bson.IO.BsonBinaryWriter.BackpatchSize()

at MongoDB.Bson.IO.BsonBinaryWriter.WriteEndArray()

at MongoDB.Bson.Serialization.Serializers.EnumerableSerializerBase`1.Serialize(BsonWriter bsonWriter, Type nominalType, Object value, IBsonSerializationOptions options)

at MongoDB.Bson.Serialization.Serializers.BsonDocumentWrapperSerializer.Serialize(BsonWriter bsonWriter, Type nominalType, Object value, IBsonSerializationOptions options)

at MongoDB.Bson.Serialization.Serializers.BsonDocumentSerializer.Serialize(BsonWriter bsonWriter, Type nominalType, Object value, IBsonSerializationOptions options)

at MongoDB.Bson.Serialization.Serializers.BsonValueSerializer.Serialize(BsonWriter bsonWriter, Type nominalType, Object value, IBsonSerializationOptions options)

at MongoDB.Bson.Serialization.Serializers.BsonDocumentSerializer.Serialize(BsonWriter bsonWriter, Type nominalType, Object value, IBsonSerializationOptions options)

at MongoDB.Bson.Serialization.Serializers.BsonValueSerializer.Serialize(BsonWriter bsonWriter, Type nominalType, Object value, IBsonSerializationOptions options)

at MongoDB.Bson.Serialization.Serializers.BsonDocumentSerializer.Serialize(BsonWriter bsonWriter, Type nominalType, Object value, IBsonSerializationOptions options)

at MongoDB.Driver.Builders.UpdateBuilder.Serialize(BsonWriter bsonWriter, Type nominalType, IBsonSerializationOptions options)

at MongoDB.Driver.Builders.BuilderBase.MongoDB.Bson.Serialization.IBsonSerializable.Serialize(BsonWriter bsonWriter, Type nominalType, IBsonSerializationOptions options)

at MongoDB.Bson.Serialization.Serializers.BsonIBsonSerializableSerializer.Serialize(BsonWriter bsonWriter, Type nominalType, Object value, IBsonSerializationOptions options)

at MongoDB.Bson.Serialization.BsonSerializer.Serialize(BsonWriter bsonWriter, Type nominalType, Object value, IBsonSerializationOptions options)

at MongoDB.Bson.Serialization.BsonSerializer.Serialize[TNominalType](BsonWriter bsonWriter, TNominalType value, IBsonSerializationOptions options)

at MongoDB.Bson.Serialization.BsonSerializer.Serialize[TNominalType](BsonWriter bsonWriter, TNominalType value)

at MongoDB.Driver.Operations.BulkUpdateOperation.UpdateBatchSerializer.SerializeRequest(BsonBinaryWriter bsonWriter, WriteRequest request)

at MongoDB.Driver.Operations.BulkUnmixedWriteOperationBase.BatchSerializer.AddRequest(BsonBinaryWriter bsonBinaryWriter, WriteRequest request)

at MongoDB.Driver.Operations.BulkUnmixedWriteOperationBase.BatchSerializer.Serialize(BsonWriter bsonWriter, Type nominalType, Object value, IBsonSerializationOptions options)

at MongoDB.Bson.Serialization.Serializers.BsonDocumentWrapperSerializer.Serialize(BsonWriter bsonWriter, Type nominalType, Object value, IBsonSerializationOptions options)

at MongoDB.Bson.Serialization.Serializers.BsonDocumentSerializer.Serialize(BsonWriter bsonWriter, Type nominalType, Object value, IBsonSerializationOptions options)

at MongoDB.Bson.Serialization.Serializers.BsonValueSerializer.Serialize(BsonWriter bsonWriter, Type nominalType, Object value, IBsonSerializationOptions options)

at MongoDB.Bson.Serialization.Serializers.BsonArraySerializer.Serialize(BsonWriter bsonWriter, Type nominalType, Object value, IBsonSerializationOptions options)

at MongoDB.Bson.Serialization.Serializers.BsonValueSerializer.Serialize(BsonWriter bsonWriter, Type nominalType, Object value, IBsonSerializationOptions options)

at MongoDB.Bson.Serialization.Serializers.BsonDocumentSerializer.Serialize(BsonWriter bsonWriter, Type nominalType, Object value, IBsonSerializationOptions options)

at MongoDB.Bson.Serialization.BsonSerializer.Serialize(BsonWriter bsonWriter, Type nominalType, Object value, IBsonSerializationOptions options)

at MongoDB.Driver.Internal.MongoQueryMessage.WriteBodyTo(BsonBuffer buffer)

at MongoDB.Driver.Internal.MongoRequestMessage.WriteTo(BsonBuffer buffer)

at MongoDB.Driver.Internal.MongoConnection.SendMessage(MongoRequestMessage message)

at MongoDB.Driver.Operations.CommandOperation`1.Execute(MongoConnection connection)

at MongoDB.Driver.Operations.BulkUnmixedWriteOperationBase.ExecuteBatch(MongoConnection connection, Batch`1 batch, Int32 originalIndex)

at MongoDB.Driver.Operations.BulkUnmixedWriteOperationBase.Execute(MongoConnection connection)

at MongoDB.Driver.Operations.BulkUpdateOperation.Execute(MongoConnection connection)

at MongoDB.Driver.Operations.UpdateOpcodeOperationEmulator.Execute(MongoConnection connection)

at MongoDB.Driver.Operations.UpdateOpcodeOperation.Execute(MongoConnection connection)

at MongoDB.Driver.MongoCollection.Update(IMongoQuery query, IMongoUpdate update, MongoUpdateOptions options)

at MongoDB.Driver.MongoCollection.Update(IMongoQuery query, IMongoUpdate update, UpdateFlags flags)

at APIManager.MongoDB.Handlers.ApiServiceHandler.SaveApiData(List`1 apiBaseData, String apiName) in d:\Projects\Cloud\ApiPortalCode\API.WebAdmin\APIManager.MongoDB\Handlers\ApiServiceHandler.cs:line 79

Asya Kamsky

unread,

Dec 25, 2015, 10:57:38 AM12/25/15

to mongodb-user

You cannot save documents larger than 16Mbs nor should you want to (normally).

It looks like what you are trying to serialize is greater than 40Mbs - are you trying to save all the entities as array elements? Can you give an example of the code and/or explain exactly what you are trying to save in the DB?

Asya

--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.

For other MongoDB technical support options, see: http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user...@googlegroups.com.
To post to this group, send email to mongod...@googlegroups.com.
Visit this group at https://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/8044ea23-fcbe-44d7-a163-6a7bc41acfc1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Asya Kamsky
Lead Product Manager
MongoDB
Download MongoDB - mongodb.org/downloads
Free MongoDB Monitoring - cloud.mongodb.com
Free Online Education - university.mongodb.com
Get Involved - mongodb.org/community
We're Hiring! - https://www.mongodb.com/careers

sunil sharma

unread,

Dec 28, 2015, 6:49:50 AM12/28/15

to mongodb-user

Hi Asya,

Thanks for your reply!

I am writing application in C#.net and in my application I have implemented logic in C# to read data from xml,csv and xml files and save this data in mongodb collection. But when data in large amount then code break and throwing this size error.

Below is the path of csv file and this file data I used to save in Mongodb but face size error.

https://drive.google.com/file/d/0B6lvaxTLjYwMNXRQRXlZdUh2T0k/view?usp=sharing

Below is the small code that i have written to save list in mongo db:

public bool SaveApiData(List<dynamic> apiBaseData, string apiName)

{

try

{

string key = apiName;

MongoCollection.Remove(Query.EQ(CommonFindByProperty, key), RemoveFlags.Single);

UpdateBuilder update = Update.Set(CommonFindByProperty, key).SetWrapped(apiName, apiBaseData);

MongoCollection.Update(Query.EQ(CommonFindByProperty, key), update, UpdateFlags.Upsert);

}

catch (Exception ex)

{

Logger.Error("An error occured during Save of Api data", ex);

return false;

}

return true;

}

Please let me know if any thing required from my side.

Best regards,

Sunil Sharma

Asya Kamsky

unread,

Dec 28, 2015, 11:59:55 AM12/28/15

to mongodb-user

What is in "apiBaseData" that you pass to this method?

I didn't look at the CSV but I suspect you may be trying to save the entire spreadsheet/CSV file as a single document when you really want to iterate over each line and save each as a single document.

If your data is a list - doesn't that seem to suggest that you should be saving as many records/documents into MongoDB as you have elements in this list? As it is now, it appears you are trying to save the entire list in a single document - but I'm guessing since I don't know how you are passing data to this function, and what part of the CSV data that is.

You had said originally you were "trying to save 14 Mb data in data in mongo db in form of List object" - what exactly do you want to see in the database? What should each single document in MongoDB represent?

Asya

To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/9be0b12c-7a13-4e1c-bdaf-b75e2cc694c0%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

sunil sharma

unread,

Dec 28, 2015, 11:51:39 PM12/28/15

to mongodb-user

Hi Asya,

Thanks for your reply!

"apiBaseData" is object of list type and it contains all data of csv file. First I have get all data in list object and then pass this object to "SaveApiData" method to save all data in mongo db. So when this method save data in mongo db at that time it breaks and gives this error.

Below is more code that I used for fetching data from csv file and add in list and then pass this list to mongo method (SaveApiData):

private List<dynamic> GetDataFromCsv()

{

var list = new List<dynamic>();

try

{

var csvData = GetDataFromCsvFile();

if (csvData == null) return list;

list.AddRange(csvData);

}

catch (Exception ex)

{

Logger.Error("An error occured during GetDataFromCsv" + ex.Message);

}

return list;

}

private IEnumerable<dynamic> GetDataFromCsvFile()

{

using (TextFieldParser parser = new TextFieldParser(FilePath))

{

parser.SetDelimiters(new string[] { "," });

parser.HasFieldsEnclosedInQuotes = true;

// Skip over header line.

parser.ReadLine();

while (!parser.EndOfData)

{

string[] fields = parser.ReadFields();

yield return new

{

Fields = fields,

};

}

Please let me know if anything required from my side.

Best regards,

Sunil Sharma

Asya Kamsky

unread,

Dec 29, 2015, 11:37:11 AM12/29/15

to mongodb-user

Sunil:

You are trying to save the entire dataset as a single document. Have you used relational databases? Do you think that you should save all this data as a single row in a table, or as many rows as there are elements in the list?

You are trying to save it all as a single record. That's wrong. You should be saving each row in the CSV file as a single document/record/row.

I'm not sure how else to say that.

Asya

To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/e21b298c-1f0b-4fc6-9664-b30a4979d81c%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

sunil sharma

unread,

Jan 4, 2016, 1:03:15 AM1/4/16

to mongod...@googlegroups.com

Hi Asya,

Thanks for your reply!

My requirement is very simple as i already explained i want to save a data may be that come from xml/csv/excel or may be from any other source. But when i am trying to save large size data then i am facing this problem.

I have only shared csv problem but this problem occurring with all types . And I have a excel file with less amount of data then it is saving perfectly in mongodb but when data is large then problem occur.

This is I think a generic problem with mongo db when we save large amount of data in collection as a single document. And my data is not a relational data. it is simple data that i am reading from excel/csv or xml files and want to save in collection as a single document.

Please let me know if you want more information. I think you can easily produce this error if you develop a C# project with Mongo db .And save some large data in mongo db.

Best regards,

Sunil Sharma

You received this message because you are subscribed to a topic in the Google Groups "mongodb-user" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/mongodb-user/K3BBN0ypl4k/unsubscribe.
To unsubscribe from this group and all its topics, send an email to mongodb-user...@googlegroups.com.

To post to this group, send email to mongod...@googlegroups.com.
Visit this group at https://groups.google.com/group/mongodb-user.

To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/CAOe6dJBHird2SBt7xoYtzefJPDnCSvqhfSfvEsv%3DqNnC%2BaABoQ%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Thanks&Regards,

Sunil

Wan Bachtiar

unread,

Jan 5, 2016, 10:29:48 PM1/5/16

to mongodb-user

My requirement is very simple as i already explained i want to save a data may be that come from xml/csv/excel or may be from any other source. But when i am trying to save large size data then i am facing this problem.

Hi Sunil,

A database is an organised collection of data. By this definition, you should organise your data in such way that you would be able to efficiently query it later. Different database systems have their own ways of organising such data. For example in a relational database, a database table consists of rows) and columns).

MongoDB is a document-oriented database. A basic unit of data in MongoDB is composed of field(s) and value(s), called a document. Documents are analogous to JSON objects but exist internally as BSON. A MongoDB document is not to be confused with file document or Google Docs’ document. See Introduction to MongoDB for more details.

So, if you would like to save a CSV file into MongoDB, one (simple) way of achieving it is to save a MongoDB document for every single line of the CSV.
Using your CSV file as an example:

UNIQUE_CARRIER,AIRLINE_ID,CARRIER,TAIL_NUM,FL_NUM
9E,20363,9E,N923XJ,3324
9E,20363,9E,N907XJ,3324

In MongoDB, you can store the file as multiple documents in a single collection. For example, the two CSV lines above can be represented as two documents:

{ UNIQUE_CARRIER: "9E", AIRLINE_ID: 20363, CARRIER:"9E", TAIL_NUM:"N923XJ", FL_NUM:3324 }
{ UNIQUE_CARRIER: "9E", AIRLINE_ID: 20363, CARRIER:"9E", TAIL_NUM:"N907XJ", FL_NUM:3324 }

See Data Modelling Introduction for other ways to model your data.
Using the examples above, you then query for certain information.

/* Find all documents containing unique_carrier value of 9E */
db.airlines.find({'UNIQUE_CARRIER': '9E'});

/* Find all documents containing tail_num value of N923XJ from airline_id 20363 */
db.airlines.find({'TAIL_NUM': 'N923XJ', 'AIRLINE_ID':20363});

As you can see, it would not be beneficial to store the entire CSV file content as a single MongoDB document. It would be equivalent to having all of the CSV lines in a single line. You need reasonable data granularity for finding certain piece of information.

Below id the error detail comes when try to save near to 14 mb data in mongo db in C#.net:- {“Size 43002467 is larger than MaxDocumentSize 16793600.”}

To ensure that a single MongoDB document cannot use excessive amount of RAM or, during transmission, excessive amount of bandwidth, there is a maximum size of 16MBs per MongoDB document (BSON). The error message that you see is related to this maximum size. See MongoDB limits and thresholds for more information.

“apiBaseData” is object of list type and it contains all data of csv file. First I have get all data in list object and then pass this object to “SaveApiData” method to save all data in mongo db.

At the moment, your method GetDataFromCsvFile() is yielding a list of strings per line, then GetDataFromCsv keep extending the list. Resulting in a list of strings of all the CSV tuples. i.e. [9E,20363,9E,N923XJ,3324,9E,20363,9E,N907XJ,3324,...]
Even if you store this list in another CSV file, it would not be efficient to find information.

Basing on the simple data model example mentioned previously, you need to modify it into a list of BsonDocuments. An example:

private IEnumerable<BsonDocument> GetDataFromCsvFile()

{
    using (TextFieldParser parser = new

 TextFieldParser(FilePath))
    {
        parser.SetDelimiters(",");
        parser.HasFieldsEnclosedInQuotes = true;
        List<string> headers = new List<string>();
        bool firstLine = true;
        while (!parser.EndOfData)
        {
            string[] fields = parser.ReadFields();
            // Read CSV headers as key for the document.
            if (firstLine)
            {
                foreach (var key in fields) { headers.Add(key); }
                firstLine = false;
                continue;
            }
            // Construct a BsonDocument to be yielded per line. 
            BsonDocument document = new BsonDocument { };
            for (int i = 0; i < headers.Count; i++) {
                document.Add(headers[i], fields[i]);
            }
            // Yield { "header1" : "tuple1", "header2" : "tuple2", ... } 
            yield return document;
        }
    }
}

You also need to update SaveApiData(). Now that you have a list of BsonDocuments, you can utilise InsertMany. For example, storing all the documents into a collection called ‘airlines’:

var collection = database.GetCollection<BsonDocument>("airlines");
collection.InsertMany(apiBaseData);

The example snippets above are written in C# .Net v4.5.2 and MongoDB Driver v2.2.0. See MongoDB C# Driver Quick Tour for more examples and snippets.

I would recommend to enrol in a free online course at MongoDB University to learn more about MongoDB. A new session has just started today so you can join straight away.

Regards,

Wan.

sunil sharma

unread,

Jan 6, 2016, 12:49:17 AM1/6/16

to mongod...@googlegroups.com

Hi Wan,

Thanks for you reply!

I have checked but not deeply that I will check later . Afetr you reply, I have found that if we have a list then we need to save every item as a single document means if in a list 10 record then 10 documents add in mongo collection. I will try this and let you know if face any problem.

Also I will join free online course at MongoDB University .

Best regards,

Sunil Sharma

--

You received this message because you are subscribed to the Google Groups "mongodb-user"
group.

For other MongoDB technical support options, see: http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to a topic in the Google Groups "mongodb-user" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/mongodb-user/K3BBN0ypl4k/unsubscribe.
To unsubscribe from this group and all its topics, send an email to mongodb-user...@googlegroups.com.
To post to this group, send email to mongod...@googlegroups.com.
Visit this group at https://groups.google.com/group/mongodb-user.

To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/df31001a-18f7-4d5b-a6a9-e275409d1db2%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Thanks&Regards,

Sunil

gokul r

unread,

Feb 21, 2018, 5:53:17 AM2/21/18

to mongodb-user

Hi i want to insert a text file having json values of full cities of size 3.92 MB but it ends up with documentsize error

On Friday, 25 December 2015 21:27:38 UTC+5:30, Asya Kamsky wrote:

Wan Bachtiar

unread,

Mar 14, 2018, 10:45:05 PM3/14/18

to mongodb-user

Hi i want to insert a text file having json values of full cities of size 3.92 MB but it ends up with documentsize error

Hi Gokul,

If the text file contains JSON values, you may be able to utilise mongoimport to insert each JSON value as a single document in a collection.

If you have any further questions, please open a new thread with the following information: