MongoDB GridFS with sharding and files more than 256K in C++ driver.

[Anti Mavrodi]

Hi All !

Please help me resolve this problem.

If MongoDB work without sharding this code work fine with any file
size.

mongo::DBClientConnection* db_mongo;
db_mongo = new mongo::DBClientConnection();
db_mongo->connect("localhost");

string md5="md5value";
GridFS gfs = GridFS(*db_mongo, "db_name", "collection_name");
BSONObj gde = BSON("md5"<<md5);
GridFile gFile=gfs.findFile(gde);
string path="/files/"+gFile.getFilename();
gFile.write(tpath);

If MongoDB work with sharding, this code work, but if file size is
more than 256K, from base to disk stored only 256K.

Tell me please how to solve this problem.
Best Regards.

[Nat]

Can you try the same from mongofiles command line to see whether it works or not? Can you also list your shard key?

--
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To post to this group, send email to mongod...@googlegroups.com.
To unsubscribe from this group, send email to mongodb-user...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/mongodb-user?hl=en.

[Anti Mavrodi]

$ ./mongofiles list
connected to: 127.0.0.1

mongos> db.runCommand({ listShards : 1});
{
    "shards" : [
        {
            "_id" : "repl1",
            "host" : "repl1/xxxx:7100:0:29::58:27018,xxxx:7100:0:30::130:27018,xxxx:7100:0:30::131:27018,xxxx:7100:0:30::134:27018"
        },
        {
            "_id" : "repl2",
            "host" : "repl2/xxxx:7100:0:29::57:27018,xxxx:7100:0:30::137:27018,xxxx:7100:0:30::136:27018,xxxx:7100:0:30::135:27018"
        }
    ],
    "ok" : 1
}

mongos> db.printShardingStatus();
--- Sharding Status ---
  sharding version: { "_id" : 1, "version" : 3 }
  shards:
    {  "_id" : "repl1", "host" : "repl1/xxxx:7100:0:29::58:27018,xxxx:7100:0:30::130:27018,xxxx:7100:0:30::131:27018,xxxx:7100:0:30::134:27018" }
    {  "_id" : "repl2", "host" : "repl2/xxxx:7100:0:29::57:27018,xxxx:7100:0:30::137:27018,xxxx:7100:0:30::136:27018,xxxx:7100:0:30::135:27018" }
  databases:
    {  "_id" : "admin",  "partitioned" : false,  "primary" : "config" }
    {  "_id" : "db_name",",  "partitioned" : true,  "primary" : "repl2" }
        db_name.collection_name.chunks chunks:
                repl2    2
                repl1    1
            { "files_id" : { "$MinKey" : true } } -->> { "files_id" : ObjectId("4f2a667de4b039bd71b65c53") } on : repl2 { "t" : 2000, "i" : 1 }
            { "files_id" : ObjectId("4f2a667de4b039bd71b65c53") } -->> { "files_id" : ObjectId("4f2acc97e4b021e8ee6e2909") } on : repl2 { "t" : 1000, "i" : 3 }
            { "files_id" : ObjectId("4f2acc97e4b021e8ee6e2909") } -->> { "files_id" : { "$MaxKey" : true } } on : repl1 { "t" : 2000, "i" : 0 }
        storage.torrents.files chunks:
                repl2    2
            { "filename" : { "$MinKey" : true } } -->> { "filename" : "XXXFile" } on : repl2 { "t" : 1000, "i" : 1 }
            { "filename" : "XXXFile" } -->> { "filename" : { "$MaxKey" : true } } on : repl2 { "t" : 1000, "i" : 2 }
    {  "_id" : "test",  "partitioned" : false,  "primary" : "repl1" }
    {  "_id" : "dbs",  "partitioned" : false,  "primary" : "repl1" }

7 февраля 2012 г. 10:03 пользователь Nat <nat....@gmail.com> написал:

[Nat]

- Can you get the result of

   use db_name

   db.collection_name.chunks.find( { "files_id" :  ObjectId("ObjectId of the file that you had the problem") } )

- Did you try it with slaveOk on? If so, can you try turning it off?

[Anti Mavrodi]

Result is

mongos> db.collection_name.files.find()

{ "_id" : ObjectId("4f30e4e7e4b03e95e35a3d83"), "filename" : "XXX", "chunkSize" : NumberLong(262144), "length" : NumberLong(334363), "md5" : "6e21023e7a70d7ba5a63dc969ec65365", "contentType" : null, "uploadDate" : ISODate("2012-02-07T08:46:31.776Z"), "aliases" : null }

mongos> db.collection_name.chunks.find()

{ "_id" : ObjectId("4f30e4e7e4b03e95e35a3d84"), "files_id" : ObjectId("4f30e4e7e4b03e95e35a3d83"), "n" : 0, "data" : BinData(0,"here_is_binary_data")}

In mongo shell all OK. slaveOk no effect.

7 февраля 2012 г. 11:04 пользователь Nat <nat....@gmail.com> написал:

--

You received this message because you are subscribed to the Google Groups "mongodb-user" group.

To view this discussion on the web visit https://groups.google.com/d/msg/mongodb-user/-/hRqn2N5BQ8wJ.

[Anti Mavrodi]

C++ driver save first 256K to disk and then an error "chunk is empty!"

7 февраля 2012 г. 11:04 пользователь Nat <nat....@gmail.com> написал:

- Can you get the result of

--

You received this message because you are subscribed to the Google Groups "mongodb-user" group.

To view this discussion on the web visit https://groups.google.com/d/msg/mongodb-user/-/hRqn2N5BQ8wJ.

[Nat]

Can you try using GridFS::storeFile instead?

---

From: Anti Mavrodi <antim...@gmail.com>

Sender: mongod...@googlegroups.com

Date: Tue, 7 Feb 2012 12:15:30 +0300

To: <mongod...@googlegroups.com>

ReplyTo: mongod...@googlegroups.com

Subject: Re: [mongodb-user] MongoDB GridFS with sharding and files more than 256K in C++ driver.

[Anti Mavrodi]

No. storeFile puts the file reference by fileName into the db. 

I try save file FROM db to disk.

7 февраля 2012 г. 13:38 пользователь Nat <nat....@gmail.com> написал:

[Nat]

I mean did you use it when storing the file to mongodb? Based on what you provided, the second chunk of data seems to be missing.

---

From: Anti Mavrodi <antim...@gmail.com>

Sender: mongod...@googlegroups.com

Date: Tue, 7 Feb 2012 13:46:48 +0300

[Anti Mavrodi]

I save a file in a database with this Java application code

                 File uploadfile = new File(file_path);
                 DB db = mongoTemplate.getDB();
                 GridFS gridFS = new GridFS(db, "collection_name");
                 GridFSInputFile file = gridFS.createFile(uploadfile);
                 file.setFilename(uploadfile.getName());
                 file.save(); 

7 февраля 2012 г. 13:52 пользователь Nat <nat....@gmail.com> написал:

[Nat]

I see. I don't see the index detail, I guess you must have created a unique index on files_id probably? If so, that would explain your case. You would need to recreate the collection with proper index and shard key.

---

From: Anti Mavrodi <antim...@gmail.com>

Sender: mongod...@googlegroups.com

Date: Tue, 7 Feb 2012 13:02:58 +0200

[Anti Mavrodi]

Thanks ! I dropped all indexes from files and chunks and then created index with command

db.collection_name.chunks.ensureIndex({files_id:1, n:1}, {unique: true});


Now all work fine !

7 февраля 2012 г. 14:06 пользователь Nat <nat....@gmail.com> написал: