Failed uploading file in GridFS "assertion: 9008 filemd5 failed" using sharding

[Ivan]

Using 2 shard servers, 1 config servers, 1 mongos.
Trying to upload a file in gridfs get an error: 9008 filemd5 failed:

# mongofiles --port 27000 --db bench put static/small/app_arts.png
connected to: 127.0.0.1:27000
assertion: 9008 filemd5 failed

Used mogno v 1.6.0 and then 1.7.3. In logs no errors.
Log mongos (1.6.0):
Thu Dec 9 14:37:04 [conn6] autosplitting bench.fs.chunks size:
2506060 shard: ns:bench.fs.chunks at: shard0001:10.0.0.3:27017
lastmod: 11|6 min: { files_id: ObjectId('4d00bb349b9ef305474fda19'),
n: 37 } max: { files_id: MaxKey, n: MaxKey } on: { files_id:
ObjectId('4d00bf60a1a2f93746a166f3'), n: 3 }(splitThreshold 2359296)
Thu Dec 9 14:37:04 [conn6] config change: { _id:
"beta-2010-12-09T11:37:04-10", server: "beta", time: new
Date(1291894624457), what: "split", ns: "bench.fs.chunks", details:
{ before: { min: { files_id: ObjectId('4d00bb349b9ef305474fda19'), n:
37 }, max: { files_id: MaxKey, n: MaxKey } }, left: { min: { files_id:
ObjectId('4d00bb349b9ef305474fda19'), n: 37 }, max: { files_id:
ObjectId('4d00bf60a1a2f93746a166f3'), n: 3 } }, right: { min:
{ files_id: ObjectId('4d00bf60a1a2f93746a166f3'), n: 3 }, max:
{ files_id: MaxKey, n: MaxKey } } } }
Thu Dec 9 14:37:04 [conn6] autosplitting bench.fs.chunks size:
2359872 shard: ns:bench.fs.chunks at: shard0001:10.0.0.3:27017
lastmod: 12|7 min: { files_id: ObjectId('4d00bf60a1a2f93746a166f3'),
n: 3 } max: { files_id: MaxKey, n: MaxKey } on: { files_id:
ObjectId('4d00bf60a1a2f93746a166f3'), n: 11 }(splitThreshold 2359296)
Thu Dec 9 14:37:04 [conn6] config change: { _id:
"beta-2010-12-09T11:37:04-11", server: "beta", time: new
Date(1291894624487), what: "split", ns: "bench.fs.chunks", details:
{ before: { min: { files_id: ObjectId('4d00bf60a1a2f93746a166f3'), n:
3 }, max: { files_id: MaxKey, n: MaxKey } }, left: { min: { files_id:
ObjectId('4d00bf60a1a2f93746a166f3'), n: 3 }, max: { files_id:
ObjectId('4d00bf60a1a2f93746a166f3'), n: 11 } }, right: { min:
{ files_id: ObjectId('4d00bf60a1a2f93746a166f3'), n: 11 }, max:
{ files_id: MaxKey, n: MaxKey } } } }
Thu Dec 9 14:37:04 [conn6] autosplitting bench.fs.chunks size:
2359872 shard: ns:bench.fs.chunks at: shard0001:10.0.0.3:27017
lastmod: 13|8 min: { files_id: ObjectId('4d00bf60a1a2f93746a166f3'),
n: 11 } max: { files_id: MaxKey, n: MaxKey } on: { files_id:
ObjectId('4d00bf60a1a2f93746a166f3'), n: 19 }(splitThreshold 2359296)
Thu Dec 9 14:37:04 [conn6] config change: { _id:
"beta-2010-12-09T11:37:04-12", server: "beta", time: new
Date(1291894624520), what: "split", ns: "bench.fs.chunks", details:
{ before: { min: { files_id: ObjectId('4d00bf60a1a2f93746a166f3'), n:
11 }, max: { files_id: MaxKey, n: MaxKey } }, left: { min: { files_id:
ObjectId('4d00bf60a1a2f93746a166f3'), n: 11 }, max: { files_id:
ObjectId('4d00bf60a1a2f93746a166f3'), n: 19 } }, right: { min:
{ files_id: ObjectId('4d00bf60a1a2f93746a166f3'), n: 19 }, max:
{ files_id: MaxKey, n: MaxKey } } } }
Thu Dec 9 14:37:04 [conn6] autosplitting bench.fs.chunks size:
2369632 shard: ns:bench.fs.chunks at: shard0001:10.0.0.3:27017
lastmod: 14|9 min: { files_id: ObjectId('4d00bf60a1a2f93746a166f3'),
n: 19 } max: { files_id: MaxKey, n: MaxKey } on: { files_id:
ObjectId('4d00bf60a1a2f93746a166f3'), n: 27 }(splitThreshold 2359296)
Thu Dec 9 14:37:04 [conn6] config change: { _id:
"beta-2010-12-09T11:37:04-13", server: "beta", time: new
Date(1291894624552), what: "split", ns: "bench.fs.chunks", details:
{ before: { min: { files_id: ObjectId('4d00bf60a1a2f93746a166f3'), n:
19 }, max: { files_id: MaxKey, n: MaxKey } }, left: { min: { files_id:
ObjectId('4d00bf60a1a2f93746a166f3'), n: 19 }, max: { files_id:
ObjectId('4d00bf60a1a2f93746a166f3'), n: 27 } }, right: { min:
{ files_id: ObjectId('4d00bf60a1a2f93746a166f3'), n: 27 }, max:
{ files_id: MaxKey, n: MaxKey } } } }
Thu Dec 9 14:37:04 [conn6] autosplitting bench.fs.chunks size:
2359872 shard: ns:bench.fs.chunks at: shard0001:10.0.0.3:27017
lastmod: 15|10 min: { files_id: ObjectId('4d00bf60a1a2f93746a166f3'),
n: 27 } max: { files_id: MaxKey, n: MaxKey } on: { files_id:
ObjectId('4d00bf60a1a2f93746a166f3'), n: 35 }(splitThreshold 2359296)
Thu Dec 9 14:37:04 [conn6] config change: { _id:
"beta-2010-12-09T11:37:04-14", server: "beta", time: new
Date(1291894624584), what: "split", ns: "bench.fs.chunks", details:
{ before: { min: { files_id: ObjectId('4d00bf60a1a2f93746a166f3'), n:
27 }, max: { files_id: MaxKey, n: MaxKey } }, left: { min: { files_id:
ObjectId('4d00bf60a1a2f93746a166f3'), n: 27 }, max: { files_id:
ObjectId('4d00bf60a1a2f93746a166f3'), n: 35 } }, right: { min:
{ files_id: ObjectId('4d00bf60a1a2f93746a166f3'), n: 35 }, max:
{ files_id: MaxKey, n: MaxKey } } } }
Thu Dec 9 14:37:04 [conn6] end connection 127.0.0.1:47046

Log mongos (1.7.3):
Thu Dec 9 16:17:54 /root/mongodb/bin/mongos db version v1.7.3, pdfile
version 4.5 starting (--help for usage)
Thu Dec 9 16:17:54 git version:
fcb5790e7528a101a49c52e442ded0e2b7a043d6
Thu Dec 9 16:17:54 sys info: Linux domU-12-31-39-06-79-A1
2.6.21.7-2.ec2.v1.2.fc8xen #1 SMP Fri Nov 20 17:48:28 EST 2009 x86_64
BOOST_LIB_VERSION=1_41
Thu Dec 9 16:17:54 [mongosMain] waiting for connections on port 27000
Thu Dec 9 16:17:54 [Balancer] balancer myid: beta:27000
Thu Dec 9 16:17:54 [websvr] web admin interface listening on port
28000
Thu Dec 9 16:22:33 [mongosMain] connection accepted from
127.0.0.1:43051 #1
Thu Dec 9 16:22:33 [conn1] creating WriteBackListener for:
10.0.0.1:27017
Thu Dec 9 16:22:33 [conn1] creating WriteBackListener for:
10.0.0.3:27017
Thu Dec 9 16:22:33 [conn1] end connection 127.0.0.1:43051
Thu Dec 9 16:24:08 [mongosMain] connection accepted from
127.0.0.1:50350 #2
Thu Dec 9 16:24:08 [conn2] creating WriteBackListener for:
10.0.0.2:27001


Configuration OS:
Gentoo. Linux beta 2.6.34-xen-domU #1 SMP Tue Aug 17 13:38:18 MSD 2010
x86_64 AMD Athlon(tm) 64 X2 Dual Core Processor 5200+ AuthenticAMD GNU/
Linux

Configuration shard:
Server 1: mongod 10.0.0.1:27017 - shard0000 (mongod)
Server 2: mongod 10.0.0.3:27017 - shard0001 (mongod)
Server 3: 10.0.0.2:27000 - mongos (mongos)
Server 3: 10.0.0.2:27001 - Config Server (mongod)

> db.shards.find()
{ "_id" : "shard0000", "host" : "10.0.0.1:27017", "maxSize" :
NumberLong(10000) }
{ "_id" : "shard0001", "host" : "10.0.0.3:27017", "maxSize" :
NumberLong(10000) }

Init sharding (mongos --port 27000 admin):
db.runCommand( { addshard : "10.0.0.1:27017", maxSize: 10000 } );
db.runCommand( { addshard : "10.0.0.3:27017", maxSize: 10000 } );
db.runCommand( { enablesharding : "bench" } );
db.runCommand( { shardcollection : "bench.fs.chunks", key: {files_id:
1, n:1} } );

[Ivan]

Also there is a bug: after a failed upload file chunks are not
deleted.

> db.fs.files.count ()
3
> db.fs.chunks.count ()
5624

Then I run the command (to upload a file, 700 megabytes):
# mongofiles -port 27000 -d bench2 put s3.avi

connected to: 127.0.0.1:27000
assertion: 9008 filemd5 failed

> db.fs.files.count ()
3
> db.fs.chunks.count ()
8414

Similarly as in 1.6.0 and in 1.7.3
In logs no errors, only lines of type
Fri Dec 10 12:24:44 [Balancer] moving chunk ns: bench2.fs.chunks
moving (ns: bench2.fs.chunks at: shard0000: 10.0.0.1:27017 lastmod: 2
| 37 min: {files_id: ObjectId ( '4 d01f075e937a5e39125951e '), n: 665}
max: {files_id: ObjectId ('4 d01f075e937a5e39125951e'), n: 683})
shard0000: 10.0.0.1:27017 -> shard0001: 10.0.0.3:27017

[Eliot Horowitz]

You have to shard just on "files_id" not "files_id" , "n"

> --
> You received this message because you are subscribed to the Google Groups "mongodb-user" group.
> To post to this group, send email to mongod...@googlegroups.com.
> To unsubscribe from this group, send email to mongodb-user...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/mongodb-user?hl=en.
>
>

[Ivan]

That is, I can not split one file into multiple shards?

[Eliot Horowitz]

Currently, yes.
The problem is we md5 the file to verify the upload, but it doesn't
work across shards yet.