Insert binary file error about use GridFs.

[Archas]

The source code: 

# -*- coding: utf-8 -*-

from __future__ import with_statement

import os,sys,shutil,threading

from time import ctime

from pymongo import Connection

from gridfs import *

def read():

    conn = Connection("2.2.2.13",30000)

    db = conn['virus']

    GridFS(db,'fs')        

    mygfs = GridFS(db,'fs')  

    tmpgfs = db ['fs.files']

    dir = "/pyApp/insert/files"

    n = 0

    

    while n < 100:

        for parent, dirnames, filenames in os.walk(dir):

            for filename in filenames:

                filePath = os.path.join(parent, filename)                                   

                with open(filePath, "rb") as file:

                    mygfs.put(file.read())

                    n+=1

                    

def main():

    print 'start at:', ctime()

    read()

    print 'All done at:', ctime()

    

if __name__ == '__main__':

    main()   

    

The error log

Traceback (most recent call last):

  File "D:\pyApp\PutInto\T-Test\i3.py", line 38, in <module>

    main()

  File "D:\pyApp\PutInto\T-Test\i3.py", line 33, in main

    read()

  File "D:\pyApp\PutInto\T-Test\i3.py", line 24, in read

    mygfs.put(file.read())

  File "C:\Python27\lib\site-packages\gridfs\__init__.py", line 116, in put

    grid_file.close()

  File "C:\Python27\lib\site-packages\gridfs\grid_file.py", line 220, in close

    self.__flush()

  File "C:\Python27\lib\site-packages\gridfs\grid_file.py", line 202, in __flush

    root=self._coll.name)["md5"]

  File "C:\Python27\lib\site-packages\pymongo\database.py", line 340, in command

    msg, allowable_errors)

  File "C:\Python27\lib\site-packages\pymongo\helpers.py", line 126, in _check_command_response

    raise OperationFailure(msg % response["errmsg"])

pymongo.errors.OperationFailure: command SON([('filemd5', ObjectId('4f9919e83e06fa10040002a0')), ('root', u'fs')]) failed: exception: chunks out of order

 

The last sentence.....

[Eliot Horowitz]

Did you shard gridfs? If so, what shard key?

If its not sharded, can you send the indexes on the files and chunks
collections?

> --
> You received this message because you are subscribed to the Google Groups
> "mongodb-user" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/mongodb-user/-/qbmkT0x1ASkJ.
> To post to this group, send email to mongod...@googlegroups.com.
> To unsubscribe from this group, send email to
> mongodb-user...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/mongodb-user?hl=en.

[Archas]

fs.chunks is be sharded, and the key is file_id;   the fs.files is not be sharded;

the fs.chunks indexes is: file_id;

在 2012年4月27日星期五UTC+8下午12时01分36秒，Eliot Horowitz写道：

> mongodb-user+unsubscribe@googlegroups.com.

[Eliot Horowitz]

I've only seen this when sharding on something else for fs.files.
You sure its file_id?

Can you send db.printShardingStatus()

>> > mongodb-user...@googlegroups.com.

>> > For more options, visit this group at
>> > http://groups.google.com/group/mongodb-user?hl=en.
>

> --
> You received this message because you are subscribed to the Google Groups
> "mongodb-user" group.
> To view this discussion on the web visit

> https://groups.google.com/d/msg/mongodb-user/-/ZuQSLirajVgJ.

>
> To post to this group, send email to mongod...@googlegroups.com.
> To unsubscribe from this group, send email to

> mongodb-user...@googlegroups.com.

[Archas]

mongos> db.printShardingStatus()

--- Sharding Status --- 

  sharding version: { "_id" : 1, "version" : 3 }

  shards:

{  "_id" : "s1",  "host" : "shard1/8.8.8.22:27017,8.8.8.23:27017" }

{  "_id" : "s2",  "host" : "shard2/8.8.8.22:27018,8.8.8.23:27018" }

{  "_id" : "s3",  "host" : "shard3/8.8.8.22:27019,8.8.8.23:27019" }

  databases:

{  "_id" : "admin",  "partitioned" : false,  "primary" : "config" }

{  "_id" : "virus",  "partitioned" : true,  "primary" : "s1" }

virus.fs.chunks chunks:

s2 3450

s1 3450

s3 3456

too many chunks to print, use verbose if you want to force print

{  "_id" : "db",  "partitioned" : false,  "primary" : "s1" }

[Archas]

[Archas]

[Greg Studer]

Unfortunately there were too many chunks to print individually, so
there wasn't any shard key info listed. Can you send the output of :

mongo <mongos>
> use config
> db.collections.find()

Which should have the info.

[Archas]

mongos> use config

switched to db config

mongos> db.collections.find()

{ "_id" : "virus.fs.chunks", "lastmod" : ISODate("1970-01-16T11:28:32.773Z"), "dropped" : false, "key" : { "files_id" : 1 }, "unique" : false }

[Greg Studer]

Are you still getting this error, and is it only for certain files?
If you're still seeing the error, there should be a corresponding
"should have chunk: " message in the mongod logs - it would be good to
see this message. Do these errors occur if you turn off balancing?

[Archas]

I'm still get this error.

The mongos' log has no error info.

This error appears because of mongodb or pymongo, I can not ensure .

The shard of mongodb has no problem

The insert code is tested by other mongodb

so, I'm befuddlement.