getting writeError (mongo/util/concurrency/rwlock.h:204) during massive insert files to mongoDB with gridfs

[roman.k...@gmail.com]

Hi, All!

A have a python script witch takes a huge list of images and move them in a circle to mongoDB using gridfs (it stores files in two mongoDB collections, files and chunks).

After few thousands of successful transfers I am starting to get following error:

assertion C:\data\mci\77b70aa30ca8170c4ee9bee2ba42cf1f\src\src\mongo/util/concurrency/rwlock.h:204

I tried to debug it and it looks, that it becomes fail during chunk insert, but why this happens I don't know. 

I test script on my local mongoDB and here is a code on python, which I use

import pymongo
from pymongo import MongoClient
import gridfs

fs = gridfs.GridFS(db)

for image in images_list:

  with open(vers['path'], 'rb') as fileObj:
    try:
        # try:
        try:
            grid_in = fs.new_file(filename=image[1]['name'],
                                  metadata={"imageId": image[1]['_id'],
                                            "version": version})
            grid_in.write(fileObj)
            file_id = grid_in._id
        finally:
            grid_in.close()

  except pymongo.errors.WriteError as we:
    print('%s %s:%s' % (image[1]['_id'], we, vers['path']))
    grid_in.abort()
    continue

fileObj.closed



I also tried gridfs.put instead of new_file and write and also GridFSBucket.open_upload_stream and GridfsBuscket.upload_from_stream - result always the same. 
Since some moment I am getting error, which a mentioned above

Do you know why this happens and how can I solve it?

Or maybe you can at least answer following question:

- can be some problems because of inserting the same images few times? or images with same filename?
- can it be error because of access to mongoDB (lock mechanizm or smth) and how can I then change it to avoid any conflicts? 

If you have some other ideas, they all are welcome. 

Thank you!

[Asya Kamsky]

You might look in the mongod logs to see what it says.  It might be that you are out of some resource on the system.

Asya

--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.
 
For other MongoDB technical support options, see: https://docs.mongodb.com/manual/support/
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+unsubscribe@googlegroups.com.
To post to this group, send email to mongod...@googlegroups.com.
Visit this group at https://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/797c142a-1015-4fe0-82bb-a51ec8ad22ff%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Asya Kamsky
Lead Product Manager
MongoDB
Download MongoDB - mongodb.org/downloads
We're Hiring! - https://www.mongodb.com/careers