mongoimport is not working properly while running using python script

[Praneeth Tvss.]

Hi everyone,
I am doing `mongoimport` with `upsert` option of a file using `python` script. While running the script, I am getting an error.

command = mongoimport --host <host:port> -u <username> -p <password> --authenticationDatabase admin -d <db_name> -c <collection name> --file myfile.json --upsertprocess = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, shell=True)
for data in process.stdout:
    print data

- When I run the above program, I am getting the following error:

connected to: <server>:27017
Failed: error processing document #165: unexpected EOF
imported 0 documents

 

- If I run the command through the command line, it is working fine.

$ mongoimport -u <username> -p <password> --host <server>:27017 -d <db name> -c <coll name> --authenticationDatabase admin --file myfile.json --upsert
2018-03-26T06:07:24.421+0200    connected to: <server>:27017
2018-03-26T06:07:26.222+0200    imported 166 documents

- I want to run this mongoimport using python script only. 
- Can anyone please help me with this issue ?.

Thanks,

Praneeth. 

[Praneeth Tvss.]

Sorry guys, miswritten the code snippet. The following is the correct ones:

command = mongoimport --host <host:port> -u <username> -p <password> --authenticationDatabase admin -d <db_name> -c <collection name> --file myfile.json --upsert
process = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, shell=True)
for data in process.stdout:
    print data

Thank you...

[Praneeth Tvss.]

Everyone,

Could you guys please help me on this. I was stuck here...

Thank you...

[Kevin Adistambha]

Hi Praneeth

I tried your exact code, and other than adding quotes on the command variable, it works as expected:

$ cat test.json
{"_id":0.0,"a":"Zulu"}
{"_id":1.0,"a":"Yankee"}
{"_id":2.0,"a":"November"}
{"_id":3.0,"a":"Whiskey"}
{"_id":4.0,"a":"Golf"}

$ cat test.py
import subprocess
command = 'mongoimport --host localhost:27017 -d test -c test --file test.json --upsert'
process = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, shell=True)
for data in process.stdout:
    print data

$ python test.py
2018-04-17T14:43:47.273+1000    connected to: localhost:27017

2018-04-17T14:43:47.332+1000    imported 5 documents

$ mongo --eval "db.test.find()"
MongoDB shell version v3.6.4
connecting to: mongodb://127.0.0.1:27017
MongoDB server version: 3.6.4
{ "_id" : 0, "a" : "Zulu" }
{ "_id" : 1, "a" : "Yankee" }
{ "_id" : 2, "a" : "November" }
{ "_id" : 3, "a" : "Whiskey" }
{ "_id" : 4, "a" : "Golf" }

The error you’ve seen seems to say that there’s some anomaly in your myfile.json in line 165 (Unexpected EOF).

I can reproduce the error you’re seeing in both Python and mongoimport if I remove the last } character:

$ cat test.json
{"_id":0.0,"a":"Zulu"}
{"_id":1.0,"a":"Yankee"}
{"_id":2.0,"a":"November"}
{"_id":3.0,"a":"Whiskey"}
{"_id":4.0,"a":"Golf"

$ mongoimport -d test -c test --upsert --file test.json
2018-04-17T14:49:36.814+1000    connected to: localhost
2018-04-17T14:49:36.815+1000    Failed: error processing document #5: unexpected EOF
2018-04-17T14:49:36.815+1000    imported 0 documents

Could you double check that your myfile.json contains well-formed JSON data and also free of unexpected invisible characters?

Best regards
Kevin

​

[Praneeth Tvss.]

Thanks a lot, Kevin!!!

But When I imported the same file from the command line, then it is not showing any error. All the documents are loaded to MongoDB.
When I include that in the script, it is showing error.
Why is this happening?

Thank you!!!

[Kevin Adistambha]

Hi Praneeth,

Are you using the same version of mongod and mongoimport? You can check by running mongod --version and mongoimport --version in the console.

Otherwise, can you verify that the import was done correctly using mongoimport, and the same documents are imported using the Python script?

Another possibility is that you have multiple mongoimport executable in your system, and you’re calling different ones from the Python script and the console. Try to modify your Python script to execute mongoimport --version to verify.

Best regards
Kevin

​

[michae...@aon.at]

I had the same problem. Turns out the file was still open in python :-)

Now it works.

[Kevin Adistambha]

Hi Michael

In terms of the ultimate goal of inserting data into the database, I think it’s probably easier to perform the operation from within Python using Pymongo (e.g. using either insert or bulk insert) rather than calling an external program from inside Python to do it.

Having said that, do you have a specific reason for using mongoimport from inside Python?

Best regards
Kevin

​

[Praneeth Tvss.]

Hi Kevin,

We will try using pymongo to insert data