"ObjectId" in the results of _id property and "formatting problems" on the mongoexport to CSV file

[Marcello de Sales]

Hello Everyone,

I have generated 1M random items for my experiments using the Java
driver... I tried to use the export to CSV format and had "problems":

1. The _id column has the value "ObjectId(value)" enclosing the
value... I wanted only the "value" to be used...
2. The format of the list of items after trying to export to CSV is
not what I thought (using mongoexport)...

Here's an example of an item generated:

> db.SondeDataContainer.findOne()
{"_id" : ObjectId( "e46f40079365ac4a3f063400") ,
"sensor_ip_address" : "192.168.0.151" , "message_id" : "d8a7e831-
ddd0-4c8e-a152-5bcd9c18ef05" , "transaction_time" : 1252812168025 ,
"fact_time" : 1252812057000 , "data" : {"temperature" : "28.62" ,
"sp_condition" : "179.6" , "condition" : "76.8" , "resistence" :
"378.41" , "salinitude" : "0.09" , "pressure" : "0.86" , "depth" :
"1.125" , "ph" : "2.67" , "pH_mv" : "-50.2" , "odo_sat" : "2.8" ,
"odo_condition" : "65.37" , "turbidity" : "0.2" , "battery" : "7.0"}}

Exported to CSV in the following format with the command: mongoexport
--csv -f "_id,transaction_time,fact_time,data.temperature,
data.sp_condition,data.condition....data.battery"

Here's the output...

_id,transaction_time,fact_time,data
ObjectId( "e46f40070066ac4a00003f00" ),1252812168025,1252812119000,
{ "temperature" : "65.92", "sp_condition" : "70.6", "condition" :
"124.7", "resistence" : "2920.35", "salinitude" : "0.09", "pressure" :
"0.154", "depth" : "0.66", "ph" : "2.4", "pH_mv" : "-61.3",
"odo_sat" : "30.3", "odo_condition" : "18.95", "turbidity" : "0.1",
"battery" : "7.9" }
ObjectId( "e46f40070066ac4a00013f00" ),1252812168025,1252812119000,
{ "temperature" : "36.34", "sp_condition" : "84.1", "condition" :
"170.5", "resistence" : "2199.32", "salinitude" : "0.09", "pressure" :
"1.812", "depth" : "0.684", "ph" : "6.11", "pH_mv" : "-78.9",
"odo_sat" : "66.7", "odo_condition" : "11.27", "turbidity" : "0.3",
"battery" : "1.3" }

I'd like to have the actual names of the columns like
_id, transaction_time, fact_time, data.sp_condition, data.resistence,
data....., data.battery
"e46f40070066ac4a00003f00",1252812168025,1252812119000,65.92,70.6,
"124.7", ..., ...., "7.9"

thanks,

Marcello

[Eliot Horowitz]

Fixed: http://jira.mongodb.org/browse/SERVER-293

[Marcello de Sales]

Hi Eliot,

What has been fixed? The jira instance does not describe the problem and solutions... Please let us know what happened... Was it fixed as I described? What about the ObjectId column value for the _id objects?

Thanks,

Marcello

--
Best Regards,

            Marcello de Sales - Software Engineer
     (   (
     _)__)         M.S. in Computer Science Graduate Student
  .-"(    `-..-.
  |`-._` _.-'|  )     Department of Computer Science
  L    ""   J  /
  |  JEE5   |.'      San Francisco State University (SFSU)
.-J    Dev  F-.
(   `-.___.-'   )  San Francisco, California 94132
`-._________.-'
             Board Member of the SF Bay area ACM Group

          __     __)                               __
         (, /|  /|                  /) /)      (__/  )   /)
           / | / |  _   __  _   _  // // ___     /  _   //  _  _
        ) /  |/  |_(_(_/ (_(___(/_(/_(/_(_)   ) /  (_(_(/__(/_/_)_
       (_/  http://userwww.sfsu.edu/~msales  (_/

"Brick walls are not there to keep us out. The Brick Walls are there to give us a chance to show how badly we want something. The Brick Walls are there to stop the people who don't want it badly. They are there to stop OTHER people."~ Randy Pausch

[Eliot Horowitz]

The problem is that mongoexport didn't support embedded fields.
So didn't support things like: "data.sp_condition"
That was a bug and fixed.

I changed the output for ObjectId as well.

http://github.com/mongodb/mongo/commit/8d07d50c48f76617b1a43d9e96efd47c890f6e8e

On Sun, Sep 13, 2009 at 7:04 AM, Marcello de Sales

[Marcello de Sales]

Thanks Eliot for the clarifications! I will checkout the latest code and build mongoDB to have the changes...

Marcello

[Eliot Horowitz]

It will be in the Monday mornings nightly if you can wait for that.
Or if you tell me what platform you're on I can push that nightly now.

On Sun, Sep 13, 2009 at 7:15 AM, Marcello de Sales

[Marcello de Sales]

Thanks Eliot... I hadn't seen your email :S I already checked out the source-code for a Ubuntu LInux 32bits on a VirtualBox and started building it... However, if you can make that nightly build quicker :) (I have only 1.5GB available), then please let me know and I prefer to download it :)

[Eliot Horowitz]

Kicked off.
You can watch progress here: http://buildbot.mongodb.org/waterfall
You're looking for the nightly 32-bit linux to finish.

On Sun, Sep 13, 2009 at 7:48 AM, Marcello de Sales

[Marcello de Sales]

Cool... I finished building mine and will test it... However, I will also download the nightly build you suggested... Thanks Eliot...