I'm using node-csv-parser to read in csv data and store it in mongodb using mongoose. However I'm trying to speed up the import, and I want to evaluate using the native save method exposed by
node-mongodb-native
, accessed in mongoose usingModel.collection
. (This at the advice of a mongo engineer I spoke to at office hours at Mongo HQ).
node-csv-parser
fires adata
event every time it reads in a new line of the csv. Inside this event I read in the line of data, create a new data point from it, and save it in mongo. I can save the data point just fine inside thedata
event using my mongoose modelTestDataPoint
. However if I try to instead create a javascript object for the data point and save it usingTestDataPoint.collection.save
, I get the error:RangeError: Maximum call stack size exceeded
.I've tried calling this native save in all sorts of different ways, including getting the collection directly through
mongoose.connection.db.collection("testdatapoints")
and by sending it out to a queue provided by the async module, but always with more or less the same result. I can successfully save a data point using the native driver elsewhere in my code, and even inside theend
event of the csv import, just not inside thedata
event.I have also determined through logging that on my current setup (Ubuntu 12.04 on a 64-bit AMD processor, 8 gb RAM), the code iterates through 154 lines of the csv before throwing the stack error, however no data is written to the db from inside this
data
event. It seems there is some kind of recursion happening(?) unintentionally, or perhaps it is some kind of bug between node-csv-parser and node-mongodb-native.To clarify, with my (edited/updated) sample code below, the log repeats 154 times:
about to call native save
just called native saveand then says:
in native save callback for dataPoint: 1
Native save failed, error:RangeError: Maximum call stack size exceeded
in native save callback for dataPoint: 2
Native save failed, error:RangeError: Maximum call stack size exceededup to dataPoint 154, then it repeats the "about to call/just called" another 154 times, then logs the errors for dataPoints 155-308 and so on (I have a lot of data points to import). This 154 number is pretty repeatable, thought I've observed it going through 155 lines at a time once or twice.
It was suggested that I wrap the save call in a
process.nextTick()
to clear the stack. When I try that, in my log I can see that the save is called 154 times, thenprocess.nextTick()
is called 154 times, then the RangeError is logged 154 times, and the sequence repeats.I'm running node 0.8.2, mongoose 2.7.2, and mongodb 2.0.4.
csv()
.fromPath(path)
.on("data", (data, index) ->
newDataPoint =
dataPoint: data[1]
testTime: data[2]/3600
cycleIndex: data[3]
console.log "about to call native save"
# TestDataPoint is my my mongoose model, which saves fine, but
# this call throws the RangeError: Maximum stack size exceeded
TestDataPoint.collection.save newDataPoint, safe:true, (err, dataPoint) ->
console.log "in native save callback for dataPoint: " + data[1]
if err
console.log "Native save failed, error:" + err
console.log "just called native save"
.on("end", (count) ->
newDataPoint =
dataPoint: 100 # dummy values
testTime: 200
cycleIndex: 300
# This call works, saves the data point
TestDataPoint.collection.save newDataPoint, safe:false, null
)
.on("error", (err) ->
console.log err
)I'm pretty much out of ideas and will gladly provide more detail if it will help figure out how to save natively.
--
--
http://mongoosejs.com
http://github.com/learnboost/mongoose
You received this message because you are subscribed to the Google
Groups "Mongoose Node.JS ORM" group.
To post to this group, send email to mongoo...@googlegroups.com
To unsubscribe from this group, send email to
mongoose-orm...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/mongoose-orm?hl=en
--