Re: [mongoose] Calling Model.collection.save() inside node-csv-parser: RangeError: Maximum call stack size exceeded

1,033 views
Skip to first unread message

Aaron Heckmann

unread,
Jul 20, 2012, 10:42:09 AM7/20/12
to mongoo...@googlegroups.com
Mongoose wraps the save in process.nextTick and casts each value appropriately first.

In the past I've triggered driver range errors by saving certain subclassed js types instead of first casting them to their native js version. have you tried saving empty docs? if that's ok i'd next try bringing each property back in one by one.


On Jul 20, 2012, at 6:10 AM, Eli Leland <e...@subwaylabs.com> wrote:

I'm using node-csv-parser to read in csv data and store it in mongodb using mongoose. However I'm trying to speed up the import, and I want to evaluate using the native save method exposed by node-mongodb-native, accessed in mongoose using Model.collection. (This at the advice of a mongo engineer I spoke to at office hours at Mongo HQ).

node-csv-parser fires a data event every time it reads in a new line of the csv. Inside this event I read in the line of data, create a new data point from it, and save it in mongo. I can save the data point just fine inside the data event using my mongoose model TestDataPoint. However if I try to instead create a javascript object for the data point and save it usingTestDataPoint.collection.save, I get the error: RangeError: Maximum call stack size exceeded.

I've tried calling this native save in all sorts of different ways, including getting the collection directly through mongoose.connection.db.collection("testdatapoints") and by sending it out to a queue provided by the async module, but always with more or less the same result. I can successfully save a data point using the native driver elsewhere in my code, and even inside the end event of the csv import, just not inside the data event.

I have also determined through logging that on my current setup (Ubuntu 12.04 on a 64-bit AMD processor, 8 gb RAM), the code iterates through 154 lines of the csv before throwing the stack error, however no data is written to the db from inside this data event. It seems there is some kind of recursion happening(?) unintentionally, or perhaps it is some kind of bug between node-csv-parser and node-mongodb-native.

To clarify, with my (edited/updated) sample code below, the log repeats 154 times:

about to call native save
just called native save

and then says:

in native save callback for dataPoint: 1
Native save failed, error:RangeError: Maximum call stack size exceeded
in native save callback
for dataPoint: 2
Native save failed, error:RangeError: Maximum call stack size exceeded

up to dataPoint 154, then it repeats the "about to call/just called" another 154 times, then logs the errors for dataPoints 155-308 and so on (I have a lot of data points to import). This 154 number is pretty repeatable, thought I've observed it going through 155 lines at a time once or twice.

It was suggested that I wrap the save call in a process.nextTick() to clear the stack. When I try that, in my log I can see that the save is called 154 times, then process.nextTick() is called 154 times, then the RangeError is logged 154 times, and the sequence repeats.

I'm running node 0.8.2, mongoose 2.7.2, and mongodb 2.0.4.

csv()
   
.fromPath(path)
   
.on("data", (data, index) ->

        newDataPoint
=
            dataPoint
: data[1]
            testTime
: data[2]/3600
            cycleIndex
: data[3]

        console
.log "about to call native save"

       
# TestDataPoint is my my mongoose model, which saves fine, but
       
# this call throws the RangeError: Maximum stack size exceeded
       
TestDataPoint.collection.save newDataPoint, safe:true, (err, dataPoint) ->
            console
.log "in native save callback for dataPoint: " + data[1]
           
if err
                console
.log "Native save failed, error:" + err

        console
.log "just called native save"

   
.on("end", (count) ->
        newDataPoint
=
            dataPoint
: 100            # dummy values
            testTime
: 200
            cycleIndex
: 300

       
# This call works, saves the data point
       
TestDataPoint.collection.save newDataPoint, safe:false, null
   
)
   
.on("error", (err) ->
       console
.log err
   
)

I'm pretty much out of ideas and will gladly provide more detail if it will help figure out how to save natively.

--
--
http://mongoosejs.com
http://github.com/learnboost/mongoose
You received this message because you are subscribed to the Google
Groups "Mongoose Node.JS ORM" group.
To post to this group, send email to mongoo...@googlegroups.com
To unsubscribe from this group, send email to
mongoose-orm...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/mongoose-orm?hl=en
 
 
 

Eli Leland

unread,
Jul 20, 2012, 2:39:48 PM7/20/12
to mongoo...@googlegroups.com
Aaron you nailed it. My sample code did not include a call where I grab an integer property from another Mongoose model object and make it a field in my dataPoint object (mea culpa for not including that in the sample). Wrapping that call in parseInt() fixed the problem.

Thanks very very much for your help.

Aaron Heckmann

unread,
Jul 25, 2012, 12:29:37 PM7/25/12
to mongoo...@googlegroups.com
Good to hear. In Mongoose 3x this problem should never occur since we're removing the subclasses Numbers.
--
Reply all
Reply to author
Forward
0 new messages