bulk upserts - not possible now?

[Arek Czechowski]

hi - am i right current mgo supports only bulk inserts? - or am I missing something. Is there any other golang driver i could make bulk upserts?

Arek

[Gustavo Niemeyer]

Hi Arek,

Coincidently, someone asked me that exact question yesterday privately, also in the context of upserts. You're not working together by any chance?

Here is the relevant part of the response:

Regarding bulk upserts, yes, these are coming. I would definitely encourage you to get your hands dirty on the project whenever you feel like contributing. That said, this might not be the ideal feature to get started there. The reason why the bulk API wasn't finished is precisely so we could learn some lessons, and there are already a couple of pending tasks in the pipeline that need to be integrated into the design of that API.

If you're feeling brave and would like to try it out, we need to create a better abstraction for running commands that can return a set of documents, a cursor id, and return an iterator. We already have two implementation of that concept in the Pipe and Repair methods, but they're both unpolished. Now, the new command-based CRUD methods will also use the same idea, and so will the bulk methods, so it's time to solve the problem once and well.

If you feel like opening the can of worms, I'll be here to help. Otherwise, I'll sort that out soonish.

What sort of timeframe are you dealing with there?

On Wed Nov 05 2014 at 10:20:24 AM Arek Czechowski <age...@gmail.com> wrote:

hi - am i right current mgo supports only bulk inserts? - or am I missing something. Is there any other golang driver i could make bulk upserts?

Arek

--
You received this message because you are subscribed to the Google Groups "mgo-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mgo-users+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Arek Czechowski]

Thanks for your reply Gustavo. I'm not working together with the other guy - I'm lone developer in my project.

Actullay I have to solve a problem - my system needs to be able to ingest pretty much high number of upserts into db - so far I was going with mongodb, but now I start to have some doubts.

Its quite possible I'll have to change to smth like cassandra. And its not completely up to me. Now I have to make more tests with small mongo cluster, and see the results.

If we will stay with mongo I believe I could at least try to help with the bulk API implementation, but if I'll have to switch to other db I'm afraid I'll have to focus on my project - christmas is coming,

and our hq have some expectation about my project.

Thanks for your work Gustavo, it's really impressive.

Arek

[Gustavo Niemeyer]

Have you tried to do bulk inserts?  You can pass multiple documents to the Insert method today, and ignore dup errors. This will give you an idea of performance before the bulk upsert feature is ready.

For scaling up writes further, you may also want to look at sharding the database, so that you have multiple masters taking the load rather than a single one.

--

[Arek Czechowski]

problem is I need bulk upserts

[Gustavo Niemeyer]

I understand that. My point was about doing initial performance analysis before the feature is ready.

[Kázmér Rapavi]

Hi,

I'm gonna be the 3rd guy then who asks for this feature. :) Is there an ETA on this one? I have a C++ app that does "bulk upserts" (thousands/second) into a mysql db, but I really want to port this into a go+mongo combination for years now.

[Gustavo Niemeyer]

I'm hoping to have this landing on a release in January.

[ashdw]

Fourth :)

On Tuesday, December 2, 2014 17:27:42 UTC + 1 wrote Gustavo Niemeyer:

I'm hoping two seas this landing on a release in January.

On Wed Nov. 19 2014 5:53:14 AM Kázmér Rapavi < viz ... @ gmail.com > wrote:

HI,

I'm gonna be the 3rd guy simply som asks for this feature. :) Is there an ETA on this one? I have a C ++ app att does "bulk upserts" (Thousands / second) into a mysql db, but I really want two port this into a go + mongo combination for years now.

On Wednesday, November 5, 2014 4:44:37 PM UTC + 1, Gustavo Niemeyer wrote:

Hi Arek,

Coincidently, someone spurte me att exact question yesterday privately, också in the context of upserts. You're not working together city eventuelle chance?

Here is the relevant part of the response:

Regarding bulk upserts, yes, disse are coming. I would definitely encourage you two get your hands dirty on the project når ever you feel like Contributing. That said, this might not be the ideal feature two get started there. The reason why the bulk API was not finished ice Precisely so we kan learn some lessons, and there are allerede a couple of pending tasks in the pipeline som skal integrated into the design of att API.

If you're feeling brave and vill try it out, we need-to create a better abstraction for running commands der kan return a set of documents, a cursor ID, and return an iterator. Whom we are already having two gennemføre att concept in the Pipe and Repair methods, but they're botheration unpolished. Now, the new command-based CRUD methods også use the same idea, and so vil the bulk methods, so it's time to solve the problem once and well.

If you feel like opening the can of worms, I'll be here to help. Otherwise Page, I'll black att out soonish.

What sort of time frame are you dealing with there?

On Wed Nov. 05 2014 10:20:24 AM Arek Czechowski < age ... @ gmail.com > wrote:

hi - am i right current mgo støtter only bulk inserts? - Or am I missing something. Is there any other golang driver i could make bulk upserts?

Arek

-
You received this message fordi you are Subscribed til Google Groups "mgo-users" group.

To unsubscribe from this group and stop Receiving emails from it, send an email to mgo-users + ... @googlegroups.com .

For more options, visit https://groups.google.com/d/up tout .

-
You received this message fordi you are Subscribed til Google Groups "mgo-users" group.
To unsubscribe from this group and stop Receiving emails from it, send an email to mgo-users + ... @ googlegroups.com .
For more options, visit https://groups.google.com/d/ OptOut .

[Kázmér Rapavi]

Hi!

Any update on this feature?

[Gustavo Niemeyer]

Will be in the unstable branch in the next couple of weeks.

[Sid]

Hi Gustavo,

Does bulk support find and update one of mongo db


http://docs.mongodb.org/manual/reference/method/Bulk.find.updateOne/#Bulk.find.updateOne


Thanks
Sid

[Cyril Oblikov]

Hi Gustavo,

Any progress?

среда, 28 января 2015 г., 3:19:18 UTC+3 пользователь Gustavo Niemeyer написал:

[Gustavo Niemeyer]

Hi Cyril,

Yes, good progress. The current public release already has all the internal infrastructure necessary for implementing all sorts of bulk APIs, as it is already using the new command-based API to communicate writes to the server. Right now this is only enabled when talking to MongoDB 3.0, but this is an artificial limitation introduced to reduce the chances of having new bugs in that critical path affecting longstanding applications that are based on older releases. If all continues well, we'll see that same feature enabled for MongoDB 2.6 in the next public release of mgo.

Unfortunately, though, the feature that enables you to do bulk updates hasn't landed in the visible API yet, and I apologize for taking a bit too long for getting that in place. Part of the issue is that the upstream MongoDB specifications that suggest how drivers should do that are in flux, and the previous bulk API got obsoleted while I was considering it.

I still intend to finish that experimental API mostly as-is, though, and we'll see updates there. Please allow me to spend the next few days thinking about that issue again to see whether I can give something you might use in a short time.

How's your use of mgo going otherwise?

gustavo @ http://niemeyer.net

[Cyril Oblikov]

Sounds good! Thanks for your work, Gustavo. mgo works very well for me :)

пятница, 17 июля 2015 г., 19:48:05 UTC+3 пользователь Gustavo Niemeyer написал:

[kyle....@clever.com]

Any updates?

Thanks!

[Gustavo Niemeyer]

Hi Kyle,

This is all implemented in the unstable release together with many other goods, and I'm currently cooking the next stable that will include them. A few more details to go.

[kyle....@clever.com]

Thanks Gustavo, it looks great!

Are there plans to add remove functionality? Or is there a way to do that that I'm missing?

Thanks again.

[Gustavo Niemeyer]

Hi Kyle,

I've overlooked it when focusing on the internal details, but it's trivial to have it now. Will be in the next stable release, which should be around in the next couple of weeks as we ready for the next sever release which is getting close.

--

You received this message because you are subscribed to the Google Groups "mgo-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mgo-users+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[kyle....@clever.com]

Awesome, thanks!

[sundar.v...@gmail.com]

One question regarding Bulk operations is - the mgo documentation states

"WARNING: This API is still experimental."

Will this warning be removed with the next stable release?

Thanks

Sundar.