why mongorestore is applying one oplog entry at a time?

[vaibhav singh]

Hi,

We have observed that mongorestore is very slow while applying oplog for big deployments. while going through mongorestore commands code we found that it is applying one entry at a time.

Is there any specific for not using array of oplogs instead of one oplog entry at a time? 

code we are talking about is from repo https://github.com/mongodb/mongo-tools and file /mongorestore/oplog.go 

func (restore *MongoRestore) RestoreOplog() error {
	log.Logv(log.Always, "replaying oplog")
	intent := restore.manager.Oplog()
	if intent == nil {
	// this should not be reached
	log.Logv(log.Always, "no oplog file provided, skipping oplog application")
	return nil
	}
	if err := intent.BSONFile.Open(); err != nil {
	return err
	}
	if fileNeedsIOBuffer, ok := intent.BSONFile.(intents.FileNeedsIOBuffer); ok {
	fileNeedsIOBuffer.TakeIOBuffer(make([]byte, db.MaxBSONSize))
	}
	defer intent.BSONFile.Close()
	// NewBufferlessBSONSource reads each bson document into its own buffer
	// because bson.Unmarshal currently can't unmarshal binary types without
	// them referencing the source buffer
	bsonSource := db.NewDecodedBSONSource(db.NewBufferlessBSONSource(intent.BSONFile))
	defer bsonSource.Close()
	rawOplogEntry := &bson.Raw{}
	var totalOps int64
	var entrySize int
	oplogProgressor := progress.NewCounter(intent.BSONSize)
	if restore.ProgressManager != nil {
	restore.ProgressManager.Attach("oplog", oplogProgressor)
	defer restore.ProgressManager.Detach("oplog")
	}
	session, err := restore.SessionProvider.GetSession()
	if err != nil {
	return fmt.Errorf("error establishing connection: %v", err)
	}
	defer session.Close()
	for bsonSource.Next(rawOplogEntry) {
	entrySize = len(rawOplogEntry.Data)
	entryAsOplog := db.Oplog{}
	err = bson.Unmarshal(rawOplogEntry.Data, &entryAsOplog)
	if err != nil {
	return fmt.Errorf("error reading oplog: %v", err)
	}
	if entryAsOplog.Operation == "n" {
	//skip no-ops
	continue
	}
	if !restore.TimestampBeforeLimit(entryAsOplog.Timestamp) {
	log.Logvf(
	log.DebugLow,
	"timestamp %v is not below limit of %v; ending oplog restoration",
	entryAsOplog.Timestamp,
	restore.oplogLimit,
	)
	break
	}
	totalOps++
	oplogProgressor.Inc(int64(entrySize))
	err = restore.ApplyOps(session, []interface{}{entryAsOplog})
	if err != nil {
	return fmt.Errorf("error applying oplog: %v", err)
	}
	}
	if fileNeedsIOBuffer, ok := intent.BSONFile.(intents.FileNeedsIOBuffer); ok {
	fileNeedsIOBuffer.ReleaseIOBuffer()
	}
	log.Logvf(log.Info, "applied %v ops", totalOps)
	return nil
	}

[Dan Pasette]

The reason it was switched to apply one operation at a time is described here: https://jira.mongodb.org/browse/TOOLS-1365 and https://jira.mongodb.org/browse/SERVER-23326.

However, as noted in SERVER-23326, the server can detect where batches of CRUD ops can be applied, so I believe it is possible this optimization could be reintroduced if mongorestore is connected to a server version greater than 3.4.0.  Would you be willing to create a feature request in the TOOLS project?

Dan

--
You received this message because you are subscribed to the Google Groups "mongodb-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-dev+unsubscribe@googlegroups.com.
To post to this group, send email to mongo...@googlegroups.com.
Visit this group at https://groups.google.com/group/mongodb-dev.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-dev/52f8b544-ec41-4c1a-9455-2824da08c40c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.