dynamic matching for change streams

[krittin phornsiricharoenphant]

Hi,

I'm playing around this new feature and has a few questions.

1. Which model mongo use to reactive things? polling/push model.

if I need to watch on dynamic condition such as matching dynamic keys like this.

(For Nodejs driver)

var pipeline = [
  {
    $match: {
       $or: [ { 'fullDocument.key': { $in:  dynamicKeys } }, { operationType: 'delete' } ]
  }
]

const changeStream = coll.watch(pipeline, { fullDocument: 'updateLookup' }) // rewatch according to new keys when dynamicKeys changed

// unsubscribe old watch or wait a little longer then unsubscribe... merge 2 intersects sequence of changes

2. How can I collect all sequences of change without missing any? It's seem rewatch has no guarantee to not missing change notification from already watching keys.

Thank you so much.

[Wan Bachtiar]

1. Which model mongo use to reactive things? polling/push model.

Hi Krittin,

MongoDB drivers implement changeStream as an abstraction of a TAILABLE_AWAIT cursor. It’s an efficient pull model with periodic wait. If the cursor is at the end of the data, block for a while rather than returning no data. See also OP_QUERY.

if I need to watch on dynamic condition such as matching dynamic keys like this.

Could you elaborate further on the use case ? i.e. why would you require to match different values of keys dynamically ?

Based on your example snippet code, there are two things that you need to consider:

• By filtering on dynamic key, the application could have missed certain event because it is filtered out.
• There may be a case where update events, even with fullDocument:'updateLookup' option, won’t return the fullDocument field. This is because it returns the lookup version of the most current majority-committed version of the updated document. If there is an interleaving operation that would delete the document, the update event will not contain fullDocument. See also Lookup Full Document for Update Operations.

An alternative way that you could consider is to use a consistent field (i.e. _id, type, etc) for filtering, and ignore irrelevant events on the application side.

1. How can I collect all sequences of change without missing any? It’s seem rewatch has no guarantee to not missing change notification from already watching keys.

You can resume a Change Stream to continue from application’s last seen event. You would have to cache the resume token, and utilise resumeAfter parameter. Please note that Change streams are resumable, as long as the oplog has enough history to locate the last operation that the application received.

Regards,
Wan.

​