Extract data from mongo collection using aggregation ( $project )

[Nezar Oussama]

Hello,
I've created pipeline who extracts data from Mongo (source) then write the result in big query sink.
In mongodb query I'm using this query: {"$project": {"_id": 1, "alfa_id2": "$alfa_id"}}
Include _id and rename alfa_id to alfa_id2 I'm getting this error: unknown operator: $project, I've tried multiple syntaxes like:

``[{"$project": {"_id": 1, "alfa_id2": "$alfa_id"}}] error : unknown operator: $project

[ { "_id": 1, "alfa_id2": "$alfa_id" } ] no error but i have nothing in output 0 rows.

{ "_id": 1, "alfa_id2": "$alfa_id" } no error but i have nothing in output 0 rows.

db.getCollection('alfa').aggregate([{$project: {'_id': 1,'alfa_id2': '$alfa_id'}}]) error can't use db.getCollection('alfa') ...`` and others

Anyone who  has had this type of problem ? Is there another way to use aggregation in mongodb source query pls?

Thank you !

[Vitalii Tymchyshyn]

CDAP uses Mongo Hadoop support to read big data from MongoDB.

The Mongo Hadoop driver is using "find" call (see https://github.com/mongodb/mongo-hadoop/blob/20208a027ad8638e56dfcf040773f176d6ee059f/core/src/main/java/com/mongodb/hadoop/input/MongoInputSplit.java#L281).

This call does not support aggregations (see https://www.mongodb.com/docs/drivers/java/sync/current/fundamentals/aggregation/).

To add to that, it's usually tricky to support aggregations in big data as aggregations require preprocessing and thus are often not splittable for big data processing.

Best regards, Vitalii Tymchyshyn