Aggregate query using Mongo java driver does not come back

76 views
Skip to first unread message

Ravindranath K

unread,
Feb 11, 2018, 4:18:26 PM2/11/18
to mongodb-user

I am running an aggregate query with an "out" stage using Mongo java driver which does not come back and keeps on running, however the collection name given in the "out" stage is created may be in 1 min. I don’t see any errors in the logs and it suggests that the query is completed successfully.
I see the same kind of behavior in one of the Mongo client Robo Mongo as well.

 

Looks to me like a strange behavior, any inputs on this would be much appreciated.

Asya Kamsky

unread,
Feb 12, 2018, 3:46:42 AM2/12/18
to mongodb-user
Could you provide some additional information, like what version of
MongoDB is this, what version of the driver, what exactly is the code
you are running in Java and can you recreate the same behavior using
just mongo shell with the same aggregation?
> --
> You received this message because you are subscribed to the Google Groups
> "mongodb-user"
> group.
>
> For other MongoDB technical support options, see:
> https://docs.mongodb.com/manual/support/
> ---
> You received this message because you are subscribed to the Google Groups
> "mongodb-user" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to mongodb-user...@googlegroups.com.
> To post to this group, send email to mongod...@googlegroups.com.
> Visit this group at https://groups.google.com/group/mongodb-user.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/mongodb-user/89cbc43a-31a0-49ee-861f-ac70fd91d154%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.



--
Asya Kamsky
Lead Product Manager
MongoDB
Download MongoDB - mongodb.org/downloads
We're Hiring! - https://www.mongodb.com/careers

Ravindranath K

unread,
Feb 12, 2018, 12:16:39 PM2/12/18
to mongodb-user
MongoDB - 3.4.10
Mongo Java Driver - 3.4.2
Robo 3T - 1.1.1

I am  basically trying to join data in 4 different collections (customer, product, transaction, service-request).
I tried in Mongo shell and it is working fine there.

Below query works fine when the volume is low, however when volume is high I have this issue.

customer - 1 million
product - 2 million
transaction - 5 million
service-request - 2 million

Java code

List<Bson> aggPipeline = Arrays.asList(
 
Aggregates.unwind("$XRef"),
 
Aggregates.lookup("product", "XRef.pkey", "rkey", "products"),
 
Aggregates.group(new Document("mid", "$mid").append("Customer", "$Customer"), Accumulators.addToSet("XRef", "$XRef"), Accumulators.addToSet("products", "$products")),
 
Aggregates.unwind("$products", new UnwindOptions().preserveNullAndEmptyArrays(true)),
 
Aggregates.unwind("$products", new UnwindOptions().preserveNullAndEmptyArrays(true)),
 
Aggregates.lookup("transaction", "products.pkey", "rkey", "products.transactions"),
 
Aggregates.lookup("service-request", "products.pkey", "rkey", "products.srs"),
 
Aggregates.group(new Document("mid", "$_id.mid").append("Customer", "$_id.Customer").append("XRef", "$XRef"), Accumulators.addToSet("products", "$products")),
 
Aggregates.project(new Document("_id", "$_id.mid").append("mid", "$_id.mid").append("Customer", "$_id.Customer").append("XRef", "$_id.XRef").append("products", "$products")),
 
Aggregates.out("customer_transactions_insert")
 
);

customerInsertStgCollection
.aggregate(aggPipeline).allowDiskUse(aggregationDiskUse).toCollection();


Sample document in customer collection

{
   
"mid" : "1",
   
"Customer" : {
       
"source_id" : "CRM",
       
"customer_id" : 2,
       
"first_name" : "ANASTASIA",
       
"middle_name" : "",
       
"last_name" : "Deleon",
       
"dob" : ISODate("1826-01-01T00:00:00.000Z"),
       
"gender" : "Male",
       
"address" : [
           
{
               
"source_id" : "CRM",
               
"address_id" : 2,
               
"address_type" : "Office",
               
"address1" : "27652 East Armenia Ln.",
               
"address2" : "dMtzrNFqjL",
               
"address3" : "ZcGoPTNZrE",
               
"city" : "St. Petersburg",
               
"state" : "MT",
               
"country" : "US",
               
"zipcode" : 5136
           
}
       
]
   
},
   
"XRef" : [
       
{
           
"SSID" : 1,
           
"ID" : "2",
           
"pkey" : "1-2"
       
}
   
]
}

Sample product document

{
   
"product_id" : "3071",
   
"issue_date" : ISODate("2016-03-06T00:00:00.000Z"),
   
"valid_till_date" : Date(253370764800000),
   
"account_status" : "open",
   
"customer_id" : "2",
   
"SSID" : 1,
   
"pkey" : "1-3071",
   
"rkey" : "1-2"
}

Sample transaction document

{
   
"transaction_id" : "3",
   
"customer_id" : "62",
   
"product_id" : "670",
   
"transaction_amount" : 2975.0,
   
"transaction_date" : ISODate("2016-09-02T00:00:00.000Z"),
   
"SSID" : 1,
   
"pkey" : "1-3",
   
"rkey" : "1-670"
}

Sample service-request document

{
    "sr_id" : "2602",
   
"customer_id" : "2",
   
"product_id" : "3071",
   
"sr_open_date" : ISODate("2016-04-06T00:00:00.000Z"),
   
"sr_close_date " : ISODate("2016-04-06T00:00:00.000Z"),
   
"sr_type" : "Complaint",
   
"sr_cust_feedback" : "the complaint was addressed in timely manner to my satisfaction",
   
"SSID" : 2,
   
"pkey" : "2-2602",
   
"rkey" : "1-3071"
}

Aggregate query from Robo 3T

db.getCollection('customer').aggregate([
{
    $unwind
: "$XRef"
},
{
    $lookup
: {
       
from: "product",
        localField
: "XRef.pkey",
        foreignField
: "rkey",
       
as: "products"
   
}
},
{
    $group
: {
        _id
: {mid: "$mid", Customer: "$Customer"},
       
XRef: {$addToSet: "$XRef"},
        products
:{$addToSet: "$products"}
   
}
},
{
    $unwind
: {
       
"path": "$products",
       
"preserveNullAndEmptyArrays": true
   
}
},
{
    $unwind
: {
       
"path": "$products",
       
"preserveNullAndEmptyArrays": true
   
}
},
{
    $lookup
: {
       
from: "transaction",
        localField
: "products.pkey",
        foreignField
: "rkey",
       
as: "products.transactions"
   
}
},
{
    $lookup
: {
       
from: "service-request",
        localField
: "products.pkey",
        foreignField
: "rkey",
       
as: "products.srs"
   
}
},
{
    $project
: {
        mid
: "$_id.mid",
       
Customer: "$_id.Customer",
       
XRef: "$XRef",
        products
: "$products",
        _id
: 0
   
}
},
{
    $out
: "cust_tran"
}
],
{
    allowDiskUse
: true
}
)

Ravindranath Kundena

unread,
Feb 17, 2018, 7:50:02 PM2/17/18
to mongod...@googlegroups.com

Any update on this please.


To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+unsubscribe@googlegroups.com.

To post to this group, send email to mongod...@googlegroups.com.
Visit this group at https://groups.google.com/group/mongodb-user.

Asya Kamsky

unread,
Feb 20, 2018, 3:11:21 PM2/20/18
to mongodb-user
If this works in mongo shell then it will work in Java driver.

You said
> I am basically trying to join data in 4 different collections (customer, product, transaction, service-request).
> I tried in Mongo shell and it is working fine there.
> Below query works fine when the volume is low, however when volume is high I have this issue.

When you run it in mongo shell, it comes back or it doesn't same as in
Java driver?

Asya
> --
> You received this message because you are subscribed to the Google Groups
> "mongodb-user"
> group.
>
> For other MongoDB technical support options, see:
> https://docs.mongodb.com/manual/support/
> ---
> You received this message because you are subscribed to the Google Groups
> "mongodb-user" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to mongodb-user...@googlegroups.com.
> To post to this group, send email to mongod...@googlegroups.com.
> Visit this group at https://groups.google.com/group/mongodb-user.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/mongodb-user/CA%2BDEm-e%2B2t%3DaWwxo-4Cgbs8D2UUa37RUbYxWCFWzMqdwB%2BJc2A%40mail.gmail.com.

Ravindranath Kundena

unread,
Feb 24, 2018, 5:44:08 AM2/24/18
to mongod...@googlegroups.com

Hi Asya,

Query runs perfectly fine in Mongo shell and the control comes back whereas the same query has issues when run using Java driver.

Ravi



>> To post to this group, send email to mongod...@googlegroups.com.
>> Visit this group at https://groups.google.com/group/mongodb-user.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/mongodb-user/34acc457-0437-49fb-a90d-5575a9632c92%40googlegroups.com.
>> For more options, visit https://groups.google.com/d/optout.
>
> --
> You received this message because you are subscribed to the Google Groups
> "mongodb-user"
> group.
>
> For other MongoDB technical support options, see:
> https://docs.mongodb.com/manual/support/
> ---
> You received this message because you are subscribed to the Google Groups
> "mongodb-user" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> To post to this group, send email to mongod...@googlegroups.com.
> Visit this group at https://groups.google.com/group/mongodb-user.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/mongodb-user/CA%2BDEm-e%2B2t%3DaWwxo-4Cgbs8D2UUa37RUbYxWCFWzMqdwB%2BJc2A%40mail.gmail.com.
>
> For more options, visit https://groups.google.com/d/optout.



--
Asya Kamsky
Lead Product Manager
MongoDB
Download MongoDB - mongodb.org/downloads
We're Hiring! - https://www.mongodb.com/careers

--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.

For other MongoDB technical support options, see: https://docs.mongodb.com/manual/support/
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+unsubscribe@googlegroups.com.

To post to this group, send email to mongod...@googlegroups.com.
Visit this group at https://groups.google.com/group/mongodb-user.

Asya Kamsky

unread,
Apr 15, 2018, 11:00:45 AM4/15/18
to mongodb-user
Sorry I missed this thread, but the aggregation in Java does not look
the same as aggregation in shell.

Just look at the stages, Java has after two lookup stages group and
project, where the shell has just project. It doesn't look like the
same thing.

Asya
>> >> https://groups.google.com/d/msgid/mongodb-user/34acc457-0437-49fb-a90d-5575a9632c92%40googlegroups.com.
>> >> For more options, visit https://groups.google.com/d/optout.
>> >
>> > --
>> > You received this message because you are subscribed to the Google
>> > Groups
>> > "mongodb-user"
>> > group.
>> >
>> > For other MongoDB technical support options, see:
>> > https://docs.mongodb.com/manual/support/
>> > ---
>> > You received this message because you are subscribed to the Google
>> > Groups
>> > "mongodb-user" group.
>> > To unsubscribe from this group and stop receiving emails from it, send
>> > an
>> > email to mongodb-user...@googlegroups.com.
>> > To post to this group, send email to mongod...@googlegroups.com.
>> > Visit this group at https://groups.google.com/group/mongodb-user.
>> > To view this discussion on the web visit
>> >
>> > https://groups.google.com/d/msgid/mongodb-user/CA%2BDEm-e%2B2t%3DaWwxo-4Cgbs8D2UUa37RUbYxWCFWzMqdwB%2BJc2A%40mail.gmail.com.
>> >
>> > For more options, visit https://groups.google.com/d/optout.
>>
>>
>>
>> --
>> Asya Kamsky
>> Lead Product Manager
>> MongoDB
>> Download MongoDB - mongodb.org/downloads
>> We're Hiring! - https://www.mongodb.com/careers
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "mongodb-user"
>> group.
>>
>> For other MongoDB technical support options, see:
>> https://docs.mongodb.com/manual/support/
>> ---
>> You received this message because you are subscribed to the Google Groups
>> "mongodb-user" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to mongodb-user...@googlegroups.com.
>> To post to this group, send email to mongod...@googlegroups.com.
>> Visit this group at https://groups.google.com/group/mongodb-user.
>> To view this discussion on the web visit
> --
> You received this message because you are subscribed to the Google Groups
> "mongodb-user"
> group.
>
> For other MongoDB technical support options, see:
> https://docs.mongodb.com/manual/support/
> ---
> You received this message because you are subscribed to the Google Groups
> "mongodb-user" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to mongodb-user...@googlegroups.com.
> To post to this group, send email to mongod...@googlegroups.com.
> Visit this group at https://groups.google.com/group/mongodb-user.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/mongodb-user/CA%2BDEm-ct-1K-UMPi%3DtDtMSRVVkQCVoTivxAGA%3DYFBX6rgnnnyg%40mail.gmail.com.
Reply all
Reply to author
Forward
0 new messages