Populate nested one-to-many associations

2,765 views
Skip to first unread message

Sávio Lucena

unread,
Oct 21, 2014, 9:32:21 AM10/21/14
to sai...@googlegroups.com
I understand that there is no built-in way in Sails.js/Waterline of populating deep nested associations, so I am trying to use bluebird promises to accomplish that but I'm running into a problem.

I'm successfully retrieving the user, and all the posts (populated with the images collection) associated with it (console.log shows me that everything is filled properly). However, when I override the property "post" of the user and try to assign the fully populated posts retrieved before, it does not fill properly the images property of Post.js. It is like the ORM is preventing the image collection of Post.js to be manually assigned.

What am I doing wrong? What is the best way of populating deep nested one-to-many associations? 

Bellow I've pasted all the code that I'm executing....

// Populate nested association
nested: function (req, res, next){
var username = req.param("id");

User
.findOneByUsername(username)
.populateAll()
.then(function (user){
var posts = Post.find({
"user": user.id
})
.populate('images')
.populate('category')
.then(function (posts){
return posts;
});
return [user, posts];
})
.spread(function (user, posts){
user.posts = posts; // This won't work.... It assigns all the fields properly but the images collection attribute
res.json(user);
}).catch(function (err){
if (err) return res.serverError(err);
});
}

// --- User.js Model --- //
module.exports = {
   attributes: {
        .....,
posts: {
collection: "post",
via: "user"
},
        .....
   }
}

// --- Post.js Model --- //
module.exports = {
    attributes: {
       ....,
       user: {
         model: "user"
       },
       images: {
         collection: "postImage",
         via: "post"
       },
       ....
    }
}

// --- PostImage.js Model --- //
module.exports = {

  attributes: {
    ....,
    post: {
  model: "post"
    }
  },
}

Regards,

Sávio Lucena
Message has been deleted

Gary Cox

unread,
Oct 22, 2014, 8:57:26 AM10/22/14
to sai...@googlegroups.com
Your "var posts" is actually a promise.  Try changing it up to this:

.then(function (user){
return Post.find({
"user": user.id
})
.populate('images')
.populate('category')
.then(function (posts){
return [user, posts];
});
})
.spread(function (user, posts){
user.posts = posts; // This won't work.... It assigns all the fields properly but the images collection attribute
res.json(user);
}).catch(function (err){
if (err) return res.serverError(err);
});

Sávio Lucena

unread,
Oct 22, 2014, 9:29:48 AM10/22/14
to Gary Cox, sai...@googlegroups.com
Gary,

Inside .spread() function I do have user and posts filled correctly, both promises have been resolved without any problems and those variables are populated.

What happens is when I’m assigning the posts variables to user.posts, everything is overwritten (as it is suppose to be) BUT the images collection attribute. As I said it before, it is like the ORM forbids manual  assignment of collection attributes.

Thank you for your time,

Sávio Lucena

Sávio Lucena

unread,
Oct 22, 2014, 9:31:48 AM10/22/14
to Gary Cox, sai...@googlegroups.com
Gary,

One more thing, I tried to change to the way you suggested and as I expected I’ve got the same results.

Thank you,

Sávio Lucena

Erk Struwe

unread,
Oct 22, 2014, 1:08:58 PM10/22/14
to sai...@googlegroups.com, gary.l...@gmail.com
I experienced this once before so I definitely +1 this. As I suspect, the solution (or reason) to this is obvious to many Sails devs but not obvious to us... :-( Would be great if someone could explain this... 

 

Sávio Lucena

unread,
Nov 6, 2014, 10:13:55 AM11/6/14
to Gary Cox, sai...@googlegroups.com
Any follow up on that issue?!

Thanks,

Sávio Lucena

On Oct 28, 2014, at 9:28 AM, Sávio Lucena <sav...@gmail.com> wrote:

Yeah, the more we take out of the equation the merrier.

I took out the afterDestroy, and got the same results as before. Still not working...

Thanks,

Sávio Lucena
 
On Oct 28, 2014, at 9:22 AM, Gary Cox <gary.l...@gmail.com> wrote:

I know thats aiming high, I just wanted to ensure that isn't the case.  Ill be sending this email chain to Mike who will be looking at the attempts we tried.

On Tue Oct 28 2014 at 7:21:12 AM Gary Cox <gary.l...@gmail.com> wrote:
Can you comment out the afterDestroy and give it a run?

On Tue Oct 28 2014 at 7:19:42 AM Sávio Lucena <sav...@gmail.com> wrote:
Hey Gary,

Thanks again for your effort.

I haven’t overridden the ToJson method.

As you requested, the ‘User’ model:

module.exports = {

  attributes: {
//Login attributes
email: {
type: "email",
unique: true
},
password: {
type: "string",
protected: true 
},
social: {
collection: "SocialLogin",
via: "user"
},
username: {
type: "string",
unique: true
},
fullName:  "string",
shortBio: "string",
image: {
model: "userImage"
},
paypalEmail: {
type: "email",
unique: true
},
timezone: "integer",
posts: {
collection: "post",
via: "user"
},
availability: "array",
appointments: "array", 
  },

  beforeCreate: function(user, cb) {  
  if (typeof user.password != "undefined"){
  bcrypt.genSalt(10, function(err, salt) {
     bcrypt.hash(user.password, salt, function(err, hash) {
       if (err) {
         return cb(err);
       }else{
         user.password = hash;                    
         cb(null, user);
       }
     });
   });
  } else{
  cb(null, user);
  }  
  },

  afterDestroy: function(users, cb){
    var count = 0;
    users.forEach(function (user, index, usersArr){
      UserImage.destroy({"user": user.id}).exec(function (err){
        if (err) return cb(err);
        Post.destroy({"user": user.id}).exec(function (err){
        if (err) return cb(err);
        if (++count == users.length) return cb(null, users);
        });        
      });
    });  
  }

};

Thanks,

Sávio Lucena

On Oct 28, 2014, at 8:59 AM, Gary Cox <gary.l...@gmail.com> wrote:

Can you send me the model for User? Im curious what your ToJson override looks like.  I showed Mike your issue last night and he suspected it might be the ToJson override on the model.



On Wed Oct 22 2014 at 2:53:22 PM Sávio Lucena <sav...@gmail.com> wrote:
Gary,

That is nice, where did you get that information from?! Is there a ETA on that function?!

Taking the populateAll() out ORM won’t let me assign to post, the same way that user.posts don’t let me assign user.posts.images.

There is the output that I get removing populateAll():

{
  • username: “jose.wellinson",
  • email: “jose...@gmail.com",
  • createdAt: "2014-10-22T01:09:10.500Z",
  • updatedAt: "2014-10-22T01:09:12.377Z",
  • id: 1,
  • availability: 
    [
    • {
      • weekday: 0,
      • from: "1970-02-01T15:00:00.637Z",
      • to: "1970-02-01T15:30:00.637Z"
      }
    ],
  • appointments: [ ]
} 

One interest thing that might give you or someone else any hints is that if I try to assign posts to any other attribute of user even a non pred-defined attribute (which is not mapped in the model), it works just fine….

Take a look when I assign to email attribute for instance:

{ username: 'savio.lucena',
  email: 
   [ { images: 
        [ { filename: '979d29f1-925e-43b5-94c3-d3825eb2f74f.JPG',
            original: [Object],
            type: 'image/jpeg',
            post: 1,
            createdAt: '2014-10-22T01:09:10.651Z',
            updatedAt: '2014-10-22T01:09:12.289Z',
            id: 1,
            normal: [Object],
            thumbnail: [Object] } ],
       category: 
        { text: '2D/3D & Video',
          createdAt: '2014-10-21T19:53:03.040Z',
          updatedAt: '2014-10-21T19:53:03.040Z',
          id: 1 },
       user: 1,
       title: 'Lesson2',
       price: 0,
       description: '',
       createdAt: '2014-10-22T01:09:10.584Z',
       updatedAt: '2014-10-22T01:09:10.584Z',
       id: 1 } ],
  createdAt: '2014-10-22T01:09:10.500Z',
  updatedAt: '2014-10-22T01:09:12.377Z',
  id: 1,
  availability: 
   [ { weekday: 0,
       from: '1970-02-01T15:00:00.637Z',
       to: '1970-02-01T15:30:00.637Z' } ],
  appointments: [] } 

Or even an attribute that is not defined in the User’s model:

{ username: 'savio.lucena',
  email: 'sav...@gmail.com',
  createdAt: '2014-10-22T01:09:10.500Z',
  updatedAt: '2014-10-22T01:09:12.377Z',
  id: 1,
  availability: 
   [ { weekday: 0,
       from: '1970-02-01T15:00:00.637Z',
       to: '1970-02-01T15:30:00.637Z' } ],
  appointments: [],
  newField: 
   [ { images: 
        [ { filename: '979d29f1-925e-43b5-94c3-d3825eb2f74f.JPG',
            original: [Object],
            type: 'image/jpeg',
            post: 1,
            createdAt: '2014-10-22T01:09:10.651Z',
            updatedAt: '2014-10-22T01:09:12.289Z',
            id: 1,
            normal: [Object],
            thumbnail: [Object] } ],
       category: 
        { text: '2D/3D & Video',
          createdAt: '2014-10-21T19:53:03.040Z',
          updatedAt: '2014-10-21T19:53:03.040Z',
          id: 1 },
       user: 1,
       title: 'Lesson2',
       price: 0,
       description: '',
       createdAt: '2014-10-22T01:09:10.584Z',
       updatedAt: '2014-10-22T01:09:10.584Z',
       id: 1 } ] }

So it seems like, fields that are defined by as collection in a model cannot be manually assigned.

Thank you,

Sávio
 


On Oct 22, 2014, at 3:53 PM, Gary Cox <gary.l...@gmail.com> wrote:

I can see what you need is coming soon

<Screen Shot 2014-10-22 at 1.53.09 PM.png>

On Wed, Oct 22, 2014 at 11:07 AM, Sávio Lucena <sav...@gmail.com> wrote:
I tried using cloneDeep like "user.posts = _.cloneDeep(posts)” but the results were even worse, user.posts came empty ( lol ).

After put some thought about what you told regarding reference issues, instead of cloning the posts, I tried to clone the user object, as it seems like it is forbidden this assignment, 

“var cloneuser = _.cloneDeep(user);
cloneuser.posts = posts;
res.json(cloneuser);”

and send it with res.json(). It worked but I lost all the instance methods of user model and with it some default features of sails, like attribute protection ( toJSON() ), which I intend to keep.

Does that behavior tell you something else that might help to properly solve this problem?!

Thank you,

Sávio


On Oct 22, 2014, at 12:30 PM, Gary Cox <gary.l...@gmail.com> wrote:

Yeah I noticed that.  I was thinking it could be a reference issue with js.  A slice would give you a new array, but that didnt appear to work either.  Another option you can try is using lodash's clone.  This has the ability to deep clone.


On Wed, Oct 22, 2014 at 10:03 AM, Sávio Lucena <sav...@gmail.com> wrote:
Still getting the same results.

As you can see from the log examples before, the assignment is actually being done, as the “category” attribute (one-to-one relationship) which had only the model_Id is being populated with the post fetched data.  But the “image” attribute (one-to-many relationship) of post is not being kept by the user.post.

What might that be?

Thank you,

Sáivo Lucena

On Oct 22, 2014, at 11:45 AM, Gary Cox <gary.l...@gmail.com> wrote:

I wonder if this is more of a reference issue, just for giggles, try this in your spread

user.posts = posts.slice();



On Wed, Oct 22, 2014 at 9:02 AM, Sávio Lucena <sav...@gmail.com> wrote:
Gary,

To illustrate the problem, I’ve added some console.log so you can see exactly what is happening and what I am looking for.

.then(function (user){
var posts = Post.find({
"user": user.id
})
.populate('images')
.populate('category')
.then(function (posts){
return posts;
});
return [user, posts];
})
.spread(function (user, posts){
console.log(user.toJSON()); // Console.log - [1] user object before the assignment
console.log(posts); // Console.log - [2] post object to be assigned
user.posts = posts;  
console.log(user.toJSON()); // Console.log - [3] user object after being assigned
res.json(user);
}).catch(function (err){
if (err) return res.serverError(err);
});


**************************************CONSOLES LOG**************************************
[1] - { social: [],
  posts: 
   [ { title: 'Lesson2',
       price: 0,
       category: 1,
       description: '',
       user: 1,
       createdAt: '2014-10-22T01:09:10.584Z',
       updatedAt: '2014-10-22T01:09:10.584Z',
       id: 1 } ],
  username: ‘jose.wellinson',
  email: ‘jose...@gmail.com',
  createdAt: '2014-10-22T01:09:10.500Z',
  updatedAt: '2014-10-22T01:09:12.377Z',
  id: 1,
  availability: 
   [ { weekday: 0,
       from: '1970-02-01T15:00:00.637Z',
       to: '1970-02-01T15:30:00.637Z' } ],
  appointments: [],
  image: undefined }

[2] - [ { images: 
     [ { filename: '979d29f1-925e-43b5-94c3-d3825eb2f74f.JPG',
         original: [Object],
         type: 'image/jpeg',
         post: 1,
         createdAt: '2014-10-22T01:09:10.651Z',
         updatedAt: '2014-10-22T01:09:12.289Z',
         id: 1,
         normal: [Object],
         thumbnail: [Object] } ],
    category: 
     { text: '2D/3D & Video',
       createdAt: '2014-10-21T19:53:03.040Z',
       updatedAt: '2014-10-21T19:53:03.040Z',
       id: 1 },
    user: 1,
    title: 'Lesson2',
    price: 0,
    description: '',
    createdAt: '2014-10-22T01:09:10.584Z',
    updatedAt: '2014-10-22T01:09:10.584Z',
    id: 1 } ]

[3] - { social: [],
  posts: 
   [ { title: 'Lesson2',
       price: 0,
       category: 
        { text: '2D/3D & Video',
          createdAt: '2014-10-21T19:53:03.040Z',
          updatedAt: '2014-10-21T19:53:03.040Z',
          id: 1 },
       description: '',
       user: 1,
       createdAt: '2014-10-22T01:09:10.584Z',
       updatedAt: '2014-10-22T01:09:10.584Z',
       id: 1 } ],
  username: ‘jose.wellinson',
  email: 'jose...@gmail.com',
  createdAt: '2014-10-22T01:09:10.500Z',
  updatedAt: '2014-10-22T01:09:12.377Z',
  id: 1,
  availability: 
   [ { weekday: 0,
       from: '1970-02-01T15:00:00.637Z',
       to: '1970-02-01T15:30:00.637Z' } ],
  appointments: [],
  image: undefined }

Thank you,

Sávio Lucena

On Oct 22, 2014, at 10:41 AM, Gary Cox <gary.l...@gmail.com> wrote:

Thanks man, I don't understand why waterline would re-query after you resolved the promise.  Makes no sense. 
--
Thank you,
Gary Cox




--
Thank you,
Gary Cox




--
Thank you,
Gary Cox




--
Thank you,
Gary Cox



Steffen Konerow

unread,
Nov 7, 2014, 6:49:47 AM11/7/14
to sai...@googlegroups.com
Sávio, I borrowed your code for a similar purpose and it works like a charm for me, using sails 0.10.5 - Did you perhaps override toJSON in your model to remove properties like posts or images?

Sávio Lucena

unread,
Nov 7, 2014, 8:47:30 AM11/7/14
to Steffen Konerow, sai...@googlegroups.com
Steffen,

As I said to Gary before, “unfortunately” I haven’t change anything related to any of the default attribute methods ( toJSON() / toObject() ).

I’ve just finished up a little debug with node-inspector, and I’m guessing what is deleting the post.images collection from its object is the default toObject() method:



I still haven’t figured out why is the "properties” variable is empty which is causing the if statement to be true along with the fact that “images” is a collection, but I hope someone out there can give me an answer sooner.

Thanks,

Sávio Lucena
Reply all
Reply to author
Forward
0 new messages