Revised many to many problem, even stranger

[whostheJBoss]

After severe headaches, I set up a new, clean site using the post
example from earlier. (no comments or multiple manytomany this time,
the problem runs deeper, it appears)

Here is the Transfer config:

<package name="users">
<object name="user" table="users"
decorator="components.decorators.users">
<id name="userID" type="numeric" />
<property name="email" type="string" column="email"
nullable="false" />

<onetomany name="posts" lazy="true" proxied="true">
<link to="posts.post" column="userID"/>
<collection type="array">
</collection>
</onetomany>
</object>
</package>

<package name="posts">
<object name="post" table="posts">
<id name="postID" type="numeric" />
<property name="postTitle" type="string"
column="postTitle" nullable="false" />
<manytomany name="tags" table="posttags"
proxied="true" lazy="true">
<link to="posts.post" column="postID"/>
<link to="tags.tag" column="tagID"/>
<collection type="array">
</collection>
</manytomany>
</object>
</package>

<package name="tags">
<object name="tag" table="tags"
decorator="components.decorators.tags">
<id name="tagID" type="numeric" />
<property name="tagTitle" type="string"
column="tagTitle" nullable="false" />
</object>
</package>

I have one handler with one method:

<!--- // default method --->
<cffunction name="default" access="public" returntype="void"
output="false" cache="true">
<cfargument name="Event" type="coldbox.system.beans.requestContext">
<cfset var rc = event.getCollection()>

<cfscript>
//get user
user = instance.Transfer.get("users.user", 1);
rc.user = user;

//get posts array from user
rc.myPosts = user.getPostsArray();

//get post
post = instance.Transfer.get("posts.post", 1);

//get tag
tag = instance.Transfer.get("tags.tag", 1);

//add tag to post
post.addTags(tag);

//add post to request collection
rc.post = post;

//save post
instance.Transfer.save(post);

</cfscript>

<cfset Event.setLayout(rc.viewPrefix & "Layout.Default")>

</cffunction>

My database is:

[posts]
postID
userID
postTitle

[users]
userID
email

[tags]
tagID
tagTitle

[posttags]
postTagID
postID
tagID

Using the exact code above, my tags are not added to the database, and
the dump below shows no tags in the array:

<cfdump var="#rc.post.getTagsArray()#">

However, if I comment out this line to be:

// instance.Transfer.save(post);

The dump of <cfdump var="#rc.post.getTagsArray()#"> shows the new tag
added to the array.

As soon as I try to save the object, though, the tags are gone and not
saved to the database.

Now, if I comment out this line to be:

// rc.myPosts = user.getPostsArray();

And then uncomment the save line again to be:

instance.Transfer.save(post);

Then the save works and the tags are added to the database and still
in the rc.post object after save. So, just calling getPostsArray()
causes any subsequent tags added to post not to save. They are added
to the object correctly, but as soon as save() is called they are
dropped from the array and not saved.

Alternatively, if I create the post object before I call getPostsArray
():

//get post
post = instance.Transfer.get("posts.post", 1);

//get posts array from user
rc.myPosts = user.getPostsArray();

Then the save works and the tags show up.

So, the behavior is:

Tags will not save or stay added to the post object after a
Transfer.save(post) if user.getPostsArray() has ever been called
before the post object has been created.

These are the only files in my clean site.

The problem here is that I want to get a list of posts, so I need to
call getPostsArray(), then I want to click to go to a form to edit the
post and add tags. The problem comes when I submit the form. The tags
are not saved since I had called getPostsArray() first to get the list
of posts on the page before. It doesn't matter if I call getPostsArray
() in the same handler, as in my example above, or on a completely
different handler. Once this method has been called at all, my tags
will not stay added to the object after a save().

Perhaps I'm again missing something glaring or a ColdBox / Transfer
setting, but as of right now, this is odd behavior to me.

[whostheJBoss]

I will add one more thing... the save() only fails after the call of
getPostsArray() if the post I am trying to add tags to was included in
the array that was retrieved.

So, if getPostsArray() returns posts 1 and 3, then tags added to those
two posts will not save. Tags added to post 2, which wasn't returned
by getPostsArray(), will save.

Thoughts?

[whostheJBoss]

Ok, another update... if I set proxied="false" on:

<onetomany name="posts" lazy="true" proxied="true">
<link to="posts.post" column="userID"/>
<collection type="array">
</collection>
</onetomany>

Then it works! I can call getPostsArray() first and still add tags.
So, having the onetomany posts object on users.user proxied causes the
odd behavior. I don't think this should be happening.

Yes, it works, but this means my call to getPostsArray() gives me an
array of all of my actual posts.post objects instead of the proxied
object, which I need. The posts.post objects are far too heavy when
retrieving 100 (or in the case of my application, 500) objects. I need
the proxy.

Shouldn't I be able to call getPostsArray() and get proxied objects
without this preventing me from adding tags.tag objects to the
manytomany relationship?

Seems like I should...

I'd like to know what is is in the mechanics of making the parent's
(users.user) relationship with the parent (posts.post) of the
manytomany (tags.tag) proxied that breaks the adding of the manytomany
on the child object (posts.post)

[whostheJBoss]

I have written a custom TQL that handles my problem and all is great,
so this is not a question I *need* the answer to, but I'm still quite
curious WHY this is happening. Any thoughts on why calling
getPostsArray() first breaks the functionality to add a tag?

[whostheJBoss]

I hate to beat this horse, but I'm still having this issue. The
problem is even more prevalent now. I have many situations where I
need to first get the array of items before using them, but doing so
causes any add (or remove) of the many to many objects not to save.
Any thoughts at all on this? Sorry and thanks!

[Mark Mandel]

The fact you are using Railo and that Transfer really isn't well tested (or tested at all) on that platform spring to mind....

Mark

--
E: mark....@gmail.com
T: http://www.twitter.com/neurotic
W: www.compoundtheory.com

[whostheJBoss]

You seem to give me that answer a lot :)

I'd say it's pretty well tested, seeing as I've tried just about
everything you can try with Transfer on Railo and have repeatedly
reported any odd behavior.

So, doesn't this post count as me officially reporting odd behavior to
you?

I can't do a lot more than write up a huge post with all the details
and submit it to the person who created the framework... as I've been
doing.

So, what's the next step?

My problem is outlined pretty well...

As it stands, if you call getWHATEVERArray() on an object... that
disables the ability to save any many-to-many relationships on that
object.

I'll test this on CF8 and let you know... will you take a look at it
if the results are the same?

On Oct 27, 7:21 pm, Mark Mandel <mark.man...@gmail.com> wrote:
> The fact you are using Railo and that Transfer really isn't well tested (or
> tested at all) on that platform spring to mind....
>
> Mark
>

> E: mark.man...@gmail.com

[Mark Mandel]

The only thing I can think of is somehow the Lazy Load xIsLoaded() isn't being set to true somehow.

On Wed, Oct 28, 2009 at 2:00 PM, whostheJBoss <dotf...@changethings.org> wrote:

You seem to give me that answer a lot :)

I'd say it's pretty well tested, seeing as I've tried just about
everything you can try with Transfer on Railo and have repeatedly
reported any odd behavior.

Considering, as far as I am aware, none of the unit tests have ever been run on Railo, I would consider that 'not tested'.

Quite honestly I've never stated that Railo was a supported platform either.

So, doesn't this post count as me officially reporting odd behavior to
you?

I can't do a lot more than write up a huge post with all the details
and submit it to the person who created the framework... as I've been
doing.

So, what's the next step?

My problem is outlined pretty well...

As it stands, if you call getWHATEVERArray() on an object... that
disables the ability to save any many-to-many relationships on that
object.

I'll test this on CF8 and let you know... will you take a look at it
if the results are the same?

Considering I expect a lot of people have done this on CF8, I will be surprised if it doesn't work.  That being said, I've been surprised before.

Mark

--
E: mark....@gmail.com

[whostheJBoss]

Ok, I have tested this on CF8 and CF9 and I get identical behavior as
I do with Railo. If I make a call to the SOMEPARENT.getWHATEVERArray()
function, subsequent saves of many-to-many WHATEVER to SOMEPARENT do
not save. The first addWHATEVER() works, but after that they fail to
save.

In this case, calling user.getPostsArray(); causes user.addPosts
(newPost); not to save after the first attempt.

p.s. I probably get under your skin at times, sorry! Anyway, by
testing I hadn't meant unit tests, I'd just meant that I'd tried the
functionality in various ways to test if it worked in production, not
that the code was stable via unit test results.

> Considering I expect a lot of people have done this on CF8, I will be
> surprised if it doesn't work.  That being said, I've been surprised before.

Well... surprised? :)

Maybe I'm oblivious to something huge here, but... maybe try my
example and see? It's a pretty easy example..

[Mark Mandel]

So this is still directly related to having the object as a proxy, as opposed to having it as a full blown object?

(Just curious how the performance is of 500 objects, I'd be doing that more as a query, but I digress....)

So some things to check out:

<cfoutput>
Is the object directy: #post.getIsDirty()# <br/>
Is the m2m loaded? : #post.getTagsIsLoaded()# <br/>
</cfoutput>

Those are the only 2 things that should stop the SQL from firing.

What SQL are you seeing in the debug when you run your test case?

Mark

[Mark Mandel]

Somehow 'dirty' came out as 'directy'

I have no idea why... :(

Mark

[whostheJBoss]

Performance is horrid with that many objects. I have since switched to
using a query for the large collections (which also fixes the original
issue of this post). I had a 2900% increase in performance. I hadn't
noticed the performance problems originally, since the objects were
still in memory after creation, so they were loading instantly on the
view page. After reinitializing my application and then trying to load
the objects, the problem is apparent. I was only testing with 5 or 10
in the beginning so the problem floated by unnoticed, so I'm glad I
switch to a query anyway. Still, the original many-to-many add / get
array problem persists in the cases where I only have a few objects. I
have been using a query for those as well, but would still like it to
work through Transfer if possible, as I would like to use some of the
objects.

To answer your question, yes, this is only happening when the objects
are proxied. I will check the results of getIsDirty() and
getTagsIsLoaded(), but I should let you know that I have tried
running .loadTags(); before calling getTagsArray(), but the issue is
unaffected. What results should I have for those two?

Oh, a sort of side-note, but having run these queries via TQL to patch
the problem, I notice that they are run each time and are not cached.
I have taken to copying the generated SQL from Transfer out into a
normal <cfquery> so that I can enable caching when I need to. Is there
a way to turn on caching for TQL queries?

On Nov 2, 1:56 pm, Mark Mandel <mark.man...@gmail.com> wrote:
> So this is still directly related to having the object as a proxy, as
> opposed to having it as a full blown object?
>
> (Just curious how the performance is of 500 objects, I'd be doing that more
> as a query, but I digress....)
>
> So some things to check out:
>
> <cfoutput>
> Is the object directy: #post.getIsDirty()# <br/>
> Is the m2m loaded? : #post.getTagsIsLoaded()# <br/>
> </cfoutput>
>
> Those are the only 2 things that should stop the SQL from firing.
>
> What SQL are you seeing in the debug when you run your test case?
>
> Mark
>

> On Tue, Nov 3, 2009 at 7:40 AM, whostheJBoss <dotfus...@changethings.org>wrote:
>
>
>
>
>
>
>
> > Ok, I have tested this on CF8 and CF9 and I get identical behavior as
> > I do with Railo. If I make a call to the SOMEPARENT.getWHATEVERArray()
> > function, subsequent saves of many-to-many WHATEVER to SOMEPARENT do
> > not save. The first addWHATEVER() works, but after that they fail to
> > save.
>
> > In this case, calling user.getPostsArray(); causes user.addPosts
> > (newPost); not to save after the first attempt.
>
> > p.s. I probably get under your skin at times, sorry! Anyway, by
> > testing I hadn't meant unit tests, I'd just meant that I'd tried the
> > functionality in various ways to test if it worked in production, not
> > that the code was stable via unit test results.
>
> > > Considering I expect a lot of people have done this on CF8, I will be
> > > surprised if it doesn't work.  That being said, I've been surprised
> > before.
>
> > Well... surprised? :)
>
> > Maybe I'm oblivious to something huge here, but... maybe try my
> > example and see? It's a pretty easy example..
>
> --

> E: mark.man...@gmail.com

[Mark Mandel]

On Tue, Nov 3, 2009 at 9:25 AM, whostheJBoss <dotf...@changethings.org> wrote:

Performance is horrid with that many objects. I have since switched to
using a query for the large collections (which also fixes the original
issue of this post). I had a 2900% increase in performance. I hadn't
noticed the performance problems originally, since the objects were
still in memory after creation, so they were loading instantly on the
view page. After reinitializing my application and then trying to load
the objects, the problem is apparent. I was only testing with 5 or 10
in the beginning so the problem floated by unnoticed, so I'm glad I
switch to a query anyway. Still, the original many-to-many add / get
array problem persists in the cases where I only have a few objects. I
have been using a query for those as well, but would still like it to
work through Transfer  if possible, as I would like to use some of the
objects.

Generally speaking setting up relationships so that they have a huge number of objects is a bad idea...
 

To answer your question, yes, this is only happening when the objects
are proxied. I will check the results of getIsDirty() and
getTagsIsLoaded(), but I should let you know that I have tried
running .loadTags(); before calling getTagsArray(), but the issue is
unaffected. What results should I have for those two?

Dirty should be 'true', and isLoaded() should also be true.
 

Oh, a sort of side-note, but having run these queries via TQL to patch
the problem, I notice that they are run each time and are not cached.
I have taken to copying the generated SQL from Transfer out into a
normal <cfquery> so that I can enable caching when I need to. Is there
a way to turn on caching for TQL queries?

There isn't a way to cache TQL results as of yet.

Mark

--
E: mark....@gmail.com

[whostheJBoss]

Yes, I've decided now to use the many-to-many for editing only, so
anything POST/write is going to be using addTags(), and anything GET/
read is going to be TQL / SQL. That way I never call getTagsArray()
(since there will be hundreds), but I can still use addTags() to add
them. Since the tags are lazy-loaded and proxied, they don't add much
weight to the objects at all.

So any thoughts on if isLoaded() comes back false? And still comes
back false after calling loadTags() ?

On Nov 2, 2:27 pm, Mark Mandel <mark.man...@gmail.com> wrote:

> E: mark.man...@gmail.com

[Mark Mandel]

So any thoughts on if isLoaded() comes back false? And still comes
back false after calling loadTags() ?

I only have thoughts if it is actually doing that...

Mark

--
E: mark....@gmail.com

[whostheJBoss]

Before calling getTagsArray():
false - getIsDirty()
false - getTagsIsLoaded()

After calling getTagsArray():
true - getIsDirty()
true - getTagsIsLoaded()

Immediately after first save (save works):
false - getIsDirty()
true - getTagsIsLoaded()

Before second save:
true - getIsDirty()
true - getTagsIsLoaded()

After second save (save fails):
false - getIsDirty()
false - getTagsIsLoaded()



On Nov 2, 2:27 pm, Mark Mandel <mark.man...@gmail.com> wrote:

> E: mark.man...@gmail.com

[whostheJBoss]

It was:

Before calling getTagsArray():
false - getIsDirty()
false - getTagsIsLoaded()

After calling getTagsArray():
true - getIsDirty()
true - getTagsIsLoaded()

Immediately after first save (save works):
false - getIsDirty()
true - getTagsIsLoaded()

Before second save:
true - getIsDirty()
true - getTagsIsLoaded()

After second save (save fails):
false - getIsDirty()
false - getTagsIsLoaded()

On Nov 2, 3:22 pm, Mark Mandel <mark.man...@gmail.com> wrote:
> > So any thoughts on if isLoaded() comes back false? And still comes
> > back false after calling loadTags() ?
>
> I only have thoughts if it is actually doing that...
>
> Mark
>
> --

> E: mark.man...@gmail.com

[Mark Mandel]

What is the second save? I only see one save in the original code?

Mark

--
E: mark....@gmail.com

[whostheJBoss]

Oops, I had a slight omission in the original post (or one of those
following it, not sure at what point I came across this aspect of the
behavior), the first addTags() after calling the array works,
subsequent calls won't.

Sorry, so I meant when running the event that does the save a second
time.

Order:
1.) Call an event that uses getTagsArray();
2.) Call an event that uses addTags(tag); to add a tag, the save works
and tag shows up
3.) Call an event again that uses addTags(tag); to add a tag, the save
does not work, tag does not show up

So, the first time I call the addTags() method after having called
getTagsArray(), anything after that won't.

On Nov 2, 3:44 pm, Mark Mandel <mark.man...@gmail.com> wrote:
> What is the second save? I only see one save in the original code?
>
> Mark
>

[Mark Mandel]

So what happens between:

Before second save:
true - getIsDirty()
true - getTagsIsLoaded()

--> dump out the tags - are all the tags there? or did addTags() not work?

After second save (save fails):
false - getIsDirty()
false - getTagsIsLoaded()

What is weird is the getTagsIsLoaded() is being reset back to 'false', after the save.

Mark

--
E: mark....@gmail.com

[whostheJBoss]

addTags() works the first time, when I dump getTagsArray() after the
first save, they show up.

Any add or remove after that is not reflected in the object.

On Nov 2, 4:09 pm, Mark Mandel <mark.man...@gmail.com> wrote:
> So what happens between:
>
> Before second save:
> true - getIsDirty()
> true - getTagsIsLoaded()
>
> --> dump out the tags - are all the tags there? or did addTags() not work?
>
> After second save (save fails):
> false - getIsDirty()
> false - getTagsIsLoaded()
>
> What is weird is the getTagsIsLoaded() is being reset back to 'false', after
> the save.
>
> Mark
>

[Mark Mandel]

That doesn't make sense how an add/remove doesn't fire on the object... even if the data is all wonky, the add/remove should still work.  the proxy() method on the ObjectProxy always goes *somewhere* even if that somewhere goes to an error.

Mark

--
E: mark....@gmail.com

[whostheJBoss]

That's why I'm asking you :)

If I use addTags() and then save it, a dump of the object shows no new
tags.

However, if I use addTags() and then dump the object, before saving,
there are tags...

On Nov 2, 4:25 pm, Mark Mandel <mark.man...@gmail.com> wrote:
> That doesn't make sense how an add/remove doesn't fire on the object... even
> if the data is all wonky, the add/remove should still work.  the proxy()
> method on the ObjectProxy always goes *somewhere* even if that somewhere
> goes to an error.
>
> Mark
>

[Mark Mandel]

Okay, so you didn't really answer my question then, maybe I wasn't clear.

Before second save:
true - getIsDirty()
true - getTagsIsLoaded()

--> dump out the tags - are all the tags there?

--> add some more tags
--> dump it again, are the tags there?

After second save (save fails):
false - getIsDirty()
false - getTagsIsLoaded()

--> dump it again, I assume the tags that were added are now gone?

Mark

--
E: mark....@gmail.com

[whostheJBoss]

Ok, I'll be more clear:

When I start, my post object has has three tags attached as many-to-
many with the following ids: 1,2,3

Before step 1:
false - isdirty
false - isloaded

Step 1.) call event with:
post = Transfer.get("posts.post", 1);
rc.tags = post.getTagsArray();

View:
Loop through rc.tags and output tagID to show:
1
2
3

The tags show up.

After step 1:
true - isdirty
true - isloaded

Step 2.) call event with:
tag1 = Transfer.get("tags.tag", 4);
tag2 = Transfer.get("tags.tag", 5);
post = Transfer.get("posts.post", 1);
post.addTags(tag1);
post.addTags(tag2);
Transfer.save(post);

View:
Loop through rc.tags and output tagID to show:
1
2
3
4
5

The new tags (4,5) show up.

After step 2:
false - isdirty
true - isloaded

Step 3.) call event with:
tag1 = Transfer.get("tags.tag", 6);
tag2 = Transfer.get("tags.tag", 7);
post = Transfer.get("posts.post", 1);
post.addTags(tag1);
post.addTags(tag2);
Transfer.save(post);

View:
Loop through rc.tags and output tagID to show:
1
2
3
4
5

The new tags (6,7) do not show up.

After step 3:
false - isdirty
false - isloaded

So, only the first addTags() with save() works after calling
getTagsArray();

On Nov 2, 5:10 pm, Mark Mandel <mark.man...@gmail.com> wrote:
> Okay, so you didn't really answer my question then, maybe I wasn't clear.
>
> Before second save:
> true - getIsDirty()
> true - getTagsIsLoaded()
>
> --> dump out the tags - are all the tags there?
> --> add some more tags
> --> dump it again, are the tags there?
>
> After second save (save fails):
> false - getIsDirty()
> false - getTagsIsLoaded()
>
> --> dump it again, I assume the tags that were added are now gone?
>
> Mark
>

[Mark Mandel]

On Tue, Nov 3, 2009 at 12:25 PM, whostheJBoss <dotf...@changethings.org> wrote:

Step 3.) call event with:
tag1 = Transfer.get("tags.tag", 6);
tag2 = Transfer.get("tags.tag", 7);
post = Transfer.get("posts.post", 1);
post.addTags(tag1);
post.addTags(tag2);

What happens if you dump the object at this point?
 

Transfer.save(post);

--
E: mark....@gmail.com

[whostheJBoss]

1
2
3
4
5

6
7

It shows all 7 tags. However, the 6 and 7 do not stay saved. After save
() is called, they are gone again.

On Nov 2, 6:51 pm, Mark Mandel <mark.man...@gmail.com> wrote:

> On Tue, Nov 3, 2009 at 12:25 PM, whostheJBoss <dotfus...@changethings.org>wrote:
>
> > Step 3.) call event with:
> > tag1 = Transfer.get("tags.tag", 6);
> > tag2 = Transfer.get("tags.tag", 7);
> > post = Transfer.get("posts.post", 1);
> > post.addTags(tag1);
> > post.addTags(tag2);
>
> What happens if you dump the object at this point?
>
> > Transfer.save(post);
>
> --

> E: mark.man...@gmail.com

[Mark Mandel]

Okay, so the addTags() *does* work, just something is the save() is going fubar.

(Probably has to do with cache synchronisation).

I'm ripping apart Transfer at the moment, I have an idea what it may be... I'll have a look.

Mark

--
E: mark....@gmail.com

[whostheJBoss]

Excellent! Thank you! Even though I've worked around the problem, it
would be great if this worked, I've been quite curious as to why it
doesn't.

p.s. If you are ripping it apart, perhaps TQL query caching would be
possible? :)

On Nov 3, 2:31 pm, Mark Mandel <mark.man...@gmail.com> wrote:
> Okay, so the addTags() *does* work, just something is the save() is going
> fubar.
>
> (Probably has to do with cache synchronisation).
>
> I'm ripping apart Transfer at the moment, I have an idea what it may be...
> I'll have a look.
>
> Mark
>