First Pass: TransferObjectProxies in SVN

[Mark Mandel]

All,

Just letting you know that I'm committing the first pass of TransferObjectProxies into SVN.

*** This is a ColdFusion 8.0.1 only release ***
(Yes, the updater is important, as it changes the way you can call <cfinvoke> in a way that makes it *much* easier to use with onMissingMethod)

Ticket:
http://tracker.transfer-orm.com/issue.cfm?p=89977683-A728-9CD3-ABD9545A91734422&i=0E4A1E3A-B6DA-8C18-05388BDA56C5FACE

From the title, that probably doesn't make a huge amount of sense, so let me explain what they do:

So, the classic example - you have a onetomany, say of Users-o2m->Orders.

Normally this would be a pretty bad thing, as users keep making orders, and they could start ranging into the hundreds of orders, and then you have to create and populate 400 objects when the you want to get the User.

So first level, is obviously, lazy loading.  So we can configure that collection to be lazy loaded, in which case, we can request the User, and the 400 Orders, don't get loaded.

BUT when we do request the Orders, all 400 objects get loaded at once, which brings the server to its knees.

Now we have a second level, we can go:

<onetomany lazy="true" proxied="true">
...
<onetomany>

Now, instead of querying for all the data of the 400 objects, it just queries for the IDs of the Orders, and then creates a TransferObjectProxy for each record it finds.

Only when a method is called on the Proxy that requires data it doesn't know about is called on it, will it load up the TranferObject it represents.

This means you could do something like:

user = transfer.get("User", 1);
len = ArrayLen(user.getOrderArray());

And there would be no loading of the underlying TransferObjects.  You could also use find() or contains(), without worrying about the objects being loaded.

This should be a huge performance boost for many applications.

Now that all being said, lazy, and proxied can be used separately, or together, totally at your discretion.

Things not yet tested:
Cloning of proxies - in theory, should be fine, but I haven't tested it yet, and there are some pieces I will want to test to ensure that if you clone() an un-loaded proxy, it won't load, but just create a new proxy.
Proxies with composite keys - again, should be fine, but I haven't tested it yet, either.

Feel free to play with these, even in areas I haven't tested, and let me know what issues you hit.

Mark

--
E: mark....@gmail.com
W: www.compoundtheory.com

[Matt Quackenbush]

That is bad-ass, Mark.  This one feature alone might be enough to get me to cough up the coin for an upgrade to 8.0.1.

[James Allen]

Su-bloody-perb mate!!!

 

This is exactly what I need as I now don’t have to worry about calling a gateway method to obtain a recordset and thus the size of a related collection.

 

Really clever thinking Mark.. Love it!

[Robert Rawlins]

Damn right, this is a superb addition, I’ll suck down the SVN a little later and give it a shot.

 

What I love best is that this shouldn’t require any changes, apart from the XML, right?

 

Nice work Mark, well done.

 

Rob

[Sean Corfield]

On Wed, Jul 2, 2008 at 12:32 AM, Mark Mandel <mark....@gmail.com> wrote:
> Just letting you know that I'm committing the first pass of
> TransferObjectProxies into SVN.

Awesome!

> *** This is a ColdFusion 8.0.1 only release ***

Do you mean it will no longer work on CFMX7 or do you just mean it
won't work on CF8.0.0? (I understand that proxies will not work on
CFMX7).

> Only when a method is called on the Proxy that requires data it doesn't know
> about is called on it, will it load up the TranferObject it represents.

Could you expand on this a little? Suppose I have a decorator. Can I
call my decorator methods without it loading data (as long as those
methods in turn do not call an underlying getter/setter)? Just trying
to get clarity on what exactly triggers the load... all generated
get*() and set*() methods? Anything else? Since the proxy has the PK,
does calling getId() - or whatever method binds to the PK - cause a
data load? I'd hope not.

I think this is an excellent enhancement and I can imagine using
lazy="false" proxied="true" a lot.
--
Sean A Corfield -- (904) 302-SEAN
An Architect's View -- http://corfield.org/

"If you're not annoying somebody, you're not really alive."
-- Margaret Atwood

[Mark Mandel]

> *** This is a ColdFusion 8.0.1 only release ***

Do you mean it will no longer work on CFMX7 or do you just mean it
won't work on CF8.0.0? (I understand that proxies will not work on
CFMX7).

This means it won't work on CF7, or CF8, only CF8.0.1.

It relies on 2 pieces of functionality - onMissingMethod() and the ability to pass in numeric keyed argument collections to <cfinvoke> (rather than the named keys that are usual for argument collections), which enables us to do much faster proxying with onMM.
 

> Only when a method is called on the Proxy that requires data it doesn't know
> about is called on it, will it load up the TranferObject it represents.

Could you expand on this a little? Suppose I have a decorator. Can I
call my decorator methods without it loading data (as long as those
methods in turn do not call an underlying getter/setter)? Just trying
to get clarity on what exactly triggers the load... all generated
get*() and set*() methods? Anything else? Since the proxy has the PK,
does calling getId() - or whatever method binds to the PK - cause a
data load? I'd hope not.

This is a good point to consider (and this thread will probably result in the documentation).

The Proxy will proxy both the TO and the Decorator.

So, if you are just using TO, it will be:
TransferObjectProxy -loads->TransferObject

If you are using a Decorator:

TransferObjectProxy -loads->Decorator -contains->TransferObject

Make sense?

Now, there are several things the TOP (TransferObjectProxy) knows about, and therefore won't load on:

getIsPersisted() - it knows it's true, so unless it's loaded, it will return 'true' without loading.
getIsDirty() - it knows this is false, so unless it's loaded, it will return 'false' without loading.
getClassName() - it knows what class it represents, so it won't be loaded if this is called.
getPrimaryKeyValue() - the TOP stores the primary key value, so it knows about it, so it won't be loaded.
(with the combination of this, and the above, you can call equalsTransfer() without worrying about loading)

Finally, the TOP knows about any properties that are used on it either as a struct key, or as a ordering property, so if the TOP is used in a collection that is order by 'Foo', calling getFoo() on the TOP will not cause a load, as it already has that information.

Side note: please do realise that CFC creation and population does not have its expense, so even if you are creating 1,000,000 TO proxies, it may well be slow.  So still load test, and make sure that it performs well, etc. There is no silver bullet ;o)

Glad you guys like the enhancement! 1.1 is really about making Transfer more scalable, and easier to tune in the long run.

(These can be seen on the issues tracker, but filter by milestone on the ticket list)
Tickets left in 1.1
- Complete Transfer Object Proxies
- Cache introspection, i.e. number of items, hits, misses etc
- Some bug fixes
- I had an idea for a good performance enhancement on object loading, but it's a question of how much time I have.

I'm also working on updating the project tracker, so you can actually see easier what tickets have been assigned to the 1.1 milestone.

I've set myself the goal of having the Final 1.1. done by August 8th, which is deliberatley a short time frame.  I'm hoping to be in a RC stage within the next week or two.

[Elliott Sprehn]

On Jul 2, 11:46 am, "Sean Corfield" <seancorfi...@gmail.com> wrote:
>
> > *** This is a ColdFusion 8.0.1 only release ***
>
> Do you mean it will no longer work on CFMX7 or do you just mean it
> won't work on CF8.0.0? (I understand that proxies will not work on
> CFMX7).

My understanding is that it will only work in CF8.0.1 because it uses
the positional invocation model when proxying function calls with
OnMissingMethod(). Previous versions required that <cfinvoke> take
named arguments that mapped to the names of the arguments on the real
method. 8.0.1 allows passing the arguments in the collection provided
to OnMissingMethod directly to <cfinvoke>.

There's a note in the 8.0.1 release notes about changes in the way
arguments are passed.

[Sean Corfield]

On Wed, Jul 2, 2008 at 3:08 PM, Mark Mandel <mark....@gmail.com> wrote:
> This means it won't work on CF7, or CF8, only CF8.0.1.

OK, I'll rephrase that: does this mean Transfer no longer supports
CFMX7 at all? (or is this just for the proxy functionality?)

(It doesn't bother me since I'm on 8.0.1 - except for one server I've
been meaning to rebuild which is on 8.0.0 - but I thought there were
still some Transfer users on CFMX7?)

> If you are using a Decorator:
>
> TransferObjectProxy -loads->Decorator -contains->TransferObject

Ah, so *any* method call will force a load even if it doesn't actually
"need" to? (excepting the ones listed below)

> getPrimaryKeyValue() - the TOP stores the primary key value, so it knows
> about it, so it won't be loaded.

Hmm, is this a new method in Transfer 1.1? Is it only on the proxy or
is it generated into the TO as well? Seems like a useful addition and
I could see writing code that relies on this to avoid loading objects
behind a proxy...

> Finally, the TOP knows about any properties that are used on it either as a
> struct key, or as a ordering property, so if the TOP is used in a collection
> that is order by 'Foo', calling getFoo() on the TOP will not cause a load,
> as it already has that information.

OK, that's sweet.

> Side note: please do realise that CFC creation and population does not have

*not*?? I assume that was a typo...

> its expense, so even if you are creating 1,000,000 TO proxies, it may well
> be slow. So still load test, and make sure that it performs well, etc.

[Mark Mandel]

Sounds like I'm having a bad explanation day....

On Thu, Jul 3, 2008 at 9:43 AM, Sean Corfield <seanco...@gmail.com> wrote:

On Wed, Jul 2, 2008 at 3:08 PM, Mark Mandel <mark....@gmail.com> wrote:
> This means it won't work on CF7, or CF8, only CF8.0.1.

OK, I'll rephrase that: does this mean Transfer no longer supports
CFMX7 at all? (or is this just for the proxy functionality?)

(It doesn't bother me since I'm on 8.0.1 - except for one server I've
been meaning to rebuild which is on 8.0.0 - but I thought there were
still some Transfer users on CFMX7?)

No, this means that the proxy feature is only available on 8.0.1.  Everything else on Transfer still works on 7, and will continue to be supported.
 

> If you are using a Decorator:
>
> TransferObjectProxy -loads->Decorator -contains->TransferObject

Ah, so *any* method call will force a load even if it doesn't actually
"need" to? (excepting the ones listed below)

Yes, that is true.
 

> getPrimaryKeyValue() - the TOP stores the primary key value, so it knows
> about it, so it won't be loaded.

Hmm, is this a new method in Transfer 1.1? Is it only on the proxy or
is it generated into the TO as well? Seems like a useful addition and
I could see writing code that relies on this to avoid loading objects
behind a proxy...

Sorry, no, that is not a new method.

I meant it as 'whatever the method is that get's generated from your <id> element'.

so if you have <id name="fooid" ... />

and you call getFooID() on the TOP, it won't load the object.

*not*?? I assume that was a typo...

> its expense, so even if you are creating 1,000,000 TO proxies, it may well
> be slow.  So still load test, and make sure that it performs well, etc.

Having a bad day...

Side note: please do realise that CFC creation and population does have its expense, so even if you are creating 1,000,000 TO proxies, it may well be slow.  So still load test, and make sure that it performs well, etc. There is no silver bullet ;o)

That should be better.

Mark

[Mark Mandel]

I just tested composite keys, passed with flying colours (well, green
actually... that's a unit testing joke ;o) )

Mark

[Peter Bell]

Hmmmm, unit testing humor is my limit. If I see a cobol or fortran
joke, I'm off this list :-)

[Jon Messer]

This is really great, it will allow for some relationships that I wouldn't have even tried otherwise.

There is a bug in the first set up I tried though that boils down to this (I think) if the proxied object has a collection and for some reason that (those) collections aren't themselves marked as proxied then an sql error is thrown. If the collections of the proxied object are proxied then it works, but if a compositeid has a collection as part of it then that collection can't be proxied or lazy.

Here is a simple config that shows this behavior :




<object name="Address" table="Address" decorator="model.Address">
    <id name="AddressId" type="numeric" generate="false"/>
</object>

<object name="Person" table="Person" decorator="model.Person">
    <id name="PersonId" type="numeric" generate="false"/>
    <onetomany name="Addresses" lazy="true" proxied="true">
        <link to="model.PersonsAddress" column="PersonId"/>
        <collection type="array" />
    </onetomany>
</object>

<object name="PersonsAddress" table="Person_Address" decorator="model.PersonsAddress">
    <compositeid>
        <parentonetomany class="model.Person" />
        <manytoone name="Address" />
    </compositeid>
    <manytoone name="Address">
        <link to="model.Address" column="AddressId"/>
    </manytoone>
</object>


person.getAddressesArray() throws error


if I redefine PersonsAddress like this it works

<object name="PersonsAddress" table="Person_Address" decorator="model.PersonsAddress">
                <id name="PersonsAddressId" column="id" type="numeric" generate="false"/>
                <manytoone name="Address" lazy="true" proxied="true">
                    <link to="model.Address" column="AddressId"/>
                </manytoone>
</object>


Again this looks fantastic and I know it isn't ready for prime time yet, but I just thought I'd share the first snag I ran into.

[Sean Corfield]

On Thu, Jul 3, 2008 at 2:48 AM, Peter Bell <pb...@systemsforge.com> wrote:

> Hmmmm, unit testing humor is my limit. If I see a cobol or fortran
> joke, I'm off this list :-)

A COBOL programmer, a FORTRAN programmer and a Smalltalk programmer
walked into a bar...

Just j/k :)

[Sean Corfield]

On Wed, Jul 2, 2008 at 7:43 PM, Mark Mandel <mark....@gmail.com> wrote:
> No, this means that the proxy feature is only available on 8.0.1.
> Everything else on Transfer still works on 7, and will continue to be
> supported.

OK, that's what I *hoped* you meant :)

> I meant it as 'whatever the method is that get's generated from your <id>
> element'.
>
> so if you have <id name="fooid" ... />
>
> and you call getFooID() on the TOP, it won't load the object.

Thanx for the clarification. Again, that's what I *hoped* you meant :)

[Mark Mandel]

Jon,

I'm wondering if this is similar to the <compsoiteid> bug that Philip
got a while ago:
http://groups.google.com/group/transfer-dev/browse_thread/thread/47d3a1f5aa591f07

What was the error message that you got?

And what is the code you used to get the error?

Mark

[Jon Messer]

To get the error all you have to do is call the get{ProxiedCollection}Array() on an object for which the proxied collection object has non-proxied children. If all of the children of the proxied object are also proxied it works fine.

I did some digging and was able to see that in TransferSelector.buildColumnStruct doesn't add any tables to the column struct for children of the proxied collection. But then in TransferSelector.buildSelect it is trying to index into the column struct based on the key being one of those non-proxied childrens tables.

I was gonna give figuring it out a go, but got busy and it's pretty gnarly code, not gnarly as in bad, just intense  :)


The actual error I got was :


Element Address is undefined in a CFML structure referenced as part of an expression.
 
The error occurred in /Library/WebServer/Documents/cf-tools/transfer/transfer/com/sql/TransferSelecter.cfc: line 656
Called from /Library/WebServer/Documents/cf-tools/transfer/transfer/com/sql/TransferSelecter.cfc: line 469
Called from /Library/WebServer/Documents/cf-tools/transfer/transfer/com/sql/TransferSelecter.cfc: line 538
Called from /Library/WebServer/Documents/cf-tools/transfer/transfer/com/sql/TransferSelecter.cfc: line 581
Called from /Library/WebServer/Documents/cf-tools/transfer/transfer/com/sql/TransferSelecter.cfc: line 96
Called from /Library/WebServer/Documents/cf-tools/transfer/transfer/com/sql/TransferSelecter.cfc: line 47
Called from /Library/WebServer/Documents/cf-tools/transfer/transfer/com/sql/SQLManager.cfc: line 87
Called from /Library/WebServer/Documents/cf-tools/transfer/transfer/com/dynamic/TransferPopulator.cfc: line 100
Called from /Library/WebServer/Documents/cf-tools/transfer/transfer/com/dynamic/DynamicManager.cfc: line 93
Called from /Library/WebServer/Documents/cf-tools/transfer/transfer/com/Transfer.cfc: line 764


I could zip up a test bed if that helps...

[Mark Mandel]

Question - is it non-proxied, or is it just a case of non-lazy?

Mark

[Jon Messer]

Ah good catch. It happens if the child is non-lazy.

The reason it happened to me is because the m2o relationship is part of the compositeid, so it has to be non-lazy.

[James Allen]

Oh go on I want to hear the punchline! :)

-----Original Message-----
From: transf...@googlegroups.com [mailto:transf...@googlegroups.com]
On Behalf Of Sean Corfield
Sent: 03 July 2008 22:40
To: transf...@googlegroups.com
Subject: [transfer-dev] Re: First Pass: TransferObjectProxies in SVN

[Mark Mandel]

This should be fixed now.

Have a play with it, and let me know if you find any more issues.

Mark

[Jon Messer]

I hate to be the bearer of bad news but I updated to rev 527 and this still fails one of the 2 conditions that threw errors.

I've added a test to your suite for the condition where the child is non-lazy and proxied that was one of the 2 error conditions. You fixed the other one which was for lazy and proxied.


/transfer/test/resources/transfere.xml

        <package name="messer">
            <object name="Address" table="tbl_A">
                <id name="id" type="UUID" generate="true" column="a_id"/>
                <property name="stringValue" type="string" column="a_value"/>
            </object>

            <object name="Person" table="tbl_B">
                <id name="id" type="UUID" generate="true" column="b_id"/>
                <property name="stringValue" type="string" column="b_value"/>

                <onetomany name="Addresses" lazy="true" proxied="true">

                    <link column="lnkid" to="messer.PersonsAddress"/>
                    <collection type="array">
                    </collection>
                </onetomany>

                <onetomany name="NonLazyProxiedAddresses" lazy="true" proxied="true">
                    <link column="lnkid" to="messer.PersonsNonLazyProxiedAddress"/>
                    <collection type="array">
                    </collection>
                </onetomany>

            </object>

            <object name="PersonsAddress" table="tbl_C">
                <compositeid>
                    <parentonetomany class="messer.Person"/>

                    <manytoone name="Address"/>
                </compositeid>

                <property name="id" type="UUID" column="c_id"/>
                <property name="stringValue" type="string" column="c_value"/>

                <manytoone name="Address" lazy="true" proxied="true">

                    <link column="lnkid2" to="messer.Address"/>
                </manytoone>
            </object>


            <object name="PersonsNonLazyProxiedAddress" table="tbl_d">
                <compositeid>
                    <parentonetomany class="messer.Person"/>

                    <manytoone name="Address"/>
                </compositeid>

                <property name="id" type="UUID" column="d_id"/>
                <property name="stringValue" type="string" column="d_value"/>
                <manytoone name="Address" proxied="true">
                    <link column="lnkid2" to="messer.Address"/>
                </manytoone>
            </object>

        </package>


and add this to /transfer/test/transfer/cases/proxy/ProxyTest.cfc

<cffunction name="testMesserNonLazyProxy" hint="" access="public" returntype="void" output="false">
    <cfscript>
        var transfer = getTransferE();
        var reget = 0;

        p = transfer.new('messer.Person');
        p.setStringValue('Messer');

        a = transfer.new('messer.Address');
        pa = transfer.new('messer.PersonsNonLazyProxiedAddress');

        pa.setAddress(a);
        pa.setParentPerson(p);
        transfer.cascadeSave(p);

        transfer.discardAll();

        reget = transfer.get("messer.Person", p.getID());

        AssertTrue(p.getIsPersisted(), "should be persisted");

        AssertEquals("should be 1", 1, ArrayLen(reget.getNonLazyProxiedAddressesArray()));

        AssertTrue(reget.getNonLazyProxiedAddresses(1).hasAddress(), "should have address");
    </cfscript>
</cffunction>


hope that helps?

[Mark Mandel]

Sod.

Jon, what error message do you get?

If something goes wrong, I need to know the error message... it saves
me hours of time ;o)
http://groups.google.com/group/transfer-dev/web/how-to-ask-support-questions-on-transfer

Mark

[Jon Messer]

Running the modified unit test this is the error. It's the same against my code execept it says "Address" instead of "tbl_A".

Test	testMesserNonLazyProxy(test.transfer.cases.proxy.ProxyTest)
Message	Element tbl_A is undefined in a CFML structure referenced as part of an expression.
Type	Expression
Tag Context	/Library/WebServer/Documents/cf-tools/transfer/transfer/com/sql/TransferSelecter.cfc:738 /Library/WebServer/Documents/cf-tools/transfer/transfer/com/sql/TransferSelecter.cfc:551 /Library/WebServer/Documents/cf-tools/transfer/transfer/com/sql/TransferSelecter.cfc:620 /Library/WebServer/Documents/cf-tools/transfer/transfer/com/sql/TransferSelecter.cfc:663 /Library/WebServer/Documents/cf-tools/transfer/transfer/com/sql/TransferSelecter.cfc:96 /Library/WebServer/Documents/cf-tools/transfer/transfer/com/sql/TransferSelecter.cfc:47 /Library/WebServer/Documents/cf-tools/transfer/transfer/com/sql/SQLManager.cfc:87 /Library/WebServer/Documents/cf-tools/transfer/transfer/com/dynamic/TransferPopulator.cfc:100 /Library/WebServer/Documents/cf-tools/transfer/transfer/com/dynamic/DynamicManager.cfc:93 /Library/WebServer/Documents/cf-tools/transfer/transfer/com/Transfer.cfc:764 /Library/WebServer/Documents/cf-tools/transfer/test/resources/defs/messer.Person$998ECE4F9DF3D0428DD05A6D602B4591.transfer:150 /Library/WebServer/Documents/cf-tools/transfer/test/resources/defs/messer.Person$998ECE4F9DF3D0428DD05A6D602B4591.transfer:616 /Library/WebServer/Documents/cf-tools/transfer/test/transfer/cases/proxy/ProxyTest.cfc:289 D:\wwwroot\cfunit\src\net\sourceforge\cfunit\framework\tester.cfm:84 D:\wwwroot\cfunit\src\net\sourceforge\cfunit\framework\tester.cfm:29 D:\wwwroot\cfunit\src\net\sourceforge\cfunit\framework\TestResult.cfc:72 D:\wwwroot\cfunit\src\net\sourceforge\cfunit\framework\TestResult.cfc:48 D:\wwwroot\cfunit\src\net\sourceforge\cfunit\framework\tester.cfm:20 D:\wwwroot\cfunit\src\net\sourceforge\cfunit\framework\TestSuite.cfc:261 D:\wwwroot\cfunit\src\net\sourceforge\cfunit\framework\TestSuite.cfc:253 D:\wwwroot\cfunit\src\net\sourceforge\cfunit\framework\TestRunner.cfc:80 /Library/WebServer/Documents/cf-tools/transfer/test/index.cfm:123

[Mark Mandel]

Cool thanks.

I'm getting the same thing, so now I know that we're on the same page.

Mark

[Mark Mandel]

Jon,

That should be fixed.

Thank for your effort in testing this stuff out. The number of
combinations in which these things can be used makes testing every
single combination tricky (especially when you add in composite keys).

Mark

[Jon Messer]

All green, thanks for your amazingly quick support Mark...

[Mark Mandel]

TransferObjectProxies are ready for usage.

Cloning has been tested, so feel free to Proxy Away!

Mark