Decorator after populate method?

[John Whish]

Hi, I know there is the configure method of a decorator which runs
before the object is populated. Is there a method that runs after the
object is populated? I have a onetomany relationship which creates
lots of objects which are basically a "name-value" pair, which could
have a performance impact (the cache has to run out of space at some
point!) so I'd like to parse these into a simple struct which is
stored in the User object and then destroy all the onetomany objects.

Transfer.xml Snippet:
<package name="member">
<object name="User" table="USERS" decorator="com.model.member.User">

<id name="UserID" column="user_id" type="GUID" generate="true" />
<property name="Email" column="user_email" type="string"
nullable="false" />

<onetomany name="UserAttributes" lazy="false" proxied="false">
<link to="member.UserAttributes" column="user_id" />
<collection type="struct">
<key property="Key" />
</collection>
</onetomany>

</object>

<object name="UserAttributes" table="USERATTRIBUTES">
<id name="UserAttributeID" column="userattribute_id" type="numeric"
generate="true" />
<property name="Key" column="userattribute_key" type="string" />
<property name="Value" column="userattribute_value" type="string" />
</object>

</package>

Thanks!

[Brian Kotek]

How large is "a large number of objects"? What you probably need to do is revisit the relationship since you shouldn't be using oneToMany in situations where there are a large number of objects. Use manyToOne.

But you can override the configure() method in the Decorator. From the documentation:

Setting up values in the Decorator

configure() on the TransferDecorator will be run when the object is first created. You can overwrite this when you implement your own Decorator CFC, and is a perfect place to set default values for your Transfer Object.

It should be noted that configure() is run before the object is populated, and therefore, any values it sets will be overwritten by those retrieved from the database.

You might also look at using the Transfer Proxies for this relationship instead of the real objects.

[John Whish]

Hi Brian, I might be going down completely the wrong path with this.

I'm expecting 200,000 users logged in at once. The idea is to "encapsulate what changes" and stick all user attributes that aren't common to all users into another class. Users will probably only have 3-4 specific attributes (although as the system grows it will no doubt be added to).

As I understand it, 200,000 x 4 is a lot of objects to store in the transfer cache so it will be constantly instantiating those objects. Originally I was using the proxy before I decided to try storing a struct directly in the User object.

As I get the user object when they log in I don't see how I could use a manyToOne in this scenario.

I got the idea from the heads up OOA&D guitar store example, if that helps.

[Bob Silverberg]

Hmm, maybe you should just forego Transfer's composition and deal with it directly in the decorator.  That sounds like a more scalable solution to me.

--
Bob Silverberg
www.silverwareconsulting.com

[John Whish]

I had thought about doing a good ol' cfquery to pull out the values from the db and just populate that way. Would certainly be quick.

[Bob Silverberg]

I think it might even make more sense, because I'm not sure that you're going to get any value from composition anyway.  It doesn't sound like you have a bunch of actual objects composed into your User object, but rather a bunch of properties.  So it might make more sense to treat them as properties rather than as composed objects.  I'm sure you could do something fun with onMissingMethod to allow them to be treated as properties and keep the extensibility that it sounds like your striving for.

Bob

On Fri, Mar 13, 2009 at 11:52 AM, John Whish <john....@googlemail.com> wrote:

I had thought about doing a good ol' cfquery to pull out the values from the db and just populate that way. Would certainly be quick.

--
Bob Silverberg
www.silverwareconsulting.com

[John Whish]

Hi Bob, yeah the objects are completely dumb so there is not value to using composition. Using a query just seemed a bit hacky to me, having said that if you think it's OK, then it's definitely good enough for me :)

Thanks for your thoughts.

[John Whish]

Another solution I thought of was to just have all the getters and setters for all the users properties which would be unused for most users. That doesn't seem very clean to me either.

[Adam Drew]

@Brian,

Are you advising the use of a manyToOne instead of oneToMany with
large collections strictly because of transfer performance issues? Or
is there some other relationship guidance behind that?

I understand the potential for issues with only having oneToMany
relationships.. but so far, I have been working on a fairly complex
model with over 200 objects.. some of those objects have oneToManys
with collections of 10K+,

I use lazy, proxied, oneToManys on those transfer objects. Then i use
a TIBO for displaying, paginating, or filtering those large
collections where performance issues arise... It's not done yet, but
honestly I'm very pleased with the overall functionality and
performance of this approach thus far, however the TIBO is not a part
of the transfer.... so i understand that, as a best practice with a
strict transfer implementation.. that a manytoOne maybe the best route
for performance concerns until future features are added... However,
The TIBO is a topic i had brought up in the past on the list.. and is
something I plan on posting a component someplace one day for anyone
here who is interested when I can get the implementation cleaned-up
and stable enough for what I need to do with it.. hopefully it works
for someone other then myself.

I guess I'm just curious if I'm missing some caveats about the
oneToMany+TIBO approach I've been spending some time on... I have have
not yet worked it out on manyToOnes.. but plan on doing so... and I
would be interested to hear your input on this topic in regards to
oneToManys vs manyToOnes...

Regards,
Adam

On Mar 13, 8:15 am, Brian Kotek <brian...@gmail.com> wrote:
> How large is "a large number of objects"? What you probably need to do is
> revisit the relationship since you shouldn't be using oneToMany in
> situations where there are a large number of objects. Use manyToOne.
>
> But you can override the configure() method in the Decorator. From the
> documentation:
> Setting up values in the Decorator
>

> configure()<http://docs.transfer-orm.com/html/transferapi/transfer/com/TransferDe...>on

> the TransferDecorator will be run when the object is first created.
> You
> can overwrite this when you implement your own Decorator CFC, and is a
> perfect place to set default values for your Transfer Object.
>

> It should be noted that configure() is run *before* the object is populated,

> and therefore, any values it sets will be overwritten by those retrieved
> from the database.
>
> You might also look at using the Transfer Proxies for this relationship
> instead of the real objects.
>

[John Whish]

@Adam, that TIBO sounds interesting. Does it pick up the 'custom' behaviour from the decorator or do you end up duplicating code?

[Adam Drew]

Yes... basically it "wraps" the TransferDecorator using CF8's
onMissingMethod() functionality.... I would suggest that if you
haven't yet, go check out the work Peter Bell did with the coldfusion
IBO on riaforge.org.. The TIBO started as a slightly modified IBO, it
now has a fairly similar api, but extends my BaseObject and also
incorporates my own Paginator and Filter objects, its implementation
in Transfer is all is based on the abstract transfer service/gateway/
decorator ideas that Bob S. and Paul M. and others have been
graciously sharing with us for a while (great work!!!)....

At the moment, I am using the TIBO mainly in my UI while i develop it
rather then for things going on in my service layer, but I don't see
any reason I couldn't or shouldn't also be able to use it anywhere i
need to as well..
However, right now I am in the middle of fleshing out the basic filter/
paginator functionality when it comes to user sessions & other caching
issues...... I think I have it mostly licked, but I need to do some
more work.
Another topic for another day maybe.

Basically, It all looks something kinda like this...
__________________________


AbstractDecorator:
getDisplayTitle() : string
return getTransferObject().getName();
__________________________

UserDecorator -> AbstractDecorator
getDisplayTitle(); //overrides abstractdecorator,
return getTransferObject().getUsername();
__________________________

BaseObject:
init(args);
configure();
get(val);
set(val);
new(object);
add(scope,val);
exists(val);

onMissingMethod() :
get[PropertyName] or get[InstanceVar]
set[PropertyName] or set[InstanceVar]
new[Transient] // (Paginator/Filter)
__________________________

TIBO -> BaseObject
configure();
load(gateway,paginator,filter);
loadStruct(pos);
filter(string);

reset();
next();
populate();
moveTo(pos);

getPKValue()
getIsPersisted() : Always TRUE
getIsDirty()

getPosition();
getCount();
getPage();

getPaginator();
getFilter();
__________________________


Controller:
ibo = userService.getIterator();
__________________________

View:
ibo.reset();
#ibo.getPaginator().render()#
if (ibo.next()) {
#ibo.getPKValue()#
#ibo.getDisplayTitle()#
#ibo.getIsPersisted()#
}

Hope that helps you visualize it a little bit.. its a lot more
complicated .. so you should start by checking out the work by Peter,
Bob, and Paul on riaforge if you haven't already....

Please excuse any typos and psudo code, been a busy day..

Regards,
-adam

[Brian Kotek]

I'm advising against it because it's the incorrect use of oneToMany. Technically it's a performance issue, but that's not Transfer's fault. It's an issue of using the wrong kind of relationship for the situation.

Because a oneToMany creates a parent-child relationship, if you have 10,000 related objects in the collection, you're absolutely going to have major, major problem. Whether you use lazy or proxied objects isn't going to make a difference. There are internal situations where Transfer will check things about the relationship such as determining if an item is already in the collection. That means loading ALL 10,000 of the related objects. With proxy objects it may load 10,000 proxies (maybe Mark can confirm), but that's still a gigantic number of CFCs being instantiated.

In essence, don't assume anything about how or when Transfer is going to trigger a load of all 10,000 of those children. If you have a OneToMany, and one side of that relationship has too many objects (and by too many, I mean any number that is more than you would want transfer to create in one shot), then don't use a OneToMany. Use a ManyToOne, i.e. person.setCompany(company).

OneToMany, is meant for very specific, tightly related relationships. For me, that means something like 10 or maybe 100 related objects at most. The actual number is up to you but following my own advice, I would not really want Transfer to be creating more than 100 related objects.

For example:

A User has a BillingAccount. If your business model is such that most BillingAccounts have one User, or if most BillingAccounts have only a few Users, then use a OneToMany if you wish. i.e. User.setParentBillingAccount().

However, a User also has an AccountType. AccountType is probably quite generic and could be associated with thousands or tens of thousands of Users. In such a case, do NOT use a OneToMany. Use a ManyToOne, i.e. User.setAccountType().

[Brian Kotek]

If any individual User only has a few attributes, then a oneToMany is fine. If a User had 5,000 attributes, it would not be fine.

200,000 instances is a lot, but "a lot" depends on your hardware, RAM, heap size, etc. With 200,000 simultaneous connections, I'm assuming this is a monster server. So it will probably be ok. Before you go too far I'd just run a load test and see how it works.

[Peter Bell]

I haven't done any load testing on ColdFusion or Transfer with anywhere near that number of users, but as Brian said I'd do a really quick spike and some load testing before you go too far with this. Mark, shoot me down if I'm wrong, but I wouldn't call that number of simultaneous users a typical use case for Transfer (or even for a ColdFusion app). If you're really gonna have that kind of load out of the gate, you might find the speed of development benefits of CF are outweighed by the performance benefits of (say) a java based app. Also I'd be looking seriously at clustering strategies (I don't know how easy/possible it is to cluster with Transfer) as I'm not sure if you'd want 200,000 logged in users running on a single machine (assuming they interact with the server with some frequency). 

Best Wishes,

Peter

[Adam Drew]

Thanks Brian.. your response is in line with my general understanding
of the o2m vs m2o relationships, and I was by no means blaming
Transfer for anything. I think I hijacked the thread and took it off
topic, but your post reminds me of a good blog I read and must have
forgotten about o2m vs m2o... maybe it was from you or maybe it was
Brian Rinaldi, IDK.. either way, I clearly remember reading a similar
relationship guidance for transfer models.

But what motivated me to deviate from that and possibly even missuse
the o2m, as you have pointed out, was this particular case:

Lets say I am a processing job shop, I receive multiple lots of
different types of object made of different types of materials
manufactured by different companies and sent to me from different
customers every day, of every week, of each month, of every year. I
process these parts, and then I ship them back, and collect payment..
these "parts" are complex and so is my "process".. I have many
customers.. some of my customers have 10,000 different parts, with
different revisions, and they will request a quote or send these parts
in each revision for me to "process"....

so I said to myself.. these are tightly related relationships.... this
isn't a user has an account type.. in a "known" system where things
can be defined as m2o or in some other way.....

and i thought:

Self, A customer has many parts

, then I thought...

Self, this part belongs to this customer.

but then I found out that a customer generated by transfer doesn't
know diddly about the parts which it could have when it's defined as
m2o ..

then I said "Self, WTF am I to do?"

Is this a cust->part = o2m or a part->cust = m2o?

To find out if I have in fact taken the wrong approach with my TIBO
work-around is exactly why I'm here.. I don't know enough about this
to go-it-alone... so all of your input matters, and I appreciate it!


so, lets get back to my case and say, "a customer has many parts"....
but lets say I never really 'ask' transfer to give me *all of those
parts only 1 of them when requested via lookups done with querys
generated via TQL... in this TIBO I inject a cloned empty transfer
object of that concrete type.. This gives me a new, technically
decorated "TransferObject" which I can work efficiently with in my UI,
Controller, Service, Gateway, Transfer.. it is a "vitural" transfer
object which I re-populate via a cached query generated via TQL, or
sql, or a file or directory query.. and then I override some of base
transfer properties to "simulate" a real transfer object.... so far
this is working prety nicly as a o2m with 10k+ objects using 1 cfc,
not 10,000+... FYI my current dev box is running cf 8.0.1 | jvm
1.6.12 @ 1.3gb heap 256perm | transfer 1.1 | cs 1.2 | cb 2.6.3 | majik
0.7 | jquery 1.3.2 on IIS 7 / WIN7x64 - Intel QuadCore @ 2.6ghz w/8gb
ram.

although I understand that I can't assume that it won't happen, I am
NOT actually ask transfer to generate 10,000 cfcs in that relationship
for me while still getting the o2m transfer functionality,
performance, and ancillary support functions I need when and where I
need them..

I don't know if the above context helps you see my motivation in a
different light.. but it's also worth noting that In general, as a
rule of thumb, transfer is very good at doing it's job... maybe I just
have problems understanding some of my percieved special
relationships.. the only thing I could say is that maybe o2o or bi-m2m
support would could helpe creative developers come up with other ways
of dealing with my issue.

I think with the o2m+TIBO, I have attempted to beat the system by
having all of the transfer functionality that i need, when and where I
need it.. even if i'm designing around my own created issues with the
miss-use of a relationship.. I learning something.


Thanks & have a good weekend all!
-adam

[Mark Mandel]

Yeah... I have to wonder at 200,000 simultaneous single users on a
single server, regardless of the application server, let alone the
framework. If you have that high a load, I'd be looking at a
clustered solution right off the bat.

Would anyone disagree with me?

Mark

--
E: mark....@gmail.com
W: www.compoundtheory.com

[Brian Kotek]

Either a cluster of separate servers or a monster system running numerous instances.

[John Whish]

Hi All, thanks for the input - I really appreciate it. 

I should have said that I'm looking at implementing this across 3 CF enterprise servers and then having the load balancer point users to a specific server and keep them on it (mind's gone blank but I think this is called sticky session). So, really I suppose it is 200,000 users divided across 3 servers.

I know that the performance issue is a ColdFusion not Transfer. I'd love to be able to use something like hibernate / Java / groovy but sadly I don't have the time to learn the skills well enough to use them on a project of this scale. I've gone for the rather pragmatic "better the devil you know" approach. This is partly because I believe (and I may be proved wring!) that CF9 will be faster at creating objects (although probably nowhere near Java) and I can always swap out the DAL to use hibernate at later date.

I've been giving this a lot of thought over the weekend and decided to drop the Transfer relationship and do it some other way, so it's good to see that you guys agree that using composition here is not the way to go. 

@Adam, I did have a look at using Transfer with an IBO a while ago but didn't explore it as far as you have, do I'd be interested to see it once you're ready to release it.