Match my $800 donation for Transfer caching improvements

[Brian G]

Mark and I chatted tonight and discussed the possibilities of
restructuring Transfer's caching architecture to be pluggable as he
posted over the weekend. The idea is, out of the box, to provide a
Javaloader 1.0/eHcache implementation that would be more flexible and
would solve the memory issues that many of us are seeing. This
pluggable architecture will support any caching system - Java-based
solutions like eHcache or ColdFusion solutions like Coldbox's cache or
anything else.

I'm putting up $800, or half the money needed, to pay for Mark's time
to solve this issue. If someone will match my contribution, Mark will
schedule the time now and we'll have a solution before the end of
October.

Who will match me and support Mark's great work on Transfer?


Brian

[Brian G]

So how many of you have saved even one day of writing SQL or objects
by hand because of Transfer? How about two days? Maybe a week? Has
your entire team probably gained a month of productivity by, day in
and day out, using Transfer instead of doing it by hand?

Tell your boss that an $800 donation to an Open Source product (or
hell, 2 x $400 or 4 x $200) will greatly improve the package that your
application depends on. Reward Mark for answering all of our inane
questions. Support open source authors who make our lives easier.

Nobody has $800 to match my offer? For shame...


Brian

[Josh Nathanson]

How about 20 people for $40 or 40 people for $20? At least you'd get
something that way.

I just donated $40, hopefully that'll get things started.

The donation link is at the Transfer homepage, right hand column:

http://www.transfer-orm.com/

-- Josh

[Nando]

Yeah, a lower amount spread out among a lot of people would help. I'm in for $50 bucks. Hit hard here by the economic slowdown. Gimme til the day after tomorrow to process it.

How about a plea that everyone on the list respond to this thread and contribute something to the effort. Just hit the reply button, sit for a moment of reflection like I just did, and type in a pledge. Almost everyone can afford something.

Nando

--

Nando M. Breiter
The CarbonZero Project
CP 234
6934 Bioggio
Switzerland
+41 76 303 4477
na...@carbonzero.ch

[Gert Franz]

Mark,

 

we just finished the EHCache – clustered cache implementation for Railo. So that you can use a distributed or synchronized cache across a cluster and store for example all CachedWithin queries in the cluster cache. If you like I can provide some information and source code about that, as a contribution to it as well…

 

Greetings from Switzerland

Gert Franz

 

Railo Technologies       Professional Open Source

skype: gert.franz            ge...@getrailo.com

+41 76 5680 231               www.getrailo.com

[Will Swain]

Hi,

 

Thanks for all your work on Transfer Mark.

 

Will there be a CF9 compatable stable release by the time CF9 is officially released (whenever that is - I have a feeling it will be fairly soon)? I seem to recall that you mentioned only the bleeding edge version worked with cf9 at the moment, and indeed I had to upgrade to that version to get it running on my test platform.

 

Apologies if this has already been covered - I did a quick search on the mailing list archives and couldn't see anything.

 

Cheers

 

Will

 

[Aurélien DELEUSIÈRE]

Hello all -

Because I think transfer is a great framework and I believe that Mark does a very good job, I've just donated $800 to support this cache enhancement.

In fact,I'm working with transfer more than one year today and I have many websites and RIA in production status and it's a real nightmare about memory leaks and CPU bottlenecks. I've deactivated the cache to speed up my applications... (for example : paris-exception.fr is powered by CFMX8, MachII and transfer connected with several CRM legacy applications,  welcome to Paris to every one ;-) ).

Thanks,
Aurelien, France, Paris

2009/9/21 Brian G <brian-...@vfive.com>

--
Aurélien Deleusière
Mobile :  +33 (0)6 83 78 83 42

ad e-consulting
expertise 2.0

104, Grande Rue - 92310 Sèvres
Fax : + 33 (0)1 49 66 15 53
S.A.R.L. au capital de 8500€
R.C.S. Nanterre 50177609000018

[Dorioo]

Hmm. I think you meant:

http://www.paris-exception.com/

Without the "www", it shows up as "under construction".

- Gabriel

2009/9/29 Aurélien DELEUSIÈRE <adele...@gmail.com>:

[Aurélien DELEUSIÈRE]

yeah i've just see and open a ticket... ;-) thanks!

2009/9/29 Dorioo <dor...@gmail.com>

[Mark Mandel]

I would like to release a stable version with a rewritten pluggable Cache implementation, but with MAX, cf.Objective(ANZ), ColdSpring, JavaLoader and the various other community stuff I am doing I am stretched a little thin at the moment.  Hence the call for the bounty.  It enables me to keep providing for the Community while also letting me put food on the table ;o)

At this stage it will most likely be after cf.Objective(ANZ), depending on some other factors.

Mark

--
E: mark....@gmail.com
T: http://www.twitter.com/neurotic
W: www.compoundtheory.com

[Mark Mandel]

...and I just saw the bounty come through! Wow!

I'll write more in the other thread, but we'll start nailing down some dates.

Mark

[Mark Mandel]

Wow!

What a way to wake up in the morning!

I will coordinate with Brian G (I can never spell your last name Brian ;o) ), and lock down some dates for getting this stuff done and out the door.

Which will also give us a date for a CF9 stable release.

Thanks guys! We get to build some cool stuff here!

Mark

2009/9/30 Aurélien DELEUSIÈRE <adele...@gmail.com>

[Brian G]

Thanks to those who stepped up (whether it be $20 or $800) to make
this happen: YOU GUYS ROCK! I've just sent my $800 (917AUD) to Mark
so he'll have a little more than we initially bargained for, which
should keep him motivated for when he feels like going to bed. :)


Brian (last name rhymes with Ghirardelli)

On Sep 29, 3:44 pm, Mark Mandel <mark.man...@gmail.com> wrote:
> Wow!
>
> What a way to wake up in the morning!
>
> I will coordinate with Brian G (I can never spell your last name Brian ;o)
> ), and lock down some dates for getting this stuff done and out the door.
>
> Which will also give us a date for a CF9 stable release.
>
> Thanks guys! We get to build some cool stuff here!
>
> Mark
>

> 2009/9/30 Aurélien DELEUSIÈRE <adeleusi...@gmail.com>

>
>
>
>
>
> > yeah i've just see and open a ticket... ;-) thanks!
>
> > 2009/9/29 Dorioo <dor...@gmail.com>
>
> >> Hmm. I think you meant:
>
> >>http://www.paris-exception.com/
>
> >> Without the "www", it shows up as "under construction".
>
> >> - Gabriel
>

> >> 2009/9/29 Aurélien DELEUSIÈRE <adeleusi...@gmail.com>:

> >> > Hello all -
>
> >> > Because I think transfer is a great framework and I believe that Mark
> >> does a
> >> > very good job, I've just donated $800 to support this cache enhancement.
>
> >> > In fact,I'm working with transfer more than one year today and I have
> >> many
> >> > websites and RIA in production status and it's a real nightmare about
> >> memory
> >> > leaks and CPU bottlenecks. I've deactivated the cache to speed up my
> >> > applications... (for example : paris-exception.fr is powered by CFMX8,
> >> > MachII and transfer connected with several CRM legacy applications,
> >> welcome
> >> > to Paris to every one ;-) ).
>
> >> > Thanks,
> >> > Aurelien, France, Paris
>

> >> > 2009/9/21 Brian G <brian-goo...@vfive.com>

> E: mark.man...@gmail.com

[Mark Mandel]

First of all - big thanks to Brian for organising this in the first place.

Next, thanks to the people who donated, Brian, Josh and Aurelien, you guys are amazing. Seriously. Amazing.

Gert, thanks for the offer, I may take you up on that, I will see how I go with eHCache.

(Un)Fortunately, October is completely full with stuff, with MAX and whatnot going on, so I've locked down the first week in November to roll this out.

It's a bit far away, I know, but literally yesterday all my time got sucked up out of October, which is just one of those things that happens.

Expect to see a lot of emails as we hit November, as the pluggable cache architecture will probably deprecate a lot of the caching config that currently exists in the Transfer XML, and move it into whatever files the pluggable cache will take - so I will be looking for some direction from the community on that.

Exciting, Exciting stuff.

Mark

--
E: mark....@gmail.com

[Will Swain]

Brilliant. Thanks Mark - I missed the bounty thread but have now read through it.

 

Cheers

Will

---

From: transf...@googlegroups.com [mailto:transf...@googlegroups.com] On Behalf Of Mark Mandel
Sent: 29 September 2009 23:40
To: transf...@googlegroups.com
Subject: [transfer-dev] Re: Transfer and CF9

size=2 face=Arial>No virus found in this incoming message.
Checked by AVG - www.avg.com
Version: 8.5.409 / Virus Database: 270.13.113/2400 - Release Date: 09/29/09 05:54:00

[Aurélien DELEUSIÈRE]

Hello Mark -

I didn't have time to get back on this. The amazing guy here is you, you're always available, getting response for everyone and everything...

If I can help you for testing, tell me I'll try to take some time on this (hot) topic.

Thanks

2009/9/30 Mark Mandel <mark....@gmail.com>

[Mark Mandel]

Starting on this today.. more details will be forthcoming.

Mark

2009/10/2 Aurélien DELEUSIÈRE <adele...@gmail.com>

[Aurélien DELEUSIÈRE]

Good news!

Let me know if I can help...

2009/11/1 Mark Mandel <mark....@gmail.com>

[Mark Mandel]

I'm about to go to bed, but I'll give you all an update.

The <objectCache> section now looks akin to this:

    <objectCache>
      <defaultcache provider="transfer.com.cache.provider.EHCacheProvider">
        <setting name="config" value="/test/resources/ehcache.xml"/>
      </defaultcache>
      <cache class="none.Basic" provider="transfer.com.cache.provider.NoCacheProvider"/>
      <cache class="none.Child" provider="transfer.com.cache.provider.NoCacheProvider"/>
  </objectCache>

So the 'provider' attribute specifies what Cache Provider, which is a CFC that extends AbstractBaseProvider, which can be found here:
http://svn.riaforge.org/transfer/transfer/branches/pluggable_cache/com/cache/provider/AbstractBaseProvider.cfc

The <setting> values get pass to the init() of the Provider.

(Some more basic statistic based methods will be added later for simple reporting, and tied back into the CacheMonitor)

So you see you can set up a defaultCacheProvider, and also use a specific CacheProvider for specific classes as well - so you can mix and match caches (possibly at your own peril ;o) )

Because you can also extend the Provider yourself, you can do all sorts of weird and wonderful things.

I'm writing up an EHCache one as the default, which will have some limitations as to the platform it can do things on (No CF7, quite possibly not going to work on some shared hosts, due to classpath restrictions), so I'll be looking for other people to do some intergration as well (A ColdBox cache adapter would be really cool, or any other cache framework).  The only major dependency is that the cache framework has to be able to tell you when something gets discarded, as that is how Objects know to drop collections and the like when things get deleted/discarded from the cache.

Anyway, it's almost midnight here, I'm gonna grab some shut eye... and then get up tomorrow morning and rip apart Transfer some more as the whole nation stops for a horse race...

Mark

[Mark Mandel]

Just letting you all know this is alive and kicking.

I'm just polishing off the final tests, and resolving any issues that I have found.

While this is a break in backwards compatibility, this has simplified much of Transfer architecture, and streamlined it very nicely.

Should see some code that is ready for testing in a day or two.

Mark

[Aurélien DELEUSIÈRE]

Hello Mark -

Thanks for the work. I've checked out the trunk. I'm going to switch this today. I'm discovering the ehcache.xml config (first time). If you have a simple ready to use xml file I'm intereted in.

Thanks -

Aurelien

2009/11/4 Mark Mandel <mark....@gmail.com>

[Mark Mandel]

Aurelien,

The code in progress is not stored in the trunk, it is stored in a branch.
http://svn.riaforge.org/transfer/transfer/branches/pluggable_cache/

I'm not 100% sure this is ready for general usage yet, as there are several unit test suites that are not yet completing.  Use very much at your own risk.

The default ehcache config is found here:
http://svn.riaforge.org/transfer/transfer/branches/pluggable_cache/com/cache/provider/ehcache-lib/ehcache.xml

Although there are plenty of examples in the EHCache documentation.

Mark

2009/11/5 Aurélien DELEUSIÈRE <adele...@gmail.com>

[Aurélien DELEUSIÈRE]

Yes, this is the branch I've checked out and I found this xml too.

I'm conscient that is not final, I'm using this on my own dev env. I'm just curious to see how is the pain... ;-)

I've started to play with, no "big" problem for the moment. The application starts.

The events binding seems not working. Will it work or I've to investigated on ehCache side ?

ApplicationObserver.cfc->init(application.oTransfer) :

// auto register this observer

oTransfer.addBeforeCreateObserver(this);

oTransfer.addAfterCreateObserver(this);

oTransfer.addBeforeUpdateObserver(this);

oTransfer.addAfterUpdateObserver(this);

oTransfer.addBeforeDeleteObserver(this);

oTransfer.addAfterDeleteObserver(this);

Thanks!

2009/11/5 Mark Mandel <mark....@gmail.com>

[Mark Mandel]

The EventManager Unit Tests are the ones that are currently failing at the moment ;o)

[Mark Mandel]

Just to keep updating -

I'm still working on this. EventManager Unit Tests are passing, but a few are failing in Composite Keys.

cf.Objective(ANZ) is taking up a lot of time this week, but I hope I have some time to keep working.

Mark

[Mark Mandel]

Okay guys, feel free to test away!!!

Only caveat - I've yet to come up with a common API for statistics (So the Monitor doesn't work), and also how to expose the cache providers to the outside world, so you can interact with them natively if need be, but that shouldn't stop you getting stuck in.

Mark

[Aurélien DELEUSIÈRE]

great... I'll restart my test this evening (it's 10:00am here)... Tomorrow is a bank day I'll restart Friday and keep you informed!

First results of load test I ran last week are better with ehCache in term of memory usage and recycling. Seems to be encouraging...

Thanks

Aurélien

2009/11/10 Mark Mandel <mark....@gmail.com>

[Mark Mandel]

I also ripped out a stack of code in Transfer that is no longer needed as the caching layer is nowhere near as complicated as it once was - which should also streamline things a lot.

Mark

2009/11/10 Aurélien DELEUSIÈRE <adele...@gmail.com>

[Brian G]

I worked with Mark this afternoon to convert my system and get it
running on the new pluggable architecture. Pretty straightforward and
I've documented it for anyone else who wants to give it a quick go:

http://www.ghidinelli.com/2009/11/18/using-transfer-ehcache

Hope that helps! I've got it on my staging server now and will likely
try it on production starting this weekend.


Brian

[Mark Mandel]

Nice writeup!

I should do my own blog post too! :D

Mark

--

Before posting questions to the group please read:

http://groups.google.com/group/transfer-dev/web/how-to-ask-support-questions-on-transfer

You received this message because you are subscribed to the Google Groups "transfer-dev" group.
To post to this group, send email to transf...@googlegroups.com
To unsubscribe from this group, send email to transfer-dev...@googlegroups.com
For more options, visit this group at http://groups.google.com/group/transfer-dev?hl=en

[Mark Mandel]

Just writing everything in this thread, so there is only 1 place to look.

There is now a TransferFactory.shutdown() method that needs to be run when a TransferFactory is destroyed/reinitialised.

This calls shutdown() on any of the configured CacheProviders, in case they need this.

Therefore you will need to call TransferFactory.shutdown() onApplicationStop() in your Application.cfc, so that when the application times out, the cache is also expired.

This is only neccessary if you are using the EHCacheProvider, but other cache providers in the future may also need this functionality.

Mark

[Aurélien DELEUSIÈRE]

I'm back on this today, I were finally too busy last week.

Ready to use cut and paste:

<cffunction name="onApplicationEnd" returnType="void">

<cfargument name="applicationScope" required=true/>

<cfscript>

arguments.applicationScope.tranferFactory.shutdown();

</cfscript>

</cffunction>

2009/11/23 Mark Mandel <mark....@gmail.com>

--

[Mark Mandel]

Should that be 'transferFactory' ? ;)

Mark

2009/11/23 Aurélien DELEUSIÈRE <adele...@gmail.com>

[Brian G]

This would impact unit tests that instantiate Transfer as well,
right? For example, I have coldspring instantiate my Transfer
instance and I instantiate CS to run unit tests over and over and
over. I can see how a suite of tests could exhaust the memory
quickly.

How would you suggest handling this? Some kind of generic teardown?


Brian

[Mark Mandel]

On Tue, Nov 24, 2009 at 7:52 AM, Brian G <brian-...@vfive.com> wrote:

This would impact unit tests that instantiate Transfer as well,
right?   For example, I have coldspring instantiate my Transfer
instance and I instantiate CS to run unit tests over and over and
over.  I can see how a suite of tests could exhaust the memory
quickly.

How would you suggest handling this?  Some kind of generic teardown?

Sounds good to me :D

I also didn't mention - the monitor is also been completed in SVN, although the API is slightly different (check out the CFC, it's pretty simple).

The monitor report custom tags have also been updated.  You can't get at Cache settings with the new monitor because each cache is so different.  I may look at down the line having a capability in the provider to output a description of the cache settings, or something similar, for reporting purposes... (depends on demand I guess).

Mark
 

Brian

> Therefore you will need to call TransferFactory.shutdown()
> onApplicationStop() in your Application.cfc, so that when the application
> times out, the cache is also expired.

--

Before posting questions to the group please read:
http://groups.google.com/group/transfer-dev/web/how-to-ask-support-questions-on-transfer

You received this message because you are subscribed to the Google Groups "transfer-dev" group.
To post to this group, send email to transf...@googlegroups.com
To unsubscribe from this group, send email to transfer-dev...@googlegroups.com
For more options, visit this group at http://groups.google.com/group/transfer-dev?hl=en

[Brian G]

On Nov 23, 2:38 pm, Mark Mandel <mark.man...@gmail.com> wrote:
> > How would you suggest handling this?  Some kind of generic teardown?
>
> Sounds good to me :D

Working like this (based on http://www.mail-archive.com/transf...@googlegroups.com/msg00730.html):

<cffunction name="setUp" returntype="void" access="public">
<cfscript>
variables.beanFactory = createObject("component",
"coldspring.beans.DefaultXmlBeanFactory").init();
variables.beanFactory.loadBeansFromXmlFile("/setup/coldspring.xml",
true);

variables.transferFactory = variables.beanFactory.getBean
("ormService");
variables.transferFactory.getTransaction().advise(this, "^test");
</cfscript>
</cffunction>

<cffunction name="tearDown" output="false" access="public"
returntype="any" hint="">
<!--- shutdown transfer cache --->
<cfset variables.transferFactory.getTransfer().discardAll() />
<cfset variables.transferFactory.shutdown() />
</cffunction>

<cffunction name="testPrepareEventThingy" output="false"
access="public" returntype="void">
...
...
...

<cftransaction action="rollback" />
</cffunction>

The only bummer to this approach is putting the rollback at the bottom
of every single test function. Is there a way to make the advice run
a function with the rollback automatically when it completes?

Thanks!


Brian

[Mark Mandel]

Just got my own write up done:
http://www.compoundtheory.com/?action=displayPost&ID=447

Mark

--

Before posting questions to the group please read:
http://groups.google.com/group/transfer-dev/web/how-to-ask-support-questions-on-transfer

You received this message because you are subscribed to the Google Groups "transfer-dev" group.
To post to this group, send email to transf...@googlegroups.com
To unsubscribe from this group, send email to transfer-dev...@googlegroups.com
For more options, visit this group at http://groups.google.com/group/transfer-dev?hl=en

[Mark Mandel]

About to go to bed - but just committed to SVN a AbstractBaseAsyncDiscardProvider - that allows for object discards to be fire asynchronously.

This is a big plus for the EHCacheProvider (which uses this), because it won't lock down a object load while waiting for a discard to fire, which can happen when doing a get() request.

More tomorrow...

Mark

[Mark Mandel]

If anyone is using the EHCacheProvider, I would suggest updating.

Elliot Sprehn and I found some bugs and performance tweaks together today that should fix several things that have been committed to SVN.

Mark

[Aurélien DELEUSIÈRE]

Hello Mark, hello all,

I've been working on my memory leak this week, it's a real nightmare to identify my problem. I've learn a lot thanks to excellent blog about JRockit and co (I feel less alone...).

I moved to the ehCache version (clearly faster), but my issue is still the same (see attached with 10 users during 20 minutes...). The point is when I deactivate the cache, there is no leak. When the cache is activated former one or new one, there is a leak of cfTransferObject2ecfc47864934 objects (thousands growing). No doubt there is a problem in our own code, I continue my investigation but it's hard to define where it is...

By the way, I try to move to CF9. I've got good results without any changes. It seems that the memory is still here but very very little.

With CF9, I've see this problem :

SLF4J: Class path contains multiple SLF4J bindings.

SLF4J: Found binding in [jar:file:/C:/dev/bin/ColdFusion9/runtime/../lib/slf4j-log4j12-1.5.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: Found binding in [jar:file:/C:/dev/workspace/lib/transfer.eh/com/cache/provider/ehcache-lib/slf4j-log4j12-1.5.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.

I don't know if there is something to do with.

I'll keep you informed...

Aurelien

2009/12/14 Mark Mandel <mark....@gmail.com>

--

[Aurélien DELEUSIÈRE]

Sometime I get this message, do you have any idea?

12/18 18:14:11 Error [cfthread-0] - TRANSFER.COM.CACHE.PROVIDER.ABSTRACTBASEASYNCDISCARDPROVIDER.FIREDISCARDEVENT.15398515.1: Could not find the ColdFusion component or interface transfer.com.events.collections.AfterDiscardObserverCollection.

[Mark Mandel]

Aurelien,

What are your cache settings?

Mark

2009/12/19 Aurélien DELEUSIÈRE <adele...@gmail.com>

[Mark Mandel]

Is this on CF8, or on CF9?

How is Transfer mapped?

Mark

2009/12/19 Aurélien DELEUSIÈRE <adele...@gmail.com>

Sometime I get this message, do you have any idea?