By the Poolside, activations and scalability

[Florin Marcus]

Probably some of you have been already seeing the excellent series of videos by Chris Muir on Application Module Pooling, I think these videos are a definitely a must-watch for everyone who takes ADF seriously. Don't be fooled by the "Essentials" tag and rule it as 'beginner level' , because is nothing like that.

That being said, while this topic is still hot, I would like to raise few questions in relation with AM activation/passivation life-cycle:

1. Is common knowledge that passivation happens under high loads, that developers should test their functionality for activation-safety. But passivation can also happen under low or normal loads, when an AM instance is inactive for a certain interval of time (controlled though 'Idle Instance Timeout'  setting) . These kind of passivation is meaningless from my perspective: Why do I need to save server memory when the server is NOT under high load and when there is plenty of HEAP available? There is already a timeout option on HTTP session(web.xml)  who  nicely cleans resources and returns AM's back to the pool, without passivating. 

Once I've thought that by having AM timeout larger than HTTP session timeout we can prevent this unnecessary passivation lifecycle, but it turns out is more complex than that. Currently we are using jbo.ampool.timetolive=-1 and jbo.ampool.maxinactiveage=<ridiculously large number> and in the last weeks there were no activations in production system.

2. Furthermore, if the ADF application doesn't require a fail-over clustering solution, I see no use for activations.

A bold statement indeed, but as long as activations increase scalability by saving server memory, but with the increase of of database/network operations, I would say: "no, thank you". Why would I be worried about saving memory, when 32 Gigs of RAM are cheaper than a consultant daily rate?

All ADF systems I was involved in had the number of users clearly known in advance, so we could scale AM pools accordingly. 

3. As long as I can suppress activations from happening, would that mean I don't need to to check my code for activation-safety? For those who never really went though this process in full in their applications, I must say that fixing passivation bugs is far from straightforward, I was spending an important amount of time with trying to figure some cases out. 

4. Chris is asserting that a user session is consuming approximately 1MB RAM and I would back such statement myself. But only if AM has 'Disconnect Application Module Upon Release' (jbo.doconnectionpooling=true) enabled. Otherwise, server's memory consumption goes rocket-high and is not because of ADF, but since without this setting, JDBC driver is not closing connections, statements or result sets.

Of course, this is not related to ADF directly, is a 'problem' of Oracle JDBC driver: for instance, take any Java Web application supporting thousands of users and stop releasing the connections after each request and you will soon find out that the application will be chocking with a hundred of users.  I suspect this detail wasn't foreseen when AM pooling was architected.

Thanks,

Florin

P.S.

I was aiming for a more dramatic effect, as the things went quiet for some time around here.

[Jean-Marc Desvaux]

Florin,

Interesting point on activation/passivation.

I agree that it could be avoided in most cases, however I think you have also pointed to the danger of doing it : It would probably lead to being less rigorous on checking for activation-safety and being exposed for some reason to a hard-to-debug application if for some reason passivation has to be enabled in the future (increase in no of users, new reasons to enable passivation in a new version, etc..).

Having built your application highly dependent on an AM setting will just delay the debugging effort to a moment where you were not prepared to do so.

Our ADF developers have win some good fights recently to sort passivation issues and I agree with you it can be something hard to debug and test.

I must say that I would not feel confident if these activation/passivation issues were sorted out just by adding memory and disabling activation/passivation for the application.

That's why I still think it is important to make sure activation/passivation is enable and tested in all cases and eventually make any AM parameter change part of pure AM tuning.

-Jean-Marc

[Florin Marcus]

Thanks for your reply,  Jean-Marc,

I would say that activations should be avoided in all cases, since activations are an overhead as it re-queries all active view objects of current screen.

When a system is under heavy load, when the number of database operations are at highest,  how are activations suppose to improve your scalability by further increasing the number of queries fired against the database?!   I see activations as  'last line of defence', preferable to  server crash, something that should happen under exceptional circumstances only.  

On the other hand, there is no option to suppress activations from happening. The only thing you can do is  configure your AMs to handle a number of users without passivating, and as long as the number of users is not increasing, passivations/activations won't happen. (This is the main statement of my post and I am trying to get other people opinion on it).

Furthermore,  we need a distinction between minimising activations, on one hand, and testing your code for activation safety, on the other hand. If I understand correctly, you are referring to lack of activation-safety as being a potential danger, in case you might need passivations in the future.  I do agree is better to have your code tested against activation-safety - in case you might need a fail-safe clustering solution in the future. But I fail to see any other reason for activation-safety.  There are few ADF applications out there without activation-safe support and fixing all these problems will take time and effort, so minimising no of activations will minimise the errors coming from lack of activation-safe testing.

There is nothing preventing you from having your ADF code as activation-safe, while aiming for minimising (or 'de-facto' suppressing) the number of activations.

Thanks,

Florin

--
--
You received this message because you are subscribed to the ADF Enterprise Methodology Group (http://groups.google.com/group/adf-methodology). To unsubscribe send email to adf-methodolo...@googlegroups.com
 
All content to the ADF EMG lies under the Creative Commons Attribution 3.0 Unported License (http://creativecommons.org/licenses/by/3.0/). Any content sourced must be attributed back to the ADF EMG with a link to the Google Group (http://groups.google.com/group/adf-methodology).

---
You received this message because you are subscribed to the Google Groups "ADF Enterprise Methodology Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to adf-methodolo...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Simon Haslam]

This is an interesting debate Florin!

 

What we are really discussing here is the legacy of a database-centric state managment approach dating back maybe 15 years. I too only see the point of passivation/activation to handle failure conditions. Arguably in the early days it may have helped compensate for memory limitations on 32 bit platforms, and generally smaller amounts of memory, but even then it degrades very unhappily under load.

 

This mid-tier state does not belong at all in the database IMO. I did some fairly simple tests in around 2008 comparing database-backed passivation to file-based (with passivate on every request to prove the point) and the performance difference (not unsurprisingly) was significant. Of course you then need a means of replicating the files (e.g. clustered filesystems come with their own problems) and the serialization is still happening which isn't ideal.

 

What does suprise me is that there's still no sign of Coherence in ADF (unless I've missed it!). At around a similar time Clemens Utschig put together some kind of prototype for caching the BC instances. Coherence would seem a much more natural means for managing state (though obviously wasn't available to Oracle in the BC4J days) and you would have thought you could replicate the BC classes with minimal change to ADF. I don't suppose Coherence solves the activation-safety issue (or perhaps not in all cases) but then if you were using Coherence*Web for HTTP session replication too maybe that would.

 

 

Simon

[John Sivertsen]

Thanks for opening this debate.
I have many of the same thoughts about activation/passivation mentioned here.

My experience with it is the following:
AM activation/passivation gives no benefit to performance or scalability in normal ADF projects.
It is a source of hard to find errors. The errors are often hard or impossible to fix without doing a major refactoring.
It causes unnecessary load to the database.
It makes it necessary to have "double up" on Weblogic managed server instances for testing. One with AM pooling turned on and one with it turned off.

I also find the subtlety the subject is threated in Fusion Developer's Guide strange. In version 11.1.1.6 there is only one mention of it, before the chapter about the subject on page 1500+. The same goes for JDeveloper. There is no UI switch for turning AM pooling on or off. The only way is to use -D startup parameter.

Has anyone tried to persuade Oracle to give the option to disable AM activation/passivation for application that don't have High Availability requirements?
Besides, activation-safety is no insurance that HA is achieved. Session scoped managed beans must also be made serializable and use ComponentReference API if appropriate etc.



// John

[Florin Marcus]

It has been more than 6 months since I've started this topic, 6 months since we took the decision of suppressing activations on three production deployments, one of which handles thousands of users daily.

All three systems are perfectly fine and we haven't seen a single activation since.

We moved on and  applied the  same strategy of suppressing AM recycling to another 5 ADF systems without encountering any problem.

Therefore, we can stand up and say that activations (AM Recycling)  are an unnecessary complication outside fail-over clusters.

Thanks,

Florin

[Leon Dorfling]

Hi Florin,

Can you give feedback wrt any performance gains please?

Regards

Leon

[Renan Monteiro de Castro]

Hello Florin,

Just adding my little contribution to the conversation. Some weeks ago we had serious trouble with the activation mechanism, http 500 errors were randomly popping out to our users. After some research I've discovered that the records inserted in PS_TXN were holding a negative value in COLLID column causing framework issues while trying to recover it (oracle.jbo package was at debug level). After two days of intense debugging here is what I did:

• Dropped PS_TXN table and PS_TXN_SEQ sequence
• After doing that the errors were still showing up in our logs
• I've did a full clean on our application and deployed it again
• The error disappeared after this...

I've attached all the information I had about the problem but Oracle said that without a way to reproduce it would be hard to proceed with the SR (#SR 3-9939732556).

My general concern is that activations adds an unjustified amount complexity in terms of what framework has to do to keep it working and what it brings back in terms of benefits. I still have no clue about what happened that time as well as Oracle engineers.

Att,

Renan Monteiro de Castro - Linkedin - Twitter

MG (OI) - (55)(32) 8827-4118

SP (VIVO) - (55)(11) 9-9727-6856

SP (TIM) - (55)(11) 9-5898-8794

Skype: renan.monteiro.castro

[Ayhan Güngör]

Hi Renan,

a little off topic; but we had the same issue about negative values for COLLID field at one of our client's production system about a year ago.

after some debugging/brainstorm sessions, we discovered that ps_txn_seq.nextval was over INTEGER.MAX_VALUE and somehow adf started to insert negative values to COLLID(number(10)) field, and it could not activate back because next sequence value and COLLID value in ps_txn was different.

solution was setting cycle flag and max value as INTEGER.MAX_VALUE for ps_txn_seq.

regards,

[Renan Monteiro de Castro]

Hello Ayhan,

Thank you for your information, I suggested incrementing the sequence to value close to ten digits to Oracle team but I got no answer atm. I'll add your information to the SR in order to help them diagnosting the issue.

Att,

Renan Monteiro de Castro - Linkedin - Twitter

MG (OI) - (55)(32) 8827-4118

SP (VIVO) - (55)(11) 9-9727-6856

SP (TIM) - (55)(11) 9-5898-8794

Skype: renan.monteiro.castro

[Florin Marcus]

Hi Leon,

Before answering, let me clarify few general elements:

The option of suppressing activations is available to only those who choose to ignore "Fusion Developer Guide" recommendations regarding AM Pooling

and use  "Disconnect Application Module Upon Release", jbo.doconnectionpooling=true. 

In this case, you won't need activations because you don't need to cleanup memory. That's because you don't cache tons of memory per user.

Furthermore, using jbo.doconnectionpooling=true is THE ONLY WAY to have a scalable ADF application. Is the difference between running your application with 4 servers, instead of 20.

Because this setting makes ADF behave like any other framework out there, when dealing with JDBC operations. 

Because ADF Framework Best Practices (as described in Fusion Dev Guide) cannot simply disregard underlying technologies best practices (JDBC in this case). 

Because a development framework becomes irrelevant once the application is live - the way technologies work together is all that matters.

That being said, here is a short answer to your question:

1. Activations are a performance overhead because when data  gets activated, the current user screen will be re-queried (transparently, of course). 

Activations artificially increase the number of database operations in the wrong moment: when number of users  is at highest, when the database server is already under fire.

2. Activations are a development overhead, increasing development time with around 10%.

Hope this helps,

Florin

[Jean-Marc Desvaux]

Florin,

Some comments on what we see with jbo.doconnectionpooling=true and false.

This is not to prove anything but to share what we see when we monitor with ADF version 11.1.2.4.0 with this setting set to true and also when set to false.

In both cases we have set jbo.ampool.timetolive=-1 and jbo.ampool.maxinactiveage=<lots of milliseconds> to avoid passivation.

The use case monitored is : A page is accessed with many tables and VOs involved. We click on a table row, then click on another one. At each click, user waits at least 5 seconds.

In case 1/. with jbo.doconnectionpooling=true :

The charts shows a new db connection and also a passivation occuring at each click.

It seems it is reflecting well what the ADF Dev guide says, i.e :

Extract from "44.2.6 What You May Need to Know About How Database and Application Module Pools Cooperate" :

[..]

If jbo.doconnectionpooling=true, then each time a user session finishes using an application module (typically at the end of each HTTP request), the application module instance disassociates itself with the JDBC connection it was using on that request and it returns it to the JDBC connection pool. The next time that application module instance is used by a user session, it will reacquire a JDBC connection from the JDBC connection pool and use it for the span of time that application module is checked out of the application module pool (again, typically the span of one HTTP request).

[..]

.. with the AM checking out of the pool translating to a passivation at each HTTP request.

The end result will be high scalability but slow performance for the user at each request.

In case 2/. with jbo.doconnectionpooling=false :

There is no passivation reported at each user request.

Maybe it is a bug with 11.1.2.4 or maybe I am missing something....

Do you also see these passivations when set to true ? If you do, do you also see a non negligible loss of performance on the user side ?

Thanks 

~Jean-Marc

On Tuesday, January 20, 2015 at 11:34:41 PM UTC+4, Florin Marcus wrote:

Hi Leon,

Before answering, let me clarify few general elements:

The option of suppressing activations is available to only those who choose to ignore "Fusion Developer Guide" recommendations regarding AM Pooling

and use  "Disconnect Application Module Upon Release", jbo.doconnectionpooling=true. 

In this case, you won't need activations because you don't need to cleanup memory. That's because you don't cache tons of memory per user.

Furthermore, using jbo.doconnectionpooling=true is THE ONLY WAY to have a scalable ADF application. Is the difference between running your application with 4 servers, instead of 20.

Because this setting makes ADF behave like any other framework out there, when dealing with JDBC operations. 

Because ADF Framework Best Practices (as described in Fusion Dev Guide) cannot simply disregard underlying technologies best practices (JDBC in this case). 

Because a development framework becomes irrelevant once the application is live - the way technologies work together is all that matters.

That being said, here is a short answer to your question:

1. Activations are a performance overhead because when data  gets activated, the current user screen will be re-queried (transparently, of course). 

Activations artificially increase the number of database operations in the wrong moment: when number of users  is at highest, when the database server is already under fire.

2. Activations are a development overhead, increasing development time with around 10%.

Hope this helps,

Florin

[..]

[Florin Marcus]

Hi Jean-Mark,

I think you are missing an argument from your configuration: jbo.txn.disconnect_level = 1.

Andrejus did some research on this matter, testing these settings under heavy load:

http://andrejusb.blogspot.co.uk/2011/11/stress-testing-oracle-adf-bc_16.html

Activations and passivations are controlled through : jbo.ampool.timetolive=-1 and jbo.ampool.maxinactiveage=<ridiculously large number>

These two settings would be enough to suppress activations, but you might run out of memory when number of users increases.

Actually you will see memory increasing  even when the same number of users will use the application for several hours.

Is why you will also need jbo.doconnectionpooling = true and jbo.txn.disconnect_level = 1).

Hope this helps,

Florin

 

--

[Jean-Marc Desvaux]

Florin,

We missed that one indeed ! And had no excuse to forget this one with Andrejus blog....

I am taking the opportunity to resume the whole discussion for any new reader :-

---------------------

If you build an ADF application and have absolutely no intention to run your application in a fail-over cluster environment, you should considerer setting these parameters for better performance and scalability and at the same time remove many hard to test & solve passivation-safe coding :-

jbo.ampool.timetolive=-1

jbo.ampool.maxinactiveage=<ridiculously large number> 

jbo.doconnectionpooling=true

jbo.txn.disconnect_level = 1

--------------------

Thanks Florin.

[koen.verhulst]

>> Has anyone tried to persuade Oracle to give the option to disable AM activation/passivation for application that don't have High Availability requirements?

I've been reading this post and I must say I totally agree with the majority of the comments posted over here: AM activation/passivation introduces unnecessary complexity.

Besides that ... It's often hard to solve problems due to AM a/p.  

Maybe someone from oracle can give some feedback/suggestions/ ... ?

 

Op vrijdag 11 juli 2014 17:08:19 UTC+2 schreef John Sivertsen:

[Duncan Mills]

I'm coming to the debate late, however, I'm a little puzzled that folks are talking about this feature in relation only to HA.  The basic mechanism of pooling is what allows you to multiplex your database connections under load, even on a single server.  I would routinely expect a 20-1 ratio of HTTP sessions to database connections and you need passivation / activation built in for that to work.  That is assuming of course that you like the luxury of working against an apparently stateful session.

In my experience, the passivation process is not a source of problems, it is, however, often a forcing function for detecting sloppy programming by developers who don't understand the basics of the environment in which they are working  (not pointing fingers here - I only work with teams within Oracle)

Now you can debate that for a non-HA setup, the database as a passivation store is overkill and I'll accept that, but the point remains that passivation/activation has a place even in a single node system. I guess that it all depends on what your most precious resource is - that's tuning for you.

Duncan

--

[Rahul Ganesh]

All,

I have worked on a project where we used dynamic shell tabs. The application seemed to work fine with AM pooling. However gradually we realized even with a single user in the system if a tab was left untouched for more than 30 -40 mins (note this is a tab left untouched while user is still active on other tabs in the system) Upon activation of the tab system reloads data and brings in all or no data in data grids bound to VOs. Upon digging further we found that the VOs lost  bind variables added via ADF API during activation cycle- perhaps it passivated without the values.. This is something that Duncan points to as being sloppy code not using view criteria. There after we changed the code  to use view criteria tested the application; to our surprise this still did not resolve 100% of the issues. There were instances where the bind variables were still lost and upon drilling down it looked like the session (HTTP, DB and the Tab) somehow never remained in sync during the cycle. All we could do is tuning the application to minimize pool recycles and reduce the activation cycles.

We never found the actual cause of this but one of the bugs with Oracle on Dynamic Shell tabs revealed that it was not meant to be stateful and once we close the tabs the sequencing somehow lead to data mix up and the shell tab code needed a fix. While this became overly complex to debug, we resorted to limiting the passivation-activation cycles as much possible.  We were a bit baffled and perplexed on ADF's extremely complex AM pooling and almost difficult to resolve problems leading every one including a CIO ask why we cannot get rid of the AM pooling. I can understand the frustration everyone has.

While Mr. Mills rightly pointed it may be due to coding/programming issues of developers but having worked with some of best developers around (some of who actually came from Oracle), it was clear that the framework is not really doing justice to developers as it cannot leave this at hands of developers unless its easy to debug and documented. 

It would have been easy to handle pooling via JDBC pooling and DB can scale and handle large number of connections.  Also with proper GC tuning its possible to keep the system out of stress. Keeping 20 : 1 ratio is possible without AM pooling and also system could have better stability without AM pooling. I hope Oracle does take a note of this and makes ADF (which is a great framework in all other respect) more performant and takes off or at least stabilizes this overhead.

Note- I have worked up till on 11.1.1.7 as we parallel develop using WebCenter. I am not sure if the AM pooling feature improved in newer versions though I hope it would have.

[Florin Marcus]

Hi Duncan,

Thank you for  taking the time to share your views on this matter. It looks like  there is a gap between Oracle's views on ADF performance and the views of people coming from real-world projects, a gap that we need to close it eventually .  I may sound a bit ridiculous to some, saying there are parts where we disagree with Oracle on how ADF should be used. At the end of the day ADF is their framework, so they should know better.

But this is not about how to use ADF. This is about how ADF uses underlying technologies. If you take a look at how memory looks like for an ADF application (top image only please http://postimg.org/image/3r7e2m6tb ), you will notice that 99% of the memory is occupied by non-ADF classes.  So, if some ADF tunings recommended by Oracle's documentation make those technologies perform worse, then yes, we can say documentation is wrong. And when I say 'perform worse', I refer to memory consumption, request throughput, or number of database operations .

I will start by saying that what we are questioning here is the need of recycling application module instances, not the fact that each client should have its own application module instance. Statefulness of application modules is not debatable here. 

I will be taking on each of your arguments at the time:

[Duncan's quote]

I'm a little puzzled that folks are talking about this feature in relation only to HA. The basic mechanism of pooling is what allows you to multiplex your database connections under load, even on a single server.  I would routinely expect a 20-1 ratio of HTTP sessions to database connections and you need passivation / activation built in for that to work.That is assuming of course that you like the luxury of working against an apparently stateful session

[/Duncan's quote]

I would agree with 20-1 ratio of HTTP sessions to database connections, this is close to what we see ourselves in ADF production systems.

But we achieve this by recycling database connections after each request (jbo.doconnectionpooling=true) and not by recycling application module instances. And we do not give up the luxury of a stateful session.

We do think there are enough arguments to say that  "Application Module Recycling" concept is unnecessary, actually a bad idea nowadays. It may have made lot of sense at the time it was introduced (when Robbie Williams was still part of Take That), but since then JDBC technologies evolved substantially, leaving "Application Module Recycling" behind. 

"Application Module Recycling" (activation/passivation) only acts as  memory cleanup mechanism and of course is increasing scalability.

But all it does is to solve a problem created by default Application Module settings:  holding the database connection between requests.

And holding connections between requests is the source of all evil. 

Here is why:

The argument Dev Guide uses for holding  db connections between requests is to keep PreparedStatements and ResultSets opened so queries can be reused when user is doing repetitive work.

My assumption is that  Oracle engineers came up with this idea while Oracle JDBC driver was perhaps at version 8i or 9i and these drivers were slow, but they had the advantage of minimum memory consumption. Starting from version 10g, JDBC architecture changed dramatically when Oracle engineers decided to trade memory for performance. Therefore JDBC became 30% faster, but memory consumption went rocket high (more info here).

This decision dynamited the concept of Application Module with sticky connections. From that point on, the argument of keeping PreparedStatements open became a problem and the only way to accommodate more users was to recycle application modules more often.

Activations turned into a memory cleaner, because it so happens that when an application module is switched from a user to another,  JDBC Prepared Statements are closed, which releases char[] and byte[] arrays from the memory. Worth mentioning that JDBC driver is the biggest memory consumer in an ADF application, you can check the earlier memory screenshot and you will notice that byte[] and char[] arrays consume aprox half of the memory alone. Those arrays are used internally as buffers by PreparedStatements (more info here).

Fusion Dev Guide argues that by closing JDBC connections after each request, there might be a performance overhead because releasing a connection back to the pool implies that Prepared Statements are being closed. This proves how outdated this writing really is: since then Prepared Statement Caching were introduced.

Therefore, if you don't shoot yourself in the foot by holding connections between requests, you don't need memory cleanups, since memory is being released naturally after each request.

Is true that AM recycling releases VO/EO cache as well. But memory consumed by EO/VO cache compared with the one consumed by PreparedStatements is irrelevant.

[Duncan's quote]

In my experience, the passivation process is not a source of problems, it is, however, often a forcing function for detecting sloppy programming by developers who don't understand the basics of the environment in which they are working  (not pointing fingers here - I only work with teams within Oracle)

[/Duncan's quote]

Lets assume this scenario:

Lets say we have an application with 100 users, where each user is constantly browsing the application and each user fires 1000 queries within an hour. 

Lets further assume the users have a normal 8 hours work program daily.

This would mean that database server will handle 100 x 1000 x 8 = 800 000 queries in a day. That is without activations happening.

With AM pooling and sticky connections you will have to use AM recycling, because you end up without memory. Lets say you will be limiting the number of AMs to 50, 

so then each two users will share one AM instance on average.

But the trouble with activations is that is releasing any client specific state on passivation (EO/VO cache). So than  the state will be re-queried again, next request after a passivation. 

So in the worst case scenario, the server  will have to endure 2 x 800 000  = 1 600 000 queries in a day, presuming that each user will click with a same frequency.

Of course this is an exaggeration for the sake of simplicity, the number might be somehow smaller, but anyhow larger than 800 0000 queries that the business requires.

In conclusion, by preferring AM recycling to DB recycling you will:

- consume more memory, because of the incorrect way of managing  JDBC statements.

- activations/passivation will release memory, but it  increases the number of queries against the database.

- you are saving memory by hitting in your most precious resource : database server. In practice most application use a Weblogic cluster, but a single Database server instance.

- it destroys the EO/VO cache every time passivation happens and that will increase the response time of the requests triggering activations.

- you will never really know how much memory you really need per user, because more time the user spends within the application, more screens the user opens, more memory consumes.

This doesn't happen when connections are being closed after each request, the memory increase when user is constantly opening new screens is much smaller.

In this big world of web technologies where even the mighty J2EE looks small , ADF is the only framework I know not releasing database connections after each request.

If anyone can come up with an example, I would love to hear it.

ADF framework support the best practice, is only Fusion Dev guide that is kept outdated.

We did log few bugs with Oracle Support, but we are still waiting for a reaction from Dev Team:

Bug 20196697 : TUNING APPLICATION MODULE POOLS AND CONNECTION POOLS
Bug 20180377 : DOCUMENTATION ENHANCEMENT REQUEST: FUSION DEV GUIDE 11G AND 12C: AM TUNING

And I totally agree with your  statement  "sloppy programming by developers who don't understand the basics of the environment in which they are working". 

Is exactly what we try to do here. Understand the basics.

Thanks,

Florin