jBASE 4.1.5.17 - does anyone face "out of memory" problems?

886 views
Skip to first unread message

Pawel (privately)

unread,
Feb 4, 2009, 9:26:58 AM2/4/09
to jBASE
Hi,

I know that it sounds unbeliveable for most of you, but I would like to
share with problems that we started to face on our test servers about
2-3 days ago.

We started to run out of memory errors on our test servers, eg.:
jsh t24fe ~ -->jdiag
** Warning [ PERFORM_ERROR ] **
Unix error number 0 while attempting PERFORM , Line 111 , Source jsh.b
Trap from an error message, error message name = PERFORM_ERROR
Line 111 , Source jsh.b
jBASE debugger->q
Are you sure ?y
.

We have several instances of some banking product on each test server.
We run batch processing (COB) on these test servers - usually 2-3 envs
in the same time on one, single AIX "test" server.
Our specificity is that we run these COBs using multiple agents - say 20
processes, giving 60 agents in total (60 jBASE processes running in the
same time on one test machine).

Today I was first time informed about problem and we have checked memory
allocation. It showed that we simply run out of it (physical and swap
runs out).

We have also found out that some of jBASE (COB) processes are consuming
large amounts of memory, for example:
16 eoyrx08 61482 633 (521) 1025 26K 9.38M 395K 6541 1212M 35m
2 SLEEP tSA 8 (BATCH.JOB.CONTROL,593)
26 t24ferx 503830 786 (776) 228 266K 12.2M 1.33M 1814 1321M 34m
2 SLEEP tSA 7 (BATCH.JOB.CONTROL,323)
28 t24ferx 430118 808 (799) 200 240K 17.6M 1.28M 1813 1200M 45m
2 tSA 6 (BATCH.JOB.CONTROL,322)

I know that somebody can start to suggest me that our local C/C++ code
is causing memory leaks. Please belive me that we do not run any C/C++
code during batch processing. We have only 1 (MQ) library written in C
and interfaced (DEFC) to jBASE. It was done by external vendor and is
LIVE since 5 years. It was thoroughly tested few years ago against
memory leaks. You have to belive me, but this library does not run
during COB. It is used only by some online processes.

Therefore I claim that something must be wrong with jBASE. My guess is
that jBASE does not free "transaction buffer" (does not downsize it once
transaction is finished).

There are some (single threaded) jobs during our COB that create huge
transactions (eg. 900K changes in one transaction). It seems to me that
"changes" buffer is never downsized or this memory simply "leaks"
somehow.

Does anyone face(d) similar problems?

jBASE version 4.1.5.17 (Major 4.1 , Minor 5.17 , Patch 5690 (Change
52756)), AIX 5.3.0.0-06.

Kind regards
Pawel

----------------------------------------------------
Nużą Cię utarte scenariusze?
Wymyśl własną grę flashową i wygraj główną nagrodę 5.500 Euro:
http://klik.wp.pl/?adr=http%3A%2F%2Fwhosegame.pl%2Fcontestcard.php%3Fcontest%3D55&sid=631


Pawel (privately)

unread,
Feb 4, 2009, 9:32:10 AM2/4/09
to jBASE
More details:

By the way:
our memory on one of the servers is 24 GB (physical) + 10 GB (swap).

It should be enough for 60 jBASE processes, should not it?

We are exporting LDR_CNTRL=MAXDATA=0x80000000@USERREGS in our .profiles.

ulimit on test environment processes is:
jsh t24fe ~ -->ulimit -a
time(seconds) unlimited
file(blocks) unlimited
data(kbytes) 2097152
stack(kbytes) 4194304
memory(kbytes) unlimited
coredump(blocks) 2097151
nofiles(descriptors) 4096

Kind regards
Pawel
Twoja rodzina na bliscy.pl
Zobacz:
http://klik.wp.pl/?adr=http%3A%2F%2Fcorto.www.wp.pl%2Fas%2Fbliscy.html&sid=628


Jim Idle

unread,
Feb 4, 2009, 4:10:50 PM2/4/09
to jB...@googlegroups.com
Pawel (privately) wrote:
Hi,

I know that it sounds unbeliveable for most of you, but I would like to 
share with problems that we started to face on our test servers about 
2-3 days ago.

We started to run out of memory errors on our test servers, eg.:
jsh t24fe ~ -->jdiag
 ** Warning [ PERFORM_ERROR ] **
Unix error number 0 while attempting PERFORM , Line   111 , Source jsh.b
Trap from an error message, error message name = PERFORM_ERROR
Line 111 , Source jsh.b
jBASE debugger->q
Are you sure ?y
  
Pawel,

Have you set the environment variable for AIX that causes memory allocation to be from the top of the HEAP down? It changes the allocation algorithm for AIX and is a huge improvement over the standard one. In fact, I suspect that as the engineers produced this new algorithm for allocation patterns like your batch job, that the standard one no longer tries to deal with it, though the literature does not indicate this:

Improvements to malloc subsystem
The number of malloc-related environment variables supported by AIX 5L Version 5.3 has been reduced to three and the attributes they can assume have been redefined. These environment variables are: MALLOCTYPE, MALLOCOPTIONS, and MALLOCDEBUG. MALLOCOPTIONS is a new environment variable that has been added to take care of all current and future options to the MALLOCTYPE allocators. It supplants the MALLOCBUCKETS, MALLOCMULTIHEAP, and MALLOCDISCLAIM that have been deprecated.

The three environment variables have the following definitions and some of the attributes they can assume:

MALLOCTYPE Used to specify allocator type. Attributes include: 3.1, default allocator, watson, and user.
MALLOCOPTIONS Used to specify allocator options. Attributes include: buckets, disclaim, threadcache, and multiheap.
MALLOCDEBUG Used to specify debugging options. Attributes include: catch_overflow, report_allocations, Malloc Log, Malloc Trace, and validate_ptrs.

The following enhancements have also been incorporated: ... etc.

I think that you want:

export MALLOCTYPE=watson

in your .profile

However there are other options to tune that, including a malloc cache and so on, so read up on it. It is safe to go right ahead and try that simple export, but don't give up on it if it seems not to improve things right away.




.

We have several instances of some banking product on each test server. 
We run batch processing (COB) on these test servers - usually 2-3 envs 
in the same time on one, single AIX "test" server.
Our specificity is that we run these COBs using multiple agents - say 20 
processes, giving 60 agents in total (60 jBASE processes running in the 
same time on one test machine).

Today I was first time informed about problem and we have checked memory 
allocation. It showed that we simply run out of it (physical and swap 
runs out).
  
Also check your local settings here. A lot of malloc algorithms pre-allocate swap space in case they NEED to swap. Hence you can run out of swap even though you have not actually used any. For this, you just allocate a huge amount of swap (disk is cheap) or find the tuning options to tell the system to run in optimistic mode for swap, which means it doesn't pre-allocate the swap and you are on your own if you run out of it.

We have also found out that some of jBASE (COB) processes are consuming 
large amounts of memory, for example:
  16    eoyrx08   61482  633 (521) 1025  26K 9.38M  395K 6541 1212M   35m  
2 SLEEP tSA 8 (BATCH.JOB.CONTROL,593)
  26    t24ferx  503830  786 (776)  228 266K 12.2M 1.33M 1814 1321M   34m  
2 SLEEP tSA 7 (BATCH.JOB.CONTROL,323)
  28    t24ferx  430118  808 (799)  200 240K 17.6M 1.28M 1813 1200M   45m  
2 tSA 6 (BATCH.JOB.CONTROL,322)
  
You have to be a bit careful here as you need to distinguish between memory that is shared by all processes (for mmap() of files and libraries) and real data memory consumed by your application. top seems to be reasonably good at distinguishing though.

However, what usually happens here is poor application programming. You can of course only inspect your local code but you should look for subroutines that do things like logging but never free up the variables they are using to accumulate log records, things like that.


I know that somebody can start to suggest me that our local C/C++ code 
is causing memory leaks. Please belive me that we do not run any C/C++ 
code during batch processing. We have only 1 (MQ) library written in C 
and interfaced (DEFC) to jBASE. It was done by external vendor and is 
LIVE since 5 years. It was thoroughly tested few years ago against memory leaks. You have to belive me, but this library does not run 
during COB. It is used only by some online processes.
  
As the only person to have written an MQ interface was me, I have supreme confidence in your MQ links ;-)
Therefore I claim that something must be wrong with jBASE. 
I see how you got here, but I would be extremely surprised :-) In 19 years, this has rarely been the case, though Greg and I had to fly all over the world to show that it wasn't more times than I can remember ;-). That doesn't mean it isn't a fault there, but it almost always isn't.

My guess is 
that jBASE does not free "transaction buffer" (does not downsize it once 
transaction is finished).
  
Nope, the system wouldn't run for more than 5 minutes if that were the case.
There are some (single threaded) jobs during our COB that create huge 
transactions (eg. 900K changes in one transaction). It seems to me that 
"changes" buffer is never downsized or this memory simply "leaks" 
somehow.
  
Well, the best answer to that is of course to fix the things, as it is a stupid design [well, it isn't a design ;-)], but they are probably not your programs. However, while the internal buffer might grow, even if it were never shrunk, it would be reused, not lost and re-allocated and would soon reach the maximum that your application needed and stay there. Hence, if this were the issue it would be because your application was just growing the transaction forever.

However, I suspect that the buffer allocation for such large transactions might be the root of your problem and that if you change the malloc algorithm to Watson, it will have a much easier time of it.

Does anyone face(d) similar problems?

jBASE version 4.1.5.17 (Major 4.1 , Minor 5.17 , Patch 5690 (Change 
52756)), AIX 5.3.0.0-06.
  
Yes - it is usually the allocation algorithm you are using and rogue applications :-)

Jim

Jim Idle

unread,
Feb 4, 2009, 4:16:39 PM2/4/09
to jB...@googlegroups.com
Pawel (privately) wrote:
> Hi,
>
> I know that it sounds unbeliveable for most of you, but I would like to
> share with problems that we started to face on our test servers about
> 2-3 days ago.
>
> We started to run out of memory errors on our test servers, eg.:
> jsh t24fe ~ -->jdiag
> ** Warning [ PERFORM_ERROR ] **
> Unix error number 0 while attempting PERFORM , Line 111 , Source jsh.b
> Trap from an error message, error message name = PERFORM_ERROR
> Line 111 , Source jsh.b
> jBASE debugger->q
> Are you sure ?y
> .
I meant to add that if valgrind can be compiled on your system, then you
could run it against a jBASE program that exited normally. You will see
that some allocations are left to the system to reclaim, but nothing
from the application. However valgrind does have a useful heap
examination tool that can identify where memory is being held out on the
stack. I have only ever used it in Linux however.

There are other tools for AIX, but I think that most of them would
require source code. What does Temenos help desk say by the way? They
are only going to have to ask the jBASE guys, but we have been through
this sort of thing lots of times.

Jim

Jim Idle

unread,
Feb 4, 2009, 4:26:32 PM2/4/09
to jB...@googlegroups.com
Pawel (privately) wrote:
> More details:
>
> By the way:
> our memory on one of the servers is 24 GB (physical) + 10 GB (swap).
>
> It should be enough for 60 jBASE processes, should not it?
>
It isn't the number of processes but what they are all doing mate ;-).
In fact, while you would think that 10GB of swap was enough, you should
at least have an equal amount as the memory (24GB), to cater for
marginal conditions. While the old rule of thumb of 3X main memory gets
a bit out of whack at 24GB, I would allocate that anyway myself. How
much does 64GB of disk cost you, even at IBM prices. Also, when you are
getting these issues, you should add on lots of it anyway so you can
diagnose without the system crashing (though it may go very slowly).

> We are exporting LDR_CNTRL=MAXDATA=0x80000000@USERREGS in our .profiles.
>
>
Hmmmm. Did someone tell you to try that?

You mgiht also try 64K datasegment page sizes and different limits as
follows:

LDR_CNTR=MAXDATA=0XB0000000@DSA@TEXTPSIZE=4K@STACKPSIZE=64K@DATAPSIZE=64K

(You may want MAXDATA to be lower, and you might wnat to unset this to
use watson, but please consult the IBM manual about that as I don't
remember off-hand) But it won't help if you are not using the watson
allocator I suspect. The real problem here is the application expecting
to create a 1GB in memory transaction, not trying to tune the system so
that it can support that :-) Are they TEMENOS supplied routines that do
that, because you should be complaining about them if it is. Your own
batch programs you can fix.

> ulimit on test environment processes is:
> jsh t24fe ~ -->ulimit -a
> time(seconds) unlimited
> file(blocks) unlimited
> data(kbytes) 2097152
>

You might need data to be unlimited.


> stack(kbytes) 4194304
> memory(kbytes) unlimited
> coredump(blocks) 2097151
> nofiles(descriptors) 4096
>
> Kind regards
> Pawel
>

JIm

Mike Preece

unread,
Feb 4, 2009, 11:31:40 PM2/4/09
to jBASE
What is the CoB job that fails?

Is it a report job that has gone wrong somehow, generating
ridiculously large transactions?

Is it localdev or core?

If it is a report, should it run in CoB or can it be set to run on an
"A"dhoc basis instead?
> Jim- Hide quoted text -
>
> - Show quoted text -

Pawel (privately)

unread,
Feb 5, 2009, 10:34:55 AM2/5/09
to jBASE
Hi guys,

We are trying to find out details (locate processes=jobs which utilise
so much memory).

We did not apply "export MALLOCTYPE=buckets" to .profiles as HelpDesk
suggested. This option is exported only on our one test area, which is
rarely processed.

I read http://www.redbooks.ibm.com/redbooks/pdfs/sg247463.pdf and:
"The current default allocator is the Yorktown allocator. With respect
to speed, the Yorktown allocator does not efficiently handle repeated
small to medium size requests. This deficiency was previously addressed
by adding the Malloc Buckets algorithm. Malloc Buckets, however,
provides no way to consolidate freed memory into the heap. The new
bucket allocator will allow freed memory to be reclaimed. Through the
use of the new bucket allocator, the Watson Allocator handles small
requests quickly and with comparatively little wasted memory. The Watson
Allocator also performs quicker than the default allocator, and with
less internal fragmentation than the old bucket allocator.

The Watson Allocator can be configured in three distinct ways to try and
identify which sections reveal the largest gains. It can be enabled with
caching mechanisms (a per thread cache and an adaptive heap cache) and
with the new bucket allocator. It can be enabled without the caching
mechanisms and with the new bucket allocator. It can also be enabled
without the caching mechanisms and without the new bucket allocator. The
new bucket allocator and thread cache have been adapted to work with the
Yorktown allocator."

It seems to me that we should try what Jim suggested: "export
MALLOCTYPE=watson" and perhaps some MALLOCOPTIONS settings.

Let's leave it for the second, because something interesting shown on
our LIVE area today in the very morning. Session that runs TSM produced
following output:
jsh techuser ~ -->START.TSM
START.TSM
Phantom process started on process id 2449420
[2449420] Done : tSA 1
jsh techuser ~ -->Process ID 1232928 , port 794 , hangup
Program source name CLEAR.TOKENS , line 26
Recursive debugger calls - program aborting
Process ID 5054700 , port 794 , hangup
Program source name CLEAR.TOKENS , line 27
Recursive debugger calls - program aborting
jBASE: Segmentation violation. Aborting
cp: ../bnk.data/int.data/DM.TEMP/DC.CARD.ISSUE.HIS.DM: No such file or
directory
jBASE: Segmentation violation. Aborting
jBASE: Attempting to free NULL pointer at
jediTransaction.c,1636(EB.TRANS.JBASE,
26)
jsh techuser ~ -->Process ID 7737374 , port 805 , hangup
Program source name F.READ , line 7

You can ignore hangups, but I am worried about these jBASE errors
(Segmentation violation / Attempting to free NULL pointer at
jediTransaction.c,1636). This does not sound good to me. We do not know
which processes thrown these messages, but likely they were COB agents.

I do not know yet wheter physical / swap memory run out yesterday on
PROD, but it quite unlikely (total memory of LIVE system is 2-3 times
bigger than on test machines).

I would like to mention one fact from the past.

During "start of year" (2nd January) processing we faced 1 "little"
problem:
a) one of the single threaded jobs did a large transaction (over 900k of
changes) - we have already requested to improve this core EOY job
b) then later one of the batch sessions (agent) failed with
SUBROUTINE_CALL_FAIL error. There was nothing wrong with our libraries -
called object was there and routine that failed was successfully called
by other COB agents. Only one COB agent noted SUBROUTINE_CALL_FAIL
error, which seemed to be very strange. We have raised that and CSHD
conclusion was: "agent run out of shared memory" (ulimit is unlimited on
LIVE) so use "slibclean" periodically to reclaim memory.
I think now that agent which failed on 2nd January performed in previous
steps large transaction, means allocated large "transaction buffer" and
finally got SUBROUTINE_CALL_FAIL on one of the following jobs (not
immediately).
That is why I suggested that "transaction buffer" may not get downsized
or leaks memory. I also guess that it may not be a leak, but default
MALLOC allocator fault. I am not sure if Watson will help, but reading
Jim's emails we will give it try.

We need to do more analysis and testing, so I will come back with
conclusion. We have red alert now and are trying to understand what is
going on with memory utilisation :)

Kind regards
Pawel

Pawel (privately)

unread,
Feb 5, 2009, 3:22:42 PM2/5/09
to jBASE
Hi,

Story continues...

We have found out that during night/batch processing memory utilisation
grows significantly after SSELECT is performed by the session. Moreover
these memory blocks seem not to be reclaimed until process dies.

We did a simple test after my colleague suggestion. Here are details:
a) session A: login to jshell, check your port via WHO command
b) session B: run SSELECT on large table with many IDs, preferably right
justified or with some non-standard characters like asterrisk, comma,
dot, question mark, etc.
c) session A: quickly, meanwhile, run MW42
-p<port_number_from_session_A_WHO> and observe memory allocation

Process will hit some number and stay at this level, say 200 MB which is
still quite too much for me.

Run a SSELECT on smaller table with lower number of IDs and observe
memory allocation. Memory usage will stay at the same level, say 200 MB.

Run a SSELECT on largest table you have with many IDs. You will see that
additional memory is allocated at some point - say you will finally
reach 400 MB.

This sorting buffer seems never to shrink.

I know that previously I suggested that perhaps "transaction buffer" is
not downsized, but we can easily tell that session that runs SSELECTs
can allocate more and more memory.

I am confused and still does not know wheter memory leaks or not.

We have also quickly changed to "MALLOCTYPE=watson" on test area and
here is interesting output:
jsh techuser ~ -->SSELECT <largetable>
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at
jQLSortProcessor.cpp,721(select.b,145)

1496042 Records selected

>CLEARSELECT

Please remember that nothing was displayed on "default" allocator. Are
we right that something is wrong and memory leaks? Why errors are thrown
on non-standard memory allocator?

Kind regards
Pawel

Jim Idle

unread,
Feb 5, 2009, 5:23:30 PM2/5/09
to jB...@googlegroups.com
Pawel (privately) wrote:
> Hi guys,
>
> We are trying to find out details (locate processes=jobs which utilise
> so much memory).
>
> We did not apply "export MALLOCTYPE=buckets" to .profiles as HelpDesk
> suggested. This option is exported only on our one test area, which is
> rarely processed.
>
> I read http://www.redbooks.ibm.com/redbooks/pdfs/sg247463.pdf and:
...

> It seems to me that we should try what Jim suggested: "export
> MALLOCTYPE=watson" and perhaps some MALLOCOPTIONS settings.
>
Yep, start without options though. I see your next post, but I will
address that in a reply to that post.

> Let's leave it for the second, because something interesting shown on
> our LIVE area today in the very morning. Session that runs TSM produced
> following output:
> jsh techuser ~ -->START.TSM
> START.TSM
> Phantom process started on process id 2449420
> [2449420] Done : tSA 1
> jsh techuser ~ -->Process ID 1232928 , port 794 , hangup
> Program source name CLEAR.TOKENS , line 26
> Recursive debugger calls - program aborting
>
Yeah - there is some serious problem going on, but to be honest this is
probably just out of memory again. There is very little you can do as a
programmer once you are out of memory. Even trying to print a message
will run out of memory unless you know you can free some. Some people
guard against this by keeping a small allocation of memory to be freed
in case of an abort, in the hope they can use it to get enough memory to
recover or show a message.

The key to this is the recursive debugger calls. This means that being
out of memory the program tried to access a memory pointer that wasn't
valid (it should not really do this, but as you can't really do anything
at this point, it is a moot point) - there is a trap routine that sees
this invalid access and it aborts the program and enters the debugger.
However, being out of memory, the debugger tries to use some memory and
accesses an invalid pointer, which means the trap triggers and it tries
to enter the debugger, where it detects the recursion and just aborts as
there is no other option.


> Process ID 5054700 , port 794 , hangup
> Program source name CLEAR.TOKENS , line 27
> Recursive debugger calls - program aborting
> jBASE: Segmentation violation. Aborting
>

This is the out of memory stuff.


> cp: ../bnk.data/int.data/DM.TEMP/DC.CARD.ISSUE.HIS.DM: No such file or
> directory
>

This is a bug in your application, which is not detecting that the
program did not finish correctly and is trying to copy the results of it
anyway.


> jBASE: Segmentation violation. Aborting
> jBASE: Attempting to free NULL pointer at
> jediTransaction.c,1636(EB.TRANS.JBASE,
> 26)
> jsh techuser ~ -->Process ID 7737374 , port 805 , hangup
> Program source name F.READ , line 7
>

This is basically all the same thing. I will defer until answering your
next message, which is where the problem is I think.


> You can ignore hangups, but I am worried about these jBASE errors
> (Segmentation violation / Attempting to free NULL pointer at
> jediTransaction.c,1636). This does not sound good to me. We do not know
> which processes thrown these messages, but likely they were COB agents.
>

IT is just taht you were out of memory and things are trying to clean
up. These are the symptoms not the cause.


> I do not know yet wheter physical / swap memory run out yesterday on
> PROD, but it quite unlikely (total memory of LIVE system is 2-3 times
> bigger than on test machines).
>
> I would like to mention one fact from the past.
>
> During "start of year" (2nd January) processing we faced 1 "little"
> problem:
> a) one of the single threaded jobs did a large transaction (over 900k of
> changes) - we have already requested to improve this core EOY job
>

Did that happen?


> b) then later one of the batch sessions (agent) failed with
> SUBROUTINE_CALL_FAIL error. There was nothing wrong with our libraries -
> called object was there and routine that failed was successfully called
> by other COB agents. Only one COB agent noted SUBROUTINE_CALL_FAIL
> error, which seemed to be very strange. We have raised that and CSHD
> conclusion was: "agent run out of shared memory"

Shared memory is not used for this. That was either their
mis-description of the problem or you are recalling what happens on
UniVerse ;-). Basically this can only really happen when there is no
real memory, whic to confuse things, is called virtual :-), for the
process to map in the subroutine, or allocate the descriptor for it and
so on.

All your problems are likely caused by the answer to the next post you made.

> (ulimit is unlimited on
> LIVE) so use "slibclean" periodically to reclaim memory.
>

This is probably what they mean. There is a fundamental design issue
with AIX and when it feels it can get rid of shared objects. Whatever
IBM try to claim about this, they avoid the question "Hmm, then why does
no other UNIX suffer from this issue?"

> I think now that agent which failed on 2nd January performed in previous
> steps large transaction, means allocated large "transaction buffer" and
> finally got SUBROUTINE_CALL_FAIL on one of the following jobs (not
> immediately).
> That is why I suggested that "transaction buffer" may not get downsized
> or leaks memory. I also guess that it may not be a leak, but default
> MALLOC allocator fault.

Yep. But of course something is causing you to need huge amounts of
memory. I suspect that it is a bug in JQL, which we can find a work
around for (am I on the clock yet?) in the next email.

> I am not sure if Watson will help, but reading
> Jim's emails we will give it try.
>

It will definitely help generally, but it is more likely to expose the
real problem, which reading your next email, it seems it has :-)

Jim

Jim Idle

unread,
Feb 5, 2009, 6:13:56 PM2/5/09
to jB...@googlegroups.com
Pawel (privately) wrote:
Hi,

Story continues...

We have found out that during night/batch processing memory utilisation 
grows significantly after SSELECT is performed by the session. Moreover 
these memory blocks seem not to be reclaimed until process dies.
  
There are only two conclusions:

1) You get back the SELECT list of 2 million items and use it in some way but you never get rid of it in the application. If you repeat this, the select lists (or rather, what you are doing with them) will eventually eat all the memory;
2) It is a bug in jQL, which is not releasing memory blocks and you are EXECUTING the SELECT in the same process space as the calling program, which means that the calling program is accumulating the lost memory;

While 1) is definitely a possibility, my guess at the moment is at 2).
We did a simple test after my colleague suggestion. Here are details:
a) session A: login to jshell, check your port via WHO command
b) session B: run SSELECT on large table with many IDs, preferably right 
justified or with some non-standard characters like asterrisk, comma, 
dot, question mark, etc.
c) session A: quickly, meanwhile, run MW42 
-p<port_number_from_session_A_WHO> and observe memory allocation

Process will hit some number and stay at this level, say 200 MB which is 
still quite too much for me.
  
That's how much it takes if you are doing a SORT - you must accumulate all the keys and essentially make a second pass. That's one reason that I continuously tell people not to use SSELECT unless there really is some logical reason why the records must be processed in this sort Given the sort algorithm used though, this should grow somewhat linearly.

Run a SSELECT on smaller table with lower number of IDs and observe 
memory allocation. Memory usage will stay at the same level, say 200 MB.

Run a SSELECT on largest table you have with many IDs. You will see that 
additional memory is allocated at some point - say you will finally 
reach 400 MB.

This sorting buffer seems never to shrink.
  
Now, you could be seeing one of two things here, so don't jump to conclusions yet.

1) As above, the items are sorted, but now you have created a SELECT list which contains all those items, so of course the memory does not shrink. Use CLEARSELECT or runa command to utilize this list and the memory shoudl reduce (note, because of the way allocators work, it might not come back to where you think it would, but it should come back quite a bit;
2) The SSELECT, or the select list management code in jBASE is not working correctly and is simply not releasing space. If you try using the list and virtually nothing changes in memory usage, then this may be the issue.
I know that previously I suggested that perhaps "transaction buffer" is 
not downsized, but we can easily tell that session that runs SSELECTs 
can allocate more and more memory.
  
Well, we can get to this conclusion, but we have not yet done so as your methodology isn't quite correct.

I am confused and still does not know wheter memory leaks or not.
  
I understand, it is a difficult thing to pin down is memory (pun intended).
We have also quickly changed to "MALLOCTYPE=watson" on test area and 
here is interesting output:
jsh techuser ~ -->SSELECT <largetable>
jBASE: Attempting to free NULL pointer at 
jQLSortProcessor.cpp,721(select.b,145)
jBASE: Attempting to free NULL pointer at 
jQLSortProcessor.cpp,721(select.b,145)

 1496042 Records selected

  
CLEARSELECT
    
Please remember that nothing was displayed on "default" allocator. Are 
we right that something is wrong and memory leaks? Why errors are thrown 
on non-standard memory allocator?
  
And herein lies the rub. I think that you may well have uncovered a problem in jQL. The reason that using a different allocator shows this issue is that it is probably behaving better. NOw, don;t worry so much about that as jBASE is intercepting the bug and preventing the free(NULL) from causing any problems. However, this is very suspicious and would tend to give you the feeling that it is trying to free the WRONG pointer and therefore not freeing the correct one! The two things that MAY not be being freed might be huge memory allocations. Of course, it could just be that the program is trying to release something that it never allocated in the first place. There is no way to know without source code. Perhaps someone like Pat, who reads this list, could take a look?

Here is how to proceed:

1) If at all possible and this is the case in ALL situations for everyone, NEVER use SSELECT to generate a processing list UNLESS the records you need to process MUST be processed in sorted order (there are of course some times when this is necessary), for the following reasons:

a) It takes a lot more memory to do the sort;
b) And more importantly, assuming that you are generating keys from the same file that you are going to process, you have just changed the disk access pattern from sequential (as in group has order) to random (as in wherever the primary keys happen to has in the file). This will have a huge impact on the READ times for the records and the issue will increase quadratically as the key set gets larger. I cannot emphasize enough what the performance impact of this is.

2) Now, in the past, the {S}SELECT would be performed in a separate process to the program that does the EXECUTE. This would mask any issues of not releasing memory because at process end, UNIX will reclaim the memory heap anyway and you will just be left with the memory usage for the select list, which will be given up when the batch process ends.

However, in 4.1, for performance reasons, these programs are all executed within the same process space, so any memory that is not freed will never be reclaimed and will gradually make a process run out of memory.

To test whether this is happening, we need to force the SSELECT to EXECUTE in a new process. If this is your program, then you can use EXECUTE CHAR(255):'k':'SSELECT .... but if it is not your process, then we will have to get tricky. However, you can redo your SSELECT testing described above by running a program that does an EXECUTE as shown here rather than the SSELECT command at jSHELL. If there is a SSELECT memory leak, then the memory used by the calling program will not be expanded by the SSELECT program, which will return the select list and the calling program will use that. Make the program repeat the SSELECT 10 times or as many as is needed to show that memory remains stable. Then, either take off the CHAR(255):'k' piece, or run SSELECT from the jSHELL lots of times and show that memory increases and never reduces.

Report the findings to TEMENOS.

Because you are in a very problematic situation, you should force the SSELECT (or SELECT if you can change it to use this) to execute in a separate process all the time. In jBASE 3.x there was an environment variable you could use to force a list of commands to execute in a new process all the time, but in 4.1, I do not know if this is the case. I will try to find out in the docs, but as they are now ONLY in stupid PDF and some .DOC formats, they cannot be searched on the web site. TEMENOS help desk should be able to tell you this, or maybe the jBASE guys on here already know that you can do this (sorry that my memory fails me here). The envvar for 3.1 was: http://www.jbase.com/knowledgebase/manuals/3.0/30manpages/man/env2_JBCNOINTERNAL.htm

I think that it is also possible to delete the .so version of the command and this will force EXECUTE to run the executable version. However, if the executable version only calls the .so version then this will not work. There was some vacillation on whether it should do one or the other. Find the SELECT binary and find the SELECT.so and see if they are of similar size (make sure you find real binaries and not any symbolic links). However, if you have a test machine, just rename the .so versions of SELECT and SSELECT and see if they SELECT and SSELECT commands still work. If they do, then rerun on your test machine with these .so file renamed. You should find that your problems go away if the issues is SSELECT not freeing memory.

Finally, if you have remote access available and you really are that in need of help, you could always, err, PAY <gasp> me to come in and at least diagnose the problem properly for you!

Finally, finally, on your test machine, make sure you have the latest AIX 5.3 patches loaded (see my past diatribes about the illogic of not applying patches to operating systems) and the latest version of jBASE 4.1 and see if this helps in any way at all!

I will post more here if I work out how to guarantee that your SELECT and SSELECT will execute in an external process.

Jim

Jim Idle

unread,
Feb 5, 2009, 7:13:34 PM2/5/09
to jB...@googlegroups.com
Jim Idle wrote:
Pawel (privately) wrote:
Hi,
    


I will post more here if I work out how to guarantee that your SELECT and SSELECT will execute in an external process.
Before I answer this, I meant to say that all evidence tells me that you should immediately switch to the Watson allocator - the issue may well be alleviated, or even fixed by that. Don't worry about the free(NULL), but do report it.

So, the problem with just removing the .so version of SELECT and SSELECT is that the incoming and outgoing SELECT lists that these programs generate can ONLY be passed around in memory. Which means that just renaming the .so will mean the SSELECT is executed out of process but the resulting select list will be discarded. So, first try the Watson allocator (it almost certainly will not be worse), and if that does not help, try this.

Until TEMENOS can fix the memory leak (assuming for the moment that there is one), we have to get very tricky, but this will immediately cure your problem if jQL memory leaks are the issue. Make sure you try this in a test installation of course as a mistake would screw up badly.

What we need to do is get SELECT and SSELECT to execute externally from the batch program. Now if the real problem is just that your SSELECT is just running out of memory because it is just too damned big, then this won't help at all, or it might help short term because there is a bit more memory available in the process when it is tand alone, but will run out anyway later when the file gets bigger. If it is just too big a query, then you will have to stop doing SSELECT or get to jBASE 5 and use 64 bits and so on (though that may not help). Basically, if the SSELECT accumulates 2GB of memory allocations, then it will just die with out of memory anyway. This is why the Watson allocator may just stop the problem.

So, to do this, you need the following:

  • Create a new SSELECT and SELECT command as BASIC programs and make sure that wherever you put the resulting programs, they are found in the PATH before the real SELECT and SSELECT.
  • Create secondary SSELECT-2 and SELECT-2 programs, that will really run SELECT and SSELECT commands - delete the .so versions of these programs after they are compiled (note we do not want to use EB.COMPILE and so on for these programs, but the jcompile command);
  • These programs should work in conjunction with a UNIX directory, where they can store temporary lists, choose a directory that is backed by reasonably well performing disks;
  • The replacement SELECT and SSELECT (note it can be one program that is symbolically linked to the other and uses SENTENCE() to work out which it is), picks up the command that was EXECUTED via the SENTENCE() command so it knows what to pass on;
  • The new SELECT and SSELECT should also look to see if there is an incoming select list from the batch program and if there is, it should use WRITELIST to save it in the temporary Unix directory;
  • Unfortunately, the replacement programs may have to look at stacked DATA too and deal with that
  • The replacement program now does EXECUTE "SSELECT-2 ":restofcommand  (or SELECT-2)
  • Because there are no .so versions of SSELECT-2 and SELECT-2, they will execute in a new process;
  • These programs read in any incoming select list using READLIST and make it active
  • They also stack any incoming DATA
  • Then they change their PATH (might also need LD_LIBRARY_PATH or LIBPATH) environment variable so that the real SSELECT and SELECT now come first in the list
  • They execute SSELECT with the original command line
  • They pick up the generated select list and WRITELIST it to the temporary Unix directory, then stop gracefully
  • The replacement SSELECT takes over again, uses READLIST to pick up the resulting list and make it active, then exits gracefully.


OK, so that sounds complicated and of course this is just a work around to try and prove things one way or another. However, it isn't very many more lines of jBC code than the description above and can be implemented without disturbing the real SELECT and SSELECT. It has the advantage that if you are not leaking memory but the query is too big for in process, then the external one will probably work and will always tart with clean memory, but if you ARE leaking memory, AND it is jQL that is doing it, then you will no longer leak it.

I realize that that program is probably a lot easier for me to write, but I am sure you can do it. Just test it a lot in your own login before changing the .profile for everyone on the test machine, then get everyone to try it until you are happy with putting it live.

Of course, whether this helps or not, you need to be getting TEMENOS to help you fix any problem; I am sure that they will.

Jim

mike ryder

unread,
Feb 6, 2009, 12:47:37 AM2/6/09
to jBASE
Hi Pawel,

I know from previous posts that this is T24 related.

You don't state which T24 release or on which table you are doing the
select or even if this is Temenos core code or your own local code.

The reason why this may be significant is as follows:

1. SSELECT on SECURITY related tables can cause a problem. I have seen
such on SECURITY.POSITION. It occurs because the dictionary says that
the ID is right justified in 35 spaces and all of the keys take up 35
characters and you run out of memory. Changing the dictionary to left
justified solves the problem.

2. If this is local dev, then doing an internal select can be much
faster. Where you expect to retrieve more than 50% of the keys and the
condition is based on the key, then an internal select with code will
be much faster and not hog the memory.

3. In local dev, many people try to do all of the processing in one
job when it would be more performant to select in one agent process
and process the keys in another. As an addendum to this, if you do
humongous processing in a job with transaction turned on all sorts of
memory issues occur.

4. The selection operation used by EB.READLIST has changed in later
releases (R6 or maybe R7) where internal select is used if the select
statement is called without parameters

5. I believe that SSELECT can be called as an internal selection
rather than an external selection which means that all of the keys are
not require to be stored before they can be processed/

Hope this is of interest
Mike

Pawel (privately)

unread,
Feb 6, 2009, 9:02:37 AM2/6/09
to jBASE
Hi Mike,
Hi Jim,

I more or less understood, which kind of workaround Jim is suggesting.
Problem is officially reported via CSHD and we are waiting. Nothing is
confirmed, it happened many times that I was mistaken or made wrong
assumptions/suggestions.
I just wonder now how it can happen that we did not add new
environments, neither release significant changes to the software (as
far as I know) and memory finally run out. Perhaps growing data volumes
can explain that? We regularly refresh test areas with LIVE system. So
SSELECTs are launched on bigger and bigger files.

We are running T24 R06004 with many patches installed. Bank did not
decide to install T24 Service Packs and it was in my opinion bad
decision to me. So it happens that we are discovering already discovered
and patched things, which is not funny at all :(

1. SSELECTs are performed by core and (unfortunately) local
developments. Select I have shown (on watson) was done on
FBNK.ACCOUNT.DEBIT.INT table. It is regular J4, not distributed file. We
tried SSELECTs on few tables and effect was the same.

2. I generally agree, but your statement is true only when you do not
use Oracle driver. I think that SELECT/READNEXT combinations should be
avoided. Perhaps in the future Temenos will optimize SELECT <file> /
SSELECT <file> and getting keys on jBASE will be so fast as on Oracle? I
am not an expert, but I belive that they have field to improvements and
it will be easier to do something to tune commands like SELECT than to
tune internal SELECT/READNEXT. Perhaps in jBASE 6 SELECT on distributed
file will launch multiple threads and partfiles will be scanned in
parallel? It is just a stupid example, because I do not expect such
improvements (who SELECTs distributed files?)
By the way: we do not run on Oracle, but would like to keep portability.

3. Which kinds of memory issues? Either memory is correctly retained or
not. We try not to do stupid things and use multiagent jobs as much as
we can. We try not to create large transactions and find old local jobs
that do.
I personally think that memory must be somehow given back even if 1 of
the jobs is stupid. Why job #2 is limited by previously executed, stupid
job? I expect that if job #1 allocates 300 MB, then job #2 will be
executed with (more or less) same initial memory utilization.

4. R06, I have seen that. Did they introduce that because memory leaked
or internal SELECT is faster? I am joking of course. We did not test
SELECT much, because we seen that memory utilization during COB
increases after SSELECTs are run. Perhaps SELECT leaks also something.

5. ? Hmmm... You need to first collect all keys and store them in memory
somehow, then to sort them with some algorithm, right? Sorting process
may require more memory than just size of the keys, but this memory
should be given back to process once SSELECT is completed. I think that
it is not (fully?) in jBASE 4.1.5.17.

Let's leave this subject for confirmation :) As I said in the begining I
was mistaken many times.

Kind regards
Pawel
----------------------------------------------------
Najlepsi skoczkowie z Polski
na zawodach w Wiśle i Szczyrku
musisz tam być razem z nami!
Kliknij: http://klik.wp.pl/?adr=http%3A%2F%2Fcorto.www.wp.pl%2Fas%2Fnarta.html&sid=633


Pawel (privately)

unread,
Feb 6, 2009, 12:14:06 PM2/6/09
to jBASE
Hi,

This is definetely not easy subject - that is why I am coming back.

I was complaining on memory leaks, but there may not be any memory leaks
:)

Can anyone explain me what "Free" and "Used" means in below output:
858 16 rx0044 3981556 Port 858 details at 06:08:08 06
FEB 2009
Thread type Normal
Usage Count 72 from 100K
Application Open Files 14654 , Actual O/S Open Files 181
Memory: Free 43,755,392 Used 2,095,398,288
READ's 192042 , WRITE's 14167
DELETE's 1315 , CLEARFILE's 0
EXECUTE's 3 , INPUT's 0
OPEN's 14676
---- jsh -Jb -c tSA 60
Program performing EXECUTE/PERFORM at jsh.b,59
---- tSA 60
Program running normally at F.WRITE,72

I am asking because if process will free say 1,8 GB situation will not
change from OS level. Memory once allocated (disregarding from "Free" /
"Used" jBASE indicator) seems not to be given to other processes. Why?

In this way it may happen that multiple processes had their own "peaks"
in processing and each one allocated (but is not longer using) 1 GB.
Then memory runs out.

So why "Free" is not free? I am completely lost :(

Kind regards
Pawel

Dnia 6-02-2009 o godz. 1:47 mike ryder napisał(a):

Jim Idle

unread,
Feb 6, 2009, 5:00:44 PM2/6/09
to jB...@googlegroups.com
Pawel (privately) wrote:
Hi Mike,
Hi Jim,

I more or less understood, which kind of workaround Jim is suggesting. 
Problem is officially reported via CSHD and we are waiting. Nothing is 
confirmed, it happened many times that I was mistaken or made wrong 
assumptions/suggestions.
  
Sure - as I said, it might be that the way your data has expanded has just caused the sort to run out of memory, rather than there being any leaks. Change to the watson allocator (have you tried this?), could make a huge difference because it knows how to coagulate lots of mallocs back together when they are freed.

I just wonder now how it can happen that we did not add new 
environments, neither release significant changes to the software (as 
far as I know) and memory finally run out. Perhaps growing data volumes 
can explain that? We regularly refresh test areas with LIVE system. So 
SSELECTs are launched on bigger and bigger files.
  
Yes - eventually, even the most efficient system will run out of space.

To be honest, a big issue here is the whole design thing. This stuff should be running via message queues, distributing the work piece by piece and running huge serial selects, but there isn't much to be done about that.

We are running T24 R06004 with many patches installed. Bank did not 
decide to install T24 Service Packs and it was in my opinion bad 
decision to me. So it happens that we are discovering already discovered 
and patched things, which is not funny at all :(
  
Yes - the illogic of not installing patches for known bugs is endemic in the industry. "How do we know it won't break something else?" is the cry, but the answer is that "This is rare, but what you do know 100% absolutely, is that you have bugs on your system which you are subjecting yourself to; are you crazy?"

1. SSELECTs are performed by core and (unfortunately) local 
developments. Select I have shown (on watson) was done on 
FBNK.ACCOUNT.DEBIT.INT table. It is regular J4, not distributed file. We 
tried SSELECTs on few tables and effect was the same.
  
Did you try changing that field to be left justified?

2. I generally agree, but your statement is true only when you do not 
use Oracle driver. I think that SELECT/READNEXT combinations should be 
avoided. Perhaps in the future Temenos will optimize SELECT <file> / 
SSELECT <file> and getting keys on jBASE will be so fast as on Oracle?
It would be anyway with the requisite indexes.
 I 
am not an expert, but I belive that they have field to improvements and 
it will be easier to do something to tune commands like SELECT than to 
tune internal SELECT/READNEXT. Perhaps in jBASE 6 SELECT on distributed 
file will launch multiple threads and partfiles will be scanned in 
parallel? It is just a stupid example, because I do not expect such 
improvements (who SELECTs distributed files?)
By the way: we do not run on Oracle, but would like to keep portability.
  
A query based select will never beat a good hand crafted data selector. You should always write these in jBC with SELECT and readnext and either use the built in indexing, or some application specific stuff of your own. All 'queries' should be disk bound, but otherwise instant. The select engine in 4.1 and above can get close to parity if you have created the right indexes, but if your data gets too big, then this can be moot. Hand code, always, no questions. The COB runs every day that the bank exists and if it takes you three weeks to write the same query in jBC (It won't) then the overall savings in time and money are too good to turn down.

3. Which kinds of memory issues? Either memory is correctly retained or 
not. We try not to do stupid things and use multiagent jobs as much as 
we can. We try not to create large transactions and find old local jobs 
that do.
I personally think that memory must be somehow given back even if 1 of 
the jobs is stupid. Why job #2 is limited by previously executed, stupid 
job? I expect that if job #1 allocates 300 MB, then job #2 will be 
executed with (more or less) same initial memory utilization.
  
Such attributes do not transfer between processes ;-). The problem with the COB idea of multi-processing is that it isn't really multi-processing when everything is waiting for one SELECT. A SELECT-READNEXT could start queuing with the first result - however a clever B correlative could be made to do the same thing.

4. R06, I have seen that. Did they introduce that because memory leaked 
or internal SELECT is faster? I am joking of course. We did not test 
SELECT much, because we seen that memory utilization during COB 
increases after SSELECTs are run. Perhaps SELECT leaks also something.
  
Different thing altogether :-)

5. ? Hmmm... You need to first collect all keys and store them in memory 
somehow, then to sort them with some algorithm, right? Sorting process 
may require more memory than just size of the keys, but this memory 
should be given back to process once SSELECT is completed.
Depends a little on how the query runs, but if the result of the SSELECT is just the keys, then the memory used at the end will be the size of the key set. However, if you are asking for data columns as well, then you might find the memory requirements are greater because it will avoid re-reading the file if it can and things like that.

 I think that 
it is not (fully?) in jBASE 4.1.5.17.
  
Can't be certain yet. The memory allocation tables have a huge effect on this and is one of the big advantages of watson, which I hope you are at least going to try as soon as possible on your test machine. I have explained this in the past, but the amount of memory that is in use, isn't always the size of the heap. Suppose I allocate 5 bytes, then I allocate 50MB, then I allocate 10bytes, then I free teh 50Mb. The 50 Mb becomes available to future mallocs, but the 10 bytes are still there. So we have a big gap in there. Now, even the current malloc algorithm has guards against simplistic things like this example, but watson is much better at dealing with the whole problem. Problems occur such as allocation and reallocating a growing buffer. eventually you free the old buffer, but need a new bigger one and can never reuse the previous memory, so you run out not so much of memory, but places to put your huge buffer. This is the kind of thing that watson is much better at dealing with, and it is the kind of thing going on in your SSELECT.

Let's leave this subject for confirmation :) As I said in the begining I 
was mistaken many times.
  
Please try watson and tell us the results on your test system.

Jim

Jim Idle

unread,
Feb 6, 2009, 5:13:27 PM2/6/09
to jB...@googlegroups.com
Pawel (privately) wrote:
> Hi,
>
> This is definetely not easy subject - that is why I am coming back.
>
> I was complaining on memory leaks, but there may not be any memory leaks
> :)
>
Nope. It would be that your query is just working on a data set that is
too big. Changing to SELECT READNEXT and forgetting about sorting would
cure that.

> Can anyone explain me what "Free" and "Used" means in below output:
>
Yes ;-)

> 858 16 rx0044 3981556 Port 858 details at 06:08:08 06
> FEB 2009
> Thread type Normal
> Usage Count 72 from 100K
> Application Open Files 14654 , Actual O/S Open Files 181
> Memory: Free 43,755,392 Used 2,095,398,288
> READ's 192042 , WRITE's 14167
> DELETE's 1315 , CLEARFILE's 0
> EXECUTE's 3 , INPUT's 0
> OPEN's 14676
> ---- jsh -Jb -c tSA 60
> Program performing EXECUTE/PERFORM at jsh.b,59
> ---- tSA 60
> Program running normally at F.WRITE,72
>
> I am asking because if process will free say 1,8 GB situation will not
> change from OS level. Memory once allocated (disregarding from "Free" /
> "Used" jBASE indicator) seems not to be given to other processes. Why?
>
This is the info that is coming back from the mallinfo() system call. It
is telling you that the heap size contains 43,755,392 bytes that have
been placed back into the allocation pool but coudl not be released back
to the system. It is also telling you about a huge amount of space that
is in use. This only tells you that it is in use though, not that it was
leaked. We know that it has not been free()d yet, but that might be
normal for a huge SSELECT.

There is a lot more to memory management than I could explain here, but
you need to think of it more in terms of address space than some
physical quantity that is doled out or given back.


> In this way it may happen that multiple processes had their own "peaks"
> in processing and each one allocated (but is not longer using) 1 GB.
> Then memory runs out.
>
> So why "Free" is not free? I am completely lost :(
>

The free table is giving you the amount of free bytes, but it is not
telling you about their order and structure. That could be 43,755,392
blocks all with one byte each, which means if you ask for 2 bytes, then
the system has to tell you it is out of memory. It won't all be 1 byte
blocks of course as the minimum allocation is probably 16 bytes or more,
but unless there is a contiguous block available of the size you need
then the computer say naahhh!

Change to the watson alocator and see if it makes a differnce, but noce
your oprocess is using 2,095,398,288 then it really is using that. We
cannot determine if it needs that to process the data or it is not
freeing something it should be. Only proper analysis tools can tell you
that.

If you want to know more about all this stuff, try:

http://publib.boulder.ibm.com/infocenter/systems/index.jsp?topic=/com.ibm.aix.genprogc/doc/genprogc/sys_mem_alloc.htm

Jim

pat

unread,
Feb 6, 2009, 7:53:25 PM2/6/09
to jBASE
Yep, the 'free' ( null pointer ) above is incorrect
The particular memory allocation it is 'free'ing at the time was
already freed a couple of instructions earlier
[ 'no names, no pack drill' ;-) ]

So although this 'double free'ing is an issue that will be corrected,
it does not account for the excessive memory use.

( Hopefully ) A clarification on the 'memory used' figure shown by
'mw42' when the '-m' option is not used

The 'memory figure' displayed is the 'high watermark'

If the 'm' option is used, or you type 'm' ( without the quotes )
whilst 'mw42' is running, the 'memory' will be displayed in two
columns as 'Free memory' and 'Memory currently being used'

The combined total of these two values should add upto the 'high
watermark value' ( displayed without the 'm' option )

Viewing only the 'high watermark value' explains why you are not
seeing this value increase, until the process actually requires more
memory than it has previously used.

'memory requests' will generally be satisfied by obtaining memory from
the 'Free memory'. Only when the process' memory requirement cannot be
satisfied from the 'Free memory',
will additional memory be requested from the 'system', and
consequently the 'high watermark value' will show an increase

This memory will be 'given' to the 'Free memory' when it is no longer
required, rather than being released back to the system

So basically what I am trying to say is :

The memory value shown by 'mw42' ( without the 'm' option ) is NOT
the amount of memory currently being used by the process, but is the
maximum amount of memory that has been required by the process at some
previous point, ie this displayed value does NOT indicate a 'memory
leak'

Pat.
> jBASE: Attempting to free NULL pointer at ...
>
> read more »

Greg Cooper

unread,
Feb 6, 2009, 8:05:03 PM2/6/09
to jB...@googlegroups.com
The output from the WHERE (V) command needs some clarification. I've repeated the output below.....

The line that says "Memory: Free 43,755,392   Used 2,095,398,288" comes, I believe, directly from a call to mallinfo().

This means that according to the OS there is 2GB of memory in use that nobody else can use and there is about 42Mb of memory in the free space chain that will get re-used.

While Pat is mostly correct in saying that the combined figures represent a "high watermark", there is in fact only 42Mb for re-use. The other 2GB is still marked by the OS as being in use and so will never be re-used.

For example, if you run another program that requires 100 Mb, then as the 2GB is still marked as being in use, the OS will have to create more space to accomodate this and so the "high watermark" will become even higher.

It looks like a memory leak to me.

It could also be named COMMON variables having a very large string inside them.

Greg


Can anyone explain me what "Free" and "Used" means in below output:
858    16        rx0044      3981556  Port 858 details at 06:08:08 06
FEB 2009
                                        Thread type Normal
                                        Usage Count 72 from 100K
                                        Application Open Files 14654 , Actual O/S Open Files 181
                                        Memory: Free 43,755,392  Used 2,095,398,288
                                        READ's 192042 , WRITE's 14167
                                        DELETE's 1315 , CLEARFILE's 0
                                        EXECUTE's 3 , INPUT's 0
                                        OPEN's 14676
                                    ---- jsh -Jb -c tSA 60
                                        Program performing EXECUTE/PERFORM at jsh.b,59
                                    ---- tSA 60
                                        Program running normally at F.WRITE,72

From: pat <pat...@gmail.com>
To: jBASE <jB...@googlegroups.com>
Sent: Friday, 6 February, 2009 19:53:25
Subject: Re: jBASE 4.1.5.17 - does anyone face "out of memory" problems after SSELECT? (was: Re: jBASE 4.1.5.17 - does anyone face "out of memory" problems?)

Pawel (privately)

unread,
Feb 10, 2009, 9:28:40 AM2/10/09
to jBASE
Hi,
 
Thank you for all responses. I am reading AIX materials and started from IBM link that Jim sent (malloc subsystem explanations).
 
Is it all about:

Malloc Disclaim

Malloc Disclaim is an optional extension of the malloc subsystem, providing the user with a means to enable the automatic disclaiming of memory returned by the free subroutine. This is useful in instances where a process has a high paging-space usage, but is not actually using the memory.
Malloc Disclaim is not enabled by default. It can be enabled and configured prior to process startup through the MALLOCOPTIONS environment variable, as follows:
MALLOCOPTIONS=disclaim
We are going to test "Watson" more carefully and play with MALLOCOPTIONS. Be sure that you will hear me again :)
 
Kind regards
Pawel
> read more Âť



----------------------------------------------------
Weź udział w akcji: Zakochana Polska!
Prześlij "serducho miłości" - Kliknij:
http://klik.wp.pl/?adr=http://corto.www.wp.pl/as/14luty.html&sid=636

Jim Idle

unread,
Feb 10, 2009, 4:30:47 PM2/10/09
to jB...@googlegroups.com
Pawel (privately) wrote:
Hi,
 
Thank you for all responses. I am reading AIX materials and started from IBM link that Jim sent (malloc subsystem explanations).
 
Is it all about:

Malloc Disclaim

Malloc Disclaim is an optional extension of the malloc subsystem, providing the user with a means to enable the automatic disclaiming of memory returned by the free subroutine. This is useful in instances where a process has a high paging-space usage, but is not actually using the memory.
Malloc Disclaim is not enabled by default. It can be enabled and configured prior to process startup through the MALLOCOPTIONS environment variable, as follows:
MALLOCOPTIONS=disclaim
We are going to test "Watson" more carefully and play with MALLOCOPTIONS. Be sure that you will hear me again :)

This won't help you I think because that isn't what is happening. The output of WHERE (V isn't telling you that there is a lot of memory that has been freed but not disclaimed, it is telling you that it is in use. I think that there has been some confusion over what Pat said about the mw42 command output vs your WHERE (V output. That (WHERE output) is NOT a high watermark, it is telling you what is actually used!

In fact this option may make performance worse for you in some cases, because unless your system has begun page swapping, there is no need to return the address space to the system.

Be careful with your experimentation. First just turn on watson, then start playing with MALLOCOPTIONS - these things are not really there to just play with to be honest, you really need to understand how the whole thing works to get any benefit other than randomly ;-)

You just need to turn on watson now and see if there is any difference. It think that there will be and that things won't be any worse. If there is a memory leak, then nothing will change really (though it might allow the process to get through the work). Otherwise you either need help to show definitively that this is a memory leak, or you need to get some serious input from TEMENOS technicians (not help desk answers I am afraid).

Jim

Pawel (privately)

unread,
Feb 11, 2009, 6:12:37 AM2/11/09
to jBASE
Hi Jim,

After deeper observations we think that this is exactly what is happening. I have shown one stupid process that allocated almost 2 GB of memory (soon it crashed), but this is really exceptional. Our COB sessions (tSA) allocate a lot of memory on SSELECTs and free most of it immediately. I think that you can easily notice that sorting, not keys selections increases memory consumption.

I am not showing here output from WHERE (V for all processes, but "Free" gives high amounts in total.

We have analysed also nmon results and you can observe that memory consumption during COB is always growing. There are almost no points when it falls down (there are such points in fact, because COB is usually stopped once during night and processes stopped). This is well visible, but totals of "Free" in processes (at some point of time) may give serious amounts.

I am not really 100% sure how memory is allocated by malloc - in pages, right? So will MALLOCOPTION=disclaim give back (fully) unused pages to the OS? Will it happen only when there is out of physical memory?

Additionally: how can we control page size? (size of chunks of memory being allocated?) Is it advisable to change it?

I am going to give you results of some simple program soon. We will try to test allocators on our test server over a weekend.

I know that .profile changes should be considered as dangerous and well pretested (considered as configuration changes).

We did not get response from CSHD yet, but I am sure they will answer shortly.

Kind regards
Pawel
----------------------------------------------------
Weź udział w akcji: Zakochana Polska!
Prześlij "serducho miłości" - Kliknij:

Jim Idle

unread,
Feb 11, 2009, 3:57:58 PM2/11/09
to jB...@googlegroups.com
Pawel (privately) wrote:
Hi Jim,

After deeper observations we think that this is exactly what is happening.
No, not quite. As I say, the reason that subsequent allocations cannot reuse space in the free chain is because of pattern of allocation. That is the main reason to use watson as an allocator. Hence you get a lot of accumulation in the free space chain. Now, it could be that option gives some relief, but the issue is really the allocation pattern. The way watson doles out address space is key to the resue of blocks that are already in the free space chain.

Hence you should try:

1) Switch to watson without other MALLOCOPTS and obsrver the differences;
2) Add options singly to see their individual effect;
3) combine options;

The key here is to make one change at a time of course. Have you done any of this yet?

I have shown one stupid process that allocated almost 2 GB of memory (soon it crashed), but this is really exceptional.
That is good.

Our COB sessions (tSA) allocate a lot of memory on SSELECTs and free most of it immediately. I think that you can easily notice that sorting, not keys selections increases memory consumption.
When you sort, you can either make one pass and accumulate the sort keys and primary key or you can accumulate the output at the same time. I think that SSELECT does the latter, hence you must store all referenced elements and need that much memory. As the keys and list grow the must be reallocated and the old smaller block is released. IF that smaller block is never big enough to satisfy a subsequent malloc, then it will sit in the free chain unused, but still require address space. Hence the option you are looking at MAY allow some of this to return to they system, but the address space is in system pages. Hence increasing the page size as mentioned in an earlier reply. may encourage better fits within the watson allocator, but using the standard allocator, may not help much anyway.


I am not showing here output from WHERE (V for all processes, but "Free" gives high amounts in total.
Sure, but they are not causing you any problems per se. This isn't the root problem, but a symptom. Also, you must asses the free chain as a percentage of the allocation used. What looks like a large number may be a small percentage. Further, because of statistical aberrations, once rogue process could throw out your entire assessment.


We have analysed also nmon results and you can observe that memory consumption during COB is always growing. There are almost no points when it falls down (there are such points in fact, because COB is usually stopped once during night and processes stopped). This is well visible, but totals of "Free" in processes (at some point of time) may give serious amounts.

I am not really 100% sure how memory is allocated by malloc - in pages, right?
YEs, hence for large allocations, using the 64 page size may help. But there are systems pages, the malloc allocator and then the malloced piece. They all interact.

So will MALLOCOPTION=disclaim give back (fully) unused pages to the OS? Will it happen only when there is out of physical memory?
I have not looked at the precise description of this option, but the short description implies that it tries to give back pages as soon as nothing occupies them when free is called. Because the reallocs are relatively large, then this might give back pages to the system. Try it, but only once you have tried watson, then tried watson with the bigger page allocation; then try the combinations.


Additionally: how can we control page size? (size of chunks of memory being allocated?) Is it advisable to change it?
Changing the page size may help, it may make things give up earlier. Only experimentation will show this. The settings for MALLOCOPT to change the page size for larger allocations was in my earlier posts in this thread.


I am going to give you results of some simple program soon. We will try to test allocators on our test server over a weekend.

I know that .profile changes should be considered as dangerous and well pretested (considered as configuration changes).
Only for the live system of course and they are not dangerous if they are pre-tested ;-). On your test system you should be willing to experiment after taking a backup and expecting that you will screw things up a few times before finding the right options.

We did not get response from CSHD yet, but I am sure they will answer shortly.
Perhaps a nudge is in order if you are in dire straights.

Jim

Pawel (privately)

unread,
Feb 11, 2009, 7:50:53 PM2/11/09
to jBASE
Welcome back,

I suggested MALLOCOPTION=disclaim, because we have noticed that on 1 of test machines memory situation was like that:
01:02 AM: ~ jBASE Free in total 350 MB, jBASE Used in total 5 GB
03:20 AM: ~ jBASE Free in total 8 GB, jBASE Used in total 7 GB (more "Free" than "Used", this is not my mistake)

Numbers came from totaled Free and Used of WHERE (V (for all jBASE processes). I guess that memory consumption in COB must be looking similar in other banks (by the way: I previously never checked such things, so it would be great if somebody of group members could check it and share here).

Above output suprised me seriously. Shown numbers are not exact - I tried to recall them and they are quite precise. Situation at 3:20 AM was exactly like that (server has 24 GB of physical mem + 10 GB of swap).

We have also noticed that memory allocation growth (jump) is on SSELECTs. So I am guessing that freed memory should formulate continious blocks.

That is why I claimed all the time that "Free" memory is not given back to other processes and this is our problem. I understood that this may be problem of memory fragementation, but I think that yorktown allocator never gives memory back unless (probably) MALLOCOPTION=disclaim is set. That why I asked about memory page size. I am thinking about this whole stuff and can not imagine such a big fragmentation (notice that at 3:20 AM are processes which for example allocated and used 300 MB, but immediately went back to 100 MB used and 200 MB free).

We have checked also wheter some processes try to allocate more than 2 GB and there are such COB agents, but hopefully not many. We need to do more analysis carefully. Testing of Watson is scheduled to weekend - you can imagine now how people from test team are protesting against such changes :)

I forgot to tell, but QA area was all the time working on MALLOCTYPE=buckets. We did this setting, because people from CSHD gave us such advice. We wonder if it could somehow accelerate our problems or not. Recently we are facing out of memory problems on test servers and we did not (!) increase memory/swap size.

End of topic for the moment, because thread will be qualified as spam ;)

Kind regards
Pawel



Kind regards
Pawel

Jim Idle

unread,
Feb 11, 2009, 9:07:28 PM2/11/09
to jB...@googlegroups.com
Pawel (privately) wrote:
Welcome back,

I suggested MALLOCOPTION=disclaim, because we have noticed that on 1 of test machines memory situation was like that:
01:02 AM: ~ jBASE Free in total 350 MB, jBASE Used in total 5 GB
03:20 AM: ~ jBASE Free in total 8 GB, jBASE Used in total 7 GB (more "Free" than "Used", this is not my mistake)
As I say, it is possible that that option would allow the system to reuse the pages, but the real problem is the allocation pattern. You need to use watson and gather the same information.


Above output suprised me seriously. Shown numbers are not exact - I tried to recall them and they are quite precise. Situation at 3:20 AM was exactly like that (server has 24 GB of physical mem + 10 GB of swap).

We have also noticed that memory allocation growth (jump) is on SSELECTs. So I am guessing that freed memory should formulate continious blocks.
Well, it will be as many blocks as were required to be re-allocated, but when freed were not big enough to satisfy further growth.

That is why I claimed all the time that "Free" memory is not given back to other processes and this is our problem.
It's not quite how it works, but it is a problem, that watson should address by allowing the processes to reuse the freed blocks (because it can amalgamate them better upon free())

I understood that this may be problem of memory fragementation, but I think that yorktown allocator never gives memory back unless (probably) MALLOCOPTION=disclaim is set.
Well, the disclaim is only going to let things shrink when it can. Given the allocation pattern, my feeling is that it would not make much difference, but it might.
That why I asked about memory page size. I am thinking about this whole stuff and can not imagine such a big fragmentation (notice that at 3:20 AM are processes which for example allocated and used 300 MB, but immediately went back to 100 MB used and 200 MB free).
Different allocation patterns allow different reuse, and of course if a new process starts up then it starts from scratch.

Speculation isn't going to help you basically, you have to test it.


We have checked also wheter some processes try to allocate more than 2 GB and there are such COB agents, but hopefully not many. We need to do more analysis carefully. Testing of Watson is scheduled to weekend - you can imagine now how people from test team are protesting against such changes :)
Well, they shouldn't be - they obviously don't understand the process. That's what your test system is for and I don't know the details of why you cannot just switch that on in your test machine now, but I am sure it is politics rather than technical points.

I forgot to tell, but QA area was all the time working on MALLOCTYPE=buckets. We did this setting, because people from CSHD gave us such advice. We wonder if it could somehow accelerate our problems or not. Recently we are facing out of memory problems on test servers and we did not (!) increase memory/swap size.
Well, you are using different policies on your test and live machines now, so the QA doesn't mean anything :-). You need your QA and live machines running the same parameters exactly, other than when you are testing new parameters for going live with.

Jim
Reply all
Reply to author
Forward
0 new messages