SELECT limitations

[ppriv...@wp.pl]

Hi!

I would like to ask if this is jBASE software bug or problem is caused
by other source (fe. user's limits):

jsh sandie ~ -->SELECT REALLY.BIG.(DISTRIBUTED).FILE BY @ID
jBASE: Unable to allocate 114439047 bytes, errno = 12 at
jlibBStore.c,347(ENQRUN
.b,995)

Program 'SELECT' at pid 1802262 - Memory error
Program source name ENQRUN.b , line 995
Memory fault(coredump)

REALLY.BIG.DISTRIBUTED file does not neccesarilly have to be
distributed to encounter this problem. Even when it is normal 2GB less
jBASE file (over 2000000 records) I got the same result. Does anybody
know SELECT limitations? I am not aware of them, so please let me know.

Here are some server's / user data:
jsh sandie ~ -->ulimit -a
time(seconds) unlimited
file(blocks) unlimited
data(kbytes) unlimited
stack(kbytes) 32768
memory(kbytes) unlimited
coredump(blocks) 2097151
nofiles(descriptors) 2000
jsh sandie ~ -->jdiag
jdiag - jBASE diagnostic '$Revision: 1.12 $'

System Information
==================

System : AIX server1 1.5 0053C6DA4C00
OS Release : 5.1.0.0
UNIX User : MIGTEST (uid 507, euid 507)
Tty name : /dev/pts/17
Time : Fri May 6 11:58:11 2005

Environment
===========

JBCRELEASEDIR : '/usr/jbc'
JBCGLOBALDIR : '/usr/jbc'
HOME : '/PATH/bnk/bnk.run'
JEDIFILEPATH : '/PATH/bnk/bnk.run'
JEDIFILENAME_MD : '/PATH/bnk/bnk.run/VOC'
JEDIFILENAME_SYSTEM : '/jbase'
JBCBASETMP : '/PATH/bnk/bnk.run/jBASEWORK/tmp_2383960'
JBCNOINTERNAL : Not Set
JEDI_NOSHMEM : '1'
RELEASE Information : Major 4.0 , Minor 5 , Patch 0215
Spooler dir (JBCSPOOLERDIR) : '/jbase/jspooler'
JBCEMULATE : 'prime'
Object path (JBCOBJECTLIST) :
'/PATH/bnk/bnk.run/globuspatchlib:/PATH/bnk/bnk.run/lib:/PATH/bnk/bnk.run/globuslib'
Program dir (JBCDEV_BIN) : '/PATH/bnk/bnk.run/bin'
Subroutine dir (JBCDEV_LIB) : '/PATH/bnk/bnk.run/lib'
There are No warnings, jBASE seems to be loaded correctly

Regards
Pawel

[Greg Cooper]

I can't give you an exact value I'm afraid, but the numbers go something like this.

 

The SELECT uses normal OS heap space to do it's work. On 32 bit systems the limit for heaps space is approximately 1Gb per process. This is a hardware limitation of 32 bit virtual memory operating systems and will apply regardless of what your ulimit settings are, or how much real memory you have installed, or how much swap space you have configured etc.

 

While building up its list of selected item id's, jBASE will have to constantly re-size this list. So if the list is sized at say 300 Mb , and it needs 301 Mb, it calls the OS function realloc() to resize it. You might get lucky and find that realloc() just appends another 1Mb to the end, but if the heapspace has become fragmented it might have to allocate 301 Mb from new, copy the old 300Mb to the new 301 Mb, and free the old 300 Mb. Hence in this second scenario although we only require 301 Mb, for an instant we need (300 + 301) Mb.

 

To make matters worse, in the above example, let us say we now need 302 Mb. It can't use the 301 Mb space it has been using previously, it can't use the 300 Mb space it free'd earlier, so it has to find a new 302 Mb space, and now you'll have fragmentation of 300 Mb not-used, 301 Mb not used and 302 Mb in use. You might get lucky and the OS will append the unused 300 Mb to the unused 301 Mb to create a new unused 601 Mb unused heap space.

 

So how much space jBASE can use for it's select list depends upon how clever the OS is at re-using fragmented workspace. jBASE will of course try to keep this fragmentation down by ensuring when it does this realloc() of workspace it add some "extra for good luck" so it doesn't call realloc() too often hence my examples of using 1Mb increments were for illustration only.

 

 

There will be some 64 bit versions of jBASE available soon which will circumvent the problem, but don't hold your breath waiting for them -- and you'll need 64 bit hardware of course !

 

I guess the answer is to SELECT each distrubuted file individually and write out the select lists, use a Unix (AIX)  command such as 'cat' to concatenate the results and then you can have a select list up to the size supported by your file system, for AIX 5 that's probably a 64 bit file system. Of course the problem is, once you've got this huge list created, let's say it is 500Mb of item id's, will you have subsequent problems with a 1Gb heap space limitation when you try to later use that select list ? If the resultant select list is less than 300 Mb I'd say "probably not", more than 600 Mb I'd say "probably yes" , but don't quote me as these are pure guesstimates.

 

Ooops, I've gone on a bit haven't I ;-( Hope this helps even if it doesn't give an exact answer which I'm sure you'd prefer.

Greg Cooper, McFatty Limited.

ppriv...@wp.pl wrote:

[Sandie]

Hi!

Thanks for quick reply Greg. Just before moment I have learned new
usefull AIX command - lsconf. At the end of this email you can find
command's output.

I think I have clearly understood your email. Please notice that my AIX
server is running in 64bit kernel mode. So 1GB heap memory limit is not
the case (check also server's statistics - there is plenty of free
memory). Again I am asking - is it jBASE's software bug? I produced
list of IDs for mentioned table using SELECT/READNEXT jBASIC statements
(program went fine). Whole list of IDs takes approx. 80 MB (IDs were
saved to file in separate lines).

Why simple SELECT consumes so much memory? Why did it crash? Has anyone
of you encountered similar problem? Any other ideas?

jsh sandie ~ -->lsconf
System Model: IBM,7028-6C4
Machine Serial Number: 653C6DA
Processor Type: PowerPC_POWER4
Number Of Processors: 4
Processor Clock Speed: 1000 MHz
CPU Type: 64-bit
Kernel Type: 64-bit
LPAR Info: 1 NULL
Memory Size: 16384 MB
Good Memory Size: 16384 MB
Firmware Version: IBM,RR020815_srv1
Console Login: enable
Auto Restart: true
Full Core: false

Network Information
Host Name: server1
IP Address: xxx.xxx.xxx.xxx
Sub Netmask: xxx.xxx.xxx.xxx
Gateway: xxx.xxx.xxx.xxx
Name Server:
Domain Name:

Paging Space Information
Total Paging Space: 3840MB
Percent Used: 1%

Volume Groups Information
==============================================================================

rootvg:
PV_NAME PV STATE TOTAL PPs FREE PPs FREE
DISTRIBUTION
hdisk0 active 542 2
00..00..00..00..02
==============================================================================

localvg:
PV_NAME PV STATE TOTAL PPs FREE PPs FREE
DISTRIBUTION
hdisk7 active 271 20
00..00..00..00..20
hdisk8 active 546 5
00..00..05..00..00
hdisk9 active 546 6
00..00..06..00..00
==============================================================================

hsgvg:
PV_NAME PV STATE TOTAL PPs FREE PPs FREE
DISTRIBUTION
hdisk13 active 203 2
00..00..00..00..02
hdisk6 active 203 2
00..00..00..00..02
==============================================================================

hsgnewvg:
PV_NAME PV STATE TOTAL PPs FREE PPs FREE
DISTRIBUTION
hdisk11 active 677 16
00..00..00..00..16
hdisk14 active 677 16
00..00..00..00..16
==============================================================================

INSTALLED RESOURCE LIST

The following resources are installed on the machine.
+/- = Added or deleted from Resource List.
* = Diagnostic support not available.

Model Architecture: chrp
Model Implementation: Multiple Processor, PCI bus

+ sys0 00-00 System Object
+ sysplanar0 00-00 System Planar
+ mem0 00-00 Memory
+ proc0 00-00 Processor
+ L2cache0 00-00 L2 Cache
+ proc1 00-01 Processor
+ proc2 00-02 Processor
+ proc3 00-03 Processor
* pci0 00-400000000110 PCI Bus
* isa0 1G-18 ISA Bus
+ fda0 01-D1 Standard I/O Diskette Adapter
* siokma0 01-K1 Keyboard/Mouse Adapter
+ sioka0 01-K1-00 Keyboard Adapter
+ sioma0 01-K1-01 Mouse Adapter
+ ppa0 01-R1 CHRP IEEE1284 (ECP) Parallel Port
Adapter
+ sa0 01-S1 Standard I/O Serial Port
+ tty0 01-S1-00-00 Asynchronous Terminal
+ sa1 01-S2 Standard I/O Serial Port
+ sa2 01-S3 Standard I/O Serial Port
* ide0 1G-19 ATA/IDE Controller Device
+ cd0 1G-19-00 IDE CD-ROM Drive I (650 MB)
* pci2 1G-10 PCI Bus
* scsi4 1H-08 Cambex Fibre Channel I/O
Controller
* hdisk1 1H-08-00-1,0 Compaq HSV110 Enterprise Virtual
Array
* hdisk2 1H-08-00-2,0 Compaq HSV110 Enterprise Virtual
Array
* hdisk5 1H-08-00-3,0 DEC HSG80 Command Console LUN
* hdisk6 1H-08-00-3,7 DEC HSG80 RAID Array
* hdisk10 1H-08-00-5,0 DEC HSG80 Command Console LUN
* hdisk11 1H-08-00-5,1 DEC HSG80 RAID Array
* hdisk14 1H-08-00-5,2 DEC HSG80 RAID Array
* pci3 1G-12 PCI Bus
+ ent0 1L-08 10/100 Mbps Ethernet PCI Adapter
II
(1410ff01)
* pci4 1G-14 PCI Bus
+ scsi0 1S-08 Wide/Ultra-3 SCSI I/O Controller
+ hdisk0 1S-08-00-8,0 16 Bit LVD SCSI Disk Drive (36400
MB)
+ hdisk7 1S-08-00-9,0 16 Bit LVD SCSI Disk Drive (36400
MB)
+ hdisk8 1S-08-00-10,0 16 Bit LVD SCSI Disk Drive (73400
MB)
+ hdisk9 1S-08-00-11,0 16 Bit LVD SCSI Disk Drive (73400
MB)
* ses0 1S-08-00-15,0 SCSI Enclosure Services Device
+ scsi1 1S-09 Wide/Ultra-3 SCSI I/O Controller
* pci5 1G-16 PCI Bus
* pci1 00-400000000112 PCI Bus
* pci6 10-10 PCI Bus
+ scsi2 11-08 Wide/Ultra-3 SCSI I/O Controller
+ rmt0 11-08-00-0,0 SCSI 4mm Tape Drive (20480 MB)
+ scsi3 11-09 Wide/Ultra-3 SCSI I/O Controller
* pci7 10-12 PCI Bus
+ ent1 14-08 10/100 Mbps Ethernet PCI Adapter
II
(1410ff01)
* pci8 10-16 PCI Bus
* scsi5 1D-08 Cambex Fibre Channel I/O
Controller
* hdisk3 1D-08-00-1,0 Compaq HSV110 Enterprise Virtual
Array
* hdisk4 1D-08-00-2,0 Compaq HSV110 Enterprise Virtual
Array
* hdisk12 1D-08-00-3,0 DEC HSG80 Command Console LUN
* hdisk13 1D-08-00-3,6 DEC HSG80 RAID Array
* hdisk15 1D-08-00-5,0 DEC HSG80 Command Console LUN

Thanks in advance.

Regards
Pawel

[Greg Cooper]

It could be a bug in jBASE I cannot discount that.

 

However it could still be the problems I spoke of before.

 

Even though you are running a 64 bit kernel, the version of jBASE you are using is 32 bit only, meaning once the process is running it is still constrained by a 32 bit address space, which when split into stack, heap and text space gives only 1Gb maximum per process. To make use of 64 bits you need 64 bit hardware, 64 bit OS and 64 bit application.

 

Try these things.

 

While the SELECT is executing, try running  repeatedly a "ps -el | grep SELECT" command and you can see how big your process is , maybe with memory fragmentation it is getting larger than you expect. At least you can see if the SELECT bombs out even if it isn't using much memory.

 

Then try the following jBASE program which builds up a list of 2 million record keys at 40 bytes each. From the 'ps' output you can see it eventually used up 36253 pages at 4k each which is 141 Mb for what should only be 80 Mb of data. This isn't too bad actually, it shows my Linux machine is working quite well. However on your AIX it might show the heap space manager isn't so good at re-using space and hence it becomes a jBASE "feature" rather than a bug.

 

Hope these help. Other than the above, well, it's down to asking jBASE.

 

Greg

linux3-mcfatty-~/greg: cat testprog.b
    list = ""
    FOR keycount = 1 TO 2000000
        list<-1> = "about a 40 byte key" "L#40"
        IF MOD(keycount,100000) EQ 0 THEN
            PRINT "list size ":LEN(list):" , number keys ":keycount "ML,"
            EXECUTE CHAR(255):"k":"ps -el | grep testprog"
        END
    NEXT keycount
    EXECUTE CHAR(255):"k":"ps -el | grep testprog"
linux3-mcfatty-~/greg: jbc testprog.b -o testprog
linux3-mcfatty-~/greg: ./testprog
list size 4099999 , number keys 100,000
0 S   502  9278  3683  0  80   0 -  9273 wait   pts/1    00:00:00 testprog
list size 8199999 , number keys 200,000
0 S   502  9278  3683  0  82   0 - 10901 wait   pts/1    00:00:00 testprog
list size 12299999 , number keys 300,000
0 S   502  9278  3683 89  85   0 - 12505 wait   pts/1    00:00:00 testprog
list size 16399999 , number keys 400,000
0 S   502  9278  3683 99  85   0 - 12505 wait   pts/1    00:00:01 testprog
list size 20499999 , number keys 500,000
0 R   502  9278  3683 99  85   0 - 15070 -      pts/1    00:00:01 testprog
list size 24599999 , number keys 600,000
0 R   502  9278  3683 89  85   0 - 15070 -      pts/1    00:00:01 testprog
list size 28699999 , number keys 700,000
0 R   502  9278  3683 99  85   0 - 19176 -      pts/1    00:00:02 testprog
list size 32799999 , number keys 800,000
0 R   502  9278  3683 99  85   0 - 19176 -      pts/1    00:00:02 testprog
list size 36899999 , number keys 900,000
0 S   502  9278  3683 89  85   0 - 19176 wait   pts/1    00:00:02 testprog
list size 40999999 , number keys 1,000,000
0 S   502  9278  3683 98  85   0 - 19176 wait   pts/1    00:00:02 testprog
list size 45099999 , number keys 1,100,000
0 S   502  9278  3683 99  85   0 - 25744 wait   pts/1    00:00:03 testprog
list size 49199999 , number keys 1,200,000
0 S   502  9278  3683 88  85   0 - 25744 wait   pts/1    00:00:03 testprog
list size 53299999 , number keys 1,300,000
0 S   502  9278  3683 96  85   0 - 25744 wait   pts/1    00:00:03 testprog
list size 57399999 , number keys 1,400,000
0 S   502  9278  3683 99  85   0 - 25744 wait   pts/1    00:00:04 testprog
list size 61499999 , number keys 1,500,000
0 S   502  9278  3683 99  85   0 - 25744 wait   pts/1    00:00:04 testprog
list size 65599999 , number keys 1,600,000
0 S   502  9278  3683 94  85   0 - 25744 wait   pts/1    00:00:04 testprog
list size 69699999 , number keys 1,700,000
0 S   502  9278  3683 99  85   0 - 25744 wait   pts/1    00:00:05 testprog
list size 73799999 , number keys 1,800,000
0 S   502  9278  3683 99  85   0 - 36253 wait   pts/1    00:00:05 testprog
list size 77899999 , number keys 1,900,000
0 S   502  9278  3683 93  85   0 - 36253 wait   pts/1    00:00:05 testprog
list size 81999999 , number keys 2,000,000
0 S   502  9278  3683 98  85   0 - 36253 wait   pts/1    00:00:05 testprog
0 S   502  9278  3683 98  84   0 - 36253 wait   pts/1    00:00:05 testprog
linux3-mcfatty-~/greg:

[Jim Idle]

From: Greg Cooper [mailto:gr...@mcfatty.com]
Sent: Friday, May 06, 2005 4:32 AM
To: jB...@googlegroups.com
Subject: Re: SELECT limitations

 

I can't give you an exact value I'm afraid, but the numbers go something like this.

 

The SELECT uses normal OS heap space to do it's work. On 32 bit systems the limit for heaps space is approximately 1Gb per process. This is a hardware limitation of 32 bit virtual memory operating systems and will apply regardless of what your ulimit settings are, or how much real memory you have installed, or how much swap space you have configured etc.

 zzzzzzzzzzzzZZZZZZZZZZZ all true of course, however perhaps the issue is that what is being done probably exceeds reasonable performance expectations even if it did not blow up.

jsh sandie ~ -->SELECT REALLY.BIG.(DISTRIBUTED).FILE BY @ID
jBASE: Unable to allocate 114439047 bytes, errno = 12 at
jlibBStore.c,347(ENQRUN
.b,995)

Program 'SELECT' at pid 1802262 - Memory error
Program source name ENQRUN.b , line 995
Memory fault(coredump)

This is possibly being caused by the sort part of it (which you could verify by removing the BY clause). If they need to have the ID in sorted order, then an index may circumvent the issue. Create a single valued index on the ID, then use KEY-SELECT. The index overhead should be minimal once it is created.

Now, why do they wish to sort by the Item ID?

Jim

[Jim Idle]

> -----Original Message-----
> From: Sandie [mailto:ppriv...@wp.pl]
>
> Thanks for quick reply Greg. Just before moment I have learned new
> usefull AIX command - lsconf. At the end of this email you can find
> command's output.
>
> I think I have clearly understood your email. Please notice that my
AIX
> server is running in 64bit kernel mode. So 1GB heap memory limit is
not
> the case (check also server's statistics - there is plenty of free
> memory).

I am afraid that this does not make any difference as you are using a 32
bit version of jBASE, so the kernel runs it in 32 bit mode and there is
no difference and no extra space available, you have the same limits as
if you were running a 32 bit system.

64 bit machines are not the panacea the hardware manufacturers would
like you to believe of course. In some case they encourage you to keep
using more and more resources because you can, instead of looking at
what you are doing with a more critical eye.

You could try the 64 bit jBASE 4.1 build, which would make it all go
away I believe. However, really you should look at the validity of what
is being done. Perhaps you can explain WHY this needs to be done and we
can show you a better/faster/simpler way.

I think that the indexes mentioned in an earlier post would solve the
issue of the SELECT, but if you are then going to traverse the file in
something other than the ID hashing order, on such a big file,
especially on distributed files, this will take an age. There are tricks
in the distributed file key generation that can help enormously with
this, but what is driving the need for that sort order?

> Again I am asking - is it jBASE's software bug?

Extremely unlikely, though the jBASE 3.x SELECT statement can be pretty
inefficient memory wise. You might make sure that you AIX system is
patched up to date, but I don't think it is that either. jBASE 4.1
solves all of that, even in 32 bit mode, though at some point you would
still run out of memory.

> list of IDs for mentioned table using SELECT/READNEXT jBASIC
statements
> (program went fine). Whole list of IDs takes approx. 80 MB (IDs were
> saved to file in separate lines).
>
> Why simple SELECT consumes so much memory? Why did it crash? Has
anyone
> of you encountered similar problem? Any other ideas?

Greg did explain why it seemed to take so much memory at a system level,
however you are also SORTING the keys too, which requires more memory
and as I mentioned the jBASE 3.x SELECT can be fairly inefficient at
this. jBASE 3.x uses the runtime libraries to build the list and sort,
which have to assume all sorts of generic possibilities for dynamic
growth and shrinkage (just like Greg), 4.1 is more attuned to what
SELECT//SSELECT actually requires.

>
> jsh sandie ~ -->lsconf

As mentioned, the limitation is what can be used rather than what is
available unfortunately.

I suggest then:

jsh $ CREATE-INDEX -mv FILE SID BY 0 # Which is done once only

jsh $ KEY-SELECT FILE SID

349823487237487 Items selected.
>

However, then review why this key order is a requirement - it might be
that it actually isn't necessary.

Jim

[Jim Idle]

Jim Idle wrote:
> > -----Original Message-----
> > From: Sandie [mailto:ppriv...@wp.pl]
> >
> > Thanks for quick reply Greg. Just before moment I have learned new
> > usefull AIX command - lsconf. At the end of this email you can find
> > command's output.
> >
> > I think I have clearly understood your email. Please notice that my
> AIX
> > server is running in 64bit kernel mode. So 1GB heap memory limit is
> not
> > the case (check also server's statistics - there is plenty of free
> > memory).

Sandie,

Any replies to my questions yet?

[Sandie]

Hi!

I have made test and everything went fine (as you can see memory
management is a little bit worse than in Linux). You can find test
output at the end of this email.

I think, that you have correctly suggested that jBASE programs are not
64bit ones. I am not sure now how can I verify this parameter. "file"
command says nothing about kind of executable - "/usr/jbc/bin/SELECT:
executable (RISC System/6000) or object module".

Regards
Pawel

PS. I was absent on Monday - sorry for discussion delay.

jsh sandie ~ -->testprog

list size 4099999 , number keys 100,000

240801 A 442 778378 1327122 25 72 20 47511400 13588
pts/36 0:00 testprog
[...output has been partially cut off...]
240801 A 442 778378 1327122 60 90 20 47511400 140140
pts/36 0:09 testprog

list size 77899999 , number keys 1,900,000

240801 A 442 778378 1327122 114 117 20 47511400 144144
pts/36 0:09 testprog

list size 81999999 , number keys 2,000,000

240801 A 442 778378 1327122 83 101 20 47511400 148148
pts/36 0:11 testprog
240801 A 442 778378 1327122 90 105 20 47511400 148148
pts/36 0:11 testprog
jsh sandie ~ -->

[Sandie]

Hi Jimmy!

As I have written I was absent till Monday. Below you can find my
answers.

Sometimes you are made to use "bought", closed-source code and you can
do nothing with that (you can just complain or ask about changes).
Simple SELECT BY @ID was used only to show the problem. I could use
even simpler SELECT, but I have noticed that SELECT BY @ID crashes
"faster".

I know there is rather no need to use such SELECTs in production
environment, but I try to asses chances that production, maily
closed-source code will fail. If jBASE external SELECT's are unstable
for large files I can not sleep well. I think you can agree with me?
SELECT/READNEXT commands works well for us, but what can we do with
plenty of "black-boxes"? My answer is nothing :(

I should notice (I did not want to, now I have to) that we are using
jBASE shipped with one Temenos banking application. I remember some
strange issues with this jBASE release; for example: compiler's
national characters enconding problem, strange versioning (jBASE
reports itself as 4.1) and others. I think all these "strange" issues
might have been solved already, but our jBASE version is frozen. Our
management staff (I support them here) will not agree to exchange
jBASE's release on production system without testing.

So Jim what can you advise us? Do not you think that production code
that is released by jBASE should not end with coredumps? What can we do
now? Normally users enters some selection criteria to our SELECTs, but
data growth is so amazing, that I wonder when SELECT will crash.

Waiting for reply :)

Regards
Pawel

PS. I will make sure if jBASE code is 32bit. Maybe one of you can point
me to proper command?

[Simon Verona]

It sounds like you are using the Temenos only release 4.0. AFAIK this
wasn't available outside of Temenos.

I think you will need to contact Temenos support direct on this issue.

Regards


Simon Verona
Dealer Management Services Ltd
The Old Cross
Woodfall Lane
Low Bradfield
S6 6LB

Tel: 0870 080 2300
Fax: 0870 169 6747
Email: si...@dmservices.co.uk

-----Original Message-----
From: Sandie [mailto:ppriv...@wp.pl]

Sent: 10 May 2005 09:16
To: jB...@googlegroups.com
Subject: Re: SELECT limitations

[Sandie]

Hi Simon!

Thanks for the advice. Sometimes I wonder if they treat specifically
their customers ;) Should not they send important issues/patches to
their customers? They do, but as far as I know partially. Patches come
only to banking application. I have never heard of jBASE patch (I have
limited informations :)). I bet (sorry for that) that publicly
available jBASE code is far better tested that one (specific) we own.

Jim, do you think is it our specific jBASE issue?

Regards
Pawel

PS. I like Temenos people (I know few of them personally), but rather
dislike Temenos "customer care" policy.

[Jim Idle]

Given the information you have provided, I can absolutely assure you
that you are running a 32 bit version of jBASE, which is actually called
4.0.

However, as you are running Globus, then you should have access to
TEMENOS support, who have the source code for both the application and
that should mean they can fix your issue. The Globus code uses rather
too many SSELECT statements on large files and at the end of the day it
is difficult to avoid the core dump without impacting performance all
over the place.

It is very rare for things to get so big that they core dump though - I
would strongly suggest investigating this with your Globus/jBASE support
people.

Jim

[Jim Idle]

It could be specific to 4.0, but I doubt it rally. It is more likely
something to do with the interaction with Globus.

Jim

> -----Original Message-----
> From: Sandie [mailto:ppriv...@wp.pl]
> Sent: Tuesday, May 10, 2005 2:26 AM
> To: jB...@googlegroups.com
> Subject: Re: SELECT limitations
>
>

[Sandie]

Hi Jim!

First of all I would like to explain jBASE versioning. According to the
informations I got on jBASE Techlist some time ago (about 2003-08-09,
thread entitled "Temenos & JBase") it seems to me that our jBASE
version (bundled with Temenos banking product) is indepedently numbered
than regular jBASE. Group members told me that time, that our "4.0.6.2"
is not truly 4th one - it is (specially modified) 3.xx (that time
regular 4.xx was just expected, not available).

Sorry for telling you that we are using so called "4.1". It is in fact
"4.0.5.3" (I mean "3.xx" publicly available). I wrote version
information from memory and I made a little mistake. To be accurate -
jDiag presents our jBASE version as "Major 4.0 , Minor 5 , Patch 0215".

So Jim, do you know is this bug (?) was discovered in 3.xx? Maybe it
has been fixed years ago and we have unpatched jBASE release? I will
try to check jBASE's 3.xx patch log (if there is any).

I would like also to mention that I have searched jBASE Techlist
archives. I have found my old email there :) - see
http://listserver.jbase.com/mailman/htdig/tech/2003/001309.html. Bruce
Willmore advised us to increase kernel's Shared Memory size. I do not
know AIX kernel details at all, but I will push admins to read more
about AIX kernel (maybe it is ). Is it possibility that coredumps occur
because of this setting? (I do not think so) I think we have never
touched such parameter.

Regards
Pawel

[Jim Idle]

Pawel,

Yes - I understand the version you are on and it is indeed a slightly
modified version of 3.x - basically it has a bit more UniVerse
compatibility than standard 3.x. But it is kept at least as up to date
as 3.x.

The situation you are seeing is not a bug as such, just that the query
eventually runs out of memory - it is possible that you could get
TEMEMNOS to make the SELECT statement more memory efficient but that
isn't so easy to do. The issue is really caused by how Globus files are
organized and you will need to seek a solution through them. What do
your Globus/TEMENOS support people say?

It is possible that ulimit is smaller than it should be, but I strongly
suspect that Greg's initial analysis was correct. The advice to increase
the size of AIX shared memory space was unfortunately incorrect - this
will have no effect on your query failing with no more memory available
as it is a process local limit.

I would ask about moving to the 4.1 release (at least over the long
term), which will give you all sorts of improvements for Globus (but I
don't know how close they are to Globus being release on 4.1).

A coredump occurs for a number of reasons, but ultimately because a
pointer to memory was de-referenced and that pointer was pointing to an
illegal address. This can happen because a program keeps allocating more
and more memory and does not check for an allocation failure so tries to
reference memory address 0, or simply a bug where a pointer gets
corrupt. This is not happening in your query though (I strongly suspect
it isn't at least).

To be honest, with jBASE 3.1 and those files, it is quite possible that
there is no perfect solution. You may have to define indexes and use
them where you know you can.

At this stage though your only real resort is TEMENOS themselves. Sorry
that we cannot be of further assistance.

Jim



> -----Original Message-----
> From: Sandie [mailto:ppriv...@wp.pl]
> Sent: Thursday, May 12, 2005 1:20 AM
> To: jB...@googlegroups.com
> Subject: Re: SELECT limitations
>
>

[mike ryder]

You should put this query to helpdesk_at_temenos.com
It occurs when the ID is right justified and the declared length in the
dictionary is say 50 chars - this particularly occurs with
SC.VALUATION.EXTRACT and associated tables.

hth
Mike