jbase problems show-items-locks

2,480 views
Skip to first unread message

adrian

unread,
Aug 5, 2011, 11:39:18 PM8/5/11
to jBASE
When running many threads say to update customer (1,000.000) some of
the tsa's show SLEEP in the mw42 view.

Looking at the show-items-locks we are see many times lately the
RECORDKEY is blank in the locking tables, when this is happening the
agents are only processing 120 per minute while if we see no blank
keys then we can process 20,000 + a minute.

Have we a problem with our Jbase version or is this normal to see
records in the lock table with blanks , I must say i have never seen
this over the pass years.

We are not using the jbase locking but the UNIX locking (set at
24,000) on a HP box as the jDLS was causing problems. The unix locking
has been running fine for over a month now.

Could somebody knowing Jbase explain many thanks.

Sorry we are using JR type files database and R10 with many patches.

System Information
==================

System : HP-UX vic-samt B.11.31.U ia64
UNIX User : aatkinso (uid 187, euid 187)
Tty name : /dev/pts/0
Time : Fri Aug 5 20:31:32 2011

Environment
===========

JBCPORTNO : Not Set
TAFC_HOME : '/app/tafc/r10SP8/R10'
JBCGLOBALDIR : '/app/tafc/r10SP8/R10'
WARNING: JBCDATADIR is not set, Default '/app/tafc/r10SP8/R10/
jbase_data'
WARNING: JBCDATADIR is subdirectory of JBCGLOBALDIR
HOME : '/app/t24/dm5/bnk.run'
JEDIFILEPATH : '/app/t24/dm5/bnk.run'
JEDIFILENAME_MD : '/app/t24/dm5/bnk.run/VOC'
JEDIFILENAME_SYSTEM : '/app/t24/dm5/bnk.run/SYSTEM'
RELEASE Information : Major 10.0 , Minor 0.8 , Patch (Change
92702)
Spooler dir (JBCSPOOLERDIR) : '/usr/spool/jspooler'
JBCEMULATE : 'prime'
Object path (JBCOBJECTLIST) : '/app/t24/dm5/bnk.run/ccslib:/app/t24/
dm5/bnk.run/GR0800004lib:/app/t24/dm5/bnk.run/t24lib:/app/t24/dm5/
bnk.run/GR0800005lib:/app/t24/dm5/bnk.run/GR0800006lib:/app/t24/dm5/
bnk.run/radlib:/app/t24/dm5/bnk.run/cardlib:/app/t24/dm5/bnk.run/
usrdlib:/app/t24/dm5/bnk.run/GR0800012lib:/app/t24/dm5/bnk.run/atmlib'
jBASE Compiler Run-time : '/app/tafc/r10SP8/R10/config/
system.properties'
Program dir (JBCDEV_BIN) : '/app/t24/dm5/bnk.run/ccsbin'
Subroutine dir (JBCDEV_LIB) : '/app/t24/dm5/bnk.run/ccslib'
Max open files : 8192
jsh aatkinso ~ -->

VK

unread,
Aug 10, 2011, 10:16:42 AM8/10/11
to jBASE
Hi,

Clarify several things please:

> many threads say to update customer

How many? Which method is used? Service OFS.MESSAGE.SERVICE I presume?

> RECORDKEY is blank in the locking tables

Which tables exactly? F.LOCKING or F.JOB.LIST.n?

> JR type files database

Secure updates or non-secure?

Last but not least - updating 1m customers looks like a one-time task,
not a regular one, isn't it?

VK

adrian

unread,
Aug 10, 2011, 1:45:56 PM8/10/11
to jBASE
We are using 10 threads at the moments

we are using JR type out of the box from Temenos. (non-secure)

we are using DM.SERVICE.DATA.FILE which we are using on a RAM drive.

as you can see below in the show-item-locks you can see blank keys.

31 9358 ../bnk.data/st/
FBNK_CUSTOMER_CHARGE
5383596
0x57a7c41c,W ---
31 9358 ../bnk.data/st/
FBNK_CUSTOMER_ROLE
0x02300000,W ---
31 9358 ../bnk.data/st/
FBNK_RELATION_CUSTOMER
5383596
0x57a7c41c,W ---
31 9358 ../bnk.data/st/
FBNK_RELATION_CUSTOMER
11561555
0x5e90b3b0,W ---
31 9358 ../bnk.data/st/
FBNK_CUSTOMER#NAU
5383596
0x57a7c41c,W


33 9367 ../bnk.data/of/
F_OFS_REQUEST_DETAIL_06
MBDM112090115838227.05
0x31fc7d36,W ---
33 9367 ../bnk.data/eb/
F_DM_SERVICE_DATA_FILE
CCS.DM.CUSTOMER.MEMA-20110810037483638476
0x59b0e064,W ---
33 9367 ../bnk.data/eb/
F_JOB_LIST_3
1139
0x6883ccaf,W ---
34 9371 ../bnk.data/eb/
F_LANGUAGE
0x0000001c,R ---
34 9371 ../bnk.data/st/
FBNK_CUSTOMER#NAU
4872036
0x04b9c9ed,W ---
34 9371 ../bnk.data/of/
F_OFS_REQUEST_DETAIL_08
MBDM112090521738237.01
0x5c4e0da1,W ---
34 9371 ../bnk.data/eb/
F_DM_SERVICE_DATA_FILE
CCS.DM.CUSTOMER.MEMA-20110810037483537290
0x0f29a5e6,W ---
34 9371 ../bnk.data/eb/
F_JOB_LIST_3
250
0x7b754110,W ---


AA

yep , onlive it will be a one off but now its run once every two
weeks.
> > jsh aatkinso ~ -->- Hide quoted text -
>
> - Show quoted text -

VK

unread,
Aug 11, 2011, 9:58:33 AM8/11/11
to jBASE
Hi,

F_JOB_LIST looks ok... 2 files with no IDs are F_LANGUAGE for one
session and FBNK_CUSTOMER_ROLE for another (though it looks not being
DM-related - maybe the problem is here? both use CUSTOMER file)...

Might be that some core code issued F.READU without supplying ID (or
supplying a COMMON variable that contains an empty string). Put that
query to Temenos helpdesk.

What's the task is BTW? Update of which fields in CUSTOMER? Might be
an easier solution to that...

Another note - non-secure is really not secure :(( Consider using
secure mode (and pay something in performance) or J4.

Yet another note... is DM something called "migration tool"?

And - your OFS.REQUEST.DETAIL is distributed? Just curious why.
Accumulating over 2Gb a day?

VK

adrian

unread,
Aug 13, 2011, 9:46:13 PM8/13/11
to jBASE
With Temenos we have found there is a problem at the Jbase level as we
have patched the F.READU program and there were no NULL records being
delivered here.
The null problem is seen more when there are a lots of locks happening
for a application, and AA and AZ have this so it is easier to spot.

The F.OFS.REQUEST.DETAIL on some of out loads takes this file to 15gb
and bigger.

Regards and Thanks
AA
> > > - Show quoted text -- Hide quoted text -

VK

unread,
Aug 15, 2011, 10:34:52 AM8/15/11
to jBASE
Hi,

> have patched the F.READU program and there were no NULL records being
> delivered here.

What about READU? A lot of routines in core T24 still use it.

VK

VK

unread,
Aug 15, 2011, 10:56:28 AM8/15/11
to jBASE
... and don't forget MATREADU (though used in less number of cases)...

BTW, it was always interesting to me - what's the idea of having lock
with empty record key? Does it have any sense from DBMS point of view?

VK

On Aug 14, 3:46 am, adrian <ar_atkin...@yahoo.com> wrote:

Jim Idle

unread,
Aug 15, 2011, 2:27:58 PM8/15/11
to jb...@googlegroups.com
Are you sure you are not just seeing group/binary locks?

Jim

> --
> Please read the posting guidelines at:
> http://groups.google.com/group/jBASE/web/Posting%20Guidelines
>
> IMPORTANT: Type T24: at the start of the subject line for questions
> specific to Globus/T24
>
> To post, send email to jB...@googlegroups.com To unsubscribe, send
> email to jBASE-un...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/jBASE?hl=en

Jim Idle

unread,
Aug 15, 2011, 2:29:48 PM8/15/11
to jb...@googlegroups.com
An empty record key is just a key without any characters. It is somewhat
pointless, but must be supported as people inventing their own 'security'
used it a lot in the old days of Pick.

Jim

> -----Original Message-----
> From: jb...@googlegroups.com [mailto:jb...@googlegroups.com] On Behalf
> Of VK
> Sent: Monday, August 15, 2011 7:56 AM
> To: jBASE
> Subject: Re: jbase problems show-items-locks
>

> dm5/bnk.run/GR0800004lib:/app/t24/dm5/bnk.run/t24lib:/app/t24/
> > > > > > dm5/
> > > > > >
> bnk.run/GR0800005lib:/app/t24/dm5/bnk.run/GR0800006lib:/app/t2
> > > > > > 4/dm5/
> > > > > >
> bnk.run/radlib:/app/t24/dm5/bnk.run/cardlib:/app/t24/dm5/bnk.r
> > > > > > un/
> > > > > >

> usrdlib:/app/t24/dm5/bnk.run/GR0800012lib:/app/t24/dm5/bnk.run/atmlib'
> > > > > > jBASE Compiler Run-time     : '/app/tafc/r10SP8/R10/config/
> > > > > > system.properties'
> > > > > > Program dir (JBCDEV_BIN)    : '/app/t24/dm5/bnk.run/ccsbin'
> > > > > > Subroutine dir (JBCDEV_LIB) : '/app/t24/dm5/bnk.run/ccslib'
> > > > > > Max open files              : 8192 jsh aatkinso ~ -->- Hide
> > > > > > quoted text -
> >
> > > > > - Show quoted text -- Hide quoted text -
> >
> > > - Show quoted text -
>

Pawel (privately)

unread,
Aug 15, 2011, 3:56:16 PM8/15/11
to jb...@googlegroups.com
Hi,

I hope that I will finally see an (environment variable?) option in jBASE, which will cause jBASE to abort when empty id is used in database operation. All cases with an empty id I have met so far were caused by software bugs. Moreover they were relatively hard to track - usually it was complex flow to recreate problem. I do not see any reason in keeping 'empty id functionality' available for T24.
It would be nice if jBASE team could go forward and throw away some unnecessary, 'old days' things :)

Another issue that can be caused by buggy software is to put field/value/subvalue markers. Some jBASE commands will be later confused. jBASE gurus: throw these possibilities away please!

Kind regards
Pawel

Dnia 15-08-2011 o godz. 20:29 Jim Idle napisaďż˝(a):


> An empty record key is just a key without any characters. It is somewhat
> pointless, but must be supported as people inventing their own 'security'
> used it a lot in the old days of Pick.
>
> Jim
>
>
>
> > -----Original Message-----
> > From: jb...@googlegroups.com [mailto:jb...@googlegroups.com] On Behalf
> > Of VK
> > Sent: Monday, August 15, 2011 7:56 AM
> > To: jBASE
> > Subject: Re: jbase problems show-items-locks
> >
> > ... and don't forget MATREADU (though used in less number of cases)...
> >
> > BTW, it was always interesting to me - what's the idea of having lock
> > with empty record key? Does it have any sense from DBMS point of view?
> >
> > VK
> >

> > > > Another note - non-secure is really not secure :(( �Consider using


> > > > secure mode (and pay something in performance) or J4.
> > >
> > > > Yet another note... is DM something called "migration tool"?
> > >
> > > > And - your OFS.REQUEST.DETAIL is distributed? Just curious why.
> > > > Accumulating over 2Gb a day?
> > >
> > > > VK
> > >
> > > > On Aug 10, 7:45�pm, adrian <ar_atkin...@yahoo.com> wrote:
> > >
> > > > > We are using 10 threads at the moments
> > >
> > > > > we are using JR type out of the box from Temenos. (non-secure)
> > >
> > > > > we are using DM.SERVICE.DATA.FILE which we are using on a RAM
> > drive.
> > >
> > > > > as you can see below in the show-item-locks you can see blank
> > keys.
> > >

> > > > > ďż˝ ďż˝ ďż˝ 31 ďż˝ ďż˝ ďż˝ 9358 ../bnk.data/st/ FBNK_CUSTOMER_CHARGE
> > > > > 5383596
> > > > > 0x57a7c41c,W ďż˝ ďż˝ ďż˝ ďż˝---
> > > > > ďż˝ ďż˝ ďż˝ 31 ďż˝ ďż˝ ďż˝ 9358 ../bnk.data/st/ FBNK_CUSTOMER_ROLE
> > > > > 0x02300000,W ďż˝ ďż˝ ďż˝ ďż˝---
> > > > > ďż˝ ďż˝ ďż˝ 31 ďż˝ ďż˝ ďż˝ 9358 ../bnk.data/st/ FBNK_RELATION_CUSTOMER
> > > > > 5383596
> > > > > 0x57a7c41c,W ďż˝ ďż˝ ďż˝ ďż˝---
> > > > > ďż˝ ďż˝ ďż˝ 31 ďż˝ ďż˝ ďż˝ 9358 ../bnk.data/st/ FBNK_RELATION_CUSTOMER
> > > > > 11561555
> > > > > 0x5e90b3b0,W ďż˝ ďż˝ ďż˝ ďż˝---
> > > > > ďż˝ ďż˝ ďż˝ 31 ďż˝ ďż˝ ďż˝ 9358 ../bnk.data/st/ FBNK_CUSTOMER#NAU
> > > > > 5383596
> > > > > 0x57a7c41c,W
> > >
> > > > > ďż˝ ďż˝ ďż˝ 33 ďż˝ ďż˝ ďż˝ 9367 ../bnk.data/of/
> > > > > F_OFS_REQUEST_DETAIL_06
> > > > > MBDM112090115838227.05
> > > > > 0x31fc7d36,W ďż˝ ďż˝ ďż˝ ďż˝---
> > > > > ďż˝ ďż˝ ďż˝ 33 ďż˝ ďż˝ ďż˝ 9367 ../bnk.data/eb/ F_DM_SERVICE_DATA_FILE
> > > > > CCS.DM.CUSTOMER.MEMA-20110810037483638476
> > > > > 0x59b0e064,W ďż˝ ďż˝ ďż˝ ďż˝---
> > > > > ďż˝ ďż˝ ďż˝ 33 ďż˝ ďż˝ ďż˝ 9367 ../bnk.data/eb/
> > > > > F_JOB_LIST_3
> > > > > 1139
> > > > > 0x6883ccaf,W ďż˝ ďż˝ ďż˝ ďż˝---
> > > > > ďż˝ ďż˝ ďż˝ 34 ďż˝ ďż˝ ďż˝ 9371 ../bnk.data/eb/ F_LANGUAGE
> > 0x0000001c,R
> > > > > ---
> > > > > ďż˝ ďż˝ ďż˝ 34 ďż˝ ďż˝ ďż˝ 9371 ../bnk.data/st/ FBNK_CUSTOMER#NAU
> > > > > 4872036
> > > > > 0x04b9c9ed,W ďż˝ ďż˝ ďż˝ ďż˝---
> > > > > ďż˝ ďż˝ ďż˝ 34 ďż˝ ďż˝ ďż˝ 9371 ../bnk.data/of/
> > > > > F_OFS_REQUEST_DETAIL_08
> > > > > MBDM112090521738237.01
> > > > > 0x5c4e0da1,W ďż˝ ďż˝ ďż˝ ďż˝---
> > > > > ďż˝ ďż˝ ďż˝ 34 ďż˝ ďż˝ ďż˝ 9371 ../bnk.data/eb/ F_DM_SERVICE_DATA_FILE
> > > > > CCS.DM.CUSTOMER.MEMA-20110810037483537290
> > > > > 0x0f29a5e6,W ďż˝ ďż˝ ďż˝ ďż˝---
> > > > > ďż˝ ďż˝ ďż˝ 34 ďż˝ ďż˝ ďż˝ 9371 ../bnk.data/eb/
> > > > > F_JOB_LIST_3
> > > > > 250
> > > > > 0x7b754110,W ďż˝ ďż˝ ďż˝ ďż˝---

> > > > > > > System ďż˝ ďż˝ ďż˝ ďż˝ ďż˝ ďż˝ ďż˝ ďż˝ ďż˝ ďż˝ ďż˝: HP-UX vic-samt B.11.31.U ia64
> > > > > > > UNIX User ďż˝ ďż˝ ďż˝ ďż˝ ďż˝ ďż˝ ďż˝ ďż˝ ďż˝ : aatkinso (uid 187, euid 187)
> > Tty
> > > > > > > name ďż˝ ďż˝ ďż˝ ďż˝ ďż˝ ďż˝ ďż˝ ďż˝ ďż˝ ďż˝: /dev/pts/0
> > Time
> > > > > > > : Fri Aug �5 20:31:32 2011
> > >
> > > > > > > Environment
> > > > > > > ===========
> > >
> > > > > > > JBCPORTNO ďż˝ ďż˝ ďż˝ ďż˝ ďż˝ ďż˝ ďż˝ ďż˝ ďż˝ : Not Set
> > TAFC_HOME
> > > > > > > : '/app/tafc/r10SP8/R10'
> > > > > > > JBCGLOBALDIR ďż˝ ďż˝ ďż˝ ďż˝ ďż˝ ďż˝ ďż˝ ďż˝: '/app/tafc/r10SP8/R10'


> > > > > > > WARNING: JBCDATADIR is not set, Default
> > '/app/tafc/r10SP8/R10/
> > > > > > > jbase_data'
> > > > > > > WARNING: JBCDATADIR is subdirectory of JBCGLOBALDIR
> > HOME
> > > > > > > : '/app/t24/dm5/bnk.run'

> > > > > > > JEDIFILEPATH ďż˝ ďż˝ ďż˝ ďż˝ ďż˝ ďż˝ ďż˝ ďż˝: '/app/t24/dm5/bnk.run'
> > > > > > > JEDIFILENAME_MD ďż˝ ďż˝ ďż˝ ďż˝ ďż˝ ďż˝ : '/app/t24/dm5/bnk.run/VOC'
> > > > > > > JEDIFILENAME_SYSTEM ďż˝ ďż˝ ďż˝ ďż˝ : '/app/t24/dm5/bnk.run/SYSTEM'
> > > > > > > RELEASE Information ďż˝ ďż˝ ďż˝ ďż˝ : Major 10.0 , Minor 0.8 ,


> > Patch
> > > > > > > (Change
> > > > > > > 92702)
> > > > > > > Spooler dir (JBCSPOOLERDIR) : '/usr/spool/jspooler'

> > > > > > > JBCEMULATE ďż˝ ďż˝ ďż˝ ďż˝ ďż˝ ďż˝ ďż˝ ďż˝ ďż˝: 'prime'


> > > > > > > Object path (JBCOBJECTLIST) :
> > > > > > > '/app/t24/dm5/bnk.run/ccslib:/app/t24/
> > > > > > >
> > dm5/bnk.run/GR0800004lib:/app/t24/dm5/bnk.run/t24lib:/app/t24/
> > > > > > > dm5/
> > > > > > >
> > bnk.run/GR0800005lib:/app/t24/dm5/bnk.run/GR0800006lib:/app/t2
> > > > > > > 4/dm5/
> > > > > > >
> > bnk.run/radlib:/app/t24/dm5/bnk.run/cardlib:/app/t24/dm5/bnk.r
> > > > > > > un/
> > > > > > >
> > usrdlib:/app/t24/dm5/bnk.run/GR0800012lib:/app/t24/dm5/bnk.run/atmlib'

> > > > > > > jBASE Compiler Run-time ďż˝ ďż˝ : '/app/tafc/r10SP8/R10/config/
> > > > > > > system.properties'
> > > > > > > Program dir (JBCDEV_BIN) ďż˝ ďż˝: '/app/t24/dm5/bnk.run/ccsbin'
> > > > > > > Subroutine dir (JBCDEV_LIB) : '/app/t24/dm5/bnk.run/ccslib'
> > > > > > > Max open files ďż˝ ďż˝ ďż˝ ďż˝ ďż˝ ďż˝ ďż˝: 8192 jsh aatkinso ~ -->- Hide
> > > > > > > quoted text -
> >


Jim Idle

unread,
Aug 15, 2011, 4:31:12 PM8/15/11
to jb...@googlegroups.com
Nice idea, but I seem to remember that in fact I/we had to put this back in
specifically for T24. All the things you mention, and anything that you
could mention are unfortunately relied upon implicitly by almost all
software. If I had a dollar for all the Universe bugs we had to recreate so
that we could port things, then there would be a run on dollar bills.

The system is what it is, and there is no changing it. This is not the jBASE
guys saying this, this is all the users.


Jim

> -----Original Message-----
> From: jb...@googlegroups.com [mailto:jb...@googlegroups.com] On Behalf
> Of Pawel (privately)
> Sent: Monday, August 15, 2011 12:56 PM
> To: jb...@googlegroups.com
> Subject: RE: RE: jbase problems show-items-locks
>
> Hi,
>
> I hope that I will finally see an (environment variable?) option in
> jBASE, which will cause jBASE to abort when empty id is used in
> database operation. All cases with an empty id I have met so far were
> caused by software bugs. Moreover they were relatively hard to track -
> usually it was complex flow to recreate problem. I do not see any
> reason in keeping 'empty id functionality' available for T24.
> It would be nice if jBASE team could go forward and throw away some
> unnecessary, 'old days' things :)
>
> Another issue that can be caused by buggy software is to put
> field/value/subvalue markers. Some jBASE commands will be later
> confused. jBASE gurus: throw these possibilities away please!
>
> Kind regards
> Pawel
>

> > > > > Another note - non-secure is really not secure :(( Consider


> > > > > using secure mode (and pay something in performance) or J4.
> > > >
> > > > > Yet another note... is DM something called "migration tool"?
> > > >
> > > > > And - your OFS.REQUEST.DETAIL is distributed? Just curious why.
> > > > > Accumulating over 2Gb a day?
> > > >
> > > > > VK
> > > >
> > > > > On Aug 10, 7:45 pm, adrian <ar_atkin...@yahoo.com> wrote:
> > > >
> > > > > > We are using 10 threads at the moments
> > > >
> > > > > > we are using JR type out of the box from Temenos. (non-
> secure)
> > > >
> > > > > > we are using DM.SERVICE.DATA.FILE which we are using on a RAM
> > > drive.
> > > >
> > > > > > as you can see below in the show-item-locks you can see blank
> > > keys.
> > > >

> > > > > > 31 9358 ../bnk.data/st/ FBNK_CUSTOMER_CHARGE
> > > > > > 5383596


> > > > > > 0x57a7c41c,W ---
> > > > > > 31 9358 ../bnk.data/st/ FBNK_CUSTOMER_ROLE
> > > > > > 0x02300000,W ---

> > > > > > 31 9358 ../bnk.data/st/ FBNK_RELATION_CUSTOMER
> > > > > > 5383596
> > > > > > 0x57a7c41c,W ---
> > > > > > 31 9358 ../bnk.data/st/ FBNK_RELATION_CUSTOMER
> > > > > > 11561555
> > > > > > 0x5e90b3b0,W ---
> > > > > > 31 9358 ../bnk.data/st/ FBNK_CUSTOMER#NAU
> > > > > > 5383596
> > > > > > 0x57a7c41c,W
> > > >
> > > > > > 33 9367 ../bnk.data/of/
> > > > > > F_OFS_REQUEST_DETAIL_06
> > > > > > MBDM112090115838227.05
> > > > > > 0x31fc7d36,W ---
> > > > > > 33 9367 ../bnk.data/eb/ F_DM_SERVICE_DATA_FILE
> > > > > > CCS.DM.CUSTOMER.MEMA-20110810037483638476
> > > > > > 0x59b0e064,W ---
> > > > > > 33 9367 ../bnk.data/eb/
> > > > > > F_JOB_LIST_3
> > > > > > 1139
> > > > > > 0x6883ccaf,W ---
> > > > > > 34 9371 ../bnk.data/eb/ F_LANGUAGE
> > > 0x0000001c,R
> > > > > > ---
> > > > > > 34 9371 ../bnk.data/st/ FBNK_CUSTOMER#NAU
> > > > > > 4872036
> > > > > > 0x04b9c9ed,W ---
> > > > > > 34 9371 ../bnk.data/of/
> > > > > > F_OFS_REQUEST_DETAIL_08
> > > > > > MBDM112090521738237.01
> > > > > > 0x5c4e0da1,W ---
> > > > > > 34 9371 ../bnk.data/eb/ F_DM_SERVICE_DATA_FILE
> > > > > > CCS.DM.CUSTOMER.MEMA-20110810037483537290
> > > > > > 0x0f29a5e6,W ---
> > > > > > 34 9371 ../bnk.data/eb/
> > > > > > F_JOB_LIST_3
> > > > > > 250
> > > > > > 0x7b754110,W ---

> > > > > > > > System : HP-UX vic-samt B.11.31.U
> > > > > > > > ia64 UNIX User : aatkinso (uid 187,
> euid
> > > > > > > > 187)
> > > Tty
> > > > > > > > name : /dev/pts/0
> > > Time
> > > > > > > > : Fri Aug 5 20:31:32 2011
> > > >
> > > > > > > > Environment
> > > > > > > > ===========
> > > >
> > > > > > > > JBCPORTNO : Not Set
> > > TAFC_HOME
> > > > > > > > : '/app/tafc/r10SP8/R10'
> > > > > > > > JBCGLOBALDIR : '/app/tafc/r10SP8/R10'


> > > > > > > > WARNING: JBCDATADIR is not set, Default
> > > '/app/tafc/r10SP8/R10/
> > > > > > > > jbase_data'
> > > > > > > > WARNING: JBCDATADIR is subdirectory of JBCGLOBALDIR
> > > HOME
> > > > > > > > : '/app/t24/dm5/bnk.run'

> > > > > > > > JEDIFILEPATH : '/app/t24/dm5/bnk.run'
> > > > > > > > JEDIFILENAME_MD : '/app/t24/dm5/bnk.run/VOC'
> > > > > > > > JEDIFILENAME_SYSTEM :
> '/app/t24/dm5/bnk.run/SYSTEM'

> > > > > > > > RELEASE Information : Major 10.0 , Minor 0.8 ,


> > > Patch
> > > > > > > > (Change
> > > > > > > > 92702)
> > > > > > > > Spooler dir (JBCSPOOLERDIR) : '/usr/spool/jspooler'

> > > > > > > > JBCEMULATE : 'prime'


> > > > > > > > Object path (JBCOBJECTLIST) :
> > > > > > > > '/app/t24/dm5/bnk.run/ccslib:/app/t24/
> > > > > > > >
> > > dm5/bnk.run/GR0800004lib:/app/t24/dm5/bnk.run/t24lib:/app/t24/
> > > > > > > > dm5/
> > > > > > > >
> > > bnk.run/GR0800005lib:/app/t24/dm5/bnk.run/GR0800006lib:/app/t2
> > > > > > > > 4/dm5/
> > > > > > > >
> > > bnk.run/radlib:/app/t24/dm5/bnk.run/cardlib:/app/t24/dm5/bnk.r
> > > > > > > > un/
> > > > > > > >
> > >
> usrdlib:/app/t24/dm5/bnk.run/GR0800012lib:/app/t24/dm5/bnk.run/atmlib'

> > > > > > > > jBASE Compiler Run-time :
> > > > > > > > '/app/tafc/r10SP8/R10/config/ system.properties'
> > > > > > > > Program dir (JBCDEV_BIN) :


> '/app/t24/dm5/bnk.run/ccsbin'
> > > > > > > > Subroutine dir (JBCDEV_LIB) :
> '/app/t24/dm5/bnk.run/ccslib'

> > > > > > > > Max open files : 8192 jsh aatkinso ~ -->-
> > > > > > > > Hide quoted text -
> > >
>
>

Pawel (privately)

unread,
Aug 15, 2011, 4:47:07 PM8/15/11
to jb...@googlegroups.com
Hi,

> > > With Temenos we have found there is a problem at the Jbase level as
> > we
> > > have patched the F.READU program and there were no NULL records being
> > > delivered here.

Could you please elaborate kindly? Do you know what was the reason? What was fixed? Was it problem strictly related to HP-UX?

Jim suggested that these could be group/binary locks and I would agree with him.
I think that binary locks are listed like that under SHOW-ITEM-LOCKS (at least on jBASE 4.1).

You have also said that:


> > > > > > > We are not using the jbase locking but the UNIX locking (set
> > > > > > > at
> > > > > > > 24,000) on a HP box as the jDLS was causing problems. The
> > unix
> > > > > > > locking has been running fine for over a month now.

Our LIVE system is P5, AIX, jBASE 4.1 and we are also sucessfully running UNIX locking. However we have observed performance and scalability problems on P7. There are 2 facts about P7, AIX 6.1 and jBASE 4.1:
a) jRLA (pthreads) locks are bit faster than UNIX locks on P7/AIX6.1/jBASE 4.1 - proven with jBASE test programs
b) group locks in 4.1 are always taken as UNIX locks disregarding wheter you run jRLA or not. We have got information that this changed (was improved) in jBASE 5+/TAFC. Can jBASE 5+/TAFC use pthreads for group/binary locks?

In our case high 'lock collision' ratio on group locks is killing performance. JOB.LIST.x files suffer from multiple, concurrent 'SELECTs SAMPLE' (~ full scan) and multiple, concurrent READUs/WRITEs. There should be an option to jBASE added - to traverse file with no locking on groups ('dirty traversal'). It seems that this new option should be used on JOB.LIST.x files and would improve performance a lot. Each process (agent) could do 'dirty READNEXT traversal', try to lock record id and move
forward if locked / not existing.

Can somebody answer me below question? Scenario:
1. Assume you have file with 3 groups - 8192 bytes each group (2 blocks in jBASE 4.1)
2. Assume there are 35 records in total - 10, 10, 15
3. Assume there are 15 concurrent processes doing: SELECT, READNEXT (up to 1000 iterations), READU followed by READNEXT (when record locked) and SLEEP 1 or doing processing (if lock sucessfully aquired)
4. Assume response time is no limitation (say file put to RAM disk, striped SSD with multiple physical disks, OS/RAID caching enabled, etc.)
5. Assume transaction is simple / lightweight and is quickly commited.
6. Asumme all processes start exactly in the same moment.

Can (theoretically) other processes be starving? (due to greedy process #1 which start, will constantly lock group #1 preventing other processes to move forward - it will be obtaing id, processing transaction and quickly commiting)

Of course in realistic world no process will be starving, but may be significantly delayed? Am I right?

By the way: we observe very higher System:User ratio (eg. System=90, User=10). Simplifying System means - 'I am not doing a (user) job for you my friend'.

Kind regards
Pawel


Dnia 15-08-2011 o godz. 20:27 Jim Idle napisaďż˝(a):


> Are you sure you are not just seeing group/binary locks?
>
> Jim
>
> > -----Original Message-----
> > From: jb...@googlegroups.com [mailto:jb...@googlegroups.com] On Behalf
> > Of VK
> > Sent: Monday, August 15, 2011 7:35 AM
> > To: jBASE
> > Subject: Re: jbase problems show-items-locks
> >
> > Hi,
> >
> > > have patched the F.READU program and there were no NULL records being
> > > delivered here.
> >
> > What about READU? A lot of routines in core T24 still use it.
> >
> > VK
> >

> > > > Another note - non-secure is really not secure :(( �Consider using


> > > > secure mode (and pay something in performance) or J4.
> > >
> > > > Yet another note... is DM something called "migration tool"?
> > >
> > > > And - your OFS.REQUEST.DETAIL is distributed? Just curious why.
> > > > Accumulating over 2Gb a day?
> > >
> > > > VK
> > >
> > > > On Aug 10, 7:45�pm, adrian <ar_atkin...@yahoo.com> wrote:
> > >
> > > > > We are using 10 threads at the moments
> > >
> > > > > we are using JR type out of the box from Temenos. (non-secure)
> > >
> > > > > we are using DM.SERVICE.DATA.FILE which we are using on a RAM
> > drive.
> > >
> > > > > as you can see below in the show-item-locks you can see blank
> > keys.
> > >

> > > > > ďż˝ ďż˝ ďż˝ 31 ďż˝ ďż˝ ďż˝ 9358 ../bnk.data/st/ FBNK_CUSTOMER_CHARGE
> > > > > 5383596


> > > > > 0x57a7c41c,W ďż˝ ďż˝ ďż˝ ďż˝---
> > > > > ďż˝ ďż˝ ďż˝ 31 ďż˝ ďż˝ ďż˝ 9358 ../bnk.data/st/ FBNK_CUSTOMER_ROLE
> > > > > 0x02300000,W ďż˝ ďż˝ ďż˝ ďż˝---

> > > > > ďż˝ ďż˝ ďż˝ 31 ďż˝ ďż˝ ďż˝ 9358 ../bnk.data/st/ FBNK_RELATION_CUSTOMER
> > > > > 5383596


> > > > > 0x57a7c41c,W ďż˝ ďż˝ ďż˝ ďż˝---

> > > > > ďż˝ ďż˝ ďż˝ 31 ďż˝ ďż˝ ďż˝ 9358 ../bnk.data/st/ FBNK_RELATION_CUSTOMER
> > > > > 11561555
> > > > > 0x5e90b3b0,W ďż˝ ďż˝ ďż˝ ďż˝---
> > > > > ďż˝ ďż˝ ďż˝ 31 ďż˝ ďż˝ ďż˝ 9358 ../bnk.data/st/ FBNK_CUSTOMER#NAU
> > > > > 5383596
> > > > > 0x57a7c41c,W
> > >
> > > > > ďż˝ ďż˝ ďż˝ 33 ďż˝ ďż˝ ďż˝ 9367 ../bnk.data/of/
> > > > > F_OFS_REQUEST_DETAIL_06
> > > > > MBDM112090115838227.05


> > > > > 0x31fc7d36,W ďż˝ ďż˝ ďż˝ ďż˝---

> > > > > ďż˝ ďż˝ ďż˝ 33 ďż˝ ďż˝ ďż˝ 9367 ../bnk.data/eb/ F_DM_SERVICE_DATA_FILE
> > > > > CCS.DM.CUSTOMER.MEMA-20110810037483638476
> > > > > 0x59b0e064,W ďż˝ ďż˝ ďż˝ ďż˝---
> > > > > ďż˝ ďż˝ ďż˝ 33 ďż˝ ďż˝ ďż˝ 9367 ../bnk.data/eb/
> > > > > F_JOB_LIST_3
> > > > > 1139


> > > > > 0x6883ccaf,W ďż˝ ďż˝ ďż˝ ďż˝---

> > > > > ďż˝ ďż˝ ďż˝ 34 ďż˝ ďż˝ ďż˝ 9371 ../bnk.data/eb/ F_LANGUAGE
> > 0x0000001c,R
> > > > > ---
> > > > > ďż˝ ďż˝ ďż˝ 34 ďż˝ ďż˝ ďż˝ 9371 ../bnk.data/st/ FBNK_CUSTOMER#NAU
> > > > > 4872036
> > > > > 0x04b9c9ed,W ďż˝ ďż˝ ďż˝ ďż˝---
> > > > > ďż˝ ďż˝ ďż˝ 34 ďż˝ ďż˝ ďż˝ 9371 ../bnk.data/of/
> > > > > F_OFS_REQUEST_DETAIL_08
> > > > > MBDM112090521738237.01
> > > > > 0x5c4e0da1,W ďż˝ ďż˝ ďż˝ ďż˝---
> > > > > ďż˝ ďż˝ ďż˝ 34 ďż˝ ďż˝ ďż˝ 9371 ../bnk.data/eb/ F_DM_SERVICE_DATA_FILE
> > > > > CCS.DM.CUSTOMER.MEMA-20110810037483537290
> > > > > 0x0f29a5e6,W ďż˝ ďż˝ ďż˝ ďż˝---
> > > > > ďż˝ ďż˝ ďż˝ 34 ďż˝ ďż˝ ďż˝ 9371 ../bnk.data/eb/
> > > > > F_JOB_LIST_3
> > > > > 250
> > > > > 0x7b754110,W ďż˝ ďż˝ ďż˝ ďż˝---

> > > > > > > System ďż˝ ďż˝ ďż˝ ďż˝ ďż˝ ďż˝ ďż˝ ďż˝ ďż˝ ďż˝ ďż˝: HP-UX vic-samt B.11.31.U ia64
> > > > > > > UNIX User ďż˝ ďż˝ ďż˝ ďż˝ ďż˝ ďż˝ ďż˝ ďż˝ ďż˝ : aatkinso (uid 187, euid 187)
> > Tty


> > > > > > > name ďż˝ ďż˝ ďż˝ ďż˝ ďż˝ ďż˝ ďż˝ ďż˝ ďż˝ ďż˝: /dev/pts/0
> > Time

> > > > > > > : Fri Aug �5 20:31:32 2011
> > >
> > > > > > > Environment
> > > > > > > ===========
> > >
> > > > > > > JBCPORTNO ďż˝ ďż˝ ďż˝ ďż˝ ďż˝ ďż˝ ďż˝ ďż˝ ďż˝ : Not Set
> > TAFC_HOME
> > > > > > > : '/app/tafc/r10SP8/R10'
> > > > > > > JBCGLOBALDIR ďż˝ ďż˝ ďż˝ ďż˝ ďż˝ ďż˝ ďż˝ ďż˝: '/app/tafc/r10SP8/R10'


> > > > > > > WARNING: JBCDATADIR is not set, Default
> > '/app/tafc/r10SP8/R10/
> > > > > > > jbase_data'
> > > > > > > WARNING: JBCDATADIR is subdirectory of JBCGLOBALDIR
> > HOME
> > > > > > > : '/app/t24/dm5/bnk.run'

> > > > > > > JEDIFILEPATH ďż˝ ďż˝ ďż˝ ďż˝ ďż˝ ďż˝ ďż˝ ďż˝: '/app/t24/dm5/bnk.run'
> > > > > > > JEDIFILENAME_MD ďż˝ ďż˝ ďż˝ ďż˝ ďż˝ ďż˝ : '/app/t24/dm5/bnk.run/VOC'
> > > > > > > JEDIFILENAME_SYSTEM ďż˝ ďż˝ ďż˝ ďż˝ : '/app/t24/dm5/bnk.run/SYSTEM'

> > > > > > > RELEASE Information ďż˝ ďż˝ ďż˝ ďż˝ : Major 10.0 , Minor 0.8 ,


> > Patch
> > > > > > > (Change
> > > > > > > 92702)
> > > > > > > Spooler dir (JBCSPOOLERDIR) : '/usr/spool/jspooler'

> > > > > > > JBCEMULATE ďż˝ ďż˝ ďż˝ ďż˝ ďż˝ ďż˝ ďż˝ ďż˝ ďż˝: 'prime'


> > > > > > > Object path (JBCOBJECTLIST) :
> > > > > > > '/app/t24/dm5/bnk.run/ccslib:/app/t24/
> > > > > > >
> > dm5/bnk.run/GR0800004lib:/app/t24/dm5/bnk.run/t24lib:/app/t24/
> > > > > > > dm5/
> > > > > > >
> > bnk.run/GR0800005lib:/app/t24/dm5/bnk.run/GR0800006lib:/app/t2
> > > > > > > 4/dm5/
> > > > > > >
> > bnk.run/radlib:/app/t24/dm5/bnk.run/cardlib:/app/t24/dm5/bnk.r
> > > > > > > un/
> > > > > > >
> > usrdlib:/app/t24/dm5/bnk.run/GR0800012lib:/app/t24/dm5/bnk.run/atmlib'

> > > > > > > jBASE Compiler Run-time ďż˝ ďż˝ : '/app/tafc/r10SP8/R10/config/
> > > > > > > system.properties'
> > > > > > > Program dir (JBCDEV_BIN) ďż˝ ďż˝: '/app/t24/dm5/bnk.run/ccsbin'
> > > > > > > Subroutine dir (JBCDEV_LIB) : '/app/t24/dm5/bnk.run/ccslib'
> > > > > > > Max open files ďż˝ ďż˝ ďż˝ ďż˝ ďż˝ ďż˝ ďż˝: 8192 jsh aatkinso ~ -->- Hide
> > > > > > > quoted text -

Pawel (privately)

unread,
Aug 15, 2011, 5:06:34 PM8/15/11
to jb...@googlegroups.com
Hi,

I think I understand, but recreating (Universe) problems to make software portable seems to be a bit ridiculous. Not sure of the nature of problems that you recreated, but I take your point. However....

Is Universe still alive? (I am trying not to sound as ignorant!)
How many Universe customers are migrating to jBASE? How many already migrated?
Could not we finally forget about these UV compatibility issues?
<ignorant>UV seems to be dead to me :)</ignorant>
UV seems to be dead for many other customers.

Were efforts spent to be (fully) Universe compatible worth of it? Would not be better to give advices to guys who port their own software and to fix T24 to run correctly on jBASE? I bet that T24 R06 will not run on Universe :)
Migration is always a big project. I know that you were trying to make it easier.

I recall one clever Temenos guy whom we met this year at Temenos Community Forum in Lisbon. He showed us T24 compiled as Java native application (yes! :)). It looked impressive (not because it was Java). He shown us also jBASIC grammar. We have asked him - "could you share"? Answer was no: "it took over 2 years (or so) to create this bloody grammar". Explanation was simple - you can not imagine how some of the constructs execute and that some are valid. Temenos has spent a lot of time on
replicating jBASIC/compiler behaviour instead of telling some of the gus - please correct your nasty program, which does not work as intended (or miracurously works as intended).

Kind regards
Pawel

Dnia 15-08-2011 o godz. 22:31 Jim Idle napisaďż˝(a):

> > Dnia 15-08-2011 o godz. 20:29 Jim Idle napisaďż˝(a):

Ross Ferris

unread,
Aug 15, 2011, 7:19:44 PM8/15/11
to jb...@googlegroups.com
You don't need an environment variable for EVERYTHING ... why not just attach a trigger to the file, which would give you greater granularity than a system wide environment variable, and give yiou greater control over what does happen when an "event of interest"(aka trigger) takes place

Ross Ferris
Stamina Software
Visage > Better by Design!


-----Original Message-----
From: jb...@googlegroups.com [mailto:jb...@googlegroups.com] On Behalf Of Pawel (privately)
Sent: Tuesday, 16 August 2011 5:56 AM
To: jb...@googlegroups.com
Subject: RE: RE: jbase problems show-items-locks

Hi,

I hope that I will finally see an (environment variable?) option in jBASE, which will cause jBASE to abort when empty id is used in database operation. All cases with an empty id I have met so far were caused by software bugs. Moreover they were relatively hard to track - usually it was complex flow to recreate problem. I do not see any reason in keeping 'empty id functionality' available for T24.
It would be nice if jBASE team could go forward and throw away some unnecessary, 'old days' things :)

Another issue that can be caused by buggy software is to put field/value/subvalue markers. Some jBASE commands will be later confused. jBASE gurus: throw these possibilities away please!

Kind regards
Pawel

> > > > Another note - non-secure is really not secure :((  Consider
> > > > using secure mode (and pay something in performance) or J4.
> > >
> > > > Yet another note... is DM something called "migration tool"?
> > >
> > > > And - your OFS.REQUEST.DETAIL is distributed? Just curious why.
> > > > Accumulating over 2Gb a day?
> > >
> > > > VK
> > >
> > > > On Aug 10, 7:45 pm, adrian <ar_atkin...@yahoo.com> wrote:
> > >
> > > > > We are using 10 threads at the moments
> > >
> > > > > we are using JR type out of the box from Temenos. (non-secure)
> > >
> > > > > we are using DM.SERVICE.DATA.FILE which we are using on a RAM
> > drive.
> > >
> > > > > as you can see below in the show-item-locks you can see blank
> > keys.
> > >
> > > > >       31       9358 ../bnk.data/st/ FBNK_CUSTOMER_CHARGE
> > > > > 5383596
> > > > > 0x57a7c41c,W        ---
> > > > >       31       9358 ../bnk.data/st/ FBNK_CUSTOMER_ROLE
> > > > > 0x02300000,W        ---
> > > > >       31       9358 ../bnk.data/st/ FBNK_RELATION_CUSTOMER
> > > > > 5383596
> > > > > 0x57a7c41c,W        ---
> > > > >       31       9358 ../bnk.data/st/ FBNK_RELATION_CUSTOMER
> > > > > 11561555
> > > > > 0x5e90b3b0,W        ---
> > > > >       31       9358 ../bnk.data/st/ FBNK_CUSTOMER#NAU
> > > > > 5383596
> > > > > 0x57a7c41c,W
> > >
> > > > >       33       9367 ../bnk.data/of/
> > > > > F_OFS_REQUEST_DETAIL_06
> > > > > MBDM112090115838227.05
> > > > > 0x31fc7d36,W        ---
> > > > >       33       9367 ../bnk.data/eb/ F_DM_SERVICE_DATA_FILE
> > > > > CCS.DM.CUSTOMER.MEMA-20110810037483638476
> > > > > 0x59b0e064,W        ---
> > > > >       33       9367 ../bnk.data/eb/
> > > > > F_JOB_LIST_3
> > > > > 1139
> > > > > 0x6883ccaf,W        ---
> > > > >       34       9371 ../bnk.data/eb/ F_LANGUAGE
> > 0x0000001c,R
> > > > > ---
> > > > >       34       9371 ../bnk.data/st/ FBNK_CUSTOMER#NAU
> > > > > 4872036
> > > > > 0x04b9c9ed,W        ---
> > > > >       34       9371 ../bnk.data/of/
> > > > > F_OFS_REQUEST_DETAIL_08
> > > > > MBDM112090521738237.01
> > > > > 0x5c4e0da1,W        ---
> > > > >       34       9371 ../bnk.data/eb/ F_DM_SERVICE_DATA_FILE
> > > > > CCS.DM.CUSTOMER.MEMA-20110810037483537290
> > > > > 0x0f29a5e6,W        ---
> > > > >       34       9371 ../bnk.data/eb/
> > > > > F_JOB_LIST_3
> > > > > 250
> > > > > 0x7b754110,W        ---
> > > > > > > System                      : HP-UX vic-samt B.11.31.U
> > > > > > > ia64 UNIX User                   : aatkinso (uid 187, euid
> > > > > > > 187)
> > Tty
> > > > > > > name                    : /dev/pts/0
> > Time
> > > > > > > : Fri Aug  5 20:31:32 2011
> > >
> > > > > > > Environment
> > > > > > > ===========
> > >
> > > > > > > JBCPORTNO                   : Not Set
> > TAFC_HOME
> > > > > > > : '/app/tafc/r10SP8/R10'
> > > > > > > JBCGLOBALDIR                : '/app/tafc/r10SP8/R10'
> > > > > > > WARNING: JBCDATADIR is not set, Default
> > '/app/tafc/r10SP8/R10/
> > > > > > > jbase_data'
> > > > > > > WARNING: JBCDATADIR is subdirectory of JBCGLOBALDIR
> > HOME
> > > > > > > : '/app/t24/dm5/bnk.run'
> > > > > > > JEDIFILEPATH                : '/app/t24/dm5/bnk.run'
> > > > > > > JEDIFILENAME_MD             : '/app/t24/dm5/bnk.run/VOC'
> > > > > > > JEDIFILENAME_SYSTEM         : '/app/t24/dm5/bnk.run/SYSTEM'
> > > > > > > RELEASE Information         : Major 10.0 , Minor 0.8 ,
> > Patch
> > > > > > > (Change
> > > > > > > 92702)
> > > > > > > Spooler dir (JBCSPOOLERDIR) : '/usr/spool/jspooler'
> > > > > > > JBCEMULATE                  : 'prime'
> > > > > > > Object path (JBCOBJECTLIST) :
> > > > > > > '/app/t24/dm5/bnk.run/ccslib:/app/t24/
> > > > > > >
> > dm5/bnk.run/GR0800004lib:/app/t24/dm5/bnk.run/t24lib:/app/t24/
> > > > > > > dm5/
> > > > > > >
> > bnk.run/GR0800005lib:/app/t24/dm5/bnk.run/GR0800006lib:/app/t2
> > > > > > > 4/dm5/
> > > > > > >
> > bnk.run/radlib:/app/t24/dm5/bnk.run/cardlib:/app/t24/dm5/bnk.r
> > > > > > > un/
> > > > > > >
> > usrdlib:/app/t24/dm5/bnk.run/GR0800012lib:/app/t24/dm5/bnk.run/atmlib'
> > > > > > > jBASE Compiler Run-time     :
> > > > > > > '/app/tafc/r10SP8/R10/config/ system.properties'
> > > > > > > Program dir (JBCDEV_BIN)    : '/app/t24/dm5/bnk.run/ccsbin'
> > > > > > > Subroutine dir (JBCDEV_LIB) : '/app/t24/dm5/bnk.run/ccslib'
> > > > > > > Max open files              : 8192 jsh aatkinso ~ -->-
> > > > > > > Hide quoted text -
> >

adrian

unread,
Aug 15, 2011, 7:53:56 PM8/15/11
to jBASE

Jim,

Thanks for your reply, what is happening is that when these blank keys
locks show in show-item-locks the throughput of the system is hit
bady. Also noticed that the more locks a transaction makes then the
blank keys start appearing.

Is is then the problem that the JR files group lock?

Never seen blank keys before unless there is a bug in the code, but
this is happening for almost all T24 applications (high volume) loads,
with more than 4 file locks.

We are using mostly JR type files with some J4 files.

AA
> ...
>
> read more »- Hide quoted text -

Jim Idle

unread,
Aug 16, 2011, 11:19:27 AM8/16/11
to jb...@googlegroups.com
There is in fact an option to traverse files without taking locks if you can
'know' that there are no updates. It has to be supported by the target file
type (J4, etc).

http://jbase.markmail.org/search/?q=fastscan


Jim

> -----Original Message-----
> From: jb...@googlegroups.com [mailto:jb...@googlegroups.com] On Behalf

> Of Pawel (privately)
> Sent: Monday, August 15, 2011 1:47 PM
> To: jb...@googlegroups.com
> Subject: RE: RE: jbase problems show-items-locks
>

> > > > > Another note - non-secure is really not secure :(( Consider


> > > > > using secure mode (and pay something in performance) or J4.
> > > >
> > > > > Yet another note... is DM something called "migration tool"?
> > > >
> > > > > And - your OFS.REQUEST.DETAIL is distributed? Just curious why.
> > > > > Accumulating over 2Gb a day?
> > > >
> > > > > VK
> > > >
> > > > > On Aug 10, 7:45 pm, adrian <ar_atkin...@yahoo.com> wrote:
> > > >
> > > > > > We are using 10 threads at the moments
> > > >
> > > > > > we are using JR type out of the box from Temenos. (non-
> secure)
> > > >
> > > > > > we are using DM.SERVICE.DATA.FILE which we are using on a RAM
> > > drive.
> > > >
> > > > > > as you can see below in the show-item-locks you can see blank
> > > keys.
> > > >

> > > > > > 31 9358 ../bnk.data/st/ FBNK_CUSTOMER_CHARGE
> > > > > > 5383596


> > > > > > 0x57a7c41c,W ---
> > > > > > 31 9358 ../bnk.data/st/ FBNK_CUSTOMER_ROLE
> > > > > > 0x02300000,W ---

> > > > > > 31 9358 ../bnk.data/st/ FBNK_RELATION_CUSTOMER
> > > > > > 5383596
> > > > > > 0x57a7c41c,W ---
> > > > > > 31 9358 ../bnk.data/st/ FBNK_RELATION_CUSTOMER
> > > > > > 11561555
> > > > > > 0x5e90b3b0,W ---
> > > > > > 31 9358 ../bnk.data/st/ FBNK_CUSTOMER#NAU
> > > > > > 5383596
> > > > > > 0x57a7c41c,W
> > > >
> > > > > > 33 9367 ../bnk.data/of/
> > > > > > F_OFS_REQUEST_DETAIL_06
> > > > > > MBDM112090115838227.05
> > > > > > 0x31fc7d36,W ---
> > > > > > 33 9367 ../bnk.data/eb/ F_DM_SERVICE_DATA_FILE
> > > > > > CCS.DM.CUSTOMER.MEMA-20110810037483638476
> > > > > > 0x59b0e064,W ---
> > > > > > 33 9367 ../bnk.data/eb/
> > > > > > F_JOB_LIST_3
> > > > > > 1139
> > > > > > 0x6883ccaf,W ---
> > > > > > 34 9371 ../bnk.data/eb/ F_LANGUAGE
> > > 0x0000001c,R
> > > > > > ---
> > > > > > 34 9371 ../bnk.data/st/ FBNK_CUSTOMER#NAU
> > > > > > 4872036
> > > > > > 0x04b9c9ed,W ---
> > > > > > 34 9371 ../bnk.data/of/
> > > > > > F_OFS_REQUEST_DETAIL_08
> > > > > > MBDM112090521738237.01
> > > > > > 0x5c4e0da1,W ---
> > > > > > 34 9371 ../bnk.data/eb/ F_DM_SERVICE_DATA_FILE
> > > > > > CCS.DM.CUSTOMER.MEMA-20110810037483537290
> > > > > > 0x0f29a5e6,W ---
> > > > > > 34 9371 ../bnk.data/eb/
> > > > > > F_JOB_LIST_3
> > > > > > 250
> > > > > > 0x7b754110,W ---

> > > > > > > > System : HP-UX vic-samt B.11.31.U
> > > > > > > > ia64 UNIX User : aatkinso (uid 187,
> euid
> > > > > > > > 187)
> > > Tty
> > > > > > > > name : /dev/pts/0
> > > Time
> > > > > > > > : Fri Aug 5 20:31:32 2011
> > > >
> > > > > > > > Environment
> > > > > > > > ===========
> > > >
> > > > > > > > JBCPORTNO : Not Set
> > > TAFC_HOME
> > > > > > > > : '/app/tafc/r10SP8/R10'
> > > > > > > > JBCGLOBALDIR : '/app/tafc/r10SP8/R10'


> > > > > > > > WARNING: JBCDATADIR is not set, Default
> > > '/app/tafc/r10SP8/R10/
> > > > > > > > jbase_data'
> > > > > > > > WARNING: JBCDATADIR is subdirectory of JBCGLOBALDIR
> > > HOME
> > > > > > > > : '/app/t24/dm5/bnk.run'

> > > > > > > > JEDIFILEPATH : '/app/t24/dm5/bnk.run'
> > > > > > > > JEDIFILENAME_MD : '/app/t24/dm5/bnk.run/VOC'
> > > > > > > > JEDIFILENAME_SYSTEM :
> '/app/t24/dm5/bnk.run/SYSTEM'

> > > > > > > > RELEASE Information : Major 10.0 , Minor 0.8 ,


> > > Patch
> > > > > > > > (Change
> > > > > > > > 92702)
> > > > > > > > Spooler dir (JBCSPOOLERDIR) : '/usr/spool/jspooler'

> > > > > > > > JBCEMULATE : 'prime'


> > > > > > > > Object path (JBCOBJECTLIST) :
> > > > > > > > '/app/t24/dm5/bnk.run/ccslib:/app/t24/
> > > > > > > >
> > > dm5/bnk.run/GR0800004lib:/app/t24/dm5/bnk.run/t24lib:/app/t24/
> > > > > > > > dm5/
> > > > > > > >
> > > bnk.run/GR0800005lib:/app/t24/dm5/bnk.run/GR0800006lib:/app/t2
> > > > > > > > 4/dm5/
> > > > > > > >
> > > bnk.run/radlib:/app/t24/dm5/bnk.run/cardlib:/app/t24/dm5/bnk.r
> > > > > > > > un/
> > > > > > > >
> > >
> usrdlib:/app/t24/dm5/bnk.run/GR0800012lib:/app/t24/dm5/bnk.run/atmlib'

> > > > > > > > jBASE Compiler Run-time :
> > > > > > > > '/app/tafc/r10SP8/R10/config/ system.properties'
> > > > > > > > Program dir (JBCDEV_BIN) :


> '/app/t24/dm5/bnk.run/ccsbin'
> > > > > > > > Subroutine dir (JBCDEV_LIB) :
> '/app/t24/dm5/bnk.run/ccslib'

> > > > > > > > Max open files : 8192 jsh aatkinso ~ -->-
> > > > > > > > Hide quoted text -
>
>
>

Jim Idle

unread,
Aug 16, 2011, 11:33:21 AM8/16/11
to jb...@googlegroups.com
Hmmm - I have an ANTLR grammar for "mvbasic" it took about 2 days to produce
it. I suspect that I won't be using it commercially any more so I may share
it. It is much improved over the yacc one I wrote for the jBASE compiler.

To be honest, the idea that any large program could be inspected and
improved to get rid of the 'dross' doesn’t work. The programs are generally
way too complicated and involved. For this reason I doubt the Thierry will
get the Java version to work 100%, though I wish him all good luck with it.
But generally it is easier for the runtime guys to accommodate a program
than have the application programmers fix their code.

Most of the UniVerse compatibility stuff was done for T24 in fact.

Jim

> -----Original Message-----
> From: jb...@googlegroups.com [mailto:jb...@googlegroups.com] On Behalf
> Of Pawel (privately)
> Sent: Monday, August 15, 2011 2:07 PM
> To: jb...@googlegroups.com
> Subject: RE: RE: RE: jbase problems show-items-locks
>
> Hi,
>

Jim Idle

unread,
Aug 16, 2011, 11:38:05 AM8/16/11
to jb...@googlegroups.com
In the small amount of testing I have done, the JR files are not very good -
you might try going back to J4 files and using FASTSCAN (see earlier email).
I suspect that you are in fact seeing 'group' locks, which are taken on the
hash bucket to prevent the 'group' being changed while in use. FASTSCAN
allows you to avoid taking the lock, but you must require that there are no
updates to a file.

I think that at one point, we changed J4 files so that if you made them read
only, then they would not use binary/group locks. chmod a-r should tell you.
This could well have changed of course and probably is not implemented for
JR files.

Jim

Pawel (privately)

unread,
Aug 16, 2011, 3:48:17 PM8/16/11
to jb...@googlegroups.com
Hi,

We use triggers and they work like a charm :) Proposed approach with trigger could be used, but sounds like workaround of problem and would impact on performance negatively.
T24 has a lot of files - it would mean that we need to create a trigger for each one (we do not want to have blank ids in our database!). Solution would not laso work for tempfiles that can potentially be created by the system. Therefore it does not sound interesting and I would expect system solution.

I think that there is no better (and more efficient) place to add something like:
if (recordid[0] == '\0' && crash_on_empty_ids_enabled) { exit(-1); }
in right places of libjbase / (jEDI) driver(s) :)

Null ids are causing jBASE to be inconistent in a way that on non-MV database (Oracle, SQL Server, DB2) will not accept empty ids as primary keys. Such WRITEs with empty id will fail on Oracle if I remember well. RECORDID primary key on Oracle was mandatory, thus non nullable.

Ross: could you explain me why would you need to create records with blank ids? (or record ids with field/multi/value separators?)

New environment variable has been proposed, because jBASE configuration is built around environment variables. I would of course perfer some more sophisticated and documented :) configuration file.

Kind regards
Pawel

NB. I like jBASE db very much, but I expect it to be superfast and reliable :)

Dnia 16-08-2011 o godz. 1:19 Ross Ferris napisaďż˝(a):

> Dnia 15-08-2011 o godz. 20:29 Jim Idle napisaďż˝(a):

> > > > > Another note - non-secure is really not secure :((  Consider


> > > > > using secure mode (and pay something in performance) or J4.
> > > >
> > > > > Yet another note... is DM something called "migration tool"?
> > > >
> > > > > And - your OFS.REQUEST.DETAIL is distributed? Just curious why.
> > > > > Accumulating over 2Gb a day?
> > > >
> > > > > VK
> > > >
> > > > > On Aug 10, 7:45 pm, adrian <ar_atkin...@yahoo.com> wrote:
> > > >
> > > > > > We are using 10 threads at the moments
> > > >
> > > > > > we are using JR type out of the box from Temenos. (non-secure)
> > > >
> > > > > > we are using DM.SERVICE.DATA.FILE which we are using on a RAM
> > > drive.
> > > >
> > > > > > as you can see below in the show-item-locks you can see blank
> > > keys.
> > > >

> > > > > >       31       9358 ../bnk.data/st/ FBNK_CUSTOMER_CHARGE
> > > > > > 5383596
> > > > > > 0x57a7c41c,W        ---
> > > > > >       31       9358 ../bnk.data/st/ FBNK_CUSTOMER_ROLE
> > > > > > 0x02300000,W        ---
> > > > > >       31       9358 ../bnk.data/st/ FBNK_RELATION_CUSTOMER
> > > > > > 5383596
> > > > > > 0x57a7c41c,W        ---
> > > > > >       31       9358 ../bnk.data/st/ FBNK_RELATION_CUSTOMER
> > > > > > 11561555
> > > > > > 0x5e90b3b0,W        ---
> > > > > >       31       9358 ../bnk.data/st/ FBNK_CUSTOMER#NAU
> > > > > > 5383596
> > > > > > 0x57a7c41c,W
> > > >
> > > > > >       33       9367 ../bnk.data/of/
> > > > > > F_OFS_REQUEST_DETAIL_06
> > > > > > MBDM112090115838227.05
> > > > > > 0x31fc7d36,W        ---
> > > > > >       33       9367 ../bnk.data/eb/ F_DM_SERVICE_DATA_FILE
> > > > > > CCS.DM.CUSTOMER.MEMA-20110810037483638476
> > > > > > 0x59b0e064,W        ---
> > > > > >       33       9367 ../bnk.data/eb/
> > > > > > F_JOB_LIST_3
> > > > > > 1139
> > > > > > 0x6883ccaf,W        ---
> > > > > >       34       9371 ../bnk.data/eb/ F_LANGUAGE
> > > 0x0000001c,R
> > > > > > ---
> > > > > >       34       9371 ../bnk.data/st/ FBNK_CUSTOMER#NAU
> > > > > > 4872036
> > > > > > 0x04b9c9ed,W        ---
> > > > > >       34       9371 ../bnk.data/of/
> > > > > > F_OFS_REQUEST_DETAIL_08
> > > > > > MBDM112090521738237.01
> > > > > > 0x5c4e0da1,W        ---
> > > > > >       34       9371 ../bnk.data/eb/ F_DM_SERVICE_DATA_FILE
> > > > > > CCS.DM.CUSTOMER.MEMA-20110810037483537290
> > > > > > 0x0f29a5e6,W        ---
> > > > > >       34       9371 ../bnk.data/eb/
> > > > > > F_JOB_LIST_3
> > > > > > 250
> > > > > > 0x7b754110,W        ---

> > > > > > > > System                      : HP-UX vic-samt B.11.31.U
> > > > > > > > ia64 UNIX User                   : aatkinso (uid 187, euid
> > > > > > > > 187)
> > > Tty
> > > > > > > > name                    : /dev/pts/0
> > > Time
> > > > > > > > : Fri Aug  5 20:31:32 2011
> > > >
> > > > > > > > Environment
> > > > > > > > ===========
> > > >
> > > > > > > > JBCPORTNO                   : Not Set
> > > TAFC_HOME
> > > > > > > > : '/app/tafc/r10SP8/R10'
> > > > > > > > JBCGLOBALDIR                : '/app/tafc/r10SP8/R10'


> > > > > > > > WARNING: JBCDATADIR is not set, Default
> > > '/app/tafc/r10SP8/R10/
> > > > > > > > jbase_data'
> > > > > > > > WARNING: JBCDATADIR is subdirectory of JBCGLOBALDIR
> > > HOME
> > > > > > > > : '/app/t24/dm5/bnk.run'

> > > > > > > > JEDIFILEPATH                : '/app/t24/dm5/bnk.run'
> > > > > > > > JEDIFILENAME_MD             : '/app/t24/dm5/bnk.run/VOC'
> > > > > > > > JEDIFILENAME_SYSTEM         : '/app/t24/dm5/bnk.run/SYSTEM'
> > > > > > > > RELEASE Information         : Major 10.0 , Minor 0.8 ,


> > > Patch
> > > > > > > > (Change
> > > > > > > > 92702)
> > > > > > > > Spooler dir (JBCSPOOLERDIR) : '/usr/spool/jspooler'

> > > > > > > > JBCEMULATE                  : 'prime'


> > > > > > > > Object path (JBCOBJECTLIST) :
> > > > > > > > '/app/t24/dm5/bnk.run/ccslib:/app/t24/
> > > > > > > >
> > > dm5/bnk.run/GR0800004lib:/app/t24/dm5/bnk.run/t24lib:/app/t24/
> > > > > > > > dm5/
> > > > > > > >
> > > bnk.run/GR0800005lib:/app/t24/dm5/bnk.run/GR0800006lib:/app/t2
> > > > > > > > 4/dm5/
> > > > > > > >
> > > bnk.run/radlib:/app/t24/dm5/bnk.run/cardlib:/app/t24/dm5/bnk.r
> > > > > > > > un/
> > > > > > > >
> > > usrdlib:/app/t24/dm5/bnk.run/GR0800012lib:/app/t24/dm5/bnk.run/atmlib'

> > > > > > > > jBASE Compiler Run-time     :
> > > > > > > > '/app/tafc/r10SP8/R10/config/ system.properties'
> > > > > > > > Program dir (JBCDEV_BIN)    : '/app/t24/dm5/bnk.run/ccsbin'
> > > > > > > > Subroutine dir (JBCDEV_LIB) : '/app/t24/dm5/bnk.run/ccslib'
> > > > > > > > Max open files              : 8192 jsh aatkinso ~ -->-

David McGehee

unread,
Aug 16, 2011, 8:50:06 PM8/16/11
to jb...@googlegroups.com
Depending upon your file by file requirements, 1 trigger file in a pre-write position might serve for all. Of course, again depending upon what you want to do, it might be the same amount of work to a) handle the failed write (null key) as b) handling the null key BEFORE you write. If a general solution works for you a single trigger which transformed the null into a specific key via an algorithm works fine. We use triggers extensively and they are simple (keep it that way) and very effective.

-----Original Message-----
From: jb...@googlegroups.com [mailto:jb...@googlegroups.com] On Behalf Of Pawel (privately)
Sent: Tuesday, August 16, 2011 2:48 PM
To: jb...@googlegroups.com
Subject: RE: RE: RE: jbase problems show-items-locks

Hi,

We use triggers and they work like a charm :) Proposed approach with trigger could be used, but sounds like workaround of problem and would impact on performance negatively.
T24 has a lot of files - it would mean that we need to create a trigger for each one (we do not want to have blank ids in our database!). Solution would not laso work for tempfiles that can potentially be created by the system. Therefore it does not sound interesting and I would expect system solution.

I think that there is no better (and more efficient) place to add something like:
if (recordid[0] == '\0' && crash_on_empty_ids_enabled) { exit(-1); }
in right places of libjbase / (jEDI) driver(s) :)

Null ids are causing jBASE to be inconistent in a way that on non-MV database (Oracle, SQL Server, DB2) will not accept empty ids as primary keys. Such WRITEs with empty id will fail on Oracle if I remember well. RECORDID primary key on Oracle was mandatory, thus non nullable.

Ross: could you explain me why would you need to create records with blank ids? (or record ids with field/multi/value separators?)

New environment variable has been proposed, because jBASE configuration is built around environment variables. I would of course perfer some more sophisticated and documented :) configuration file.

Kind regards
Pawel

NB. I like jBASE db very much, but I expect it to be superfast and reliable :)

Mark Hogden

unread,
Aug 17, 2011, 12:45:53 AM8/17/11
to jb...@googlegroups.com
Just to add that hammering the lock table (for example doing thousands of file opens in a short period of time) under 5.2.x on Windows 'seems' to cause locking to lose its way, regardless of file type.


Jim,

AA

--

John Watson

unread,
Aug 17, 2011, 5:59:26 AM8/17/11
to jBASE
Hi Pawel,

You have said

'In our case high 'lock collision' ratio on group locks is killing
performance. JOB.LIST.x files suffer from multiple, concurrent
'SELECTs SAMPLE' (~ full scan) and multiple, concurrent READUs/WRITEs'

I'm not sure what release you use at the moment but I would only
expect to see the above problem towards the end of a job when a high
number of agents are being used - at the start of the job a list is
created and each agent takes 1/number of agents * total number of
records as the agent work load - there should be no duplication of
records across agents at this initial stage (but of course there could
be a duplication of groups)

When this workload is consumed a full select will take place on the
JOB.LIST file and hence we can have more issues with collisions on
groups, records processed by other agents etc.

Are you sure that the problem is actually with the JOB.LIST table and
not locks on BATCH.STATUS or BATCH - there have been problems in this
area in much older releases and they should now be addressed.

If the issue is with group locks on the JOB.LIST file and the overhead
really is substantial then perhaps there is a logical argument to
oversize the file to minimise the number of records in a group - to be
worthwhile the increase in select time would have to outweigh the time
lost to group locks.

Cheers,

John.

John Watson

unread,
Aug 17, 2011, 5:51:01 AM8/17/11
to jBASE
Hi Adrian,

My understanding is that when a JR file feels the need to grow there
will be a group lock in place to secure the re-distributed data from
that group and for the new target groups. The duration of the lock may
seem quite lengthy as the entire re-distribution must happen before
the lock is released - are you seeing the group locks only in files
with growth?

I can see a benefit in the JR files for Non-Stop etc. but as with
everything there is a cost - in this case possibly performance. I'm
sure it has been said many times here before that a well sized hash
file will outperform most types of file storage.

Cheers,

John.

Jim Idle

unread,
Aug 17, 2011, 12:28:36 PM8/17/11
to jb...@googlegroups.com
I think that we might already have this as an envvar/config option but I
cannot find it in any docs, so you will have to ask via the support channel.

Jim

> -----Original Message-----
> From: jb...@googlegroups.com [mailto:jb...@googlegroups.com] On Behalf
> Of David McGehee
> Sent: Tuesday, August 16, 2011 5:50 PM
> To: jb...@googlegroups.com
> Subject: RE: RE: RE: jbase problems show-items-locks
>

Jim Idle

unread,
Aug 17, 2011, 12:35:26 PM8/17/11
to jb...@googlegroups.com
Actually, the locking behavior you suggest will only be the case so long
as the select is in hash order and not a SSELECT. A SSELECT will cause
different batch processes to access the same groups simultaneously almost
immediately, with a probability related to the number of groups (assuming
well sized) and the number of batch processes.

Also don't forget that batch inputs will be processed better if you make
the files read only and do not delete anything until a cleanup at the end
(of all processing). Knowing that there is no problem with read/update
locking, J4 files will not take the locks (unless someone changed this in
jBASE 5 for instance). Also, you should look in to FASTSCAN as application
programmers.

Jim

> -----Original Message-----
> From: jb...@googlegroups.com [mailto:jb...@googlegroups.com] On Behalf
> Of John Watson
> Sent: Wednesday, August 17, 2011 2:59 AM
> To: jBASE
> Subject: Re: jbase problems show-items-locks
>

VK

unread,
Aug 17, 2011, 9:53:49 AM8/17/11
to jBASE
Hi John,

What about a thought of changing the whole scheme of agents' work?
Like: starting a job (on a SELECT stage) "main" process creates not a
single F.JOB.LIST file but several ones - each for one agent to use
exclusively. Of course situation of dying agent is to be handled as
well (as well as parallel execution of different jobs). But I believe
finally we might achieve better overall performance.

Offtopic:

Another thing that I personally would like to see in T24 COB - you can
set number of agents for particular time but not for particular job...
(you know sometimes it has to be only one)

VK

Charlie Noah

unread,
Aug 17, 2011, 4:41:48 PM8/17/11
to jb...@googlegroups.com
Hi VK,

Would you mind checking your time settings in your email client (or perhaps the problem is with your ISP's email server)? This email arrived a few minutes ago with a time stamp of 08:53 (I got it at approximately 15:33 CST, United States), and appeared way up in my inbox, which is sorted by date/time. Just a nit, I know, but I get a lot of these, especially from across the pond. ;^)

Thanks,
Charlie Noah

Jim Idle

unread,
Aug 17, 2011, 5:16:36 PM8/17/11
to jb...@googlegroups.com

It is because Google preserves the time stamp of the email when it was received, but it won’t actually go out until it is approved. The timestamps are correct.

 

jim

 

From: jb...@googlegroups.com [mailto:jb...@googlegroups.com] On Behalf Of Charlie Noah
Sent: Wednesday, August 17, 2011 1:42 PM
To: jb...@googlegroups.com
Subject: Re: jbase problems show-items-locks

 

Hi VK,

Pawel (privately)

unread,
Aug 17, 2011, 7:14:57 PM8/17/11
to jb...@googlegroups.com
Hi,

I may be wrong, but is not that effect of file handles pooling? As far as I remember jBASE keeps physically not more than X (say 1000) files open. Closing a handle / file means loosing its lock, right? (not 100% wheter I am right)

Kind regards
Pawel

Dnia 17-08-2011 o godz. 6:45 Mark Hogden napisaďż˝(a):

> > > Dnia 15-08-2011 o godz. 20:29 Jim Idle napisaďż˝(a):

> > read more Âť- Hide quoted text -

Pawel (privately)

unread,
Aug 17, 2011, 8:01:53 PM8/17/11
to jb...@googlegroups.com
Hi guys,

Thank you very much for discussion, which was started by Adrian. I should likely create separate thread not to abuse his original post. Anyway...

Unfortunately I do not understand FASTSCAN interface. I know that there is JBASE_DISTRIB_FASTSCAN environment variable. It is already set to 1 in our config. I am not sure however of full meaning of the variable. Jim explained partly in the past and I could not understand difference / benefits from normal file traversal. I should also check my work mail - perhaps I have explanation and simply do not remember anymore.

T24 R06 is currently doing something like that (worker thread concept):
1. Allocate JOB.LIST.x file name (name completely unpredictable - can be JOB.LIST.1 or JOB.LIST.7 or something else if job is really old)
2. Store ids/transaction data to be processed. Single recordid may contain from 1 to X transaction data inside.
3. Each agent:
SELECT joblist SAMPLE number
process transaction/transaction(s) that are not being processed (locked)
again SELECT joblist SAMPLE number....
and so on, and so on...

Above pattern could be changed to avoid unnecessary SELECT joblist SAMPLE number. Say agent could keep his own, local list of items (SELECT joblist SAMPLE number result):
a) if item not existent remove it from locally kept list (already processed by other agent)
b) if item locked keep it still on the list and count (being processed, but agent may fatal out for any reason (DEADLOCK eg.))
c) if item not locked - process and remove from the list
d) if list empty or count<number - SELECT joblist SAMPLE number
5000 as number should be fine (should be fine for 500 agents 10 servers each)
Unnecessary SELECTs should be avoided. JOBLIST files can not be marked

I would prefer also that JOB.LIST names are unique and PREDICTABLE! JOB.LIST file is though thing for jBASE, because of right sizing and variable data volume. You know however that there are big jobs with more transactions and smaller jobs with less transactions - but you can not size joblists well! Currently joblist files needs to be sized (more or less) the same way, which is not perfect. AFAIK JR is not available for jBASE 4.1.

Last thing (jBASE 4.1)
We see that SELECT joblist SAMPLE number (traversing file) causes a lot of locks to be taken (each group is binary locked). I will try to paste truss output. That is normal, right? FASTSCAN could change it? (but joblist files are SELECTed and updated at the same time!)

NB. Jim I saw your (very old) post - http://groups.google.com/group/jBASE/browse_thread/thread/6133536abb4c30e7/0d4703621957b4d7?hl=en&lnk=gst&q=file+system#0d4703621957b4d7. Perhaphs you would like to give us trial of you "J5" filetype? :)

I will try to give you more information about (P7) problems we face.

Kind regards
Pawel


Dnia 17-08-2011 o godz. 18:35 Jim Idle napisaďż˝(a):

John Watson

unread,
Aug 18, 2011, 4:37:40 AM8/18/11
to jBASE
Hi VK,

The advantage of a single JOB.LIST is that a job can be stopped and
started with a different number of agents as desired and the job will
complete - remeber that the tSM / tSA concept is not only for COB.

The fact that each agent initially takes a different chunk of the job
list should minimise this problem and if processing time per record is
similar then the agents should all finish around the same time and the
subsequent full select issue should not be a problem.

Of course there may be other ways to manage this and if any T24 system
designers read this forum then hopefully they can include some ideas -
but remember that the really big T24 clients now use Oracle or (would
you believe it) MSSql so this group issue is not a problem (they get
to have other problems :-) )

On your other issue you could perhaps create a new job that does
something like the following;

* This routine will modify the COB TSA.SERVICE record to allow the
server
* and workload profile used to be modified whilst the COB is running.
* The intention is that agents can be increased/decreased as required
* and servers can be disabled/added for the duration of certain jobs.
* the routine would be added to the relevant batch record before and
* after the job to have modified agents.
* The server name and workload profile to use will be held in the DATA
field of the
* BATCH record relevant to this routine.

You could then insert this job wherever you wanted to modify the
number of agents.

Of course it would be nice if T24 allowed the number of agents to be
stored in the relevant PGM.FILE - but as it took over 5 years for the
BULKING parameter to be moved here I wouldn't hold my breath...

Cheers,

John.

Charlie Noah

unread,
Aug 18, 2011, 11:26:16 AM8/18/11
to jb...@googlegroups.com
Hi Jim,

I remember when we ported to Jbase from Universe in 1998 that we had to fix a ton of things UV let us get away with, but Jbase didn't. It was a royal pain and a very stressful time, but I really liked that Jbase forced us to do things "right".

Regards,
Charlie Noah

VK

unread,
Aug 19, 2011, 3:08:25 AM8/19/11
to jBASE
Hi all,
thanks John. And - I agree that assigning and managing "exclusive"
chunks of ID pools would require a lot of intelligence to be added to
tSM.
BTW, I remember that at early days of Oracle driver people still kept
their F.JOB.LIST.n files as J4 ones...

to Charlie Noah: I post via Web interface, not e-mail client.

To Pawel: you mentioned T24 compiled for Java. Yes that thing exists,
it's called TAFJ, was started to be developed about 5-6 years ago and
I wonder if any real customer uses that.
But that's off-topic of course; write to me privately if you would
like to know more.

VK

John Watson

unread,
Aug 19, 2011, 4:52:15 AM8/19/11
to jBASE
Hi Pawel,

I am sure there is an environment variable that allows the lock to be
kept if a file handle is closed - this was introduced as the T24 OPF
routine only has an array of 99 to hold the opened files. Although I
guess it may be different destroying an array reference of an open
file to physically closing that file.

Cheers,

John.

John Watson

unread,
Aug 19, 2011, 5:38:27 AM8/19/11
to jBASE
Hi Pawel,

The BATCH.JOB.,CONTROL works nearly as you describe;

The select JOB.LIST SAMPLE is simply to see if there is work
remaining, it does not create the workload for an agent; instead a
UNIQIE chunk of the JOB.LIST is taken for processing. This can be seen
in the COMO where it says;
* Processing - is the total number of records in the JOB.LIST at that
time
* Offset - is the starting position in the list of records that this
agent will process INT(((Total Number to Process / Number of Sessions)
* My session Number) - (Total Number to Process / Number of Sessions))
* Number of Sessions - the number of agents currently contributing to
this job.

The records processed by the agent will be the section of the job list
starting at the offset and will contain (processing/number of
sessions) records - this effectively means that each agent does
process a specific workload so no need to hold a sperate file for each
agent.

This only happens if there are 10*the number of agents worth of
records to process, in this case the whole list is selected by each
agent.

Other than the SAMPLE issue the mechanism works more or less as you
describe.

You are quite right that sometimes a row in the job list file may
contain a number of explicit IDs - this is a mechanism to group
particular kinds of data and avoud locking.

If you really want to prove this then create a sample set of routines
where the record routine simply prints the ID to be processed - you
will see there is no conflict until near the end of the job (less than
the required number of records to make the splitting worthwhile)

The JOB.LIST file number is the lowest unused number found from
checking the F.LOCKING table - if you have orphan records in here then
the job list files could be high. There are times when a particular
job could get different job files from one run to the next - this is
if different BATCH records share the same stage and hence the jobs can
run concurrently.

HTH

John.


On Aug 18, 1:01 am, "Pawel (privately)" <pprivat...@wp.pl> wrote:
> Hi guys,
>
> Thank you very much for discussion, which was started by Adrian. I should likely create separate thread not to abuse his original post. Anyway...
>
> Unfortunately I do not understand FASTSCAN interface. I know that there is JBASE_DISTRIB_FASTSCAN environment variable. It is already set to 1 in our config. I am not sure however of full meaning of the variable. Jim explained partly in the past and I could not understand difference / benefits from normal file traversal. I should also check my work mail - perhaps I have explanation and simply do not remember anymore.
>
> T24 R06 is currently doing something like that (worker thread concept):
> 1. Allocate JOB.LIST.x file name (name completely unpredictable - can be JOB.LIST.1 or JOB.LIST.7 or something else if job is really old)
> 2. Store ids/transaction data to be processed. Single recordid may contain from 1 to X transaction data inside.
> 3. Each agent:
> SELECT joblist SAMPLE number
> process transaction/transaction(s) that are not being processed (locked)
> again SELECT joblist SAMPLE number....
> and so on, and so on...
>
> Above pattern could be changed to avoid unnecessary SELECT joblist SAMPLE number. Say agent could keep his own, local list of items (SELECT joblist SAMPLE number result):
> a) if item not existent remove it from locally kept list (already processed by other agent)
> b) if item locked keep it still on the list and count (being processed, but agent may fatal out for any reason (DEADLOCK eg.))
> c) if item not locked - process and remove from the list
> d) if list empty or count<number - SELECT joblist SAMPLE number
> 5000 as number should be fine (should be fine for 500 agents 10 servers each)
> Unnecessary SELECTs should be avoided. JOBLIST files can not be marked
>
> I would prefer also that JOB.LIST names are unique and PREDICTABLE! JOB.LIST file is though thing for jBASE, because of right sizing and variable data volume. You know however that there are big jobs with more transactions and smaller jobs with less transactions - but you can not size joblists well! Currently joblist files needs to be sized (more or less) the same way, which is not perfect. AFAIK JR is not available for jBASE 4.1.
>
> Last thing (jBASE 4.1)
> We see that SELECT joblist SAMPLE number (traversing file) causes a lot of locks to be taken (each group is binary locked). I will try to paste truss output. That is normal, right? FASTSCAN could change it? (but joblist files are SELECTed and updated at the same time!)
>
> NB. Jim I saw your (very old) post -http://groups.google.com/group/jBASE/browse_thread/thread/6133536abb4.... Perhaphs you would like to give us trial of you "J5" filetype? :)
>
> I will try to give you more information about (P7) problems we face.
>
> Kind regards
> Pawel
>
> Dnia 17-08-2011 o godz. 18:35 Jim Idle napisa (a):
> > > John.- Hide quoted text -

adrian

unread,
Aug 19, 2011, 9:18:10 AM8/19/11
to jBASE
We have been informed by Temenos to change the JR files which are used
for migration to J4 because there is a performance problem with the JR
files type. The throughput of the JR we have found are much slower
compared to the J4 files and there is also a problem with exclusive
reads with JR. (SP8 and SP11 have this problem).
Any have a formula to convert JR files to J4?
AA
> >http://groups.google.com/group/jBASE?hl=en- Hide quoted text -

Jim Idle

unread,
Aug 19, 2011, 12:37:50 PM8/19/11
to jb...@googlegroups.com
The far better re-design is to change to using message queues, which means
you won't have any locks and won't be splitting huge select lists. The
batch jobs could probably run in real time with a bit of database thought.
However, even though I have been advising message queue based distributed
processing for at least a decade - only one of you has done it (and he
will never go back).

Jim

> -----Original Message-----
> From: jb...@googlegroups.com [mailto:jb...@googlegroups.com] On Behalf
> Of VK
> Sent: Friday, August 19, 2011 12:08 AM
> To: jBASE
> Subject: Re: jbase problems show-items-locks
>

Jim Idle

unread,
Aug 19, 2011, 12:39:34 PM8/19/11
to jb...@googlegroups.com
He means the internal closing of OS file handles to cache the number of
available file handles. However, that does not release any locks because
for a start, if the file has any locks, another handle is closed instead.
I am not even sure that that happens any more in fact as most OSs can
handles many thousands of file handles.

Jim

> -----Original Message-----
> From: jb...@googlegroups.com [mailto:jb...@googlegroups.com] On Behalf
> Of John Watson
> Sent: Friday, August 19, 2011 1:52 AM
> To: jBASE
> Subject: Re: jbase problems show-items-locks
>

Jim Idle

unread,
Aug 19, 2011, 12:43:54 PM8/19/11
to jb...@googlegroups.com
Yes, I posted as much here about 2 years ago I think. The command jrf -H4
is what you need. then set the defaults to create J4 files, not JR files.
Reply all
Reply to author
Forward
0 new messages