question about data alignment / API QUSCUSAT
Jeff Berman
I'm trying to use API QUSCUSAT to change two user space attributes. The
way you do this is by sending up to three variable length records to the
API. According to IBM's documentation, "Each variable length record
must be 4-byte aligned. If not, unpredictable results may occur."
(http://publib.boulder.ibm.com:80/cgi-bin/bookmgr/BOOKS/QBKAMQ01/4.3.3)
Well, I tried putting the keyword "align" next to the data structure
that contains my variable length records, but the API returned with
errors. However, when I removed the "align" keyword, it worked
beautifully.
Can anybody explain why it works without data alignment and fails with
it, when the API clearly requires alignment?? I'm quite confused.
Maybe I don't understand exactly how data alignment works.
Any insight would really be greatly appreciated. Thank you!
Jeff
Following is the code I used (it looks pretty bad in a proportional
font, sorry):
* Parms used to change the user space attributes
d pSpcAttr ds align <--- this fails
d spc#Recs 10i 0 inz(2)
d key1 10i 0 inz(3) self-extending
d dtalen1 10i 0 inz(%size(dta1))
d dta1 1 inz('1')
d key2 10i 0 inz(2) inital value
d dtalen2 10i 0 inz(%size(dta2))
d dta2 1 inz(x'00')
d pLibName s 10
[snip]
* Make user space self extending.
c eval qusbavl = 0
c call 'QUSCUSAT'
c parm pLibName (output parm)
c parm spcName (usrspc name)
c parm pSpcAttr
c parm apiErr (error struct)
--
(Please remove the computer from my address to reply)
Paul Nicolay
Euh... aren't userspaces always 'self-extending' ?
Regards,
Paul
-------------------------------
Jeff Berman wrote in message ...
Hi everyone,
Jeff
[snip]
The contents of this message express only the sender's opinion.
This message does not necessarily reflect the policy or views of
my employer, Merck & Co., Inc. All responsibility for the statements
made in this Usenet posting resides solely and completely with the
sender.
Tim
an API performs its output to a user space. If you are simply using a user
space for your own purposes, you must extend them explicitly.
"Paul Nicolay" <removethis....@merck.com> wrote in message
news:8529uo$f7q$1...@merck.com...
Paul Allen
not) regardless of whether the code accessing the User Space is an API or a user
program.
Ian Stewart
> No, user spaces are not self extending. The only time they auto-extend is if
> an API performs its output to a user space. If you are simply using a user
> space for your own purposes, you must extend them explicitly.
Er, yes they can be. By default when created, they are not (unfortunately I
can't see this point documented, but experimentation reveals this to be the
case), but this can be (as the original poster of this thread is doing)
specified by a call to QUSCUSAT.
This effect can be used quite nicely if you, say, create a user space, specify
it as auto-extending, retrieve a pointer to it ... voila! Automagically
extending array/structure/whatever ...
-Ian.
--
Ian Stewart
i...@incognito.co.nz
Ian Stewart
I can confirm that (on V4R1 anyway) it works as you've discovered, NOT as
documented in the API manual. Presumably IBM need to correct the manual!
Couple of additional comments:
I would personally not favour using the ALIGN keyword in this situation,
though it does achieve the alignment (theoretically) required by the API.
As I read it, the beginning of each "record" was to be aligned,
conceptually, if a record type had an item of length 8, or multiple items,
the result might not be as intended. But of course there isn't such a
record defined for this API (yet).
I'm also wondering why you specify the initial value in the QUSCUSAT call.
That attribute is specified on the QUSCRTUS call, so the only conceivable
reason to specify it on QUSCUSAT would be to change what was specified
originally. BUT, if you specified something different, note that the
initial size of the user space may (probably will) be larger than the
initially requested size, and the new initial value will only take effect
for extra allocations required beyond this initial allocation. It would
appear to me that this could result in something unexpected if you are
changing the initial value, or if you aren't trying to change it, at least
it would be redundant? Sort of begs the question as to why they even allow
it to be modified later ...
-Ian.
Jeff Berman wrote:
--
Ian Stewart
i...@incognito.co.nz
Jeff Berman
Many, many thanks for your input. Ian, I'm doing exactly what you
suggested: Creating a user space, retrieving a pointer to it, and then
using the space as storage for my own structures. :-)
Well, I still don't understand this thing about using the ALIGN keyword,
but it seems to be working fine without specifying that, so I guess I'll
let it go.
Ian, when I create the user space, I'm specifying the inital value as
x'00'. Is this intial value somehow stored internally as an attribute
of the user space? The reason I'm specifying the initial value on the
QUSCUSAT API is because my understanding was that this initial value
parameter referred to the value of any newly extended portions of the
user space (assuming auto-extending is turned on). I guess I could have
tested it, but I read the documentation that way. In other words, I was
afraid that this scenario could occur: a user space is created with an
initial value of x'FF', it is then configured to auto-extend itself,
then data is placed into the space so that the size of the original
space is exceeded, so then the space is auto-extended with an initial
value of something other than x'FF'. That's why I was explicitly
stating the initial value via QUSCUSAT. Is this not necessary?
Also, if I may ask another question, if more than one job are all
pointing to the same user space (via QUSPTRUS), is there any way to
serialize their access to the user space? Apparently, while QUSCHGUS
obeys locks placed on the user space, assigning a pointer to the user
space doesn't.
Our problem is that we're using a user space to hold a sequence number,
and several jobs will all be adding a value of one to that sequence
number and then using that resulting number for their particular unit of
work. I'm going to have to test this out, but my fear is that this
could happen:
Job #1 increments the sequence number in the user space to, say, 1001;
then job #2 increments it to 1002 before job #1 can move a copy of it to
a non-volatile variable. Finally, job #1 moves a copy of the number
into a non-user space-mapped field and erroneously assigns the value
1002 to its current unit of work.
If we could lock the space somehow before incrementing/retrieving the
number and then unlock it when we're done, that would be perfect.
Unfortunately, that doesn't seem to be possible. Does anybody have any
ideas here? How do you synchronize your programs when you have multiple
jobs all performing identical tasks like this? I guess we could use a
data area, but performance is crucial here, and I fear that data area
access will be too slow.
Or, is this scenario impossible because of the way the AS/400 doles out
time slices? If I have the code (where usrSpcField is mapped to the
user space):
c eval usrSpcField = usrSpcField + 1
c eval tempSeq# = usrSpcField
Is it even possible for another job to run in-between the execution of
those two statements? If not, then I guess we don't have a problem.
Thank you very, very much for any insight into what's going on here.
I'm very grateful for any help.
Jeff
In article <3876EB35...@incognito.co.nz>, Ian Stewart
<i...@incognito.co.nz> wrote:
--
Tim
space that is shared among processes because, as you have surmised, the very
scenario you described can arise. If one job changes the value of a user
space through a pointer, then every other job that has a pointer to that
user space will be affected. Furthermore traditional object locking
mechanisms won't help because once a job has a pointer, it can do what it
will to the data in a user space.
You would be better off to store the value of your sequence number in a data
area and use the built-in data area locking support.
"Jeff Berman" <je...@visualcities.imac.com> wrote in message
news:jeff-31F7ED.13224509012000@news...
> Everyone,
Paul Nicolay
This can be solved by using semaphores.... but as another poster suggested,
why don't you use a data area ?
Kind regards,
Paul
---------------------
Jeff Berman wrote in message ...
Everyone,
...snip...
Also, if I may ask another question, if more than one job are all
pointing to the same user space (via QUSPTRUS), is there any way to
serialize their access to the user space? Apparently, while QUSCHGUS
obeys locks placed on the user space, assigning a pointer to the user
space doesn't.
The contents of this message express only the sender's opinion.
Ian Stewart
[snip]
> Ian, when I create the user space, I'm specifying the inital value as
> x'00'. Is this intial value somehow stored internally as an attribute
> of the user space? The reason I'm specifying the initial value on the
> QUSCUSAT API is because my understanding was that this initial value
> parameter referred to the value of any newly extended portions of the
> user space (assuming auto-extending is turned on). I guess I could have
> tested it, but I read the documentation that way. In other words, I was
> afraid that this scenario could occur: a user space is created with an
> initial value of x'FF', it is then configured to auto-extend itself,
> then data is placed into the space so that the size of the original
> space is exceeded, so then the space is auto-extended with an initial
> value of something other than x'FF'. That's why I was explicitly
> stating the initial value via QUSCUSAT. Is this not necessary?
My understanding is that the intiial value attribute specified when the space
is created will be retained. I haven't found any reason to set it to anything
other than x'00'. I would suggest that respecifying it with QUSCUSAT is not
necessary. Someone else may be able to shed more light on this ... if I've got
it wrong.
OK, I'd say that it's entirely possible that the above statements could be
interrupted! I would expect it would happen in reality very rarely (which
makes it even worse - you could operate for some time before you mysteriously
got an occasional problem with this!)
Something else to be aware of: I don't know about RPG, but you have to be
careful with C because the optimiser may theoretically cache the updated value
in a register until some later point in time ... you might think you've updated
memory, but it won't have, and another process could get the old value! This
can be managed in C ... anyone know whether this could be a problem with RPG.
Also, if memory is updated in a user space in this way, is there any guarantee
that it's available immediately to other processes? I'd assume so ... but
someone who knows more about this than me might like to comment ...
I'm not sure what the overhead of a lock data area/retreive/unlock would be,
but I would not expect it to be extraordinarily slow. (I.e. probably faster
than the other obvious method of placing it in a file, retrieving the record
with a lock, updating it, though you'd perhaps need to try it each way and time
it to really tell - I just assume the data area would be quicker.) Using a
data area would be plain and simple to most RPG programmers too, whatever hoops
you'll have to jump through to make the user space method work might not be ...
which might also be worthy of some consideration.
Either way, I expect you'll have to synchronise your processes somehow. I
notice in the MI Functional Reference that there are LOCK (Lock Object), LOCKOL
(Lock Object Location) and LOCKSL (Lock Space Location) instructions, it
appears that they are available from bound programs, though I don't know
whether any of them would be accessible from RPG. I haven't had time to figure
the distinction between the latter two, but one or the other may be promising.
It says they're "symbolic" locks - my interpretation is that they won't prevent
anything from happening unless the application(s) manage them correctly, i.e.
each program would have to place a lock before interrogating/updating a
particular part of your user space. It does appear however that you can get
your process to synchronously wait on the lock, which would be essential for it
to work effectively.
Another option may be the Unix-like semaphores. I've never used them, and
perhaps they're implemented using the above instructions anyway. OTOH, they
might be accessible from RPG where some of the above might not be. (Lots of
speculation on my part, haven't had time to experiment, though I'm likely to as
this coincides with something I'm thinking about at the moment ...)
Whether in practise this works out any differently in performance terms than
using a data area is something I have no idea about at this stage ... good
luck!
Jeff Berman
I posed the same question to a friend of mine, and he suggested using
LOCKSL, which I implemented today. You *can* call it from RPG. You
have to prototype the function in RPG, and then create your program
using the QC2LE (this is from memory) binding directory, which will bind
in whatever service program contains LOCKSL.
Performance was much, much faster than using *LOCK IN and OUT with a
data area. And just to make sure it all worked, I ran two processes,
each of which contend for the same user space, and paused one of them
just after placing the lock. Sure enough, the other job stopped dead in
LCKW status.
The only difficult part, which I must confess I didn't address, is what
to do if the wait time expires. There's no way to place an indicator on
a CALLP statement. My friend suggested using the RPG condition handler,
but I'm not familiar with that at all. I guess in higher OS releases
you can do something like "CALLP(e) SomeProcedure", and then use a
built-in function to detect if it ended in error, but we have to compile
back to v3r7.
Anyway, thank you for all your help. You really do know your stuff!
It's a true honor talking tech with you.
Jeff
In article <387BABEE...@incognito.co.nz>, Ian Stewart
<i...@incognito.co.nz> wrote:
> I notice in the MI Functional Reference that there are LOCK (Lock
> Object), LOCKOL (Lock Object Location) and LOCKSL (Lock Space
> Location) instructions, it appears that they are available from bound
> programs, though I don't know whether any of them would be accessible
> from RPG.
>
>
Karl Hanson
>
<snip>
>
> Ian, when I create the user space, I'm specifying the inital value as
> x'00'. Is this intial value somehow stored internally as an attribute
> of the user space? The reason I'm specifying the initial value on the
> QUSCUSAT API is because my understanding was that this initial value
> parameter referred to the value of any newly extended portions of the
> user space (assuming auto-extending is turned on).
You are correct - the initial value is used both for the initial
allocation when the space is created, and for subsequent extensions (if
any).
>
> Also, if I may ask another question, if more than one job are all
> pointing to the same user space (via QUSPTRUS), is there any way to
> serialize their access to the user space? Apparently, while QUSCHGUS
> obeys locks placed on the user space, assigning a pointer to the user
> space doesn't.
>
<snip>
As Ian mentioned, the Lock Space Location (LOCKSL) is an option,
although I'm uncertain if this can be accessed in RPG. The counterpart
is UNLOCKSL. These are fairly efficient (low overhead) operations, and
sometimes are used internally by OS/400. You can find out more in the
MI Library Reference book:
http://publib.boulder.ibm.com/cgi-bin/bookmgr/BOOKS/QBJADR00/CCONTENTS
There are a number of other options. The best overall reference is
probably Ed Prosser's Shared Memory Concepts whitepaper at:
http://www.as400.ibm.com/WHPAPR/sharedmem.htm
--
Karl Hanson
Ian Stewart
> Damn, Ian, but you are GOOD. :-)
Oops! A fan club?!?!! :-)
> I posed the same question to a friend of mine, and he suggested using
> LOCKSL, which I implemented today. You *can* call it from RPG. You
> have to prototype the function in RPG, and then create your program
> using the QC2LE (this is from memory) binding directory, which will bind
> in whatever service program contains LOCKSL.
That's good. I presume most stuff in QC2LE is available to RPG, but I
hadn't checked to see if it was in there or not.
> Performance was much, much faster than using *LOCK IN and OUT with a
> data area. And just to make sure it all worked, I ran two processes,
> each of which contend for the same user space, and paused one of them
> just after placing the lock. Sure enough, the other job stopped dead in
> LCKW status.
That's good to know. For my purposes, a data area wouldn't do anyway, but I
will definitely want very low overhead too as I'm working towards something
I expect to be general purpose and which might get heavily used in an
application.
> The only difficult part, which I must confess I didn't address, is what
> to do if the wait time expires. There's no way to place an indicator on
> a CALLP statement. My friend suggested using the RPG condition handler,
> but I'm not familiar with that at all. I guess in higher OS releases
> you can do something like "CALLP(e) SomeProcedure", and then use a
> built-in function to detect if it ended in error, but we have to compile
> back to v3r7.
I'm on the V3R7 version of the compilers too (my machine is on V4R1 - likely
to be upgraded soon to V4R4, but I still would want to target back to V3R7
anyway ...)
I haven't tried the condition handlers either, but presumably they would
allow you to intercept and handle the situation before the standard RPG
error stuff kicks in. I just wrote a small function in C to bind into a
trigger program (trying to catch the invocation stack of whatever process is
doing something naughty to a file!) and the main consideration was that it
wouldn't want to break anything - in that, I used the direct exception
monitor feature, which as far as I know isn't available in RPG. It seems
most like the CL MONMSG in concept - works quite nicely. I've got a global
monitor of virtually anything covering the entire function so that I can
quietly exit without upsetting anything if something really unexpected
happens and a couple of more specific ones to trap errors on the MI calls.
More specifically related to your concern: check out the LOCKSL2 variant
which allows the specification of the timeout value. Some quick
calculations on the maximum allowable time (documented in the MI Functional
Reference) indicates that you could get it to wait for up to nearly 9 years
(that's if I got my arithmetic right, and presumably if you could keep your
machine running continuously for that long!)
Of course, that won't be much help if you *want* a timeout (though obviously
you get to choose how long it is with LOCKSL2) ... if you don't want it to
wait forever, then you'll probably end up reading about the condition
handlers!
Gernot Langle
> < snip >
>
> I'm on the V3R7 version of the compilers too (my machine is on V4R1 - likely
> to be upgraded soon to V4R4, but I still would want to target back to V3R7
> anyway ...)
>
> < snip >
Ian, don't go to V4R4 - you will not be able to target back to V3R7; only V4R3
will let you do that.
--
Gernot Langle
PARAS Solutions, Inc.
http://www.parassolutions.com
mailto:gla...@parassolutions.com
Ian Stewart
Thanks for the warning! I remember seeing someone with a problem with that
posting here a while back, I got the impression it was a "bug" rather than a
"feature"!
It might be a tough decision; with the 9401-150 I get to pay for each upgrade, no
subscription, so I'll have to weigh the benefits. I have to balance the need to
keep reasonably current with my clients' machines with the requirements for some
(currently hypothetical) software I want to develop.
I might go V4R4 anyway, but develop my software only using interfaces available on
V3R7 unless it's absolutely essential to use a later one for a particular
feature. If I get it ready in the next year or so (!) and if it sells AND if I
get demand for V3R7 support (or even V3R2) then I might address the problem then.
At that point, if I get enough interest in the product, I could justify buying a
slightly larger machine (which I'd probably go with the latest version on) and
then backdate the 150 to V4R1 or V3R7 (which I also have for this machine). V3R2
support would be a different problem again; I am only aware of one CISC machine
still running locally (and that for not much longer) so maybe it won't be too big
an issue in another year or so.
Gernot Langle wrote:
--
Ian Stewart
i...@incognito.co.nz
Anton Gombkötö
Didn't someone tell you about KISS? Keep it simple....
I'm happy to see that someone is willing to pay for research on new
solutions for such every-day tasks, congratulations!
1) if you really want to lock a user space, then allocate it. When you
have an *EXCL lock on the space, nobody else can access it. This will
slow down the thing so that the user space advantage (if there is one)
is gone.
2) normally data structures with the based(ptr) option are used to
access user spaces via pointers. One should suggest that ADD 1 to a
subfield of such a data structure should result in an immediate change
in the user space so that no other job has a chance to retrieve the
old value. But i wouldn't bet on it.
I do expect the use of QUSRTVUS and QUSCHGUS to be very much slower
than almost any other solution. And, of course, there definitely is
the danger of duplicate numbers. (Danger is the wrong word, danger
expresses that it might happen, but this will happen.)
Normally one has a file with a record in it containing such a number.
When programs open this file for update, records are locked after the
READ/CHAIN and stay locked till the record *is* updated. Other jobs
wait until their WAITRCD job attribute is exceeded. This happens on
thousands of machines every day almost every second.
I don't think it is a good idea to reprogram record lock / object lock
logic in application programs.
I do not expect the approach via DB/2 to be that slower than the user
space solution that it pays off ever.
4) Another solution is to have progarms receiving such numbers from a
data queue. Another program can fill the data queue up with unique
numbers. A bit tricky, but useful in some environments.
5) i expect the x'00' (or any other value) to be necessary to
overwrite the contents on the disk. A user space has no structure
(from the OS), so if i create a user space with 16 MB and have no
initial value, i'd surely be able to see parts of objects that existed
before. These data has to be destroyed prior to giving a mortile a
pointer to that space.
0.02 Euro
Anton Gombkötö
Jeff Berman
In article <388507b9...@news.chello.at>,
Gombk...@nospam.assoft.com (Anton Gombkötö) wrote:
> 1) if you really want to lock a user space, then allocate it. When you
> have an *EXCL lock on the space, nobody else can access it. This will
> slow down the thing so that the user space advantage (if there is one)
> is gone.
Good idea, except that the lock is only obeyed when you use QUSCHGUS,
not when you access it via pointers.
> Normally one has a file with a record in it containing such a number.
> When programs open this file for update, records are locked after the
> READ/CHAIN and stay locked till the record *is* updated. Other jobs
> wait until their WAITRCD job attribute is exceeded. This happens on
> thousands of machines every day almost every second.
True, but it's also fairly slow. After implementing LOCKSL in my
program, I was able to process over one million transactions per hour.
That's over one million increments of the sequence number field. I
shudder to think how performance would be affected if we changed that to
one million CHAIN/UPDATEs.
> I do not expect the approach via DB/2 to be that slower than the user
> space solution that it pays off ever.
Why not? We're talking about disk I/O versus RAM access, aren't we?
> 4) Another solution is to have progarms receiving such numbers from a
> data queue. Another program can fill the data queue up with unique
> numbers. A bit tricky, but useful in some environments.
Now that's an interesting idea. I wonder what the performance would be?
Of course there'd be the overhead of that second job to populate the
numbers.
My $.02...
Jeff
Anton Gombkötö
<je...@visualcities.imac.com> wrote:
>Ummm.... who wants to take this one?
>
>In article <388507b9...@news.chello.at>,
>Gombk...@nospam.assoft.com (Anton Gombkötö) wrote:
>
>> 1) if you really want to lock a user space, then allocate it. When you
>> have an *EXCL lock on the space, nobody else can access it. This will
>> slow down the thing so that the user space advantage (if there is one)
>> is gone.
>
>Good idea, except that the lock is only obeyed when you use QUSCHGUS,
>not when you access it via pointers.
I know, or better, i believe to know as i haven't tried yet; my answer
was unclear. I meant that nobody can retrieve a pointer with QUSPTRUS
while another job has a *EXCL lock on the space.
>
>> Normally one has a file with a record in it containing such a number.
>> When programs open this file for update, records are locked after the
>> READ/CHAIN and stay locked till the record *is* updated. Other jobs
>> wait until their WAITRCD job attribute is exceeded. This happens on
>> thousands of machines every day almost every second.
>
>True, but it's also fairly slow. After implementing LOCKSL in my
>program, I was able to process over one million transactions per hour.
>That's over one million increments of the sequence number field. I
>shudder to think how performance would be affected if we changed that to
>one million CHAIN/UPDATEs.
I am not afraid about it and wouldn't hesitate to implement it this
way (again).
>
>> I do not expect the approach via DB/2 to be that slower than the user
>> space solution that it pays off ever.
>
>Why not? We're talking about disk I/O versus RAM access, aren't we?
No, definitely not. The file is in RAM, especially when you update it
a million times per hour.
>
>> 4) Another solution is to have progarms receiving such numbers from a
>> data queue. Another program can fill the data queue up with unique
>> numbers. A bit tricky, but useful in some environments.
>
>Now that's an interesting idea. I wonder what the performance would be?
>Of course there'd be the overhead of that second job to populate the
>numbers.
Not bad. We didn't measure it down to the single thing. The generation
of numbers was the problem (had to be in sequence; no holes wanted,
but there were holes, when the transaction didn't complete due to
errors, so we used these numbers again) ; that wasn't that easy. So it
was a good idea to do this at another time, having the numbers handy
for the jobs needing them.
We have in the same package the DB-solution for other keys, and this
is also quite fast. We do not have a million updates per hour, as our
2500 clients aren't able to perform 400 transactions per hour. They
have to fill in fields.. :-)
>
>My $.02...
>
>Jeff
$0.02 + Euro 0.02 = ?
:-)
Tony
Karl Hanson
>
> On Wed, 19 Jan 2000 05:57:38 GMT, Jeff Berman
> <je...@visualcities.imac.com> wrote:
>
> >Ummm.... who wants to take this one?
> >
> >In article <388507b9...@news.chello.at>,
> >Gombk...@nospam.assoft.com (Anton Gombkötö) wrote:
> >
> >> 1) if you really want to lock a user space, then allocate it. When you
> >> have an *EXCL lock on the space, nobody else can access it. This will
> >> slow down the thing so that the user space advantage (if there is one)
> >> is gone.
> >
> >Good idea, except that the lock is only obeyed when you use QUSCHGUS,
> >not when you access it via pointers.
> I know, or better, i believe to know as i haven't tried yet; my answer
> was unclear. I meant that nobody can retrieve a pointer with QUSPTRUS
> while another job has a *EXCL lock on the space.
>
From my reading of the API documentation I don't think this is true. No
lock is shown in the "Authorities and Locks" section (whereas QUSCHGUS
gets a *EXCLRD lock), and there is this statement: "The QUSPTRUS API
even returns a pointer to an object that is subject to an exclusive
(*EXCL) lock." I believe this means one *can* retrieve a pointer even
when another job holds an exclusive lock.
<snip>
--
Karl Hanson
Tim
user space are "well-behaved," they will attempt to allocate the user space
before they attempt to modify it, thereby assuring proper intergrity of the
space.
But I will say once again, the original poster asked this question because
he wanted to increment a value in a user space in order to generate a
sequential key for a data base. This task is much better suited to a data
area which enforces proper locking
"Karl Hanson" <kcha...@us.ibm.com> wrote in message
news:3885C9B1...@us.ibm.com...
Njål Fisketjøn
Gombkötö) wrote:
>>> 4) Another solution is to have progarms receiving such numbers from a
>>> data queue. Another program can fill the data queue up with unique
>>> numbers. A bit tricky, but useful in some environments.
>>
>>Now that's an interesting idea. I wonder what the performance would be?
>>Of course there'd be the overhead of that second job to populate the
>>numbers.
>Not bad. We didn't measure it down to the single thing. The generation
>of numbers was the problem (had to be in sequence; no holes wanted,
>but there were holes, when the transaction didn't complete due to
>errors, so we used these numbers again) ; that wasn't that easy. So it
>was a good idea to do this at another time, having the numbers handy
>for the jobs needing them.
Did you use a keyed data queue so that you could put the uncompleted
transaction numbers back on the data queue again?
Njål Fisketjøn, FIGU DATA AS
n...@figu.no
http://www.robin.no/~nfisketj
Njål Fisketjøn
<je...@visualcities.imac.com> wrote:
>Damn, Ian, but you are GOOD. :-)
>
>I posed the same question to a friend of mine, and he suggested using
>LOCKSL, which I implemented today. You *can* call it from RPG. You
>have to prototype the function in RPG, and then create your program
>using the QC2LE (this is from memory) binding directory, which will bind
>in whatever service program contains LOCKSL.
>
FYI: No need to use C service programs.
LOCKSL (LOCKSL1 and LOCKSL2) can be used as a system built-in (as can
all MI instructions with members in QSYSINC/MIH).
Search http://www.deja.com for system built-in for sample ILE RPG code
using built-ins.
Ian Stewart
[snip]
> I know, or better, i believe to know as i haven't tried yet; my answer
> was unclear. I meant that nobody can retrieve a pointer with QUSPTRUS
> while another job has a *EXCL lock on the space.
*Assuming* they couldn't, this would rely on the fact that they resolved the
pointer to the user space each time it was accessed; I think this would tend
to negate the gain in performance by using the pointer in the first place ...
Ian Stewart
[snip]
> > I do not expect the approach via DB/2 to be that slower than the user
> > space solution that it pays off ever.
>
> Why not? We're talking about disk I/O versus RAM access, aren't we?
Disk, cached by RAM, in both cases. I think the real difference is in the
overhead of the database I/O API's (both in the calling of them itself, and
the amount of work they probably have to do given the general nature and
wider scope of their functions). Also, I'm unsure whether the system
is/isn't more likely to switch tasks when you do file I/O (perhaps not if
it's already available in RAM?)
Jeff Berman
> > > I do not expect the approach via DB/2 to be that slower than the user
> > > space solution that it pays off ever.
> >
> > Why not? We're talking about disk I/O versus RAM access, aren't we?
>
> Disk, cached by RAM, in both cases.
My understanding was that files open for update did not use record
blocking. In other words, when you issue the record update, the OS
actually writes it out to disk at that time. Conversely, if the file is
opened for output only then sequential file processing can be used, and
groups of records will be cached first and then written out to disk in
one I/O instruction.
Is this not so?
Also, I haven't been able to find any documentation stating exactly when
user space changes are written to disk when the change is made via
pointers. The QUSCHGUS (Change User Space) API lets you specify how you
want the OS to handle this, but the pointer method doesn't. I kind of
assumed that it would function similarly to QUSCHGUS' "0" method: "Does
not force changes - normal system management writes the changes to
auxiliary storage".
No?
Thanks,
Jeff
Jeff Berman
<n...@figu.no> wrote:
> FYI: No need to use C service programs.
>
> LOCKSL (LOCKSL1 and LOCKSL2) can be used as a system built-in (as can
> all MI instructions with members in QSYSINC/MIH).
Damn, Njål, but you are good, too!! :-) That is really cool; I had no
idea. Thank you.
It took a while to locate the exact article on deja news, so for the
benefit of others, here is the relevant article.
Again, thank you!
Jeff
-----------
<<<<< START OF QUOTE >>>>>
In article <3648264e...@news.allianse.no>,
nfis...@hesgrp.com (Njal Fisketjon (Njål Fisketjøn)) wrote:
> The MI Functional Reference manual says for the CPRDATA (Compress Data)
> ILE access
> CALLBI built-in number for CPRDATA is 107.
> CPRDATA(compress_data_template: address)
>
> The RPG/400 Reference manual says for the EXTPROC keyword (in chapter 15)
> ...
> If the name specified for EXTPROC ... starts with "CEE" or an underscore
> ('_'), the compiler will treat this as a system built-in. If it is not
> actually a system built-in, then a warning will appear in the listing;
> .....
> I wonder if this means that you can use *any* MI function from ILE RPG that
> has ILE access specified, or if it is only C that can call them.
>
> I know that I can use some of the functions by linking in the C service
> programs (QC2UTIL1 & 2), but some MI functions are not present in those >
service programs.
> ...
Njål, any ILE language (except possibly ILE CL) can call the MI
builtins. The actual name of the builtin function for CPRDATA is
_CPRDATA; ILE RPG will recognize this as a builtin.
In general, for MI builtin function XXXXX, the "EXTPROC" name is
'_XXXXX' (one underscore). For some MI functions (e.g. RSLVSP) there
are several BIFs: _RSLVSP1, _RSLVSP2 etc.
There are also some MI builtins that are versions of C runtime
functions. For example "memcpy" can be accessed as function 'memcpy' or
as builtin '__memcpy'. I think this type of builtin always has a double
underscore preceding the name.
I wish I knew of a better way to find out the exact name for MI
builtins, but here's how I do it: look in QSYSINC/MIH <BIFNAME> and find
the "#pragma linkage" for the bif. For example: in QSYSINC/MIH CPRDATA,
there is a line #pragma linkage( _CPRDATA, builtin )
This file will also show that there's a wrapper function in the C
runtime for the MI builtins. For example, there is a function cprdata()
- it has different parameters from _CPRDATA, I guess to make it handier
to call. (It's in QC2UTIL1).
--
Barbara Morris IBM Toronto Lab RPG Compiler Development
<<<<< END OF QUOTE >>>>>
Karl Hanson
>
> This is exactly the case. However if all the programs that reference the
> user space are "well-behaved," they will attempt to allocate the user space
> before they attempt to modify it, thereby assuring proper intergrity of the
> space.
Agreed. And the "well-behaved" nature of the application could involve
object level locks, space location locks (LOCKSL), or even other
synchronization methods such as data queues (mentioned by others).
>
> But I will say once again, the original poster asked this question because
> he wanted to increment a value in a user space in order to generate a
> sequential key for a data base. This task is much better suited to a data
> area which enforces proper locking
This comes down to a question of simplicity IMHO. A data area (*DTAARA)
is in reality a "system space" vs a user space - both are MI object type
0x19. So overhead of managing data in either should be roughly the
same, the difference being the pathlength of data area commands/APIs vs
direct pointer access. True the data area mgmt interfaces give proper
lock enforcement, but this could also be built into the application that
accesses the user space (the well-behaved aspect above).
Someone else asked about forcing user space changes to disk (aka
non-volatile storage). Assuming the application has a system pointer to
the user space, this could be done using the ENSOBJ (ensure object) MI
intruction. See the MI Library Reference for details:
http://publib.boulder.ibm.com/cgi-bin/bookmgr/BOOKS/QBJADR00/CCONTENTS
>
> "Karl Hanson" <kcha...@us.ibm.com> wrote in message
> news:3885C9B1...@us.ibm.com...
> > Anton Gombkötö wrote:
> > >
> > > On Wed, 19 Jan 2000 05:57:38 GMT, Jeff Berman
> > > <je...@visualcities.imac.com> wrote:
> > >
> > > >Ummm.... who wants to take this one?
> > > >
> > > >In article <388507b9...@news.chello.at>,
> > > >Gombk...@nospam.assoft.com (Anton Gombkötö) wrote:
> > > >
> > > >> 1) if you really want to lock a user space, then allocate it. When
> you
> > > >> have an *EXCL lock on the space, nobody else can access it. This will
> > > >> slow down the thing so that the user space advantage (if there is
> one)
> > > >> is gone.
> > > >
> > > >Good idea, except that the lock is only obeyed when you use QUSCHGUS,
> > > >not when you access it via pointers.
> > > I know, or better, i believe to know as i haven't tried yet; my answer
> > > was unclear. I meant that nobody can retrieve a pointer with QUSPTRUS
> > > while another job has a *EXCL lock on the space.
> > >
> >
> > From my reading of the API documentation I don't think this is true. No
> > lock is shown in the "Authorities and Locks" section (whereas QUSCHGUS
> > gets a *EXCLRD lock), and there is this statement: "The QUSPTRUS API
> > even returns a pointer to an object that is subject to an exclusive
> > (*EXCL) lock." I believe this means one *can* retrieve a pointer even
> > when another job holds an exclusive lock.
> >
> > <snip>
> >
> > --
> >
> > Karl Hanson
--
Karl Hanson
Anton Gombkötö
On Thu, 20 Jan 2000 11:03:05 +1300, Ian Stewart <i...@incognito.co.nz>
wrote:
>"Anton Gombkötö" wrote:
>
>[snip]
>
>> I know, or better, i believe to know as i haven't tried yet; my answer
>> was unclear. I meant that nobody can retrieve a pointer with QUSPTRUS
>> while another job has a *EXCL lock on the space.
>
Anton Gombkötö
calm.
On Thu, 20 Jan 2000 00:22:20 +0100, Njål Fisketjøn <n...@figu.no>
wrote:
>On Wed, 19 Jan 2000 08:10:22 GMT, Gombk...@nospam.assoft.com (Anton
Anton Gombkötö
It is not possible to retrieve a pointer with QUSPTRUS while another
job is holding a *EXCL lock.
Believing is good, trying is better... :-))
On Wed, 19 Jan 2000 08:26:57 -0600, Karl Hanson <kcha...@us.ibm.com>
wrote:
>Anton Gombkötö wrote:
>>
>> On Wed, 19 Jan 2000 05:57:38 GMT, Jeff Berman
>> <je...@visualcities.imac.com> wrote:
>>
>> >Ummm.... who wants to take this one?
>> >
>> >In article <388507b9...@news.chello.at>,
>> >Gombk...@nospam.assoft.com (Anton Gombkötö) wrote:
>> >
>> >> 1) if you really want to lock a user space, then allocate it. When you
>> >> have an *EXCL lock on the space, nobody else can access it. This will
>> >> slow down the thing so that the user space advantage (if there is one)
>> >> is gone.
>> >
>> >Good idea, except that the lock is only obeyed when you use QUSCHGUS,
>> >not when you access it via pointers.
>> I know, or better, i believe to know as i haven't tried yet; my answer
>> was unclear. I meant that nobody can retrieve a pointer with QUSPTRUS
>> while another job has a *EXCL lock on the space.
>>
>
Karl Hanson
lock on a user space, while a program in job 2 that calls QUSPTRUS
successfully gets a pointer to it.
--
Karl Hanson
Ian Stewart
> My understanding was that files open for update did not use record
> blocking. In other words, when you issue the record update, the OS
> actually writes it out to disk at that time. Conversely, if the file is
> opened for output only then sequential file processing can be used, and
> groups of records will be cached first and then written out to disk in
> one I/O instruction.
>
> Is this not so?
I think there are two levels of buffering happening here. My impression is
that the compilers/system provide support for blocking multiple records for
input or output, and submitting them to the OS as a block, instead of making an
individual call for each record. This can occur when the system doesn't need
to negotiate locks (as in input-only or record addition) or provide immediate
feedback due to invalid key errors.
That is, if you write to a file which has a logical over it with a unique key
specification, I think you'll find that the compiler blocking is forced off at
file open time in any case.
But as far as I know, none of this implies that the changes are synchronously
forced to disk even if the compiler isn't doing the blocking. To do that, I
think you can specifiy a FRCRATIO of 1 on the file, or do something like FEOD
from RPG.
I think people get confused as to the difference between making the record
available to OS/400 (i.e. so that it can be observed by another program) and
forcing it to disk, hence taking the more drastic step of forcing it to disk.
This can impact performance badly (read horrendous here). I've seen programs
doing FEOD take a very long time to process transactions (they aren't doing
anything but waiting on the disk most of the time - hardly any CPU usage!)
OTOH, I wonder whether a FEOD forces *all* updates to disk for the file at that
point, not just those from the program concerned - and in any case, it'll be
keeping the disk(s) pretty busy, all of which is likely to affect anything else
on the system too ...
My understanding of the situation is that to ensure the availability of data to
other programs, it is only necessary to specify SEQONLY(*NO) on an override at
open time; this will prevent the blocking, but not physically force the data to
disk.
> Also, I haven't been able to find any documentation stating exactly when
> user space changes are written to disk when the change is made via
> pointers. The QUSCHGUS (Change User Space) API lets you specify how you
> want the OS to handle this, but the pointer method doesn't. I kind of
> assumed that it would function similarly to QUSCHGUS' "0" method: "Does
> not force changes - normal system management writes the changes to
> auxiliary storage".
>
> No?
I think we would have to assume normal management. I noticed Karl's reply re
ENSOBJ MI instruction, I guess that gives some form of control. I assume this
is synchronous; the QUSCHGUS also has an option for asynchronous - I can't see
how that could be simulated though.
This leads to a question (maybe Karl can help here): what happens if the system
goes down following a change to a user space via a pointer? If it had started,
but not completed any sequence of updates it knew about, would it flag it as a
damaged object? If not, I guess the only really safe way would be for the user
program to set a flag in the space, use ENSOBJ to force it to disk, then do its
stuff, and later clear the flag. I presume that for complete safety the
clearing of the flag would need to be bracketed by calls to ENSOBJ, to avoid
the danger of the pages being updated in an unexpected sequence (I presume the
system can optimise head movement by grouping updates based on physical
location??) And of course, proper attention would also have to be paid to
locks ...
Thomas
As interesting as this thread has been, there still seems no answer for
your actual question. (Nor do I have one.)
Has it been resolved other than 'I get the same result'? (Which is
valuable by itself.) Any IBM response?
Tom Liotta
In article <jeff-066BFA.00390806012000@news>,
Jeff Berman <je...@visualcities.imac.com> wrote:
> Hi everyone,
>
> I'm trying to use API QUSCUSAT to change two user space attributes.
The
> way you do this is by sending up to three variable length records to
the
> API. According to IBM's documentation, "Each variable length record
> must be 4-byte aligned. If not, unpredictable results may occur."
>
(http://publib.boulder.ibm.com:80/cgi-bin/bookmgr/BOOKS/QBKAMQ01/4.3.3)
>
> Well, I tried putting the keyword "align" next to the data structure
> that contains my variable length records, but the API returned with
> errors. However, when I removed the "align" keyword, it worked
> beautifully.
>
> Can anybody explain why it works without data alignment and fails with
> it, when the API clearly requires alignment?? I'm quite confused.
--
Tom Liotta
AS/400 systems programmer
Sent via Deja.com http://www.deja.com/
Before you buy.
Paul Nicolay
You could use semaphores to serialize the access to the userspace (I use it
for avoiding conflicts on shared memory). Just check the UNIX APIs.
Regards,
Paul
---------------------------
Thomas wrote in message <868kh9$779$1...@nnrp1.deja.com>...
Jeff:
As interesting as this thread has been, there still seems no answer for
your actual question. (Nor do I have one.)
Has it been resolved other than 'I get the same result'? (Which is
valuable by itself.) Any IBM response?
Tom Liotta
Thomas
True enough, BUT...
That's not what the original question was about (as far as I can see).
The question was about alignment and QUSCUSAT. If alignment is
*supposed* to be done but it is currently not enforced, then a program
written today might crash after the next PTF "fixes" the alignment
issue. Or perhaps alignment is somehow occurring naturally in the
programs mentioned in this thread and a minor change to one of them will
have "unpredictable results" (although the point about specifying the
ALIGN keyword doesn't seem to fit with that).
I'm still curious.
Tom Liotta
In article <869laj$g3c$1...@merck.com>,
--
Larry Loen
>
> Jeff:
>
> As interesting as this thread has been, there still seems no answer for
> your actual question. (Nor do I have one.)
>
> Has it been resolved other than 'I get the same result'? (Which is
> valuable by itself.) Any IBM response?
>
> Tom Liotta
>
> Jeff Berman <je...@visualcities.imac.com> wrote:
> > Hi everyone,
> >
> > I'm trying to use API QUSCUSAT to change two user space attributes.
> The
> > way you do this is by sending up to three variable length records to
> the
> > API. According to IBM's documentation, "Each variable length record
> > must be 4-byte aligned. If not, unpredictable results may occur."
> >
> (http://publib.boulder.ibm.com:80/cgi-bin/bookmgr/BOOKS/QBKAMQ01/4.3.3)
> >
> > Well, I tried putting the keyword "align" next to the data structure
> > that contains my variable length records, but the API returned with
> > errors. However, when I removed the "align" keyword, it worked
> > beautifully.
> >
> > Can anybody explain why it works without data alignment and fails with
> > it, when the API clearly requires alignment?? I'm quite confused.
>
Well, I work at IBM, but don't take this as an official response.
However, the first question is "what is really being produced by the
compiler" for the "D" specs when 'align' is and is not specified.
I would not immediately blame the API.
It would seem to me likely that since the structure starts with a
four byte integer, then the structure is probably aligned that way
(a free compiler choice, and better performance if the first
and subsequent fields are four byte aligned).
One question is whether, for some reason, the compiler is doing
some not-expected padding when 'align' is specified. This
might cause the interface call to fail simply because everything got
moved two to four bytes "father down" if you understand what I mean.
If this is what is happening, it would be as if the compiler
inserted:
d pad 10i 0
between the pSpcAttr and spc#Recs.
The API, after all, does not react to source code; it can only react
to what the compiler produces. I would start by looking into the
"size" of SpcAttr and see whether it "grows" when 'align' is specified
versus when it does not.
Ian Stewart
> Paul:
>
> True enough, BUT...
>
> That's not what the original question was about (as far as I can see).
> The question was about alignment and QUSCUSAT. If alignment is
> *supposed* to be done but it is currently not enforced, then a program
> written today might crash after the next PTF "fixes" the alignment
> issue. Or perhaps alignment is somehow occurring naturally in the
> programs mentioned in this thread and a minor change to one of them will
> have "unpredictable results" (although the point about specifying the
> ALIGN keyword doesn't seem to fit with that).
>
> I'm still curious.
>
> Tom Liotta
Before all this digression began, I think we established fairly conclusively
that it is as observed by the original poster: ie. it actually *breaks* if
it's aligned as documented. I did a test myself (V4R1) and verified the
layout of the parameter block with the debugger. "Correctly" aligned (as
per the book) it broke, when not aligned (i.e. no padding between the
"subrecords") it worked. So there's only one way to get it to work ...
I'd suggest that it'd be easier for IBM to fix this by simply removing the
bogus note about the alignment from the manual - any change to enforce that
alignment is almost certainly going to break some existing code.
Jeff Berman
rchland.vnet.ibm.com wrote:
> I would start by looking into the
> "size" of SpcAttr and see whether it "grows" when 'align' is specified
> versus when it does not.
Hi Larry, thanks for giving your input. Ok, I'll take a look at what
happens internally to the data structure when it compiles, and I'll post
my findings back here.
The only thing I can think of, and maybe this is what you're saying, is
this:
The documentation says "Each variable length record must be 4-byte
aligned. If not, unpredictable results may occur."
My structure, to refresh your memory, looks like this:
* Parms used to change the user space attributes
d pSpcAttr ds align
d spc#Recs 10i 0 inz(2)
d key1 10i 0 inz(3) self-extending
d dtalen1 10i 0 inz(%size(dta1))
d dta1 1 inz('1')
d key2 10i 0 inz(2) inital value
d dtalen2 10i 0 inz(%size(dta2))
d dta2 1 inz(x'00')
Ok, so maybe the problem is that although IBM states that each *variable
length record* must be aligned, I am beginning the alignment with the
fixed portion of the parameter. Remember, this parameter consists of a
fixed portion (the number of variable length records to follow) plus the
variable length records themselves. If there was a way to align a
subfield then perhaps this would work. Well, I don't know, I'm just
sort of making this up.
Anyway, I'll let you guys know what happens.
Thanks!
Jeff
Ian Stewart
[snip]
> Ok, so maybe the problem is that although IBM states that each *variable
> length record* must be aligned, I am beginning the alignment with the
> fixed portion of the parameter. Remember, this parameter consists of a
> fixed portion (the number of variable length records to follow) plus the
> variable length records themselves. If there was a way to align a
> subfield then perhaps this would work. Well, I don't know, I'm just
> sort of making this up.
Jeff,
I actually tried it myself before posting my original reply on this ... as
you've noticed above, one of my concerns was that the align keyword could
conceptually align fields other than the intiial key of each of the
"subrecords". However, due to the size and attributes of the fields
concerned it works out as you inteded in this case.
Checking the layout using a hex display from the debugger, I concluded that
your "aligned" structure was what the documentation required. I also
concluded that the "unaligned" version definitely wasn't as per the
documentation in layout, but was the only arrangement that worked!
So two distinct pieces of evidence point to it being as per your original
observation: the aligned structure doesn't work, and the unaligned one
does! I find it hard to believe that the unaligned one works
coincidentally; looking at it logically, if it expected the key and data
length of the second structure to be three bytes offset from what you
provided, it would get both binary fields (the key and the length) wrong,
and miss the initial value byte. While testing this, I was also wanting to
verify the other point I made, ie. that the initial allocation would have
already been made with the value specified when the space was created, and
that a change specified here would only take effect on the next physical
extension. I verified that to be the case by creating a space with an
initial value of x'00' and then specifying a value of x'FC' on the call to
change the attribute. I observed the x'FC' values only after forcing an
extension following that call to the QUSCUSAT API.
Given that, I'd be most surprised if it didn't really require it to be
non-aligned ...