V8.3-1H1 SYS V12 patch causes exec alignment faults using RMS Global buffers
Hein RMS van den Heuvel
update
kit introduces a problem causing EXEC mode Alignment when using RMS
files with global buffers (SEQ, REL, IDX)
For systems with multiple CPUs and highly concurrent RMS global buffer
usage this in turn may cause high MPsync activity waiting for a
Memory
Management spinlock which can lead to significant performance
problems.
This problem was (best I know) first analyzed by our friends at Parsec
HP knows about this for at least a week but so far (9-dec-2010) has
failed to retract the patch or at the very least put out a stern
warning.
The fault occurs in the rms cache module for each record access.
This can leads to a few thousand faults/sec for low use which is no
problem
But on busy/fast/concurrent system can lead to 80,000 fault/sec or
more.
It will happen for any file organization (SEQ, RELATIVE, INDEXED)
When using ANALYZE /SYSTEM the FLT TRACE will report entries like
9-DEC 00:06:20.951820 00 FFFFFFFF.809F7C90 RMS+0011DB90
000007FD.BFF4EEA4 Exec 21A00442 FFFFFFFF.7A8E6D38
RM0CACHE + 00004F90 / GET_GBPB + 00000180
Using SDA> SHOW PROC/RMS=(GBH,GBDSUM) one can see that the failign VA
is 0x24 bytes into the currently accessed GBD (middle of GBD
$Q_REL_ADDR?)
Official Patch name: VMS831H1I_SYS-V1200 posted: 2010/09/07
contained in VMS831H1I_UPDATE-V0900 posted: 2010/10/28
Regards,
Hein van den Heuvel
Rich Jordan
Thanks for the heads-up.
Jan-Erik Soderholm
Is this a IA64-only problem or also an Alpha problem ?
The patch name seems to be a IA64 patch (?)
VAXman-
I think this is a patch exclusively for V8.3-1H1 which would imply that it
is just Itanium. That said, the impact of the alignment fault fixup on an
Itanium is significantly greater than it is on Alpha. It may not be very
significant for some but if one is using global buffers to tweak the most
out of RMS activity by using global buffers, then why cripple it even the
slightest with alignment fault fixups; especially, when they were *NOT* an
issue previously.
--
VAXman- A Bored Certified VMS Kernel Mode Hacker VAXman(at)TMESIS(dot)ORG
All your spirit rack abuses, come to haunt you back by day.
All your Byzantine excuses, given time, given you away.
Arne Vajhøj
> The OpenVMS patch SYS V12 which is contained in the latest V900
> update
> kit introduces a problem causing EXEC mode Alignment when using RMS
> files with global buffers (SEQ, REL, IDX)
>
> For systems with multiple CPUs and highly concurrent RMS global buffer
> usage this in turn may cause high MPsync activity waiting for a
> Memory
> Management spinlock which can lead to significant performance
> problems.
>
> This problem was (best I know) first analyzed by our friends at Parsec
> HP knows about this for at least a week but so far (9-dec-2010) has
> failed to retract the patch or at the very least put out a stern
> warning.
:-(
Arne
JF Mezei
> This problem was (best I know) first analyzed by our friends at Parsec
> HP knows about this for at least a week but so far (9-dec-2010) has
> failed to retract the patch or at the very least put out a stern
> warning.
See, there are advantages to preventing customers from freely accessing
patches ! This way, they don't get the bad untested ones :-) :-) :-) :-)
H Vlems
JF you're an optimist ;-)
Hein RMS van den Heuvel
<heinvandenheu...@gmail.com> wrote:
> The OpenVMS patch SYS V12 which is contained in the latest V900
> update kit introduces a problem causing EXEC mode Alignment when using RMS
> files with global buffers (SEQ, REL, IDX)
> HP knows about this for at least a week but so far (9-dec-2010) has
> failed to retract the patch or at the very least put out a stern
> warning.
[ this may be a re-post. Thought I submitted this earlier, but I don't
see it pop up ]
1) The good news.
I received a prompt reply, and the problem is understood
"This problem got introduced when we changed two fields in GBD
structure namely GBD$W_NUMB and GBD$W_SIZE to Longword. Due to this
change GBD$Q_REL_ADDR which was previously Quadword aligned became
longword aligned"
That change was surely needed to support the increased max MBC = 255
(was 127)
An official patch is expected in Q1CY11. Request remedial image if
need be.
More important than the fix (imho) is fair warning before considering
this update.
"When using RMS global buffers ...."
2) The bad news.
- RMSDEF.STB was NOT updated
- ANALYZE/SYSTEM - RMS was NOT updated
- No update for the RMS.EXE Image file identification: "X-47"
- While the MBC chance was very nicely documented (below) it's
implications were not.
3) The worse news.
- This can cause system crashes (BUGCHECKFATAL) for those
appliactions / tools which interpret and analyze GBD structure... such
as my 'GBD' tool, on Alpha and Itanium.
If an application relies on GBD (and BDB) layout, then this is no
indication of a change.
RMS.EXE, BID/BLN stayed the same (increased size change happened to
'fit' in rounded up allocation).
The only thing to test for is the RMS.EXE image link date. :-(
fwiw... I suspect I'll go for a heuristic data value test in my code.
I already had it decide between V7 and V8 on the fly. Will have to add
an other test or two.
hth,
Hein van den Heuvel
From SYS - V12 - notes....
5 NEW FUNCTIONALITY AND/OR PROBLEMS ADDRESSED IN THE
VMS831H1I_SYS-V1200 KIT
5.1 New functionality addressed in this kit
5.1.1 MBC Maximum value increased to 255
5.1.1.1 Functionality Description:
An RMS application can specify the number of blocks
that
it wishes RMS to transfer with each I/O operation, via
the Multi Block Count (MBC) field of the RAB structure,
RAB$B_MBC.
Please refer Section 7.11 of OpenVMS Record Management
Services Reference Manual for more information on
RAB$B_MBC.
The maximum value of this field is currently 127
blocks.
With this change, the limit is now raised to 255
blocks.
For applications reading or writing sequential files,
this change helps increase I/O performance by
transferring more blocks in a single I/O operation.
This
change is applicable in all versions of VMS starting
from
V8.3.
Restriction :
If you intend to use a Multi Block Count (RAB$B_MBC)
value of more than 127 to open a file in shared mode in
a
VMS cluster, you should apply the latest RMS or SYS
patches (dated July 2010 or later) on ALL THE NODES in
the VMS cluster where the file will be shared.
JF Mezei
> "This problem got introduced when we changed two fields in GBD
> structure namely GBD$W_NUMB and GBD$W_SIZE to Longword.
isn't changing the size of fields in core OS software something that is
normally avoided at all costs ?
Did they create GBD$L_NUMB and GBD$L_SIZE overlays to map the longword
version while keeping the $W_ in situ for compatibility reasons ?
Excuse my ignorance here, but wouldn't changing such field size or field
order in RMS have major impacts on so many components of VMS and
applications that it would br frowned upon because it would require a
recompile of so many applications ?
Or is this the type of change that could be made because those symbols
are dynamically linked at image activation time ?
> That change was surely needed to support the increased max MBC = 255
> (was 127)
since both 127 and 255 comfortably fit within a word, why lenghten it to
longword ? Or do they refer to number of bytes to be transfered instead
of blocks ?
VAXman-
>Hein RMS van den Heuvel wrote:
>
>> "This problem got introduced when we changed two fields in GBD
>> structure namely GBD$W_NUMB and GBD$W_SIZE to Longword.
>
>isn't changing the size of fields in core OS software something that is
>normally avoided at all costs ?
Not necessarily. Many data structure fields have changed over recent
releases to promote them to naturally aligned locations.
>Did they create GBD$L_NUMB and GBD$L_SIZE overlays to map the longword
>version while keeping the $W_ in situ for compatibility reasons ?
Good question. The GBD is strictly an RMS concept, so if the size of
these fields was changed, they would typically only affect what is in
RMS. If there is software that referred to GBD$W_NUMB/GBD$W_SIZE, it
would need to be modified to refer to GBD$L_NUMB/GBD$L_SIZE as well as
promote word references (ex. MOVW) to longword references (ex. MOVL).
>Excuse my ignorance here, but wouldn't changing such field size or field
>order in RMS have major impacts on so many components of VMS and
>applications that it would br frowned upon because it would require a
>recompile of so many applications ?
No. I recently completed development of RMS-CDC for Attunity, I can
tell you that RMS data structures have changed release to release in
recent VMS releases and even over patches for a release. V7.2-1 was
particularly onerous with regard to the latter.
>Or is this the type of change that could be made because those symbols
>are dynamically linked at image activation time ?
The symbols are defined in SYS$LOADABLE_IMAGES:RMSDEF.STB. You can
see them if you were to create a .MAP file from the .STB.
$ LINK/NOEXE/MAP SYS$LOADABLE_IMAGES:RMSDEF.STB
Software that is linked against RMSDEF.STB would see these symbols
resolved at LINK time.
>> That change was surely needed to support the increased max MBC = 255
>> (was 127)
>
>since both 127 and 255 comfortably fit within a word, why lenghten it to
>longword ? Or do they refer to number of bytes to be transfered instead
>of blocks ?
127(x7F)*512(x200)=65024(xFE00) fits in a word; 255(xFF)*512(x200)
=130560(x1FE00) and does not.
The problem is that adding 2 bytes to each of the GBD$W_xxxx fields
upset the rest of the data structure alignment. An alignment filler
longword should have been introduced to keep the structure aligned.
Presently, a quadword field is now aligned on x4 and xC instead of
x0 and x8.
Arne Vajhøj
> 2) The bad news.
>
> - RMSDEF.STB was NOT updated
> - ANALYZE/SYSTEM - RMS was NOT updated
> - No update for the RMS.EXE Image file identification: "X-47"
> - While the MBC chance was very nicely documented (below) it's
> implications were not.
>
> 3) The worse news.
>
> - This can cause system crashes (BUGCHECKFATAL) for those
> appliactions / tools which interpret and analyze GBD structure... such
> as my 'GBD' tool, on Alpha and Itanium.
Not good.
It is a myth that everything was perfect in the old days,
but does anyone remember anything similar happened before?
Arne
Arne Vajhøj
> Hein RMS van den Heuvel wrote:
>
>> "This problem got introduced when we changed two fields in GBD
>> structure namely GBD$W_NUMB and GBD$W_SIZE to Longword.
>
> isn't changing the size of fields in core OS software something that is
> normally avoided at all costs ?
The general rule has always been:
- user mode images using documented interfaces should
continue to work forever
- the inner mode, undocumented, linked with the system etc.
could break anytime and needed to be rebuild and restested
for each VMS upgrade
> Did they create GBD$L_NUMB and GBD$L_SIZE overlays to map the longword
> version while keeping the $W_ in situ for compatibility reasons ?
I don't think that would help much. Sure people could just
rebuild, but if people were that far in the process changing
the constants before rebuild is no big thing.
> Excuse my ignorance here, but wouldn't changing such field size or field
> order in RMS have major impacts on so many components of VMS
VMS is rebuild for a new VMS rebuild anyway.
> and
> applications that it would br frowned upon because it would require a
> recompile of so many applications ?
Could be.
But it is not that shocking with a need to rebuild this
type of apps.
That does not excuse the lack of documentation of the
change.
Arne
Hein RMS van den Heuvel
> In article <4d028091$0$23318$c3e8da3$a9097...@news.astraweb.com>, JF Mezei <jfmezei.spam...@vaxination.ca> writes:
>
> >Hein RMS van den Heuvel wrote:
>
> >> "This problem got introduced when we changed two fields in GBD
> >> structure namely GBD$W_NUMB and GBD$W_SIZE to Longword.
>
that is
JF>normally avoided at all costs ?
For documented interfaces, yes. And for those compatibility is
critical.
64-bit Descriptors, and the 64-bit RMS RAB/FAB/NAM blocks are recent
examples.
There it is a goal, but not a requirement to retain compatibilty.
GBD and other RMS runtime blocks are internal use only structure.
Very internal.
It is not even defined in LIB.MLB, only in RMSINTSTR which is not
shipped.
JF>Did they create GBD$L_NUMB and GBD$L_SIZE overlays to map the
longword
JF>version while keeping the $W_ in situ for compatibility reasons ?
Yes, and that was done for several fields such as GBD$L_ALLOC_ADDR
Apparently this is was nit deemed useful for these 2 fields.
IMH0 it is better that $W_ symbols disappear as a potential warning
that the structure has changed IF the program is rebuild.
>Excuse my ignorance here, but wouldn't changing such field size or field
>order in RMS have major impacts on so many components of VMS and
Yes, that can happen but then the kit just ships many images.
For this kit OpenVMS engineering failed to rebuild SDA.
Brian> Software that is linked against RMSDEF.STB would see these
symbols
> resolved at LINK time.
Yes, but OpenVMS engineering failed to ship a matching RMSDEF.STB!
Brian> The problem is that adding 2 bytes to each of the GBD$W_xxxx
fields
> upset the rest of the data structure alignment.
Because the GBD was padded out to 128 bytes (to make sure adjacent
GBDs do not end up in a single CPU cache line), the extra used bytes
did NOT trigger a change to GBD$C_BLN so that did not help my program.
My program embedded its own GBD structure layout, because RMSINTSTR is
not exposed.
I now feel nicely vindicated to have added 'defensive' code to the
program in question.
It calls SYS$GETSYI asking for SYI$_BUGCHECKFATAL and then proceeds
with:
if (bugcheckfatal && ! quiet) {
fprintf (stderr, "GBD: Denied! SYSGEN BUGCHECKFATAL = 1. Override
with -q if you feel lucky\n");
return 16;
}
Grins,
Hein
VAXman-
>On Dec 10, 3:11=A0pm, VAXman- @SendSpamHere.ORG wrote:
>> In article <4d028091$0$23318$c3e8da3$a9097...@news.astraweb.com>, JF Meze=>> >> "This problem got introduced when we =A0changed two fields in GBD
>> >> structure namely GBD$W_NUMB =A0and GBD$W_SIZE to Longword.
Which poses even more of an issue for people like me who would read the
.STBs on the fly to resolve symbols. No corresponding .STBs; incorrect
symbol values.
>Brian> The problem is that adding 2 bytes to each of the GBD$W_xxxx
>fields
>> upset the rest of the data structure alignment. =A0
>
>Because the GBD was padded out to 128 bytes (to make sure adjacent
>GBDs do not end up in a single CPU cache line), the extra used bytes
>did NOT trigger a change to GBD$C_BLN so that did not help my program.
Yeah, that's a good thing and a bad thing. I think the BLN should hold
the structure size. Allocate/deallocate memory for them using a round-
up algorithm.
>My program embedded its own GBD structure layout, because RMSINTSTR is
>not exposed.
You don't LINK against RMSDEF.STB for this?
>I now feel nicely vindicated to have added 'defensive' code to the
>program in question.
>It calls SYS$GETSYI asking for SYI$_BUGCHECKFATAL and then proceeds
>with:
>
>if (bugcheckfatal && ! quiet) {
> fprintf (stderr, "GBD: Denied! SYSGEN BUGCHECKFATAL =3D 1. Override
>with -q if you feel lucky\n");
> return 16;
> }
>
>Grins,
>Hein
I never feel lucky!
JF Mezei
> Which poses even more of an issue for people like me who would read the
> .STBs on the fly to resolve symbols. No corresponding .STBs; incorrect
> symbol values.
You greatly disapoint me Mr VAXman. At your level of kernel hacking, I
would have expected you to know all thew .STBs by heart and not need to
read them to resolve symbols.
:-)
VAXman-
>VAXman- @SendSpamHere.ORG wrote:
>
>> Which poses even more of an issue for people like me who would read the
>> .STBs on the fly to resolve symbols. No corresponding .STBs; incorrect
>> symbol values.
>
>would have expected you to know all thew .STBs by heart and not need to
>read them to resolve symbols.
>
>:-)
I don't have to, only the software has to! :)
One of the most visible place where I do this is in my SYMBOL utility.
One can use SYMBOL's /EXECUTIVE qualifier and specify the name of some
symbol (for example, and relative to this thread, GBD$Q_REL_ADDR) and
SYMBOL will create a DCL symbol with its value.
DS10$ SYMBOL/SET/EXECUTIVE GBD$Q_REL_ADDR
DS10$ SHOW SYMBOL GBD*
GBD$Q_REL_ADDR == 32 Hex = 00000020 Octal = 00000000040
If this patch were to supply a new RMSDEF.STB, I suspect that the very
same commands above would result in outputting:
GBD$Q_REL_ADDR == 36 Hex = 000000244 Octal = 00000000044