SCSI reservation conflict errors with AIX, Symmetrix and Powerpath
dbaronica
The similar problem is detected in our machine to:
- RS/6000-p610 connected to the EMC Symmetrix 8430 via EMC Connectrix
switches .
- AIX 6228 FC adapters (microcode 382A1)
- AIX 5.1 with some required patches for application needs
First SC_DISK_ERR2 is detected, following the LVM_IO_FAIL, then
JFS_LOG_WRITE_ERR, JFS_FSCK_REQUIRED.
After that all is repeated several times, and results in file
system corupted.
I would like to know how did you solve the problem (if you are)?
Thanks and regards,
Damir
--
Posted via http://dbforums.com
Florian Heigl
> The similar problem is detected in our machine to:
You didn't actually notice that thread was a year old, hmmm?
> - RS/6000-p610 connected to the EMC Symmetrix 8430 via EMC Connectrix
> switches .
> - AIX 6228 FC adapters (microcode 382A1)
> - AIX 5.1 with some required patches for application needs
>
> First SC_DISK_ERR2 is detected, following the LVM_IO_FAIL, then
> JFS_LOG_WRITE_ERR, JFS_FSCK_REQUIRED.
> After that all is repeated several times, and results in file
> system corupted.
>
> I would like to know how did you solve the problem (if you are)?
At our site we didn't experience those problems, BUT
I had similar errors in our hp-ux environment on lower-end clariions.
*if* You're running in a FC-AL 'quickloop' topology, and there's a few
boxes on it, You might actually experience what is similar to
broadcast storms on good old ethernet.
The solution in our case was to
+ decrease the maximum amount of reqests per fc-path to lower the load
on the fibre channel and to
+ increase the pv timeouts to avoid corruption (*errr* right now 180s.
Do a lsattr -El on Your 6228
On my cambex adapter here, the value is
# lsattr -El scsi1 |grep exec_throttle
exec_throttle 256 Max commands per port
Formula is:
the maximum number of request the symmetrix takes per port by two in
case there's a trespass happening (I/Os will have to be re-routed) by
the number of hosts on those host adapters.
Ok, actually it's not that simple, for the switches in-between limit
the amount of traffic, but rather start careful and check if problems
cease.
Check Your microcode levels with ibm and emc^2, try the above, and
also maybe take one box to an port adapter of it's own for
cross-checking and consider Your cables broken.
Errr....
Just open a call.
Good luck.
dbaronica
We are running Point-to-Point connection to the FC Switch, not the
FC-AL loop.
What I have found is that FC 6228 in p-Series may lead to unpredictable
results including system crash, system hang, data error, unwarranted
error log entries, etc. (APAR NUMBER: IY33105) if the
devices.pci.df1000f7.com fileset is at levels below 5.1.0.28.
In our machine it is actually 5.1.0.10 - probably we need to upgrade
this, but we are still waiting for the answer from IBM Support.
In the mean time we added VCM support to the ports on EMC box, so each
machine can see only the disks attached to it. There are some ideas that
cfgmgr or snap on the other machine can cause the disk reservation on
the disks attached to the host where we encounter the problem.
We will watch if the problem is appearing again. Actually, I was trying
to see if the Gregory and Swen solved the problem and how (probably
they are).
Best regards,
Gregory Beckowski
worked with both support groups for over a year before I posted my
original message with no success. Lots of finger pointing and no
solutions, only the following recommendation.
Stop performing volume assignment operations (or anything for that
matter) that will cause a state change notification (SCN) in the
fabric during time of high I/O. For example, adding new disk-to-host
paths will cause re-zoning in the fabric which causes SCN. The SCN
causes some type of reset in the fabric.
One oddity though, we have two FC adapters in these hosts and they are
connected to two different fabrics. They are also running PowerPath.
Our expectation was that PowerPath should handle such events since it
handles complete fabric outages.
That correlation we are fairly confident of, ie, re-zoning/SCN and
high I/O causing reservation conflicts. We are quite disappointed
since we were sold on the fact we could make changes whenever but now
we limit them to off-hours because we have had databases come down too
much. Since we have implemented this policy we have not had a database
come down due to 0118 errors.
However we still do get 0118 errors at certain times. I have been
looking for a tool which will show the reservation status of a disk
(cleared, set, which WWN, etc...) so we can monitor the disks. At
this point I do not know enough to write my own so any guidance in
this regard with be appreciated.
Greg
dbaronica <membe...@dbforums.com> wrote in message news:<2920456.1...@dbforums.com>...
Destiny's Lost Child
two HBAs onto different fabrics + powerpath balancing between them)
and we came to the conclusion that the crashs were down to paging
requests timing out to external disks. This was because our systems
with only hd6 on internal disk never crashed (although they did get
SCSI_ERR2 messages).
However, this appears to only be a partial solution as we found that
removing one powerpath adapter removed made the host stable again
(sort of defeats the purpose of powerpath)
We have never had a satisfactory answer from either IBM or EMC about
whether our partial conclusion could be correct or not but we stopped
all our plans to use external boot devices.
Gregory Beckowski
the system (rootvg). I have been focusing on db problems when
SCSI_ERR2s occur but if one of the disks 'hit' was a paging disk the
system would lock or crash.
Over the past year we have upgraded all our systems (last few will be
done in 2 weeks) from AIX 4.3.3-10/PowerPath 2.0.2 to AIX
5.1.0-03/PowerPath 3.0.3. SYMCLI and EMC ODM extensions have been
upgrade also. System and adapter firmware have not been upgraded
(probably this fall). Also, come July we will be replacing our
8000-series and Brocades with DMX and McData.
With all these changes and our policy of limiting zoning changes, it
is going to be difficult or impossible to determine what the issue is
and if it was fixed in any of these upgrades.
destinys...@theironpig.com (Destiny's Lost Child) wrote in message news:<786bcb1.03060...@posting.google.com>...
dbaronica
We made some testing on other systems and here is result:
Configuration:
* RS/6000-p690 with two LPAR, 4 FC 6228 in 1st LPAR, 2FC in 2nd
* EMC DS32 SAN Switch
* 2x DMX2000, 2x8 FC ports/box, 6 for SAN, 2 for SRDF, VCM on ports
* PowerPath 3.0.3.
* AIX 5.2-ML00
Under heavy load (dd, 40-120 MB/sec) we made 'mkbcv -a' on Defined BCV
disks and almost always got SCSI_ERR_2.
After installing ML01 for AIX 5.2 we can't get that error with the same
testing. That is leading as to the solution that new ML is solving the
problem on the AIX side, but we are still checking (and testing).
The same is tested with
* RS/6000-p610, 2x6228 FC, PowerPath 3.0.3, AIX 5.1-ML00
First we got SCSI_ERR2, and after applying the ML04 for AIX 5.1 -
results is same - no errors.
Now we are testing how is Oracle 9.2.0.3. working with AIX 5.2-ML01 and
probably we will upgrade our production soon :-).
Regards,
Swen Jung
...
> > In the mean time we added VCM support to the ports on EMC box, so each
> > machine can see only the disks attached to it. There are some ideas that
> > cfgmgr or snap on the other machine can cause the disk reservation on
> > the disks attached to the host where we encounter the problem.
Hi,
I found the following critical problems with IBM FC 6227 and 6228:
- DATA MISCOMPARE ON FIBRE CHANNEL ATTACHED DEVICE,
- SYSTEM CRASH IN FIBRE CHANNEL DRIVER,
- ENSURE QUICKER I/O COMPLETION ON ADAPTER MALFUNCTION,
- IO FAILURES OR HANG WHILE DOING IO TO FIBRE CHANNEL DEVICES,
- UNEXPECTED DATA ON FIBRE CHANNEL HANDLING IOCB ERRORS,
- DATA CORRUPTION WHILE BUILDING LARGE DATABASE
(http://techsupport.services.ibm.com/server/fixes)
This AIX filesets needs to be current for to solve this issues: (But
not the SCSI reservation conflict)
- Device Drivers for 6227
devices.pci.df1000f7.rte
devices.pci.df1000f7.com
devices.pci.df1000f7.diag
- Device Drivers for 6228
devices.pci.df1000f7.com
devices.pci.df1000f9.rte
devices.pci.df1000f9.diag
- Other commonly needed filesets for 6227/6228
devices.common.IBM.fc.rte
devices.fcp.disk.rte
devices.fcp.disk.array.diag
devices.fcp.disk.array.rte
devices.fcp.tape.rte
devices.scsi.disk.rte
===============================================================================
For to solve the SCSI reservation conflict I have the following tip's:
If you are using EMC Solutions Enabler (TimeFinder, SRDF, ..), you
need Gatekeeper-Devices for the communication between the Host and the
Symmetrix.
If you have no or to few GK's, the SYMCLI will use data-devices
instead. This will cause reservation conflicts.
If you are running loops with symmir/symrdf -g <group> verify, your
loop should include a sleep for a minute or longer. Otherwise you
needs a lot of GK's.
In the SYMAPI options file, you could change the parameter
SYMAPI_WAIT_ON_LOCKED_GK to ENABLE, too.
Don't share GK's. Assign a GK to only one Host-Adapter ! To use two
Host-Adapters for a GK could cause reservation conflicts.
If you are running EMC Solutions Enabler, you should upgrade to
Version 5.0.3 or 5.1.1.
-------------------------------------------------------------------------------
You NEED to upgrade PowerPath for AIX to Version 3.0.3 !!!
Upgrade the EMC ODM Definitions for AIX 4.x to Version 4.3.3.4, too.
Version 5.0.0.0 for AIX 5.x has not been updated till now.
-------------------------------------------------------------------------------
If you have any HACMP System connected to the Symmetrix, you must
configure the files: "/etc/cluster/lunreset.lst" and
"/etc/cluster/disktype.lst" to issue a LUN reset !!!
Otherwise the default target reset for OEM Disks will be issued. This
means: The whole FA will receive a target reset and ALL disk
reservations for ALL other hosts will be cleared, too !!
(For more Info's refer to IBM's HACMP Installation and Administration
Guide, Appendix G: OEM Disk Accommodation)
-------------------------------------------------------------------------------
Upgrade the following Microcodes / Firmware to the current Levels:
- Fibre Channel Adapter
- System/Service-Processor
- Switch
- EMC Symmetrix
-------------------------------------------------------------------------------
I have seen reservation conflicts in case of the following hardware
problems, too:
- SC/LS Adapters are in use
- bad cable
- Adapter will be defect, soon.
- GBIC will be defect, soon.
Regards,
Swen
Swen Jung
PowerPath 3.0.4 for AIX and Solutions Enabler 5.2 are available....
Regards,
Swen
Gregory Beckowski
We do symmir loops (and sleeps 60+ seconds) to wait for sync'ng and
splitting disks. And at one point we had no GKs because we were told
we did not need them. Now we have about 6 GKs per host, not per HBA.
I never understood why the SYMCLI commands need to clear a
reservation in order to access their data in the Symm.
We also ran in to the HACMP and LUN reset vs. FA reset issue. We
never setup those cluster OEM files (until recently) and in the past
our HACMP detail log shows the SCSI command sent was a LUN reset not a
BUS reset.
But there is still no way of directly checking the reservation state
of a disk. All these settings and upgrades and test do not directly
monitor how the reservations are being managed by the initiator
(Powerpath/FC driver/HBA) and the target (Symmetrix software).
Looking at the SCSI spec only seems to show an indirect method, ie,
try to get the reservation and the return code will tell you if you
have or some other HBA does.
Greg
ChS....@t-online.de (Swen Jung) wrote in message news:<fbc62f87.03060...@posting.google.com>...
Swen Jung
with EMC SolutionsEnabler you can check if any Symmetrix devices are
reserved by AIX.
First, I have created and activated the VG 'testvg':
#> lsvg -p testvg
testvg:
PV_NAME PV STATE TOTAL PPs FREE PPs FREE
DISTRIBUTION
hdiskpower0 active 259 259
52..52..51..52..52
hdiskpower1 active 259 259
52..52..51..52..52
With 'symdev -sid XXX -resv list' you are able to see all reservations
in the Symmetrix:
#> symdev -sid 739 -resv list
Symmetrix ID : 000182504739
Device Name Device Reservation
---------------------------- -------------
----------------------------
Physical Sym Config Initiator SA :P Type
---------------------------- -------------
----------------------------
/dev/rhdiskpower0 0006 RDF1+Mir 00 01A:0 Group
/dev/rhdisk7 0006 RDF1+Mir 00 12A:0 Group
/dev/rhdiskpower1 000E RDF1+Mir 00 01A:0 Group
/dev/rhdisk8 000E RDF1+Mir 00 12A:0 Group
When you varyoff the VG the reservations will be cleared by your node:
#> varyoffvg testvg
#> symdev -sid 739 -resv list
Symmetrix ID : 000182504739
No Symmetrix devices were found to have reservations.
(You can use 'sympd -sid XXX -resv' list to check only the hdisks
visible to the node running that command.)
Best Regards,
Swen
gbeck...@rohmhaas.com (Gregory Beckowski) wrote in message news:<35e7709.03061...@posting.google.com>...
Gregory Beckowski
No physical devices were found to have reservations.
I just found an EMC knowledge base article (emc13776) which states
that the resv option can only see exclusive reservations. If
PowerPath is installed (which all our systems have), PowerPath changes
the reservation to a group reservation and SYMCLI cannot see those.
The versions referenced are older (PPath 2.0.2 and SYMCLI 4.x)
Your output shows power devices in the resv output though.
What versions are in your environment, ours is:
AIX 5.1-03
SYMCLI V5.2.0.0 (Edit Level: 425)
PowerPath 3.0.3
microcode 5568
I just ran symdev -resv list on a large number of our boxes and it
says no reservations found.
Greg
ChS....@t-online.de (Swen Jung) wrote in message news:<fbc62f87.03062...@posting.google.com>...