Field Notice: FN - 63380 - ASR9000 DC Power Module Replacement Recommended to Prevent Failure of PEM on Lightly Loaded Chassis - Upgrade Program Available

[Minneapolis Cisco dealers]

November 16, 2011

---

NOTICE:

THIS FIELD NOTICE IS PROVIDED ON AN "AS IS" BASIS AND DOES NOT IMPLY ANY KIND OF GUARANTEE OR WARRANTY, INCLUDING THE WARRANTY OF MERCHANTABILITY. YOUR USE OF THE INFORMATION ON THE FIELD NOTICE OR MATERIALS LINKED FROM THE FIELD NOTICE IS AT YOUR OWN RISK. CISCO RESERVES THE RIGHT TO CHANGE OR UPDATE THIS FIELD NOTICE AT ANY TIME.

Revision History

Revision	Date	Comment
1.0	16-NOV-2011	Initial Public Release

Products Affected

Products Affected	Top Assembly Part #
A9K-1.5KW-DC(=)	341-0337-01, 341-0337-02
A9K-2KW-DC(=)	341-0325-01, 341-0325-02

Problem Description

1. DC power module might fail during power-on in an ASR9000 chassis operating under a low-load condition, or it might cause one or more adjacent power modules to fail if one component in the circuit fails. This issue can defeat the redundant power system in the ASR9000 chassis, resulting in a single point of failure.
2. Once a single DC power feed fails, the expected behavior is that the DC power module generates a power feed fail alarm and the power LED turns off in order to notify the user to take corrective action. The issue identified here is that the DC power module might take longer or it might not generate a power feed failure alarm or event (%PLATFORM-PWRMON-2-MODULE_FAILURE). Also, the power status LED might remain lit for a failed DC power feed. The power module continues to provide DC power to the router via the other working DC power feed and does not affect router functionality.

Note: The two symptoms described above are not safety issues. ASR9000 AC power modules are not affected.

Background

1. Multiple power modules in a router convert battery power 48V DC to power line cards. Conversion is done by switching the DC power passing through FET (Field Effect Transistor) and controlling the drive current to FET using a micro-controller at high speed or frequency.
   Incorrect FET drive settings in the power module firmware lead to current imbalance in low-load conditions. A low load condition is defined as any one power module sourcing (outputting) less than 10 ampere of current. This output can be determined by monitoring power module output using a system CLI command. The condition is more likely to occur with frequent power cycles.
   To fix this issue a new version (-02) of the DC power module has been implemented that includes the following:
   1. A new version of the power module's internal firmware has been incorporated to correct the FET drive setting which led to a current imbalance in low-load condition. The power module firmware change is not field upgradable.
   2. Design enhancements have been made to the DC power module in order to improve long term reliability by implementing changes that support the D-share model of load balancing. 
      
      
2. The ASR9000 router features two power input feeds (A and B) to each power module. In few power breaker configurations, the input power feeds (either A or B) to multiple modules that are tied together at a common switch or breaker. If power is disrupted at the breaker to any one feed, the power failure detection and LED circuit does not discharge fast enough, which keeps the LED lit and prevents power failure alarm.
   To fix this issue a new version (-03) of the DC power module has been implemented that includes changes to the-02 version update to fix issue 1 above plus the following:
   1. Firmware update to change DC PEM parameter setting.
   2. The fix from -02 version to -03 also includes HW design improvement.

In summary, improvements to address issue 1 are incorporated with TAN 341-0325-02 and newer versions. Improvements to address issue 2 are incorporated with TAN 341-0325-03 and newer versions.

Problem Symptoms

1. One or more DC Power module might not provide power to the ASR9000 router on power up.
2. After a single power feed failure, the %PLATFORM-PWRMON-2-MODULE_WARNING syslog alarm or event might not be generated, and/or the power LED might remain lit. There is no functional impact to the power module or router.

Here is an example of the alarm or event with power feed B disrupted:

RP/0/RSP0/CPU0:Apr 19 13:13:46.688 BST: pwrmon[289]: 
   %PLATFORM-PWRMON-2-MODULE_WARNING :
   Power-module 0/PM5/SP warning condition raised
RP/0/RSP0/CPU0:Apr 19 13:13:46.688 BST: pwrmon[289]: 
   %PLATFORM-PWRMON-2-MODULE_WARNING_REASON :
   Power-module 0/PM5/SP warning condition reason: 
   Module feeder-B input voltage is missing or outside design limits 

Workaround/Solution

All 2KW (TAN:341-0325-01) and 1.5KW (TAN:341-0337-01) DC power modules shipped from February 2009 to January 31st 2011 might be subject to these failures. Cisco recommends proactively replacing all -01 versions of these power modules. 

For customers with 2KW (TAN:341-0325-02) and 1.5KW (TAN:341-0337-02) DC power modules, Cisco recommends taking no action at this time and replacing only if the problem symptom observed is no power feed failure alarm or power LED remaining lit for an extended time. 

Cisco plans to ship 2KW (TAN:341-0325-03) and 1.5KW (TAN:341-0337-03) versions (or newer) to customers as part of FN replacement. Version -02 and version -03 DC power module can coexists in the same router. 

Use the How To Identify Hardware Levels section to locate the PID, serial number, and TAN/version of the power module. Use the following table below to identify suspect power module and corrective action. 

Part Number or TAN	Steps	Action
341-0337-01 (1.5KW ASR9K DC power module)	Check Serial Number Validation tool	If affected, request upgrade by submitting the form in the Upgrade Program section of this document.
341-0325-01 (2KW ASR9K DC power module)	Check Serial Number Validation tool	If affected, request upgrade by submitting the form in the Upgrade Program section of this document.
341-0337-02 (1.5KW ASR9K DC power module)	No check required	Action required only if no power feed fail alarm or event is generated and/or the power LED remains lit after the power feed failure.**
341-0325-02(2KW ASR9K DC power module )	No check required	Action required only if no power feed fail alarm or event is generated and/or the power LED remains lit after the power feed failure.**
341-0337-02 with Deviation D119654 or 341-0337-03 or higher (1.5KW ASR9K DC power module)	No check required	Not affected. No action required.
341-0325-02 with Deviation D119654 or 341-0325-03 or higher (2KW ASR9K DC power module)	No check required	Not affected. No action required.

** ASR9000 router operation not affected. 

Note: The Serial Number Validation (SNV) tool is updated on monthly basis. If the SNV tool marks the power module as affected but with part number -02 with deviation D119654 or -03, the DC power module is good, and no replacement or upgrade is required. 
No Cisco internal DC power module will be allowed for replacement via this upgrade program. 

For information on replacing the DC power module, refer to the Removing and Replacing AC or DC Power System Components section of the Cisco ASR 9000 Series Aggregation Services Router Hardware Installation Guide.

How To Identify Hardware Levels

The HW PID, version/TAN, and serial number can be identified using one of these methods:

1. Physical inspection
   - OR -
2. Command line interface (CLI) commands
   

Note: Deviation is identified only by physical inspection of the label.

1. Physical inspection:

   A picture of suspect A9K-2KW-DC power module is shown below. See PID, serial number, TAN and deviation location in photographs below for A9K-2KW-DC power module (A9K-1.5KW-DC Power module is similar, except TAN is 321-0337-01).
   
   
   
   Note: A few A9K-2KW-DC power module might be incorrectly labeled as A2K-2KW-DC power module. Use the PID information printed on the faceplate of the PEM and the serial number and TAN from the label.

2. CLI commands:
   
   The PID, serial number, and TAN/version is obtained by running the show inventory and show diag commands with administrator privileges.
   
   Here is an example of the show inventory command for the 1.5KW DC power module:
   
   
   
   Here is an example of show inventory command for the 2KW DC power module:
   
   
   
   Here is an example of the show diagnostic command for the 1.5KW DC power module:
   
   

Upgrade Program

A9K-2KW-DC=, A9K-1.5KW-DC=

If you want to check status on a previously booked upgrade, please refer to the Status Tool (Please note: you must have a sales order number in addition to a CCO User ID and Password to access this site):http://tools.cisco.com/qtc/status/tool/action/LoadOrderQueryScreen For questions about an upgrade order already submitted, please send an email with your order number in the subject line to:upgrades...@cisco.com You will receive a response within 24 hours Monday-Friday not including US Holidays. Note: Fields marked with an asterisk (*) are required fields. Requestor Information *Name *E-mail Address TAC SR Number Customer Shipping Information *Company *Address Address_line2 *City *State/Province *ZIP/Postal Code *Country  Select a Country AFGHANISTAN ALBANIA ALGERIA AMERICAN SAMOA ANDORRA ANGOLA ANGUILLA ANTARCTICA ANTIGUA ARGENTINA ARMENIA ARUBA AUSTRALIA AUSTRIA AZERVBAIJAN BAHAMAS BAHRAIN BANGLADESH BARBADOS BELARUS BELGIUM BELIZE BENIN BERMUDA BHUTAN BOLIVIA BOSNIA BOTSWANA BOUVET ISLAND BRAZIL BRITISH INDIES BRUNEI DARUSSALAM BULGARIA BURKINA FASO BURUNDI CAMBODIA CAMEROON CANADA CAPE VERDE CAYMAN ISLANDS CENTRAL AFRICA CHAD CHILE CHINA CHRISTMAS ISLAND COCOS COLOMBIA COMOROS CONGO CONGO, THE DEMOCRATIC REPUBLIC OF THE COOK ISLANDS COSTA RICA COTE D IVOIRE CROATIA CUBA CYPRUS CZECH REPUBLIC DENMARK DIJIBOUTI DOMINICA DOMINICAN REPUBLIC ECUADOR EGYPT EL SALVADOR EQUATORIAL ERITREA ESTONIA ETHIOPIA FALKLAND ISLANDS FAROE ISLANDS FIJI FINLAND FRANCE FRENCH GUIANA FRENCH POLYNESIA FRENCH SOUTHERN TERR GABON GAMBIA GEORGIA GERMANY GHANA GIBRALTAR GREECE GREENLAND GRENADA GUADELOUPE GUAM GUATEMALA GUINEA GUINEA-BISSAU GUYANA HAITI HEARD ISLAND HONDURAS HONG KONG HUNGARY ICELAND INDIA INDONESIA IRAN, ISLAMIC REPUBLIC OF IRAQ IRELAND ISRAEL ITALY JAMAICA JAPAN JORDAN KAZAKHSTAN KENYA KIRIBATI KOREA, DEMOCRATIC PEOPLE'S REPUBLIC OF KOREA, REPUBLIC OF KUWAIT KYRGYZSTAN LAO PEOPLE LATVIA LEBANON LESOTHO LIBERIA LIBYAN ARAB LIECHTENSTEIN LITHUANIA LUXEMBOURG MACAO MACEDONIA, THE FORMER YUGOSLAV REPUBLIC OF MADAGASCAR MALAWI MALAYSIA MALDIVES MALI MALTA MARSHALL ISLAND MARTINQUE MAURITANIA MAURITIUS MAYOTTE MEXICO MICRONESIA, FEDERATED STATES OF MOLDOVA MONACO MONGOLIA MONTSERRAT MOROCCO MOZAMBIQUE MYANMAR NAMBIA NAURU NEPAL NETHERLANDS NETHERLANDS ANTILLES NEW CALEDONIA NEW ZEALAND NICARAGUA NIGER NIGERIA NIUE NORFOLK ISLAND NORTHERN COMMONWEALTH NORWAY OMAN PAKISTAN PALAU PALESTINIAN TERRITORY,OCCUPIED PANAMA PAPUA NEW GUINEA PARAGUAY PERU PHILIPPINES PITCAIRN POLAND PORTUGAL PUERTO RICO QATAR REUNION ROMANIA RUSSIAN FEDERATION RWANDA SAINT HELENA SAINT KITTS SAINT LUCIA SAINT PIERRE SAINT VINCENT SAMOA SAN MARINO SAO TOME SAUDI ARABIA SENEGAL SERBIA AND MONTENEGRO SEYCHELLES SIERRA LEONE SINGAPORE SLOVAKIA SLOVENIA SOLOMON ISLAND SOMALIA SOUTH AFRICA SOUTH GEORGIA AND THE SOUTH SANDWICH ISLANDS SPAIN SRI LANKA SUDAN SURINAME SVALBARD AND JAN MAYEN SWAZILAND SWEDEN SWITZERLAND SYRIAN ARAB TAIWAN TAJIKISTAN TANZANIA THAILAND TIMOR-LESTE TOGO TOKELAU TONGA TRINIDAD AND TOBAGO TUNISIA TURKEY TURKMENISTAN TURKS AND CAICOS TUVALU UGANDA UKRAINE UNITED ARAB EMIRATES UNITED KINGDOM UNITED STATES UNITED STATES MINOR OUTLYING ISLANDS URUGUAY UZBEKISTAN VANUATU VATICAN CITY VENEZUELA VIET NAM VIRGIN ISLANDS, BRITISH VIRGIN ISLANDS, U.S. WALLIS AND FUTUNA WESTERN SAHARA YEMEN ZAMBIA ZIMBABWE  Product Product *Quantity *Serial# 2 A9K-2KW-DC= A9K-1.5KW-DC= Customer Contact Information *First Name *Last Name *Phone 1 Ext.  Fax 1 Ext.  *E-Mail   Please use the following format: us...@domain.com *Upgrade Order Reference Number   Please provide a number that you can use when inquiring about order status Notes 1 For phone and fax, include 011 and the country code outside North America. 2 The serial number input field for each Product ID can hold up to 4,000 characters, including commas and white space. For longer lists of serial numbers, please submit additional requests. 3 For customers in Japan only *** please enter the building and the floor in the address field. Also, enter the contact person's name, the telephone number and the e-mail address in the appropriate fields..

For More Information

If you require further assistance, or if you have any further questions regarding this field notice, please contact the Cisco Systems Technical Assistance Center (TAC) by one of the following methods:

• Open a service request on Cisco.com
• By email
• By telephone