Switch Port xx is down (AutomaticBadSpeedOrWidth)

How to enable an Infiniband SUN DCS 36p switch port when it 's auto-disabled (Doc ID 1605955.1) Applies to: Sun Network QDR InfiniBand Gateway Switch - Version All Versions and later Exadata Database Machine X2-2 Hardware - Version All Versions and later Exadata X3-2 Hardware - Version All Versions and later All Platforms autodisable feature has been introduced only in firmware 2.1 and does not apply to firmware 1.3. Correct way to account for this is to check and ensure whether any auto-disabled ports exist and if present then re-enable using enableswitchport --automatic 'before' up/downgrading fw to a different version. This will ensure compatible settings when moving between different fw. autodisable feature is in the 2.x series and should be in the /conf/disabledport.conf if autodisable is set for this port. This should not be here for the 1.x series and looks like a artifact from the downgrade Our internal team is looking at how to handle this in the future Note: Using ibportstate to administer a port will not be preserved through subsequent reboots, hence should not be relied on. You should always use en(dis)ableswitchport. since /conf/disabledports.conf does not apply to 1.3, you can safely delete it for now and issue a switch reset command 'ibdevreset Switch' or simply reboot the switch. for switch with version 2.1.3-4 run enableswitchport --automatic command and port number Symptoms   listlinkup show that port 29 is autodisabled. # listlinkup Connector  4B Present <-> Switch Port 27 is up (Enabled) Connector  5B Present <-> Switch Port 29 is down (AutomaticBadSpeedOrWidth) <--------------- Connector  6B Present <-> Switch Port 36 is up (Enabled)   Changes To enable the port , you first need to determine the firmware version that 's running on the switch using the "version" command. I included below the output for version: 2.1.3-4 and version: 1.3.3-2 which is supported as per the latest release notes.   version SUN DCS 36p version: 2.1.5-1 version SUN DCS 36p version: 2.1.3-4 version SUN DCS 36p version: 1.3.3-2    Cause The issue can be caused by a bad partner link or misconfiguration  Solution  For version 1.3.3-2 ,   2.1.3-4 and above , make sure the file /conf/disabledports.conf doesn't exist . # version SUN DCS 36p version: 1.3.3-2 Build time: Apr  4 2011 11:15:19 SP board info: Manufacturing Date: 2011.12.25 Serial Number: "NCD7K0152" Hardware Revision: 0x0006 Firmware Revision: 0x0000 BIOS version: SUN0R100 BIOS date: 06/22/2010 # getportstatus 29 Port status for connector 3B Switch Port 29 Adminstate:......................Disabled (AutomaticBadSpeedOrWidth) LinkWidthEnabled:................1X or 4X LinkWidthSupported:..............1X or 4X LinkWidthActive:.................4X LinkSpeedSupported:..............2.5 Gbps or 5.0 Gbps or 10.0 Gbps LinkState:.......................Down PhysLinkState:...................Disabled LinkSpeedActive:.................2.5 Gbps LinkSpeedEnabled:................2.5 Gbps or 5.0 Gbps or 10.0 Gbps   This below command works on version: 2.1.3-4 but not on version: 1.3.3-2 # enableswitchport --reason=AutomaticBadSpeedOrWidth 29 Invalid reason AutomaticBadSpeedOrWidth Usage: enableswitchport [--reason=reason] connector | [ibdevicename] port Values for ibdevicdename: Switch Values for port: 1-36 Values for connector: 0A-17A, 0B-17B Values for reason: Blacklist, Partition   Somehow /conf/disabledports.conf had these settings.  Only possibility I can think of is this system had a fw with the autodisable feature with 2 ports that had this condition and then the system was downgraded and this file did not get updated.   the file should be renamed or removed completely. cat /conf/disabledports.conf # List of Disabled ports # Format: # ibdev port Adminstate #Switch 34 AutomaticBadSpeedOrWidth #Switch 29 AutomaticBadSpeedOrWidth after removing or renaming the file /conf/disabledports.conf, enable the port as follow enableswitchport Switch 29 getportstatus 29 Port status for connector 5B Switch Port 29 Adminstate:......................Enabled LinkWidthEnabled:................1X or 4X LinkWidthSupported:..............1X or 4X LinkWidthActive:.................1X LinkSpeedSupported:..............2.5 Gbps or 5.0 Gbps or 10.0 Gbps LinkState:.......................Active PhysLinkState:...................LinkUp LinkSpeedActive:.................10.0 Gbps LinkSpeedEnabled:................2.5 Gbps or 5.0 Gbps or 10.0 Gbps ibportstate 29 query PortInfo: # Port info: Lid 29 port 0 LinkState:.......................Active PhysLinkState:...................LinkUp LinkWidthSupported:..............1X or 4X LinkWidthEnabled:................1X or 4X LinkWidthActive:.................4X LinkSpeedSupported:..............2.5 Gbps or 5.0 Gbps or 10.0 Gbps LinkSpeedEnabled:................2.5 Gbps or 5.0 Gbps or 10.0 Gbps LinkSpeedActive:.................10.0 Gbps #listlinkup Connector  4B Present <-> Switch Port 27 is up (Enabled) Connector  5B Present <-> Switch Port 29 is up (Enabled) Connector  6B Present <-> Switch Port 36 is up (Enabled)   to disable the port again you can use disableswitchport switch 29 Now for version SUN DCS 36p version: 2.1.3-4 and above # listlinkup Connector  5B Present <-> Switch Port 29 is down (AutomaticBadSpeedOrWidth) <---------- # enableswitchport  --automatic Switch 29 Enable connector 5B Switch port 29 Adminstate:......................Enabled LinkWidthEnabled:................1X or 4X LinkWidthSupported:..............1X or 4X LinkWidthActive:.................4X LinkSpeedSupported:..............2.5 Gbps or 5.0 Gbps or 10.0 Gbps LinkState:.......................Down PhysLinkState:...................PortConfigurationTraining LinkSpeedActive:.................2.5 Gbps LinkSpeedEnabled:................2.5 Gbps or 5.0 Gbps or 10.0 Gbps NeighborMTU:.....................2048 OperVLs:.........................VL0-7 #  ibportstate 3 29 PortInfo: # Port info: Lid 3 port 29 LinkState:.......................Down PhysLinkState:...................Polling LinkWidthSupported:..............1X or 4X LinkWidthEnabled:................1X or 4X LinkWidthActive:.................4X LinkSpeedSupported:..............2.5 Gbps or 5.0 Gbps or 10.0 Gbps LinkSpeedEnabled:................2.5 Gbps or 5.0 Gbps or 10.0 Gbps LinkSpeedActive:.................10.0 Gbps #  ibportstate 3 29 PortInfo: # Port info: Lid 3 port 29 LinkState:.......................Down PhysLinkState:...................Polling LinkWidthSupported:..............1X or 4X LinkWidthEnabled:................1X or 4X LinkWidthActive:.................4X LinkSpeedSupported:..............2.5 Gbps or 5.0 Gbps or 10.0 Gbps LinkSpeedEnabled:................2.5 Gbps or 5.0 Gbps or 10.0 Gbps LinkSpeedActive:.................10.0 Gbps # #  ibportstate 3 29 PortInfo: # Port info: Lid 3 port 29 LinkState:.......................Active PhysLinkState:...................LinkUp LinkWidthSupported:..............1X or 4X LinkWidthEnabled:................1X or 4X LinkWidthActive:.................4X LinkSpeedSupported:..............2.5 Gbps or 5.0 Gbps or 10.0 Gbps LinkSpeedEnabled:................2.5 Gbps or 5.0 Gbps or 10.0 Gbps LinkSpeedActive:.................2.5 Gbps Peer PortInfo: # Port info: Lid 3 DR path slid 65535; dlid 65535; 0,29 port 2 LinkState:.......................Active PhysLinkState:...................LinkUp LinkWidthSupported:..............1X or 4X LinkWidthEnabled:................1X or 4X LinkWidthActive:.................4X LinkSpeedSupported:..............2.5 Gbps or 5.0 Gbps or 10.0 Gbps LinkSpeedEnabled:................2.5 Gbps or 5.0 Gbps or 10.0 Gbps LinkSpeedActive:.................2.5 Gbps ibwarn: [2687] validate_speed: Peer ports operating at active speed 1 rather than  4 (10.0 Gbps) [root@scam07sw-ibb0 IBdata]# ibportstate 3 29 PortInfo: # Port info: Lid 3 port 29 LinkState:.......................Down PhysLinkState:...................Disabled LinkWidthSupported:..............1X or 4X LinkWidthEnabled:................1X or 4X LinkWidthActive:.................4X LinkSpeedSupported:..............2.5 Gbps or 5.0 Gbps or 10.0 Gbps LinkSpeedEnabled:................2.5 Gbps or 5.0 Gbps or 10.0 Gbps LinkSpeedActive:.................2.5 Gbps [root@scam07sw-ibb0 IBdata]# This might explain why the port got disabled in the first place. You have an SDR link speed on peer port instead of QDR. It could be either a cable or HCA problem.  try switching cables?

Negative Values of USABLE_FILE_MB

When you lose multiple disks from multiple failure groups, then you could lose both the primary and the redundant copies of your data. In addition, if you do not have enough capacity to restore redundancy, then Oracle ASM can continue to operate. However, if another disk fails, then the system may not be able to tolerate additional failures.
 
The V$ASM_DISKGROUP view contains the following columns that contain information to help you manage capacity:
 
SQL> SELECT name, type, total_mb, free_mb, required_mirror_free_mb,
     usable_file_mb FROM V$ASM_DISKGROUP;

NAME         TYPE     TOTAL_MB    FREE_MB REQUIRED_MIRROR_FREE_MB USABLE_FILE_MB
------------ ------ ---------- ---------- ----------------------- --------------
DATA         NORMAL      4194304   189923328   398204   2260992  
-931394

 

• REQUIRED_MIRROR_FREE_MB indicates the amount of space that must be available in a disk group to restore full redundancy after the worst failure that can be tolerated by the disk group without adding additional storage. This requirement ensures that there are sufficient failure groups to restore redundancy. Also, this worst failure refers to a permanent failure where the disks must be dropped, not the case where the disks go offline and then back online.
  The amount of space displayed in this column takes the effects of mirroring into account. The value is computed as follows:
• A normal redundancy disk group with more than two REGULAR failure groups
  The value is the total raw space for all of the disks in the largest failure group. The largest failure group is the one with the largest total raw capacity. For example, if each disk is in its own failure group, then the value would be the size of the largest capacity disk.
• A high redundancy disk group with more than three REGULAR failure groups
  The value is the total raw space for all of the disks in the two largest failure groups. 
• USABLE_FILE_MB indicates the amount of free space, adjusted for mirroring, that is available for new files to restore redundancy after a disk failure. USABLE_FILE_MB is computed by subtracting REQUIRED_MIRROR_FREE_MB from the total free space in the disk group and then adjusting the value for mirroring
• TOTAL_MB is the total usable capacity of a disk group in megabytes. The calculations for data in this column take the disk header overhead into consideration. The disk header overhead depends on the number of Oracle ASM disks and Oracle ASM files. This value is typically about 1% of the total raw storage capacity.
• FREE_MB is the unused capacity of the disk group in megabytes, without considering any data imbalance.

USABLE_FILE_MB calculation:
(FREE_MB - REQUIRED_MIRROR_FREE_MB) / 2 = USABLE_FILE_MB => Normal redundancy
(FREE_MB - REQUIRED_MIRROR_FREE_MB) / 3 = USABLE_FILE_MB => High redundancy

Due to the relationship between FREE_MB, REQUIRED_MIRROR_FREE_MB, and USABLE_FILE_MB, USABLE_FILE_MB can become negative. Although this is not necessarily a critical situation, it does mean that:

• Depending on the value of FREE_MB, you may not be able to create new files.
• The next failure might result in files with reduced redundancy.

If USABLE_FILE_MB becomes negative, it is strongly recommended that you add more space to the disk group as soon as possible.