M2.devices in X7 and how to monitor and replace faulty disks

Oracle Exadata Database Machine X7 systems comes with two internal M.2 devices that contain the system area. In all previous systems, the first two disks of the Oracle Exadata Storage Server are system disks and the portions on these system disks are referred to as the system area.

Note:

Oracle Exadata Rack and Oracle Exadata Storage Servers can remain online and available while replacing an M.2 disk.

This section contains the following topics:

• Monitoring the Status of M.2 Disks
• Replacing a M.2 Disk Due to Failure or Other Problems

Monitoring the Status of M.2 Disks

You can monitor the status of a M.2 disk by checking its attributes with the CellCLI LIST PHYSICALDISK command.

The disk firmware maintains the error counters, and marks a drive with Predictive Failure when the disk is about to fail. The drive, not the cell software, determines if it needs replacement.

• Use the CellCLI command LIST PHSYICALDISK to determine the status of a M.2 disk:

 

CellCLI> LIST PHYSICALDISK WHERE disktype='M2Disk' DETAIL
         name:                           M2_SYS_0
        deviceName:                  /dev/sdm
        diskType:                      M2Disk
         makeModel:                    "INTEL SSDSCGJK150G7"
         physicalFirmware:         N2010112
         physicalInsertTime:      2017-07-14T08:42:24-07:00
         physicalSerial:            PHDW7082000M150A
         physicalSize:               139.73558807373047G
         slotNumber:                  "M.2 Slot: 0"
         status:                failed

         name:                  M2_SYS_1        
         deviceName:            /dev/sdn
         diskType:              M2Disk
         makeModel:             "INTEL SSDSCKJB150G7"
         physicalFirmware:      N2010112
         physicalInsertTime:    2017-07-14T12:25:05-07:00
         physicalSerial:        PHDW708204SZ150A
         physicalSize:          139.73558807373047G
         slotNumber:            "M.2 Slot: 1"
         status:                normal

Replacing a M.2 Disk Due to Failure or Other Problems

Failure of a M.2 disks reduces redundancy of the system area, and can impact patching, imaging, and system rescue. Therefore, the disk should be replaced with a new disk as soon as possible. When a M.2 disk fails, the storage server automatically and transparently switches to using the software stored on the inactive system disk, making it the active system disk.

An Exadata alert is generated when an M.2 disk fails. The alert includes specific instructions for replacing the disk. If you have configured the system for alert notifications, then the alert is sent by e-mail to the designated address. M.2 disk is hot-pluggable and can be replaced when the power is on. After the M.2 disk is replaced, Oracle Exadata System Software automatically adds the new device to the system partition and starts the rebuilding process.

1. Identify the failed M.2 disk.

   CellCLI> LIST PHYSICALDISK WHERE diskType=M2Disk AND status!=normal DETAIL
            name:                      M2_SYS_0
             deviceName:              /dev/sda
             diskType:                M2Disk
             makeModel:              "INTEL SSDSCKJB150G7"
            physicalFirmware:          N2010112
            physicalInsertTime:        2017-07-14T08:42:24-07:00
            physicalSerial:            PHDW7082000M150A
            physicalSize:              139.73558807373047G
            slotNumber:                "M.2 Slot: 0"
          status:                    failed - dropped for replacement
   

2. Locate the cell that has the white LED lit.
3. Open the chassis and identify the M.2 disk by the slot number in Step 1.
4. The amber LED for this disk should be lit to indicate service is needed.
   M.2 disks are hot pluggable, so you do not need to power down the cell before replacing the disk.
5. Remove the M.2 disk:
   1. Rotate both riser board socket ejectors up and outward as far as they will go.
      The green power LED on the riser board turns off when you open the socket ejectors.
   2. Carefully lift the riser board straight up to the remove it from the sockets.
6. Insert the replacement M.2 disk:
   1. Unpack the replacement flash riser board and place it on an antistatic mat.
   2. Align the notch in the replacement riser board with the connector key in the connector socket.
   3. Push the riser board into the connector socket until the riser board is securely seated in the socket.
      Caution:

      If the riser board does not easily seat into the connector socket, verify that the notch in the riser board is aligned with the connector key in the connector socket. If the notch is not aligned, damage to the riser board might occur.
   4. Rotate both riser board socket ejectors inward until the ejector tabs lock the riser board in place.
      The green power LED on the riser board turns on when you close the socket ejectors.
7. Confirm the M.2 disk has been replaced.

   CellCLI> LIST PHYSICALDISK WHERE DISKTYPE=M2Disk DETAIL
        name:                  M2_SYS_0 
       deviceName:            /dev/sdm   
      diskType:              M2Disk   
      makeModel:             "INTEL SSDSCKJB150G7"   
      physicalFirmware:      N2010112    
      physicalInsertTime:    2017-08-24T18:55:13-07:00   
      physicalSerial:        PHDW708201G0150A   
      physicalSize:          139.73558807373047G   
      slotNumber:            "M.2 Slot: 0"   
      status:                normal   
   
      name:                  M2_SYS_1   
      deviceName:            /dev/sdn   
      diskType:              M2Disk   
      makeModel:             "INTEL SSDSCKJB150G7"    
      physicalFirmware:      N2010112   
      physicalInsertTime:    2017-08-24T18:55:13-07:00   
      physicalSerial:        PHDW708200SZ150A   
      physicalSize:          139.73558807373047G   
      slotNumber:            "M.2 Slot: 1"   
      status:                normal 
   

8. Confirm the system disk arrays are have an active sync status, or are being rebuilt.

# mdadm --detail /dev/md[2-3][4-5]
/dev/md24:
      Container : /dev/md/imsm0, member 0
     Raid Level : raid1
     Array Size : 104857600 (100.00 GiB 107.37 GB)
  Used Dev Size : 104857600 (100.00 GiB 107.37 GB)
   Raid Devices : 2
  Total Devices : 2

               State  : active
 Active Devices  : 2
Working Devices  : 2
 Failed Devices  : 0
   Spare Devices : 0  

            UUUID : 152f728a:6d294098:5177b2e5:8e0d8c6c
   Number    Major    Minor    RaidDevice    State
    1           8         16             0       active sync  /dev/sdb
    0           8           0            1       active sync  /dev/sda
/dev/md25:
      Container : /dev/md/imsm0, member 1
     Raid Level : raid1
     Array Size : 41660426 (39.73 GiB 42.66 GB)
  Used Dev Size : 41660524 (39.73 GiB 42.66 GB)
   Raid Devices : 2
  Total Devices : 2

               State  : clean
 Active Devices  : 2
Working Devices  : 2
 Failed Devices  : 0
   Spare Devices : 0  

             UUID : 466173ba:507008c7:6d65ed89:3c40cf23
   Number    Major    Minor    RaidDevice    State
 1           8         16        0      active sync  /dev/sdb
 0           8         0         1      active sync  /dev/sda

Storage Cell reports error RS-7445 [Serv MS Leaking Memory]

Bug 19790644 - RS-7445 [SERV MS LEAKING MEMORY]

Issue is a memory leak in the Java executable.

This bug affects systems running with JDK 7u51 or later versions (1.7.0_55-b13) which are 11.2.3.3.1 and 12.1.1.1.1
This is relevant for all versions 11.2.3.3.1 to 12.1.2.1.1 [Release 11.2 to 12.1]  (excluding  11.2.3.3.0 or 12.1.1.1.0)
Systems running 11.2.3.3.0 or 12.1.1.1.0 are not affected as they use 1.7.0_25-b15 

Cause:

MS process will be consuming memory (up to 2GB).  Normally MS will use around 1GB of memory but because of the bug, the memory allocated can grow upt to 2GB.

Normal memory usage:

ps -feal|grep java
0 S root     18585 13652  0  80   0 - 15319 pipe_w 15:21 pts/1    00:00:00 grep java
0 S root     27960 27958  0  80   0 - 292553 futex_ Jun17 ?       01:45:06 /usr/java/default/bin/java -Xms256m -Xmx512m -XX:-UseLargePages -Djava.library.path=/opt/oracle/cell/cellsrv/lib -Ddisable.checkForUpdate=true -jar /opt/oracle/cell/oc4j/ms/j2ee/home/oc4j.jar -out /opt/oracle/cell/cellsrv/deploy/log/ms.lst -err /opt/oracle/cell/cellsrv/deploy/log/ms.err
292553 * 4096 = 1142MB (1GB).

Larger values will indicate memory leak.
When using command pmap -x <MS process pid>,  if memory leak is still present  it will report a large number of 64 memory chunks:

Address           Kbytes     RSS   Dirty Mode   Mapping
0000000000400000       4       4       0 r-x--  java
0000000000600000       4       4       4 rw---  java
00000000019c3000   85212   83816   83816 rw---    [ anon ]
00000000dae00000   46080   45856   45856 rw---    [ anon ]
00000000ddb00000   37888       0       0 -----    [ anon ]
00000000e0000000  175104  174900  174900 rw---    [ anon ]
00000000eab00000  174080       0       0 rw---    [ anon ]
00000000f5500000   87552   87552   87552 rw---    [ anon ]
00000000faa80000   87552       0       0 -----    [ anon ]
00007f261c000000   38384   37488   37488 rw---    [ anon ]
00007f261e57c000   27152       0       0 -----    [ anon ]
00007f2624000000   58488   56628   56628 rw---    [ anon ]
00007f262791e000    7048       0       0 -----    [ anon ]
00007f262c000000   65524   65444   65444 rw---    [ anon ]
00007f262fffd000      12       0       0 -----    [ anon ]
00007f2634000000   65528   65528   65528 rw---    [ anon ]
00007f2637ffe000       8       0       0 -----    [ anon ]
00007f263c000000   65536   65528   65528 rw---    [ anon ]
00007f2644000000   65536   65360   65360 rw---    [ anon ]
00007f264c000000   65528   65520   65520 rw---    [ anon ]
00007f264fffe000       8       0       0 -----    [ anon ]
00007f2654000000   65456   65456   65456 rw---    [ anon ]

Solution:

1. Error is ignorable as MS service will be re-started automatically, which will reset the process and memory used.
2. While patching the storage cell with one-off patches is not generally recommended, if there are issues where the MS service is not automatically re-started, the JDK needs upgraded on the Storage Cell
    Use Patch 20328167: TRACKING BUG FOR JDK 1.7.0.72- B33 PATCH (wrapper for 20328167: Oracle JDK 7 Update 72 b33 or later)
3. If no other issues are being seen, the recommended action is to wait for Exadata Cell Software version 12.1.2.1.2 or later.
 

Asmcmd daemon consuming High CPU

ASMCMD commands when executed with parameters are leaving the asm connection open "asmcmd daemon" and consumes high CPU usage.

Bug 28019068 - EXADATA: ASMCMD CONSUMING HIGH CPU IN 18C

$ ps -ef | grep asmcmd | grep -v grep
root      224347      1  3 03:20 ?        00:00:00 asmcmd daemon
grid   234123      1  2 03:20 ?        00:00:00 oracle+asm_asmcmd
pstack output:

#0  0x00007f4954b00050 in __open_nocancel () from /lib64/libpthread.so.0
#1  0x00000000005622de in PerlIOUnix_open ()
#2  0x0000000000563c74 in PerlIOBuf_open ()
#3  0x0000000000565795 in PerlIO_openn ()
#4  0x000000000053b030 in Perl_do_open6 ()
#5  0x00000000005274d2 in Perl_pp_open () 
#6  0x00000000004cefad in Perl_runops_standard ()
#7  0x0000000000443622 in S_run_body ()
#8  0x000000000044350b in perl_run ()
#9  0x000000000041de78 in main ()
You can see open ( ) call and leave pool connection open.

Workaround:

1. When you kick off an ASMCMD command, it actually establishes a connection to the ASM instance. To disable connection pooling, use the --nocp parameter to the ASMCMD tool:

 $ asmcmd --nocp <parameters>

2. Use commands within ASMCMD command line instead of passing parameters to avoid  pool connections open.