Spectre / Meltdown - Intel processor vulnerabilities [CVE-2017-5715 CVE-2017-5753 CVE-2017-5754]

Oracle Exadata Database Machine with respect CVE-2017-5753 (Spectre v1), CVE-2017-5715 (Spectre v2), and CVE-2017-5754 (Meltdown) Intel processor vulnerabilities Status:

Patch Availability Table

Affected Products	Patch Availability
	CVE-2017-5715	CVE-2017-5753	CVE-2017-5754
Oracle Exadata Database Machine (compute nodes/storage servers)	18.1.5.0.0 12.2.1.1.7 Note 1 Note 5 Note 6	18.1.4.0.0 12.2.1.1.6	18.1.4.0.0 12.2.1.1.6
Sun Data Center InfiniBand Switch 36 (NM2-36P)	Not Required Note 2	Not Required Note 2	Not Required Note 2
Cisco Catalyst C4948/C4948E-F-S	Not Required Note 3	Not Required Note 3	Not Required Note 3
Cisco Nexus 93108TC-EX	May be Required Note 4	May be Required Note 4	May be Required Note 4

Note 1: Install updated ACFS drivers before updating Exadata database servers to >= 18.1.5.0.0 or >= 12.2.1.1.7, and use the most recent patchmgr/dbserver.patch (Patch 21634633 version 18.1.5.0.0 / 5.180529 or later) to perform the database server updates.  Updated ACFS drivers are needed whether or not ACFS is in use in order for the most efficient CVE-2017-5715 mitigation to be in place once >= 18.1.5.0.0 or >= 12.2.1.1.7 is installed.  See section Required ACFS Driver Updates below for details.
Note 2: The InfiniBand Switch component (NM2-36P) is not currently believed to be impacted by these vulnerabilities.

Required ACFS Driver Updates

Install updated ACFS drivers before updating Exadata database servers to >= 18.1.5.0.0 or >= 12.2.1.1.7. Note the following:

1. Updated ACFS drivers are needed for the most efficient CVE-2017-5715 mitigation to be in place once >= 18.1.5.0.0 or >= 12.2.1.1.7 is installed.
2. Updated ACFS drivers are needed whether or not ACFS is in use.
3. In an OVM configuration dom0 may be updated to >= 18.1.5.0.0 or >= 12.2.1.1.7 before the ACFS drivers are updated in domU.

Updated ACFS drivers are available with the following:

• Grid infrastructure 18.3.0.0.180717 and later version 18 quarterly updates
• Grid infrastructure 12.2.0.1.180717 and later version 12.2.0.1 quarterly updates
• Grid infrastructure 12.1.0.2.180831 and later version 12.1.0.2 quarterly updates
• Grid infrastructure 12.1.0.2.180717 plus patch 23312691

How to Verify Proper Mitigation


To verify your systems have the proper mitigations in place after the ACFS driver and Exadata updates are performed, run the following command:

$ grep . /sys/devices/system/cpu/vulnerabilities/*

The expected output depends on the hardware version, server type, and system configuration.  If the output shown on your system does not match 

X7 hardware - storage server and database server (non-OVM and domU)

Expected vulnerability mitigation output:

$ grep . /sys/devices/system/cpu/vulnerabilities/*
/sys/devices/system/cpu/vulnerabilities/meltdown:Mitigation: PTI
/sys/devices/system/cpu/vulnerabilities/spectre_v1:Mitigation: lfence
/sys/devices/system/cpu/vulnerabilities/spectre_v2:Mitigation: IBRS, IBRS_FW, IBPB

X6 and earlier hardware - storage server and database server (non-OVM and domU)

Expected vulnerability mitigation output:

$ grep . /sys/devices/system/cpu/vulnerabilities/*
/sys/devices/system/cpu/vulnerabilities/meltdown:Mitigation: PTI
/sys/devices/system/cpu/vulnerabilities/spectre_v1:Mitigation: lfence
/sys/devices/system/cpu/vulnerabilities/spectre_v2:Mitigation: Full generic retpoline, IBRS_FW, IBPB

All hardware - database server (dom0)

Expected vulnerability mitigation output:

# grep . /sys/devices/system/cpu/vulnerabilities/*

/sys/devices/system/cpu/vulnerabilities/meltdown:Vulnerable

/sys/devices/system/cpu/vulnerabilities/spectre_v1:Mitigation: lfence

/sys/devices/system/cpu/vulnerabilities/spectre_v2:Mitigation: Full generic retpoline

Note that the output for dom0 incorrectly indicates the system is vulnerable to Meltdown (CVE-2017-5754).  Exadata database servers configured with OVM use the Xen hypervisor and guest VMs in HVM mode.  Xen with guests in HVM mode is not vulnerable to Meltdown.

Troubleshooting unexpected mitigation

If Spectre v2 mitigation does not match the expected output shown above, then confirm "imageinfo -ver" indicates the installed Exadata versions is >= 18.1.5.0.0 or >= 12.2.1.1.7.  If the proper Exadata version is installed, then follow these guidelines:
Storage servers
This situation should never occur on storage servers.  Contact Oracle Support.

Database servers (non-OVM and domU)
Perform the following verification steps on each database server:

1. Verify the patch containing the updated ACFS drivers has been installed in the grid infrastructure home.  If the following OPatch command does not return output then the ACFS patch is not installed. 
   
   $ <gihome>/grid/OPatch/opatch lsinventory -bugs_fixed | grep ^27463879
   27463879 27463879 Tue May 15 20:31:10 UTC 2018 TRACKING BUG FOR RECOMPILING USM DRIVERS WITH
    
   If the patch is not installed, perform the following steps: 1) install the ACFS patch, as indicated in the "Required ACFS Driver Updates" section above; 2) reboot (required for the kernel to re-enable proper mitigation); 3) verify proper mitigation. This may be done in a rolling manner.
   
   If the patch is installed, but it was installed after Exadata software was upgraded to >= 18.1.5.0.0 or >= 12.2.1.1.7, then reboot is required for the kernel to re-enable proper mitigation.
   
   Otherwise, proceed to the next troubleshooting step.
   
2. Verify the state of the ACFS driver using the acfsdriverstate command.
   
   $ <gihome>/bin/acfsdriverstate version
   ACFS-9325: Driver OS kernel version = 4.1.12-94.8.2.el6uek.x86_64.
     
   If Driver OS kernel version = 4.1.12-32, then the updated ACFS driver is not loaded.  This scenario may occur if the ACFS patch was installed before database server update, but the database server update was performed with an older patchmgr/dbserver.patch.  To load the proper ACFS driver perform the following steps as root: 1) stop clusterware; 2) run "<gihome>/bin/acfsroot install" to load the proper drivers; 3) reboot (required for the kernel to re-enable proper mitigation); 4) verify proper mitigation.  This may be done in a rolling manner.
   
   If Driver OS kernel version = 4.1.12-94.8.2 or a higher kernel version (note that this will not necessarily match the installed kernel version), then the correct driver is loaded.  Proceed to the next troubleshooting step.
   
3. Review /var/log/messages for output referring to "Spectre V2" logged shortly after the time for the current system boot, indicating a module is loaded "not compiled with retpoline compiler", similar to the following:
   
   kernel: badmodule: loading module not compiled with retpoline compiler.
   kernel: Spectre V2 : Disabling Spectre v2 mitigation retpoline.
   kernel: Spectre V2 : Spectre v2 mitigation set to IBRS.
     
   If /var/log/messages does not contain any message referring to "Spectre V2" then one of the following conditions exist: 1) /var/log/messages file was rotated - review an older messages file; or 2) Exadata software has not been upgraded to >= 18.1.5.0.0 or >= 12.2.1.1.7.
   
   If the message reported is "oracleoks: loading module not compiled with retpoline compiler", then it indicates the proper ACFS driver is not in place.  Review the previous troubleshooting steps.
   
   If the message refers to any other module, then it is likely caused by user-installed software that supplies a module that has not been compiled with a retpoline-aware compiler.  Contact the vendor of that kernel module to obtain an update.
   
   For retpoline mitigation to be active, kernel modules/drivers that contain code needing retpolines must be compiled with a retpoline-aware compiler. Loading a module needing retpolines that was not compiled with a retpoline-aware compiler (e.g. an older ACFS driver, or a third-party module) will cause the kernel to disable retpoline mitigation systemwide, and fallback to a different mitigation (e.g. IBRS), which may have higher than expected performance impact on some systems. Review /var/log/messages, as shown above, for output showing the kernel disabling retpoline because a module was not compiled with a retpoline-aware compiler. All kernel modules delivered with Exadata 18.1.5.0.0 and 12.2.1.1.7 have been compiled with a retpoline-aware compiler. The updated ACFS drivers discussed above have been compiled with a retpoline-aware compiler.
   
   
4. If the previous troubleshooting steps do not resolve the issue, then Contact Oracle Support.

M2.devices in X7 and how to monitor and replace faulty disks

Oracle Exadata Database Machine X7 systems comes with two internal M.2 devices that contain the system area. In all previous systems, the first two disks of the Oracle Exadata Storage Server are system disks and the portions on these system disks are referred to as the system area.

Note:

Oracle Exadata Rack and Oracle Exadata Storage Servers can remain online and available while replacing an M.2 disk.

This section contains the following topics:

• Monitoring the Status of M.2 Disks
• Replacing a M.2 Disk Due to Failure or Other Problems

Monitoring the Status of M.2 Disks

You can monitor the status of a M.2 disk by checking its attributes with the CellCLI LIST PHYSICALDISK command.

The disk firmware maintains the error counters, and marks a drive with Predictive Failure when the disk is about to fail. The drive, not the cell software, determines if it needs replacement.

• Use the CellCLI command LIST PHSYICALDISK to determine the status of a M.2 disk:

 

CellCLI> LIST PHYSICALDISK WHERE disktype='M2Disk' DETAIL
         name:                           M2_SYS_0
        deviceName:                  /dev/sdm
        diskType:                      M2Disk
         makeModel:                    "INTEL SSDSCGJK150G7"
         physicalFirmware:         N2010112
         physicalInsertTime:      2017-07-14T08:42:24-07:00
         physicalSerial:            PHDW7082000M150A
         physicalSize:               139.73558807373047G
         slotNumber:                  "M.2 Slot: 0"
         status:                failed

         name:                  M2_SYS_1        
         deviceName:            /dev/sdn
         diskType:              M2Disk
         makeModel:             "INTEL SSDSCKJB150G7"
         physicalFirmware:      N2010112
         physicalInsertTime:    2017-07-14T12:25:05-07:00
         physicalSerial:        PHDW708204SZ150A
         physicalSize:          139.73558807373047G
         slotNumber:            "M.2 Slot: 1"
         status:                normal

Replacing a M.2 Disk Due to Failure or Other Problems

Failure of a M.2 disks reduces redundancy of the system area, and can impact patching, imaging, and system rescue. Therefore, the disk should be replaced with a new disk as soon as possible. When a M.2 disk fails, the storage server automatically and transparently switches to using the software stored on the inactive system disk, making it the active system disk.

An Exadata alert is generated when an M.2 disk fails. The alert includes specific instructions for replacing the disk. If you have configured the system for alert notifications, then the alert is sent by e-mail to the designated address. M.2 disk is hot-pluggable and can be replaced when the power is on. After the M.2 disk is replaced, Oracle Exadata System Software automatically adds the new device to the system partition and starts the rebuilding process.

1. Identify the failed M.2 disk.

   CellCLI> LIST PHYSICALDISK WHERE diskType=M2Disk AND status!=normal DETAIL
            name:                      M2_SYS_0
             deviceName:              /dev/sda
             diskType:                M2Disk
             makeModel:              "INTEL SSDSCKJB150G7"
            physicalFirmware:          N2010112
            physicalInsertTime:        2017-07-14T08:42:24-07:00
            physicalSerial:            PHDW7082000M150A
            physicalSize:              139.73558807373047G
            slotNumber:                "M.2 Slot: 0"
          status:                    failed - dropped for replacement
   

2. Locate the cell that has the white LED lit.
3. Open the chassis and identify the M.2 disk by the slot number in Step 1.
4. The amber LED for this disk should be lit to indicate service is needed.
   M.2 disks are hot pluggable, so you do not need to power down the cell before replacing the disk.
5. Remove the M.2 disk:
   1. Rotate both riser board socket ejectors up and outward as far as they will go.
      The green power LED on the riser board turns off when you open the socket ejectors.
   2. Carefully lift the riser board straight up to the remove it from the sockets.
6. Insert the replacement M.2 disk:
   1. Unpack the replacement flash riser board and place it on an antistatic mat.
   2. Align the notch in the replacement riser board with the connector key in the connector socket.
   3. Push the riser board into the connector socket until the riser board is securely seated in the socket.
      Caution:

      If the riser board does not easily seat into the connector socket, verify that the notch in the riser board is aligned with the connector key in the connector socket. If the notch is not aligned, damage to the riser board might occur.
   4. Rotate both riser board socket ejectors inward until the ejector tabs lock the riser board in place.
      The green power LED on the riser board turns on when you close the socket ejectors.
7. Confirm the M.2 disk has been replaced.

   CellCLI> LIST PHYSICALDISK WHERE DISKTYPE=M2Disk DETAIL
        name:                  M2_SYS_0 
       deviceName:            /dev/sdm   
      diskType:              M2Disk   
      makeModel:             "INTEL SSDSCKJB150G7"   
      physicalFirmware:      N2010112    
      physicalInsertTime:    2017-08-24T18:55:13-07:00   
      physicalSerial:        PHDW708201G0150A   
      physicalSize:          139.73558807373047G   
      slotNumber:            "M.2 Slot: 0"   
      status:                normal   
   
      name:                  M2_SYS_1   
      deviceName:            /dev/sdn   
      diskType:              M2Disk   
      makeModel:             "INTEL SSDSCKJB150G7"    
      physicalFirmware:      N2010112   
      physicalInsertTime:    2017-08-24T18:55:13-07:00   
      physicalSerial:        PHDW708200SZ150A   
      physicalSize:          139.73558807373047G   
      slotNumber:            "M.2 Slot: 1"   
      status:                normal 
   

8. Confirm the system disk arrays are have an active sync status, or are being rebuilt.

# mdadm --detail /dev/md[2-3][4-5]
/dev/md24:
      Container : /dev/md/imsm0, member 0
     Raid Level : raid1
     Array Size : 104857600 (100.00 GiB 107.37 GB)
  Used Dev Size : 104857600 (100.00 GiB 107.37 GB)
   Raid Devices : 2
  Total Devices : 2

               State  : active
 Active Devices  : 2
Working Devices  : 2
 Failed Devices  : 0
   Spare Devices : 0  

            UUUID : 152f728a:6d294098:5177b2e5:8e0d8c6c
   Number    Major    Minor    RaidDevice    State
    1           8         16             0       active sync  /dev/sdb
    0           8           0            1       active sync  /dev/sda
/dev/md25:
      Container : /dev/md/imsm0, member 1
     Raid Level : raid1
     Array Size : 41660426 (39.73 GiB 42.66 GB)
  Used Dev Size : 41660524 (39.73 GiB 42.66 GB)
   Raid Devices : 2
  Total Devices : 2

               State  : clean
 Active Devices  : 2
Working Devices  : 2
 Failed Devices  : 0
   Spare Devices : 0  

             UUID : 466173ba:507008c7:6d65ed89:3c40cf23
   Number    Major    Minor    RaidDevice    State
 1           8         16        0      active sync  /dev/sdb
 0           8         0         1      active sync  /dev/sda

Storage Cell reports error RS-7445 [Serv MS Leaking Memory]

Bug 19790644 - RS-7445 [SERV MS LEAKING MEMORY]

Issue is a memory leak in the Java executable.

This bug affects systems running with JDK 7u51 or later versions (1.7.0_55-b13) which are 11.2.3.3.1 and 12.1.1.1.1
This is relevant for all versions 11.2.3.3.1 to 12.1.2.1.1 [Release 11.2 to 12.1]  (excluding  11.2.3.3.0 or 12.1.1.1.0)
Systems running 11.2.3.3.0 or 12.1.1.1.0 are not affected as they use 1.7.0_25-b15 

Cause:

MS process will be consuming memory (up to 2GB).  Normally MS will use around 1GB of memory but because of the bug, the memory allocated can grow upt to 2GB.

Normal memory usage:

ps -feal|grep java
0 S root     18585 13652  0  80   0 - 15319 pipe_w 15:21 pts/1    00:00:00 grep java
0 S root     27960 27958  0  80   0 - 292553 futex_ Jun17 ?       01:45:06 /usr/java/default/bin/java -Xms256m -Xmx512m -XX:-UseLargePages -Djava.library.path=/opt/oracle/cell/cellsrv/lib -Ddisable.checkForUpdate=true -jar /opt/oracle/cell/oc4j/ms/j2ee/home/oc4j.jar -out /opt/oracle/cell/cellsrv/deploy/log/ms.lst -err /opt/oracle/cell/cellsrv/deploy/log/ms.err
292553 * 4096 = 1142MB (1GB).

Larger values will indicate memory leak.
When using command pmap -x <MS process pid>,  if memory leak is still present  it will report a large number of 64 memory chunks:

Address           Kbytes     RSS   Dirty Mode   Mapping
0000000000400000       4       4       0 r-x--  java
0000000000600000       4       4       4 rw---  java
00000000019c3000   85212   83816   83816 rw---    [ anon ]
00000000dae00000   46080   45856   45856 rw---    [ anon ]
00000000ddb00000   37888       0       0 -----    [ anon ]
00000000e0000000  175104  174900  174900 rw---    [ anon ]
00000000eab00000  174080       0       0 rw---    [ anon ]
00000000f5500000   87552   87552   87552 rw---    [ anon ]
00000000faa80000   87552       0       0 -----    [ anon ]
00007f261c000000   38384   37488   37488 rw---    [ anon ]
00007f261e57c000   27152       0       0 -----    [ anon ]
00007f2624000000   58488   56628   56628 rw---    [ anon ]
00007f262791e000    7048       0       0 -----    [ anon ]
00007f262c000000   65524   65444   65444 rw---    [ anon ]
00007f262fffd000      12       0       0 -----    [ anon ]
00007f2634000000   65528   65528   65528 rw---    [ anon ]
00007f2637ffe000       8       0       0 -----    [ anon ]
00007f263c000000   65536   65528   65528 rw---    [ anon ]
00007f2644000000   65536   65360   65360 rw---    [ anon ]
00007f264c000000   65528   65520   65520 rw---    [ anon ]
00007f264fffe000       8       0       0 -----    [ anon ]
00007f2654000000   65456   65456   65456 rw---    [ anon ]

Solution:

1. Error is ignorable as MS service will be re-started automatically, which will reset the process and memory used.
2. While patching the storage cell with one-off patches is not generally recommended, if there are issues where the MS service is not automatically re-started, the JDK needs upgraded on the Storage Cell
    Use Patch 20328167: TRACKING BUG FOR JDK 1.7.0.72- B33 PATCH (wrapper for 20328167: Oracle JDK 7 Update 72 b33 or later)
3. If no other issues are being seen, the recommended action is to wait for Exadata Cell Software version 12.1.2.1.2 or later.
 

Asmcmd daemon consuming High CPU

ASMCMD commands when executed with parameters are leaving the asm connection open "asmcmd daemon" and consumes high CPU usage.

Bug 28019068 - EXADATA: ASMCMD CONSUMING HIGH CPU IN 18C

$ ps -ef | grep asmcmd | grep -v grep
root      224347      1  3 03:20 ?        00:00:00 asmcmd daemon
grid   234123      1  2 03:20 ?        00:00:00 oracle+asm_asmcmd
pstack output:

#0  0x00007f4954b00050 in __open_nocancel () from /lib64/libpthread.so.0
#1  0x00000000005622de in PerlIOUnix_open ()
#2  0x0000000000563c74 in PerlIOBuf_open ()
#3  0x0000000000565795 in PerlIO_openn ()
#4  0x000000000053b030 in Perl_do_open6 ()
#5  0x00000000005274d2 in Perl_pp_open () 
#6  0x00000000004cefad in Perl_runops_standard ()
#7  0x0000000000443622 in S_run_body ()
#8  0x000000000044350b in perl_run ()
#9  0x000000000041de78 in main ()
You can see open ( ) call and leave pool connection open.

Workaround:

1. When you kick off an ASMCMD command, it actually establishes a connection to the ASM instance. To disable connection pooling, use the --nocp parameter to the ASMCMD tool:

 $ asmcmd --nocp <parameters>

2. Use commands within ASMCMD command line instead of passing parameters to avoid  pool connections open.
 

Physical disk in failed state but online on MegaRaid on CELL

Physical disk in failed state but online on MegaRaid on CELL

BUG ID:  25632147

Workaround:



 1. Copy the cell disk config.xml file (Take backup)


 [root@jfclcx0024 config]# cp cell_disk_config.xml* /tmp
 [root@jfclcx0024 config]# ls /tmp/cell_disk_config.xml*
 /tmp/cell_disk_config.xml /tmp/cell_disk_config.xml_
 /tmp/cell_disk_config.xml__


 2. Remove the config file


 [root@jfclcx0024 config]# rm cell_disk_config.xml*
 rm: remove regular file `cell_disk_config.xml'? y
 rm: remove regular file `cell_disk_config.xml_'? y
 rm: remove regular file `cell_disk_config.xml__'? y



 3. Stop the celld services.


 [root@jfclcx0024 config]# service celld stop
 Stopping the RS, CELLSRV, and MS services...
 The SHUTDOWN of services was successful.


 4. Start the cell services


 [root@jfclcx0024 config]# service celld start
 1: 108 usec
 1: 76 usec
 Starting the RS, CELLSRV, and MS services...
 Getting the state of RS services... running
 Starting CELLSRV services...
 The STARTUP of CELLSRV services was not successful.
 CELL-01537: Unable to read the cell_disk_config.xml file because the file is missing or empty.
 Starting MS services...
 The STARTUP of MS services was successful.



 5. Check the status of the cell services, cellsrv will not be started


 [root@jfclcx0024 config]# service celld status
 rsStatus: running
 msStatus: running
 cellsrvStatus: stopped
 [root@jfclcx0024 config]#
 [root@jfclcx0024 config]#
 6. copy the backup config.xml file to its original location.


 [root@jfclcx0024 config]# cp /tmp/cell_disk_config.xml .



 7. Restart the cell services.


 [root@jfclcx0024 config]# service celld restart
 Stopping the RS, CELLSRV, and MS services...
 The SHUTDOWN of services was successful.
 Starting the RS, CELLSRV, and MS services...
 Getting the state of RS services... running
 Starting CELLSRV services...
 The STARTUP of CELLSRV services was successful.
 Starting MS services...
 The STARTUP of MS services was successful.
 8. Confirm the cell services are up
 [root@jfclcx0024 config]# service celld status
 rsStatus: running
 msStatus: running
 cellsrvStatus: running
 [root@jfclcx0024 config]#


 9. Now check the status of the physical disk, if the status is normal.


 [root@jfclcx0024 config]# cellcli -e list physicaldisk 8:10 detail
 name: 8:10
 deviceId: 25
 deviceName: /dev/sdk
 diskType: HardDisk
 enclosureDeviceId: 8
 errOtherCount: 0
 luns: 0_10
 makeModel: "HGST H7280A520SUN8.0T"
 physicalFirmware: P9E2
 physicalInsertTime: 2017-02-26T07:23:06+00:00
 physicalInterface: sas
 physicalSerial: P1PMBV
 physicalSize: 7.153663907200098T
 slotNumber: 10
 status: normal
 

CLSU-00107: operating system function: open failed; failed with error data: 2; at location: SlfFopen1

CLSU-00107: operating system function: open failed; failed with error data: 2; at location: SlfFopen1

CLSU-00101: operating system error message: No such file or directory

When executing the asmcmd command in GI,one got the error Can't open '/opt/oracle/log/diag/asmcmd/user_grid/weasel1xa.rjf.com/alert/alert.log' for append.
This is pointing to the Oracle base and not the Oracle home for directory location. 
 
[grid@weasel1xa ~]$ export DBI_TRACE=1
[grid@weasel1xa ~]$ asmcmd
   DBI 1.616-ithread default trace level set to 0x0/1 (pid 2193 pi 7fa010) at DBI.pm line 278 via asmcmdshare.pm line 270
Can not create path /opt/oracle/log/diag/asmcmd/user_grid/weasel1xa.rjf.com/alert
Can not create path /opt/oracle/log/diag/asmcmd/user_grid/weasel1xa.rjf.com/trace
Can't open '/opt/oracle/log/diag/asmcmd/user_grid/weasel1xa.rjf.com/alert/alert.log' for append
CLSU-00100: Operating System function: open failed failed with error data: 2
CLSU-00101: Operating System error message: No such file or directory
CLSU-00103: error location: SlfFopen1

As per documentation:

Under certain circumstances, $ORACLE_BASE and $ORACLE_HOME can be set to override the default locations of the alert.log and trace.trc files.
http://docs.oracle.com/cd/E11882_01/server.112/e18951/asm_util001.htm#OSTMG94362
 
 
"log" directory was missing in asmcmd log location path /opt/oracle/log/diag/asmcmd ($ORACLE_BASE/log/diag/asmcmd)

$ls -l /opt/oracle
total 48
drwxr-x--- 4 grid oinstall 4096 Oct 31 11:27 admin
drwxr-x--- 2 grid oinstall 4096 Oct 31 11:27 audit
drwxr-x--- 6 grid oinstall 4096 Oct 31 11:29 cfgtoollogs
drwxr-xr-x 2 grid oinstall 4096 Oct 31 11:32 checkpoints
drwxr-xr-x 3 oracle oinstall 4096 Nov 4 16:48 core
drwxrwxr-x 3 grid oinstall 4096 Oct 30 17:54 crsdata
drwxrwxr-x 12 grid oinstall 4096 Nov 1 16:26 diag
drwxr-xr-x 3 oracle oinstall 4096 Oct 31 13:44 opatchauto
drwxr-xr-x 10 oracle oinstall 4096 Oct 31 16:17 ora_sw
drwxr-xr-x 5 oracle oinstall 4096 Nov 5 14:10 product
drwxr-xr-x 3 grid oinstall 4096 Oct 24 14:14 weasel1xa
drwxr-xr-x 3 root root 4096 Oct 31 13:41 weasel1xa.rjf.com

Solution: Issue resolved after creating the directory structure /opt/oracle/log/diag/asmcmd where "log" subdirectory was included.
 

Hybrid Storage Pools

Using Hybrid Storage Pools:

 Most of the storage available on the market today utilizes a small amount of NRAM devices (usually 1, 8 or even 16GB in size) as the first cache tier of the system and also hard disk drivers. NRAM devices are expensive, and disk drivers’ performance is affected by seek operations, rotation and transfer times, which can result in I/O bottleneck and performance problems. Oracle ZFS Storage Appliance implements a Hybrid Storage Pool architecture designed to work with multiple tiers of storage media to maximize the performance for the virtualized environment.

First Tier: DRAM (large L1 cache) – DRAM memory and high-optimized and low-latency solid state disks combined with ZFS file systems architecture (ReadZilla) accelerate read-cache operations for the virtualized environment. Unlike traditional NAS architecture, Oracle ZFS Storage Appliance and its Hybrid Storage Pools utilize DRAM devices as the main cache device of the system. DRAM devices are cheaper, faster and deliver higher performance than NVRAM, so they are well matched for random I/O workloads – the kind of workloads performed by hypervisors/virtualized environments. DRAM devices are also used by the Adaptive Replacement Cache (ARC), which is part of the ZFS file system architecture model and intelligently managed by instructions provided by multiple cache algorithms.

Second Tier: SSDs (large L2 cache) – High-optimized and low-latency solid state disks combined with ZFS file systems log architecture (ZIL or LogZilla) to accelerate write-cache operations. These provide excellent performance and fast response for writing operations performed by applications and databases running in virtualized environment. The Hybrid Storage Pools are SSD devices designed to provide fast writing operations (100 times faster than traditional disk drivers) with low latency. Inside of the Hybrid Storage Pool architecture, SSD devices host the ZFS ZIL log (known as LogZilla or ZFS Intent Log), which is part of the ZFS file system architecture and mainly responsible for accelerating the synchronous writing operations requested by the critical applications and databases running in virtualized environments. Also, the SSDs are utilized by the Layer 2 Adaptive Replacement Cache (L2ARC), which is an extension of the ARC (main cache of the system) and hosts the read log devices for the ZFS architecture.

Third Tier: Disk Pools – Disk pools are composed of high-performance (15000 rpm) and/or high capacity (7200 rpm) disks that are protected by different RAID levels and intelligently managed by the ZFS file system. Disk pools are designed to archive the application data, providing continuously high I/O rates for different types of workloads, even when utilizing high-capacity disks (7200 rpm). Disk pools can optionally be configured with 15000 rpm disks, which provide the highest performance and thousands of IOPS for datastores typical of virtualized environments.