Sunday, October 14, 2018

Spectre / Meltdown - Intel processor vulnerabilities [CVE-2017-5715 CVE-2017-5753 CVE-2017-5754]

Oracle Exadata Database Machine with respect CVE-2017-5753 (Spectre v1), CVE-2017-5715 (Spectre v2), and CVE-2017-5754 (Meltdown) Intel processor vulnerabilities Status:


Patch Availability Table

Affected Products

Patch Availability

 CVE-2017-5715CVE-2017-5753CVE-2017-5754
Oracle Exadata Database Machine
(compute nodes/storage servers)
18.1.5.0.0
12.2.1.1.7
Note 1
Note 5
Note 6
18.1.4.0.0
12.2.1.1.6
18.1.4.0.0
12.2.1.1.6
Sun Data Center InfiniBand Switch 36 (NM2-36P)
Not Required
Note 2
Not Required
Note 2
Not Required
Note 2
Cisco Catalyst C4948/C4948E-F-S
Not Required
Note 3
Not Required
Note 3
Not Required
Note 3
Cisco Nexus 93108TC-EX
May be Required
Note 4
May be Required
Note 4
May be Required
Note 4




Note 1: Install updated ACFS drivers before updating Exadata database servers to >= 18.1.5.0.0 or >= 12.2.1.1.7, and use the most recent patchmgr/dbserver.patch (Patch 21634633 version 18.1.5.0.0 / 5.180529 or later) to perform the database server updates.  Updated ACFS drivers are needed whether or not ACFS is in use in order for the most efficient CVE-2017-5715 mitigation to be in place once >= 18.1.5.0.0 or >= 12.2.1.1.7 is installed.  See section Required ACFS Driver Updates below for details.
Note 2: The InfiniBand Switch component (NM2-36P) is not currently believed to be impacted by these vulnerabilities.


Required ACFS Driver Updates

Install updated ACFS drivers before updating Exadata database servers to >= 18.1.5.0.0 or >= 12.2.1.1.7. Note the following:
  1. Updated ACFS drivers are needed for the most efficient CVE-2017-5715 mitigation to be in place once >= 18.1.5.0.0 or >= 12.2.1.1.7 is installed.
  2. Updated ACFS drivers are needed whether or not ACFS is in use.
  3. In an OVM configuration dom0 may be updated to >= 18.1.5.0.0 or >= 12.2.1.1.7 before the ACFS drivers are updated in domU.
Updated ACFS drivers are available with the following:
  • Grid infrastructure 18.3.0.0.180717 and later version 18 quarterly updates
  • Grid infrastructure 12.2.0.1.180717 and later version 12.2.0.1 quarterly updates
  • Grid infrastructure 12.1.0.2.180831 and later version 12.1.0.2 quarterly updates
  • Grid infrastructure 12.1.0.2.180717 plus patch 23312691
How to Verify Proper Mitigation


To verify your systems have the proper mitigations in place after the ACFS driver and Exadata updates are performed, run the following command:
$ grep . /sys/devices/system/cpu/vulnerabilities/*
The expected output depends on the hardware version, server type, and system configuration.  If the output shown on your system does not match 
X7 hardware - storage server and database server (non-OVM and domU)
Expected vulnerability mitigation output:
$ grep . /sys/devices/system/cpu/vulnerabilities/*
/sys/devices/system/cpu/vulnerabilities/meltdown:Mitigation: PTI
/sys/devices/system/cpu/vulnerabilities/spectre_v1:Mitigation: lfence
/sys/devices/system/cpu/vulnerabilities/spectre_v2:Mitigation: IBRS, IBRS_FW, IBPB
X6 and earlier hardware - storage server and database server (non-OVM and domU)
Expected vulnerability mitigation output:
$ grep . /sys/devices/system/cpu/vulnerabilities/*
/sys/devices/system/cpu/vulnerabilities/meltdown:Mitigation: PTI
/sys/devices/system/cpu/vulnerabilities/spectre_v1:Mitigation: lfence
/sys/devices/system/cpu/vulnerabilities/spectre_v2:Mitigation: Full generic retpoline, IBRS_FW, IBPB
All hardware - database server (dom0)
Expected vulnerability mitigation output:
# grep . /sys/devices/system/cpu/vulnerabilities/*
/sys/devices/system/cpu/vulnerabilities/meltdown:Vulnerable
/sys/devices/system/cpu/vulnerabilities/spectre_v1:Mitigation: lfence
/sys/devices/system/cpu/vulnerabilities/spectre_v2:Mitigation: Full generic retpoline
Note that the output for dom0 incorrectly indicates the system is vulnerable to Meltdown (CVE-2017-5754).  Exadata database servers configured with OVM use the Xen hypervisor and guest VMs in HVM mode.  Xen with guests in HVM mode is not vulnerable to Meltdown.
Troubleshooting unexpected mitigation
If Spectre v2 mitigation does not match the expected output shown above, then confirm "imageinfo -ver" indicates the installed Exadata versions is >= 18.1.5.0.0 or >= 12.2.1.1.7.  If the proper Exadata version is installed, then follow these guidelines:
Storage servers
This situation should never occur on storage servers.  Contact Oracle Support.

Database servers (non-OVM and domU)
Perform the following verification steps on each database server:
  1. Verify the patch containing the updated ACFS drivers has been installed in the grid infrastructure home.  If the following OPatch command does not return output then the ACFS patch is not installed. 
    $ <gihome>/grid/OPatch/opatch lsinventory -bugs_fixed | grep ^27463879
    27463879 27463879 Tue May 15 20:31:10 UTC 2018 TRACKING BUG FOR RECOMPILING USM DRIVERS WITH
     
    If the patch is not installed, perform the following steps: 1) install the ACFS patch, as indicated in the "Required ACFS Driver Updates" section above; 2) reboot (required for the kernel to re-enable proper mitigation); 3) verify proper mitigation. This may be done in a rolling manner.

    If the patch is installed, but it was installed after Exadata software was upgraded to >= 18.1.5.0.0 or >= 12.2.1.1.7, then reboot is required for the kernel to re-enable proper mitigation.

    Otherwise, proceed to the next troubleshooting step.
  2. Verify the state of the ACFS driver using the acfsdriverstate command.
    $ <gihome>/bin/acfsdriverstate version
    ACFS-9325: Driver OS kernel version = 4.1.12-94.8.2.el6uek.x86_64.
      
    If Driver OS kernel version = 4.1.12-32, then the updated ACFS driver is not loaded.  This scenario may occur if the ACFS patch was installed before database server update, but the database server update was performed with an older patchmgr/dbserver.patch.  To load the proper ACFS driver perform the following steps as root: 1) stop clusterware; 2) run "<gihome>/bin/acfsroot install" to load the proper drivers; 3) reboot (required for the kernel to re-enable proper mitigation); 4) verify proper mitigation.  This may be done in a rolling manner.

    If Driver OS kernel version = 4.1.12-94.8.2 or a higher kernel version (note that this will not necessarily match the installed kernel version), then the correct driver is loaded.  Proceed to the next troubleshooting step.
  3. Review /var/log/messages for output referring to "Spectre V2" logged shortly after the time for the current system boot, indicating a module is loaded "not compiled with retpoline compiler", similar to the following:
    kernel: badmodule: loading module not compiled with retpoline compiler.
    kernel: Spectre V2 : Disabling Spectre v2 mitigation retpoline.
    kernel: Spectre V2 : Spectre v2 mitigation set to IBRS.
      
    If /var/log/messages does not contain any message referring to "Spectre V2" then one of the following conditions exist: 1) /var/log/messages file was rotated - review an older messages file; or 2) Exadata software has not been upgraded to >= 18.1.5.0.0 or >= 12.2.1.1.7.

    If the message reported is "oracleoks: loading module not compiled with retpoline compiler", then it indicates the proper ACFS driver is not in place.  Review the previous troubleshooting steps.

    If the message refers to any other module, then it is likely caused by user-installed software that supplies a module that has not been compiled with a retpoline-aware compiler.  Contact the vendor of that kernel module to obtain an update.

    For retpoline mitigation to be active, kernel modules/drivers that contain code needing retpolines must be compiled with a retpoline-aware compiler. Loading a module needing retpolines that was not compiled with a retpoline-aware compiler (e.g. an older ACFS driver, or a third-party module) will cause the kernel to disable retpoline mitigation systemwide, and fallback to a different mitigation (e.g. IBRS), which may have higher than expected performance impact on some systems. Review /var/log/messages, as shown above, for output showing the kernel disabling retpoline because a module was not compiled with a retpoline-aware compiler. All kernel modules delivered with Exadata 18.1.5.0.0 and 12.2.1.1.7 have been compiled with a retpoline-aware compiler. The updated ACFS drivers discussed above have been compiled with a retpoline-aware compiler.

  4. If the previous troubleshooting steps do not resolve the issue, then Contact Oracle Support.