Wednesday, July 4, 2018

Storage Cell reports error RS-7445 [Serv MS Leaking Memory]


Bug 19790644 - RS-7445 [SERV MS LEAKING MEMORY]

Issue is a memory leak in the Java executable.

This bug affects systems running with JDK 7u51 or later versions (1.7.0_55-b13) which are 11.2.3.3.1 and 12.1.1.1.1

This is relevant for all versions 11.2.3.3.1 to 12.1.2.1.1 [Release 11.2 to 12.1]  (excluding  11.2.3.3.0 or 12.1.1.1.0)
Systems running 11.2.3.3.0 or 12.1.1.1.0 are not affected as they use 1.7.0_25-b15 

Cause:


MS process will be consuming memory (up to 2GB).  Normally MS will use around 1GB of memory but because of the bug, the memory allocated can grow upt to 2GB.

Normal memory usage:

ps -feal|grep java
0 S root     18585 13652  0  80   0 - 15319 pipe_w 15:21 pts/1    00:00:00 grep java
0 S root     27960 27958  0  80   0 - 292553 futex_ Jun17 ?       01:45:06 /usr/java/default/bin/java -Xms256m -Xmx512m -XX:-UseLargePages -Djava.library.path=/opt/oracle/cell/cellsrv/lib -Ddisable.checkForUpdate=true -jar /opt/oracle/cell/oc4j/ms/j2ee/home/oc4j.jar -out /opt/oracle/cell/cellsrv/deploy/log/ms.lst -err /opt/oracle/cell/cellsrv/deploy/log/ms.err
292553 * 4096 = 1142MB (1GB).

Larger values will indicate memory leak.
When using command pmap -x <MS process pid>,  if memory leak is still present  it will report a large number of 64 memory chunks:
Address           Kbytes     RSS   Dirty Mode   Mapping
0000000000400000       4       4       0 r-x--  java
0000000000600000       4       4       4 rw---  java
00000000019c3000   85212   83816   83816 rw---    [ anon ]
00000000dae00000   46080   45856   45856 rw---    [ anon ]
00000000ddb00000   37888       0       0 -----    [ anon ]
00000000e0000000  175104  174900  174900 rw---    [ anon ]
00000000eab00000  174080       0       0 rw---    [ anon ]
00000000f5500000   87552   87552   87552 rw---    [ anon ]
00000000faa80000   87552       0       0 -----    [ anon ]
00007f261c000000   38384   37488   37488 rw---    [ anon ]
00007f261e57c000   27152       0       0 -----    [ anon ]
00007f2624000000   58488   56628   56628 rw---    [ anon ]
00007f262791e000    7048       0       0 -----    [ anon ]
00007f262c000000   65524   65444   65444 rw---    [ anon ]
00007f262fffd000      12       0       0 -----    [ anon ]
00007f2634000000   65528   65528   65528 rw---    [ anon ]
00007f2637ffe000       8       0       0 -----    [ anon ]
00007f263c000000   65536   65528   65528 rw---    [ anon ]
00007f2644000000   65536   65360   65360 rw---    [ anon ]
00007f264c000000   65528   65520   65520 rw---    [ anon ]
00007f264fffe000       8       0       0 -----    [ anon ]
00007f2654000000   65456   65456   65456 rw---    [ anon ]


Solution:

1. Error is ignorable as MS service will be re-started automatically, which will reset the process and memory used.
2. While patching the storage cell with one-off patches is not generally recommended, if there are issues where the MS service is not automatically re-started, the JDK needs upgraded on the Storage Cell
    Use Patch 20328167: TRACKING BUG FOR JDK 1.7.0.72- B33 PATCH (wrapper for 20328167: Oracle JDK 7 Update 72 b33 or later)
3. If no other issues are being seen, the recommended action is to wait for Exadata Cell Software version 12.1.2.1.2 or later.
 

1 comment: