9.5.  Use "mustgather" documentation and/or the IBM Support Assistant to collect and analyze diagnostic data or submit data to IBM Support.

[Note]

Out of Memory exceptions in WebSphere Application Server

An Out of Memory exception can be caused by numerous reasons. A memory leak can be indicated by the steady increase in the memory usage without de-allocation, as shown in figure below in IBM Pattern Modeling and Analysis Tool in ISA. It is expected that the Java heap usage fluctuates up and down because the de-allocation of memory is only triggered when the heap usage reaches certain levels.

Figure 9.6. Chart graph of a memory leak

Chart graph of a memory leak


Figure above is similar to a saw, with slow increase in the memory (objects allocation) and then a sudden decrease (garbage collection). If in every cycle, the heap usage after garbage collector execution shows a steady increase, it can also indicate a memory leak condition.

Another type of out of memory issue can be caused by heap fragmentation. Heap fragmentation is caused by the allocation and de-allocation of large objects in a short amount of time. The heap will have enough space to accommodate new large objects, but that space is not contiguous, causing an Out of Memory exception. In such cases, a steady increase pattern is not seen in the chart, so then the analysis tools will help in determining if heap fragmentation is the problem.

An out of memory condition is most commonly detected when an OutOfMemoryError exception is shown in the SystemErr.log file, as shown below:

[6/29/12 18:40:31:919 EDT] 0000008f SystemErr     R java.lang.OutOfMemoryError: Java heap space
[6/29/12 18:40:31:922 EDT] 0000008f SystemErr     R     at com.ibm.AllocateObject.doWork(AllocateObject.java:45)
[6/29/12 18:40:31:924 EDT] 0000008f SystemErr     R     at com.ibm.BaseServlet.service(BaseServlet.java:72)
[6/29/12 18:40:31:925 EDT] 0000008f SystemErr     R     at javax.servlet.http.HttpServlet.service(HttpServlet.java:668)
...
[6/29/12 18:40:32:602 EDT] 0000008f SystemErr     R     at com.ibm.io.async.AbstractAsyncFuture.invokeCallback(AbstractAsyncFuture.java:217)
[6/29/12 18:40:32:603 EDT] 0000008f SystemErr     R     at com.ibm.io.async.AsyncChannelFuture$1.run(AsyncChannelFuture.java:205)
[6/29/12 18:40:32:604 EDT] 0000008f SystemErr     R     at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1783)
					

After an Out Of Memory exception is detected, WebSphere Application Server by default creates three log files. The files are one Java Core, one Portable Heap Dump (or simply Heap dump) and one System Dump (native platform core dump), as shown in the native_stderr.log

JVMDUMP039I Processing dump event "systhrow", detail "java/lang/OutOfMemoryError" at 2012/06/29 18:39:48 - please wait.
JVMDUMP032I JVM requested System dump using '/opt/IBM/WebSphere/AppServer/profiles/AppSrv01/core.20120629.183948.19153.0001.dmp' in response to an event
JVMDUMP010I System dump written to /opt/IBM/WebSphere/AppServer/profiles/AppSrv01/core.20120629.183948.19153.0001.dmp
JVMDUMP032I JVM requested Heap dump using '/opt/IBM/WebSphere/AppServer/profiles/AppSrv01/heapdump.20120629.183948.19153.0002.phd' in response to an event
JVMDUMP010I Heap dump written to /opt/IBM/WebSphere/AppServer/profiles/AppSrv01/heapdump.20120629.183948.19153.0002.phd
JVMDUMP032I JVM requested Java dump using '/opt/IBM/WebSphere/AppServer/profiles/AppSrv01/javacore.20120629.183948.19153.0003.txt' in response to an event
JVMDUMP010I Java dump written to /opt/IBM/WebSphere/AppServer/profiles/AppSrv01/javacore.20120629.183948.19153.0003.txt
JVMDUMP032I JVM requested Snap dump using '/opt/IBM/WebSphere/AppServer/profiles/AppSrv01/Snap.20120629.183948.19153.0004.trc' in response to an event
JVMDUMP010I Snap dump written to /opt/IBM/WebSphere/AppServer/profiles/AppSrv01/Snap.20120629.183948.19153.0004.trc
JVMDUMP013I Processed dump event "systhrow", detail "java/lang/OutOfMemoryError".
					

Using these files together with the native_stderr.log, which contains the verbose garbage collection output (if enabled), you can have a first look. These files and logs help diagnose what is accumulating in the memory and causing the Out of Memory exception.

The following steps help identify the likely cause of the Out Of Memory Exception:

  1. Download the portable heap dump file (.phd) and the native_stderr.log file to a computer with ISA installed. The System Dump file (.dmp) is only needed in case the Out Of Memory exception is being caused by something outside the Java Heap, such as the JVM itself or native libraries.

  2. Open the native_stderr.log file in the ISA tool IBM Monitoring and Diagnostic Tools for Java - Garbage Collection and Memory Visualizer.

  3. The tool provides views of the heap usage and reports possible issues and recommendations, as shown in the figure below:

    Figure 9.7. IBM Monitoring and Diagnostic Tools for Java - Garbage Collection and Memory Visualizer

    IBM Monitoring and Diagnostic Tools for Java - Garbage Collection and Memory Visualizer


  4. To understand what objects are getting filled up in memory, use a tool to process the contents of the Portable Heap Dump and allow navigation and drill-down, such as IBM Heap Analyzer or IBM Monitoring and Diagnostic Tools for Java - Garbage Collection and Memory Analyzer. Figure below shows an example of IBM Heap Analyzer:

    Figure 9.8. IBM Heap Analyzer

    IBM Heap Analyzer


Professional hosting         Free 'Oracle Certified Expert Web Services Developer 6' Guide     Free SCDJWS 5.0 Guide