Chapter 7. Problem Determination

7.1.  Configure, review and analyze logs (e.g., Web server, IBM WebSphere Application Server Network Deployment V7.0, first failure data capture (FFDC)).

Components and log files

During runtime, each WebSphere Application Server process creates log files that report the actions being performed. Each of these logs can be very important to troubleshooting a problem.

Figure 7.1. Log Files

Log Files


There are also many log files that are not shown in this picture. The logs shown in this picture are the logs that you should look at first when dealing with a problem during runtime.

The logs shown in the table below will exist for each profile, and as such will exist in the particular profile's profile_home/logs directory. Other than activity.log, which only exists once per node, each server on your node will have its own copy of these runtime log files.

Table 7.1.  WebSphere logs: Overview

GroupLog Files (default names)Description

Service log

activity.log

Binary log file contains data from each JVM in a node, for analysis using Log Analyzer. Contains system messages from all application servers and the node agent for a given node. Contains messages produced by instrumented applications

The service log is a binary-format log that contains more or less the same messages as the SystemOut JVM log, plus a few extra serviceability messages.

The benefit of the service log is that it can be used in conjunction with the Log Analyzer, which can compare logged messages with a "symptom database". The symptom database correlates messages to known problems. Viewing your service log with Log Analyzer is a good first step in problem determination, because it can compare your situation to a list of known WebSphere problems.

JVM logs

SystemOut.log, SystemErr.log

Contain all messages sent to the Java System.out and System.err streams.

The most commonly used logs are the JVM logs, often referred to as "standard out" and "standard error". They contain messages written to the System.out and System.err streams, respectively. Each Java process has its own JVM logs. Exceptions are written to these logs from Exception.printStackTrace().

The JVM logs are a good place to look for detailed information when there is a problem with an Application Server. Node Agent and Deployment Manager processes also write to JVM logs. You should look at these if the Node Agent or Deployment Manager is unable to start, similar to the trace file in older versions of WebSphere Application Server.

A log can be configured to roll over at specific time intervals, or when the file reaches a certain size. The two can also be combined. You could, for example, configure the log to roll over each day at midnight, unless the file grows larger than 10MB, in which case it rolls over immediately.

You can also specify how many "historical", meaning previously rolled-over, log files to keep.

Native process logs

native_stdout.log, native_stderr.log

Contain messages sent to stdout and stderr from native code segments, including the JVM.

The native process logs contain messages written by native code segments, including the JVM itself. Given the relatively small amount of native code compared to Java code, very few messages are written to these logs. Since the files are small, they do not have rollover capability.

One component that does use a reasonable amount of native code is the security subsystem. If you are having security-related problems, you should take a look at the native logs.

Embedded HTTP server logs

http_access.log, http_error.log

Contain all requests to the embedded HTTP server.

These logs are disabled by default, but can be enabled using the Administrative Console or wsadmin.sh. They can be particularly useful when you want to verify that a request is reaching the Application Server, or track the progress of a particular request as it moves through your environment.

HTTP server plug-in log

http_plugin.log

Contains data about the operation of the HTTP server plug-in module.

Although it runs inside the web server process, the plug-in writes to its own log file.

The log contains messages about the plug-in's startup process and runtime events. If your web server does not start, you should look for errors in this file, and in the web server's log files. If there seems to be a communication problem between the plug-in and an Application Server, you should also examine this file.

Command-line program logs

startServer.log, startNode.log, addNode.log, <command>.log

Contain data about the execution of individual command-line utilities.

Most of the command-line utilities, such as startServer.sh or addNode.sh, write data to their own log files. These logs will appear in your profile's "logs" directory, and have the same name as the utility. For most utilities, you can specify an alternate log file on the command line if you prefer.

System application and sample application logs

<name>_deploy.txt, <name>_config.txt

Deployment and configuration logs for each of the enterprise applications installed by the WebSphere Application Server installer (Administrative Console, samples, etc)


The JVM logs are the most useful of the runtime logs. They contain information about the server runtime, and any messages that the application writes to System.out or System.err. You will also find exception and stack trace information in these logs. The native process logs contain information that gets logged by native code, such as the JVM itself, or some parts of the security implementation.

First Failure Data Capture (FFDC)

WebSphere Application Server V7 includes a feature called First Failure Data Capture (FFDC). The FFDC feature runs in the background and collects events and errors that occur during WebSphere Application Server runtime. The information that it collects are written to log files in the WAS_install_root/profiles/profile_name/logs/ffdc directory.

FFDC does not affect the performance of WebSphere Application Server and should not be disabled. The FFDC logs will not, most likely, be useful in your problem determination efforts. However, they might be useful to the WebSphere Application Server support team if you open a PMR.

There are three FFDC configuration files in the WAS_install_root/properties directory. The only file that you should modify is the ffdcRun.properties file. You can add the ExceptionFileMaximumAge property to the file. This property specifies the number of days that an FFDC log remains in the WAS_install_root>/profiles/profile_name/logs/ffdc directory before it is deleted. As part of your diagnostic data collection plan, you might want to modify the ExceptionFileMaximumAge property to ensure that the FFDC files remain on your system for a certain time period. You should not modify any other properties unless you are asked to do so by the WebSphere Application Server support team.

Professional hosting         Free 'Oracle Certified Expert Web Services Developer 6' Guide     Free SCDJWS 5.0 Guide