Chapter 7. Problem Determination

7.1.  Enable high performance extensible logging (HPEL) and view HPEL data.

[Note]

In WebSphere Application Server V8, an alternative to the basic log and trace facility is offered, called High Performance Extensible Logging (HPEL). It provides three repositories:

  • Log data repository: A storage facility for log records, typically information stored in SystemOut.log, SystemErr.log, or java.util.logging at level DETAIL or higher.

  • Trace data repository: A storage facility for trace records, typically information written to java.util.logging below level DETAIL.

  • Text log: Plain text file for log and trace records. Provided for convenience.

Figure 7.1. HPEL repositories

HPEL repositories


Now the data is stored in proprietary binary format, rather than text format as in basic logging. (The only exception is for the text log repository.) In this way, the following benefits are achieved:

  • There is no more text parsing.

  • More data is available due to the fact that truncation does not need to be done.

  • Data is not formatted unless necessary.

  • There is no need to clear log files before server start, for example, to diagnosis a problem.

  • Trace speed is improved and more data can be available (it has half of the impact of basic tracing).

  • It is a common solution between z/OS and distributed platform.

  • Applications running with HPEL run faster than with basic logging.

To read the log and trace records in this new format, a new command called logViewer.sh was introduced. It reads the data from repositories, formats it, and displays it to the administrator.

Enable and configure HPEL

  1. If necessary, start the application server.

  2. Log in to the administrative console, specifying an appropriate user ID and password if administrative security is enabled. Next, click Troubleshooting > Logs and Trace > server1.

    Figure 7.2. 


    Then do the following:

    1. Click Switch to HPEL Mode.

      Figure 7.3. 


    2. Click Configure HPEL Text Log.

      Figure 7.4. 


      Then uncheck Enable text log and click OK.

      Figure 7.5. 


    3. Click Configure HPEL logging.

      Figure 7.6. 


      Then change Maximum log size from 50 to 20 and click OK.

      Figure 7.7. 


    4. Click Configure HPEL trace.

      Figure 7.8. 


      Then, set Log record purging policy to Begin cleanup of oldest records when oldest records reach age limit. Next, set Log record age limit to 12 hours.

      Figure 7.9. 


    5. Click Save to save the changes to the master configuration.

  3. Log out of the administrative console.

  4. Recycle (stop/start) the application server. If administrative security is enabled, you will be required to specify an appropriate user ID and password.

    test317:~ # /opt/IBM/WebSphere/AppServer/profiles/AppSrv01/bin/stopServer.sh server1
    ADMU0116I: Tool information is being logged in file
               /opt/IBM/WebSphere/AppServer/profiles/AppSrv01/logs/server1/stopServer.log
    ADMU0128I: Starting tool with the AppSrv01 profile
    ADMU3100I: Reading configuration for server: server1
    ADMU3201I: Server stop request issued. Waiting for stop status.
    ADMU4000I: Server server1 stop completed.
    								

    test317:~ # /opt/IBM/WebSphere/AppServer/profiles/AppSrv01/bin/startServer.sh server1
    ADMU0116I: Tool information is being logged in file
               /opt/IBM/WebSphere/AppServer/profiles/AppSrv01/logs/server1/startServer.log
    ADMU0128I: Starting tool with the AppSrv01 profile
    ADMU3100I: Reading configuration for server: server1
    ADMU3200I: Server launched. Waiting for initialization status.
    ADMU3000I: Server server1 open for e-business; process id is 58067
    								

Use the HPEL command line log viewer

Since you enabled HPEL and restarted the server, the legacy WebSphere Application Server logging information is no longer available. The command line HPEL log viewer provides a powerful, yet simple solution for viewing logging information.

  1. Open a command prompt and change to the application server profile bin directory, for example:

    test317:~ # cd /opt/IBM/WebSphere/AppServer/profiles/AppSrv01/bin
    								 

  2. Run the following command to obtain help on the log viewer, including an explanation of the options:

    
    test317:/opt/IBM/WebSphere/AppServer/profiles/AppSrv01/bin # ./logViewer.sh help
    
    LogViewer
    Use the logViewer command to view, query and filter the HPEL log and trace repositories.
    
    
    logViewer.bat|sh [options]
    
    
    Options:
    
    -repositoryDir <directory name>
             Specifies the path to the repository directory. In the case where you
             want to query both the log and trace data together, provide the path to
             the parent directory, which contains both the log data and tracedata
             directories. If you use the default repository location,
             profile_root/logs/application_server/, and run this tool from the profile
             bin directory, then this argument is optional. The tool checks the default
             location if one is not provided. If multiple application servers exist in
             this profile with HPEL repositories, you are prompted to select which
             server log and trace repository you want to view.
    ...
    ...
    
    								

  3. Create a legacy format log file and compare its contents to the output of several HPEL log viewer commands.

    1. Run the following command to create a legacy format log file (legacyFormat.log) that contains only log records (INFO, WARNING, and SEVERE):

      test317:/opt/IBM/WebSphere/AppServer/profiles/AppSrv01/bin # ./logViewer.sh -outLog ../logs/legacyFormat.log -minLevel INFO -maxLevel SEVERE
      
      Using /opt/IBM/WebSphere/AppServer/profiles/AppSrv01/logs/server1 as repository directory.
      Operation Complete
      Processed 213 records in 0.155 seconds (1,374.194 records per second).
      											

    2. Run the following command to view only the log records for thread 0:

      test317:/opt/IBM/WebSphere/AppServer/profiles/AppSrv01/bin # ./logViewer.sh -thread 0
      
      Using /opt/IBM/WebSphere/AppServer/profiles/AppSrv01/logs/server1 as repository directory.
      ************ Start Display Current Environment ************
      WebSphere Platform 8.0.0.4 [ND 8.0.0.4 cf041228.02] running with process name test317Cell01\exampleNode01\server1 and process id 58067
      Host Operating System is Linux, version 3.0.34-0.7-default
      Java version = 1.6.0, Java Compiler = j9jit26, Java VM name = IBM J9 VM
      was.install.root = /opt/IBM/WebSphere/AppServer
      user.install.root = /opt/IBM/WebSphere/AppServer/profiles/AppSrv01
      Java Home = /opt/IBM/WebSphere/AppServer/java/jre
      ws.ext.dirs = /opt/IBM/WebSphere/AppServer/java/lib:/opt/IBM/WebSphere/AppServer/profiles/AppSrv01/classes:/opt/IBM/WebSphere/AppServer/classes:/opt/IBM/WebSphere/AppServer/lib:/opt/IBM/WebSphere/AppServer/installedChannels:/opt/IBM/WebSphere/AppServer/lib/ext:/opt/IBM/WebSphere/AppServer/web/help:/opt/IBM/WebSphere/AppServer/deploytool/itp/plugins/com.ibm.etools.ejbdeploy/runtime
      Classpath = /opt/IBM/WebSphere/AppServer/profiles/AppSrv01/properties:/opt/IBM/WebSphere/AppServer/properties:/opt/IBM/WebSphere/AppServer/lib/startup.jar:/opt/IBM/WebSphere/AppServer/lib/bootstrap.jar:/opt/IBM/WebSphere/AppServer/lib/jsf-nls.jar:/opt/IBM/WebSphere/AppServer/lib/lmproxy.jar:/opt/IBM/WebSphere/AppServer/lib/urlprotocols.jar:/opt/IBM/WebSphere/AppServer/deploytool/itp/batchboot.jar:/opt/IBM/WebSphere/AppServer/deploytool/itp/batch2.jar:/opt/IBM/WebSphere/AppServer/java/lib/tools.jar
      Java Library path = /opt/IBM/WebSphere/AppServer/lib/native/linux/x86_64/:/opt/IBM/WebSphere/AppServer/java/jre/lib/amd64/default:/opt/IBM/WebSphere/AppServer/java/jre/lib/amd64:/opt/IBM/WebSphere/AppServer/bin:/usr/lib:
      Orb Version = IBM Java ORB build orb626ifx-20120410.00 (SR2FP1+191257)
      ************* End Display Current Environment *************
      [7/6/13 8:40:21:381 EDT] 00000000 ManagerAdmin  I   TRAS0017I: The startup trace state is *=info.
      [7/6/13 8:40:21:384 EDT] 00000000 ManagerAdmin  I   TRAS0111I: The message IDs that are in use are deprecated
      [7/6/13 8:40:21:413 EDT] 00000000 ModelMgr      I   WSVR0800I: Initializing core configuration models
      ...
      ...
      											

    3. Run the following command to view only WARNING messages:

      test317:/opt/IBM/WebSphere/AppServer/profiles/AppSrv01/bin # ./logViewer.sh -level WARNING
      
      Using /opt/IBM/WebSphere/AppServer/profiles/AppSrv01/logs/server1 as repository directory.
      ************ Start Display Current Environment ************
      WebSphere Platform 8.0.0.4 [ND 8.0.0.4 cf041228.02] running with process name test317Cell01\exampleNode01\server1 and process id 58067
      Host Operating System is Linux, version 3.0.34-0.7-default
      Java version = 1.6.0, Java Compiler = j9jit26, Java VM name = IBM J9 VM
      was.install.root = /opt/IBM/WebSphere/AppServer
      user.install.root = /opt/IBM/WebSphere/AppServer/profiles/AppSrv01
      Java Home = /opt/IBM/WebSphere/AppServer/java/jre
      ws.ext.dirs = /opt/IBM/WebSphere/AppServer/java/lib:/opt/IBM/WebSphere/AppServer/profiles/AppSrv01/classes:/opt/IBM/WebSphere/AppServer/classes:/opt/IBM/WebSphere/AppServer/lib:/opt/IBM/WebSphere/AppServer/installedChannels:/opt/IBM/WebSphere/AppServer/lib/ext:/opt/IBM/WebSphere/AppServer/web/help:/opt/IBM/WebSphere/AppServer/deploytool/itp/plugins/com.ibm.etools.ejbdeploy/runtime
      Classpath = /opt/IBM/WebSphere/AppServer/profiles/AppSrv01/properties:/opt/IBM/WebSphere/AppServer/properties:/opt/IBM/WebSphere/AppServer/lib/startup.jar:/opt/IBM/WebSphere/AppServer/lib/bootstrap.jar:/opt/IBM/WebSphere/AppServer/lib/jsf-nls.jar:/opt/IBM/WebSphere/AppServer/lib/lmproxy.jar:/opt/IBM/WebSphere/AppServer/lib/urlprotocols.jar:/opt/IBM/WebSphere/AppServer/deploytool/itp/batchboot.jar:/opt/IBM/WebSphere/AppServer/deploytool/itp/batch2.jar:/opt/IBM/WebSphere/AppServer/java/lib/tools.jar
      Java Library path = /opt/IBM/WebSphere/AppServer/lib/native/linux/x86_64/:/opt/IBM/WebSphere/AppServer/java/jre/lib/amd64/default:/opt/IBM/WebSphere/AppServer/java/jre/lib/amd64:/opt/IBM/WebSphere/AppServer/bin:/usr/lib:
      Orb Version = IBM Java ORB build orb626ifx-20120410.00 (SR2FP1+191257)
      ************* End Display Current Environment *************
      [7/6/13 8:40:22:329 EDT] 00000000 WSKeyStore    W   CWPKI0041W: One or more key stores are using the default password.
      [7/6/13 8:40:23:628 EDT] 00000000 ThreadPoolMgr W   WSVR0626W: The ThreadPool setting on the ObjectRequestBroker service is deprecated.
      [7/6/13 8:40:28:287 EDT] 00000010 J2EEServiceMa W   ASYN0080W: Registration of the JTA service might not be honored because the service was registered after asynchronous beans was used.
      [7/6/13 8:40:28:299 EDT] 00000010 J2EEServiceMa W   ASYN0080W: Registration of the JavaCompContextMgr service might not be honored because the service was registered after asynchronous beans was used.
      [7/6/13 8:45:34:255 EDT] 00000005 FfdcProvider  W com.ibm.ws.ffdc.impl.FfdcProvider logIncident FFDC1003I: FFDC Incident emitted on /opt/IBM/WebSphere/AppServer/profiles/AppSrv01/logs/ffdc/server1_6e00e877_13.07.06_08.45.34.2471181567819540010440.txt com.ibm.ws.wsgroup.bb.BBPostingMsg 65
      [7/6/13 9:28:31:350 EDT] 00000025 CoordinatorCo W   HMGR0152W: CPU Starvation detected. Current thread scheduling delay is 5 seconds.
      Operation Complete
      Processed 6 records in 0.325 seconds (18.462 records per second).
      											

    4. View the legacy format log file that you created earlier:

      test317:/opt/IBM/WebSphere/AppServer/profiles/AppSrv01/bin # cat ../logs/legacyFormat.log
      
      ************ Start Display Current Environment ************
      WebSphere Platform 8.0.0.4 [ND 8.0.0.4 cf041228.02] running with process name test317Cell01\exampleNode01\server1 and process id 58067
      Host Operating System is Linux, version 3.0.34-0.7-default
      Java version = 1.6.0, Java Compiler = j9jit26, Java VM name = IBM J9 VM
      was.install.root = /opt/IBM/WebSphere/AppServer
      user.install.root = /opt/IBM/WebSphere/AppServer/profiles/AppSrv01
      Java Home = /opt/IBM/WebSphere/AppServer/java/jre
      ws.ext.dirs = /opt/IBM/WebSphere/AppServer/java/lib:/opt/IBM/WebSphere/AppServer/profiles/AppSrv01/classes:/opt/IBM/WebSphere/AppServer/classes:/opt/IBM/WebSphere/AppServer/lib:/opt/IBM/WebSphere/AppServer/installedChannels:/opt/IBM/WebSphere/AppServer/lib/ext:/opt/IBM/WebSphere/AppServer/web/help:/opt/IBM/WebSphere/AppServer/deploytool/itp/plugins/com.ibm.etools.ejbdeploy/runtime
      Classpath = /opt/IBM/WebSphere/AppServer/profiles/AppSrv01/properties:/opt/IBM/WebSphere/AppServer/properties:/opt/IBM/WebSphere/AppServer/lib/startup.jar:/opt/IBM/WebSphere/AppServer/lib/bootstrap.jar:/opt/IBM/WebSphere/AppServer/lib/jsf-nls.jar:/opt/IBM/WebSphere/AppServer/lib/lmproxy.jar:/opt/IBM/WebSphere/AppServer/lib/urlprotocols.jar:/opt/IBM/WebSphere/AppServer/deploytool/itp/batchboot.jar:/opt/IBM/WebSphere/AppServer/deploytool/itp/batch2.jar:/opt/IBM/WebSphere/AppServer/java/lib/tools.jar
      Java Library path = /opt/IBM/WebSphere/AppServer/lib/native/linux/x86_64/:/opt/IBM/WebSphere/AppServer/java/jre/lib/amd64/default:/opt/IBM/WebSphere/AppServer/java/jre/lib/amd64:/opt/IBM/WebSphere/AppServer/bin:/usr/lib:
      Orb Version = IBM Java ORB build orb626ifx-20120410.00 (SR2FP1+191257)
      ************* End Display Current Environment *************
      [7/6/13 8:40:21:381 EDT] 00000000 ManagerAdmin  I   TRAS0017I: The startup trace state is *=info.
      [7/6/13 8:40:21:384 EDT] 00000000 ManagerAdmin  I   TRAS0111I: The message IDs that are in use are deprecated
      [7/6/13 8:40:21:413 EDT] 00000000 ModelMgr      I   WSVR0800I: Initializing core configuration models
      ...
      ...
      											

  4. Run the following command to retrieve all log records NOT from loggers beginning with com.ibm:

    test317:/opt/IBM/WebSphere/AppServer/profiles/AppSrv01/bin #  ./logViewer.sh -excludeLoggers com.ibm.*
    
    Using /opt/IBM/WebSphere/AppServer/profiles/AppSrv01/logs/server1 as repository directory.
    Operation Complete
    Processed 0 records in 0.04 seconds (0 records per second).
    								

  5. Extract a specified set of log records to a new repository and a text file.

    1. Run the following command to extract a repository of just WARNING and SEVERE messages and save the resulting file in a new directory:

      test317:/opt/IBM/WebSphere/AppServer/profiles/AppSrv01/bin # ./logViewer.sh -minLevel WARNING -maxLevel SEVERE -extractToNewRepository ../logs/newHPELRepository
      
      Using /opt/IBM/WebSphere/AppServer/profiles/AppSrv01/logs/server1 as repository directory.
      Operation Complete
      Processed 6 records in 0.163 seconds (36.81 records per second).
      											

    2. Run the following command to export the contents of the resulting repository to a text format log file:

       test317:/opt/IBM/WebSphere/AppServer/profiles/AppSrv01/bin # ./logViewer.sh -repositoryDir ../logs/newHPELRepository -outLog ../logs/newFormat.log
      
      Operation Complete
      Processed 6 records in 0.029 seconds (206.897 records per second).
       											

    3. View the resulting log file, for example:

      test317:/opt/IBM/WebSphere/AppServer/profiles/AppSrv01/bin # cat ../logs/newFormat.log
      
      ************ Start Display Current Environment ************
      WebSphere Platform 8.0.0.4 [ND 8.0.0.4 cf041228.02] running with process name test317Cell01\exampleNode01\server1 and process id 58067
      Host Operating System is Linux, version 3.0.34-0.7-default
      Java version = 1.6.0, Java Compiler = j9jit26, Java VM name = IBM J9 VM
      was.install.root = /opt/IBM/WebSphere/AppServer
      user.install.root = /opt/IBM/WebSphere/AppServer/profiles/AppSrv01
      Java Home = /opt/IBM/WebSphere/AppServer/java/jre
      ws.ext.dirs = /opt/IBM/WebSphere/AppServer/java/lib:/opt/IBM/WebSphere/AppServer/profiles/AppSrv01/classes:/opt/IBM/WebSphere/AppServer/classes:/opt/IBM/WebSphere/AppServer/lib:/opt/IBM/WebSphere/AppServer/installedChannels:/opt/IBM/WebSphere/AppServer/lib/ext:/opt/IBM/WebSphere/AppServer/web/help:/opt/IBM/WebSphere/AppServer/deploytool/itp/plugins/com.ibm.etools.ejbdeploy/runtime
      Classpath = /opt/IBM/WebSphere/AppServer/profiles/AppSrv01/properties:/opt/IBM/WebSphere/AppServer/properties:/opt/IBM/WebSphere/AppServer/lib/startup.jar:/opt/IBM/WebSphere/AppServer/lib/bootstrap.jar:/opt/IBM/WebSphere/AppServer/lib/jsf-nls.jar:/opt/IBM/WebSphere/AppServer/lib/lmproxy.jar:/opt/IBM/WebSphere/AppServer/lib/urlprotocols.jar:/opt/IBM/WebSphere/AppServer/deploytool/itp/batchboot.jar:/opt/IBM/WebSphere/AppServer/deploytool/itp/batch2.jar:/opt/IBM/WebSphere/AppServer/java/lib/tools.jar
      Java Library path = /opt/IBM/WebSphere/AppServer/lib/native/linux/x86_64/:/opt/IBM/WebSphere/AppServer/java/jre/lib/amd64/default:/opt/IBM/WebSphere/AppServer/java/jre/lib/amd64:/opt/IBM/WebSphere/AppServer/bin:/usr/lib:
      Orb Version = IBM Java ORB build orb626ifx-20120410.00 (SR2FP1+191257)
      ************* End Display Current Environment *************
      [7/6/13 8:40:22:329 EDT] 00000000 WSKeyStore    W   CWPKI0041W: One or more key stores are using the default password.
      [7/6/13 8:40:23:628 EDT] 00000000 ThreadPoolMgr W   WSVR0626W: The ThreadPool setting on the ObjectRequestBroker service is deprecated.
      [7/6/13 8:40:28:287 EDT] 00000010 J2EEServiceMa W   ASYN0080W: Registration of the JTA service might not be honored because the service was registered after asynchronous beans was used.
      [7/6/13 8:40:28:299 EDT] 00000010 J2EEServiceMa W   ASYN0080W: Registration of the JavaCompContextMgr service might not be honored because the service was registered after asynchronous beans was used.
      [7/6/13 8:45:34:255 EDT] 00000005 FfdcProvider  W com.ibm.ws.ffdc.impl.FfdcProvider logIncident FFDC1003I: FFDC Incident emitted on /opt/IBM/WebSphere/AppServer/profiles/AppSrv01/logs/ffdc/server1_6e00e877_13.07.06_08.45.34.2471181567819540010440.txt com.ibm.ws.wsgroup.bb.BBPostingMsg 65
      [7/6/13 9:28:31:350 EDT] 00000025 CoordinatorCo W   HMGR0152W: CPU Starvation detected. Current thread scheduling delay is 5 seconds.
      											

Use the HPEL log viewer included in the WebSphere administrative console

  1. Log in to the administrative console, specifying an appropriate user ID and password if administrative security is enabled. Next, click Troubleshooting > Logs and Trace > server1.

    Then click View HPEL logs and trace.

    Figure 7.10. 


  2. Expand the Content and Filtering Details area of the page. Click each server instance (start time) to view the corresponding log messages.

    Figure 7.11. 


    Then collapse the Content and Filtering Details area.

  3. Click the Server Instance information button to view information about the current run of the application server such as the process ID (PID).

    Figure 7.12. 


    Then close the Server Instance information pop-up window.

  4. Click the (early) log message TRAS0017I to view explanations, user actions, and so on.

    Figure 7.13. 


    Then click Close to remove the pop-up window.

  5. Locate a log entry indicating activity on a thread other than 00000000. Then select the entry and click Show Only Selected Threads to filter the list to include only records generated by the same thread.

    Figure 7.14. 


    Finally, click Show All Threads to once again display messages generated by all threads.

  6. Show only WARNING and higher level messages.

    1. Expand the Content and Filtering Details area of the page.

    2. Uncheck System out and System err.

    3. Under Logs and trace, set Minimum level to WARNING and Maximum level to FATAL.

      Figure 7.15. 


    4. Click the Apply button to filter the list. It may take several seconds for the filtering to be applied. Then collapse the Content and Filtering Details area of the page and view the resulting messages.

  7. Log out of the administrative console.

  8. Optionally, stop the application server.

Professional hosting         Free 'Oracle Certified Expert Web Services Developer 6' Guide     Free SCDJWS 5.0 Guide