WebLogic Server along with the JVM offers various ways to
collect logs, debug traces, diagnostic images, thread dumps, and much more.
However, useful data can easily get buried with lots of noise, unneeded data
that won't help resolving a particular problem.
Performance Issues
We will start with performance issues such as long running
requests or server hangs. The best way to narrow down where the issue
lies is to dump the java threads and
parse them through a thread dump analyzer tool such as ThreadLogic. Looking
at the server log files, or enabling some debugging features would be premature
since the first step is to identify the thread(s) that are stuck to determine
what is being executed and relevant JEE resources. The log could also alert
about stuck threads once they have been stuck for over the maximum allowed time
(see MaxStruckThreadTime) but
unless thread dumps are examined, it won't be possible to determine possible
patterns of issues and threads' relationship around locked objects.
Memory Leaks
Another common performance issue involves memory leaks. If
you encounter OutOfMemory java.lang exceptions you should issue a heap dump and
analyze it with a tool such as VisualVM or the Eclipse
MAT. See my post on Heap Dumps analysis for
more details. Heap dumps are snapshots of memory certain times. Heap
dumps are to the Java process memory (non native) what thread dumps are to Java
process threads. Enabling -verbose:gc is also an effective way to monitor
heap usage at runtime. The output of the GC stats can be redirected to a
file so that not to clog the server logs. This can be done by setting
-Xloggc with HotSpot JVM. It's recommended to
include -XX:+PrintGCDetails and -XX:+PrintGCTimeStamps to log time
stamps and detailed GC activities as well. Troubleshooting guide for HotSpot VM details
these flags among others useful flags for diagnosis purposes.
Applications, 3rd parties, and configuration issues
The following chart summarizes what could be collected to debug
each of the listed WLS component. This list is by no means exhaustive.
Many more debug flags could be used for these listed components.
This chart contains examples of useful debug options and links to
additional resources.
For the following, Redirect stdout and stderr logging should
be enabled. This can be done via the Administration Console in the
advanced section of the servers logging tab or at startup in the java command
line with -Dweblogic.StdoutDebugEnabled=true. In addition, the severity
level of the log file should be set to debug.
WLS comp. or 3rd party resource
|
Examples of Debug options and flags
|
MyOracleSupport related resources
(login required)
|
EJB/Web Serv.
|
-Dweblogic.webservice.verbose=true
-Dweblogic.webservice.client.verbose=true
To log SOAP requests and response messages
Trace in servers log files
|
|
JMS
|
-Dweblogic.debug.DebugJMSStore
To log info on load, store events and transaction records into servers
log files
|
|
SSL
|
-Dssl.debug=true
-Dweblogic.security.SSL.verbose=true
Trace in servers log files
|
|
Clustering
|
-Dweblogic.debug.DebugReplication for high level replication
info
-Dweblogic.debug.DebugClusterAnnouncements to log info on
announcement, StateDump and attributes messages sent or received by multicast
Trace in servers log files
|
|
JTA
|
weblogic.JTAXA & weblogic.JTA2PC for runtime issues,
example:
JAVA_OPTIONS="-Dweblogic.Debug=weblogic.JTAXA, weblogic.JTA2PC"
Trace in servers log files
|
|
JDBC
|
-Dweblogic.debug.DebugJDBCSQL=true
to print information about JDBC methods invoked with
their arguments, return values, and
thrown exceptions
Trace in servers log files
|
|
Oracle JDBC Driver (*)
|
-Doracle.jdbc.Trace=true to enable jdbc logging
Trace is file defined in oracle.jdbc.LogFile
|
|
Deployment (Classloader)
|
Set the following
options to true to report which classes are getting loaded
-Dweblogic.utils.classloaders.GenericClassLoader.Verbose
-Dweblogic.utils.classloaders.ChangeAwareClassLoader.Verbose
-Dweblogic.utils.classloaders.ClasspathClassFinder
-Dweblogic.utils.classloaders.DefaultFilteringClassLoader.Verbose
-Dweblogic.utils.classloaders.FilteringClassLoader.Verbose
-Dweblogic.utils.classloaders.FilteringClassLoader.ResourceDump
Trace in servers log files
|
|
Proxy plug-in
|
Debug="ALL" in the proxy configuration file
Trace in file set in WLLogFile defined inside proxy
configuration file
|
|
(*) Enabling JDBC Driver debug flags is verbose so it's
recommended to set logging properties to appropriate levels such as FINE,
SEVERE, FINEST, INFO etc. based upon the debugging requirement.
WLDF
The Weblogic Diagnostic Framework can
be used to collect metrics, setup watch and notifications, and to define
instrumentation. With WLS 12.1.2, you can set the level (Low, Medium or
High) of built-in modules that
will gather metrics, or disable them completely. WLDF collects runtime
statistics on JDBC, JTA and JVM and more. You can then use the Monitoring Dashboard to
view and navigate through the collected data. WLDF is not an alternative
to debug options. Debug options activate tracing that is coded as part of
WLS whereas WLDF collects and monitors runtime mbeans values and can activate
notifications with defined rules. They are used for different purposes.
The WLDF built-in configurations (Low, Medium, & High) can
be cloned into a new configuration, and used as templates for creating custom
WLDF configurations. The Low volume built-in is enabled by default in
production mode. One new feature with WLS 12.1.2 is Runtime control which gives the
ability to activate or deactivate diagnostic system modules dynamically at
runtime wihtout making any domain configuration change.
JFR
JFR recording data can
also be captured through WLDF diagnostic images based on WLDF watch rules so
that JVM runtime system information can be analyzed along with recording of WLS
components diagnostic data. Once extracted from the diagnostic image
capture, the JFR file can be analyzed using Java Mission Control.
WLST
WLST diagnostic commands can
be used to extract data from the diagnostic archive (event and harvested metric
data) in either XML form or CSV, based on the function
you use. You can also dump the diagnostics data from a harvester to a
local file.
The following WLST functions allow to capture an image and to
download image files from a remote system over WLST:
One major difference between the diagnostic framework and the
debug options is that debugs are enabled so that an issue can be reproduced and
traced whereas the diagnostic image capture is used as a server-level dump for
post-failure analysis in a similar way heap dumps are used post memory leak
failures.
___________________________________________________________________________
JSTACK
jstack can be use to dump the java threads. While working with WebLogic Server, you can dump the java threads with various tools such as WLST. jstack can do just that but differs from <ctrl>+<break> or kill -3 since it doesn't dump a heap summary along with the threads.
In addition, jstack can print lock information with -l option and the list of synchronizers that may be exclusively owned by a thread. Such information can be very useful while monitoring concurrent access by threads.
jstack comes as part of the JDK installed with WebLogic Server. With instances of WLS running, you can open a new shell, execute the setDomain script and run jstack or any tool that's part of the JDK package. Thread dumps taken by jstack can be redirected to files, e.g. jstack <pid> > myTD.txt.
PIDs can be obtained on Solaris/Linux with the ps (report process status) command, e.g. ps -ef | grep java, and from the task managed on Windows. In task manager you might need to go to View and Select Columns to add PID or process identifiers.
______________________________________________________________________
Thread dumps
Introduction
A Java thread dump is a snapshot what every thread in the JVM is
doing at a particular point in time. Each thread in the JVM is listed with it's
name and id, it's current state and the Java call stack showing what monitor it
has locked or is waiting on. This is especially useful if your Java application
sometimes seems to hang when running under load, as an analysis of the dump
will show where the threads are stuck, either by deadlock or other thread
contention.
How to get a thread dump
Unix and Linux
On most flavors of Unix kill -3 <pid> will generate a thread
dump. To get the java process pid use ps -ef | grep java. The dump is directed
to stdout so redirecting to a file is neccessary. Since 1.6 the Sun/Oracle JDK
has the jstack utility to do the same thing.
Enter the
following at the command line at startup:
bin\jrcmd.exe <pid> print_threads
The
thread dump appears at the command line.
Windows:
If the application was
started from a command window you can select the window and use CTRL+Breakbut
the output will likely overflow the window. Jstack is available if your using
the Sun JDK. Otherwise use PsExec.exe ( http://technet.microsoft.com/en-us/sysinternals/bb897553.aspx )
to execute SendSignal.exe.
How many thread dumps do I need
Often a single thread dump will reveal the problem, this is
especially true with deadlocks where two or more threads are waiting for locks
obtained by each other. But in other cases, such as threads waiting on other
processes, like IO from a database, you won't be able to detect it with a
single thread dump. In this case multiple thread dumps taken over time will
show the same thread waiting for the process to complete during that time span.
Understanding
Thread States
Before starting on dissecting thread dumps it is very
important to get a good understanding of the common thread states.
- NEW : A thread is in a new state when
it has just been created and the start() method hasn't been invoked
- RUNNING : A thread is in a runnable
state when the start method has been invoked and the JVM actively
schedules it for execution
- BLOCKED / WAITING FOR MONITOR ENTRY: Thread enters
this state when it isn't able to acquire the necessary monitors to enter a
synchronized block. A thread waits for other threads to reliquish the
monitor. A thread in this state isnt doing any work
- WAITING : A thread enters this state when it relinquishes
its control over the monitor by calling Object.wait(). It waits till the
time other threads invoke Object.notify() or Object.notifyAll(). In this
state too the thread isnt doing any work
- TIMED WAITING : A thread enters this state
when it relinquishes its control over the monitor by calling
Object.wait(long). It waits till the time other threads invoke
Object.notify() / Object.notifyAll() or when the wait time expires. In
this state too the thread isnt doing any work
- SLEEPING / WAITING ON CONDITION: A thread
sleeps when it calls Thread.sleep(long). As the name suggests the thread
isnt doing in this state either
- Terminated
- Other
As you can see from the above list a thread is only doing active work when it
is in the RUNABBLE state. In all other states the thread is just wasting CPU
cycles. For maximum throughput from an application it is necessary to ensure
that the maximum number of threads are in the RUNNABLE state and fewer threads
in WAITING FOR MONITOR ENTRY, SLEEPING, WAITING or TIMED WAITING state.
l Threads
present in a typical weblogic thread dump
l JVM
threads { GC threads, Code optimisation etc }
l ListenThread.Default
l Weblogic.socket.Muxer
l Weblogic.kernel.System
l Weblogic.admin.RMI
l Weblogic.admin.HTTP
l Weblogic.kernel.Defaul
l VDE
Transaction Processor Thread
l Weblogic.health.CoreHealthMonitor
l Miscellaneous
Threads ( Timer, Security etc. )
Analyzing the thread dumps
You can read a thread dump with a text editor, and many people use
only this.
The Beginning of The Thread Dump
The thread dump starts with the
date and time of the dump, and the version number of the JRockit JVM used (see Listing 20-1).
Listing 20-1
The initial information of a thread dump
===== FULL THREAD DUMP ===============
Wed Feb 21 13:46:45 2007
BEA JRockit(R) R27.1.0-109-73164-1.5.0_08-20061129-1428-windows-ia32
Format
of thread dumps
Full
thread dump Java HotSpot(TM) Client VM
(1.5.0_04-b05 mixed mode, sharing):
"Thread-1" prio=5 tid=0x00a995d0 nid=0x1300 in Object.wait()[0x02d0f000..0x02d0fb68]
at java.lang.Object.wait(Native Method)
- waiting on <0x22aaa8d0> (a org.tw.testyard.thread.Drop)
at java.lang.Object.wait(Unknown Source)
at org.tw.testyard.thread.Drop.take(Drop.java:14)
- locked <0x22aaa8d0> (a org.tw.testyard.thread.Drop)
at org.tw.testyard.thread.Consumer.run(Consumer.java:15)
at java.lang.Thread.run(Unknown Source)
"Thread-0" prio=5 tid=0x00a88440 nid=0x6a4 waiting
on condition[0x02ccf000..0x02ccfbe8]
at java.lang.Thread.sleep(Native Method)
at org.tw.testyard.thread.Producer.run(Producer.java:24)
at java.lang.Thread.run(Unknown Source)
|
In the thread dump snippet above you can observe the following
- The
thread dump starts with "Full thread dump", followed by a list
of threads currently being executed.
- There
are 2 application threads called Thread-1 and Thread-0. These are the
default names which the JVM handed over to the threads.
- "Thread-1"
is waiting for a notification after it called Object.wait() on the Drop
object.
- Similarly,
"Thread-0" is sleeping on a condition after it called
Thread.sleep
- At
this particular instant, there are no threads in the runnable state and
hence the application isn't doing any real work.
Although
thread dumps also lists the state of the system threads we are not going to
look deeper into system threads.
Thread Dump: There are 3 tool are available in the market to analysis the thread dump.
1.Samurai.
2.Thread Dump Analyzer
3.Thread Logic.
1. Samurai:
Samurai description
A GUI based tail, thread dump analysis tool
Samurai is a tabbed GUI tail application. Samurai will colors
idle threads in gray,
blocked threads in red and all the
running threads in green.
There are three result views and Samurai shows "Table view" by default. In many case you are just interested in the table view and the sequence view.
Use the table view to decide which thread needs be inspected, the sequence view to understand the thread's behavior. You should take care especially threds always in red.
Here are some key features of "Samurai":
Picks thread dumps from std/stderr output and colorize them.
Visualize the "-verbose:gc" log.
Works as a GUI based "tail -f"
Requirements:
JDK 1.4 or later
_______________________________________________________________________________
Thread Logic:
Download:
https://java.net/projects/threadlogic/downloads
Thread Dump Analysis is a key tool for performance tuning and troubleshooting
of Java based applications. The current set of TDA tools (Samurai/TDA) dont
mine the thread dumps or provide a more detailed view of what each thread is
doing while just limiting themselves to reporting the state (locked/waiting/running)
or the lock information.
Once a thread dump is parsed and threads details are populated, each of the
thread is then analyzed against matching advisories and tagged appropriately.
The threads are also associated with specific Thread Groups based on
functionality or thread group name. Both the advisories and grouping are
managed via xml definition files which can be modified or extended.
Each of the advisory has a health level indicating severity of the issue found,
pattern, name, keyword and related advice.
Thread Groups
One can see the thread groups are divided into two buckets - WLS and non-WLS
related threads. The JVM threads, LDAP and other unknown custom threads go
under the non-WLS bucket while all the WLS, Muxer, ADF, Coherence, Oracle,
SOA, JMS, Oracle Adapter threads are all under the WLS bucket. The
classification can be changed by modifying the GroupsDefn xml files.
The health levels (in descending of severity) are FATAL (meant for Deadlocks,
STUCK, Finalizer blocked etc), WARNING, WATCH (worth watching), NORMAL
and IGNORE. Based on the highest severity of threads within a group, that
health level gets promoted to the Thread Group's health level and same is
repeated at the thread dump level. There can be multiple advisories tagged to a
Thread, Thread Group and Thread Dump.
Each of the advisory gets triggered based on either call execution patterns
observed in the thread stack or presence of other conditions (thread blocked or
multiple threads blocked for same lock can trigger BlockedThreads Advisory).
Sometimes a thread might be tagged as IGNORE or NORMAL based on its
execution logic or might be tagged more specifically as involved in JMS send or
receive client or a Servlet thread. The advisories are generated based on an
advisory xml map that is extensible.
___________________________________________________________________________
Heap Dump
Heap dump:
A Heap Dump is a snapshot of the Java process heap memory at a
given time. It's a useful tool to troubleshoot memory leaks and other
memory related issues. Heap dumps are to memory what thread dumps are to
java threads.
There are different ways to dump the java heap used by a Java
application such as WebLogic Server. Heap dumps can be triggered automatically by JVMs such as
HotSpot or JRockit when an OutOfMemory event occurs.
Example:
java -XX:+HeapDumpOnOutOfMemoryError
-XX:HeapDumpPath=D:\myApp\hprof-dumps myApp
This command will generate a heap dump file in the defined heap
dump directory when an OOM is thrown.
The file will be in HPROF binary format (*.hprof). If no
path is defined the file will be created in the directory from which the java
process was launched. The format of the file with JRockit is jrockit_pid,hprof, pid being
the process ID of this JVM instance. With HotSpot the format of the file
will be java_pidpid.hprof.
Example:
java.lang.OutOfMemoryError: Java heap space
Dumping heap to D:\java_pid4148.hprof ...
Heap dump file created [24921731 bytes in 1.400 secs]
For WebLogic Server these properties, HeapDumpOnOutOfMemoryError and HeapDumpPath(optional), can be added directly
to the startup script that contain the Java command that launches WLS. On
Windows the script (with default name startWebLogic.cmd) lives in the domain bin directory and the
parameters can be directly added to the existing JAVA_OPTIONS.
With a JRockit VM process the following command can also be used
to generate a heap dump.
Example:
jrcmd pid hprofdump filename=name_of_dump_file
However if you are using this command against a non JRockit
process the following error will come up:
java.io.IOException: Command failed in target
VM
You can also manually generate a heap dump with tools such as jmap or VisualVM.
Example:
jmap -heap:live,format=b,file=filename pid
This command will generate a heap dump file in binary format and
will only count live objects (the live option is optional). Live objects
are used by applications and generally reside in the Young Generation part of
the heap.
As illustrated in the screenshot below, generating a heap dump
with VisualVM is very simple. VisualVM is shipped as part of the HotSpot
JVM (/bin/jvisualvm.exe). However, VisualVM can connect to java
processes that started with other JVM such as JRockit.
Once generated, the heap dump will be parsed as shown below:
VisualVM enables you to browse heap dumps
as shown above with the Classes view where you can easily identify the number
and percentage of instances referred by each class.
jhat can also be used
to analyze heap dump files.
Example:
jhat dumpfile
Reading from dumpfile...
Dump file created Fri May 17 15:25:41 PDT 2013
Snapshot read, resolving...
Resolving 580390 objects...
Chasing references, expect 116
dots.............................................
.......................................................................
Eliminating duplicate references................................................
....................................................................
Snapshot resolved.
Started HTTP server on port 7000
Server is ready.
Then you can open http://localhost:7000/ to navigate through
the content of the dump file to identify objects with high number of
occurrences.
Finally, as shown below, the OpenSource Eclipse
MAT offers a very rich GUI that includes histograms and leak
hunter actions, overview and leak suspect reports and much more.
__________________________________________________________________________________________
jhat - Java Heap Analysis Tool
jhat [ options ] <heap-dump-file>
PARAMETERS
Options, if
used, should follow immediately after the command name.
heap-dump-file
Java binary heap
dump file to be browsed. For a dump file that contains multiple heap dumps, you
may specify which dump in the file by appending "#<number> to the
file name, i.e. "foo.hprof#3".
The jhat command parses a java heap dump file
and launches a webserver. jhat enables you to browse heap dumps using your
favorite webbrowser. jhat supports pre-designed queries (such as 'show all
instances of a known class "Foo"') as well as OQL (Object Query Language) - a SQL-like query
language to query heap dumps. Help on OQL is available from the OQL help page
shown by jhat. With the default port, OQL help is available at
http://localhost:7000/oqlhelp/
There are several ways to generate a java heap dump:
·
Use jmap -dump
option to obtain a heap dump at runtime;
·
Heap dump will be generated when OutOfMemoryError is thrown by
specifying -XX:+HeapDumpOnOutOfMemoryError VM option;
NOTE: This tool
is experimental and may not be available in future versions of the
JDK.
OPTIONS
-stack false/true
Turn off
tracking object allocation call stack. Note that if allocation site information
is not available in the heap dump, you have to set this flag to false. Default
is true.
-refs false/true
Turn off
tracking of references to objects. Default is true. By default, back pointers
(objects pointing to a given object a.k.a referrers or in-coming references)
are calculated for all objects in the heap.
-port port-number
Set the port for
the jhat's HTTP server. Default is 7000.
-exclude exclude-file
Specify a file
that lists data members that should be excluded from the "reachable
objects" query. For example, if the file lists java.lang.String.value, then,
whenever list of objects reachable from a specific object "o" are
calculated, reference paths involving java.lang.String.value field will not considered.
-baseline baseline-dump-file
Specify a
baseline heap dump. Objects in both heap dumps with the same object ID will be
marked as not being "new". Other objects will be marked as
"new". This is useful while comparing two different heap dumps.
-debug int
Set debug level
for this tool. 0 means no debug output. Set higher values for more verbose
modes.
-version
Report version
number and exit.
-h
Output help
message and exit.
-help
Output help
message and exit.
-J<flag>
Pass
<flag> to the Java virtual machine on which jhat is run. For example,
-J-Xmx512m to use a maximum heap size of 512MB.
_____________________________________________________________________________
CORE DUMP
Core dump is usually a Binary file which gets generrated by the operating system when JVM or any other
process crashes. some times it also happens that the jvm will not be able to generate the crash dump.
Path:
Windows: c:\Documents and settings\All Users\Application Data\Microsoft\Dr Watson.
By default in Unix based operating systems the core dump files are created in the directory where the Java
program/server was started even sometimes it is generated in the "/Tmp" directory of the operating system.
but we can change the location to create crash/Heap dump.
-XX:HeapDumpPath=/opt/app/Prefered Location and -XX:+Heap Dump jvm Options.
Filename:
Sund JDK: "hs_err_pid<weblogicPID>.log
for JRockit Jvm also generates a textual file with name "*.dump"
Flags:
Sun JDK JAVA_OPTIONS -XX:+showMessageBoxOnError.
JRockit JAVA_OPTIONS -Djrockit.waitonerror.
Cause JVM Crash.
1.Native Code causes the JVM crash (c/C++,JNI-Javanative interface APIs)
2.JDBC Drivers Specially the Native drivers
3.JVM code optimization
4.Less Memory availability for native area of a java process.
5.Application Server Native Performance Pack Libraries.
6.JVMs library itself can cause the Crash.
7.High CPU Utilization by the threads.
Tools.
1. window's : Dr.Watson start->run->drwtsn32.
2. Solaris : Pstack and Pmap
3. Linux : lsstack
