Thread Dump Skills Part-1

Thread Dump Skills are Very Important and Mandatory Skills in Today's J2EE Middleware world.
This skill help's us in identifying  the Performances and Root causes of  most of the Server/Infrastructure related issues. What we should aim to learn:

      1)      Thread Dump overview & fundamentals understanding 
      2)      Thread Dump generation techniques and available tools

      3)      Thread Dump format Sun HotSpot

      4)      Thread Stack Trace explanation and interpretation

      5)      Thread Dump analysis and correlation techniques

  6)  Thread Dump common problem patterns (Thread race, deadlock, hanging IO calls, garbage collection OutOfMemoryError problems, infinite looping etc.)

      7)      Thread Dump examples via real life case studies

  


Before we jump in to the Internals of the Thread Dumps You need to understand the basics of the JVM environment

Java VM overview

The Java virtual machine is really the foundation of  weblogic EE platform. This is where your middleware and applications are deployed and active.
The JVM provides the middleware software and your Java / Java EE program with:

-        A runtime environment for your Java / Java EE program (bytecode format)

-        Several program features and utilities (IO facilities, data structure, Threads management, security, monitoring etc.)

-        Dynamic memory allocation and management via the garbage collector


Your JVM can reside on many OS (Solaris, AIX, Windows etc.) and depending of your physical server specifications, you can install 1...n JVM processes per physical / virtual server.

JVM and Weblogic interactions

Find below a diagram showing you a high level interaction view between the JVM, webloic and application(s).
 
This is showing you a typical and simple interaction diagram between the JVM, middleware and application. As you can see, the Threads allocation for a standard Java EE application are done mainly between the middleware kernel itself and JVM 

Also, please note that certain Threads are managed internally within the JVM itself such as GC (garbage collection) Threads in order to handle concurrent garbage collections.


Since most of the Thread allocations are done by the Java EE container, it is important that you understand and recognize the Thread Stack Trace and identify it properly from the Thread Dump data. This will allow you to understand quickly the type of request that the Java EE container is attempting to execute.


From a Thread Dump analysis perspective, you will learn how to differentiate between the different Thread Pools found from the JVM and identify the request type.

Weblogic servers uses WorkManager for carrying out/executing all the work internally

WLS Work Managers Key Components

•          New Thread Pool Implementation: Single internally managed thread pool and priority-based request queue service all application requests

•          Request “Priority” dynamic and internally computed to meet application-defined goals

•          Thread Count Self-Tuning

•          Self-tuning thread pool monitors overall throughput every two sec

•          Present thread count, measured throughput, and past history determines if thread count needs to change

•          New threads automatically added/removed as needed

Benefits Administrators and Operators  - no need to conduct tedious performance testing or guesswork just to pick a static thread pool size that does not adapt to changing workloads

Basics of Thread States

A Thread can be in one of the following states:

1. New
2. Runnable
3. Non-Runnable
• Sleep for a time duration
• Wait for a condition/event
• Blocked for a lock
4. Dead

Following image shows the transition between thread states.

 


In a thread dump, we are looking at threads that have been created already and are either in running or non-running states. So, the new (unless a new thread just got created at the exact moment the thread dump was generated) & dead states are really not of value or present in a thread dump.


Running state implies the thread is actively working on something. Coming to the non-runnable states, its possible a thread has nothing to do and sleeps for some duration and periodically checks for condition to start work. Wait for a condition implies the thread is waiting for some form of notification or an event and can start work once there is a green light. Its much more efficient to use waiting for a condition pattern instead of regular sleep-wake up pattern for optimal usage of resources. If there are multiple threads regularly doing a sleep and wake up periodically, they can be optimized to wake up on notify of an event and only one thread would be successful in getting the notify call instead of all doing the regular check for event in the sleep pattern.


Blocked implies it cannot proceed with its work till it can obtain a lock which is currently held by someone else. This is similar to obtaining a lock to a Critical Region or Semaphore (in OS semantics) before proceeding with the work.

JVM Thread Dump – what is it?

A JVM Thread Dump is a snapshot taken at a given time which provides you with a complete listing of all created Java Threads.

Thread Dumps

A Thread Dump is a brief snapshot in textual format of threads within a Java Virtual Machine (JVM). This is equivalent to process dump in the native world. Data about each thread including the name of the thread, priority, thread group, state (running/blocked/waiting/parking) as well as the execution stack in form of thread stack trace is included in the thread dump. All threads – the Java VM threads (GC threads/scavengers/monitors/others) as well as application and server threads are all included in the dump. Newer versions of JVMs also report blocked thread chains (like ThreadA is locked for a resource held by ThreadB) as well as deadlocks (circular dependency among threads for locks).