Friday, December 12, 2008

Mapping java thread to pstack and pmap

Starting from solaris9, we have a one-one threading model. Solaris library libthread.so schedules the solaris threads to run on LWPs. LWPs are mapped 1-1 with kernel threads. The kernel threads are the ones that are scheduled on processor's dispatch queue. With no debugger handy, if we need comprehensive information related to a thread or set of threads, we would need:
1) java thread dumps(kill -3 pid)
2) pstack (pstack pid)
3) pmap (pmap )


Here is an example on how we can related all the three outputs:
1) Java thread dump:
This is single thread stack trace taken from java thread dump:
The below thread has a java level thread identifier "0x00651988" that is passed as argument for the native thread call.
It also has native thread id as nid=0x2c6 that corresponds to a LWP.

"Thread-646" daemon prio=10 tid=0x00651988 nid=0x2c6 runnable [0x629ff000..0x629ff8f0] at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:129) at java.net.SocketInputStream.read(SocketInputStream.java:182) at com.novell.ldap.asn1.ASN1Identifier.(Unknown Source) at com.novell.ldap.Connection$ReaderThread.run(Unknown Source) at java.lang.Thread.run(Thread.java:595)


2) Pstack Output
The corresponding native thread for the above java thread is shown below: (obtained from executing #pstack pid) nid=0x2c6, the corresponding decimal representation for 0x2c6 is 710, so we got to look for thread# 710 from pstack


----------------- lwp# 710 / thread# 710 -------------------- ff31f4a8 read (32, 629fec98, 1)
fead346c JVM_Read (32, 629fec98, 1, 8460, 651988, ff0148f8) + b8
6f89c1a4 Java_java_net_SocketInputStream_socketRead0 (651a44, 629ff500, 32, 629ff4f8, 0, 1) + 1fc
f960fe6c ???????? (d1265768, d12653b0, d1301f58, 0, 1, 0)
fa6946e4 ???????? (ffffffff, d1265308, 1, 1, 651988, ff150000)
f904e644 ???????? (d1265768, b6, 0, 8388, 80000000, 1)
f90058b8 ???????? (d1265768, b6, 0, f9015d28, 74f6f258, 629ff640)
f90058b8 ???????? (d1301f40, b7, 0, f9015ce0, d1379800, 629ff6c8)
f9005764 ???????? (d1265830, 7082c430, 0, f9015f40, 1, 629ff790)
f9005c2c ???????? (0, ff013928, 0, f9016230, 7ce8, 629ff808)
f9000218 ???????? (629ff8f0, 629ffaf0, a, 7082d668, f900a840, 629ffa08)
…………..
feadb2e4 __1cKJavaThreadDrun6M_v_ (651988, ff01882c, 6f7c, 0, 6c00, 8c00) + 2b0
fee71a00 __1cG_start6Fpv_0_ (651988, 732b, fefc0000, 0, 4ff4, 4c00) + 208
ff395370 _lwp_start (0, 0, 0, 0, 0, 0)


A pstack is native representation of the java thread stack trace. There are 6 arguments for each function that are loaded into the registers. This is Solaris standard register allocation model (there are 6 registers for parameter passing) In the above example we see 6 arguments as:
fee71a00 __1cG_start6Fpv_0_ (651988, 732b, fefc0000, 0, 4ff4, 4c00) + 208
fee71a00 is the program counter. (Virtual address space where the library is loaded)
651988 is the first arguments thats pushed to register0 and note this is the tid from java thread dump (tid=0x00651988)


3) pmap output (pmap pid)
We can trace the library loaded based on the native stack with help of pmap. Refering the same example:
ff395370 _lwp_start (0, 0, 0, 0, 0, 0)
Check the PC value "ff395370" and see where it fits in pmap:
pmap output indicates as below: (We know ff395370 fits in this range)

FF380000 96K r-x-- /usr/lib/libthread.so.1
FF3A8000 8K rwx-- /usr/lib/libthread.so.1

This tells _lwp_start function is from libthread.so library


Similarly:
6f89c1a4 Java_java_net_SocketInputStream_socketRead0 (651a44, 629ff500, 32, 629ff4f8, 0, 1) + 1fc
Based on pmap output, "6f89c1a4" fits in:
6F890000 72K r-x-- /WebLogic/utils/bea922/jdk150_10/jre/lib/sparc/libnet.so
6F8B0000 8K rwx-- /WebLogic/utils/bea922/jdk150_10/jre/lib/sparc/libnet.so

This tells the Java_java_net_SocketInputStream_socketRead function was from libnet.so


Finally,
fead346c JVM_Read (32, 629fec98, 1, 8460, 651988, ff0148f8) + b8
Based on pmapt output "fead346c" fits in:
FE800000 7936K r-x-- /WebLogic/utils/bea922/jdk150_10/jre/lib/sparc/server/libjvm.so
FEFC0000 400K rwx-- /WebLogic/utils/bea922/jdk150_10/jre/lib/sparc/server/libjvm.so

This tells JVM_Read function is from libjvm.so

2 comments: