GenerationalGC Vs G1GC

This blog is to just showcase difference between generational GC Vs G1GC on JDK1.7.0_45. For comparison, I took JMC JFR (flight recording) for 3mins with default GC Vs G1GC on WebLogic 12.1.2

Just fyi, the following JVM arguments have to be added to collect JFR on hotspot.
-XX:+UnlockCommercialFeatures -XX:+FlightRecorder
And the following JVM arguments have to be added for enabling G1GC
-XX:+UseG1GC

Here is a comparison of snapshots. Notice the number of GC occurrences and the GC times.

GC times with G1GC

GC times with generational GC

Garbage Collection with generational GC

Garbage Collection with G1GC

There were 13 garbage collections (from young generation) Vs 3 garbage collections with G1 GC.
However the total GC times is higher with G1GC . This is an example where perhaps we don’t need G1GC.
G1GC makes sense on JVMs with high number of full GCs (collection in tenured/old generation).
In this blog, the idea was to just showcase the basic differences. A better comparison would be to check GC activity on a busy system (probably taken during a load test)

Getting Fusion Middleware Application MBeans

To get fusion middleware MBeans (eg: BPEL properties, Mediator properties), the WLST shell that needs to be invoked is different from the regular weblogic WLST shell.

For querying the SOA MBeans, the following shell needs to be invoked.

SOA_ORACLE_HOME\common\bin\wlst.cmd (eg: C:\oracle\Middleware\Oracle_SOA1\common\bin\wlst.cmd)

fmwMBean.py
===================
connect('weblogic','password','t3://Adminhost:port')
domainRuntime()
SOAInfraConfigobj = ObjectName('oracle.as.soainfra.config:Location=soa_server1,name=soa-infra,type=SoaInfraConfig,Application=soa-infra')
print 'Common Properties for soa_server1'
print 'AuditLevel at SOAConfig (Global)~', mbs.getAttribute(SOAInfraConfigobj,'AuditLevel')
print 'CallbackServerURL~', mbs.getAttribute(SOAInfraConfigobj,'CallbackServerURL')
print 'GlobalTxMaxRetry~', mbs.getAttribute(SOAInfraConfigobj,'GlobalTxMaxRetry')
print 'GlobalTxRetryInterval~', mbs.getAttribute(SOAInfraConfigobj,'GlobalTxRetryInterval')

#Get BPEL Properties
BPELMBeanobj = ObjectName('oracle.as.soainfra.config:Location=soa_server1,name=bpel,type=BPELConfig,Application=soa-infra')
print 'BPEL Properties for soa_server1'
print 'DispatcherEngineThreads~', mbs.getAttribute(BPELMBeanobj,'DispatcherEngineThreads')
print 'DispatcherInvokeThreads~', mbs.getAttribute(BPELMBeanobj,'DispatcherInvokeThreads')
print 'DispatcherSystemThreads~', mbs.getAttribute(BPELMBeanobj,'DispatcherSystemThreads')
print 'LargeDocumentThreshold~', mbs.getAttribute(BPELMBeanobj,'LargeDocumentThreshold')

#Get Mediator Properties
MediatorMbeanobj = ObjectName('oracle.as.soainfra.config:Location=soa_server1,name=mediator,type=MediatorConfig,Application=soa-infra')
print 'Mediator Properties for soa_server1'
print 'AuditLevel for Mediator~',mbs.getAttribute(MediatorMbeanobj,'AuditLevel')
print 'ContainerIdLeaseTimeout~', mbs.getAttribute(MediatorMbeanobj,'ContainerIdLeaseTimeout')
print 'ResequencerWorkerThreadCount~', mbs.getAttribute(MediatorMbeanobj,'ResequencerWorkerThreadCount')

#Get HumanWorkFlow Properties
Humanworkflowobj = ObjectName('oracle.as.soainfra.config:Location=soa_server1,name=human-workflow,type=HWFMailerConfig,Application=soa-infra')
print 'Human Workflow Properties for soa_server1'
print 'Email from address~', mbs.getAttribute(Humanworkflowobj,'ASNSDriverEmailFromAddress')

#Get Adapter Properties for soa_server1
print 'Adapter Properties for soa_server1'
AdapterMbeanobj = ObjectName('oracle.as.soainfra.config:Location=soa_server1,name=adapter,type=AdapterConfig,Application=soa-infra')
print 'GlobalInboundJcaRetryCount~', mbs.getAttribute(AdapterMbeanobj,'GlobalInboundJcaRetryCount')
==============

The above script can be executed as:
c:\$SOA_ORACLE_HOME\common\bin\wlst.cmd fmwMBean.py > wlst.out

This data can be generated into an excel file (without the need to access em).
The above script generates values with a delimiter ~
The below script generates an excel file
Get text2xls.pl from :
http://www.perlmonks.org/bare/?node_id=72985
If you haven't used perl before, download it from ActivePerl site and to run a perl script, just do $ perl script.pl
To run the current script:

C:\WLST>perl text2xls.pl -i wlst.out -o test1.xls -d ~ (wlst.out is generated from above wlst script and -d option is the delimiter)
Starting text2xls.pl
infile = wlst.out
outfile = test1.xls
Finished text2xls.pl

This would generate an excel file (test1.xls) with MBean values without the need for em.

Both the above two operations (generating wlst.out and converting wlst.out to excel) can be wrapped in a batch/shell script for better automation.

References: http://biemond.blogspot.com/2010/02/invoking-fmw-application-mbeans-in.html
http://www.perlmonks.org/bare/?node_id=72985

WebLogic Plan file merger with webdescriptor

Ever seen this error when a plan file is applied to an application on weblogic10.3?

<[WebAppModule(test:test.war)] Error parsing descriptor in Web appplication "C:\webapp\test.war"
weblogic.descriptor.DescriptorValidateException: The following failures occurred:
-- EnvEntryType is a required value that has not been specified.
at weblogic.descriptor.internal.AbstractDescriptorBeanHelper.validateSubTree(AbstractDescriptorBeanHelper.java:113)

This is because of some wierd whitespace issue that weblogic is not able to parse the webdescriptor file or the plan file.
The same plan and application works on wls11g (Either wls11g is more forgiving or does a better parsing)
You are better off to redo the plan xml or webdescriptor file, if you encounter the above error.

Oracle Enterprise Manager perl Syntax

We needed to enable metric "Log File Pattern Matched Line Count" from OEM.
We were looking for java.lang.OutOfMemory string within our weblogic log file.
As first time user, I went ahead and added the string name and really did not care to check the "Ignore Pattern in Perl" that had "%" by default.



This was telling the EM to ignore virtually everything and no wonder we did not receive alerts.
The right to look for a string "java.lang.OutOfMemoryError" was:

Jrockit JRCMD

If you are one of those shops where remote access from JRMC (Jrockit Management Console) is not possible, jrcmd comes to rescue.
All we need is access through SSH.
This came in real handy for our environments where remote access is not allowed.

$ ./Jrockit_home/bin/jrcmd PID heap_diagnostics
This would print heap summary, Memory layout (similar to pmap in solaris) and details object statistics

If you are new to Jrockit and familiar with other JVM tools like Jhat or MAT,and would need to do an offline analysis, then a heap dump can be generated in HPROF format.

$ ./Jrockit_home/bin/jrcmd PID hprofdump filename=heapdump.hprof
(heapdump.hprof is created from JVM process spawning directory. For weblogic it will be under domain directory - assuming startscript is triggred from there)
This hprof can be further analyzed with tools like jhat or MAT (Eclipse Memory analyzer tool).
MAT would provide all sorts of info from leak suspects to Accumulated Objects by Class as shown above.

Another neat feature from JRMD is baselining the heap and native memory.
$./Jrockit_home/bin/jrcmd PID print_memusage scale=M baseline
This prints base line of memory usage (both java and native heap) something like:



Subsequent runs of same command would baseline to first snapshot and shows the added java & native heaps.



Reference:
https://www.packtpub.com/oracle-jrockit-the-definitive-guide/book

WebLogic & Oracle RAC

It's a common scenario to have weblogic hosted apps connect to Oracle RAC in the backend. From weblogic perspective, there are two ways to connect to Oracle RAC.

1)   Use WebLogic Multipool:   This would mean all the failover/loadbalancing of JDBC calls will be taken care by weblogic JDBC framework - built within application server.
2)   Connect-time failover :   This would mean, all failover/loadbalancing of jdbc calls will be taken care by Oracle driver.

Now what is better approach?   Based on testing and feedback from Oracle, multipool approach is a better choice.

In case of connect-time failover scenario, failover can take lot of time (max would depend on tcp timeout).
Oracle FCF (fast connection failover) would help to some extent in reducing the failover time, but still the failover would be connection by connection instead of whole pool atonce.
Failover in case of multipool is faster compared to FastConnection Failover feature from Oracle DB.
Also, XA connections are better taken care of w.r.t multipools since weblogic framework would make sure all JDBC calls within an XA transaction would reach the same Oracle server.(which is not the case with Oracle thin driver)

JVM memory profiling (jmap & jhat)

The newer versions of Sun JDK (1.5, 1.6) has better memory profiling tools available. One such combination is jmap and jhat.
Both of the tools are now part of JDK1.6 distribution (located under javahome/bin folder)
To get to a probable source of memory leak (or to figure out which objects are consuming most part of the heap) we have these JDK builtin tools for rescue.
Essentially, we can get the heap snapshot with jmap. we can get a heap dump from jmap which can be read by jhat.
Here is how it looks:  (I am trying to profile my weblogic11g process and its processID is 23707)

To get current heap usage stats :
===================================

# ./jmap -heap 23707   (23707   being java process ID )


To get list of all objects on the heap (both live and not reachable)
This is my WebLogic11g process.

# ./jmap -histo 23707   (23707   being java process ID )


To get a heap dump :
(I have specified file name as myheap.bin)
# ./jmap -dump:format=b,file=myheap.bin 23707
Dumping heap to /weblogic11g/jdk160_11/bin/myheap.bin ...
Heap dump file created

To read the heap dump thats generated, jhat can be used as follows :
# ./jhat myheap.bin
Reading from myheap.bin...
Dump file created Mon Aug 24 16:20:26 EDT 2009
Snapshot read, resolving...
Resolving 546388 objects...
.......................
Snapshot resolved.
Started HTTP server on port 7000
Server is ready.

jhat would start a webserver and all object allocations can be viewed and queried. The object allocation can be accessed through browser as http://myIP:7000

Here is a snapshot with default access. we find basic allocation information and also different options to query the objects.
We can get object allocation by size as :


If you prefer a GUI based view, you can check MAT(from eclipse) or visualGC.