WebLogic Performance Tuning

The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remain at the sole discretion of Oracle.

2

|

© 2011 Oracle Corporation – Proprietary and Confidential

Extreme (WLS/Java) Performance Workshop WebLogic Server Performance Tuning Jonathan Leung FMW APAC Performance Specialist

Copyright © 2012, Oracle and/or its affiliates. All rights reserved.

Agenda • • • • • • • •

WebLogic Server Core Tuning Web Application Tuning JDBC Tuning EJB/MDB Tuning JMS Tuning Application Tuning Timeout Tuning Diagnosing Performance Issues


WebLogic Server Core Tuning Network-related Tunables

• WebLogic Server 10.3.2+ • The maximum number of threads is 4.

• WebLogic Server 10.3.1 (and below) • # of muxer threads = # of cpus + 1 • On larger systems, with multicore CPUs, could easily have over 100 muxer threads • Higher CPU utilization due to fat lock contention while synchronizing • Limit the # of Muxer threads: -Dweblogic.SocketReaders=4 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.

WebLogic Server Core Tuning Network Tuning • Always verify that WebLogic is using NativeIO. The following message indicates it is in use • • Be careful when switching from 32bit to 64bit JVM. The 32bit WebLogic Server download does not contain the 64bit native libraries needed for the Socket Muxer.


WebLogic Server Core Tuning Exalogic Optimizations • Turn on ―ExalogicOptimizationsEnabled‖ flag under Domain->Configuration->General • This will enable the following core optimizations • Pure java NIO Socket Muxer • Scattered Reads • Gathered Writes • JSSE as the SSL Provider • Lockless Request Manager • More aggressive self-tuning thread algorithm


WebLogic Server Core Tuning Monitor Channel Connections • Ensure we don’t have too many or uneven tcp connection counts across managed servers


WebLogic Server Core Tuning Threads • Monitor Stuck Threads, Queue Length, Throughput


WebLogic Server Core Tuning STUCK Threads • WLS monitors its threads to detect threads where a single request is taking “too long” to process. • “Too long” is defined by Stuck Thread Max Time (600 s)

• What does WLS do when it detects such a thread? • Marks the thread as STUCK • WLS logs a warning message with a stack trace of the thread at that instant in time • Increments the stuck thread count of the appropriate Work Manager • Takes Failure Action if count >= Stuck Thread Count


WebLogic Server Core Tuning STUCK Threads (cont’d) • WebLogic Server will NOT kill a STUCK thread • A STUCK thread is not necessarily stuck • Just means it is long-running • Just means that at the point in time detected, the logged stack trace is where the thread was currently executing • It may be doing something else immediately before or after!

• When you encounter a STUCK thread • Take multiple thread dumps at regular intervals • Looking at the stack trace of the STUCK thread and its progress over time is key to diagnosing the issue.

• WLDF can be configured to automatically create an incident that will include the series of thread dumps • Oracle SOA Suite domains do this OOTB Copyright © 2012, Oracle and/or its affiliates. All rights reserved.

WebLogic Server Core Tuning Work Manager Tuning • Create Work Managers to avoid thread starvation, only as needed. • Fair Share work manager is typically what you want

• Monitor • Pending Requests • Max Wait Time • Current Wait Time


WebLogic Server Core Tuning Work Manager Tuning (cont’d)


WebLogic Server Core Tuning Work Manager Constraint Tuning • Max Threads Constraint • Limit the number of concurrent threads processing requests for the Work Manager • Use to configure MDB concurrency • Default is 16

• Capacity Constraint • Limits the number of total requests (currently processing and queued) to avoid resource issues. When the capacity is reached • HTTP requests get an HTTP 503 response • RMI clients get a QueueThrottleException response


WebLogic Server Core Tuning Work Manager Constraint Tuning (cont’d) • Min Threads Constraint • Guarantees thread availability to process requests, thus avoiding thread starvation • Use only if absolutely required • Keep value very low (e.g., 3) • Min Threads Constraint work is immediately scheduled if constraint not yet met • Setting too high affects work for other work managers


WebLogic Server Core Tuning Standalone Java Clients • Prefer wlfullclient.jar • Replacement for weblogic.jar • Use WebLogic JarBuilder utility to generate • Clients that use WLST need a full install

• If jar size is critical, use wlthint3client.jar • wlclient.jar uses the Java ORB • Silently converts t3 URLs to use IIOP. • Generally slower than wlfulllclient.jar • Not as scalable for multi-threaded clients

• wlthint3client.jar replaces wlclient.jar • Uses t3 runtime  faster, better scalability! Copyright © 2012, Oracle and/or its affiliates. All rights reserved.

Agenda • • • • • • • •



Web Application Tuning JSP and Servlet Change Detection Tuning • WLS can automatically detect and reload Servlets/JSP that have changed • Servlet Reload Check Seconds (servlet-reload-check-secs) • Entire web application is reloaded when change detected!

• JSP Page Check Seconds (pageCheckSeconds) • JSP page is recompiled and reloaded

• Default Behavior • Development Mode, check every second • Production Mode, never check • Values in weblogic.xml override defaults • Be careful when moving from Development to Production Copyright © 2012, Oracle and/or its affiliates. All rights reserved.

Web Application Tuning JSP Compilation Tuning

• By default, JSP files are compiled on demand • Can cause poor response time for first access

• To avoid this initial overhead, use WLS tools to precompile JSP files prior to deployment. Precompile JSPs from the command line: java weblogic.appc hrWeb.war


Web Application Tuning JSP Compilation Tuning (cont’d)

• Can force WLS to never look for JSP source files by using JspClassServlet • Precompile the JSP files • Map jsp extension to JspClassServlet JSPClassServlet weblogic.servlet.JSPClassServlet JSPClassServlet *.jsp

• No need to package JSP files in WAR Copyright © 2012, Oracle and/or its affiliates. All rights reserved.

Web Application Tuning JSP and Servlet Output Buffer Tuning

• Web Container buffers output while building response • Data flushed to client once buffer is full • Cannot forward request once data sent to client • Set buffer size accordingly • ServletResponse.setBufferSize(int) •

• WLS default buffer size is 12kb

• Avoid calling flush() and close() on ServletOutputStream Copyright © 2012, Oracle and/or its affiliates. All rights reserved.

Web Application Tuning HttpSession Replication Tuning

• WLS replicates only session’s new/changed attributes • Keep session objects as small as possible • Better to store lots of attributes rather than one

• Put only Serializable objects in the session • Failure to do so will prevent replication

• Monitor primary distributions across cluster • Uneven distributions likely means the Load Balancer/Proxy isn’t properly distributing the load


Web Application Tuning HttpSession Replication Tuning (cont’d)


Web Application Tuning HttpSession Replication Tuning (cont’d)

• Make sure requests with existing sessions stick to server where primary is located • Remote session ―fetch‖ and secondary relocation will have significant performance implications • WebLogic Web Server Plug-ins provide sticky routing • No need for HWLB sticky sessions in front of web servers

• When not using Web Servers with WL Plug-in, HWLB should always use Sticky Sessions • Active Cookie Persistence is recommended Copyright © 2012, Oracle and/or its affiliates. All rights reserved.

Web Application Tuning Exalogic Session Replication Optimizations • Turn on ―ExalogicOptimizationsEnabled‖ flag under Domain>Configuration->General • This will automatically enable Lazy Deserialization • Create Replication Channel • Environment->Servers->->Protocols->Channels • Enable the use of this replication channel • Cluster->Configuration->Replication

• Cluster Replication Ports • Environment->Servers->->Configuration>Cluster • Configure replication port range (Eg: 8001-8010)


Web Application Tuning Offload HTTP Sessions to Coherence*Web

• WLS performance can be impacted if too many sessions remain in memory • Consider Coherence*Web for apps with lots of users, large sessions, or long session timeouts • Offloads sessions into a Data Grid • Easy to scale by increasing the number of Coherence nodes in the Grid • Allows sharing sessions across applications


Agenda • • • • • • • •



JDBC Tuning Connection Pool Sizing • Creating DB connections is time consuming • Ideally, set Min = Max to avoid creating connections on demand • If DB connections are limited, set Min to handle normal load and max to handle peak loads, enable pool shrinking

• Most applications need only one connection per DB per concurrent request being processed • Typically, that means one per execute thread. • Consider to prevent contention with limited numbers of DB connections


JDBC Tuning Connection Pool Sizing (cont’d) • Monitor Data Source statistics in the Console • Active Connections High Count • Waiting on Connection High Count • Wait Seconds High Count

• Tune accordingly • Maximum Capacity • Connection Reserve Timeout (default: 10 sec) Copyright © 2012, Oracle and/or its affiliates. All rights reserved.

JDBC Tuning Statement Cache Tuning • WLS caches JDBC Statement objects • Cache is per connection • # of cached statements = # of conns * cache size • Tune according to hit/miss ratio • WLS Console


JDBC Tuning Statement Cache Tuning (etc.) • Databases also cache parsed SQL statements • Oracle DB caches SQL in Shared Pool • SQL cache shared across all connections • Work with your DBA to optimize database parsing of SQL


JDBC Tuning Exalogic Optimizations

• SDP-enabled JDBC thin driver • JDBC URL: Replace “(PROTOCOL = TCP)” with “(PROTOCOL = SDP)” • Requires Oracle DB (Exadata) with • SDP Listener configured (listener.ora) • Use ActiveGridlink data source instead of Multi Datasource


JDBC Tuning Miscellaneous Tuning • Connection Creation Retry Seconds (default = 0) • If database is not available for a Data Source, WLS will fail to boot if set to zero—the default! • Set to a non-zero value to allow the server to boot and periodically retry to create the connection pool

• Seconds to Trust Idle Pool Connection (default = 10 s) • Works in conjunction with Test Connections On Reserve • Can reduce connection testing queries significantly

• Consider LLR for JTA transactions • Only helps with XA transactions involving multiple resources • Last Logging Resource (non-XA) must be a database


JDBC Tuning Miscellaneous Tuning (cont’d) • Remove Infected Connections Enabled (default = on) • Used to clean up connections where app accessed underlying physical connection object • Can improve performance to disable when using physical connections to access driver-specific extensions • Use only when safe!!! • App doesn’t retain reference to the Connection, Statements, or ResultSet objects • App doesn’t change the state of the connection • Effective database user ID • Auto commit state


Agenda • • • • • • • •



EJB Tuning Stateless Session Beans

• The defaults are typically good enough • initial-beans-in-free-pool = 0 • max-beans-in-free-pool = 1000

• There is a cost with creating a new instance in the pool and thus tuning the initial-beans-infree-pool might be beneficial if the runtime cost is too much • This may increase the time to startup but reduces the runtime cost



• Tuning based on Pool Miss Ratio • Pool Miss Ratio = (Pool Total Miss Count / Pool Total Access Count) * 100 • These values are available as MBeans (console, wlst, jmx) • A high pool miss ratio indicates an improperly tuned pool where it doesn’t find an instance in the pool




EJB Tuning MDB Tuning

• Maximum MDB concurrency determined by: • Is MDB using a custom work manager with a maxthreads-constraint? • Yes • max-threads-constraint < max-beans-in-free-pool? • Yes : Max concurrency = max-threads-constraint • No : Max concurrency = max-beans-in-free-pool

• No • Is max-beans-in-free-pool > 16? • Yes: Max concurrency = 16 • No: Max concurrency = max-beans-in-free-pool


EJB Tuning MDB Tuning (cont’d)

• To handle errors in MDBs, you should: • For transactional delivery, use setRollbackOnly() to force the message back onto the source queue • For non-transactional delivery, raise a RuntimeException to prevent message acknowledgement. • EJB spec requires destroying the MDB instance • Use weblogic.ejb.NonDestructiveRuntimeException to prevent WLS from destroying the MDB instance

• Use Foreign JMS Server for remote destinations Copyright © 2012, Oracle and/or its affiliates. All rights reserved.

Agenda • • • • • • • •



JMS Tuning Inter-Application Messaging Patterns Producing Applicatio n

Producing Applicatio n

Producing Applicatio n SAF

Consumin g Applicatio n

Consumin Consumin g g Applicatio Applicatio n n Pull Model Conceptual Model Push Model When you have control over the consuming application, prefer the Pull Model as it is simpler and more efficient Copyright © 2012, Oracle and/or its affiliates. All rights reserved.

JMS Tuning Understanding JMS Transactions • JMS supports two types of transactions • Transacted Sessions – local transaction only involving JMS operations on a single JMS Session • JTA Transactions – global JTA transaction that can include other transactional (i.e., XA) resources

• Transacted Sessions and JTA transactions are mutually exclusive • JTA transaction must use createSession (false, …)

• Messages sent within a transaction are acknowledged on commit Copyright © 2012, Oracle and/or its affiliates. All rights reserved.

JMS Tuning Choosing a Persistent Store Type • WebLogic Server supports two types of persistent stores for messages • File Store • Database Store

• File Store is generally faster • Database Store is generally easier for HA • JMS Store must be highly available for failover • File Store  shared disk (e.g., SAN) • Database Store  database is accessible


JMS Tuning JMS Wrappers • WLS JMS Wrappers provide: • JMS Connection, Session, & Producer pooling • Testing/Re-establishment of JMS connection • Automatic transaction enlistment for XA providers

• To use: • Map Connection Factory & Destination to app using: • entries in deployment descriptors, or • @Resource annotations with dependency injection

• Write normal JMS code to send a message: • Create new connection, session, and producer for each request • Close objects at the end of each request Copyright © 2012, Oracle and/or its affiliates. All rights reserved.

JMS Tuning JMS Checklist • Consumers must be able to keep up with producers over extended period of time • Avoid intra-cluster message routing for the best performance and scalability • Use JMS Wrappers to send messages in server-side apps • Prefer asynchronous consumers • Use MDBs for consumers running in a WLS server • Use Message Listeners for standalone consumers Copyright © 2012, Oracle and/or its affiliates. All rights reserved.

Agenda • • • • • • • •



Application Tuning Application Performance Problems • Most application performance issues are typically contention-related: • Overuse of synchronization • Insufficient resources (e.g., database connections) • Overloaded backend systems

• Load testing can identify most contention • Take thread dumps at regular intervals (5-10 secs)

• Attach a profiler such as JRockit Mission Control or Yourkit


Application Tuning Coding for Performance • Cache JNDI InitialContext, Data Source, JMS Connection Factory and Destination objects • Use the Service Locator pattern to cache lookups, or • Use Dependency Injection

• Use Coherence to cache frequently used data • Can be used explicitly in your application • Can be used as a JPA Level 2 cache


Agenda • • • • • • • •



Timeout Tuning Setting Network Timeouts • Set appropriate CONNECT and READ timeouts to prevent threads hanging indefinitely • RMI - No default timeouts when using JNDI • Set connect and read timeouts using JNDI InitialContext properties • weblogic.jndi.WLContext.REQUEST_TIMEOUT – This is the connect timeout • weblogic.jndi.WLContext.RMI_TIMEOUT – This is the read timeout • Future versions will use: • weblogic.jndi.WLContext.CONNECT_TIMEOUT • weblogic.jndi.WLContext.RESPONSE_READ_TIMEOUT Copyright © 2012, Oracle and/or its affiliates. All rights reserved.

Timeout Tuning Setting Network Timeouts (cont’d) • HTTP – set timeouts for outbound calls • Set programmatically • URLConnection.setConnectTimeout(int milliseconds) • URLConnection.setReadTimeout(int milliseconds)

• If using default weblogic.net.http implementation, default values can be set using system properties • -Dweblogic.http.client.defaultConnectTimeout=3000 • -Dweblogic.http.client.defaultReadTimeout=30000


Timeout Tuning Setting Network Timeouts (cont’d) • JDBC – set timeouts to prevent database issues from hanging threads in the server • Set Oracle Thin Driver properties • oracle.net.CONNECT_TIMEOUT=3000 • oracle.jdbc.ReadTimeout=30000

• WebLogic Console • Set Invocation Timeout Seconds in order to avoid scenarios where a managed server might be physically disconnected from the network and yet the Admin Server tries to contact it causing threads to hang indefinitely • DomainConfigurationGeneralAdvanced Copyright © 2012, Oracle and/or its affiliates. All rights reserved.

Timeout Tuning RJVM Heartbeat Tuning • RJVM – logical, persistent connection between two WebLogic endpoints • Connection health is determined via heartbeats • By default, heartbeats sent every 60 seconds (Period Length) • By default, connection is declared unhealthy and closed after 4 missed heartbeats (Idle Periods Until Timeout) • By default, can take 240 seconds (60*4) to detect an unhealthy connection and potentially fail over

• Way too long, should be tuned down • Idle Periods Until Timeout minimum is 4 • Tune down Period Length instead • ServerConfigurationTuningAdvanced


Agenda • • • • • • • •


Copyright © 2012, Oracle and/or its affiliates. All rights reserved. `

Diagnosing Performance Issues Thread Dumps • Take multiple thread dumps at regular intervals • Lots of ways to get thread dumps • • • • • •

kill -3 (best) jstack (Hotspot) jrcmd print_threads (JRockit) Ctrl-Break on Windows (only foreground process) Visual VM WLST


Diagnosing Performance Issues Thread Dumps (cont’d) • Use ThreadLogic to analyze thread dumps (shameless plug) • It can analyze and provide recommendations automatically based on common patterns • It can do a ―diff‖ of multiple thread dumps over time and provide recommendations • http://java.net/projects/threadlogic


Diagnosing Performance Issues Thread Dumps (cont’d)


Diagnosing Performance Issues Thread Dumps (cont’d)


Diagnosing Performance Issues Heap Dumps • A common cause of performance problems is frequent, long GC pauses • Likely due to heavy memory usage or memory leak

• Heap dump is a memory snapshot at a point in time • Several ways to get an HPROF heap dump • jmap –dump:format=b,file=heapdump.hprof (Hotspot) • jrcmd hprofdump filename=heapdump.hprof (JRockit) • jextract corefile [zipfile] (IBM) • Add the following to the JVM options first : • -Xdump:system:defaults:request=exclusive+prepwalk • -XX:+HeapDumpOnOutOfMemoryError


Diagnosing Performance Issues Heap Dumps (cont’d) • Various tools to analyze HPROF heap dumps • • • •

Eclipse Memory Analyzer Tool (MAT) Visual VM Java Heap Analysis Tool (JHAT) YourKit

• These tools can show • Which objects make up the heap • Compare and ―diff‖ multiple heap dumps to detect memory leaks

• MAT is probably the most popular and easiest to use


Diagnosing Performance Issues Heap Dumps (cont’d)


Diagnosing Performance Issues Heap Dumps (cont’d)


Diagnosing Performance Issues Network Performance Troubleshooting • Best way to troubleshoot a potential network issues is with a packet capture • To capture the data • tcpdump (Unix) • Make sure to output to a file • tcpdump –i bond0 -w file.pcap

• Wireshark (Windows)

• Use Wireshark to visually analyze the packet capture


Diagnosing Performance Issues Network Performance Troubleshooting (cont’d) • tcptrace • neat utility that can analyze a packet capture and provide statistics such as packet loss and retransmissions in a nicely laid out format • Prebuilt Windows binary exists as well (uses cygwin)

• Fiddler • Very useful Web Debugging HTTP proxy that intercepts browser request/responses


Diagnosing Performance Issues Network Performance Troubleshooting (cont’d)






Diagnosing Performance Issues Other Useful Profiling Tools • JRockit Mission Control • OOTB Flight Recorder has very minimal performance overhead and can be used in production • Analyze GC behavior, potential latency issues

• Visual VM • YourKit • AD4J


Diagnosing Performance Issues Other Useful Profiling Tools (cont’d) • OS Tools • truss/strace – trace system calls. • Very useful to see whether performance issues are at the lowest level

• top, ps, prstat, vmstat, mpstat, iostat, netstat, etc. – Monitor performance at a system level. • Useful for monitoring CPU, disk I/O, NFS statistics, network I/O, virtual memory statistics, processor statistics, and process statistics (such as how much CPU is consumed per thread)


Questions?


WebLogic Performance Tuning

WebLogic Performance Tuning

Suggest Documents

Performance and Tuning for Oracle WebLogic Server

Oracle WebLogic Thread Pool Tuning

Oracle Weblogic 11g Admin, Configuration and Performance tuning

BEA WebLogic Event Server Performance - Complex Event ...

ETL Performance Tuning Tips

MySQL Performance Tuning - Percona

Performance Tuning - 3cyl.com

ABAP Performance Tuning

Java Performance Tuning

JDBC Performance Tuning

Hibernate Performance Tuning - databene

Two Stroke Performance Tuning

Automated Performance Tuning

Memcached Performance Tuning

Two Stroke Performance Tuning

OpenGL Performance Tuning - AMD

Beginning Performance Tuning - Proligence

Collaborative Agent Tuning: Performance

Oracle Database Performance Tuning Guide

Oracle SQL Performance Tuning - Earthlink

Oracle Database 11g: Performance Tuning

Tomcat Performance Tuning and Troubleshooting

Performance Tuning of Scientific Applications

Java Performance Tuning and Optimization