Performance Monitoring for the Java Virtual Machine

Performance Monitoring for the ™ Java Virtual ™ Machine (JVM ) Brian Doherty, Ben Kim, David Stoutamire J ava P erformance E ngineering S un Micros ys tems , Inc. | J avaO ne 2003 | S es s ion 2690

Presentation Goal

L earn about non- intrus ive performance monitoring and the tools available for monitoring production s ys tems utilizing the J ava ™ s oftware s tack.

2

| J avaO ne 2003 | S es s ion 2690

Speaker’s Qualifications and D avid are all members of • B rian, B en, ™

the J ava P latform P erformance E ngineering group at S un Micros ys tems • W e repres ent expertis e in performance meas urement, analys is , and tuning of indus try s tandard benchmarks , cus tomer applications , the J ava ™ clas s libraries , and the J ava HotS pot™ virtual machine

3


Performance Measurement— A Product Lifecycle Process

D on’t s top monitoring your application jus t becaus e it’s been deployed! O bs erve your production applications us ing minimally invas ive tools that are becoming better integrated acros s the s oftware s tack.

4


Agenda • W hat is performance monitoring • Monitoring J ava ™ applications ─ J ava ™ s oftware s tack ─ D is tributed applications

• D emons tration ─ T ools ─ A nalys is ─ T uning

5


Monitoring Goals • Maintain and manage s ys tem health

– D etection of performance events – D etection of application progres s or livenes s

• P erformance tuning

– D rilling down to find caus es – Identifying s olutions

6


Performance Management Process Start Analysis Design Code Benchmark

Performance OK

Monitor 7


Yes

Deploy

No

Profile

Profiling Vs. Monitoring • P roduction vs . D evelopment

– P roduction s ys tems have more res trictive meas urement needs – P roduction loads vary over time – P roduction loads difficult to s imulate

8


Profiling Vs. Monitoring

Intrus ion D etail A ccuracy R equired K nowledge

9


Profiling High High High High

Monitoring Low Low Low Low

Complexities ™

T he J ava S oftware S tack Application Application Server Virtual Machine Operating System Hardware

10


Complexities T he D eployment A rchitecture

C lient T ier

11


W eb T ier

J 2E E ™ A pplication S erver T ier

D atabas e T ier

Monitoring Tools • Network management s ys tems • O S monitoring tools • J V M monitoring tools • J 2E E ™ monitoring tools

12


Demo

What We Saw • S ymptoms ─ E xces s ive memory utilization ─ R es tart

• T ools s een here ─ R untime.freeMemory() ─ gcore ─ S erviceability A gent • Identified s ource of memory cons umption

14


Other Tools • P roduction ─ - verbos e:gc

• D evelopment ─ - verbos e:gc ─ J V MP I/J V MD I profiling and debugging tools

15


Demo

What We Saw • S ymptoms ─ Incons is tent res pons e times

• T ools s een here ─ V is ualG C • Identified explicit S ys tem.gc() calls

17


Other Tools • P roduction ─ - verbos e:gc ─ mps tat—identify s ingle threaded G C ─ ps tack—print O S thread dump ─ S IG INT —print J ava thread dump

• D evelopment ─ - X prof ─ J V MP I/J V MD I profiling and debugging tools ─ S erviceability A gent 18


Demo

What We Saw • S ymptoms ─ P oor s ys tem utilization ─ L ow throughput

• T ools us ed ─ Y A X (perfmeter) • C P U only ~ 50% utilized

─ S IG INT (thread dump) • T hreads waiting on locks • Identified s ource of lock contention 20


Other Tools • P roduction ─ B ytecode ins trumentation ─ prs tat / top ─ ps tack

• D evelopment ─ - X prof ─ J V MP I/J V MD I profiling and debugging tools

21


What We Covered ™

Monitoring the J ava S oftware S tack Application Application Server Virtual Machine Operating System Hardware

22


What We Covered ™


• UNIX s tat tools – vms tat, ios tat, mps tat, nets tat ks tat

• UNIX proc tools – ps tack, pmap, prs tat/top, trus s /s trace

• Hardware counters – cpus tat, bus s tat

• Mis c. tools – S E T oolkit 23


What We Covered ™


24


• jvms tat tools • C ommand line options – - verbos e:[gc|clas s |jni] – - X prof – - X X options

• S erviceability A gent • J S R 174 interfaces

What We Covered ™

Monitoring the J ava S oftware S tack Application Application Server

• J MX interfaces – S NMP , C IM/W B E M

• B ytecode ins trumentation – V arious commercial tools

• J V MP I, J V MD I tools Virtual Machine Operating System Hardware

25


– C ommercial and free

Summary • P erformance meas urement needs to be a product lifecycle proces s

─ Not jus t a development proces s

• Non- intrus ive tools are critical for

meas urement of production applications • T ools integrated acros s the s oftware s tack eas e the meas urement and analys is proces s

26


Resources • HotS pot performance documentation http://java.s un.com/docs /performance

• jvms tat http://www.s un.com/developers /cools tuff

• S olaris ™ performance documentation http://docs .s un.com

• J S R 174: Monitoring and Management ™ S pecification for the J ava virtual machine

27


Q&A