Using Compositional Verification to Manage Change in Large-Scale ...

Using Compositional Verification to Manage Change in Large-Scale Complex IT Systems (Work in progress) Radu Calinescu1 , Shinji Kikuchi2 and Kenneth Johnson1 1 Computer

Science Research Group, Aston University, UK 2 Fujitsu

R. Calinescu, S. Kikuchi & K. Johnson ()

Laboratories Limited, Japan

Compositional Verification for LSCITS

M ONTEREY 2012

1 / 25

Motivation Formal modelling & verification is successfully used to establish correctness and QoS properties of an increasing number of systems

static, infrequent changes highly dynamic, continual changes


small to large systems

very large systems

symbolic model checking runtime verification & adaptation

compositional verification


(LSCITS) ?

M ONTEREY 2012

2 / 25

Motivation Formal modelling & verification is successfully used to establish correctness and QoS properties of an increasing number of systems small to large systems static, infrequent changes highly dynamic, continual changes

very large systems

symbolic compositional model checking verification ↓ runtime verification → (LSCITS) & adaptation ?

Can composition & runtime verification techniques be integrated to handle LSCITS? R. Calinescu, S. Kikuchi & K. Johnson ()


M ONTEREY 2012

2 / 25

Research questions

Can composition & runtime verification techniques be integrated to handle large scale and continual change: 1

How large are the systems that can be verified using compositional verification?

2

How to extend compositional verification “proof rules” for LSCITS?

3

How to automate the calculation of the minimal sequence of verification steps to be re-done after different types of system change?

4

What application domain(s) can provide suitable case studies?



M ONTEREY 2012

3 / 25

A first application domain: cloud computing

(potentially) very large, continually changing, quantifiable QoS R. Calinescu, S. Kikuchi & K. Johnson ()


M ONTEREY 2012

4 / 25

We started much smaller. . . Service (logical) configuration Service

Applica5on

Web

Database

Web 1

Web 2

App 1

App 2

Web 3

Web 4

App 3

App 4

DB 1

DB 2

VMA,1

VMA,2

VMA,3

VMA,4

VMB,1

VMB,2

VMB,3

VMB,4

VMC,1

VMD,1

Server C

Server D

Server A

Server B

Cloud infrastructure configuration

(but know we can scale up the case study easily) R. Calinescu, S. Kikuchi & K. Johnson ()


M ONTEREY 2012

5 / 25

We are interested in (probabilistic) safety properties Service (logical) configuration Service

Applica5on

Web

Database

Web 1

Web 2

App 1

App 2

Web 3

Web 4

App 3

App 4

DB 1

DB 2

VMA,1

VMA,2

VMA,3

VMA,4

VMB,1

VMB,2

VMB,3

VMB,4

VMC,1

VMD,1

Server C

Server D

Server A

Server B


The probability that the service does not fail within X days is at least Y R. Calinescu, S. Kikuchi & K. Johnson ()


M ONTEREY 2012

6 / 25

“Monolithic” verification

We establish correctness/QoS properties of a large system by verifying that the parallel composition of n interacting models of its components and environment satisfy a set of requirements:

M1 k M2 k . . . k Mn |= R



M ONTEREY 2012

7 / 25

Monolithic verification {disks = NDISK , cpus = NCP U , mem = NM EM }

{disks = NDISK −1, cpus = NCP U , mem = NM EM }

{disks > 0, cpus = NCP U , mem = NM EM }

{disks > 0, cpus > 0, mem = NM EM }

{disks = 2, {disks = NDISK −2, {disks = 2, {disks = 1, cpus = NCP U , cpus = NCP U , cpus = NCP U , cpus = NCP U , mem = NM EM } mem = NM EM } mem = NM EM mem = NM EM } detect}

{disks = 0∨ cpus = 0∨ mem = 0}

{disks > 0, cpus = 1, {disks > 0, mem = NM EM cpus = 1, detect} mem = NM EM }

{disks > 0, cpus > 0, mem = 2, detect}

{disks > 0, cpus > 0, mem = 2}


{disks > 0, cpus > 0, mem > 0}

models are finite-state transition graphs—nodes correspond to relevant states of the real system component and are labelled with basic properties that hold in these states R. Calinescu, S. Kikuchi & K. Johnson ()


M ONTEREY 2012

8 / 25

Monolithic verification

{disks = 2, −2, {disks = 2, {disks = 1, cpus = NCP U , cpus = NCP U , cpus = NCP U , mem = NM EM } mem = NM EM mem = NM EM } detect}

5

p

1

op

0.9 detect 0.1

1.0 warn 0.995

0.005 disk op


1.0 server down


0.95 detect 0.05

1.0 warn

0.001 cpu op

0.999

{disks > 0, {disks > 0, cpus > 0, cpus > 0, 0.002 {disks > 0, cpus > 0, transitions between statesmem may=be with probabilities, 2} annotated mem = 1} mem = 2, detect}

actions,0.9 ... 0.1

detect

1.0 warn


0.998

0.002 mem op 0.998


mem op M ONTEREY 2012

9 / 25

Monolithic verification {web = 2, app = 2}

{web = 0, app = 0}

server down

wa rn serv e

{web = 2, app = 2}

1.0 app down

0.15 vm migrate 0.85

r up

1.0 web down

{web = 2, app = 2}

{web > 0, app > 0}

{web > 0, app = 2}

0.95

vm op

0.95 0.05

1.0

vm op 0.95

0.05

1.0 web up

app up {web > 0,

vm op app = 0} {web > 0, app = 1} 0.05

1.0 web up 1.0

{web = 1, app = 2} {web = 1, app = 2}

0.95

{web = 1, app > 0}

0.95

app down 1.0 web up

1.0 Interacting models have shared actions, which vm op vm opannotate transitions 0.05 app up that must be taken at the same time. . . 0.95 {web = 1, 0.05


vm op


app = 0}

1.0

M ONTEREY 2012

10 / 25

Monolithic verification {web = 2, app = 2}

{web = 0, app = 0}

1.0 web down

server down

wa rn

{disks = NDISK , cpus = NCP U , mem = NM EM }


{disks = 2, {disks = NDISK −2, {disks = 2, {disks = 1, cpus = NCP U , cpus = NCP U , cpus = NCP U , cpus = NCP U , mem = NM EM } mem = NM EM } mem = NM EM mem = NM EM } detect}

0.005

0.995 {disks > 0, cpus = NCP U , mem = NM EM }

0.999 {disks > 0, cpus > 0, mem = NM EM }

0.998

0.005 0.995 0.995 disk op disk op

0.001

0.001 0.999 0.999 cpu op cpu op

0.005 disk op

0.001 cpu op

0.9 detect 0.1

1.0 warn 0.995

0.002 0.998 0.998 mem op mem op

0.002 mem op

1.0

{web = 2, app = 2}

0.95 detect 0.05

1.0 warn

0.9 detect 0.1

0.95

vm op


0.998

1.0 web up 1.0

vm op 0.95

app up

{web > 0, app = 1} 0.05

1.0 web up 1.0

0.001 cpu op {disks > 0, cpus > 0, 0.002 mem = 1}

0.002 mem op 0.998

0.95 0.05

{web > 0, vm op app = 0}

server down

0.999

1.0 warn

{web > 0, app > 0}

{web > 0, app = 2}

0.05

1.0

k

{web = 1, app = 2} {web = 1, app = 2}

0.95 vm op

mem op

{web = 1, app > 0}

0.95 0.05

0.05

app up {web = 1, app = 0}

vm op

{web = 0, app = 2}

0.05 {web = 0, app = 1}

1.0 web up 1.0

{web = 0, app > 0}

server up

app down 1.0 web up 1.0

vm op 0.95

{web = 1, app = 1} 0.05

{disks > 0, cpus > 0, mem > 0}

1.0

app down

0.15 vm migrate 0.85


{disks > 0, cpus > 0, mem = 2, detect}

0.002

disk op

serv er u p


0.005

{web = 2, app = 2}

0.95

app down 1.0 web down 1.0

vm op

app up

0.95 vm op 0.05

{web = 0, app = 0}

1.0 web down 1.0 app down

MserverA

k

Mweb+appA

. . . but all other transitions can be taken in any order! R. Calinescu, S. Kikuchi & K. Johnson ()


M ONTEREY 2012

11 / 25


s0


no disk

1

0.001

s1

disk1 op 0.005

0.995

disk2 op


It is possible to model certain types of nondeterminism. . .

s2



M ONTEREY 2012

12 / 25


s0


no disk

1

0.001

s1

disk1 op 0.005

0.995

disk2 op


s2


It is possible to model certain types of nondeterminism. . . . . . and to establish the minimum probability that the service does not fail over all possible choices of actions


M ONTEREY 2012

12 / 25

Monolithic verification Service (logical) configuration Service

Applica5on

Web

Database

Web 1

Web 2

App 1

App 2

Web 3

Web 4

App 3

App 4

DB 1

DB 2

VMA,1

VMA,2

VMA,3

VMA,4

VMB,1

VMB,2

VMB,3

VMB,4

VMC,1

VMD,1

Server C

Server D

Server A

Server B


MserverA k MserverB k MserverC k MserverD k Mweb+appA k Mweb+appB k MdbC k MdbD k Mservice has 176, 381, 406, 182, 650 states and 1, 444, 659, 335, 770, 680 transitions



M ONTEREY 2012

13 / 25

Monolithic verification Service (logical) configuration Service

Applica5on

Web

Database

Web 1

Web 2

App 1

App 2

Web 3

Web 4

App 3

App 4

DB 1

DB 2

VMA,1

VMA,2

VMA,3

VMA,4

VMB,1

VMB,2

VMB,3

VMB,4

VMC,1

VMD,1

Server C

Server D

Server A

Server B


MserverA k MserverB k MserverC k MserverD k Mweb+appA k Mweb+appB k MdbC k MdbD k Mservice has 176, 381, 406, 182, 650 states and 1, 444, 659, 335, 770, 680 transitions Out-of-memory when verifying the safety property “the probability that the service does not fail over a 24-hour time period is at least 0.995” (using PRISM on Macbook Pro laptop with 8GB memory) R. Calinescu, S. Kikuchi & K. Johnson ()


M ONTEREY 2012

13 / 25

Compositional verification: some key advances Hoare logic

Kwiatkowska et al ’s probabilistic assume-guarantee

Notation: {P }Q{R}

Notation: hAi≥pA M1 hGi≥pG

Proof rule:

{P }Q{R}, {R}S{T } {P }Q; S{T }

Proof rule:

htrueiM1 hAi≥pA , hAi≥pA M2 hGi≥pG htrueiM1 k M2 hGi≥pA

1985 1969

2010

Pnueli’s modular temporal reasoning Notation: hAiM1 hGi Proof rule:


htrueiM1 hAi, hAiM2 hGi htrueiM1 k M2 hGi


M ONTEREY 2012

14 / 25

Compositional verification

web down

app down warn

server down

warn

app down

web down

web down

warn, server down

warn, server down

+ (a) Aerr 1 : server down

app down

web down, app down

warn + (b) Aerr 2 :warn

+ (c) Aerr 3 : (app down web down | web down+ app down) (web down | app down)∗

Probabilistic safetywebproperties are specified as regular expressions that web up down app down app up define action sequences which violate the properties app up web down

web down app up

web down, app up


app down

web up

web up

app down

web down, app down


M ONTEREY 2012

15 / 25

Compositional verification

app down warn

server down

warn

web down

app down

web down

web down

warn, server down

warn, server down

+ (a) Aerr 1 : server down

web down, app down

warn + (b) Aerr 2 :warn

app down

+ (c) Aerr 3 : (app down web down | web down+ app down) (web down | app down)∗

Probabilistic safetywebproperties are specified as regular expressions that web up down app down app up define action sequences which violate the properties app up

web down

app down

web up

Associated finite automata ofapp the models they web(instead up web down deterministic down app up characterise) are composed with the models of other components web down, app up


web down, app down


M ONTEREY 2012

15 / 25

Compositional verification htrueiMserverA hA1 , A2 i≥pA1 ,pA2 hA1 , A2 i≥pA1 ,pA2 Mweb+appA hA3A , A4A , A5A i≥pA3 ,pA4 ,pA5 htrueiMserverA k Mweb+appA hA3A , A4A , A5A i≥pA3 ,pA4 ,pA5 htrueiMserverB hA1 , A2 i≥pA1 ,pA2 hA1 , A2 i≥pA1 ,pA2 Mweb+appB hA3B , A4B , A5B i≥pA3 ,pA4 ,pA5 htrueiMserverB k Mweb+appB hA3B , A4B , A5B i≥pA3 ,pA4 ,pA5 htrueiMserverC hA1 , A2 i≥pA1 ,pA2 hA1 , A2 i≥pA1 ,pA2 MdbC hA6C i≥pA6 htrueiMserverC k MdbC hA6C i≥pA6

,

,

,

htrueiMserverD hA1 , A2 i≥pA1 ,pA2 hA1 , A2 i≥pA1 ,pA2 MdbD hA6D i≥pA6 htrueiMserverD k MdbD hA6D i≥pA6

,

hA3A,A4A,A5A,A3B,A4B,A5B,A6C,A6Di≥pA3,pA4,pA5,pA3,pA4,pA5,pA6,pA6 Mservice hGi≥pG htrueiMhGi≥pG



M ONTEREY 2012

16 / 25

Compositional verification Verified model MserverA−D Mweb+appA−B

MdbC−D Mservice

No states 570 54

13 1035

Result pA1 = 1 − 3.84E-14 pA2 = 1 − 3.45E-9 pA3 = 1 − 6.25E-6 pA4 = 0.9975 pA5 = 0.9975 pA6 = 0.95 pG = 0.997488

Approach expected to scale to much larger systems. . .



M ONTEREY 2012

17 / 25

Compositional verification Verified model MserverA−D Mweb+appA−B

MdbC−D Mservice

No states 570 54

13 1035

Result pA1 = 1 − 3.84E-14 pA2 = 1 − 3.45E-9 pA3 = 1 − 6.25E-6 pA4 = 0.9975 pA5 = 0.9975 pA6 = 0.95 pG = 0.997488

Approach expected to scale to much larger systems. . . . . . but should avoid re-executing all verification steps (at runtime) each time there is a change in the system



M ONTEREY 2012

17 / 25

Managing change with minimal re-verification Consider all models M, properties P and verification steps V = 2P × M × 2P A compositional verification task is a sequence of verification steps cv = hv1 , v2 , . . . , vn i ∈ seqV where vi = (Ai , Mi , Gi ), 1 ≤ i ≤ n A1 = {htruei} Ai ⊆ ∪i−1 j=1 Gj , i > 1



M ONTEREY 2012

18 / 25

Managing change with minimal re-verification Consider all models M, properties P and verification steps V = 2P × M × 2P A compositional verification task is a sequence of verification steps cv = hv1 , v2 , . . . , vn i ∈ seqV where vi = (Ai , Mi , Gi ), 1 ≤ i ≤ n A1 = {htruei} Ai ⊆ ∪i−1 j=1 Gj , i > 1 We want to compute ∆cv after different types of system change



M ONTEREY 2012

18 / 25

Managing change with minimal re-verification Service (logical) configuration Service

Applica5on

Web

Database

Web 1

Web 2

App 1

App 2

Web 3

Web 4

App 3

App 4

DB 1

DB 2

VMA,1

VMA,2

VMA,3

VMA,4

VMB,1

VMB,2

VMB,3

VMB,4

VMC,1

VMD,1

Server C

Server D

Server A

Server B


E.g., cv = hv1 , v2 , . . . , v9 i, where: v1 = ({htruei}, MserverA , {hA1 i≥pA1 , hA2 i≥pA2 }) ... v5 = ({hA1 i≥pA1 , hA2 i≥pA2 }, Mweb+appA , {hA3A i≥pA3 , hA4A i≥pA4 , hA5A i≥pA5 }) ... v9 = ({hA3A i≥pA3 , hA4A i≥pA4 , hA5A i≥pA5 , hA3B i≥pA3 , hA4B i≥pA4 , hA5B i≥pA5 , hA6C i≥pA6 , hA6D i≥pA6 }, Mservice , {hGi≥pG }) R. Calinescu, S. Kikuchi & K. Johnson ()


M ONTEREY 2012

19 / 25

Managing change: failure/departure of Mi ∆cv = reverify (i + 1, h i, {g ∈ Gi • (g, htruei)})

reverify (j, steps, changes) =   if j = n + 1  steps, reverify (j + 1, steps, changes), if Aj ∩ changed = ∅ =   reverify (j + 1, steps a hv 0 i, changes ∪ c ), otherwise j j where changed = {a ∈ P | ∃ a0 ∈ P • (a, a0 ) ∈ changes};

cj = {(g, g 0 ) ∈ Gj ×Gj0 | regex(g 0 ) = regex(g) ∧ prob(g 0 ) < prob(g)}



M ONTEREY 2012

20 / 25

Managing change: failure/departure of MdbD Service (logical) configuration Service

Applica5on

Web

Database

Web 1

Web 2

App 1

App 2

Web 3

Web 4

App 3

App 4

DB 1

VMA,1

VMA,2

VMA,3

VMA,4

VMB,1

VMB,2

VMB,3

VMB,4

VMC,1

Server A

Server B

Server C

X

DB 2

VMD,1

Server D


∆cv = hv90 i v90 = ({hA3A i≥pA3 , hA4A i≥pA4 , hA5A i≥pA5 , hA3B i≥pA3 , hA4B i≥pA4 , hA5B i≥pA5 , hA6C i≥pA6 , htruei}, Mservice , {hGi≥pG0 }) Result: pG0 = 0.94998805 R. Calinescu, S. Kikuchi & K. Johnson ()


M ONTEREY 2012

21 / 25

Managing change: component change Mi → Mi0

∆cv

= verify change(i, Mi0 ) = = reverify (i + 1, h(Ai , Mi0 , Gi0 )i, {(g, g 0 ) ∈ Gi × Gi0 | regex(g 0 ) = regex(g) ∧ prob(g 0 ) < prob(g)}),



M ONTEREY 2012

22 / 25

0 Managing change: MserverA → Mserver A Service (logical) configuration Service

Applica5on

Web

Database

Web 1

Web 2

App 1

App 2

Web 3

Web 4

App 3

App 4

DB 1

DB 2

VMA,1

VMA,2

VMA,3

VMA,4

VMB,1

VMB,2

VMB,3

VMB,4

VMC,1

VMD,1

Server C

Server D

Server A

Server B


NDISK = 3, pdisk

fail

0 0 = 0.0000635294 −→ NDISK = 4, pdisk

fail

= 0.0000823529

0 ∆cv = verify change(1, Mserver ) = reverify (2, hv10 i, ∆G1 ), A 0 0 , hA2 i≥p 0 }) where v10 = ({htruei}, Mserver , {hA1 i≥pA A A 1

2

Result: ∆G1 = ∅ so ∆cv = hv10 i R. Calinescu, S. Kikuchi & K. Johnson ()


M ONTEREY 2012

23 / 25

Managing change: further change patterns 1

new component Mnew ∆cv = verify join(i1 , h(Anew , Mnew , Gnew )i, ∅), where Mi1 , Mi2 , . . . , Mim are the models that depend on Mnew (see paper for definition of verify join))

2

component choice: Mi −→ one of Mi1 , Mi2 , . . . Mim ∆cv = verify change(i, Mi1 ) a verify change(i, Mi2 ) a . . . . . . a verify change(i, Mim ),



M ONTEREY 2012

24 / 25

Summary Early indication that assume-guarantee compositional verification scales well First set of “(LSCITS) change patterns” mapped to sequences of verification steps that need to be re-executed Ongoing work: extension to other change patterns (e.g., requirements change) additional case studies integration with change detection (e.g., based on online model learning)



M ONTEREY 2012

25 / 25