Assessment of formal nuclear I&C logic models using behavior

Download PDF
0 downloads 0 Views 680KB Size Report
Comment
include Matlab/Simulink3, Ptolemy II4, Modelica5 and IEC 61499 ...... [2] International Standard IEC 61499-1: Function Blocks – Part 1: Architecture,. Second ...
Assessment of formal nuclear I&C logic models using behavior examples Igor Buzhinsky

School of Electrical Engineering

Research report, SAUNA project (SAFIR2018 program) Espoo 07.11.2018

aalto university school of electrical engineering

abstract

Author: Igor Buzhinsky Title: Assessment of formal nuclear I&C logic models using behavior examples Date: 07.11.2018

Language: English

Number of pages: 3+26

Department of Electrical Engineering and Automation This report is written within the scope of the SAUNA (“Integrated safety assessment and justification of nuclear power plant automation”) project, a part of SAFIR2018 program. One of the goals of the project is to improve the reliability of safety assurance of nuclear instrumentation and control (I&C) systems by means of model checking, a particular formal verification technique. Previously, to this end, a graphical tool MODCHK was developed in VTT Technical Research Centre of Finland Ltd. This tool allows the user to visually model I&C logic diagrams, verify these diagrams with model checker NuSMV and observe verification results. The goal of this report is to examine the issues of modeling nuclear I&C systems in MODCHK by comparing formal models against simulation models which have been used as the reference for formal modeling. In a more general context, the report proposes a methodology of correctness assessment of modular formal models which were developed based on simulation models. Essentially, the proposed methodology is based on comparing the behavior of the formal model prepared in MODCHK with the one observed in behavior traces collected from the runs of the simulation model. The performed case study is based on the generic PWR simulation model, which has been provided to Aalto University by Fortum Power and Heat Oy. The results reveal a number of modeling issues, especially ones involving processing of continuous data. Upon discovery, all these issues can be resolved by adjusting the MODCHK model being assessed, which shows that MODCHK is applicable for modeling I&C networks with real-valued signals.

Keywords: formal verification, model checking, simulation, nuclear power plants, nuclear I&C systems

iii

Contents Abstract

ii

Contents

iii

1 Introduction

1

2 Background 2.1 Model checking . . . . . . . . . . . . . 2.2 Model checking of nuclear I&C systems 2.3 Apros process simulator . . . . . . . . 2.4 Related work . . . . . . . . . . . . . .

. . . .

2 2 2 4 5

3 Methodology 3.1 Stage 1: model conversion . 3.2 Stage 2: trace recording . . 3.3 Stage 3: trace discretization 3.4 Stage 4: converting traces to 3.5 Stage 5: trace comparison . 3.6 Stage 6: issue detection . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

6 7 9 9 10 11 12

4 Case study 4.1 Generic PWR model . . . . . . . . . . . . . . . . . . . . 4.2 Applying the methodology . . . . . . . . . . . . . . . . . 4.3 Discovered issue classes . . . . . . . . . . . . . . . . . . . 4.3.1 Issues related to basic block implementation . . . 4.3.2 Issues related to basic block use . . . . . . . . . . 4.3.3 Issues related to precision . . . . . . . . . . . . . 4.3.4 Issues related to the simulation model . . . . . . 4.4 Concrete discovered issues . . . . . . . . . . . . . . . . . 4.4.1 S1 (activation of emergency pumps) . . . . . . . . 4.4.2 S2 (closing emergency feed water lines) . . . . . . 4.4.3 S3 (preventing low pressure in steam generators) . 4.4.4 S4 (opening emergency feed water lines) . . . . . 4.4.5 S5 (reactor and turbine trip) . . . . . . . . . . . . 4.4.6 S6 (pressurizer liquid level control) . . . . . . . . 4.4.7 S7 (pressurizer pressure control) . . . . . . . . . . 4.4.8 S8 (feed water tank level control) . . . . . . . . . 4.5 Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . .

14 14 14 14 15 15 15 15 16 16 17 18 18 18 19 19 20 20

. . . . . . . . . . . . formal . . . . . . . .

. . . . . . . . . . . . . . . models . . . . . . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

5 Conclusions

22

References

23

1

1

Introduction

In the Finnish nuclear industry, the means on ensuring reliability of instrumentation and control (I&C) systems of nuclear power plants (NPPs) include model checking [13, 4], a formal verification technique. The established practice which is applied by VTT Technical Research Centre of Finland Ltd.1 while performing model checking in customer projects and research is to formally model the I&C system as a graphical network of interconnected basic blocks (processing entities with memory) in a software tool called MODCHK [22], and then to model-check the formal model using this tool. Within the SAUNA (“Integrated safety assessment and justification of nuclear power plant automation”) project, wherein VTT and Aalto University collaborate on nuclear I&C formal verification research, one of the tasks is to develop a means of assessing the correctness of I&C system models based on examples of correct behavior of nuclear I&C systems. Previously, only basic blocks were assessed [7], and the goal of the present report is to develop an approach of assessing entire I&C networks. While in the industrial setting examples of correct system behavior can be provided by vendors, in this report we assume that these examples (or, from now on, traces) originate from a simulation model of a nuclear I&C system. This assumption allows controlling trace recording and acquiring the amount of traces sufficient for analysis. Apros, a continuous process simulator with a support of nuclear automation systems, is used as a simulation environment, and the simulation model employed in the case study is a generic model of an NPP. The contribution of the report is twofold. The first contribution is a methodology of assessing the correctness of formal control logic models which are based on function blocks, assuming the availability of behavior traces. This methodology involves conversion of the traces into formal models, which are then used to compare the output values in these traces with the ones produced by the formal model of the I&C system. The scope of possible application of the methodology is wider than nuclear I&C verification. The second contribution, which has a direct connection to the SAUNA project, is the list of common issues identified in MODCHK models, and the results of analysis of these issues. The rest of the report is structured as follows. In Section 2, basic concepts used throughout the report are introduced. Then, Section 3 introduces the proposed methodology, Section 4 evaluates the methodology on a case study, and Section 5 concludes the report.

1

https://www.vttresearch.com/

2

2 2.1

Background Model checking

Model checking [13, 4] is a technique of correctness assurance of hardware and software systems by means of exhaustive state space exploration of the formal model of the system. In model checking, the system to be checked is modeled formally using either state machines or textual languages. The latter are employed by model checkers such as NuSMV [11] and SPIN [17]. Requirements to be checked are also modeled formally, which is done using formal languages such as linear temporal logic (LTL), computation tree logic (CTL) and property specification language (PSL). These languages allow expressing requirements involving dependencies between the state of the system in different time instants. In particular cases, for example when verifying LTL requirements, if a requirement is violated, model checkers are able to return a counterexample – a behavior trace of the formal model which proves the violation of the requirement.

2.2

Model checking of nuclear I&C systems

Model checking is widely used in ensuring safety and correctness of industrial automation systems, especially PLCs [15, 24, 20]. Since 2008, model checking has been used in the Finnish nuclear industry to verify I&C systems [19, 22, 23]. In VTT, the tool MODCHK [22] is used for this task. MODCHK is a graphical environment where the user can specify the behavior of basic (elementary) function blocks, create modular I&C networks out of these blocks, specify formal requirements, verify them and view animated verification results. A screenshot of MODCHK is given on Fig. 1. On the left, the library of basic function blocks is visible. In the main part of the screen, an I&C network is shown. This particular network displays a fragment of one of the models of the system which will be described in Section 4. On the other hand, MODCHK can be treated as a front-end to a command-line model checker NuSMV [12]. In particular, basic blocks in MODCHK are specified directly as NuSMV modules (although their visual representation is specified in SVG). A basic block is an entity which operates in discrete steps and is able to keep memory between steps. Each basic block has a number of input and output variables, each of which can be either Boolean or integer. Integer variables can be used to approximate continuous signals. In MODCHK, each variable V is associated with a Boolean signal V_FAULT, which is needed for fault processing [21], a common measure to increase the reliability of nuclear I&C logic. Moreover, each input variable is associated with a Boolean signal V_CONNECTED, which tells whether this input variable is connected to the basic block in a block network. Additionally, each basic block may have Boolean or integer parameters, whose values are specified for each particular basic block instance (i.e., concrete occurrence of the basic block in a block network) and are constant for each instance. On each step, the basic block relates given input values with corresponding output values (this mapping of inputs to outputs is allowed to be nondeterministic).

3

Figure 1: MODCHK tool. As an example, consider the unit delay block with Boolean input and output variables. This block returns FALSE on the first step, and the previous input value on each next step. Below, it is provided in the NuSMV format: 1 2 3 4 5 6 7 8 9 10 11 12

MODULE UNIT_DELAY(INPUT, INPUT_FAULT, INPUT_CONNECTED) VAR last: boolean; lastFault: boolean; DEFINE OUTPUT := last; OUTPUT_FAULT := lastFault; ASSIGN init(last) := FALSE; next(last) := INPUT; init(lastFault) := FALSE; next(lastFault) := INPUT_FAULT;

An I&C block network (from now on, also block network or simply network) is a collection of interlinked instances of basic blocks or other block networks. These instances are linked to each other by directed connections. Block networks have input and output variables just like basic blocks. Input and output variables, which are graphically represented as input and output ports of basic blocks or nested networks, are allowed to be the starting or terminal points of connections, respectively. MODCHK is able to generate modular NuSMV models from I&C block networks, automatically run NuSMV on these models against the requirements specified in LTL, CTL or PSL, and display verification results. If a requirements is not satisfied,

4 a counterexample can be visually shown in the model by highlighting the values of each signal for each step of the counterexample.

2.3

Apros process simulator

Discrete-state formal models such as the ones which can be created in MODCHK are used in formal verification. If the purpose of the model is instead to simulate the behavior of the original system, discretization of continuous values is not needed, and the flow of time can be modeled with a significantly higher precision. From now on, such models will be referred to as simulation models. Apros2 is a continuous process simulator which has a support of NPP modeling. A simulation model in Apros is a collection of interconnected networks, wherein each network models either plant structure and processes or automation logic. Often, plant and automation networks are connected in a closed loop. Each network in Apros, like in MODCHK, is composed of a number of interconnected basic blocks. Unlike in MODCHK, Apros basic blocks can operate with real values. A screenshot of Apros is given in Fig. 2.

Figure 2: Apros 6 Nuclear. MODCHK and Apros are partially integrated: in VTT, a library of MODCHK basic block has been developed to mimic the behavior of Apros basic blocks. This allows formal verification of Apros models once they are transformed to MODCHK ones. Current verification practices in VTT include the use of only open-loop models, wherein the I&C (automation) logic is considered separately from physical processes, and the latter are replaced with arbitrary, nondeterministic sensor measurements 2

http://www.apros.fi/en/

5 within specified ranges. In this report, this procedure is assumed to be manual, which means that corresponding basic blocks are created in MODCHK and interlinked to repeat the connections in the Apros model. A more recent version of MODCHK supports import of I&C block network models from Apros.

2.4

Related work

In the work [7], which has been prepared during the previous year on the SAUNA project, the problem of ensuring the reliability of basic block models is considered. Using Apros basic blocks for the case study and applying formal synthesis, this work proposes a methodology to assess the correctness of NuSMV basic block models which have been manually prepared based on specification (in the Finnish nuclear industry, the original basic block implementations are often closed-source). Being a continuation of [7], the present report considers the correctness of the entire MODCHK model with respect to the original simulation model. Assessing formal models on a more general level than the one of basic blocks improves the overall reliability of assessment.

6

3

Methodology

In this report, we assume that two different models of the system are available: 1. The closed-loop simulation model S of the system, which is represented as a number of function block diagrams. In this report, we will consider simulation models developed in Apros. Other possible simulation environments include Matlab/Simulink3 , Ptolemy II4 , Modelica5 and IEC 61499 development environments such as nxtSTUDIO6 . 2. The discrete-state formal model M of the system, which also consists of function block diagrams. Only the controller part of this model is needed. The formal model is assumed to be based on the simulation model: in the ideal case this means that each of the basic blocks in the formal model corresponds to a similar basic block in the simulation model, and the connections between the blocks in two models are identical. In our case study, formal models are manually prepared in the MODCHK tool [22], which is used in the SAUNA project. The goal of the proposed methodology is to evaluate to which extent the approximation of S by M is adequate and to identify the patterns of discrepancies between these model, hereinafter referred to as issues. This will be done using the MODCHK tool [22] as an example. We will distinguish issue instances, which are concrete discrepancies with identified causes in a certain block network, and issue classes, to which issue instances are grouped based on similarity of their causes. In general, issue classes may be arranged in a hierarchy, which can be represented as a tree. We also allow an issue to belong to several issue classes if it has multiple causes. Fig. 3 shows a tree of issues which has been obtained based on a case study in Section 4. The idea of the methodology brings the approach from the work [7] (see Section 2.4) to a broader level: instead of assessing each of the basic blocks in M , the formal model M is assessed as a whole. As a result, the proposed approach can additionally check correctness of basic block connections, instantiation, and the general adequacy of M with respect to S. A straightforward solution would be to generalize the approach from [7]: automatically synthesize models of entire block networks, not just single basic blocks. Since S is available, traces showing the behavior of the system can be recorded from this model, and then formal synthesis tools (such as [25, 16, 14, 10]) can be applied to construct a state machine M 0 mimicking the behavior of S. However, this approach turns out to be not only computationally complex (formal synthesis tools are not sufficiently powerful yet), but also methodologically complex given the availability of S. Suppose that the set of behaviors of the generated state machine model M 0 is compared with the one of M . This comparison can be implemented as a run of model checking for an invariant which checks that the output 3

https://www.mathworks.com/products/simulink.html http://ptolemy.berkeley.edu/ptolemyII/ 5 https://www.modelica.org/ 6 http://www.nxtcontrol.com/en/engineering/ 4

7 Issues 1. Issues related to basic block implementation

2. Issues related to basic block use

3. Issues related to precision

4. Issues related to the simulation model

1.1. Basic block initialization

2.1. Incorrect parameters

3.1. Insufficient value precision

4.1. Error in the simulation model

1.2. Incorrect processing of continuous signals in basic blocks

2.2. Incorrect connections

3.2. Insufficient time precision

4.2. Problematic initial state of the simulation model

2.3. Lack of basic blocks

Figure 3: Hierarchical arrangement of issues based on their causes. sequences of M and M 0 are identical whenever the input sequence is. If this invariant is violated, the model checker (e.g., NuSMV) will return a counterexample composed of behavior traces of M and M 0 with different outputs. Unfortunately, since M 0 has been obtained by generalizing the traces of S, the M 0 part of the trace is not guaranteed to be adequate, i.e., correspond to a behavior of S. On the other hand, consider a different approach which does not involve construction of M 0 from traces. Suppose that all the values within a single trace τ of S are discretized, making the obtained trace τ 0 comparable with the traces of M . Then M is executed on the input values stored in τ 0 , and the outputs of M and τ 0 are compared. This procedure is repeated for all available traces of S. Similarly to the approach outlined above, traces of M 0 can be shown to diverge from the ones obtained based on S and independently from M , but now each alternative trace is known to correspond to a behavior of S since it has been obtained from such a behavior by discretizing it. This leads to higher reliability. Below, this alternative methodology is explained in detail.

3.1

Stage 1: model conversion

On the first stage of the methodology, the formal model M is converted to the format of a model checker. In our case, MODCHK has a built-in generator of textual NuSMV models. Fig. 4 shows a fictitious example of an I&C block network in MODCHK. This network returns either the sum or the difference of its two integer inputs IN0 and IN1 depending on the value of the third Boolean input SWITCH. Below, the NuSMV model produced from this network is given. 1 2

MODULE PlusMinus(IN0, IN0_FAULT, IN0_CONNECTED, IN1, IN1_FAULT, IN1_CONNECTED, SWITCH, SWITCH_FAULT, SWITCH_CONNECTED)

8

Figure 4: PlusMinus block network. 3

VAR

4

SW: BINARY_SWITCH(SWITCH, SWITCH_FAULT, TRUE, PLUS.ADDER_OUTPUT_SIGN, PLUS.ADDER_OUTPUT_SIGN_FAULT, TRUE, MINUS.ADDER_OUTPUT_SIGN, MINUS.ADDER_OUTPUT_SIGN_FAULT, TRUE, FALSE); PLUS: ADDER_3(0, FALSE, FALSE, IN0, IN0_FAULT, TRUE, IN1, IN1_FAULT, TRUE, 1, 1, 1); MINUS: ADDER_3(0, FALSE, FALSE, IN0, IN0_FAULT, TRUE, IN1, IN1_FAULT, TRUE, 1, 1, -1); DEFINE OUT := SW.BSWITCH_OUTP_SIGN; OUT_FAULT := SW.BSWITCH_OUTP_SIGN_FAULT;

5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29

MODULE main VAR PlusMinus001 : PlusMinus(IN0, IN0_FAULT, IN0_CONNECTED, IN1, IN1_FAULT, IN1_CONNECTED, SWITCH, SWITCH_FAULT, SWITCH_CONNECTED); IN0: -10..10; IN0_FAULT: boolean; IN0_CONNECTED: boolean; IN1: -10..10; IN1_FAULT: boolean; IN1_CONNECTED: boolean; SWITCH: boolean; SWITCH_FAULT: boolean; SWITCH_CONNECTED: boolean;

In the listing, each basic block instance in the network corresponds to a NuSMV VAR declaration (lines 4–11), where parameters and connections of this instance are specified. In module main, the ranges for all input variables are provided (lines 21–29).

9

3.2

Stage 2: trace recording

On the second stage of the methodology, the simulation model S is executed multiple times, and the relevant data is recorded as behavior traces. These data include input and output parameters of the I&C block network to be examined. In addition, all outputs of internal basic blocks are also recorded: later, if a discrepancy between outputs in two behavior traces is found, basic block outputs will allow the analyst to identify the exact location in the network causing the problem. Formally, a trace element is defined as a tuple (I, O, B), where I is a list of m inputs, O is a list of n block network outputs, and B is a list of k basic block outputs. A raw trace, or a trace which is recorded from S, is a finite sequence of trace elements τ = ((I1 , O1 , B1 ), ..., (I` , O` , B` )), where ` is its length. When a raw trace is recorded, the time between two adjacent trace elements is ∆t. Since simulation clock and the wall clock may have different rates, ∆t is measured in simulation time. As a technique of trace recording, various approaches can be used. To record different traces, the performed runs of S must also differ. Generally, this can be achieved by applying changes to any part of S which is outside the block network to be examined. Such changes can be related to the plant model (including its initial state) and the models of other block networks, and are specific for each new trace recording. In this report, we apply a technique similar to the one used for trace recording in previous SAUNA works [6, 5], since it is easy to implement for an Apros model and works well on our case study. It involves changing the initial plant state and adding small changes, called mutations, to other block networks. Ideally, the collected traces must cover various input scenarios to be examined. Ensuring such coverage is beyond the scope of this report. Some experimental results on various trace collection strategies are reported in [3].

3.3

Stage 3: trace discretization

To allow comparison of the traces collected from S and the ones of M , the former must be discretized. A discretized trace is a finite sequence τ 0 = ((I10 , O10 , B10 ), ..., (I`0 , O`0 , B`0 )) which differs from τ by applying discretization to each real value to cast it to an integer. Value discretization is needed since the most popular model checkers, such as NuSMV, work exclusively with discrete models.7 Since rounding commonly causes precision loss, value scaling [22] may be reasonable. If v is a real value belonging to the interval [vmin , vmax ], its discretized version v 0 is computed by rounding sv · v to the closest integer, where sv is the scaling coefficient of v: for each value, it is chosen to ensure a sufficient number of possible values of v 0 . For example, if v ∈ [0.1, 0.2], choosing sv = 1000 will result in v 0 ∈ {100, 101, ..., 200}. To improve the clarity of discretization for analysts, sv can be chosen as a power of 10. On the other hand, creating the formal model M based on S involves similar discretization, and thus the same value of sv must be used while both creating M and obtaining τ 0 . Finally, note that the use of large integer ranges has a shortcoming: model checking complexity 7

nuXmv model checker (https://nuxmv.fbk.eu/) partially supports real values, but is available exclusively for non-commercial and academic purposes.

10

I0 O0 B0

Table 1: Example of a discretized trace. Element index 1 2 3 IN0 5 5 5 IN1 7 7 8 SWITCH FALSE TRUE TRUE OUT 12 −2 −3 PLUS.ADDER_OUTPUT_SIGN 12 12 13 MINUS.ADDER_OUTPUT_SIGN −2 −2 −3 SW.BSWITCH_OUTP_SIGN 12 −2 −3

of M increases. An example of a discretized trace for the PlusMinus block network (Fig. 4) with ` = 3 is given in Table 1.

3.4

Stage 4: converting traces to formal models

On the fourth stage, each available discretized trace is converted to a trace-specific formal model. This process is straightforward and involves specifying the values of all parameters within the trace for each discrete time instant (thus, one discrete step of the formal model will correspond to the amount of time ∆t in the simulation model). Trace model generation was implemented within the toolset EFSM-Tools.8 As an example, the NuSMV model for the trace from Table 1 is given below: MODULE TRACE VAR step: 1..3; ASSIGN init(step) := 0; next(step) := step < 3 ? step + 1 : 3; DEFINE IN0 := case step in 1..3: 5; esac; IN0 := case step in 1..2: 7; step in {3}: 8; esac; SWITCH := case step in {1}: FALSE; step in 2..3: TRUE; esac; OUT := case step in {1}: 12; step in {2}: -2; step in {3}: -3;

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22

8

https://github.com/ulyantsev/EFSM-Tools

11 esac; PLUS__ADDER_OUTPUT_SIGN := case step in 1..2: 12; step in {3}: 13; esac; MINUS__ADDER_OUTPUT_SIGN := case step in 1..2: -2; step in {3}: -3; esac; SW__BSWITCH_OUTP_SIGN := case step in {1}: 12; step in {2}: -2; step in {3}: -3; esac;

23 24 25 26 27 28 29 30 31 32 33 34 35 36

At this point, it is worth mentioning that infinite model behaviors are usually considered in model checking. Yet, for the purpose of trace execution, the behavior of the model after the trace ends is not important. In the listing above, the last element of the trace is looped over (line 6).

3.5

Stage 5: trace comparison

On the fifth stage, each model of discretized trace τ 0 is composed with M , and both models are executed on the input sequence of τ 0 . Model execution was automated by running NuSMV in the interactive mode (command line option “-int”). The tool used to record traces is available online.9 Model composition to be executed is automatically obtained from M and τ 0 . An example of a model composition obtained from the PlusMinus system (Fig. 4) and the trace from Table 1 is given below: MODULE PlusMinus ... (as defined in previous listings)

1 2

MODULE TRACE ... (as defined in previous listings)

3 4

MODULE main VAR trace: TRACE; PlusMinus001: PlusMinus( trace.IN0, FALSE, TRUE, trace.IN1, FALSE, TRUE, trace.SWITCH, FALSE, TRUE );

5 6 7 8 9 10 11 12

Then, the values of all variables stored in τ 0 are recorded during NuSMV execution and transformed into plain text human-readable tables. In particular, the variables in O0 and B 0 are taken twice: from τ 0 and M . An example of such a table for the PlusMinus system (Fig. 4) is given below: 9

https://github.com/igor-buzhinsky/synthesis-aided-basic-block-assurance

12 1 2 3 4

IN0 5 5 5

IN1 7 7 8

SWITCH 0 1 1

[ 12 -2 -3

OUT] 13 -2 -3

[ 12 12 13

PLUS] 13 13 13

[ -2 -2 -3

MINUS] -2 -2 -3

[ 12 -2 -3

SW] 13 -2 -3

On top, you see a header with column names. For each variables from O0 and B 0 , a pair of columns, combined in square brackets, is given. The left column of each pair gives the output of the formal model M , and the right column gives the output of the discretized trace τ 0 . In the header, the values PLUS.ADDER_OUTPUT_SIGN, MINUS.ADDER_OUTPUT_SIGN and SW.BSWITCH_OUTP_SIGN are abbreviated as PLUS, MINUS and SW respectively, and the output OUT is identical to SW. In the rest of the table, Boolean values are shown as 0 and 1. In this fictitious case, the output of M corresponds to the trace from Table 1 (this trace has been previously introduced as a valid trace of M ), but the trace output is different for variables OUT, PLUS and SW. The analyst would then examine such differences on the next stage of the methodology. An alternative solution would be to model-check the model composition to identify output deviations automatically. In the simplest case, this can be done with the following LTL formula:   G

^

vM = vτ 0  ,

v∈O0 ∪B 0

where G is the temporal operator with the meaning “always”, and vM and vτ 0 are the values of v in M and τ 0 respectively. However, output deviations were found to be easily visible in value tables, making the solution based on model checking less convenient to use.

3.6

Stage 6: issue detection

The last stage of the methodology is issue detection based on value tables obtained on Stage 5. It is based on manual analysis of output deviations between the model and the traces, which involves understanding the causes of such deviations, determining whether these deviations can be regarded as issues, grouping patterns of issues into issue classes, and disabling certain issues to be able to see more issues. The simulation model S operates with real values, and only after that trace collection discretization is applied. In contrast, the formal model M operates with initially discrete (integer) values. This distinction makes precision a frequent cause of output deviations. If such deviations are minor and their connection with more serious behavior differences (such as an activation of a protection function in M but not in a simulation trace) cannot be established, there is no need to recognize them as issues. They can be either retained or hidden by increasing the scaling coefficient of the related variables. If a particular deviation is regarded by the analyst as serious, the following actions are performed: 1. The causes of the deviation at the same element of the same trace are searched. For example, if a difference in SW.BSWITCH_OUTP_SIGN is identified, the outputs

13 of the blocks which flow into SW are examined (in the case of PlusMinus, these outputs would be PLUS.ADDER_OUTPUT_SIGN and MINUS.ADDER_OUTPUT_SIGN). If either of these outputs differ, the same examination is applied to the origin of these outputs. Eventually, this procedure can identify the basic blocks whose output deviations cannot be explained by output deviations of their inputs. 2. Other traces are looked for the presence of similar output deviations. Hypotheses are made regarding the causes of the issue being defined. These hypotheses are checked by examining basic block models (which, e.g., can reveal a fault in such a model) and the system in general (which, e.g., can reveal incorrect connections between basic blocks). 3. If the identified issue cannot be yet attributed to the issue classes defined so far, a new issue class is created describing the issue, including its circumstances and causes. Otherwise, if the issue is related to one or more issues which were previously identified and grouped into an issue class, if necessary, this issue class is extended to encompass the circumstances and causes of the newly discovered issue. 4. To simplify the search of other issues, the discovered issue can be excluded from the value table. This can be done, e.g., by increasing value precision, fixing basic block connections in M , changing basic block parameters in M , modifying basic block models. 5. Once the issue is excluded, the analysis is repeated starting from Stage 5. If the exclusion of the issue is too laborious to be done, the analyst just ignores output deviations which can be recognized as related to this issue and similarly returns to Stage 5.

14

4

Case study

This section evaluates the proposed methodology on several subsystems of an NPP model.

4.1

Generic PWR model

The case study used in this report is based on the generic PWR model, provided to Aalto University by Fortum Power and Heat Oy10 , a Finnish power company. This Apros model represents an NPP with a pressurized water reactor (PWR). The generic PWR model includes over 40 I&C networks which are executed in closed loop together with the plant model, which is also represented by multiple plant networks. Among the available I&C networks, or subsystems, we only consider eight, which were examined in previous SAUNA works [6, 5]. The generic PWR model is confidential and only basic information about it may be disclosed – for this reason we will refer to the considered subsystems as S1, ..., S8. Their purpose is to implement protection functions of the NPP (S1–S4), shut down reactor and turbines in case of an emergency (S5), control the pressurizer liquid level and pressure (S6, S7) and the feed water tank level (S8). S1, ..., S8 were modeled in MODCHK based on Apros diagrams, and in this section of the report the adequacy of these formal models is evaluated. For more information about the generic PWR model and its subsystems, see the works [6, 5, 9, 8, 18].

4.2

Applying the methodology

Stage 1 of the methodology (model conversion) was done by applying the internal NuSMV model generator of MODCHK. On Stage 2 (trace recording), 240 traces were recorded for each of the subsystems from the Apros model with the frequency of one trace element per simulation second (∆t = 1 s). The changes introduced to the simulation model to record each new trace were designed to allow sufficient variability in input sequences of subsystems. Stages 3 (trace discretization), 4 (converting traces to formal models) and 5 (trace comparison) were performed using the software tools mentioned in Section 3. Stage 6 (issue detection) was performed manually for each subsystem. The analysis for each of them took around 4–8 person hours.

4.3

Discovered issue classes

Below, we provide the list of issues discovered during Stage 6 of the methodology. They are classified into issue classes which were already shown in Fig. 3. These findings allow making conclusions regarding the suitability of MODCHK for modeling continuous systems, which will be given in Section 5. 10

https://www.fortum.fi/

15 4.3.1

Issues related to basic block implementation

Issue class 1.1. Basic block initialization. Some basic blocks (such as delays) do not simply operate as functions, but store data which may affect their future behavior. In MODCHK model, these data are often initialized with default values, such as FALSE or 0. An issue occurs when the formal model starts with a default value, but some traces recorded from the simulation model begin with a different value. Issue class 1.2. Incorrect processing of continuous signals in basic blocks. This issue class comprises issues related to the difficulty of expressing continuous logic (such as the one of a PID controller) in discrete-state basic block models, where both variable values and time are discrete. 4.3.2

Issues related to basic block use

Issue class 2.1. Incorrect parameters. Parameter values of basic block instances do not correspond to the ones in the simulation model. Issue class 2.2. Incorrect connections. Connections between basic blocks do not correspond to the ones in the simulation model. Issue class 2.3. Lack of basic blocks. Some basic blocks are present in the simulation model but missing in the formal model, either mistakenly or to reduce modeling efforts. 4.3.3

Issues related to precision

Issue class 3.1. Insufficient value precision. Rounding real values to integers causes errors in processing real data. If the model contains functions which are activated when certain variables reach certain thresholds, these functions can be delayed (or activated earlier) for multiple cycles, not activated when they must, or activated spuriously. Issue class 3.2. Insufficient time precision. Trace elements are recorded with a constant frequency. In the simulation model, some condition may have a rising edge followed by a falling edge with both these events happening between recording the adjacent trace elements. As a result, these events cannot be detected in the formal model, and thus functions triggered by this condition are not activated (while they are in the simulation traces). Note that issues of this class are more caused by the methodology of model assessment rather than the formal model: even though time precision cannot be made infinite, events lost due to such issues are still possible in the formal model, the only difference is that they last for at least one discrete step. 4.3.4

Issues related to the simulation model

Issue class 4.1. Error in the simulation model. The simulation model may contain an error, which may be either related to modeling or the modeled system.

16 The formal model may either repeat this error (making the detection of the issue more difficult) or not. Issue class 4.2. Problematic initial state of the simulation model. Certain states of the simulation model may not allow certain functionality to be achieved, e.g., if some protection function is already activated, there may be no means to repeat this activation. Such issues relate to both the simulation model and also to the trace collection procedure.

4.4

Concrete discovered issues

In this subsection, a list of discovered issues is provided for each subsystem. The issues are given in the order of their discovery and are attributed to one or more issue classes (see Section 4.3). 4.4.1

S1 (activation of emergency pumps)

Issue class: 1.1. Basic block initialization Issue description: In MODCHK, the previous values of delay basic blocks are always initialized with FALSE, while in traces these values may be TRUE immediately. Thus, in MODCHK it may take the number of cycles equal to the length of the delay to make the delay block output TRUE, which is not the case for the simulation model. In S1, this makes a protection function activated earlier in the simulation model than in MODCHK. How the issue was excluded: The issue was retained. Issue class: 2.3. Lack of basic blocks Issue description: In Apros, measurement basic blocks are used to bind the values in plant networks to the inputs of automation (I&C) networks. In addition, these blocks model the measurement process, and in particular they will smoothen rapid changes of measured values. Such changes occur when the plant state of the simulation model is changed to record a new trace. As a result, the real and measured values may significantly differ in the beginning of the traces. Measurement blocks were not modeled in MODCHK. Hence, the MODCHK I&C network uses raw input values while the corresponding Apros network uses measured ones. Significant differences in inputs sometimes lead to major behavior differences, e.g., an activation of a protection function in only one of the networks. How the issue was excluded: Measured values were included into the set of parameters recorded in traces. These values were fed to the MODCHK model instead of actual values.

17 Issue class: 3.1. Insufficient value precision Issue description: In the MODCHK model, a protection function may be activated several steps later than in the Apros model or not activated at all. This is caused by rounding errors near the threshold which is checked by a limit value checker basic block and is needed to be passed to activate the function. How the issue was excluded: Relevant parameters were scaled in the MODCHK model and while discretizing traces to make precision errors negligible. Issue class: 2.1. Incorrect parameters Issue description: One of the actions related to a protection function was never performed in the MODCHK model due to a misconfigured Boolean parameter of a basic block related to control. How the issue was excluded: The value of the incorrectly specified parameter was changed to the one from the Apros model. 4.4.2

S2 (closing emergency feed water lines)

Issue class: 3.1. Insufficient value precision Issue description: A protection function may fail to activate in the MODCHK model due to rounding and further loss of precision during arithmetical operations, while it is activated in the Apros model. How the issue was excluded: Relevant parameters were scaled in the MODCHK model and while discretizing traces to make precision errors negligible. Issue class: 2.1. Incorrect parameters; 2.2. Incorrect connections; 4.1. Error in the simulation model Issue description: This issue is a result of an interplay of an error in the Apros model and an attempt to model the I&C network in MODCHK in order to comply with the specification. While the MODCHK model complies with the specification, the Apros model does not. The differences between the models are located in adder function blocks, which can actually produce linear combinations of their arguments (the coefficients thereof are provided as parameters). In Apros, these adder blocks produce the negation of the correct output. However, in MODCHK, these blocks produce correct outputs, which is a result of (1) a different assignment of connections to their input ports, (2) a different order of their parameters (linear combination coefficients), and (3) a different mapping of parameters to input ports. How the issue was excluded: The issue was retained.

18 4.4.3

S3 (preventing low pressure in steam generators)

Issue class: 1.2. Incorrect processing of continuous signals in basic blocks Issue description: The implementation of the gradient basic block in MODCHK was not properly working. As a result, a rapid change of a value was not smoothened. How the issue was excluded: Due to the complexity of implementing the gradient block in NuSMV to work adequately in various situations, the implementation of the block was corrected specifically for this subsystem. 4.4.4

S4 (opening emergency feed water lines)

Issue class: 3.1. Insufficient value precision Issue description: The activation of a protection function may be delayed for multiple time steps due to rounding of values near the threshold of a limit value checker basic block. How the issue was excluded: Relevant parameters were scaled in the MODCHK model and while discretizing traces to make precision errors negligible. Issue class: 1.1. Basic block initialization Issue description: Due to the memory of the delay basic block model being always initialized with FALSE, a protection function is activated spuriously on the first step of the behavior of the MODCHK model. In the Apros model, when the safety function does not need to be activated, the delay block stores the value TRUE. How the issue was excluded: The delay basic block model was modified specifically for this subsystem to be initialized with TRUE. Issue class: 3.2. Insufficient time precision Issue description: An event (certain signal reaching a certain value) which happens in the Apros model is not noticed in the MODCHK model. As a result, a protection function is not activated in a MODCHK trace while it does in the corresponding Apros trace. How the issue was excluded: The issue was retained. 4.4.5

S5 (reactor and turbine trip)

Issue class: 2.3. Lack of basic blocks Issue description: Measurement basic blocks are missing in the MODCHK model. The issue is identical to the issue of the same class described in Section 4.4.1. How the issue was excluded: Measured values were included into the set of parameters recorded in traces. These values were fed to the MODCHK model instead of actual values.

19 Issue class: 1.1. Basic block initialization Issue description: In the MODCHK model, the memory of the gradient basic block, which is essentially its instantaneous output, is initialized with 0. This does not correspond to the simulation model. As a result, significant differences in the output of the gradient basic block can be observed for multiple time steps. Eventually, the outputs become almost identical. How the issue was excluded: The issue was retained. Issue class: 1.2. Incorrect processing of continuous signals in basic blocks Issue description: The issue is identical to the issue of the same class described in Section 4.4.3. How the issue was excluded: Due to the complexity of implementing the gradient block in NuSMV to work adequately in various situations, the implementation of the block was corrected specifically for this subsystem. The resultant implementation differs from the one created for S3. 4.4.6

S6 (pressurizer liquid level control)

Issue class: 4.2. Problematic initial state of the simulation model; 1.1. Basic block initialization Issue description: In all the traces recorded from the Apros model, the protection function has already been activated, and there is no situation when this function is switched off and then activated again (i.e., the action related to this function is performed always). In contrast, the initial state of the formal model assumes that the protection function is not yet activated, i.e., the action related to this function is performed only when a particular condition is met. As a result, the output controlled by this action differs in the traces of the MODCHK and the Apros models. How the issue was excluded: The issue was retained. 4.4.7

S7 (pressurizer pressure control)

Issue class: 2.3. Lack of basic blocks Issue description: The hysteresis basic block was not modeled in MODCHK to reduce the modeling effort. Consequently, the MODCHK model may respond to input changes faster than the Apros one. How the issue was excluded: The issue was retained.

20 Table 2: Numbers of occurrences of each issue class. Issue class Number of occurrences 1.1. Basic block initialization 4 1.2. Incorrect processing of continuous signals in basic 2 blocks 2.1. Incorrect parameters 2 2.2. Incorrect connections 1 2.3. Lack of basic blocks 3 3.1. Insufficient value precision 3 3.2. Insufficient time precision 2 4.1. Error in the simulation model 1 4.2. Problematic initial state of the simulation model 1 4.4.8

S8 (feed water tank level control)

Issue class: 3.2. Insufficient time precision Issue description: In the Apros model, a certain signal reaches the threshold of a limit checker, leading to a discrete control action which is active for the rest of the trace. The signal passes the threshold again in less than a second, and thus the event of reaching the threshold is not noticed in the MODCHK model. Hence, the MODCHK model does not perform the control action. How the issue was excluded: The issue was retained.

4.5

Analysis

Table 2 shows the number of occurrences of each issue class described in Section 4.3. Also accounting for the nature of the revealed issues, the following observations can be made: 1. A large group of issues is related to manual modeling. They may include deliberate model simplifications, such as omitting certain basic blocks, or errors while designing the block network (e.g., misconfigured basic block instances). Luckily, a more recent version of MODCHK supports model import from Apros, which will reduce manual work and the related chance of making an error. On the other hand, some modeling errors in MODCHK can be revealed by model checking this model and getting unusual results or counterexamples. In our case study, this stage was not performed, which may have lead to a higher abundance of manual modeling issues. 2. Faults in basic block models are relatively rare compared to other manual modeling issues. The work [7] proposes a much more meticulous method specifically to reveal issues in basic block models, yet they were not encountered in the context where these blocks were used to model I&C networks of the generic PWR model (except the issues which lead to one-cycle output differences of basic blocks, but such situations were not considered as issues in this report).

21 3. The second large group of issues is related to operations with real values and their approximations with integers. It includes value and time precision issues and issues related to incorrect processing of continuous signals in basic blocks. To some extent, these issues can be fixed by modifying the scaling coefficients of real variables (see Section 3.3). 4. In particular, the time precision issue may cause the formal model to behave differently from the simulation model due to a missed event, but in fact this does not hinder model checking capabilities: for example, in LTL model checking, since all model behaviors are checked, they will include the ones where this event occurs. A possible exception is related to a situation where continuous aspects of model behavior are verified (although this is not common in practice). In such situations the time corresponding to a discrete step of the formal model (in this report, ∆t = 1 s was used) should be reduced, and the basic block models with nontrivial processing of continuous signals (such as the gradient basic block) should be properly tested.

22

5

Conclusions

In this report, a methodology has been proposed to assess the adequacy of formal models of nuclear I&C systems which are developed in the software tool MODCHK. This methodology compares the formal model with behavior examples of the actual system or its simulation model. The case study, which has been performed on eight I&C networks of an NPP model, has revealed various issues in formal models, including the ones connected with manual modeling and processing of continuous signals. The list of found issues points to the most common sources of errors in formal models and thus can be helpful in improving the reliability of I&C models even when behavior examples are not available. In general, the tool MODCHK has been found applicable for modeling I&C networks working with real-valued data. The developed methodology can also be applied outside the nuclear I&C verification context since models based on function blocks are quite common (for example, they are used in the international standards IEC 61499 [2] and IEC 61131 [1]), as well as simulation models of automation systems. If a simulation model is available, a volume of behavior examples can be collected which is sufficient for proper analysis, and the developed methodology can be applied to reveal issues of a particular formal model of a function block network.

23

References [1] International Standard IEC 61131-3: Programmable controllers – Part 3: Programming languages, Second edition. International Electrotechnical Commission, Geneva, 2003. [2] International Standard IEC 61499-1: Function Blocks – Part 1: Architecture, Second edition. International Electrotechnical Commission, Geneva, 2012. [3] D. Avdyukhin, D. Chivilikhin, G. Korneev, V. Ulyantsev, and A. Shalyto. Plant trace generation for formal plant model inference: methods and case study. In IEEE 15th International Conference on Industrial Informatics (INDIN 2017), pages 746–752. IEEE, 2017. [4] C. Baier and J.-P. Katoen. Principles of model checking. MIT press, 2008. [5] I. Buzhinsky, A. Pakonen, and V. Vyatkin. Explicit-state and symbolic model checking of nuclear I&C systems: A comparison. In 43rd Annual Conference of the IEEE Industrial Electronics Society (IECON 2017), pages 5439–5446. IEEE, 2017. [6] I. Buzhinsky, A. Pakonen, and V. Vyatkin. Scalable methods of discrete plant model generation for closed-loop model checking. In 43rd Annual Conference of the IEEE Industrial Electronics Society (IECON 2017), pages 5483–5488. IEEE, 2017. [7] I. Buzhinsky, A. Pakonen, and V. Vyatkin. Synthesis-aided reliability assurance of basic block models for model checking purposes. In International Symposium on Industrial Electronics (ISIE 2018), pages 669–674. IEEE, 2018. [8] I. Buzhinsky and V. Vyatkin. Automatic inference of finite-state plant models from traces and temporal properties. IEEE Transactions on Industrial Informatics, 13(4):1521–1530, 2017. [9] I. Buzhinsky and V. Vyatkin. Modular plant model synthesis from behavior traces and temporal properties. In 22nd IEEE International Conference on Emerging Technologies and Factory Automation (ETFA 2017), pages 1–7. IEEE, 2017. [10] C.-H. Cheng, C.-H. Huang, H. Ruess, and S. Stattelmann. G4LTL-ST: Automatic generation of PLC programs. In International Conference on Computer Aided Verification, pages 541–549. Springer, 2014. [11] A. Cimatti, E. Clarke, E. Giunchiglia, F. Giunchiglia, M. Pistore, M. Roveri, R. Sebastiani, and A. Tacchella. NuSMV 2: An opensource tool for symbolic model checking. In International Conference on Computer Aided Verification, pages 359–364. Springer, 2002.

24 [12] A. Cimatti, E. Clarke, F. Giunchiglia, and M. Roveri. NuSMV: a new symbolic model checker. International Journal on Software Tools for Technology Transfer, 2(4):410–425, 2000. [13] E. M. Clarke, O. Grumberg, and D. Peled. Model checking. MIT press, 1999. [14] P. Faymonville, B. Finkbeiner, and L. Tentrup. BoSy: An experimentation framework for bounded synthesis. In International Conference on Computer Aided Verification, pages 325–332. Springer, 2017. [15] G. Frey and L. Litz. Formal methods in PLC programming. In IEEE International Conference on Systems, Man, and Cybernetics (SMC), volume 4, pages 2431–2436. IEEE, 2000. [16] G. Giantamidis and S. Tripakis. Learning Moore machines from input-output traces. In International Symposium on Formal Methods, pages 291–309. Springer, 2016. [17] G. J. Holzmann. The model checker SPIN. IEEE Transactions on Software Engineering, 23(5):279–295, 1997. [18] D. Kleyko, E. Osipov, N. Papakonstantinou, V. Vyatkin, and A. Mousavi. Fault detection in the hyperspace: Towards intelligent automation systems. In IEEE 13th International Conference on Industrial Informatics (INDIN 2015), pages 1219–1224. IEEE, 2015. [19] J. Lahtinen, J. Valkonen, K. Björkman, J. Frits, I. Niemelä, and K. Heljanko. Model checking of safety-critical software in the nuclear engineering domain. Reliability Engineering & System Safety, 105:104–113, 2012. [20] T. Ovatman, A. Aral, D. Polat, and A. O. Ünver. An overview of model checking practices on verification of PLC software. Software & Systems Modeling, 15(4):937–960, 2016. [21] A. Pakonen and K. Björkman. Model checking as a protective method against spurious actuation of industrial control systems. In 27th European Safety and Reliability Conference (ESREL 2017), pages 3189–3196. CRC Press, 2017. [22] A. Pakonen, T. Mätäsniemi, J. Lahtinen, and T. Karhela. A toolset for model checking of PLC software. In IEEE Conference on Emerging Technologies & Factory Automation (ETFA 2013), pages 1–6. IEEE, 2013. [23] A. Pakonen, T. Tahvonen, M. Hartikainen, and M. Pihlanko. Practical applications of model checking in the Finnish nuclear industry. In 10th International Topical Meeting on Nuclear Plant Instrumentation, Control and Human Machine Interface Technologies (NPIC & HMIT 2017), pages 1342–1352. American Nuclear Society, 2017.

25 [24] O. Pavlovic and H.-D. Ehrich. Model checking PLC software written in function block diagram. In Third International Conference on Software Testing, Verification and Validation (ICST 2010), pages 439–448. IEEE, 2010. [25] V. Ulyantsev, I. Buzhinsky, and A. Shalyto. Exact finite-state machine identification from scenarios and temporal properties. International Journal on Software Tools for Technology Transfer, 20(1):35–55, 2018.