Maturity Models in an Operational Context - Software Engineering ...

Maturity Models in an Operational Context Rich Caralli

Introductions Rich Caralli

CEWM Teams:

Technical Director

Cyber Enterprise and Workforce Management (CEWM) Directorate CERT Program

Enterprise Threat and Vulnerability Analysis Immersive Learning Technologies Resilient Enterprise Management Workforce Development

Agenda Maturity Models 101 The CERT® Resilience Management Model The SEI Smart Grid Maturity Model Summary

Maturity Models in an Operational Context

Maturity Models 101

Maturity Modeling 101 Traceable to Philip Crosby – ―Quality Is Free‖ – The Quality Management Maturity Grid

Focused on defining how to institutionalize a focus on quality Set the tone for applying a ―maturity‖ approach to all processes that have a quality requirement Influenced the later development of other maturity models

Published in 1979 by McGraw-Hill

SEI Maturity Models SEI published the first capability maturity model in 1995: Guidelines for Improving the Software Process Focused on measuring the maturity of software process capabilities from ad-hoc to disciplined Process maturity is the focus of the progression of the model

Process maturity is measured by how well software engineering processes are performed and institutionalized

Not all maturity models are the same Models based on CMMI measure process maturity by examining the degree to which institutionalizing factors are present Other maturity models measure other attributes such as: – Technological progression – Progression of controls or practices – Progression of traits or attributes, such as quality or categories of management

This can be confusing: often maturity models are ―branded‖ as being produced in the likeness of CMMI, but in fact are NOT capability maturity models and do not measure process maturity

Rise of maturity models Measurement is driving the need for new maturity models Serve as a barometer for where organizations are now and where they want to go New models emerging in the security and resilience space—focused on operational processes (continuous) rather than closed-ended processes such as software engineering (i.e., something is produced at the end of the process)

Separating good and bad models becoming more difficult

Maturity Models in an Operational Context

CERT-RMM and Operational Risk Management

An expanding operational risk environment

Operational risk and resilience

Actions of people

Systems and technology failures

Failed internal processes

External events

Operational resilience emerges from effective operational risk management

Operational resilience and convergence Organization Mission

Operational Resilience

Supply Chain Management

Security Management

Business Continuity

IT Operations Management

Materials Management

Operational Risk Management

Convergence directly affects the level of operational resilience. Level of operational resilience affects the ability to meet organizational mission.

What is CERT®-RMM? CERT-RMM is a maturity model for managing and improving operational resilience.

Guides implementation and management of operational resilience activities

Converges key operational risk management activities: security, BC/DR, and IT operations Defines maturity through capability levels (like CMMI) Enables measurement “…an extensive super-set of the things an organization could do to be more resilient.”

- CERT-RMM adopter

Improves confidence in how an organization responds in times of operational stress

CERT-RMM Quick View Engineering

Operations Management

ADM

Asset Definition and Management

AM

Access Management

CTRL

Controls Management

EC

Environmental Control

RRD

Resilience Requirements Development

EXD

External Dependencies Management

RRM

Resilience Requirements Management

ID

Identity Management

RTSE

Resilient Technical Solution Engineering

IMC

Incident Management & Control

SC

Service Continuity

KIM

Knowledge & Information Management

PM

People Management

Enterprise Management

TM

Technology Management

COMM

Communications

VAR

Vulnerability Analysis & Resolution

COMP

Compliance

EF

Enterprise Focus

Process Management

FRM

Financial Resource Management

MA

Measurement and Analysis

HRM

Human Resource Management

MON

Monitoring

OTA

Organizational Training & Awareness

OPD

Organizational Process Definition

RISK

Risk Management

OPF

Organizational Process Focus

Positioning models in the lifecycle

Process institutionalization in CERT-RMM Processes are acculturated, defined, measured, and governed

Level 3 • Defined Level 2 • Managed Level 1

Practices are performed Practices are incomplete

• Performed

Higher degrees of institutionalization translate to more stable processes that • produce consistent results over time • are retained during times of stress

Level 0 • Incomplete

Capability levels are used in CERT-RMM to measure process institutionalization

Example: Measuring Capability Level 2 in Service Continuity (SC) Process Area Generic Goals GG1

Achieve Specific Goals √

GG1.GP1

Perform Specific Practices

√

GG2

Institutionalize a Managed Process

√

GG2.GP1

Establish Process Governance

√

GG2.GP2

Plan the Process

√

GG2.GP3

Provide Resources

√

GG2.GP4

Assign Responsibility

√

GG2.GP5

Train People

√

GG2.GP6

Manage Work Product Configurations

√

GG2.GP7

Identify and Involve Relevant Stakeholders

√

GG2.GP8

Monitor and Control the Process

√

GG2.GP9

Objectively Evaluate Adherence

√

GG2.GP10

Review Status with Higher-Level Managers

√

Capability level 2 is achieved by •

•

Generic Practices

Sustaining capability level 1 plus

Satisfying generic goal 2 by performing the associated 10 generic practices

CERT-RMM Sample Class A Appraisal Results

CERT-RMM Users Group (RUG) Innovation in supporting model adoption Year-long series of 4 workshops helps participating organizations use CERT-RMM to make improvements 1. Establish objectives & scope

4. Evaluate results; plan next steps

2. Define project & initiate diagnostic

3. Plan actions & measures; define process

First year participants: • United States Postal Inspection Service • Discover Financial Services • Lockheed Martin • Carnegie Mellon University ISO • CERT

Resilience measurement research Ongoing research at CERT is exploring operational resilience measurement. This first publication – Defines key measurement terminology and concepts – Derives high-level objectives for measuring resilience – Presents a methodology and template for defining measurements to address specific management objectives

www.cert.org/resilience

Maturity Models in an Operational Context

Improving the Smart Grid using SGMM

What Is the Smart Grid Maturity Model?

SGMM is a management tool that provides a common framework for defining key elements of a smart grid transformation and helps utilities develop a programmatic approach to track their progress

SGMM History 2007

2008

Global Intelligent Utility Network Coalition (GIUNC) develops SGMM GIUNC: • CenterPoint Energy • Progress Energy • DONG Energy • North Delhi Power Ltd • Country Energy • Sempra Energy • Pepco Holdings • IBM • APQC

2009

2010

Utilities use SGMM v1.0

2011

Utilities use v1.1

Software Engineering Institute serves as model steward SEI releases SGMM v1.1 product suite Certification program for SGMM Navigators begins SEI releases SGMM v1.2 product suite

v1.2

SGMM at a Glance

8 Domains: Logical groupings of smart grid related capabilities and characteristics

175 Characteristics: Features you would expect to see at each stage of the smart grid journey

6 Maturity Levels: Defined sets of characteristics and outcomes

Not a process model – a lightweight set of ordered characteristics that reflect maturity of smart grid implementation

The Smart Grid Maturity Model – Domains Strategy, Mgmt & Regulatory Vision, planning, governance, stakeholder collaboration

Organization and Structure Culture, structure, training, communications, knowledge mgmt

Grid Operations Reliability, efficiency, security, safety, observability, control

Work & Asset Management Asset monitoring, tracking & maintenance, mobile workforce

Technology IT architecture, standards, infrastructure, integration, tools

Customer Pricing, customer participation & experience, advanced services

Value Chain Integration Demand & supply management, leveraging market opportunities

Societal & Environmental Responsibility, sustainability, critical infrastructure, efficiency

The Smart Grid Maturity Model – Levels PIONEERING

OPTIMIZING

INTEGRATING

ENABLING

INITIATING

DEFAULT

Breaking new ground; industry-leading innovation

Optimizing smart grid to benefit entire organization; may reach beyond organization; increased automation Integrating smart grid deployments across the organization, realizing measurably improved performance Investing based on clear strategy, implementing first projects to enable smart grid (may be compartmentalized) Taking the first steps, exploring options, conducting experiments, developing smart grid vision Default level (status quo)

SGMM Sample Compass Assessment Results

Point Range

Meaning

≥ 0.70

Green reflects level compliance within the domain

≥ 0.40 and < 0.70

Yellow reflects significant progress

< 0.40

Red reflects initial progress

=0

Grey reflects has not started

Conclusions -1 Maturity models can help an organization develop a path to improvement Maturity models focused on process institutionalization can help an organization measure the degree to which they inculcate something— this is the ―way we do things around here‖ Increasing institutionalization stabilizes processes so that they – Produce consistent and repeatable results over time – Are retained during times of stress – Can be instrumented for performance and effectiveness measures – Can be continuously improved

CERT-RMM promotes institutionalization of operational risk management practices toward the goal of managing operational resilience

Conclusions -2 CERT-RMM promotes convergence of operational risk management activities like security, business continuity, and IT ops management to take advantage of synergies and efficiencies SGMM is focused on providing a transformative path for organizations looking to improve the ―smart gridness‖ of their power grids

SGMM is not focused on process maturity—the dimension measured in SGMM is the sophistication and maturity of practices and technologies Both models are focused on improving operational capacity—whether managing system assets in operations or improving the operational sophistication and capacity of a power grid

Questions?