Is the DSML small enough, leaving out language features that do not contribute to the purpose of the language? -> Identify important quality goals and decide ...
Evaluating Domain-Specific Modelling Solutions Parastoo Mohagheghi1, Øystein Haugen2 1,2 SINTEF,
Norway 1 Norwegian University of Science and Technology 2 University of Oslo
DE@ER’10 ICT
1
A few words about us and our area of interests SINTEF is the biggest research organization in Scandinavia and the 4th largest in Europe: 2000 employees, around 600 in Oslo Research in multiple areas SINTEF ICT with 260 employees
Model-driven engineering is the focus of our group: Developing languages Developing tools Developing methodologies Application in different areas: SOA, embedded software, etc.
EU research projects and industry projects
ICT
2
“Model” in this context A description or representation of a software system or its environment for a certain purpose, developed using a modelling language and thus conforming to a metamodel. A model may consist of several diagrams where each diagram type gives a different view on the described system. For example UML 2.0 has 13 diagram types such as use case
diagram, class diagram etc.
It is common to model a software system at different abstraction levels. For each purpose of modelling, a suitable language is important. ICT
3
What is a Domain-Specific Language (DSL)? A Domain-Specific Language (DSL) is typically a small, highly focused language used to solve some clearly identifiable problems in a domain. The contrast is to a General-Purpose Language (GPL) that is
supposed to be useful for multiple domains.
DSLs may be graphical or textual, fall under various categories of languages (OO, event-oriented, etc). Examples are SQL for data manipulation, DOORS for requirement management etc. (horizontal) or languages for vertical domains such as health care.
ICT
4
Domain-Specific Modelling solutions A Domain-Specific Modelling Language uses domainspecific notations to provide intuitive descriptions of a system that are easily understood by domain experts. In our projects we have focused on Model-Driven Engineering (MDE) approaches with generating artefacts from models and performing various analysis on models. DSMLs are therefore viewed as an MDE approach which raises the abstraction level even more by using problem domain concepts.
ICT
5
Advantages and challenges of using domain-specific languages Advantages: The goal of DSLs is to increase the productivity of software
engineers by targeting specific domain needs. Close to the domain notations; thus easier to be used by domain experts. Often with precise semantics that allows automatic generation and verification.
Challenges: DSL development is hard, requiring both domain knowledge and
language development expertise. Domains evolve, thus DSLs should evolve as well. Responsibility for maintenance. It is often far from evident that a DSL might be useful or that developing one might be worthwhile ICT
6
An iterative development process for DSLs Identify need for a DSL
Requirements v1
Develop DSL
DSL v1
Try DSL (by experts)
Requirements v2
Revise DSL
DSL v2
Try DSL (by users)
Requirements v3
Revise DSL
ICT
7
The context of research Several DSMLs and editors and transformations (generally referred as DSM solutions) have been developed in the MODELPLEX project. A network modelling language in Telefonica for modelling and
generating configuration files, a security modelling UML profile in Thales, a performance modelling DSL in SAP for business performance simulation and analysis
a DSM solution for specifying signalling at railway stations and generating source code has been developed in the ITEA-MoSiS project. Generating routing tables, allocating trains to routes and testing
ICT
8
Questions to be answered Several questions to answer: Is the DSM solution easily usable by the intended domain experts? Does the DSM solution provide appropriate built-in abstractions
and notations for building applications in the specific domain? Does the DSM solution serve the purpose of the development such as generating relevant artefacts? Is the DSM solution maintainable and evolvable when the domain evolves? Is the DSML small enough, leaving out language features that do not contribute to the purpose of the language?
-> Identify important quality goals and decide how to evaluate them.
ICT
9
State of the art analysis Related work can be discussed in several dimensions: evaluating languages in general, evaluating modelling languages, evaluating domain-specific languages.
Some work have classified the required quality characteristics from different viewpoints. Others have just listed some important quality characteristics. Very little work on how to achieve a desired quality characteristics.
ICT
10
Example: Howatt classes of criteria for evaluating languages (2001) Language Design and Implementation Criteria: Is the language formally defined? Can a fast, compact compiler be
written to generate efficient, compact code?
Human Factors Criteria: These criteria are used to assess the human interface or the user-
friendliness of a language.
Software Engineering Criteria: These assess those aspects of a language that enhance the
engineering of good software; for example supporting portability, reliability and maintainability of the software.
Application Domain Criteria: These criteria assess how well a language supports programming
for specific applications. ICT
11
Lindland et al.’s quality model for conceptual models (1994) Domain semantic quality
appropriateness
appropriateness
Model
Audience
Language
syntactic quality
appropriateness
interpretation Syntactic quality -> syntactic correctness Semantic quality -> validity and completeness Pragmatic quality -> comprehension ICT
12
Other characteristics Having right data necessary constructs and their semantics. Completeness in capturing all concepts.
Accuracy of concepts to present the developed system and helping in designing it. Flexibility to model different systems and ease of change. Understandability in the ease of read and conveying the meaning of the underlying system.
Level of detail and needed training Seamlessness mapping concepts in the problem space to implementations in the solution
space
Simplicity no unnecessary complexity, including being small and memorable. ICT
13
Tool Developers (TD) debugger, library, intuitive UI
ease of implementing language features / generating compilers, unambiguous, formalism
ease of learning/understanding models and other usability concerns, increased productivity, effort needed to model, debug or generate artefacts
End Users (EU)
Domain Experts (DE)
evolution, scalability, reusability, modularity
reliability, maintainability, completeness, correctness, complexity, performance of generated artefacts
Quality Experts (QE)
DSM Solution domain / system appropriateness, consistency, orthogonality, accurateness, cost-effectiveness, specific requirements
Software Engineers (SE)
mappings, metamodels, integration, standards, extensibility, exchanging artefacts, interoperability
Evaluating a DSM solution From multiple viewpoints ICT
Other languages / tools (O) 14
Methods of evaluation Qualitative methods case studies; including comparative ones that compare using a
DSM solution with other approaches, analysis of a language and the DSM solution by experts for various characteristics, monitoring or interviewing users.
Quantitative methods collecting metrics from models or metamodels, performing surveys among users, controlled experiments on models and metamodels,
Combined: Prototyping phase; qualitative or some predictive metrics Usage phase: both; involving users ICT
15
Examples of metrics Time and effort required to model, debug, and generate artefacts Performance of the code that results from models Model metrics; mostly size Usability metrics time to learn or perform tasks, user steps to perform a task and
layout appropriateness
Number of concepts and the relations between these concepts in the DSML
ICT
16
Case study 1: network modelling tool developed in Telefónica using Eclipse GMF
ICT
17
Some requirements of the DSML solution and a first evaluation Requirements: A visual, user-friendly interface; Scalability – enabling thousands of model elements to be
managed; Interoperability with other tools and standards (CIM was used) Flexibility – enabling the rapid adaptation of the tool to support new abstractions (preferably done by the engineers themselves); Support for model validation and checking; Support for generation.
A questionnaire was developed based on the requirements that were answered by the domain experts and pilot users. ICT
18
Examples of questions Stake holder End Users
Software Engineeri ng Software Engineeri ng Other tools
Question based on the requirement
Results
Not enough, largely due to the sheer size of the metamodel which resulted in having to add a large number of connection and node tools. Is the DSM solution flexible? Not enough. A more dynamic, metamodel-driven tool generation approach is needed. Does the DSM solution Modelling at different abstraction levels provide reuse possibilities? is applied to increase reusability of elements. Is the DSML compatible with Many tools used in the network other tools? management domain are based on CIM, but as the DSML transforms the CIM metamodel into EMF, this leads to compatibility issues with CIM-based offthe-shelf products that need to be resolved. • Is the DSML tool easy to use? • Is the UI acceptable?
ICT
19
Identified challenges The large number of modelling abstractions and relationships in the CIM model We restricted to 200 elements but this is too many.
Developing a DSML in Eclipse required high language and tool expertise. Not for non-technical experts.
Changes to the metamodel which happen frequently in the domain required considerable effort in updating the tool and the developed models became corrupted due to changes.
ICT
20
Case study 2: TCL- Train Control Language; a process for evaluation 1. Identify stakeholders
tool developers, signalling engineers (end-users that will model the stations), station deployers that will generate required source code, testers who will generate test cases from the models, railway authorities the standardization organs and national authorities that define safety requirements.
2. Identify purposes for developing a DSML
using visual models that are easier to draw and maintain; automatic generation of artefacts such as interlocking source code and test cases should be automatically generated; reducing errors in models by improving the consistency and completeness of the specifications; saving effort by reducing the human effort to implement new railway stations. ICT
21
Process continued 3. Identify quality goals based on the purposes of stakeholders Quality goal #1: TCL diagrams should be similar to existing
diagrams to be accepted by domain experts and be trusted. However, TCL can be more expressive than existing diagrams. Quality goal #2: Small stations should be covered completely so that models can be used as complete signalling documentation of these. Quality goal #3: TCL and tools must prevent specifying unsafe models and reduce the need for inspections. Quality goal #4: Generated code should be identical to manually written code to facilitate evaluation of the generated code for completeness and improve perceived usefulness of the DSL.
ICT
22
Process continued 4. Identify means (practices) to achieve quality goals and evaluation methods:
Quality goal #1: TCL diagrams should be similar to existing diagrams. Practice: walking through existing examples together with Station deployers. Evaluation method: performing visual comparison of models developed with the first version of TCL with existing diagrams
ICT
23
Summarizing Stake holder
Requirement
Means (Practice)
End user
TCL and tools should prevent specifying unsafe models.
the Adding well-formedness Validate by rules to the language. Also constraints and adding constraints to the inspections running test cases. TCL specifications.
End user
Small stations should be covered completely.
Small stations are Models developed identified and their models with the first version of TCL are compared are reviewed. with existing diagrams. the Add constraints to the Validate and TCL specifications. Also, constraints the integrate the safety inspect standards in the necessary development process. steps in the development process.
Other
Models are compliant with safety standards.
Evaluation method
ICT
24
Some important characteristics of DSLs Domain-appropriateness: A DSL must be powerful enough to capture the major domain
concepts and should match the mental representation of the domain.
Formal and accurate: DSM solutions are typically used for prediction or simulation, as
well as code generation, test generation and execution.
Proper layout: For a DSL with a diagrammatical syntax
Integrating DSLs with other languages There is often a need for integrating DSLs with other ones. Find
solutions such as using standards, mapping to a formal langauge etc. ICT
25
experience, The whole picture: Quality in Modelling familiarity with domain, powerful, self-descriptive processes, languages and tools. easy to learn, Modelling languages problem appropriateness conformance to the language
Model Quality assurance
technical appropriateness
effort-saving, unambiguous
Modelers
Quality of transformations
Tools
Processes
Software
ICT
26
Conclusions State of the work: Identifying quality characteristics and some methods of evaluation Classifying them according to stakeholders’ interest A few case studies; identified some challenges in developing DSLs A framework for defining quality models based on goals, purposes,
stakeholders, means and evaluation methods
Future work: Perform more case studies: We work on several UML profiles DSLs in different projects More state of the art analysis Relate quality characteristics to evaluation methods and steps of
development (integrate development and assessment)
ICT
27
Thank you! Questions? Comments? http://quality-mde.org/ ICT
28
