Test Automation with TTCN-3 - ictss-sbmf-sast

Competence Center MOTION

Fraunhofer FOKUS

Test Automation with TTCN-3: State of the Art and a Future Perspective Ina Schieferdecker ICTSS, Natal, Brazil, Nov. 9, 2010

ICTSS 2010, Test Automation with TTCN-3, Slide 1


Fraunhofer FOKUS

My TTCN-3 Background  Standardization at ETSI since 1999  Tool development at FOKUS with TU Berlin, then Testing Technologies  Test suite development and testing with numerous industrial partners  TTCN-3 syllabus and certificate with GTB, iSQI, and ETSI  TTCN-3 beyond in research: – test suite quality: coverage, metrics, guidelines, refactorings – testing of embedded systems: real-time, continuous – performance testing and benchmarking ICTSS 2010, Test Automation with TTCN-3, Slide 4


Fraunhofer FOKUS

It’s not my first keynote …

… and indeed the second on TTCN-3 … but first outside the TTCN-3 User Conference – … – Anthony Wiles: TTCN-3: Past and Future, 2007. – Andreas Ulrich: Leveraging TTCN-3 in SW development projects, 2007. – …

… I will add thoughts on uptakes and perspectives

Disclaimer: What will be said is my personal view only – and in particular not the view of the ETSI maintenance team or of anybody else mentioned beforehand. ICTSS 2010, Test Automation with TTCN-3, Slide 5


Who are you ?

1. Are you a tester ?

2. Do you work on test automation ?

3. Have you heart about TTCN-3 ?

4. Do you used/use TTCN-3 ?


Fraunhofer FOKUS


Outline

 Introduction

 TTCN-3 Status + New Developments

 TTCN-3 Perspective

 Outlook


Fraunhofer FOKUS


Outline

 Introduction



 Summary


Fraunhofer FOKUS


Fraunhofer FOKUS

What is TTCN-3 – see www.ttcn-3.org  Testing and Test Control Notation Version 3  Internationally standardized testing language  Product of the ETSI (European Telecommunication Standards Institute) Technical Committee MTS (Methods for Testing and Specification)  A programming language that has been used for more than 15 years in standardization as well as industry  Specifically designed for testing and certification  Constantly developed and maintained at ETSI by a team of leading testing experts from industry, research institutes, and academia  A testing technology that applies to a variety of application domains and types of testing  Knowledge of TTCN-3 is valuable both for employees as well as employers due to its wide applicability  Offers potential for reducing training and test maintenance costs significantly  Proven to work in very large and complex industrial tests, e.g., of 3G network elements ICTSS 2010, Test Automation with TTCN-3, Slide 9


Fraunhofer FOKUS

What is TTCN-3 – see www.ttcn-3.org  Testing and Test Control Notation Version 3  Internationally standardized testing language  Product of the ETSI (European Telecommunication Standards Institute) Technical Committee MTS (Methods for Testing and Specification)  A programming language that has been used for more than 15 years in standardization as well as industry  Specifically designed for testing and certification  Constantly developed and maintained at ETSI by a team of leading testing experts from industry, research institutes, and academia  A testing technology that applies to a variety of application domains and types of testing  Knowledge of TTCN-3 is valuable both for employees as well as employers due to its wide applicability  Offers potential for reducing training and test maintenance costs significantly  Proven to work in very large and complex industrial tests, e.g., of 3G network elements ICTSS 2010, Test Automation with TTCN-3, Slide 10


Fraunhofer FOKUS

TTCN-3 in a Nutshell (1)  High-level, technologyindependent language for abstract test specifications  data templates allow structuring and reusability of test data  matching mechanism to compare expected and received data  message-, procedure-based and real-time ports  parallel test components ICTSS 2010, Test Automation with TTCN-3, Slide 11

11


Fraunhofer FOKUS

TTCN-3 in a Nutshell (1)  High-level, technologyindependent language for abstract test specifications  data templates allow structuring and reusability of test data  matching mechanism to compare expected and received data  message-, procedure-based and real-time ports  parallel test components ICTSS 2010, Test Automation with TTCN-3, Slide 12

12


Fraunhofer FOKUS

TTCN-3 in a Nutshell (2) type definitions:  boolean, integer, float, bitstring, charstring, octectstring, hexstring  record, set, enumeration, union programming constucts:  message: send/receive  procedure: call/getcall, reply/getreply, raise/catch  if-then-else, loops: for, while, do-while  functions, alternatives  component/port/timer control Predefined functions:  type conversion, lengthof (string), sizeof (records), … Overview: e.g. TTCN-3 Quick Reference Card ICTSS 2010, Test Automation with TTCN-3, Slide 13


TTCN-3 in a Nutshell (3)  Test technology for local/distributed testing, for static/dynamic testing, for interface-/API-/protocoloriented testing  Concrete test implementation: adapter and codecs  Execution environments based on TRI, TCI


Fraunhofer FOKUS


Fraunhofer FOKUS

Some history Dave Rayner Ostap Monkevich

Anthony Wiles Dieter Hogrefe Jens Grabowski Ina Schieferdecker

TTCN-1

TTCN-2/-2++

ISO/IEC 9646 1990

ISO/IEC 9646-3 1998

ITU-T X.290 1992

ETSI TR 101 666 1999


Proposal for New Version 1998

Anthony Wiles Colin Willcock Jens Grabowski Ekkart Rudolph Paul Baker Ina Schieferdecker Gyorgy Rethy …

TTCN-3 ETSI ES 201 873 2000 ITU-T Z.140 2001

...


Fraunhofer FOKUS

What is Test Automation  ”The management and performance of test activities to include the development and execution of test scripts so as to verify test requirements, using an automated test tool”. – Dustin, Rashka & Paul  ”Testing supported by software tool.” – Faught, Bach  ”Testing is automation, automation, automation.” – Harman testexecution execution  Automated Automated test  Test cases to test code/scripts  Test platforms  Automated test case generation  Models to test cases  Model-based testing  Automated test framework generation  Metamodels to test models ? ICTSS 2010, Test Automation with TTCN-3, Slide 16


Fraunhofer FOKUS

Areas of Test Execution Automation  Automated tests mainly on testing that requires repeated effort of similar tests cases – – – – – –

Regression testing Portability testing Performance and stress testing Configuration testing Smoke testing ...

 Automated tests needed where manual tests impossible – – – – – – –

Fast tests Precise tests Distributed tests Coordinated tests Remote tests Tests via heterogeneous interfaces ...



Fraunhofer FOKUS

Views on Test Execution Automation  Some believe that test execution automation is so expensive relative to its value that it should be used sparingly.  Others, such as advocates of agile development, recommend automating 100% of all tests.  Challenges with test execution automation are that it requires – Automated test oracles – Automated test validation – Automated test evolution – … ICTSS 2010, Test Automation with TTCN-3, Slide 18


Fraunhofer FOKUS

Some points to make  Test execution automation is not only about scripting.  It is about – – – – –

Test design and modelling Test quality and validation Test execution for various target systems, on various test devices, etc. Test reuse …

 It is also needs to be integrated with – – – –

Test process management Defect tracking Configuration management …



Prospects of Test Execution Automation        

More tests are run per build Tests can be more complex Tests run much faster Tests are much less error-prone to operator error More effective and efficient use of testers Better co-operation with developers Improved system reliability Improved quality of test


Fraunhofer FOKUS


Fraunhofer FOKUS

Pitfalls of Test Execution Automation

 You can go from test planning to test execution with no manual intervention.  One tool can support all test requirements.  There will be an immediate reduction in test effort and test schedule.  Test automation tools are easy to use.  Test automation compensates for bad requirements or bad design.

 …



TTCN-3 based test execution automation

Domain-specific solutions TTCN-3 test suites Domain-specific adaptations TTCN-3 test frameworks Generic test automation platform TTCN-3


Fraunhofer FOKUS


Fraunhofer FOKUS

Challenges in Test Execution Automation  Suitability of the test automation framework for iterative development  Evolving tests (incl. test requirements designs, models and code)  Adaptability of the test framework  Flexibility in the test execution environment  Re-usability of the test framework across different projects  Test reuse  Efficient test behavior and test data management  First-order test behavior and test data in test specifications


? (?) ()




Fraunhofer FOKUS

Adaptability of the test framework – a TTCN-3 view: from … TCI

Test Logging

Test Management

TRI

Control

Behavior

Components

Types, Data

Ports

Timer

System Adaptor System Adaptor System Adaptor

Platform Adaptor Platform Adaptor Platform Adaptor

System Under Test (SUT)

 Current TRI/TCI: – Telecommunication oriented: bitstrings – Imply limitations ICTSS 2010, Test Automation with TTCN-3, Slide 24

Codecs Codecs Codecs

Component Handling

Selected Adapters: CAN, Ethernet, FTP, GPIB, HTTP, IDL, IP, LTE RRC/S1/X2, MOST, Native C Release Management, RS232, SCP, SQL, SSH, TCP, Telnet , TLS, UDP, SCTP, USB

I. Schieferdecker, T. Vassiliou-Gioles: Tool Supported Test Frameworks in TTCN-3. – 8th Intern. Workshop in Formal Methods in Industrial Critical Systems, Røros, Norway, June 2003, ENTCS (80), Elsevier Science.


Fraunhofer FOKUS

Adaptability of the test framework – a TTCN-3 view: … to

TCI

Test Management

Test Logging

Component Handling

 Adaptor

TRI

Control

Behavior

Components

Types, Data

Ports

Timer

interfaces dedicated to the domain

Protocol adaptation Embedded systems adaptation Software adaptation System Under Test (SUT)


requirements


Reuse of test artefacts

Fraunhofer FOKUS

 Vertical: reuse between testing levels or types – e.g., component and integration testing, functional and performance testing

 Horizontal: reuse between products in the same domain or family – e.g., standardized test suites, tests for product families, or tests for product lines;

TTCN-3 embedded

 Historical: reuse between product generations – e.g., regression testing

www.temea.org


Karinsalo, M. and P. Abrahamsson (2004). Software reuse and the test development process: A combined approach. In ICSR, Volume 3107 of Lecture Notes in Computer Science, pp. 59–68. Springer.

26


Outline

 Introduction + History



 Summary


Fraunhofer FOKUS


Fraunhofer FOKUS

Commercial TTCN-3 tools: www.ttcn-3.org TTCN-3 Compilers and Interpreters  Exhaustif/TTCN: compiler (C++) produced by Métodos y Tecnología (MTP), Spain.  OpenTTCN: interpreter (C, Java, C# interfaces) produced by OpenTTCN Ltd, Finland.  MessageMagic: compiler (C/C++, Java, C#) produced by ELVIOR, Estonia.  Real Time Developer Studio: modelling tool including TTCN-3 compiler by PragmaDev, France.  TAU Tester: compiler (C) by IBM.  TTCN-3 toolbox: compiler (C) by Danet Group, Germany.  TTCN-3 Express: compiler (C#) by Fraunhofer FIRST and Metarga GmbH, Germany.  TTworkbench: compiler (C, Java) by Testing Technologies, Germany. TTCN-3 Generators  Qtronic by Conformiq OY, Finland. generate complete TTCN-3 test suites from e.g., UML, Java, or C# models.  MaTeLo by All4Tec, France (TTCN-3 test suites from usage models specified using Markov chains).  MOTES by ELVIOR, Estonia (from the state model of the SUT)



Fraunhofer FOKUS

Open source TTCN-3 tools: www.ttcn-3.org   

  

LoongTesting testing platform including TTCN-3 compiler and integrated development environment by Information Processing Center of USTC, China. BBT TTCN-3 Compiler, by BroadBit, Hungary. TRex: by University of Göttingen to provide IDE functionality for TTCN-3 core notation, and to support assessment and automatic restructuring of TTCN-3 test suites. (opensource Eclipse plug-in). T3doc by Federico Engler and further developed by ETSI. for generating HTML documentation via tagged TTCN-3 comments. Codec generator by IRISA as part of T3DevKit. It automatically generates a codec based on TTCN-3 type module(s), C++ codec functions. T3DevLib by IRISA as part of T3DevKit. It allows the development or integration of Codec, SUT and Platform Adapter implementations written in C++.

... and more academic prototype/research tools (guideline checking, quality analysis, ...) ICTSS 2010, Test Automation with TTCN-3, Slide 29


Fraunhofer FOKUS

Selected test devices with TTCN-3 support – www.testingtech.com

 DCT2000© (Catapult Communications Corporation)  E6651A Mobile WiMAX Test Set (Agilent Technologies)  G35 Protocol Analyzer (Tektronix)  MiNT T2230/1 AIME - 802.16e (Aeroflex)  Nexus8610 Traffic Simulation System (Nexus Telecom)  MobileRobot (ServiceForce.Com GmbH)  Testerlyzer (Ruetz Systems Solutions GmbH)  …



Fraunhofer FOKUS

A seamless chain of TTCN-3 tools

Developers Perspective for Modification

Result Analyzer

Test Execution

Test Campaign Designer (Test Automation)

Test Parametrization


Test Report

Online Logging, Filter, Reporting


A Selection of TTCN-3 Users  Telecommunication Vendors

 Service Provider

 Test Device Vendors  Automotive and Avionics  Hardware Vendors  System Integrators  Standardisation


Fraunhofer FOKUS


Fraunhofer FOKUS

Telecom example: Performance Benchmarking Sample SUT: IMS components Test IMS components separately and in different configurations

Definitions using TTCN-3: • standard call scenarios (e.g. voice call, conference call, application call, push-to-talk call) • traffic set as an aggregation of call scenarios

I-CSCF

HSS

P-CSCF

...


...

Performance: • 5.000 - 10.000 IMS subscribers (per server) • Up to 250 requests per second (per server)

SuT reference Implementation

S-CSCF

control and coordination UE emulations

Benchmark Test Platform

UE emulations


Fraunhofer FOKUS

TTCN-3 domains: Automotive  Car communication systems – Daimler, Volkswagen, SiemensVDO – edutainment bus system

 Car-to-car communication

Telematics Applications in the Cockpit • Audio (CD / Radio), Video • Telephone, SMS • Navigation • Speech recognition • User interface for body electronic

– SIM-TD field experiments

 Standardization groups: CD Changer

– AUTOSAR consortium – MOST cooperation

Head Unit

Test er

Amplifier / Tuner

MOST Bus

Speaker ICTSS 2010, Test Automation with TTCN-3, Slide 34


TTCN-3 domains: further examples  Medicine – SiemensMED (image processing) – HL7 eHealth protocols (Interoperability)

 Power transmission and distribution: – SiemensPTD (safe and reliable energy system)

 Financial data warehouse: – International bank (functional / regression testing)

 Avionics – European Space Agency

 Railways – Dutch railways


Fraunhofer FOKUS


Google analytics for www.ttcn-3.org  Statistics since June 2008


Fraunhofer FOKUS


TTCN-3 today  A successful testing technology – Used in telecommunication, software industry, automotive, etc.  A textual and graphical test scripting language – Human readable  A test implementation language – Automated test execution is built-in  A test realization framework – A variety of ready-to-use tools and test assets provided by an agile community  A philosophy – Specifically made for testers


Fraunhofer FOKUS


Fraunhofer FOKUS

Challenges forTTCN-3 based Test Execution Automation

1. The home-grown test-tool syndrom

2. The home-grown test-standard mafia

3. The technology ignorants

2010



Fraunhofer FOKUS

Mastering the power  Establishment of a TTCN-3 syllabus and certificate – Cooperation of ETSI, GTB and iSQI – accredited TTCN-3 training providers – several certificates issued – layered approach proposed  Improvement of TTCN-3 tool interoperability – Development of a TTCN-3 reference test suite – 1st TTCN-3 Plugtest event with commercial tools at ETSI (2009)  TTCN-3 user conferences – Europe: since 2004 (F, D, E, S), next: 2011 (SLO) – Asia: 2007 (China), 2009 (India), 2010 (China) ICTSS 2010, Test Automation with TTCN-3, Slide 39

39


Fraunhofer FOKUS

Major New Features in TTCN-3 v3  Test components – named test components – alive test components – test component inheritance

 Communication – multicast and broadcast

 Types – implicit (unnamed) types within structured types – regular expressions for subtyping ICTSS 2010, Test Automation with TTCN-3, Slide 40

 Templates –templates in to and from fields –dynamic generation of templates (local templates, template returning functions and template variables)

 Logging – extended log statement – logging interface


Fraunhofer FOKUS

More flexible handling of test data: Dynamic template handling  Allows to declare and use – template variables – local templates – template-type return “values” (for functions) template MyMessageType t_MyMessage (MyType myPar) := { …} function MyFunction (MyType MyPar1) runs on MyCompType return template MyType2 { var MyType2 vl_MyType2; var template MyMessageType vl_MyMessage := t_MyMessage (MyPar1); template MyMessageType tl_MyMessage (MyType myPar) := { …} vl_MyMessage.myField := (1,2,3,4,5); vl_MyMessage.someField.anotherField := ?; P.receive (vl_MyMessage); if (cond) { return ? } else { return (1,2,3,4,5) } } ICTSS 2010, Test Automation with TTCN-3, Slide 41


Fraunhofer FOKUS

Major New Features in TTCN-3 v4  TTCN-3 language – Import of import statements

 Extension packages – Advanced Parameterization – Behaviour Types – Configuration and Deployment

 TRI/TCI – C# mapping

Support – Real-Time and Performance Testing Support



Fraunhofer FOKUS

Extension packages #1: Advanced Parameterization  Formal parameters of kind “type” for: type, template, and behaviour definitions  Examples: type record Data { Header hdr, p_PayloadType payload} function f_myfunction (in MyList p_list, in p_myType p_elem) return p_myType {return p_list[0] + p_elem)} f_myfunction ({1,2,3,4}, 5)



Fraunhofer FOKUS

Extension packages #2: Behaviour types

 Variables may carry “behaviour” (functions/altsteps etc.)  Example: type function MyFuncType ( in integer p1 ); function f_myFunc1 ( in integer p1 ) {…}; … var MyFuncType v_func; v_func := f_myFunc1; apply (v_func(0));


// formal type definition (w/o body) // concrete behaviour


Outline

 Introduction



 Summary


Fraunhofer FOKUS


TTCN-3 Perspective  TTCN-3 as target for model-based testing  TTCN-3 for embedded systems  Future TTCN-3


Fraunhofer FOKUS


Fraunhofer FOKUS

TTCN-3 in MBT

 TTCN-3 is used in several domains as binding link between modelling and execution  Commercial tools like Qtronic (Conformiq) and TTmodeler (TestingTech) do generate TTCN-3 code for test execution  Lots of academic prototype tools

 Selection of industrial case studies: see e.g. European D-MINT project www.d-mint.org


49


Fraunhofer FOKUS

MBT example: Trimek/Datapixel production engineering case study • SUT: Coordinates Measuring Machines (CMM) control software (CDMS) for controlling a measuring system • Focus: test case derivation from UML models • Models in use: UML class, sequence, state diagrams

SUT



Fraunhofer FOKUS

MBT example 2: City street lights case study • SUT: Eliko street lighting control system feeder box control unit (FBCU) • Models for the SUT: UML state charts, produced with tool Poseidon • Elvior test generator derives TTCN-3 test cases from state charts SUT



Fraunhofer FOKUS

TTCN-3 embedded  Extensions for testing – real time systems (RT-TTCN-3) – continuous systems (Continuous TTCN-3)  RT TTCN-3 Concepts – clock as a common basis for time measurement. – timestamp redirection for exact time measurement of message interaction.

 Continuous TTCN-3 Concepts – sampled clock as a common basis for discretization and stream definitions. – sampled streams that provide a data structure to define, access and manipulate discretized signal values and their history in time. – hybrid automata that provides a control flow structure to enable and control the simultaneous stimulation and evaluation of stream ports. ICTSS 2010, Test Automation with TTCN-3, Slide 52


Fraunhofer FOKUS

TTCN-3 embedded: Hybrid Behavior Specification

cont{ var integer x :=0; onentry{Brake.value := 0.0} inv{ Brake.value = 10.0)

Cont. Test Behavior

[g1]  Modes represent macro states of the test system  Repetitive execution of the mode’s content (micro states) – onentry and onexit are executed once – inv, until, assertions and assignments are executed continuously

 Parallel and sequential composition (using par and seq)



Fraunhofer FOKUS

TTCN-3 embedded sample

 Test execution platforms  Vector Canoe  Matlab Simulink  DSpace real time environment



Fraunhofer FOKUS

My panel slides from 2000 at TestCom, Ottawa:

Too early that time And still today: TTCN-3 is a brand, major updates are new editions ICTSS 2010, Test Automation with TTCN-3, Slide 55


Outline

 Introduction



 Summary


Fraunhofer FOKUS


Fraunhofer FOKUS

Conclusions

 1980s – today: long history and value  Wide range of applicability – different communication paradigms and testing types – various domains – used in industry and research

 Created by an international community, wide-spread expertise, commercial and open source tools  TTCN-3 is mature, well-established and in progress for the future



TTCN-3 as a testing middleware


Fraunhofer FOKUS


TTCN-3 Sources

 Online information  www.ttcn-3.org  TTCN-3 User Conference  2011 in Bled  TTCN-3 Standards, Papers, Book  http://www.ttcn.de/  Quick Reference  http://www.blukaktus.com/  Exercises and Tooling  research licenses ICTSS 2010, Test Automation with TTCN-3, Slide 59

Fraunhofer FOKUS


Fraunhofer FOKUS

The test specification and test implementation language



Fraunhofer FOKUS

Please be all invited to come to Berlin …



Thank you for your attention ! Further Questions ?


Fraunhofer FOKUS


Fraunhofer FOKUS

Contact Prof. Dr.-Ing. Ina Schieferdecker

FOKUS Fraunhofer Institute for Open

Phone: +49 30 34 63 7241 Mobile: +49 175 260 30 21

Communication Systems FOKUS Kaiserin-Augusta-Allee 31 10589 Berlin, Germany

Email:

ina.schieferdecker@ fokus.fraunhofer.de Tel:

+49 (30) 34 63 – 7000

Fax:

+49 (30) 34 63 – 8000

Web: www.fokus.fraunhofer.de


Test Automation with TTCN-3 - ictss-sbmf-sast

Test Automation with TTCN-3 - ictss-sbmf-sast

Suggest Documents

Test Automation Framework - ADFAHRER

Pen Test Automation - WordPress.com

Test Automation Snake Oil

Test Automation in Practice - swerl

Technical Debt in Test Automation

FREE Automation Test Assessment - Bitpipe

Exploratory Test Automation - Cem Kaner

Intelligent Test Automation - Google Sites

360Â° Test Automation fact sheet

Test Automation in Practice - swerl

Read Test Automation using Selenium WebDriver with ... - Google Sites

ePub Test Automation using Selenium WebDriver with ... - Google Sites

eBook Download Test Automation using Selenium WebDriver with Java

[PDF] Download Test Automation using Selenium WebDriver with Java

Test Automation using Selenium WebDriver with Java - Google Sites

[PDF] Download Test Automation using Selenium WebDriver with Java

[PDF] Download Test Automation using Selenium WebDriver with Java

PDF Test Automation using Selenium WebDriver with ... - Google Sites

[PDF]Read Test Automation using Selenium WebDriver with Java ...

Making Test Automation Work in Agile Projects - Agile Testing with ...

Experiments with High Volume Test Automation - Cem Kaner

[PDF]Download Test Automation using Selenium WebDriver with Java ...

Test Automation with TTCN-3 - ictss-sbmf-sast

[PDF] Download Test Automation using Selenium WebDriver with Java