Automated Model-Based Android GUI Testing using

ASE 2016

Automated Model-Based Android GUI Testing using Multi-Level GUI Comparison Criteria Young-Min Baek, Doo-Hwan Bae Korea Advanced Institute of Science and Technology (KAIST) Daejeon, Republic of Korea

{ymbaek, bae}@se.kaist.ac.kr

Agenda

1. 2. 3. 4.

Introduction Background Overall Approach Methodology A. Multi-level GUI Comparison Criteria B. Testing Framework

5. 6. 7. 8.

Experiments Results Conclusion Discussion

Automated Model-based Android GUI Testing using Multi-level GUI Comparison Criteria

2

Introduction

Automated Model-based Android GUI Testing using Multi-level GUI Comparison Criteria

3

Introduction

Background

Overall Approach

Methodology

Experiments

Results

Conclusion

Discussion

How Can We Model an Android App? 

State-based models for Model-Based GUI Testing (MBGT) *,**  Model AUT’s GUI states and transitions between the states  Generate test inputs, which consist of sequences of events, considering stateful GUI states of the model e7

e6

s1

Wonderful App

Welcome!

abstraction

Let’s start our app!

e10

e5 e9

s3 application under test (AUT)

s2

s1

s2

e1 e2

s0 e3 e4

Start!

e7

e11

e8

s4

test input generation

s0

e10

e3

s3

GUI model

test input

5 states, 11 transitions

e3  e10  e7

* D. Amalfitano, A. R. Fasolino, P. Tramontana, B. D. Ta, and A. M. Memon, “MobiGUITAR: Automated Model-Based Testing of Mobile Apps,” IEEE Software, vol. 32, issue 5, pp 53-59, 2015. ** T. Azim and I. Neamtiu, “Targeted and Depth-first Exploration for Systematic Testing of Android Apps,” OOPSLA 2013.

4

Introduction

Background

Overall Approach

Methodology

Experiments

Results

Conclusion

Discussion

Define GUI States of a GUI Model 

A GUI Comparison Criterion (GUICC)  A GUICC distinguishes the equivalence/difference between GUI states to update the model. 1) Equivalent GUI state discarded

GUI Comparison Criterion (GUICC) e.g., Activity name, enabled widgets

Wonderful App

Wonderful App

compare

Unfortunately, Wonderful App has stopped.

GUI state 1

modeling

2) Different GUI state

e1

OK

Welcome!

Welcome!

Let’s start our app!

Let’s start our app!

Start!

=

e2

transition event

Start!

... en

GUI state 1

GUI state 2 5

Introduction

Background

Overall Approach

Methodology

Experiments

Results

Conclusion

Discussion

Influence of GUICC on Automated MBGT 

GUICC determines the effectiveness of MBGT techniques. Weak GUICC for Android

Strong GUICC for Android

Activity name

Observable graphical change

MainActivity

MainActivity

MainActivity

* P. Aho, M. Suarez, A. Memon, and T. Kanstrén, “Making GUI Testing Practical: Bridging the Gaps,” Information Technology – New Generations (ITNG), 2015.

6

Introduction

Background

Overall Approach

Methodology

Experiments

Results

Conclusion

Discussion

Influence of GUICC on Automated MBGT 

GUICC determines the effectiveness of MBGT techniques. Weak GUICC for Android

Strong GUICC for Android

Activity name

Observable graphical change

discarded

discarded

Infinite number of GUI states state 1 (MainActivity)

state 1

state 0

state 2

MBGT techniques should carefully consider the GUICC for model generation * P. Aho, M. Suarez, A. Memon, and T. Kanstrén, “Making GUI Testing Practical: Bridging the Gaps,” Information Technology – New Generations (ITNG), 2015.

7

Introduction

Background

Overall Approach

Methodology

Experiments

Results

Conclusion

Discussion

GUI Comparison Criteria (GUICC) for Android 

Used GUICC by existing MBGT tools for Android apps

1st Author

Venue/Year

Tool

Type of model

GUICC

M. L. Vasquez L. Ravindranath E. Nijkamp

MSR/2015 MobiSys/2014 Github/2014

MonkeyLab VanarSena SuperMonkey

statistical language model OCR-based EFG state-based graph

S. Hao

MobiSys/2014

PUMA

state-based graph

D. Amalfitano

ASE/2012

AndroidRipper

GUI tree

T. Azim

OOPSLA/2013

A3E

Activity positions of texts Activity cosine-similarity with a threshold composition of conditions Activity

Activity transition graphs extended deterministic 1. They have not clearly explained why those GUICC were selected. W. Choi OOPSLA/2013 SwiftHand labeled transition systems (ELTS) 2. They utilize only a fixed single GUICC for modelfinite generation, IEEE Software/ state machine D.no Amalfitano MobiGUITAR matter how AUTs behave. 2014 (state-based directed graph)

enabled GUI elements widget values (id, properties) values of class attributes observable state (structural)

P. Tonella

ICSE/2014

Ngram-MBT

statistical language model

W. Yang

FASE/2013

ORBIT

finite state machine

C. S. Jensen

ISSTA/2013

Collider

finite state machine

set of event handlers

R. Mahmood

FSE/2014

EvoDroid

interface model/ call graph model

composition of widgets

8

Introduction

Background

Overall Approach

Methodology

Experiments

Results

Conclusion

Discussion

Goal of This Research 

Motivation  Dynamic and volatile behaviors of Android GUIs make accurate GUI modeling more challenging.  However, existing MBGT techniques for Android: are focusing on improving exploration strategies and test input generation, while GUICC are regarded as unimportant. have defined their own GUI comparison criteria for model generation



Goal  To conduct empirical experiments to identify the influence of GUICC on the effectiveness of automated model-based GUI testing.

9

Background

Automated Model-based Android GUI Testing using Multi-level GUI Comparison Criteria

10

Introduction

Background

Overall Approach

Methodology

Experiments

Results

Conclusion

Discussion

GUI Graph 

A GUI graph G is defined as G = (S, E)  S is a set of ScreenNodes (S = {s1, s2, ..., sn}), n = # of nodes.  E is a set of EventEdges (E = {e1, e2, ..., em}), m = # of edges.  A GUI Comparison Criterion (GUICC) represents a specific type of GUI information to distinguish GUI states. (e.g., Activity name)

EventEdge e1

ScreenNode s1

ScreenNode s2

GUICC 11

Introduction

Background

Overall Approach

Methodology

Experiments

Results

Conclusion

Discussion

GUI Graph 

An example GUI graph

distinguished by GUICC

EventEdge 2

5

ScreenNode 3

7

1 10 8 9 4

6 s Normal state s Terminated state s Error state

* The generated GUI models are visualized by GraphStream-Project, http://graphstream-project.org/

12

Methodology • Overall Approach • Multi-level GUI Comparison Criteria • Automated GUI Testing Framework

Automated Model-based Android GUI Testing using Multi-level GUI Comparison Criteria

13

Introduction

Background

Overall Approach

Methodology

Experiments

Results

Conclusion

Discussion

Our Approach 

Automated model-based Android GUI testing using Multi-level GUI Comparison Criteria (Multi-level GUICC) Multi-level GUICC

definition

Systematic investigation with real-world apps

multiple GUICC

GUI model generator model management

GUI model

GUI analysis

• feature extraction • event inference

test results

Test input generator exploration strategies

test inputs

event sequence generation

Test executor test execution log collection GUI extraction

test results Multi-level GUI graphs

Compare test effectiveness:

GUI modeling, code coverage, error detection

14

Introduction

Background

Overall Approach

Methodology

Experiments

Results

Conclusion

Discussion

Design Multi-level GUICC 

Overview of the investigation on the behaviors of commercial Android applications  The multi-level GUI comparison technique was designed based on a semi-automated investigation with 93 real-world Android commercial apps registered in Google Play.

Google Play

UIAutomator Dumpsys

240 apps

Manual exploration & GUI extraction

Target app selection

Selected 93 apps

GUI information classification

Define multi-level GUICC

Multi-level GUI Comparison Model 15

Introduction

Background

Overall Approach

Methodology

Experiments

Results

Conclusion

Discussion

Investigation on GUIs of Android Apps 

Step 1. Target app selection  Collect the 20 most popular apps in 12 categories (Total 240 apps) Exclude category because they are usually not native apps Exclude apps that were downloaded fewer than 10,000 times

 Finally select 93 target apps Category

# of apps

Category

# of apps

Book

13

Shopping

9

Business

12

Social

7

Communication

10

Transportation

13

Medical

9

Weather

10

Music

10 Selected 93 apps for the investigation

16

Introduction

Background

Overall Approach

Methodology

Experiments

Results

Conclusion

Discussion

Investigation on GUIs of Android Apps 

Step 2. Manual exploration  Manually visit 5-10 main screens of the apps in an end user’s view  Examine the constituent widgets in GUIs of the main screens Use UIAutomator tool to analyze the GUI components of the screens Extract system information via dumpsys system diagnostics & Logcat

Go back Select a button Add

Delete

Extract GUI structure

Parse

UIAutomator GUI dump XML

GUI hierarchy, widget components, properties, values

Parse dumpsys Text description

visited screen

Logcat

Extract system information

system states, logs activity, package info system logs

17

Introduction

Background

Overall Approach

Methodology

Experiments

Results

Conclusion

Discussion

Investigation on GUIs of Android Apps 

Step 2. Manual exploration  Manually visit 5-10 main screens of the apps in an end user’s view  Examine the constituent widgets in GUIs of the main screens Use UIAutomator tool to analyze the GUI components of the screens Extract system information via dumpsys system diagnostics & Logcat UIAutomator dump

Dumpsys

adb shell /system/bin/uiautomator/ dump /data/local/tmp/uidump.xml LinearLayout RelativeLayout LinearLayout Button

Button

Button

adb shell dumpsys window windows > /data/local/tmp/windowdump.txt mDisplayId

LinearLayout EditText

ImageView

mAnimLayer mConfiguration mFocusedApp mCurrentFocus

18

Introduction

Background

Overall Approach

Methodology

Experiments

Results

Conclusion

Discussion

Investigation on GUIs of Android Apps 

Step 3. Classification of GUI information  Find hierarchical relationships from extracted GUI information  Filter out redundant GUI information that highly depends on the device or the execution environment (e.g., coordinates)  Merge some GUI information into a single property Package Activities

Android package

combine

AppName, AppVersion, CachedInfo

combine

Launchable, CurrentlyFocused Activity

combine

Coordination, Orientation, TouchMode, WidgetClass, ResourceId

1

*

Android Activity 1

Widgets

*

Layout widgets 1

*

Control widgets 1

Values

combine

Coordination, WidgetClass, ResourceId {Long-clickable, Clickable, Scrollable, ...}

*

Widget properties-values

19

Introduction

Background

Overall Approach

Methodology

Experiments

Results

Conclusion

Discussion

Investigation on GUIs of Android Apps 

Step 4. Definition of GUICC model  Design a multi-level GUI comparison model that contains hierarchical relationships among GUI information Define 5 comparison levels (C-Lv) Define 3 types of outputs according to the comparison result T: Terminated state, S: Same state, N: New state

A GUI comparison model using multi-level GUICC for Android apps

20

Introduction

Background

Overall Approach

Methodology

Experiments

Results

Conclusion

Discussion

Multi-level GUICC 

Compare GUI states based on a chosen C-Lv  As C-Lv increases from C-Lv1 to C-Lv5, more concrete (finegrained) GUI information is used to distinguish GUI states.

C-Lv1

C-Lv2

Compare package

Compare activity

C-Lv3

C-Lv4

C-Lv5

Compare layout widgets

Compare executable widgets

Compare text contents & items

21

Introduction

Background

Overall Approach

Methodology

Experiments

Results

Conclusion

Discussion

Multi-level GUICC 

Compare GUI states based on a chosen C-Lv  As C-Lv increases from C-Lv1 to C-Lv5, more concrete (finegrained) GUI information is used to distinguish GUI states. Max C-Lv C-Lv1

C-Lv2

Compare package

Compare activity

C-Lv3

C-Lv4

C-Lv5

Compare layout widgets

Compare executable widgets

Compare text contents & items

can distinguish Calculator

cannot distinguish

Calculator

3