Mobile Testing as a Game Customer Orientation
Lean Six Sigma
Agile Processes
Thomas M. Fehlmann Euro Project Office AG E:
[email protected] H: www.e-p-o.com
Process Project Estimations Metrics
Transfer Functions
1
ThomasFehlmann Fehlmann Dr. Dr. Thomas Customer Orientation
Lean Six Sigma
Agile Processes
Project Estimations
Transfer Functions
1981: 1991: 1999: 2001: 2003: 2004: 2007: 2011: 2012: 2013:
Dr. Math. ETHZ Six Sigma for Software Black Belt Euro Project Office AG, Zürich Akao Price 2001 for original contributions to QFD SwissICT Expert for Software Metrics, SwiSMA QFD Architect QFD Institute Deutschland CMMI for Software – Level 4 & 5 Net Promoter® Certified Associate Galorath Partner – SEER for Software IntLock eMetrics Partner & Distributor 2
Agenda Customer Orientation
Apps Testing
Lean Six Sigma
Test Strategy Agile Processes
Project Estimations
Transfer Functions
Recruiting Testers The Testing Game 3
Agenda Customer Orientation
Apps Testing
Lean Six Sigma
Agile Processes
Project Estimations
Transfer Functions
Test Strategy Apps Testing Recruiting Testers The Testing Game 4
What an Apps does Customer Orientation
Data Display From a Database By Search Parameters
Lean Six Sigma
Data Capture
Agile Processes
Incentives for Visitors
Logistics, Consumption Preferences, Whishes Club Admission, Bars Based on Geodata
Project Estimations
Transfer Functions
Communicate User locate each other Bargain Offers Ads, Marketing
Few Data
A lot of Fun! 5
Where do you need Apps? Customer Orientation
Lean Six Sigma
Agile Processes
Project Estimations
Transfer Functions
6
Where do you need Apps? Customer Orientation
Lean Six Sigma
Agile Processes
Project Estimations
Transfer Functions
7
Where do you need Apps? Customer Orientation
Lean Six Sigma
Agile Processes
Project Estimations
Transfer Functions
8
Example: The Ticket Apps Customer Orientation
Lean Six Sigma
Agile Processes
Customer’s Voice: Give me a ticket subito! Without much ado and questions if I simply want to go home User Stories 1) Q001 Prepare Destinations 2) Q002 Find Route
Project Estimations
Transfer Functions
3) Q003 Authentication
4) Q004 Issue Ticket
As a Traveler
I want to be able to store my preferred destinations Traveler locate nearest station with GPS Transportation give user access Service to their preferred Provider payment options Traveler get a valid ticket
such that they are valid for the Ticket Shop
so that I no longer have to pay fees when caught without tickets that's being served I immediately can see right now whether it's right all payments are they can manage traceable in Ticket spending Shop with settings from I no longer have to pay Ticket Shop fees when caught without tickets 9
The Ticket Apps 18 Entry (E) + 14 eXit (X) + 3 Read (R) + 3 Write (W) = 38 CFP Prepare
Customer Orientation
App User
Prepare Destinations
Standard Stations
GIS Application
Find Route
Ticket Shop
Timetable
Authentication
SIM Card
1.// Add New Destination 2.// Check whether Destination Exists 3.// Collect Matching Destinations 4.// Show Matching Destinations
Lean Six Sigma Payments
5.// Select Exact Destination 6.// Record Selected Destinations 7.// Authorize Automatic Payment 8.// Read SIM Card 9.// Enable Ticket Shop
Agile Traveling Processes
10.// Enable Payments
11.// Ask App for Ticket 12.// Propose Destinations 13.// Proposed Destinations
Project Estimations
14.// Select Destination
16.// Date & Time 17.// Ask GIS
Transfer Functions
18.// Nearest Boarding Station 19.// Search Connection 20.// Next Connection 21.// Request Ticket
22.// Send Pricing Proposal
10
The Ticket Apps – Objects of Interest Customer Orientation
Description
Type
Wants to buy a ticket to ride
Device User
Store a list of destinations that are valid for "getting home"
Functional Process
Standard Stations
List of standard stations eligible as travel destination
Persistent Data
GIS Application
Finds nearest origin station
Other Application
Find Route
Determine route being traveled
Functional Process
Ticket Shop
External ticket shop able to identify the App’s user
Other Application
Finds connections between stations at a given time & day
Other Application
Phone's GPS
Provides GPS coordinates where phone is located
Device User
Local Time
Synchronized time service located in smartphone
Device User
App User Lean Six Sigma
Agile Processes
Project Estimations
Transfer Functions
Prepare Destinations
Timetable Service
11
The Ticket Apps – Prepare Destinations Prepare Customer Orientation
App User
Prepare Destinations
Ticket Shop
Standard Stations
1.// Add New Destination 2.// Check whether Destination Exists 3.// Collect Matching Destinations
Lean Six Sigma
4.// Show Matching Destinations 5.// Select Exact Destination 6.// Record Selected Destination
Agile Processes
Preparation: Project Estimations
Transfer Functions
Store standard destination stations on Smartphone
After checking with the Ticket Shop whether they exist or which stop to select Involves Ticket Shop and local data ‘Standard Stations” 12
The Ticket Apps – Get the Ticket! Traveler Customer Orientation
App User
Find Route
Standard Stations
Ticket Shop
GIS Application
Timetable Service
Phone's GPS
Local Time
7.// Ask App for a Ticket 8.// Propose Destinations 9.// Proposed Destinations 10.// Select Destination
Lean Six Sigma
11.// GPS Coordinates 12.// Date & Time 13.// Ask GIS 14.// Nearest Boarding Station
Agile Processes
15.// Search Connection
Find Route
16.// Next Connection
17.// Request Ticket
Project Estimations
Nearest station by GIS to stored destination only 18.// Send Pricing Proposal Searches whether connection is available at that time of day
19.// Proposed Price 20.// Confirm Price
Ticket Request sent to Ticket Shop 21.// Confirm Transaction
Transfer Functions
22.//stored Issue Ticketin Ticket Shop for issuing ticket • Uses User Profile 23.// Show Ticket 13
Do Measure, Don’t Estimate 12 Entry (E) + 9 eXit (X) + 1 Read (R) + 1 Write (W) = 23 CFP Customer Orientation
Lean Six Sigma
Functional Size is measured by counting Data Movements as defined in the International Standard ISO/IEC 19’761 Count the number of data movements in the diagram needed for fulfilling all Functional User Requirements (FUR)
Test Size is measured by similar rules Agile Processes
Project Estimations
Transfer Functions
Count the number of sequences executed by all test cases in a test story
Enables comparison of functional size with test size and defect density
2Entry (E) + 3 eXit (X) + 1 Read (R) + 3 Write (W) = 9 CFP App User
E
Fun Proc
1. // Data Movement
W R E
X
Data Store
Fun Proc
Application
2. // Data Movement 3. // Data Movement 4. // Data Movement
W
5. // Data Movement
6. // Data Movement 7. // Data Movement?
X
8. // Data Movement
X W
9. // Data Movement 10. // Data Movement
Test Case Size = 4 Count Functional Size according ISO/IEC 19761 (“COSMIC”)
14
Exercise: What Apps could you think of? Customer Orientation
Hint: Start with an existing application Make it available on some mobile device
Lean Six Sigma
Adapt to Screen Size
Localize, use Geographical Information Systems Agile Processes
Use Camera, Near Field Communication, Voice Personalize, because you know the SIM identity • SIM = Subscriber Identity Module
Project Estimations
Limit the number of Functional Processes to the Minimum!
Transfer Functions
15
Does Your Apps Work as Expected? Customer Orientation
Something’s gone wrong?
Lean Six Sigma
Agile Processes
Project Estimations
Transfer Functions
16
What is a Test? Customer Orientation
A Software Test has Several Test Stories
Lean Six Sigma
• Explaining the Value for the Customer • Weighted by Customer’s Priority for the Test Story
Many Test Cases per Test Story • For various kind of test data Agile Processes
Test Size • Number of data movements executed by test cases
Project Estimations
Test Coverage • Percentage of data movements covered with test cases
An Outcome Transfer Functions
• Passed or Failed – Passed: All responses according expectations – Failed: at least one test case didn’t yield the expected response 17
What is a Test Case? Customer Orientation
A Test Case has Entry Data (“Test Data”)
Lean Six Sigma
Agile Processes
• Explaining the environment for the test case • Typically valid, invalid, borderline data • Normal and disturbed communication services
A known Expected Response • The response of the system is known in advance
Project Estimations
Transfer Functions
A known sequence of data movements executed • Defining Test Coverage • Each Test Case has a Size
18
What is a Test Case? Customer Orientation
The sequence of data movements can be visualized in the sequence diagram per test case Find Route
Lean Six Sigma
GIS Application
Phone's GPS
Local Time
8.// GPS Coordinates 9.// Date & Time
Agile Processes
10.// Ask GIS
11.// Nearest Boarding Station Project Estimations
Test Case Measurements for Test Story CT-A.1 CT-A.1 Find Nearest Station
Test Story No. 1 Functional User Requirements R001: Propose Standard Destinations R002: Find Nearest Boarding Station R003: Process Payment R004: Issue Ticket R005: Show Ticket
CT-A.1.1 Enter GPS exactly
Transfer Functions
E012
CT-A.1.2 Enter GPS nearby CT-A.1.3 Enter malformatted GPS
E006
CT-A.1.4 Enter far away GPS position
E006,E007,X004,E008
CT-A.1.5 GPS not working Test Story Contribution (CFP):
E005,E006
Returns correct station
E012,X009
Returns nearest station Returns no station
E012,X009
E006 2
Expected Response
E012,X009
Returns nearest station plus warning Returns warning
6
0
1
6
Test Size
19
Sample Test Stories Test Story Customer Orientation
Case 1
Test Data
Expected Response
Case 2
CT-A.1.1 Enter GPS exactly
Returns correct station
CT-A.1.2 Enter GPS nearby
Returns nearest station
CT-A.2.1 Select time of service
Returns next available connection
CT-A.2.2 Select time out of service
Return next best served station
Destination stored
CT-A.3.2 Enter invalid station name
Destination rejected
CT-B.1.1 Station of origin known
Asks for destination
CT-B.1.2 No destination selected
Stops without contacting Ticket Shop
CT-B.2 Get Ticket
CT-B.2.1 Both stations known
Asks to confirm price
CT-B.2.2 Boarding station undefined
Doesn't ask for destination
CT-B.3 Price Calculation
CT-B.3.1 Both stations known
Presents price of next actual connection
CT-B.3.2 Both stations known
Display intermediate change stations
CT-B.4 Issue Ticket
CT-B.4.1 Issue successful, ticket shown
All credentials visible
CT-B.4.2 Ticket Shop blocked
Explanation given
CT-B.5 Payment Tests
CT-B.5.1 Credit limit exceeded
Returns exceed notice
CT-B.5.2 Payment service out of order
Returns warning
CT-A Prepare CT-A.1 Find Nearest Station CT-A.2 Served Stations only
CT-A.3 Enter New Destination CT-A.3.1 Enter valid station name CT-B Ticketing CT-B.1 Select Destination Lean Six Sigma
Test Data
Expected Response
Agile Processes
Eight Test Stories for the Ticket Apps Project Estimations
Each test story has several test cases
Each test case has defined test data The expected response is known
Transfer Functions
20
What is Test Coverage? Customer Orientation
Lean Six Sigma
Number of Data Movements executed by Test Cases
Goal Profile of FURs as seen by Customers
Agile Processes
Project Estimations
Transfer Functions
Effective Profile for FUR Coverage by Test Stories
21
What is Test Priority? Test Story Customer Orientation
CT-A Prepare
Case 1
Test Data
Expected Response
CT-A.1 Find Nearest Station
CT-A.1.1
Enter GPS exactly
Returns correct station 0.30
CT-A.2 Served Stations only
CT-A.2.1
Select time of service
Returns next available connection 0.17 14
CT-A.3 Enter New Destination
CT-A.3.1
Enter valid station name
Destination stored
0.24
14
CT-B.1.1
Station of origin known
Asks for destination
0.30
21
CT-B.2 Get Ticket
CT-B.2.1
Both stations known
Asks to confirm price0.35
22
CT-B.3 Price Calculation
CT-B.3.1
Both stations known
Presents price of next0.45 actual connection 30
CT-B.4 Issue Ticket
CT-B.4.1
Issue successful, ticket shown
All credentials visible 0.37
25
CT-C.1.1
User agrees to T&Cs
Payment is processed0.33
19
CT-C.2.1
Credit limit exceeded
Returns exceed notice 0.39
25
CT-B Ticketing CT-B.1 Select Destination Lean Six Sigma
CT-C Payment CT-C.1 Contract Validation Agile Processes
Project Estimations
Transfer Functions
CT-C.2 Payment Tests
22
The Test Story Profile is the unique solution 𝒙 of 𝑻𝒙 = 𝒚 Where 𝒚 is the Goal Profile as seen by customers and 𝑻 is the frequency of data movements executed by all test cases in the Test Coverage Matrix The Test Story Profile 𝒙 is the (relative) priority for the test stories as seen by customers
22
How is Test Priority Calculated? Customer Orientation
Lean Six Sigma
The Frequency Matrix 𝑻
The Goal Profile 𝒚
Agile Processes
Project Estimations
Transfer Functions
The Calculated Solution 𝒙
The Effective Coverage 𝑻𝒙
23
What is the Goal Profile? Kano Approach is simple: Position topics in a frame
y Customer Satisfaction
Customer Orientation
5,5
• Delighters Lean Six Sigma
BD1
• One-to-One Requirements
• Expectations Agile Processes
Project Estimations
Transfer Functions
3,3
BD2
2,2 Kano Profile = 0.50
1,1
Kano Profile = 0.25
You can also use NPS (Net Promoter Score)… Business Drivers
BD3
Kano Profile = 0.75
Read out profile using grid
4,4
Kano Profile = 0.00
0,0
Technical Excellence
x
x y Measured
BD1 Have Ticket to Ride 0.45 11.0 3.6 4.1 BD2 Use Standard Settings 0.70 17.1 1.8 2.8 13.9 2.7 3.4 BD3 Buy with Few Clicks 0.56 24.6
24
Software Metrics and Gaming Customer Orientation
What is so exciting about a Game?
Lean Six Sigma
Agile Processes
Project Estimations
Transfer Functions
25
Agenda Customer Orientation
Apps Testing
Lean Six Sigma
Test Strategy Agile Processes
Project Estimations
Transfer Functions
Recruiting Testers The Testing Game 26
Agenda Customer Orientation
Apps Testing
Lean Six Sigma
Agile Processes
Project Estimations
Transfer Functions
Test Strategy Strategy Test Recruiting Testers The Testing Game 27
What is a Defect? Customer Orientation
Developers find Mistakes and eliminate them
Lean Six Sigma
Agile Processes
Testers find Defects Developers eliminate them
Project Estimations
Transfer Functions
Customers find Failures to Perform Supporters assess them, Developers eliminate them, and Testers repeat testing! 28
Goal of Software Testing Customer Orientation
Lean Six Sigma
Agile Processes
Find and remove
Defects before they become
Project Estimations
Failures to Perform Transfer Functions
29
Defect Classification Customer Orientation
Lean Six Sigma
A-Defects Missing, undetected requirement; “Anforderung” in German Can have disastrous effects on success on an Apps Detectable by Market Test; difficult to find before release
B-Defects Agile Processes
Project Estimations
Transfer Functions
Missed requirement A defect that became a failure to perform Detectable by Application Testing; seen by users
C-Defects Violation of Consistency, or requirements’ Contradiction Detectable by Peer Reviews, or Integration Testing Seen by developers only; relates to Technical Debt 30
Test Strategy Customer’s Needs
Lean Six Sigma
Customer’s Needs
Customer Orientation
Business Process Implementation
A-Defects Market Tests
Functional Requirements & Implementation FURs
Agile Processes
Application Tests
Software Services
Service-Oriented Architecture Transfer Functions
B-Defects
Software Services
Project Estimations
FURs
FURs
C-Defects
Service Integration Tests
31
Focus on B–Defects Customer Orientation
B-Defects can be located in the sequence diagram They affect at least one data movement
Lean Six Sigma
Thus they can be counted That count can be compared to functional size Defect Density Data Movements
Project Estimations
Transfer Functions
E001 X001 E002 X002 E003 W001
Write (W)
Read (R)
Data Movement Sub-Process Description
eXit (X)
Name Label 1) 2) 3) 4) 5) 6)
Move Type Entry (E)
Agile Processes
1 Add New Destination Enter a new destination station Check whether Destination Exists 1 be booked in the shop Check with ticket shop whether the entered destination exists and can Collect Matching Destinations Collect names and locations of destination stations that match 1the user's input, or indicate that user's destination i Show Matching Destinations Show list of eligible destination stations 1 1 Select Exact Destination Select destination exactly from list, if needed Record Selected DestinationsRecord it in the smartphone's store 1 CFP per Data Movement Type
Total CFP
18 14 3 3
3838 32
Functional Coverage Test Case Measurements for Test Story CT-A.1 Customer Orientation
User Stories
CT-A.1 Find Nearest Station
Q001: Prepare Destinations
CT-A.1.1 Enter GPS exactly
X005,R002,E005,X009,E009,X006
Q002: Find Route
Q003: Authentication E006,X005,R002
CT-A.1.2 Enter GPS nearby Lean Six Sigma
Agile Processes
CT-A.1.3 Enter malformatted GPS
E006
CT-A.1.4 Enter far away GPS position
E007,E006,X004
CT-A.1.5 GPS not working
E005,E006
E006
Which data movements are necessary to execute a FUR? Click, and
Project Estimations
Count App User
Find Route
Authentication
Payment Services
Phone's GPS
10.// Enable Payments Transfer Functions
14.// Select Destination 15.// GPS Coordinates
33
Test Coverage – Part 1 Test Stories for Capability Testing
Add Test Case Column Delete Empty Test Cases
Customer Orientation
Test Cases Test Story CT-A Prepare
Lean Six Sigma
Case 1
Test Data
Expected Response
Case 2
Test Data
Expected Response
CT-A.1 Find Nearest Station
CT-A.1.1
Enter GPS exactly
Returns correct station
CT-A.1.2
Enter GPS nearby
Returns nearest station
CT-A.2 Served Stations only
CT-A.2.1
Select time of service
Returns next available connection
CT-A.2.2
Select time out of service
Return next best served station
CT-A.3 Enter New Destination
CT-A.3.1
Enter valid station name
Destination stored
CT-A.3.2
Enter invalid station name
Destination rejected
CT-B.1.1
Station of origin known
Asks for destination
CT-B.1.2
No destination selected
Stops without contacting Ticket Shop
CT-B.2 Get Ticket
CT-B.2.1
Both stations known
Asks to confirm price
CT-B.2.2
Boarding station undefined
Doesn't ask for destination
CT-B.3 Price Calculation
CT-B.3.1
Both stations known
Presents price of next actual connection
CT-B.3.2
Both stations known
Display intermediate change stations
CT-B.4 Issue Ticket
CT-B.4.1
Issue successful, ticket shown
All credentials visible
CT-B.4.2
Ticket Shop blocked
Explanation given
CT-C.1.1
User agrees to T&Cs
Payment is processed
CT-C.1.2
User does not click "agree"
User is prompted for cancelation
CT-C.2.1
Credit limit exceeded
Returns exceed notice
CT-C.2.2
Payment service out of order
Returns warning
CT-B Ticketing CT-B.1 Select Destination
CT-C Payment CT-C.1 Contract Validation
Agile Processes
CT-C.2 Payment Tests Add Application Test Story Delete Application Test Story
Project Estimations
Test Case Measurements for Test Story CT-A.1 Transfer Functions
The test stories cover the FURs by the data movements executed by test cases Test Story No. 1 User Stories
CT-A.1 Find Nearest Station
Q001: Prepare Destinations
CT-A.1.1 Enter GPS exactly
X005,R002,E005,X009,E009,X006
Q002: Find Route
Q003: Authentication
Q004: Issue Ticket
Expected Response
E006,X005,R002
E012,X009
Returns correct station
11
E012,X009
Returns nearest station
2
Returns no station
1
Returns nearest station plus warning
5
Returns warning
3
Test Size
22
CT-A.1.2 Enter GPS nearby CT-A.1.3 Enter malformatted GPS
E006
CT-A.1.4 Enter far away GPS position
E007,E006,X004
CT-A.1.5 GPS not working Test Story Contribution (CFP):
E005,E006
E012,X009
E006 8
5
3
6
CFP
34
Test Coverage – Part 2 Customer Orientation
The Test Coverage matrix visualizes how test stories cover Functional User Requirements (FURs) Effective Coverage should match Goal Test Coverage
Lean Six Sigma
Small Convergence Gap becomes green
Agile Processes
Project Estimations
Transfer Functions
35
Exercise: Test Stories for Your Apps Customer Orientation
Define a few FURs for your App Prioritize them in view of the App’s users
Lean Six Sigma
Using the Kano Method (use tool QfdKanoPriority.xlsm)
Check whether the data movements you’ve defined in the sequence diagram cover the customer’s need Agile Processes
Play your FURs on the sequence diagram Use the sample App provided (use tool TicketFur.xlsm)
Project Estimations
Now define a few Test Stories for your FURs Which FURs do your Test Stories support?
Transfer Functions
You must measure the Test Cases
Using the Test Coverage tool (QfdTestCoverage.xlsm) 36
Agenda Customer Orientation
Apps Testing
Lean Six Sigma
Test Strategy Agile Processes
Project Estimations
Transfer Functions
Recruiting Testers The Testing Game 37
Agenda Customer Orientation
Apps Testing
Lean Six Sigma
Agile Processes
Project Estimations
Transfer Functions
Test Strategy Recruiting Testers Recruiting Testers The Testing Game 38
Go to the Gemba Customer Orientation
Lean Six Sigma
Agile Processes
Project Estimations
Transfer Functions
The king put on his shoes and went down where pigs wallow in the mire under the hot sun 39
Net Promoter® Survey Customer Orientation
The Ultimate Question : If a friend or relative asks you: How likely is it that you recommend our Ticket Shop?
Lean Six Sigma
Agile Processes
Please use a score between 0 (sure not!) until 10 (for sure!) Sure not! 0
Rather not 1
2
3
Don’t know 4
5
Probably yes 6
7
8
Absolutely! 9
10
Why did you select this score? Project Estimations
Please explain your score in a few words
Transfer Functions
40
Net Promoter® Score Customer Orientation
Sure not! 0
Rather not 1
2
Don’t know
3
4
5
Probably yes
6
7
8
Absolutely! 9
10
Lean Six Sigma
Detractors Agile Processes
Project Estimations
Neutrals
Promoters
Net Promoter Score = %Promoters – %Detractors NPS = 40%:
Everything’s fine!
NPS = 0%:
Something’s wrong
NPS < 0%:
All hands off board!
Transfer Functions
41
NPS Survey Execution Customer Orientation
Net Promoter® Score Surveys How likely is a recommendation?
Lean Six Sigma
Agile Processes
Project Estimations
Open question for the score’s reasons Classify the responses by the most frequent Customer’s Needs, for our Ticket App: CN1: Have Ticket to Ride CN2: Use Standard Settings CN3: Buy with Few Clicks
Frequency count Total of classified responses
Transfer Functions
Positive vs. Negative Give me a ticket subito! Without much ado and questions if I simply want to go home!
42
WOM Economics – Words of Mouth Customer Orientation
Loyal Customers Repurchase
Lean Six Sigma
Buy New Product Lines Recommend, and Give Feedback
Agile Processes
Recommendations Help increase future business
Project Estimations
Help improve the product or service
Transfer Functions
43
Turn Detractors into Promoters! Customer Orientation
Promoter is worth much more than the business generated Every positive Word of Mouth may yield new business
Lean Six Sigma
Agile Processes
Project Estimations
Transfer Functions
Detractors do more harm than the value of lost business by depreciatory comments Can spoil your business model, e.g., in Software for B2B:
Value of Promoter:
€
838'591
Value of Detractor:
€
-506'442
Value of turning Detractor -> Promoter:
€
1'345'033 44
NPS Survey Analysis Profile Customer Orientation
Customer Segments CS1 Commuter/Business CS2 Private Traveler CS3 Shopping Travelers CS4 Leisure Travelers
Categorized Response Count
Net Positive Response Count
Lean Six Sigma
Agile Processes
Project Estimations
Transfer Functions
!
?
45
Identify the Values of Your Customers Business Drivers - Profile
Agile Processes
Project Estimations
Business Drivers CN1 Have Ticket to Ride CN2 Use Standard Settings CN3 Buy with Few Clicks
1
1
Management Adjustments
Lean Six Sigma
Negative Satisfaction is stretched exponentially!
Satisfaction Gap
Customer Orientation
Importance
Combined from NPS Survey
S
0.41 0.52
0.93
0.73 0.59
1.32
0.55 0.62
1.17 2
* 1.0 * 1.0 * 1.0
NPS Profile 0.47 0.66 0.59
-0.50 0.52
0.50
0.67
0.59
0.57
0.55
0.62 1.03
0.60 0.97
0.50 0.67
Compare Importance with Satisfaction Here not good enough, in general!
You can also add new strategic goals Transfer Functions
Be it to address new customers, or To comply with legal constraints such as the need for a click to confirm payment
46
Recruit Your Testers Customer Orientation
First use own resources Testers who know the process
Lean Six Sigma
Second, ask Promoters whether they want to participate You have to explain them what software test means
Third, ask Detractors that constructively criticize Agile Processes
Project Estimations
Transfer Functions
Need special treatment Priv ate Trav eler
4
Too many clicks, and alw ay s the same inputs
Commuter/Business
9
Can print my ticket at home
Priv ate Trav eler
10
Well done the ticket counter at home!
Shopping Trav elers
3
Takes too long
Leisure Trav elers
8
Must constantly ty pe the same again, loose time to get a ticket
Leisure Trav elers
9
Giv es me a ticket subito! Without much ado and questions if I simply w ant to go home!
Priv ate Trav eler
9
I like calling back stored routes
Priv ate Trav eler
9
Can prepare at home a note, it's quick 47
Test Interface Preparations Customer Orientation
Lean Six Sigma
Agile Processes
Project Estimations
Transfer Functions
Tester must be able to see selected sequences in the diagram Tester can “walk” on a data movement when planning or executing tests
Find Route
GIS Application
Phone's GPS
Local Time
8.// GPS Coordinates 9.// Date & Time 10.// Ask GIS 11.// Nearest Boarding Station
Functionality and defects affecting functionality become visible
Test data must be provided for installation or manual entry 48
Test Result Reporting Preparations Customer Orientation
Find Route
GIS Application
Lean Six Sigma
Phone's GPS
Local Time
8.// GPS Coordinates 9.// Date & Time 10.// Ask GIS
Agile Processes
Project Estimations
11.// Nearest Boarding Station
Tester must be able to mark the data movement where a defect has been detected A SharePoint Apps allows recording observations and susceptible behavior Defects need being confirmed by test management
Transfer Functions
49
Defect Reporting
18 Entry (E) + 14 eXit (X) + 3 Read (R) + 3 Write (W) = 38 CFP Prepare
App User
Prepare Destinations
Standard Stations
GIS Application
Find Route
Ticket Shop
Timetable
Authentication
Paymen
SIM Card
1.// Add New Destination
Customer Orientation
2.// Check whether Destination Exists 3.// Collect Matching Destinations 4.// Show Matching Destinations 5.// Select Exact Destination
Payments
Lean Six Sigma
6.// Record Selected Destinations 7.// Authorize Automatic Payment 8.// Read SIM Card 9.// Enable Ticket Shop
Traveling
Agile Processes
10.// Enable Payments 11.// Ask App for Ticket 12.// Propose Destinations
13.// Proposed Destinations 14.// Select Destination
Project Estimations
16.// Date & Time 17.// Ask GIS 18.// Nearest Boarding Station 19.// Search Connection 20.// Next Connection
Transfer Functions
21.// Request Ticket 22.// Send Pricing Proposal
23.// Proposed Route Contract
24.// Cancel Traveling 25.// Confirm Travel Contract
50
Defect Reporting Customer Orientation
Lean Six Sigma
Enter defect descriptions manually Enter defect descriptions in a defect reporting table Then enter the reference to the defect in the test story table containing the test case that identified the defect
You can do the same graphically Double-click on test case
Agile Processes
Project Estimations
Transfer Functions
Test Case Measurements for Test Story CT-B.4
User Stories
CT-B.4 Issue Ticket
Q001: Prepare Destinations
CT-B.4.1 Issue successful, ticket shown
E011,X008,E010,X007,E009,X006,X009
CT-B.4.2 Ticket Shop blocked
X008,E012
CT-B.4.3 Payment processed
X008
CT-B.4.4 Ticket Shown
X008
Q002: Find Route
E010,X007
Q004: Issue Ticket
X009,E012,X008
X009
E011,X008
X014,W003,E018 X009
CT-B.4.5 Payment Test Story Contribution (CFP):
Q003: Authentication
X008 11
2
X009 6
6
51
Defect Reporting Customer Orientation
Lean Six Sigma
Agile Processes
Project Estimations
Transfer Functions
52
Defect Reporting Test Case Measurements for Test Story CT-B.4 Customer Orientation
Lean Six Sigma
User Stories
CT-B.4 Issue Ticket
Q001: Prepare Destinations
CT-B.4.1 Issue successful, ticket shown
E011,X008,E010,X007,E009,X006,X009
CT-B.4.2 Ticket Shop blocked
X008,E012
CT-B.4.3 Payment processed
X008
CT-B.4.4 Ticket Shown
X008
Q002: Find Route
E010,X007
Q003: Authentication
Q004: Issue Ticket
X009,E012,X008
X009
E011,X008
X014,W003,E018 X009
CT-B.4.5 Payment
X008
Test Story Contribution (CFP):
11
2
X009 6
6
Agile Processes
Project Estimations
CFP
Defects Observed
Data Movements Affected
Name Label
Description
Name Label
#002
Takes boarding station wrong
X007
11 2 8 Transfer Functions
Wrong Price
Search Connection
2 2 25
1
Defect Count 53
The Apps Testing SharePoint Customer Orientation
Lean Six Sigma
Agile Processes
Project Estimations
Transfer Functions
54
Agenda Customer Orientation
Apps Testing
Lean Six Sigma
Test Strategy Agile Processes
Project Estimations
Transfer Functions
Recruiting Testers The Testing Game 55
Agenda Customer Orientation
Apps Testing
Lean Six Sigma
Agile Processes
Project Estimations
Transfer Functions
Strategy TheTest Testing Game Recruiting Testers The Testing Game 56
Preparations Customer Orientation
You need to set up a minimum set of test stories Covering the FURs
Lean Six Sigma
Select testers from promoters Only from promoters; or from detractors that complain
Agile Processes
No neutrals
Make the sequence diagrams available to testers Project Estimations
Transfer Functions
Document the FURs well Explain what a defect is: a response different from the expected: no response, a wrong response, a system crash Allow testers to mark defects visually in sequence diagrams
Set up your award system
57
Level 1: The Test Executor Customer Orientation
Scoring: Every data movement executed in a test counts 1 point
Lean Six Sigma
A defect suspected counts 10 point A defect confirmed counts 30 point Points are cut in half when uncovering same defects again
Agile Processes
Challenges: Test result capture is separated from executing the App
Project Estimations
Transfer Functions
Ideally, it’s integrated; however, that work only in very special environments
Solution: Switch between App execution and text result capture 58
Level 2: The Test Case Creator Customer Orientation
Adding test cases changes the test story profile Award closing the convergence gap
Lean Six Sigma
Do not inhibit opening the convergence gap – somebody might have found a novel way of using the App
Test Case Creator defines test data and expected response Agile Processes
Test Case Creator sees the Test Coverage Matrix and where the convergence gap opens Can continue to execute tests and get scores for it
Project Estimations
Transfer Functions
Note that test cases can only be executed after the proposed test data has been approved and made available by test management
59
Level 2: The Test Case Creator Customer Orientation
Scoring: Every test case created counts 10 points
Lean Six Sigma
Agile Processes
A test case accepted counts 40 points Closing the convergence gap by 0.1 adds 20 points
Challenges: A Test Case is only as good as test data are reproducible Direct contact and discussions might be necessary for scoring
Project Estimations
Transfer Functions
Solution: Establishing a group chat among Test Case Creators might help stirring competition 60
Level 3: The Test Story Designer Customer Orientation
Test Story design is changing the value for the customer Make hurdles high: a minimum of three test cases per story
Lean Six Sigma
Agile Processes
For awards, the new test story must add value • Probably one alone will not do • Convergence gap must not suffer • Test stories without test cases are useless
Test Story Designers earn their awards from the ability to create new test cases and execute more tests
Project Estimations
Transfer Functions
61
Level 3: The Test Story Designer Customer Orientation
Scoring: A proposed test story with three test cases counts 10 points
Lean Six Sigma
A test story accepted counts 50 points Closing the convergence gap by 0.1 adds 25 points • Normally you need more than one test story to close the gap
Agile Processes
Challenges: Creating test stories means a lot of work and small return
Project Estimations
Transfer Functions
Solution: For an accepted new test story there are many opportunities for test cases that count high • Especially if that helps increasing the convergence gap 62
The Testing Interface Customer Orientation
The sequence stops after each data movement
Find Route
Lean Six Sigma
GIS Application
Phone's GPS
Local Time
8.// GPS Coordinates 9.// Date & Time
Agile Processes
10.// Ask GIS
11.// Nearest Boarding Station Project Estimations
Transfer Functions
Test Case Measurements for Test Story CT-A.1
Test Story No. 1 User Stories
CT-A.1 Find Nearest Station
Q001: Prepare Destinations
CT-A.1.1 Enter GPS exactly
X005,R002,E005,X009,E009,X006
Q002: Find Route
Q003: Authentication
Q004: Issue Ticket
Expected Response
CFP
E006,X005,R002
E012,X009
Returns correct station
11
E012,X009
Returns nearest station
2
Returns no station
1
Returns nearest station plus warning
5
Returns warning
3
Test Size
22
CT-A.1.2 Enter GPS nearby CT-A.1.3 Enter malformatted GPS
E006
CT-A.1.4 Enter far away GPS position
E007,E006,X004
CT-A.1.5 GPS not working Test Story Contribution (CFP):
E005,E006
E012,X009
E006 8
5
3
6
63
The Testing Interface Customer Orientation
The tester stops the Walker when perceiving a defect
Find Route
Lean Six Sigma
GIS Application
Phone's GPS
Local Time
8.// GPS Coordinates 9.// Date & Time
Agile Processes
10.// Ask GIS
11.// Nearest Boarding Station Project Estimations
Transfer Functions
Test Case Measurements for Test Story CT-A.1
Test Story No. 1 User Stories
CT-A.1 Find Nearest Station
Q001: Prepare Destinations
CT-A.1.1 Enter GPS exactly
X005,R002,E005,X009,E009,X006
Q002: Find Route
Q003: Authentication
Q004: Issue Ticket
Expected Response
CFP
E006,X005,R002
E012,X009
Returns correct station
11
E012,X009
Returns nearest station
2
Returns no station
1
Returns nearest station plus warning
5
Returns warning
3
Test Size
22
CT-A.1.2 Enter GPS nearby CT-A.1.3 Enter malformatted GPS
E006
CT-A.1.4 Enter far away GPS position CT-A.1.5 GPS not working Test Story Contribution (CFP):
E007,E006,X004 E005,E006
E012,X009
E006 8
5
3
6
64
Rewards Customer Orientation
External testers can be rewarded By public acknowledgment
Lean Six Sigma
Agile Processes
By reduced service rates By other fringe benefits
Financial rewards can be considered Makes external testers similar to external consultants Should rely on high score
Project Estimations
Transfer Functions
65
Overview Voice of the Customer (VoC)
CN VoC
#NPS Customer Orientation
MT CN
FUR CN Customer's Needs (CN)
Lean Six Sigma
Market Test (MT) #A-Defects
SwS FUR Agile Processes
UTS FUR
Func. User Reqs. (FUR) #CFP
User Test Stories (UTS)
Project Estimations
#B-Defects
Software Service (SwS) Transfer Functions
ITS SwS
#CFP
Integration Test Stories (TS) #C-Defects
66
Conclusions Customer Orientation
Testing mobile Apps can be crowd-sourced It requires sound preparation
Lean Six Sigma
Agile Processes
It is difficult to find good testers You save on establishing testing platform for a huge variety of smartphone but you invest into happy promoters
You link testing to product or service promotion Potential benefits are huge
Project Estimations
Risks are considerable
Most important risk: Transfer Functions
Your marketing department has no idea what SW testing is! 67
Questions? Customer Orientation
Lean Six Sigma
Agile Processes
Project Estimations
Transfer Functions
68
