How to Create a Performance-Based Certification Program for ...

How to Create a Performance-Based Certification Program for Software and Technology

Performance-Based Certification White Paper

How to Create a Performance-Based Certification Program for Software and Technology

Contents Abstract ....................................................................... 2 Introduction ................................................................. 3 The Value of Performance-Based Certifications ............ 4 Graphic A: Bloom’s Taxonomy ...................................... 5 Problem: Lack of Information ....................................... 6 Solution: Your “How-to” Guide ..................................... 6 Due Diligence ............................................................... 6 Planning and Designing the Test ................................... 8 Content and Test Writing ............................................. 9 Creating Internally vs. Outsourcing ............................. 10 Testing and Launching ................................................ 11 Considerations ........................................................... 12 Graphic B: Example Performance-Based Certification Program Ecosystem Conclusion ................................................................. 12 Author/Contributors .................................................. 14 For More Information ................................................ 15 Endnotes .................................................................... 17

12

1

How to Create a Performance-Based Certification Program for Software and Technology

Abstract Certifications are a crucial aspect of the world of technology today. Not only do certifications perform a gatekeeping function to ensure that skilled users are able to interact with crucial software and applications, they also provide a system of distinguishing various levels of knowledge. However, the content and testing method of a certification assessment is crucial to determining its value. Different assessment tools extract different types and levels of knowledge from candidates. Certification program managers should strive to utilize the most comprehensive testing method. A performance-based assessment is a testing method that places users in an environment that emulates on the job tasks and captures a wide spectrum of candidate ability and comprehension. There are a lack of resources available explaining how to design and launch a performance-based certification program. This paper will delve into relevant research on the subject of designing and launching a performance-based certification program.

2

How to Create a Performance-Based Certification Program for Software and Technology

Introduction When it comes to certifications, there are essentially three primary stakeholders that benefit from the value that a certification provides. The three stakeholders involved in a certification program are the certification holder, the hiring organization, and the technology provider or certification body. Each stakeholder receives distinct benefits from a certification program. Perhaps the most obvious benefactor of a certification is the certification holder. As a certification holder, one has more leverage as a potential employee than a non-certified professional because certifications provide a competitive advantage in the workplace. Not only are certified individuals more likely to be hired but also they are also more likely to be paid more than their uncertified counterparts. An IT salary report released by Foote Partners in 2015 revealed that in terms of pay raises, “salaries of employees with non-certified skills went up nearly 4% in the past year compared to 10% for certified skills.” This is because certifications provide more specialized proof of a candidate’s skill level than other methods can, even college degrees. In fact, a 2013 research study published by Lumina Foundation found that only 10% of business leaders surveyed believed that college degrees provided students with the right skills at the right level. Apart from pay rate, certified individuals reportedly enjoy “higher job satisfaction...and greater confidence in their ability to do their jobs compared to their uncertified peers.” While college degrees often serve as a baseline credential for a role, college educations can vary in terms of content and rigor. Having a streamlined and centralized certification that applicants can strive to obtain effectively levels the playing field. Certifications are also valuable to a hiring organization. An IT Salary Survey conducted by InformationWeek released in 2014 suggested that IT management valued certification within an organization more than two times an MBA. Certifications are critical to hiring organizations because of the need for skill validation and gauging the expertise level of employees. There are additional benefits to hiring and maintaining certified employees. For example, with training and experience, certified employees are often capable of doing better and more efficient work, ultimately ensuring high-caliber expertise to make things work correctly. Additionally, certifications can create a more objective hiring process within an organization. Hiring organizations benefit from hiring certified individuals because they have already demonstrated their skill level and this practice can help create a more objective hiring system. Lastly, certifications are not only valuable to the certification holder and the hiring organization but also to the technology provider/certification body providing or overseeing the certification. The technology provider or certification body reaps many benefits from having experts of their particular technology in the field. Having people who are certified to use and have mastered an organization’s technology not only reduces support requests from users, but can also be leveraged as a marketing tool. Someone who is certified in a particular technology is effectively an expert who will often refer to that technology, purchase more of that technology, and evangelize it in various organizations or projects over time. This creates a type of promoter effect, directly from the mouth of the experts themselves. Further, a certification lends validity and promotes a leadership position within the competitive landscape. The value of certification

3

How to Create a Performance-Based Certification Program for Software and Technology does not lie only in the hands of the person being certified but also the organization that has hired a certified individual as well as the certification body.

The Value of Performance-Based Certifications While there is multidimensional value in certifications, not all certifications are created equally. For example, some certifications rely on low tech, multiple-choice tests, while others require candidates to actively demonstrate on-the-job skills in a highly realistic environment. While a multiple-choice assessment is valuable in providing insight into a candidate’s theoretical knowledge of a technology, used alone it’s not the strongest indicator of a candidate’s ability to actually perform the necessary functions to solve operational issues. In fact, research suggests that many software opinion leaders believe that multiple choice tests alone may not actually “provide proof of the candidate’s true knowledge, skill and ability...especially...for IT-oriented certification programs.” Performance-based assessments require candidates to demonstrate actual skills. These are high fidelity tests that allow test moderators to view the candidate in the technologies present in the actual work environment. An expert in the world of testing, researcher Judith Hale, Ph.D., authored a must read book for any fledgling program, titled, Performance-Based Certification: How to Design a Valid, Defensible, Cost-Effective Program. In it she explains that fidelity is the “degree to which the assessment device reflects the performance of a task or demonstration of knowledge in a real workplace setting.” There is a need for high fidelity assessments in the world of certifications in order to protect a company or technology’s integrity. Performancebased certifications help fill a crucial industry gap by ensuring that certification holders have the actual skills necessary to perform well on the job. Performance-based testing also ensures there is little risk of breach in the testing process, that the certification actually proves real skills, and also provides the ability to create test variability. However designing a certification program is an involved process and it can be difficult for an organization to know where to begin. One of the ways to think of the benefits of this type of certification program is to consider it in terms of Bloom’s Taxonomy. Bloom’s Taxonomy is a framework developed in the 1950’s by a group of educators and revised in 2001 that ranks educational objectives. The framework presents six different types of learning goals that represent increasing levels of understanding and comprehension.

4

How to Create a Performance-Based Certification Program for Software and Technology

Bloom’s Taxonomy

Figure 1: Bloom's Taxonomy Anderson, L.W. (Ed.), Krathwohl, D.R. (Ed.), Airasian, P.W., Cruikshank, K.A., Mayer, R.E., Pintrich, P.R., Raths, J., & Wittrock, M.C. (2001).

Learning taxonomies typically present three different domains of knowledge: the cognitive domain, the affective domain, and the psychomotor domain. The cognitive domain deals with the ability to recall specific terms and details and this type of information is sometimes referred to as knowledge-based information. The levels of the cognitive domain, range from remembering to evaluating. The affective domain refers to emotional reactions and is concerned with attitudes. The levels of the affective domain range from receiving to characterizing. Finally the psychomotor domain refers to action-based knowledge and skills, like being able to perform a specific function. There are seven different levels of the psychomotor

5

How to Create a Performance-Based Certification Program for Software and Technology domain from perception to origination. Performance-based assessments test for skills across these domains. According to Judith Hale, performance-based testing allows organizations to “screen job applicants, measure learning, and determine proficiency or competence for job placement and recognition.” Being able to verify that a professional has the right skills is critical. To learn more about the value of performance-based testing and certifications, read the white paper, “The Value of Performance-Based Certification & Training for the IT Industry.”

Problem: Lack of Information Although there is significant evidence that performance-based certifications are valuable, there is a lack of generally available information regarding the process of designing and launching a performance-based certification program. Approaching this process thoroughly and correctly is crucial to a program’s success.

Solution: Your “How-to” Guide This paper will attempt to provide a definitive guide for designing and launching an effective performance-based certification program, leveraging knowledge and experiences from both academics and practitioners of performance-based testing. The general steps to start the process of creating a performance-based certification program are not very different from that of any type of test writing or planning. First, complete due diligence research and determine if a performance-based certification program is really the right fit for your organization and its needs. The due diligence portion of the program design should produce an identified program driver, a clear purpose behind the certification program, and assurance of key stakeholder support. Then, it is critical to plan exactly how to launch the certification program and what it will look like. This will involve determining a few different components such as the requirements for candidacy, the delivery method, and mapping out the end state. The following suggested steps provide a framework for how to approach the design and launch of a certification program.

Due Diligence The due diligence portion of the certification program design process includes a few different deliverables. This first step requires determining the driver of a certification program–the purpose of the certification and ensuring that all stakeholders are on board with the program.

6

How to Create a Performance-Based Certification Program for Software and Technology

The driver of a program is the catalyst for why it’s being created or what the program is being created in response to. Hale writes that often “certification programs are the result of external pressures on organizations to ensure that individuals perform to a standard.” However, this is not the only possible driver. Being able to understand what the driver is or what the certification program is in response to is critical to the program’s success. The driver presents three different outputs: 1. Goal statement 2. Metrics for analyzing the effectiveness of a program 3. Agreement of target audience and stakeholders involved in the program Some examples of certification program drivers include protecting the public and demonstration of professional status and supporting universal standards. Other reasons for creating a program may include marketing a type of technology or creating a more objective hiring process. Strengthening hiring processes as well as leveraging certifications as a marketing tool can make an organization stand out in a competitive market. However, the driver will vary depending on organization, goals, and industry. Having a clear idea of what the driver is behind a certification program will shape many of the program’s requirements and features. Understanding and defining the purpose behind a certification is critical to its development. There are gateway certifications, certifications that recognize demonstrated performance, and certifications that recognize different levels of capability. A gateway certification essentially acts as a prerequisite for the authorization of a particular position or task. The purpose of a gateway certification is to assess a certain level of skill before allowing an individual to complete a particular task or maintain a particular job role. On the other hand, certifications that are intended to recognize demonstrated performance allow candidates to complete the work prior to receiving a certification but the candidate must demonstrate their on-the-job performance in order to receive the certification. Finally certifications that recognize different levels of accomplishments serve as proof of skill level and may, for example, distinguish a novice programmer from an expert. The purpose behind a certification program is intertwined with other defining traits, like whether or not there will be multiple levels of certification or how often recertification will be required if at all. Determining the purpose behind a certification program at the beginning stages will inform the design and execution of the program. Another important factor to consider in starting a certification program is determining whether key organizational players are on board with the program. Ensuring that the company’s leadership agrees with and understands the value of performance-based certification is critical. This is the job of the certification program manager and they must secure agreement among stakeholders on the goals and standards of the certification. In today’s tech world, organizations are deliberately moving away from testing methods that don’t accurately represent and capture a candidate’s expertise. The higher fidelity of performance-based certification tests requires candidates to demonstrate procedural knowledge and helps address this issue. However, key

7

How to Create a Performance-Based Certification Program for Software and Technology organizational leadership must understand this aspect of a performance-based certification program or the program may not succeed.

Planning and Designing the Test Once due diligence is completed, it is necessary to plan the content and requirements of the certification program’s assessment. When determining the requirements of a new certification program, a few different questions need to be answered. Some of the potential requirements include: the requirements for candidacy, the standards of the test, and the content of the test. The requirements may be a product of factors such as the delivery method and the desired endstate. The majority of certification programs maintain a mix of different requirements. These requirements should also be revisited when choosing the requirements for maintaining credentials and/or recertification. Determining the requirements for the certification program will make it easier to assess what specific tasks and skills needs to be tested within the assessment. After deciding on the certification requirements, it is important to determine the delivery method of the assessment. According to Rich Raposa, certification manager at Hortonworks who launched a performance-based certification program, finding a delivery platform “is a large hurdle and the most critical decision to make…*and+ will determine almost everything that follows.” The delivery method will be closely related to factors such as proctoring and administration. An assessment delivered in person would more than likely require a person at the physical testing site to administer and oversee the test. However, a test delivered purely online would require different controls in order to ensure security. Determining the best methods for proctoring and administration will go hand in hand with the delivery method. Selecting a delivery method in the case of performance-based assessments presents many questions because there are a few different options. With the right resources, you may choose to build the platform internally, but without the necessary internal resources, you may select to work with a third party vendor. Choosing what method to use becomes easier if the end state process has been mapped out. Mapping out the end state will require a few different considerations. A good starting point for mapping out the end state is to have answers to the following questions: 

Will this be a proctored exam?



Will this be formatted as part of a curriculum strategy and tracked through a learning management system (LMS)?



Is there a deadline or critical launch date?



What needs to be done in order to meet the launch date?



Will there be multiple phases of the certification program? If so there should be established differences in what needs to be included in each phase of the launch.

8

How to Create a Performance-Based Certification Program for Software and Technology Determine what the bare minimum is to launch and work from there. Once the requirements of the end-state have been decided, it will be easier to make decisions regarding the delivery method.

Content and Test Writing The next important step in the planning and design process is figuring out what the scope and content of the test should be. Writing the test content often requires leveraging one or more subject matter experts (SME) who can help a certification program manager determine not only what tasks a certification should entail, but help create the content. For technology organizations without a dedicated resource, the SME can often be a technical trainer, tenured customer support technician, or sales support engineer. These types of SMEs can help contextualize and provide specific technical knowledge. The SME is particularly helpful with the job task analysis (JTA), another critical step in creating the certification so that it’s relevant to the end users. A JTA determines what people do, the working conditions involved, and the required knowledge and skills. This type of analysis assists in the process of developing specific standards for the content of the assessment. The steps involved with performing a job task analysis are as follows: 

Phase one: Create a list of tasks describing what people do, the conditions, and any required skills



Phase two: verify the degree to which a larger group agrees with the initial description and rank the tasks by importance, frequency, difficulty



Phase three: Convert the list of tasks into performance standards that include conditions, tools, triggers, criteria for judging quality

The different phases of the job task analysis create “as complete a picture as possible of the task, the environment in which it is performed, and the stakeholders’ expectations. It also describes the organization’s expectations in sufficient detail to support assessing candidates’ ability to satisfy the standards.” Beverly van de Velde, leader of the Global IT Certification Program at Symantec Corporation, explains that the importance of the job task analysis is that it measures "how" to complete a task relevant to the job. This might also require defining the workflow and expected order to complete tasks, depending on whether sequence is more important than or related to output. Now that the requirements of the test have been determined, the scope of what the certification must cover is clear. Once the JTA has been completed and a clear list of performance standards has been determined, test writing can begin. Writing the test will involve determining a scenario that captures all of the requirements decided on in the job task analysis. Ideally, the SME will need to advise on the scenario and its details. Scenario development is the most critical aspect of developing a performance-based test because it must test all of the required skills and the content of the job task analysis. Keep in mind that when writing the test, the question of how

9

How to Create a Performance-Based Certification Program for Software and Technology each step discussed in scenario development or skill will be graded is critical. For performancebased assessments, the answers need to be written into the test item, as verifiable end-states. Additionally, the test will ultimately need to be scalable as well.

Creating Internally vs. Outsourcing There are distinct benefits and difficulties associated with considering creating a performancebased test internally or launching it with third party vendors. Creating a performance-based assessment internally from scratch is a massive undertaking that requires resources and time. From outlining goals and objectives, writing test items and connecting pieces like test provisioning, administration, grading, proctoring (online or physical), a learning management system (LMS) and credentialing, a certification program manager must have the right resources in place. One way to determine if a program should be built and delivered by internal resources is to measure the deliverable against the company’s core competencies. For example, technology companies like RedHat and Cloudera both created and maintain their own performance-based certification infrastructure and delivery systems. These companies have internal teams and expertise that they leverage and dedicate to creating “fromscratch” tools required to run an end-to-end certification environment and integrate into peripheral systems. Apart from leveraging core competencies, there are other reasons to outsource parts of the program development. For example, Hale lists gaining scalability and the ability to downsize, avoiding long-term liability and cost of developing internal structure to support a program, and gaining access to other technologies. For the right organization with the right level of manpower, budget and existing capability to administer the exam, creating the assessment internally may be an efficient and effective decision. However, launching a performance-based certification program is a costly and time-consuming undertaking if systems and resources are not already in place to support the assessment. In this case, the use of a third party vendor will prove more efficient. In years past, creating performance-based certification was a costly endeavor with no standardization and minimal resources readily available to aid a program manager. In the age of outsourcing, with technological advancements and economical advantages, using third party vendors to create a performance-based certification is feasible today. The most obvious benefit of this option is that it will save the organization time, money, and effort. A number of the key components of a performance-based certification program can be handled externally. Hale states that everything from operating the program to developing content to maintaining server infrastructure and providing course facilitation can be outsourced for a certification program. The list below presents some of the key steps involved in the development of a performancebased certification program.

10

How to Create a Performance-Based Certification Program for Software and Technology What can be Outsourced? 

Due Diligence: Determining driver, requirements, ensuring stakeholder agreement



Planning and Designing: o

Planning - Mapping out the end state

o

Designing 

Scenario development



Proctoring and administration



Grading 

o

A factor to be considered when comparing vendors. Scaling requires the ability to grade efficiently, often automatically.

Testing (beta) and Launching: 

Internal SME run-through



Pilot program



Outsourcing testing administration and analysis



Hosting/launching can be outsourced on a vendor’s platform



Marketing the launch

According to Judith Hale, most of the steps in a training or testing program can be outsourced with the critical exception of the program’s leadership and processes that are training department strengths. This advice stems from the organization’s need to retain control over the program and budget needs. Outsourcing various steps to external vendors can save an organization crucial time and money. While some companies like RedHat have the internal resources and structure to support creating a program internally, this is not the case for many companies. With the lengthy process involved in creating and launching a certification program, saving effort where possible can help add more time for testing and perfecting the performance-based certification program.

Testing and Launching The final step in the certification program development process is testing and launching. Danielle Urban, a Certification Team Lead at kCura, recommends that organizations dry test the assessment over and over internally and pilot it with a small group. Additionally, Urban recommends that program managers be sure to obtain feedback on the flow and feasibility of the exam and reinvent the assessment as necessary. Symantec’s van de Velde has a similar

11

How to Create a Performance-Based Certification Program for Software and Technology recommendation. Once the exam has been developed, the program sponsor should further validate the exam performance by conducting a beta exam. That would be followed by psychometric analysis, forms equating, and establishing the score required to pass the exam. After thorough testing and revising, a soft launch of the assessment can be completed. Once the test has been launched, other possible considerations are preparing promotion and marketing activities to support the launch of the assessment.

Considerations Some common mistakes that organizations run into when establishing a certification program are: understanding the main reason or driver behind the certification program, underestimating the level of discipline that the certifying body requires, and a lack of reasonable expectations. Creating and launching a performance-based certification program is a major undertaking that requires extensive planning and support. Understanding the amount of work that is required to design and launch an effective performance-based certification program is critical to its success. Some of the specific challenges with developing a performance-based certification include access to SME time, vendor selection, the test item writing process, scalability, and security.

Example Performance-Based Certification Program Ecosystem

Figure 2: PBT Ecosystem Graphic, TrueAbility Inc.

Conclusion Certification programs serve many important functions, like gatekeeping and security when it comes to securing access to sensitive technology. Apart from their usefulness, certifications provide value to certification holders, hiring organizations, and certification bodies. Considering the significant role that certifications play in the world of technology today, ensuring that certifications are reliable and user-friendly is critical. However, not all certification assessments are equally accurate or telling. While some certifications are awarded to candidates completing multiple-choice tests that focus on theoretical knowledge, performance-based certifications utilize assessments that require candidates to demonstrate their skills in an environment that closely resembles on-the-job tasks.

12

How to Create a Performance-Based Certification Program for Software and Technology The process of designing and launching a performance-based certification program closely resembles that of most test writing processes, however, there are many key considerations involved in building this type of exam. Certification program managers have many important questions to answer, such as what steps to complete internally and which to outsource as well as whether to write a multiple-choice test or performance-based scenario. Many of the key considerations stem from the job task analysis process. Writing a performance-based assessment involves answering many complex questions such as how will the test be administered and how it will be graded. Developing and launching a performance-based certification program provides value to all stakeholders. However, the benefits of these certification programs will benefit a company or certification body in the long term.

13

How to Create a Performance-Based Certification Program for Software and Technology

Author/Contributors Written by: Sarah Ann Cantu, Greehey Scholar Contributor: Judith Hale, Ph.D., founder, The Institute for Performance Improvement (www.TIfPI.org), and author Performance-Based Certification: How to Design a Valid, Defensible, Cost-Effective Program Contributor: Rich Raposa, Ph.D., Certification Leader, Hortonworks (NASDAQ: HDP) Contributor: Beverly van de Velde, Global Leader, IT Certification Program, Education Services, Symantec Corporation (NASDAQ: SYMC), Member, Performance Testing Council Contributor: Danielle Urban, Certification Manager, kCura Contributor: Dr. Wallace Judd, President, Authentic Testing, Founder & Member, Performance Testing Council Contributor: Frederick “Suizo” Mendler, CEO/Co-Founder, TrueAbility, Inc., Member, Performance Testing Council

14

How to Create a Performance-Based Certification Program for Software and Technology

About Performance Testing Council This white paper was created with the support of the Performance Testing Council and its members. The Performance Testing Council (http://www.performancetest.org/) is a non-profit consortium dedicated to innovation in the design, development, and delivery of performance testing. We share our experiences, knowledge, and passion in the practice of performance testing and help each other’s testing efforts. Our mission is to Identify, develop and support evaluation of Performance Testing tools, metrics, resources

Encourage and support development of a theoretical base for Performance Testing

Evangelize and educate stakeholders in advantages, value and appropriate use of Performance Testing

The Performance Testing Council goals are to Share Best Practices by creating an Indexed list of best practices in the field of performance testing. Establish Common Delivery Standards for required resources, data transmission, and file formats enabling a single test source to be delivered by multiple vendors. Learn from Experts by having access to materials that introduce authors, item writers, and managers to the field of performance testing. Define and drive research that will resolve the major open issues of performance testing. Expand the Marketplace allowing vendors and suppliers to showcase performance testing tools and services for our members.

15

How to Create a Performance-Based Certification Program for Software and Technology

For More Information TrueAbility is a trusted global performance-based certification and training platform provider, helping companies manage and execute fully scalable, performance-based exams securely from anywhere in the world. To learn more about how your company can leverage performance-based certification and training, visit trueability.com or email [email protected]

www.trueability.com

© Copyright TrueAbility, Inc. 2016. TrueAbility, Inc, 1150 N Loop 1604 W Suite 108-244; San Antonio TX, 78248 Produced in the United States of America. All Rights Reserved. TrueAbility and the TrueAbility logo are trademarks of TrueAbility, Inc. in the United States, other countries, or both. Other company, product and service names may be trademarks or service marks of others.

16

How to Create a Performance-Based Certification Program for Software and Technology

Endnotes 1. Foote Partners, IT Skills Demand and Pay Trends Report, http://www.footepartners.com/fp_pdf/FooteNewsrelease_2Q15ITSkillsTrends_082620 15v1.pdf (July 2015). 2. The 2013 Lumina Study of the American Public’s Opinion on Higher Education and U.S. Business Leaders Poll on Higher Education: What America Needsto Know About Higher Education Redesign, Lumina Foundation, http://www.luminafoundation.org/files/resources/2013-galluplumina-foundationreport.pdf (February 2014). 3. Prometric, IT Certification Research: Results from an Online Survey for Prometric, https://www.prometric.com/en-us/itcc/documents/prometricitsurveyreportd022413v2.pdf (February 2013). 4. 2014 US IT Salary Survey, InformationWeek, (May 2014). 5. Blooms Taxonomy, Anderson, L.W. (Ed.), Krathwohl, D.R. (Ed.), Airasian, P.W., Cruikshank, K.A., Mayer, R.E., Pintrich, P.R., Raths, J., & Wittrock, M.C. (2001). A taxonomy for learning, teaching, and assessing: A revision of Bloom's Taxonomy of Educational Objectives (Complete edition) 6. Greg Shaffer, Certification Exams: Time to Move on From Multiple Choice, Certification Magazine, http://certmag.com/certification-exams-time-to-move-on-from-multiplechoice/ (June 2014). 7. Judith Hale presentation 8. Vanderbilt University Center for Teaching, Bloom’s Taxonomy https://cft.vanderbilt.edu/guides-sub-pages/blooms-taxonomy/ 9. Judith Hale, Performance-Based Certification: How to Design a Valid, Defensible, CostEffective Program (2012). 10. Judith Hale, Outsourcing Training & Development: Factors for Success (2006).

17