SHADOWBOXING WITH DATA: A FRAMEWORK FOR INFORMING THE CRITICAL ANALYSIS OF PERFORMANCE AND PERFORMANCE MEASURES Donald J. Winiecki
The practice of performance improvement requires measuring before and after conditions to determine if changes have occurred as a result of an intervention. Understanding how to take, make, interpret, and use measurements can go a long way toward improving performance improvement work and improving conditions for clients. The risk of not adequately analyzing measures leads to shadowboxing with data, where performance measures may not be equated with authentic performance issues.
ARE WE ACTUALLY MEASURING what we purport to be measuring? Here I would like to provide some new tools and a framework that will guide performance improvement practitioners in the analyses of a workplace. Called shadowboxing with data, this framework points up several practices of management and labor that can obscure what is actually being done while at the same time producing what appear to be good numbers in evaluations of workplace conduct. Shadowboxing with data was produced from critical social science research and was initially applied to model the actions of members of workplace organizations typified by multiple layers of continuous measurement and evaluation processes (Winiecki, 2006a, 2008). Here, I propose its use as a framework to guide the practice of rooting out hidden issues.
IDENTIFYING THE CONCEPT: SHADOWBOXING WITH DATA One of the most common and most economical means for facilitating performance is to provide workers with data, information, and feedback. When we do so, we are also telling them what we expect, how their conduct will
be measured, and how they are seen in terms of the metrics and measures. With all of this, we expect that people will modify their behavior so as to accomplish their work in ways that satisfy the organization’s expectations, which is reflected in evaluation evidence. As we continue to provide data, information, and feedback, we expect the recipients to continually refine and improve their conduct in accord with what we provide to them. In other words, we expect that the measurements we reflect back to them are accurate reflections of what the people are actually doing (this is the common issue of validity in measurements) and they are altering their conduct to improve their practice in the way we define improvement rather than to accomplish some other ends. It is the second of these that I prioritize here. But first I wish to address a contemporary definition of rationality and rational action as it exists in economically influenced fields like performance improvement. Rationality is thought and action that follow a regular pattern that may be abstracted into rules and provide individuals with increased or perhaps maximized payoff or reward for the effort expended in the action context. In this formulation, payoff or reward need not be limited to monetary items. Anything that an individual considers to be
Performance Improvement, 48, no. 2, February 2009 International Society for Performance Improvement Published online in Wiley InterScience (www.interscience.wiley.com) • DOI: 10.1002/pfi.20044 ©2009
31
rewarding he or she will do, whether it is a reduction in the difficulty of effort, avoiding some undesired state, exerting power over others, maintaining status differentials, or these sorts of things in combination (Coleman, 1990). Returning to the main issue, following Gilbert’s behavioral engineering model, we expect that providing individuals with a systemic array of environmental supports—including instrumentation and incentives and perhaps training and personnel selection for desired capacities and motives—will lead to some measurable improvement in practice or decrease in problems or errors. We determine this from a calculation of worth in terms of a ratio of costly inputs to valued outputs or outcomes (Gilbert, 1996). While this is a very conventional and respected set of ideas and practices, it also ignores something that we should be especially attuned to: the actual practices of individuals whose actions are supposed to be the target of measurement and evaluation. That is, the numbers that stand for measurements are assumed to be accurate transcriptions of conduct. This is, unfortunately, not a very safe assumption. It is not a safe assumption when impediments to performance exist, owing to the highly creative, emergent, and contextually dependent practices of rational individuals who may be modifying their behavior to produce the numbers or measurements they want or which are desired by their supervisors, rather than producing performance that is accurately transcribed into measurements (Winiecki, 2006a, 2006b, 2008). The result is what may be called shadowboxing with data. In shadowboxing with data, individuals who know what sorts of measured outputs are desired (by themselves or their organization) may modify their practice so as to produce “good numbers” rather than what we consider to be “good performance.” But because the numbers are assumed to stand for performance, a performance improvement practitioner may miss the process and consider everything to be operating as expected when, in fact, improvements may not only be possible but important to the well-being of individuals and the organization. You might ask how producing good numbers might vary from producing good performance. There are lots of examples in contemporary society. One is high-stakes testing in schools, where teachers’ jobs might be threatened if students do not pass tests with administratively defined score thresholds. This situation can quite reasonably result in the use of instructional methods designed to increase the likelihood of students’ reaching those scores despite limited support from parents and very limited instructional resources and time—that is, teaching to the test. In this example, we see teachers taking the limited support and resources they are provided and producing what
32
www.ispi.org
•
DOI: 10.1002/pfi
•
FEBRUARY 2009
appears to be evidence of useful learning. Metaphorically, they project this evidence (the test scores) on a sort of screen that obscures or produces an abstraction or a shadow of what they are really doing. Because we do not see the details of what teachers or students are actually doing, we interpret the shadow or numbers to mean what we assume test scores are supposed to mean (evidence of valued learning) and we react (or shadowbox) with the numbers in a way that reifies what we want to believe rather than what might actually exist, missing an opportunity for performance improvement in the process. Examples that are closer to the normal work of a performance improvement practitioner might help to clarify the concept and bring us closer to being able to apply the framework in our work. The following examples are taken from a multiyear ethnographic research project conducted in call centers (see Winiecki, 2006a, for details on this research and other findings from it). All corporate and personal names in these examples are pseudonyms.
MANAGEMENT: WITHHOLDING RESOURCES AND PRACTICING DESPOTIC HEGEMONY TO GET MORE FOR LESS DeliveryWorldwide is an international package delivery company with eight domestic call centers and several located in Europe and Asia. A substantial component of DeliveryWorldwide’s business success was built on overnight delivery of documents across great distances. The increasing use of email to deliver documents has cut into the company’s core business, and the economic downturn has further reduced the amount of business the company does with its traditional clientele. As a result, it has closed several of its call centers, and there are fears that the company will move other call centers to low-cost centers abroad. This turn of events has produced several outcomes for agents at several of the domestic call centers operated by DeliveryWorldwide. First, incoming calls have increased by as much as 50% because calls formerly routed to the closed call centers are now routed to centers that are still open. Second, employees are told that the training and equipment budgets have been cut and employees no longer receive instruction and guided practice on new software tools deployed by the company. They have also been told that their already obsolete desktop computers will not be replaced for at least the next two fiscal years. These old PCs are substantially slowed by increased use of Web-based software and tracking systems, which adversely affects agents’ ability to work calls within the company’s stipulated time frame. Finally, employees are
told that the company has cut the bonus program through which outstanding employees are rewarded for their performance. I observed the manager of one of DeliveryWorldwide’s call centers emphatically urging agents to work faster despite the lack of instruction and adequate equipment and incentives, and obliquely threatening employees with job loss if statistics showed that they were unable to keep up with the quantity of incoming calls. (The threats are couched in reference to “market forces” and economic hardship on the part of the company, a practice known as “despotic hegemony” [Burawoy, 1983, p. 603, in Littler, 1990, p. 62].) Faced with a very real threat to their employment, agents largely responded by increasing their measures of productivity, but not without substantial cost in terms of stress, an increase in health problems, and associated health insurance claims, and other tactics to cope with the situation. Leaked documents and information from high-level company insiders also showed that the company’s CEO was courting a buyout from a larger delivery company and sought to produce evidence that the company was able to handle increased customer service demands with a smaller workforce. In this case, the company CEO may have manipulated information and practices to facilitate personal profit, gains, and self-interests. Employees and even the eventual parent company were made to shadowbox with data and respond to what appeared to be the case rather than what actually existed. While the long-term effects are palpable but unmeasured, the short-term and self-interested actions of individuals hurt employees and their willingness to perform, producing an ongoing fatalism and unwillingness to perform customer service. This is evidenced by the following episode. Lonnie, a new team leader but an individual who has been working at this DeliveryWorldwide call center for over five years, had three new agents on her team. She asked that all of them be assigned to cubicles within earshot of her cubicle, so, as she joked, “I can bring ’em up right.” In many cases, the advice Lonnie provided to these agents had to do with more efficient uses of the databases for looking up and performing data entry tasks. However, in many other cases, her advice oriented to more affective matters. One day, Bambi, the newest of these agents, was especially exasperated after a long call with an angry customer. When this call ended, she let out a loud and anguished sigh. Lonnie, working on performing a quality evaluation of recorded calls, shut off the tape recorder and motioned to Bambi to come to her desk. As Bambi approached, Lonnie said, without looking up from the form she was writing on, “You care too much.” Bambi asked, “What do you mean, I care too much?” (Lonnie exhibits “not caring too much” in her conduct.)
L: Just what I said. You care too much. B: Yeah. I heard you, but what do you mean? Aren’t we supposed to help them? I get frustrated when I can’t, and especially when they’re mad at me for not being able to do anything! L: Yeah, sure. But you’re getting frustrated because you can’t help them the way you want to or the way they want you to. You’re here to help them the way the company wants you to [help them]. B: [But] I wanna help them. That’s what I’m here for, aren’t I? To do customer service? L: No. You’re here to do it fast, and that’s all. You’re here to do company service. You gotta stop caring. Just then, Fiona, another agent on Lonnie’s team who sits adjacent to Lonnie’s cubicle, finished a call and started to collect her belongings to go on her lunch break. As she stood up, she entered the conversation: Fiona: Yeah. You can’t care about ’em [the customers]. The easiest thing to do is just punch in and punch out. Turn off your emotions when you come in. Just do the job. Bring in your trashy novels to read during lunch and breaks and don’t care. Look at Toni, Blaise, and Pat [three other agents who practice what Lonnie is describing]. They just check in and check out. When they punch in, they leave [their emotions] at the door. When you [work for DeliveryWorldwide], that’s how you survive. Long-term employees have learned not to care about their work; instead they perform work robotically and in many cases do not actually answer the customer’s questions—the single most common complaint by customers to call centers. What the company rewards is not what customers say they want, but workers make the very rational decision to robotically follow rules despite the costs to them and perhaps the customer. While the costs to long-term performance may be plain once such information is uncovered, the performance technologist who looks at statistics alone as a measure of performance would see acceptable outputs and perhaps find no reason to look further.
Labor: Accounting for Unavoidable Problems to Reflect the Agent’s “Real Ability” At DeliveryWorldwide, the official evaluation of employees is conducted by one person: the call center supervisor. Evaluations are accomplished using several forms of data: • Data collected from a computer system that both channels calls to agents and continuously observes the Performance Improvement
•
Volume 48
•
Number 2
•
DOI: 10.1002/pfi
33
agents’ work, computing statistics like the average amount of time each agent takes to complete calls, the amount of time each agent is logged in and ready to accept calls, attendance records, and the like • “Quality” evaluations of an agent’s work that are accomplished by individual team leaders in the call center, who are supposed to follow a rubric created by the company • Subjective assessments of each agent’s conduct in seven categories (time management, job knowledge, productivity, quality, communication skills, professionalism, personal responsibility) by the supervisor Focusing on the quality evaluations performed by team leaders, I observed a team leader, Hal, conduct quality evaluations. After tapping into the phone of an agent with nearly six years of experience, he waited until a new call started and began quickly scribbling notes on the evaluation form. After the call was completed, he remarked to me: That wasn’t a very good [call]. She provided incorrect information when she answered the customer’s question about shipment of live animals. The only animals we allow are turtles—don’t ask me why. I don’t know. It’s not a very common question and the company keeps changing their policy so that’s not too big a deal—but it’ll hurt her.
As the next call began, Hal started scribbling notes again. From the team leader’s desk, we could hear her typing very firmly (hard!) on her keyboard and telling the customer in a very tired-sounding voice, “I’m sorry but my computer’s acting up again. . . . This should only be a minute.” After a few seconds we heard her say in a loud and anguished voice, “I HATE MY COMPUTER!” Hal nodded. “[It] sounds like she’s having a really bad day.” (Hal indicated that he could hear a change in the hiss in the phone line, indicating the agent had pressed her mute button. Thus, the customer did not hear her exclamation.) Just then, the agent’s team leader walked up behind us and, noticing her name on the form Hal was writing on, said, “Oh! Evaluating my girl! How’s she doin’?” Hal remarked, “Not too good.” Then crumpling the rating form, he said, “I think we’re done for the night.” When I asked about this, Hal told that it was late in the day and at the end of the agents’ workweek—a time, he assured me, when any agent really starts to show fatigue. “Plus, her computer is acting up, and it’s probably not fair to do a rating of her today. She’s a good agent, but things just aren’t good right now.”
34
www.ispi.org
•
DOI: 10.1002/pfi
•
FEBRUARY 2009
At the most, only 10 calls per agent are rated for quality each month at DeliveryWorldwide. Agents at this call center are under pressure to answer more calls than ever before, and they commonly take up to 200 calls each day (around 4,000 per month) without the benefit of adequate environmental supports. Thus, quality evaluations account for only about .0025% of the calls worked by the agent. Therefore, an otherwise competent agent’s monthly quality rating could be severely harmed by rating calls performed in a brief period when both technical problems and understandable fatigue are impeding performance. Given these conditions, it is entirely rational and understandable that Hal, or any other team leader, would employ his or her personal knowledge of a worker’s overall knowledge, skill, and attitude when creating (or not creating) empirical evidence of an agent’s work. Any performance issues in this situation are not appropriately leveled at the workers but rather at the workplace environment. However, simply looking at the evaluation data produced would not allow one to identify that anything is amiss.
Labor: Overworking to Ensure the Numbers Look Good MHealth is a small call center operated as part of a regional health insurance company. With only four to five agents (depending on the time of day), small issues can rapidly snowball into problems that affect all agents and perhaps bring the call center to a standstill. Having won a contract to service a portion of the insurance claims of a large state organization, the company had to agree to contract stipulations indicating that it had to maintain a particular service level. (Service level is the percentage of incoming calls answered within a fixed period of time. It is always expressed as a ratio of the percentage of incoming calls answered in the required period. For example, if 80% of incoming calls are to be answered within 15 seconds of ringing, service level is expressed as 80:15 or “80 in 15.”) Shortly after winning this contract, the computer system servicing the call center began operating so slowly that agents could not complete the data entry for a call quickly enough to ensure the call center would meet its contractual obligations for service level. Rather than 80:15, the service level was in some instances 10:120. When Kam, the call center manager, was told that the computer system would not be replaced any time soon for financial reasons, she implored agents to be vigilant of their time working each call so they could meet the required service level. At one staff meeting, she harangued the staff, pointing at a printout of the previous day’s call statistics (which documented failure to meet any of the organizational expectations):
You have to be able to see yourself in the data. . . . This [stabbing the page with her finger] is how we see you, and this [continuing the stabbing gesture] is how you should be seeing yourselves. Each of you knows better than I do what’s going on minute by minute and if you’re aware of how we’re seeing you, you can adapt to the current conditions in order to make sure the stats come out right!
In other words, rather than supporting worker performance with adequate instrumentation, the organization threw responsibility back onto the employees. In fact, employees did adapt to the situation where the computer system could not keep up with the speed demanded of them in their work. Several agents independently produced a paper form on which they could hand-write the data required by the database more quickly than if they used the computer during a call. When call volume decreased or during breaks, lunch, or even after punching out of work for the day, these agents would enter the handwritten data from their paper forms into the database. They shared their forms with other workers and collectively were able to regularly approach or occasionally even meet the service level demanded by the contract. While supporting the practice, Kam distanced herself from responsibility, and in fact it was the workers themselves who overworked in the face of her harangue to “adapt to the current conditions to make sure the stats come out right.” Even after the company’s financial crisis subsided, workers were continually expected to adapt in this way and work extra time without compensation. The company officer ultimately responsible for the call center explicitly refused to repair or replace the computer system, saying that “the call statistics look good! There isn’t any problem here; otherwise, it would show up in the stats.” In fact, the problem did show up, except it did not show up in what the company considered important data. Attrition in the call center was very high: 200% in the year in which the episode occurred. Although the statistics continued to look good for the effort of the agents, the added stress and fatigue that agents suffered was very high as they waited for the company to hire new agents, train them, and then wait for these new agents to come up to speed. The cycle of attrition continued as management allowed itself to shadowbox with data, unwilling to pay attention to how those data were actually produced.
Labor: Hiding Overwork to Make Yourself “Look Good” Workers often criticize their organizations and management for being overbearing, thoughtlessly demanding, and
concerned only about the bottom line at the employees’ expense. Although this may very well be the case in some instances, it has been my experience that while management is indeed oriented to the bottom line, many managers and administrators are not happy that economic necessities of business may contribute to decreased quality of life for their employees. Regardless, workers often respond to management’s despotic hegemony in ways that amplify their own stresses. Consider the following example. BigTech is a multinational computer equipment manufacturer that operates several call centers in the United States and abroad and is aggressively seeking to outsource almost all of its customer-facing work to contractors in South Asia. Sal works as a case manager in one of BigTech’s U.S. call centers. Case managers are a combination of secondlevel support (addressing more difficult issues that could not be solved by agents who directly accept customer calls) and fulfillment officer who schedules on-site repairs and delivery of replacement parts and equipment to the customer’s site. Aware of the outsourcing initiative, Sal wants to maintain employment and sees the best way to do this is to be promoted into managerial ranks where he would manage distributed projects. However, to do so, he has been told by his supervisor that he must produce an array of personnel evaluations and statistical evidence of high productivity over a prolonged period, which purport to document his ability to manage many tasks on deadline and under budget. Despite his supervisor’s advice, what Sal decides to do is to overwork and accept undercompensation with the idea that it will allow him to produce the numbers the company wants to see and eventually put him into a position where he can move into a different job category and bolster his long-term employability. He does this by obscuring from the official record the facts of his conduct. This will, he believes, make him look like “management material” because the records will show that he is doing more work in less time and for a lower budget than others. Sal’s actions are seen to be rational on the grounds that he believes reporting his overworking and being fully compensated for his labor are the result of fears about losing his job altogether. In his actions, Sal makes the company shadowbox with data he has intentionally fabricated but that do not reflect the truth about his practices. While one could say that overworking and accepting undercompensation for his labor is Sal’s own doing, the choice is perhaps the least distasteful from among a set of bad options—options that a performance-oriented workplace should perhaps not present to a worker. This is especially the case when one considers Gilbert’s (1996) use of the term leisure: the ability to decide what to do
Performance Improvement
•
Volume 48
•
Number 2
•
DOI: 10.1002/pfi
35
The numbers are treated as the principal point of interest. While one cannot say that the work itself is unimportant, it is apparently not as important as the numbers.
when one has a choice. In fact, while Sal technically has a choice of what to do, he considers his options to be very limited, and he chooses the least-bad option he thinks is available rather than the best alternative. Quite different from the aphorism that what gets measured is what gets done, in all of the examples provided here, individuals set about to produce what is imputed to be evidence of good performance—evidence that appears in the form of statistics—but not necessarily to produce good performance itself. More than the question of validity, these examples document how individuals can affect the numbers when they combine their knowledge and understanding of the assumed importance and credibility placed on statistical representations of work, with particular desires that lead them to produce “good numbers” but not necessarily good or desirable performance. With these situations, those individuals not only affect what is apparently happening but also obscure authentic performance problems and make performance improvement a difficult proposition (Winiecki, 2006b).
contingencies they determine actually exist, and adopt actions they either expect or have learned will provide some benefit, even if that benefit is not the most desirable outcome or is gained by deceit. As evidenced above, the contexts, contingencies, and outcomes of these actions are widely variant and often serve neither the individual nor the organization very well in the long term. In all cases, however, “good numbers” are influential because these individuals have learned that their colleagues and organizations seem to have a propensity to shadowbox with data rather than actually support good performance. Generalizing from the examples provided above allows us to create the diagram in Figure 1 and an explanation of shadowboxing with data. Figure 1 schematically represents a collection of data (the sphere, labeled C) visible through a semitransparent screen. However, while the data C is visible as a single unit through the screen, the schematic shows that the appearance of data is actually the creator’s desired affectation (A) of existing performance in the form of data (B). Figure 1 displays A, B, and C as one unit because the actual mechanisms through which they are combined and presented are highly variable. One has to be skeptical of the data and look at the processes by which they are produced to determine if they are legitimate reflections of performance. The main idea is that if one simply accepts the unitary appearance of evaluation data (that is, as the data are made to appear by the creator), the viewer can be made to shadowbox with it or act as if the creator’s desired affectation of the data is in fact real. When the viewer accepts the creator’s affectation, his or her actions in accord with
THE FRAMEWORK AND ITS USE: SHADOWBOXING WITH DATA All of the examples share several features. First, in all cases the numbers are treated as the principal point of interest. While one cannot say that the work itself is unimportant, it is apparently not as important as the numbers to the individuals in the situations described. Second, the people and organizations reflected in these examples are acting within imposed constraints to fulfill what they decide to be the right thing (for whatever reason). These rational individuals interpret the context in which they are working and respond to the system and
36
www.ispi.org
•
DOI: 10.1002/pfi
•
FEBRUARY 2009
FIGURE 1. DIAGRAM OF SHADOWBOXING WITH DATA
it will serve to substantiate it—to make it real—in ultimate effects (Merton, 1949). Such perceptions and actions following the creator’s constructions serve as the foundation for action that effectively obscures the creator’s creativity and makes his or her desires real. This phenomenon challenges the performance improvement professional to remain skeptical of what the data appear to indicate and to look into the processes by which data are produced. This skepticism is akin to a critical social science orientation—an orientation that does not try to deny the value of data but rather allows one to understand the data from their very foundation and methods of formation prior to accepting them or not and perhaps acting to change the situation in response. By understanding the foundation and methods of formation for data, the performance improvement professional can see through any shadowboxing screens that may have been erected by members of an organization and, in so doing, activate the potential to engineer worthy performance for individual workers, organizations, and society at large (Winiecki, 2008).
WHAT THIS IS AND WHAT IT IS NOT The framework made possible by all of this is not a model or a theory––it does not claim to explain, predict, or control and it does not tell us what to do or how to do it. Rather it is a framework or a tool to think with that may help one organize one’s practice in new ways as one goes
about the work of analyzing organizations to identify points at which improvements can be made.
References Coleman, J. (1990). Foundations of social theory. Cambridge, MA: Harvard University Press. Gilbert, T. (1996). Human competence: Engineering worthy performance (tribute ed.). Washington, DC: International Society for Performance Improvement. Littler, C. (1990). The labour process debate: A theoretical review, 1974–88. In D. Knights & H. Willmott (Eds.), Labour process theory (pp. 46–94). London: Macmillan. Merton, R. (1949). Social theory and social structure: Toward the codification of theory and research. New York: Free Press. Winiecki, D. (2006a). Discipline and governmentality at work: Making the subject and subjectivity in modern tertiary labour. London: Free Association Books. Winiecki, D. (2006b). Systems, measures and workers: Producing and obscuring the system and making systemic performance improvement difficult. In J. Pershing (Ed.), The handbook of human performance technology (3rd ed., pp. 1224–1250). San Francisco: Jossey-Bass/Pfeiffer. Winiecki, D. (2008). An ethnostatistical analysis of performance measurement. Performance Improvement Quarterly, 20(3/4), 185–209.
DONALD J. WINIECKI, EdD, PhD, is a professor in the Instructional and Performance Technology Department and an adjunct professor of sociology at Boise State University. His doctor of education is in instructional technology, and his doctor of philosophy is in sociology. His research focuses on the practices, and the affects and effects, of measurement and technologies in social systems. He teaches courses on ethnographic research in organizations; needs assessment; and science, technology, and engineering in society. He may be reached at
[email protected].
Performance Improvement
•
Volume 48
•
Number 2
•
DOI: 10.1002/pfi
37
