2014 IEEE Fourth International Conference on Big Data and Cloud Computing
A Dynamic Social Network Experiment with Multi-Team Systems Andrew Pilny, Alex Yahja*, Marshall Scott Poole
Melissa Dobosh
Department of Communication *National Center for Supercomputing Applications University of Illinois at Urbana-Champaign Urbana, Illinois, USA
[email protected],
[email protected],
[email protected]
Communication Studies University of Northern Iowa Cedar Falls, Iowa, USA
[email protected]
Abstract—This paper describes the use of VBS High-Fidelity 3D Game to perform experiments on multi-team systems. Multiteam systems (MTS) are a natural part of human social phenomena and online social networks as people form groups with shared goals and interests. We gathered data on human players (on communications and interactions) who were engaged in a VBS game scenario. Using Relational Event Modeling (REM), we analyzed the results. The results suggest some synchronization and cross-team communication have both direct effects with team performance and, in some cases, can moderate the effect of false information in environments of uncertainty.
II. DYNAMIC SOCIAL NETWORK EXPERIMENTATION People naturally form groups, cliques, or teams around shared goals or interests: this breaks down social networks, whether online or offline, into multiple smaller groupings of actors. As such, social network research questions can be more precisely framed at the level of multiple interacting smaller groups. Indeed, the interactions among groups are the source of productivity or even survival. For instance, during the Pleistocene epoch, human hunters and gatherers went for mammoths for much-needed meat for food, skins for mats, and ivory for arts. Mammoth hunting is by no means an individual task, it requires multiple action-packed teams working together. In common tribes, an elder wizard team would predict mammoth paths and would talk with a spotter team that performed reconnaissance. Moreover, a wolf-dog team would scare mammoths into a trap and would interact with a spear team who did the actual spearing in a time-sensitve, life-critical choereogaphy. Inspired by the awesome hunting sight, an artist team would later document this exploit in mammoth cave painting and figurines.
Keywords—multi-team system, social network, social media, relational event modeling, sequential structural signatures
I. INTRODUCTION Social networks in real life or mediated by social media involve actions (not just enduring relationships) in activities and may contain groups or teams which can interact with each other. Organizations and businesses are often intentionally forming teams to increase effectiveness and leverage teamwork as the whole is greater than the sum of its parts. The study of how teams interact internallly and with other teams is known as Multi-Team Systems (MTS), a concept articulated by Mathieu, Marks, and Zaccaro [1].
Hence here we operate on the Multi-Team Systems level (MTS-level) in analyzing social interactions and networks with the focus on social actions or sequential sequence(s) of actions. We argue that it is this action or a sequence(s) of actions (a sequence can contain only one action) that constitute a fundamental unit of analysis and the appropriate level of social network analysis. Research on social networks has typically focused on structures of nodes (actors) and edges (enduring relationships) which has produced valuable insights, but to make progress we need to focus on dynamic actions. For instance, the finding that structural holes of leaders in groups were negatively associated with performance [2] is valuable. However, reanalyzing structural holes from an action perspective, rather than a structure perspective, can inform researchers on the dynamics of how structural holes emerge and how this emergence might influence team performance. The action perspective might also explain studies such as one that found that networks and networking do not matter [3] by a revelation that the strength of the strongest ties (which the paper found to be correlated with best team performance) could actually be the strengh of strongest actions. Teams that are worse than the sum of its individual capacities do exist for
In this paper, we report an experiment in how multi-team systems perform in a coordination task scenario using VBS2 High-Fidelity Game. Two groups of two individuals took part in a scenario in which they were required to accomplish intrateam and inter-team goals, possibly under environments of uncertainty. We capture their communication within and between teams and their mission performance. Using Relational Event Modeling (REM), we analyzed the structure of their communication network, the relationship between their communication and performance, and the effects of uncertainty caused by false information. First, we will describe the problem we are addressing. Next, we will describe VBS2 Game, its capabilities, and suitability for multi-team experiments. Then we will describe our experiment setting. Following the description of experiment results, we conclude with assessments, implications, and future work.
978-1-4799-6719-3/14 $31.00 © 2014 IEEE DOI 10.1109/BDCloud.2014.81
587
various reasons (e.g., unclear decision making power, responsibility, and rewards [4]), but with action as a fundamental unit of analysis, we could delve deeper into more complex issues (e.g., what exactly is the decision making power? what decision?). A suite of interacting sequential actions allows for more accurate and measurable reinterpretations of the degree of how strong or weak a tie actually is. Such a perspective redefines previously static views of concepts like centrality, structural holes, cognitive load, and leadership. For example, one of the results reported on this paper, the correlation between synchrony and performance resonates with the results reported in [5], which found that synchronous trading actions and communications correlated with less transaction loss.
Fig. 1. VBS2 High-Fidelity 3D Simulation Game
With VBS2 Mission Editor, we implemented a scenario in which two squads (two teams) in two vehicles, starting from different locations, have a mission of maneuvering safely on the roads to arrive at a rendezvous point about four kilometers away. Each squad consists of a captain (a.k.a. commander) and a specialist (who is also a driver). One squad is called Phantom, the other Stinger; one can have a role of Phantom Commander, Phantom Driver, Stinger Commander, or Stinger Driver. (In the real world, a squad consists of a squad leader and two fire-teams with four persons each, totaling 9 persons, but we simplify the setup to manage the time and cost of the experiments.) VBS2 provides the players physics-based simulations of vehicles and steering wheels to drive as avatars. The VBS2 environment of 3D roads, weapons, buildings, and other objects is also interacted within a similar manner to the real world.
Unlike in the Pleistocene epoch, the observation of social phenomena in the real world can be facilitated by modern audiovisual recordings. This is, however, still time-consuming and ethics precludes experimentation on some real-world social interactions. The focus on dynamic interactions is important: a small change of real-world social interactions that seems harmless can have unintended, emergent consequences (i.e., the butterfly effect). For instance, in the business world, A/B Testing is regularly used to experiment with the responses of social media users for marketing campaigns and custom user interface elements. This A/B Testing on social phenomena via social media, however, could have unintended consequences in the real world (e.g., ethical dilemmas, consumer backlash). A virtual game in which the game mirrors real world situations and social interactions represents a Goldilocks zone in which human experiments can be performed on virtual situations (game scenarios) where the unintended consequences can be drastically reduced as the virtual situations vanish after the game is over and do not carry over into the real world.
The squads need to clear the roads of bombs/IEDs and enemies for emergency supply vehicles to traverse afterwards. The squads are given clues--which may be true or false information—along the roads. They also need to report what they are seeing and get a secret message from an informant along the roads. Finally, they need to coordinate their maneuvers to arrive at the same time near the rendezvous point to maximize their firepower advantage over the enemies near the rendezvous point. Communication is thus vital in coordinating action for the teams.
Research on correlations about the states of social phenomena is valuable, but here we focus on microlevel interactions among group members and among groups which shape the states of social networks. Social actions and interactions shape perceived network structures, which in turn influence the next interactions [6]. Structurational theory of networks provides a similar view of this cyclical process [7].
The scenario is played twice: Mission 1 and then Mission 2 with information accuracy manipulation in Mission 1. An accurate condition means that all of the clues received in the mission (e.g., location of artifacts, presence of bombs) are true. In the inaccurate condition, teams in Mission 1 are given two inaccurate clues (e.g., clues of artifacts when they are in fact not present) or, more accurately, two false information bits. The goal of the manipulation is to increase uncertainty and decrease trust in the MTS.
III. VBS2 GAME SCENARIO VBS2 Game (Bohemia Interactive Simulations https://bisimulations.com) is used to test social interactions as it offers a physically and visually realistic 3D environment. It simulates the kinematics and mechanics of vehicles, persons, projectiles, and other objects in high fidelity. More importantly, VBS2 facilitates the creation and manipulation of mission scenarios so that researchers can experiment with dependent and independent variables. In the game, we record gameplay visuals and communications using Fraps video recording software (Beepa Pty Ltd http://fraps.com). Fraps is capable of recording the gameplay from the viewpoint of each player. As a backup, we also record audio communication using CNR (Calytrix http://www.calytrix.com/products/cnr).
The teams used a chain network for communication where only the captain of a squad can talk to the captain of the other squad while the driver can only talk to his or her captain in the same squad. The scenario reflects a chain of command structure common in military teams. Squads needed to perform three key tasks: (1) Neutralize bombs and enemies along the roads, (2) record secret messages and artifacts (artifact name, time, confirming/disconfirming the presence of the artifact, and location of the artifact), and (3) coordinate to synchronize arrival at the rendezvous point. Tasks (1) and (2) are intra-team tasks (component team tasks), while Task (3) is the inter-team or MTS-level tasks.
The authors would like to thank the US Army Research Lab (ARL) for funding support. The views expressed are those of the authors and do not necessarily reflect the official policy or position of ARL or US Government.
588
from one team to another as a liaison. The captain initiated relay SSS represents the tendency of a captain sending a message to one squad and then another. This represents leadership in cross-squad coordination. Captain popularity is another SSS, which measures the likelihood of a captain to receive a message from one squad and then receiving a message from the other squad.
Participants in our experiments are 52 undergraduate students who formed 26 experiment sessions. On the same day, the students played two game sessions in a row, Mission 1 and then Mission 2. Each experiment session lasted for 30 minutes. Before the experiment, the students went through a 30-minute training session to get acquainted with basic game mechanics. They also filled out surveys before and after each session. Based on the Fraps recordings and squad logs, two human raters score the player performance for speed (how fast a team arrives at the rendezvous area), thoroughness (how accurate a team kept its log), efficacy (how effective a team neutralizes bombs and overcomes enemy ambushes without taking much damage), and coordination (how well teams coordinate maneuvers to sync arrival at the rendezvous point). These are outcomes or performance measures of the teams. The interrater reliability of these outcome scoring were verified to be good with all Cronbach’s alpha equal to or greater than .90.
2.
For synchronization, intra-squad communication is measured for general reciprocity SSS and intra-team synergy SSS. General reciprocity represents the tendency for individuals to reciprocate message. Intrateam synergy is the likelihood that the other squad will communicate within their squad, when one squad previously sends a message within their team. It measures the extent to which teams are sending intrateam messages at similar rates.
IV. RELATIONAL EVENT MODELING A social network is typically modeled as nodes and edges, where nodes represent entities or actors and edges represent relationships. Activities within a social network however consist of actions. Actions can be viewed as discrete events. Relational events are events or actions that depend on other events or actions in the past. Each event is assumed to be independent of all other events except for the sequence of past events that occurred within a short period of time. A significant pattern of actions in the ongoing stream of interaction is known as a Sequential Structural Signature (SSS). For example, if an interaction is heavy on reciprocity where it is likely that actor B replies to actor A after A sends a message to B, then we will have a reciprocity interaction pattern or a Reciprocity SSS.
V. RESULTS Table 1 shows the average parameter estimates from Mission 1 and Mission 2. The parameter estimates are interpreted as logged multipliers for the hazard involving interacting senders and receivers. There are a total of 26 experiment results. On average, certain SSSs were more and less likely to be enacted than random chance alone. For instance, MTSs, on average, did not create structures characterized by inter-squad communication, as evident with the negative average parameter estimates across the boundary spanning leadership parameters; in other words, across captain gatekeeping relay, captain-initiated relay, and captain popularity SSSs. This suggests that MTS or boundary spanning leadership is a learned process that does not come easily, which comes across as no surprise as teamwork is difficult and multiteams working coherently is even harder.
A survival function is used to incorporate past relational events to model and estimate the likelihood of a future event. This method is known as Relational Event Modeling (REM) and is described by Butts in [8] in more detail. Applied to communication interactions, REM computes the estimate of the likelihood of a future event or interaction being associated with a pattern of past interactions or SSSs. The software for REM is implemented in RelEvent, an R package for the estimation of REM http://cran.rproject.org/web/packages/relevent/index.html.
The reciprocity sequential structure signature (SSS), however, had significant positive values (Mission 1 MLE avg. = 0.86, loglikelihood = 2.36, Mission 2 MLE avg. = 0.90, loglikelihood = 2.46). That is, when messages were sent between individuals, they over twice as more likely to be reciprocated. The intra-team synergy SSS has fairly similar values across missions (Mission 1 MLE avg. = 0.28, loglikelihood = 1.32, Mission 2 MLE avg. = 0.12, loglikelihood = 1.13), suggesting that squads engaged in similar intra-team behavior for communication within each squad across missions.
A. Sequential Structural Signatures (SSSs) to Test Based on our experiment results, we evaluated two categories of SSSs: 1.
Synchronization.
Cross-team communication. Boundary spanning activities [9] across teams represent cross-team communication, including the captain relaying messages across teams, the captain initiating messages to teams, and the captain receiving messages from teams. Table 1 shows the illustration with the arrows denoting the direction of message sent and C denoting the captain. The captain gatekeeping relay SSS describes the tendency--the significant interaction sequence--for captains to relay messages
589
TABLE I.
MLE ESTIMATES OF SSSS
TABLE II.
Average MLE Estimates
SSS
Counts
Mean
Std. Dev.
(pos.): 4 (neg.): 8
M1: -0.22 M2: -0.35
M1: 0.47 M2: 0.73
(pos.): 2 (neg.): 11
M1: -0.38 M2: -0.52
M1: 0.69 M2: 0.70
(pos.): 2 (neg.): 7
M1: -0.12 M2: -0.52
M1: 0.42 M2: 0.97
SSS
Captain gatekeeping relay
Captain-initiated relay
Cross-team M1
Synchronization M1
Cross-team M2
Synchronization M2
.80
.34
.70
.31
.86
-.09
.90
.22
.78
.06
.93
.09
.01
.95
.13
.88
.14
.91
.18
.87
Captain gatekeeping relay
Captain-initiated relay
Captain popularity
General reciprocity
EXPLORATORY FACTOR ANALYSIS
Captain popularity
General reciprocity
(pos.): 24 (neg.): 0
M1: 0.86 M2: 0.90
M1: 0.43 M2: 0.33 Intra team synergy
Intra-team synergy (pos.): 14 (neg.): 4
M1: 0.28 M2: 0.12
M1: 0.53 M2: 0.47 Note: Loadings over .40 are bolded
M1 = Mission 1, M2 = Mission 2, (pos.) = number of significant positive estimates, (neg.) = number of significant negative estimates
To estimate the effects of each relational event dimension on team performance, we used advanced moderation analysis developed by Andrew Hayes [10] in PROCESS, a macro extension for SPSS and SAS. More specifically, we used Model 2 as described in Figure 2, which estimates both direct and moderation effects of cross-team communication and synchronization.
The high standard deviation values--most were nearly double the mean—suggested that MTSs were heterogeneous with the SSSs they created. In other words, there was a lot of variability in SSSs. To understand the relationship between SSSs and team performance, we conducted an exploratory factor analysis (EFA) to reduce the SSSs into a manageable set of dimensions. Using varimax rotation, the results of the EFA clearly suggested that two factors, what we called synchronization and cross-team communication, are evident in the data. Table 2 shows the factor loadings across each mission, suggesting that MTSs were consistent in their patterns of communication interactions.
In Figure 2, we let X represent the manipulation effect of information accuracy. M and W represent the factor scores of cross-team communication and synchronization. This way, we can see if those two factors are not only related to team performance (Y), but also if they moderate the effect of the main information accuracy manipulation.
To create new factor variables corresponding to synchronization and cross-team communication, we saved the regression coefficients from each team on each factor. As such, a higher factor score corresponding to the two factors indicates that the MTS had higher MLE estimates for those SSSs, while a lower score indicates the opposite.
590
Efficacy M1
Thoroughness M1
Coordination M2
Efficacy M2
Thoroughness M2
MODERATION ANALYSIS PREDICTING TEAM PERFORMANCE
Coordination M1
TABLE III.
3.2**
.74**
.67**
3.59**
.86**
.83**
Cross-team factor score
.92**
.01
.05
.53
-.08
.02
False information
-.93*
-.37*
-.24*
-.66
-.18†
-.15
-.03
.02
.21**
.42
.32**
.28
.83**
.04
.16**
.70*
.05
.04
False information X Synch. factor score (Int2)
.03
.04
-.02
.04*
-.19*
-.07
R2
.62
.76
.64
.51
.38
.33
R2 due to Int1
.01
.00
.14**
.01
.22**
.09
Variable
Constant
False information X Crossteam factor score (Int1) Synch. factor score Fig. 2. Conceptual and Statistical Models
Our results suggested that the SSSs corresponding to crossteam communication and synchronization had some positive relationships with performance outcomes. For instance, crossteam team coordination ( = .92, p < .01) and synchronization ( = .83, p < .01) had a positive relationship with coordination in Mission 1, while synchronization had additional positive relationships with thoroughness ( = .16, p < .01) and coordination in Mission 2 ( = .70, p < .05). Perhaps more interesting is the moderation effects of the relational event patterns, as shown in Table 3. For instance, the main manipulation of false information had negative relationships with all the performance metrics in Mission 1, but only slightly with efficacy in Mission 2. In some instances, relational event patterns were able to mitigate the effects of the inaccurate information condition. For example, the interaction between cross-team communication and false information had a positive relationship with thoroughness in Mission 1 ( = .21, p < .01), suggesting that teams that frequently communicated with one another were able to do better when they received the false information condition. The same relationship was true with efficacy in Mission 2 ( = .32, p < .01). Indeed, the interaction added an extra 14% and 22% explanation in variance in each case.
R2 due to Int2
.01
.01
.01
.00
.15*
.01
R2 due to both
.02
.01
.15*
.01
.28*
.09
噛
p < .10, * p < .05, **p < .01.
On the other hand, while synchronization had positive direct effects with team performance, its interaction with the false condition was less obvious. For instance, whereas the main effect of the false information condition had a negative effect on efficacy in the second mission ( = -.18, p < .10), the effect increased as teams were more synchronized, as evident in the negative interaction effect ( = -.19, p < .10).
591
first mission. To bring this finding into more light, consider the following cross-team interaction and how it can reduce uncertainty in team: PC to PD: I just called that in Phantom Driver. PC to SC: Stinger Commander, how is your first task going? SC to PC: One second here, just gotta get longitude and latitude. Umm, we had a donkey cart in the middle of the road, but we recorded it and everything is good to go. PC to SC: Yea we had a fire extinguisher. SC to PC: 10-4, we are ready to move on. PC to PD: Phantom Driver, it’s a little more up here [IED], but don’t move too much further. Here, we can see the leadership role enacted by PC, not only directing his own squad’s driver, but also checking in the other squad’s progress. While these SSSs might be overlooked, they may reduce uncertainty and help make sense of the tasks in the mission. These structures were common in this particular MTS and might represent one of the reasons why this MTS scored higher in thoroughness across both mission one (Thoroughness = 0.79, average = 0.67) and mission two (Thoroughness = 0.94, average = 0.76).
Fig. 3. Interaction Plots on Efficacy
There are however several limitations in our current work. While Butt’s model [8] pioneered formal modeling of dynamic relational events, other researchers have described some of its shortfalls and suggested improvements: Stadtfeld [11] enhanced REM with stochastic actor-oriented modeling, Quintane et al. [12] allowed exploitation of full information in relational event sequences, and Leenders et al. [13]. Hence we will employ more refined models as our next step. Research on multi-team systems requires many observations and good analyses of relational event data. In our experimental setting, our sample size was limited to 26 missions, leaving us with a limited set of tools for traditional inferential analysis. We may also reconstruct game logs (VBS2 game logs are encrypted) to automated raters rather than human raters, or use different game engines such as Unity http://unity3d.com.
The interaction effects are plotted in Figure 3 to demonstrate the effects of the information manipulation and relational event pattern interactions on team efficacy. These results demonstrate that some effects could carry over to social cooperation and social networks domain. The synchronization SSSs correlation with multi-team performance indicates that the performance of social networks could be improved with general reciprocity of messages and actions and with internal team synergistic communications. The cross-team communication SSSs (captain gatekeeping relay SSS, captaininitiated relay SSS, and captain popularity SSS) correlate with the ability of the teams to handle false information suggests that in social network groups should talk with other groups, spanning boundaries with high betweenness centrality ideally, to mitigate the uncertainties and false information in the environment.
Our finding of the relationships between SSSs and multiteam performance and the moderation effects of certain SSSs could imply that the current emphasis on relational structures or relationships for social cooperation and networks are misplaced. The problem might not be in relationship but in action, or rather, a sequence of actions. Rather than social network, we should be talking about social action or, more precisely, social sequential structural signatures. Social sequential structural signatures within and among teams give rise to networks within and among teams that influence MTS [14]. There is a vital need to understand and design good MTSs to make human organizations and societies work better, and SSSs of dynamic actions and communication acts provide a pathway to achieve this better than so far unquestioned quasistatic definition of relationships in conventional social network analysis.
DISCUSSION With VBS2 High-Fidelity 3D Simulator, the current research investigated the SSSs that emerge in MTSs and the relationship between those SSSs and multi-team performance outcomes. Our results indicate some of the positive direct effects on multi-team performance of synchronization and cross-team communication. These direct effects were more likely to occur when teams were communicating within teams and in synchrony with one another by reciprocating messages at similar rates. Perhaps more interesting is how relational event patterns or SSSs might influence performance when teams are embedded in environments of uncertainty. For instance, consider our findings on how cross-team communication moderated the effect of the false information condition on thoroughness in the
592
REFERENCES [1]
[2]
[3]
[4]
[5]
[6] [7]
[8]
[9]
J. Mathieu, M.A. Marks, and S.J. Zaccaro, S. J. “Multiteam systems,” in International handbook of industrial work and organizational psychology (Vol. 2, pp. 289-313). Thousand Oaks, CA: SAGE, 2001. J.N. Cummings and R. Cross. “Structural properties of work groups and their consequences for performance,” in Social Networks, 25(3), 197210, 2003. Y-A de Montjoye, A. Stopczynski, E. Shmueli, A. Pentland, and S. Lehmann, “The Strength of the Strongest Ties in Collaborative Problem Solving,” Nature Scientific Reports, June 2014, doi:10.1038/srep05277 J.R. Hackman, “Why Team Don’t Work,” in R.S. Tindale et al. (editors), Theory and Research on Small Groups. New York: the Plenum Press, 1998. S. Saavedra, K. Hagerty, and B. Uzzi, “Synchronicity, instant messaging, and performance among financial traders,” in the Proceedings of the National Academy of Sciences, vol. 108, no. 13, pp. 5296-5301, March 29, 2011. K.M. Carley. “A Theory of Group Stability,” American sociological Review, 56(3), 331-354, 1991. S.R. Corman and C.R. Scott. “Perceived networks, activity foci, and observable communication in social collectives,” Communication Theory, 4(3), 171-190. doi: 10.1111/j.1468-2885.1994.tb00089.x, 1994. C.T. Butts. “A relational event framework for social action,” Sociological Methodology, 38(1), 155-200. doi: 10.1111/j.14679531.2008.00203.x, 2008
[10] [11]
[12]
[13]
[14]
593
R.B. Davison and J.R. Hollenbeck. “Boundary spanning in the domain of multiteam systems,” in S. Zaccaro, M. Marks & L. DeChurch (Eds.), Multiteam systems: An organization form for dynamic and complex environments (pp. 323-364). New York: Routledge, 2011. A.F. Hayes, “Introduction to mediation, moderation, and conditional process analysis,” New York: The Guilford Press, 2013. C. Stadfield. “Events in social networks: A stochastic actor-oriented framework for dynamic event processes in social networks,” Scientific Publishing, 2012. E. Quintane, G. Conaldi, M. Tonellato, and A. Lomi, “Modeling relational events: A case study on an Open Source Software Project,” Organizational Research Methods, 17(1), 23-50. doi: 10.1177/1094428113517007, 2014. R. Leenders, L.A. DeChurch and N. Contractor. “Once upon a time: Understanding team dynamics as relational event networks,” Organizational Psychology Review, 1-40, 2013. M.S. Poole and N. Contractor. “Conceptualizing the multiteam system as an ecosystem of networked groups,” in S. J. Zaccaro, M. A. Marks & L. A. DeChurch (Eds.), Multiteam systems: An organization form for dynamic and complex environments (pp. 193-225). New York: Routledge, 2012.
