Knowledge creation processes in Information ...

Knowledge creation processes in Information Systems and Management: lessons from simulation studies Stefano ZAa,1 and Paolo SPAGNOLETTI a a

CeRSI – LUISS Guido Carli University, Rome, Italy

Abstract. This paper aims to contribute to the debate on the relationship between Information Systems (IS) and other Management (MGT) fields of studies. We present the preliminary results of the publication statistics, co-citation, and cross-citation analyses performed on a dataset of 54 and 169 “simulation related studies” published in top IS and MGT journals in the last thirty years. The analysis shows that this stream of research allows to comparatively analyze the evolutionary trends of research outcomes and impact, and to better understand the cumulative tradition of the knowledge creation process in the IS and MGT communities. Although research in IS is unlikely to build on and cite prior research, its influence on other fields is proportionally similar in magnitude to the external influence of MGT studies. Keywords. simulation, cross-citation, network analysis, hierarchic complex systems

Introduction The relationship between Information Systems (IS) studies and other sub-fields of management has been widely debated with some controversial positions. This debate started at the infancy of the IS field, when the concepts of “work points” and “reference points” have been introduced to refer to the structure and the nature of the field [1]. Some authors have advocated a two-way relationship in which IS plays the role of both a referee discipline and a reference discipline [2]. Other studies have found evidences that IS has left a modest imprint on other sub-fields of Management [3]. However there is a common agreement on the potential of IS in terms of its external influence on other fields [4]. Previous studies on this topic are based either on conceptual analysis [2], or on the quantitative analysis of citations trends [3], and in some cases on post-hoc content analysis aimed to understand how IS works are being utilized by other disciplines [4]. These studies share an holistic approach to the study of this phenomenon, in which the whole body of knowledge developed in the IS field has been considered as unit of analysis instead of focusing on a specific topic. These contributions aim to provide a general assessment of the IS field in order to define general guidelines for enhancing the discipline. We claim that a further contribution to this debate can be provided by gaining insights into the dynamics of knowledge creation processes in scholar communities. In this paper we start a discussion on this matter by showing how a citation analysis focused on a specific topic can help in the exploration of such dynamics. We thence begin an investigation on the relationship between the knowledge creation process of a research community and the impact of its research results. To achieve this goal we first restrict the scope of our analysis to a specific stream of research, which represents the source of data for this study. This stream of research must be i) old enough to show its evolutionary trends, ii) present in two different fields of research to enable a comparative analysis, iii) narrow enough for allowing the adoption of both quantitative and qualitative bibliometric analysis methods and techniques. According with these criteria, our case study addresses the knowledge creation process of the IS and the Management (MGT) research communities within the particular domain of “simulation related studies”. This stream of research includes both articles adopting simulation as a research method and articles providing contributions at the meta-level. Simulation is intended as using computer software to model the operation of “real-world” processes, systems, or events 1

Corresponding Author.

[5]. Simulation methods have been adopted as theory

development methods in both Management and Information Systems studies. Davis et al. [6] have discussed the strengths and weaknesses of simulation methods for theory development and have defined a roadmap for executing and evaluating research using simulation methods in the broad Management field. Simulation is also considered as being one of the available evaluation methods for design research studies [7][8]. Our assumption is that a comparative exploration of trends and connections between articles referred to “simulation related studies” in the IS and MGT fields can set up the basis for a subsequent discussion on the emergent aspects of their generation processes. The paper starts with a description of the research method and the dataset. Then the results of the analysis are illustrated by the means of descriptive data and network diagrams. Finally a discussion summarizes the findings by providing insights and suggestions for further investigation.

1. Method and Data ISI (Institute for Scientific Information) Web of Science [9] was used to conduct the searches and retrieve publication data taking into account the first three citation databases: Science Citation Index Expanded, Social Sciences Citation Index and Arts & Humanities Citation Index. They fully cover nearly 12,000 major journals adding up to over 40 million searchable records [9]. In addition, ISI Web of Science search seems to merge duplicate results when searching from multiple indexes, so there was no great fear of redundant data in the search results. These citation indexes also contain a record of the references cited by the authors of the covered publications. This enables the use of cited reference searches and various citation analyses. For this work, we performed two queries on ISI Web of Science search, both based on “what” (topic) and “where” (publication name) searching attributes. For both queries the topic is the same whereas they differ for the set of selected journals. With respect to the topic, we adopted the classification provided by Davis et al. [6], about the different simulation approaches. Therefore the following keywords have been identified: "system dynamics", "NK fitness landscape", "genetic algorithm", "cellular automata", and "stochastic processes", plus the more generic term “simulation”. With respect to the journals’ selection, we adopted the Senior Scholars’ Basket of Journals published by AIS2 for the IS query. Whereas, for the MGT query we based the selection on the list of top journals published by the Italian National Agency for the Evaluation of Universities and Research Institutes (ANVUR3). The two final search strings are formulated as: IS query: Publication Name=("European Journal of Information Systems" OR "Information Systems Journal" OR "Information Systems Research" OR "Journal of AIS" OR "Journal of Information Technology" OR "Journal of MIS" OR "Journal of Strategic Information Systems" OR "MIS Quarterly") AND Topic=("system dynamic*" OR "NK fitness landscape" OR "genetic algorithm*" OR "cellular automat*" OR "stochastic process*" OR "simulation*") MGT query: Publication Name=("academy of management review" OR "academy of management journal" OR "organization science" OR "organizational behavior and human decision processes" OR "journal of organizational behavior management" OR "strategic management journal" OR "behaviour & information technology") AND Topic=("system dynamic*" OR "NK fitness landscape" OR "genetic algorithm*" OR "cellular automat*" OR "stochastic process*" OR "simulation*")

Asterisk-characters are used to include singulars search and plurals of each search term. The search is not limited to any specific year, thus everything from 1985 to 2012 has been retrieved and no other restrictions are applied. All searches were performed on June 14th, 2012. This resulted in 54 hits for the IS query (with 1080 citations) and 169 for the MGT query (with 5076 citations). Furthermore the “Create Citation Report”-link was used to get detailed data on citations to the retrieved publications. The relative paucity of simulation studies, which are not a mainstream topic in both IS and MGT, provides us the opportunity to perform a longitudinal

2 3

http://home.aisnet.org/displaycommon.cfm?an=1&subarticlenbr=346 http://www.anvur.org/

analysis over a thirty year period without losing the possibility for further studies to analyze details of articles’ connections. This also does not avoid us to analytically generalize our findings. Before to start the analysis of the results, we performed the same queries without any filter on the topic. In this way we obtained the following output in terms of the two overarching sets: 

IS query: 3.462 articles, cited 95.580 times



MGT query: 9.556 articles, cited 527.821 times

Therefore the ratio of publications in IS to MGT is about 1 to 3, whereas it is about 1 to 5 with respect to the number of citations. These proportions reflect the ones founded in the subset of simulation related articles. For further analyses the search results of the queries were exported from the ISI Web of Science in text format as full records, including cited reference data. These data have been then imported into a bibliometric analysis tool called Sitkis [10]. Sitkis is a free Java-based program that utilizes Microsoft Access database to store and analyze data exported from the ISI Web of Science. Various analyses can be made with Sitkis using the exported data. After analyzing the articles, Sitkis can generate some text file containing a matrix representation of some citation analysis results. These text files can be imported to UCINET 6, a social network analysis software, to make them visualized in NetDraw (included with UCINET 6). NetDraw was then used to draw the various network diagrams provided in this work.

2. Analysis and results 2.1. Publication statistics There are two categories of bibliometric indicators: descriptive indicators, and relational indicators having an analytical function [11]. We start our analysis by presenting some general figures representing descriptive indicators. Even though the first article in IS was published in 1988, the trend in this area actually starts in 1994, almost 10 years later with respect to the MGT area (1986). This delay also reflects the different ages of these two fields. The number of publications per year and their citations trends for both IS and MGT areas are depicted in Figure 1. The number of publications and citations per year in the two sets shows a similartrend. Indeed the publishing rate increases as a sine wave trend with peaks in 1996, 2003, and 2010 in both cases (left side of Figure 1). Whereas the number of citations increases yearly in a continuous manner. Two are the main considerations about the topic: 

on the basis of the number of publications per year, simulation studies still represent a research topic in which both communities show a growing interest;



the trend of the number of citations of papers in the two sets, is a proxy of the importance of these contributions for the whole research community .

Nr. of Publications per Year

Nr. of Citations per Year

18

800

16

700

14

600

12

500

10

400

8

300

6 4

200

2

100

0

0

IS publications

MGT publications

IS Citations

MGT Citations

Figure 1. Number of publications and citation per year.

2.2. Co-Citation analysis The ISI Web of Science citation indexes also record the citations to references made by retrieved publications. By analyzing these citations, the most influential works in the IS and in the MGT areas can be identified. Table 2 lists the most cited references, among those retrieved with the IS query, that are cited at least 3 times by other articles belonging to the same set. On the other side, Table 3 refers to the MGT query, and shows the references cited at least 13 times. These thresholds have been fixed for obtaining a similar number of papers (14 and 13 for IS and MGT respectively). By observing these two tables, a high degree of convergence emerges in the MGT area in which some influential works are cited about 30 times. On the other side, in the IS area the maximum number of repeated citations is 5. The co-citation analysis can depict better this behavior considering the co-citation network. Co-citation analysis is a relational bibliometric indicator, which provides a picture of scientific activity and helps in monitoring and identifying emerging research topics and the relevant contributors [11]. Co-citation analysis involves tracking pairs of papers that are cited together in the source publications, forming relations and clusters of research [12]. Figure 2 and 3 show the co-citation network of the most cited references for IS and MGT subsets respectively. The nodes represent the common references and the ties, with their thickness, show how many times two articles are cited together. Despite the number of nodes is quite similar the co-citation networks are fairly different. Within the IS set, the number of ties is 25 and their maximum thickness is 4 (4 times two references are cited together). Moreover the network is composed by two separated graphs (also identified by different colors in table 2). On the other side, in the MGT area there are 62 ties and two references are cited together 14 times. This result suggests to further analyze the extent to which the most popular IS references are sparse and scattered in comparison with the MGT area. In order to investigate this issue we evaluate the connectedness or density index of both graphs. The level of connectedness (also named density) can be considered as the simplest index of the amount of relationships among the nodes in a network. It is defined as the ratio of observed links to the total number of links possible [13, 14]. Connectedness is a variable that takes on a value of 0 when there are no links in the network and it reaches a value of 1 when every node is linked with everyone. To calculate the level of connectedness C, the first step is to identify the number of possible ties T in the network. T can be calculated considering the total number of nodes (named N) and the directionality of the graph: 

T=N*((N-1)/2) for a non-directional graph



T=N*(N-1) for a directional graph.

The co-citation network is a non-directional graph, then the connectedness or density for both networks is: 

for IS C=25/[14*((14-1)/2)]=0,27;



for MGT C=62/[13*((13-1)/2)]=0,79.

As suggested before, the connectedness index confirms that the level of common references usage is higher in MGT than in IS. This parameter does not take into account the “quality” (size: how many times two nodes are cited together) of each tie. From the IS co-citation network there is only one co-cited “triangle” within the selection of the most cited references: Diamantopoulos et al. 2001, Jarvis et al. 2003 and Hardin et al. 2008. They are cited together 3 times in the retrieved publications. There are also two additional co-cited “triangles” which are less frequent and belong to two separated sub-graphs; one is formed by Majchrzak et al. (2000), Mayer et al. 1995 and Fornell et al. 2001 and the second one by DeMarco 1982, Forrester 1961, and Boehm 1981. Table 2. The most cited references (cited by the search result publications for IS area) Citations 5

4

4

Author(s)

Title

Year

Type

Diamantopoulos, A; Winklhofer,

Index construction with formative indicators: an alternative to scale

2001

Journal article

H.M.

development

Jarvis, C.B.; MacKenzie, S.B.;

A critical review of construct indicators and measurement model

2003

Journal article

Podsakoff, P.M.

misspecification in marketing and consumer research

Majchrzak, A.; Rice, R.E.;

Technology adaptation: the case of a computer-supported inter-

2000

Journal article

Malhotra, A.; King, N.; Ba, S.L.

organizational virtual team

Ba, SL; Stallaert, J.; Whinston,

Research commentary: introducing a third dimension in information

A.B.

systems design the case for incentive alignment

3

Boehm, B.W.

3

Cohen J.

3 3

3

2001

Journal article

An experiment in small-scale application software engineering

1981

Journal article

Statistical power analysis for the behavioral sciences

1988

Book

DeMarco, T.

Controlling software projects: management, measurement & estimation

1982

Book

Dewan, S.; Mendelson, H.

User delay costs and internal pricing for a service facility

1990

Journal article

3

Fornell, C.; Larcker, D.F.

Evaluating structural equation models with unobservable variables and

1981

Journal article

3

Forrester, J.W.

Industrial dynamics

1961

Book

3

Hardin, A.M. ; Chang, J.C.J. ;

Formative vs. reflective measurement: comment on marakas, johnson, and

2008

Journal article

Fuller, M.A.

clay (2007)

3

Jöreskog, K.G.; Yang, F.

Structural equation modeling

1996

Book Chapter

3

Malone, T.W.

Modeling coordination in organizations and markets

1987

Journal article

3

Mayer, R.C.; Davis, J.H.;

An integrative model of organizational trust

1995

Journal article

Auctions and bidding - a primer

1989

Journal article

measurement error

Schoorman, F.D. 3

Milgrom, P.

Figure 2. Co-citation network of the most cited reference for IS query

In the MGT co-citation network, the more co-cited “triangle” is composed by: March 1991, Nelson and Winter 1982, and Cyert and March 1963. These references are cited together in 12 publications. If we set the minimum number of cited times to 7, all the nodes are connected through one or more ties, except Baron and Kenny 1986, and Locke and Latham 1990 (cited only two times together also with the remaining publications). As mentioned before, the nodes in the MGT co-citations network are more strongly connected (high density degree and great ties thickness) than in the IS co-citation network. This means that in MGT, the research community has identified a common set of references on the topic. Table 3. The most cited references (cited by the search result publications for the MGT area) Citations Authors 28

Nelson, R.R.; Winter

Title

Year

Type

An evolutionary theory of economic change

1982

Book

A behavioral theory of the firm

1963

Book

Organizations

1958

Book

S.G. 27

Cyert, R.M. ; March, J.G.

22

March, J.G.; Simon, H.A.

21

March J.G.

Exploration and exploitation in organizational learning

1991

journal article

16

Levitt B; March J.G.

Organizational learning

1988

journal article

15

Baron, R.M.; Kenny,

The moderator mediator variable distinction in social psychological-research - conceptual,

1986

journal article

D.A.

strategic, and statistical considerations

Cohen M.D., March

A garbage can model of organizational choice

1972

journal article

15

J.G. and Olsen J.P.

15

Thompson, J. D.

Organizations in action

1967

Book

14

Levinthal D.A.

Adaptation on rugged landscapes

1997

journal article

14

Rivkin J.W.

Imitation of complex strategies

2000

journal article

13

Gavetti G. and

Looking forward and looking backward: cognitive and experiential search

2000

journal article

A theory of goal setting & task performance

1990

Book

Dynamic capabilities and strategic management

1997

journal article

Levinthal D. 13

Locke, E.A.; Latham, G.P.

13

Teece D.J.; Pisano G; Shuen A.

Figure 3. Co-citation network of the most cited reference for MGT/OS/OB query

2.3. Cross-Citation analysis The co-citation analysis shows the main network of references cited by the articles retrieved in the two areas: Information Systems (Figure 2) and Management (Figure 3). It can be considered as the basis (mainly outside the set of retrieved publications) upon which the authors have built their contributions. However it is also interesting to observe whether the selected publications do relate to each other. By tracing these links it is possible to identify the evolution paths of the knowledge creation process and to compare the two fields of study. This can be achieved by adopting the cross-citation analysis. This analytical technique depicts whether and to what extent a publication within a set is cited (does have an impact) by the others in the same set. The cross-citation network is created as a result of the cross-citation analysis. It is a way for having an idea on the “relative impact” of each publication inside the same set. The nodes represent a publication and the ties, formed by arrows, indicate “who cites whom”. For example, if the paper A cites paper B there is an arrow from A to B. The cross-citation network can be considered as a snapshot of the citations links among the members of the same community. For describing this kind of graph we adopt some concepts borrowed from the social network analysis studies [14, 15]. Figure 4 shows the cross-citation network for the IS field. There are 20 nodes and 13 ties, which are distributed in 4 Isolated Dyads (a pair of nodes linked together but isolated from the rest of the network) and 3 Trees (a subset of node with minimal connections; if any single node is removed the result will separate the tree structure in two parts). Figure 5 shows the crosscitation network for the MGT field. There are 79 nodes with 111 ties, distributed in 10 Isolated Dyads, 1 Tree, and two groups of nodes: the smaller is formed by 11 nodes and 13 ties and it is quite similar to a tree structure; the bigger includes 44 nodes and 85 ties and it is a typical “scale-free” or “Power law” structure, in which very few nodes have a huge number of connections, some nodes with moderate connections and many with very few ties.

Figure 4. Cross-citation network of the most cited reference for IS query

Figure 5. Cross-citation network of the most cited reference for MGT/OS/OB query

From figure 5 it is possible to observe a hierarchical structure between articles grounded on previous articles that are recursively grounded on additional levels. The graph suggests that the members of the MGT community build their body of knowledge incrementally, by taking into account the previous works done within the community itself. A totally different behavior can be observed in the IS field. In fact, the cross-citation graph in figure 4 shows a sparse network with very few ties linking a small number of papers. The transitivity degree is used in social network analysis for describing the extent to which nodes are taken into account by other nodes in the same set. We start from the following definition [14]:  Triad: a triple of actors with their ties  Transitivity: the triad involving actors A, B, and C is transitive if whenever A--> B and B --> C then A --> C (in our case, if A cites B and B cites C then A cites C). Starting from these definitions we can introduce the transitivity degree as the number of transitivity triads or paths present in the graph divided by the potential number of them. This index is equal to 1 when each triad in a directional graph is transitive. We used UCINET for calculating this index with a resulting value of 0 for the IS field cross-citation network and 0,2 for the MGT field. These values confirm the hypothesis on the behavior of the MGT research community described before.

Finally, we adopt cross-citation analysis for investigating the reciprocal influence between the two communities. With this purpose we merged the two datasets and we built a holistic graph, using different colors for characterizing the original set of each node. Figure 6 shows on the left side the IS publications (blue) whereas on the right side the MGT (red) ones. From this graph is interesting to observe that 10 out of 11 “bridges”, meaning the group members with a link to a member of a different group, come from the IS field and are linked towards the MGT field. Furthermore the most cited publications in the MGT area are often linked to (cited from) articles in the IS area. This highlights the different behaviors of these research communities with respect to the knowledge creation process in this specific domain. Indeed the IS community has the propensity to cite external publication, even though in this case we are considering only the MGT ones.

Figure 6. Cross-citation network of the most cited reference in the IS (blue) and MGT (red) fields

3. Conclusion In this paper we present the preliminary results of a bibliometric analysis of “simulation related studies” published in top IS and MGT journals in the last thirty years. Although a discussion of these findings against previous works on the knowledge creation processes of scholar communities is out of the scope of this chapter, it can be considered as a first step towards a better understanding of such complex phenomenon. In fact a deeper exploration of the mechanisms leading to similar evolutionary trends starting from different cumulative research traditions emerges as the main finding of our analysis. The concept of hierarchy in complex systems [16] can provide the lens for interpreting the results of our cross-citation analysis that shows how the majority of scholarly contributions in the MGT field are firmly grounded on previous studies published within the same field. As opposed to this hierarchic incremental behavior characterizing the knowledge creation process in the MGT field, IS studies seldom refer to a hierarchical body of knowledge. When this is the case, MGT studies act as a reference discipline upon which new knowledge is created in the IS domain. In spite of this different behavior, citation data show that the two body of knowledge have a similar impact on subsequent research. In fact, the number of citations from other fields can be considered as a proxy of the research importance [17] and the analysis of this research stream shows a similar ratio number of publications versus number of citations. This means that although research in IS is unlikely to build on and cite prior research, its influence on other fields is proportionally similar in magnitude to the external influence that MGT studies. This conclusion asks for further investigations on the relationship between the knowledge creation process of a research community and its external influence and evolutionary dynamics. A qualitative analysis on the same dataset or the analysis of further streams of research can provide additional insights on this research problem and contribute to the debate on the development of the IS field.

References 1.

Culnan, M.J., Swanson, E.B.: Research in Management Information Systems, 1980-1984: Points of work and reference. Mis Quarterly. 10, 288–302 (1986).

2.

Baskerville, R., Myers, D.: Information systems as a reference discipline. MIS Quarterly. 26, 1–14 (2002).

3.

Wade, M., Biehl, M., Kim, H.: Information Systems is not a Reference Discipline (And What We Can Do About It). Journal of AIS. 7, 247–269 (2006).

4.

Grover, V., Ayyagari, R., Gokhale, R., Lim, J.: A Citation Analysis of the Evolution and State of Information Systems within a Constellation of Reference Disciplines. Journal of AIS. 7, 270–325 (2006).

5.

Law, A.M., Kelton, D.W.: Simulation modeling and analysis (2nd ed.). McGraw-Hill, New York (1991).

6.

Davis, J.P., Eisenhardt, K.M., Bingham, C.B.: Developing Theory Through Simulation Methods. Academy of Management Review. 32, 480–499 (2007).

7.

Hevner, A.R.A.R., March, S.T.S.T., Park, J., Ram, S.: Design Science in Information Systems Reserach. MIS Quarterly. 28, 75–105 (2004).

8.

Spagnoletti, P., Baskerville, R., Marco, M.D.: The Contributions of Alessandro D’Atri to Organization and Information Systems Studies. In: Baskerville, R., De Marco, M., and Spagnoletti, P. (eds.) Designing Organizational Systems, LNISO vol. 1. pp. 1–18. Springer, Heidelberg (2012).

9.

ISI: ISI Web of Knowledge [v.5.6] - Web of Science Home.

10.

Schildt, H.A.: SITKIS: Software for Bibliometric Data Management and Analysis v2.0. Institute of Strategy and International Business, Helsinki (2006).

11.

Gauthier, É.: Bibliometric analysis of scientific and technological research: A user’s guide to the methodology. Statistics Canada (1998).

12.

Garfield, E.: Co-citation analysis of the scientific literature: Henry Small on mapping the collective mind of science. Current Comments. 9, 3–13 (1993).

13.

Tutzauer, F.: Toward a theory of disintegration in communication networks. Social Networks. 7, 263–285 (1985).

14.

Wasserman, S., Faust, K.: Social network analysis: Methods and applications. Cambridge University Press (1994).

15.

Richards, W.: The NEGOPY network analysis program. Social Networks. 3, 215–223 (1981).

16.

Simon, H.A.: The Sciences of the Artificial. MIT Press (1996).

17.

Loebbecke, C., Leidner, D.: The Contribution of Top IS Publications to Subsequent Research: A Citation Analysis. Communications of the Association for Information Systems. 30, 423–438 (2012).