Nov 7, 2007 - paper we extend our field work with IT helpdesk staff to examine .... structures. Each site contains one director, hardware technicians,.
Structuring Cross-Organizational Knowledge Sharing Kevin F. White, Wayne G. Lutters Department of Information Systems, UMBC 1000 Hilltop Circle, Baltimore, MD 21250, USA
{kwhite2, lutters}@umbc.edu These systems are not without limitations. The benefits of EROMS to in-house experts are bounded, and are primarily derived from reducing the interruptions caused by repetitive information requests. When multiple experts exist within an organization they often collaborate on answers to difficult problems. However, when this expertise is exhausted “organizational dysfunction” occurs [1]. Dysfunction is the result of the inability to obtain answers to problems efficiently and in usable form from internal sources. The information seeker must then develop new information-seeking strategies, often obtaining outside help via Internet repositories or outside consultants, both of which tend to be costly in both time and money.
ABSTRACT Ontology development is fundamental to most knowledge management efforts. When approached in a formal knowledge engineering manner the resulting ontology usually becomes brittle when spanning even a modest number of groups within a single organization. It breaks entirely when scaled to multiple, heterogeneous organizations. A promising alternative is the bottom-up approach such as can be found in social tagging systems (e.g., del.ico.us), but little research has examined the utility of these systems for knowledge reuse activities. In this paper we extend our field work with IT helpdesk staff to examine the drivers for natural ontology development. We found that a balance between some degree of external order while maintaining local flexibility was required. This information space is navigated via social relations, especially expert referral. We examine the user-centered design criteria for both mid-level ontology development and related expert profile management.
Such difficulties are exacerbated in small organizations. Firstly, expertise recommender systems deployed within small organizations lack the benefits associated with their use in larger organizations. Employees’ knowledge of each other’s expertise often makes identifying and locating an expert a trivial task that does not require the overhead of a recommender system. Secondly, groupware systems in general, and ER-OMS systems in particular, are notoriously expensive and time consuming to build and maintain. Given the costs and benefits of an ER-OMS, it is frequently prohibitive to deploy such a system within small organizations. However, without an ER-OMS in place small organizations are at a disadvantage as they have no less need to preserve organizational memory than their larger counterparts.
Categories and Subject Descriptors H.5.3 [Information Interfaces and Presentation]: Group and Organization Interfaces – computer supported collaborative work
General Terms Management, Documentation, Design, Human Factors
A potential solution to this problem is to expand ER-OMS beyond an organization’s boundary by developing information partners with companies operating in similar domains and with similar information needs. Doing so increases the quantity of potential collaborative experts, disperses time consuming knowledge capture activities over a larger group, and reduces the costs associated with system building and maintenance.
Keywords Cross-organization, organizational memory, knowledge management, expertise recommendation, profiles, ontology, trust, socio-technical, social computing, qualitative research
1. INTRODUCTION Knowledge management is a known difficult problem. Expertise recommender and organizational memory systems (ER-OMS) have long been promised to provide a method for preserving organizational knowledge and assisting in locating experts [10]. OMS provide the opportunity to store, search, and reuse information after those who originally developed the knowledge have left the organization. Likewise, ER systems assist in finding a domain expert when knowledge cannot be located in an information repository [21]. Taken together, ER-OMS can improve the effectiveness and efficiency of problem solving activities by reducing knowledge recreation events.
Before designing and implementing a cross-organizational memory system it is necessary to explore existing knowledge reuse practices. Areas of keen interest are the processes of knowledge capture, storage and organization as well as expertise identification and selection. Additionally, developing trust in knowledge developed outside of one’s own organization must be assessed. These interests drove our empirical inquiry. After discussing our motivation to implement a crossorganizational memory system, we describe our research sites. We then frame our work in the relevant literature followed by our findings within three areas: organization of captured knowledge, information necessary for knowledge and expertise selection, and trust in other sites’ knowledge. We conclude with a brief discussion on the implications our findings have on crossorganizational ER-OMS design.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. GROUP’07, November 4–7, 2007, Sanibel Island, Florida, USA. Copyright 2007 ACM 978-1-59593-845-9/07/0011...$5.00.
187
Each of these questions is critical to understanding the design of a future cross-organizational ER-OMS. Attempting to answer them, we examined current organizational practices and preferences, including information seeking strategies for both public and knowledge repositories, and methods of assessing the relevance and trustworthiness of the resultant resources. This was done within five technical support departments in public schools.
2. MOTIVATION Existing research on ER-OMS has concentrated on implementations within sizeable organizations where the quantity of experts is sufficient to warrant an internal system. However, the vast majority of organizations are small, containing a limited pool of experts. While this decreases the need for an internal EROMS, it increases their reliance on external sources of support and the risk of losing significant knowledge when members leave. In an effort to provide small organizations similar knowledge management benefits as their larger counterparts, we have been engaged in an on-going project designed to understand the utility of building and implementing a cross-organizational ER-OMS as described in [34,35]. Briefly, most businesses contain cost centers, such as IT, that are necessary overhead to provide competitive advantage to the firm. We claim that developing information sharing collaborations between these groups can trim overhead costs by increasing their productivity and reducing their need to use expensive external support. While it is possible to develop such information partnerships between organizational units, it is difficult to do so across firms, especially when they are in direct competition with each other. Useful alternative venues to explore are public service organizations, such as municipal governments, non-profit foundations, places of worship, police/fire departments, and schools. While these organizations are competing for resources at some level, they often have pre-established partnerships and a culture that permits, even encourages, dialog between them. Within each domain these parallel organizations share common organizational structures, recurring problems, and information needs, thus making them excellent sites in which to explore the merits of a cross-organizational ER-OMS. As the design of ER-OMS moves from internal to crossorganization new dynamics are introduced that must be taken into consideration during system design and implementation. Specifically, the goal of the research described here was to understand: • how knowledge captured in a cross-organizational memory system should be organized; • what contextual details about partner sites and their members are necessary to assist the information-seeker in reusing knowledge and selecting experts; • how knowledge developed in partner sites is evaluated and deemed trustworthy. Table 1: Research Site Comparison
The study of ER-OMS is not new (e.g., [3,4,23,28]), nor is the study of ontology building (e.g., [6,16,36]), however, our emphasis on parallel organizations is novel. Our findings describe the challenges that exist when extending traditional ER-OMS implementations for use between multiple parallel organizations.
3. SITE DESCRIPTION As noted, the initial parallel organizations selected for this research were IT departments in public school districts. Technical support environments have proven to be rich venues for exploring both ER and OMS design issues (e.g., [18,25]) due to their high volume and diversity of questions, collaborative problem solving approaches, expertise-sensitive work, time critical responses and limited confidentiality concerns. Five IT departments were selected for this study based on their established affiliations and their willingness to collaborate with each other both for the near-term study and any resulting longterm partnership. As shown in table 1, all five sites share common organizational structures. Each site contains one director, hardware technicians, and a network administrator. Part-time technicians consisting of parents or teachers with interest in technology are used to supplement full time staff. Two of the sites, Lakeview and Homesdale, also employ a full-time database administrator and network security administrators. This difference, while notable, is representative of what is likely to be found in school districts across Pennsylvania and provided opportunity to compare findings from smaller sized school districts (Prairieville, Jefferson, and Newburry) to larger districts (Lakeview and Homesdale) in order to assess strategies that may differ based on size and departmental make up. In addition to similar organizational structures, the sites also support similar computer systems, software, and network systems.
Lakeview
Homesdale
Newburry
Prairieville
Jefferson
Computers
5,000
4,500
2,700
1,500
1,700
IT Staff
22
28
6
6
6
IT Staff to Computers
1 : 227.3
1 : 160.7
1 : 450
1 : 250
1 : 283
Faculty
850
940
590
275
320
IT Staff to Faculty
1 : 38.6
1 : 33.6
1 : 98.3
1 : 45.8
1 : 58.3
Computer Models
HP
Dell
Dell
Dell
HP, white box
Server Models
HP
Compaq
Compaq
Compaq
Compaq
Server OS
Windows
Windows
Novell
Windows
Windows
Infrastructure
HP,
3Com, Cisco
3Com, Cisco
3Com,
Cisco
Cisco
Cisco
188
semi-structured interviews provided insight into thought processes, search behaviors and preferences, knowledge organization, domain information, potential sources that could be used to automate profiles, and how trust is judged for each source of located knowledge. Direct observations provided an opportunity to confirm and expand on data extricated through semi-structured interviews as well as to gather additional information which may have been overlooked. Clarifications and additional information was gathered where necessary through email correspondence. Semi-structured interviews, e-mail, and field notes were transcribed and content coded for analysis using a Grounded Theory approach [31].
For example, each site supports primarily Windows 2000 operating systems, a combination of Cisco and 3Com networking equipment, and similar software titles. Given the same overall goal (to provide technical support for faculty, staff and students), and significant similarities in the type of equipment, software, users, and tasks supported within each separate entity, there is a high probability of encountering similar problems throughout their work day. These similarities create an environment ripe with knowledge sharing potential. When coupled with each sites’ willingness to share knowledge with each other, these school districts provided a nearly ideal environment to explore how knowledge captured in a cross-organizational memory system should be organized, what kinds of information about partner sites and their employees is necessary to assist the information-seeker in selecting experts to collaborate in developing solutions to problems, and to understand how trust in knowledge developed within partner sites is assessed.
In addition to interviews and observations, we gathered approximately 75 knowledge articles developed from across all 5 sites. The articles represented a range of topics, including question-answer pairs describing hardware and software problems and solutions, procedural documents (such as how to image a computer), and policy decision documents. Participants were asked to complete an adaptation of a pile-sort exercise, such as described in [7,8,19], in which each participant was asked to place the 75 articles within a hierarchical scheme. This exercise was used to test and further refine an ontology developed from semistructured interviews and field observations.
4. STUDY DESIGN As previously stated, our research had several goals. First, we were interested in how knowledge is currently captured and organized within each individual organization, and the techniques most frequently used to find needed information prior to the introduction of a new cross-organizational ER-OMS. Understanding current processes within each site prior to the design of a new system is necessary to improve user buy-in and lower the cost of transition to a unified ER-OMS. It is also critical to develop an ontological framework within which captured knowledge can be organized to provide an intuitive process for (re)finding captured information.
5. ONTOLOGY REVIEW The study of ontology is deeply rooted in information and library sciences, which have used ontological research to form catalogues and store bibliographic information along with full text documents [16]. The definition of ontology is murky, ranging from its more traditional roots where it refers to the development of a hierarchical structure within which information is organized, to the more contemporary definition, “a set of classes, relations, functions, axioms and instances [12]” which form an “explicit definition of a conceptualization [14]” While both are relevant to knowledge management activities, it is the former that we concentrate on here.
Secondly, we needed to understand the types of “people information” required to assist a knowledge seeker in matching their needs to a list of available experts. Knowing a person’s background and areas of expertise satisfies only a partial need of expertise-matching in cross-organizational ER-OMS. It is also necessary to capture site information in order to narrow the pool of potential collaborators by taking into account environmental factors that influence problems and potential resolutions within each environment. Combining user profiles with site profiles is one of the challenges that makes cross-organizational, knowledge based expertise recommendations a particularly interesting problem. Thus, we needed to understand what informationseekers need to know about a partner site’s work environment to maximize context necessary to solve a problem within their own site.
Knowledge management is concerned with the representation of tacit and explicit knowledge – how it is created, stored and organized, and relocated for future reuse. As knowledge encoded in computer-readable forms increases, tools are necessary to effectively search through storage units to find and extract information [16]. The primary difficulty is in organizing and coding the information such that effective search and retrieval is made as straightforward as possible – a problem that has only increased with the introduction of a variety of media such as voice and video recordings. Controlled vocabularies and hierarchical coding schemes are two of the most difficult problems of forming an ontological scheme [32]. This is particularly true in dynamic fields in which new concepts are regularly added, requiring new branches within the agreed upon schema to be formed and additional controlled vocabulary to be added [16]. Additionally, because the way in which people think and organize items differ, ontology building becomes a socio-technical activity.
Finally, the utility of providing a list of potential experts together with environmental factors is limited without trusting other sites’ knowledge. Therefore, we needed to explore how trust is assessed and the type of information which, if present, may serve to increase the level of trust. Given our desire to understand current processes and potential sources of information in situ, we chose to use a primarily qualitative field research design. Data collection involved a series of semi-structured interviews and direct observations. Each interview lasted between 45 and 75 minutes per participant (n=20), while observations lasted between one and two days per site. Participants represented a cross-section of each site’s technical support department, and included one person from each position (director, technician, database administrator, etc.). The
In their research Berlin et al. exposed that people have very different preferences in organizing information [6]. Their work highlights how each individual’s organizational preferences result in difficulties when extended to group use. When working alone, each person is able to locate needed information based on their own preferences, whereas when storage activities are defined by a
189
group additional mental load is placed on all team members attempting to conform to group standards which are in constant conflict with their own preferences. Such conflict sometimes results in the use of localized stores maintained by each individual, which will in turn result in inconsistencies and outdated data between the two sources [28].
6. ONTOLOGY FINDINGS Although we recognize the benefits of the contemporary view of ontology as described by Gruber in [14] and briefly discussed in section 5, our focus was on devising a taxonomy and hierarchical structure to store and organize knowledge which could later be formalized within an ontological language. It is from this viewpoint that we use the term “ontology.”
Several techniques of storing knowledge in a shared repository have been developed to mitigate the problem of individual preferences leaking into a shared ontology, causing it to become less useful. One technique is to employ a small number of librarians or knowledge engineers to categorize and store knowledge within the established hierarchy. Our research focuses on small companies in which the overhead of providing a librarian, even on a part-time basis, to organize knowledge and maintain ontology is unlikely given the personnel costs involved.
One of the challenges in creating ontology is establishing how many layers should be present in the hierarchy. Too many layers and the hierarchy becomes too fine grained, leading to incorrect placement of knowledge articles. It also makes browsing more time consuming and mentally taxing than necessary. Too few levels can lead to a feeling of chaotic organization as articles with different topics become mixed within the same structure, and creates the problem of wading through articles to find those that may be relevant [6]. The objective, then, is to find the correct balance between the two extremes. In order to arrive at a consensus on the number of hierarchical levels to sufficiently organize captured knowledge we relied heavily on semistructured interviews and observations, followed by a card sort activity. Participants unanimously specified that there should be no more than 3 to 4 subcategories within each top level classification.
Instead of providing librarians, Wu et al. suggests that it is possible to provide multiple ontologies within the same system [35]. Their “document co-organization” research allows each person to organize knowledge captured within a repository in a personal hierarchy. Each team member’s individualized hierarchy is then combined using a variety of algorithms to create a selforganizing single ontology. This solves the conflict inherent in creating a single ontology and forcing all members of a group to conform to specified standards, while still providing a group view of all available stored knowledge. An added benefit, as described in their study, is that using peers’ personal hierarchies is more useful than using search functions to find needed information. Their research, however, focused on a classroom-based project in which the use of documents stored in a repository throughout a semester would be used to complete a final paper. In real knowledge consuming organizations, however, information needs are seldom known ahead of time. This is particularly true in a technical support environment where future information needs are often unknown and where information needs change on a regular basis.
I’d say 3 levels. Maybe 4. Yeah, 3 levels and you better be in the area of what you are looking for, because then you start to get lost and feeling like you are wasting too much time. Technician, Prairieville
Throughout the observations conducted while technicians were searching for information in Internet repositories we noted that they would follow referenced documents located throughout each article (where available). In doing so, the technicians followed as many as 4 successive links, with each link opening a new web page before returning to the original document or search. When asked about their behavior, they explained that after 3 to 4 links they began to feel that they were getting lost, and that the likelihood of finding relevant information degraded. Further, they explained that linked documents could “go on endlessly, eventually linking to an entirely different topic”. If a linked document was relevant, they felt that they would find it within their original search.
Another potential solution is to provide a system capable of suggesting the correct placement of documents within a hierarchy, as described in Kao’s KMDocTEr system [17]. KMDocTEr requires a set of controlled vocabulary and/or thesaurus which can be used to automatically search a document and recommend, based on its content, keywords that can be used within future search criteria as well correct placement of the document within a specified hierarchy. Controlled vocabularies provide a standardized set of terms for indexing and retrieval and can be used to organize information and relationships between documents according to pre-established taxonomies [16]. While KMDocTEr is capable of suggesting potential keywords and placement within an ontology, final selection is left to the individual [17]. The primary drawback to this system is establishing a sufficiently large and broad set of controlled words or thesaurus such that the system is able to accurately predict logical placement within the hierarchy. While KMDocTEr is advertised as reducing the overhead of training users on the appropriate placement of their documents within a hierarchical schema and employing system librarians, the overhead is replaced by maintaining controlled vocabulary and thesauri.
We used the combination of semi-structured interviews and observations to create a preliminary ontological scheme. The hierarchy developed by each participant was straight forward. Each participant’s desired hierarchy resembled that described by their peers, changing slightly in vocabulary alone (such as educational software, academic software, and teaching software). This was the case not only within employee’s job role, but across the entire department. Following initial development of a paper-based hierarchical model, participants were provided an opportunity to examine and suggest changes to the proposed schema. Minor changes were requested, such as the inclusion of “operating systems” under both computers and networking instead of as a separate entity as had initially been proposed. To test the ontology and determine the frequency that participants would place articles into the same category, we selected 75 knowledge articles from Internet knowledge bases as well as from school district sources. The articles represented a variety of question-answer pairs, excluding common break-fix articles (such
190
as how to replace a hard drive), as well as policy and procedure documentation gathered from the IT Directors. Participants were asked to place them within the proposed hierarchy, using a process similar to a card sort exercise as suggested by [15]. 61 of 75 articles (81.3%) were placed in the same location by all participants. Articles that were not placed in the same location were located within one level (n=11, 14.6%) or two levels (n=3, 4%) of each other, but were in the same branch. Feedback from participants was encouraging. They generally agreed that the hierarchy was sufficient to provide organization without causing significant difficulty in deciding which directory to place an article within. The only significant comment received during this process was the desire by 4 participants to have a miscellaneous folder in which topics that, as one Jefferson technician put it, “doesn’t seem to belong anywhere else.” When this desire was taken back to all participants for comment, it was criticized. Participants stated that any article that they are likely to develop would be able to fit into one of the top level branches, and that adding a miscellaneous folder would result in management overhead and some amount of chaos:
From a high level view, it appeared that participants were rallying against formation of an ontological structure altogether. However, search engines use ontological schemes (often combined with thesauri) in order to increase accuracy and probability of locating needed information. Ontology and search are therefore tightly intertwined. When unpacked, instead of rallying against organizing knowledge participants were in contention of being required to conform to a group organizational schema and of taking the time to correctly categorize information within the group ontology. Their preference was to obscure ontology and to rely on automated classification that provides the perception of an ontology-free system. Participants’ arguments lends itself to the timeless organize and browse vs. search debate such as described by Mackinlay and Zellweger in [20]. The concept of social tagging, such as found in del.icio.us, may provide an opportunity to quickly categorize knowledge during storage activities so that captured knowledge can be relocated by others while at the same time avoiding the need to conform to a group organizational scheme and also partially occlude the hierarchy from immediate view. Such an approach may be particularly beneficial within a homogeneous department which is likely to use specific jargon and error codes within documents, as is likely to be found within technical support environments. In addition to occluding ontological structure, social tagging may prove useful in moving formal ontological schemas towards a situated ontology, enabling tagged information to be used as a boundary object by providing abstraction and modularity of tagged information as described by Floyd et al. in [13]. However, as captured knowledge moves from use within a single department, such as technical support, towards use across different departments such as teaching staff or personnel, knowledge of specific domain jargon decays, reducing ability for non-group members to find required information. As our ultimate goal is to provide a true organizational memory system that can be used throughout an organization, it is likely that portions of knowledge articles will be used outside ones own department. Future study will need to target the use of social tagging mechanisms as a method of avoiding structured hierarchy in deference to personal organization styles while still maintaining the ability for others’ to locate necessary documents.
No, because that would just fill up with crap. Who wants to look through that, it would be a real mess – you’d have to look through a billion articles all having different topics. Besides, that would mean that someone would have to go through once in a while, pull things out and reorganize it. I sure wouldn’t want to do that. Technician, Prairieville
The conflict between those that desired a miscellaneous container and those that did not is consistent with existing literature such as [6] in which Berlin discussed the ways in which personal organization styles caused conflict when applied to a group setting. While an ontology is likely to evolve as new needs emerge, the general acceptance and high percentage of agreement during our article placement exercise leads us to believe that the hierarchy developed for this department is sufficient to use, at least during early stages of system use, though a process should be present to contact the ER-OMS manager(s) to request new branches and levels within the organizational system.
6.1 Resisting Top-Down Ontology Development Perhaps the most interesting result from the ontology portion of our research was our participants’ perception that no ontological scheme was necessary. As we were asking questions about organizational preferences, 85% (n=17) questioned the rationale of organizing data within a specified group hierarchy, preferring instead a single space in which all documents would be placed. This may largely be a result of participants’ predisposition to use a search engine to find relevant documents.
7. ER PROFILE REVIEW While ontology is a key component to information location events, knowledge is inherently embedded in the social world and should not be ignored as part of knowledge location and acquisition processes. Collaboration with domain experts provides opportunities to obtain references to captured information as well as to locate and reuse tacit knowledge.
Well if you could have a knowledge base search engine I just don’t see the relevance in organizing anything. To me it would just take up more time and cause us aggravation to have to put things in a specific folder. Because search engines now are so good at finding stuff that we need, and it would take more time to have to manually search through documents and try to find the right directory. Technician, Prairieville
Thus far we have discussed ontology development as the first part of the knowledge location equation. Expert recommendation systems address the second half of the equation by assisting information-seekers in locating people that are knowledgeable in specific areas for the purposes of collaborating on an answer to a question or problem. This process is accomplished through identification of people that have domain expertise and are likely to be able to answer questions or solve problems. This process can be completed via specific recommendations or lists of experts presented in a yellow page style index as discussed in [30]. In the case of specific recommendations, when more than one potential expert matches an information-seeker’s needs, a list can be
All I do is open Google…I just do a generic search for keywords that I think are most relevant to what I’m looking for. So I’m just not sure that it would be worth the headache since I’d just use a search engine to find what I need anyway. Network Administrator, Newburry
191
push and pull methods of matching information-seekers with potential collaborators. They also suggest that Answer Garden style systems (e.g. [1]) may not be successful when used for large-scale KM deployments due to requirements that make it necessary to use discussions as a process of handling difficult problems.
presented to the user from which selection(s) can ultimately be made [23]. In small organizations the difficulty of finding collaborative partners tends to be insignificant. Research has found that an organization of 200 – 300 people is small enough that people still know each other well enough to have a reliable grasp of each person’s area of expertise [11]. However, as an organization becomes more distributed, as is often the case in today’s society, or grows beyond 300 people, the difficulties inherent in expertise location becomes significantly larger. A system is then necessary to support expertise locating tasks. This “system” is available in several different forms, including the “information concierge” in which individuals are tasked with knowing the experts within each domain [5], and computer-based systems which can assist the information-seeker in finding a person with whom to collaborate. As organizations become dispersed or large, more reliance is placed on the computer-based approach. Because our research is focused on combining multiple organizations together to form a knowledge partnership, it is the systems supported view that we focus on here.
As the process of locating potential collaborators moves from a localized activity to one that stretches beyond organizational borders, challenging design requirements emerge. Connecting people across different organizations requires not only a sociotechnical process of finding mutually beneficial experts, but also ensuring a sufficient environmental match to provide adequate common ground to maximize the return on invested time. Our challenge is therefore twofold: to find appropriate markers to use for matchmaking activities that are relevant to people, their areas of expertise, and information needs, and to supplement those potential linkages with environmental preconditions which may impact expert selection. Throughout our research we paid close attention to work artifacts that might be useful in aiding recommendation processes. Many techniques have been developed to link people based on documents that they have written or read. For example, Reichling et al.’s ExpertFinding framework used a text analysis user matching algorithm capable of creating profiles and recommending actors based on documents [29]. In particular, the researchers acknowledged information-seekers need to know attributes of potential collaborators, to know how those properties can be modeled and reused for expertise recommendation purposes, and to provide a mechanism to gather those attributes automatically such as through text based files that they own. While automated profiling based on text was shown to be promising, our technical support departments are largely void of written documents, even within the director’s office. Most of the decisions that are made are done so verbally, with reliance on school board policy and memory of each historical decision. Network directors also had little written documentation, mostly consisting of network maps that were once created digitally but have since been modified by notes captured overtop of the digital document and then hung up on their bulletin board. The primary source of documentation come from technicians when they document processes used to solve problems. To date this has been a sporadic, largely abandoned activity. Instead they tended to (re)search for solutions to pressing problems. In several cases technicians carried around a folder stuffed with printed articles and hand written notes. Although eventual use of a crossorganizational ER-OMS may provide sufficient documents to enable expertise recommendation processes via text analysis such as suggested by [29], little is available for start-up processes.
Matching information-seekers to experts is not a trivial task. It requires an iterative process of finding people that are likely to have required knowledge, narrowing the list to those who the information-seeker is willing to approach, selecting the expert(s), and finally contacting the selected expert [22]. Three methods are used to build profiles: manual generation in which workers are asked to complete a survey or rate items, automated profiles which are built from watching user actions as they perform routine work or by mining artifacts located in databases such as human resources, or a hybrid approach that combines manual and automated techniques [21]. Manual profiles, while perhaps the easiest to design for, requires user investment. Survey or ranking exercises must be completed and updated as interests and areas of expertise change. It is therefore preferable to use automated processes whenever possible, though they are more time consuming to design. In [21], Marwick suggests that the state of the art process of profile-building is to use a hybrid approach. Although this provides the best balance between the work required by users and data captured within a profile, it increases design and maintenance costs.
8. ER PROFILE FINDINGS 8.1 Developing Profiles Many techniques for matching people together have been suggested. Existing literature has focused primarily on connecting people working on similar projects, having similar areas of interest, or connecting experts with information-seekers based on social networks and probability that an expert will have needed knowledge. Research in this area is not new (e.g. [24,25,26]).
Providing links between information-seekers and experts based on search history has excellent potential in this environment. All five sites within this study used a proxy server that required authentication before access to Internet resources could be achieved. Through capture and analysis of search history data, it is likely that connections between people will be among the most successful expert locating processes, providing opportunities for limited automated profiles. It is likely that automated processes will need to be supplemented with manual profiles techniques both in the short and long term.
Perhaps closest in spirit to our research is Reichling and Veith’s work on expertise sharing between a major European national industry association (NIA) and its 3,000 member companies [30]. Unlike our project, the researchers focused on NIA member sites locating expertise within NIA in an effort to make complex department structures more transparent, whereas we are interested in a fully meshed system of ER and memory sharing between all partner organizations. Regardless, we value their key insight yellow page systems can be used in combination with ER systems to provide transparency in complex environments to provide both
192
know maybe a 7 or an 8. So a level below from him… And maybe that’s not really fair, but that’s just the way I feel. And then as you keep moving out; it’s kind of like the ring of friends on phones. Unless I develop a relationship with Bob’s friend, if Bob’s friend trusts someone it wouldn’t necessarily mean that I would. It would probably go down… to a point that trust isn’t there by default anymore. And I’m not sure how far out that would be, maybe 2 to 3 people. Technician, Jefferson
8.2 Selection Through Social Computing While providing expertise recommendation based purely on the likelihood of a person’s expertise is important, selection of collaborative partners tends to be a social activity [9]. In order to provide a basis for selection activities, we looked at how collaboration currently takes place both within each organization and across organizations when such interactions occur. One of the well researched methods of social interaction occurs via e-mail channels. It is therefore not a surprise, particularly within a technology department, that collaborative connections are built and maintained through that medium. E-mail is used frequently when the asker believes that their question is straightforward with a likelihood of a quick answer. However, when the question is involved or requires rich context or multiple communicative interactions will be required, use of direct communication via face-to-face interaction or the telephone is generally used.
If it is entered into the database from someone in our organization that means that it is pretty valid…And so if someone here relies on maybe someone from a different department here or another school district, that would tell me that the person is probably pretty good too, and I would be likely to trust them and want to work with them too. So it’s almost like I’m adopting someone and working with them based purely on how I feel about my own colleague. And I would do it almost without questioning a lot or having to test their answers on a test box before really using it. And that’s a pretty powerful statement. Network Administrator, Homesdale
Most of the time just send one of my friends, my colleagues, an email message…But if it is a big problem that needs to get solved right away or I need to explain the problem and get feedback then I’ll call them…And then we can hash through whatever the problem is…Because on e-mail it can be more time consuming to describe a problem and then you end up with an answer that doesn’t really make enough sense, but you know from the answer you get that they really do know the answer, but just don’t understand what you are saying. Technician, Lakeview
Aside from selection based on social closeness, our participants had personal weighting schemes at the ready that combined both static and dynamic characteristics to arrive at selection preference. Interestingly, the weighting given to each of these characteristics tends to be the same within a position as opposed to by type of knowledge partnership desired. Technicians were very interested in the number of years of field experience, and consider experience within the education environment different from that outside of education. Experience was closely followed by certifications, and finally formal education. Meanwhile, network and database administrators valued certifications above years of experience, followed by education. Finally, IT Directors preferred to collaborate with employees with significant formal education, followed by experience and certifications. This demonstrates the unique profile requirement needs based on the information-seekers position; it cannot be said that one size fits all.
Well most of the time if my staff doesn’t know the answer or I’m dealing with something that they wouldn’t really be able to help with, like a policy decision, then I call one of the other tech directors…Because then we can discuss the pros and cons...Thing[s] can get complicated quickly and it is so dependent on how things are set up. So e-mail just doesn’t work well when there are a lot of situational things that affect the needs that you have. But if it’s just a quick question, like what do you think about product xyz then it is different, it’s pretty straight forward and there’s not a lot of interpretation there. IT Director, Prairieville
You can say that you’ve been to the courses, been to the classes and I’d probably look for that to make sure that they have some classical training, but I’m more interested in experience.…How long have they been working in the field? I myself have a lot of field experience. I don’t have no pieces of paper, but I’m just as good as people that do have paper. And maybe it’s because I only have an associate’s degree, but real education just doesn’t really apply here. It’s okay to have some theory to help you figure out problems, but it’s the experience that makes or breaks you. Technician, Newburry
Perceived social closeness is important for selection of collaborative partners; answers provided by friends come at a lower cost and expected reciprocity than those that come from people that are unknown, and inherent trust in the correctness of answers are greater [2]. But social closeness is not the only method of assessing trust in answers and experts. Our participants identified additional characteristics that they indicated would effect their selection of collaborative partners, including trust by proxy, experience, certifications, formal education, and position title. Participants indicated a strong desire to have these characteristics included within profiles to aid in appropriate selection of collaborative partners.
You go to a training and there is this guy that has this cert this and he has an alphabet after his name but when it comes to understanding real life situations… they just look at you and scratch there head because they don’t understand that you have a classroom full of kids and so you need to limit access but the teacher needs to get to it. But they will know how to do it even if they don’t understand why. And for example, database problems can be real theoretical, it isn’t likely that two of us even using the same database program will set it up exactly the same way, so being able to fall back on the theory, the certifications, is really critical. So for me I’d want certifications listed and I’d put a heavy weight on that, and … field experience, how to apply it. Database Administrator, Lakeview
Within small organizations it is increasingly likely that all members of the company know each other fairly well. Trust is quickly established or voided based on previous interactions. As organizations become more dispersed and / or partnerships are established across organizational boundaries, trust can be inferred – what we call here trust by proxy, explained best by our participants. So in that same situation, because I trust Bob I would assume that if Bob is talking to someone in another district and trusts that person, I should probably trust them too. It might not be the same level of trust though. Like let’s say I trust Bob an 8 or 9 out of 10. I would probably trust Bob’s friend that I don’t
Education man, education is really important. Because I deal a lot with politics and so there has to be an understanding of all of the different impacts of what we do and the repercussions.
193
organizations, matching site profiles becomes almost as critical as matching its people. For example:
Education gives you the ability to think through those issues and make informed decisions...And the combination of education and field experience is just awesome. Certifications, that’s great for my network guys, or even in some cases my techies, but for me it isn’t relevant at all. IT Director, Prairieville
I had a situation with our HP laptops and SATA, where if you image the laptop with SATA enabled the computer wouldn’t boot up until you go into the BIOS and turn SATA off. But with SATA off DVD’s would run really choppy. And so we had to figure out how to image a computer with SATA enabled on a Windows 2000 system. And so if your system gave me a list of people to call and none of them run Windows 2000 and HP laptops, there isn’t much probability that they would be able to help me solve the problem. I need to know about their environment in order to make an intelligent decision about who to call. Technician, Jefferson
The final component requested by our participants to assist them in selecting a collaborative partner is the person’s title. Title is related to both issues of trust and social closeness as well as in determining the likelihood that a person will have needed knowledge within a specific domain. For instance, if an employee carries the title of technician, it is more likely that they will have crossed paths with similar problems as other technicians, while less likely that they will be able to address network or database problems and even less likely that they will be directly involved in policy decisions. This is perceived as true, even though there is a large amount of cross-breeding (technicians completing network related functions such as setting up wireless networks, as was the case in Newburry). Nonetheless, there is a social barrier for a technician to go to another organization’s network administrator or IT Director – doing so would inevitably cause relationship problems with the technicians own department.
It’s just a matter of knowing what a district uses. Like if every district would put out a list of what they have, like we use Novel, we use NEC projectors, we use IBM laptops. And then you can look at that and say we have something in common. And call them up and say hey, have you experienced this problem, how have you handled this. Network Administrator, Newburry
Site information required for matchmaking purposes was fairly straight forward, and required two components: general component information such as types of computers used, operating systems, network equipment, and software titles, and general district information such as number of computers and people. The size of the district is primarily important to the IT Director. The explanation given was simply that both policy decisions and interests in new technologies (two areas of primary interest) are largely related to the size of their environment.
I’d be pretty upset if someone did that. If they didn’t come to me for an answer and let me make the contact if I didn’t know the answer. Because that’s my job, right? And it’s just not their place to do that. And I wouldn’t call another IT Director, and I really don’t think that one of our network technicians would go around me and do that either, unless I’m on some kind of long time leave or something. So if a list was displayed of people to contact you can almost just rule out people that don’t have the same title. Network Administrator, Homesdale
Well schools the same size as ours are more likely to be doing the same thing. We’ll have similar control issues. Like whether there are enough people to be able to support personal equipment or install software. And we can afford different things…A smaller school district might not be able to afford a Packeteer or might be able to support personal software because their IT department, they might be a little bit smaller than us but their people go around further than ours. So the size of the district really does matter. IT Director, Homesdale
The inclusion of titles within profiles is not as straightforward as it would appear. Within a single organization, titles are very static and indicate a job function throughout a company. However, when working with multiple organizations titles are nuanced. For instance, Prairieville employs network technicians which carry with it roughly the same job functions as computer technicians in Homesdale and Jefferson, and computer specialists within Newburry. In sites where there is no formal network administrator, IT Directors may perform the function, or it may be split between computer technicians and helpdesk personnel. Larger public school districts may separate network electronics administration functions and server administration functions. Thus, job categorizations may need to be defined anew within a unified cross-organizational ER-OMS, providing the opportunity for participants to select each duty that they perform instead of simply specifying their job title.
Conversely, both network administrators and technicians were primarily concerned with the type of equipment as opposed to quantities. Configurations on network electronics, processes of creating printer shares or setting security on network folders, for example, are highly dependent on the operating system being used. Similarly, issues that technicians come across are frequently dependent upon the make of computer system. Problem encountered with software titles are almost always unique to the title. Creating environment profiles could be an extremely time consuming task – and one that is likely to change on a yearly basis as new equipment is purchased and old equipment retired. By happenstance, our data collection activities coincided with an annual survey required by the state’s Department of Education. The results of the survey are compiled and made publicly available, aggregated by school district.
Each of the characteristics that we have described (education, certifications, experience, and title) are tracked within personnel databases. Although it would be valuable to form a direct link to personnel databases in order to establish automated profiles, it is unlikely that such access would be granted given the confidentiality of personnel records. Without such a link reliance on manual profiles would be necessary.
The state does it kind of in their [state] survey to get a profile for each school. What kind of connections do you have in each school, what kind of pc’s you are using, what kind of infrastructure equipment. So you get a profile so that if I have a question, okay who all are the schools that have HP electronics and who can I talk to on that end of it. So if you get a profile for every district and you can start drilling down from there... So if you know that kind of stuff you can go to those
8.3 Site Profiles Requirements of ER-OMS developed for use within a single site are unconcerned with capturing site profiles for use in expertise selection as such information is understood by their employees. However, when expanding the system for use between
194
Throughout this work we have described the ways in which crossorganizational ER-OMS differ from traditional systems designed for in-house use, and have suggested potential solutions to design requirements elicited from participants engaged in real work and with actual knowledge needs.
different schools and have common issues with that. IT Director, Lakeview
In addition, each school district is required by state auditors and their insurers to maintain an inventory of all computer systems. Between the state required surveys which are available to the public, and inventory databases available to each district, it is conceivable to automate site characteristic profiles with little additional interaction from employees.
Specifically, our research suggests that: •
Including site information as a component of expert recommender systems will enhance the ability to select collaborative partners and increase efficacy of knowledge sharing between organizations.
•
Including personal characteristics such as level of education or certifications will assist information-seekers during selection processes when recommendations contain unknown individuals.
•
Trust and relevance of stored knowledge degrades as it moves outside an organization and outside the purview of a social network. New processes of establishing real or artificial trust need to be developed to enhance crossorganizational knowledge sharing.
•
Rigidity of forced ontologies remains a difficult problem which may influence creation of individual repositories and cause group memory repositories to become incomplete and outdated. Social tagging offers opportunity to overcome rigidity, reduce cognitive load, and form boundary objects within a situated ontology.
9. CONCLUSION ER-OMS are designed to meet an organization’s needs by providing a repository for organizing and storing knowledge that can then be acted upon to identify trends and future needs, solve current problems, and reduce overall operating costs by minimizing the need to recreate previously developed procedures and solutions. Although it is a technical system designed to support business needs, it is also entrenched in nuanced social processes that must be identified and taken into consideration during system design and implementation. Our research has been grounded in the pre-collegiate educational environment. Thus, while we caution the reader that our findings are specific to pre-college level educational institutions, many of the results may be transferable to similar situations. Our primary contribution was to expand prior research by identifying issues that must be addressed when designing and building an ER-OMS for use beyond organizational boundaries. As we have discussed, expanding ER-OMS to crossorganizational partnerships brings new challenges and system requirements. Among these challenges is the need to provide not only personal profiles that are used to select collaborative partners, but also to provide site characteristics in order to maximize contextual appropriateness of the knowledge sharing link. Site characteristics, personnel characteristics, and individualized areas of expertise are combined in a triumvirate that can be used by information-seekers during collaborative partner selection.
The work presented in this paper has begun the process of exploring the challenges and design requirements of crossorganizational knowledge sharing. We plan on extending this research by implementing a functional cross-organizational EROMS to test design criteria and to continue to explore the dynamics of sharing knowledge between small organizations. Future research will also focus on the efficacy of using social tagging mechanisms to form ontological schemas. Of particular interest is whether such socially formed schemas can successfully be used as boundary objects to enable knowledge reuse between disparate groups.
Matchmaking activities continues to be rooted in the social world. Although trust is often a non-issue within an organization’s boundaries due to the understanding of colleagues’ areas and levels of expertise, methods of trust-building and expertise selection processes must be included in cross-organizational EROMS wherever possible. Although some trust can be inferred through the presentation of social networks, as the expert becomes further removed from an information-seekers personal social network trust is likewise diminished.
10. REFERENCES [1] Ackerman, M.S. (1998). Augmenting Organizational Memory: A Field Study of Answer Garden. TOIS'98, 16(3), 203-224. [2] Ackerman, M.S., & D.W. McDonald. (1996). Answer Garden 2: Merging Organizational Memory with Collaborative Help. CSCW’96, 97-205. [3] Ackerman, M.S., & C.A. Halverson. (1998). Considering an Organization’s Memory. CSCW’98, 39– 48. [4] Ackerman, M.S., & C.A. Halverson. (1999). Organizational Memory: Processes, Boundary Objects, and Trajectories. HICCS’99, 1067 – 1079. [5] Allen, T.J. (1977). Managing the Flow of Technology. Cambridge, MA: MIT Press. [6] Berlin, L.M., R. Jeffries, V.L. O'Day, Andreas Paepcke, & Cathleen Wharton. (1993). Where Did You Put It? Issues in the Design and Use of a Group Memory. SIGCHI’93, 23-30. [7] Boster, J.S. (1986). Exchange of Varieties and Information between Aguaruna Manioc Cultivators. American Anthropoligist, 88(2), 428-436.
The process of designing a hierarchical structure to store explicit knowledge was fairly straightforward, though additional modifications to the hierarchy are expected throughout system implementation and maintenance. The participants in our study preferred occluding any hierarchical schema, avoiding the need to conform to rigid group standards while maintaining their ability to (re)locate stored knowledge via search engines. Social tagging systems such as del.ico.us may provide a mechanism to move placement and storage from a visible process towards a black box approach, allowing each member to tag captured knowledge with keywords of their own choosing. Future work is required to more fully understand the potential of social tagging processes when applied beyond a homogeneous group.
195
[23] McDonald, D.W. (2000). Supporting Nuance in Groupware Design: Moving from Naturalistic Expertise Location to Expertise Recommendation. Doctoral Dissertation, University of California, Irvine. [24] McDonald, D.W., & M.S. Ackerman. (1998). Just Talk to Me: A Field Study of Expertise Location. CSCW’98, 315324. [25] McDonald, D.W., & M.S. Ackerman. (2000). Expertise Recommender: A Flexible Recommendation System and Architecture. CSCW’00, 231-240. [26] Middleton, S.E., N.R. Shadbolt, & D.C. de Roure. (2004). Ontological User Profiling in Recommender Systems. ACM Transactions on TOIS, 22(1), 54-88. [27] Pentland, B. (1992). Organizing Moves in Software Support Hotlines. Administrative Science Quarterly, 37(4), 527-548. [28] Pipek, V., J. Hinrichs, & V. Wulf. (2001). Sharing Expertise: Challenges for Technical Support. Cambridge, MA: MIT Press. [29] Reichling, T., K. Schubert, & V. Wulf. (2005). Matching Human Actors Based on Their Texts: Design and Evaluation of an Instance of the Expertfinding Framework. GROUP’05, 61-70. [30] Reichling, T., & M. Veith. (2005). Expertise Sharing in a Heterogeneous Organizational Environment. ECSCW'05, 325-345. [31] Strauss, A., & J. Corbin. (1998). Basics of Qualitative Research. Newbury Park, CA: Sage Publications. [32] Sure, Y., S. Staab, & R. Studer. (2002). Methodology for Development and Employment of Ontology Based Knowledge Management Applications. SIGMOD Record, 31(4), 18-23. [33] White, K.F., & W.G. Lutters. (2004). Behind the Curtain: Lessons Learned from a Wizard of Oz Field Experiment. ACM SIG-GROUP Bulletin, 24(3), 129-136. [34] White, K.F., & W.G. Lutters. (2005). Insightful Illusions: Requirements Gathering for Large-Scale Groupware Systems. GROUP’05, 348-349. [35] Wu, H., M.D. Gordon, & K. DeMaagd. (2004). Document Co-Organization in an Online Knowledge Community. CHI’04 Extended Abstracts, 1211-1214.
[8] Boster, J.S., & J.C. Johnson. (1989). Form or Function: A Comparison of Expert and Novice Judgments of Similarity among Fish. American Anthropologist, 91(4), 866-899. [9] Brown, J.S., & P. Duguid. (2000). The Social Life of Information: Harvard Business School Press. [10] Conklin, J. (1996). Designing Organizational Memory: Preserving Intellectual Assets in a Knowledge Economy. Group Decision Support Systems, 1-35. [11] Davenport, T.H., & L. Prusak. (2004). Working Knowledge: How Organizations Manage What They Know. Ubiquity, 1(24). [12] Ding, Y., & S. Foo. (2002). Ontology Research and Development. Part I -- a Review of Ontology Generation. Journal of Information Science, 28(2), 123-136. [13] Floyd, C., & S. Ukena. (2005). On Designing Situated Ontologies for Knowledge Sharing in Communities of Practice. PHISE'05, 559-569. [14] Gruber, T.R. (1995). Toward Principles for the Design of Ontologies Used for Knowledge Sharing. International Journal of Human-Computer Studies, 43(5-6), 907-928. [15] Henderson, S. (2004). How Do People Organize Their Desktops? CHI’04 Extended Abstracts, 1047-1048. [16] Jurisica, I., J. Mylopoulos, & E. Yu. (1990). Using Ontologies for Knowledge Management: An Information Systems Perspective. ASIS’90, 482-496. [17] Kao, A., L. Quach, S. Poteet, & S. Woods. (2003). User Assisted Text Classification and Knowledge Management. CIKM’03, 524-527. [18] Lutters, W.G., & M.S. Ackerman. (2002). Achieving Safety: A Field Study of Boundary Objects in Aircraft Technical Support. CSCW’02, 266-275. [19] Lutters, W.G., M.S. Ackerman, J. Boster, & D.W. McDonald. (2000). Mapping Knowledge Networks in Organizations: Creating a Knowledge Mapping Instrument. AMCIS’00, 2014-2018. [20] Mackinlay, J.D., & P.T. Zellweger. (1995). Browsing vs. Search: Can We Find a Synergy? CHI’95 Conference Companion, 179-180. [21] Marwick, A.D. (2001). Knowledge Management Technology. IBM Systems Journal, 40(4), 814-830. [22] McDonald, D.W. (1999). Recommending Expertise in an Organizational Setting. CHI’99, 63-64.
196
