Physician's Information Customizer (PIC): Using a ... - CiteSeerX

Proceedings of the International Conference on Medical Informatics (MEDINFO’95), July 1995.

Physician’s Information Customizer (PIC): Using a Shareable User Model to Filter the Medical Literature Wanda Pratt, M.S. and Ida Sim, M.D. Section on Medical Informatics, Stanford University School of Medicine. MSOB X-215 Stanford, California 94305, USA The practice of medicine is information-intensive. From reviewing the literature to formulating therapeutic plans, each physician handles information differently. Yet rarely does a representation of the user’s information needs and preferences, a user model, get incorporated into information management tools even though we might reasonably expect better acceptance and effectiveness if the tools’ presentation and processing were customized to the user. We developed the Physician’s Information Customizer (PIC), which generates a shareable user model that can be used in any medical information-management application. PIC elicits the stable, long-term attributes of a physician through simple questions about his/her specialty, research focus, areas of interest, patient characteristics (e.g. ages), and practice locale. To show the utility of this user model in customizing a medical informatics application, PIC custom-filters and ranks articles from Medline, using the user model to determine what would be most interesting to the user. Preliminary evaluation on all 99 unselected articles from a recent issue of six prominent medical journals shows that PIC ranks 66% of the articles as the user would. This demonstrates the feasibility of using easily acquired physician attributes to develop a user model that can successfully filter articles of interest from a large undifferentiated collection. Further testing and development is required to optimize the custom filter and to determine which characteristics should be included in the shareable user model and which should be obtained by individual applications. 1. INTRODUCTION A major challenge in medical informatics is the seamless integration of computer applications into the hectic day-today work of health care providers. This integration should be driven not by hardware or software imperatives but by a model of the needs and preferences of the user, a user model. If this user model were shareable, the same information about a clinician may be used to customize any information management tool that the clinician may use. Tools and interfaces such as those for education, decision support, and information retrieval (Figure 1) would all look and interact more as the user would prefer, and seamless integration from the user’s perspective would be facilitated. Our project focused on the use of a shareable user model to assist with the particularly information-intensive task of keeping up with the medical literature. In 1989 alone, 360,000 new articles were indexed in Medline, an online biomedical bibliographic service. However, for any one physician, only a small fraction of these articles are of interest. Precious time and energy must be spent to identify these articles and to ensure that no important articles are overlooked. We designed the Physician’s Information Customizer (PIC) to assist physicians in this informationintensive task by generating a shareable model of the user’s stable attributes and using it to automatically select articles most likely to be of interest to that user. By answering only simple questions about his/her specialty, information preferences, and interests, a physician using PIC would be able to save time and effort in keeping up with the medical literature. Previous work on tools for developing shareable user models include GUMS [1], UM [2], BGP-MS [3], and UMT [4]. As PIC does, these systems capture what Rich calls the long-term characteristics of users [5], and they use both explicitly acquired and inferred information to form a user model. All except UM use Rich’s user stereotype method, in which a user is inferred to belong to a particular class and thus inherits the stereotyped characteristics of that class. PIC applies this method to the medical domain. Stable attributes of physician users are used both explicitly and implicitly via stereotypes to generate shareable user models of physicians.

Applications Information Retrieval

Static User Profile Dynamic Patient Record Current Needs

Article Critiquing Shareable User Model

Decision Support

Medline Citations

Information Filtering User Interfaces

Filtered Citations

Educational Tools

= components of the Physician’s Information Customizer

Figure 1: Role of a Shareable User Model in Medical Informatics Applications 2. PROJECT GOALS The primary functional goal of PIC was to help a physician keep up with the medical literature by providing a file of recent Medline citations that have been scored and ranked for that specific user. Our goal was not to develop a comprehensive literature filtering system, but to demonstrate that a shareable user model could be used to customize a literature filtering system. In particular, the heuristics we used do not provide a complete mapping to the more than 130,000 Medical Subject Heading (MeSH) terms used in the UMLS Metathesaurus [6]. We have focused only on rules that exploit attributes of the user model to discriminate among the citations in our test set of 99 articles. The coverage could easily be expanded by adding more rules to link the user model to MeSH terms. The primary design goal of PIC was to generate an intermediate representation of the physician’s needs and preferences, which could be used to customize many different medical informatics applications. To facilitate the reusability of the user model, PIC needed to be modular and easily expandable. We also realized that the heuristics used to generate the user model might not be correct for every physician, so we allowed physicians to modify the user model if they desire. A search of the literature confirms our belief that the user model captures only an initial view of the user’s needs and preferences [5,8]; application-specific and iterative run-time user modeling can build on this preliminary model to refine a dynamic model of the user. 3. SYSTEM DESCRIPTION Architecture. PIC consists of three modules: the model generator, the custom filter, and the user interface (Figure 2). The model generator takes the user profile and generates a more detailed user model in a frame-based representation language; this model can then be exported for use by any application. The custom filter uses the user model to score, rank, and select citations from a Medline search, yielding a customized citation file for the user. The model generator and custom-filter modules were both developed using the CLIPS expert system shell; they share an ontology, but are otherwise independent. The Hypercard user interface is for acquiring the user profile, but it also allows users to create and view a user profile; generate, view and edit the shareable user model; customize a citation file; and view the customized citation file. The user profile is obtained by asking the user as few questions as possible about the areas shown in Figure 3. We focused on questions that would be most relevant for filtering the literature.

2

Model Generator

User

Shareable User Model

Profiles

Hypercard Interface

Unscored Citations Custom Filter

Customized Citation File

Program Modules Inputs and Outputs

Figure 2: System Architecture

Patient Characteristics

Practice Locale

Yes

Introduction

Specialty

Seeing No Patients

Research/Clinical Interests Policy Interests

Other Interests

Profile Complete

Administrative Interests

Figure 3: User Interface Flow Diagram Data Structure and Algorithms. Since our primary design goal was to develop a user model of physicians that would be shareable among many different applications, we chose the general frame-based representation language for CLIPS to represent keywords, citations, and physician attributes. We used the production rule facility in CLIPS to represent the rules for generating the user model and rules for custom-filtering the citations. The model generator uses stereotyping rules to extrapolate from the user profile to a richer model of the physician. For example, when a physician states that her practice is located in a rural community, PIC infers that some of the physician’s patients are employed as farm workers. Other rules map from the shareable user model to desired keywords. For example, one rule concludes that "agrochemicals" and "insecticides" are desired keywords for physicians whose patients are farm workers. The weight of the keyword is incremented by an importance factor whenever an interest in the keyword is concluded. The importance factors range from 1 to 5 in accordance with the keyword’s significance in differentiating potentially interesting from uninteresting articles. For instance, if a physician is merely interested in cardiology, the keywords associated with cardiology would only be incremented by 1; whereas if the physician’s specialty is cardiology, the cardiology keywords are incremented by 5. When the appropriate weights have been established for all the desired keywords (keywords are all MeSH terms), each citation’s score is incremented by the weight of every desired keyword it contains. Each citation is then ranked from the highest to the lowest score, and those articles with scores above a threshold are saved to the customized citation file.

3

4. EVALUATION Users were given only the same citation information that PIC uses, and their ranking of citations by interest level was compared to PIC’s ranking for them. The beginning citation set was all the 99 articles and editorials from a recent edition of each of The New England Journal of Medicine, Annals of Internal Medicine, American Journal of Public Health, Journal of General Internal Medicine, Science, and Nature. We excluded letters and news articles because they are usually not relevant to patient care. These six journals represent prominent journals in clinical internal medicine, in general medicine, and in general biomedicine, and they approximate the breadth of articles that may be of interest to physicians in internal medicine. Subjects. The subjects were eleven internal medicine physicians ranging from non-practicing academicians to fulltime clinicians. Most were practicing physicians interested in primary care, medical informatics and medical education. Ten of the eleven were engaged in research. Procedure. The subjects entered their user profile into PIC and were given only the title, journal name, keywords, and publication type for each of the 99 articles. The abstract was not provided since PIC is not capable of interpreting text. Each user ranked the citations from 1 to 4 according to the scale in Table 1. They did this without seeing the PIC ranking for their user profile. Ranking 1

Interest Level High

2

Moderate

3

Low

4

Not at all

Explanation I am very interested in this article. It would be a major error if PIC did not select this article for me. I am interested in this article. It would be an error if PIC did not select this article for me. I am somewhat interested in this article, but I’d only scan this article if I had the time. It would only be a minor error if PIC did not select this article for me. I am not interested in this article. It would be an error if PIC selected this article for me.

Table 1: Article Ranking System Calculation of Similarity Scores. We used two metrics for evaluating PIC’s performance: an ordinal and a binary similarity score. For both metrics, the percentage of all citations that were assigned to each interest level was determined for each user. The PIC-ranked citations for that user were divided in a similar proportion into the four groups. For each citation that PIC placed in the same group that the user did, a full match was allotted. For each citation that was one group away from where the user placed it, a half-match was allotted. The ordinal similarity score is the total number of matches divided by the total number of citations (99). It reflects PIC’s ability to order the articles as the users did. The binary similarity score is the percentage of citations that PIC correctly classified as to whether the user would read it (ranked 1-3) or not read it (ranked 4) and reflects PIC’s ability to discriminate which articles are of interest. Results. Of the articles ranked 1 to 3 by the users, 69% were correctly ranked as 1 to 3 by PIC. Of the articles ranked 4 by the users, 69% were correctly ranked as 4 by PIC. PIC was thus moderately good at the binary discrimination of what is and is not of interest. The ordinal similarity score was 66%, showing moderately good ordering ability. No significant difference (p > .10) was found between the scores of those who ranked the citations before or after using PIC, showing that entering the user profile did not bias the user’s scoring of the citations. Further evaluation. Comparing PIC’s performance to users who can read the abstract when ranking citations would be a more realistic measure of the real-world performance of PIC. We could also compare the rankings and keyword scores with the results from commercial systems such as Knowledge Finder. 5. DISCUSSION PIC shows how a user model can be used to custom-filter relevant citations from Medline. From the answers to a few simple questions about a physician’s interests and practice characteristics, PIC generates a shareable user model of the physician and uses this intermediate representation to select from a large file of recent Medline citations those citations that are likely to be of interest to that particular physician Shareable User Model. A logic-based representation, as opposed to a probabilistic representation, was chosen to facilitate sharing of the knowledge in the user model. However, the limited expressiveness of the representation

4

language for CLIPS severely hampered our ability to fully represent the physician’s attributes and our heuristics. For example, we could not represent the simple constraint that a physician’s specialty must be an instance of the class of medical specialties. A richer representation language such as the Knowledge Interchange Format (KIF) [7] would be essential for further work on the shareable user model. The level of generalization that is appropriate for the shareable user model is unclear. Should a physician’s interest in a specific disease be coded in the shareable user model, or should it be coded as an application-specific attribute? An analysis of the user modelling needs of each type of information-management application would clarify the desired contents of the shareable user model. The user model should be enriched by letting users express their interests while browsing medical terms and by incorporating questions about presentation preferences. Applications can then use dynamic run-time information to customize the user model even further. The usefulness of shareable user models will be fully demonstrated only by using the model to customize other information management applications and then evaluating the results as we have done here for the literature filtering task. The potential for these models to enable seamless integration from the user’s perspective can only be explored when several applications use the same shareable user model. Empirical data about the success of shareable models is sparse and the health care domain will be a challenging test-bed for these ideas. Literature Filtering Task. The literature filtering task was most constrained by the poor semantic representation of the citations. Our approach relied on MeSH keywords, which are often used inconsistently and often do not identify concepts in their proper context, especially with general concepts. The performance of PIC was particularly hindered by these limitations of MeSH indexing because of the very general interests of the users tested (e.g. primary care and medical education). In addition, many rules were required to map from the model of the user to the desired keywords. Exploiting the tree structure of the MeSH headings would reduce the number and complexity of these rules, but the inherent weaknesses of MeSH indexing would still be present. At the present state of technology, however, MeSH indexing is the most comprehensive indexing system available. This particular application of the shareable user model could also benefit from dynamic user model refinement. Conclusion. Preliminary evaluation showed that PIC was able to rank articles similar to how the users would rank them. This illustrates the advantages of using a physician user model to custom-filter relevant journal articles. Future work should apply this user model to customizing other information management applications in the medical domain in order to establish the proper form of this shareable user model of physicians. 6. REFERENCES 1. Finin TW. GUMS - A General User Modeling Shell. User Models in Dialog Systems, Kobsa, A. and Wahlster, W. (Eds.). Springer Verlag, London. 2. Kay J. UM: A Toolkit for User Modelling. Proceedings of the Second International Workshop on User Modeling, Honolulu, Hawaii. 3. Kobsa A. Modeling the User’s Conceptual Knowledge in BGP-MS, A User Modeling Shell System. Computational Intelligence (6). 4. Brajnik G; Tasso C. A Flexible Tool for Developing User Modeling Applications with Nonmonotonic Reasoning Capabilities. UM92 - Third International Workshop on User Modeling. Deutsches Forschungszentrum fur Kunstliche Intelligenz, Saarbrucken. 5. Rich E. Stereotypes and User Modelling. User Models in Dialog Systems. Kobsa, A. and Wahlster, W. (Eds.). Springer Verlag, London. 6. McCray AT; Aronson AR; Browne AC; Rindfleisch TC; Razi A; Srinivasan S. UMLS Knowledge for Biomedical Language Processing. Bull Med Libr Assoc. April 1993; 81(2). 7. Genesereth MR; Fikes RE. Knowledge Interchange Format, Version 3.0 Reference Manual. Technical Report Logic-92-1, Stanford University Logic Group, 1992. 8. Kok AJ. A review and synthesis of user modelling in intelligent systems. The Knowledge Engineering Review 1991; 6(1):21-47. 7. ACKNOWLEDGMENTS The authors thank Nora Sweeny for her editing advice, and M. Walker, L. Fagan and S. Shiffman for comments on an earlier version of this paper. Computing facilities were provided by the CAMIS resource, through grant LM05305 from the National Library of Medicine.

5