284
April 2003
Family Medicine
Facu lty Development
Clinical Research in Family Medicine: Quantity and Quality of Published Articles Joel Merenstein, MD; Goutham Rao, MD; Frank D’Amico, PhD
Background and Objectives: Publication of clinical research in peer-reviewed journals is an important measure of scholarly productivity. This study determined the quantity and quality of original clinical research published by family physicians. Methods: We surveyed clinical research papers published in the year 2000 in four leading family medicine research journals and research originating in a family practice institution but published in 16 non-family medicine journals. All were selected on the basis of relevance to family physicians and “impact factor.” The relevance and validity of papers was assessed using previously established criteria. Results: The survey of family medicine journals revealed a total of 170 original research articles. Ninety eight were from academic family practice programs, and the remaining 72 were from other medical specialties or health care institutions. Most of the papers were cross-sectional surveys. There were seven qualitative studies, six randomized controlled trials, and no systematic reviews from family practice programs in these journals. Eight of the articles were from practice-based research networks. A total of 79 articles were considered relevant or highly relevant, and 22 of these were also considered valid (Patient-oriented Evidence That Matters or POEMs). The survey of 16 non-family medicine journals revealed 37 clinical research papers: 16 surveys, nine prospective cohort studies, seven randomized controlled trials, three systematic reviews/meta-analysis, one qualitative study, and one casecontrol study. There were nine “highly relevant” papers—seven could be classified as POEMs. Conclusions: Most clinical family medicine research uses less-rigorous study designs, such as the crosssectional survey. The majority of papers do not meet established criteria for relevance and validity. There are no standards or comparable studies to compare these results to prior years or to other disciplines. (Fam Med 2003;35(4):284-8.)
A recent report by Weiss indicated that the number of publications by Society of Teachers of Family Medicine (STFM) members is declining.1 This decline, Weiss believes, if sustained “would threaten the academic viability of our discipline.”Weiss also questioned the need for additional family medicine journals—“W here would we get the high-quality family medicineauthored articles necessary to fill other journals? . . .” STFM leaders in an accompanying editorial proposed that producing high-quality relevant research is difficult but must remain part of our mission.2 By contrast, Dickinson noted that the quality, if not the quantity, of family medicine research submitted for the STFM Annual Spring Conferences is improving.3 Ewigman be-
From the University of Pittsburgh Medical Center, St Margaret.
lieves that we are on track in developing research in our discipline.4 Such contrasting opinions must also take into account the cessation of publication of the Archives of Family Medicine because “The current quantity and quality of scientific research by family physicians is simply not yet sufficient to support three research-based family medicine journals”5 and the recent announcement by the editor of the Journal of Family Practice that the journal would no longer publish original research. In an editorial in the last issue of the Archives, Weiss stated that “We need good family medicine research to identify the best way to practice, and unless family medicine research defines the best way for family physicians to practice, the standard of care for family physicians will be determined by someone else.”6 Opinions about the state of family medicine research are clearly numerous, and there are reports of the productivity of re sear ch in f amily medicine, 7 ,8 but a
Faculty Development systematic assessment of both published research quantity and quality is lacking. Our paper aims to fill this void. Our feeling is that clinical or patient-oriented research is most relevant to practicing family physicians. This, therefore, is the focus of our assessment. A critical review of publications in our own journals and publications by family physicians in other journals should provide a valuable gauge of the state of research in our specialty and, indirectly, whether our own research guides the way we practice. Our study, therefore, identified the number of clinical research articles published by family medicine researchers in family medicine journals and non-family medicine journals and assessed the relevance and validity of those articles. Methods Retrieval of Articles The principal US family medicine research journals that were in print in the year 2000 were the Journal of Family Practice, the Journal of the American Board of Family Practice, the Archives of Family Medicine, and Family Medicine.All clinical research articles published in these four journals in the year 2000 were retrieved. We defined “clinical research” in the following way: the study of actual human patients or the records of actual human patients or theoretical models that deal primarily with patient care or decisions that affect patients directly (eg, decision analysis). By this definition, for example, studies dealing with the practice patterns of physicians, the effect of public policy on outcomes, or educational interventions for medical students or residents were excluded. The titles, authors, and institutional affiliation of papers were collected and entered into a database. Papers were classified as originating in a family practice institution if at least one of the authors was affiliated with a family medicine department, division, residency, health center, or private practice. The number of papers originating in a practice-based research network was recorded separately. We also retrieved articles originating in family medicine institutions but published in non-family medicine journals. Since it would be impossible to survey all nonfamily medicine publications, we selected those journals most likely to have content relevant to family physicians and with the highest “impact factor”13 —a measure of the average frequency with which articles in a particular journal are cited each year. It is a gauge of the importance or “impact” of a journal. Using the criteria of relevance to family physicians and highest impact factor, the 16 non-family medicine journals listed in Table 1 were selected and searched by hand for the year 2000.
Vol. 35, No. 4
285
Table 1 Non-Family Medicine Journals From Which Family Medicine-authored Papers Were Identified • • • • • • • • • • • • • • • •
New England Journal of Medicine Journal of the American Medical Association Lancet Annual Reviews of Medicine Annals of Internal Medicine Archives of Internal Medicine American Journal of Medicine British Medical Journal American Journal of Obstetrics and Gynecology British Journal of Obstetrics and Gynecology Obstetrics and Gynecology Pediatrics Journal of Pediatrics Journal of the American Geriatrics Society American Journal of Sports Medicine Journal of General Internal Medicine
Evaluation of Articles The study design used in each paper was determined independently by two of the authors. Disagreements were settled by the third author, a biostatistician. Each paper was put in one of the following categories: randomized controlled trials, prospective cohort studies, case-control studies, meta-analyses/systematic reviews, case series, surveys, and qualitative research. Case reports were excluded. The relevance and validity of each paper was assessed independently by two authors using the evidence-based medicine wor kshee ts designed by Slawson and Shaughnessy10 that are based on the work of Sackett et al.11 Articles were classified as “highly relevant” if they met all three relevance criteria, “relevant” if they at least described a problem frequently encountered by family physicians (second relevance criterion), and otherwise “irrelevant” (Table 2). Articles were classified as “valid” if they met the key validity criteria of the appropriate
Table 2 Classification of and Criteria for Relevance Relevant=criterion #2 only Highly relevant=all three criteria 1. Did the authors study an outcome patients would care about? 2. Does the article address a specific clinical question that you encounter frequently in your practice? 3. Will this information, if true, require you to change your current practice?
286
April 2003
Family Medicine
worksheet without any “fatal” flaws. Lack of proper blinding, for example, would make a randomized-control trial fatally flawed. A complete description of validity criteria can be found elsewhere.12 Disa greements about relevance or validity were settled by consensus in discussions with the third author. Highly relevant and valid papers were classified as Patient-oriented Evidence That Matters (POEMs).12
Cross-sectional surveys were more common than other designs (Table 4). The majority of the 20 randomized-controlled trials were from non-family practice institutions. Among the 170 clinical studies, 79 (46%) were rated as either relevant or highly relevant. Twenty-two (13%) were considered highly relevant and valid, ie, POEMs (Table 5).
Results Papers From Family Medicine Journals A total of 170 clinical research articles were found published in the four family medicine journals during 2000. Ninety eight (58%) originated from family practice departments or residencies. Of these, eight involved practice-based research networks. The remaining 72 articles (42%) were from family medicine researchers, other academic departments, or other health care institutions. The Journal of Family Practice published 57% of the 98 papers originating from family medicine programs. (Table 3).
Papers in Non-Family Medicine Journals The hand search of the 16 journals selected on the basis of impact factor yielded an additional 37 clinical research papers that used the following designs: 16 surveys, nine prospective cohort studies, seven randomized controlled trials, two systematic reviews, one metaanalysis, one qualitative study, and one case-control study. Relevance assessment revealed 12 irrelevant papers, 16 relevant papers, and nine highly relevant papers. Seven papers (19%) were classified as POEMs. In addition to the clinical research articles, there were 15 health policy and seven educational research articles.
Table 3 Breakdown of Clinical Research by Family Medicine Journal and Type of Institution Total 74
Family Medicine 56 (57%)
Non-Family Medicine 18 (25%)
Archives of Family Medicine
74
26 (27%)
48 (67%)
Journal of the American Board of Family Practice
17
13 (13%)
4 (6%)
5
3 (3%)
2 (3%)
170
98 (58%)
72 (42%)
Journal Journal of Family Practice
Family Medicine Total
Discussion Only 58% of clinical research papers published in the four leading family medicine journals originated in family practice institutions. Are family physicians sending their best papers elsewhere? The answer is probably no, since only 37 clinical research papers were found in 16 non-family medicine journals with a highimpact factor. Nineteen percent of these were classified as POEMs, compared with just 12% of clinical research papers published by family physicians in family medicine journals. The majority of published papers we reviewed used a survey design, the least sophisticated of research methodologies. Qualitative research 14 has been proposed as an ideal design, and practice-based research networks15 have been proposed as an ideal vehicle for Table 5
Table 4 Frequency Distribution of Research Design in Family Medicine Journals Research Design Survey/cross section Cohort Randomized trial Qualitative Case-control Systematic review Case series Decision analysis
Total 96 30 20 10 6 6 1 1 170
Family Medicine 60 (61%) 19 (19%) 6 (6%) 7 (7%) 5 (5%) 1 (1%) 0 0 98 (58%)
Non-Family Medicine 36 (50%) 11 (15%) 14 (19%) 3 (4%) 1 (1%) 5 (7%) 1 (1%) 1 (1%) 72 (42%)
Frequency Distribution of Relevance and Validity of Papers in Family Medicine Journals by Type of Institution
Relevance Irrelevant
Total (n=170) 91
Family Practice Institution (n=98) 43 (44%)
Non-Family Practice Institution (n=72) 48 (67%)
Relevant
34
22 (22%)
12 (17%)
Highly relevant
45
33 (34%)
12 (17%)
Highly relevant and valid (POEM)
22
12 (12%)
10 (14%)
Faculty Development family medicine research. Few papers in our survey included either. It is encouraging, on the other hand, that about half of the articles published in family medicine journals were relevant for family physicians and that there were 22 POEMs. Unfortunately, only 12 of these articles came from family practice institutions. Comparative Studies There are no compatible standards for comparison of our results to determine if the number of publications was appropriate or how our results compare to the numbers in other disciplines. We only have data on non-primary care disciplines. Specifically, Powner and Kellum22 studied the research productivity of critical care physicians by looking at the three leading journals of that specialty. In 1999, critical care specialists published a total of 408 research articles; 169 were from US institutions. A number of previous reports, however, do address the issue of research productivity among family physicians. Geyman and Berg 16 reviewed articles published in the Journal of Family Practice’s first 15 years and described a subset for the 5 years from 1984–1988. There were an average of 150 papers published per year from 1984–1988. Of these, 63 per year would meet our criteria for clinical research, and six of these would be randomized trials. We do not know how many of these were from family practice programs. By comparison, our study revealed that in the year 2000, the Journal of Family Practice published 74 clinical research articles, including five randomized controlled trials (three from family practice institutions). Unlike during 1984–1988, there were, of course, two additional family medicine journals in 2000. Only an additional 42 clinical articles, however, were published outside of the Journal of Family Practice. It is interesting that in an editorial accompanying a similar article by Geyman and Berg, Pellegrino17 asserted that: A more-detailed analysis of content of the Journal of Family Practice and other journals, together with some objective assessment of quality and significance, is definitely in order. Were family practice to undertake such a study a few years from now, formally and in detail, it would do itself a service.
Muncie et al reviewed the designs and methods of the North American Primary Care Research Group abstracts for 1977–1987.8 Then, as now revealed by this paper, cross-sectional surveys were the most-common research design. Frey18 expressed concern at that time about the preponderance of the less rigorous survey design:
Vol. 35, No. 4
287
Such trends may produce research that lacks the methodologic strength to move medicine forward with as much certainty as is necessary to change the practice of medicine and the delivery of medical care.
In an analysis published in 1992, Ingram 19 noted that half of the publications by family medicine faculty were published in non-family medicine journals. Clinical research was not specifically studied. A more-recent paper revealed that neither family practice departments that made research a high priority nor those that made it a low priority published a significant number of papers.20 The editors of “Evidence-based Practice” found that only 2.6% of the 8,085 articles they reviewed in a 6month period could be classified as POEMs.21 This is a much lower percentage than in our study but cannot be compared because of their very large denominator and broad selection of journals. Limitations This study has a number of limitations. First, no single measure can encompass all the research being conducted by family physicians, and there may be an increasing body of research in progress that is yet to be published. Second, there is the possibility, though unlikely, that the year 2000 was a notably weak year for family medicine research. Third, we looked specifically at clinical research using a definition some may criticize as too “narrow.” Fourth, it is possible that family physicians are doing a considerable amount of goodquality research in the fields of medical education, health policy, practice management, etc, and we did not include such research. Fifth, we only searched the four leading family medicine research journals and 16 important journals that family physicians are likely to read. It is possible that there is a large volume of highquality clinical research published in other journals. Finally, the evidence-based medicine tools we used in our relevance and validity assessment have their own shortcomings. While all papers meeting POEM criteria are of practical importance to practicing family physicians, papers that do not meet POEM criteria are often still valuable. We came across many papers that addressed important questions that may not apply to the immediate care of patients. Further, important clinical questions are often first addressed through lessrigorous and less-expensive research designs such as the case-control methodology. These often form the basis for more-intensive study through randomized-controlled trials, etc. The POEM criteria do not take this progression into account. According to the criteria, for example, no case-control study about a new therapy can ever be considered a POEM.
288
April 2003
Family Medicine
Conclusions We can conclude that some family medicine researchers are asking appropriate, relevant questions. There is, however, a lack of research that uses high-quality designs and is highly relevant and valid. With 129 departments and 466 family practice residency programs, we should be able to sustain more than one clinical research journal and produce more patient-oriented clinical research studies in 1 year. Let us also keep in mind that concerns about the quantity and quality of research are not unique to family medicine.23 Whatever one’s perspective on the quantity and quality of family medicine research, it is hoped that this paper encourages family physicians to take a critical look at the research activities of our discipline. Few would argue that there is no room for improvement. Corresponding Author: Address correspondence to Dr Merenstein, University of Pittsburgh Medical Center, St Margaret, 3937 Butler Street, Pittsburgh, PA 15201. 412-622-7343. Fax: 412-621-8235.
[email protected].
REFERENCES 1. Weiss BD. Publications by family medicine faculty in the biomedical literature: 1989–1999. Fam Med 2002;34(1):10-6. 2. Rosenthal MP, Dickinson PW, Bogdewic SP, Sherwood RA. The state of family medicine research, or, stop, hey, what’s that sound? Everybody look what’s going down! Fam Med 2002;34(1):48-50. 3. Dickinson WP. Research at the STFM 2001 Annual Spring Conference. Fam Med 2001;33(7):539. 4. Ewigman B. Reinventing family practice again. J Fam Pract 2001; 50(7):586-7. 5. DeAngelis CD. Dear readers (editorial). Arch Fam Med 2000;9:961. 6. Weiss BD. Why family practice research? (editorial). Arch Fam Med 2000;9:1103-7.
7. Nathan DG. Clinical research: perceptions, reality, and proposed solutions. JAMA 1998;280(16):1427-31. 8. Muncie HL, Sobal J, DeForge BR. NAPCRG abstracts 1977–1987: analysis of research designs and methods. Fam Med 1990;2(2):125-9. 9. Sonis J, Joines J. The quality of clinical trials published in the Journal of Family Practice, 1977–1991. J Fam Pract 1994;39(3);225-35. 10. Slawson D, Shaughnessy A. Critical appraisal worksheets: Information Mastery Working Group. Adapted from material developed at McMaster University, 1997. 11. Sackett DL, Richardson WS, Rosenberg W, Haynes RB. Evidence-based medicine: how to practice and teach evidence-based medicine. New York: Churchill Livingston, 1997. 12. Slawson DC, Shaughnessy AF, Ebell MH, Barry HC. Mastering medical information and the role of POEMs—Patient-oriented Evidence That Matters. J Fam Pract 1997;45(3):195-6. 13. ISI. Th e imp act facto r. www. is in et.co m/isi/ho t/es s ays / journalcitationreports/7.html. Accessed February 11, 2002. 14. Murphy E, Mattson B. Qualitative research and family practice: a marriage made in heaven? Fam Pract 1992;9(1):85-91. 15. Green LA, Dovey SM. Practice-based primary care research networks: they work and are ready for full develo pment and suppo rt. BMJ 2001;322(7286):567-8. 16. Geyman J, Berg AO. The Journal of Family Practice 1974–1978. Window to an evolving academic discipline. J Fam Pract 1989;28(3): 301-4. 17. Pellegrino ED. Research in family practice as reflected by the Journal of Family Practice. J Fam Pract 1984;18(1):21-2. 18. Frey JJ. Five years of Family Medicine. Fam Med 1986;18(3):121. 19. Ingram TG. A cross-sectional analysis of family medicine publications in the indexed medical literature. Fam Med 1992;24(4):303-6. 20. Mainous AG III, Hueston WJ, Ye X, Bazel C. A comparison of family medicine research in research-intense and less-intense institutions. Arch Fam Med 2000;9:1100-4. 21. Ebell MH, Barry HC, Slawson DC, Shaughnessy A. Finding POEMs in the medicine literature. J Fam Pract 1999;48(5):350-5. 22. Powner DJ, Kellum JA. Declining critical care research publications b y au th ors fro m US in s titution s, 1 99 0– 19 99 . Acad M ed 2001;76(12):1261-3. 23. Campbell EG, Weissman JS, Moy E, BlumenthalD. Status of clinical research in academic health centers: views from the research leadership. JAMA 2001;286(7):845-6.
