Scoring Key. Criteria based on Drummond & Jefferson[7], Centre for Reviews and Dissemination[8], Evers et al.[9], and Machado[10]. Scores. Criteria. 0. 0.5. 1.
Appendix 2 - Quality Assessment Scoring Key Criteria based on Drummond & Jefferson[7], Centre for Reviews and Dissemination[8], Evers et al.[9], and Machado[10]. Criteria 1. Is there a research question or definition of study aim? 2. Are the primary outcome measure(s) for the economic evaluation stated? 3. Is the study sample provided and described? 4. Is the health information system being evaluated described? 5. Is the study time horizon stated? 6. Are the data collection methods described? 7. Are the analysis methods used described? 8. Are the results reported clearly with any caveats? 9. Do conclusions follow from the results and answer the study question/objective? 10. Are generalizability issues addressed along with limitations?
0 No.
Scores 0.5 Yes, but needed to be inferred from text.
1 Yes, clearly stated.
No.
Yes, but needed to be inferred from text.
Yes, clearly stated.
No.
Yes, sampling frame or population described.
No, not clearly.
Yes, type of system clear but not described.
Yes, sample characteristics and size stated. Yes, system clearly described in terms of functionality.
No.
Yes, but not clear.
Yes, clearly stated.
No, not clearly.
Yes, somewhat but not fully.
Yes, clearly described.
No, not clearly.
Yes, somewhat but not fully.
Yes, clearly described.
No, incomplete reporting.
Yes, results partially reported.
Yes, results clearly reported (e.g. in tables).
No, not clearly.
Yes, to some extent but not fully.
Yes, conclusions match results and answer main questions.
No.
Yes, somewhat but missing clear statements.
Yes, clear statements on generalizability of results and limitations.
For each paper, two reviewers independently assigned scores of 0, 0.5, or 1 to each item. This gave a maximum quality score range of 0 to 10. If the two reviewers’ scores for a specific item were within one point, an average score was used. If the score differed by more than one point, the reviewers met to reach consensus.
Summary of Quality The majority of studies scored very well in terms of having all criteria items reported. 33 out of 42 papers had scores above 8, with 11 papers achieving a score of 10. 8 papers had overall scores between 5 and 7.5, indicating adequate quality. Only one paper was determined to be of low quality in terms of the items we were looking for. This study scored poorly for reporting results, conclusions and generalizability. Across all ten criteria items, the two items with the lowest overall scores were for descriptions of analysis methods and reporting generalizability/limitations. Virtually all papers received a score of 1 for stating research aims and primary outcome measures. However, it should be noted that this quality assessment considered whether each item was reported but not the actual content of the item. For example, a study may have reported a data collection method but we did not assess whether the method was appropriate or comprehensive. Some individual items were analyzed in greater detail during the synthesis portion of the review.
![Appendix 2 [PDF]](https://m.moam.info/img/260x300/appendix-2-pdf_647e28d1098a9e464d8b45a4.jpg)
![Appendix 2 [PDF]](https://m.moam.info/img/260x300/appendix-2-pdf_64869400098a9e25508b460e.jpg)
