CLSI evaluation protocols

CL AL IBN IMC A NL AI GS ES MU E SN T

CLSI evaluation protocols By Dan Tholen

O

ver the years Clinical and Laboratory Standards Institute (CLSI, formerly NCCLS) has developed a comprehensive series of reference documents to promote best practices in laboratory testing throughout the world, using a consensusdriven process that balances the viewpoints of industry, government, and the healthcare professions. The evaluation protocol (EP) documents were among the first documents developed by an area committee of volunteer experts representing some of the most accomplished and dedicated laboratory scientists in the clinical field. The CLSI consensus process is rigorous to assure scientific integrity, practicality, and proper regard for public health and safety. This article details the relationship between CLSI EP documents and a laboratory’s needs for conducting different types of studies to evaluate the performance of instruments and clinical assays, as outlined in EP19-R — A Framework for NCCLS Evaluation Protocols; A Report. This report categorizes each EP guideline with respect to its evaluation type, which takes place when an assay is considered for use or placed in use by a laboratory. CLSI EP documents provide recommendations for the assessment of laboratory test procedures in a way that defines laboratory best practices grounded on scientific and statistical theory. CLSI evaluation protocols are primarily intended to be practical in their use in the clinical laboratory since all clinical laboratories must perform method-evaluation studies prior to using a new or revised test system for reporting patient test results. Manufacturers can also use these protocols as a means of generating performance data for their in vitro diagnostic (IVD) devices and/or assays in a form to which the laboratory can relate. In essence such documents provide a framework for evaluation studies that should result in greater consensus and standardization of practice.

For user verification of performance Accreditation requirements (particularly CLIA ’88) include the need to verify the performance of a method prior to use on patient samples. EP15-A2 — User Verification of Performance for Precision and Trueness; Approved Guideline — Second Edition and EP12-A — User Protocol for Evaluation of Qualitative Test Performance; Approved Guideline have been designed for these purposes. Basic procedures to meet accreditation requirements are detailed but can be modified to meet laboratories’ particular circumstances. All accredited clinical laboratories should use such documents routinely. EP10-A2 — Preliminary Evaluation of Quantitative Clinical Laboratory Methods; Approved Guideline — Second Edition provides a simple but powerful protocol for an early evaluation of a new instrument or for checking performance after major maintenance. EP15-A2 includes a five-day testing protocol and simplified worksheets for all data gathering, statistical calculations, and

tests of observed precision and trueness. A computer spreadsheet is provided to simplify implementation. This document is primarily intended for use when an established method is initially set up in the laboratory; however, it may also be used to verify method performance after corrective action following a failed proficiency-testing event. EP12-A provides a protocol designed to optimize the experimental design for the evaluation of qualitative tests; to better measure performance; and to provide a structured data analysis. This guideline provides evaluation protocols for the demonstration of qualitative test performance. EP12 is written for clinical laboratory personnel who are the end-users of such tests. Demonstration of test performance by the user can satisfy internal (as well as external) expectations that the test performs acceptably in meeting the user’s clinical and analytical goals. EP10-A2 gives a minimal procedure that can be used as preliminary indication of performance, and includes sample data sheets that simplify analysis of the data and assist in determining the possible causes of imprecision. This might not satisfy accreditation requirements for verification but can be useful as a troubleshooting tool, for a quick check when considering adoption of a method, or when modifying methods.

For more detailed understanding Laboratories may need to know more about their methods for specific applications or may need to meet more detailed accreditation requirements. These EP documents can also be used for describing performance for laboratories that develop or modify methods. They include the following: EP6-A — Evaluation of the Linearity of Quantitative Measurement Procedures: A Statistical Approach; Approved Guideline for verifying calibration or checking linearity across the measuring range. This guideline describes the required conditions, materials, and easy-to-follow instructions for evaluating linearity using the polynomial method, which is statistically rigorous and can be easily implemented by spreadsheet programs. For ease of use, the linearity protocol is illustrated in a flowchart in which each task has a parenthetical note referring to the appropriate section in the document (see Figure 1); references to appropriate sections in the text are in parentheses. EP17-A — Protocols for Determination of Limits of Detection and Limits of Quantitation; Approved Guideline for determining or verifying the lower limit of detection or LoD and lower limit of quantitation or LoQ of a method. The limits of detection and quantitation are critical because detecting extremely small amounts of an analyte can be necessary to define disease states, screen for disease, identify significant exposure, or to reveal the presence or absence of toxins, pollutants, carcinogens, contaminants, infectious agents, and illicit drugs. These two critical performance characteristics are also important in laboratory examinations for tumor markers, hormones, agents Continues on page 40

38

August 2006

■

MLO

www.mlo-online.com

LAB MANAGEMENT of infectious diseases, therapeutic drugs, and other tests where low values separate subjects into different disease or exposure categories. The EP17 protocols are practical for use with common clinical laboratory devices and include easy-to-follow flowcharts showing the steps involved in determining and/or verifying limits of detection and quantitation (see Figure 2). EP9-A2 — Method Comparison and Bias Estimation Using Patient Samples; Approved Guideline — Second Edition for a more rigorous verification of trueness, of bias relative to a reference procedure. This would also be useful when migrating from one method to another, or for aligning two different devices for the same measurement. For carrying out method-comparison evaluations, an overview of the experiment, sample data recording and calculation sheets, and an overview flowchart and a detailed flowchart for preliminary data examination are included. As an additional aid, a sample scatter plot and bias plot are introduced for those who are unfamiliar with these procedures (see Figure 3). EP21-A2 — Estimation of Total Analytical Error for Clinical Laboratory Methods; Approved Guideline for determining the total error of measurement, to check against laboratory goals for error. These protocols can be used to judge the clinical-laboratory acceptability of new quantitative analytical methods by using patient specimens or to monitor an assay’s total analytical error by using quality-control samples, and are applicable in settings with only one instrument, as well as to those with multiple laboratories and multiple instruments. EP18-A — Quality Management for Unit-Use Testing; Approved Guideline for quality control of unit-use devices. This is a comprehensive and flexible guideline that describes quality-management models to identify potential sources of errors in unit-use test systems related to specimen collection through reporting of results. The recommendations in this guideline are applicable to various devices and settings and are practical to implement, so that sources of error (potential failure modes) are identified, understood, and managed. This document complements the other CLSI document for quality control, C24-A2 — Statistical Quality Control for Quantitative Measurements: Principles and Definitions; Approved Guideline — Second Edition.

Figure 1. Flowchart for EP6 linearity procedure Design the study (2&3) and goal for error (nonlinearity and repeatability [5.3])

C

Conduct measurements according to design (3.8) Plot data on XY graph and assess visually (4.1)

Evidence of extreme nonlinearity (4.1)?

Yes

Find problem and correct

No Two or more

Yes

Outliers (4.1, 4.2)?

How many (4.1,4.2)?

No

B

One

Conduct !st-, 2nd-, and 3rd-order least-squares regression (4.3.2)

Remove outlier

Any nonlinear coefficients significant (4.3.2)?

Check repeatability against goal (4.4)

No

Stop

Error OK

Method is linear

Error not OK

Yes

A Choose the nonlinear model with the lowest (5.3.3) Calculate difference between nonlinear model and kinear model at all levels (5.3.3)

Find problem and correct

C

Not OK Any differences larger than error goal (5.3.3)

No

Check repeatability against goal (5.4)

OK

Method is linear

Stop

Yes Method is significantly monlinear over tested change

Can the range be reduced (5.3.3)?

No

Method is nonlinear

Yes Eliminate a point at high or low and where nonlinearity appears greatest (5.3.3)

B 40

August 2006

■

MLO

Stop

References to appropriate sections of the EP6 document appear in parentheses ()

For detailed descriptions of performance There are a number of other EP documents designed for detailed insight into method performance. This can be necessary for laboratories that modify methods or develop in-house methods. These documents are also used extensively by IVD manufacturers and by the U.S. Food and Drug Administration to review applications for approval or clearance of new methods. EP5-A2 — Evaluation of Precision Performance of Quantitative Measurement Methods; Approved Guideline — Second Edition gives a rigorous protocol for determining the precision of quantitative methods across the entire measuring range. Included are guidelines for the duration, procedures, materials, data summaries, and interpretation techniques that are adaptable for the widest possible range of analytes and device complexity. A balance is created in the document between complexity of design and formulae, and simplicity of operation. EP7-A — Interference Testing in Clinical Chemistry; Approved www.mlo-online.com

CLSI EP DOCUMENTS lection and analysis worksheets; and detailed examples. There is a balance between consistency of structured protocols and flexibility to accommodate the technology being evaluated. EP14-A2 is of use for evaluating reference materials (or proficiency-test samples) and determining whether they perform the same as patient samples. The document is informative about the impact of matrix effects on the assessment of the quality of laboratory performance, and provides guidance in evaluating the bias in analyte measurements that is due to the sample matrix (physiological or artificial) when two measurement procedures are compared. In summary, the CLSI EP series is applicable for most laboratory tests and measurements and provides invaluable insight into method performance. The use of these protocols, and comments and participation in the ongoing efforts to improve them, are encouraged. 

Figure 2. Flowchart for determination of LoD and LoQ B Choose desired β level Test lowest (untested) positive level

Use parametric statistics (mean and SDs)

Yes

Are data normally distributed?

Produce new material at LoDT

Calculate DS,β =AV• -pβ

LoDT =LoB+cβ SDST

Yes

Yes

LoDT =LoB+DS,β

Measure > 20 times obtain SDST

Can we assume that the SDS at LoDT is the same?

Use nonparametric statistics (ranked data)

No

Calculate Trial LoDT =LoB+cβ SDS

Is SDS constant?

No

LoDT =LoB+cβ SDST

Yes

LoD =LoDT

Try again with higher level material

Is achieved β percentile greater than LoB

D Determine goal for Total Error (TE)

Will you establish or verify LoQ?

Verifiy

Select samples at claimed LoQ and obtain n>25 replicates

Dan Tholen, MS, Traverse City, MI, is a consultant and trainer in statistical methods for laboratory quality. His clients include those in the commercial, professional, and governmental areas, in all fields of laboratory measurements. He is a member of the CLSI Area Committee on Evaluation Protocols.

CLSI welcomes comments and questions about the documents; this feedback serves as the basis for updated document editions. All comments and responses are formally addressed and published in the next edition of the document. For more information about Clinical and Laboratory Standards Institute references and best practices, visit www.clsi.org or call 610-688-0100.

Establish Count number of results that meet error goal

Calculate TE for LoD test results

LoQ=LoD

Yes

Figure 3. The overview flowchart of protocol from EP9-A2 for the examination of process

Is TE less that goal?

Is the number greater than Table 2?

No

Choose higher level material

LoQ is not verified

Test material n>40 replicates Calculate TE for new level test results

Device familiarization No

No Yes

Options: Contact manufacturer. Test highter level material to establish LoQ

Is TE less than goal?

Yes

LoQ+New level

Stop

Claimed LoQ is verified

Guideline and EP14-A2 — Evaluation of Matrix Effects; Approved Guideline — Second Edition both provide protocols to help identify sources of error that can affect patient care and/or assess the suitability of a method. EP7-A gives a protocol for detecting interfering substances and describing the effect of those substances. This guideline contains background information on interference testing concepts; tables of recommended test concentrations for analytes and potential interference; data colwww.mlo-online.com

Data collection Test duplicates within method outlier Check (X&Y) (Section4.1) Plot Scatter (YvsX (Section 4.2) . vs. concentration[(Y-X)vs.X] Plot bias Visual Linearity Check (section 4.3)

Not OK

Is remaining range adequate?

Truncate, if possible

No Yes OK More than one outlier Visual outlier between Investigate method check (section 4.4) Investigate One outlier or none 2 Yes r less Can range Adequate range test be extended? (section 4.5) than 0.95 r2greater than or. equal to 0.95 No Use partitioned biases Compute regression procedure (section 6.2) (section 5.1) Visual uniform scatter . check (section 5.25)

Not OK

Use partitioned residuals procedure (section 6.3)

Use linear regression procedure (section 6.1) Compare average and maximum bias estimates to claims or internal criteria (section 7)

MLO

■

August 2006

41