Use of precision profiles to evaluate precision of the ... - CiteSeerX

ClinicalChemistry 42:7 1068-1073

(1996)

Use of precision profiles to evaluate precision of the automated leukocyte differential WOLFGANG

The

HUBL,*

Lus

TLUSTOS,

commonly

used methods of assessing the precision of leukocyte differential have certain drawbacks that affect the validity and comparability of results. In the present report, we introduce a procedure based on building precision profiles from a large number of withinrun imprecision experiments. The profiles are fitted to the function for the CV of proportions, which yields the number of theoretically differentiated leukocytes. Differences between fitted curves are evaluated for statistical significance by the F-test. As an example, we compared the precision of two hematology analyzers, a flow-cytometric technique involving fluorescence-labeled monoclonal antibodies, and the manual differential. We were able to establish definite differences in precision between different analyzers and different leukocyte classes. Our data also indicated that conventional within-run imprecision studies may completely misjudge analyzer precision. Furthermore, we could demonstrate that the precision of analyzers that analyze a fixed amount of blood rather than a fixed number of leukocytes is strongly influenced by the leukocyte count of the sample, leading to high imprecision for leukopenic samples. We believe the proposed procedure is a useful addition to currently used protocols; it yields clear results and creates a statistical basis of comparison between various instrwnents and techniques of differentiation.

count

TERMS:

#{149} method

hematology . flow cytometry comparison studies

#{149} leukocyte

Imprecision of the relative cell count is one of the main criteria in describing and comparing the performance of modern hematology analyzers. However, the currently used evaluation protocols show substantial shortcomings. Essentially, investigators have relied on two methods to evaluate imprecision. Some [1, 2] used the technique recommended by the National Committee for Clinical Laboratory Standards (NCCLS), evaluating pair differences obtained from duplicate runs [3], which can be

Central Lab, ViIhelminenspital, Montleartstr. *Author for correspondence. Fax Int + 431 Received August 2, l995; accepted February

37, A-i 171 Vienna, 49150 2776. 20, 1996.

PETER

MICHAEL

BAYER

conveniently combined with accuracy assessment. Because the NCCLS procedure involves a large number and great variety of samples, precision results are usually derived from analyses performed over several days to ensure that the data cover a variety of operating conditions. Although this technique is methodologically sound, it also has its drawbacks. The resulting CVs are strongly influenced by the mean size of the respective leukocyte class-usually being higher for minor leukocyte classes [1], which is misleading, as it wrongly suggests that differentiation of these populations is less precise. Second, the method may fail to reveal precision performance in the infrequent high- or low-range samples. Furthermore, duplicate analysis becomes considerably expensive and time-consuming when evaluating flow-cytometric methods involving manual sample preparation. More frequently, precision performance is determined on the basis of within-run imprecision obtained by replicate analysis of one or few samples [2, 4-8]. Complete misinterpretation of the data may result if only one sample is used because, for one thing, the conditions in effect at the time of that single run may not be representative of the usual operating conditions and, for another, the results may be distorted by specific sample characteristics. Therefore, this procedure is no longer recommended for method evaluation [9]. Moreover, testing an analyzer on the basis of only one sample detracts from the comparability of results, as the precision strongly depends on the percentage of the respective leukocyte class present [3, 10]. Using one normal, one low-range, and one high-range sample instead of a single sample may reduce some of these problems, but a valid comparison between different analyzers will, in most cases, remain impossible. After the realization that conventional tests were inadequate, the precision performance of immunoassays has for some time been based on precision profiles [Il, 12]. In the present study, we introduce a procedure in which we adapted the principle of precision profiles to the evaluation of the leukocyte differential of hematology analyzers. This method allows the evaluation of precision performance over the whole relative count range, facilitates statistical comparison between different instruments and techniques, and relates the performance to that of the manual differential, which makes its results easy to interpret. As an example, we present an evaluation of the neutrophil, lym-

the automated

INDEXING

and

Austria.

1068

Clinical Chemirtry

42, No. 7, 1996

1069

phocyte, and monocyte count imprecision of two hematology analyzers: the Coulter STKS and the Roche Cobas Argos 5 Diff. In addition, we related monocyte imprecision to that of a flow-cytometric technique involving fluorescence-labeled monoclonal antibodies, a technique that has been proposed as a new reference method for monocyte counting [6, 13].

E > C.)

Materials and Methods The hematology analyzers were used with the reagents recommended and supplied by their manufacturers. Samples were analyzed by standard operating procedures. The Coulter STKS with VCS technology (Coulter Electronics, Hialeah, FL) classifies leukocytes by measuring low-frequency impedance, highfrequency conductivity, and laser light scatter. It was used with the software revision 1G 1. The STKS typically differentiates 8192 events in all but severely leukopenic samples. The Cobas Argos 5 Duff (Hofftnann-La Roche, Montpellier, France) measures impedance and optical transmission of leukocytes after staining of eosinophils. Basophils are detected in a specific channel after lysis of all other cells. In contrast to the STKS, the Argos analyzes a fixed volume of diluted whole blood, up to a maximum of 16 624 evaluated events. The number of leukocytes differentiated is, therefore, influenced by the white blood cell (WBC) count of the sample. In addition, the number can vary to some extent between analyzer units and depends on the adjustment of the flow cell. Therefore, the Cobas Argos can be programmed to report the number of differentiated leukocytes for each sample, which revealed that the number of leukocytes differentiated by our Cobas Argos 5 Duff unit could be estimated by dividing the WBC count by 2.1, with small variations between different samples. Venous blood was drawn into standard evacuated 3-mL K3EDTA tubes. Samples were maintained at room temperature and tested between 1 and 3 h after collection. The WBC count of the samples was between 4 X i0 and 15 x io cellsfL. Each sample was analyzed 15 times in the manual mode on the STKS, whereas with the Cobas Argos the autosampling mode was used. To determine the precision of the manual differential, 15 slides were prepared for each specimen by using a spinning technique (Microx spinner; Omron, Tokyo, Japan). After fixing and staining with a modified Wright method (Hema-Tek, Ames Automatic Stain; Miles, Slough, UK), a technician counted a 400-cell differential for each slide. In this way, 20 normal and abnormal samples from the daily routine samples were analyzed with each method. To assess the precision of flow cytometry, only 10 samples were prepared 15 times and analyzed a Coulter Epics Profile II flow cytometer on various days. Dual staining with fluorescence-labeled monoclonal antibodies was performed as described previously [13]. In brief, 100 r.tL of EDTA-anticoagulated whole blood was incubated for 15 mm at room temperature with a combination of 20 L of CD45-fluorescein isothiocyanate (FITC; Immunotech, Paris, France; antileukocyte antibody) and 20 L ofCDl4-phycoerythrin (PE; Immunotech; antimonocyte antibody). Subsequently, erythrocytes were lysed by adding 2 mL of an ammonium chloride-based lysing solution (An-Der-Grub Bio Research, Kaumberg, Austria). After two

>

C,

20

0

40

60

80

25 20

i5 >

C.) 10

5 0 3

6

9

12

15

18

Mean Relative Cell Count (%) Fig. 1. Precision profiles of neutrophil, lymphocyte, and monocyte Counts. Each data point represents the results of one sample analyzed 15 times. Curves were fitted to the data points by using the function for the CV,,.The STKS proved significantly more precise than the Argos for neutrophils and monocytes, whereas there was no difference for lymphocytes (dashed line, fitted curve for the STKS; solid line, all other methods: fitted curves of the STKS and Argos are closely adjacent for lymphocytes; to optimize ‘axis scaling, not all data points are shown for the manual differential).

washing steps, samples were analyzed on the Profile II, which was programmed to count 20 000 events. No samples were used from patients with hematologic disease, as they might have yielded atypical results and, in any case, results of automatic differentiation would not have been adequate for these samples. To represent various operating conditions, we distributed the precision experiments over the whole instrument evaluation period. The resulting CVs were plotted against the mean relative cell counts (Fig. 1) and the data for each analyzer were fitted to a

1070

H#{252}bl et al.: Precision

function

the CV of proportions

describing

is the standard

error

(CVi,):

SE 100 X__t

CV(%)= SE

profiles

p

as given in NCCLS

of proportions

document H2OA [3]; it defines single proportion:

the confidence

intervals

for a

where n = number of observed cells, p = relative cell count (%), and q = 100 - p. Curve fitting used the CV as the dependent variable (usually designated as y; CVs as obtained from experiments) and p as the independent variable (usually designated as x; mean relative counts as obtained from experiments). The function for CV was used to describe the relation between the CV and p. It includes one parameter (n), which is estimated in the curvefitting process. The resulting value for n is equivalent to the number of theoretically differentiated leukocytes, which is a theoretical index of the precision of the method. The curve fitting was performed with commercially available curve-fitting computer software, which includes the Marquardt algorithm for nonlinear regression (Fig-P for Windows; Biosoft, Cambridge, UK). Nonlinear regression iteratively determines the values of the parameters that minimize the sum of the squares (SS) of the distances of the data points to the curve:

Because

=

sum [(y

the error values yielded

-

y)2I by the curve-fitting

cannot be used in further formal statistical F-test was chosen to compare data from determining whether the fit for the two significantly better than that for the pooled would indicate a difference between the compare two data sets, they are first fitted

process

calculations [14], the different analyzers, separate data sets is data [14, 15], which two data sets: To separately, yielding

SS1 and SS2 and having the degrees of freedom df1 and df2. The overall value for the SS and the dfare the sums of the individual values from each fit:

SS,.,

=

SS1 + 5S2

df,=df+dfi In the next step, the data are pooled and fitted simultaneously, yielding SS1,,,1 and df001. To determine whether the separate fit is significantly better than the pooled fit, the F ratio is calculated: F

-

(SSr,OOI

-

differential

Results and Discussion Although in the majority of cases the result of the absolute leukocyte differential will be of greater value to the clinician than the relative cell count, we decided to use the latter for the purpose of evaluating the differentiating performance because the absolute count results will be affected by the performance of the WBC channel [16]. In addition, it was important to us to relate the performance of the analyzers to that of the manual differential and the Profile II flow cytometer, quires using the relative count.

SE=

SS

of the leukocyte

SS,p)/(dfp,,,,i df,) SS/df,

-

The F value can be converted to a P value by using a statistical table. The numerator has df001 - dfep degrees of freedom; the denominator has dfsep degrees of freedom. A large F value and a corresponding low P value indicate that the separate fit is better than the pooled fit and thus that there is a significant difference between the two data sets [14, 15].

PRECISION

which

also re-

PROFILES

The precision profiles for neutrophils, lymphocytes, and monocytes are presented in Fig. 1. As expected, they show that precision increased with increasing cell percentages in all methods investigated, but they also show that considerably different CVs may be obtained even for samples with similar mean relative cell counts determined by the same analyzer. For example, as shown in Fig. 1, the STKS analyzed samples with nearly identical mean relative monocyte counts (7.3 1-7.65%), producing CVs varying between 3.88% and 12.42%. This shows that not only the relative cell count but also many other sample characteristics have a great influence on the resulting precision. Hence conventional imprecision studies, which are usually based on determining the within-run precision of only a few samples [2, 4-7], may result in misjudgment of analyzer performance and, thus, may distort analyzer comparison studies. Another problem arises if only normal samples or control materials are used to test precision [2, 5], which may create an unrealistic impression of analyzer performance. Therefore, a large number of within-run imprecision experiments should be performed with normal and abnormal samples covering a wide range of relative cell counts. Realizing this, Warner and Reardon performed within-run precision experiments with a larger number of samples [8]. However, they then simply averaged the obtained CVs, which strikes us as dissatisfactory because much information is lost. We decided to combine the results of the precision experiments for each analyzer by fitting the precision profiles to the function for the CV (Fig. 1), which is derived from the function for the standard deviation of proportions [3], which yields the number of theoretically differentiated leukocytes. However, this number does not represent the actual number of analyzed cells and is not equivalent to an n-value but, rather, should be regarded as a precision index that is influenced by the precision of all components of the differentiation process, from the sample mixing to the distinction and enumeration of leukocyte subclasses. Although the function of the CV describes the precision of the relative cell count on a purely statistical basis and therefore is bound to be an oversimplification, the precision index yielded by curve fitting proved to be a valuable tool for interpreting and comparing precision results. It revealed a significantly higher precision of the STKS for neutrophils and monocytes, the precision index being 4069 and 3423, respectively, whereas that of the Argos was 2391 and 1329 (P

C.) 10

A

A A

5

A

0

3

0

6

9

12

15

18

21

Mean Relative Cell Count (%) Fig. 2. Argos precision

was operated in the manual mode because this allowed us to use a careful manual overhead mixing procedure and because less blood is aspirated this way. The Argos 5 Diff was used in the autosampling mode because it was unable to differentiate sam-

profiles

corrected

for WBC

counts.

To compensate for the impact of the samples’ WBC counts, the original CV5 of the within-run precision studies of the Cobas Argos 5 Diff were corrected for the actual number of differentiated leukocytes. CVs were adapted to 8192 differentiated cells, which is equivalent to the number of cells differentiated by the Coulter STKS and thus facilitates comparison of both techniques of differentiation. After adaptation, the precision of the STKS was no longer superior for neutrophils and monocytes, whereas the Argos now showed a significantly higher precision for lymphocytes, suggesting that the originally superior precision of the STKS had been reached primarily by differentiating a higher number of cells rather than by a more precise algorithm or technique of differentiation (solid line, fitted curve for measured data; dashed line, for corrected data).

In summary, the procedure presented for the assessment of analyzer precision has its shortcomings. It is more laborintensive than other methods, it requires a computer for calculating results, and the fitting of results to the above-described function simplifies the characteristics of the differentiation process. On the other hand, this method describes the precision of automated hematology analyzers clearly and comprehensively and allows statistical comparison between different analyzers. The precision index proved to be a highly useful parameter: It comprises the results of all precision runs in one figure and relates the performance to that of the manual differential, which gives an expressive picture of analyzer performance. The method also enables evaluation of the differentiating technique, which may be of greater interest than analyzer performance. Therefore, we consider this a useful addition to conventional methods of precision assessment.

We thank Gabriele NUMBER

OF

Thum

for excellent

technical

assistance.

REPLICATES

We deliberately limited our within-run precision experiments to 15 replicates to minimize artifacts and to perform all analyses from one standard blood tube. Other investigators have analyzed the same sample up to 30 times [2, 6, 8]. Although no details on sample mixing were given, it is clear that they must have been mixed repeatedly. This, however, does not reflect normal operating conditions, and the theoretical question may be raised of whether precision results obtained by replicate analysis are representative of single-sample analysis. The STKS

References 1. Bentley SA, Johnson A, Bishop CA. A parallel evaluation of four automated hematology analyzers. Am J Clin Pathol 1993;100:

626-32. 2. Swaim WR. Laboratory and clinical evaluation of white blood cell differential counts. Am J Clin Pathol 1991;95:381-8.

3. NCCLS. Approved

Standard H20-A. Reference leukoCyte differential count (proportional) and evaluation of instrumental methods.

Villanova, PA: National Committee for Clinical Laboratory Standards,

1992.

Clinical Chemirtry

4. Brigden ML, Page NE, Graydon C. Evaluation of the Sysmex NE-8000. Am J Clin Pathol 1993;100:618-25. 5. Cung-Yi H, Chiu-Hwa W, Hwei-Mei C, Ming-Ching S. Evaluation of performance for automated differential leucocyte counting on Sysmex NE-8000 by NCCLS recommended protocol H20-T. Clin Lab Haematol 1993;15:287-99. 6. Goosens W, Van Hove L, VerwilghenRL. Monocytecounting: discrepancies in results obtained with different automated instruments. J Clin Pathol 1991;44:224-7. 7. Robertson EP. An evaluation of leucocyte analysis on the Coulter STKS. Clin Lab Haematol 1992;14:53-68. 8. Warner BA, Reardon DM. A field evaluation of the Coulter STKS. Am J Clin Pathol 1991;95:207-17.

counting.

In: Koepke

11.

JA, ed.

Differential

leukocyte

counting. Skokie, IL: College of American Pathologists. 1979:3945.

1073

Ekins RP. The precision profile: its use in assay design, assessment and quality control. In: Hunter WM, Corrie JET, eds. lmmunoassays for clinical chemistry, Livingstone, 1983:76-105.

12.

Sadler

WA, Smith

2nd ed. Edinburgh: Churchill

MH. Use and abuse

of imprecision

profiles:

some pitfalls illustrated by computing and plotting confidence intervals. Clin Chem 1990;36:1346-50. 13. H#{252}bI W, Hauptlorenz S, Tiustos L, Jilch R, Fischer M, Bayer PM. Precision and accuracy of monocyte counting. Am J Clin Pathol 1995;103:167-70. 14. Motulsky regression:

9. NCCLS. Tentative Guideline EP5-T2. Evaluation of precision performance of clinical chemistry devices. Villanova, PA: National Committee for Clinical Laboratory Standards, 1992. 10. R#{252}mke CL. The statistically expected variability in differential leukocyte

42, No. 7, 1996

Hi, Ransnas LA. Fitting curves to data using nonlinear a practical and nonmathematical review. FASEB J

1987;1:365-74.

15. Ratkowsky 0. Comparing parameter estimates from more than one data set. In: Nonlinear regression modelling: a unified and practical

approach.

New York: Dekker,

1983:135-52.

16. R#{252}mke CL. Imprecision of ratio-derived differential counts.

Blood Cells 1985;11:311-4.

leukocyte