Method validation and verification in liquid scintillation counting using the longterm uncertainty method (LTUM) on two decades of proficiency test data F. Verrezen, M. Vasile, H. Loots & M. Bruggeman
Journal of Radioanalytical and Nuclear Chemistry An International Journal Dealing with All Aspects and Applications of Nuclear Chemistry ISSN 0236-5731 J Radioanal Nucl Chem DOI 10.1007/s10967-017-5436-2
1 23
Your article is protected by copyright and all rights are held exclusively by Akadémiai Kiadó, Budapest, Hungary. This e-offprint is for personal use only and shall not be selfarchived in electronic repositories. If you wish to self-archive your article, please use the accepted manuscript version for posting on your own website. You may further deposit the accepted manuscript version in any repository, provided it is only made publicly available 12 months after official publication or later and provided acknowledgement is given to the original source of publication and a link is inserted to the published article on Springer's website. The link must be accompanied by the following text: "The final publication is available at link.springer.com”.
1 23
Author's personal copy J Radioanal Nucl Chem DOI 10.1007/s10967-017-5436-2
Method validation and verification in liquid scintillation counting using the long-term uncertainty method (LTUM) on two decades of proficiency test data F. Verrezen1 • M. Vasile1 • H. Loots1 • M. Bruggeman1
Received: 2 June 2017 Ó Akade´miai Kiado´, Budapest, Hungary 2017
Abstract Results from proficiency tests gathered over the past two decades by the laboratory for low level radioactivity measurements for liquid scintillation counting of 3H (184 results) and 14C (74 results) are used to verify the validated measurement methods used by the laboratory, in particular the estimated uncertainty budget of the method and its reproducibility and stability. A linear regression approach is used for the analysis of the results, described in the literature as the long term uncertainty in measurement method. The present study clearly indicates the advantages of using proficiency test results in identifying possible constant or proportional bias effects as well as the possibility to compare the laboratory performance with the performance of peer laboratories. Keywords Liquid scintillation counting Measurement uncertainty Proficiency test Tritium Radio-carbon Validation Verification
Introduction Analytical laboratories trying to obtain or maintain a quality assurance (QA) system are confronted with the numerous requirements imposed by the relevant QA management system they have chosen to implement. Whether it is an ISO-9001 [1], ISO-17025 [2], ISO-15189
& F. Verrezen
[email protected] 1
Low-level Radioactivity Measurements Laboratory, Belgian Nuclear Research Center, SCK-CEN, Boeretang 200, 2400 Mol, Belgium
[3] or similar quality management system, each comes with its own flavor of requirements regarding method validation and method verification. As a consequence numerous guidelines have been published on how to efficiently set up method validation, such as the Eurachem guide on the fitness for purpose of analytical methods [4]. Most of these guidelines focus on intra-laboratory validation against certified reference materials (CRMs) of the key parameters listed in Table 1. However, the importance of inter-laboratory validation and verification methods are quickly gaining in popularity in recent literature, as for instance proposed in the AOAC International Guideline How to meet ISO-17025 Requirements for Method Verification [5]. Equally important, if not even more, is the establishment of an overall uncertainty budget for the validated method, capable of predicting the uncertainty linked with an individual measured value, without over nor under estimating the given uncertainty. Paragraph 5.4.6.3. of the ISO17025:2005 norm clearly states that ‘‘when estimating the uncertainty of measurement, all uncertainty components which are of importance in the given situation should be taken into account using appropriate methods of analysis’’ [2]. Normative documents, such as the ISO-11929:2010 [6] give detailed requirements on the calculation of detection limits and uncertainty values and several guidelines give guidance on how to set up such an uncertainty budget for both the modelling approach [7] and the empirical approach [8]. Despite all available guides and norms, setting up a reliable uncertainty budget (and with that the correct way of reporting quantitative results of chemical or radiochemical analysis) seems to be the major problem that analytical laboratories need to overcome in method validation. This is clearly illustrated in Fig. 1, representing the
123
Author's personal copy J Radioanal Nucl Chem Table 1 Parameters required as part of a full method validation Main parameter—[related parameter(s)] Selectivity, specificity Limit of detection LOD, limit of quantification LOQ Linearity—(working range) Accuracy Precision—(repeatability, reproducibility) Sensitivity Ruggedness, robustness—(matrix interference)
results of 46 participating laboratories to a proficiency test of 3H in urine samples by liquid scintillation counting (organized by the ‘‘Association pour la Promotion du Controˆle de Qualite´ des Analyses de Biologie Me´dicale en Radiotoxicologie’’, PROCORAD, France in 2016). The range of uncertainties reported varies between 57 Bq/L up to 1200 Bq/L, despite the fact that all laboratories used a similar measurement technique (liquid scintillation counting) and the target value (5770 ± 197 Bq/L) should have been easy to measure and was far from the method’s detection limit.
These results also show a very important advantage of inter-laboratory testing, since no amount of intra-laboratory validation could ever reveal the large discrepancies between calculated and actual uncertainty. Furthermore, once the laboratory has built a history of several years of proficiency test results, additional valuable information can be obtained by analyzing these results with simple mathematical tools such as regression calculation. Depending on the number of data points and the difference in the sample matrix and activity levels, information regarding the validation parameters linearity, ruggedness and precision (especially reproducibility and bias) can be derived. In 2002 Meijer et al. used a linear regression model (y = bx ? a) to evaluate the laboratory values (y) of 82 participating laboratories as a function of target values (x) over a period of 4 years [9]. The same method (called long term uncertainty in measurement or LTUM for short) was applied by Matar et al. to the results of 43 different biological routine analytes [10]. This present study aims to apply the LTUM method to the results of over two decades of participation in proficiency tests in the field of liquid scintillation counting for routine 3H and 14C analysis.
Fig. 1 Results of a 2016 proficiency test on 3H in urine samples organized by PROCORAD
123
Author's personal copy J Radioanal Nucl Chem
Theory The long-term uncertainty in measurement (LTUM) method published by Meijer et al. [9] uses linear regression on pairs of data points, relating the laboratory result (y) to the consensus value (x). The consensus value can either be the target value as given by the organizer of the proficiency test, or (by lack of a target value) the group mean value or other agreed upon value of the measurand. Ideally the laboratory results should always be equal to the target value and the resulting relation should be given by the equality line (y = x). In reality though, lab results will be different from the target values and the regression equation will be of the type (y = bx ? a) as shown in Fig. 2. In this equation, ‘a’ represents the intercept, equivalent to a constant bias (BC), while ‘b’ represents the slope, equivalent to a possible proportional bias (BP). The sum of the constant bias and the proportional bias, termed as the total bias (BT), is what is usually referred to as ‘systematic error’. Finally, the scatter of the individual points around the regression line is an indication of the random error (RE) caused by measurement uncertainty and reproducibility. The contributions of the constant bias (BC), proportional bias (BP) and random error (RE) to the total error of the measurement (TE) can be calculated from the regression
parameters as a simple square root of the sum of squares of the individual components LTCV and BT, multiplied by a coverage factor k: pffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi TE ¼ k LTCV2 þ BT2 ð1Þ with LTCV (the long term coefficient of variation) defined as: . Syx b LTCV ¼ 100 ð2Þ x In the above formula Sy/x is the residual standard deviation (or the variability) of the regression line, defined as: vffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi " #ffi u P 2 X u 1 ½ ðx xÞðy yÞ ðy yÞ2 Syx ¼ t P ð n 2Þ ðx xÞ2 ð3Þ The term BT (total bias) can be calculated as: pffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi pffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi BT ¼ BC2 þ BP2 ¼ a2 þ BP2 with BP (proportional bias) equal to: rffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi ð n 1Þ ðb 1Þ2 S2x BP ¼ n
ð4Þ
ð5Þ
Fig. 2 Linear regression analysis according to the LTUM method
123
Author's personal copy J Radioanal Nucl Chem
Fig. 3 LTUM analysis on the proficiency test results for
14
C by liquid scintillation counting (1990–2017)
Experimental The Laboratory for low level radioactivity measurements (LRM) of the SCK-CEN has been participating on a yearly basis in a variety of proficiency tests since 1990. One of the more important and long running proficiency tests to which the laboratory has participated is the ‘‘Intercomparision in radiotoxicology—3H and 14C in urine’’ test, organized yearly and offering several human urine samples spiked with 3H, 14C, 3H/14C mixtures and tritiated Thymidine (PROCORAD, France). Other proficiency tests the laboratory participated in concerned water samples spiked with 3 H or 14C organized by the International Atomic Energy Agency (IAEA in the ALMERA framework, Austria), Bundesamt fu¨r Strahlenschutz (BfS, Germany), Institut de Radioprotection et de Suˆrete´ Nucle´aire (IRSN, France) and the World Health Organization (WHO, Switzerland).
Results and discussion For the present study all available results for our laboratory (LRM) were used starting from 1990 up to 2017 for 3H and 14C measured by liquid scintillation counting. Values that were clearly aberrant (e.g. z-scores outside the range of |-3 to 3|)
123
and for which a motivated cause for the deviation had been found (e.g. typing error in transferring results) were excluded. This yielded 184 results for 3H in the activity range 6.7 Bq/L up to 1.1 MBq/L and 74 results for 14C in the activity range 2.5 up to 9540 Bq/L. All of these results were statistically analyzed using the LTUM method and linear regression. Figure 3 shows an example of the LTUM analysis for all of the proficiency test results for 14C by liquid scintillation counting in which the laboratory has participated in the period 1990 up to 2017. The graph shows the relation of the reported value against the target values given by the organizer of the proficiency test. The regression line calculated for the laboratory results (solid line) is very close to the equality line (dotted line), indicating that the method used by our laboratory does not suffer from any constant nor any proportional bias. The graph also shows the good linearity of the method in the full range of activity concentrations (0–10 kBq/L). The calculated values for LTCV and BT are 2.4 and 2.4% respectively, indicating an overall average measurement uncertainty TE equal to 3.4% (for k = 1). This is in good agreement with the value of the expanded uncertainty derived from the validation study, which estimated the total uncertainty budget for 3H measurements at 5% (for a coverage factor k = 1, taking into account uncertainty contributions introduced by counting
Author's personal copy J Radioanal Nucl Chem
Fig. 4 LTUM analysis on the proficiency test results for 3H by liquid scintillation counting (1990–2017)
statistics, background correction, detector calibration, sample and cocktail volume). When analyzing the results for 3H however, comparing the lab results against the target values (shown in Fig. 4), a definite proportional bias could be distinguished. The calculated values for LTCV and BT are 8.3 and 25.4% respectively, with an average overall uncertainty TE (k = 1) of 26.7%. However, when analyzing the same results of the lab against the group mean value (average value of all participating laboratories), no obvious bias could be found with values calculated for LTCV and BT equal to 6.5 and 2.4% respectively. Looking at the data represented in Fig. 4, it was obvious that this discrepancy was caused by a cluster of points around the 30–35 kBq activity concentration. Examining the origin of these points in more detail revealed that all of them belonged to a single type of proficiency test (tritiated thymidine in urine samples) organized by the same provider in five consecutive years (1999–2003). For this test, the organizer each time used the same reference material that had been certified by the manufacturer with what seems to be a wrong activity value. This observation is confirmed by the fact that the bias is only present when comparing lab results with target values and not when comparing lab results with the consensus value (in this case the average group mean).
After elimination of the data points related to the tests with the aberrant target values, the analysis of our laboratory results no longer showed any bias (constant nor proportional), and a LTCV value of 4.7% and BT value of 7.3% are obtained. This translates into a calculated overall uncertainty value TE of 8.7%, which again is in good agreement with the value estimated in the validation (10%). All other results of the same test by this provider prior to 1999 and after 2003 (when they purchased a new reference material) are in good agreement with both the target value and the consensus value.
Conclusions The Long Term Uncertainty in Measurement (LTUM) method provides an easy yet powerful way to evaluate proficiency test results, merely by performing a simple regression analysis on the data available. It allows laboratories to compare their results not only with the target values given by the organizer but also with a consensus value (such as the group average) of the participating laboratories, directly comparing the laboratory performance to peer laboratories in the same domain.
123
Author's personal copy J Radioanal Nucl Chem
Whenever the laboratory has collected a set of results by participating in a sufficient number of proficiency tests, the information given by the LTUM method can be directly linked to some of the important parameters that are needed in a full method validation or subsequent verification study. It gives a good estimate about the long-term stability (and thus reproducibility), a good estimate about linearity and range, a clear indication of possible bias (both constant and proportional) and a confirmation on the validity of the calculated uncertainty budget (however it should be noted that it is not a substitute for a full uncertainty budget study and cannot give any information about the individual components in the total uncertainty). It also gives additional information about trueness and precision and, depending on the proficiency tests included in the analysis, possibly even information about ruggedness. Acknowledgements The authors of this paper would like to acknowledge all organizers of proficiency tests. Without their effort, the validation and verification of laboratory methods would be much harder.
References 1. ISO 9001 (2015) Quality management systems—requirements 2. ISO-17025 (2005) General requirements for the competence of testing and calibration laboratories
123
3. ISO-15189 (2012) Medical laboratories—requirements for quality and competence ¨ rnemark U (eds) (2nd ed) (2014) Eurachem 4. Magnusson G, O guide: the fitness for purpose of analytical methods—a laboratory guide to method validation and related topics. ISBN 978-9187461-59-0. www.eurachem.org. Accessed 28 May 2017 5. Weitzel JML, Lee SM, Smoot M, Viafara N and Brodsky M (2007) How to meet ISO-17025 requirements for method verification. AOAC International. www.aoac.org. Accessed 28 May 2017 6. ISO-11929 (2010) Determination of the characteristic limits (decision threshold, detection limit and limits of the confidence interval) for measurements of ionizing radiation—Fundamentals and application 7. GUM (2008) Evaluation of measurement data—guide to the expression of uncertainty in measurement. www.bipm.org. Accessed 28 May 2017 8. Ellison SLR, Williams A (eds) (3th ed) (2012) Eurachem/CITAC Guide CG 4; Quantifying uncertainty in analytical measurement. ISBN 978-0-948926-30-3. www.eurachem.org. Accessed 28 May 2017 9. Meijer P, de Maat MP, Kluft C, Haverkate F, van Houwelingen HC (2002) Long-term analytical performance of hemostasis field methods as assessed by evaluation of the results of an external quality assessment program for antithrombin. Clin Chem 48(7):1011–1015 10. Matar G, Poggi B, Mely R, Bon C, Chardon L, Chikh K, Renard AC, Sota C, Eynard JC, Cartier R, Cohen R (2015) Uncertainty in measurement for 43 biochemistry, immunoassay, and hemostasis routine analytes evaluated by a method using only external quality assessment data. Clin Chem Lab Med 53(11):1725–1736
