this development and is called scientific computing. This field consists of numerical mathematics, computer science and the relating engineering discipline.
On the Need of Certification in Computational Electromagnetics based Engineering Services Sebastian Lange, Martin Schaarschmidt and Frank Sabath Bundeswehr Research Institute for Protective Technologies and NBC Protection (WIS) Humboldtstrasse 100, 29633 Munster, Germany which achieves quality by organising activities and related resources as processes. In order to guarantee a high quality which is planned, controlled and permanently improved laboratories are accredited by national authorities. Quality characteristics are defined by international norms and recognised to specific tasks by a third party (accreditation service). EMC testing laboratories are accredited against the ISO 17025 norm. Such a laboratory can demonstrate to its customer that it has been successful at fulfilling the requirements of international accreditation standards. A confirmation for an independent evaluation is the confirmation of specific requirements like measurements against the IEC EN 61000 family. Such a recognition by a third
Abstract—During the last five decades computing power has become more widely available. Consequently the solving of practical engineering and scientific problems has shifted towards virtual computer based assessments. A new field based on mathematical models and modern computers arose from this development and is called scientific computing. This field consists of numerical mathematics, computer science and the relating engineering discipline. This field differs from laboratory experiments and theoretical work which are the common forms of engineering work. Nowadays computational electromagnetics is a self-contained engineering service. This paper approaches the problem of providing proof of reliability and creating confidence in this special industrial sector.
I.
I NTRODUCTION
Both theory and laboratory experiment are traditional forms of engineering. During the last five decades computational power has become widely available and scientific computing became an additional part of engineering and science. Nowadays this field is more and more important during product development and production. Digital mockups and scientific computing may be used in order to •
reduce product development costs by minimising the number of physical prototypes and real testing time
•
increase product quality by a higher variance of design alternatives
•
reduce the development time by identifying possible issues in early project phases
Fig. 1.
Credibility in Science and Engineering
party reduces risks like product failure, health risks, reputation loss and also allows to meet legal requirements. An accredited or equivalent operated laboratory is an integral part of the QMS of its organisation. This makes the laboratory processes available for the management and yields transparency during development and production. Virtual testing is performed by numerical analysts according to its laboratory equivalent, but due to a lack of standardised processes scientific computing does not inspire a comparable confidence to that in laboratory measurements. The goal of virtual tests without the need of validation measurements in a professional and industrial environment makes process based scientific computing embedded in a QMS with a third party certification unavoidable.
Against the background of the high importance of numerical results in product development and design a quantification and recognition of quality in this area is needed. Theoretical work may be proofed by mathematical arguments, while a accreditation is possible for testing laboratories. Computer based virtual testing based upon scientific computing and digital mockups is widely performed by in-house and external service providers additionally to experimental testing. Most companies operate a quality management system (QMS) which consist of four main components: •
quality planning,
•
quality control,
•
quality assurance,
A. Liability of Products and Services
•
quality improvement.
Product liability is the responsibility of a manufacturer or vendor of goods to compensate for injury and damage caused by defective merchandise that it has provided for sale. Beyond that, strict liability defines an absolute legal responsibility for
A QMS is operated by most organisations in almost all industrial sectors. A QMS is understood as a process approach
978-1-4799-6616-5/15/$31.00 ©2015 IEEE
277
an injury or a damage that can be imposed on the wrongdoer without proof of carelessness or fault. Both product and strict liability are effective methods of consumer protection. Figure 2 presents the relationship of privity in liability cases. In
real-world problem in the best way possible. For that reason scientific computing became dependent on
Fig. 3. Non-conformities and risk during product engineering, based on[25]
Fig. 2.
proper tools and well-educated analysts. This dependency is the reason why a pure validation of software a pure certification of personnel is not sufficient. Both together yields a certification of the whole service. Such services are utilised in wide areas of engineering industries. Numerical computations are performed during the whole product engineering process. Numerical results are used during the concept, development and design phase of a product. These results have an effect on decisions belonging to the process for realisation of a product. Figure 3 exemplifies non-conformity costs and the risk during all phases. This means, that an early failure by computation may become really expensive. On the other hand a mitigation of risks may be possible with a quality assurance in all phases especially the earliest. In this manner a QMS for scientific computing and related engineering services targets cost and risk reduction.
Relationship of Privity
the manner common for most sectors of industry QMS are implemented to prevent or decrease liability costs by increasing product quality. Assuming that methods of scientific computing are used by numerical analysts in research and development divisions a link between quality and computing exists. Liability and reversal of evidence are big challenges in engineering, production and service. The manufacturer or vendor has the possibility to free himself from liability if he proves that the state of scientific and technical knowledge was not such to enable a defect to be discovered. For that reason a well-established QMS which covers all processes is needed. Against the background of liability the highest goal is given by traceability in order to avoid costs and risks. Traceability in laboratory experiment is fulfilled by calibration to a standard. In this way an unbroken chain of comparisons relating to an instrumental measurement to a standard is given. This implements a chain of custody in the laboratory framework. Scientific computing as a process is neglected often. A negative example is a well educated analyst with suitable equipment, who works with an “problem in - solution out” workflow. Such an input-computing-output pattern is a widely used approach in scientific computing. It can be assumed, that these results are correct and state-of-the art, but it can’t be proven. In order to provide transparency and traceability a process based workflow with a documented and recorded path between problem description and result is needed. Furthermore a third party certification may assure a high efficiency of a process based scientific computing approach.
C. Conformity Assessment - The ISO 17000 Family Conformity is the compliance of products or services with specified requirements. A well known example is the European CE-mark or national German GS-mark which indicate compliance wit EU requirements or the German Equipment and Product Safety Act. Furthermore ’conformity’ describes services, processes, products etc. which fulfil several requirements stipulated by contracts, norms or by law. Legal requirements apply in governmental areas like consumer or environmental protection, healthcare etc. The private sector seeks to prove that their goods or services meet specific standards of quality or demonstrate expertise in a specialized area. Not only specific requirements which are stipulated by law, in contracts or fulfilled voluntarily are understood as conformity assessment, but the conformity assessment itself. Figure 4 gives an overview about the conformity assessment toolbox. It involves tasks like certification, inspection, testing, and of course calibration. These assessments may be performed by the supplier (ISO 17050) or service provider itself or by an independent third-party test and calibration laboratory (ISO 17025), inspection body (ISO 17020) or certification body (ISO 17021). The ISO 17000 norm-family starts with the Terms and the ISO 17011 defines the requirements for accreditation bodies. The ISO 17065 defines the requirements for bodies certifying
B. Scientific Computing as a Service Scientific computing is based on the engineering discipline, numerical mathematics and computer science. Both computers and software are used as equipment by numerical analysts. Usually numerical methods are not developed by the analyst itself. Instead a lot of specialised software tools exist. For that reason it may be assumed, that there are proper tools with an correct behaviour for numerical assessments. The analyst has to ensure that a adequate tools are used in their valid regime and that the model assumptions are correct and represent the
278
“A laboratory’s fulfilment of the requirements of ISO/IEC 17025:2005 means the laboratory meets both the technical competence requirements and management system requirements that are necessary for it to consistently deliver technically valid test results and calibrations. The management system requirements in ISO/IEC 17025:2005 (Section 4) are written in language relevant to laboratory operations and meet the principles of ISO 9001:2008 Quality Management Systems - Requirements and are aligned with its pertinent requirements.”
Fig. 4.
Against the background of the nature of scientific computing a certification as a service with respect to ISO 17065 and an accreditation with respect to ISO 17021 is proposed here. A running and registered QMS is a framework and a fundamental requirement.
The ISO 17000 family, adapted from [26]
products, processes and services. The evaluation activities during certification shall meet applicable requirements of the relevant standards. The minimum requirements are given by •
ISO 17025 for testing,
•
ISO 17020 for inspections or
•
ISO 17021 for managements system auditing.
II.
Engineering services provide and utilise competences, knowledge and engineering experience in order to produce issue-related solution for technical difficulties. This services can be provided for internal or external customers. The VDI guideline 4510 [10] defines service as follows: Service [. . . ] is an activity, which is instructed and paid by a customer/user, who in return acquires an immaterial service as a result of the activity - e.g. calculation of a bolted joint [. . . ].
In this manner service providers may be certified (ISO 17065) by the conformity to a QMS based upon an external audit (ISO 17021) with respect to a QMS standard like the NAFEMS Quality Standard Supplement 001 which is conform to the internationally recognised quality standard ISO 9001. Figure 5 describes the hierarchy of conformity assessment. A QMS is needed to establish a basis. This may be done by a registration. Thus registration gives a written assurance about conformity of the QMS to a normative standard for example ISO 9001. The accreditation is a formal recognition about the competence to carry out specific tasks. In the case of EMC testing laboratories the accreditation requirements are defined by ISO 17025 and for certification bodies by ISO 17021. The last step is the certification of products, services and processes with respect to ISO 17065. Here certification means a written assurance that a product, process, service, or person’s qualification meets specified requirements. A relationship between
This definition satisfies scientific computing exactly. Figure 6 gives an overview of competencies. These include as well engineering, economical, and social contents. Engineering
Fig. 6.
Fig. 5.
R EQUIREMENTS TO E NGINEERING S ERVICE P ROVIDERS
From base competence to expertise [10]
knowledge, of course, represents the base competence. But with respect to the nature of provision of services this profession includes also methodical, professional and social skills. A further extension of communication, national and international management yields a status of an expert. This paper aims at the basis competencies as requirements for certification of services in scientific computing. The VDI 4510 standard is used as a classification of service providers in competencies, fields of knowledge and service classes. From this requirements for a certification with respect to ISO 17065 are defined.
Hierarchy of Conformity Assessment
ISO 9001 QMS and and accredited testing laboratory is given in the Joint IAF-ILAC-ISO Communiqu´e on the Management Systems Requirements of ISO/IEC 17025:2005, General requirements for the competence of testing and calibration laboratories:
279
III.
T ESTING AND A SSESSMENT
environment. Both content and form of reports are determined. Here a documentation of test conditions and results with respect to an estimated uncertainty is demanded. The archiving of records is demanded also. In order to ensure that records cannot be altered or lost an audit-proof archive is required also. This ensures integrity and security of all records. Furthermore accredited laboratories are distinguished by preventive and corrective actions, complaint handling, supplier and subcontractor management, non-discriminatory conditions, and internal audits. These laboratory requirements are embedded in all quality related workflow steps of an accredited laboratory. The unique steps are documented in written procedures. Some processes like control of nonconforming testing, calibration and maintenance of equipment, validation of procedures and analytical methods, or a creation of controlled environmental conditions are demanded. A sufficient qualification of personnel or material is additionally required in accredited testing laboratories. These arrangements allow a high transparency and traceability in all processes of an experimental laboratory and yield repeatability of assessments. A lot of work was done in accreditation of EMC testing laboratories worldwide during the last two decades. Quality assurance and quality management are widely accepted in laboratory experiment. Even university labs spend a high amount of money and time for implementing and accreditation, cf. [11]-[20] .
The customer is the arbiter of quality, where quality means a degree to which a set of inherent characteristics fulfils needs or expectations. For that reason the creation of confidence in products or services is one of the most important goals on global market. In order to achieve such objectives or satisfy legal regulations assessments and testing are performed in all phases of product engineering. Nowadays testing is performed or supported by computational methods as “virtual testing”. A process of quality assurance and accreditation for laboratory testing started in the 1980s and still goes on. Today accredited EMC testing laboratories are usual and inevitable. Otherwise virtual testing, in the best case, is performed with respect to the rules of Good Scientific Practice (GSP). In contrast a Good Laboratory Practice (GLP) exits and meets the requirements of ISO 17025 on a conceptual level abstractly. GSP is a resolution which stipulates that researchers must comply rules of science and research in the entire range of all scientific disciplines and a canon of scientific ethics. Is GSP sufficient as quality assurance in engineering? It is not sufficient for EMS laboratory experiments and assessments. Thus it appears that it is not sufficient for scientific computing under the assumption that the result significance is the same like the laboratory one. A. Accredited EMC Laboratories - ISO 17025 ISO 17025 consists of management and technical requirements. The technical requirements concern the competence of the staff, equipment, testing methodology, reporting of tests, and calibration results, where the management requirements are related to the laboratories QMS. The main difference between a good laboratory practice and a formal accreditation with respect to ISO 17025 is the higher amount of documentation. Of course a good laboratory is well equipped, validated and uses qualified personnel. But sometimes the results are not sufficient documented. Such an environment with a formal documentation and recording is required by ISO 17025 and yields to maximal traceability.
B. Virtual Testing Scientific computing aims confident testing and assessment of engineering issues. Figure 8 shows the process of computeraided engineering (CAE). The device under test is modelled as a digital mockup using computer aided design (CAD). Dis-
Fig. 8. Fig. 7.
CEM workflow
cretisation is the next phase which yields a mesh or grid for the numerical solver. Both CAD modelling and discretisation are quality relevant steps in the whole CAE-process. A CAD bases digital mockup is the representation of a real-wolrd application and the following discretisation represents a translation to the mathematical problem. The CAD abstraction and discretisation are strongly dependent from numerical method and problem class. Different numerical methods need different models and discretisations. In the case of a standard Finite Difference Time Domain (FDTD) structured grid (hexahedron) is used for computation, but methods like Finite Element Method (FEM) needs in a classical volume modelling a unstructured mesh (tetrahedron). Such local methods discretise the volume and solve the differential equations exactly. On the other hand global methods like the Method of Moments (MoM) or
Sample/Result Workflown in Accredited Laboratories
The path of a sample and of the related testing results through the entire laboratory system shows the impact and influence of accreditation, see figure 7. Such a workflow is a derives from ISO 17025 requirements. An accredited laboratory processes samples in a stipulated modality. Standardisation begins with the sampling, where a sampling plan and a sampling documentation exists. This is often done using formalised inventor’s notebooks. The samples are individually identified and the protection of sample integrity is the main goal during sample handling and preparation. The assessment is dominated by surveillance of quality of the testing results. Not only sampling and testing is standardised but also reporting and record maintenance is formalised in an accredited
280
Multilevel-Fast Multipole Method (MLFMM) base on unstructured surface discretisations in their classical implementation. Such methods solve integral equations. All this and a lot more things have to be considered during the pre-processing phase. All model assumptions are implemented in the digital mockup, in a second step in the discretisation and in the boundary conditions. Without a standardised environment the pre-processing, which means the entire process between the real-world engineering issue and defined numerical problem, is not transparent. This means that CAE and scientific computing is neither repeatable nor traceable. It can be assumed that a verified and validated software implementation produces correct results under the assumption of a straight configuration and installation. Consideration of wrong solver choice, ignored warning or an imprecise solver may affect the quality of results. This quality risk creates the need of preventive and corrective actions in the CAE-process. The post-processing is the last phase in the CAE-process. Here some quality risks arise from wrong scaling or result choice and of course by wrong smoothing and presentation of results. Such erroneous results may incorporate into reports and are in the worst case never detected. With the intention to create confidence in numerical results traceability, transparency and repeatability must be realised on the same level like in an accredited EMC testing laboratory. For that reason a controlled environment for scientific computing is needed. Such an environment demands also an adapted QMS, technical requirements and of course sufficient equipped and qualified analysts. In order to achieve the same confidence in numerical results like in measurements the accurateness and carefulness of the virtual laboratory system shall be appropriate to an experimental EMC testing laboratory. This begins with an QMS and is perpetuated in a certified service.
Methods T.E.A.M-Problems [23] or the EM Programmer’s Notebook [22] at IEEE. Most examples were used as pure code benchmarks. Of course benchmarking is a really important issue, but under the assumption of well verified and validated codes there is still the human factor and the environment. The idea of Round Robin Tests is a possibility to introduce proficiency in the developed QMS of a virtual laboratory. Furthermore a Round Robin Testing allows to operate more sophisticated examples. Especially coupled approaches like MLFMM - FEM - Cable Solver may benefit from Round Robin Testing and improve proficiency and expert knowledge of the analyst. In order to give an example on possibilities of computational electromagnetics a minesweeper vessel is shown in figure 9. The vessel is exposed to a high-altitude electromagnetic pulse (E1-short time) and figure 10 shows the electric field at the afterdeck. The simulation was performed using a subgrid
Fig. 9.
Minesweeper Vessel
C. Round Robin Tests A Round Robin Test is a kind of proficiency tests in experimental methodology. This interlaboratory test is a well known and accepted method of quality assurance for accredited test and calibrations laboratories. A Round Robin Test is one of the requirements of ISO 17025. These tests involve multiple independent scientific groups performing tests with a variety of methods and equipment. For example the analysis of a sample is performed by different laboratories using different methods. The requirements for proficiency testing are stipulated in ISO 17043. The purpose of Round Robin Tests in computational electromagnetics is the determination of usability of numerical results in a formal permit. The U.S. Food and Drug Adminstration promotes the Critical Path Initiative Program to advance the application of CFD technology in the development and evaluation of medical devices. Both automotive and aerospace & defence industries are along with the medical industry the most sophisticated branches with respect to a permit of products and readiness for marketing. Possible consequences in liability scenarios is also comparable. For that reason the idea of Round Robin Testing in computations electromagnetics make sense. A lot of work on validation and verification of numerical codes was performed since the 1980s. One of these results were the IEEE standards [5], [6] which give recommendations and yielded the FSV-validation. The International Compumag Society has implemented the Testing Electromagnetic Analysis
Fig. 10.
Electrical field at afterdeck
FDTD algorithm. The issue is not the modelling of a digital mockup of the hulk, but the real issue of an analyst is given by the needed assumptions during modelling which consist of a decision between rejecting or accepting of geometrical features. Such a complex problem, especially when cabling and harnesses are considered, cannot be validated by a quick measurement. Especially the constructional level and the model accuracy are the biggest pre-processing issues. Computing of such models is possible due to computing power and sufficient methods. Such virtual assessments will dominate the next decades.
281
IV.
CEM S ERVICE P ROVIDER C ERTIFICATION
[4]
A certification on the basis of NAFEMS QSS 001 [7], [8] was proposed in [24] as a third party certification on the basis of a supplement to ISO 9001. Such a certification needs a implementation of a QMS. The benefit is not the certificate, but the implementation of processes which are monitored. Due to the certification the entire CAE-process has to be embedded in a quality controlled environment which is equivalent to the workflow shown in figure 7. Testing and performing the process shown in figure 8 is the same - computing is here equivalent to measuring. QMS requirements result in a standardised sampling and sample handling. Also a standardised reporting and maintenance of records is a quality related attribute of a professional engineering service provider which aims proficiency and repeatability in engineering progress. A service provider certification is equivalent to a product certification based upon ISO 17065. A product certification is confirmed by an accredited testing laboratory with respect to ISO 17025 and a service certification is confirmed by ISO 17021 by a accredited certification body. This happens on the basis of an external QMS audit. In both cases a running QMS is needed. With respect to scientific computing a NAFEMS QSS 001 based QMS is required in [24].
[5] [6] [7] [8] [9]
[10] [11]
[12]
[13]
[14]
V.
C ONCLUSION
The confidence between experimental and computed results was compared in this paper. The need of formal quality assurance and certification of scientific computing services was derived from product liability and the needs of an engineering service provider on the global market. The idea of confidence in results was derived from accreditation of EMC testing laboratories. Especially the idea of a Round Robin Test in CEM as a proficiency test is proposed here. Furthermore the need of transparent, traceable and repeatable virtual tests was discussed and a possible solution was given by a service certification. The general idea of certification compared to the situation in EMC testing laboratories. Under the tacit assumption of an equivalence between accredited experimental results and computed results the same or equivalent quality standards must exist. A responsible usage of numerical results in projects which may cause human injuries or public distresses demand a qualified confidence in the computed results. Industry sectors like aerospace & defence, medicine or automotive bear high risks. For that reason authorisation and final approval of new products has the highest requirements. In this manner product engineering based upon laboratory experiments is protected by accreditation, on the other side numerical results are not. Scientific computing will enhance in the next two decades. Of course education, method development and the validation of implementations is important but the new issue in scientific computing is the creation of controlled environment where numerical results can be produced with the same confidence like experimental work.
[15]
[16] [17]
[18]
[19] [20]
[21]
[22]
[23] [24]
[25] [26]
R EFERENCES [1] [2] [3]
ISO/IEC 9001:2008 Quality management systems - Requirements ISO/IEC 17021 Conformity assessment Requirements for bodies providing audit and certification of management systems EN ISO/IEC 17025:2005 General requirements for the competence of testing and calibration laboratories
282
EN ISO/IEC 17065:2012 Conformity assessment - Requirements for bodies certifying products, processes and services IEEE 1597.1-2008 IEEE Standard for Validation of Computational Electromagnetics Computer Modeling and Simulations IEEE 1597.2-2010 IEEE Recommended Practice for Validation of Computational Electromagnetics Computer Modeling and Simulations NAFEMS QSS 001:2007 Engineering Simulation - Quality Management Systems - Requirements J. Smith Quality Assurance Procedures for Engineering Analysis, NAFEMS Ltd. 1999 Duffy, A.; et al. The feature selective validation (FSV) method, Electromagnetic Compatibility, 2005. EMC 2005. 2005 International Symposium on , vol.1, no., pp.272,277 Vol. 1, 8-12 Aug. 2005 Engineering services and requirements to engineering service providers VDI 4510, May 2006 Hansen, D.Lessons learned from running and auditing EU EMC Labs. In: Electromagnetic Compatibility, 2001. EMC. 2001 IEEE International Symposium on. IEEE, 2001. S. 74-79. Kumar, D. Strategic approach to EMC Test Laboratory Accreditation in compliance with ISO/IEC 17025 standard. In: ElectroMagnetic Interference and Compatibility (INCEMIC), 2006 Proceedings of the 9th International Conference on. IEEE, 2006. S. 233-236. Nezri, Z.; Jacobson, S. Accreditation of EMC testing laboratoriesworldwide trends and Israeli status. In: Electromagnetic Compatibility, 2003. EMC’03. 2003 IEEE International Symposium on. IEEE, 2003. S. 678-679. Friesen, D. M.; Dobson, S. Implementing an ISO/IEC 17025 quality system in a commercial EMC test laboratory environment. In: Electromagnetic Compatibility, 2003 IEEE International Symposium on. IEEE, 2003. S. 177-180. Zapata-Garcia, D.; Llaurad´o , M.; Rauret, G. Experience of implementing ISO 17025 for the accreditation of a university testing laboratory. Accreditation and quality assurance, 2007, 12. Jg., Nr. 6, S. 317-322. Rodima, A., et al. ISO 17025 quality system in a university environment. Accreditation and quality assurance, 2005, 10. Jg., Nr. 7, S. 369-372 Hullihen, K.; Fithsimmons, V.; Fisch, M. R. Establishing an ISO 17025 compliant laboratory at a University. Engineering Technology Opens the Door to a World of Opportunity, International Journal of Modern Engineering 2009, S. 55-63. PRITZKOW, J. Practical experience of the laboratories in implementing the ISO/IEC 17025. Accreditation and Quality Assurance: Journal for Quality, Comparability and Reliability in Chemical Measurement, 2003, 8. Jg., Nr. 1, S. 25-26. Islin, H.; Andersen, T. The process of management review. Accreditation and Quality Assurance, 2008, 13. Jg., Nr. 3, S. 157-160. Halevy, A. The benefits calibration and testing laboratories may gain from ISO/IEC 17025 accreditation. Accreditation and quality assurance, 2003, 8. Jg., Nr. 6, S. 286-290. Arvas, E.; Sevgi, L. A Tutorial on the Method of Moments [Testing Ourselves]. Antennas and Propagation Magazine, IEEE, 2012, 54. Jg., Nr. 3, S. 260-275. Woo, A. C., et al. EM programmer’s notebook-benchmark radar targets for the validation of computational electromagnetics programs. Antennas and Propagation Magazine, IEEE, 1993, 35. Jg., Nr. 1, S. 84-89. International Compumag Society, Testing Electromagnetic Analysis Methods (T.E.A.M.), http://www.compumag.org/jsite/team.html Lange, S.; Schaarschmidt, M.; Sabath, F. An approach on quality assurance in computational electromagnetics. In: Electromagnetic Compatibility, 2014 International Symposium on. IEEE, 2014. S. 240-245. Clark, K. B.; Wheelwright S. C. Managing New Product and Process Development USA 1992/2010 ISO/IEC 17000:2004 Conformity assessment – Vocabulary and general principles
