Template of Manuscripts for IREE

International Review of Electrical Engineering (I.R.E.E.), Vol. xx, n. x

Robust estimates for validation performance indexes of electric network models Jacopo Bongiorno1, Andrea Mariscotti2 Abstract – The validation of an electric traction network simulator is addressed by selecting the testing conditions and considering three performance indexes, which evaluate both simply amplitude and other curve features. A common interpretation scale is proposed to verify the agreement between them. Moreover, since the most complex cannot be fully used without a graphical representation of its output, this is simplified, in order to conclude the validation with an estimate of accuracy that is as close as possible to the classical declaration that accompanies measuring instrument. Copyright © 2014 Praise Worthy Prize S.r.l. - All rights reserved.

Keywords: Electrical simulation, Electric networks, Guideway transportation systems, Validation

List of symbols ADMi i-th component of Amplitude Difference Measure index ADM mean value of Amplitude Difference Measure index DFT Discrete Fourier Transform FDMi i-th component of Feature Difference Measure index FDM mean value of Feature Difference Measure index FSV Feature Selective Validation method GDMi i-th component of Global Difference Measure index GDM Mean value of Global Difference Measure index IDFT Inverse Discrete Fourier Transform mj j-th component of measured data MTL Multiconductor Transmission Line N number of components of a curve or vector ODM contribution to ADM of the Offset Difference between curves oj j-th component of simulated data RMP Modified Pendry index U Theil index V&V validation and verification Zp pantograph impedance

I. Introduction Numeric models are being used more and more often for the assessment of system performance, reliability, safety in normal and exceptional conditions or configurations for a wide range of systems. Correspondingly, simulation models shall undergo a verification and validation process, evaluating performance for the field of application [1][2].

Manuscript received April 2014, revised April 2014

Depending on it, model suitability may be expressed in terms of accuracy, robustness, reliability [3][4]. The V&V process for a simulation tool is grounded on the initial definition of its intended use: modeled network elements and necessary information, solution methods for frequency and/or time domain simulation, inclusion of non-linearity, identification of output quantities and how they are reported, possibility of executing parametric and sensitivity analysis. The most relevant part of the validation process is the characterization of model accuracy using reference data based on dynamic techniques, i.e. executing test runs on selected cases. The use of simulation tools aims at replacing experimental methods and measurement campaigns with significant savings in terms of time and cost. Focusing on electric traction networks and electrical interoperability phenomena, the use of simulation was already accepted and suggested [4]-[7] for the following phenomena:  useful voltage, i.e. the average pantograph voltage available when absorbing traction power, calculated per train or per network area;  power factor and displacement factor for ac systems;  harmonics and inter-harmonics, caused by the interaction of distorting loads and generators, namely trains and substations;  dynamic interaction between supply network and trains, with possible electrical instability, resonances, supply distortion and reactive power flow. Simulation allows testing configurations that cannot be reproduced in practice, but represent exceptional worstcase events and parameter combinations. Especially when results are used for safety-related decisions, simulation models shall be validated for their adequacy, accuracy, range of validity, as any other instrument [3]. Model accuracy is evaluated by comparison with reference data, using dynamic techniques, executing test

Copyright © 2014 Praise Worthy Prize S.r.l. - All rights reserved

J. Bongiorno, A. Mariscotti

runs on reference cases. Simulated and measured data shall be compared, covering various configurations and parameter combinations, including preliminary evaluation of the metrological quality of experimental data. Data may for clarity be assumed as electrical quantities (e.g. voltage, current, impedance) represented as frequencydomain spectra [7], normally characterized as amplitude and phase response. The complexity of network models on one side and the probabilistic nature of the experimental results on the other require that consistency evaluation is carried out, minimizing indeterminacy, e.g. identifying system configurations with known and stable quantities for network, train operating conditions, driving style, etc. Shape and characteristics of curves under comparison orient the choice towards specific performance indexes [8][9], preferable for several reasons: robustness to noise, adequate response for peaks and slopes, ability to cope with relatively large uncertainty of experimental data. More than one performance index is useful to crosscheck indexes themselves and avoid biasing and distortion of judgment and validation results. Comparing performance indexes is of course not straightforward, because of different ranges and different sensitivities to curve characteristics: they were tested extensively on sample curves in [8]. This work addresses the comparison of selected performance indexes clarifying some points:  complex and powerful analysis tools and indexes often give articulated, and thus possibly ambiguous, answers; on the other hand, too a simple evaluation of model accuracy is often simplistic and misses to fully cover model behavior, leaving dark areas in its characterization; three performance indexes are considered with increasing complexity, Theil, Modified Pendry and FSV;  curve features are considered for a problem where abundance of experimental results and knowledge of underlying physical mechanisms are available; thus, not only the validation of the traction supply network simulator traXsim [10] is concluded, but is in turn used for a better understanding of index response;  the best representation of index results is also considered, especially for FSV, using statistical quantities and histograms to define a validation statement that is simple (robust mean) and conveys statistical significance by using confidence interval. The test case is based on the measurements made at the Velim test centre in Czech Republic in 2014.

II. Validation method A straightforward comparison valid for all kinds of curves is based on classical distance measures, e.g. rms and absolute error. For dynamic system responses, other


features may be considered, such as resonance and antiresonance frequencies, factor of merit, overall shape in terms of positive and negative slopes. When observing frequency-response curves some features normally capture observer’s eye and may be used to quantify the degree of similarity, with strong and weak points:  the eye concentrates on peak positions and slopes, ignoring exact values; visual evaluation selects the most relevant behavior and trend, rejecting many details with adverse influence;  the amount and structure of data may be too large and complex to be visually compared, and feature selection and extraction shall be implemented. The overall judgment is based on many results and cases, which shall be correctly weighted and combined to represent in the best way model adequacy and accuracy: graphical and numerical means are used, such as histograms, mean values, dispersions or spreads. Model adequacy and accuracy are evaluated by means of performance indexes that measure the degree of similarity between simulation and experimental data, i.e. the distance between two vectors, o (simulation output) and m (measured data). Index selection is based on previous works and experience [8]: two indexes were identified, Theil U [11] and Modified Pendry RMP [8][12], with a define linear scale between 0 and 1, that behaved consistently with benchmark curves. U 

N 1

 i0

(oi  m i )

2

  

N 1

R MP 



i0

N 1

 i0

2

oi 

  L oi

N 1

L oi  L mi

N 1

 i0

 L mi



2  mi  

(1)

(2)

i0

Where L  x  / x in (2) represents the normalized first derivative of either the model output o or experimental data m. Aiming at evaluating more completely curve features that usually draw expert’s attention, the Feature Selective Validation (FSV) is also considered [13][14]. This method has undergone an intense phase of analysis and validation itself and is now established by the IEEE Std. 1597.2 [13], giving thus the dignity and authoritativeness of a normative reference. The method split the data vector in three parts, “dc”, “low” and “high” intervals. The first two are identified as “trend” data, while the latter corresponds to “feature” data; separation is performed by computing the Fourier transform of original data, but the focus is on slow and fast changes of shape, not to be interpreted as frequency (by the way the original data may already be frequency-domain vectors). Trend and feature are then evaluated by two indexes, the ADM (Amplitude Difference Measure) and FDM (Feature Difference Measure); both are combined in the GDM (Global Difference Measure). Data processing and calculations are done with a proprietary code that was International Review of Electrical Engineering, Vol. xx, n. x


validated using the IEEE Std. 1597.2 data and GVT results (GCEM Validation Tool) developed by UPC [15]. FSV formulation is reported for completeness, with N the number of points of the curves. The process begins applying DFT to the data and separating DC, LO and HI parts of the spectrum; with a successive IDFT of these partial vectors dc, lo, hi vectors are obtained in the data original domain used then for index calculation. The announced ADM, FDM and GDM indexes may be calculated as follows:

ADM

i



o lo , i  m lo , i 1 N

N 1

 j0

ODM



i

o lo , j  m lo , j

1

N 1

 i0

ADM 



2 i



2

3 i



6

 ADM

i

N 1

 j0

7 .2

N 1

1 i

j0

(7)

1

2 i

 FDM

N

(9)



Quality descriptor Excellent (E) Very Good (VG) Good (G) Fair (F) Poor (P) Very Poor (VP)

FSV

Theil and Pendry

Lower bound

Upper bound

Lower bound

Upper bound

0.0 0.1 0.2 0.4 0.8 1.6

0.1 0.2 0.4 0.8 1.6 

0.0 0.1 0.3 0.5 0.7 0.9

0.1 0.3 0.5 0.7 0.9 1.0

Lacking quantitative equivalence between Theil and Modified Pendry indexes on one side and FSV on the other, the correspondence between the two is based on the interpretation scales and the related categories, from Excellent to Very Poor.

III. Case study and experimental data FDM

(10)

i

i0

2

1

3 i

N 1

( ADM i )  ( FDM i )

GDM 

(8)

 , j  m hi  , j o hi

 FDM

N



(6)

 , j  m hi  ,j o hi

j0



 2 FDM

i

TABLE 1. INTERPRETATION SCALE FOR FSV, THEIL AND PENDRY INDEXES

o lo , j  m lo , j

N 1



FDM 

GDM

(5)

i

i0

 , i  m hi  , i o hi N

FDM

(4)

o dc ,i  m dc ,i

 , i  m hi  ,i o hi N

FDM

(3)

i

o lo ,i  m lo ,i N

FDM

ODM

N 1

1 N

1 i

exp

i

o dc ,i  m dc ,i N

FDM

 ODM

dividing by the differential of the independent variable, that would make them difference quotients; the word “derivative” will be used to indicate them indifferently. Before proceeding, the interpretation of index values shall be clarified. FSV, as established by IEEE Std. 1597.2 [13], is classified using a nearly geometric progression of the interval widths shown in Table 1. For Theil and Mod. Pendry the fact that they are almost linear between 0 and 1 suggests an even subdivision of intervals. For similarity with FSV six subintervals are selected, as shown in [16]. The determination of lower and upper bounds of each interval is nearly uniform, with smaller first and last ones [14]. A compression of the first Excellent interval was observed for Theil and Pendry indexes using absolute values in their expressions; the phenomenon was explained by observing the behavior of data point close to zero and the folding of the probability distribution for small negative values. The consequence is that the first interval is underutilized and that the judgment on similarity is thus slightly unfavorably biased.

2

(11)

N 1

 GDM i

(12)

i0

The scale factors of 2, 6 and 7.2 in (6), (7) and (8) were chosen empirically based on the considered data at the time the method was developed in IEEE Std. 1597.2. Derivatives are calculated as a difference, using an interval that is 2 for “lo” and 3 for “hi”, but without Copyright © 2014 Praise Worthy Prize S.r.l. - All rights reserved

The used electric network module is part of traXsim simulator [10]; line sections are modeled with MTL equations and Kirchhoff and component equations for other lumped circuits [17]. The solution principle was described in [17]-[19]. While solution methods are quite established and characterized by negligible uncertainty, how parameters values are assigned is relevant, taking into account approximations, variability, non-linearity, etc. Sources of error and uncertainty may be minimized by using reliable values and approaching modeling with the necessary level of abstraction. For track parameters (e.g. rail-to-earth conductance) different degrees of approximations may be adopted and are expected to be satisfactory, depending on the use of results. The International Review of Electrical Engineering, Vol. xx, n. x


simulator in its last implementation was already used for the analysis of power supply networks tested within the EU Project EUREMCO [20]. To this aim a first validation was performed with data recorded during long test runs along the Italian high-speed railway line [21]: the fitting of simulated and experimental curves was generally good and allowed also to preliminarily test various performance indexes. However, the Italian network was modeled using average values for various elements (such as cable connections, supply transformers, etc.), because of lack of specific information for the used network section. To proceed towards a more reliable validation, minimizing uncertainty of system parameters, new tests were performed under well-monitored conditions. The test site was the VUZ centre in Velim and its test ring in Czech Republic in 2014 (see Figure 1). Preliminary low voltage measurements were initially performed to characterize the test ring, using two different configurations (open and short circuit) fitting the 1-day time slot available with no traffic. These results were used for the validation [16].

amounting to one locomotive length are commonplace, also because when the direction of travel is reversed the driver changes cabin (or even changes locomotive, in multi-loco train consists) and a new reference point is taken without saying when e.g. stopping at signals. For the selected test locations recordings were checked that they belong to the same location and that locations are correctly positioned with respect to the test ring chainage, having unambiguously established where the km 0.000 reference point is. Recordings for each location were limited to 1 s of time duration: it is possible to assume that all data samples refer to an average position with acceptable error, observing that at the maximum speed of 100 km/h, the train travels 27.8 m in 1 s, so about one loco length [16]. Earthing system extension and conductive coupling between earthed circuits are probably one of the major modeling challenges, also because the effect is not immediately evident. Additionally, the influence of the HV feeder is difficult to model because only nominal values were known, and a rough estimate of the 110 kV network indicates the possibility of resonances in the same frequency range where the supply transformer has its own resonance in short-circuit configuration (around 7 kHz). Feeding cables connecting the transformer to the overhead line (positive terminal) and track (negative terminal) were already included and validated when preliminary Low Voltage tests were performed.

IV. Qualitative and quantitative validation Figure 1. The Velim test ring with substation (circle), test positions (filled squares) and other positions (hollow squares).

The dynamic validation is then performed selecting pantograph impedance curves Zp as the quantity for validation, because of insensitivity to current and voltage intensity, provided a minimum signal-to-noise ratio. Accurate determination of train position is of paramount importance; it is known from transmission line theory [22] that Zp resonances are related to line parameters and length, while anti-resonances are influenced by loco position. For more complex lines loaded by cables and transformers, as well as from the locomotive filter, behavior is not so simple. When antiresonances move, also the curve sides joining resonances and anti-resonances move, and this explains why small positioning errors may significantly affect accuracy. The reasons for positioning errors may be many, such as simple annotation mistakes, discrepancies in reference longitudinal positions, and even using a GPS, insufficient number of satellites, poor signal quality and displacement of the GPS antenna with respect to the pantograph in operation. In many cases, practically speaking, errors


In this section a preliminary visual validation is performed at selected loco positions, presenting to the reader the typical Zp curve shape and similarity. The validation is performed using simple and complex indexes and their results are compared. Then a new method for evaluating FSV indexes is proposed and verified, evaluating its uncertainty. Positions along the test ring were selected to probe all possible resonances and anti-resonances of the line, distributed with the maximum spatial extension for the two straight sections A and B where tests took place. Four positions were selected in each section A and B, identified by chainage: A1=km 7.1, A2=km 7.4, A3=km 8.0 and A4=km 8.3; B1=km 1.6, B2=km 1.3, B3=km 1.0 and B4=km 0.4. IV.1. Qualitative visual evaluation Measured Zp amplitude and phase curves are shown in Figure 2 and Figure 3 respectively for sections A and B and compared with respective simulations curves. Figures that distinguish between locations and allow a more accurate visual comparison were reported in [16].

International Review of Electrical Engineering, Vol. xx, n. x


(a)

(a)

(b)

(b)

Figure 2. . Pantograph impedance comparison (a) magnitude and (b) phase, in test section A (km 8.3, 8, 7.1 from darkest to lighter line), measured (thin lines) and simulated (thick lines).

Figure 3. Pantograph impedance comparison (a) magnitude and (b) phase, in test section B (km 1.6, 1, 0.1, from darkest to lighter line), measured (thin lines) and simulated (thick lines).

Curves in section A spot out the already mentioned influence of HV network feeding the transformer primary side, with phase rotation at resonance around 10 kHz and thus anticipating the preceding and following antiresonances. Conversely in section B the expected shift of anti-resonance with train position is visible, the depth of the anti-resonance dip inversely proportional to line losses. Both line resonances at 2 and 18.4 kHz are fixed while position is varying, as expected, and the slight change of peak height is due again to line losses, accounting for different length of traction line between substation and each position. There is generally good agreement between simulation and measurements, although the amplitude of the resonance at 10 kHz and the adjacent anti-resonances are characterized by the largest error; the error increases when the train is closer to substation at km 13. The resonance at 10 kHz is due to the substation transformer and influenced by the HV line frequency response, that couldn’t be accurately modeled (a simplified model was based on short-circuit power and rough estimates of line inductance and capacitance).

For section B curves are in substantial agreement both in phase and amplitude, with negligible differences also at resonances and anti-resonances; some differences are observed for the phase slightly increasing with frequency. Based on visual comparison the model can be considered validated. However visual comparison might have some weak points and validation is too subjective, without quantitative assessment of similarity.


IV.2. Quantitative numeric evaluation Differences between curves are assessed based on the three indexes and their average values interpreted using the common interpretation scale of Table I. Agreement between them is considered and commented. Numeric values of Theil, Mod. Pendry and FSV indexes appear in Table 2 and Table 3 for section A and Table 4 and Table 5 for section B, both for Zp magnitude and phase. As anticipated, reported FSV values are mean values, abandoning histograms and their evaluation that showed to be subjective. To avoid discontinuities that adversely



affect indexes based on derivatives, the unwrapped phase is used. Indexes are calculated excluding low-frequency components below 1.6 kHz due to on-board converters. Judgments following the interpretation scale in Table 1 are reported below numeric index values. TABLE 2 VALIDATION INDEXES FOR MAGNITUDE OF ZP (SECTION A) Magnitude Theil Mod. Pendry ADM FDM GDM

km 7.1 0.186 (VG) 0.279 (VG) 0.270 (G) 0.186 (VG) 0.358 (G)

km 7.4 0.152 (VG) 0.275 (VG) 0.248 (G) 0.223 (G) 0.366 (G)

km 8.0 0.259 (VG) 0.305 (G) 0.397 (G) 0.469 (F) 0.674 (F)

km 8.5 0.269 (VG) 0.330 (G) 0.470 (F) 0.516 (F) 0.764 (F)

TABLE 3 VALIDATION INDEXES FOR PHASE OF ZP (SECTION A) Phase Theil Mod. Pendry ADM FDM GDM

km 7.1 0.201 (VG) 0.135 (VG) 0.345 (G) 0.337 (G) 0.538 (F)

km 7.4 0.195 (VG) 0.100 (VG) 0.299 (G) 0.313 (G) 0.480 (F)

km 8.0 0.178 (VG) 0.046 (E) 0.372 (G) 0.297 (G) 0.523 (F)

km 8.5 0.169 (VG) 0.054 (E) 0.324 (G) 0.276 (G) 0.465 (F)

TABLE 4 VALIDATION INDEXES FOR MAGNITUDE OF ZP (SECTION B) Magnitude Theil Mod. Pendry ADM FDM GDM

km 1.6 0.135 (VG) 0.174 (VG) 0.183 (VG) 0.210 (G) 0.314 (G)

km 1.3 0.280 (VG) 0.199 (VG) 0.356 (G) 0.399 (G) 0.594 (F)

km 1.0 0.245 (VG) 0.190 (VG) 0.305 (G) 0.389 (G) 0.552 (F)

km 0.4 0.249 (VG) 0.207 (VG) 0.332 (G) 0.374 (G) 0.551 (F)

TABLE 5 VALIDATION INDEXES FOR PHASE OF ZP (SECTION B) Phase Theil Mod. Pendry ADM FDM GDM

km 1.6 0.139 (VG) 0.091 (E) 0.227 (G) 0.228 (G) 0.364 (G)

km 1.3 0.133 (VG) 0.042 (E) 0.265 (G) 0.243 (G) 0.399 (G)

km 1.0 0.121 (VG) 0.036 (E) 0.224 (G) 0.230 (G) 0.358 (G)

km 0.4 0.125 (VG) 0.067 (E) 0.247 (G) 0.245 (G) 0.388 (G)


Indexes which evaluate amplitude differences, as Theil and ADM are in substantial agreement, identifying the most relevant differences in section A near the substation and in section B for the km 11.975 location, close to the inner ring entry point. However, the range of Theil values falls in the “Very Good” score for all positions, whereas ADM has a more pronounced variation and selectively indicates a generally “Good” score with a “Fair” mark for the position closest to substation. Additionally, all indexes agree to indicate a better similarity in section B. In general there is no complete agreement between indexes that have a different structure, i.e. the simplest (Theil, and then Pendry) versus the most complex one (FSV, with ADM, FDM and GDM indexes). It is underlined that FSV is a very complex index that should be evaluated in a combined numeric and graphical form to correctly include spread and dispersion of point similarity into account. The evaluation with confidence histograms is strongly recommended by the IEEE Std. 1597.2 [13] and aims at discarding from judgment scarcely populated histogram bins. The simplified approach using mean value of FSV indexes and evaluating them following the same scale used to obtain histogram classes, can bring to pessimistic results, as pointed out also in [13]. Furthermore, index values expressed as judgment score are not suitable in case a precise evaluation and comparison is needed. It is also apparent that “Good” ADM scores in section A1 and A3 are not representative of the same type of difference between curves. It is thus necessary to use reliable numeric indexes, considering the highest number of features that characterize the difference between curves giving an all-comprehensive judgment. IV.3. Proposed robust estimation of FSV indexes The simplified approach presented in sec. IV.2 is useful from a practical viewpoint, giving a single score that weights each feature without user’s judgment bias. However, as already commented, the use of mean values leads to a pessimistic judgment of curve similarity, in particular using GDM, that is a convex function (square root of sum of squared ADM and FDM), resulting always in values larger than individual ADM and FDM terms, usually stepping back by one category. The IEEE Std. 1597.2 proposes the same interpretation scale (see Table 1) for all FSV indexes without coping with this unavoidable biasing and inconsistency. The aim of the present work is to propose a robust estimate of FSV vectors, which allow a fully numeric evaluation of results. Four considerations are done:  histograms which would represent expert’s judgment, but are obtained by a numerical equivalent, are affected by a larger dispersion if compared to experts’ evaluations [23];



 



histogram distributions are typically not symmetric and this is a further element against using only mean value of indexes[24][25]; the tallest bar of the histogram (the mode) does not indicate always the correct and most suitable judgment, being advisable to include at least the two adjacent categories to interpret shape and skew; GDM results depend on the degree of correlation between ADM and FDM [26].

The purpose is to join the useful information given by histograms and remove outliers which might affect index results, without any “human action”, in order to have a

reliable measure of curve similarity, supported by a confidence interval. The proposed robust estimation of FSV indexes begins with ADM and FDM histogram calculation as in the IEEE standard. Then categories shorter than 30% of the tallest category are identified and related points removed. Index histograms are then redrawn using a higher number of bins splitting those of Table 1 as in [27] (2 bins for Excellent and Very Good, 4 for Good and Fair, 2 for Poor) reaching a more accurate evaluation of mean value and confidence interval.

Figure 4. FSV indexes for Zp amplitude curves in section A and B: original mean value (square), original median value (diamond), robust estimation of index: mean value (star) and its confidence interval (marked with a vertical line and two crosses).

Figure 5. FSV indexes for Zp phase curves in section A and B: original mean value (square), original median value (diamond), robust estimation of index: mean value (star) and its confidence interval (marked with a vertical line and two crosses)



International Review of Electrical Engineering (I.R.E.E.), Vol. xx, n. x

The considered confidence interval is the data interval that corresponds to the assigned probability of 95% for a given distribution. Using the level of confidence of 95% is what is normally referred to as k=2 coverage factor under the assumption of Gaussian distribution. The robust mean value is in general lower than the original mean value and higher than the median, that include outliers, confirming the fact that the two are pessimistic and optimistic estimates, respectively. The median value is more robust and reliable in the presence of outliers, but is too optimistically biased, even including a slight pessimistic biasing due to folding of negative points typical of some indexes. In general, using all index samples for first-order moment calculation does not correctly form a final judgment that is in-line with the full evaluation of histogram characteristics. A comprehensive comparison includes amplitude and phase and in case of different, or even opposite, behaviors a more accurate investigation is necessary. Phase curves give in general lower confidence intervals than amplitude curves, indicating a more reliable estimate. The only relevant difference in phase curves was due to the influence of slightly larger losses in the measured results. When the robust mean value falls in “Good” or better categories, but the confidence interval is quite wide, portions of the curve are in reality dissimilar, even if on average the correspondence is satisfactory. Filtering data with a 30% threshold has the consequence that FDM results are more uniform and miss the known model deviation near the substation. It is thus evident that filtering outliers is a powerful instrument to ensure robustness of judgment with a fully numeric approach, but at the risk of loosing selectivity. Additionally, the 30% threshold is aligned with the indication contained in the IEEE 1597.2 standard, but different threshold may be advisable for different classes of problems.

V. Conclusions The problem of the quantitative validation of an electric network simulator was considered. The validation process requires various steps since the beginning of the design of the simulator and the implementation of the single modules. In the last phase the simulator response is evaluated against experimental reference data, aiming at verifying the largest number of different system configurations and parameter combinations. The output variables are in principle all the variables in the system; to limit complexity and duration of validation, the size of experimental data and the cost of measurements, the most significant quantities shall be identified. In the present case, with measurements made on-board locomotive, the pantograph impedance curve was selected, i.e. the ratio of the spectra of pantograph voltage and current.

Manuscript received April 2014, revised April 2014

After visual inspection and comparison between simulated and experimental data, quantitative validation is developed based on the comparison of several performance indexes (already evaluated and analyzed in other publications). The use of more than one index ensures that the final judgment on similarity between simulated and experimental curves is well grounded and any abnormal sensitivity to some curve characteristic is identified. The considered performance indexes are Theil, Modified Pendry and FSV. Although the most powerful and complete FSV allows the use of histograms, they are used as a second-order tool for comparison between solutions, but the evaluation of similarity is kept numerical and supported by the use of an interpretation scale (shown in Table 1). Results are analyzed and discussed, including the statistical behavior of indexes: to cope with the significant dispersion of index values, robust estimates are also proposed (median value and outliers removal). Some considerations were formulated on the suitability of interpretation scales for the various indexes, that shall cover the same scale of values with similar distribution for the comparison to be consistent. In this case FSV necessarily worsens the judgment when going from separate ADM and FDM indexes, to the combined GDM, which by its own formulation as convex function is always larger than the two. Dispersion of FSV results is in general slightly larger than that of scores of human experts. The considered indexes are subdivided into two groups, separating those evaluating amplitude difference (Theil and ADM) and curve feature difference (Modified Pendry and FDM), finding a substantial agreement. The global GDM, as said, leads to a worse judgment by its own construction. It may be concluded that the simulator is validated with an overall performance that depends on the used indexes, it is “Very Good” using Theil index, between “Excellent” and “Good” using Mod. Pendry index and between “Very Good” and “Fair”, using mean values of ADM and FDM. For FSV indexes it is underlined that using the robust estimate based on median instead of mean, the result improves staying between “Very Good” and “Good”, with some “Excellent” cases for phase curves; similarly, when mean values are calculated using 30% filtered indexes. Consequently, the agreement between indexes looking at the resulting numeric values and their respective interpretation scales is quite consistently within 50% of the width of a classification bin. The use of the robust mean and related confidence interval can be used to effectively declare the judgment on validated model and the related uncertainty.

VI. References [1]

M.U. Brandolini, C. Briano, E. Briano and R. Revetria, “VV&A of Complex Modeling and Simulation Systems: Methodologies



[2] [3]

[4]

[5] [6]

[7]

[8]

[9]

[10] [11] [12] [13]

[14]

[15] [16]

[17] [18] [19]

[20] [21]

[22]

[23]

[24]

and Case Studies,” International Journal of Mathematics and Computers in Simulation, Vol. 2, n. 4, 2008, pp. 338-348. O. Balci, “Verification, validation, and accreditation,” Winter Simulation Conf. (Cat. n. 98CH36274), 1998, vol. 1, pp. 41–48. A. Mariscotti, “On the validation of models of large complex electrical systems,” International Journal of Measurement Technologies and Instrumentation Engineering, Vol. 5, n. 1, Jan.-March 2015. Railway applications – Fixed installations – Requirements for the validation of simulation tools used for the design of traction power supply systems, CENELEC C20 working doc., May 2012. European Union, Technical Specifications for Interoperability – Energy System, 2002/733, 30-05-2002. Railway applications. Power supply and rolling stock. Technical criteria for the coordination between power supply (substation) and rolling stock to achieve interoperability, CENELEC EN 50388 Standard, Aug. 2005. A. Mariscotti, “Measuring and Analyzing Power Quality in Electric Traction Systems”, International Journal of Measurement Technologies and Instrumentation Engineering, Vol. 2 n. 4, Oct.-Dec. 2012, pp. 21-42. J. Bongiorno and A. Mariscotti, “Evaluation of performances of indexes used for validation of simulation models based on real cases,” International Journal of Mathematical Models and Methods in Applied Sciences, Vol. 9, 2015, pp.22-43. R. Jauregui and F. Silva, “Numerical Validation Methods”, in Numerical Analysis - Theory and Application, ISBN 978-953307-389-7, InTech, (2011). traXsim (integrated railway simulator): [online] www.traxsim.eu, www.astm-e.ch H. Theil, Economics and Information Theory, Amsterdam, North-Holland, 1967. J.B. Pendry, “Reliability factors for LEED calculations,” Journal of Physics – Part C: Solid State Physics, Vol. 13, pp. 937-944, 1980. IEEE Std.1597.2, “Recommended Practice for Validation of Computational Electromagnetics Computer Modeling and Simulations”, 2010. A.P. Duffy, A.J.M. Martin, A. Orlandi, G. Antonini, T.M. Benson and M.S. Woolfson, “Feature Selective Validation (FSV) for Validation of Computational Electromagnetics (CEM). Part-I-The FSV Method”, IEEE Trans. on Electromagnetic Compatibility, Vol. 48, no. 3, Aug. 2006. pp. 449-459. GCEM, “Group de compatibilitat electromagnetica. [Online]. Available http://www.upc.edu/web/gcem/ J. Bongiorno, A. Mariscotti and N. Pasquino, “Validation of railway electric simulator using scale and slope performance indexes”, IMCAS’15, Salerno, 25-29 June, 2015, in Recent Researches in Electrical and Computer Engineering, ISBN 9781-61804-315-3. C. Paul, Multiconductor transmission lines, New York, J. Wiley & Sons, 1991. CCITT Directive Vol. II  Calculating induced voltages and currents in practical cases, Geneva, 1989 A. Mariscotti and P. Pozzobon, “Determination of the Electrical Parameters of Railway Traction Lines: Calculation, Measurement and Reference Data”, IEEE Trans. on Power Delivery, vol. 19 n. 4, Oct. 2004 pp. 1538-1546. European Project EUREMCO, [online] www.euremco.eu J. Bongiorno and A. Mariscotti, “Experimental validation of the electric network model of the Italian 2x25 kV 50 Hz railway,” presented at the 20th Imeko TC4 International Symposium, Benevento (Italy), Sept. 15-17, 2014. J. Holtz and H.-J. Klein, “The propagation of harmonic currents generated by inverter-fed locomotives in the distributed overhead supply system,” IEEE Trans. on Power Electronics, Vol. 4, n. 2, Apr. 1989, pp. 168-174. A. J. M. Martin, “Quantitative data validation (automated visual evaluations)”, Ph.D. dissertation, De Montfort Univ., Leicester, United Kingdom, 1999. D. Di Febo, F. de Paulis, A. Orlandi, G. Zhang, H. Sasse, A. P. Duffy, L. Wang, and B. Archambeault, “Investigating Confidence Histograms and Classification in FSV: Part I. Fuzzy FSV,” IEEE


Trans. Electromagnetic Compatibility, vol. 55, no. 5, pp. 917– 924, Oct. 2013. [25] G. Zhang, H. Sasse, A. P. Duffy, L. Wang, D. Di Febo, F. de Paulis, A. Orlandi, and B. Archambeault, “Investigating Confidence Histograms and Classification in FSV: Part II-Float FSV,” IEEE Trans. on Electromagnetic Compatibility, vol. 55, no. 5, pp. 925–932, Oct. 2013. [26] G. J. Hankins and D. M. Lewis, “Validating the FSV Method using reverberation chamber measurements,” IEEE Intern. Symp. on Electromagnetic Compatibility, 2010, pp. 737–742. [27] J. Bongiorno and A. Mariscotti, “Simplified validation of simulation models of electric traction networks,” 12th Intern. Conf. on Modern Electrified Transport MET 2015, Oct. 4-7, 2015, Trogir, Croatia, pp. 22-29.

VII. Authors’ information 1DITEN – University of Genova, Via Opera Pia 11A, 16145 Genova, Italy 2ASTM, Via Comacini 7, 6830 Chiasso, Switzerland

Jacopo Bongiorno was born in Genova in 1987. He received the degree in Electrical Engineering in 2012 from the University of Genova Italy. As a grant holder he has worked in national and international research programs. He is presently PhD student at University of Genova. His research interests are modeling and testing of electrical and power systems, in particular electric transportation systems, and the related activities of verification and validation of simulation models. Andrea Mariscotti was born in Genova in 1968. He obtained a degree in Electronic Engineering and a Ph.D. in Electrical Engineering from the University of Genova, Italy, in 1991 and 1996, respectively. His major fields of study are electromagnetic compatibility, analysis of interference, design of instrumentation, and modeling and testing of power and electronic systems, in particular for transportation applications. He is presently on leave from the University of Genova, working at ASTM on innovative products and tools for transportation and energy applications. He is author or co-author of about 150 papers published in international journals or presented at international conferences. He authored two books on Electromagnetic Compatibility in Railways and on RF and Microwave Measurements. Prof. Mariscotti is an IEEE Senior Member and member of the Italian Electrical and Electronics Measurement Group (GMEE).
