Divide and conquer? Assessing k-means clustering of demand data in simulations of the British electricity system. Richard Green†, Iain Staffell† and Nicholas Vasilakos†‡* †
‡
Department of Economics, University of Birmingham, Birmingham, B15 2TT, United Kingdom.
Norwich Business School, University of East Anglia, Norwich, NR4 7TJ, United Kingdom.
Corresponding author:
[email protected]
Abstract We apply a k-means clustering algorithm to partition national electricity demand data for Great Britain and obtain a set of representative demand profiles for each year over the period 1994-2005. We then use a simulated dispatch model to assess the accuracy of these daily profiles against the complete dataset on a year to year basis. We find that the use of data partitioning does not compromise the accuracy of the simulations for most of the main variables considered, even with significant intermittent wind generation. This technique yields significant gains in terms of computational speed, allowing more complex Monte Carlo simulations to be performed.
JEL Codes: D43, L13, L94, Q41, Q42 Keywords: Demand data, k-means, clustering, data mining, computational efficiency, electricity modelling.
*
This research is supported by the Research Councils‟ Energy Programme and by our industrial partners through the Supergen Flexnet Consortium, grant number EP/E04011X/1. The views expressed are ours alone. 1
1
Introduction
Any model involves compromises. The modeller must often choose between a simple model that neglects some key features of the real world or a more complex one that requires a long time to solve. In the case of the electricity industry, the choice is frequently between a “merit-order stack” model that simply meets each period‟s demand from the cheapest power stations, ranked in order of increasing variable cost, and a more sophisticated “dispatch model” that takes into account the temporal nature of demand and plant operation via the time-lag and cost of starting and stopping plants. As the proportion of intermittent renewable generation increases in many countries such as the UK, the demand to be met by conventional plant is likely to become more variable, making these effects more important. Many of the input parameters into these models will be uncertain, and repeated simulations might be required to represent this uncertainty. With a complex model covering a full year plant dispatch it can easily become too costly, in terms of computing power and time, to run sufficient simulations to capture this uncertainty with Monte Carlo or sensitivity analysis. This paper presents a way of identifying a small number of “typical” days that can represent a year of operation, allowing a dispatch model to be run very fast and thus many times for stochastic or other simulations. To do this we apply a k-means clustering algorithm, a method that has been used widely in various disciplines to separate (often large) agglomerated datasets into a number of smaller groups. Membership for each cluster is decided using a similarity measure – in this case the Euclidean distance of each data point from the group mean. The objective of the process is to allocate each observation between groups in such a way so that this distance is minimised. Clustering has been used to represent electricity demand data in previous studies, but this is the first exercise that we are aware of that validates their use by comparing the results from clustered data to those from a full year‟s simulation. We create and use clusters for twelve separate years of data, thus allowing us to illustrate the reliability of this method, in terms of the differences from those obtained from the original data, and how much they vary from year to year. The data we use is partly drawn from Green and Vasilakos (2010), a simulation of wind output and demand patterns for Great Britain in the year 2020. This data is based on hourly national electricity demand and weather data from 1994 to 2005, hence giving twelve years calibrated to the same overall level of demand and wind capacity. We combine this data with a scenario for generation capacity and a dispatch model to calculate the costs and operating patterns of meeting demand from day to day. We run the model for each year of data with different numbers of clusters. We find that the clustered data give a remarkably accurate simulation for many key variables, including the average cost of electricity, carbon emissions and generator revenues. Other variables, which are more dependent on extreme events, such as generator start-ups (often in response to a short-lived spike in demand) are less accurately modelled, and so the method is not applicable to every problem that we might wish to apply it to. Nonetheless, we find that using six or ten clusters to represent a year of data gives a remarkable saving in computer time, and recommend that this method be considered to allow the complexities of electricity dispatch to be represented in studies that require many repeated simulations of an electricity market. The paper starts with an overview of previous work in the field. We then introduce the data that we use as an example in the following descriptions of the clustering technique and a means for producing representative profiles from these clusters. We then explain how the validity of these 2
clustered profiles was tested using a generator dispatch model, and follow this with results from the validation.
2
Previous work
Many problems in electricity are stochastic. Sometimes the law of large numbers can be used to model a simpler deterministic equivalent; for example, the incidence of mechanical problems that may make generating units unavailable can be represented by scaling down the available capacity of every unit by a fixed proportion. Such techniques are not always appropriate, however: Borenstein et al. (2002) point out that scaling available capacity in a deterministic manner fails to capture the convexity of the marginal cost curve. Their simulations of competitive prices (as a counter-factual to be compared with actual prices during the California electricity crisis) were based on 100 repeated model runs in which each unit had a probability of being completely unavailable. The trade-off required to make the problem manageable was that they modelled each hour separately and did not attempt to include the impact of start-up costs in their model. In using Monte Carlo methods to estimate marginal costs, they follow researchers such as Baughman and Lee (1992), who calculate electricity market spot prices and Mazumdar and Chrzan (1995), who estimate generation costs. Researchers in hydro-dominated systems have developed stochastic techniques for optimising the use of water when future inflows are uncertain, as described by Førsund (2007). These rely on backwards induction to find the optimal output pattern across (and responding to) a range of possible future inflows. More recently, uncertainty over wind speeds has become an important factor for countries with a high (or rising) proportion of wind generation. Müsgens and Neuhoff (2006) model the additional costs imposed on the German electricity system when thermal plants are committed on the basis of wind forecasts made 24 hours ahead rather than on the more accurate forecasts available 4 hours before real time. Neuhoff et al. (2008) use repeated model runs to find the optimal deployment patterns for wind and other generators across Great Britain in the presence of transmission constraints. Fuel prices differ from plant availability and the weather in that they are uncertain, rather than risky, and so we should not expect past experience to give us the full probability distribution for the future. Even so, stochastic models based upon some probability distribution may act as a guide to possible outcomes or help illustrate the underlying forces at work. Green (2008) uses a Monte Carlo analysis with variable fuel prices to compare the impact of carbon trading (with a stochastic permit price related to the fuel prices) and a carbon tax on the profits of high- and lowcarbon generators. Once again, each period within a year is effectively modelled separately, neglecting inter-temporal constraints to produce a tractable problem. Roh et al. (2009) also model a combined generation-transmission expansion planning problem with Monte Carlo methods, but introduce a scenario reduction technique to reduce the computational burden of a large number of different cases. Scully et al. (1992) discuss a semiguided Monte Carlo technique that can be used with a full inter-temporal scheduling model and greatly reduce the number of runs required to produce “reliable” production cost results. Our work is in this vein, finding a way to combine a sophisticated system model with a limited set of input data and still obtain reliable results. Data partitioning methods of the kind we propose have been used widely in the general energy economics literature, mostly as part of wider forecasting frameworks. A good number of papers show how data clustering can be used to explain sources of retail and wholesale price 3
variation, or simply how these algorithms can be used to facilitate a more complex computational framework. Most conclude that the use of clustering algorithms results in small forecasting errors and a substantial increase in computational speed. For instance, Álvarez et al. (2007) apply two separate clustering algorithms on time series data to forecast prices for the Spanish electricity market. Their results indicate that both algorithms performed very well in identifying usable patterns on the data and with small average errors. Balachandra and Chandru (1999) use k-means partitioning to estimate 9 representative load curves for the electricity system in Karnataka in 1994, which they then use to explain the sources of variation in hourly demand. Although the aim of our paper is not to explain patterns of demand variation per se, the clustering algorithm they have applied is very similar to ours. Their discussion on the performance of the method also indicates that the use of clustering methods did not result to substantial losses of forecasting accuracy (when compared with actual demand data for that year), with 86% of the percentage error for the estimated hourly loads falling within the ±10% range. The choice of a partitioning as a means to model electricity demand data is also discussed in Marton et al. (2007) who use hierarchical clustering methods to construct representative load curves for the province of Ontario, Canada. Gerbec et al. (2003) apply and compare two different partitioning algorithms (fuzzy c-means and hierarchical clustering) on a sample of 288 measured load profiles of eligible retail consumers obtained from Slovenian distribution companies. Both methods produced similar results (although not identical, in terms of the optimal number of clusters and group membership of individual observations), with the fuzzy c-means algorithm supporting 8 groups as opposed to 6 when the k-means algorithm was used.
3
Generating clustered demand data
The basis for our work is a set of national hourly demand and wind generation data, which represents the British electricity system in 2020 and is drawn in large part from Green and Vasilakos (2010). That paper took 12 years of hourly wind speed data from the British Atmospheric Data Centre, covering a number of weather stations around Great Britain. These data were cleaned to interpolate missing observations (where just one or two consecutive observations were missing) or to fill them in on the basis of regressions against the wind speeds at other stations (where the gap was longer). The hourly wind speeds were then adjusted for the height of a typical wind turbine, and a power curve was used to calculate the output (per MW of capacity) as a function of the wind speed. The wind speed for each station was scaled up or down by a constant (station-specific) percentage if this was required to give a plausible average load factor for the wind turbines in that area – many weather stations are located in sites which are (locally) insufficiently windy for a wind farm. We assumed that the wind speed at a suitable generation site was related to, but higher than, the speed at the nearby weather station. We used a similar method to obtain offshore wind outputs from coastal weather stations. The per-MW outputs were multiplied by a total of 11 GW of onshore and 19 GW of offshore capacity, distributed around Great Britain in line with British Wind Energy Association predictions. The wind data for 1994–2005 were then combined with hourly demand data for the same period, produced by combining the half-hourly figures given by National Grid. We assumed future demand growth of 0.9% per year, giving a total demand in 2020 of 382 TWh in normal weather conditions. The individual demand figures for each year in our data were scaled by the ratio of the actual demand to the weather-adjusted demand for that year, giving a range across the twelve years 4
of 382.2 ± 2.6 TWh. This preserves the day-to-day and year-to-year variation in demand due to weather conditions, ensuring that our hourly wind outputs are matched with a compatible demand for electricity. We repeat our simulations with both the gross demand figures (effectively assuming that there is no wind power at all) and the demand net of wind output. Figure 1 summarises the two data sets, showing the general structure of demand throughout the day and the range in demand levels seen over the 12 years. The net data (with wind subtracted) shares the same intra-daily structure as the gross data, albeit with far greater variability between days. A notable feature, and one which will present serious technical challenges for the GB system, is that peak demand can coincide with a lull in wind, as can troughs in demand with full wind output. Based on the estimated output from 30 GW of wind capacity, peak GB demand falls by just 2.3 GW from 67.4 GW to 65.1 GW net of wind, whereas the minimum level of demand falls from 24.0 GW down to just 1.5 GW. 70
70
(a) Gross demand (GW)
50 40 30 20 10 0
(b)
60
Net demand (GW)
Percentiles: 85-95% 75-85% 65-75% 55-65% 45-55% 35-45% 25-35% 15-25% 5-15% Max/Min
60
50
40 30 20 10 0
0
4
8
12
16
20
Hour of the day
0
4
8
12
16
20
Figure 1: The distribution of demand across the 4,383 days of (a) gross national demand, and (b) demand net of wind output.
3.1 Clustering method Data partitioning works by allocating individual observations into smaller groups in such a way that (a) observations that exhibit some common feature (built-in within the algorithm) are grouped together and (b) the groups that are created are dissimilar enough from each other to make them distinct. To achieve this, a clustering algorithm applies a measure of dissimilarity to establish group membership. The k-means algorithm used in this exercise employs as a dissimilarity measure the Euclidean distance of each demand vector (containing 24 hourly demand points) to allocate daily demand profiles to k groups. Membership for each group is decided as a solution to the minimization problem: (1) where is a 24-dimensional demand vector and is the mean for group i to which this particular vector has been allocated. In other words, all j vectors are allocated between the k groups in such a way so that they are the closest possible to their group mean. We initialise the procedure by defining a set of k demand vectors from the dataset as centroids for each cluster. The algorithm then allocates the demand vectors between the clusters so that the objective function in (1) is minimised. The new centroid is then recalculated for each group as the average of the values of all vector 5
members in each hour-dimension. The procedure repeats itself until convergence is achieved. A different value of k would naturally produce a somewhat different set of clusters, and so we experiment with a wide range of values.
3.2 Clustered demand data This technique was applied to our data, matching both sets of demand (gross and net of wind) to 5, 6, 8, 10, 15 and 25 clusters. Figure 2 gives an example of how six clusters were distributed across the twelve years of gross demand data. Even though the algorithm did not take explicit account of any meta-data (such as ambient temperatures or dates), these clusters show strong correlations with season and day of the week (including national holidays). Dividing this data into more clusters simply increased the number of bands between seasons: 8 clusters had two grades of shoulder day, 10 had three grades, and so on. Beyond this we found that very small clusters of anomalous days begin to form, e.g. for Christmas, New Year, and extreme weather events. Winter weekend
Winter weekday
Shoulder weekend
Shoulder weekday
Summer weekend
Summer weekday
2004 2002 2000 1998 1996 1994 January February
March
April
May
June
July
August September October November December
Figure 2: Spectrogram showing the distribution of six clusters fitted to the gross data set.
The clusters are distributed similarly between years. The boundaries between seasons show relatively little variation, and they correlate well with temperature records for the UK (Met Office, 2011). For example, March 1997 was unusually warm, and May 1996 was colder than average, hence both were predominantly classified as having shoulder days. The labels given in Figure 2 were manually assigned to the clusters based on our observations. Figure 3 shows that a high proportion (82–94%) of the days within each cluster fall into their „assigned‟ season; where, based on this data (from an electricity-consumption point of view), we defined winter as running from the 8th of November to the 16th of March, and summer from the 5th of May to the 27th of September.
6
Winter
Shoulder
Summer WD = Weekday WE = Weekend
100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% Winter WD WE
Shoulder WD WE
Summer WD WE
Figure 3: The seasonal distribution of days within in each cluster shows a good correlation across all twelve years.
The strong influence of stochastic output from wind turbines meant that clusters for net demand data did not fall into such clear-cut patterns. Figure 4 shows the distribution of ten clusters; there is still a visible distinction between summer (light) and winter (dark), but the clusters are also determined by the pattern of wind output during the day (colours / hatching). From our observations of the profiles within each cluster, we found that they converged on three broad patterns of wind output: predominantly windy during the morning (A–C); predominantly windy during the evening (D–G); or no overall pattern (H–J). The profiles only loosely adhere to these categories, and so they were simply assigned letters rather than descriptive names in Figure 4. A
B
C
D
E
F
G
H
I
J
2004 2002 2000 1998 1996
1994 January
February
March
April
May
June
July
August September October November December
Figure 4: The distribution of ten clusters fitted to the demand net of wind output.
3.3 Producing demand profiles Once the individual days have been grouped into clusters, there are several methods for reducing each cluster down to a single, representative profile. The main options are to average, selectively filter, or randomly choose from the days within the cluster; each of which has its relative merits. For example, averaging allows every single day to contribute to the final result, but incurs the penalty of temporal smoothing. By selecting a single random day from each cluster, any smoothing effects are eliminated, but other problems are incurred. The total level of demand over the entire period will not be preserved, although this can be remedied by scaling the total demand of each day to equal the average demand from all days within its cluster. The main drawback is the sheer variation in end results that is obtained depending on which particular days were chosen. We could instead choose the specific day that best represents the cluster, namely the one that is closest to the cluster‟s mean (as defined by the same process that derives the clusters). However, 7
as the cluster mean is smoothed out, the day that is most similar to it will be an unusually smooth day, and thus will poorly represent the entire data set. A less extreme option is to filter out days from within each cluster, for example removing those which deviate most from the cluster‟s average, or which have markedly different total levels of demand. This also appears to offer no benefits, presumably because removing the influence of even „spurious‟ data does not bring the resulting profile closer to the complete data set. The principle of the first option – including all the data with no discrimination – was found to work best. A basic implementation consists of calculating the average demand level from the first hour of all days within the cluster and assigning that to hour 1 of the resulting profile, then averaging all levels from the 2nd hours to give hour 2, and so on. This naturally preserves the total energy demand over the entire day, but has a smoothing effect on the resulting profile. Extreme values at both ends of the spectrum (e.g. unusually high and low demands at 4am) cancel each other out, as do temporal differences. Inter-temporal smoothing has a greater effect, as the rate of change of demand shows greater swings than the levels of demand itself. For example, if the evening peak on one day occurs at 8pm, the change in demand from 7 to 8pm will be positive, and from 8 to 9 pm will be negative. When this day is averaged with another where the evening peak occurs at 9pm, the ramp rates from 8 to 9pm will cancel each other out, resulting in a plateau rather than a spike. Ideally this situation should not occur as the clusters were formed based on similarity in both demand and ramp rates, however it is inevitable when small numbers of clusters are under consideration. This is reflected in our actual data set, where morning and evening peaks get broadened out over time and lessened in intensity, as demonstrated in Figure 5. 65 06/01/1995 21/02/2003 Average
Demand (GW)
60
55
50
45
12
14
16
18 Hour
20
22
24
Figure 5: Example of an evening peak being smoothed out by averaging two otherwise similar days.
A better alternative is therefore to average the days within each cluster based on both their levels of demand and the changes between these levels. Since the mean of the changes between two adjacent hours is equal to the change between their mean levels, the average changes provide no extra information. Instead we consider a subset of the changes: either those which have the same sign as the mean change in that hour (i.e. the dominant / mean change); or those which have the most common sign in that hour (i.e. the common / median change). In other words, if the average profile has an increase in demand between 7 and 8pm, the dominant profile takes the average of all positive changes between those hours. It is possible that between those hours the majority of profiles have a small negative change which is outweighed in magnitude by the minority of large positive changes. 8
In this case, the common profile would instead be determined by the average of the negative changes, as illustrated in Figure 6. In both cases this gives us 23 hour-to-hour changes between adjacent periods. Given these changes, the demand in the first hour of the day is sufficient to determine the rest of the profile, and this is chosen so that the mean demand over the profile equals the mean over the entire cluster. We compare these options (the profiles built from the dominant and the commonest changes in demand) with the profile given by the cluster mean and with equally weighted mixes of dominant/mean and common/median. 65
65
(a)
(b) 60 Demand (GW)
Demand (GW)
60
55
50
06/01/1995
45
55
50
Level average Dominant ramp Common ramp
45
21/02/2003 18/02/2005 40
40
0
2
4
6
8
0
10 12 14 16 18 20 22 24 Hour
2
4
6
8
10 12 14 16 18 20 22 24 Hour
Figure 6: Three sample daily profiles (a), and the merged profiles that result from the three grouping methods (b).
The three sample profiles given in Figure 6a are similar throughout the day except between 16:00 and 17:00, which is highlighted. Between these hours, demand increases substantially on 06/01/1995, but decreases slightly on the other two days. This gives a marked difference in the common and dominant merged profiles shown in Figure 6b: the dominant profile rises by 5.6 GW, whereas the common profile falls by 0.2 GW. This highlights a drawback of using the dominant changes, as the demand level rises for three consecutive hours, from 16:00 to 19:00 and peaks at a higher level than any of the constituent days it is meant to represent. To balance this increase in evening demand, the whole profile was normalised downwards (to maintain the correct overall level of demand), meaning that it tracks about 1.5 GW lower than the average profile for the earlier part of the day. In contrast, the profile built from the common ramps shows similar levels of demand to the simple average, but sharpens the evening peak at 19:00. This is the method used in this paper. Finally, when implementing the clusters it is important to assign a weighting factor, w, to each of the resulting profiles, equal to the number of days that were contained within its cluster. All results derived from that profile are then multiplied by its weighting factor, giving the same results as if the simulation were run on w repetitions of the profile. This emphasises the importance of the more „typical‟ clusters which contain the majority of the data (e.g. summer weekdays), and prevents the outlying clusters (shoulder weekends, anomalous weather events) from dominating.
3.4 Clustered demand profiles Figure 7 gives the profiles that were formed from the 6 gross clusters shown in Figure 2, produced from the common ramp rates.
9
60
Demand (GW)
50 40 30
Winter weekend (457 days) Winter weekday (1071 days) Shoulder weekend (353 days) Shoulder weekday (693 days) Summer weekend (583 days) Summer weekday (1226 days)
20 10 0
0
3
6
9
12 Hour
15
18
21
24
Figure 7: Six daily profiles that represent the gross UK electricity demand in 2020. The names we assigned each profile are given along with their weighting factors, showing how many days they represent.
Demand (GW)
Similarly, Figure 8 gives the profiles that were formed from the 10 net clusters shown in Figure 4, separated by the pattern of wind output. This categorisation of profiles was somewhat arbitrary, as it was not possible to divide the clusters into clearly defined groups. Net demand on a winter‟s day with relatively high wind output is indistinguishable from a summer‟s day with relatively little wind. We therefore characterised the profiles based on the shape of the demand profiles, loosely correlating to windy nights (left), windy afternoons (right), and „other‟ patterns (centre).
60
60
50
50
40
40
30
30
20
20 D (344 days) E (378 days) F (492 days) G (288 days)
A (323 days) B (328 days) C (443 days)
10
H (300 days) I (745 days) J (742 days)
0
10 0
0
3
6
9
12 15 18 21 24 Hour
0
3
6
9
12 15 18 21 24 Hour
0
3
6
9
12 15 18 21 24 Hour
Figure 8: Ten daily profiles that represent UK electricity demand in 2020, net of 30 GW of wind output.
A seemingly negative impact of using the common or dominant ramp rates can be seen between the last and first hours of profile B (i.e. hours 23 and 24 = 0 in Figure 8). The late evening demand is on a steady downward trajectory, then drops abruptly from 38 to 20 GW, rising slightly to 22 GW in hour 1. The method described in the previous section does not force the profiles to begin and end on similar values, so if they are wrapped around as in Figure 8, two days of the same profile following one after the other do not appear to be „natural‟. Especially large ramps can be seen in profiles B and C, or ramps with the wrong apparent sign in profiles I and J. An attempt to remove this bias was made by including the ramp rate from hour 23 to hour 0 in the optimisation process of producing the common and dominant ramp profiles. A constant bias 10
was added to all the ramp rates so as to alter the change between the end and beginning of the profiles. For example, in profile B every ramp rate was reduced by 0.47 GW, so that the change from hours 23 to 0 reduced from –16.7 GW to –5.5 GW. This however proved to be counterproductive, and greatly reduced the accuracy of the clusters in terms of the final results, and so the anomalous end-of-day boundaries were retained.
4
Validation of the clusters
Having obtained sets of clusters, our aim is to determine whether a small number of clusters can be used as an acceptable representation of a full year of data in a simple dispatch model that estimates grid operation patterns and costs. Our simulations use a standard linear programming approach, with a program derived from the MAGIC simulator in the GAMS model library (Day and Williams, 1982). The program aims to minimise the total cost of meeting the demand for electricity, subject to constraints on the maximum and minimum generation from each unit in operation. The cost of power includes a no-load cost for keeping each unit running, a cost per MWh of output and a start cost, each of which depends on the type of power station considered. Each model run consists of one or more complete days of data, and to avoid the possibility of starting or finishing the period with an unrealistic amount of plant in operation, the solution is constrained to start and end with the same stations in use (since the demand at midnight and at 11 pm is similar). We treat each year of data as a separate test, which allows us to report distributions for our results over the 12 years we have available. For each year, the base case consists of running the model for the entire year as a single run. We then run the model for a small set of clusters, running each day separately. We weight the results for each cluster by the number of days it represents to obtain a value (average or total, as appropriate) for the year as a whole. The key results are the demand for power (which will equal the base case, by construction), the output by plant type, the average cost of electricity and carbon emissions. We also measure the marginal cost (which would equal the market price in a perfectly competitive market), the number and cost of plant starts, and the profits each type of plant would make. We have taken generating capacity figures from National Grid‟s Seven Year Statement for 2010 and applied our own judgement for investment and plant retirements in the years to 2020. Plant costs are largely based on Mott MacDonald‟s (2010) generation cost update for the Department of Energy and Climate Change, which gives projected fuel prices and overall thermal efficiencies for different types of plant. We use central prices of £20.40/MWh for gas, £9.52/MWh of coal, and £30.00/tonne of CO2, based on the Budget 2011 announcement of the Carbon Price Support scheme. We follow Troy (2008) and Meibom (2008) in deriving start-up costs, attempting to capture both fuel costs (including carbon) and the cost of increased wear and tear from heating and cooling the station. Table 1 gives the types of dispatchable plant considered; run-of-river and pumped storage hydro were reduced to a single unit each due to having no lower limit on output, and were constrained to optimise their outputs given 25 and 15 GWh of available storage which was replenished at constant rates to give average load-factors of 0.415 and 0.150 respectively.
11
Unit Total Minimum No-load Incremental Number Start-up capacity capacity stable cost cost of units cost (£) (MW) (GW) generation (£/hour) (£/MWh) Nuclear 500 20 10 75% 4,000,000 3510 7.39 CCGT (large) 750 28 21 50% 79,564 2226 48.42 CCGT (small) 350 22 7.7 50% 26,727 1039 53.64 Coal (large) 525 44 23.1 33% 198,528 3356 51.37 Coal (small) 150 20 3 33% 60,385 959 55.96 OCGT 30 70 2.1 10% 377 89 85.85 Oil 50 144 7.2 10% 596 320 116.81 Run-of-river hydro 1360 1 1.36 0% 0 0 0 Pumped hydro 2750 1 2.75 0% 0 0 0 Table 1: Parameters for the dispatchable generator types on the UK grid with central fuel prices and a £30/T carbon price. Generator type
5
Results
The initial series of tests consisted of 7,560 model runs: 2 types of demand data (gross and net of wind); 12 years of data; 7 sizes of cluster (k = 1, 5, 6, 8, 10, 15, 25). The trivial case of k = 1 (i.e. no clustering of the data) was included as a control experiment; 15 clustering methods, including variations and combinations of those listed in Section 3.3; 3 carbon prices (£10, £30, £50 / tonne CO2), to cover the range of roles for coal and CCGT plant. One of the complete year-long profiles could be processed in 19–22 minutes, whereas the average for a cluster of 6 daily profiles was 19 seconds, rising to 26 seconds for a cluster of 10 days. Because most of the calculations were for small groups of clustered data, it was possible to complete these model runs in under a day of computing time, in spite of the number of trials undertaken. Running the same number of trials using a full year‟s data would have taken approximately two months. It should be noted that approximately 10 seconds of each model run was spent reading and writing Microsoft Excel files, so with simpler file handling we expect clustering could increase processing speed by a factor of over 100. Of the various cluster sizes and methods, we found that 6 clusters gave the best trade-off of speed and accuracy for the gross data, while 10 clusters were required to cope with the more variable nature of the net data. In both cases, merging the clusters based on the most common ramp rate yielded the best results, in terms of closeness to the true values and minimising variation in closeness across different metrics, across years and carbon prices. Figure 9 gives a comparison of the clustering method to the complete data sets for a selection of metrics with the gross data. Each plot gives a fan chart, showing the range of results from the 12 years of data against the number of clusters. The accuracy for cost metrics is very good, with an average error of 0.4±0.2% for 6 clusters. For average and marginal carbon intensities, annual outputs and revenue, these average errors increased to between 3 and 7%. Remarkably, for these metrics, a single daily profile created from a whole year‟s observations gives very similar results to those from 10 or even 25 clusters.
12
101%
101%
Average cost of electricity (avg. £44.70/MWh)
105% 104% 103% 102% 101% 100% 99% 98% 97% 96% 95%
Marginal cost of electricity (avg. £57.38/MWh)
100%
100%
99%
99% 1
5
6
8
10
15
25
1
110%
110%
105%
105%
100%
100%
95%
95%
Total annual CCGT output (avg. 175.7 TWh)
90%
5
6
8
10
15
25
5
6
8
10
15
25
Annual CCGT revenue (avg. £11.6 bn) 1
5
6
8
10
15
1 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0%
90% 1
Average carbon intensity (avg. 405 kg/MWh)
25
5
6
8
10
15
CCGT Start-ups (avg. 53 per plant per year)
1
5
6
8
10
15
Figure 9: Comparisons of the results from clustered data with different values of k (horizontal axis) to the values obtained with the gross annual data sets for a selection of metrics.
A notable problem can be seen in the lower right plot of Figure 9; the simulation with clustered data predicted few or no start-ups for CCGT and coal plants, compared to averages of 53 and 2 start-ups per plant per year for the complete data sets. Because of this, the total cost of start-ups was underestimated by an average of £47m per year, which accounted for more than half of the £80m underestimate in the total cost of generation, or 0.3 percentage points of the total error. This highlights an inherent drawback of clustering; it is not able to represent low frequency events such as plant starts, outages, or peak requirements. Figure 10 repeats the comparison given in Figure 9, showing the accuracy of the clustering method when using the demand data net of wind. The additional stochastic variation introduced with the wind demand meant that in most cases the clusters were less accurate for bulk properties such as average costs and carbon emissions; however, the output, revenue and number of start-ups for different plant types were more accurately modelled with the net demand than they had been with the gross demand data.
13
25
25
102%
105% 104% 103% 102% 101% 100% 99% 98% 97% 96% 95%
Average cost of electricity (avg. £41.55/MWh)
101%
100%
99%
98% 1
5
6
8
10
15
25
110%
100%
95%
Total annual CCGT output (avg. 155.1 TWh)
90% 1
5
6
8
10
15
1 105% 104% 103% 102% 101% 100% 99% 98% 97% 96% 95%
105%
25
105% 104% 103% 102% 101% 100% 99% 98% 97% 96% 95%
Marginal cost of electricity (avg. £55.37/MWh)
5
6
8
10
15
25
5
6
8
10
15
1 200% 180% 160% 140% 120% 100% 80% 60% 40% 20% 0%
Annual CCGT revenue (avg. £10.1 bn) 1
Average carbon intensity (avg. 318 kg/MWh)
25
5
6
8
10
15
CCGT start-ups (avg. 120 per plant per year)
1
5
6
8
10
15
Figure 10: Comparisons of the results from clustered data with different values of k (horizontal axis) to the values obtained with the net annual data sets for a selection of metrics.
The benefit of clustering, rather than creating a single profile from a whole year‟s data, is clearly visible with the net demand data, given the step-change in results going from 1 up to 5 clusters. With the gross data, this was only pronounced in a selection of metrics, for example the number of OCGT start-ups and marginal carbon intensity. Overall, moving from 1 to 6 or more clusters reduced the average error by a fifth with the gross data and by three-fifths with the net data, averaged over all the metrics considered. We note that the errors for some variables continue to fall by noticeable amounts as we move from 10 to 25 profiles, although this is not the case for the marginal cost or revenue estimates. Similarly, using the commonest ramp rate method of merging clusters into a single profile was seen to reduce the average errors by two-fifths compared to using the average profile, and by one-fifteenth compared to selecting six or ten random days from within each cluster. In almost all of the metrics considered, the errors introduced through clustering were determined to be normally distributed by the Jarque-Bera test at the 5% significance level, albeit with means that were often non-zero. The only exceptions were in metrics that related to extreme events, for example the number of CCGT start-ups shown in Figure 9.
6
Sensitivity to input data
When these simulations were repeated with carbon costs of £10 and £50 per tonne of CO 2, the accuracy of the clusters improved in many respects. It can be calculated from Table 1 that a £30 carbon cost produces a relatively rare situation with our fuel price assumptions where the incremental costs of coal and CCGT plants intersect. Between a carbon price of £24.15 and £34.92 per tonne, large CCGT plant have the lowest incremental cost, followed by large coal, then small CCGT and small coal. This scenario results in larger errors in the ratio of coal to gas used in the supply mix, as a minor change in the demand curve could result in substantial numbers of coal plant starting up instead of CCGT, or vice versa. 14
25
25
As a test of the robustness of the clustering method, we repeated the tests of Section 5 with carbon prices ranging from £10 to £50 per tonne, in steps of £1. A standard desktop computer performed the tests using the optimum clustering method (common ramp) and number of clusters (6 for gross data, 10 for net); while a small parallel computer cluster was used to run the model with the full annual data sets. Figures 11 and 12 illustrate how the clusters compare to the full simulations with gross and net demand respectively. The solid line with error bars depicts the results from the complete data sets, while the shaded areas give the ten-percentile intervals for the clustered results.
600
£50
£45
£40
-£0.20
-£0.30 -£0.40 £15
£20 £25 £30 £35 £40 Carbon price (£ / tonne CO₂)
£45
500
450 400 350
Residuals
-£0.10
£10
Annual results, with 5–95% limits
300
£0.00
£35
Clustered results, with 5–95% limits 550
-£0.50 £50
£10
£15
£20 £25 £30 £35 £40 Carbon price (£ / tonne CO₂)
£45
30 20 10 0 -10 -20 -30 £50
Residuals
Average Carbon Intensity (kg/MWh)
Average Cost of Electricity (£/MWh)
£55
750
£0.20
700
£0.15 Annual coal profit (£bn)
650 600 550 500 450 400
Annual results, with 5–95% limits
£0.10 £0.05 £0.00 -£0.05 -£0.10 -£0.15
100
300
50 0
-50 £15
£20 £25 £30 £35 £40 Carbon price (£ / tonne CO₂)
£45
£0.08
-£0.20
£0.04
-100 £50
£0.00 -£0.04
£10
£15
£20 £25 £30 £35 £40 Carbon price (£ / tonne CO₂)
£45
-£0.08 £50
Figure 12: Comparisons of the results from clustered days and net annual data sets across a range of carbon prices.
15
Residuals
350
£10
Clustered results, with 5–95% limits
Residuals
Marginal Carbon Intensity (kg/MWh)
Figure 11: Comparisons of the results from clustered days and gross annual data sets across a range of carbon prices. The deviation of the clusters from the true values is shown underneath each plot on a separate scale.
Both the average and marginal cost of electricity progress smoothly upwards with increasing carbon cost, which is reliably replicated by the clusters in all cases. The clusters gradually drift further away from the true values in absolute terms, but the average absolute errors are small, reaching maximum values of £0.38–1.07 per MWh at £50 per tonne of carbon (relative errors of 0.7–1.6%). The clusters always under-predict costs, which is something that can easily be corrected for. The average and marginal carbon intensity display more complex patterns across the range of carbon prices considered. Average intensity falls in stepped increments, with particularly strong jumps occurring at £20, £25 and £35 per tonne due to the interaction between fuel prices and plant efficiencies (given in Table 1). The clustered demand profiles are able to replicate the general pattern of the average carbon intensity (Figure 11), but the locations of these step changes are wrongly predicted. For a narrow range of carbon prices, the clustered results over-predict emissions, instead of their more usual slight under-prediction. The marginal carbon intensity is more reliably predicted throughout the whole range, however the absolute errors are of larger magnitude (both for the gross and net data). Output from CCGT and coal plant types follow the same pattern as the average carbon intensity (with CCGT output going as the inverse of coal and carbon); and so these exhibit the same structural errors between carbon prices of £15 and £25 per tonne. The frequency of plant start-ups is not reliably modelled at any carbon price with the low number of clusters chosen. It remained at zero for all trials with the gross data, and varied between 0 and 3 times the true value with the net data. Finally, operator profit took a relatively complicated form (as in Figure 12) but was represented relatively well by the clusters. For both CCGT and coal the deviations increase substantially at carbon costs above £35 per tonne. Table 2 summarises the accuracy of the clusters over all carbon costs, giving the mean, standard deviation and maximum relative error; defined as the difference between clustered and true values relative to the true value, over the range of £10 to £50 per tonne of carbon. Some general trends can be seen: costs and average carbon intensity are modelled with the greatest precision, followed by marginal carbon and plant outputs, and again followed by generator revenues and profits. The relative error in profits is influenced upwards by the fact that the true values (coming from the annual data) were often around zero, e.g. for coal plants between £20 and £35 per tonne).
Metric Average cost Marginal cost Average carbon Marginal carbon CCGT output Coal output CCGT profit Coal profit
Average relative error –0.5 ± 0.2% –0.8 ± 0.6% –1.8 ± 2.0% –3.5 ± 5.2% –3.5 ± 4.8% –7.0 ± 6.8% –4.3 ± 5.3% –7.4 ± 9.2%
Maximum relative error 0.9% 1.9% 5.9% 15.1% 16.0% 19.2% 29.1% 32.3%
Table 2: Summary statistics for the deviations between clustered results and those from annual profiles.
What can be concluded from this exercise is that the way in which the clusters deviate – in terms of systematic or random errors and in terms of the sign or magnitude of the error – cannot (yet) be predicted in advance. This is an issue that we suggest researchers take into consideration 16
by testing the accuracy of clustering against the complete data set at selected points throughout their range of input parameters before embarking on a large stochastic study with clustered data.
7
Conclusions
This paper has used the k-means clustering techniques to derive small sets of electricity demand profiles that can be used in a simulation model to represent a full year of data. This allows a complex model to be run many times to consider the impact of variations in other input parameters, such as fuel prices. If a large time-series of operation has to be simulated, restrictions on computing time might force researchers to use a simpler model, or consider fewer sets of the other variables. We found it straightforward to obtain clusters from simulated electricity demand and wind output data for Great Britain in 2020, and to derive appropriate daily profiles for gross demand and demand net of wind. We found that six profiles gave an adequate representation of gross demand, while ten were required for the more variable demand net of wind output. These clusters were able to replicate the results of simulations based on the entire data set to within an accuracy of ±10% in all cases, and to within ±1% when estimating the average or marginal costs of electricity. The use of clustered data presents researchers with a trade-off: a reduction in the absolute accuracy of results against the ability to conduct Monte Carlo or sensitivity studies with one hundredth of the computing resource. The reduction in uncertainty gained by using more complex models or broader sets of inputs outweighs that introduced by clustering; and provided that the errors introduced by clustering are investigated and understood for the specific scenario, it should be possible to qualitatively (if not quantitatively) correct for them. The clustering technique is clearly ill-suited to modelling rare events and their consequences, but our work shows that it can be used to good advantage for simulating normal operations.
8
References
Balachandra P. and V. Chandru (1999), “Modelling electricity demand with representative load curves”, Energy, Vol. 24, No 3, pp 219-23 Baughman, M.L., and W.W. Lee (1992) “A Monte-Carlo Model for Calculating Spot Market Prices of Electricity”, IEEE Transactions on Power Systems, vol. 7, is. 2, pp. 584-90 Borenstein, S., J. Bushnell and F. Wolak (2002) “Measuring Market Inefficiencies in California's Restructured Wholesale Electricity Market” American Economic Review vol. 92, no. 5 (December), pp 1376-1405 Day, R.E. and H.P. Williams (1982) MAGIC: The design and use of an interactive modeling language for mathematical programming. Tech. rep., Department Business Studies, University of Edinburgh Førsund, F.R. (2007) Hydropower Economics, New York, Springer Gerbec, D. S. Gašperič And F. Gubina, (2003) Comparison of Different Classification Methods for the Consumers' Load Profile Determination, 17th International Conference on Electricity Distribution proceedings, Barcelona Green, R.J. (2008) “Carbon Tax or Carbon Permits: The Impact on Generators‟ Risks,” Energy Journal, vol. 29, no. 3, pp. 67-89 Green, R.J. and N. Vasilakos (2010) “Market Behaviour with Large Amounts of Intermittent Generation” Energy Policy, vol. 38, no. 7, pp. 3211-3220
17
Martínez-Álvarez, F., A. Troncoso, J. C. Riquelme, and J. M. Riquelme (2007) “Partitioningclustering techniques applied to the electricity price time series” IDEAL'07 Proceedings of the 8th international conference on Intelligent data engineering and automated learning, Berlin, Springer-Verlag Marton, C.H., A. Elkamel and T.A. Duever (2008), “An order-specific clustering algorithm for the determination of representative demand curves”, Computers and Chemical Engineering, No 32, pp. 1365–1372 Mazumdar, M and L. Chrzan (1995) “Monte Carlo Stochastic Simulation of Electric Power Generation System Production Costs Under Time-Dependent Constraints” Electric Power Systems Research, vol. 35, no. 2, pp. 101-8 Meibom, P., R. Barth, H. Brand, B. Hasche, H. Ravn and C. Weber (2008) “All Island Grid Study”. Wind Variability Management Studies Workstream 2B Met Office. (2011) “Historic station data”. Cited June 2011; URL: http://www.metoffice.gov.uk/climate/uk/stationdata/ Mott MacDonald (2010) UK Generation Costs Update, Brighton, Mott MacDonald Müsgens, F., and K. Neuhoff (2006) Modelling Dynamic Constraints in Electricity Markets and the Costs of Uncertain Wind Output, Electricity Policy Research Group working paper CWPE 06010 and EPRG 0514, University of Cambridge Neuhoff, K., J. Cust and K. Keats (2008) “Implications of intermittency and transmission constraints for renewables deployment”, in M. Grubb, T. Jamasb and M. Pollitt (eds.) Delivering a Low Carbon Electricity System: Technologies, Economics and Policy, Cambridge, Cambridge University Press Roh, J.H. and M. Shahidehpour (2009) “Market-Based Generation and Transmission Planning With Uncertainties” IEEE Transactions on Power Systems vol. 24, no. 3, pp. 1587-98 Scully A, A. Harpur, K.D. Le, J.T. Day, M.J. Malone, T.E. Mousseau, S. Hirakawa, A. Brennan, C.J. Stansberry, T.M. Jackson, T.R.W. Russell and D.J. Anelli (1992) “Using a SemiGuided Monte-Carlo Method for Faster Simulation of Forced Outages of Generating-Units” IEEE Transactions on Power Systems, vol. 7, no. 3, pp. 1313-21 Troy, N., E. Denny and M. O‟Malley (2008) “The Relationship Between Base-load Generation, Start-up Costs and Generation Cycling”. Presented at the 28th USAEE/IAEE North American Conference, New Orleans
18
