Page 1





CONTENTS 1 1.1 1.2 1.3

INTRODUCTION: WHAT IS IT ALL ABOUT? ____________________________________________ 4 Data: the new oil of the 21st century? ........................................................................................ 6 Convergence of structured and unstructured data ................................................................. 7 Reading guide .............................................................................................................................. 9

2 THE POWER SYSTEM AND ITS CHALLENGES _________________________________________ 10 2.1 The use of data in the power system ........................................................................................ 14 3 3.1 3.2 3.3 3.4

THE IMPACT OF DATA ANALYTICS ON FORECASTING AND PLANNING ____________________ 16 Asset management: investments and maintenance ............................................................... 17 Renewable generation: resource planning and dispatch ....................................................... 23 Demand forecasting .................................................................................................................... 24 Summary for forecasting and planning .................................................................................... 25

4 4.1 4.2 4.3 4.4 4.5 4.6

THE IMPACT OF DATA ANALYTICS ON OPERATIONS ____________________________________ 26 Operations in the power system ............................................................................................... 27 The impact of analytics on grid operations ............................................................................. 30 The impact of analytics on commercial wholesale operations ............................................. 32 The impact of analytics on retail operations ........................................................................... 33 Aggregating demand response in smart energy systems .................................................... 34 Summary for operations ............................................................................................................ 37

5 5.1 5.2 5.3 5.4 5.5

THE IMPACT OF DATA ANALYTICS ON SETTLEMENT ______________________________ 38 Settlement in unbundled markets ...................................................................................................................... 38 The impact of data analytics on wholesale settlement ........................................................................ 39 The impact of smart metering ................................................................................................................ 40 Advanced energy contracts ........................................................................................................................ 41 Summary for settlement ................................................................................................................. 46

6 CONCLUSION AND SUMMARY _______________________________________________________ 48 6.1 Applying data analytics in the electricity sector ....................................................................... 49 6.2 Final reflection ............................................................................................................................ 50 7 REFERENCES/SOURCES _______________________________________________________ 51 8 END NOTES ___________________________________________________________________ 52

1 - INTRODUCTION: WHAT IS IT ALL ABOUT? Data is a buzz word in many industries, and indeed advanced data analytics — combining data sources in creative ways, automatic decision making through machine learning, etc. — is at the pinnacle of this applied scientific development that is currently transforming many industries.


n marketing and (online) sales, data analytics is booming. Knowing and serving customers as best as you can, remains the core of any business and being able to use huge amounts of consumer data, find correlations, improve hit rates and the value offered to customers is increasingly becoming a strong source of income.

04 ENERGY Data analytics in the electricity sector

A well-known example is Amazon, which generates almost half of its sales through recommendations it makes to its customers based on data about their buying behaviors and previous purchases as well as purchases from comparable customers. The use of data analytics to reduce costs, improve efficiency and quality is increasing rapidly in operational processes. This ranges from applications in logistics planning and optimization to automation of manufacturing using machine learning.

Sometimes, just the access to data already has the potential to transform a whole industry. The success of Airbnb for example, is primarily based on giving ‘amateur’ tourist accommodation providers access to the market by disclosing information to tourists about available lodgings. Thus, by drastically lowering entry barriers for new entries (private accommodations) to enter the market, Airbnb has disrupted the hotel and tourist accommodation markets. Uber has done the same with UberPOP/UberX, by allowing self-employed taxi drivers entry to the market and generally charging much lower fares than normal taxis1. Data and smart algorithms allow Uber drivers efficient access to clients, driving down prices and disrupting the existing market. At the same time, clients perceive the information they get when the taxi arrives, its fares, etc. as an increase of the quality of service.

Most examples of big data and data analytics are about consumers, and consumers act as the ‘linking pin’ between different sets of data. In the business to business context data analytics already has a history and wide spread use, especially in logistics and operational management. But new analytical methodologies and algorithms, for example machine learning, are having a huge impact in these fields. In fact, most examples in this paper are about applications in business processes. So, it is quite logical that data analytics and the possibilities of ‘Big Data’2 are a major strategic issue in business communities. Data analytics can be a game changer, either by setting costs to new unprecedented levels, by offering new value to customers or by creating new markets or business models that do both. So, how will this impact the electricity industry?3

The potential of data analytics is huge. Keeping to the example of the taxi industry: a study from MIT using Ride-Share Vehicles and data analytics showed that, in the City of New York “3,000 four-passenger cars could serve 98 percent of taxi demand in New York City, with an average wait-time of only 2.7 minutes”. Currently there are nearly 14,000 taxis that perform this task (1).

Data analytics in the electricity sector ENERGY 05

1.1 Data: the oil of the 21st century? Some see data as the new ‘oil’: a valuable ‘raw material’ that will be vital for the survival of companies in the coming decades. They see access to data as a commodity that needs to be secured, not unlike countries that try to secure access to energy and raw materials to fuel their economies. Whether it is justified to view data as the ‘oil’ of the 21 century, i.e. a commodity that needs to be secured, remains to be seen. Contrary to this view, many others argue that sharing data creates even more value. Data is not becoming scarce; it is becoming abundant. The term ‘Big Data’ implies that there is no shortage of data and—unlike oil—data is not being consumed during its use. It can be distributed, shared and copied indefinitely. The saying “To share knowledge is to multiply it” applies equally to data. The value of combining two different datasets might be much greater than each individual set on its own. This applies especially to the ‘traditional’ Big Data (if you already can call it this way). Big Data is often being characterized by having more volume, velocity, variety and veracity than can be processed with traditional means, and is mostly concerned with finding correlations in large quantities of consumer related data. More data means finding more useful correlations and making already known correlations more reliable and specific and thus more valuable. st

However, there is a catch. In a competitive market success is determined by relative advantage. You don’t have to be good; you have to be better than your competitors. Having data your competitors do not have can be more beneficial than sharing this data with them. Sharing data will likely result in a better offer to your customers, but so will the offer of your competitors. Thus, there is a possibly that you will lose your competitive advantage through sharing data.

06 ENERGY Data analytics in the electricity sector

Combining data with the data of only a selected number of competitors will give a competitive advantage over those that do not participate in the exchange4. This can be an effective strategy if this results in an increase in market share of all collaborating participants at the cost of the non-participating companies. For the non-participating competitors, there will be an incentive to follow the same path and start a ‘data consortium’ among themselves to remain competitive. The forces driving cooperation between companies to share data and information thus look like those that are at play when industrial standards are established. However unlike with standardization, presence is not enough in ‘data consortia’; the partner that would like to join most likely must bring its own data, or something else that will benefit the consortium. So, securing data for yourself will likely remain important. Sharing huge amounts of even non-sensitive data does have challenges. As it is not economical to send over huge amounts of data back and forth, this either implies a shared data warehouse with the consortium, which implies a huge commitment and lock in; or it implies allowing the consortium partners access to your data systems with their analytical algorithms, which requires quite some trust in the intent and capability between all consortium partners. Nevertheless, many companies are putting effort into building platforms that will allow sharing data with others5. Will access to data truly function as the new ‘oil’ of the 21st century? Unlike oil, data is not an economically scarce consumable resource. However, (access to) data does represent a huge economic asset that can be used to get access to even larger sets of data. This reasoning applies to all forms of data: from generic data sets about consumer behavior and preferences; to large volumes data about assets.

1.2 Convergence of structured and unstructured data One of the largest drivers behind the Big Data trend is the ability to draw conclusions from unstructured data, like texts, pictures, videos, social media and behavior. This is basically being done by adding structure to this data, for example linking it to each other through correlations, or quantifying it through the weights in neural networks. There are many applications where unstructured data is being analyzed, from self-learning algorithms translating texts form one language into another without a beforehand fixed set of grammar rules and dictionary to the automatic generation of juridical advice and self-driving cars. A huge application is marketing. For the first time in history creating structure in consumer data can be used to create an increasingly complete picture of (potential) customers, customer groups and their preferences. This offers huge benefits compared to the traditional approaches to consumer profiling based on the often infrequent direct contact between thebusiness and its customers. In a world where competition is ever increasing, this customer knowledge is increasingly becoming a vital asset to companies.

Analytics of structured data has a history much longer than that of unstructured data, but also this field is undergoing rapid developments. An example of analysis of structured data is Process Mining (2). Process mining analyses business processes based on actual records generated during the process, to create a ‘picture’ of how the business process functions, including all hick ups, bottlenecks and loops. The results of the analysis are then used to optimize the process and significantly increase efficiency and the quality of its output. In asset-rich environments sensors generate the major portion of structured data. Sensors of all kinds are becoming so inexpensive that they can be used in virtually everything, from sensors in clothes to monitor health and the effectiveness of a sports training, to sensors in flower pots that can be used to administer the optimal amount of water. The amount of data generated by these sensors is huge. While this sensor data may consist of ‘structured’ time series, the context and metadata is often missing or difficult to trace, adding an unstructured element to this data.

Data analytics in the electricity sector ENERGY 07





Too big, too complex, too fast Big data will underpin new waves of productivity, growth and consumer surplus

McKinsey 2011

Secure data access!

Collecting, cleaning, processing, analysing, visualising To extract meaningful value from big data, you need optimal processing power, analytics capabilities and skills IBM 2014

Develop smart tools!

THE RESULT SECURING BENEFITS Insight, optimisation, automated decision-making Make the data work: Leveraging data in the core process to create new services, gain efficiency, reduce costs and improve performance.

Integrate analytics in core processes!

Figure 1 - Big data and data analytics aim to create value. Many applications are generic. Application of data analytics to the core processes requires more specific effort but has a higher potential.

The difference between structured data and unstructured data is disappearing. For example, in asset management, structured and metadata will not only improve knowledge, but will also ‘institutionalize’ practical knowledge6, often considered as the most unstructured knowledge there is and something traditional tools like expert systems have struggled with. This will result in more refined asset management strategies that will take more practical aspects into consideration. The use of real time sensor data in maintenance allows for increasingly ‘leaner and meaner’ maintenance strategies without sacrificing on quality.

08 ENERGY Data analytics in the electricity sector

Many tools and applications of Big Data and data analytics for the analysis of both structured and unstructured data are generic and are applied in different business sectors. They concern generic aims like marketing and sales, procurement or process management that have similar goals for all industries. The challenge that many companies are now facing is how to apply data analytics to their business specific and technical processes, products and services in order to stay competitive. Companies start to implement data analytics to optimize their core processes, increase service levels and/or develop revolutionary new ways of working, new products and new services. This step is challenging because it requires a combination of ICT knowledge, data analytics skills as well as domain knowledge to know where data analytics brings most value and to interpret the results, as illustrated in figure 1.

Data analytics has such a huge impact on businesses that it becomes easy to lose track of the obvious: no matter how much data and how advanced the analytics, the value of data lies in the conclusions arising from the analysis. To reuse the example of Amazon; it is the appropriate recommendation that saves the customer time and reduces the risk of a bad buy that encourages him/her to make another purchase.

1.3 Reading guide This paper is organized into 6 chapters. While this chapter (chapter 1) reflects on high over developments concerning data analytics in general, chapter 2 introduces the most relevant developments within the electricity sector, its stakeholders and the use of data in the power system. Chapter 2 ends with a global process segmentation of the electricity sector that is used to segment the impact of data analytics on different elements of the power industry. These are described in chapters 3, 4 and 5. Chapter 3 highlights some examples of the impact data analytics is having on forecasting and planning in different aspects of the sector: investment decisions and maintenance; renewable generation forecasting and demand forecasting. Chapter 4 analyses the impact on respectively network, wholesale and retail operations, coming together in how these work together in smart energy operations involving demand response. Chapter 5 starts with a short introduction on settlement (e.g. resolving and remunerating of contracts, agreements & billing) in unbundled energy markets and analyses how data analytics might create new markets and how these translate into the retail market with the help of smart meters and advanced energy contracts possibly involving block chain technologies. Chapter 6 eventually summarizes the impact of data analytics on the electricity sector overall and highlights specific aspects for the different stakeholders in the electricity value chain.

Data analytics in the electricity sector ENERGY 09

2 - THE POWER SYSTEM AND ITS CHALLENGES While data analysis always was an important part of econometrics and logistics, the availability of huge amounts of data through the Internet and social media, data analytics - and related associated aspects like machine-learning - is becoming a force that is transforming industries.


o understand the transformational effect of data analytics on an industry, it is important to know the structure, processes and other developments and challenges in that industry. So, a short introduction on the structure, processes and developments of the electricity industry is described. The electricity sector is going through a transition towards a more renewable electricity system, while trying to remain reliable and affordable. This process of the energy transition and the balancing act between renewable, reliable and affordable is referred to as ‘the energy trilemma’7. This transition is also often described by the trends 'decarbonization', 'decentralization' and 'digitalization'.

10 ENERGY Data analytics in the electricity sector

‘Digitalization’ is the main topic of this paper. ‘Decarbonization’ refers to the shift from fossil fuels to renewables. For the power sector, this shift towards renewable generation translates into an increasing amount of variable i.e. ‘non-controllable’ power sources like wind and solar, and by a corresponding shift to smaller and more dispersed power generation (often called DER: Distributed Energy Resources8). This trend is referred to as ‘decentralization’. The power sector, including utilities as well as system and network operators, are used to making huge decisions with investment horizons that span over decades. However, in many regions in the world, the stable environment in which such decisions are possible is disappearing (3) because of the energy transition.

Governmental policies are driving the large-scale implementation of renewables with mixed success9. Developments such as electric mobility might speed up, or stall; the use of residential solar the use of solar panels on residential rooftops is expanding exponentially; market models around flexibility, designed to cope with (local) variable renewable energy sources and that include the network are being designed (4). And because of the increase of power generation by wind and solar, the electricity system itself is becoming more volatile. All these unknowns make long term investment decisions hazardous. So, in many regions in the world, stakeholders in the electricity sector—utilities as well as network and system operators—are forced to make fundamental choices. Traditionally, utilities, at least the ones in Europe, used to used to make their largest profit with their power generation portfolio. Since 2008 that has changed drastically10 and they are forced to rethink their complete business models, acknowledging the fact that the liberalization of generation, combined with the increase of renewable generation result in a lasting shift of profitability from the beginning of the electricity value chain towards the end. The split between the traditional power generation and the retail and grid activities of E.ON into respectively Uniper and E.ON; and of RWE into respectively RWE and Innogy are examples of this on an organization level.

However, at the at the end of the value chain, competition for consumers is fierce and might eventually result in a shakeout, if the utilities fail to diversify and create new value. Data analytics will be a part of this new value, as will be demonstrated in the following chapters. Network and system operators often operate in a strict regulatory framework and have an assigned task to fullfil. However, the dimensions and metrics they use in designing and operating their grid will need to change. This is due to the emergence of distributed generation, less predictable and more correlated loads, and load situations that can change much faster than before, which may be caused by massive installations of solar panels or charging stations of electric vehicles in a neighbourhood. In the following chapters the role of data analytics in changing grid operations is analyzed. In summary, both utilities and grid operators face challenging times as their environment is changing and they are forced to cope with growing uncertainty and complexity, changing policies, regulations, market models and fast technological developments. Because of variable renewable power generation, the system is becoming more volatile and these developments will accelerate the adoption of data analytics in both network operations and emerging energy parties, such as the aggregators and energy service companies.

Data analytics in the electricity sector ENERGY 11


In unbundled energy markets, there is a strict distinction between commercial stakeholders and market facilitating and ‘shared infrastructure’ management parties, such as distribution network operators (DSOs49) and transmission system operators (TSOs). Commercial stakeholders are ‘governed’ by a competing market: competition should ensure their cost efficiency and their customer focus. Grid operators have a natural monopoly. To ensure their cost efficiency and proper execution of their tasks, they are governed by regulation and monitored by external (governmental) agencies, referred to as the ‘Regulators’. Transmission and distribution system/network operators (TSOs and DSOs) have the task to facilitate the market by transporting and distributing the electricity through their grids and by maintaining the integrity of the system through several technical and administrative functions. These include maintenance and grid expansion to provide sufficient capacity; the procurement of ancillary services to keep the system stable; and facilitate market processes that are necessary for the market to function. Examples of these facilitating services include providing metering services and facilitating the allocation and reconciliation of energy use and generation. However, regulating TSOs and DSOs by strictly defining their tasks, services and price levels does have drawbacks. DSOs and TSO are by regulation not allowed to benefit specific market parties—including end—users, they cannot differentiate their offering to specific individual wishes of end-users. All market parties—including end users—should be treated equally. For relative simple needs, that all end-users share, like a grid connection with a fixed capacity for a fixed price, this works well. However sometimes more complex situations arise, like a congestion in the grid that affects only a limited number of users. This strict approach leads to major inefficiencies as the TSO/DSO are not allowed to make use of the different needs and wishes of individual customers and therefore have to fall back to the ‘one size fits’ all approach. By redefining this ‘one size fits all’ service level as the best ‘collective service’ possible to which end-users are entitled but can negotiate about, this paradox can be broken. TSO/DSOs still have an obligation to offer this ‘best collective service’ if end users if requested. However, it becomes negotiable. In case of network capacity problems, a TSO/DSO can offer to compensate end-users in the problem area to reduce (or increase) their load/production through an auction mechanism. End-users that accept, will offer the flexibility while being compensated, while end-users that do not or cannot accept, will not be affected by the capacity problems. This mechanism ensures an optimal allocation of scarce flexibility, not only for the DSO, but also for the TSO and market parties. Within the USEF framework (4) this concept has been extensively developed, including fail safe mechanisms for DSOs; the definition of stakeholders like aggregators that facilitate the interaction with end-users; and definitions of processes and protocols in a reference implementation that has been and is continued to be tested in several demonstration projects among Europe.

12 ENERGY Data analytics in the electricity sector

STAKEHOLDERS IN THE POWER SYSTEM The power system knows many stakeholders and the most important ones are described below. The role of the Transmission System Operator (TSO) is to transport energy from centralized producers to dispersed industrial prosumers and Distribution System Operators over its high-voltage grid. The TSO also operates interconnectors that link to other high-voltage grids in neighbouring regions and countries. The TSO is responsible for keeping the system in balance by deploying regulating capacity, reserve capacity, and incidental emergency capacity (In some countries this task is performed by an Independent System Operator, the ISO). The role of the distribution network operator (DNO) or Distribution System Operator (DSO) is to cost-effective transfer and distribute energy in a given region over the distribution grid to and from end users and for the connections to and from the transmission grid. The difference between a DSO and a DNO is that the DSO is able to perform grid capacity management (i.e. activate demand response and local generation to support the distribution grid). A Balance Responsible Party (BRP) is responsible for actively balancing supply and demand for its portfolio of producers, aggregators, and prosumers. The BRP forecasts its portfolio’s energy

demand and supply and seeks the most economical solution for the energy to be supplied. The BRP can source the requested energy on behalf of the Supplier in two ways: directly, by dispatching power plants with which it has a contractual agreement, or indirectly, by trading on the energy markets. The role of the Supplier is to source, supply, and invoice energy to its customers. The Supplier and its customers agree on commercial terms for the supply and procurement of energy. In this paper often utility is used to describe a supplier that also owns generation. The role of the Aggregator is to accumulate flexibility (controlled changes in power demand or generation) from Prosumers sell it to the BRP, the TSO and/or the DSO. The Aggregator’s goal is to maximize the value of that flexibility, taking into account customer needs, economic optimization, and grid capacity. An Energy Service Company (ESCo) offers auxiliary energy-related services to Prosumers but is not directly active in the electricity value chain. An ESCo may provide for example data related services like insight services and energy management services, but may also offer installation services (like condensing boilers,

Data analytics in the electricity sector ENERGY 13

2.1 The use of data in the power system Knowing what customers value is indispensable in any industry, even for an industry delivering a commodity like power. This holds true for the commercial players in the electricity value chain and even for regulated monopolies like grid operators. Data analytics is a huge enabler for this. Data analytics applied to other generic business areas; such as the redesign and optimization business processes (‘process mining’), logistics, procurement, pricing, risk management, and many more become increasingly relevant to both utilities and grid operators. But, as mentioned in section 1.2, ‘Big Data’ and data analytics will also have specific benefits for specific sectors. For a sector that is going through a fundamental change like the power sector, this is especially true. Given the challenges the power system and its stakeholders are facing, it is not difficult to understand the huge interest from power industry stakeholders in Big Data and data analytics as a source for possible answers, solutions and tools that will help during the transition.

The power system started more than one hundred years ago, and has been expanding ever since. It is possible that there are still some components in the existing power grid that are more than 80 years old. Companies have been merged into large utilities and a lot of data has been lost or confined in legacy systems. This data, so called ‘dark data’ is very difficult and expensive to access. Most new data on the other hand, are sensor data from sensors within the grid and at its edges. For example, Phasor Measurement Units (PMUs) and other sensors, smart metering at end users and even directly from equipment of customers if they allow it (like solar panel inverters, combined heat and power installations, etc.). A complicating factor with this kind of data is that it is often context specific, thus not easily shared or interpreted. It usually consists in the form of time series. To get the most out of this kind of data it is not sufficient to just know what is being measured, but also consider the physics of the measured devices, its location and relation to other devices, its environment, logic, business context like contracts, market prices, etc. Figure 2 shows a schematic diagram of the basic process in the power industry. Data plays an important role in all of them. These processes use data to give information about either the future, the current situation (now) or the past. Processes dealing with the ‘future’ are about predicting what might happen and to plan accordingly; processes about ‘now’ are about operations and control and act on what currently is happening; and processes about the ‘past’ are mainly about settling of what has happened between stakeholders, like for example billing. While it is possible to make this ‘rough’ distinction between processes, it should be noted that this is a simplification and that the impact of data analytics will link these processes closer together. For example, ‘Settlement’ might be based on real time dynamic prices that are influenced by operations. ‘Operations’ might be based on probabilistic calculations of a set of forecasted scenarios and ‘Forecasting and Planning’ will make use of real-time operational data and historical data collected from the system. But nevertheless, it is a very practical distinction that will be used to structure this paper.

14 ENERGY Data analytics in the electricity sector


Now (real time)


TIME ENERGY SYSTEM Regulated: Network and system operators

Non-regulated: Generator, retailer, aggregator


(residential, industrial, producer)



(dispatch, control, support, maintenance, workforce management)

(data allocation, contracts & billing)


FORECASTING AND PLANNING (loads, asset status, contracting, investment decisions)



(products, services)

(information, marketing)


(energy service company, independent service provider)


Figure 2 - Overview of the purpose of the use of data in the power system set in a timeframe: is data used for past actions (settlement); is data used for future actions (forecasting and planning). The 'effect' on other stakeholders (customers and third parties) is taken into account. Customer satisfaction is in the end a result of his expectation and what he/she perceives they're receiving.

Mapping the same approach of future’, now and past regarding services to the customer gives an indication of how these specific processes might help in creating value to the customer (the end users of the electricity system). Carefully managing customer expectations, as well as meeting them when the moment is due, will ultimately will lead to satisfaction and customer retention11 (see figure 2). This paper is about data analytics in the electricity sector and it mostly considers stakeholders in the electricity

value chain: utilities, retailers, aggregators and grid and system operators. It is however important to notice that there are more companies connected to the electricity value chain, like independent service providers and energy service companies. The next section will address the basic processes as mentioned in figure 2:  forecasting and planning  operation and  settlement

Data analytics in the electricity sector ENERGY 15

3 - THE IMPACT OF DATA ANALYTICS ON FORECASTING AND PLANNING One of the main applications of data analytics is forecasting. In the electricity sector the application of forecasting ranges from predicting power flows minutes ahead, to economic developments in the energy market spanning decades; and, from lifetime predictions of large assets, to predictions of behaviors and preferences of consumers.


ata analytics in the power system usually is supplemented by physical models or scenarios that describe (possible) external developments, such as new technologies, changing regulation, growth rates or disruptive events. Figure 3 shows an overview of common processes in the energy industry linked to their associated time scale. A general rule of thumb is that the shorter the time horizon the more automated and integrated the forecasting will be into the process, driven by both necessity and availability of data.

16 ENERGY Data analytics in the electricity sector

This chapter does not have the ambition to give an extensive overview of all developments regarding forecasting in the electricity sector. It does however discuss the impact and opportunities data analytics is having on a selected couple of examples that are indicative for the wider developments. For each example, the role of data analytics regarding the example is described, followed by a discussion of some new developments and opportunities.

3.1 Asset management: investments and maintenance Introduction and current situation In power systems, asset management is usually defined as the activities and practices through which a network operator (TSO and/or DSO12) develops and manages its assets and their associated performance, risks and expenditures over their lifecycle to fulfil the requirements and needs of grid users. Asset management is concerned with policy development, decision-making and managing of grid investments and maintenance. While the focus of this section is mainly on the grid operators, most of it equally applies to other stakeholders in the power system which manage assets with a long lifetime. Network operators, as well as other asset managers, must deal with increasing uncertainties related to regulations, developments in the energy market and the use of electricity. Examples include the changes of grid codes and market design, the emergence of local energy markets, integration of renewables (solar and wind power generation) and storage, and the increase of electric transportation.

A network operator is expected to deal adequately with all types of uncertainties associated with these developments. That is, a network operator must be effective in forecasting and in least-regret decision-making regarding future utilisation of his network. For example, decision-making regarding investments, reinforcements, replacements and depreciations. More data is becoming available due to the extensive monitoring of existing assets and the development of new data collection techniques/technologies13. The tools used to interpret this data are becoming more sophisticated and data-driven. This data is helping in finding new correlations between environmental data and asset utilization, and remaining lifetime as well as forecasting imminent failures. Early experiences are helping to prioritize the deployment of the next sensors. Unfortunately, from a data point of view—assets have a technical lifetime of multiple decades and most of the healthy asset population is still not being monitored. So, in practice most information about assets and asset lifetime comes from investigations of incidental failures.

Data analytics in the electricity sector ENERGY 17

Investments Maintenance Resource planning Dispatch planning Execution Balancing minutes MINUTES

hours HOURS

days DAYS

months MONTHS

years YEARS

decades DECADES

Figure 3 - Typical timing scope for important processes in the power system related to forecasting

This data is supporting the execution of asset maintenance, for which there are many philosophies. Most of these philosophies can be categorized into three basic types, each having their own merits and drawbacks. This is dependent on the assets age and use, the probability and consequence of failure, the availability of data, costs, etc.: Corrective maintenance (for example ‘run to failure’) Preventive maintenance (for example periodic maintenance)  Predictive maintenance (for example maintenance; reliability-centered maintenance)  

The available data and data analytics basically play a role on two levels. First, they allow for selecting the right approach for each individual asset and/or group/cluster of assets. Second, they play a major role in the maintenance methodologies themselves. With the more advanced methodologies, aspects that have predictive qualities about future possible failures are monitored and help decide about preventive maintenance and when (plannable) corrective maintenance measures are required.

18 ENERGY Data analytics in the electricity sector

New developments and opportunities Within the power industry, especially the network operators, there are some hurdles for asset management to become more data driven. One major hurdle is that grid assets have a very long (technical) lifetime, some ranging over 60 years. Most failures are incidents with external causes, e.g. by storms, floods, building and digging activities, although aging and wear caused by increasing loads are beginning to have a larger impact. There is extensive knowledge and information about failure mechanisms and core failure causes available, gained by power failure investigations. However, there is surprisingly little ‘raw’ data about asset failures. The cause of this is the long lifetime of assets and the fact that most of them are not monitored. Especially little data is available of the ‘large healthy population’, which makes the application of many analytical algorithms like ‘rare event’ statistics difficult.

Failures are often difficult to compare. Grid assets may operate under very different circumstances and in very different environments. In combination with the limited amount of failures, it is a challenge to gain enough statistical relevant data to use common ‘Big Data’ techniques to correlate data of asset failures with data about their historical use and environment. While the number of assets being monitored is increasing, eventually solving the blind spot of the ‘healthy population’. However, the number of failures, and thus data about failures, will remain limited. So, while there is no doubt that asset management —especially asset maintenance—will benefit from data analytics, it needs to face two main challenges to be able for data analytics to fulfil the promise many people in the sector believe: 1. How to gain more relevant data about asset failures and especially near ‘misses’—to optimize maintenance (obviously without creating more failures). 2. How to maximize the learning from the data that is available by ‘generalizing’ it somehow, so it can be linked to data from assets operating in different situations under circumstances, as well as generally be applied to all assets considering all different uses and circumstances. The first challenge can be greatly reduced if grid operators would share data about failures—as well as their healthy asset population—to some extent. Section 1.1 ‘outlined that there is a lot of value in sharing data if that gives a competitive advantage. For grid operators (who do not directly compete against each other on the reliability of their grids), there would be great value in sharing information about asset failures. Sharing data with (energy-intensive) industry may help increase both the (healthy) population data and the failure data. Standardization, recommended practices and independent IT-platforms may support such developments. The oil industry recognized decades ago that they were not competing on reliability of their assets, and are now sharing reliability data together in the OREDA organization (Offshore and onshore Reliability Data)14.


SMART CABLE GUARD EQUIPMENT: a cable sensor and a receiver

If you know what the future holds, you can anticipate on what it will bring.

A medium voltage network that is continuously monitored by a fully automated system which immediately gives a warning when problems arise. This is a dream long held by many grid operators. Part of this dream is now realized by Smart Cable Guard®.

This is the first sentence of the DNV GL website on Smart Cable Guard (11). Smart Cable Guard is a monitoring and prediction solution that has been developed over a period of more than 10 years by DNV GL in cooperation with the Technology Foundation of the Dutch government and Dutch grid operators and STW. It is using continuous measurements at both ends of a medium voltage cable to discover patterns in frequency, location, and magnitude of partial discharges within the cable. These patterns are used to predict the remaining life time of the cable, and more precisely can give an indication of where and when a fault will occur in the cable. Smart Cable Guard is an example that shows that the application of data analytics can be very specific and that applying data analytics has to be one with care. For each type of cable, Smart Cable Guard requires specific algorithms tweaked to that specific type of cable. For example, in XLPE cables the interval between the first sign of partial discharge activity and the average failure is 2 months, while in old PILC cables this is 16 years, because of its self-healing properties. Using the advanced pattern recognition algorithms of Smart Cable Guard, the precision for forecasting a failure can be increased to respectively 10 days for XLPE cables and 3 years for PILC cables (15). Another additional feature of the measurements is that the location of a fault can pinpointed at the moment it occurs. So, if a cable is being cut by for example digging activities, it is immediately known exactly where the fault has occurred and thus how to isolate it to save precious customer minutes lost.

Data analytics in the electricity sector ENERGY 19

Impact of asset data (its condition, health, use and environment) is translated into probabilities and eventually into risks to the integrity of the system.














Figure 4 - A risk-based asset management decision process (5) and the application of (historical) failure data by factorizing this into core probabilities and risks

The second challenge can only be addressed by recognizing that the availability of (failure) data will remain an issue and thus a different approach is required than with normal big data applications, that presume an ‘abundance’ of data. The required methodologies need to make optimal use of the relative small amount that is available. For example, as assets will be more and more monitored, new tools, such as ‘rare event’ statistics, can be applied, giving better predictions of the asset lifetime. Increased monitoring also means that failure data and data about the circumstances and environment before and during the failure, will become available. However, for this to become truly valuable, this needs to be generalized to be applicable to the whole asset population, i.e. assets operating under different circumstances (see figure 5 for an example).

20 ENERGY Data analytics in the electricity sector

Although data analytics promises to find predictive structures in the data itself, combining it with fundamental knowledge about the system has huge benefits. This “knowledge analytics” can be represented by a Bayesian network, which is essentially a complete representation of the interconnections between different features of a system that lead to performance failures. The interconnections in a Bayesian network are represented by conditional probabilities (e.g., the probability of an interconnect failure given the probabilities of encapsulation, metal frame, and glass failures), which are derived from data, experience, and models. Using Bayesian networks in this way offers several advantages: they can be used to perform sensitivity analyses, thus providing leading indicators of failures; they can be used to conduct dynamic risk management using sensor inputs.

With the help of Bayesian networks, raw data from among others power failure investigations, as well as other sources, like sensors, can be unraveled or factorized into root causes and root mechanisms that impact the probability of failures. These factors can then be compared with factors of other failures and recombined and used to predict the probability of failures of other assets16, even assets in different circumstances. The advantage of this methodology is that these ‘root’ factors (i.e. core aspects affecting failure) can be updated with new specific data as it becomes available (using for example Bayesian statistics) and thus an ever increasing ‘knowledge base’ is created.


In addition, the opportunity to combine data from multiple sources (e.g. multiple cooperating grid operators from all over the world), stakeholders multiplying knowledge on their existing assets might prove invaluable to speed up the learning experience of new, relatively unfamiliar asset types, like DC equipment and batteries, making their application in the future much more likely to be successful.

Temperature Vapor phase salt

Material susceptibility

Impact in transportation

Atmospheric corrosion Galvanic coupling to steel


Manufacturing infant defect

Impact in installation

Mechanical force Metal frame failure Glass failure

Encapsulant failure Interconnect failure

Figure 5 - Example of a Bayesian network for the failure mechanisms of solar panels, able to combine data with fundamental knowledge of the system15. 'Backpropagation' techniques might be used to reverse experience data to 'root causes'. These kinds of 'empirical' models can also be used to establish links between data of assets operating in completely different circumstances.

Data analytics in the electricity sector ENERGY 21

22 ENERGY Data analytics in the electricity sector

3.2 Renewable generation: resource planning and dispatch Introduction and current situation Probably the most forecasted system is the weather. Weather prediction has a history as old as humanity itself. In the electricity sector, extensive use is being made of weather forecasts, historically mostly for load forecasting, but forecasting the generation of weather dependent renewable energy is becoming more and more crucial. Generally, weather forecasts from meteorological institutes are used as the basis for the forecasting of renewable energy as well as demands. It is not difficult to notice how many ways of forecasting the weather there are; from detailed physical modelling to empirical methods and looking at trends. Nowadays practically all of them use a lot of different sources of digitalized data, such as pressure, wind speeds, temperatures, data from satellites etc. Numerical Weather Prediction (NWP) is a container for many computer aided predictions based on physical modelling, parameterization, statistics etc. An interesting observation concerning NWP is that the best results are achieved when using models with many slightly different initial states, known as ‘ensemble forecasting’, and even using different models, such as ‘super ensemble forecasting’. Power generation from wind and solar is obviously highly dependent on the weather. Data from specialized weather forecasts, (for example for a wind farm these could be wind speeds, wind speed variations, wind direction, pressures, temperatures, etc.) is combined with characteristics of the specific wind farm, like historical performance, reactiveness to wind variations (ramping), probability of icing dependent on wind and temperatures. All this data and models are combined to form a generation profile for the next period (which could be 15 minutes ahead to a week).

New developments and opportunities Weather forecasts will continue to become more accurate as the amount of input data computing power is increasing. More detailed data is becoming available. Examples of this are the Sentinel Satellites from the EU’s Copernicus program; more local sensors and weather stations, allowing much higher detail of the f orecasting models (from 5 km to 2 km resolution); high quality imagery of cloud coverage and waves between offshore wind turbines, giving information about wakes behind the turbines, etc. However, being a chaotic system, each slight improvement of forecasting will require an exponentially increasing amount of data and computing power and large jumps in the increase of accuracy of weather predictions are not expected, not even through the application of new methodologies or forecasting techniques like deep learning. Though especially for predicting ‘rare’ events, like storms or floods that might put the electricity system at risks, these ‘slight’ improvements still might prove to be quite significant. The use of weather predictions to calculate the power output profile of a wind turbine or solar panel, and especially a whole wind or solar farm is a much younger science, which is increasingly becoming more critical for wind farm operations in recent years. Wind (and solar) energy is affecting the power system and power markets, and operations are shifting from maximizing energy yield to integrating the farm into the electricity system and market17. The need to translate weather forecasts to power output is fast developing and is gaining importance. Some examples are the re-evaluation and adjustments of assumptions made in the first generations of models; the development of advanced models, able to accurately model turbulence and wakes and thus the influence of wind turbines on each other; and models and methodologies to estimate the effect of clouds and cloud movement on the power output of solar farms.

Data analytics in the electricity sector ENERGY 23

3.3 Demand forecasting

day approach’ based on historical data, prove to be surprisingly accurate for a large number of consumers.

Introduction and current situation As with renewable generation, electricity demand is highly influenced by the weather, especially, heating and cooling. Variations in demand used to be the main source of uncertainty in the electricity system, thus forecasting load always has been an important topic in electricity systems. Both short term forecasting, necessary for contracting and planning of dispatch of generation; as well as long term demand, influencing investment decisions in power generation and transmission and distribution networks.

These methods take a ‘top down’ approach (see figure 6), using (well-known) patterns in overall loads as a basis for forecast. These methods prove to be powerful. The demand of even a few hundreds of households follows a regular pattern, and on a scale of millions of households, industry and commercial buildings, the demand is regular and shows a well-known pattern that is easy to forecast using the prementioned methods. Size (i.e. the number of customers) is an important tool to manage risks for balance responsible parties, and thus a source of competitive advantage.

Long term forecasting in general relies on scenario planning, based on general and local economic developments, correlated with data from industry, commercial and residential demand and possible technological developments like for example energy savings, the use of electric vehicles and solar energy.

However, demand is becoming more correlated due to the ongoing electrification of heat (heat pumps, air-conditioning) and transport (electric vehicles). Local (solar) generation is also increasingly occurring in distribution grids, so the statistical averaging out will become less in the future. More relevant, because expanding the network capacity to cope with this synchronization of demand and generation in the capillaries of the network is very expensive, these capacity limitations will lead to the necessity for local ‘balancing’ (i.e. congestion management), and thus for local demand forecasts consisting of much smaller numbers of end users.

Short term load forecasting depends on statistical learning algorithms, regression models, neural networks and other pattern recognition methods18. Because aggregated demand has a very periodical (i.e. day/ week/season) and stable behavior, even relative simple methods that exploit this behavior, like the ‘similar




Top system level


Figure 6 - Traditional demand forecasting in power systems relies on known patterns overall (i.e. aggregated) loads. 'Big data' - as applied in for example marketing - is trying to create insights in unknown patterns by aggregating data from users. As local forecasting becomes more important because of infrastructure constraints, this will lead to an increased application of 'Big data' techniques in demand forecasting.

24 ENERGY Data analytics in the electricity sector

New developments and opportunities An alternative approach to top down forecasting considers the use and demand of individual appliances, correlated with other demand, events and local weather forecasts, before being aggregated into a forecast of (part of) the demand. The main advantage of this so called ‘end-use method’ is that patterns and correlations can be found on appliance level. Therefore, forecasts can be made much more local and flexible, for example considering local weather variations, events and scenarios. As more demand data becomes available for analysis, due to the presence of smart meters and the metering of individual devices among others, this approach is becoming more accurate and therefore feasible. The ‘top down’ methodologies in load forecasting will remain dominant for a long time. However, because the aforementioned increase in correlation in electric demand and importance of local forecasting of smaller numbers of end-users, it is likely that they will be gradually being complemented by bottom up approaches, like the end-use method and ‘big data’ approaches; where correlations in the use of appliances and external factors are identified by combining a lot of data from individual users, building up to a picture of the total system (see figure 6). The combination of such a top down and a bottom up approach is much more flexible than the traditional top down approach alone, because it can handle forecasting smaller number of end users and can consider changing local circumstances like the local weather, events and local trends, while still retaining the accuracy in forecasting large number of end users

3.4 Summary for forecasting and planning Forecasting in power systems is applied to a wide field of issues and different time horizons. This chapter touched upon a few applications: Long term forecasting necessary for investment decisions and maintenance strategies (often asset management); forecasting the output of wind and solar farms; and demand forecasting. In practice, combining datasets still turns out to be hard, due to different regional conventions, different time scales and differences in meta information. Nevertheless, in each of the examples raised, data becomes increasingly available as data from different sources are disclosed. Also, more sensors from ranging from usage in network equipment, to weather sensors in the field to smart meters measuring small scale demand are deployed. All these added together consequently i mprove the integrity of data. While the data analytic tools that are applied in each case are comparable, the developments in each area are facing different challenges. Where asset management faces increasing amount of data, only a very limited number of incidents happen. Data analytics is like searching for a needle in a hay stack. With the exception of predicting extreme weather conditions, forecasting renewables and demand is a more 'continuous' process. Demand forecasting potentially will face increasingly synchronized demand from charging electric vehicles and heat pumps, including possible ‘feedback loops’ caused by demand response reacting on energy prices and local network constrains, leading to a relative increase of bottom up modelling of demand at the expense of the traditional top down statistical approach.

Data analytics in the electricity sector ENERGY 25

4 - THE IMPACT OF DATA ANALYTICS ON OPERATIONS The area that probably will be influenced most by data analytics is 'Operations'. Historically analytics of operational processes was manual work and thus labour intensive and expensive. This meant that it was predominantly a tool for analysts in staff departments to underpin strategic decisions or to design more efficient operational processes.


s developments go on, data gathering becomes more automated, continuous and 'complete'; methods and tools become more sophisticated and standardized (e.g. Six Sigma) and more and more processes are optimized. In many (automated) manufacturing facilities a lot of data is being gathered continuously within the process. This data is periodically analysed by experts and used to redesign or ‘tweak’ the operational processes to optimize it, e.g. by insulating the process from disturbances.

26 ENERGY Data analytics in the electricity sector

The result of these exercises is an efficient and optimized, but rather ‘frozen’ process that does not respond well to changes in the process’ environment and ‘non-filtered’ disturbances. A developing trend in manufacturing (e.g. in the semi-conductor industry or the solar panel manufacturing industry) is that not only the data gathering is automated and continuous, but also the analytics. By continuously analysing the data and adjusting the processes accordingly, the analytics and optimization becomes part of the system itself and able to adopt to changing circumstances.

At the same time, more sensors in the system and new ‘smart’ control algorithms and better and faster simulation techniques allow sub-processes (and components) to become more ‘aware’ of what is happening outside their direct environment, and take this into account in their control and optimization actions. So, on the one side data analytics will allow for automating parts of the (previously manual) ‘top down’ optimization and tweaking of processes and systems. This becomes an integral part of the system or process itself, as is shown in figure 7. On the other side process control (e.g. security of components in grids) is continuously expanding its scope and able to take more and more information into account. The result of both these developments will be the automation of operational decision making and in processes that will become more robust, flexible and adaptive to external influences. Increasing efficiency does no longer automatically come at the expense of loss of flexibility.

4.1 Operations in the power system The developments towards more autonomous operational decision making, as described in the previous introduction, makes sense only if there are continuously changing circumstances to which the operations need to adapt to, as is the case in the power sector. Among the largest challenges the power system is facing, is the incorporation of more renewable energy sources, while remaining dependable and affordable. While this will include controllable renewable energy sources like biomass and geothermal, the bulk of the energy will be provided only when there is wind and sunshine. Wind and solar are energy sources that are geographically distributed and vary with changing weather conditions. At the same time, the electricity demand is changing, mostly due to the electrification of transport and heating. This means that the variables in the power system, i.e. the size and direction of power flows, the relative balance, and the power quality will change continuously. Also, the long-term developments in both demand and generation are much more uncertain than they used to be.

Data analytics in the electricity sector ENERGY 27



(e.g. business process redesign, Six Sigma etc.) Process improvements, tweaks to reduce disturbances

Continuous analytics, adaptive control, situational awareness to continuously optimize the process or system







Subprocess or component

Subprocess or component

Subprocess or component

Subprocess or component

Subprocess or component

Subprocess or component



Figure 7 - A significant part of the 'manual' process analytics will be automated and become part of the process itself. At the same time component/step control will become more aware of its environment and other components in the system (situational awareness).

The traditional solution to cope with such uncertainty is to increase the network capacity and/or the generation capacity. However, this solution is becoming increasingly expensive for the described uncertainty, because of two reasons:  Variable

renewable generation (i.e. wind and solar) is highly correlated. New electric demand (electric vehicle charging as well as heat pumps) is also potentially highly correlated. To accommodate peaks caused by this correlated demand and this (local) generation would require an enormous increase in grid transmission and distribution capacity, which would have a low degree of utilization.  As the future becomes less predictable, the risks that traditional large and long term investments in both generation as transmission capacity become ‘stranded assets’ are increasing drastically, leading to a relative higher attractiveness of more flexible and short term solutions.19

28 ENERGY Data analytics in the electricity sector

The power system can therefore benefit greatly from the integration of data analytics supported optimization into operational processes, as described in the introduction of this chapter. However, it can only do so if certain conditions are met. The most important condition is that there should be enough (near) real-time data from sensors in the system to assess the situation and status. Second, there should be enough possibilities or ‘buttons to press’ to adapt the system to cope with the changing environment, i.e. ‘actors. Finally, the right models and ‘meta-models’ should be available to respectively give the gathered sensor data a context and relevance; issue the right command to the ‘actors’; and adapt the models themselves to learn and keep the system optimal under changing circumstances. All these conditions are beginning to be satisfied. The number of sensors in the grid is rapidly increasing. Today, smart meters for example can act as the receptors on the outer edges of power systems, and an increasing number of smart devices will do the same. Within the grid, Phasor Measuring Units (PMUs) are installed at strategic locations, measuring all aspects of the power flow.

At the same time ‘actors’—the tools to influence and optimize the grid and power system—are being developed. Automated demand response (DR) is gaining renewed interest, for example the possibility for smart charging electric vehicles. Load Tap Changers (LTC), smart transformers and the control of the reactive power through inverters (of e.g. solar panels) can make the grid much more bi-directional. Concepts like self-healing grids are being developed. For example, by automatically reconfiguring a (meshed or ring) grid as a response to very fast fault passage indicators, the effects of a failure in the grid can be reduced to a minimum, or even avoided. Grid maintenance, as well as the maintenance of power plants, is more and more planned and executed based on the outcome and analytics of operational and other data (such as market data). So, both the development and deployment of sensorsin the power system as the development of ‘actors’ is ongoing. To interpret and relate sensor measurements and other data from many different sources and locations, to draw conclusions from them and eventually translate these into appropriate ‘actions’, models are inevitable. For power transmission and distribution systems, ‘state estimation’20 and load flow models come to mind, but depending on the application also other kinds of models will be used. Utilities and aggregators are using models relating weather data, data about loads, measured grid frequency and other sources to predict imbalance and intraday market prices and optimize their generation and load portfolio accordingly, e.g. through demand response. Many other examples of models are available, especially for specialized situations or local decision making21. While these models themselves often are based on physics, they need to be adjusted and tweaked to adapt to constantly changing (often unmeasured) parameters. This process relies heavily on automated data analytics and machine learning and assures that the automated decision making is adapting constantly to the changing circumstances. Examples are the constantly changing portfolios of retailers, aggregators and trading companies; the changing load and/or generation at grid connections because consumers install solar panels, a heat pump or an electric vehicle. The models used for automated decision making will adapt to this. They will ‘learn’ and adapt to these changing circumstances.


In (12) some practical applications of data analysis in distribution grids are analysed. One typical example considers the detection of “non-technical losses” (NTL), in particular electricity theft by tampering with or bypassing the meters. This includes use of electricity for illegal drugs cultivation. Typical indirect methods use periodic behavior of currents (aligned with growing patterns), using data mining techniques. Ideally the exact location of the perpetrators is identified. The newly proposed methods use well established load flow analysis to estimate “normal conditions” in terms of active and reactive power and voltage and expected (technical) losses. This can be done per distribution substation and per feeder, based on smart meter data. Once deviations in the voltage profiles exceed a certain threshold localisation is done. Note that even the smart meters do not identify the power that is withdrawn illegally (by definition) but they still measure voltages. Petr Kadurek based his method on smart meter data, in particular voltage data. Newer methods may have to rely on or include PQ-data (Power Quality), since illegal growing tends to use LED lighting, using much less power (so smaller power deficits and smaller voltage drops), but added harmonics due to the electronics, THD (Total Harmonic Distortion) may be used as a base for analysis, comparing simulations and measurements.

Data analytics in the electricity sector ENERGY 29

There are already many different techniques that have been developed and are now available. For example, the Bayesian method to ‘tune’ model parameters and evolutionary algorithms for various outcomes, such as minimizing risks by optimizing over possible hazardous scenarios, neural nets for identifying and recognizing patterns and anticipating accordingly, etc. These ‘meta- models’ sometimes will be a separate model or algorithm ‘wrapped’ as a separate layer around the system model, and sometimes will be an integrated part of the model of the—to be managed—system itself. In the next three sections examples of the possible impact of data analytics on grid operations, commercial wholesale operations and retail operations are given respectively.

4.2 The impact of analytics on grid operations For both utilities as grid operators, the ability to optimize and fine-tune their operations is ever increasing as more buttons and dials become available. However, as they control increasingly detailed aspects of the power system, they increasingly require more and different kinds of data and more advanced models and data analytics to do so. So, to fully benefit from data analytics, it is inevitable that the analytical processes, as well as their eventual results—i.e. the decisions and actions—are automated. A typical example is the ELVIS information system of FinGrid, applying among others, IBM’s Dr. Watson, to enhance many grid related applications22, in operations as well as in asset management. DSOs need to automate data analytics for optimizing operations to a much larger degree than utilities or TSOs. This is due to the size and numbers of the issues they might face. Instead of facing a few ‘global’ problems, DSOs face multiple relatively small local problems. For example, capacity problems due to the intermittence of solar power generation or the sudden charging of electrical vehicles will likely appear in many parts of the grid simultaneously. Each of these individual problems will be too small to justify manual decision making. So, what kinds of operational decisions can be made? Grid automation includes tap changers and smart transformers, managing voltage levels in the grid so power can flow ‘two ways’. It includes automated grid

30 ENERGY Data analytics in the electricity sector

reconfiguration of loops to reduce both grid losses as well as reduce the effects of potential outages and the effects of outages themselves, in the context/case of self-healing grids. It might include activating automated demand response and local storage, that will help DSOs to reduce peak loads to avoid grid losses and capacity problems. Also, the management of the maintenance work force will be optimized using more data and smart analytics. The operations department within the DSO will get a more strategic role as grid automation and the use of the flexibility of customers’ assets will gain importance and prove to be a viable alternative to investments in excess capacity to accommodate the increasing amount of variable renewable generation and demand. This will be achieved by incorporating more automated analytics and decision support, both on a local and on a global level, to utilize the potential of flexible demand of electric vehicles, heat pumps, building climate installations and some industrial processes to balance against the variations in the power generation of renewables, reduce resistive losses and implement c oncepts like self-healing grids. A schematic of how this might work is shown in figure 8. Through ‘faster-than-real-time’ simulations e.g. based on load flow simulations and state estimation it will be

possible to incorporate scenarios and risk in operations23, gaining large operational benefits both in cost reductions as in quality or risk reduction (e.g. less customer minutes lost). Basically, operations can be optimized using several forecast scenarios and potential risks, including risks associated to using ICT itself. The simulation model(s) itself can be very fast, and do not need to be very accurate or complicated. This may be due to several factors. For example, data from strategically placed measurements will be used to continuously recalibrate them, and they can draw from previously compiled more detailed simulations. These models will be able to prevent grid instabilities that otherwise might emerge because of unexpected interference of behavior of smart devices24. This behavior is simulated by 'digital twins', digital models that are compiled beforehand and then tested and certified so they can be trusted to react the same as as their real counterparts and thus can be used to study and examine their behavior on the grid and on other digital twins. They can be published for use by other grid operators, for example using blockchain technology to ensure their integrity. For other stakeholders in the power system, data analytics and automated decision making can play a similar role in operations. Automation already plays a major part in power trading (similar as in stock market trading) and in control of power plants, and is likely to expand further.


Activities to reduce risks

GRID OPERATIONS (DMS, switching, disturbances, events, workforce development)

DATA ACQUISITION (measurements)

Activities to optimize operations



OTHER PROCESSES (strategy, asset management investments)


Figure 8 - Risk-based grid operations: a schematic of how faster than real time simulations, continuously calibrated with measurements and data from the grid can be used to continuously assess and minimize risk and optimize the grid configuration, work force deployment, etc.

Data analytics in the electricity sector ENERGY 31

4.3 The impact of analytics on commercial wholesale operations Like network operation, commercial operations will be highly affected by data analytics, even more than it already is. Commercial operations include (renewable) power generation, trading, electricity supply and aggregation. The commercial domain in the electricity sector can roughly be divided into a wholesale part, where energy parties interact with each other, and a retail part, where energy is sold to (or bought from) essentially non-energy companies and citizens. While this division is blurring as companies and citizens are getting more involved in ‘traditional’ wholesale activities, such as power generation; and changing business models of retailers and aggregators are integrating wholesale and retail processes more and more, this division is still quite insightful. As discussed in chapter 3, data analytics play a major part in forecasting renewable generation and demand, as well as in market price forecasting. As forecasting is being developed to become increasingly sophisticated, it results in temporary advantages to the companies who forecast best. However, markets and especially market prices, are dependent on the actions of other stakeholders and their forecasts. So, while market forecasting is tremendously important to the relative competitive advantage of the participants, it is an arms race that does not really change the system overall. In operations, as in other domains, it is the actions that count. More data, and especially more real time data will give traders, retailers and aggregators a better insight in their current and future market positions and risks, leading to a need to develop the tools to swiftly act upon these insights. To optimize their market positions, they will need to adjust their portfolio of generation, demand and contracts, by trading among each other. This will lead to new tradable products and more differentiated markets. Just like the increase in solar energy will likely lead to a further differentiation of tradable futures beyond the current ‘base’, ‘peak’ and ‘16h-peak’ products, more (real time) data will lead to a differentiation short term tradable products (and possibly new market places) to utilize the results of these analytics.

32 ENERGY Data analytics in the electricity sector

An example of such a product might be a kind of ‘reinsurance’ of imbalance risks that might appear around the balancing and program responsibility mechanisms. Traders trade energy in time blocks called ‘Imbalance Settlement Period’ (ISP, in most countries this is 15 minutes). If the physical demand and generation in an ISP does not match, they must compensate the TSO for solving the created problem. To solve these imbalances the TSO has contracted power generation, and in some countries, the TSO has organized an ‘automatic semi-real-time auction’ (the imbalance market) where market parties can offer power to the TSO to restore the balance. The cost the TSO incurs (and forwards to the responsible party that caused the imbalance) is only known after the ISP is closed. For many smaller parties25 there are benefits to lock this price in a trade with another party (which can more easily bear or hedge this risk, e.g. because of aggregation or available flexibility) for a price before or even during an ISP. Numerous other possible new products and markets will emerge as a response to data analytics and data exchange. For example, around Power Purchase Agreements (PPAs) between renewable generators, storage and industry (6), especially industry able to response to fluctuations in renewable output: Industry would secure renewable energy, while the renewable generator would benefit from being shielded from the fluctuations of the power market by synchronous generation of renewable energy. Other examples might be around differentiation of energy by source, location and/or even application; or around products where risks of intermittency of renewables are shared and/or mitigated through ‘private’ capacity products; or—as will be discussed in section 5.4 'Advanced energy contracts'— integrate the value chain by utilizing new retail products based on demand response and local storage owned by end-users.

4.4 The impact of analytics on retail operations In the electricity sector retail is generally reserved to indicate electricity sales to commercial and residential end-users, and often also includes energy services and energy data services to end-users. Although through developments such as local generation (the emergence of prosumers26) and demand response (end-users offering flexibility to the electricity sector), the distinction between retail and wholesale is becoming blurred. The impact of data analytics on retail operations will be much more visible to the public than the impact of data analytics on wholesale and grid operations. Like in all other consumer sectors, data analytics and big data allow retailers and aggregators to much better tweak their propositions to individual consumers, creating higher marketing efficiencies and sales,27 and eventually leading to smart energy contracts, that respect the individual wishes and requirements of consumers instead of the ‘one size fits all’ energy contracts that are dominant today. On appliance level, smart thermostats exist already for a number of years, monitoring the behavior of residents in a building and optimize heating accordingly28. ‘Boxes’ (physical or virtual cloud-based) that analyse smart meter data go way beyond merely showing energy use and are able to disaggregate smart meter data into the load of individual appliances using self-learning (non-intrusive) load monitoring algorithms29. The change in retail operations that will likely have the biggest impact on the other parts of the electricity value chain, will likely come from retailers and aggregators aggregating flexibility from end users, using demand response. They might use this flexibility to optimize trading on electricity markets; sell it to grid operators as emergency power, or to solve congestions - adopting the concept of flexibility as a new commodity. Alternatively, they might use it to tune their client's individual energy use to their specific wishes, thus using flexibility as diversification opportunity in the retail market. In section 4.5 ‘Aggregating demand response in smart energy systems’ and 5.4 ‘Advanced energy contracts’, this is further discussed.

Eventually consumers might become a digital Gladstone Gander, the lucky bird from the Donald Duck cartoons. ‘Gladstone Gander services’ imply that eventually your ‘digital’ life will become that of Gladstone Gander. Artificial Intelligence modelled around you, will ‘tweak’ cyber space around you so it seems everything will work in your favour. All information (like product and service propositions) will be fed to you exactly when you subconsciously want it. The more information is known about you the better and more easy your (digital) life will become (at a price).

Data analytics in the electricity sector ENERGY 33

4.5 Aggregating demand response in smart energy systems In the previous sections, a number of subjects have been addressed that often are associated with ‘smart grids’, ‘smart markets’ or ‘smart energy systems’, including automated demand response. The previous paragraphs did not address an important aspect of demand response (DR) , which is what demand response involves—devices and processes of—end users that have their own requirements and conditions. They likely will demand compensation for their services to the system, even if these conditions are met. The exchange of these DR services are transactions and smart grid concepts that take this into account are often referred to as ‘transactional’ smart grids). However, these requirements and conditions are not fixed and will depend on the end users’ continuously changing needs and circumstances. Demand response is the most economically feasible solution to provide flexibility to accommodate higher penetrations of renewable variable energy from the wind and the sun30. It is in many ways already an established business. For decades, day and night tariffs have existed in many countries. Direct control of both devices and processes, like air conditioners and aluminum smelters, has been done for a long time. More recently, specialized independent aggregators are emerging, acting as ‘brokers’ between (industrial) end users and TSOs for ancillary services like emergency power31.

34 ENERGY Data analytics in the electricity sector

While DR has a huge potential, for it to be responsive enough to add value to the continuously changing power system, demand response will need a much higher degree of automation and automated decision making than what is commonly commercially available now. This needs to be on all levels – from the individual device level, to the decision to dispatch hundreds of megawatts by changing the charging behavior of electric vehicles. Behavior. Therefore, data analytics will play a major part in the future of demand response. As mentioned in the introduction of this section, the involvement of multiple stakeholders makes demand response complicated. Flexibility is a resource owned by the customer that must be bought (back) from the customer32. This is such a complicated and specialized task that often this is tasked to a dedicated role: the role of aggregator. An aggregator aggregates demand response and other sources of flexibility to sell to other stakeholders in the power system and to make sure that all possible obligations with these stakeholders are fulfilled (such as balance responsibility and capacity constrains). Aggregators that focus on aggregating many small end-users need to be able to monitor and control hundreds of thousands to millions of devices, considering the specific circumstances of each individual device in (near) real time. This requires an enormous amount of very fast data analytics and communication. However, depending on the design of such an aggregator system, a lot of this data analytics can be done by the devices themselves (a phenomenon called ‘edge computing’).

A demand response system of an aggregator must address two issues: 

Control: How to control the demand (or local generation) of all the customers Value allocation: How to reward these preferably in relation to merit

These two aspects of demand response are highly intertwined as the control mechanism must control demand response in such a way that most value is created, meaning customers and devices that are least disadvantaged (i.e. require least compensation) will be dispatched first. Historically the emphasis of demand response was either on control (e.g. direct control) or on the value allocation (e.g. day/night tariffs), leading to respectively dispatchable and non-dispatchable demand response. Dispatchable demand response, sometimes called explicit demand response, focuses on the control system, often simplifying the value allocation system to a fixed fee to the participating customers. It is difficult to adjust dispatchable DR programs to changing desires and circumstances of customers without involving a more advanced value allocation system. A customer that does not mind the air-conditioning to be switched off one day, might find it uncomfortable the next.

Dispatchable DR thus can feel quite intrusive to customers, although many adaptions have been made to soften this. This is especially true for the United States where dispatchable DR is much more common than in most parts of the world. Examples range from remotely controlled air-conditioning to load shedding contracts with large aluminium smelters. In the Unites States experience shows that the reliability of dispatchable demand response is as reliable as controlled generation. Non-dispatchable demand response, sometimes referred to as implicit demand response, focuses on the value allocation system, often relying on the behavior of customers to adjust their demand in response to price incentives. This makes non-dispatchable demand response programs not reliable enough to rely on for grid safety and stability, limiting its use and value to a ‘support’ role. Examples are day/night tariffs and critical peak pricing. However, this division between dispatchable and non-dispatchable DR is disappearing. Dispatchable and non-dispatchable DR, where respectively a control signal or a price signal is communicated are essentially 'one-way communication' concepts. Two-way communications in combination with data analytics and local intelligence is changing this rapidly.

Data analytics in the electricity sector ENERGY 35


EXAMPLE: Dispatchable demand response Load (e.g. electric boilers) is scheduled to the - for the DSO, TSO or BRP - optimal time

Power (kW)

ADVANTAGES  Technically relatively easy to implement  Direct control of loads DISADVANTAGES to evaluate the value for the power system and the individual customer  Difficult to apply to more than one use of flexibility  Difficult Time

EXAMPLE: Non-dispatchable demand response Time of use pricing ADVANTAGES  Relatively straightforward  Lots of experience with day/night tariffs  Customers are free to react on incentive

(Value to suupplier)

Price (€)


DISADVANTAGES direct control of loads (uncertainty of the response and ability to 'fine tune')  Possible instability problems if automation is applied on a large scale  No


Value (€)

Result of bidding of supply and demand


EXAMPLE: An integrated approach Market-based coordination (day-ahead markets, for small scale: PowerMatcher, Intelligator)

Supply Price Demand

Power (kW)

ADVANTAGES optimization' ane enumeration possible between stakeholders

 'Multi-goal

DISADVANTAGES anticipation to (global) changes in the system should be implemented locally (through bids), making local intelligence more crucial

 Timing:

Figure 9 - Examples of demand response concepts: each of them communicates on two of three possible axes: power, value and time. Each concept is missing one of these axes, which is the root of its disadvantages. Dispatchable demand response misses value; non-dispatchable demand response misses power and a market approach misses time, resulting in planning challenges.

36 ENERGY Data analytics in the electricity sector

Traditional dispatchable demand response controls power directly: For each point in time the amount of controllable power is controlled by the central control system (as shown in the first graph in figure 9). Traditional non-dispatchable demand response sets the value of power directly (for example hourly prices, shown in the second graph in figure 9). One solution for the integration of dispatchable and non-dispatchable demand response lies in recognizing that combining them results in a mechanism very much resembling a market (see third graph in figure 9). Because the wishes and requirements of individual consumers can be implemented in the demand response scheme, retailers or aggregators will be able to offer much more tailored energy contracts to their customers. They can offer services that optimize the electricity use of a customer to make greatest use of the solar energy from his neighbourhood, while offering his neighbour a service that minimizes his bill (see also chapter 5). At the same time the remaining flexibility can be used to offer services to the grid operator to avoid congestions and to optimize the aggregator’s position on the markets. An example of a pilot project where these propositions were implemented is discussed on page 42 (PowerMatchingCity).

4.6 Summary for operations In general, the impact of data analytics on operations can be characterized by leaving behind the old ‘paradigm’ that a choice needs to be made between efficiency and flexibility, or at least bring this paradigm to a next level. Periodic analyses made by staff departments to optimize processes are replaced by machine learning algorithms, using online data that continuously will tweak and optimize the processes. For the power system, which used to handle variations in demand and disturbances with capacity and redundancy, this change comes just in time. As the variations in the system will become dominated by renewable generation as well as demand with very synchronous behavior, this traditional response of increasing capacity becomes increasingly more expensive, both in regard to variable loads caused by variable renewables and synchronous demand as with regard to the risk of stranded assets, that turn out to be obsolete before their anticipated end of life, due to the fast changing (economic or regulatory) environment.

For most parts of the power system the necessary prerequisites are available or soon will be:  Sensors

to sense the ‘state’ of the system, environment and relevant conditions  Actors that affect the system and can optimize operation  Models to translate sensor data into the system and derive the appropriate actions  Meta models to adjust and the models to capture higher order changes (machine learning techniques) For network operations, this will result in a much deeper control at the edges of the network, incorporating demand response and control of decentral generation. Influencing demand and local generation means that not only the grid itself is considered, but also the specific circumstances of end-users’ processes and energy markets (either directly or through intermediaries like aggregators). This requires very fast models utilizing for example precompiled simulations of more complicated aspects of the system system, such as using certified ‘digital twins’. Data analytics in wholesale operations also will utilize sensors such as smart meters (and many other sensors), but then to create and utilize this data to formulate and optimize opportunities to trade eventually leading to new tradable products and new markets; for example by activating demand response and small scale residential storage to optimize intraday and balancing positions. These products coming from wholesale optimization, are materializing in new retail products. Already energy suppliers are diverging their product range (e.g. offering hourly wholesale prices to their clients). Data analytics will tweak retail propositions to redistribute costs and risks between the energy supplier/aggregator, wholesale traders/generators and end-users. It will incorporate demand response, possibly automatically dispatched through ‘smart contracts’, making sure to incorporate the specific wishes of the consumer, for ex- ample using real time renewable energy from a specific wind turbine.

Data analytics in the electricity sector ENERGY 37

5 - THE IMPACT OF DATA ANALYTICS ON SETTLEMENT Energy settlement is the process by which the difference between electricity generated and electricity sold is reconciled. The process consists mainly of allocating metering data to the responsible market parties and associated contracts and, afterwards, the associated payments.


ata analytics in forecasting is mostly concerned with using existing data to forecast what this data will be in the future. Data analytics in operations is mostly concerned with translating both existing data and forecast data into control and optimization actions. Like operations, data analytics applied to settlement is mostly concerned with optimization, but on a longer timeframe: optimizing energy transactions and creating new competitive energy services and propositions.

38 ENERGY Data analytics in the electricity sector

5.1 Settlement in unbundled markets In an unbundled electricity market, the buying and selling of electricity and gas is done by commercial companies, while the transport and distribution of the energy is done through a public network, managed by independent transmission and/or distribution system operators that are regulated and monitored by government agencies (Regulators). Besides managing the system and network, transmission and distribution system operators are usually tasked to facilitate the market.

Because energy flows through the public network, following the laws of physics and not necessarily according to contractual arrangements, these flows cannot be physically traced back to the buyers and s ellers of this energy. Therefore, settlement in an unbundled electricity market is an elaborate and data intensive administrative process. Besides the billing to energy consumers by energy suppliers, it consists of processing the meter data of all generation feeding into the grid and all demand drawn from it and allocate this to the responsible commercial parties, as well as using this data to settle the deviations of the physical power flows with all commercial contracts and obligations that were traded bilaterally or on the energy markets prior to the physical delivery (often referred to as 'imbalance'). Small, usually residential users have a special place in this process. In most countries, their energy use is considered through an artificial standardized profile, which once a year is ‘scaled’ based on their total measured annual consumption and then settled between the responsible energy suppliers in a process called reconciliation. Smart meters (see also section 5.3) will allow the energy consumption and generation of these small end-users to be included into the ‘real time’ allocation of the electricity. In most countries, the energy market is discretized into 15-minute time slots called ‘Imbalance Settlement Period’ or ISP33.

5.2 The impact of data analytics on wholesale settlement Data analytics is a mathematical toolbox. It is the use of these tools that bring the benefits. So, how does this relate to settlement, being basically the process of calculating (and remunerating) the difference between previously negotiated contracts and agreements and the actual measured energy flows using metering data? Obviously, a lot of data is being processed during settlement, and data analytics on settlement data can extract more data that brings value for other processes, like forecasting, marketing and sales. Settlement itself however is an administrative process, and there appears to be little room for applying new insights that might result in benefits within the wholesale process itself. Which of course does not imply that data analytics cannot be used to make this process itself more efficient, faster and secure. However, at the boundaries of the settlement processes things are changing. The most evident being the use of 15-minute smart meter data in the allocation process, with most of the potential impact likely to be most felt on retail billing and retail contracts. More and faster data might lead to a better insight in the positions and risks of electricity traders, especially in relation to imbalance settlement as mentioned in section 4.3.

Data analytics in the electricity sector ENERGY 39

This scenario may lead to new risk mitigation and trade options, for example pre-, or real-time settlement imbalance markets, where imbalance risks can be hedged; and new demand response contracts targeting specific residential end-users, such as electric vehicle owners. And finally, the emergence of (independent) aggregators, which focus on flexibility instead on bulk energy sales, imply thatimplies that a separation needs to be made between the electricity supplied by the traditional energy supplier and the flexibility created by the aggregator. Sometimes installing a separate meter will be sufficient to distinguish between the energy supplied by different suppliers, but in the future, more often than not data analytics will need to be applied. An example would be when there is a need to determine a base line for explicit demand response sold by an aggregator to the transmission system operator as emergency power. The base line being the energy demand (or generation) that would have been supplied by the energy supplier if the aggregator would not have activated the demand response (7).

Settlement based on smart meter data (i.e. ‘billing’) involves huge streams of data and requires corresponding investments in ICT infrastructure, but processing this data for settlement still can be considered ‘standard’ ICT. The impact of allocation on smart meter data instead of settlement on standardized average profiles can be huge, because this means that it becomes possible to directly link the settlement between energy suppliers and their consumers (billing) with the settlement of the rest of the electricity system (allocation and reconciliation). In other words, it becomes possible to create innovative energy contracts for consumers based on real prices and risks on energy markets (instead on merely yearly averages). So, smart metering allows for advanced time of use contracts, dynamic pricing and other contracts based on optimizing energy use on energy price and risks or for example, contractually link the energy use to renewable intermittent renewable energy sources like a nearby wind farm as will be discussed in the next section.

5.3 The impact of smart metering Smart metering is often seen as a way to make consumers aware of their energy use and, hopefully, create the interest to reduce it. Data analytics is seen as a tool to give consumers the necessary insight and options to reduce their energy use. These include various visualization tools; load disaggregation (NIALM34); benchmarking with peer groups; concrete and consumer specific proposed actions35; etc. And indeed, smart boxes, thermostats and monitors that offer such functionality find their way towards consumers. Data analysis of smart meter data also has a potential beyond the energy domain: Smart Cities. Detailed data of energy consumption and generation might play a role in monitoring and optimizing public services other than energy supply, for example: traffic control, security, healthcare, etc. However, this is beyond the scope of this paper.

40 ENERGY Data analytics in the electricity sector

There are already a number of energy suppliers that pass day-ahead market prices to their customers, and earn their margins from fixed fees.36. Because they ‘spread’ the risks of trading on the day ahead market over their customers instead of hedging this risk by spreading it over their sources/procurement contracts, they can be structurally cheaper than traditional suppliers as end-users, are generally less affected by higher energy prices for a short duration than the energy supplier is, thus an overall smaller ‘insurance cost’ should be paid. Still, the marketing of these kind of energy contracts to risk-averse end-users still is a major challenge.

5.4 Advanced energy contracts As mentioned in the previous section, smart meter data combined with data analytics will pave the way for new advanced energy contracts. Besides the previous mentioned financial and risk-based energy contracts (such as retail prices based on the day ahead market), smart meter data combined with data analytics will allow energy suppliers and aggregators to further differentiate their offerings on other values than price and risk. In liberalized energy markets, where energy suppliers find it difficult to differentiate their offers, this is a major issue. To avoid direct competition on price, in a retail market that is becoming more and more transparent, energy suppliers need to differentiate their products from their competitors and find niches in the market37. While the amount of possible energy supply services is limitless, there are only two basic parameters that are connected to the energy supply itself that can be used to differentiate services from those of competitors. These are the price (actually price/risk) and the source of the energy38, which data analytics will have a major impact on. Propositions based on price/risk try to optimize the energy bill for the customer, considering his risk appetite and ability to reduce this risk in other ways. The rule of thumb is that the higher the risk, the lower the expected bill will be. An example is dynamic pricing, where the consumer price is varying every 15 minutes based on actual market prices or renewable generation. Consumers that are very aware, or have flexible loads that might be adjusted automatically (e.g. EV charging) would benefit with a variable, but on average lower energy bills39. On the risk-averse extreme, this could be a fixed price contact, which does not change regardless of the amount of energy used (e.g. a capacity contract, only limited by a special maximizing switch in the connection to the grid). The consumer then knows exactly how much he is going to pay irrelevant of how much energy he uses. The risk for the energy supplier will be higher (maybe limited by the special digital or physical ‘fuse’) and this will be included in the price. Of course, energy suppliers will be able to use data analytics to keep this risk as low as possible.

Propositions based on sourcing take the source of the energy into account. Originally end users could only choose one type of energy, which was differentiated into ‘green’ and ‘grey’ energy. Currently, more variations are emerging. Specifically for electricity, these can be traced back to either only power only from national wind farms; or to a specific farm with solar panels chosen by the consumers themselves. For electricity that is generated under the appropriate criteria a ‘guarantee of origin’ is issued by a certification body to the generator. Users buying this energy will receive the appropriate certificates40 and when the energy is consumed, the certificates are destroyed. This certification system ensures that the total amount of electricity used matches the total amount of certified electricity produced within a certain predefined period, usually a year. Big Data and data analytics can make this system much more refined. By combining generation data with consumption data, it becomes possible to differentiate this certification not only on source, but also in time41, so consumers can choose to buy real-time solar power and, when the sun sets, supplement it by the local wind farm or battery storage. Consumers thus can choose to become truly ‘independent’ from fossil fired power plants. Energy suppliers/aggregators might differentiate themselves by becoming facilitators of inter-customer trade, using timed dependent information of origin and destination to allow customers to exchange electricity among each other including their collectively owned wind turbine42. While these examples might seem a little exotic, they were demonstrated in quite some smart energy pilots. For example, in PowerMatching City II43, automatic demand response based on either dynamic pricing or the availability of local energy from solar panels— including wholesale settlement—was demonstrated, using heat pumps, micro CHP’s and electric vehicles (8) (9). And while the current public interest remains low, the interest from some companies and public bodies wanting to go beyond ‘merely’—on average—renewable to fully—real-time—renewable, is increasing.

Data analytics in the electricity sector ENERGY 41


As mentioned in paragraph 4.5, dispatchable demand response and non-dispatchable demand response can be integrated through a market mechanism in such a way that demand can be operated as a virtual power plant, while still enabling the participating clients the freedom not to participate when they are not flexible enough or if the price is too low. PowerMatching technology as applied in PowerMatchingCity (part of the EU FP6 project INTEGRAL), basically communicates the energy need of a device to the system based on an economic measure. For example, a heat pump/heat buffer combination can offer to use energy for a price that depends on the fill level of the heat buffer. If the buffer is almost empty, it would make a bid to ‘buy’ energy even for a high price. If near full, it would offer to buy energy only if the price is low. One of the advantages of such a system is that all devices will make a bid to use or produce electricity just a few minutes in advance. This means that the system is basically a ‘feed forward control system’, with all the benefits for stability. The bid curves can easily be aggregated to form bid curves on higher levels, like households, aggregated to district level, to aggregator or retailer portfolio level etc. Control or dispatch can then be done by communicating the price associated with the required power demand or generation. For traders, this system acts as if dispatching through a virtual power plant. Access to the bid curve of the whole portfolio provides a relation between power change and price change. By changing the electricity price of the consumers (the retail price through the aggregator) traders unlock this flexibility. It can be used to optimize the trader’s portfolio or it can be sold to the intraday or balancing markets. For the aggregator or retailer, this demand response opens a new dimension for the development of energy products and contracts based on the preferences their customers, up to the point to facilitate their consumers to organize their own energy (sub)community. The ‘default’ option is that devices are optimized on the retail price set by the aggregator. This price is the effect of the aggregator’s effort to optimize the load and local generation on behalf of its customers. Consumers and local generators that are the aggregator’s customers which (automatically) react on this price will receive a lower energy bill. In addition, the aggregator’s customers can react on other information as well. Consumers that would like to maximize the fraction of a certain kind of energy (for example solar) or certain sources such as locallygenerated energy can do so. If this energy is available they can change their bids accordingly, by using real time certificates, or just measuring data.

42 ENERGY Data analytics in the electricity sector

For grid operators, short term capacity management is made available. The aggregated bid curve can be used to set local prices to make sure that the total load (or generation) will be within all capacity boundaries. Within an area with an expected congestion area there is another price than outside? Communicating this price to the devices around the (expected) congested area will avoid overloading. The project PowerMatchingCity has already demonstrated that smart grids are technically feasible.  Flexibility has an economic value for the economy  Energy services can be created that meet the needs of consumers  Market barriers that impede the monetization of flexibility can be eliminated relatively cheaply




Data analytics in the electricity sector ENERGY 43

The role of blockchain in retail As in other sectors, blockchain is receiving a lot of attention in the power and utility sector as well. It can be used in many applications, from cryptocurrencies to facilitate energy trade among citizens to digital models of power electronic equipment for use in stability assessments by grid operators as discussed in paragraph 4.2 ‘The impact of analytics on grid operations’. Although the main application is seen in energy or power transactions between end-users (or their equipment when talking about the internet of things) and/or stakeholders in the electricity system such as retailers of the TSOs. Blockchain basically is a shared, distributed ledger of blocks of transactions, or other entries to the ledger, that are chained to each other. Each block has a hash (basically a digital fingerprint that depends on the content of the block) and contains the hash of the previous block. So, once an entry is recorded in a block, it cannot be altered anymore. Most blockchains are permissioned blockchains, meaning that all participants are known and trusted. There are some applications—like Bitcoin or Ethereum—that are permissionless (open) and everyone can make transactions as well as add new blocks to the chain, as long as you can ‘prove that you have an interest in upkeeping the integrity of the chain’, by showing a proof of effort44 (or e.g. a proof of stake) and the newly added block has been verified by others. This process of adding new blocks to the chain is called ‘mining’ and successful adding a new block to the chain is rewarded, for example by gaining bitcoins.

44 ENERGY Data analytics in the electricity sector

Blockchain (or alternatively tangle45) is considered as an important piece of the puzzle in the how the power sector might develop. Just as the shift from fossil to renewable electricity generation is breaking the business logic of ever-increasing scale in central (thermal) power generation, blockchain based transactions might break the business logic in energy supply and billing46, making blockchain a potential game changer, especially in energy retail, which allows the emergence of new players in the market, among which (cooperatives of) their own clients—active consumers—themselves. Consumers are actively organizing themselves into cooperatives, installing solar panels on their roofs or collectively build wind farms to generate and supply themselves with renewable energy. Blockchain (and tangle) technologies have many characteristics that seem like a natural fit with this development. Organizing a cooperative requires a high degree of organization and dedication. Not only in initiating such initiatives, but especially in maintenance and administration. Blockchain might facilitate a major part of this administration, such as the automatic registration (and remuneration) of energy transactions. Many applications using blockchain already have been developed that are, or can be used for energy applications; ranging from trusted peer to peer energy transactions and billing applications47 to certification.

For a successful and secure permissionless open blockchain application quite a lot of decentral computing power is required. Creating a separate blockchain for each application seems impractical and not very secure. It seems plausible that many blockchain applications will share a common blockchain, and a limited number of platforms/’tokens’ that support users to make their own blockchain applications will appear. Examples of such platforms that support multiple blockchain applications with one blockchain that is large enough to ensure the security of the application are Ethereum and Electron.

Using a blockchain to facilitate peer to peer trade between end users to circumvent the role of an energy supplier that does not take this ‘imbalance risk’ into account, would still require a contract with large party that can insure these risks. Such a blockchain will be an interesting tool for the energy supplier/aggregator/ balance responsible party to make their processes more efficient and secure, but it will be just an evolution of the existing business model of energy supplier/aggregator, enhancing their services to facilitate their customers to have more choice in selecting the source of the energy they use.

There are more hurdles to overcome before blockchain can break ‘the business logic in energy supply and energy billing’. For example; exchanging electricity between ‘peers’ uses the electricity network. The network requires a continuously balance of the power in and output. So, if the two ‘peers’ are not exactly matching their power generation and use, others that use the network are affected.

However, it is only a matter of time before blockchain applications appear that might be able to create portfolios and incorporate data analytics to calculate and share this risk among all participants, thus creating a virtual balance responsible party. It might even incorporate instructions for controlled demand response and local generation providing the flexibility and control to offset the imbalance risks of both demand, for example, to incorporate the variable supply of local renewable energy supply. Blockchain used in such a fashion has the potential to put the power system completely on a new footing.  

In most countries, it is obligatory to be ‘insured’ against this risk of not complying with your agreed demand and supply contracts (thus creating an administrative imbalance). Short imbalances are collectively taken care of by the TSO, longer imbalances are taken care of by the energy supplier (and corresponding balance responsible party)48.

Data analytics in the electricity sector ENERGY 45

5.5 Summary for settlement Settlement consists mainly of allocating meter data and their corresponding energy exchange to the responsible market parties and associated contracts and, afterwards, the associated payments. While data analytics can achieve benefits in optimizing these processes, it is in these contracts and agreements between electricity traders, energy suppliers and end-users where the major developments will be. The main application of data analytics will be driven by the changes at the edges of the settlement processes, caused by the development of smart metering and the rise of new business models focusing on alternative value models rather than the economies of scale of traditional energy suppliers. There are already some energy suppliers who have started to make use of smart meter data to build their retail products on, like charging their customers wholesale prices.  

46 ENERGY Data analytics in the electricity sector

Data analytics will enhance this trend and will allow suppliers or aggregators to enter a business niche that is focussing on energy of ‘higher value’ to their customers, for example by operating demand response or (local) storage tweaked to the wishes of these customers, using data from many sources, among which data from smart meters. So, the development of new business models, away from the economies of scale of traditional energy suppliers, focussing on ‘premium’ flexible energy and the application of data analytics go hand in hand and it is difficult to see which one enables the other. Applications like blockchain, tangle and the internet of things likely will play a role in this, likely to make existing settlement processes much more efficient and secure, but there is a small chance that blockchain technology will disrupt the existing business models of energy suppliers by automating their tasks and enabling end-users to trade among themselves, skipping the middle man.


Another major transformational trend is the development of the Internet of Things (IoT). It has many faces and applications, for example optimizing supply chains and logistics50 and media (sharing data between different devices etc.). Energy is seen as an important application, especially in unlocking large amounts of small scale flexibility from devices such as heat pumps and electric vehicles. The Internet of Things is about devices communicating with each other. Of course, for the IoT to create value, this communication has to have sensible content, that has meaning to the ‘receiver’. In other words, the IoT devices need to have something in common to talk over. While this value can be derived from the end user (e.g. streaming media, remotely controlled thermostats or an IoT coffee machine connected to the alarm clock), it becomes interesting if this communication becomes more autonomous, such as optimizing logistic chains in the industrial IoT, or indeed exchanging energy and optimizing the power system. Energy use, and especially flexibility in energy use, is something all devices share, and provided it is a scarce resource is an ‘interesting’ topic between devices to communicate. While the relative increase in value of this energy IoT will diminish as more appliances offer flexibility, it still might offer a kick-start to a much wider IoT.

The value of connectivity vs the value of optimization

Totalvalue valueofofthe thenetwork network Total

Internet of Things: Internet of Things: value proportional proportional to Value ton·log(b) n· log(n)

The value of connectivity vs. the value of optimization Added value Added value of of smart energy energy to to the smart Internet of Things the internet of things

Energy optimization:


Scaling up networks. As energy is a 'topic' all devices share, its initial value is relatively high. However, as networks grow, its relative value will decline. Contrary to Metcalfe's law51, all devices offer the same thing: flexibility in energy use that can be used to optimize the power system.

Value proportunal Energy opimization: to valuelog(n) proportional to log(n)

Number of connected devices n

Number of connected devices n

Data analytics in the electricity sector ENERGY 47

6 - CONCLUSION AND SUMMARY Data analytics has, and will continue to have, an enormous impact on the electricity system and its stakeholders. To benefit from it, companies will strive to connect data sources, sources within their company, external public sources and also sources from other companies, including sharing data with competitors.


n all the examples mentioned, data analytics is bridging borders between types of data such as structured and unstructured data; between departments and functions within companies; between companies and their customers; and even between competitors. In this sense data (and data analytics) is more of a ‘binding force’ that binds businesses (and other stakeholders) together into a much larger network than it is a fuel that drives the business itself. So, coming back to the initial question – ‘will data be the oil of the 21st century?’ As a transformational force, the impact of data might be comparable to that of oil in the 20th century. However, it will do this by binding stakeholders together instead of stakeholders competing over scare resources.

48 ENERGY Data analytics in the electricity sector

Competing in a data and data analytics environment therefore has a completely different logic compared to competing over scarce resources, and seeing data as a scarce resource to be secured is a recipe for failure. Competing in such an environment is about using data and data analytics to create a bond and relationship with other stakeholders, and not only customers. Synergy between data is the cement (or duct tape) that ties everything together. Companies should evaluate their ambition in what role they want to take in this ‘structure’ bound together by data, and what value they want to add to it, compared to companies with comparable ambitions; instead of merely looking at linear value chains and their added value to their direct customers.

Network operators, as well as energy suppliers, need to make fundamental choices: Be an essential—but rather unglamorous ‘brick’ in the foundation of the building, offering essential basic platform services (which are likely to be regulated eventually if they are essential). Or, instead, going for a glamorous, high risk, high reward strategy in ‘the top of the building’, offering ‘Gladstone Gander’ kind of services using state of the art advanced artificial intelligence; or will it be something completely unprecedented and new?

6.1 Applying data analytics in the electricity sector In the electricity sector the rise of data and data analytics coincides with the transition towards a more renewable energy system, though the speeds of both developments differ radically. Data and data analytics will play a major part in how, and how fast—the energy transition will develop. While many of the data analytic methods and algorithms are built upon similar principles, their application in different domains face different challenges and hurdles and domain knowledge is invaluable in the development and application of these algorithms. Nevertheless, in all applications it will allow a much more responsive behavior, by binding together all elements from the electricity value chain. Because of the difficulties to store electricity in large quantities, processes in the electricity sector are—more than in other sectors—divided into processes anticipating and preparing for the future; operatingthe system in (near) real time; and settling the difference between the two with hindsight. Data and data analytics do make the distinction between these processes less.

Data analytics in the electricity sector ENERGY 49

Data analytics is most widely applied in forecasting on many different time scales, and developments in forecasting are developing fast. Forecasting is widely used in the electricity sector, and its application will only increase and become more sophisticated. Better forecasting will allow the operation of the power system to become more effective and efficient. It will be operations that will be transformed the most by data analytics and its associated technologies. Data analytics allows for the optimization activities, traditionally conducted ‘ad hoc’ or by staff departments, to become part of the core processes itself. Thus, the core processes can be continuously ‘tweaked’ to the changing environment, though, as will all specialization, the necessary infrastructure likely will reduce the flexibility to react to ‘higher’ level strategic changes. Nevertheless, as the electricity sector goes through a transition, the flexibility through data analytics is a huge merit for all its stakeholders. It allows retailers and aggregators to implement automatic demand response and generation that reacts to variable renewable generation, as well as grid operators to ‘tune in’ to these developments and together with energy suppliers and aggregators, to include transmission and distribution in optimizing electricity supply. The boundaries between operations and settlement will become smaller and smaller. Especially at the ‘edges’ of the settlement processes, commercial stakeholders, such as energy suppliers and aggregators will make use of data analytics to diversify and escape the ‘commodity trap’ by developing new energy products, services and business models, giving end users more choice. Decentralized ledger technologies, such as blockchain and tangle, might facilitate this, mostly by adding cyber security, though the main potential of these technologies lies in administrating with n x m transactions, where a ‘central authority’ (like an energy supplier or grid operator) no longer is necessary.

50 ENERGY Data analytics in the electricity sector

6.2 Final reflection Important themes such as the availability of data, quality of data, privacy and cyber security should not be neglected. They pose important requirements that need to be satisfied for data analytics to reach its potential. However, as such, in this paper they are considered solvable, and thus outside the focus of this paper, which aims to give a sense of this potential and its impact on the electricity sector. This is illustrated with several examples. This paper tries to show in which direction developments are heading, though it remains unclear where it will end. In 2008 a very interesting observation was made in an article called ‘The end of theory’ (10), stating a quite convincing argument that data and data analytics would make scientific theory obsolete: “Given enough data, the data will speak for itself”. Naturally this caused a lot of discussion, but also raised new questions. The development of theory started with trying to explain observations, first through myths and legend, and the development of the ‘scientific method’ through verification and especially falsification of observations and correlations, making theoretical hypotheses that pass more trustworthy. In principle, the same mechanism can be applied to data, ever expanding the database giving more and more context and finding and refining more and more relations between parameters. Nevertheless, hypotheses also act as a guide to scientific research. It gives something to verify or falsify by experiments (i.e. by observations). So, a question that emerges is what will be the new guide to the development of knowledge? Or will it merely be a process of filling in the empty spots in the databases and finding more and more complicated correlations, using more and more advanced artificial intelligence?

7 - REFERENCES/SOURCES (1) Rus, Daniela. Study: Carpooling apps could reduce traffic 75%. www.csail.mit.edu. [Online] 3 January 2017 https://www.csail.mit.edu/ridesharing_reduces_traffic_300_percent (2) Aalst, prof. dr. WMP van der. http://www.processmining.org/. Process Mining. [Online] 2014 http://www.processmining.org/ (3) DNV GL. 10 Technology Trends Creating a New Power Realy. www.dnvgl.com. [Online] 2016 https://www.dnvgl.com/energy/publications/download/technology-outlook-2025-energy.html (4) USEF. www.USEF.info. http://www.usef.info/Downloads.aspx. [Online] 2016. (5) The application of health and risk indices as a decision-support tool for utilities. Vermeer, M.E., Schuddebeurs, J.D. and Wetzer, J.M. London : IET Digital Library, 2016 Asset Management Conference (AM 2016), 2016 page 31 (6) World business council for sustainable development. Corporate Renewable Power Purchase Agreements www.wbcsd.org. [Online] 26 October 2016 http://www.wbcsd.org/Clusters/Climate-Energy/Resources/Corporate_Renewable_PPAs_Scaling_up_globally (7) USEF. USEF: Workstream on aggregator implementation models https://www.usef.energy. [Online] September 2017. https://www.usef.energy/news-events/publications/ (8) Kamphuis, R., et al. Real-time trade dispatch of a commercial VPP with residential customers in the PowerMatchingCity SmartGrid living lab. [Online] 2013 IET Conference Publications. 2013. 1-4. 10.1049/cp.2013.0666 (9) Bliek, F., et al. PowerMatchingCity, a living lab smart grid demonstration. Innovative Smart Grid Technologies Conference Europe (ISGT Europe), 2010 IEEE PES. 1 - 8. 10.1109/ISGTEUROPE.2010.5638863, 2010 (10) Anderson, Chris. The end of theory: the data deluge makes scientific method obsolete. Wired. June 2008 (11) DNV GL. https://www.dnvgl.com/publications/smart-cable-guard-2-0-17603 www.dnvgl.com [Online] DNV GL, 2018 (12) Kadurek, Petr. Data Applications for Advanced Distribution Networks Operation (PhD thesis) Enschede: Ipskamp drukkers, 2013 (13) DNV GL. https://www.dnvgl.com/publications/making-renewables-smarter-104362 https://www.dnvgl.com/publications/. [Online] 11 2017 (14) Sridhar, Narasi. 2017 Frank Newman Speller Award: Knowledge-Based Predictive Analytics in Corrosion. 2017, Vol. 74, 2 (15) Qian, Y., et al. Risk on failure, based on PD measurements in actual MV PILC and XLPE power cables Jicable'15. Versailles, France, June 2015

Data analytics in the electricity sector ENERGY 51

8 - END NOTES This comes of course at the expense of the taxi drivers. Whether this is completely ethical is another issue. Around Uber, but also around Airbnb there are many controversies. They are transforming their industries to such an extent that many think that new regulation is required. 2 ‘Big Data’ is usually characterized by Volume, Velocity, Variety and sometimes Veracity. In other words, the data is too much, too fast, too divers and with a too low ‘integrity’ to be handled with ‘traditional’ ICT technology. 3 Note: A fundamental difference between these markets and electricity is that besides better service and more choice Uber and Airbnb main breakthrough is a much more efficient matching between demand and supply that includes new participants that did previously not participate. The electricity system is characterized by a per definition (near) perfect match of demand and supply, that includes all grid connected generation and demand. So, while the Uber and Airbnb business models can and likely will be transposed to the energy sector, their impact will be different. Data analytics does enable other business models that might have a similar impact on the electricity markets as Uber did on the taxi market. 4 Management of the shared data might also be an issue, as this can lead to distrust and if one of the participating partners would take this role. Most likely this would be done by an independent third party (e.g. a joint venture of the participants, or an independent service provider). 5 For example, DNV GL has set up its Veracity data analytics platform for external clients to store, analyse and share data among each other, even across industries and benefit from the data analytics capabilities it offers. 6 In 2008 Chris Anderson published an article in ‘Wired’ predicting the End of Theory. “With enough data, the numbers speak for themselves.” For practical experience this would be even truer. 7 More information about the challenges the electricity sector is facing can be found in other DNV GL white and position papers, among which the Energy Transition Outlook: https://eto.dnvgl.com. 8 DER (distributed energy resources) can be generation, like CHP units, solar and wind generation. It includes also potential flexible loads, like charging of electric vehicles and demand response (both industrial as residential). 9 The main message is that policies are often changing every few years. Still successful examples are the feed in tariffs boosting solar in Germany and the current tenders for off-shore wind projects in the North Sea, driving down costs. On the other side there is the ETS scheme that still does not really take off. 10 Another dimension in the transformation of the energy supply is formed by the effects of the unbundling of distribution and production/supply function of utilities and the liberalization of the latter since 2000. This has contributed to an overinvestment and consequent crisis in the traditional fossil fuelled power generation. 11 Customers’ expectations as well as the perceived value of the services they received are influenced by many factors, like past experiences, information from competitors, reactions from friends and family, the media, etc. The reversed mechanism: a dissatisfied customer ‘running away’, because the service/product did not live up to their expectation, is even stronger. 12 DSO stands for Distribution System Operator. In this document the term DSO is chosen over DNO (Distribution Network Operator), because Distribution Network Management will require more and more active control of assets and services, for example for capacity management, voltage control and other power quality issues. 13 For example the use of drones for automatic inspections of wind turbine blades, see (13). 14 See www.oreda.com 15 See (14). The graph is based on internal analysis by Narasi Sridhar. 16 A (simplified) example could be: P(failure near sea shore) = P(failure because of salt) + P(failure because of humidity) – P(failure salt - failure humidity). By factorizing these probabilities in elements and recombining the elements, data from different failures can be reused to anticipate failures in other situations and possible other kinds of assets. 1

52 ENERGY Data analytics in the electricity sector

For a wind/solar farm that is rewarded per MWh in a feed in or subsidy scheme, it is more important to maximize the output using the available wind, than it is to know exactly what this output will be in the next four hours (for investment decisions is important to know the average yield of a farm). A wind or solar farm that is (partly) participating in the electricity market, where energy is traded beforehand, this becomes much more important. 18 For example ARIMA models (autoregressive integrated moving average) as well as neural networks applied to aggregated data demand data to extract trends and periodic patterns. 19 This also applies to long term energy contracts. This is for example illustrated by the increase in intraday trading. In parts of Western Europe TSO’s are finding it increasingly challenging to include all the changes in the BRP’s energy programs to the transport prognosis that is caused by intraday trading. 20 State estimators basically estimate the state of a grid (i.e. the complex voltages of all nodes and the related currents) by minimizing the errors between the results of load flow calculations and actual (synchronized) measurements. Both state estimation and load flow simulations are widely used, but for distribution grids mostly ‘off line’ situations once per e.g. month. And not continuously as input for active grid management. 21 For example, DNV GL developed ‘Kermit’ and ‘Pulse Chlorination’. ‘Kermit’ is a model for grid stability, based on the simulation of inertia and frequency control. ‘Pulse Chlorination’ is a commercial product DNV GL able to minimize the amount of chlorine necessary to avoid bio fouling in cooling water outlets of power plants and factories, based on the monitoring and modelling of clams. 22 Reference: http://www.fingrid.fi/en/grid_projects/ELVIS/Pages/default.aspx; and https://www.fingrid.fi/en/pages/news/news/2016/fingrids-elvis-project-completed 23 With the help of mathematical developments like stochastic collocation; splitting techniques (rare event simulations) able to immensely speed up Monte Carlo kind of simulations; and dimensionality reduction of scenario’s; it becomes much easier to quickly identify and assess potential critical situations. 24 For this the behaviour of smart devices need to be known, and especially how smart devices might react to one another. It is viable that the model based control includes a model of these devices, merely that this model can identify predefined ‘instability states’ from more detailed simulations using digital twins (certified digital models) of the smart devices and inverters that behave exactly like the physical original (within the simulated scope). More on the verification of these digital models and their physical counterpart, using PHIL (Power Hardware In the Loop) and CHIL (Control Hardware In the Loop) methodologies can be found in the DNVGL position paper ‘Grid Cybernetics’. 25 Like ‘flexibility providers’ such as greenhouse CHP units; industrial and agricultural cooling houses as well as other demand response and small flexible generation. 26 Consumers that also produce energy e.g. with solar panels. 27 For example, there supposedly is a correlation between switching from energy supplier and the installation of solar panels 2 months earlier, so information of solar panel installation provides a perfect lead to energy retailers. 28 NEST is probably the most well-known. 29 Called Non-Intrusive Load Monitoring (NILM) or Non-Intrusive Appliance Load Monitoring (NIALM). 30 This depends of course on the demand. While in principle all demand is flexible for the right price, this applies mostly to processes where the ‘objective’ can be decoupled from the power use. Examples are heating and cooling processes (like air conditioners), where the heat or cold can be buffered, or charging electric vehicles, where the charging and driving happen on different times. 31 See e.g. https://www.enernoc.com/ and http://www.energy.actility.com/ 32 Assuming of course a basic level of quality and reliability of the power system, where power is considered a commodity and not as a luxury. 33 This used to be called Program Time Unit or PTU. In most countries this is 15 minutes, is some this is 30 minutes (e.g. the United Kingdom) or 1 hour (e.g. Spain). 17

Data analytics in the electricity sector ENERGY 53

NIALM: Non-Intrusive Appliance Load Monitoring: disaggregation of metering data into the energy use of individual appliances that themselves are not monitored. 35 For example to replace an old refrigerator when it becomes economical beneficial to replace it. 36 E.g. Easy Energy: https://www.easyenergy.com. 37 While a traditional energy supplier’s business model focusses on economics of scale and quantity, An (independent) aggregator can be seen as an energy ‘supplier’ that is trying to differentiate its energy based special (time dependent) requirements: i.e. flexibility. This flexibility is created by aggregating and controlling flexible demand and generation. 38 Of course besides the energy itself there are other differentiation values. For example providing excellent customer contacts; providing information and advice on energy use; give welcome or retention presents for new resp. existing customers; create a corporate image fitting the life style of customers; and many more all are important and are used in creating ‘value’ to consumers, but all of these services and/or attributes are not directly related to the energy itself. 39 Because the consumer is taking the risk of the fluctuating market prices, this does not necessarily mean that the energy supplier will receive lower margins. Normally the supplier would have much better capabilities to manage this risk, because of the large portfolio he has. However, if the consumer can adjust his demand to the fluctuating prices (Demand Response) it will result in a net gain. 40 The certificates and the energy can be traded independently. 41 See also GPX (www.gpx.eu) and PowerMatching City II (www.powermatchingcity.nl) 42 The energy supplier becomes an aggregator and—until blockchain will take over—will perform the settlement and responsibilities to the system (like balance responsibility). 43 See www.powermatchingcity.nl 44 With proof of effort, a successful miner will create a new block of entries including a hash, that miners only can solve by trial and error in order to win. If the new solution in verified, the winner will add a new block with entries to the blockchain, etc. 45 A tangle—as developed and applied in Iota—is basically a network of hashed ‘transactions’ instead of blocks of transactions in a single chain. This allows parallel verification of transactions done by the parties doing the transactions themselves, instead of being performed by separate ‘miners’. This allows for smaller transactions and overall less mining costs (in Iota transaction can be done at a cost of verifying two other transactions). This might avoid the huge energy consumption currently required by bitcoin miners to supply the ‘proof of effort’ (i.e. solve the hash puzzle). See https://iota.org/ and https://iota.org/IOTA_Whitepaper.pdf 46 Among others creating economies of scale with huge volumes and centrally standardized processes. 47 E.g. PowerPeers, GPX, Innogy EV Charging, Grid+. Though most of these are based on a private blockchain, i.e. blockchains where transactions (or the information) are stored in a blockchain for security reasons, but instead of competing miners there are only a couple of central authorities (or just one) that validates all transactions and puts them in the chain. 48 The TSO procures ancillary services like frequency containment reserve (FCR) to compensate balance disturbances resulting in frequency deviations. If the imbalance is longer than the imbalance settlement period (ISP, in most countries 15 minutes), then one or more balance responsible parties (BRP, often an energy supplier) did not act according previous negotiated contracts and this mismatch is produced/consumed by order of the TSO and the bill forwarded to the responsible BRP. All demand and generation fed into, or drawn from the grid has to be accounted for by a BRP. A contract with a BRP can be considered as an insurance against creating imbalance. 49 DSO stands for Distribution System Operator. In this document the term DSO is chosen over DNO, because Distribution Network Management will require more and more active control of assets and services, for example for capacity management, voltage control and other power quality issues. 50 The Internet of things allows to optimize value chains and logistics up to the devices of the end users (like the ‘fridge, after noticing that it is about to run out of milk, will order new milk). 51 Metcalfe’s law states that the value of a network scales with the number of connections between nodes, this scales with the number of nodes squared: n2, or, discounting for declining values of connections, scales to n*log(n). See https://en.wikipedia.org/wiki/Metcalfe%27s_law 34

54 ENERGY Data analytics in the electricity sector


DNV GL Netherlands B.V. Utrechtseweg 310-B50 6812 AR Arnhem The Netherlands Tel: +31 26 356 9111 Email: contact.energy@dnvgl.com www.dnvgl.com Author: Marcel Eijgelaar

DNV GL DNV GL is a global quality assurance and risk management company. Driven by our purpose of safeguarding life, property and the environment, we enable our customers to advance the safety and sustainability of their business. We provide classification, technical assurance, software and independent expert advisory services to the maritime, oil & gas, power and renewables industries. We also provide certification and supply chain services to customers across a wide range of industries. Operating in more than 100 countries, our experts are dedicated to helping customers make the world safer, smarter and greener. In the power and renewables industry DNV GL delivers world-renowned testing and advisory services to the energy value chain including renewables and energy efficiency. Our expertise spans onshore and offshore wind power, solar, conventional generation, transmission and distribution, smart grids, and sustainable energy use, as well as energy markets and regulations. Our experts support customers around the globe in delivering a safe, reliable, efficient, and sustainable energy supply. Group Technology & Research The objective of strategic research is through new knowledge and services to enable long term innovation and business growth in support of the overall strategy of DNV GL. Such research is carried out in selected areas that are believed to be of particular significance for DNV GL in the future. A white paper from Group Technology & Research is intended to highlight findings from our research programmes.

The trademarks DNV GL and the Horizon Graphic are the property of DNV GL AS. All rights reserved. This publication or parts thereof may not be reproduced or transmitted in any form or by any means, including photocopying or recording, without the prior written consent of DNV GL. ŠDNV GL 10 /2018

Profile for DNV GL

Data analytics in the electricity sector  

This paper tries to show in which direction developments are heading, though it remains unclear where it will end.

Data analytics in the electricity sector  

This paper tries to show in which direction developments are heading, though it remains unclear where it will end.

Profile for dnvgl