THE
INTRODUCTION
Energy Data Management systems have long been an integral and central part of the operational landscape of energy companies, but their role is now evolving dramatically in response to the growing volume and complexity of data. Let us consider forecasting as an example area. In forecasting applications, this transformation now means that data extends beyond just the traditional weather data historically utilized for power and gas consumption forecasting. Now, it also includes critical production forecasts for wind, solar, and hydro power facilities underpinned by a richer dataset that encompasses not just temperature, precipitation, and wind strength, but also real-time satellite imagery for precise forecasting, for example. The introduction of intraday and continuous power trading now also necessitates the need for near-real-time forecasting imposing more stringent demands on the update frequency and performance of the software processing these critical time series.
The need for efficient time series processing transcends forecasting. Real-time market data, emanating from continuous intraday power trading generates a vast quantity of information requiring storage and analysis for a wide range of applications such as fraud detection and trading strategy evaluation. Weather data is invaluable, not just for forecasting, but also for informing trading decisions.
Equally critical is the management of smart meter data. With the proliferation of smart metering, consumption data is pivotal across a multitude of application areas demanding meticulous data management including cleansing, anonymization, and even the generation of artificial data preserving statistical integrity. This data is vital for applications ranging from consumption-based retail pricing to identifying demand-side flexibility.
Data management is increasingly seen as a universal
task within the industry spanning the energy trading to retail business segments and from AI model development to compliance and fraud detection. At the core of this endeavor is time series data, the cornerstone of Big Data in energy-related applications, characterized by its rapid accumulation, fine granularity, and high frequency of updates - sometimes down to seconds. This exponential growth in data imposes significant performance demands on data processing necessitating efficient data cleansing, aggregation, versioning, and seamless integration with various applications across the enterprise.
Visualization of this data has also become a critical aspect of data management. Modern requirements dictate that energy data software must support userdefined graphical visualizations, allowing users to merge data from diverse sources into cohesive visual representations. This includes simple transformations
and calculations as well as the ability to effortlessly create and customize graphs and dashboards, catering to a broad spectrum of user needs.
Performance, security, and integrability of the data platforms are paramount. High performance is often linked with cloud deployment, which necessitates scalable solutions that can adjust resource allocation up and down based on processing demands. Only cloudnative platforms can prove cost efficient in the cloud and capable of using cloud elasticity. However, cloud deployment raises significant data security concerns as energy companies are wary of compromising
sensitive information, including price information or private data contained within smart meter data. This requires sophisticated solutions for data anonymization and privacy law compliance. Additionally, the ease of integrating the data platform with various other applications is crucial requiring not just standard APIs, but also compatibility with widely used tools and programming environments.
HAKOM Time Series is a company at the forefront of this evolving landscape, offering its PowerTSM® software as a cutting-edge data management platform. It has designed its solution to efficiently handle large volumes
of time series data and it emphasizes PowerTSMs® high-performance data capture, validation, and processing capabilities, tailored specifically for the energy sector. Built on a modern microservices architecture, it is platform and database independent, and offers plug-and-play extensibility, making it an ideal potential foundation for diverse time series data projects. HAKOM conceptually designed PowerTSM® to provide a potentially significant advancement over traditional data platforms, providing not just the technical infrastructure for application integration, but also a comprehensive toolkit for developing custom software
solutions. This toolkit comprises the data platform itself, an integrated application for data provisioning aimed at facilitating seamless data sharing between applications, and a range of pre-linked applications that are readily available for immediate use. The number of such applications will grow with time and its approach promises to redefine how energy companies manage and utilize their data, paving the way for creating flexible tailor-made solutions for various needs. The following chapters describe how HAKOM sees this working in more detail.
TECHNICAL COMPONENTS AND FUTURE VISION
Let’s explore why HAKOM views their data platform as technically innovative. According to HAKOM, the HAKOM PowerTSM® suite has been designed for the efficient capture, analysis, and processing of time series data, catering specifically, but not exclusively to, the energy sector. It leverages a contemporary microservices architecture ensuring platform and database technology independence, along with plug-and-play extensibility. This, HAKOM says, makes HAKOM PowerTSM® adaptable for any time series project, regardless of size.
It is cloud-native software, which means that it uniquely utilizes the cloud’s scalability and elasticity distinguishing it from traditional “lift and shift” cloud deployments. Its capability for parallel computing provides the potential to perform when managing vast datasets. Thanks to a microservices architecture, larger tasks can be distributed across multiple services.
It also employs compression techniques for data
storage, allowing it to potentially achieve up to an 80% reduction in storage needs. It supports both current and historical data, with time granularities ranging from milliseconds to years, and allowing for the definition of time series without fixed intervals. Quality indicators for each time interval are available to validate the data points (e.g., validated, entered, predicted). Time series attributes are versatile, enabling extension of master data, categorization of series, or assignment to
hierarchical structures.
HAKOM PowerTSM® also offers numerous processing capabilities for time series, including automatic interval conversion and unit conversions based on custom aggregation rules. It uses an integrated formula engine to simplify the capture and validation of time series formulas, with an easily expandable function library.
From an integration perspective, the PowerTSM® Excel Add-in provides straightforward management and linkage of time series data. Application integration is facilitated by a platform-independent, Dockerenabled REST service, ensuring full functionality integration into any application. HAKOM’s Power TSM® integrates with leading data science tools via the WebTSM Services API, offering standard connections to Python, MATLAB, and R. For data exploration, options include full integration with MS Excel and the server optimized PowerTSM® Plus application, with additional exploration capabilities through PowerBI integration.
Data visualization is critically important in today’s visually driven world. Modern users, accustomed to the interactive and visually rich environments of social media and gaming, expect smooth, intuitive, and timely interactions with systems. HAKOM has chosen Grafana for its visualization needs, valuing its flexible data representation, real-time monitoring capabilities, and ease of dashboard configuration. Its status as an open-source platform with a wide user base was also a key factor in its selection.
The option to utilize external visualization tools like Grafana and the upcoming Visplore application for time series visualization and exploration highlights the platform’s flexibility and the concept of linking multiple applications. The platform itself has no custom functionality, undergoes automatic testing, and allows continuous updates. Applications, whether public or private, are updated independently based on user needs, with HAKOM ensuring interface compatibility with each release.
These critical features lay the technical groundwork for HAKOM’s envisioned business model: the creation of a platform that facilitates the easy integration of any application and offering 3rd party applications equipped with standard plug-ins for various commonly used functional extensions. Providers of such integrated applications are informed about forthcoming changes in advance to ensure compatibility with upcoming releases.
Security is paramount. The platform is subject to regular vulnerability assessments, with immediate deployment of security patches independent of release schedules. However, recognizing the need for enhanced data exchange security, HAKOM is initiating a partnership with Intertrust Technologies aiming to secure data exchanges by validating signals to ensure that what is sent matches what is received and enhancing the security of data stored in the cloud. Intertrust’s verification means data authorized by Intertrust will be marked as Intertrust Proved in HAKOM’s storage, providing an added layer of trust and security.
DATA AVAILABILITY AND DATA PRIVACY
Time series platforms can offer significant functionality, but we believe that their value is significantly enhanced when paired with commonly used datasets. Data such as market data (prices, traded volumes), or weather data used for renewable forecasting, can be shared across multiple applications. HAKOM aims to provide users with a variety of different data types by collaborating with the Austrian company – Cognify and integrating their product Data Donkey. This collaboration exemplifies how integrating one application into the PowerTSM® platform can address the challenge faced by many energy companies: that of consolidating, validating, and making data available from diverse sources, independent of original format or location.
Cognify GmbH provides a solution to this challenge with their product Data Donkey. This solution is capable of extracting data from various sources, including data warehouses, websites, APIs, RSS feeds, social media platforms, and virtually any digitally accessible resource, and integrating it into a central database. Utilizing stateof-the-art technologies and tools, the data is validated and transformed into a user-friendly structure.
Integrating Data Donkey into PowerTSM® enables seamless streaming of data into the platform, making it accessible to other applications. End-users can conveniently select domains, data sources, and individual time series needed for their applications directly within PowerTSM®. Examples of data sources supported through the Data Donkey application range from ENTSOE, offering power plant availability, crossborder flows, network load and production data, to market data from EPEX or weather providers’ data.
Another instance of integrating a data-related application into the platform is the forecasting service from Metalogic. The mP Xnergy solution was designed
to meet various forecasting requirements, catering to smaller local municipal utilities as well as large international energy producers, suppliers, network operators, and trading companies for consumption, generation, and grid forecasts. Integrating this application into the platform will render the forecasting data accessible for numerous future use cases. The availability of these readily usable data sets significantly enhances the value of the platform, reinforcing the advantages of HAKOM’s chosen business approach: the integration of third-party services that can be offered alongside the platform.
Data privacy considerations are particularly pertinent when it comes to consumer meter data. There are numerous use cases where smart meter data is essential, such as in retail pricing creation, pattern recognition, demand response applications. For energy companies, processing smart meter data facilitates more efficient customer engagement at a personalized level, positioning them as partners in demand response. Such companies can offer recommendations for optimizing device usage and implementing energy
cost reduction strategies. An application that removes personalized information and generates synthetic datasets with equivalent statistical qualities to the original data is crucial in this context. This would enable Distribution System Operators (DSOs) and
utilities to readily share artificial datasets with research organizations, software companies, and others. This would potentially be another business case of useful integration of the 3rd party service.
COMMERCIAL APPROACH
Let us now further explore HAKOM’s business model and focus here on its commercial side. The platform operates on a subscription-based model with an easily accessible online price calculator. Calculating the necessary cloud resources for a specific use case is streamlined, relying on key data points such as the number of users, time series, their granularities, update frequencies, and retention periods. The calculator automatically determines optimal cloud resources, factoring in storage and transaction costs.
Integrated services will be conveniently listed alongside HAKOM prices in the same calculator, allowing potential customers to estimate not only platform usage costs, but also any expenses associated with the data provided by integrated services. Pricing typically includes both a flat fee as well as an access-related fee. Users can select time series from data catalogs provided by these services from directly within the platform. The access fees are only charged when the data is actually accessed and the access rules are determined based on user groups or individual time series, with support for multifactor identification.
This vision resembles the APP Store concept, where
certified third-party applications can seamlessly link to the platform, offering a curated selection of applications alongside the platform itself. Applications may be either public or private, with only public ones being available for purchase. Potential users can peruse a catalog of available services to tailor their selections to their specific requirements.
To attract more application developers, HAKOM has also proposed a program which includes free access to the platform for applications developers to create a connector with a profit-sharing model after the connector is live on the platform.
CURRENT AND FUTURE BUSINESS CASES
Now let us talk about the vision and possible future use cases where HAKOM’s concept looks promising. Numerous business cases exist where the availability of third-party services could significantly enhance the value of the data platform. However, some of these are already in motion and initial steps have already been taken by HAKOM. These business cases are described below.
Security and data protection service
Among the various applications dedicated to security, one stands out as particularly crucial given its direct impact on the reliability and security of power networks. As outlined in §14a of the Energy Industry Act (EnWG), Distribution System Operators (DSOs) are empowered to alleviate congestion by either reducing the load or temporarily disconnecting certain prosumers from the network. To execute this, DSOs must possess the capability to selectively deactivate portions of the network, relying on data of absolute reliability.
The HAKOM data platform, combined with Intertrust software, offers a robust solution to address this situation. Leveraging Intertrust technology, data encryption can be implemented across multiple tiers, spanning from individual meter points to substations or aggregated levels. DSOs retain the flexibility to determine on level the network dimming is performed, and data encryption can be aligned with the specifics of their network architecture.
Encrypted data is securely transmitted to the Power TSM, where it is decrypted and authenticated, ensuring its integrity and legitimacy. With the proliferation of distributed production and consumption, an increasing number of DSO customers will possess contracts permitting the DSO to enact §14a measures, heightening the significance of secure meter data transfer for DSOs.
This business case can be further expanded by the development of Power2Drive technology supporting flexible timing of EV Charging and other IoT technologies. These developments enhance consumers’ capacity to provide flexibility to DSOs, thereby equipping DSOs with additional tools for network management and optimization. This amplifies the volume of securely transferred meter data, making the business case forward-looking based on its support to evolving technologies.
Forecasting services
For a data platform offering access to both weather data and meter data, incorporating forecasts would be a natural progression. The aim is to provide ready-to-use forecasts for consumption or production, eliminating the need for complex modeling. This should become a straightforward feature for platform users, with specific requirements guiding the type of forecasts offered.
For instance, residential consumption forecasts typically rely on local weather data and historical consumption. By entering a few parameters, such as postal code and meter point or substation, the model could automatically select the relevant weather data and historical consumption for calculation. The platform already contains network topology and general weather data, and forecasting models can be easily accessed with minimal user input, simplifying forecasting as a complementary feature of the data platform.
However, it’s important to note that not all forecasts can be considered low-hanging fruit. Industrial consumption often requires more complex or branch-specific modeling, with additional input parameters. While this could be an extension of the business case, it’s not the primary focus initially. HAKOM aims to offer one-click forecasting on the Power TSM® platform.
Simplistic consumption forecasts based on a common model can be provided by the Metalogic forecasting service, if available on the Power TSM. While Metalogic also supports complex individual modeling, pricing for such models requires individual discussion. Conversely, simplistic forecasting can be easily included in HAKOM’s pricing structure.
With the rise of renewable production, forecasting production becomes as vital as forecasting consumption. Cognify solar forecasting, based on the same one-click principle and requiring minimal input parameters like KW peak and solar panel location, will be connected to the Power TSM.
To complete the use case, wind production forecasting availability can be considered by connecting with a service form some of multiple companies focusing on wind power forecasts.
Integration with ETRM
What role time series platforms could play in relation to an ETRM System? To answer this question, we need to step back first and look at the nature of the data which an ETRM system needs to process.
Most of the well-known legacy ETRM systems were developed by traders for traders. Traditionally, these systems primarily focused on price-related time dependent variables, which were the main concern for traders. However, with the emergence of renewable production assets, such as batteries, and new deal types like Power Purchase Agreements (PPAs), there is a growing demand for additional time series data to support trading activities, such as weather data, aggregated meter data, etc. These time series can be added to the ETRM solutions but usually stored in different structures within ETRM, separated from the price data. However, these time series can become inputs to formulas which may include price, volume data and even more. There are a lot of other calculations where time series data need to be considered holistically, independent of the nature of these data. Of course, legacy ETRM solutions may allow “work arounds” to deal with the requirement, but when the distributed assets and related deal types or formulas become common as opposed to being considered exotic, it is increasingly difficult to use those “work arounds”.
Even modern ETRM solutions often lack a comprehensive time series data model, despite its usefulness. Exceptions include solutions originally
designed for forecasting, optimization, or general data management with trading and risk capabilities added on the top. Such solutions can benefit from built-in time series management capabilities. Examples here might include IRM, now part of Kisters, or FIS Energy Portfolio Management.
But how might other trading solution address these challenges? Integration of a time series data platform into ETRM systems can provide a centralized hub for all time-dependent data required by the ETRM. This integration opens numerous business opportunities from building forward curves to conducting formulabased calculations regardless of the inputs’ nature.
One significant use case arises from consolidating multiple software components into a single, flexible platform. For instance, Volue is consolidating various products it has acquired in recent years, such as optimization tools for hydro and thermal power plants and also-trading solution, into one unified platform. In such cases, the time series data hub becomes a critical component, not just for simplifying modeling but as a mandatory aspect of the solution.
Customers who have traditionally managed time series data alongside their ETRM systems are reluctant to switch to solutions lacking time series management capabilities when trying to replace their legacy ETRM(s). Multiple replacement projects are struggling to manage this transition even in the case of powerful new ETRMs. This is where HAKOM’s Power TSM® comes into play, providing a solution according to HAKOM.
The distinction between the ETRM-related business cases and those previously discussed for forecasting and data security lies in the role of HAKOM’s Power TSM. Rather than being a foundational component, it functions as an application that sits atop the ETRM system. This positioning underscores the significance of modern architecture of the ETRM platform, allowing seamless integration with Power TSM. If the ETRM solution is cloud native platform designed for integration of third-party applications, the combined solution becomes especially compelling. Several contemporary ETRM solutions, such as Previse System or ETRMCubed adopt a similar approach offering limited out of the box functionalities while prioritizing the streamlined integration of applications by certified third parties.
With the rise of Virtual Power Plants and aggregators operating them, new trading-related business cases are emerging. These market participants, tasked with managing large volumes of meter data alongside trading activities, represent a growing market segment ripe for development.
In summary, integrating time series platforms with ETRM systems enhances their capabilities, addresses evolving data requirements, and opens new business opportunities in the energy trading landscape.
ABOUT HAKOM TIME SERIES
With more than 30 years of know-how in the energy industry, over 100 customers in 15 countries and 150+ successful projects, HAKOM has positioned itself as a strong and sought-after partner of the energy industry. Since 1991 HAKOM has been following its successful path steadily and successfully and is a reliable and innovative partner for its customers.
The smallest components of matter are elementary particles, the smallest elements of the energy industry are data with time stamps, collected in time series. The further we look into the future, the more important time series become for the energy industry and those who depend on it – Smart City, Industry 4.0, Agriculture 4.0, Smart Metering, IoT, autonomous driving or 5G mobile networks are just some of the buzzwords that will change our world forever. Tomorrow. There, time-stamped data will be the lowest common denominator, the elementary particles of the energy industry collected in time series to control and link complex systems. Without the management of time series at all levels, our modern world will no longer function technically.
This current and booming market of the future is HAKOM’s core business, making the elementary particles of the energy industry controllable and creating added value. Enrich data with time stamps so that numbers become values. This is HAKOM’s sought-after competence.
The challenges in the area of time series management,
in a disruptive market environment such as the energy industry, are enormous. In order to master them, it takes decades of expertise and a commitment to dedicate oneself exclusively to this highly complex subject with passion. The ability to validate and monitor Big Data, to analyze Big Data, no matter from which local source, system environment, formatting or screening the data material may originate and to always deliver useful results in a short time in this process, requires an innovative basic technology like HAKOM:
• Ingestion of Big Data at unbeatable processing speed,
• full database independence,
• proven energy logic,
• perfect API for easy integration into 3rd party systems,
are some core premises. HAKOM has developed its proprietary FeasyUse® technology to ensure that the whole thing remains suitable for everyday use, intuitive and easy to use without having to compromise on quality or performance. Because ingenious is simple and not complex.
ABOUT
Commodity Technology Advisory LLC
Commodity Technology Advisory is the leading analyst organization covering the ETRM and CTRM markets. We provide the invaluable insights into the issues and trends affecting the users and providers of the technologies that are crucial for success in the constantly evolving global commodities markets.
Gary Vasey heads our team, with many years in the energy and commodities markets, provides depth of understanding of the market and its issues that is unmatched and unrivaled by any analyst group.
For more information, please visit: www.comtechadvisory.com
ComTech Advisory also hosts the CTRMCenter, your online portal with news and views about commodity markets and technology as well as a comprehensive online directory of software and services providers. Please visit the CTRMCenter at: www.ctrmcenter.com
PO Box 1269, New Waverly, TX 77358
Prague, Czech Republic +420 775 718 112
ComTechAdvisory.com
Email: info@comtechadvisory.com