The data confidentiality cannot be guaranteed if there is no secure connectivity between devices [ 15 ]. To secure the communication, there is a need to adopt an authentication mechanism [ 16 ]. The authentication defines users using credentials, the authorization describes for each user his own permissions, and the accounting is responsible of supervising users [ 17 ]. Big Data technologies are a good opportunity for utilities to bring new methodologies, evaluation models and applications and improve data management in smart grids.
Big Data can be defined as a huge quantity of datasets, but in fact it includes other features.
In addition to 1 the volume, Big Data is based on 2 the variety to present various data formats structured, semi-structured or unstructured , 3 the velocity to provide timeliness requirements, 4 the value to give the ability to extract the meaning from the collected datasets, 5 the variability to provide inconsistency concept of the data, and 6 veracity to work on the trustworthiness of the data [ 18 ]. Figure 5 presents Big Data technologies for smart grid, in it different levels from data sources to visualization.
Actually, there are distinct data classes according to the type of extracted values: i Operational data which is the electrical data of the grid that represent real and reactive power flows, demand response capacity, voltage etc. Finally, v Metadata, which is used to organize and interpret all the other kind of data. All these data are collected from several sources such as meters, sensors, devices, substations, mobile data terminals, control devices, intelligent electronic devices, distributed energy resources, customer devices and historical data.
Modern information and communication technologies and advanced operation are used actually to improve smart grid reliability, persistence, efficiency and performance. Service Oriented Architecture SOA all enterprise systems combine a great number of software, each one has its own way to provide services to users.
So the problem is how to manage and maintain all these systems. As a solution, SOA makes software communicate together using a single approach which makes data integration easier and more flexible [ 19 ]. In smart grids, SOA is used essentially on demand systems. ESB brings a lot of benefits to reduce cost and time in term of management, monitoring and divergence of integration [ 20 ]. It plays a very important role in energy management systems in term of data integration, time and cost. In general, CIM help to exchange data with technical grid infrastructure. The CIM become primordial in power systems in order to guarantee the data interoperability, in the case of implementing different applications.
CIM operate in data transformation level, it is used with ESB for the normalization and standardization of the data between smart grid systems. Messaging represents communication systems based on exchanging messages. These messages include data and other information from different applications managed by messaging server [ 21 ].
So there is a need for a developed and scalable data storage mechanism to meet Big Data requirements. Distributed File System DFS is a file system that allows multiple users on multiple machines to share files and storage resources.
NoSQl databases is a new database approach to overcome the limitations of traditional relational SQL databases in the case of massive data. This kind of databases present three architectures: key-value solutions such as Dynamo and Voldemort, column-oriented solutions such as Cassandra and HBase and documents databases solutions such as MongoDB and CouchDB. The grid collects data from different sources and stores it as a huge quantity of dataset that should be easily consumable for analytics.
Analytics has a critical role to make the grid more intelligent, efficient and gainful. Figure 6 presents various kind of analytics in smart grids: i signal analytics which is based on signal processing, ii event analytics which focus on events, iii state analytics which help to have a vision about the state of the grid, iv engineering operations analytics which is responsible of the grid operating side, and v customer analytics which process customer data.
There are actually several models that can combine the various kind of the previous analytics classes such as descriptive, diagnostic, predictive, and prescriptive models. Each model describes an operation side of the grid. Descriptive models are used to describe customers behaviours in demand response programs and provide a basic understanding of their practices.
After customers description, diagnostic models come to understand particular customers behaviours and analyse their decisions. All these previous models are useful to make predictive models to predict customers decisions in the future. Finally, there is prescriptive models which are the high level of analytics in smart grid, because they affect marketing, engagement strategies and the decisions to make [ 22 ].
Big Data processing can be done in two manners: The first is batch processing, which process data in a period of time and is used for data processing without high requirements on response time. The second, is stream processing and is used for real-time applications. This kind of processing requires a very low latency of response. Data visualization has a great role, because it improves the assessment of smart grid.
Actually, there is a great number of visualization techniques based on multivariate high dimensional visualization which gives the ability to use 2D and even the 3D visualisation. But smart grids face enormous variables that complicate data presentation such as 3D Power-map etc.
Scatter diagram, parallel coordinate, and Andrew curve for example resolve the problem of high dimensional data [ 23 ]. Data transmission in Big Data plays a critical role, because it affects all the previous phases.
Many Smart Grid books include "privacy" in their title, but only touch on privacy, with most of the discussion focusing on cybersecurity. Filling this knowledge gap, . Herold, Rebecca. Data privacy for the smart grid / Rebecca Herold and Christine Hertzog. pages cm. Summary: “The Smart Grid is a convenient term to describe.
So it should maintain high bandwidth capacity and speed, data security and privacy etc. Data transmission in smart grids is based on communication technologies as described in " Communication systems ", starting by access network technologies including PLC, ZigBee, WIFI etc. Finnaly, backbone network technologies, which relay on fiber technologies, microwave link, IP-based Wavelength, Division Multiplexing WDM network and other optical technologies. Big Data technologies propose several tools, so utilities should determine which platforms and tools to deploy to meet their goals.
Previous subsections have shown that Big Data life cycle is composed of five phases: data sources, data integration, data storage, data analytics and data visualization. Big Data analytics is the most important step in the life cycle. So, depending on the analytics process, utilities can identify data to acquire and how to store it and even the visualization techniques to use.
Cloud computing components. Cloud computing relies on several concepts that are required for Big Data management in smart grid. Customer data analytics architecture. Customer data analytics use several models depending on the business goals. Proposed architecture for customer data analytics. Big Data implementation can be done using several tools depending on the targeted customer data analytics. Electrical companies should consider certain amount of precautions to choose the right analytics solutions. There are a lot of criterias to take into account in term of speed of computation, compatibility, graphic capabilities, possibility to work on the cloud etc.
For decision making applications, the Analytic Hierarchy Process AHP is considered one of the most popular MCDM methods, because it takes in consideration the quantitative and qualitative performances. The AHP model can be used for the Big Data analytics platform selection based on criteria definition including technical, social, cost and policy perspectives [ 24 ].
Table 1 describes Big Data technical perspectives, including hardware and resources configuration requirements [ 24 ]. Big Data solutions have large amount of challenges in term of storing and processing. Thus, utilities should be aware of all Big Data requirements before implementing it. Big Data solutions require high volume of data storage and high velocity in processing. So, before installing these technologies, utilities should ensure all hardware requirements. The most essential components of a Big Data system are the processing frameworks and processing engines which are responsible for computing over data.
Table 2 provides hardware requirements for the most popular Big Data processing frameworks: Hadoop, Storm, Spark and Flink. To run these Big Data engines, especially for real-time processing, utilities should dedicate additional funds and resources.
Cloud computing helps electrical companies to overcome power and cost requirements and bring a great number of benefits. Cloud computing solves many problems related to Big Data management for smart grids. It helps utilities to ensure the flexibility, agility and efficiency in terms of saving cost, energy and resources [ 25 ].
The use of cloud computing in smart grid brings enormous benefits, due to redundancy, rollback recovery and multi-location data backup which increase data fault tolerance and security [ 13 ]. Cloud computing is based on service models. These models can be offered in public, private, or hybrid manner: 1 software as a service SaaS which provides applications and make them available to customers over the Internet, 2 platform as a service PaaS delivers hardware and software tools and gives customers the ability to create their own applications, 3 infrastructure as a service IaaS offers hardware, software, servers, and other IT infrastructure components over the Internet, 4 data as a service DaaS allows customers in addition to run applications to store data on-line, 5 communication-as-a-service CaaS is useful for messaging tools including voice over IP VoIP , instant messaging IM , and video conferencing, and 6 monitoring as a service MaaS is used for security services to ensure a third party security [ 26 ].
In this section, the focus will be on customers data analytics, because it involves the smart consumers concept, which makes consumers as potential producers of clean energy, players in their consumption and also main actors in production and consumption balancing. Customer data analytics is a great opportunity for utilities to understand customer behaviour better, and be able to make strategic decisions. Big Data analytics of customers data become a necessity and not a choice for electrical companies.
Consumers are participating in smart grids as end customers through smart meters that offer them better control of their own consumption. Demand Response DR programs are used by utilities to obtain real-time information of the demand curves in the various points of consumption in order to calibrate and prognosticate more precisely. Improving customer engagement is among the motivations of DR, because it helps utilities interact with the customers energy needs even during power outage. Dynamic pricing is also involved by DR; consumption monitoring avoid usage in peak time, so customers can check prices in real time and adapt their usage according to the electrical bills [ 12 ].
All of which is only possible using customer data analytics techniques as shown in Fig. Customers data is in the order of Terabytes and in a variety of formats. So, it requires high velocity, scalability and fault tolerance in data processing, storage and visualization. Big Data implementation can be done using several tools, but the analytics tools are the most critical in business choice. Figure 9 provides several Big Data technologies that can be used to manage smart grid data. The variety of customer data sources smart meters, devices, historical data, etc. Messaging tools are the most efficient for raw data integration and hence can be used for customer data integration.
Batch processing tools Big data analytics offers a great number of methods to process data starting from batch processing. Hadoop [ 27 ] is a suitable choice for batch analytics for smart grid. Since smart grid systems are distributed geographically, distributed file systems are very useful for it. Real time processing tools Real time processing is fast in term of execution than batch processing, because it handles data with high velocity requirements using stream processing or complex event processing systems.
Real time processing can be implemented using several solutions such as S4, Splunk, Storm etc. Storm can be used with Kafka for data integration and and Hbase for data storage. Hybrid processing tools Hybrid processing can handle both batch and real time processing. Spark [ 30 ] is a framework used for batch processing, but it has also real time processing solution with Spark streaming. All that make Spark meet Big Data requirements in smart grid. Spark streaming uses real time complex event processing engine to handle velocity issues.
Flink is efficient in machine learning, because it adopts its own machine learning library called FlinkML. Smart grid systems collect huge quantity of datasets to bring smartness to the gird. In the same time this present challenges for utilities to deal with the nature, the distribution and the real-time constraints of the collected data.
In this paper we have presented an overview of the opportunities, concepts and challenges of data management in smart grids and summarized the Big Data technologies and mechanisms that can be used to handle smart grid requirements including processing, storage and even visualization.
We also provided the steps, tools and technical requirements for implementing and deploying Big Data technologies for smart grids in order to have an efficient and scalable data management. Wang W, Lu Z. Cyber security in the smart grid: survey and challenges. Comput Netw. Enabling the integrated grid: leveraging data to integrate distributed resources and customers.
Agarwal V, Tsoukalas LH. Smart grids: importance of power quality. In: Proceedings of first international conference on energy-efficient computing and networking. Berlin; Amin SM. Smart grid: overview, issues and opportunities.
Advances and challenges in sensing, modeling, simulation, optimization and control. Eur J Control. Special attention should be given to the right of portability that, we believe, would be of great importance in the context of Energy Providers relating to competition and switching from one Energy provider to another.
In Belgium, Atrias is the clearing house that has taken this in charge. The right of Portability applies without prejudice of the right to be forgotten and does not apply to processing necessary for the performance of a task carried out in the public interest or in the exercise of official authority vested in the controller. In conclusion, the massive rollout of smart meters across the European Union will bring significant benefits but will also introduce challenges to the protection of personal data.
Next to the European legislation, most of the Members States are also building a legal framework for the introduction of smart meters. About GDPR. Related Articles. GDPR Nederlands. Furthermore, since smart meters can be remotely controlled via the network, the system might also become a preferred target for terrorist attacks in the form of switching off whole areas from electricity or other energy sources.
Therefore, compliance with data security regulations is essential when designing smart metering systems. However, due to the high inconsistency of the implementation of the security requirements among the Member States and the lack of an international standard, maintaining data security within a smart metering system will become a tough challenge for an internationally operating grid provider. Due to the widespread implications of the implementation and operation of the new metering technology, a stringent and intense evaluation of its compliance with data protection principles and regulations is necessary.
Such evaluation has to initially assess whether the processing of the metering data by the data controller can be performed in accordance with data protection regulations. In another step, possible legal bases for a transfer of the metering data have to be analysed. Moreover, its compliance with core principles of the law such as data minimization, legitimate data processing, transfer only for specific purposes, and data storage being no longer than absolutely necessary for such purposes has to be examined as well.
Accordingly, legislative bodies of countries which are in the course of implementing the new technology should devote attention to the privacy issues connected to smart metering and not ignore or overlook its interferences, infringements, and dangers with respect to data privacy rights. The initial implementation efforts taken in the Netherlands impressively demonstrate that privacy implications of smart metering should be seriously taken into account from the very beginning of the implementation process. If such issues remain unsolved or even ignored by lawmakers, then the implementation process might be suddenly stopped and sent back to the beginning.
This is exactly what happened in the Netherlands, where the current, completely modified draft of the implementation package is the outcome of protests by consumer associations; as a result, the implementation was changed to provide consumers with the right to refuse installation of a smart meter. This concept, on the one hand, reflects legitimate consumer interests, but, on the other hand, makes it more difficult for the DSOs to realize the purposes and goals of the new technology, which are namely the improvement of energy efficiency, grid quality, and stability as well as customer information.
A successful macro-economic implementation, indeed, is depending on the installation of the new technology at the vast majority of the measuring points. Despite the model of the Netherlands, a privacy friendly implementation of smart metering does not necessarily have to require the explicit consent of every consumer to the installation of a smart meter, as other legal bases provided for by European data protection legislation are available, as discussed in detail below.
For purposes of assessing the legal provisions covering the implementation of Smart Metering, three different regulations have to be considered. The European Court of Human Rights has interpreted these provisions to develop a right to data protection and a catalogue of requirements by which to measure applicable national legislation.
These directives recommend the implementation of intelligent metering systems, which may be subject to an economic assessment of the long-term costs and benefits to the markets and the individual consumer. This assessment should be finalized no later than 3 September In case the results of the economic assessment are positive, paragraph 2 of Annex 1 of the Electricity Directive stipulates that at least 80 per cent of consumers shall be equipped with smart meters by the year A similar provision has also been enacted for the implementation of smart meters for measuring the consumption of natural gas.
Closely connected to the implementation of smart metering systems are provisions included in Annex 1, para. Furthermore, they shall be able by explicit agreement and free of charge to give any registered supply undertaking access to its metering data, as well as be properly informed of their actual electricity consumption and its costs, which shall be ensured frequently enough in order to enable them to actively regulate their own electricity consumption and use energy move efficiently.
Therefore, the provisions of the Data Protection Directive are applicable even if more than one person lives in a household equipped with a smart meter. According to Article 6 of the Data Protection Directive, personal data must be processed fairly and lawfully, collected for specific and legitimate purposes and may not be further processed in a way incompatible with those purposes.
In addition, data processing must be adequate, relevant, and not excessive in relation to the purposes for which the data are collected. With respect to smart metering, the processing may be based on various purposes such as the improvement of energy efficiency, metering accuracy, customer information, grid stability, as well as timely billing. Therefore, the data collected by smart meters could be legitimately processed for different purposes and, as a consequence, might be subject to different restrictions of processing and transmission.
The processing of personal data, however, does not only have to comply with the principles mentioned above, but also has to be justified under Article 7 of the Data Protection Directive. According to this provision, processing of personal data may only be legitimate if one of the following conditions is satisfied listed here in order of apparent suitability for smart metering : The addressee of the regulations concerning the legitimacy of processing is the controller of the data application.
With respect to smart metering, the DSO companies certainly qualify as controllers, since they as the owners of the meters determine the purposes and means of the processing of the data; the DSO also implicitly decides which kinds of personal data are collected and are transferred to other parties. Such parties may include energy suppliers, governmental agencies, and other third parties having an interest in the metering data.
The lawfulness of the processing and transmission of metering data has to be determined for each type of party that will process the data. We will consider each legal basis in turn. The existence of a legal obligation or legal authorization of the data application by statute is generally the most favourable and strongest legal basis for data processing.
With respect to the implementation and operation of a smart metering system, no such legislative provision currently exists on the European level. The Electricity Directive itself does not qualify, since it requires implementation by the national laws of the Member States. Therefore, it is currently up to the Member States to provide an explicit legal foundation, which could be included within statutes regulating the national electricity and gas markets.
National statutes mandating the implementation of smart metering have been enacted in some EU Member States such as in Italy and Sweden. In Austria, the statute only obliges utility companies to ensure delivery of energy at a low price and with high quality. A draft statute regulating the implementation of smart meters and implementing the EU energy directive passed the Austrian parliament in November , but lacks provisions on important issues of data protection, such as the permissible period of storage of the data or a determination of the legitimate interval for measuring the consumption data.
In order to comply with applicable legal requirements under Article 8 of the European Convention on Human Rights as set forth in the case law of the European Court of Human Rights, 6 any legislation providing a legal basis for the processing of smart metering data would have to meet the following conditions: Due to these stringent and detailed requirements that s t atutes must meet, there are a number of difficult problems to be solved for any legislation to avoid being found incompatible with Article 8. The processing as well as the transfer of data can also be legitimate if such processing is necessary for the controller to fulfil a duty or obligation set forth within a contract entered into with the data subject ie, the energy consumers.
The analysis of whether any contractual duties exist which require the processing of smart metering data, however, must be preceded by the clarification that data subjects have usually entered into two or three separate contractual relationships with the various players in the energy supply market.
Legitimacy of processing must therefore be analysed individually for any duty or obligation, as well as for each contractual relationship. According to Article 25 of the Electricity Directive, the DSO is responsible for the long-term ability of the system grid to meet reasonable demands for the distribution of electricity and for operating and maintaining under economic conditions a secure, reliable, and efficient electricity distribution system with due regard for energy efficiency.
This could be seen as a codification of the general contractual duty of a DSO to ensure a high level of grid quality as well as grid stability and security.
Therefore, the processing of smart metering data by the data controller DSO could be justified by its necessity for the performance of a contractual duty. Nonetheless, the processing could only be justified to the extent that metering data are required for the purpose of ensuring grid stability. Furthermore, the question arises whether the metering data thereby necessarily have to be processed in a personalized fashion.
By means of the implementation of new metering technology, suppliers may offer their customers various types of energy tariffs eg, so that the bill might not only depend on energy usage, but also on whether consumption takes place during the day-time, night-time, peak or off-peak, etc. Such new tariff models would require the transfer of detailed metering data to the energy supplier, which needs those data to provide its customers with accurate and timely bills.
This does not only represent a core goal of the third energy package of the EU, but also a main contractual duty of the energy supplier towards its customers. Accordingly, it could be argued that the energy supplier may lawfully receive and use the detailed metering data for billing purposes, in case sophisticated tariffs as described above are agreed with the respective customer which also requires personalized data. However, the wording of Article 7 b of the Data Protection Directive, which legitimates data processing only if necessary for the performance of a contract entered into by the controller and the data subject, could conflict with this argumentation.
As already stated above, the DSO is the data controller for the smart metering application, so that any processing or transmission of the metering data from the DSO to the energy supplier cannot, at first sight, be subsumed under the fulfilment of a contractual duty. Nevertheless, it could be argued that Article 7 b of the Directive should still apply due to a necessarily broad interpretation of this provision as a reaction to the consistent support and promotion of the unbundling of DSOs and energy suppliers by European legislation in the energy sector over the last decade.
Since the DSO and the energy supplier have to be legally or at least economically separated and independent even within the same corporate group, Article 7 b has to be understood to the effect that a transfer of data might also be legitimate on the ground of performance of a contractual duty, if the transfer is necessary for a third party in order to fulfil its contractual duty towards the data subject.