Development and evaluation of Artificial Intelligence techniques for IoT data quality assessment and curation

Abstract

Nowadays, data is becoming the new fuel for economic wealth and creation of novel and profitable business models. Multitude of technologies are contributing to an abundance of information sources which are already the baseline for multi-millionaire services and applications. Internet of Things (IoT), is probably the most representative one. However, for an economy of data to actually flourish there are still several critical challenges that have to be overcome. Among them, data quality can become an issue when data come from heterogeneous sources or have different formats, standards and scale. Improving data quality is of utmost importance for any domain since data are the basis for any decision-making system and decisions will not be accurate if they are based on inadequate low-quality data. In this paper we are presenting a solution for assessing several quality dimensions of IoT data streams as they are generated. Additionally, the solution described in the paper actually improves the quality of data streams by curating them through the application of Artificial Intelligence techniques. The approach followed in our work has been to append data quality information as metadata linked to each individual piece of curated data. We have leveraged linked-data principles and integrated the developed AI-based IoT data curation mechanisms within a Data Enrichment Toolchain (DET) that employs the NGSI-LD standard to harmonize and enrich heterogeneous data sources. Furthermore, we have evaluated our design under experimental research conditions, achieving a robust compromise between functionality and overhead. Besides, it demonstrates a stable and scalable performance.