Cloud Platform using Big Data and HPC Technologies for Distributed and Parallels Treatments

Smart farming is one of the most diverse researches. In addition, the quantity of data to be stored and the choice of the most efficient algorithms to process are important elements in this field. The storage of collected data from Internet of Things (IoT), existing on distributed, local databases and open data needs particular infrastructure to federate all these data in order to make complex treatments. The storage of this wide range of data that comes at high frequency and variable throughput is particularly difficult. In this paper, we propose the use of distributed databases and high-performance computing architecture in order to exploit multiple re-configurable computing and application specific processing such as CPUs, GPUs, TPUs and FPGAs efficiently. This exploitation allows an accurate training for an application to machine learning, deep learning and unsupervised modeling algorithms. The last ones are used for training supervised algorithms on images when it labels a set of images and unsupervised algorithms on IoT data which are unlabeled with variable qualities. The processing of data is based on Hadoop 3.1 MapReduce to achieve parallels processing and use containerization technologies to distribute treatments on Multi GPU, MIC and FPGA. This architecture allows efficient treatments of data coming from several sources with a cloud high-performance heterogeneous architecture. The proposed 4 layers infrastructure can also implement FPGA and MIC which are now natively supported by recent version of Hadoop. Moreover, with the advent of new technologies like Intel Movidius; it is now possible to deploy CNN at fog level in the IoT network and to make inference with the cloud and therefore limit significantly the network traffic that result on reducing the move of large amounts of data to the cloud.