Which tools can help to thwart “out of memory” issues during analysis of big data?

Nowadays, data management and data analysis are more and more present in the scientist’s life. Indeed, there is an increase of cutting-edge technologies in the analytical field, which provide high quality data but in high amount. In some cases, it is possible increase the data analysis power of a basic computer by parallelizing functions. However, central processing unit (CPU) computing is quickly limited. Another option is to perform analyses with the Graphics Processing Unit (GPU). However, it requires a high knowledge of coding and most usual toolboxes do not support GPU computing. That is why some chemometric strategies have to be developed to be able to analyse such amount of data with an accessible software, such as the MATLAB® environment, on an affordable workstation.
In this study, several chemometrics algorithms will be evaluated for the data analysis of an infrared chemical image of a pharmaceutical tablet. The image (3.5 million of spectra of 767 wavenumbers) has been acquired on a FT-IR Cary 670/620 Agilent series microscope (Agilent Technologies). The file size of the raw image data is ~40GB and is impossible to analyse even on a workstation due to “Out of Memory” error. The goal of this work is to show that specific chemometrics strategies coupled to a little bit of coding enables the analysis keeping only the relevant information by reducing the size of the matrix in a smart way.