Outlier Detection Method for Identifying Outliers that are not in Gaussian Distribution

Abstract

The most statistical methods demand outlier (noise) in Gaussian distribution. When outliers are not in Gaussian distribution, these methods produce bias results. We introduce an outlier detection method that performs best when the outliers are in non Gaussian distribution. The method is non-parametric and based on properties of arithmetic progression (AP). If the number of elements in AP is n, the maximumelement is a max, the minimumelement is aOT,„, andthe sum of all elements is S„.ThenRmax = amax amm an(i/?min = amax ,a™fnis always equal to 2/n. Usually, /?„,ajc>2/«implies that S n ~ amin*n a max*n $n — the maximumelementis an outlier and Rmin>2/mmp\ies that the minimum elementis an outlier. The value 2/n is nonparametric and always between 0 and 1. If t is a threshold relevant to the considered domain, the value 2/n + t can be used to identify significant outliers where 0 < t <1 -2/n. The method identifies one outlier at a time and continuous application of the method allows detection of multiple outliers. The algorithm was tested using several artificial and real data sets. The real data sets were the data which were automatically recorded with a frequency of twelve data points per day from a biogas plant, over a period of seven months. Among the different parameters, we selected the H2 content, which we expected to maintain linear behavior during the stable operation. When the outliers are non-Gaussian, the Grubbs’ test locates 0% - 17% as significant outliers at the significance level of 0.05,. With the new method, there was t, which was capable of locating more outliers than Grubbs’ test