A noise reduction method based upon statistical analysis for the detection of weak signals in discrete data

Abstract

While searching sets of discrete data contaminated heavily by noise for weak signals of correlated data points we have made use of statistic quantities to detect correlations between data points. The signal consists of linearly correlated three dimensional discrete data, forming the trajectory. The noise data are independent, uniformly distributed three dimensional random vectors, originating in spurious data whose origin is not known to us. The method is widely usable whenever the state of a stochastically driven dynamical system is to be estimated from a series of sensor data sets received at discrete instants of time, i.e. when three dimensional discrete noise corrupted data are being searched for trajectories. <br /> We divide the data set into representative sub-samples and map the data contained in such a sub-set to one value. Mappings we investigated include the eigenvalues of the covariance matrix, the dispersion of the angle between time and spatial axes, the Kolmogorov-Smirnov one-sample test and linear regression. Using Monte Carlo simulation we obtained probability density distributions for all mappings and for both cases `noise' and `noise plus trajectory' each. As we test the data sub-sets for the hypothesis `noise', we obtained the statistical type I and II errors as an indicator for the quality of a mapping. The type I error corresponds to the false alarm probability, i.e. the detection of a non-existent trajectory, and the type II error corresponds to 1 minus the detection probability, which is the probability of not detecting an existent trajectory. The aim is to achieve the highest possible detection probability while reducing the false alarm probability at the same time. We obtained receiver operating characteristics (ROC) which shows the detection probability as a function of the false alarm rate. This curve is an indicator of the quality of any detection system. In order to obtain the ROC we simulated 10^7 cut-outs containing `noise' and `noise plus trajectory', mapped each sub-sample to one value and compared the probability density distributions for the two cases. From these distributions we obtained the statistical type I and II errors corresponding to the false alarm probability and 1-detection probability. To maximize the detection probability while reducing the false alarm probability at the same time we composed a weighted combination of all mappings. Each single mapping contributes to the weighted combination according to its quality. Even mappings that are clearly not suitable as a stand-alone detector contribute and improve the performance of the combination. <br /> An application to radar data with a naval radar system have shown satisfying results to prove the applicability of the method to real data.