In data analysis, anomaly detection (also referred to as outlier detection and sometimes as novelty detection) is generally understood to be the identification of rare items, events or observations which deviate significantly from the majority of the data and do not conform to a well defined notion of normal behavior. Such examples may arouse suspicions of being generated by a different mechanism, or appear inconsistent with the remainder of that set of data.
Anomaly detection finds application in many domains including cybersecurity, medicine, machine vision, statistics, neuroscience, law enforcement and financial fraud to name only a few. Anomalies were initially searched for clear rejection or omission from the data to aid statistical analysis, for example to compute the mean or standard deviation. They were also removed to better predictions from models such as linear regression, and more recently their removal aids the performance of machine learning algorithms. However, in many applications anomalies themselves are of interest and are the observations most desirous in the entire data set, which need to be identified and separated from noise or irrelevant outliers.
Three broad categories of anomaly detection techniques exist:
Supervised anomaly detection techniques require a data set that has been labeled as "normal" and "abnormal" and involves training a classifier. However, this approach is rarely used in anomaly detection due to the general unavailability of labelled data and the inherent unbalanced nature of the classes. Semi-supervised anomaly detection techniques assume that some portion of the data is labelled. This may be any combination of the normal or anomalous data, but more often than not, the techniques construct a model representing normal behavior from a given normal training data set, and then test the likelihood of a test instance to be generated by the model. Unsupervised anomaly detection techniques assume the data is unlabelled and are by far the most commonly used due to their wider and relevant application.
Definition and terminology
Anomaly detection is the identification of observations, events, or patterns that depart substantially from normal behavior expected in a particular setting. There is no single operational definition for anomaly detection that applies to every domain, because the notion of normality depends on the data, the representation of the observations, the context in which they occur, and the assumptions made by the detection method.
An anomaly is not necessarily a data error. Depending on the application, it may represent a measurement or recording error, a change in the process that generated the data, an unusual but valid observation, or an event of interest such as fraud, equipment failure, or a security intrusion. Anomalies are also not defined solely by rarity or by distance from other observations. Different methods characterize normality using probability distributions, distances or local densities, cluster membership, boundaries around normal data, prediction errors, reconstruction errors, or other model-specific criteria.
Many anomaly-detection methods return a numerical anomaly score rather than an immediate categorical label. The score represents the degree to which an observation departs from the fitted model of normality and is converted into a binary decision by applying a decision threshold. Consequently, whether an observation is labelled anomalous depends on the model of normality, the available context, and the decision rule used.
Types of anomalies
A widely used classification distinguishes three types of anomalies:
A point anomaly is an individual observation that is considered anomalous relative to the rest of the data e.g., an unusually large transaction may be anomalous when compared with the other transactions in a dataset. A contextual anomaly, also called a conditional anomaly, is anomalous only within a particular context. An observation may therefore be normal under one set of circumstances but anomalous under another e.g., a particular temperature may be normal in summer but unusual in winter. Context may be temporal, spatial, demographic, or otherwise defined by the application. A collective anomaly is a related collection of observations that is anomalous as a whole, even when its individual observations are not anomalous by themselves e.g., an unusual subsequence in a time series or an unexpected pattern of activity in a computer network.
The appropriate category depends partly on the structure of the data. Point-anomaly methods may be sufficient for independent observations, whereas time series, spatial data, sequences, and graphs often require contextual or collective relationships to be modelled explicitly.
Related terms
The terms anomaly, outlier, and novelty are sometimes used interchangeably, but their usage varies among statistics, machine learning, signal processing, and individual application domains. An outlier commonly denotes an observation that appears inconsistent with the other observations in a dataset. Outlier-detection methods often allow such observations to be present in the data used to fit the model.
In novelty detection, the training data are generally assumed to represent normal behavior, and previously unseen test observations are assessed against the resulting model of normality. Novelty detection is consequently often formulated as a form of one-class classification. These distinctions are not applied consistently throughout the literature, and the broader term anomaly detection is frequently used to include all of these settings.
History
Intrusion detection The concept of intrusion detection, a critical component of anomaly detection, has evolved significantly over time. Initially, it was a manual process where system administrators would monitor for unusual activities, such as a vacationing user's account being accessed or unexpected printer activity. This approach was not scalable and was soon superseded by the analysis of audit logs and system logs for signs of malicious behavior.
By the late 1970s and early 1980s, the analysis of these logs was primarily used retrospectively to investigate incidents, as the volume of data made it impractical for real-time monitoring. The affordability of digital storage eventually led to audit logs being analyzed online, with specialized programs being developed to sift through the data. These programs, however, were typically run during off-peak hours due to their computational intensity.
The 1990s brought the advent of real-time intrusion detection systems capable of analyzing audit data as it was generated, allowing for immediate detection of and response to attacks. This marked a significant shift towards proactive intrusion detection.
As the field has continued to develop, the focus shifted to creating solutions that can be efficiently be implemented across large and complex networks, adapting to the ever-growing variety of security threats and the dynamic nature of modern computing infrastructures.
Applications Anomaly detection is applicable in a very large number and variety of domains, and is an important subarea of unsupervised machine learning. As such it has applications in cyber-security, intrusion detection, fraud detection, fault detection, system health monitoring, event detection in sensor networks, detecting ecosystem disturbances, defect detection in images using machine vision, medical diagnosis and law enforcement.
Intrusion detection
Anomaly detection was proposed for intrusion detection systems (IDS) by Dorothy Denning in 1986. Anomaly detection for IDS is normally accomplished with thresholds and statistics, but can also be done with soft computing, and inductive learning. Types of features proposed by 1999 included profiles of users, workstations, networks, remote hosts, groups of users, and programs based on frequencies, means, variances, covariances, and standard deviations. The counterpart of anomaly detection in intrusion detection is misuse detection.
Fintech fraud detection Anomaly detection is vital in fintech for fraud prevention.
Preprocessing Preprocessing data to remove anomalies can be an important step in data analysis, and is done for a number of reasons. Statistics such as the mean and standard deviation are more accurate after the removal of anomalies, and the visualisation of data can also be improved. In supervised learning, removing the anomalous data from the dataset often results in a statistically significant increase in accuracy.
Video surveillance Anomaly detection has become increasingly vital in video surveillance to enhance security and safety. With the advent of deep learning technologies, methods using Convolutional Neural Networks (CNNs) and Simple Recurrent Units (SRUs) have shown significant promise in identifying unusual activities or behaviors in video data. These models can process and analyze extensive video feeds in real-time, recognizing patterns that deviate from the norm, which may indicate potential security threats or safety violations. An important aspect for video surveillance is the development of scalable real-time frameworks. Such pipelines are required for processing multiple video streams with low computational resources.
IT infrastructure In IT infrastructure management, anomaly detection is crucial for ensuring the smooth operation and reliability of services. These are complex systems, composed of many interactive elements and large data quantities, requiring methods to process and reduce this data into a human and machine interpretable format. Techniques like the IT Infrastructure Library (ITIL) and monitoring frameworks are employed to track and manage system performance and user experience. Detected anomalies can help identify and pre-empt potential performance degradations or system failures, thus maintaining productivity and business process effectiveness.
IoT systems Anomaly detection is critical for the security and efficiency of Internet of Things (IoT) systems. It helps in identifying system failures and security breaches in complex networks of IoT devices. The methods must manage real-time data, diverse device types, and scale effectively. Garg et al. have introduced a multi-stage anomaly detection framework that improves upon traditional methods by incorporating spatial clustering, density-based clustering, and locality-sensitive hashing. This tailored approach is designed to better handle the vast and varied nature of IoT data, thereby enhancing security and operational reliability in smart infrastructure and industrial IoT systems.
Petroleum industry Anomaly detection is crucial in the petroleum industry for monitoring critical machinery. A 2015 paper proposed a novel segmentation algorithm using support vector machines to analyze sensor data for real-time anomaly detection.
Oil and gas pipeline monitoring In the oil and gas sector, anomaly detection is not just crucial for maintenance and safety, but also for environmental protection. Aljameel et al. propose an advanced machine learning-based model for detecting minor leaks in oil and gas pipelines, a task traditional methods may miss.
Methods Many anomaly detection techniques have been proposed in literature. The performance of methods usually depend on the data sets. For example, some may be suited to detecting local outliers, while others global, and methods have little systematic advantages over another when compared across many data sets.Anomaly detection benchmark data repository of the Ludwig-Maximilians-Universität München; Mirror at University of São Paulo. Almost all algorithms also require the setting of non-intuitive parameters critical for performance, and usually unknown before application. Some of the popular techniques are mentioned below and are broken down into categories:
Statistical
Parameter-free