An Overview of Outlier Detection Methods
Keywords:
Particleboard, urea-formaldehyde resin, curing agents, modifier-curing agent, copper composites, ionic liquid, alumina, EQCM and SEM, Machine learning, Outlier, Outlier detectionAbstract
One of the first important steps in achieving informed data analysis is detection of outliers. Even in cases where the final values are often considered to be incorrect calculations or noise, they can still provide very important information in some cases. Therefore, it is very important to detect them before modeling and analysis. In this paper, we present a structured and comprehensive review of residual detection research. There are many different methods, hence the purpose of this article is to help the novice researcher to formulate his ideas and gain an easier understanding of the various lines of research in which research has been conducted on this topic.
References
S. Ounacer, H. Ait El Bour, Y. Oubrahim, M. Y. Ghoumari, M. Azzouazi (2018) Using Isolation Forest in anomaly detection: the case of credit card transactions. 6(2), 394-400. https://doi.org/10.1007/s11779-018-0135-6
X. Li, Y. Cai, W. Zhu (2018) Power Data Cleaning Method Based on Isolation Forest and LSTM Neural Network. 11067. https://doi.org/10.1007/978-3-030-00018-9_47
E. Habler, A. Shabtai (2018) Using LSTM encoder-decoder algorithm for detecting anomalous ADS-B messages. 78, 155-173. https://doi.org/10.1016/j.cose.2018.05.009
Z. Liu, X. Liu, J. Ma, H. Gao (2018) An Optimized Computational Framework for Isolation Forest. 13 pages. https://doi.org/10.1155/2018/2318763
Chun-Hui X., S. Chen, Cong-Xiao B., L. Xing (2018) Anomaly Detection in Network Management System Based on Isolation Forest. 56-60. https://doi.org/10.1109/ICNISC.2018.00019
X. Tao, Y. Peng, F. Zhao, P. Zhao, Y. Wang (2018) A parallel algorithm for network traffic anomaly detection based on Isolation Forest. 14(11). https://doi.org/10.1177/1550147718814471
L. Liao, B. Luo (2018) Entropy Isolation Forest Based on Dimension Entropy for Anomaly Detection.
A. I. Filippov, A. V. Iuzbashev, A. S. Kurnev (2018) User authentication via touch pattern recognition based on isolation forest. 1485-1489. https://doi.org/10.1109/EIConRus.2018.8317378
L. E. Kurnianingsih, E. Nugroho, L. Widyawan, L. Lazuardi, A. S. Prabuwono (2018) Detection of Anomalous Vital Sign of Elderly Using Hybrid K-Means Clustering and Isolation Forest. 0913-0918. https://doi.org/10.1109/TENCON.2018.8650457T
Y. -J. Wu, Zhang, X. Tang (2018) Isolation Forest Based Method for Low-Quality Synchrophasor Measurements and Early Events Detection. 1-7. https://doi.org/10.1109/SmartGridComm.2018.8587434
M. U. Togbe, et al (2020) Anomaly Detection for Data Streams Based on Isolation Forest Using Scikit-Multiflow. https://doi.org/10.1007/978-3-030-58811-3_2
M. U. Togbe, Y. Chabchoub, A. Boly, R. Chiky (2020) Etude comparative des méthodes de détection d'anomalies. 109–120.
Y. Hara, Y. Fukuyama, K. Murakami, T. Iizaka, T. Matsui (2020) Fault Detection of Hydroelectric Generators using Isolation Forest. https://doi.org/10.23919/SICE48898.2020.9240331
H. Ma, B. Ghojogh, M. N. Samad, D. Zheng, M. Crowley (2020) Isolation Mondrian Forest for Batch and Online Anomaly Detection. https://doi.org/10.1109/SMC42975.2020.9283073
P. Karczmarek, A. Kiersztyn, W. Pedrycz (2020) Fuzzy Set-Based Isolation Forest. https://doi.org/10.1109/FUZZ48607.2020.9177718
S. Liu, Z. Ji, Y. Wang (2020) Improving Anomaly Detection Fusion Method of Rotating Machinery Based on ANN and Isolation Forest. https://doi.org/10.1109/CVIDL51233.2020.00-23
N. L. Fitriyani, M. Syafrudin, G. Alfian, A. Fatwanto, S. L. Qolbiyani, J. Rhee (2020) Prediction Model for Type 2 Diabetes using Stacked Ensemble Classifiers. https://doi.org/10.1109/DASA51403.2020.9317090
J. Park, V. R. Surabhi, P. Krishnamurthy, S. Garg, R. Karri, F. Khorrami (2020) Anomaly Detection in Embedded Systems Using Power and Memory Side Channels. https://doi.org/10.1109/ETS48528.2020.9131596
L. Meneghetti, M. Terzi, S. Del Favero, G. A. Susto, C. Cobelli (2020) Data-Driven Anomaly Recognition for Unsupervised Model-Free Fault Detection in Artificial Pancreas. 1, 33-47. https://doi.org/10.1109/TCST.2018.2885963
A. Dridi, C. Boucetta, S. E. Hammami, H. Afifi, H. Moungla (2021) STAD: SpatioTemporal Anomaly Detection Mechanism for Mobile Network Management. 18(1), 894-906. https://doi.org/10.1109/TNSM.2020.3048131
C. Liu, S. Yong, X. Wang, X. Zhang (2020) A Multi-feature Anomaly Detection Method Based on AETA ULF Electromagnetic Disturbance Signal. 1103-1108. https://doi.org/10.1109/ITNEC48623.2020.9085032
C. Kromer-Edwards, M. Castanheira, S. Oliveira (2020) Year, Location, and Species Information In Predicting MIC Values with Beta-Lactamase Genes. 1383-1390. https://doi.org/10.1109/BIBM49941.2020.9313331
N. Fang, X. Fang, K. Lu (2022) Anomalous Behavior Detection Based on the Isolation Forest Model with Multiple Perspective Business Processes. 3640. https://doi.org/10.3390/electronics11213640
M. Chater, A. Borgi, M. T. Taieb, K. Sfar-Gandoura, M. I. Landoulsi (2022) Fuzzy Isolation Forest for Anomaly Detection. 207, 916-925. https://doi.org/10.1016/j.procs.2022.09.147
J. Choudhury, C. Shi (2022) Enhanced Performance of Finite Boundary Isolation Forest (FBIF) for Datasets with Standard Distribution Properties. 1-5. https://doi.org/10.1109/ICECET55527.2022.9873022
P. R. Reddy, A. S. Kumar (2022) Credit Card Fraudulent Transactions Prediction Using Novel Sequential Transactions by Comparing Light Gradient Booster Algorithm Over Isolation Forest Algorithm. 563-567. https://doi.org/10.1109/ICIPTM54933.2022.9754211
E. Marcelli, T. Barbariol, V. Savarino, A. Beghi, G. A. Susto (2022) A Revised Isolation Forest Procedure for Anomaly Detection with a High Number of Data Points. 1-5. https://doi.org/10.1109/LATS57337.2022.9936964
L. Fan, J. Ma, J. Tian, T. Li, H. Wang (2021) Comparative Study of Isolation Forest and LOF Algorithm in Anomaly Detection of Data Mining. 1-5. https://doi.org/10.1109/ICBAR55169.2021.00008
F. Feng, Z. Liu, J. Zhang (2022) Detection of GPS Abnormal Data of Sanitation Vehicles Based on Isolation Forest Algorithm. 460-463. https://doi.org/10.1109/ICDACAI57211.2022.00097J
Su, J. Li (2022) An Anomaly Detection Algorithm for Multi-dimensional Segmentation Plane Isolation Forest. 89-93. https://doi.org/10.1109/CCET55412.2022.9906369
P. Yu, L. Jia (2022) Wind Power Data Cleaning Based on Autoencoder-Isolation Forest. 803-808. https://doi.org/10.1109/ICPRE55555.2022.9960342
H. H. Kilinc (2022) Anomaly Pattern Analysis Based on Machine Learning on Real Telecommunication Data. 43-48. https://doi.org/10.1109/UBMK55850.2022.9919564
X. Huang, Y. Ren, Y. He, Q. Chen (2022) Malicious Models-based Federated Learning in Fog Computing Networks. 192-196. https://doi.org/10.1109/WCSP55476.2022.10039266
H. R. Kerner, J. B. Adler (2022) Guiding Field Exploration on Earth and Mars with Outlier Detection. 5333-5336. https://doi.org/10.1109/IGARSS46834.2022.9884366
I. Zualkernan, N. Ahmed, A. Elmeligy, A. Abdelnaby, N. Sheta (2022) IoT Sensor Data Consistency using Deep Learning. 198-203. https://doi.org/10.1109/IoTaIS56727.2022.9975955
J. Gajda, J. Kwiecień, W. Chmiel (2022) Machine learning methods for anomaly detection in computer networks. 276-281. https://doi.org/10.1109/MMAR55195.2022.9874341
S. M. Malaek, E. Alipour (2022) Intelligent Flight-Data-Recorders; a Step Toward a New Generation of Learning Aircraft. 1545-1549. https://doi.org/10.1109/CoDIT55151.2022.9804136
Z. A. El Houda, A. S. Hafid, L. Khoukhi (2021) A Novel Machine Learning Framework for Advanced Attack Detection using SDN. 1-6. https://doi.org/10.1109/GLOBECOM46510.2021.9685643
H. M. Kafi, A. S. M. Miah, J. Shin, M. N. Siddique (2022) A Lite-Weight Clinical Features Based Chronic Kidney Disease Diagnosis System Using 1D Convolutional Neural Network. 1-5. https://doi.org/10.1109/ICAEEE54957.2022.9836398
E. Anello, et al. (2022) Anomaly Detection for the Industrial Internet of Things: an Unsupervised Approach for Fast Root Cause Analysis. 1366-1371. https://doi.org/10.1109/CCTA49430.2022.9966158
X. Yang, Y. Zhuang, M. Shi, X. Cao, D. Chen, Y. Tang SPiForest: An Anomaly Detecting Algorithm Using Space Partition Constructed by Probability Density-Based Inverse Sampling. https://doi.org/10.1109/TNNLS.2022.3223342
N. Sugunaraj, J. Vrtis, J. Snyder, P. Ranganathan (2022) False Data Injection Modeling & Detection for Phasor Measurement Units. 1-6. https://doi.org/10.1109/NAPS56150.2022.10012137
P. Premisha, S. Prasanth, M. Kanagarathnam, K. Banujan (2022) An Ensemble Machine Learning Approach for Stroke Prediction. 165-170. https://doi.org/10.1109/SCSE56529.2022.9905215
G. Wang, X. Mao, Q. Zhang, A. Lu Fatigue Detection in Running with Inertial Measurement Unit and Machine Learning. 85-90. https://doi.org/10.1109/ICBCB55259.2022.9802471
R. Singh, R. Deorari Enhancing Collaborative Intrusion detection networks against insider attack using supervised learning technique. 1-6. https://doi.org/10.1109/MysuruCon55714.2022.9972599
T. Yang, Z. Cai, B. Hou, T. Zhou 6Forest: An Ensemble Learning-based Approach to Target Generation for Internet-wide IPv6 Scanning. 1679-1688. https://doi.org/10.1109/INFOCOM48880.2022.9796925
F. Mouret, M. Albughdadi, S. Duthoit, D. Kouamé, J.-Y. Tourneret Robust Estimation of Gaussian Mixture Models Using Anomaly Scores and Bayesian Information Criterion for Missing Value Imputation. 827-831. https://doi.org/10.23919/EUSIPCO55093.2022.9909815
A. Mensi, M. Bicego (2021) Enhanced anomaly scores for isolation forests. 120, 108115. https://doi.org/10.1016/j.patcog.2021.108115
M. Badurowicz, P. Karczmarek, J. Montusiewicz Fuzzy Extensions of Isolation Forests for Road Anomaly Detection. 1-6. https://doi.org/10.1109/FUZZ45933.2021.9494469
T. Ahmed, A. Shah, M. Kolla, R. Yellasiri Reduction of Alert Fatigue using Extended Isolation Forest. 1-5. https://doi.org/10.1109/FABS52071.2021.9702617
C. Y. Priyanto, H. D. Purnomo Combination of Isolation Forest and LSTM Autoencoder for Anomaly Detection. 35-38. https://doi.org/10.1109/ICITech50181.2021.9590143
C. Derse, M. Baghdadi, O. Hegazy, U. Sensoz, H. N. Gezer, M. Nil An Anomaly Detection Study on Automotive Sensor Data Time Series for Vehicle Applications. 1-5. https://doi.org/10.1109/EVER52347.2021.9456629
S. Damodaran, R. Padmanabhan, R. Maahin, S. Gurugopinath A Copula-Driven Unsupervised Learning Framework for Anomaly Detection with Multivariate Heterogeneous Data. 1-6. https://doi.org/10.1109/MLSP52302.2021.9596359
S. Karthik, H. V. Supreetha, S. Sandhya Detection of Anomalies in Time Series Data. 1-5. https://doi.org/10.1109/CSITSS54238.2021.9683715
A. P. Lopes, S. Parshionikar, A. Kale, N. Sharma, A. A. Varghese Comparative Analysis of Deep Learning Techniques For Credit Card Fraud Detection. 1-5. https://doi.org/10.1109/ICAC353642.2021.9697205
T. Komárek, J. Brabec, Č. Škarda, P. Somol Threat Hunting as a Similarity Search Problem on Multi-positive and Unlabeled Data. 2098-2103. https://doi.org/10.1109/BigData52589.2021.9671958
C. McKinnon, J. Carroll, A. McDonald, S. Koukoura, C. Plumley Investigation of anomaly detection technique for wind turbine pitch systems. 277-282. https://doi.org/10.1049/icp.2021.1401
P. Witzig, E. Upenik, T. Ebrahimi Open-Set Person Re-Identification Through Error Resilient Recurring Gallery Building. 245-249. https://doi.org/10.1109/ICIP42928.2021.9506241
R. Chen, Y. Yang, M. Xia Anomaly Detection of Sensor Data Based on 1D Depth Separable Dilated Convolution Neural Network. 240-244. https://doi.org/10.1109/NetCIT54147.2021.00055
G. Lu, C. Duan, G. Zhou, X. Ding, Y. Liu (2021) Privacy-Preserving Outlier Detection with High Efficiency over Distributed Datasets. 1-10. https://doi.org/10.1109/INFOCOM42981.2021.9488710P
Ntambu, S. A. Adeshina (2021) Machine Learning-Based Anomalies Detection in Cloud Virtual Machine Resource Usage. 1-6. https://doi.org/10.1109/ICMEAS52683.2021.9692308
T. Kotsiopoulos (2021) Fault Detection on Bearings and Rotating Machines based on Vibration Sensors Data. 474-483. https://doi.org/10.1109/PIC53636.2021.9686999
W. Liu (2023) D2MIF: A Malicious Model Detection Mechanism for Federated-Learning - Empowered Artificial Intelligence of Things. 10(3), 2141-2151. https://doi.org/10.1109/JIOT.2021.3081606
R. Shrivastava (2021) Comparative study of boosting and bagging based methods for fault detection in a chemical process. 674-679. https://doi.org/10.1109/ICAIS50930.2021.9395905
P. Koch, K. Schekotihin, D. Jannach, B. Hofer, F. Wotawa (2021) Metric-Based Fault Prediction for Spreadsheets. 47(10), 2195-2207. https://doi.org/10.1109/TSE.2019.2944604
A. K. Pathak, S. Saguna, K. Mitra, C. Åhlund (2021) Anomaly Detection using Machine Learning to Discover Sensor Tampering in IoT Systems. 1-6. https://doi.org/10.1109/ICC42927.2021.9500825
A. Alghanmi, A. Yunusa-Kaltungo, R. Edwards (2021) A Comparative Study of Faults Detection Techniques on HVAC Systems. 1-5. https://doi.org/10.1109/PowerAfrica52236.2021.9543158
L. Antony (2021) A Comprehensive Unsupervised Framework for Chronic Kidney Disease Prediction. 126481-126501. https://doi.org/10.1109/ACCESS.2021.3109168
R. F. Mansour, A. E. Amraoui, I. Nouaouri, V. G. Díaz, D. Gupta, S. Kumar (2021) Artificial Intelligence and Internet of Things Enabled Disease Diagnosis Model for Smart Healthcare Systems. 45137-45146. https://doi.org/10.1109/ACCESS.2021.3066365
F. Simmini, M. Rampazzo, F. Peterle, G. A. Susto, A. Beghi (2022) A Self-Tuning KPCA-Based Approach to Fault Detection in Chiller Systems. 30(4), 1359-1374. https://doi.org/10.1109/TCST.2021.3107200
J. Chen, J. Zhang, R. Qian, Junfeng Yuan, Y. Ren (2023) An Anomaly Detection Method for Wireless Sensor Networks Based on the Improved Isolation Forest. https://doi.org/10.3390/app13020702
M. Almansoori, M. Telek (2023) Anomaly Detection using a combination of Autoencoder and Isolation Forest. 25-30. https://doi.org/10.3311/WINS2023-005
L. Utkin, A. Ageev, A. Konstantinov, V. Muliukha (2023) Improved Anomaly Detection by Using the Attention-Based Isolation Forest. 16. https://doi.org/10.3390/a16010019
S. Kabir, A. Shufian, M. S. R. Zishan (2023) Isolation Forest Based Anomaly Detection and Fault Localization for Solar PV System. 341-345. https://doi.org/10.1109/ICREST57604.2023.10070033
P. V. Baviskar, G. Singh, V. N. Patil (2023) Design of Machine Learning-Based Malware Detection Methodologies in the Internet of Things Environment. 1-6. https://doi.org/10.1109/ICONAT57137.2023.10080517
Y. Himeur, F. Fadli, A. Amira (2022) A Two-Stage Energy Anomaly Detection for Edge-based Building Internet of Things (BIoT) Applications. 180-185. https://doi.org/10.1109/ICSPIS57063.2022.10002641
N. Anand, M. A. Saifulla (2023) An efficient IDS for slow rate HTTP/2.0 DoS Attacks using one class classification. 1-9. https://doi.org/10.1109/I2CT57861.2023.10126162
C. Bannur, C. Bhat, K. Singh, S. A. Kulkarni, M. Doddamani (2023) PAACDA: Comprehensive Data Corruption Detection Algorithm. 11, 24908-24934. https://doi.org/10.1109/ACCESS.2023.3253022
S. Buschjäger, P. -J. Honysz, K. Morik (2020) Generalized Isolation Forest: Some Theory and More Applications Extended Abstract. 793-794. https://doi.org/10.1109/DSAA49011.2020.00120
D. Xu, Y. Wang, Y. Meng, Z. Zhang (2017) An Improved Data Anomaly Detection Method Based on Isolation Forest. 287-291. https://doi.org/10.1109/ISCID.2017.202
M. Wielgosz, A. Skoczen, K. Wiatr (2018) Looking for a Correct Solution of Anomaly Detection in the LHC Machine Protection System. 257-262. https://doi.org/10.1109/ICSES.2018.8507291Y
Y. Qin, Y. Lou (2019) Hydrological Time Series Anomaly Pattern Detection based on Isolation Forest. 1706-1710. https://doi.org/10.1109/ITNEC.2019.8729405
S. Luo, L. Luan, Y. Cui, X. Chai, Z. Wang, Y. Kong (2019) An Attribute Associated Isolation Forest Algorithm for Detecting Anomalous Electro-data. 3788-3792. https://doi.org/10.23919/ChiCC.2019.8866495
Z. Ding, Y. Mo, Z. Pan (2019) A Novel Software Defect Prediction Method Based on Isolation Forest. 882-887. https://doi.org/10.1109/QR2MSE46217.2019.9021215
S. Shriram, E. Sivasankar (2019) Anomaly Detection on Shuttle data using Unsupervised Learning Techniques. 221-225. https://doi.org/10.1109/ICCIKE47802.2019.9004325
U. B. Baloglu, B. Alatas, H. Bingol (2019) Assessment of Supervised Learning Algorithms for Irony Detection in Online Social Media. 1-5. https://doi.org/10.1109/UBMYK48245.2019.8965580
H. Shu, X. Zhao, H. Luo, C. Li (2019) Research on Stacking-Based Integrated Algorithm of Anomaly Detection in Production Process. 85-90. https://doi.org/10.1109/HPBDIS.2019.8735447
L.O. Iheme, Ş. Ozan (2019) Feature Selection for Anomaly Detection in Call Center Data. 926-929. https://doi.org/10.23919/ELECO47770.2019.8990454
T. Komárek, J. Brabec, Č. Škarda, P. Somol (2021) Threat Hunting as a Similarity Search Problem on Multi-positive and Unlabeled Data. 2098-2103. https://doi.org/10.1109/BigData52589.2021.9671958
W. Liu, J. He, S. Han, F. Cai, Z. Yang, N. Zhu A Method for the Detection of Fake Reviews Based on Temporal Features of Reviews and Comments.
Y. Lv, Y. Cui, X. Zhang, M. Cai, X. Gu, Z. Xiong (2019) A New Outlier Detection Method Based on Machine Learning. 1-7. https://doi.org/10.1109/ICSIDP47821.2019.9173217
L. H. M. Sepulvene, et al. (2019) Analysis of Machine Learning Techniques in Fault Diagnosis of Vehicle Fleet Tracking Modules. 759-764. https://doi.org/10.1109/BRACIS.2019.00136
Downloads
Published
Issue
Section
License
Copyright (c) 2024 Authors and Global Journals Private Limited
This work is licensed under a Creative Commons Attribution 4.0 International License.
