Soon-Hong An, Hoon-Suk Lee, Seung-Hoon Kim

KIPS Transactions on Software and Data Engineering, Vol. 12, No. 6, pp. 259-266, Jun. 2023
https://doi.org/10.3745/KTSDE.2023.12.6.259,   PDF Download:
Keywords: Concept Drift, Data Drift, Covariate Shift, Kolmogorov–Smirnov Test
Abstract

It is very difficult to measure the performance of the machine learning model in the business service stage. Therefore, managing the performance of the model through the operational department is not done effectively. Academically, various studies have been conducted on the concept drift detection method to determine whether the model status is appropriate. The operational department wants to know quantitatively the performance of the operating model, but concept drift can only detect the state of the model in relation to the data, it cannot estimate the quantitative performance of the model. In this study, we propose a performance prediction model (PPM) that quantitatively estimates precision through the statistics of concept drift. The proposed model induces artificial drift in the sampling data extracted from the training data, measures the precision of the sampling data, creates a dataset of drift and precision, and learns it. Then, the difference between the actual precision and the predicted precision is compared through the test data to correct the error of the performance prediction model. The proposed PPM was applied to two models, a loan underwriting model and a credit card fraud detection model that can be used in real business. It was confirmed that the precision was effectively predicted.

Statistics
Cite this article