Skip to main content

Monitoring Concepts

Model monitoring is an operational stage in the machine learning lifecycle that comes after model deployment. It entails monitoring your ML models for changes such as model degradation, data drift, and concept drift, and ensuring that your model is maintaining an acceptable level of performance.

What is drift?​

Machine learning models are trained with historical data, but once they are used in the real world, they may become outdated and lose their accuracy over time due to a phenomenon called drift. Drift is the change over time in the statistical properties of the data that was used to train a machine learning model. This can cause the model to become less accurate or perform differently than it was designed to.

In other words, drift is the decline in a model's ability to make accurate predictions due to changes in the environment in which it is being used.

Further resources

You can find more information about drift in the following articles:

What to monitor?​

Data Integrity​

It's important to monitor and assess the quality of the incoming data, including patterns such as limits (for example, acceptable minimum and maximum values) and data structure. Machine learning models rely on consistent and high-quality data to generate accurate predictions.

Model/Concept Drift​

Concept drift, also known as model drift, occurs when the task that the model was designed to perform changes over time. For example, imagine that a machine learning model was trained to detect spam emails based on the content of the email. If the types of spam emails that people receive change significantly, the model may no longer be able to accurately detect spam.

Feature/Data Drift​

Feature/Data drift also known as covariate shift, occurs when the distribution of the input data changes over time. For example, consider a machine learning model that was trained to predict the likelihood of a customer purchasing a product based on their age and income. If the distribution of ages and incomes of the customers changes significantly over time, the model may no longer be able to predict the likelihood of a purchase accurately.

Application Performance​

MLService deployed as REST API endpoints needs to be monitored by API peformance including number of requests, response time, failed requests and availability etc.