Statistical Techniques for Drift Detection - Learn Data Science with Travis

Travis chat

Filters

Statistical Techniques for Drift Detection

Statistical techniques for detecting data drift are essential for ensuring that machine learning models remain effective over time. These techniques typically involve comparing the distribution of the training data with the distribution of the production data to identify any significant changes.

One common method is the Population Stability Index (PSI), which measures how much a population has shifted over time. Other statistical methods include Kullback-Leibler (KL) divergence, which quantifies how one probability distribution diverges from a second expected probability distribution, and Jensen-Shannon (JS) divergence, which is a symmetrized and smoothed version of KL divergence. The Wasserstein distance, also known as the Earth Mover’s Distance, is another technique that measures the distance between two probability distributions.

These methods can be applied to various features in the dataset to monitor for drift effectively. By implementing these statistical techniques, organizations can proactively manage model performance and ensure that their models adapt to changes in the underlying data ¹².

Data Drift — Part 2: How to Detect Data Drift

Towards Data Science

A description of the Techniques to detect data drift. These include PSI, Kullback-Leibler (KL) divergence, (JS) Divergence, Wasserstein distance, PSI

Why data drift detection is important and how do you automate it in 5 simple steps

Towards Data Science

The fundamental assumption in developing any machine learning model is that the data that is used to train the model mimics the real-world data. But how do you assert this assumption after the model…

Data Drift — Part 1: Types of Data Drift

Towards Data Science

This post explains the concept of Data Drift and how it can cause model performance degradation, how to identify data drift, and model monitoring plans

Calculating Data Drift in Machine Learning using Python

Towards Data Science

This article will aim to provide the intuition and implementation associated with data drift in Python. It will cover the implementation and differences between two approaches to calculating drift…

How to Detect Data Drift with Hypothesis Testing

Towards Data Science

Data drift is a concern to anyone with a machine learning model serving live predictions. The world changes, and as the consumers’ tastes or demographics shift, the model starts receiving feature…

Drift Metrics: How to Select the Right Metric to Analyze Drift

Towards Data Science

In our last post we summarized the problem of drift in machine learning deployments (“Drift in Machine Learning: Why It’s Hard and What to Do About It” in Towards Data Science). One of the takeaways…

How to Build a Fully Automated Data Drift Detection Pipeline

Towards Data Science

Motivation Data drift occurs when the distribution of input features in the production environment differs from the training data, leading to potential inaccuracies and decreased model performance. Im...

Drift in Machine Learning

Towards Data Science

The COVID-19 pandemic has sparked a lot of interest in data drift in machine learning. Drift is a key issue because machine learning often relies on a key assumption: the past == the future. In the…

How to detect, evaluate and visualize historical drifts in the data

Towards Data Science

TL;DR: You can look at historical drift in data to understand how your data changes and choose the monitoring thresholds. Here is an example with Evidently, Plotly, Mlflow, and some Python code. The…

How to Detect Concept Drift Without Labels

Towards Data Science

In a previous article , we explored the basics of concept drift. Concept drift occurs when the distribution of a dataset changes. This post continues to explore this topic. Here, you’ll learn how to d...

Understanding Kolmogorov-Smirnov (KS) Tests for Data Drift on Profiled Data

Towards Data Science

Data drift meets data profiling Image by author TLDR: We experimented with statistical tests, Kolmogorov-Smirnov (KS) specifically, applied to full datasets as well as dataset profiles and compared r...

Unboxing the Concept of Drift in Machine Learning

Towards AI

Machine Learning Drift is a common phenomenon that occurs once the machine learning algorithm is deployed to production. It can adversely affect the overall performance of your machine-learning model ...