Efficient ML Monitoring with Evidently AI: Step-by-Step Guide for Report Generation

Hey there, AI enthusiasts! Welcome to our channel. Today, we will dive into an amazing tool that will supercharge your ML model monitoring – Evidently AI. This open-source gem helps you monitor your models, detect issues, and generate detailed reports. Let's jump right in and see how to use Evidently AI with the California Housing dataset.

Step 1: Setting Up the Environment

First things first, let's get everything set up. Open your terminal or Jupyter Lab and run this command to install the necessary libraries. I'm using Evidently AI version X.XX, along with other relevant packages.

pip install evidently==X.XX pandas numpy scikit-learn

If you have already set up these libraries on your system, you're good to go. These libraries will set the stage for our data analysis and model monitoring.

Step 2: Import the Cool Tools

Next, let's import all the tools we will be using: NumPy, pandas, scikit-learn, and Evidently AI.

import numpy as np
import pandas as pd
from sklearn.datasets import fetch_california_housing
import evidently
from evidently.report import Report
from evidently.metric_preset import DataDriftPreset

These imports will help us with data manipulation, generating reports, and running tests.

Step 3: Load and Prepare the Dataset

Time to load our dataset. We are using the California Housing dataset available in scikit-learn. Let's also add a prediction column with a bit of noise for some extra fun.

## Introduction
data = fetch_california_housing(as_frame=True)
df = data.frame
df['Prediction'] = df['MedHouseVal'] + np.random.normal(0, 0.5, df.shape[0])

## Introduction
df.rename(columns=("MedHouseVal": "target_col"), inplace=True)

Step 4: Splitting the Data

Now let's split our data into reference and current datasets. This helps us compare and analyze any changes or drifts.

## Introduction
reference, current = df.iloc[:1000], df.iloc[1000:2000]

Step 5: Generate Initial Report

Let's kick things off with the data drift report to see if there are any shifts between our reference and current datasets.

## Introduction
report = Report(metrics=[DataDriftPreset()])
report.run(reference_data=reference, current_data=current)

## Introduction
report.show()

This inline command gives us the Evidently AI report in Jupyter Lab itself. All the visualizations for each column and the data distribution are provided, making it easier to understand the data at a deeper level.

Step 6: Specific Column Analysis

We can also generate reports for specific columns. Here's how to do it for the "AveRooms" column.

## Introduction
report = Report(metrics=[DataDriftPreset(column_names=['AveRooms'])])
report.run(reference_data=reference, current_data=current)
report.show()

You'll get a detailed and beautiful chart for the specific column, along with data drift and distribution visualization.

Step 7: Customize Metrics

You can customize the metrics for specific columns as well.

## Introduction
custom_metrics = Report(metrics=[DataDriftPreset(column_names=['AveRooms', 'AveOccup'])])
custom_metrics.run(reference_data=reference, current_data=current)
custom_metrics.show()

Step 8: Exporting Reports

Finally, let's export our reports for further analysis or sharing with your team.

## Introduction
report_dict = report.as_dict()
report_json = report.as_json()

## Introduction
with open("report.json", "w") as f:
    f.write(report_json)

Your report is now ready to be shared or analyzed further.

That's it for today, folks! Evidently AI makes monitoring your ML models a breeze. By following these steps, you can generate insightful reports and keep your models in check. If you found this article helpful, give us a thumbs up and share it with your friends. Subscribe for more awesome content! Check out the GitHub link in the description for all the code. Happy coding!

Keywords

Evidently AI
ML Model Monitoring
California Housing Dataset
Data Drift
Visualization
Metrics
Report Generation

FAQ

Q: What is Evidently AI?
A: Evidently AI is an open-source tool designed for monitoring machine learning models, detecting issues, and generating detailed reports.

Q: How do I install Evidently AI?
A: You can install Evidently AI and the necessary libraries using the pip install evidently==X.XX pandas numpy scikit-learn command.

Q: What datasets can be used with Evidently AI?
A: You can use any dataset, but in this guide, we use the California Housing dataset from scikit-learn.

Q: Can Evidently AI generate reports for specific columns?
A: Yes, Evidently AI allows you to generate reports for specific columns and even customize metrics for in-depth analysis.

Q: How can I export the generated reports?
A: You can export the reports as a dictionary or a JSON file using the report.as_dict() and report.as_json() methods. Save them to a file for further analysis or sharing.

Q: Where can I find the code used in this guide?
A: You can find the code on the GitHub link provided in the description of the video.

Happy monitoring and coding!