Monitor your ML Models on scored data using IBM Watson OpenScale
IBM Watson OpenScale is a machine learning model monitoring platform for monitoring drift, quality, fairness of your machine learning models and be able to explain the outcome of the model predictions.
Not all the time customers are willing to expose their model scoring endpoints for monitoring purposes — say in some scenarios like, security restrictions, air gap deployment reasons, firewall restrictions, or in case of Batch Scoring models, where the scoring happens asynchronously, OpenScale does not have access to the scoring end point. Said that, customers are expecting to monitor their models based on the scored payload and feedback data, which when logged to OpenScale data mart and by configuring the monitors.
The solution is, with IBM Watson OpenScale, one can create a placeholder Custom ML provider and configure an Headless Subscription against it, log the payload and feedback data and thereby evaluate the monitors.
This article describes the steps involved in configuring the OpenScale Headless subscriptions through OpenScale UI through the following sections:
- Generating the training data statistics, and archives needed for Drift and Explainability monitors through notebook.
- OpenScale Headless Subscription Configuration through OpenScale UI
- Payload Logging and Feedback Logging and finally evaluate the configured monitors.
1. OpenScale Monitor Configuration Generation
The scenario here is, for OpenScale monitors like Drift and Explainability they need certain artifacts to be generated based on the training data and the scoring end point and configure them with the said monitors. A one-time artifacts generation and configuration. But the issue here is, customers are not willing to expose their training data and nor the scoring end point. So, the alternative is, use a notebook, which can be executed in their Jupyter or python environment by having access to the training data and scoring end point to generate the required artifacts. Through the said notebook, one the following configurations are generated:
- Training Data Statistics
- Explainability Monitor Archive
- Drift Monitor Archive.
Once the artifacts are generated then we move on to OpenScale UI to upload those artifacts to configure the monitors.
A sample/working instance of the training statistics notebook can be found here.
2. OpenScale Headless Subscription Configuration
2.1 OpenScale Instance Creation
If OpenScale instance is not already present, then from Cloud Pak for Data home page, navigate to the Instances page and create a new OpenScale instance.
Give a name and description to the OpenScale instance that you are creating.
Review the summary of the OpenScale instance, and then click on “Create”. Wait for couple of seconds, and the newly created instance will be listed in the instances page.
2.2 OpenScale Insights Dashboard
From the newly created instance click “Open” to navigate to OpenScale insights dashboard.
In here, please follow setting up the OpenScale Auto-Setup which should create the Credit Risk based pre-production and production subscriptions.
OpenScale Subscriptions created through Auto-Setup process.
2.3 OpenScale Headless Subscription Configuration
The workflow for creating and monitoring a Headless Subscription is as follows:
- Creation of Service Provider using Custom Environment.
- Adding an empty deployment to the Custom Service Provider.
- Upload the training data statistics which would setup the subscription and also configure fairness monitor.
- Setup the payload data schema by sending a sample model scoring request/response to OpenScale.
- Configure Explainability and evaluate an transaction explanation.
- Configure Drift and Quality monitor
- Log payload and feedback data
- Evaluate the monitors.
2.4 Creation of Custom ML Provider
Under OpenScale System setup page, click on “Add machine learning provider”
Give the provider a name and an optional description. Select “Custom Environment” and the “Environment type” as Production. Leave the authentication details as blank. This is because we are not actually subscribing any deployment.
2.5 OpenScale Headless Deployment Creation
From the insights page, click on “Add to dashboard” and select “System-managed”. Give the deployment a name and an optional description. Make sure to leave the “Model endpoint” empty. Click on Configure. Followed by a confirmation on the same.
2.6 Enabling Headless Subscription
Click on “Configure monitors” as in the previous page, which should take you to a page as below. Edit “Model input” icon…
And specify the model type details as below (as per your requirements)
From the training statistics notebook, download the training_distribution.json which contains the training statistics. Upload this file in here. Doing so, will enable the subscription and also create and configure the fairness monitor. (Note: you are not providing the actual training data location, per se).
2.7 Enable Payload Logging (output data) schema
Select the JSON payload option under “Scoring method”. In here, specify the model scoring request/response payload to establish the output data schema. As an example, for the Credit Risk sample, here is an example request/response scoring payload which is logged for setting the output data schema.
Get the confirmation as below…
Specify the model prediction and probability column names, and get a confirmation that the subscription with all the necessary details are configured.
2.8 Fairness monitor configuration
As we have uploaded the training statistics configuration JSON, which also has the fairness configuration details, the fairness monitor is auto configured as below.
2.9 Explain monitor configuration
From the training statistics configuration notebook, one of the configuration file is the Explainability archive. Upload this archive, as in here, which would configure the Explainability monitor.
To enable the payload logging schema, we have already uploaded a scoring request/response record. Let’s try to get the explanation of this transaction from the Explainability page, as below. Click on “Explain” of a given transaction.
Which would take couple of seconds to complete, and would provide you the explanation as below.
2.10 Quality Monitor Configuration
Select “Quality” from the side menu and specify the quality monitor thresholds as below.
Followed by specifying the number of records (say, 100) to be used for quality monitor evaluation. Followed by Save the configuration.
2.11 Drift Monitor Configuration
From the training statistics configuration notebook, one of the configuration files is the drift archive. Upload this archive, as in here, which would configure the Drift monitor.
Click “Next” and specify the thresholds for “model drift” and “data drift”, to get the confirmation as below.
3. Payload and Feedback Logging
From the Model Info/Endpoints page, make a note of the data mart id, subscription id, and use the code snippets to load the payload and feedback data to data mart.
4. OpenScale Monitors Evaluation
Click on the newly created deployment from the Insights dashboard and click on Evaluate Now. Make sure the count of the payload and feedback records match with the number of records that are added in the previous step.
Click on “Evaluate now” and wait for the evaluation to complete to see the monitor evaluation results as below.
5. Conclusion
Using OpenScale Headless Subscriptions one need not provide access to the model scoring endpoint and neither model training data. As part of the this article, we have seen on how to configure an OpenScale Headless Subscription and log scored payload and feedback data to OpenScale data mart and finally do an evaluation of the monitors.
Note: though this article is worded using IBM Watson OpenScale in IBM Cloud Pak for Data 4.5 release, the same set of steps are applicable for IBM Watson OpenScale provisioned in IBM Cloud as well.
6. References
- All about IBM Watson OpenScale
- OpenScale Training Statistics Notebook
- Sample notebook for performing Payload Logging and Feedback Logging to OpenScale.
That concludes this tutorial!
