Monitoring IBM SPSS Modeler Models using IBM Watson OpenScale
What it is about? ..
IBM Watson OpenScale is an AI/ML model monitoring platform to evaluate and monitor the Drift, Fairness, Quality, Explainability or any other custom metrics of your machine learning models. Evaluate, and thereby be able to assess the risk of using the machine learning models in a production environment. And once put in production environment, use OpenScale to continuously monitor your machine learning models for any threshold violations.
On the other hand, IBM SPSS Modeler provides a set of data mining tools to enable you to quickly develop predictive models and conduct analytics tasks, and thereby deploy the models into business operations to improve decision making.
This post is about describing a mechanism to use OpenScale to monitor the models that are created using IBM SPSS Modeler.
Integration Details ..
Consider a use case — Telco Customer Churn Prediction. Given a set of model features like, Customer Gender, Tenure of the connection, Does stream TV, Movies, Monthly Charges, etc, predict whether the customer would leave the service or not.
Using the training data, a model is created using IBM SPSS Modeler.
IBM SPSS Modeler is a leading visual data science and machine learning (ML) solution designed to help enterprises accelerate time to value by speeding up operational tasks for data scientists.
A model is basically a unit of work — for a given set of features, it predicts an outcome. For an application to make use of, or integrate this model we need to deploy it in an serving environment — for which, the model created using SPSS Modeler can be deployed to IBM Watson Machine Learning.
IBM Watson Machine Learning provides a full range of tools and services so that you can build, train, and deploy Machine Learning models.
Once the model is deployed, we can send the scoring payload to the model deployment to get the prediction outcomes. Something like this..
scoring_data = {
wml_client.deployments.ScoringMetaNames.INPUT_DATA: [{
"fields": ["customerID","gender","SeniorCitizen","Partner","Dependents","tenure","PhoneService","MultipleLines","InternetService","OnlineSecurity","OnlineBackup","DeviceProtection","TechSupport","StreamingTV","StreamingMovies","Contract","PaperlessBilling","PaymentMethod","MonthlyCharges","TotalCharges","Churn","SampleWeight"],
"values":[["3638-WEABW","Female",0,"Yes","No",58,"Yes","Yes","DSL","No","Yes","No","Yes","No","No","Two year","Yes","Credit card (automatic)",59.9,3505.1,None,None]]
}]
}
predictions = wml_client.deployments.score(deployment_id, scoring_data)
print(predictions)
{'predictions': [{'fields': ['customerID', 'Churn', 'Predicted Churn',
'Probability of Churn'],
'values': [['3638-WEABW', None, 'No', 0.0526309571556145]]}]}
In above, the model deployment is predicting whether the customer would Churn or not, and the confidence of that prediction. If you notice, the key thing here is, the model is predicting (Not going to Churn in this case) only the winning class. But to compute various model monitoring metrics we need winning and non-winning classes confidence (probability) values.
So, as a next step, given a model deployment prediction we would do some post-processing to deduce the winning and non-winning classes prediction. Something like this ..
def score(scoring_payload):
...
# original scoring response from the model deployment
fields = scoring_response['predictions'][0]['fields']
values = scoring_response['predictions'][0]['values']
scored_data = pd.DataFrame(values, columns = fields)
# deduce the non-winning class confidence
scored_data['Probability of No Churn'] = 1 - scored_data['Probability of Churn']
scored_data.rename(columns={'Predicted Churn': 'prediction', 'Probability of Churn': 'winning_class_prob', 'Probability of No Churn': 'losing_class_prob'}, inplace=True)
cols = ['winning_class_prob', 'losing_class_prob']
# create the probability, which would be sent along with the prediction output.
scored_data['probability'] = [[e for e in row if e==e] for row in scored_data[cols].values.tolist()]
scored_data.drop(['customerID', 'Churn', 'winning_class_prob', 'losing_class_prob'], axis=1, inplace=True)
scored_data_fields = scored_data.columns.tolist()
scored_data_values = scored_data[scored_data_fields].values.tolist()
# create the response, which would contain prediction and proability values
response_payload = {"predictions": [{"fields": scored_data_fields, "values": scored_data_values}]}
return response_payloadQuestion is, where does this scoring post-processing function comes into play? So, the idea here is, we deploy this code as a python function to IBM Watson Machine Learning, and monitor the python function deployment with IBM Watson OpenScale. Something like this ..
function_artifact = wml_client.repository.store_function(meta_props=function_meta_props, function=spss_scoring_wrapper)
function_uid = wml_client.repository.get_function_id(function_artifact)
print("Function UID = " + function_uid)
...
deployment_details = wml_client.deployments.create(function_uid, meta_props=deploy_meta)IBM Watson OpenScale is then used to configure all the monitors — namely, Fairness, Drift, Quality, Explainability, to the model evaluation outcome. As in..
Summarising ..
To summarise, the IBM SPSS Modeler Model is deployed to WML, and to hand the confidence values of winning and non-winning classes, a function wrapper is written which is further deployed to WML. Then use IBM Watson OpenScale to monitor the python function deployment, which essentially turns out to be monitoring the IBM SPSS Modeler. Something like this ..
The code that is used to manage the above setup using IBM Watson OpenScale and IBM Watson Machine Learning SDK’s can be found here. Please note, this notebook wraps binary class predictions, and for multi-class predictions, please check in IBM SPSS Modeler to know the probabilities of all classes and same flow can be applied as outlined above.
