Generating analytical reports using performance metrics of governed machine learning models
( Co-authored by Ravi Chamarthy and Marc Cassagnol )
Measuring performance of machine learning models is an important aspect of the overall AI/ML lifecycle management. Data Scientists develop models using different AI/ML frameworks (Spark mllib, Scikit-Learn, Torch, etc) but use the same training data to train the models, and the other possibility is to use different versions of training data sets but use same AI/ML model framework. Of these different ways to develop models, how to choose which model is performing better — in terms of model and data bias, drift, model quality evaluations, robustness of the model and/or other metric evaluations. And then the other aspect is the model governance — which user performed the model evaluation, which training data used, which AI/ML framework used, etc. We need a comprehensive analytical report listing the comparison of performance metrics belonging to multiple models.
This article describes the mechanism to generate analytical reports using IBM Cognos Analytics with machine learning model metrics evaluated and published from IBM Watson OpenScale to IBM Watson OpenPages.
Pre-requisite
To use IBM Cognos Analytics from IBM Watson OpenPages for machine learning model performance data, please ensure the OpenPages Reporting Framework V6 is updated to include a framework model and namespace for MRG, and re-generate the Cognos Analytics Reporting Framework.
Please reach out to IBM Support or your System Administrator if you need any guidance updating Reporting Framework V6.
Configure your model deployments with IBM Watson OpenScale for monitoring
Please refer to the earlier article Using Automation to Choose the Right Machine Learning Model for Your Production on how to build, deploy, and monitor your machine learning models using IBM Watson OpenScale.
The steps include:
- Create, Train and Build the model
- Deploy the model to any Machine Learning Serv Engine
- Subscribe the model deployment with IBM Watson OpenScale
- Associate the OpenScale Subscription with OpenPages model
- Configure the monitors
Fairness to monitor on one or more fairness attributes
Quality to monitor quality of the model
Drift to monitor to model and data drift
One or more Custom Metrics, as needed.
Evaluate the model deployment from OpenScale Model Risk Management dashboard.
Once the model is subscribed to OpenScale and evaluated, the evaluation results can be visualised like in the below screenshot.
- Publish the model performance metrics from OpenScale subscription to OpenPages model by clicking on “Send to OpenPages now” from “Actions” menu
- Visualize the metrics in OpenPages
- Repeat the steps for one or more models that we want to analyze in IBM Cognos Analytics.
Building analytical reports
In this section we will see how to build Cognos Analytics Reports with the model metrics that are published from OpenScale to OpenPages.
From OpenPages UI, perform the following set of steps to launch a new Analytics Report
- Click on Analytics from the primary menu in the OpenPages UI
- Select New/Report
- Select Blank
- Click OK
This would open Report Pages, in which you select the source for the reporting data, all the OpenPages metrics are present under MRG Reporting namespace.
Here comes the report designing part…
In the report canvas, select a List item, and select default Name and the Query Name.
- Form the Reporting Data Source, Select Model, and thereby select Model Name drag it and drop it in the List area.
- Likewise, drag and drop Model Description, Metric Name, Metric Description, Metric Value, Metric Red Threshold in the List area.
- Group Metric Name, so that the values appear per model.
- Group Metric Description, so that same description is not repeated.
- For Model Name, specify a custom filter by selecting only those models that you would like to display. In this case, models 5,6, and 31 are selected for displaying purpose.
- From the tool bar you can set a Page Break on the Model Name, as required.
- Run the report by selecting the Run option from the top toolbar.
- You would see multiple report rendering options — HTML, PDF, Excel, CSV, XML.
- Select Run HTML, and you would see the generated report in another browser window, with metrics from 3 OpenScale Subscriptions in a single report.
- The report displays fairness, quality, and drift metrics of OpenScale subscriptions in a single Cognos Report for analysis purposes.
Conclusion
As part of this article, we looked at how to generate Cognos analytical reports with machine learning model metrics published from OpenScale subscriptions to OpenPages model. IBM Cognos Reporting is a powerful analytics tool, using which one can render much more complex and deep-dive reports using multiple model performance metrics.
