domino logo
Tech Ecosystem
Get started with Python
Step 0: Orient yourself to DominoStep 1: Create a projectStep 2: Configure your projectStep 3: Start a workspaceStep 4: Get your files and dataStep 5: Develop your modelStep 6: Clean up WorkspacesStep 7: Deploy your model
Get started with R
Step 0: Orient yourself to Domino (R Tutorial)Step 1: Create a projectStep 2: Configure your projectStep 3: Start a workspaceStep 4: Get your files and dataStep 5: Develop your modelStep 6: Clean up WorkspacesStep 7: Deploy your model
Get Started with MATLAB
Step 1: Orient yourself to DominoStep 2: Create a Domino ProjectStep 3: Configure Your Domino ProjectStep 4: Start a MATLAB WorkspaceStep 5: Fetch and Save Your DataStep 6: Develop Your ModelStep 7: Clean Up Your Workspace
Step 8: Deploy Your Model
Scheduled JobsLaunchers
Step 9: Working with Domino Datasets
Domino Reference
Projects
Projects Overview
Revert Projects and Files
Revert a ProjectRevert a File
Projects PortfolioReference ProjectsProject Goals in Domino 4+
Git Integration
Git Repositories in DominoGit-based Projects with CodeSyncWorking from a Commit ID in Git
Jira Integration in DominoUpload Files to Domino using your BrowserCopy ProjectsFork and Merge ProjectsSearchSharing and CollaborationCommentsDomino Service FilesystemCompare File RevisionsArchive a Project
Advanced Project Settings
Project DependenciesProject TagsRename a ProjectSet up your Project to Ignore FilesUpload files larger than 550MBExporting Files as a Python or R PackageTransfer Project Ownership
Domino Runs
JobsDiagnostic Statistics with dominostats.jsonNotificationsResultsRun Comparison
Advanced Options for Domino Runs
Run StatesDomino Environment VariablesEnvironment Variables for Secure Credential StorageUse Apache Airflow with Domino
Scheduled Jobs
Domino Workspaces
WorkspacesUse Git in Your WorkspaceUse Visual Studio Code in Domino WorkspacesPersist RStudio PreferencesAccess Multiple Hosted Applications in one Workspace Session
Spark on Domino
On-Demand Spark
On-Demand Spark OverviewValidated Spark VersionConfigure PrerequisitesWork with your ClusterManage DependenciesWork with Data
External Hadoop and Spark
Hadoop and Spark OverviewConnect to a Cloudera CDH5 cluster from DominoConnect to a Hortonworks cluster from DominoConnect to a MapR cluster from DominoConnect to an Amazon EMR cluster from DominoRun Local Spark on a Domino ExecutorUse PySpark in Jupyter WorkspacesKerberos Authentication
On-Demand Ray
On-Demand Ray OverviewValidated Ray VersionConfigure PrerequisitesWork with your ClusterManage DependenciesWork with Data
On-Demand Dask
On-Demand Dask OverviewValidated Dask VersionConfigure PrerequisitesWork with Your ClusterManage DependenciesWork with Data
Customize the Domino Software Environment
Environment ManagementDomino Standard EnvironmentsInstall Packages and DependenciesAdd Workspace IDEsAdding Jupyter Kernels
Partner Environments for Domino
Use MATLAB as a WorkspaceUse Stata as a WorkspaceUse SAS as a WorkspaceNVIDIA NGC Containers
Advanced Options for Domino Software Environment
Install Custom Packages in Domino with Git IntegrationAdd Custom DNS Servers to Your Domino EnvironmentConfigure a Compute Environment to User Private Cran/Conda/PyPi MirrorsUse TensorBoard in Jupyter Workspaces
Publish your Work
Publish a Model API
Model Publishing OverviewModel Invocation SettingsModel Access and CollaborationModel Deployment ConfigurationPromote Projects to ProductionExport Model Image
Publish a Web Application
App Publishing OverviewGet Started with DashGet Started with ShinyGet Started with FlaskContent Security Policies for Web Apps
Advanced Web Application Settings in Domino
App Scaling and PerformanceHost HTML Pages from DominoHow to Get the Domino Username of an App Viewer
Launchers
Launchers OverviewAdvanced Launcher Editor
Assets Portfolio Overview
Model Monitoring
Model Monitoring APIsAccessing The Model MonitorGet Started with Model MonitoringModel Monitor DeploymentIngest Data into The Model MonitorModel RegistrationMonitoring Data DriftMonitoring Model QualitySetting Scheduled Checks for the ModelConfigure Notification Channels for the ModelUse Model Monitoring APIsProduct Settings
Connect to your Data
Data in Domino
Datasets OverviewDatasets Best Practices
Data Sources Overview
Connect to Data Sources
External Data Volumes
Work with Data Best Practices
Work with Big Data in DominoWork with Lots of FilesMove Data Over a Network
Advanced User Configuration Settings
User API KeysDomino TokenOrganizations Overview
Use the Domino Command Line Interface (CLI)
Install the Domino Command Line (CLI)Domino CLI ReferenceDownload Files with the CLIForce-Restore a Local ProjectMove a Project Between Domino DeploymentsUse the Domino CLI Behind a Proxy
Browser Support
Get Help with Domino
Additional ResourcesGet Domino VersionContact Domino Technical SupportSupport Bundles
domino logo
About Domino
Domino Data LabKnowledge BaseData Science BlogTraining
User Guide
>
Domino Reference
>
Model Monitoring
>
Get Started with Model Monitoring

Get Started with Model Monitoring

This section will walk you through simple steps to get started on monitoring a model. Before we get into the specific steps, let’s get a high-level overview of the Model Monitor workflow and how the Model Monitor integrates with your system.

Model Monitoring Approach

image

Model Monitoring Workflow

image

Model monitoring reads and processes the model’s training, predictions and ground truth data from one of the supported data stores. Model monitoring allows you to link different data stores to it so that it can read data from them. To see the list of supported data sources go to Ingest Data in Model Monitoring.

Use the steps below steps to quickly get started with drift detection.

1) Connect Data Source

  1. If you haven’t already set up a data source, go to the Data Sources section, and click Add Data Source.

    image

  2. Fill in the details of the storage location which you will have your data files.

2) Register A Model

  1. Go to the Model Dashboard and click Register Model.

    image

  2. Upload a Model Config file into the pop-up. A sample file can be seen in Model Registration. Alternatively, you can copy paste the JSON that represents the model config into a JSON editor, also available in the same modal window. This pop-up provides basic JSON validation as well.

  3. On clicking Register the model entry is now created in the model monitor.

  4. You should see the newly registered model listed in the Models page.

3) Add Prediction Data

  1. Click on the registered model and go to the Data Drift page.

    image

  2. Click on Register Prediction where you will see a pop-up that lets you upload a Prediction Config file. A sample Prediction Config can be seen in Monitoring Data Drift.

  3. On clicking Register, the Model Monitor will begin calculation of drift on the registered dataset.

4) Analyze Data Drift

  1. After the drift calculations are complete, you should see the divergence value for each feature in the Drift column.

  2. The default Test Type and Threshold value will be used for the computation. (You can change the defaults in Settings > Drift Tests Defaults).

  3. You can now experiment with other Test Types, Conditions and Threshold values for this dataset.

  4. You can filter the data by different Date Ranges to look at Drift and its trends for specific time periods.

  5. If your model had a timestamp column declared, then that is used to get the timestamp of different predictions in the dataset. If it was not declared, then the data’s ingestion time in the Model Monitor is taken as its time stamp.

  6. You can control which features should be excluded from the Scheduled Checks using the Enable/Disable Alerts toggle icon in the table. This is helpful to reduce alert noise.

  7. If you want to save the test types, threshold or other configs, and want to save them as the default config of the model to be used for running scheduled checks, you can save them using Other Actions > Save Checks Config.

  8. If you have made experimented with different configs and want to load the model’s default config to reset the Analyze table, you can load them using Other Actions > Restore Checks Config.

5) Add Ground Truth Data

  1. After you have added prediction data, you can ingest the model’s ground truth data to monitor the quality of predictions made by the model.

  2. For this analysis, it is necessary that a row_identifier column was declared for the model. It is used to match the ground truth labels to the model’s predictions.

  3. On the Model Quality page of the registered model, click on 'Register Ground Truth'. You will see a pop-up that lets you upload a Ground Truth Config file. A sample Ground Truth Config can be seen in Monitoring Model Quality.

    image

  4. On clicking Register, you will be taken to the Model Quality analyze page and the Model Monitor will begin calculation of model quality metrics for this dataset.

  5. You can control which metrics should be excluded from the Scheduled Checks using the Enable/Disable Alerts toggle icon in the table. This is helpful to reduce alert noise.

6) Setup Scheduled Checks and Notifications

  1. You can also setup checks that run periodically and configure who should be receiving email alert notifications if they fail.

  2. Go to the Model’s Notifications section and enter the email ids to which notifications will be sent for this model.

    image

  3. Go to the Analyze tab in Data Drift to control which features should be excluded from the Scheduled Checks using the Enable/Disable Alerts toggle icon in the table. This is helpful to reduce alert noise.

  4. Now go to the Schedule Checks tab to set the frequency and other parameters.

  5. Enter the name for the check.

  6. Configure the frequency at which it should be run.

  7. In the Select Data to Check options select one:

    1. Use data since last check time ensures only predictions with timestamp that fall after the last time the scheduled check ran are considered for the check.

    2. Data since last x time period allows to select predictions with timestamp that fall within the last specified x time period (eg. last 3 days) are considered for the check.

  8. Click Save and the Scheduled Checks get enabled. You can see the historical reports of all the runs in Checks History tab for data drift.

  9. Similarly, you can set up Schedule Checks for model quality as well.

  10. One important distinction to keep in mind is that while for data drift the timestamp of the predictions is used select data for the schedule checks, in case of model quality it is the ingestion time of the ground truth labels that is used to select data. Refer to the Setting Scheduled Checks for the model section of the documentation for more details.

System diagram with Model Monitor integration

This diagram shows one way that the Model Monitor can be integrated into your system.

image

Domino Data LabKnowledge BaseData Science BlogTraining
Copyright © 2022 Domino Data Lab. All rights reserved.