About Domino
Domino Data LabKnowledge BaseData Science BlogTraining

Domino 5.1.1 (March 2022)

This release provides security patches and several enhancements:

See also the fleetcommand-agent Release Notes.

New Features

Support for manually configuring pod attributes

Administrators can manually configure the attributes of pods in a hardware tier, including their resources, labels, and annotations. See Manually configure pod attributes in the Admin Guide for details.

Support for on-demand Open MPI clusters

Users can perform distributed CPU or GPU MPI-based training on Domino. Domino orchestrates Kubernetes level resource orchestration. This makes it easy to provision and use compute capacity for both batch and interactive workloads without DevOps overhead. Users also benefit from simplified code distribution to all cluster workers. This has been a usability issue in tradition MPI setups. Lastly, MPI clusters can use NVIDIA NCG distributed training images to make it easier to get started. See Open MPI for details.

Note

Known issues

  • Workspaces become unstoppable and their statuses don’t change when the timestamps on the run’s status changes array are incorrect.

  • In the Control Center’s navigation pane, click Assets. Domino displays a "Failed to get assets" message and takes a long time to load the assets.

  • You cannot delete, rename, or move a .gitignore file in a Domino File System project.

    • If you delete the file, it remains.

    • If you move the file, the original and target file both exist.

    • If you rename the file, the source and renamed file both exist.

  • If you rerun a successful job that accesses files from a dataset in a project owned by an organization, the job fails with file not found errors. Initial runs of the jobs succeed, but reruns fail with logging similar to the following:

    2022-10-06 04:34:32 : FileNotFoundError: [Errno 2] No such file or directory: '/domino/datasets/local/reruns_fail/file.txt'

    To work around this, start the execution as a new job.

  • A limit overcommit error (OOMKilled) is occurring frequently on the prometheus-adapter pod in Domino’s Integrated Model Monitoring. The current workaround is to increase the limits. This issue is resolved in Domino 5.5.0.

  • If you use a forward slash (/) in a Git branch name, the forward slash and any preceding text is stripped. For example, if the branch was named test/branch1, it will be renamed branch1.

    • In a workspace in a Git-based project, you can’t pull data because the branch doesn’t exist. You can push because this creates a new branch named branch1.

    • In a Domino File System project that uses imported code repositories, you can push and pull directly to and from the branch (test/branch2) if you don’t change the branch in the menu. If you change the branch, you can’t return to the selection (test/branch2) because test/ is stripped. If you push to branch2, this creates a new branch named branch2.

  • The graph views are skewed incorrectly, in Domino’s model monitoring, when normalized values are small.

  • When exporting notebooks from JupyterLab workspaces, the system shows a 403 Forbidden Error with Blocking Cross Origin message.

  • Entering a URI in the New Environment window can cause the window to go blank. This issue is resolved in Domino 5.2.0.

  • The output of the gateway/runs/getByBatchId endpoint is missing the runs and nextBatchId objects. This issue is resolved in Domino 5.2.0.

  • When viewing a raw HTML file while com.cerebro.domino.frontend.allowHtmlFilesToRender is false, Javascript can be executed. This issue is fixed in Domino 5.2.0.

  • When you set an environment as the default on the Environments page and then restart the service, Domino reverts to the default environment previously set on the Central Configuration page. This issue is fixed in Domino 5.2.0.

  • Sync operations for deployments backed by SMB file stores can sometimes result in unintended file deletions. This issue is fixed in Domino 5.2.0.

  • You cannot delete a .gitignore file in a Domino File System (DFS)-based project. This issue is fixed in Domino 5.2.0.

  • Jobs can fail or become stuck in a Pending state when using files from an S3 data source. This issue is fixed in Domino 5.2.0.

  • Entering a URI in the New Environment window can cause the window to go blank. This issue is resolved in Domino 5.2.0.

  • Apps may fail with unusual error messages. This issue is resolved in Domino 5.2.0.

  • The Domino UI may hang with a "Waiting for available socket" error. This issue is resolved in Domino 5.2.0.

  • When an API key explicitly passed as an api_key argument and the DOMINO_TOKEN_FILE environment variable is set, Domino uses the token authentication. This issue is resolved in Domino 5.2.0.

  • When Flask apps generate links, this can result in unwanted 302 redirect behavior. This issue is fixed in Domino 5.2.0.

  • When a Model API fails while retrieving files from git using SSH keys, the private key and the key password, if any, may be exposed in the Domino log. This issue is resolved in Domino 5.2.0.

  • The Import Project drop-down does not display projects owned by organizations of which you are a member. This issue is resolved in Domino 5.2.0.

  • MATLAB environment builds may fail, or MATLAB workspaces may be unresponsive.

  • You might experience slow loading times when you open the Environments, Workspaces, Model APIs, or Projects pages.

  • When starting a workspace, the application sometimes gets stuck in the Pulling status, even though the logs indicate that the workspace started successfully. If this occurs, stop and start the workspace again. This issue is fixed in Domino 5.2.0.

  • When you add environment variables to a project, they are visible in the execution_specifications collection in MongoDB.

  • After upgrading to 4.6.4 or higher, launching a job from a Git-based project that uses a shared SSH key or personal access token results in a Cannot access Git repository error.

  • In a Git-based project, requirements.txt must be in your project’s Artifacts folder in order to be picked up by an execution, instead of at the root level in your project files.

    Tip

  • After you migrate from 4.x, scheduled jobs will fail until the users that own them log out and log in again. This issue is resolved in Domino 5.3.0.

    Tip

  • Domino can become inaccessible after a fixed uptime period when running on Kubernetes 1.21 or higher. See this knowledge base article for the workaround. This issue is fixed in Domino 5.3.1.

  • Upgrading to this release from Domino 3.x or 4.x can cause Git credentials to be omitted from the dropdown when creating a new Git-based project. A workaround is available from Domino’s Customer Success team.

  • Model Monitoring data sources aren’t validated. If you enter an invalid bucket name and attempt to save, the entry will go through. However, you won’t be able to see metrics for that entry because the name points to an invalid bucket.

  • If the Model Access Token Vault migration runs more than once, it wipes tokens from Mongo and Vault, and users need to regenerate their Model Access Tokens in the UI. This is fixed in Domino 5.4.0.

Domino Data LabKnowledge BaseData Science BlogTraining
Copyright © 2022 Domino Data Lab. All rights reserved.