Persistent volume management

Overview

When not in use, Domino project files are stored and versioned in the Domino blob store. When a Domino run is started from a project, the projects files are copied to a Kubernetes persistent volume that is attached to the compute node and mounted in the run. You can run the following command to see all current persistent volumes sorted by last-used:

kubectl get pv --sort-by='.metadata.annotations.dominodatalab.com/last-used'



Definitions

  • Persistent Volume (PV)

    A storage volume in a Kubernetes cluster that can be mounted to pods. Domino dynamically creates persistent volumes to provide local storage for active runs.

  • Persistent Volume Claim (PVC)

    A request made in Kubernetes by a pod for storage. Domino uses these to correctly match a new run with either a new PV or an idle PV that has the project’s files cached.

  • Idle Persistent Volume

    A PV that was used by a previous run, and which is currently not being used. Idle PV’s will either be re-used for a new run or garbage collected.

  • Storage Class

    Kubernetes method of defining the type, size, provisioning interface, and other properties of storage volumes.




Storage workflow for Jobs

When a user starts a new job, Domino will broker assignment of a new execution pod to the cluster. This pod will have an associated PVC which defines for Kubernetes what type of storage it requires. If an idle PV exists matching the PVC, Kubernetes will mount that PV on the node it assigns to host the pod, and the job or workspace will start. If an appropriate idle PV does not exist, Kubernetes will create a new PV according to the Storage Class.

When the user completes their workspace or job, the PV data will be written to the Domino File System, and the PV will be unmounted and sit idle until it is either reused for the user’s next job or garbage collected. By reusing PV’s, users who are actively working in a project will not need to copy data from the blob store to a PV repeatedly.

A job will only match with either a fresh PV or one previously used by that project. PV’s are not reused between projects.


Storage workflow for Workspaces

Workspace volumes are handled differently than volumes for Jobs. Workspaces are potentially long lived development environments that users will stop and resume repeatedly without writing data back to the Domino File System each time. As a result, the PV for the workspace is a similarly long-lived resource that stores the user’s working data.

These workspace PVs are durably associated with the durable workspace they are initially created for. Each time that workspace is stopped, the PV is detached and preserved so that it’s available the next time the user starts the workspace. When the workspace starts again, it reattaches its PV and the user will see all of their working data saved during the last session.

Only when a user chooses to initiate a sync will the contents of their project files in the workspace PV be written back to the Domino File System. A durable workspace PV will only be deleted if the user deletes the associated workspace.




Durable Workspace volume backups on AWS

Since the data in durable workspace volumes is not automatically written back to the Domino File System, there is a risk of lost work should the volume be lost or deleted. When Domino is running on AWS, it safeguards against this by backing up the EBS volume that backs the workspace PV with EBS snapshotting to S3. If you have accidentally deleted or lost a durable workspace volume that contains data you want to recover, contact Domino support for assistance in restoring from the snapshot.




Garbage collection

Note

This feature applies to Idle PVs used for jobs & scheduled jobs, not durable workspaces.

Domino has configurable values to help you tune your cluster to balance performance with cost controls. The more idle volumes you allow the more likely it is that users will be able to reuse a volume and avoid needing to copy project files from the blob store. However, this comes at the cost of keeping additional idle PVs.

By default, Domino will:

  • Limit the total number of idle PV’s to 32. This can be adjusted by setting the following option in the central config:
Namespace Key Value
common com.cerebro.domino.computegrid.kubernetes.volume.maxIdle Number (default: 32)
  • Terminate any idle PV that has not been used in 7 days. This can be adjusted by setting the following option in the central config:
Namespace Key Value
common com.cerebro.domino.computegrid.kubernetes.volume.maxAge Number of days (default: 7d)



Volume Size Configuration

Volume size can be configured by users on a per-project basis on the Project Settings page. Administrators can set the default volume size for new projects, as well as minimum and maximum volume size allowed. You can control the minimum, maximum, and default volume size for new volumes by editing the following central config values:

Note

This configuration option doesn’t apply to durable workspaces.

Namespace Key Value
common com.cerebro.domino.workbench.project.defaultVolumeSizeGiB Volume size in GB (default 10)
common com.cerebro.domino.workbench.project.minVolumeSizeGiB Volume size in GB (default 4)
common com.cerebro.domino.workbench.project.maxVolumeSizeGiB Volume size in GB (default 200)