domino logo
About DominoArchitecture
Kubernetes
Cluster RequirementsDomino on EKSDomino Kubernetes Version CompatibilityDomino on GKEDomino on AKSDomino on OpenShiftNVIDIA DGX in DominoDomino in Multi-Tenant Kubernetes ClusterEncryption in Transit
Installation
Installation ProcessConfiguration ReferenceInstaller Configuration ExamplesPrivate or Offline Installationfleetcommand-agent release notes
Azure Deployments
Prepare for InstallationProvision Infrastructure and Runtime EnvironmentDeploy Domino
Google Cloud Deployments
Prepare for InstallationProvision Infrastructure and Runtime EnvironmentDeploy Domino
Amazon Web Services Deployments
Prepare for InstallationProvision Infrastructure and Runtime EnvironmentDeploy Domino
Configuration
Central ConfigurationNotificationsChange The Default Project For New UsersProject Stage ConfigurationDomino Integration With Atlassian Jira
Compute
Manage Domino Compute ResourcesHardware Tier Best PracticesModel Resource QuotasPersistent Volume ManagementAdding a Node Pool to your Domino ClusterRemove a Node from Service
Keycloak Authentication Service
Operations
Domino Application LoggingDomino MonitoringSizing Infrastructure for Domino
Data Management
Data in DominoData Flow In DominoExternal Data VolumesDatasets AdministrationSubmit GDPR Requests
User Management
RolesView User InformationRun a User Activity ReportSchedule a User Activity Report
Environments
Environment Management Best PracticesCache Environment Images in EKS
Backup and Restore
Backup StructureBackup LocationCustomize BackupsRun a Manual, On-Demand BackupRestore backups
Control Center
Control Center OverviewExport Control Center Data with The API
domino logo
About Domino
Domino Data LabKnowledge BaseData Science BlogTraining
Admin Guide
>
Compute
>
Adding a Node Pool to your Domino Cluster

Adding a Node Pool to your Domino Cluster

Making a new node group available to Domino is as simple as adding new Kubernetes worker nodes with a distinct dominodatalab.com/node-pool label. You can then reference the value of that label when creating new hardware tiers to configure Domino to assign executions to those nodes.

See below for an example of creating a scalable node pool in EKS.

Creating a scalable node pool in EKS

This example shows how to create a new node group with eksctl and expose it to the cluster autoscaler as a labeled Domino node pool.

  1. Create a new-nodegroup.yaml file like the one below, and configure it with the properties you want the new group to have. All values shown with a $ are variables that you must modify.

    apiVersion: eksctl.io/v1alpha5
    kind: ClusterConfig
    metadata:
      name: $CLUSTER_NAME
      region: $CLUSTER_REGION
    nodeGroups:
      - name: $GROUP_NAME # this can be any name you choose, it will be part of the ASG and template name
        instanceType: $AWS_INSTANCE_TYPE
        minSize: $DMINIMUM_GROUP_SIZE
        maxSize: $DESIRED_MAXIMUM_GROUP_SIZE
        volumeSize: 400 # important to allow for image caching on Domino workers
        availabilityZones: ["$YOUR_CHOICE"] # this should be the same AZ (or the same multiple AZ's) as your other node pools
        ami:
          $AMI_ID
        labels:
          "dominodatalab.com/node-pool": "$NODE_POOL_NAME" # this is the name you'll reference from Domino
          # "nvidia.com/gpu": "true" # uncomment this line if this pool uses a GPU instance type
        tags:
          "k8s.io/cluster-autoscaler/node-template/label/dominodatalab.com/node-pool": "$NODE_POOL_NAME"
          # "k8s.io/cluster-autoscaler/node-template/label/nvidia.com/gpu": "true" # uncomment this line if this pool uses a GPU instance type

    The AWS tag with key k8s.io/cluster-autoscaler/node-template/label/dominodatalab.com/node-pool is important for exposing the group to your cluster autoscaler.

    Note also that you cannot have compute node pools in separate, isolated AZ’s as this creates volume affinity errors.

  2. Once your configuration file describes the group you want to create, run eksctl create nodegroup --config-file=new-nodegroup.yaml.

  3. Take the names of the resulting ASG and add them to the autoscaling.groups section of your domino.yml installer configuration.

  4. Run the Domino installer to update the autoscaler.

  5. Create a new hardware tiers in Domino that references the new labels.

When finished, you can start Domino executions that use the new Hardware Tier and those executions will be assigned to nodes in the new group, which will be scaled as configured by the cluster autoscaler.

Domino Data LabKnowledge BaseData Science BlogTraining
Copyright © 2022 Domino Data Lab. All rights reserved.