fleetcommand-agent release notes

fleetcommand-agent v32 (December 2020)

Image: quay.io/domino/fleetcommand-agent:v32

Installation bundles:

• 4.3.3 images https://mirrors.domino.tech/s3/domino-artifacts/offline/opsless-v32-docker-images-4.3.3.tar

Changes:

• Fixes a memory leak in the EFS CSI driver.

fleetcommand-agent v31 (December 2020)

Image: quay.io/domino/fleetcommand-agent:v31

Installation bundles:

• 4.3.3 images https://mirrors.domino.tech/s3/domino-artifacts/offline/opsless-v31-docker-images-4.3.3.tar

Changes:

• Updates to latest build of 4.3.3.

fleetcommand-agent v30 (December 2020)

Image: quay.io/domino/fleetcommand-agent:v30

Installation bundles:

• 4.3.3 images https://mirrors.domino.tech/s3/domino-artifacts/offline/opsless-v30-docker-images-4.3.3.tar

Changes:

• Adds support for Domino 4.3.3

• The agent now supports installing Istio 1.7 (set istio.install to true), and installing Domino in Istio-compatible mode (set istio.enabled to true).

  istio:
    enabled: false
    install: false
    cni: true
    namespace: istio-system
  

• The EFS storage provider for new installs has changed from efs-provisioner to the EFS CSI driver, in order to support encryption in transit to EFS. For existing installs, this does not require any changes unless encryption in transit is desired. If a migration to encrypted EFS is necessary, please contact Domino support.

  One limitation of the new driver, compared to the previous, is an inability to dynamically create directories according to provisioned volumes. Support for pre-provisioned directories in AWS is done through access points, which must be created before Domino can be installed.

  To specify the access point at install time, ensure the filesystem_id is set in the format {EFS ID}::{AP ID}:

  storage_classes:
    shared:
      efs:
        filesystem_id: 'fs-285b532d::fsap-00cb72ba8ca35a121'
  

• Two new fields were added in order simplify DaemonSet management during upgrades for particularly large clusters. DaemonSets do not have configuration options for upgrades and pods will be replaced one-by-one. For large compute node pools, this can take a significant amount of time.

  helm:
    skip_daemonset_validation: false
    daemonset_timeout: 300
  

  Setting helm.skip_daemonset_validation to true will bypass post-upgrade validation that all pods have been successfully recreated. helm.daemonset_timeout is an integer representing the number of seconds to wait for all daemon pods in a DaemonSet to be recreated.

• 4.3.3 introduces limited availability of the new containerized Domino image builder: Forge. Forge can be enabled with the ImageBuilderV2 feature flag, although Domino services must be restarted to cause this flag to take effect. Running Domino image builds in a cluster that uses a non-Docker container runtime, such as cri-o or containerd, requires that the feature flag be enabled.

  To support the default rootless mode that Forge is configured to use, the worker nodes must support unprivileged mounts, user namespaces, and overlayfs (either natively or through FUSE). Currently, GKE and EKS do not support user namespace remapping and require the following extra configuration to properly use Forge.

  services:
    forge:
      chart_values:
        config:
          fullPrivilege: true
  

fleetcommand-agent v29 (November 2020)

Image: quay.io/domino/fleetcommand-agent:v29

Installation bundles:

• 4.3.2 images https://mirrors.domino.tech/s3/domino-artifacts/offline/opsless-v29-docker-images-4.3.2.tar

Changes:

• Updated Keycloak migration job version.

fleetcommand-agent v28 (November 2020)

Image: quay.io/domino/fleetcommand-agent:v28

Installation bundles:

• 4.3.2 images https://mirrors.domino.tech/s3/domino-artifacts/offline/opsless-v28-docker-images-4.3.2.tar

Changes:

• Adds support for Domino 4.3.2

• Adds support for encrypted EFS access by using the EFS CSI driver.

• A new istio field has been added to the domino.yml schema for testing and development of future releases. Domino 4.3.2 does not support Istio and therefore you must set enabled in this new section to false.

  istio:
    enabled: false
    install: false
    cni: true
    namespace: istio-system
  

• New fields to specify static AWS access key and secret key credentials have been added. These are currently unused and can be left unset.

  blob_storage:
    projects:
      s3:
        access_key_id: ''
        secret_access_key: ''
  

• A new field for Teleport remote access integration has been added. This is currently unused and should be set to false.

  teleport:
    remote_access: false
  

fleetcommand-agent v27 (October 2020)

Image: quay.io/domino/fleetcommand-agent:v27

Installation bundles:

• 4.3.1 images https://mirrors.domino.tech/s3/domino-artifacts/offline/opsless-v27-docker-images-4.3.1.tar

Changes:

• Fix a bug where dry-run installation could cause internal credentials to be improperly rotated.

fleetcommand-agent v26 (October 2020)

Image: quay.io/domino/fleetcommand-agent:v26

Installation bundles:

• 4.3.1 images https://mirrors.domino.tech/s3/domino-artifacts/offline/opsless-v26-docker-images-4.3.1.tar
• Latest charts http://mirrors.domino.tech/artifacts/appr/domino-appr-latest.tar.gz

Changes:

• Adds support for Domino 4.3.1

• Adds support for running Domino on OpenShift 4.4+.

• A new field has been added to the installer configuration that controls whether or not the image caching service is deployed.

  image_caching:
    enabled: true
  

• A new field has been added to the installer configuration that specifies the Kubernetes distribution for resource compatibility. The available options are cncf (Cloud Native Computing Foundation) and openshift.

  kubernetes_distribution: cncf
  

fleetcommand-agent v25 (August 2020)

Image: quay.io/domino/fleetcommand-agent:v25

Installation bundles:

• 4.3.0 images https://mirrors.domino.tech/s3/domino-artifacts/offline/opsless-v25-docker-images-4.3.0.tar
• Latest charts http://mirrors.domino.tech/artifacts/appr/domino-appr-latest.tar.gz

Changes:

• Adds support for Domino 4.3.0.

• A new cache_path field has been added to the helm configuration section. Leaving this field blank will ensure charts are fetched from an upstream repository.

  helm:
    cache_path: ''
  

• To facilitate deployment of Domino into clusters with other tenants, a new global node selector field has been added to the top-level configuration that allows an arbitrary label to be used for scheduling all Domino workloads. Its primary purpose is to limit workloads such as DaemonSets that would be scheduled on all available nodes in the cluster to only nodes with the provided label. Note that this can override default node pool selectors such as dominodatalab.com/node-pool: "platform", but does not replace them.

  global_node_selectors:
    domino-owned: "true"
  

• To facilitate deployment of Domino into clusters with other tenants, a configurable Ingress class has been added to allow differentiation from other ingress providers in a cluster. If multiple Ingress objects are created with the default class, it’s possible for other tenant’s paths to interfere with Domino and vice versa. Generally, this setting does not need to change, but can be set to any arbitrary string value (such as domino).

  ingress_controller:
    class_name: nginx
  

fleetcommand-agent v24 (July 2020)

Image: quay.io/domino/fleetcommand-agent:v24

Installation bundles:

• 4.2.4 images https://mirrors.domino.tech/s3/domino-artifacts/offline/opsless-docker-images-v24-4.2.4.tar
• Latest charts http://mirrors.domino.tech/artifacts/appr/domino-appr-latest.tar.gz

Changes:

• Adds support for Domino 4.2.3 and 4.2.4.

fleetcommand-agent v23 (May 2020)

Image: quay.io/domino/fleetcommand-agent:v23

Installation bundles:

• 4.2.2 images https://mirrors.domino.tech/s3/domino-artifacts/offline/opsless-docker-images-v23-4.2.2.tar
• Latest charts http://mirrors.domino.tech/artifacts/appr/domino-appr-latest.tar.gz

Changes:

• Adds support for Domino 4.2.2.

• The known issue with v22 around Domino Apps being stopped after upgrade has been resolved. Apps will now automatically restart after upgrade.

• The known issue with Elasticsearch not upgrading until manually restarted has been resolved. Elasticsearch will automatically cycle through a rolling upgrade when the deployment is upgraded.

• Fixed an issue that prevented the fleetcommand-agent

• Adds support for autodiscovery of scaling resources by the cluster autoscaler.

  Two new fields have been added under the autoscaler.auto_discovery key:

  autoscaler:
    auto_discovery:
      cluster_name: domino
      tags: [] # optional. if filled in, cluster_name is ignored.
  

  By default, if no autoscaler.groups or autoscaler.auto_discovery.tags are specified, the cluster_name will be used to look for the following AWS tags:

  • k8s.io/cluster-autoscaler/enabled
  • k8s.io/cluster-autoscaler/{{ cluster_name }}

  The tags parameter can be used to explicitly specify which resource tags the autoscaler service should look for. Auto scaling groups with matching tags will have their scaling properties detected and the autoscaler will be configured to scale them.

  Note that the IAM role for the platform nodes where the autoscaler is running still needs an autoscaling access policy that will allow it to read and scale the groups.

  When upgrading from an install that uses specific groups, ensure that auto_discovery.cluster_name is an empty value.

Known Issues:

• If you’re upgrading from fleetcommand-agent v21 or older, be sure to read the v22 release notes and implement the Helm configuration changes.

• An incompatibility between how nginx-ingress was initially installed and should be maintained going forward means that action is required for both Helm 2 and Helm 3 upgrades.

  For Helm 2 upgrades, add the following services object to your domino.yml to ensure compatibility:

  services:
    nginx_ingress:
      chart_values:
        controller:
          metrics:
            service:
              clusterIP: "-"
          service:
            clusterIP: "-"
  

  For Helm 3, there are two options. If nginx-ingress has not been configured to provide a cloud-native load balancer that is tied to the hosting DNS entry, then nginx-ingress can be safely uninstalled prior to the upgrade. If, however, the load balancer address must be maintained across the upgrade, then the initial upgrade after the Helm 3 migration will fail. Before retrying the upgrade, execute the following commands.

  export NAME=nginx-ingress
  export SECRET=$(kubectl get secret -l owner=helm,status=deployed,name=$NAME -n domino-platform | awk '{print $1}' | grep -v NAME)
  kubectl get secret -n domino-platform $SECRET -oyaml | sed "s/release:.*/release: $(kubectl get secret -n domino-platform $SECRET -ogo-template="{{ .data.release | base64decode | base64decode }}" | gzip -d - | sed 's/clusterIP: \\"\\"//g' | gzip | base64 -w0 | base64 -w0)/" | kubectl replace -f -
  kubectl get secret -n domino-platform $SECRET -oyaml | sed "s/release:.*/release: $(kubectl get secret -n domino-platform $SECRET -ogo-template="{{ .data.release | base64decode | base64decode }}" | gzip -d - | sed 's/rbac.authorization.k8s.io\/v1beta1/rbac.authorization.k8s.io\/v1/g' | gzip | base64 -w0 | base64 -w0)/" | kubectl replace -f -
  

fleetcommand-agent v22 (May 2020)

Image: quay.io/domino/fleetcommand-agent:v22

Installation bundles:

• 4.2.0 images https://mirrors.domino.tech/s3/domino-artifacts/offline/opsless-docker-images-v22-4.2.0.tar
• Latest charts http://mirrors.domino.tech/artifacts/appr/domino-appr-latest.tar.gz

Changes:

• Adds support for Domino 4.2.

• Adds support for Helm 3

  The helm object in the installer configuration has been restructured to accommodate Helm 3 support. There is now a helm.version property which can be set to 2 or 3. When using Helm 2, the configuration should be similar to the below example. The username and password will continue to be standard Quay.io credentials provided by Domino.

  helm:
    version: 2
    host: quay.io
    namespace: domino
    prefix: helm- # Prefix for the chart repository, defaults to `helm-`
    username: "<username>"
    password: "<password>"
    tiller_image: gcr.io/kubernetes-helm/tiller:v2.16.1 # Version is required and MUST be 2.16.1
    insecure: false
  

  When using Helm 3, configure the object as shown below. Helm 3 is a major release of the underlying tool that powers installation of Domino’s services. Helm 3 removes the Tiller service, which was the server-side component of Helm 2. This improves the security posture of Domino installation by reducing the scope and complexity of required RBAC permissions, and it enables namespace isolation of services. Additionally, Helm 3 adds support for storing charts in OCI registries.

  Currently, only gcr.io and mirrors.domino.tech are supported as chart repositories. If you are switching to Helm 3, you will need to contact Domino for gcr.io credentials. When using Helm 3, the helm configuration object should be similar to the below example

  helm:
    version: 3
    host: gcr.io
    namespace: domino-eng-service-artifacts
    insecure: false
    username: _json_key  # To support GCR authentication, this must be "_json_key"
    password: "<password>"
    tiller_image: null # Not required for Helm 3
    prefix: '' # Charts are stored without a prefix by default
  

  Migration of an existing Helm 2 installation to Helm 3 is done seamlessly within the installer. Once successful, Tiller will be removed from the cluster and all Helm 2 configuration is deleted.

Known Issues:

• Elasticsearch is currently configured to only upgrade when the pods are deleted. To properly upgrade an existing deployment from Elasticsearch 6.5 to 6.8, after running the installer use the rolling upgrade process. This involves first deleting the elasticsearch-data pods, then the elasticsearch-master pods. See the example procedure below.

  kubectl delete pods --namespace domino-platform elasticsearch-data-0 elasticsearch-data-1 --force=true --grace-period=0
  # Wait for elasticsearch-data-0 & elasticsearch-data-1 to come back online
  kubectl delete pods --namespace domino-platform elasticsearch-master-0 elasticsearch-master-1 elasticsearch-master-2
  

• An incompatibility between how nginx-ingress was initially installed and should be maintained going forward means that action is required for both Helm 2 and Helm 3 upgrades.

  For Helm 2 upgrades, add the following services object to your domino.yml to ensure compatibility:

  services:
    nginx_ingress:
      chart_values:
        controller:
          metrics:
            service:
              clusterIP: "-"
          service:
            clusterIP: "-"
  

  For Helm 3, there are two options. If nginx-ingress has not been configured to provide a cloud-native load balancer that is tied to the hosting DNS entry, then nginx-ingress can be safely uninstalled prior to the upgrade. If, however, the load balancer address must be maintained across the upgrade, then the initial upgrade after the Helm 3 migration will fail. Before retrying the upgrade, execute the following commands.

  export NAME=nginx-ingress
  export SECRET=$(kubectl get secret -l owner=helm,status=deployed,name=$NAME -n domino-platform | awk '{print $1}' | grep -v NAME)
  kubectl get secret -n domino-platform $SECRET -oyaml | sed "s/release:.*/release: $(kubectl get secret -n domino-platform $SECRET -ogo-template="{{ .data.release | base64decode | base64decode }}" | gzip -d - | sed 's/clusterIP: \\"\\"//g' | gzip | base64 -w0 | base64 -w0)/" | kubectl replace -f -
  kubectl get secret -n domino-platform $SECRET -oyaml | sed "s/release:.*/release: $(kubectl get secret -n domino-platform $SECRET -ogo-template="{{ .data.release | base64decode | base64decode }}" | gzip -d - | sed 's/rbac.authorization.k8s.io\/v1beta1/rbac.authorization.k8s.io\/v1/g' | gzip | base64 -w0 | base64 -w0)/" | kubectl replace -f -
  

• Domino Apps do not currently support a live upgrade from version 4.1 to version 4.2. After the upgrade, all Apps will be stopped.

  To restart them, you can use the /v4/modelProducts/restartAll endpoint like in the below example, providing an API key for a system administrator.

  curl -X POST --include --header "X-Domino-Api-Key: <admin-api-key>" 'https://<domino-url>/v4/modelProducts/restartAll'
  

fleetcommand-agent v21 (May 2020)

Image: quay.io/domino/fleetcommand-agent:v21

Changes:

• Adds support for Domino 4.1.10

Known issues:

• The deployed version 8.0.1 of Keycloak has an incorrect default First Broker Login authentication flow.

  When setting up an SSO integration, you must create a new authentication flow like the one below. Note that the Automatically Link Account step is a custom flow, and the Create User if Unique and Automatically Set Existing User executions must be nested under it by adding them with the Actions link.

  

fleetcommand-agent v20 (March 2020)

Image: quay.io/domino/fleetcommand-agent:v20

Changes:

• Support for 4.1.9 has been updated to reflect a new set of artifacts.

Known issues:

• The deployed version 8.0.1 of Keycloak has an incorrect default First Broker Login authentication flow.

  When setting up an SSO integration, you must create a new authentication flow like the one below. Note that the Automatically Link Account step is a custom flow, and the Create User if Unique and Automatically Set Existing User executions must be nested under it by adding them with the Actions link.

  

fleetcommand-agent v19 (March 2020)

Image: quay.io/domino/fleetcommand-agent:v19

Changes:

• Added catalogs for Domino up to 4.1.9
• Added support for Docker NO_PROXY configuration. Domino containers will now respect the configuration and connect to the specified hosts without proxy.

Known issues:

• The deployed version 8.0.1 of Keycloak has an incorrect default First Broker Login authentication flow.

  When setting up an SSO integration, you must create a new authentication flow like the one below. Note that the Automatically Link Account step is a custom flow, and the Create User if Unique and Automatically Set Existing User executions must be nested under it by adding them with the Actions link.

  

fleetcommand-agent v18 (March 2020)

Image: quay.io/domino/fleetcommand-agent:v18

Changes:

The following new fields have been added to the fleetcommand-agent installer configuration.

1. Storage class access modes

   The storage_class options have a new field called access_modes that allows configuration of the underlying storage class’ allowed access modes.

   storage_classes:
     block:
       [snip]
       access_modes:
       - ReadWriteOnce
   

2. Git service storage class

   Previously, the deployed Domino Git service used storage backed by the shared storage class. Now, the dominodisk block storage class will be used by default. If using custom storage classes, set this to the name of the block storage class. For existing Domino installations, you must set this to dominoshared.

   For new installs:

   git:
     storage_class: dominodisk
   

   For existing installations and upgrades:

   git:
     storage_class: dominoshared
   services:
     git_server:
       chart_values:
         persistence:
           size: 5Ti
   

fleetcommand-agent v17 (March 2020)

Image: quay.io/domino/fleetcommand-agent:v17

Changes:

• Added catalogs for Domino up to 4.1.8

fleetcommand-agent v16 (February 2020)

Image: quay.io/domino/fleetcommand-agent:v16

Changes:

• Added catalogs for Domino up to 4.1.7
• Calico CNI is now installed by default for EKS deployments
• AWS Metadata API is blocked by default for Domino version >= 4.1.5
• Added Private registry support in the Installer
• New Install configuration attributes (see the reference documentation for more details): - sse_kms_key_id option for Blob storage - gcs option for Google Cloud Storage - Namespaces now support optional the labels to apply labels during installation - teleport for Domino managed installations only

fleetcommand-agent v15 (January 2020)

Image: quay.io/domino/fleetcommand-agent:v15

Changes:

• Added catalog for Domino 4.1.4
• Ensure fleetcommand-agent also deletes system namespace.
• Updated version of Cluster Autoscaler to 1.13.9

fleetcommand-agent v14 (January 2020)

Image: quay.io/domino/fleetcommand-agent:v14

Changes:

• Updated version of Cluster Autoscaler to 1.13.7
• Added catalog for Domino 4.1.3