fleetcommand-agent release notes

fleetcommand-agent v38 (May 2021)

Image: quay.io/domino/fleetcommand-agent:v38

Installation bundles:

  • 4.4.2 images https://mirrors.domino.tech/s3/domino-artifacts/offline/opsless-v38-docker-images-4.4.2.tar

Changes

  • Adds support for Domino 4.4.2

  • Included in Domino 4.4.2 is opt-in support for FUSE volume usage in the compute grid. This feature currently requires configuration changes at installation or upgrade time due to the potential security implications. To enable FUSE volume support for compute grid executions, set the following:

    services:
  nucleus:
    chart_values:
      enableFuseVolumeSupport: true

  • A new configuration has been added in preparation for future feature development. These fields are a required part of the v38 schema but currently have no effect.

    certificate_management:
  enabled: true

fleetcommand-agent v37 (March 2021)

Image: quay.io/domino/fleetcommand-agent:v37

Installation bundles:

  • 4.4.1 images https://mirrors.domino.tech/s3/domino-artifacts/offline/opsless-v37-docker-images-4.4.1.tar

Changes

  • Adds support for Domino 4.4.1
  • The teleport section of the agent configuration has been deprecated in favor of the new teleport_kube_agent section.

fleetcommand-agent v34 (February 2021)

Image: quay.io/domino/fleetcommand-agent:v34

Installation bundles:

  • 4.4.0 images https://mirrors.domino.tech/s3/domino-artifacts/offline/opsless-v34-docker-images-4.4.0.tar

Changes

  • Adds support for Domino 4.4.0

fleetcommand-agent v33 (February 2021)

Image: quay.io/domino/fleetcommand-agent:v33

Installation bundles:

  • 4.4.0 images https://mirrors.domino.tech/s3/domino-artifacts/offline/opsless-v33-docker-images-4.4.0.tar

Changes

  • Adds support for Domino 4.4.0

  • New configuration options have been added for the new Teleport Kubernetes agent:

    If a deployment currently has teleport.enabled and teleport.remote_access set to true, they should be disabled and teleport_kube_agent.enabled should be set instead.

    teleport_kube_agent:
  enabled: false
  proxyAddr: teleport.domino.tech:443
  authToken: eeceeV4sohh8eew0Oa1aexoTahm3Eiha

  • Domino 4.4.0 includes support for restartable workspace disaster recovery in AWS leveraging EBS snapshots. To support this functionality, existing installations may potentially require additional IAM permissions for platform node pool instances.

    The permissions required, without any resource restriction (i.e. *), are the following:

    • ec2:CreateSnapshot
    • ec2:CreateTags
    • ec2:DeleteSnapshot
    • ec2:DeleteTags
    • ec2:DescribeAvailabilityZones
    • ec2:DescribeSnapshots
    • ec2:DescribeTags

Known Issues

  • If upgrading from Helm 2 to Helm 3, please read the release notes from v22 for caveats and known issues.

fleetcommand-agent v32 (December 2020)

Image: quay.io/domino/fleetcommand-agent:v32

Installation bundles:

  • 4.3.3 images https://mirrors.domino.tech/s3/domino-artifacts/offline/opsless-v32-docker-images-4.3.3.tar

Changes:

  • Fixes a memory leak in the EFS CSI driver.

fleetcommand-agent v31 (December 2020)

Image: quay.io/domino/fleetcommand-agent:v31

Installation bundles:

  • 4.3.3 images https://mirrors.domino.tech/s3/domino-artifacts/offline/opsless-v31-docker-images-4.3.3.tar

Changes:

  • Updates to latest build of 4.3.3.

fleetcommand-agent v30 (December 2020)

Image: quay.io/domino/fleetcommand-agent:v30

Installation bundles:

  • 4.3.3 images https://mirrors.domino.tech/s3/domino-artifacts/offline/opsless-v30-docker-images-4.3.3.tar

Changes:

  • Adds support for Domino 4.3.3

  • The agent now supports installing Istio 1.7 (set istio.install to true), and installing Domino in Istio-compatible mode (set istio.enabled to true).

    istio:
  enabled: false
  install: false
  cni: true
  namespace: istio-system

  • The EFS storage provider for new installs has changed from efs-provisioner to the EFS CSI driver, in order to support encryption in transit to EFS. For existing installs, this does not require any changes unless encryption in transit is desired. If a migration to encrypted EFS is necessary, please contact Domino support.

    One limitation of the new driver, compared to the previous, is an inability to dynamically create directories according to provisioned volumes. Support for pre-provisioned directories in AWS is done through access points, which must be created before Domino can be installed.

    To specify the access point at install time, ensure the filesystem_id is set in the format {EFS ID}::{AP ID}:

    storage_classes:
  shared:
    efs:
      filesystem_id: 'fs-285b532d::fsap-00cb72ba8ca35a121'

  • Two new fields were added in order simplify DaemonSet management during upgrades for particularly large clusters. DaemonSets do not have configuration options for upgrades and pods will be replaced one-by-one. For large compute node pools, this can take a significant amount of time.

    helm:
  skip_daemonset_validation: false
  daemonset_timeout: 300

    Setting helm.skip_daemonset_validation to true will bypass post-upgrade validation that all pods have been successfully recreated. helm.daemonset_timeout is an integer representing the number of seconds to wait for all daemon pods in a DaemonSet to be recreated.

  • 4.3.3 introduces limited availability of the new containerized Domino image builder: Forge. Forge can be enabled with the ImageBuilderV2 feature flag, although Domino services must be restarted to cause this flag to take effect. Running Domino image builds in a cluster that uses a non-Docker container runtime, such as cri-o or containerd, requires that the feature flag be enabled.

    To support the default rootless mode that Forge is configured to use, the worker nodes must support unprivileged mounts, user namespaces, and overlayfs (either natively or through FUSE). Currently, GKE and EKS do not support user namespace remapping and require the following extra configuration to properly use Forge.

    services:
  forge:
    chart_values:
      config:
        fullPrivilege: true

fleetcommand-agent v29 (November 2020)

Image: quay.io/domino/fleetcommand-agent:v29

Installation bundles:

  • 4.3.2 images https://mirrors.domino.tech/s3/domino-artifacts/offline/opsless-v29-docker-images-4.3.2.tar

Changes:

  • Updated Keycloak migration job version.

fleetcommand-agent v28 (November 2020)

Image: quay.io/domino/fleetcommand-agent:v28

Installation bundles:

  • 4.3.2 images https://mirrors.domino.tech/s3/domino-artifacts/offline/opsless-v28-docker-images-4.3.2.tar

Changes:

  • Adds support for Domino 4.3.2

  • Adds support for encrypted EFS access by using the EFS CSI driver.

  • A new istio field has been added to the domino.yml schema for testing and development of future releases. Domino 4.3.2 does not support Istio and therefore you must set enabled in this new section to false.

    istio:
  enabled: false
  install: false
  cni: true
  namespace: istio-system

  • New fields to specify static AWS access key and secret key credentials have been added. These are currently unused and can be left unset.

    blob_storage:
  projects:
    s3:
      access_key_id: ''
      secret_access_key: ''

  • A new field for Teleport remote access integration has been added. This is currently unused and should be set to false.

    teleport:
  remote_access: false

fleetcommand-agent v27 (October 2020)

Image: quay.io/domino/fleetcommand-agent:v27

Installation bundles:

  • 4.3.1 images https://mirrors.domino.tech/s3/domino-artifacts/offline/opsless-v27-docker-images-4.3.1.tar

Changes:

  • Fix a bug where dry-run installation could cause internal credentials to be improperly rotated.

fleetcommand-agent v26 (October 2020)

Image: quay.io/domino/fleetcommand-agent:v26

Installation bundles:

  • 4.3.1 images https://mirrors.domino.tech/s3/domino-artifacts/offline/opsless-v26-docker-images-4.3.1.tar
  • Latest charts http://mirrors.domino.tech/artifacts/appr/domino-appr-latest.tar.gz

Changes:

  • Adds support for Domino 4.3.1

  • Adds support for running Domino on OpenShift 4.4+.

  • A new field has been added to the installer configuration that controls whether or not the image caching service is deployed.

    image_caching:
  enabled: true

  • A new field has been added to the installer configuration that specifies the Kubernetes distribution for resource compatibility. The available options are cncf (Cloud Native Computing Foundation) and openshift.

    kubernetes_distribution: cncf

fleetcommand-agent v25 (August 2020)

Image: quay.io/domino/fleetcommand-agent:v25

Installation bundles:

  • 4.3.0 images https://mirrors.domino.tech/s3/domino-artifacts/offline/opsless-v25-docker-images-4.3.0.tar
  • Latest charts http://mirrors.domino.tech/artifacts/appr/domino-appr-latest.tar.gz

Changes:

  • Adds support for Domino 4.3.0.

  • A new cache_path field has been added to the helm configuration section. Leaving this field blank will ensure charts are fetched from an upstream repository.

    helm:
  cache_path: ''

  • To facilitate deployment of Domino into clusters with other tenants, a new global node selector field has been added to the top-level configuration that allows an arbitrary label to be used for scheduling all Domino workloads. Its primary purpose is to limit workloads such as DaemonSets that would be scheduled on all available nodes in the cluster to only nodes with the provided label. Note that this can override default node pool selectors such as dominodatalab.com/node-pool: "platform", but does not replace them.

    global_node_selectors:
  domino-owned: "true"

  • To facilitate deployment of Domino into clusters with other tenants, a configurable Ingress class has been added to allow differentiation from other ingress providers in a cluster. If multiple Ingress objects are created with the default class, it’s possible for other tenant’s paths to interfere with Domino and vice versa. Generally, this setting does not need to change, but can be set to any arbitrary string value (such as domino).

    ingress_controller:
  class_name: nginx

fleetcommand-agent v24 (July 2020)

Image: quay.io/domino/fleetcommand-agent:v24

Installation bundles:

  • 4.2.4 images https://mirrors.domino.tech/s3/domino-artifacts/offline/opsless-docker-images-v24-4.2.4.tar
  • Latest charts http://mirrors.domino.tech/artifacts/appr/domino-appr-latest.tar.gz

Changes:

  • Adds support for Domino 4.2.3 and 4.2.4.

fleetcommand-agent v23 (May 2020)

Image: quay.io/domino/fleetcommand-agent:v23

Installation bundles:

  • 4.2.2 images https://mirrors.domino.tech/s3/domino-artifacts/offline/opsless-docker-images-v23-4.2.2.tar
  • Latest charts http://mirrors.domino.tech/artifacts/appr/domino-appr-latest.tar.gz

Changes:

  • Adds support for Domino 4.2.2.

  • The known issue with v22 around Domino Apps being stopped after upgrade has been resolved. Apps will now automatically restart after upgrade.

  • The known issue with Elasticsearch not upgrading until manually restarted has been resolved. Elasticsearch will automatically cycle through a rolling upgrade when the deployment is upgraded.

  • Fixed an issue that prevented the fleetcommand-agent

  • Adds support for autodiscovery of scaling resources by the cluster autoscaler.

    Two new fields have been added under the autoscaler.auto_discovery key:

    autoscaler:
  auto_discovery:
    cluster_name: domino
    tags: [] # optional. if filled in, cluster_name is ignored.

    By default, if no autoscaler.groups or autoscaler.auto_discovery.tags are specified, the cluster_name will be used to look for the following AWS tags:

    • k8s.io/cluster-autoscaler/enabled
    • k8s.io/cluster-autoscaler/{{ cluster_name }}

    The tags parameter can be used to explicitly specify which resource tags the autoscaler service should look for. Auto scaling groups with matching tags will have their scaling properties detected and the autoscaler will be configured to scale them.

    Note that the IAM role for the platform nodes where the autoscaler is running still needs an autoscaling access policy that will allow it to read and scale the groups.

    When upgrading from an install that uses specific groups, ensure that auto_discovery.cluster_name is an empty value.

Known Issues:

  • If you’re upgrading from fleetcommand-agent v21 or older, be sure to read the v22 release notes and implement the Helm configuration changes.

  • An incompatibility between how nginx-ingress was initially installed and should be maintained going forward means that action is required for both Helm 2 and Helm 3 upgrades.

    For Helm 2 upgrades, add the following services object to your domino.yml to ensure compatibility:

    services:
  nginx_ingress:
    chart_values:
      controller:
        metrics:
          service:
            clusterIP: "-"
        service:
          clusterIP: "-"

    For Helm 3, there are two options. If nginx-ingress has not been configured to provide a cloud-native load balancer that is tied to the hosting DNS entry, then nginx-ingress can be safely uninstalled prior to the upgrade. If, however, the load balancer address must be maintained across the upgrade, then the initial upgrade after the Helm 3 migration will fail. Before retrying the upgrade, execute the following commands.

    export NAME=nginx-ingress
export SECRET=$(kubectl get secret -l owner=helm,status=deployed,name=$NAME -n domino-platform | awk '{print $1}' | grep -v NAME)
kubectl get secret -n domino-platform $SECRET -oyaml | sed "s/release:.*/release: $(kubectl get secret -n domino-platform $SECRET -ogo-template="{{ .data.release | base64decode | base64decode }}" | gzip -d - | sed 's/clusterIP: \\"\\"//g' | gzip | base64 -w0 | base64 -w0)/" | kubectl replace -f -
kubectl get secret -n domino-platform $SECRET -oyaml | sed "s/release:.*/release: $(kubectl get secret -n domino-platform $SECRET -ogo-template="{{ .data.release | base64decode | base64decode }}" | gzip -d - | sed 's/rbac.authorization.k8s.io\/v1beta1/rbac.authorization.k8s.io\/v1/g' | gzip | base64 -w0 | base64 -w0)/" | kubectl replace -f -

fleetcommand-agent v22 (May 2020)

Image: quay.io/domino/fleetcommand-agent:v22

Installation bundles:

  • 4.2.0 images https://mirrors.domino.tech/s3/domino-artifacts/offline/opsless-docker-images-v22-4.2.0.tar
  • Latest charts http://mirrors.domino.tech/artifacts/appr/domino-appr-latest.tar.gz

Changes:

  • Adds support for Domino 4.2.

  • Adds support for Helm 3

    The helm object in the installer configuration has been restructured to accommodate Helm 3 support. There is now a helm.version property which can be set to 2 or 3. When using Helm 2, the configuration should be similar to the below example. The username and password will continue to be standard Quay.io credentials provided by Domino.

    helm:
  version: 2
  host: quay.io
  namespace: domino
  prefix: helm- # Prefix for the chart repository, defaults to `helm-`
  username: "<username>"
  password: "<password>"
  tiller_image: gcr.io/kubernetes-helm/tiller:v2.16.1 # Version is required and MUST be 2.16.1
  insecure: false

    When using Helm 3, configure the object as shown below. Helm 3 is a major release of the underlying tool that powers installation of Domino’s services. Helm 3 removes the Tiller service, which was the server-side component of Helm 2. This improves the security posture of Domino installation by reducing the scope and complexity of required RBAC permissions, and it enables namespace isolation of services. Additionally, Helm 3 adds support for storing charts in OCI registries.

    Currently, only gcr.io and mirrors.domino.tech are supported as chart repositories. If you are switching to Helm 3, you will need to contact Domino for gcr.io credentials. When using Helm 3, the helm configuration object should be similar to the below example

    helm:
  version: 3
  host: gcr.io
  namespace: domino-eng-service-artifacts
  insecure: false
  username: _json_key  # To support GCR authentication, this must be "_json_key"
  password: "<password>"
  tiller_image: null # Not required for Helm 3
  prefix: '' # Charts are stored without a prefix by default

    Migration of an existing Helm 2 installation to Helm 3 is done seamlessly within the installer. Once successful, Tiller will be removed from the cluster and all Helm 2 configuration is deleted.

Known Issues:

  • Elasticsearch is currently configured to only upgrade when the pods are deleted. To properly upgrade an existing deployment from Elasticsearch 6.5 to 6.8, after running the installer use the rolling upgrade process. This involves first deleting the elasticsearch-data pods, then the elasticsearch-master pods. See the example procedure below.

    kubectl delete pods --namespace domino-platform elasticsearch-data-0 elasticsearch-data-1 --force=true --grace-period=0
# Wait for elasticsearch-data-0 & elasticsearch-data-1 to come back online
kubectl delete pods --namespace domino-platform elasticsearch-master-0 elasticsearch-master-1 elasticsearch-master-2

  • An incompatibility between how nginx-ingress was initially installed and should be maintained going forward means that action is required for both Helm 2 and Helm 3 upgrades.

    For Helm 2 upgrades, add the following services object to your domino.yml to ensure compatibility:

    services:
  nginx_ingress:
    chart_values:
      controller:
        metrics:
          service:
            clusterIP: "-"
        service:
          clusterIP: "-"

    For Helm 3, there are two options. If nginx-ingress has not been configured to provide a cloud-native load balancer that is tied to the hosting DNS entry, then nginx-ingress can be safely uninstalled prior to the upgrade. If, however, the load balancer address must be maintained across the upgrade, then the initial upgrade after the Helm 3 migration will fail. Before retrying the upgrade, execute the following commands.

    export NAME=nginx-ingress
export SECRET=$(kubectl get secret -l owner=helm,status=deployed,name=$NAME -n domino-platform | awk '{print $1}' | grep -v NAME)
kubectl get secret -n domino-platform $SECRET -oyaml | sed "s/release:.*/release: $(kubectl get secret -n domino-platform $SECRET -ogo-template="{{ .data.release | base64decode | base64decode }}" | gzip -d - | sed 's/clusterIP: \\"\\"//g' | gzip | base64 -w0 | base64 -w0)/" | kubectl replace -f -
kubectl get secret -n domino-platform $SECRET -oyaml | sed "s/release:.*/release: $(kubectl get secret -n domino-platform $SECRET -ogo-template="{{ .data.release | base64decode | base64decode }}" | gzip -d - | sed 's/rbac.authorization.k8s.io\/v1beta1/rbac.authorization.k8s.io\/v1/g' | gzip | base64 -w0 | base64 -w0)/" | kubectl replace -f -

  • Domino Apps do not currently support a live upgrade from version 4.1 to version 4.2. After the upgrade, all Apps will be stopped.

    To restart them, you can use the /v4/modelProducts/restartAll endpoint like in the below example, providing an API key for a system administrator.

    curl -X POST --include --header "X-Domino-Api-Key: <admin-api-key>" 'https://<domino-url>/v4/modelProducts/restartAll'

fleetcommand-agent v21 (May 2020)

Image: quay.io/domino/fleetcommand-agent:v21

Changes:

  • Adds support for Domino 4.1.10

Known issues:

  • The deployed version 8.0.1 of Keycloak has an incorrect default First Broker Login authentication flow.

    When setting up an SSO integration, you must create a new authentication flow like the one below. Note that the Automatically Link Account step is a custom flow, and the Create User if Unique and Automatically Set Existing User executions must be nested under it by adding them with the Actions link.

    ../_images/alternative_flow.png

fleetcommand-agent v20 (March 2020)

Image: quay.io/domino/fleetcommand-agent:v20

Changes:

  • Support for 4.1.9 has been updated to reflect a new set of artifacts.

Known issues:

  • The deployed version 8.0.1 of Keycloak has an incorrect default First Broker Login authentication flow.

    When setting up an SSO integration, you must create a new authentication flow like the one below. Note that the Automatically Link Account step is a custom flow, and the Create User if Unique and Automatically Set Existing User executions must be nested under it by adding them with the Actions link.

    ../_images/alternative_flow.png

fleetcommand-agent v19 (March 2020)

Image: quay.io/domino/fleetcommand-agent:v19

Changes:

  • Added catalogs for Domino up to 4.1.9
  • Added support for Docker NO_PROXY configuration. Domino containers will now respect the configuration and connect to the specified hosts without proxy.

Known issues:

  • The deployed version 8.0.1 of Keycloak has an incorrect default First Broker Login authentication flow.

    When setting up an SSO integration, you must create a new authentication flow like the one below. Note that the Automatically Link Account step is a custom flow, and the Create User if Unique and Automatically Set Existing User executions must be nested under it by adding them with the Actions link.

    ../_images/alternative_flow.png

fleetcommand-agent v18 (March 2020)

Image: quay.io/domino/fleetcommand-agent:v18

Changes:

The following new fields have been added to the fleetcommand-agent installer configuration.

  1. Storage class access modes

    The storage_class options have a new field called access_modes that allows configuration of the underlying storage class’ allowed access modes.

    storage_classes:
  block:
    [snip]
    access_modes:
    - ReadWriteOnce

  2. Git service storage class

    Previously, the deployed Domino Git service used storage backed by the shared storage class. Now, the dominodisk block storage class will be used by default. If using custom storage classes, set this to the name of the block storage class. For existing Domino installations, you must set this to dominoshared.

    For new installs:

    git:
  storage_class: dominodisk

    For existing installations and upgrades:

    git:
  storage_class: dominoshared
services:
  git_server:
    chart_values:
      persistence:
        size: 5Ti

fleetcommand-agent v17 (March 2020)

Image: quay.io/domino/fleetcommand-agent:v17

Changes:

  • Added catalogs for Domino up to 4.1.8

fleetcommand-agent v16 (February 2020)

Image: quay.io/domino/fleetcommand-agent:v16

Changes:

  • Added catalogs for Domino up to 4.1.7
  • Calico CNI is now installed by default for EKS deployments
  • AWS Metadata API is blocked by default for Domino version >= 4.1.5
  • Added Private registry support in the Installer
  • New Install configuration attributes (see the reference documentation for more details):
    • sse_kms_key_id option for Blob storage
    • gcs option for Google Cloud Storage
    • Namespaces now support optional labels to apply labels during installation
    • teleport for Domino managed installations only

fleetcommand-agent v15 (January 2020)

Image: quay.io/domino/fleetcommand-agent:v15

Changes:

  • Added catalog for Domino 4.1.4
  • Ensure fleetcommand-agent also deletes system namespace.
  • Updated version of Cluster Autoscaler to 1.13.9

fleetcommand-agent v14 (January 2020)

Image: quay.io/domino/fleetcommand-agent:v14

Changes: