fleetcommand-agent release notes


fleetcommand-agent v25 (August 2020)

Image: quay.io/domino/fleetcommand-agent:v25

Installation bundles:

  • 4.3.0 images https://mirrors.domino.tech/s3/domino-artifacts/offline/opsless-v25-docker-images-4.3.0.tar
  • Latest charts http://mirrors.domino.tech/artifacts/appr/domino-appr-latest.tar.gz

Changes:

  • Adds support for Domino 4.3.0.

  • A new cache_path field has been added to the helm configuration section. Leaving this field blank will ensure charts are fetched from an upstream repository.

    helm:
      cache_path: ''
    
  • To facilitate deployment of Domino into clusters with other tenants, a new global node selector field has been added to the top-level configuration that allows an arbitrary label to be used for scheduling all Domino workloads. Its primary purpose is to limit workloads such as DaemonSets that would be scheduled on all available nodes in the cluster to only nodes with the provided label. Note that this can override default node pool selectors such as dominodatalab.com/node-pool: "platform", but does not replace them.

    global_node_selectors:
      domino-owned: "true"
    
  • To facilitate deployment of Domino into clusters with other tenants, a configurable Ingress class has been added to allow differentiation from other ingress providers in a cluster. If multiple Ingress objects are created with the default class, it’s possible for other tenant’s paths to interfere with Domino and vice versa. Generally, this setting does not need to change, but can be set to any arbitrary string value (such as domino).

    ingress_controller:
      class_name: nginx
    

fleetcommand-agent v24 (July 2020)

Image: quay.io/domino/fleetcommand-agent:v24

Installation bundles:

  • 4.2.4 images https://mirrors.domino.tech/s3/domino-artifacts/offline/opsless-docker-images-v24-4.2.4.tar
  • Latest charts http://mirrors.domino.tech/artifacts/appr/domino-appr-latest.tar.gz

Changes:

  • Adds support for Domino 4.2.3 and 4.2.4.

fleetcommand-agent v23 (May 2020)

Image: quay.io/domino/fleetcommand-agent:v23

Installation bundles:

  • 4.2.2 images https://mirrors.domino.tech/s3/domino-artifacts/offline/opsless-docker-images-v23-4.2.2.tar
  • Latest charts http://mirrors.domino.tech/artifacts/appr/domino-appr-latest.tar.gz

Changes:

  • Adds support for Domino 4.2.2.

  • The known issue with v22 around Domino Apps being stopped after upgrade has been resolved. Apps will now automatically restart after upgrade.

  • The known issue with Elasticsearch not upgrading until manually restarted has been resolved. Elasticsearch will automatically cycle through a rolling upgrade when the deployment is upgraded.

  • Fixed an issue that prevented the fleetcommand-agent

  • Adds support for autodiscovery of scaling resources by the cluster autoscaler.

    Two new fields have been added under the autoscaler.auto_discovery key:

    autoscaler:
      auto_discovery:
        cluster_name: domino
        tags: [] # optional. if filled in, cluster_name is ignored.
    

    By default, if no autoscaler.groups or autoscaler.auto_discovery.tags are specified, the cluster_name will be used to look for the following AWS tags:

    • k8s.io/cluster-autoscaler/enabled
    • k8s.io/cluster-autoscaler/{{ cluster_name }}

    The tags parameter can be used to explicitly specify which resource tags the autoscaler service should look for. Auto scaling groups with matching tags will have their scaling properties detected and the autoscaler will be configured to scale them.

    Note that the IAM role for the platform nodes where the autoscaler is running still needs an autoscaling access policy that will allow it to read and scale the groups.

    When upgrading from an install that uses specific groups, ensure that auto_discovery.cluster_name is an empty value.

Known Issues:

  • If you’re upgrading from fleetcommand-agent v21 or older, be sure to read the v22 release notes and implement the Helm configuration changes.

  • An incompatibility between how nginx-ingress was initially installed and should be maintained going forward means that action is required for both Helm 2 and Helm 3 upgrades.

    For Helm 2 upgrades, add the following services object to your domino.yml to ensure compatibility:

    services:
      nginx_ingress:
        chart_values:
          controller:
            metrics:
              service:
                clusterIP: "-"
            service:
              clusterIP: "-"
    

    For Helm 3, there are two options. If nginx-ingress has not been configured to provide a cloud-native load balancer that is tied to the hosting DNS entry, then nginx-ingress can be safely uninstalled prior to the upgrade. If, however, the load balancer address must be maintained across the upgrade, then the initial upgrade after the Helm 3 migration will fail. Before retrying the upgrade, execute the following commands.

    export NAME=nginx-ingress
    export SECRET=$(kubectl get secret -l owner=helm,status=deployed,name=$NAME -n domino-platform | awk '{print $1}' | grep -v NAME)
    kubectl get secret -n domino-platform $SECRET -oyaml | sed "s/release:.*/release: $(kubectl get secret -n domino-platform $SECRET -ogo-template="{{ .data.release | base64decode | base64decode }}" | gzip -d - | sed 's/clusterIP: \\"\\"//g' | gzip | base64 -w0 | base64 -w0)/" | kubectl replace -f -
    kubectl get secret -n domino-platform $SECRET -oyaml | sed "s/release:.*/release: $(kubectl get secret -n domino-platform $SECRET -ogo-template="{{ .data.release | base64decode | base64decode }}" | gzip -d - | sed 's/rbac.authorization.k8s.io\/v1beta1/rbac.authorization.k8s.io\/v1/g' | gzip | base64 -w0 | base64 -w0)/" | kubectl replace -f -
    

fleetcommand-agent v22 (May 2020)

Image: quay.io/domino/fleetcommand-agent:v22

Installation bundles:

  • 4.2.0 images https://mirrors.domino.tech/s3/domino-artifacts/offline/opsless-docker-images-v22-4.2.0.tar
  • Latest charts http://mirrors.domino.tech/artifacts/appr/domino-appr-latest.tar.gz

Changes:

  • Adds support for Domino 4.2.

  • Adds support for Helm 3

    The helm object in the installer configuration has been restructured to accommodate Helm 3 support. There is now a helm.version property which can be set to 2 or 3. When using Helm 2, the configuration should be similar to the below example. The username and password will continue to be standard Quay.io credentials provided by Domino.

    helm:
      version: 2
      host: quay.io
      namespace: domino
      prefix: helm- # Prefix for the chart repository, defaults to `helm-`
      username: "<username>"
      password: "<password>"
      tiller_image: gcr.io/kubernetes-helm/tiller:v2.16.1 # Version is required and MUST be 2.16.1
      insecure: false
    

    When using Helm 3, configure the object as shown below. Helm 3 is a major release of the underlying tool that powers installation of Domino’s services. Helm 3 removes the Tiller service, which was the server-side component of Helm 2. This improves the security posture of Domino installation by reducing the scope and complexity of required RBAC permissions, and it enables namespace isolation of services. Additionally, Helm 3 adds support for storing charts in OCI registries.

    Currently, only gcr.io and mirrors.domino.tech are supported as chart repositories. If you are switching to Helm 3, you will need to contact Domino for gcr.io credentials. When using Helm 3, the helm configuration object should be similar to the below example

    helm:
      version: 3
      host: gcr.io
      namespace: domino-eng-service-artifacts
      insecure: false
      username: _json_key  # To support GCR authentication, this must be "_json_key"
      password: "<password>"
      tiller_image: null # Not required for Helm 3
      prefix: '' # Charts are stored without a prefix by default
    

    Migration of an existing Helm 2 installation to Helm 3 is done seamlessly within the installer. Once successful, Tiller will be removed from the cluster and all Helm 2 configuration is deleted.

Known Issues:

  • Elasticsearch is currently configured to only upgrade when the pods are deleted. To properly upgrade an existing deployment from Elasticsearch 6.5 to 6.8, after running the installer use the rolling upgrade process. This involves first deleting the elasticsearch-data pods, then the elasticsearch-master pods. See the example procedure below.

    kubectl delete pods --namespace domino-platform elasticsearch-data-0 elasticsearch-data-1 --force=true --grace-period=0
    # Wait for elasticsearch-data-0 & elasticsearch-data-1 to come back online
    kubectl delete pods --namespace domino-platform elasticsearch-master-0 elasticsearch-master-1 elasticsearch-master-2
    
  • An incompatibility between how nginx-ingress was initially installed and should be maintained going forward means that action is required for both Helm 2 and Helm 3 upgrades.

    For Helm 2 upgrades, add the following services object to your domino.yml to ensure compatibility:

    services:
      nginx_ingress:
        chart_values:
          controller:
            metrics:
              service:
                clusterIP: "-"
            service:
              clusterIP: "-"
    

    For Helm 3, there are two options. If nginx-ingress has not been configured to provide a cloud-native load balancer that is tied to the hosting DNS entry, then nginx-ingress can be safely uninstalled prior to the upgrade. If, however, the load balancer address must be maintained across the upgrade, then the initial upgrade after the Helm 3 migration will fail. Before retrying the upgrade, execute the following commands.

    export NAME=nginx-ingress
    export SECRET=$(kubectl get secret -l owner=helm,status=deployed,name=$NAME -n domino-platform | awk '{print $1}' | grep -v NAME)
    kubectl get secret -n domino-platform $SECRET -oyaml | sed "s/release:.*/release: $(kubectl get secret -n domino-platform $SECRET -ogo-template="{{ .data.release | base64decode | base64decode }}" | gzip -d - | sed 's/clusterIP: \\"\\"//g' | gzip | base64 -w0 | base64 -w0)/" | kubectl replace -f -
    kubectl get secret -n domino-platform $SECRET -oyaml | sed "s/release:.*/release: $(kubectl get secret -n domino-platform $SECRET -ogo-template="{{ .data.release | base64decode | base64decode }}" | gzip -d - | sed 's/rbac.authorization.k8s.io\/v1beta1/rbac.authorization.k8s.io\/v1/g' | gzip | base64 -w0 | base64 -w0)/" | kubectl replace -f -
    
  • Domino Apps do not currently support a live upgrade from version 4.1 to version 4.2. After the upgrade, all Apps will be stopped.

    To restart them, you can use the /v4/modelProducts/restartAll endpoint like in the below example, providing an API key for a system administrator.

    curl --include --header "X-Domino-Api-Key: <admin-api-key>" 'https://<domino-url>/v4/modelProducts/restartAll'
    

fleetcommand-agent v21 (May 2020)

Image: quay.io/domino/fleetcommand-agent:v21

Changes:

  • Adds support for Domino 4.1.10

Known issues:

  • The deployed version 8.0.1 of Keycloak has an incorrect default First Broker Login authentication flow.

    When setting up an SSO integration, you must create a new authentication flow like the one below. Note that the Automatically Link Account step is a custom flow, and the Create User if Unique and Automatically Set Existing User executions must be nested under it by adding them with the Actions link.

    ../_images/alternative_flow.png

fleetcommand-agent v20 (March 2020)

Image: quay.io/domino/fleetcommand-agent:v20

Changes:

  • Support for 4.1.9 has been updated to reflect a new set of artifacts.

Known issues:

  • The deployed version 8.0.1 of Keycloak has an incorrect default First Broker Login authentication flow.

    When setting up an SSO integration, you must create a new authentication flow like the one below. Note that the Automatically Link Account step is a custom flow, and the Create User if Unique and Automatically Set Existing User executions must be nested under it by adding them with the Actions link.

    ../_images/alternative_flow.png

fleetcommand-agent v19 (March 2020)

Image: quay.io/domino/fleetcommand-agent:v19

Changes:

  • Added catalogs for Domino up to 4.1.9
  • Added support for Docker NO_PROXY configuration. Domino containers will now respect the configuration and connect to the specified hosts without proxy.

Known issues:

  • The deployed version 8.0.1 of Keycloak has an incorrect default First Broker Login authentication flow.

    When setting up an SSO integration, you must create a new authentication flow like the one below. Note that the Automatically Link Account step is a custom flow, and the Create User if Unique and Automatically Set Existing User executions must be nested under it by adding them with the Actions link.

    ../_images/alternative_flow.png

fleetcommand-agent v18 (March 2020)

Image: quay.io/domino/fleetcommand-agent:v18

Changes:

The following new fields have been added to the fleetcommand-agent installer configuration.

  1. Storage class access modes

    The storage_class options have a new field called access_modes that allows configuration of the underlying storage class’ allowed access modes.

    storage_classes:
      block:
        [snip]
        access_modes:
        - ReadWriteOnce
    
  2. Git service storage class

    Previously, the deployed Domino Git service used storage backed by the shared storage class. Now, the dominodisk block storage class will be used by default. If using custom storage classes, set this to the name of the block storage class. For existing Domino installations, you must set this to dominoshared.

    For new installs:

    git:
      storage_class: dominodisk
    

    For existing installations and upgrades:

    git:
      storage_class: dominoshared
    services:
      git_server:
        chart_values:
          persistence:
            size: 5Ti
    

fleetcommand-agent v17 (March 2020)

Image: quay.io/domino/fleetcommand-agent:v17

Changes:

  • Added catalogs for Domino up to 4.1.8

fleetcommand-agent v16 (February 2020)

Image: quay.io/domino/fleetcommand-agent:v16

Changes:

  • Added catalogs for Domino up to 4.1.7
  • Calico CNI is now installed by default for EKS deployments
  • AWS Metadata API is blocked by default for Domino version >= 4.1.5
  • Added Private registry support in the Installer
  • New Install configuration attributes (see the reference documentation for more details): - sse_kms_key_id option for Blob storage - gcs option for Google Cloud Storage - Namespaces now support optional the labels to apply labels during installation - teleport for Domino managed installations only

fleetcommand-agent v15 (January 2020)

Image: quay.io/domino/fleetcommand-agent:v15

Changes:

  • Added catalog for Domino 4.1.4
  • Ensure fleetcommand-agent also deletes system namespace.
  • Updated version of Cluster Autoscaler to 1.13.9

fleetcommand-agent v14 (January 2020)

Image: quay.io/domino/fleetcommand-agent:v14

Changes: