fleetcommand-agent release notes¶
fleetcommand-agent v32 (December 2020)¶
Image: quay.io/domino/fleetcommand-agent:v32
Installation bundles:
- 4.3.3 images
https://mirrors.domino.tech/s3/domino-artifacts/offline/opsless-v32-docker-images-4.3.3.tar
Changes:
- Fixes a memory leak in the EFS CSI driver.
fleetcommand-agent v31 (December 2020)¶
Image: quay.io/domino/fleetcommand-agent:v31
Installation bundles:
- 4.3.3 images
https://mirrors.domino.tech/s3/domino-artifacts/offline/opsless-v31-docker-images-4.3.3.tar
Changes:
- Updates to latest build of 4.3.3.
fleetcommand-agent v30 (December 2020)¶
Image: quay.io/domino/fleetcommand-agent:v30
Installation bundles:
- 4.3.3 images
https://mirrors.domino.tech/s3/domino-artifacts/offline/opsless-v30-docker-images-4.3.3.tar
Changes:
Adds support for Domino 4.3.3
The agent now supports installing Istio 1.7 (set
istio.install
totrue
), and installing Domino in Istio-compatible mode (setistio.enabled
totrue
).istio: enabled: false install: false cni: true namespace: istio-system
The EFS storage provider for new installs has changed from
efs-provisioner
to the EFS CSI driver, in order to support encryption in transit to EFS. For existing installs, this does not require any changes unless encryption in transit is desired. If a migration to encrypted EFS is necessary, please contact Domino support.One limitation of the new driver, compared to the previous, is an inability to dynamically create directories according to provisioned volumes. Support for pre-provisioned directories in AWS is done through access points, which must be created before Domino can be installed.
To specify the access point at install time, ensure the filesystem_id is set in the format
{EFS ID}::{AP ID}
:storage_classes: shared: efs: filesystem_id: 'fs-285b532d::fsap-00cb72ba8ca35a121'
Two new fields were added in order simplify DaemonSet management during upgrades for particularly large clusters. DaemonSets do not have configuration options for upgrades and pods will be replaced one-by-one. For large compute node pools, this can take a significant amount of time.
helm: skip_daemonset_validation: false daemonset_timeout: 300
Setting
helm.skip_daemonset_validation
to true will bypass post-upgrade validation that all pods have been successfully recreated.helm.daemonset_timeout
is an integer representing the number of seconds to wait for all daemon pods in a DaemonSet to be recreated.4.3.3 introduces limited availability of the new containerized Domino image builder: Forge. Forge can be enabled with the
ImageBuilderV2
feature flag, although Domino services must be restarted to cause this flag to take effect. Running Domino image builds in a cluster that uses a non-Docker container runtime, such as cri-o or containerd, requires that the feature flag be enabled.To support the default rootless mode that Forge is configured to use, the worker nodes must support unprivileged mounts, user namespaces, and overlayfs (either natively or through FUSE). Currently, GKE and EKS do not support user namespace remapping and require the following extra configuration to properly use Forge.
services: forge: chart_values: config: fullPrivilege: true
fleetcommand-agent v29 (November 2020)¶
Image: quay.io/domino/fleetcommand-agent:v29
Installation bundles:
- 4.3.2 images
https://mirrors.domino.tech/s3/domino-artifacts/offline/opsless-v29-docker-images-4.3.2.tar
Changes:
- Updated Keycloak migration job version.
fleetcommand-agent v28 (November 2020)¶
Image: quay.io/domino/fleetcommand-agent:v28
Installation bundles:
- 4.3.2 images
https://mirrors.domino.tech/s3/domino-artifacts/offline/opsless-v28-docker-images-4.3.2.tar
Changes:
Adds support for Domino 4.3.2
Adds support for encrypted EFS access by using the EFS CSI driver.
A new
istio
field has been added to the domino.yml schema for testing and development of future releases. Domino 4.3.2 does not support Istio and therefore you must setenabled
in this new section tofalse
.istio: enabled: false install: false cni: true namespace: istio-system
New fields to specify static AWS access key and secret key credentials have been added. These are currently unused and can be left unset.
blob_storage: projects: s3: access_key_id: '' secret_access_key: ''
A new field for Teleport remote access integration has been added. This is currently unused and should be set to
false
.teleport: remote_access: false
fleetcommand-agent v27 (October 2020)¶
Image: quay.io/domino/fleetcommand-agent:v27
Installation bundles:
- 4.3.1 images
https://mirrors.domino.tech/s3/domino-artifacts/offline/opsless-v27-docker-images-4.3.1.tar
Changes:
- Fix a bug where dry-run installation could cause internal credentials to be improperly rotated.
fleetcommand-agent v26 (October 2020)¶
Image: quay.io/domino/fleetcommand-agent:v26
Installation bundles:
- 4.3.1 images
https://mirrors.domino.tech/s3/domino-artifacts/offline/opsless-v26-docker-images-4.3.1.tar
- Latest charts
http://mirrors.domino.tech/artifacts/appr/domino-appr-latest.tar.gz
Changes:
Adds support for Domino 4.3.1
Adds support for running Domino on OpenShift 4.4+.
A new field has been added to the installer configuration that controls whether or not the image caching service is deployed.
image_caching: enabled: true
A new field has been added to the installer configuration that specifies the Kubernetes distribution for resource compatibility. The available options are
cncf
(Cloud Native Computing Foundation) andopenshift
.kubernetes_distribution: cncf
fleetcommand-agent v25 (August 2020)¶
Image: quay.io/domino/fleetcommand-agent:v25
Installation bundles:
- 4.3.0 images
https://mirrors.domino.tech/s3/domino-artifacts/offline/opsless-v25-docker-images-4.3.0.tar
- Latest charts
http://mirrors.domino.tech/artifacts/appr/domino-appr-latest.tar.gz
Changes:
Adds support for Domino 4.3.0.
A new
cache_path
field has been added to thehelm
configuration section. Leaving this field blank will ensure charts are fetched from an upstream repository.helm: cache_path: ''
To facilitate deployment of Domino into clusters with other tenants, a new global node selector field has been added to the top-level configuration that allows an arbitrary label to be used for scheduling all Domino workloads. Its primary purpose is to limit workloads such as DaemonSets that would be scheduled on all available nodes in the cluster to only nodes with the provided label. Note that this can override default node pool selectors such as
dominodatalab.com/node-pool: "platform"
, but does not replace them.global_node_selectors: domino-owned: "true"
To facilitate deployment of Domino into clusters with other tenants, a configurable Ingress class has been added to allow differentiation from other ingress providers in a cluster. If multiple Ingress objects are created with the default class, it’s possible for other tenant’s paths to interfere with Domino and vice versa. Generally, this setting does not need to change, but can be set to any arbitrary string value (such as
domino
).ingress_controller: class_name: nginx
fleetcommand-agent v24 (July 2020)¶
Image: quay.io/domino/fleetcommand-agent:v24
Installation bundles:
- 4.2.4 images
https://mirrors.domino.tech/s3/domino-artifacts/offline/opsless-docker-images-v24-4.2.4.tar
- Latest charts
http://mirrors.domino.tech/artifacts/appr/domino-appr-latest.tar.gz
Changes:
- Adds support for Domino 4.2.3 and 4.2.4.
fleetcommand-agent v23 (May 2020)¶
Image: quay.io/domino/fleetcommand-agent:v23
Installation bundles:
- 4.2.2 images
https://mirrors.domino.tech/s3/domino-artifacts/offline/opsless-docker-images-v23-4.2.2.tar
- Latest charts
http://mirrors.domino.tech/artifacts/appr/domino-appr-latest.tar.gz
Changes:
Adds support for Domino 4.2.2.
The known issue with v22 around Domino Apps being stopped after upgrade has been resolved. Apps will now automatically restart after upgrade.
The known issue with Elasticsearch not upgrading until manually restarted has been resolved. Elasticsearch will automatically cycle through a rolling upgrade when the deployment is upgraded.
Fixed an issue that prevented the fleetcommand-agent
Adds support for autodiscovery of scaling resources by the cluster autoscaler.
Two new fields have been added under the
autoscaler.auto_discovery
key:autoscaler: auto_discovery: cluster_name: domino tags: [] # optional. if filled in, cluster_name is ignored.
By default, if no
autoscaler.groups
orautoscaler.auto_discovery.tags
are specified, thecluster_name
will be used to look for the following AWS tags:k8s.io/cluster-autoscaler/enabled
k8s.io/cluster-autoscaler/{{ cluster_name }}
The
tags
parameter can be used to explicitly specify which resource tags the autoscaler service should look for. Auto scaling groups with matching tags will have their scaling properties detected and the autoscaler will be configured to scale them.Note that the IAM role for the platform nodes where the autoscaler is running still needs an autoscaling access policy that will allow it to read and scale the groups.
When upgrading from an install that uses specific groups, ensure that
auto_discovery.cluster_name
is an empty value.
Known Issues:
If you’re upgrading from fleetcommand-agent v21 or older, be sure to read the v22 release notes and implement the Helm configuration changes.
An incompatibility between how
nginx-ingress
was initially installed and should be maintained going forward means that action is required for both Helm 2 and Helm 3 upgrades.For Helm 2 upgrades, add the following
services
object to yourdomino.yml
to ensure compatibility:services: nginx_ingress: chart_values: controller: metrics: service: clusterIP: "-" service: clusterIP: "-"
For Helm 3, there are two options. If
nginx-ingress
has not been configured to provide a cloud-native load balancer that is tied to the hosting DNS entry, thennginx-ingress
can be safely uninstalled prior to the upgrade. If, however, the load balancer address must be maintained across the upgrade, then the initial upgrade after the Helm 3 migration will fail. Before retrying the upgrade, execute the following commands.export NAME=nginx-ingress export SECRET=$(kubectl get secret -l owner=helm,status=deployed,name=$NAME -n domino-platform | awk '{print $1}' | grep -v NAME) kubectl get secret -n domino-platform $SECRET -oyaml | sed "s/release:.*/release: $(kubectl get secret -n domino-platform $SECRET -ogo-template="{{ .data.release | base64decode | base64decode }}" | gzip -d - | sed 's/clusterIP: \\"\\"//g' | gzip | base64 -w0 | base64 -w0)/" | kubectl replace -f - kubectl get secret -n domino-platform $SECRET -oyaml | sed "s/release:.*/release: $(kubectl get secret -n domino-platform $SECRET -ogo-template="{{ .data.release | base64decode | base64decode }}" | gzip -d - | sed 's/rbac.authorization.k8s.io\/v1beta1/rbac.authorization.k8s.io\/v1/g' | gzip | base64 -w0 | base64 -w0)/" | kubectl replace -f -
fleetcommand-agent v22 (May 2020)¶
Image: quay.io/domino/fleetcommand-agent:v22
Installation bundles:
- 4.2.0 images
https://mirrors.domino.tech/s3/domino-artifacts/offline/opsless-docker-images-v22-4.2.0.tar
- Latest charts
http://mirrors.domino.tech/artifacts/appr/domino-appr-latest.tar.gz
Changes:
Adds support for Domino 4.2.
Adds support for Helm 3
The
helm
object in the installer configuration has been restructured to accommodate Helm 3 support. There is now ahelm.version
property which can be set to2
or3
. When using Helm 2, the configuration should be similar to the below example. Theusername
andpassword
will continue to be standard Quay.io credentials provided by Domino.helm: version: 2 host: quay.io namespace: domino prefix: helm- # Prefix for the chart repository, defaults to `helm-` username: "<username>" password: "<password>" tiller_image: gcr.io/kubernetes-helm/tiller:v2.16.1 # Version is required and MUST be 2.16.1 insecure: false
When using Helm 3, configure the object as shown below. Helm 3 is a major release of the underlying tool that powers installation of Domino’s services. Helm 3 removes the Tiller service, which was the server-side component of Helm 2. This improves the security posture of Domino installation by reducing the scope and complexity of required RBAC permissions, and it enables namespace isolation of services. Additionally, Helm 3 adds support for storing charts in OCI registries.
Currently, only gcr.io and mirrors.domino.tech are supported as chart repositories. If you are switching to Helm 3, you will need to contact Domino for gcr.io credentials. When using Helm 3, the
helm
configuration object should be similar to the below examplehelm: version: 3 host: gcr.io namespace: domino-eng-service-artifacts insecure: false username: _json_key # To support GCR authentication, this must be "_json_key" password: "<password>" tiller_image: null # Not required for Helm 3 prefix: '' # Charts are stored without a prefix by default
Migration of an existing Helm 2 installation to Helm 3 is done seamlessly within the installer. Once successful, Tiller will be removed from the cluster and all Helm 2 configuration is deleted.
Known Issues:
Elasticsearch is currently configured to only upgrade when the pods are deleted. To properly upgrade an existing deployment from Elasticsearch 6.5 to 6.8, after running the installer use the rolling upgrade process. This involves first deleting the
elasticsearch-data
pods, then theelasticsearch-master
pods. See the example procedure below.kubectl delete pods --namespace domino-platform elasticsearch-data-0 elasticsearch-data-1 --force=true --grace-period=0 # Wait for elasticsearch-data-0 & elasticsearch-data-1 to come back online kubectl delete pods --namespace domino-platform elasticsearch-master-0 elasticsearch-master-1 elasticsearch-master-2
An incompatibility between how
nginx-ingress
was initially installed and should be maintained going forward means that action is required for both Helm 2 and Helm 3 upgrades.For Helm 2 upgrades, add the following
services
object to yourdomino.yml
to ensure compatibility:services: nginx_ingress: chart_values: controller: metrics: service: clusterIP: "-" service: clusterIP: "-"
For Helm 3, there are two options. If
nginx-ingress
has not been configured to provide a cloud-native load balancer that is tied to the hosting DNS entry, thennginx-ingress
can be safely uninstalled prior to the upgrade. If, however, the load balancer address must be maintained across the upgrade, then the initial upgrade after the Helm 3 migration will fail. Before retrying the upgrade, execute the following commands.export NAME=nginx-ingress export SECRET=$(kubectl get secret -l owner=helm,status=deployed,name=$NAME -n domino-platform | awk '{print $1}' | grep -v NAME) kubectl get secret -n domino-platform $SECRET -oyaml | sed "s/release:.*/release: $(kubectl get secret -n domino-platform $SECRET -ogo-template="{{ .data.release | base64decode | base64decode }}" | gzip -d - | sed 's/clusterIP: \\"\\"//g' | gzip | base64 -w0 | base64 -w0)/" | kubectl replace -f - kubectl get secret -n domino-platform $SECRET -oyaml | sed "s/release:.*/release: $(kubectl get secret -n domino-platform $SECRET -ogo-template="{{ .data.release | base64decode | base64decode }}" | gzip -d - | sed 's/rbac.authorization.k8s.io\/v1beta1/rbac.authorization.k8s.io\/v1/g' | gzip | base64 -w0 | base64 -w0)/" | kubectl replace -f -
Domino Apps do not currently support a live upgrade from version 4.1 to version 4.2. After the upgrade, all Apps will be stopped.
To restart them, you can use the
/v4/modelProducts/restartAll
endpoint like in the below example, providing an API key for a system administrator.curl -X POST --include --header "X-Domino-Api-Key: <admin-api-key>" 'https://<domino-url>/v4/modelProducts/restartAll'
fleetcommand-agent v21 (May 2020)¶
Image: quay.io/domino/fleetcommand-agent:v21
Changes:
- Adds support for Domino 4.1.10
Known issues:
The deployed version 8.0.1 of Keycloak has an incorrect default First Broker Login authentication flow.
When setting up an SSO integration, you must create a new authentication flow like the one below. Note that the
Automatically Link Account
step is a custom flow, and theCreate User if Unique
andAutomatically Set Existing User
executions must be nested under it by adding them with the Actions link.
fleetcommand-agent v20 (March 2020)¶
Image: quay.io/domino/fleetcommand-agent:v20
Changes:
- Support for 4.1.9 has been updated to reflect a new set of artifacts.
Known issues:
The deployed version 8.0.1 of Keycloak has an incorrect default First Broker Login authentication flow.
When setting up an SSO integration, you must create a new authentication flow like the one below. Note that the
Automatically Link Account
step is a custom flow, and theCreate User if Unique
andAutomatically Set Existing User
executions must be nested under it by adding them with the Actions link.
fleetcommand-agent v19 (March 2020)¶
Image: quay.io/domino/fleetcommand-agent:v19
Changes:
- Added catalogs for Domino up to 4.1.9
- Added support for Docker
NO_PROXY
configuration. Domino containers will now respect the configuration and connect to the specified hosts without proxy.
Known issues:
The deployed version 8.0.1 of Keycloak has an incorrect default First Broker Login authentication flow.
When setting up an SSO integration, you must create a new authentication flow like the one below. Note that the
Automatically Link Account
step is a custom flow, and theCreate User if Unique
andAutomatically Set Existing User
executions must be nested under it by adding them with the Actions link.
fleetcommand-agent v18 (March 2020)¶
Image: quay.io/domino/fleetcommand-agent:v18
Changes:
The following new fields have been added to the fleetcommand-agent installer configuration.
Storage class access modes
The
storage_class
options have a new field calledaccess_modes
that allows configuration of the underlying storage class’ allowed access modes.storage_classes: block: [snip] access_modes: - ReadWriteOnce
Git service storage class
Previously, the deployed Domino Git service used storage backed by the
shared
storage class. Now, thedominodisk
block storage class will be used by default. If using custom storage classes, set this to the name of the block storage class. For existing Domino installations, you must set this todominoshared
.For new installs:
git: storage_class: dominodisk
For existing installations and upgrades:
git: storage_class: dominoshared services: git_server: chart_values: persistence: size: 5Ti
fleetcommand-agent v17 (March 2020)¶
Image: quay.io/domino/fleetcommand-agent:v17
Changes:
- Added catalogs for Domino up to 4.1.8
fleetcommand-agent v16 (February 2020)¶
Image: quay.io/domino/fleetcommand-agent:v16
Changes:
- Added catalogs for Domino up to 4.1.7
- Calico CNI is now installed by default for EKS deployments
- AWS Metadata API is blocked by default for Domino version >= 4.1.5
- Added Private registry support in the Installer
- New Install configuration attributes (see the reference documentation for more details):
-
sse_kms_key_id
option for Blob storage -gcs
option for Google Cloud Storage - Namespaces now support optional thelabels
to apply labels during installation -teleport
for Domino managed installations only
fleetcommand-agent v15 (January 2020)¶
Image: quay.io/domino/fleetcommand-agent:v15
Changes:
- Added catalog for Domino 4.1.4
- Ensure fleetcommand-agent also deletes system namespace.
- Updated version of Cluster Autoscaler to 1.13.9
fleetcommand-agent v14 (January 2020)¶
Image: quay.io/domino/fleetcommand-agent:v14
Changes:
- Updated version of Cluster Autoscaler to 1.13.7
- Added catalog for Domino 4.1.3