Installation process

The Domino platform runs on Kubernetes. To simplify deployment and configuration of Domino services, Domino provides an install automation tool called the fleetcommand-agent that uses Helm to deploy Domino into your compatible cluster. The fleetcommand-agent is a Python application delivered in a Docker container, and can be run locally or as a job inside the target cluster.

../_images/install-diagram.png



Requirements

The install automation tools are delivered as a Docker image, and need to run on an installation workstation that meets the following requirements:

Additionally, you will need credentials for an installation service account that can access the Domino upstream image repositories in quay.io. Throughout these instructions, these credentials will be referred to as $QUAY_USERNAME and $QUAY_PASSWORD. Contact your Domino account team if you need new credentials.

The fleetcommand-agent needs access to two types of assets to install Domino:

  1. Docker images for Domino components
  2. Helm charts

The hosting cluster will need access to the following domains via Internet to retrieve component and dependency images for online installation:

  • quay.io
  • domino.tech
  • k8s.gcr.io
  • docker.elastic.co
  • docker.io
  • gcr.io

Alternatively, you can configure the fleetcommand-agent to point to a private docker registry and application registry for offline installation.




Pulling the fleetcommand-agent image

  1. Log in to quay.io with the credentials described in the requirements section above.

    docker login quay.io
    
  2. Find the image URI for the version of the fleetcommand-agent you want to use from the release notes.

  3. Pull the image to your local machine.

    docker pull quay.io/domino/fleetcommand-agent:v47
    



Running fleetcommand-agent commands

The default entrypoint for the fleetcommand-agent is:

"Entrypoint": [
    "python",
    "-m",
    "fleetcommand_agent"
]

This launches the Python application inside the container at /app/fleetcommand_agent. This allows you to easily run agent commands via docker run like this:

docker run --rm quay.io/domino/fleetcommand-agent:v47 $COMMAND $ARGUMENTS

The fleetcommand-agent supports the following commands:


init

Generates a template configuration file.

Arguments:

  • --file -f

    File system path to write the template to. This should be a host volume mounted to the container to persist the output.

  • --full -F

    Includes optional and advanced portions of the template. Should only be used when advanced options are needed, as configurations with these fields are more complex to maintain.

  • --version

    Domino version to generate a configuration template for.

  • --image-registry

    Provide a registry URI to prepend to Domino images to set up the template for installation from a private Docker registry. Should be used in conjunction with --full.

Example:

docker run --rm -v $(pwd):/install quay.io/domino/fleetcommand-agent:v47 init --file /install/domino.yml

run

Installs Domino into a cluster specified by a Kubernetes configuration from the KUBECONFIG environment variable. A valid configuration file must be passed in to this command.

Arguments:

  • --file -f

    File system path to the complete and valid configuration file.

  • --kubeconfig

    Path to Kubernetes configuration file containing cluster and authentication information to use.

  • --dry

    Use this mode to not make any permanent changes to the target cluster. A dry run checks service account permissions and generates detailed logs about the charts to be deployed with the given configuration. The output is written to `/app/logs and /app/.appr_chart_cache inside the container.

    Note that this option requires that the namespaces you want to use already exist

Example:

docker run --rm -v $(pwd):/install quay.io/domino/fleetcommand-agent:v47 run --file /install/domino.yml

Example dry run:

docker run --rm -v $(pwd):/install -v $(pwd)/logs:/app/logs -v $(pwd)/cache:/app/.appr_chart_cache quay.io/domino/fleetcommand-agent:v47 run --dry --file /install/domino.yml

destroy

Removes all resources from the target cluster for a given configuration file.

Arguments:

  • --file -f

    File system path to the complete and valid configuration file.

  • --kubeconfig

    Path to Kubernetes configuration file containing cluster and authentication information to use.

  • --dry

    Use this mode to not make any permanent changes to the target cluster. A dry run checks service account permissions and generates detailed logs about the charts to be deployed with the given configuration.

Example:

docker run --rm -v $(pwd):/install quay.io/domino/fleetcommand-agent:v47 destroy --file /install/domino.yml



Install process

  1. Connect to a workstation that meets the install automation requirements listed above. It is not necessary to use an account with root privileges to run the installer.

  2. Log in to quay.io with the credentials described in the requirements section above.

    docker login quay.io
    
  3. Retrieve the Domino installer image from quay.io.

    docker pull quay.io/domino/fleetcommand-agent:v47
    
  4. Initialize the installer application to generate a template configuration file named domino.yml.

    docker run --rm -it \
      -v $(pwd):/install \
      quay.io/domino/fleetcommand-agent:v47 \
      init --file /install/domino.yml
    
  5. Edit the configuration file with all necessary details about the target cluster, storage systems, and hosting domain. Read the configuration reference for more information about available keys, and consult the configuration examples for guidance on getting started.

    Note that you should change the value of name from domino-deployment to something that identifies the purpose of your installation and contains the name of your organization.

  6. Run this install script from the directory with the finalized configuration file to install Domino into the cluster. Note that you must fill in your $QUAY_USERNAME and $QUAY_PASSWORD where indicated, and also note that this script assumes your installer configuration file is in the same directory, and is named exactly domino.yml.

    #!/bin/bash
    
    set -ex
    
    kubectl delete po --ignore-not-found=true fleetcommand-agent-install
    
    kubectl create secret \
      docker-registry \
      -o yaml --dry-run \
      --docker-server=quay.io \
      --docker-username=$QUAY_USERNAME \
      --docker-password=$QUAY_PASSWORD \
      --docker-email=. domino-quay-repos | kubectl apply -f -
    
    kubectl create configmap \
      fleetcommand-agent-config \
      -o yaml --dry-run \
      --from-file=domino.yml | kubectl apply -f -
    
    cat <<EOF | kubectl apply -f -
    apiVersion: v1
    kind: ServiceAccount
    metadata:
      name: admin
    ---
    apiVersion: rbac.authorization.k8s.io/v1
    kind: ClusterRoleBinding
    metadata:
      name: admin-default
    roleRef:
      apiGroup: rbac.authorization.k8s.io
      kind: ClusterRole
      name: cluster-admin
    subjects:
    - kind: ServiceAccount
      name: admin
      namespace: default
    ---
    apiVersion: v1
    kind: Pod
    metadata:
      name: fleetcommand-agent-install
    spec:
      serviceAccountName: admin
      imagePullSecrets:
        - name: domino-quay-repos
      restartPolicy: Never
      containers:
      - name: fleetcommand-agent
        image: quay.io/domino/fleetcommand-agent:v47
        args: ["run", "-f", "/app/install/domino.yml", "-v"]
        imagePullPolicy: Always
        volumeMounts:
        - name: install-config
          mountPath: /app/install/
      volumes:
      - name: install-config
        configMap:
          name: fleetcommand-agent-config
    EOF
    
    set +e
    while true; do
      sleep 5
      if kubectl logs -f fleetcommand-agent-install; then
        break
      fi
    done
    
  7. The installation process can take up to 30 minutes to fully complete. The installer will output verbose logs and surface any errors it encounters, but it can also be useful to follow along in another terminal tab by running:

    kubectl get pods --all-namespaces
    

    This will show the status of all pods being created by the installation process. If you see any pods enter a crash loop or hang in a non-ready state, you can get logs from that pod by running:

    kubectl logs $POD_NAME --namespace $NAMESPACE_NAME
    

    If the installation completes successfully, you should see a message that says:

    2019-11-26 21:20:20,214 - INFO - fleetcommand_agent.Application - Deployment complete.
    Domino is accessible at $YOUR_FQDN
    

    However, the application will only be accessible via HTTPS at that FQDN if you have configured DNS for the name to point to an ingress load balancer with the appropriate SSL certificate that forwards traffic to your platform nodes.




Upgrading

Upgrading a Domino deployment is a simple process of running the installer again with the same configuration, but with the version field set the value of the desired upgrade version. See the installer configuration reference and the installer release notes for information on the Domino versions your installer can support.

If you need to upgrade to a newer installer version to upgrade to your desired Domino version, use the process below.

  1. Retrieve the new Domino installer image from quay.io by filling in the desired <version> value in the command below

    docker pull quay.io/domino/fleetcommand-agent:<version>
    
  2. Move your existing domino.yml configuration file to another directory, or rename it.

  3. Generate a new domino.yml configuration template by running the initialization command through the new version of the installer. This will ensure you have a configuration schema conformant to the new version.

    docker run --rm -it \
      -v $(pwd):/install \
      quay.io/domino/fleetcommand-agent:<version> \
      init --file /install/domino.yml
    
  4. Copy the values from your old configuration into the new file.

  5. When complete, run the install script from the install process, being sure to change the spec.containers.image value to quay.io/domino/fleetcommand-agent:<version> with the appropriate version.




Testing and Troubleshooting

The following tests can be run to verify that your Domino installation was successful:

  • Verify that you can login to the Domino application and that all of the navigation panel options are operational

    Failure of this test means you will need to check that Keycloak was setup properly

  • Create a new project and verify that you can launch a Jupyter/JupyterLab workspace

    Failure of this test mean you will need to check that default environment images have been loaded in the cluster

  • Verify that the Control Center UI is operational

    Failure of this test means you should check that all of the pods in the domino-deployment namespace are running correctly

  • Verify that the Dispatcher UI is operational

    Failure of this test means you should check that all of the pods in the domino-deployment namespace are running correctly

  • Verify that you can publish an app with flask or shiny apps

    Failure of this test means you should that the environment images have flask and shiny installed

  • Verify that email notifications are operational by completing a run and verifying that an email confirmation has been sent to you

    Failure of this test means you should verify there are valid email addresses for users in the admin section

  • Verify that you could launch a new model (confirm via the API tester and curl it)

    Failure of this test means you should check that Domino can create new objects in the cluster

  • Verify that datasets work by starting a workspace and adding a file to the domino/datasets/<username>/<project_name>/scratch and seeing that it shows up in Domino after closing the workspace.

    Failure of this test means you should check your data and storage configurations in the admin section