The Admin Toolkit is an always-on tool deployed alongside Domino in the Kubernetes cluster that allows Domino administrators to run a system scan on the Domino deployment to do the following:

Check for unpatched known issues
Validate the health of core services
Get information to assist troubleshooting, run remediation actions, or provide context about issues affecting the deployment
Provide the toolkit report to Domino Support as part of an incident

The toolkit has a front-end UI provided at https://<your-domino-url>/toolkit/ where you can view scan reports and initiate new scans. There is also a CLI tool to initiate scans and help manage the toolkit deployment.

You can run scans as many times as necessary because the checks performed by the toolkit are only reading information and not making any modifications. You can run a scan during any of these stages:

After a new Domino install
At the start of an incident
In the midst of an incident
After incident resolution to verify health
Before and after a Domino upgrade
Before and after a Domino migration

You can find details about the specific tests that are included in the scan at https://docs.toolkit.re.domino.tech/.

Installing, uninstalling, and upgrading the toolkit

Prerequisites

A deployment of Domino 4.x or newer. This version of the toolkit does not run on Domino 3.x or older.
Administrative access to the Kubernetes cluster that hosts Domino.
The Kubernetes control plane must be working so that the toolkit can interrogate the Kubernetes API server.
A bastion host, laptop, or other host running a Unix-like OS with kubectl access that has network connectivity to the target Kubernetes cluster.
The latest Admin Toolkit CLI script, called toolkit.sh, which you can download to your bastion host or laptop using this command:
```
curl -sSLO https://toolkit.re.domino.tech/toolkit.sh && chmod a+x ./toolkit.sh
```
Some features of the toolkit, such as automatic upgrading or automatic uploading of scan reports, require an internet connection from the Kubernetes cluster.

Install the toolkit

You can install the toolkit using the toolkit CLI script as follows:

./toolkit.sh install

If there is a legacy version of the toolkit installed and running on the cluster, the installation process removes that version before installing the latest release.

The install command also accepts the --no-ingress option that installs the toolkit with the /toolkit ingress route disabled. When installed with this option, it is not possible to use the web UI and only the CLI can be used to operate the toolkit.

Note

On a machine with access to both the internal registry and quay.io/mirrors.domino.tech, run
```
./toolkit.sh push internal-registry-host/domino-admin-toolkit
```

From your bastion host, run:

./toolkit install --image internal-registry-host/domino-admin-toolkit

Uninstall the toolkit

To uninstall the toolkit, run:

./toolkit.sh uninstall

This completely removes all Kubernetes resources associated with the toolkit. Any reports still on the toolkit’s pod storage are deleted.

Upgrade the toolkit

The toolkit can be updated independently of the Domino version deployed. Domino regularly publishes new versions of the toolkit Docker image with new and updated checks.

In the toolkit UI, click Toolkit Upgrade Check to check for a new toolkit Docker image and deploy it. You can also force an immediate check for a new version by stopping and re-starting the toolkit like this:

./toolkit stop && ./toolkit start

Note

Starting, stopping and checking the toolkit status

The toolkit is designed to remain running at all times and typically uses minimal resources in the Kubernetes cluster.

Stop the toolkit:

./toolkit.sh stop

Start the toolkit:

./toolkit.sh start

Bear in mind that, while stopped, the toolkit cannot run system scans and does not auto-update.

Get the current install and running status of the toolkit:

./toolkit status

Retrieve the UI password

When installing the toolkit with ./toolkit.sh install, the installer generates a random password that does not change for the life of the toolkit deployment. If you installed the toolkit on the command line using the CLI, this password (along with the ingress URL) is displayed at the end of the install process.

If you forget the password, you can retrieve the password using this command:

./toolkit.sh get-password

The default username is admin-toolkit.

Run a system scan

You can run a system scan using the Admin Toolkit UI or CLI.

Run a scan using the UI

Navigate to /https://<your-domino-url>/toolkit/ and log in.
Click the Run button at the bottom of the page:

When the scan has finished, the UI displays a link to the report.

Run a scan using the CLI

From your bastion host or laptop where you have toolkit.sh installed, run this command:

./toolkit.sh test

This starts a scan, displaying the progress to the console.

Run a scan on an air-gapped deployment

If your Domino deployment blocks AWS access, run the following command to output the report only in the console:

./toolkit.sh pytest

This runs the report and displays it to the console with no colorizing of the output, making it easier to redirect to a file or copy and paste from the console into a text file for sending to Domino. Contact Domino support at support@dominodatalab.com to pre-arrange a secure way to send the text file to Domino.

View the reports

Use the Admin Toolkit’s web UI to review the reports in HTML format. The UI displays a list of all the reports that have been generated by the toolkit. Click the link of the report you wish to view.

web report

Upload scan reports to Domino

Regardless of whether a scan runs from the UI or CLI, the toolkit can automatically and securely upload the report to an S3 bucket in Domino’s AWS account that is specific to your deployment.

This behavior is off by default but can be enabled by either selecting the option in the UI to enable uploads, or when using the command line by running with the --upload-report option:

./toolkit.sh test --upload-report

There are many benefits to uploading reports to Domino:

Quicker resolution of support tickets
Validating deployment configuration
Providing more proactive rather than reactive support

Domino can also use the data collected from multiple customers to better understand how customer-specific configuration and infrastructure affects the overall performance and stability of the Domino platform so that we can deliver a more robust and performant product with each release.

Important

toolkit.sh usage

The following describes the toolkit parameters.

./toolkit.sh help

toolkit.sh version: 0d5903c, 2022-11-11

Usage: toolkit.sh [command] [options]

Run with no parameters to create resources and run the tests

Commands:
  install [options]  Install and start the admin toolkit
  uninstall          Uninstall the admin toolkit, deleting all resources
  start              Starts the admin toolkit if it is stopped
  stop               Stops the admin toolkit if it is running
  status             Shows whether admin toolkit is installed and running
  test    [options]  Run tests with given options and upload html report to S3
  pytest  [options]  Run pytest directly, see https://docs.pytest.org/en/7.1.x/how-to/usage.html
  exec    [command]  Execute a command in the toolkit container (by default /bin/bash), useful for debugging
  logs               Show toolkit container logs
  get-password       Retrieve the admin toolkit UI username and password
  help               Get this help message

`install` command options:
  ./toolkit.sh install --tag|-t tag              Run specific docker image tag of the admin toolkit
  ./toolkit.sh install --daemonset               Enable daemonset functionality
  ./toolkit.sh install --daemonset-port|-d port  Set the host port the daemonset listens on. Default is port 5000.
  ./toolkit.sh install --no-ingress              Disable ingress route to toolkit pod.

`test` command options:
  ./toolkit.sh test --help                  Show help including choice of parameters
  ./toolkit.sh test --exclude salt kubeadm  Exclude tests for specific features
  ./toolkit.sh test --log-cli-level DEBUG   Show extra debugging output
  ./toolkit.sh test --upload-report         Send the completed report output to Domino's S3

Enabling/disabling reports to be sent to Domino

By default each newly-generated Toolkit report is securely sent to Domino Support and Engineering. This allows Domino engineers to gain access to the reports faster during an outage and get historical context to previous outages. You can opt out of this service (or opt in again) with an applied Kubernetes ConfigMap.

Copy and paste the following template into a file, changing the namespace to meet your deployment’s platform namespace:

kind: ConfigMap
apiVersion: v1
metadata:
  name: domino-admin-toolkit-cm
  namespace: <platform-namespace>
data:
  send_to_domino: "True"

Then apply:
```
kubectl apply -f <configmap>.yaml
```

The requirements are:

ConfigMap type
Metadata name: domino-admin-toolkit-cm
Data must only use key send_to_domino and the value is a boolean type(True or False):

send_to_domino: "True"

By default, the functionality to have reports uploaded to Domino is enabled. Opt out by setting the value to False in the ConfigMap or clicking the associated button in the Toolkit UI. Setting the ConfigMap value to True re-enables automatic sending of each report to Domino. These reports only collect deployment and service verification data and do not contain any sensitive data.

Understand and resolve failures

The Admin Toolkit documentation provides comprehensive information about the checks, their meanings, and any details on resolving issues. Using the toolkit and its documentation you can quickly identify and resolve issues, ensuring optimal performance of the Domino platform. You can access the documentation in two ways:

Publicly-accessible online documentation: https://docs.toolkit.re.domino.tech/ (always updated to the latest version of the toolkit)
Documentation included within the Domino deployment: https://<your-domino-url>/toolkit/docs/ (specific to the deployed version of the toolkit)