The Admin Toolkit is an always-on tool deployed alongside Domino in the Kubernetes cluster that allows Domino administrators to run a system scan on the Domino deployment to do the following:
Check for unpatched known issues
Validate the health of core services
Get information to assist troubleshooting, run remediation actions, or provide context about issues affecting the deployment
Provide the toolkit report to Domino Support as part of an incident
The toolkit has a front-end UI provided at https://<your-domino-url>/toolkit/ where you can view scan reports and initiate new scans. There is also a CLI tool to initiate scans and help manage the toolkit deployment.
You can run scans as many times as necessary because the checks performed by the toolkit are only reading information and not making any modifications. You can run a scan during any of these stages:
After a new Domino install
At the start of an incident
In the midst of an incident
After incident resolution to verify health
Before and after a Domino upgrade
Before and after a Domino migration
You can find details about the specific tests that are included in the scan at https://docs.toolkit.re.domino.tech/.
A deployment of Domino 4.x or newer. This version of the toolkit does not run on Domino 3.x or older.
Administrative access to the Kubernetes cluster that hosts Domino.
The Kubernetes control plane must be working so that the toolkit can interrogate the Kubernetes API server.
A bastion host, laptop, or other host running a Unix-like OS with
kubectlaccess that has network connectivity to the target Kubernetes cluster.
The latest Admin Toolkit CLI script, called
toolkit.sh, which you can download to your bastion host or laptop using this command:
curl -sSLO https://toolkit.re.domino.tech/toolkit.sh && chmod a+x ./toolkit.sh
Some features of the toolkit, such as automatic upgrading or automatic uploading of scan reports, require an internet connection from the Kubernetes cluster.
Install the toolkit
The toolkit is installed automatically by the Domino installer.
You can also install the toolkit using the toolkit CLI script as follows:
If there is a legacy version of the toolkit installed and running on the cluster, the installation process removes that version before installing the latest release.
The install command also accepts the
--no-ingress option that installs the toolkit with the
/toolkit ingress route disabled. When installed with this option, it is not possible to use the web UI and only the CLI can be used to operate the toolkit.
On a machine with access to both the internal registry and quay.io/mirrors.domino.tech, run
./toolkit.sh push internal-registry-host/domino-admin-toolkit
From your bastion host, run:
./toolkit install --image internal-registry-host/domino-admin-toolkit
Upgrade the toolkit
The toolkit can be updated independently of the Domino version deployed. Domino regularly publishes new versions of the toolkit Docker image with new and updated checks.
In the toolkit UI, click Toolkit Upgrade Check to check for a new toolkit Docker image and deploy it. You can also force an immediate check for a new version by stopping and re-starting the toolkit like this:
./toolkit stop && ./toolkit start
The toolkit is designed to remain running at all times and typically uses minimal resources in the Kubernetes cluster.
Bear in mind that, while stopped, the toolkit cannot run system scans and does not auto-update.
When installing the toolkit with
./toolkit.sh install, the installer generates a random password that does not change for the life of the toolkit deployment.
If you installed the toolkit on the command line using the CLI, this password (along with the ingress URL) is displayed at the end of the install process.
If you forget the password, or if toolkit was installed by the Domino installer, you can retrieve the password using this command:
The default username is
You can run a system scan using the Admin Toolkit UI or CLI.
Run a scan on an air-gapped deployment
If your Domino deployment blocks AWS access, run the following command to output the report only in the console:
This runs the report and displays it to the console with no colorizing of the output, making it easier to redirect to a file or copy and paste from the console into a text file for sending to Domino. Contact Domino support at email@example.com to pre-arrange a secure way to send the text file to Domino.
Regardless of whether a scan runs from the UI or CLI, the toolkit can automatically and securely upload the report to an S3 bucket in Domino’s AWS account that is specific to your deployment.
This behavior is off by default but can be enabled by either selecting the option in the UI to enable uploads, or when using the command line by running with the
./toolkit.sh test --upload-report
There are many benefits to uploading reports to Domino:
Quicker resolution of support tickets
Validating deployment configuration
Providing more proactive rather than reactive support
Domino can also use the data collected from multiple customers to better understand how customer-specific configuration and infrastructure affects the overall performance and stability of the Domino platform so that we can deliver a more robust and performant product with each release.
The following describes the toolkit parameters.
./toolkit.sh help toolkit.sh version: 0d5903c, 2022-11-11 Usage: toolkit.sh [command] [options] Run with no parameters to create resources and run the tests Commands: install [options] Install and start the admin toolkit uninstall Uninstall the admin toolkit, deleting all resources start Starts the admin toolkit if it is stopped stop Stops the admin toolkit if it is running status Shows whether admin toolkit is installed and running test [options] Run tests with given options and upload html report to S3 pytest [options] Run pytest directly, see https://docs.pytest.org/en/7.1.x/how-to/usage.html exec [command] Execute a command in the toolkit container (by default /bin/bash), useful for debugging logs Show toolkit container logs get-password Retrieve the admin toolkit UI username and password help Get this help message `install` command options: ./toolkit.sh install --tag|-t tag Run specific docker image tag of the admin toolkit ./toolkit.sh install --daemonset Enable daemonset functionality ./toolkit.sh install --daemonset-port|-d port Set the host port the daemonset listens on. Default is port 5000. ./toolkit.sh install --no-ingress Disable ingress route to toolkit pod. `test` command options: ./toolkit.sh test --help Show help including choice of parameters ./toolkit.sh test --exclude salt kubeadm Exclude tests for specific features ./toolkit.sh test --log-cli-level DEBUG Show extra debugging output ./toolkit.sh test --upload-report Send the completed report output to Domino's S3
By default each newly-generated Toolkit report is securely sent to Domino Support and Engineering.
This allows Domino engineers to gain access to the reports faster during an outage and get historical context to previous outages.
You can opt out of this service (or opt in again) with an applied Kubernetes
Copy and paste the following template into a file, changing the namespace to meet your deployment’s platform namespace:
kind: ConfigMap apiVersion: v1 metadata: name: domino-admin-toolkit-cm namespace: <platform-namespace> data: send_to_domino: "True"
kubectl apply -f <configmap>.yaml
The requirements are:
Data must only use key
send_to_dominoand the value is a boolean type(True or False):
By default, the functionality to have reports uploaded to Domino is enabled. Opt out by setting the value to
False in the
ConfigMap or clicking the associated button in the Toolkit UI. Setting the
ConfigMap value to
True re-enables automatic sending of each report to Domino.
These reports only collect deployment and service verification data and do not contain any sensitive data.
The Admin Toolkit documentation provides comprehensive information about the checks, their meanings, and any details on resolving issues. Using the toolkit and its documentation you can quickly identify and resolve issues, ensuring optimal performance of the Domino platform. You can access the documentation in two ways:
Publicly-accessible online documentation: https://docs.toolkit.re.domino.tech/ (always updated to the latest version of the toolkit)
Documentation included within the Domino deployment: https://<your-domino-url>/toolkit/docs/ (specific to the deployed version of the toolkit)
yqdependencies are removed.
The web-based UI is now password protected.
The web-based UI can be disabled using
Prerequisites now include
The web-based UI is now interactive.
--load-balanceroption is deprecated. Instead, use the new ingress route,
/toolkit, on the same domain as the deployment.
--runtime-timeoutoption sets the maximum runtime of the toolkit in seconds.