The Admin Toolkit is an always-on tool deployed alongside Domino in the Kubernetes cluster that allows Domino administrators to run a system scan on the Domino deployment to do the following:
-
Check for unpatched known issues
-
Validate the health of core services
-
Get information to assist troubleshooting, run remediation actions, or provide context about issues affecting the deployment
-
Provide the toolkit report to Domino Support as part of an incident
The toolkit has a front-end UI provided at https://<your-domino-url>/toolkit/ where you can view scan reports and initiate new scans. There is also a CLI tool to initiate scans and help manage the toolkit deployment.
You can run scans as many times as necessary because the checks performed by the toolkit are only reading information and not making any modifications. You can run a scan during any of these stages:
-
After a new Domino install
-
At the start of an incident
-
In the midst of an incident
-
After incident resolution to verify health
-
Before and after a Domino upgrade
-
Before and after a Domino migration
You can find details about the specific tests that are included in the scan at https://docs.toolkit.re.domino.tech/.
Prerequisites
-
A deployment of Domino 4.x or newer. This version of the toolkit does not run on Domino 3.x or older.
-
Administrative access to the Kubernetes cluster that hosts Domino.
-
The Kubernetes control plane must be working so that the toolkit can interrogate the Kubernetes API server.
-
A bastion host, laptop, or other host running a Unix-like OS with
kubectl
access that has network connectivity to the target Kubernetes cluster. -
The latest Admin Toolkit CLI script, called
toolkit.sh
, which you can download to your bastion host or laptop using this command:curl -sSLO https://toolkit.re.domino.tech/toolkit.sh && chmod a+x ./toolkit.sh
-
Some features of the toolkit, such as automatic upgrading or automatic uploading of scan reports, require an internet connection from the Kubernetes cluster.
Install the toolkit
You can install the toolkit using the toolkit CLI script as follows:
./toolkit.sh install
If there is a legacy version of the toolkit installed and running on the cluster, the installation process removes that version before installing the latest release.
The install command also accepts the --no-ingress
option that installs the toolkit with the /toolkit
ingress route disabled. When installed with this option, it is not possible to use the web UI and only the CLI can be used to operate the toolkit.
-
On a machine with access to both the internal registry and quay.io/mirrors.domino.tech, run
./toolkit.sh push internal-registry-host/domino-admin-toolkit
-
From your bastion host, run:
./toolkit install --image internal-registry-host/domino-admin-toolkit
Upgrade the toolkit
The toolkit can be updated independently of the Domino version deployed. Domino regularly publishes new versions of the toolkit Docker image with new and updated checks.
In the toolkit UI, click Toolkit Upgrade Check to check for a new toolkit Docker image and deploy it. You can also force an immediate check for a new version by stopping and re-starting the toolkit like this:
./toolkit stop && ./toolkit start
The toolkit is designed to remain running at all times and typically uses minimal resources in the Kubernetes cluster.
./toolkit.sh stop
./toolkit.sh start
Bear in mind that, while stopped, the toolkit cannot run system scans and does not auto-update.
./toolkit status
When installing the toolkit with ./toolkit.sh install
, the installer generates a random password that does not change for the life of the toolkit deployment.
If you installed the toolkit on the command line using the CLI, this password (along with the ingress URL) is displayed at the end of the install process.
If you forget the password, you can retrieve the password using this command:
./toolkit.sh get-password
The default username is admin-toolkit
.
You can run a system scan using the Admin Toolkit UI or CLI.
Run a scan on an air-gapped deployment
If your Domino deployment blocks AWS access, run the following command to output the report only in the console:
./toolkit.sh pytest
This runs the report and displays it to the console with no colorizing of the output, making it easier to redirect to a file or copy and paste from the console into a text file for sending to Domino. Contact Domino support at support@dominodatalab.com to pre-arrange a secure way to send the text file to Domino.
Regardless of whether a scan runs from the UI or CLI, the toolkit can automatically and securely upload the report to an S3 bucket in Domino’s AWS account that is specific to your deployment.
This behavior is off by default but can be enabled by either selecting the option in the UI to enable uploads, or when using the command line by running with the --upload-report
option:
./toolkit.sh test --upload-report
There are many benefits to uploading reports to Domino:
-
Quicker resolution of support tickets
-
Validating deployment configuration
-
Providing more proactive rather than reactive support
Domino can also use the data collected from multiple customers to better understand how customer-specific configuration and infrastructure affects the overall performance and stability of the Domino platform so that we can deliver a more robust and performant product with each release.
The following describes the toolkit parameters.
./toolkit.sh help
toolkit.sh version: 0d5903c, 2022-11-11
Usage: toolkit.sh [command] [options]
Run with no parameters to create resources and run the tests
Commands:
install [options] Install and start the admin toolkit
uninstall Uninstall the admin toolkit, deleting all resources
start Starts the admin toolkit if it is stopped
stop Stops the admin toolkit if it is running
status Shows whether admin toolkit is installed and running
test [options] Run tests with given options and upload html report to S3
pytest [options] Run pytest directly, see https://docs.pytest.org/en/7.1.x/how-to/usage.html
exec [command] Execute a command in the toolkit container (by default /bin/bash), useful for debugging
logs Show toolkit container logs
get-password Retrieve the admin toolkit UI username and password
help Get this help message
`install` command options:
./toolkit.sh install --tag|-t tag Run specific docker image tag of the admin toolkit
./toolkit.sh install --daemonset Enable daemonset functionality
./toolkit.sh install --daemonset-port|-d port Set the host port the daemonset listens on. Default is port 5000.
./toolkit.sh install --no-ingress Disable ingress route to toolkit pod.
`test` command options:
./toolkit.sh test --help Show help including choice of parameters
./toolkit.sh test --exclude salt kubeadm Exclude tests for specific features
./toolkit.sh test --log-cli-level DEBUG Show extra debugging output
./toolkit.sh test --upload-report Send the completed report output to Domino's S3
By default each newly-generated Toolkit report is securely sent to Domino Support and Engineering.
This allows Domino engineers to gain access to the reports faster during an outage and get historical context to previous outages.
You can opt out of this service (or opt in again) with an applied Kubernetes ConfigMap
.
-
Copy and paste the following template into a file, changing the namespace to meet your deployment’s platform namespace:
kind: ConfigMap apiVersion: v1 metadata: name: domino-admin-toolkit-cm namespace: <platform-namespace> data: send_to_domino: "True"
-
Then apply:
kubectl apply -f <configmap>.yaml
The requirements are:
-
ConfigMap
type -
Metadata name:
domino-admin-toolkit-cm
-
Data must only use key
send_to_domino
and the value is a boolean type(True or False):
send_to_domino: "True"
By default, the functionality to have reports uploaded to Domino is enabled. Opt out by setting the value to False
in the ConfigMap
or clicking the associated button in the Toolkit UI. Setting the ConfigMap
value to True
re-enables automatic sending of each report to Domino.
These reports only collect deployment and service verification data and do not contain any sensitive data.
The Admin Toolkit documentation provides comprehensive information about the checks, their meanings, and any details on resolving issues. Using the toolkit and its documentation you can quickly identify and resolve issues, ensuring optimal performance of the Domino platform. You can access the documentation in two ways:
-
Publicly-accessible online documentation: https://docs.toolkit.re.domino.tech/ (always updated to the latest version of the toolkit)
-
Documentation included within the Domino deployment: https://<your-domino-url>/toolkit/docs/ (specific to the deployed version of the toolkit)
-
helm
andyq
dependencies are removed. -
The web-based UI is now password protected.
-
The web-based UI can be disabled using
--no-ingress
parameter.
-
Prerequisites now include
helm
andyq
. -
The web-based UI is now interactive.
-
The
--load-balancer
option is deprecated. Instead, use the new ingress route,/toolkit
, on the same domain as the deployment. -
A new
--runtime-timeout
option sets the maximum runtime of the toolkit in seconds.