Domino application logging¶
There are two types of logs produced by the operation of Domino.
- Domino execution logs
- Domino application logs
These are the logs output by user code running in Domino as a Job, Workspace, App, or Model API. These are available in the Domino web application on the Jobs Dashboard, Workspaces Dashboard, App Dashboard, and Model API instance logs. This data is a key part of the Domino reproducibility model, and is kept indefinitely in the Domino blob store.
The system these logs are written to is defined in the installation configuration file at
All Domino services output their logs using the
standard Kubernetes logging architecture.
Relevant logs are printed to
stderr as indicated, and are captured by Kubernetes.
The most effective way to aggregate logs is to attach a Kubernetes log aggregation utility to monitor the following Kubernetes namespaces used by Domino:
This namespace hosts the core application components of the Domino application, including API servers, databases, and web interfaces. The name of this namespace is defined in the installer configuration file at
The following components running in this namespace produce the most important logs:
The nucleus-frontend pods host the frontend API server that routes all requests to the Domino application. Its logs will contain details on HTTP requests to Domino from the application or another API client. If you see errors in Domino with HTTP error codes like 500, 504, or 401, you can find corresponding logs here.
The nucleus-dispatcher pod hosts the Domino scheduling and brokering service that sends user execution pods to Kubernetes for deployment. Errors in communication between Domino and Kubernetes will result in corresponding logs from this service.
The keycloak pods hosts the Domino authentication service. The logs for this service will contain a record of authentication events, including additional details on any errors.
This pod hosts the open-source Kubernetes cluster autoscaler, which controls and manages autoscaling resources. The logs for this service will contain records of scaling events, both scaling up new nodes in response to demand and scaling down idle resources, including additional details on any errors.
Compute grid namespace
This namespace hosts user executions plus Domino environment builds. The name of this namespace is defined in the installer configuration file at
Logs that appear in this namespace will correspond to ephemeral pods hosting using work. Each pod will contain a user-defined environment container, whose logs are described above as Execution logs. There are additional supporting containers in those pods, and their logs may contain additional information on any errors or behavior seen with specific Domino executions.
Users are advised to aggregate and keep at least 30 days of logs to facilitate debugging. These logs can be harvested with a variety of Kubernetes log aggregation utilities, including: