Domino application logging

There are two types of logs produced by the operation of Domino.

  1. Domino execution logs
  2. Domino application logs



Execution logs

These are the logs output by user code running in Domino as a Job, Workspace, App, or Model API. These are available in the Domino web application on the Jobs Dashboard, Workspaces Dashboard, App Dashboard, and Model API instance logs. This data is a key part of the Domino reproducibility model, and is kept indefinitely in the Domino blob store.

The system these logs are written to is defined in the installation configuration file at blob_storage.logs.




Application logs

All Domino services output their logs using the standard Kubernetes logging architecture. Relevant logs are printed to stdout or stderr as indicated, and are captured by Kubernetes.

The most effective way to aggregate logs is to attach a Kubernetes log aggregation utility to monitor the following Kubernetes namespaces used by Domino:

  • Platform namespace

    This namespace hosts the core application components of the Domino application, including API servers, databases, and web interfaces. The name of this namespace is defined in the installer configuration file at namespaces.platform.name.

    The following components running in this namespace produce the most important logs:

    Component Logs
    nucleus-frontend The nucleus-frontend pods host the frontend API server that routes all requests to the Domino application. Its logs will contain details on HTTP requests to Domino from the application or another API client. If you see errors in Domino with HTTP error codes like 500, 504, or 401, you can find corresponding logs here.
    nucleus-dispatcher The nucleus-dispatcher pod hosts the Domino scheduling and brokering service that sends user execution pods to Kubernetes for deployment. Errors in communication between Domino and Kubernetes will result in corresponding logs from this service.
    keycloak The keycloak pods hosts the Domino authentication service. The logs for this service will contain a record of authentication events, including additional details on any errors.
    cluster-autoscaler This pod hosts the open-source Kubernetes cluster autoscaler, which controls and manages autoscaling resources. The logs for this service will contain records of scaling events, both scaling up new nodes in response to demand and scaling down idle resources, including additional details on any errors.
  • Compute grid namespace

    This namespace hosts user executions plus Domino environment builds. The name of this namespace is defined in the installer configuration file at namespaces.compute.name.

    Logs that appear in this namespace will correspond to ephemeral pods hosting using work. Each pod will contain a user-defined environment container, whose logs are described above as Execution logs. There are additional supporting containers in those pods, and their logs may contain additional information on any errors or behavior seen with specific Domino executions.

Users are advised to aggregate and keep at least 30 days of logs to facilitate debugging. These logs can be harvested with a variety of Kubernetes log aggregation utilities, including: