Before you can start using on-demand Spark clusters on Domino you need to ensure that this functionality is enabled and properly configured on your deployment.
Domino Administrators must enable on-demand Spark functionality by setting the ShortLived.SparkClustersEnabled feature flag to true.
Set ShortLived.UseNewWorkspaceChrome to true.
This enables the new Domino workspace
experience which allows for easy access to the Spark Web UI.
By default, Domino does not come with a Spark compatible Compute Environment that can be used for the components of the cluster. Without at least one such environment available, you will not be able to create a cluster.
When using on-demand Spark in Domino you will have two separate environments - one for the Spark cluster (base or worker environment) and one for the workspace/job execution (compute environment).
To create a new base Spark cluster environment, you will follow the
general
Environment Management
with the following environment_attributes.
-
Base image
Select Custom Image and specify an image URI that points to a deployable Spark image.