domino logo
Tech Ecosystem
Get started with Python
Step 0: Orient yourself to DominoStep 1: Create a projectStep 2: Configure your projectStep 3: Start a workspaceStep 4: Get your files and dataStep 5: Develop your modelStep 6: Clean up WorkspacesStep 7: Deploy your model
Get started with R
Step 0: Orient yourself to Domino (R Tutorial)Step 1: Create a projectStep 2: Configure your projectStep 3: Start a workspaceStep 4: Get your files and dataStep 5: Develop your modelStep 6: Clean up WorkspacesStep 7: Deploy your model
Get Started with MATLAB
Step 1: Orient yourself to DominoStep 2: Create a Domino ProjectStep 3: Configure Your Domino ProjectStep 4: Start a MATLAB WorkspaceStep 5: Fetch and Save Your DataStep 6: Develop Your ModelStep 7: Clean Up Your Workspace
Step 8: Deploy Your Model
Scheduled JobsLaunchers
Step 9: Working with Domino Datasets
Domino Reference
Projects
Projects Overview
Revert Projects and Files
Revert a ProjectRevert a File
Projects PortfolioProject Goals in Domino 4+Jira Integration in DominoUpload Files to Domino using your BrowserCopy ProjectsFork and Merge ProjectsSearchSharing and CollaborationCommentsDomino Service FilesystemCompare File RevisionsArchive a Project
Advanced Project Settings
Project DependenciesProject TagsRename a ProjectSet up your Project to Ignore FilesUpload files larger than 550MBExporting Files as a Python or R PackageTransfer Project Ownership
Domino Runs
JobsDiagnostic Statistics with dominostats.jsonNotificationsResultsRun Comparison
Advanced Options for Domino Runs
Run StatesDomino Environment VariablesEnvironment Variables for Secure Credential StorageUse Apache Airflow with Domino
Scheduled Jobs
Domino Workspaces
WorkspacesUse Git in Your WorkspaceUse Visual Studio Code in Domino WorkspacesPersist RStudio PreferencesAccess Multiple Hosted Applications in one Workspace Session
Spark on Domino
On-Demand Spark
On-Demand Spark OverviewValidated Spark VersionConfigure PrerequisitesWork with your ClusterManage DependenciesWork with Data
External Hadoop and Spark
Hadoop and Spark OverviewConnect to a Cloudera CDH5 cluster from DominoConnect to a Hortonworks cluster from DominoConnect to a MapR cluster from DominoConnect to an Amazon EMR cluster from DominoRun Local Spark on a Domino ExecutorUse PySpark in Jupyter WorkspacesKerberos Authentication
On-Demand Ray
On-Demand Ray OverviewValidated Ray VersionConfigure PrerequisitesWork with your ClusterManage DependenciesWork with Data
Customize the Domino Software Environment
Environment ManagementDomino Standard EnvironmentsInstall Packages and DependenciesAdd Workspace IDEs
Partner Environments for Domino
Use MATLAB as a WorkspaceCreate a SAS Data Science Workspace EnvironmentNVIDIA NGC Containers
Advanced Options for Domino Software Environment
Install Custom Packages in Domino with Git IntegrationAdd Custom DNS Servers to Your Domino EnvironmentConfigure a Compute Environment to User Private Cran/Conda/PyPi MirrorsScala notebooksUse TensorBoard in Jupyter Workspaces
Publish your Work
Publish a Model API
Model Publishing OverviewModel Invocation SettingsModel Access and CollaborationModel Deployment ConfigurationPromote Projects to ProductionExport Model Image
Publish a Web Application
App Publishing OverviewGet Started with DashGet Started with ShinyGet Started with FlaskContent Security Policies for Web Apps
Advanced Web Application Settings in Domino
App Scaling and PerformanceHost HTML Pages from DominoHow to Get the Domino Username of an App Viewer
Launchers
Launchers OverviewAdvanced Launcher Editor
Assets Portfolio Overview
Connect to your Data
Data in Domino
Datasets OverviewDatasets Best Practices
Data Sources Overview
Connect to Data Sources
External Data Volumes
Git and Domino
Git Repositories in DominoWork From a Commit ID in Git
Work with Data Best Practices
Work with Big Data in DominoWork with Lots of FilesMove Data Over a Network
Advanced User Configuration Settings
User API KeysDomino TokenOrganizations Overview
Use the Domino Command Line Interface (CLI)
Install the Domino Command Line (CLI)Domino CLI ReferenceDownload Files with the CLIForce-Restore a Local ProjectMove a Project Between Domino DeploymentsUse the Domino CLI Behind a Proxy
Browser Support
Get Help with Domino
Additional ResourcesGet Domino VersionContact Domino Technical SupportSupport Bundles
domino logo
About Domino
Domino Data LabKnowledge BaseData Science BlogTraining
User Guide
>
Domino Reference
>
Spark on Domino
>
On-Demand Spark
>
Work with Data

Work with Data

When using a Domino on-demand Spark cluster any data that will be used, created, or modified as part of the interaction must go into an external data store.

Note

Use Domino datasets

When you create a Spark cluster attached to a Domino workspace or job, any Domino dataset accessible from the workspace or job will also be accessible from all components of the cluster under the same dataset mount path. Data can be accessed using the file:/// path prefix.

For example, to read a file you would use the following.

rdd = sc.textFile("file:///path/to/file")

No additional configuration of the Spark cluster environment or the execution environment is required.

Use S3

To enable working with data in Amazon S3 (or S3 compatible object store) you must ensure that your base Spark cluster environment and compatible PySpark compute environment are configured with the Hadoop-AWS module.

The environments created when configuring prerequisites will at a minimum include Hadoop 2.7.3 client libraries which are sufficient for basic access. A number of additional commonly used features (for example, temporary credentials, SSE-KMS encryption, more efficient committers, etc) are only available in more recent Hadoop-AWS module versions.

Consult the documentation for the relevant version to determine what may be the best fit for you.

  • Hadoop-AWS Module 2.7.3

  • Hadoop-AWS Module 2.8.5

  • Hadoop-AWS Module 2.9.2

  • Hadoop-AWS Module 3.1.3

  • Hadoop-AWS Module 3.2.1

For Spark 3.1.1, a good advanced option would be Hadoop 3.2.0 or later.

S3 Usage Examples

Now that you have your environments properly setup, you can interact with S3. The following are several common access patterns.

Access bucket with AWS credentials in environment variables

import os
from pyspark.sql import SparkSession

spark = SparkSession.builder.getOrCreate()

# the default configuration will pick up your credentials from environment variables
# No additional configuration is necessary

# test reading
df = spark.read.json("s3a://bucket/prefix1/prefix2/people.json")
df.show()

Access bucket with SSE-KMS encryption

Note
import os
from pyspark.sql import SparkSession

spark = SparkSession.builder.getOrCreate()

# for write operations you will need the ARN of the key to use
# Note that the credentials used need to have proper access to use the key
kms_key_arn = "<your key ARN here>"

# configure the connector
# This example assumes credentials from environment variables so no need to configure
# Note: The encryption config is not needed for read only operations
hadoop_conf = spark.sparkContext._jsc.hadoopConfiguration()
hadoop_conf.set("fs.s3a.server-side-encryption-algorithm", "SSE-KMS")
hadoop_conf.set("fs.s3a.server-side-encryption.key", kms_key_arn)

# test reading
df = spark.read.json("s3a://bucket/prefix1/prefix2/people.json")
df.show()

# test writing
df.write.mode("overwrite").parquet("3a://bucket/prefix1/prefix2/write-test/output")

Access a bucket with Domino assumed temporary credentials

Note
import os
from pyspark.sql import SparkSession

try:
    spark.stop()
except:
    pass
spark = SparkSession.builder.getOrCreate()

#The name of one of the roles you are entitled to
profile_name="my-role-name-read-write"

# use boto3 for convenience to get credentials form credentials file populated by Domino
# can use any method desirable to extract the credentials
import boto3
role_creds = boto3.Session(profile_name=profile_name).get_credentials().get_frozen_credentials()

# configure the connector
# Use the TemporaryAWSCredentialsProvider
hadoop_conf = spark.sparkContext._jsc.hadoopConfiguration()
hadoop_conf.set("fs.s3a.aws.credentials.provider", "org.apache.hadoop.fs.s3a.TemporaryAWSCredentialsProvider")
hadoop_conf.set("fs.s3a.access.key", role_creds.access_key)
hadoop_conf.set("fs.s3a.secret.key", role_creds.secret_key)
hadoop_conf.set("fs.s3a.session.token", role_creds.token)

# test reading
df = spark.read.json("s3a://bucket/prefix1/prefix2/people.json")
df.show()

# test writing
df.write.mode("overwrite").parquet("s3a://bucket/prefix1/prefix2/write-test/output")

For full set of configuration options see the documentation for the Hadoop-AWS module.

Using Azure Data Lake Storage Gen2

To enable working with data in Azure Data Lake Storage (ADLS) Gen2 you need to configure your base Spark environment and your compute environment with the Hadoop-Azure ABFS connector.

The ABFS connector requires Hadoop 3.2+.

To accomplish this set SPARK_VERSION=3.0.0 and HADOOP_VERSION=3.2.1 when following the advanced instructions for base Spark cluster environment and compatible PySpark compute environment.

Note

Access AWS Resources from a Spark Cluster

Note

This feature requires Hadoop 2.9.2 or higher. If you’re using an earlier version of Hadoop, you’ll need to configure your base Spark environment or your PySpark environment to use Hadoop 2.9.2+.

You can configure on-demand Spark clusters in your Domino workspace to access AWS resources using temporary credentials issued by AWS. To do this, your Domino deployment must use single sign-on (SSO) with a trusted identity provider (IdP). The credentials can also be continuously refreshed, allowing your Spark cluster to have continuous access to AWS resources.

The specific credentials (and associated privileges) issued by AWS to your Spark cluster are based on role profiles defined in your IdP by a Domino administrator. These profiles include identity attributes used by AWS to issue appropriate temporary credentials corresponding to a role profile. The temporary credentials are then automatically distributed to your on-demand Spark cluster.

For more details on this credential propagation architecture, see the AWS Credential Propagation section in the Domino Administrator’s Guide.

To take advantage of this feature, you can either configure your Spark context dynamically to work with profile role credentials in your code, or configure the desired profile in your project settings. Both methods are described below and you should select the option that best matches your use case.

Configure your Spark context dynamically in your code

This method provides you with more flexibility and is recommended if you need to frequently change role profiles. Recall that your Spark clusters must use Hadoop 2.9.2 or higher and may need to be configured accordingly prior to implementing the following code snippet.

import os
from pyspark.sql import SparkSession

try:
    spark.stop()
except:
    pass

# First, set the AWS_PROFILE environment variable to the name of the profile found in the credentials file $AWS_SHARED_CREDENTIALS_FILE.
# If you're doing this in a notebook, first stop your Spark session or context for the change to take effect.
os.environ['AWS_PROFILE'] = 'name-of-profile-to-use'

# Next, configure the Spark connector by setting up the provider type and the name of the profile to use. Be sure to replace the .appName() argument with the name of your app.
spark = SparkSession.builder \
          .appName("Credential Spark Test") \
          .config("spark.hadoop.fs.s3a.aws.credentials.provider", "com.amazonaws.auth.profile.ProfileCredentialsProvider") \
          .config("spark.executorEnv.AWS_PROFILE", os.environ['AWS_PROFILE']) \
          .getOrCreate()

# Read some data from AWS (replace with your S3 URI)
df = spark.read.json("s3a://foobar/bazbux.json")
df.show()

Configure your Spark context from your Domino project settings

You can also enable this feature by adding Spark configuration options in your Domino project settings. This method provides less flexibility and is recommended for projects that will utilize one consistent role profile. To enable this feature:

  1. Go to Settings in your Domino project.

  2. Click the Integrations tab.

  3. In the Apache Spark mode section, select Domino managed on-demand cluster.

  4. In the Spark Configuration Options text area, add the keys and values specified below. Ensure one whitespace between the key and the value.

    spark.hadoop.fs.s3a.aws.credentials.provider com.amazonaws.auth.profile.ProfileCredentialsProvider
    spark.executorEnv.AWS_PROFILE name-of-profile-to-use

    spark-cred-1

  5. Go to your Domino account settings and click User Environment Variables. Under Set user environment variable, set Name to AWS_PROFILE and set Value to the name of the profile you’d like to use (name-of-profile-to-use in the previous step). Click Set Variable.

    spark-cred-2

Kerberos keytab propagation

When Kerberos Authentication is enabled either in user settings or in project settings, the uploaded keytab will be automatically distributed to all cluster containers at a well-known location.

By default, the keytab will be available at: /etc/security/keytabs/keytab

Alternatively, a Domino administrator can change the path where the keytab will be available by using the com.cerebro.domino.integrations.kerberos.keytabMountPath central configuration setting.

Domino Data LabKnowledge BaseData Science BlogTraining
Copyright © 2022 Domino Data Lab. All rights reserved.