This topic describes how to use Domino Datasets to solve problems, improve collaboration, and open new workflow possibilities in Domino.
When you start a Run or launch a Workspace, Domino copies your project files to a Domino execution. When working with large volumes of data, this presents the following potential issues:
-
By default, you can store 10,000 files in a Domino project and you might exceed the limit.
-
By default, you can only transfer individual files that are 8GB to and from your Domino project files, and you might exceed the limit.
-
The time required to transfer data to and from the Domino executions is proportional to the size of the data. It can take a long time if the size of the data is large, which can lead to long startup and shutdown times for Workspaces, Jobs, Apps, and Launchers.
You can solve these problems with Domino Datasets because:
-
Domino datasets do not have a limit on the number of files that can be stored.
-
Domino datasets do not have a limit on the size of any individual file.
-
Domino Datasets are directly attached to executions as networked filesystems, so you do not have to transfer their contents when executions start or complete.
When you want to reproduce a training experiment, you might want to version a Domino Dataset so that you can return to a specific version used in the past.
To do this with Domino:
-
Create a snapshot to create versions of a Domino dataset.
-
Use a naming convention and a folder hierarchy to organize data your way in the read/write portions of a dataset.
If you export and import project content to share data with other members of your team, the consumers of your project will receive the entire contents of your project files in their Runs and Workspaces. That works well if your project is small, simple, and narrowly scoped.
However, for large projects that produce many data artifacts, you might want to expose them to your consumers in smaller, curated subsets. You can do this with Domino Datasets.
Consider the following project.
This project has a small folder full of code and nine folders with various kinds of output data. Each data folder is larger than 10GB, and the whole project is 100GB. It would be impractical to ask your data consumers to import this project, but you also don’t want to separate the data from the code that produced it by moving the data to a different project.
You can organize the data into Datasets, with one Dataset for each type of data in which your consumers are interested.
In this example, suppose you have two colleagues who want to consume your data.
One of them is only interested in the data from the experiment1 folder, and the other is only interested in the data from experiment9.
You can create and write two Datasets with scripts like the following, where it’s assumed you have named the Datasets experiment1-data and experiment9-data.
experiment1-populate-dataset.shcp -R $DOMINO_WORKING_DIR/experiment1/. /domino/datasets/experiment1-data/experiment9-populate-dataset.shcp -R $DOMINO_WORKING_DIR/experiment9/. /domino/datasets/experiment9-data/Your consumers can then mount only the datasets in which they are interested.
If you are working with data at this scale, write it to datasets whenever you produce it, instead of storing it in your project files.
Work with local data
If you use the Domino CLI to work with projects on your local machine, you might find that storing large data files slows your download and sync operations, and fills up a lot of your local disk storage. To prevent this, store data in a Domino dataset, and reserve your project files for the scripts and documents that you want to work with locally.
To simplify your local workflow:
-
Create a Dataset in your project, and write your large data files to it.
-
After the files have been written to the Dataset, remove them from your project files.
-
Fetch a clean, lighter-weight copy of your project.
-
Update your code to reference your data files in their new location, at:
/domino/datasets/local/<dataset-name>/ -
When everything is working properly, delete copies of the project that have the large data files in them from your local machine.
Load data from external sources into Domino
If you have data in an external source from which you want to periodically fetch and load into Domino, you can set up scheduled Jobs to write to Datasets.
Suppose you have data stored in an external data source that is periodically updated. If you wanted to fetch the latest state of that file once a week and load it into a Domino dataset, you could set up a scheduled Run:
-
Create a dataset to store the data from the external source.
-
Write a script that fetches the data and writes it to the dataset.
-
Create a scheduled job to run your script with the new dataset configuration.
The following is a detailed example showing how to fetch a large, dynamic data file from a private S3 bucket with a scheduled Run once a week.
-
Create a dataset to hold the file. This example shows the Dataset named
fetched-from-s3.
For this example, assume the S3 bucket is named
my_bucketand the file you want is namedsome_data.csv. You can set up your script like this:fetch-data.pyimport boto3 import io # create new S3 client client = boto3.client('s3') # download some_data.csv from my_bucket and write to latest-S3 output mount file = client.download_file('my_bucket', 'some_data.csv', '/domino/datasets/fetched-from-s3/some_data.csv') -
Set up a scheduled job that executes this script once a week with the correct dataset configuration.