domino logo
Tech Ecosystem
Get started with Python
Step 0: Orient yourself to DominoStep 1: Create a projectStep 2: Configure your projectStep 3: Start a workspaceStep 4: Get your files and dataStep 5: Develop your modelStep 6: Clean up WorkspacesStep 7: Deploy your model
Get started with R
Step 0: Orient yourself to Domino (R Tutorial)Step 1: Create a projectStep 2: Configure your projectStep 3: Start a workspaceStep 4: Get your files and dataStep 5: Develop your modelStep 6: Clean up WorkspacesStep 7: Deploy your model
Domino Reference
Projects
Projects Overview
Revert Projects and Files
Revert a ProjectRevert a File
Projects PortfolioProject Goals in Domino 4+Jira Integration in DominoUpload Files to Domino using your BrowserFork and Merge ProjectsSearchSharing and CollaborationCommentsDomino Service FilesystemCompare File RevisionsArchive a Project
Advanced Project Settings
Project DependenciesProject TagsRename a ProjectSet up your Project to Ignore FilesUpload files larger than 550MBExporting Files as a Python or R PackageTransfer Project Ownership
Domino Runs
JobsDiagnostic Statistics with dominostats.jsonNotificationsResultsRun Comparison
Advanced Options for Domino Runs
Run StatesDomino Environment VariablesEnvironment Variables for Secure Credential StorageUse Apache Airflow with Domino
Scheduled Jobs
Domino Workspaces
WorkspacesUse Visual Studio Code in Domino WorkspacesPersist RStudio PreferencesAccess Multiple Hosted Applications in one Workspace SessionUse Domino Workspaces in Safari
Spark on Domino
On-Demand Spark
On-Demand Spark OverviewValidated Spark VersionConfigure PrerequisitesWork with your ClusterManage DependenciesWork with Data
External Hadoop and Spark
Hadoop and Spark OverviewConnect to a Cloudera CDH5 cluster from DominoConnect to a Hortonworks cluster from DominoConnect to a MapR cluster from DominoConnect to an Amazon EMR cluster from DominoRun Local Spark on a Domino ExecutorUse PySpark in Jupyter WorkspacesKerberos Authentication
Customize the Domino Software Environment
Environment ManagementDomino Standard EnvironmentsInstall Packages and DependenciesAdd Workspace IDEs
Advanced Options for Domino Software Environment
Install Custom Packages in Domino with Git IntegrationAdd Custom DNS Servers to Your Domino EnvironmentConfigure a Compute Environment to User Private Cran/Conda/PyPi MirrorsScala notebooksUse TensorBoard in Jupyter WorkspacesUse MATLAB as a WorkspaceCreate a SAS Data Science Workspace Environment
Publish your Work
Publish a Model API
Model Publishing OverviewModel Invocation SettingsModel Access and CollaborationModel Deployment ConfigurationPromote Projects to ProductionExport Model Image
Publish a Web Application
Cross-Origin Security in Domino web appsApp Publishing OverviewGet Started with DashGet Started with ShinyGet Started with Flask
Advanced Web Application Settings in Domino
App Scaling and PerformanceHost HTML Pages from DominoHow to Get the Domino Username of an App Viewer
Launchers
Launchers OverviewAdvanced Launcher Editor
Assets Portfolio Overview
Connect to your Data
Domino Datasets
Datasets OverviewDatasets Best PracticesAbout domino.yamlDatasets Advanced Mode TutorialDatasets Scratch SpacesConvert Legacy Data Sets to Domino Datasets
Data Sources OverviewConnect to Data Sources
Git and Domino
Git Repositories in DominoWork From a Commit ID in Git
Work with Data Best Practices
Work with Big Data in DominoWork with Lots of FilesMove Data Over a Network
Advanced User Configuration Settings
User API KeysOrganizations Overview
Use the Domino Command Line Interface (CLI)
Install the Domino Command Line (CLI)Domino CLI ReferenceDownload Files with the CLIForce-Restore a Local ProjectMove a Project Between Domino DeploymentsUse the Domino CLI Behind a Proxy
Browser Support
Get Help with Domino
Additional ResourcesGet Domino VersionContact Domino Technical SupportSupport Bundles
domino logo
About Domino
Domino Data LabKnowledge BaseData Science BlogTraining
User Guide
>
Domino Reference
>
Customize the Domino Software Environment
>
Environment Management

Environment Management

You can create new Domino environments and edit existing environments to meet your language and package needs. You can create or modify an environment when:

  • You must install a package for Python, R, Octave, or some other software dependency.

  • You want to cache a package or library so that it’s immediately available when you start a run.

  • You are managing an organization, and want to create a default environment across all projects.

A Domino environment is an abstraction on top of a Docker image. When Domino starts your run, it creates an isolated Docker container based on the environment associated with your project. This helps provide flexibility and versioning for your runs.

Set the default environment for a project
  1. Go to the project for which you want to set a default environment. . In the navigation pane, click Settings.

    Screen_Shot_2018-10-23_at_11.30.21_AM.png

  2. From Compute environment, select the environment.

    screen shot 2018 10 23 at 11.31.29 AM

  3. Click Manage Environment to open the environment’s Overview tab.

    Overview tab

    The Overview tab lists all metadata about your environment. Click Edit Definition to make changes. After every save, your environment’s revision number increments by one, and your Domino deployment rebuilds the environment and pushes it to the local Docker registry.

    Revisions tab

    Each time you edit an environment, you create a new revision. The Revisions tab lists all revisions of your compute environment, along with each revision’s build status, timestamp, and docker image URI. Click the gear icon to reveal additional options, including the ability to view build logs, cancel builds, or set a revision as Active.

    Projects and Models tabs

    Projects and models are listed on their own tabs, after they have been assigned an environment. Use these tabs to see who to contact to update or archive an environment.

  4. Click Duplicate Environment to clone an environment.

  5. Click Edit Definition to edit an environment’s attributes.

  6. Click Archive Environment to hide an environment. Projects already using this environment can continue to use it until a new project environment is selected.

Manage environments

In the navigation pane, click Environments to see the environments to which you have access, including: . Your deployment’s global environments. . Environments used by projects with which you are a collaborating. . Environments shared with organizations to which you belong.

Screen_Shot_2018-10-23_at_11.35.48_AM.png

Create an environment

  1. Click Create Environment.

  2. Name your new environment and define its visibility. Administrators can make the environment global, which makes it available to all users who have access to the deployment.

    After creating your environment, the Revision tab opens.

  3. Click Edit Definition to define the Dockerfile and supporting scripts and settings for the environment.

Environment definition

Base Environment

Defines the FROM line in the Dockerfile that Domino constructs for you. You can base your compute environment on your deployment default or on a custom Dockerfile URI (for example, registry.hub.docker.com/library/python:3.8-slim).

Dockerfile Instructions

Enter your Dockerfile layers. See Docker’s official site. You can also view the Dockerfile best practices.

Pluggable Notebooks / Workspace Sessions

Enter the interactive tools needed by any project using this environment. See Add Workspace IDEs.

Scripts

Enter bash code that is executed at the specified step in your experiment’s lifecycle. Domino executes these commands at runtime and runs them as root.

Pre-setup scripts

Run before the Python packages in your project’s requirements.txt are installed.

Post-setup scripts

Run after the requirements.txt installation process.

Pre Run scripts

Run after post-setup scripts.

Post Run scripts

Run at the beginning of the Stopping run state. Because of the way that Domino shuts down runs and workspaces, these scripts are subject to a runtime limit. The default limit is one hour. If the script hasn’t completed by end of the runtime limit, it will time out and the process will terminate.

Username

Specify a non-default username for the environment.

Environment variables

Set environment variables at the environment level.

Raw Dockerfiles

You might want to install packages directly to your environment. This is useful if your package installation takes a long time. Installations in a Domino environment are cached, so you won’t have to wait for the package to install every time. If you already have a Docker image you’d like to use, enter it in the preceding Base Environment field. If you don’t set this, the Domino default environment is used as your base image.

Consult the official Docker documentation to learn more about Dockerfiles:

  • Reference

  • Best practices

Do not start your Dockerfile instructions in Domino with a FROM line. Domino includes the FROM line for you, pointing to the base image specified when setting up the environment.

The most common Dockerfile instructions you’ll use are RUN, ENV, and ARG:

RUN commands execute lines of bash. For example:

RUN wget http://d3kbcqa49mib13.cloudfront.net/spark-1.5.1-bin-hadoop2.6.tgz
RUN tar xvzf spark-1.5.1-bin-hadoop2.6.tgz
RUN mv spark-1.5.1-bin-hadoop2.6 /opt
RUN rm spark-1.5.1-bin-hadoop2.6.tgz

ARG commands set build-time variables, and ENV commands set container bash environment variables. They will be accessible from runs that use this environment. For example:

ENV SPARK_HOME /opt/spark-1.5.1-bin-hadoop2.6

If you set variables in the Environment variables section of your definition, you can use an ARG statement:

ARG SPARK_HOME

This will be available for the build step. If you want the variable to be available in the final compute environment you also need to add an ENV statement referencing the argument name:

ENV SPARK_HOME=$SPARK_HOME

Examples: Package Installation

Click R Package or Python Package when editing your environment, to insert a line to install packages. Enter the names of the packages.

You can also add the commands, as in the following examples:

  • R Package Installation, example with the devtools package.

    RUN R --no-save -e "install.packages('devtools')"
  • Python Package Installation with Pip, for example with the…​ numpy package.

    RUN pip install numpy

Dockerfile best practices

  • Docker optimizes its build process by keeping track of commands it has run and aggressively caching the results. This means that if it sees the same set of commands as in a previous build, it will assume that it can use the cached version. A single new command will invalidate the caching of all subsequent commands.

  • A Docker image can have up to 127 layers, or commands. To work around this limit, you can use &&: to combine several commands into a single command.

    RUN \
      wget http://d3kbcqa49mib13.cloudfront.net/spark-1.5.1-bin-hadoop2.6.tgz && \
      tar xvzf spark-1.5.1-bin-hadoop2.6.tgz && \
      mv spark-1.5.1-bin-hadoop2.6 /opt && \
      rm spark-1.5.1-bin-hadoop2.6.tgz
  • If you are installing multiple python packages via pip, it’s almost always best to use a single pip install command. This ensures that dependencies and package versions are properly resolved. If you install via separate commands, you may end up inadvertently overriding a package with the wrong version, due a dependency specified by a later installation. For example:

    RUN pip install luigi nolearn lasagne
Domino Data LabKnowledge BaseData Science BlogTraining
Copyright © 2022 Domino Data Lab. All rights reserved.