Skip to content

Quick Start for Python and Jupyter Notebooks

Before you start, know your storage locations.

One-Time Setup Steps

  1. Login to the DCC using [ssh](../dcc/login.md#SSH-Access) <netid>@dcc-login.oit.duke.edu (VPN is not required)
Sample ssh login with multifactor 
kk338@CDSS-5630 ~ % ssh kk338@dcc-login.oit.duke.edu
Password:
Duo two-factor login for kk338
    1. Duo Push to XXX-XXX-4007
    2. Phone call to XXX-XXX-4007
    3. Phone call to XXX-XXX-9784
    4. SMS passcodes to XXX-XXX-4007 (next code starts with: 2)
Passcode or option (1-4): 1
Success. Logging you in...
Last login: Fri Nov 15 16:15:46 2024 from dcc-ondemand-01.oit.duke.edu
################################################################################
# Please report any issues or questions to: oitresearchsupport@duke.edu        #
#                                                                              #
# Duke Research Computing info and documentation: https://rc.duke.edu          #
#                                                                              #
# **NOTICE** : There will be a complete outage of OIT provided Research        #
# Computing services from 1/6/25 to 1/10/25 for routine maintenance and        #
# patching.                                                                    #
#                                                                              #
# My next patch run is Saturday (11-23-2024) at  3 AM                           #
################################################################################
712M   (3%) used of  25G available in /hpc/home/kk338
712G  (70%) used of 1.0T available in /hpc/group/rescomp
164G  (33%) used of 512G available in /hpc/group/temp
136M   (1%) used of 500T available in /hpc/dctrl/kk338
kk338@dcc-login-03  ~ $
  1. Add a symbolic link to your home directory A symbolic link is a shortcut to another file system. In this case, it will help you use file navigation in a Jupyter Notebook, which defaults to your home directory. Symbolic links are created with ln -s <full_path> <link_name> When you first login, you will be in your home directory, create your shortcuts there.
Sample Symbolic Link Creation 
kk338@dcc-login-03  ~ $ pwd
/hpc/home/kk338
kk338@dcc-login-03  ~ $ ls -l
kk338@dcc-login-03  ~ $ ln -s /hpc/dctrl/kk338 dctrl_kk338
kk338@dcc-login-03  ~ $ ls -l
lrwxrwxrwx. 1 kk338 dukeusers       16 Nov 18 10:57  dctrl_kk338 -> /hpc/dctrl/kk338
kk338@dcc-login-03  ~ $ 

kk338@dcc-login-03  ~ $ pwd
/hpc/home/kk338
kk338@dcc-login-03  ~ $ ls -l
kk338@dcc-login-03  ~ $ ln -s /hpc/group/rescomp rescomp
kk338@dcc-login-03  ~ $ ls -l
lrwxrwxrwx. 1 kk338 dukeusers       16 Nov 18 10:57  rescomp -> /hpc/group/rescomp
kk338@dcc-login-03  ~ $
  1. Install Miniconda
    Installation commands: 
    wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
sh Miniconda3-latest-Linux-x86_64.sh
    and then follow the instructions. The place to give as Miniconda install location should be /hpc/dctrl/<netid>/miniconda3 or /hpc/dctrl/<netid>/miniconda3. It will offer to update your ~/.bashrc, (init) which is what you want. Log out log back in and then you can run conda install, pip install, create environments, etc.

Managing Python Environments

  1. Creating a conda environment Users who will be using Python through SSH and OnDemand should setup virtual environments using miniconda with the desired Python version and packages. After installing Minconda, create a virtual environment. In the example below, test is the name of the environment and python=3.9 is the python version number. Remember to log out, then log back into the cluster after completing your installation of miniconda, to create your environments: 

    conda create -n test python=3.9
conda activate test
conda install ipykernel
python -m ipykernel install --user --name test

  2. Activating your environment First, to use your environment, you need to activate your environment: 

    conda activate test
    You will know your environment is activated because it will show up on your command line: 
    (test) kk338@dcc-login-03  ~ $

  3. Installing any packages needed in that environment Then, you can install any packages you need using conda or pip. If your environment is activated, conda keeps track of which packages are installed where. Some packages will require pip to install - your new conda environment has it’s own pip install.

Accessing Jupyter Lab through DCC OnDemand portal

  1. Login at https://dcc-ondemand-01.oit.duke.edu

  2. Click on Interactive Apps in the top navigation menu

  3. Click on Jupyter Lab

    a. Select the name of your DCC group (dctrl-) b. Under partition, make a selection as follows: - If you need CPU resources, select "common" - If you need GPU resources, select "gpu-common" - In case you fail to get a session within 5 minutes, try "scavenger-gpu" c. Input the number of hours you would like the server to remain active (please try to remain small, as it will continue running even if you are not using it) d. Input the desired amount of nodes, memory, and CPUs (try to start small with only a few gigabytes of memory and cores) e. Press the blue "Launch" button on the bottom of the page

  4. After pressing the blue "launch" button, your job will be queued to start a Jupyter Lab server You should see this automatically

  5. Wait a few minutes for the Jupyter Lab server to finish launching. The status will automatically change from "Starting" to "Running" when the server is ready
  6. Press the blue "Connect to Jupyter" button when the server is running to access your Jupyter Lab server