Skip to content

How to use containers in the DCC and with OnDemand

You can use Apptainer/Singularity containers to encapsulate software that you require to run your analyses. These containers can be used with normal batch and interactive Slurm jobs and they can also be used with the OnDemand server (provided they are built to support the OnDemand application).

Some basic information and links to further documentation for containers can be found at https://dcc.duke.edu/software/singularity/.

A suggested method to create containers for use with Jupyter Lab and RStudio follow.

In both cases you will be using the OIT Gitlab instance to create a project that will build your container.

  • Open https://gitlab.oit.duke.edu in your browser and click on the "Duke Shibboleth Login" button. If you don't have a GitLab account already one will be created for you.
  • You should the click on the "New Project" button and create a project. The project name will also be the name of the container image file. The project can be owned by you or a GitLab group. You may want the ownership to be a group if the image will be shared with lab members.
  • You will then need to add two files to your project: .gitlab-ci.yml and Singularity.def. You will also want to update the README.md to reflect what your container is used for. The content of the .gitlab-ci.yml file should be an exact copy of the one in the following two projects. The Singularity.def contents will vary as this is where you customize what packages you want installed in your image.

Jupyterlab project

Use the project https://gitlab.oit.duke.edu/OIT-DCC/dcc-container-example as an example for creating an image that will contain Jupyterlab and any additional Python modules you need to use.

  • Copy the contents of the .gitlab-ci.yml file to your own projects .gitlab-ci.yml file.
  • Create your own Singularity.def file using the one in the project as a template. Change the %label to reflect you or your groups information, and in the %post section after the mamba install of jupyterlab add lines that install the modules you wish to use.
  • When you push your changes to your project the .gitlab-ci.yml files precense will cause a GitLab CI process to kick off. This will build a container with the name of your project and deploy it to the DCC. It will be stored at /opt/apps/containers/community//.sif or /opt/apps/containers/community/<GitLab_Group/.sif.
  • To use your image, launch the "Jupyter Lab Singularity" app under "Interactive Apps" at https://dcc-ondemand-01.oit.duke.edu. In the "Singularity Container File" field enter the stored location of your image e.g. /opt/apps/containers/community//.sif and launch the job. Your Jupyter Lab job will be able to use the modules you built your container with.

RStudio project

Use the project https://gitlab.oit.duke.edu/OIT-DCC/dcc-container-rstudio-example as an example for creating an image that will contain RStudio and any additional R modules you need to use.

  • Copy the contents of the .gitlab-ci.yml file to your own projects .gitlab-ci.yml file.
  • Create your own Singularity.def file using the one in the project as a template. Change the %label to reflect you or your groups information, and in the %post section There is a section commented "R: Install Additional R Packages" that shows installing additional R packages.
  • When you push your changes to your project the .gitlab-ci.yml files precense will cause a GitLab CI process to kick off. This will build a container with the name of your project and deploy it to the DCC. It will be stored at /opt/apps/containers/community//.sif or /opt/apps/containers/community/<GitLab_Group/.sif.
  • To use your image, launch the "RStudio Singularity" app under "Interactive Apps" at https://dcc-ondemand-01.oit.duke.edu. In the "Singularity Container File" field enter the stored location of your image e.g. /opt/apps/containers/community//.sif and launch the job. Your RStudio job will be able to use the modules you built your container with.