Custom Jupyter Notebooks using Singularity

In a Jupyter Notebook, there are some cases where you may need a specialized software stack that does not work with the default Python environment. The vast majority of cases only require that you set up a custom environment for your set of modules. An example of that can be found here.

 

The example below demonstrates how to launch a python environment encapsulated within a Singularity (also known as Apptainer) container. In this example, we launch python from within a Singularity container on an Agate node in the A40 partition. The container is based off of Ubuntu 20.04, and includes a stack of software for performing GPU-accelerated simulations of Quantum Circuits using IBM Qiskit, Google Cirq, ProjectQ, and QuTiP.

 

• Agate cluster
• A40 GPU
• Interactive GPU partition
• JupyterLab Interface
• Open OnDemand
• Singularity (Apptainer)

 

1. Build or Locate a Singularity Image
2. Make the Custom Kernel Selectable by Assigning a Display Name
3. Edit the Kernel Description File
4. Launch a Notebook on an A40 Node
5. Select kernel

For the example below, you can use a public singularity image provided. The definition file for this example image is also provided, so you can build the image yourself. If you have never used Singularity before, please start here.

To build the image yourself, you can start with a Singularity Definition File like this one. The Definition File contains all the software packages (including JupyterLab) that are needed. If you are running singularity on your laptop, you can build a new image with the command:

singularity build my-new-image.sif my-definition-file.def

 

2. Make the Custom Kernel Selectable by Assigning a Display Name

Create a skeleton .json file for the python kernel that you want to run from a Singularity container.

python3 -m ipykernel install --user --name singularity-qiskit --display-name "Python 3.8 Singularity QISKIT"

3. Edit the Kernel Description File

In a shell (through OnDemand or ssh) open the file

​~/.local/share/jupyter/kernels/singularity-qiskit/kernel.json

{
 "argv": [
  "/usr/bin/singularity",
  "exec",
  "--nv",
  "-B",
  "/home/mygroup/myuname:/home/mygroup/myuname",
  "-B",
  "/panfs/roc:/panfs/roc",
  "-B",
  "/panfs/jay:/panfs/jay",
  "https://public.s3.msi.umn.edu/singularity-images/quantum-oct-2022.sif",
  "/usr/bin/python3",
  "-m",
  "ipykernel",
  "-f",
  "{connection_file}"
 ],
 "interrupt_mode": "message",
 "display_name": "Python 3.8 Singularity QISKIT",
 "language": "python"
}

In the example above, replace  /home/mygroup/myuname with your home directory. 

 

After you have a singularity image, note the location. It can be placed somewhere on Primary storage (home directory or group project space) or it can be placed as a publically accessible object on an S3 bucket. For example, https://public.s3.msi.umn.edu/singularity-images/quantum-oct-2022.sif 

4. Launch a Notebook on an A40 Node

From OnDemand, 

 

 

5. Connect to Jupyter Notebook

Connect to the Notebook

6. Select kernel