Running MIRACL commands on Sherlock (Stanford supercomputer)

This tutorial highlights the registration workflow but a similar approach applies to other commands.

Setting up MIRACL (first time)

Log in to Sherlock:

$ ssh -Y username@sherlock.stanford.edu

Start an interactive session:

$ sdev

Move to your scratch folder:

$ cd SCRATCH

Build or download your MIRACL container:

Building from DockerHub image:

$ apptainer build miracl.sif docker://mgoubran/miracl:latest

To download a MIRACL Apptainer binary directly, use either of the following commands:

$ wget https://huggingface.co/datasets/AICONSlab/MIRACL/resolve/dev/apptainer/versions/miracl_v242.sif

or

$ curl -L -O https://huggingface.co/datasets/AICONSlab/MIRACL/resolve/dev/apptainer/versions/miracl_v242.sif

Note

Replace the version number with the version of MIRACL you want to download.

Tip

If you have a particular Apptainer container of MIRACL that you want to use on Sherlock, just copy it to the servers directly using e.g. scp or rsync instead of pulling (downloading) the latest version of MIRACL from the Apptainer registry

Copying your data to Sherlock

Copy a folder called, e.g. input_clar with tiff files that you want to register to the Allen Atlas using scp:

$ scp -r input_clar sherlock.stanford.edu:/scratch/users/<username>/clarity_registration/.

or rsync:

$ rsync -avPhz input_clar sherlock.stanford.edu:/scratch/users/<username>/clarity_registration/.

Attention

Make sure to replace <username> with your Sherlock username

Running MIRACL in an interactive session

For quick jobs that don’t require much resources you can login to Sherlock:

$ ssh -Y username@sherlock.stanford.edu

Move to your scratch folder:

$ cd SCRATCH

Start interactive session:

$ sdev

Start Apptainer with bound data:

$ apptainer shell miracl_latest.sif bash

Note

Use –nv to forward your Nvidia GPU into the container and -B to bind volumes to the container.

Within the shell, load the GUI:

$ miraclGUI

Or use the command-line:

$ miracl lbls stats -h

Note

Please consult our Troubleshooting section if you experience problems with opening MIRACL’s GUI on Sherlock

Running SBATCH jobs

If you want to run jobs with specific resources for larger, longer jobs (e.g. running the registration workflow) you can do the following:

First get the data orientation (please check the registration tutorial for setting orientation):

$ miracl conv set_orient

After setting the orientation, a file called ort2std.txt will be created that might look like this:

$ cat ort2std.txt
tifdir=/scratch/users/username/clarity_registration/input_clar
ortcode=ARS

Use that orientation code (ARS) in your registration workflow.

First check the workflow arguments:

$ miracl flow reg_clar -h

Assuming you wanted to run this command with the following arguments, for example on your data:

$ miracl flow reg_clar -f input_clar -n "-d 5 -ch autofluo" -r "-o ARS -m combined -v 25"

Create an sbatch script named, for example reg_job.sbatch and paste the following lines:

#!/bin/bash
#SBATCH --job-name=clar_reg
#SBATCH --ntasks=1
#SBATCH --time=05:00:00
#SBATCH --cpus-per-task=12
#SBATCH --mem=32G

module load apptainer

apptainer exec ${SCRATCH}/miracl_latest.sif miracl flow reg_clar -f ${SCRATCH}/clarity_registration/input_clar -n "-d 5 -ch autofluo" -r "-o ARS -m combined -v 25"

Attention

Note that the miracl function call comes after invoking the Apptainer call apptainer exec ${SCRATCH}/miracl_latest.sif and that full file paths were used for the .sif container and the input data

This sample job (called: clar_reg) asks for 5 hours, 12 cpus and 32G of memory on one node. Adjust the requested resources based on the job you are submitting.

Next submit the sbatch script:

$ sbatch reg_job.sbatch

To check on the status of your submitted job use:

$ squeue -u $USER

Jupyter notebook

An accompanying Jupyter notebook for this tutorial can be found here.