Installing and running MIRACL

We provide instructions on how to install and run MIRACL using either of the following methods:

Important

Docker is our recommended method for running MIRACL on local machines and servers. We recommend Apptainer to run MIRACL in a cluster environment (e.g. Compute Canada).

Attention

Support for installing MIRACL locally (i.e. on your host system directly without using Docker or Apptainer) has been deprecated. Please use Docker or Apptainer instead.

Important

MIRACL has been tested on:

  • Docker (version 27.3.1, build ce12230):

    • Ubuntu: 20.04 (Focal Fossa), 22.04 (Jammy Jellyfish), 24.04 (Noble Numbat)

    • Debian: 13 (trixie)

  • Apptainer (version 1.3.5):

    • Ubuntu: 20.04 (Focal Fossa), 22.04 (Jammy Jellyfish), 24.04 (Noble Numbat)

    • Gentoo: 2.14

We strongly recommend that you use one of the above Linux versions but MIRACL will most likely work on other versions of Linux as long as Docker or Apptainer are installed and working.

Note

Hardware requirements for MIRACL are heavily dependent on the type/size of your data as well as the analyses you are running. While some modules and datasets can be less computationally expensive, others require a lot of resources. As a point of reference, the least powerful system that we are testing MIRACL on, uses a 24-Core Threadripper with 128GB RAM and two NVIDIA RTX A6000 GPUs.

We provide a installation script to automatically create a Docker image for you that can be run using Docker Compose. This method does not require a manual installation of MIRACL and works on Linux and in the WSL2 on Windows. Theoretically, MIRACL should also work on MacOS but we do not officially support it.

Tip

Docker is our recommended method for running MIRACL on local machines and servers

Docker is well suited if you want to run MIRACL on a local machine or local server. If you need to run MIRACL on a cluster, see our instructions for installing Apptainer. If you don’t have Docker installed on your computer, do that first. Make sure your installation includes Docker Compose as it is required to run the installation script we provide. Note that Docker Compose is included as part of the Docker Desktop installation by default.

Getting started

First, it is important to understand how the container is built. There is a base image in the docker folder that installs Python and dependencies. Then the Dockerfile in the base of the repository builds the Docker image from that base. When the build happens, it cats the version.txt file in the repository to save a versioned base, but then the build uses the tag revised-base-latest that is always the latest base. The base container is built from this folder and pushed manually, while the main container is built and pushed automatically via the CircleCI Recipe. Thus, if you want to update the base, you will need to see the README.md in that folder and push new images.

Build MIRACL from scratch

This will build a Docker image of MIRACL based on its latest version using our default naming scheme. For custom names and specific versions see below for our Additional build options section.

Clone the MIRACL repo to your machine:

$ git clone https://www.github.com/mgoubran/MIRACL

Change into the newly created directory where you cloned MIRACL to:

$ cd MIRACL

Build the latest MIRACL image using the installation script we provide:

$ ./install.sh

Attention

In order for the ./install.sh script to work, Docker should NOT be used with sudo. Our script checks and exits if it is being run with sudo priviledges. The reason for this behavior is that the installation script creates a user in the Docker container that matches the uid and gid of the host user which is required correct X11 forwarding. This user should NOT be root which is the case when Docker commands are executed with sudo. For more information on how to add a docker user to use Docker without sudo visit the official Docker documentation.

Error

Make sure that the script can be executed. If it can’t and you are the owner of the file, use chmod u+x install.sh to make it executable. Prefix the chmod command with sudo if you are not the owner of the file or change permissions for g and/or o.

Running the script without any flags will start an interactive installation process that will run you through an abbreviated version of the installation.

$ ./install.sh
No flags provided, starting interactive prompt...
Enter Docker image name (default: 'miracl_mgoubran_kirk_4918_img'):
Enter Docker container name (default: 'miracl_mgoubran_kirk_4918'):
Enable GPU in Docker container (required for ACE) (y/N): y
Enter the location of your data on your host system (default: None). If you choose a location, your data will be mounted at '/data' in your container: /data5/projects/

Press enter for the image and container prompts to choose the default names or enter your preferred names. GPU forwarding will be disabled by default so enter y, Y, Yes, or YES at the prompt to enable it. Lastly, you will be prompted for the location of your data on the host system. Provide the full path to your location. It will be mounted under /data in the container. Once you entered all prompts, MIRACL will be installed automatically.

Note

This installation method should be sufficient for 90% of users. However, If you require more fine grained control see our Additional build options section further below.

Once the image has successfully been built, run the container using Docker Compose:

$ docker compose up -d

Note

Note that the Docker Compose syntax is different if you installed it using the standalone method. Compose standalone uses the -compose syntax instead of the current standard syntax compose. The above command would thus be docker-compose up -d when using Compose standalone.

The container is now running and ready to be used.

Using the container

Interactively shell inside:

$ docker exec -it miracl bash

Stopping the container

Exit your container and navigate to your MIRACL folder. Use Docker Compose to stop the container:

$ docker compose down

Note

Note that the Docker Compose syntax is different if you installed it using the standalone method. Compose standalone uses the -compose syntax instead of the current standard syntax compose. The above command would thus be docker-compose up -d when using Compose standalone.

Mounting additional drives

Files that are saved while using MIRACL should be saved to volumes mounted into the container in order to make them persistent. You might have already mounted at least one data location when you used the interactive installation menu or several if you used the flags outlined in the Additional build options sections below. However, if you need to mount additional volumes, you can easily do that by adding them to the docker-compose.yml in the base directory under the volumes section.

Example:

volumes:
      - '/home/mgoubran/.Xauthority:/home/mgoubran/.Xauthority'
      - '/home/mgoubran/mydata:/home/mgoubran/mydata'  # This is the additional volume

Danger

Do not delete the .Xauthority volume that is already mounted. It mounts your .Xauthority file which is important for X11 to work correctly.

The format of mounting volumes is </host/path>:</container/path> (note that the delimiter is : but that there are no trailing /). In the above example, the host path /home/mgoubran/mydata is mounted to the container path /home/mgoubran/mydata. The names for the paths on the host system do not need to match the names for the locations inside the container. The above example could therefore also be /home/mgoubran/mydata:/data.

Additional build options

The interactive installation method described above should be sufficient 90% of users. However, if you need more finegrained control or advanced options, use the flags described in this section.

To start, use ./install.sh -h to see all optional flags:

Usage: ./install.sh [-n service_name] [-i image_name] [-c container_name] [-t {auto, x.x.x}] [-g] [-e] [-v vol:vol] [-l] [-s] [-m] [-h]

  Automatically build MIRACL Docker image with pseudo host user

Options:

  -n, name of the Docker service (randomized default: 'miracl_mgrouban_hughes_11707')
  -i, specify image name (randomized default: 'miracl_mgrouban_hughes_11707_img')
  -c, specify container name (default: 'miracl')
  -t, set when using specific MIRACL tag/version. Use 'auto' to parse from 'miracl/version.txt' or specify version as floating point value in format 'x.x.x' (default: 'latest')
  -g, enable Nvidia GPU passthrough mode for Docker container which is required for some of MIRACL's scripts e.g. ACE segmentation (default: false)
  -e, disable mounting MIRACL's script directory into Docker container. Mounting is useful if you want host changes to propagate to the container directly (default: false; set flag to disable)
  -d, set shared memory (shm) size (e.g. '1024mb', '16gb' or '512gb') which is important for e.g ACE (default: int(MemTotal/1024)*0.85 of host machine)
  -v, mount volumes for MIRACL in docker-compose.yml, using a separate flag for each additional volume (format: '/path/on/host:/path/in/container'; default: none)
  -l, write logfile of build process to 'build.log' in MIRACL root directory (default: false)
  -s, print version of build script and exit
  -m, print version of MIRACL on current Git branch and exit
  -h, print this help menu and exit

Script version: 2.0.1-beta
MIRACL version: 2.4.2

Let’s have a closer look at the most important flags.

Image and container naming

By default, the installation script will choose names for the image and container randomly, using the following syntax:

<miracl>_<usernamehostuser>_<randomname>_<randomdigitsbetween0and99999>

The image name will also be appended with <_img> do distinguish it from the container name. We do this to avoid conflicts with previous or dangling installations of MIRACL. Is you are confident that you will not duplicate your image or container names, set the -i and -c flags to choose your preferred names:

$ ./install.sh -i <image_name> -c <container_name>

Example:

$ ./install.sh -i josmann/miracl_dev_img -c miracl_dev

GPU forwarding

If you want to add GPU (Nvidia/CUDA) support to your MIRACL container, you can do so by running the installation script with the -g flag:

$ ./install.sh -g

This is required for MIRACL modules like ACE.

Disable script directory mounting

Our installation script mounts the miracl folder from inside your cloned MIRACL Git folder by default. The miracl folder contains all of MIRACL's modules and workflow code. This behavior is useful when you want to make real-time changes to e.g. a module for fit your specific needs on the fly as the changes will persist across restarts. In case this behavior is not desired, e.g. when testing changes that you don’t want to be permanent, use the -e flag to disable automatic mounting.

Set shared memory size

It is not always desirable to let your host system share all of its available memory with the MIRACL container. By default, the memory that will be shared by MIRACL is calculated as follows:

int(MemTotal/1024)*0.85 of host machine

If you want to increase or limit the memory availabe to MIRACL, set it manually using the -d flag. Example sizes are 1024mb, 16gb, or 512gb.

Example:

$ ./install.sh -d 512gb

This will set the shared memory size to 512gb.

Mount additional volumes

Use the -v flag if you need to mount several volumes or files from your host system to your MIRACL Docker container. Use a separate flag for each additional volume with this format: -v '/path/on/host:/path/in/container'. Make sure that you use a separate -v flag for each separate volume!

Example:

$ ./install -v /data5:/data5 -v /configfile:/configfile -v /data4:/additional_data

This will automatically add the volumes to your docker-compose.yml configuration under the volumes section where you can further edit or remove them as needed.

Hint

Combine any of the above flags if you want to use several of them together e.g.:

$ ./miracl -i josmann/miracl_dev_img -c miracl_dev -g -v /data:/data -v /config:/config

MIRACL versions

By default, Docker images will be built using the latest version of MIRACL. If you need to build a Docker image based on a specific version of MIRACL, do the following:

  1. Clone the MIRACL repository and navigate to the MIRACL folder:

$ git clone https://www.github.com/mgoubran/MIRACL
$ cd MIRACL

  1. Cloning the repository will download all tags/versions. List them with:

$ git tag -l

Example output:

v1.1.1
v2.2.1
v2.2.2
v2.2.3
v2.2.4
v2.2.5

  1. Decide which tag/version of MIRACL you want to use and check it out as a new branch:

$ git checkout tags/<tag_name> -b <branch_name>

Example:

$ git checkout tags/v2.2.4 -b miracl_v2.2.4

  1. If you are reverting to a version of MIRACL >= 2.2.4, you can build the image for your chosen version by running the installation script with the -t flag:

$ ./install.sh -t

Note

If you want to build an image for a version of MIRACL <= 2.2.4 either follow the build instructions of the particular version or download the latest installation script using e.g. wget https://raw.githubusercontent.com/AICONSlab/MIRACL/master/install.sh (overwrites current installation script if present) and run it with the -t flag.

  1. From here you can follow our instructions for building MIRACL from scratch starting with docker compose up -d. Our script will automatically detect the version of the branch you checked out and tag the image accordingly.