mpydocker

Launch a container for a given Docker or Apptainer image. This tool provides a sophisticated wrapper around Docker and Apptainer to create user-friendly development environments with automatic configuration for common development needs.

The container setup:

  • Automatically creates user matching host UID/GID for seamless file permissions (docker container runtime)

  • Detects and mounts NVIDIA GPUs when available

  • Configures X11/XWayland/XQuartz display support

  • Forwards SSH authentication for git operations

  • Expands environment variables in volume paths

  • Supports Jupyter Lab server integration

  • Allows custom build instructions

Usage

mpydocker [options] image

Container Runtime

The tool supports two container runtimes, Docker and Apptainer. The latter support is yet experimental. To switch between the container runtimes, use the --engine option and select either docker or apptainer. In case the option is not explicitly set, docker will be used as default.

To run a docker container:

mpydocker --engine docker ubuntu:latest

or run a apptainer container:

mpydocker --engine apptainer docker://ubuntu:latest

Restart and Attach (Docker only)

It is possible to attach interactively to a running container. The default name of a container corresponds to the name of the parent directory. The command will check if it finds a docker container, which is already running, and will open a new bash inside this container. Note that opening a interactive bash session in a running container works not for remote hosts.

Create a named container that you can reattach to:

mpydocker --name dev-env myimage:latest

Later, restart and attach with the same command. In case you forgot to specify a name upon start of the container, use docker container ls to determine the name.

Running on Remote Hosts (Docker only)

It is possible launch a container on a remote host. The only requirement is that the remote host has to have docker installed and the user has to have permission to run docker containers. The remote host is specified with the --host option. The syntax is the same as for ssh, e.g., username@hostname:port. The remote host must be reachable via ssh without password, e.g., by using ssh keys. The remote host must also have access to the docker registry to pull the image.

Using a remote host requires to launch the container in detached mode. This is possible by using --extra-args '--detach'. Furthermore, you have to modify the command to be executed in the container to something that keeps the container alive, e.g., sleep infinity.

mpydocker --host <remote> --extra-args '--detach' ubuntu:latest sleep infinity

Customize your Docker Container

It is possible to specify a reduced set of docker build instructions to build a custom image, e.g., RUN or ENV. The image will get a new tag with the suffix ‘MPY’. In case a newer image is pulled from the registry, the custom image will be build automatically. The custom image is also used to add the host user as a new user to the docker image. The reason is to enable the dynamic code feature of the OpenFOAM Foundation software, which prevents running as root in the docker image. A new user and a new group will be created, with the corresponding user id and group id of the host system user. If the user id or group id is 1000 or less, the default user named ‘ubuntu’ will be used. The user will get sudo permissions.

For example to build a custom image, by adding x11-apps package to docker container for testing X11 capabilities:

mpydocker --custom-build "RUN sudo apt-get update" --custom-build "RUN sudo apt-get install -qy x11-apps" ubuntu:latest

Customize your Apptainer Container

It is also possible to install additional packages into the apptainer container upon launch. Therefore, bash instructions has to be provided, which will be injected into the %post section of an apptainer definition file

Make sure that you have setup the user namespace mapping correctly by checking etc/subuid and etc/subgid for an entry for your username, e.g., foo:100000:65536. Otherwise, the custom build will fail.

For example to build a custom image, by adding x11-apps package to apptainer container for testing X11 capabilities:

mpydocker --engine apptainer --custom-build "apt-get update" --custom-build "apt-get install -qy x11-apps" docker://ubuntu:latest

Variable Expansion

Variable Expansion on the Host

All variables specified as command line parameters are expanded on the host system. If variables are specified in the configuration file, variable expansion on the host system is enabled for the source definition in --volume , and for the --env option.

Variable Expansion in the Image

The --volume option enable variable expansion for the target definition by inspecting the image. When specifying this options as command line arguments, the dollar sign has to be escaped with a backslash, otherwise the variables are expanded on the host system. Alternatively, you can use single quotes.

For example to mount a directory as run directory for OpenFOAM Foundation software:

mpydocker --volume /path/to/cases:\${WM_PROJECT_USER_DIR}/run <image> # or
mpydocker --volume /path/to/cases:'${WM_PROJECT_USER_DIR}/run' <image>

or to mount the current directory to the site directory of OpenFOAM Foundation software:

mpydocker --volume $PWD:'${WM_PROJECT_SITE}/${WM_PROJECT_VERSION}' <image>

Jupyter Server (Docker only)

In case you need a jupyter server to run jupyter notebooks in the container, you have to configure mpydocker with the following config.json

{
  "image": "<image:tag>",
  "extra_args": [
    "--detach",
    "-p",
    "8888:8888"
  ],
  "custom_build": [
    "RUN sudo apt-get update && sudo apt-get install -y python3 python3-pip",
    "RUN sudo pip3 install --break-system-packages Jupyterlab pandas ipywidgets scikit-learn pulp"
  ],
  "cmd": "jupyter",
  "args": [
    "lab",
    "--ServerApp.port=8888",
    "--ServerApp.ip=0.0.0.0",
    "--ServerApp.token=''",
    "--ServerApp.password=''"
  ]
}

The jupyter lab server is listening on port 8888. To access the jupyter server use Visual Studio Code or a web browser with the following url: http://127.0.0.1:8888/lab. If the server is running remotely, use the ip address or hostname of the remote machine instead: http://<ip-address>:8888/lab. The container will be started as ‘detached’ in the background. Use docker container ls to obtain the container id. Then you can use docker container stop <container id> to stop the container, or docker exec -it <container id> to attach to the server.

If you install new packages via pip install --break-system-packages <package> you have to restart the jupyter kernel afterwards.

Run a container with jupyter lab server, access at http://127.0.0.1:8888/lab:

mpydocker --config config.json --name server <image>

Display Support

Display support works on Linux systems with X11 and XWayland. On Mac OS enable in settings for XQuartz ‘Allow connections from network clients’. If a Nvidia runtime environment is available, the tool mounts mount the graphic cards (GPU) to the container automatically at runtime. If no Nvidia support is found, only the default display device is added. Note, AMD graphics cards are not supported at the moment.

For NVIDIA GPUs the propritery NVIDA driver has to be installed. To this end follow the NVIDIA installation instructions and make sure that the driver is installed properly. This can be done by running nvidia-smi on the host system.

Once, the driver is installed the NVIDIA Container Toolkit has to be installed too. To verify that the toolkit works correctly run

sudo docker run --gpus=all --runtime=nvidia --rm nvcr.io/nvidia/cuda:11.6.2-base-ubuntu20.04 nvidia-smi

Note that an update of the NVIDIA driver can cause a update or re-installation of the NVIDIA Container Toolkit too. Always verify that nvidia-smi works on the host system as well as in a container.

Configuration File

An example configuration file can be produces by mpydocker --print-config. For convenient usage create ~/.config/mpydocker/config.json:

{
  "image": "myregistry/myimage:latest",
  "volume": [
    "~/cases:${WM_PROJECT_USER_DIR}/run",
    "~/data:/data"
  ],
  "custom_build": [
    "RUN sudo apt-get update",
    "RUN sudo apt-get install -y vim"
  ]
}

Then simply run:

mpydocker