mpyworkflow

Provides subcommands for creating and managing simulation case repositories that are set up for batch-processing using a Snakemake.

Usage

mpyworkflow [options] <subcommand> [subcommand-options]

Subcommands

create

Creates a new cases project containing the configuration files for batch- processing via Snakemake. The Snakemake workflow definition is supplied as a copier template via the Git development repository for the multiphasepy Python package. The template is not packaged, so use of this command requires online access to the repository.

During creation, the user will be asked a series of questions to obtain a pre- configured workflow that meets specific needs.

Note that by default, the command will look for the template matching the installed version of multiphasepy. To obtain the latest definition, consider updating your multiphasepy installation.

The command

mpyworkflow [mpyworkflow-options] create [create-options] [destination]

will prompt questions which have to be answered to render the template

Should jobs run within an Apptainer/Singularity/Docker image?
   Yes
Which image should jobs run in?
(Use arrow keys)
   oras://<registry>/image:tag.sif
   oras://<registry>/differentImage:tag.sif
Do you intend to run the workflow on a Slurm cluster?
   Yes
What is the default partition on your cluster?
   fast-partition
What is the maximum walltime for the default partition in minutes?
   1234
...

Examples:

Creating a new cases project:

mpyworkflow create cases

Creating a new cases project using template definition from a branch, tag or commit:

mpyworkflow create --vcs-ref <branch>/<tag>/<commit> cases

update

Updates existing Snakemake configuration files for a given cases project, e.g. after updating the installation of multiphasepy.

The same options as for the mpyworkflow create command apply. Users will again be asked the same questions as during creation, but the answers given during creation are suggested as default.

Depending on whether the cases project is itself a Git repository or a plain unversioned directory, the command will behave differently. It is recommended to track cases projects via Git, because the update command will then automatically incorporate structural changes to the template, i.e. take care of moving and deleting files. If the cases project is unversioned, there may be left-over files from older workflow definitions.

Example:

Updating from within a cases project using the answers given during creation or last update:

mpyworkflow update --defaults .

collect - Create list of cases to process by scanning a directory

Collect all cases in a directory for batch processing by tools like Snakemake. This tool searches through a directory structure to find simulation cases, expands template cases to sub-cases, and stores the collection in a YAML file for further processing. The tool searches for cases from various CFD simulation packages, e.g., OpenFOAM Foundation software, Siemens STAR-CCM+, or Ansys Fluent.

The collection process:

  • Searches recursively for simulation cases

  • Expands templated cases into individual sub-cases

  • Generates a YAML file with all cases listed in the include found section

  • Supports filtering with regular expressions

  • Can stop on errors or continue searching

The command

mpyworkflow [mpyworkflow-options] collect [collect-options]

generates a YAML file (default is workflow.yml) with the following structure. The nested keys in the include section specify the path to the case. In the example below the cases are located in

  • cases/baseline/1986_Sun_and_Faeth/singlePhase,

  • cases/baseline/1986_Sun_and_Faeth/multiphase, and

  • cases/baseline/1987_Wang_et_al/dwHH.

include:
  cases:
    baseline:
      1986_Sun_and_Faeth:
        singlePhase:
          '0': true
        multiphase:
          '1': true
      1987_Wang_et_al:
        dwHH: true

If a workflow.yml file is already presented, the include dictionary will simply be added to it. The structure of the YAML file is such that it can be used for batch-processing in a Snakemake workflow.

Examples:

For collecting all cases in a directory:

mpyworkflow collect ./cases

Excluding directories matching a pattern:

mpyworkflow collect --ignore "backup|old" ./simulations

For processing only the first parameter set from templated cases:

mpyworkflow collect --test ./template-cases

junit-report

Creates a JUnit report from a Snakemake workflow. The format of the report is compatible with GitLab CI where it can be used as an artifact in order to display test results in the Merge Request details.

Running the command

mpyworkflow [mpyworkflow-options] junit-report .

at the top level of a workflow generates an XML file (default is report.xml).

A human readable report can be generated from that by

pip install junit2html
junit2html report.xml junit-report.html