sanger-tol/nf-core-modules

An nf-core modules repository hosting Nextflow DSL2 modules for the Sanger Tree of Life organization.

Modules

The module files hosted in this repository define a set of processes for software tools that allow you to share and add common functionality across multiple pipelines in a modular fashion.

We use a helper command in the nf-core/tools package that uses the GitHub API to obtain the relevant information for the module files present in the modules/ directory of this repository. This includes using git commit hashes to track changes for reproducibility purposes, and to download and install all of the relevant module files.

Install the latest version of nf-core/tools version 3.4 or later. Version 3.3 and earlier do not support sub-workflows recorded in this repository.

List the available modules:

nf-core modules --git-remote https://github.com/sanger-tol/nf-core-modules.git list remote

                                          ,--./,-.
         ___     __   __   __   ___      /,-._.--~\
   |\ | |__  __ /  ` /  \ |__) |__          }  {
   | \| |       \__, \__/ |  \ |___      \`-._,-`-,
                                          `._,._,'

   nf-core/tools version 3.4.1 - https://nf-co.re

INFO     Modules available from https://github.com/sanger-tol/nf-core-modules.git (main):

┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Module Name                ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ ancestral/extract          │
│ ancestral/plot             │
│ asmstats                   │
│ bedtools/bamtobedsort      │
│ blobtoolkit/generatecsv    │
..truncated..

Install the module in your pipeline directory:

nf-core modules --git-remote https://github.com/sanger-tol/nf-core-modules.git install asmstats

                                          ,--./,-.
         ___     __   __   __   ___      /,-._.--~\
   |\ | |__  __ /  ` /  \ |__) |__          }  {
   | \| |       \__, \__/ |  \ |___      \`-._,-`-,
                                          `._,._,'

   nf-core/tools version 3.4.1 - https://nf-co.re

INFO     Installing 'asmstats'
INFO     Use the following statement to include this module:

 include { ASMSTATS } from '../modules/sanger-tol/asmstats/main'

Use the include statement as is in your workflow file workflows/name.nf. If you want to add the module do a sub-workflow such as subworkflows/local/name/main.nf, you will need to adjust the path accordingly:
```
include { ASMSTATS } from '../../../modules/sanger-tol/asmstats/main'
```

Check that a locally installed sanger-tol module is up-to-date compared to the one hosted in this repo:

nf-core modules --git-remote https://github.com/sanger-tol/nf-core-modules.git lint asmstats

                                          ,--./,-.
         ___     __   __   __   ___      /,-._.--~\
   |\ | |__  __ /  ` /  \ |__) |__          }  {
   | \| |       \__, \__/ |  \ |___      \`-._,-`-,
                                          `._,._,'

   nf-core/tools version 3.4.1 - https://nf-co.re

   INFO     Linting pipeline: '.'
   INFO     Linting module: 'asmstats'

╭─ [!] 2 Module Test Warnings ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│              ╷                                     ╷                                                                                                                                         │
│ Module name  │ File path                           │ Test message                                                                                                                            │
│╶─────────────┼─────────────────────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╴│
│ asmstats     │ modules/sanger-tol/asmstats/main.nf │ Unable to connect to container registry, code:  404, url: https://community.wave.seqera.io/library/seqtk_perl:37201934bb74266e          │
│ asmstats     │ modules/sanger-tol/asmstats/main.nf │ Container versions do not match                                                                                                         │
│              ╵                                     ╵                                                                                                                                         │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭───────────────────────╮
│ LINT RESULTS SUMMARY  │
├───────────────────────┤
│ [✔]  34 Tests Passed  │
│ [!]   2 Test Warnings │
│ [✗]   0 Tests Failed  │
╰───────────────────────╯

Remove the module from the pipeline repository if required:

nf-core modules --git-remote https://github.com/sanger-tol/nf-core-modules.git remove asmstats

                                          ,--./,-.
         ___     __   __   __   ___      /,-._.--~\
   |\ | |__  __ /  ` /  \ |__) |__          }  {
   | \| |       \__, \__/ |  \ |___      \`-._,-`-,
                                          `._,._,'

   nf-core/tools version 3.4.1 - https://nf-co.re

INFO     Removed files for 'asmstats' and its dependencies 'asmstats'.

Sub-workflows

The sub-workflow files hosted in this repository define arrangements of existing software tools (modules) that are frequently seen across pipelines. Like modules, sub-workflows are managed with the nf-core/tools package and allow you to share and add common functionality across multiple pipelines in a modular fashion.

Sub-workflows are stored in the subworkflows/ directory of this repository.

List the available sub-workflows:

nf-core subworkflows --git-remote https://github.com/sanger-tol/nf-core-modules.git list remote

                                          ,--./,-.
         ___     __   __   __   ___      /,-._.--~\
   |\ | |__  __ /  ` /  \ |__) |__          }  {
   | \| |       \__, \__/ |  \ |___      \`-._,-`-,
                                          `._,._,'

   nf-core/tools version 3.4.1 - https://nf-co.re

INFO     Subworkflows available from https://github.com/sanger-tol/nf-core-modules.git (main):

┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Subworkflow Name               ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ ancestral_annotation           │
│ bam_samtools_merge_markdup     │
│ cram_map_illumina_hic          │
..truncated..

Install the sub-workflow in your pipeline directory:

nf-core subworkflows --git-remote https://github.com/sanger-tol/nf-core-modules.git install bam_samtools_merge_markdup

                                          ,--./,-.
         ___     __   __   __   ___      /,-._.--~\
   |\ | |__  __ /  ` /  \ |__) |__          }  {
   | \| |       \__, \__/ |  \ |___      \`-._,-`-,
                                          `._,._,'

   nf-core/tools version 3.4.1 - https://nf-co.re

INFO     Installing 'bam_samtools_merge_markdup'
INFO     Use the following statement to include this subworkflow:

 include { BAM_SAMTOOLS_MERGE_MARKDUP } from '../subworkflows/sanger-tol/bam_samtools_merge_markdup/main'

This will automatically install module (and sub-workflow) dependencies:

$ ls modules/nf-core/samtools/
faidx  markdup  merge

Import the sub-workflow in your workflow (expected to live in workflows/):

include { BAM_SAMTOOLS_MERGE_MARKDUP } from '../subworkflows/sanger-tol/bam_samtools_merge_markdup/main'

Check that a locally installed sanger-tol sub-workflow is up-to-date compared to the one hosted in this repo:

nf-core subworkflows --git-remote https://github.com/sanger-tol/nf-core-modules.git lint bam_samtools_merge_markdup

                                          ,--./,-.
         ___     __   __   __   ___      /,-._.--~\
   |\ | |__  __ /  ` /  \ |__) |__          }  {
   | \| |       \__, \__/ |  \ |___      \`-._,-`-,
                                          `._,._,'

   nf-core/tools version 3.4.1 - https://nf-co.re

   INFO     Linting pipeline: '.'
   INFO     Linting subworkflow: 'bam_samtools_merge_markdup'

╭───────────────────────╮
│ LINT RESULTS SUMMARY  │
├───────────────────────┤
│ [✔]  26 Tests Passed  │
│ [!]   0 Test Warnings │
│ [✗]   0 Tests Failed  │
╰───────────────────────╯

Remove the sub-workflow from the pipeline repository if required:

nf-core subworkflows --git-remote https://github.com/sanger-tol/nf-core-modules.git remove bam_samtools_merge_markdup

                                          ,--./,-.
         ___     __   __   __   ___      /,-._.--~\
   |\ | |__  __ /  ` /  \ |__) |__          }  {
   | \| |       \__, \__/ |  \ |___      \`-._,-`-,
                                          `._,._,'

   nf-core/tools version 3.4.1 - https://nf-co.re

INFO     Removed files for 'samtools/faidx' and its dependencies 'samtools/faidx'.
INFO     Removed files for 'samtools/markdup' and its dependencies 'samtools/markdup'.
INFO     Removed files for 'samtools/merge' and its dependencies 'samtools/merge'.
INFO     Removed files for 'bam_samtools_merge_markdup' and its dependencies 'bam_samtools_merge_markdup, samtools_faidx, samtools_markdup, samtools_merge'.

Cross-organisation sub-workflows

"Cross-organisation" sub-workflows are sub-workflows that contain components from both nf-core/modules and sanger-tol/nf-core-modules. They require the version 3.4 (or later) of the nf-core/tools package.

Writing cross-organisation sub-workflows

A reference example exists in the nf-core test repository https://github.com/nf-core-test/modules.

Write sub-workflows .nf files that refer to locations in both sanger-tol and nf-core. Example
In meta.yml:
1. Change the first line to
```
# yaml-language-server: $schema=https://raw.githubusercontent.com/sanger-tol/nf-core-modules/main/subworkflows/yaml-schema.json
```
  Example. This ensures that the right schema will be used to validate the file. This schema differs from the default one by allowing keys such as git_remote under "components", which are used to indicate modules that live in the nf-core/modules repository, see next point.
2. Add a git_remote key that maps to the nf-core modules repository.
In modules/, do not add nf-core modules. When installing a sub-workflow, the nf-core tools command will identify the nf-core modules from the git_remote key explained above, and install those modules automatically.

Testing cross-organisation sub-workflows

Tests for a cross-organisation sub-workflow also need a copy of the nf-core modules to run. We use functions from the nft-utils plugin (version 0.0.7 or later), which is declared as a test dependency in nf-test.config. The functions will automatically download (and clean up) nf-core modules when running tests. These functions must be called from the sub-workflow's tests (e.g. the main.nf.test file). Take the hic_mapping sub-workflow as an example.

In the setup phase:

Call nfcoreInitialise to initialise a new "library" directory.
Call nfcoreInstall to install all the nf-core modules you need in that library. You need to keep this list in sync with the modules declared in meta.yml.
Call nfcoreLink to link the nf-core modules from the above "library" into the test's "modules" directory.

And in the cleanup phase:

Call nfcoreUnlink.

(and that's all !)

Using cross-organisation sub-workflows in pipelines

Pipelines need a few modifications to work seamlessly with cross-organisation sub-workflows.

In .pre-commit-config.yaml, add extra lines to ignore sanger-tol modules and sub-workflows the same way nf-core ones are ignored:

--- a/.pre-commit-config.yaml
+++ b/.pre-commit-config.yaml
@@ -15,6 +15,8 @@ repos:
               .*ro-crate-metadata.json$|
               modules/nf-core/.*|
               subworkflows/nf-core/.*|
+              modules/sanger-tol/.*|
+              subworkflows/sanger-tol/.*|
               .*\.snap$
         )$
      - id: end-of-file-fixer
@@ -23,5 +25,7 @@ repos:
               .*ro-crate-metadata.json$|
               modules/nf-core/.*|
               subworkflows/nf-core/.*|
+              modules/sanger-tol/.*|
+              subworkflows/sanger-tol/.*|
               .*\.snap$
         )$

nf-test.config needs similar rules:

--- a/nf-test.config
+++ b/nf-test.config
@@ -9,7 +9,7 @@ config {
   configFile "tests/nextflow.config"

   // ignore tests coming from the nf-core/modules repo
-    ignore 'modules/nf-core/**/*', 'subworkflows/nf-core/**/*'
+    ignore 'modules/nf-core/**/*', 'subworkflows/nf-core/**/*', 'modules/sanger-tol/**/*', 'subworkflows/sanger-tol/**/*'

   // run all test with defined profile(s) from the main nextflow.config
   profile "test"

It also needs to refers to the version 0.0.7 or later of the nft-utils plugin (load "nft-utils@ line).

So does .gitattributes (to automatically collapse files changed in the sanger-tol components)

--- a/.gitattributes
+++ b/.gitattributes
@@ -2,3 +2,5 @@
*.nf.test linguist-language=nextflow
modules/nf-core/** linguist-generated
subworkflows/nf-core/** linguist-generated
+modules/sanger-tol/** linguist-generated
+subworkflows/sanger-tol/** linguist-generated

Maintenance

We aim to stay in sync with the upstream nf-core/modules repository in terms of guidelines, schemas, CI, etc. This is done by regularly comparing the core files found in both repositories.

For those who know how to exit vim, here is a for loop that runs vimdiff over all files

cd /path/to/nf-core/modules
# Flag missing files
for i in $(git ls-files | grep -v '^modules/nf-core/' | grep -v '^subworkflows/nf-core/'); do [[ -e /path/to/sanger-tol/modules/$i ]] || echo "$i needs to be copied to sanger-tol/modules"; done

# Modified files
for i in $(git ls-files | grep -v '^modules/nf-core/' | grep -v '^subworkflows/nf-core/'); do [[ -e /path/to/sanger-tol/modules/$i ]] && vimdiff <(sed 's|nf-core/modules|sanger-tol/nf-core-modules|g' $i) /path/to/sanger-tol/modules/$i; done

# Removed files
for i in $(git ls-files | grep -v '^modules/nf-core/' | grep -v '^subworkflows/nf-core/'); do [[ -e /path/to/sanger-tol/modules/$i ]] || echo "$i needs to be copied to sanger-tol/modules"; done

Differences we want to keep:

we use main, not master
we have our own token name
we only use the "ubuntu-latest" runners

We need to preserve these differences:

Our repository is sanger-tol/nf-core-modules, not nf-core/modules.
We need to install nf-core/tools in nf-test-action to support the installation of dependent modules. We then need an extra checkout of nf-core/modules and link it across.
Our main branch is main, not master.
We only use the ubuntu-latest runner.
We have a different copyright statement.
I haven't introduced = in nf-test configs because I don't think it is necessary.
The sub-workflow schema needs to allow for object dependencies. That's what allows defining the git remote.
We have a different base path for our test data.

And consider those when importing new code and configuration !

Citation

If you use the module files in this repository for your analysis please you can cite the nf-core publication as follows:

The nf-core framework for community-curated bioinformatics pipelines.

Philip Ewels, Alexander Peltzer, Sven Fillinger, Harshil Patel, Johannes Alneberg, Andreas Wilm, Maxime Ulysse Garcia, Paolo Di Tommaso & Sven Nahnsen.

Nat Biotechnol. 2020 Feb 13. doi: 10.1038/s41587-020-0439-x.

Template

This module library was created using nf-core/modules-template with this command:

copier copy --vcs-ref main gh:nf-core/modules-template ./sanger-tol-modules

You can fetch updates to the library template by running:

pipx install copier
copier update

Name		Name	Last commit message	Last commit date
.devcontainer		.devcontainer
.github		.github
.vscode		.vscode
docs/images		docs/images
modules		modules
scripts		scripts
subworkflows		subworkflows
tests/config		tests/config
.copier-answers.yml		.copier-answers.yml
.gitattributes		.gitattributes
.gitignore		.gitignore
.gitpod.yml		.gitpod.yml
.nf-core.yml		.nf-core.yml
.pre-commit-config.yaml		.pre-commit-config.yaml
.prettierignore		.prettierignore
.prettierrc.yml		.prettierrc.yml
.project_init.sh		.project_init.sh
AGENTS.md		AGENTS.md
CITATION.cff		CITATION.cff
LICENSE		LICENSE
README.md		README.md
nf-test.config		nf-test.config
ruff.toml		ruff.toml

Navigation Menu

Repository navigation

About

Repository files navigation

sanger-tol/nf-core-modules

Table of contents

Modules

Sub-workflows

Cross-organisation sub-workflows

Writing cross-organisation sub-workflows

Testing cross-organisation sub-workflows

Using cross-organisation sub-workflows in pipelines

Maintenance

Citation

Template

About

Contributors

Languages

Latest commit

History

Folders and files

Repository files navigation

sanger-tol/nf-core-modules

Table of contents

Modules

Sub-workflows

Cross-organisation sub-workflows

Writing cross-organisation sub-workflows

Testing cross-organisation sub-workflows

Using cross-organisation sub-workflows in pipelines

Maintenance

Citation

Template

About

Topics

Resources

Code of conduct

Contributing

Stars

Watchers

Forks

Contributors

Languages