unipept-database

The Unipept metaproteomics and metagenomics pipelines operate on data found in the UniProtKB database. This collection of scripts is provided to download and process that database (or others) into range of tsv-files that can directly be used by the Unipept pipeline.

The image

ghcr.io/unipept/database-tools:<version> holds the data tools in one image: pipelines/, the Rust programs they run, sa-builder from unipept-index, umgap, and opensearch/, the loader. Dockerfile builds it; a version tag on main (v2.0.0, v2.0.0-rc.1, …) publishes it through .github/workflows/image.yml, which also builds and checks it on every pull request that changes it. Push the tag only (git tag v2.0.0 && git push origin v2.0.0), without a GitHub release: unipept-desktop reads the release marked Latest, which static_database.yml sets on the monthly static database. A release for a version tag, if there is one, needs --latest=false (or Set as the latest release unticked). Mount the host's database directory at the same path, and run as the user who owns it: the image's default user is root, and a build as root leaves a database that user cannot replace.

docker run --rm --user "$(id -u):$(id -g)" --network host -v /mnt/data:/mnt/data \
  ghcr.io/unipept/database-tools:<version> \
  opensearch/load.sh --opensearch-url http://localhost:9200 --index-name uniprot_entries-2026-03 \
  --uniprot-entries /mnt/data/uniprot-2026-03/tables/uniprot_entries.tsv.lz4

With PREBUILT_BINARIES=true, as the image sets, the pipelines use the programs in target/release instead of building them with cargo.

Most important scripts

pipelines/suffix-array/build.sh

This script is responsible for parsing the UniProtKB database and generating several .tsv.lz4 files that are essential for the Unipept pipeline that generates a suffix array (see unipept-index). These files contain structured information extracted from the database in a compressed format for efficient storage and processing. Below is an overview of the generated files and what they represent:

• taxons.tsv.lz4: This file contains data on NCBI taxonomic identifiers and their corresponding scientific names, providing information about the classification of organisms.
• lineages.tsv.lz4: A detailed representation of taxonomic lineages (according to the NCBI taxonomy), mapping organisms to their hierarchical classification (e.g., kingdom, phylum, class, etc.).
• go_terms.tsv.lz4: This file contains Gene Ontology (GO) terms mapped to their full name and namespace.
• ec_numbers.tsv.lz4: This file contains Enzyme Commission (EC) numbers, mapped to their full name and namespace.
• interpro_entries.tsv.lz4: This file lists InterPro entries mapped to their full name and namespace.
• uniprot_entries.tsv.lz4: This file contains protein sequences, together with their UniProtKB accession number, as well as the associated NCBI taxon IDs, and functional annotations (GO, EC and InterPro).
• .version: Contains a reference to the version number of the UniProtKB database that was used as input to this script.

See our wiki for more information on how to run this script.

pipelines/suffix-array/build-index.sh

This script builds the suffix array the API searches from uniprot_entries.tsv.lz4. It writes sa.bin, proteins.bin, mapping.bin and kmer_table.bin to the output directory, and needs sa-builder from unipept-index on PATH, as in the image:

pipelines/suffix-array/build-index.sh --uniprot-entries /mnt/data/build/tables/uniprot_entries.tsv.lz4 --output-dir /mnt/data/build/suffix-array \
  --sparseness-factor 2 --kmer-size 5 --construction-algorithm lib-sais

pipelines/suffix-array/build-database.sh

This script runs the steps above and builds a whole database in <output-dir>/uniprot-<YYYY-MM>: suffix-array/ is the directory the API reads, and tables/ holds the entries for OpenSearch. The version comes from the UniProtKB release it downloads. It stops if a database of that version already exists. It loads nothing into OpenSearch.

docker run --rm --user "$(id -u):$(id -g)" -v /mnt/data:/mnt/data ghcr.io/unipept/database-tools:<version> \
  pipelines/suffix-array/build-database.sh --output-dir /mnt/data \
  --sparseness-factor 2 --kmer-size 5 --construction-algorithm lib-sais

pipelines/umgap/build.sh

This script is responsible for parsing datasets to generate files required by the Unipept metagenomics pipeline, which constructs a metagenomics index. The script supports two distinct modes—kmer and tryptic—to create two different types of metagenomics indices optimized for different analytical purposes.

Below is an overview of the two supported modes and the files generated:

Kmer mode:
This mode generates a metagenomics index based on k-mers (short nucleotide or peptide sequences of fixed length). This approach is highly efficient for large-scale metagenomics data and allows for rapid matching and analysis of unassembled reads within metagenomic datasets.

Tryptic mode:
This mode constructs the index based on tryptic peptide sequences. Tryptic peptides result from in silico digestion of protein sequences using trypsin digestion rules. This method is well-suited for metaproteomics workflows where peptide-level functional and taxonomic analyses are required.

Both modes output data in a binary .index file to ensure efficient storage and processing that can directly be used by the UMGAP pipeline.

See our wiki for more information on how to run this script.