How to increase parallelism of minimap2_wnode_fasta in EGAPx? #263

Activity

1. etvedte commented on Aug 10, 2026

   

   etvedte

   on Aug 10, 2026

   Contributor

   More actions

   Can you provide some more info?

   On #261, you mentioned you have a large volume of RNA-seq, which we recommended you sample down quite a bit to get a handle on your HPC configuration. How many short read/long read runs are you using? When you say "The pipeline works normally," have you completed any EGAPx runs with your downsampled data?

   What other tasks are running concurrently with minimap2_wnode_fasta? The default minimap2 parameters are being set here which means more than two processes can be running at once, but it's possible there are no available hosts depending on what else is going on concurrently. You gave the example of bam_bin running many jobs concurrently, and since that is a task that runs on a single cpu the slurm scheduler might be prioritizing those jobs.

   If you are willing to share some of your configuration files (e.g. your process_resources.config and slurm.config files) and share some of the details of the processes that are going on concurrently that might help shed light on the problem.
2. zhoudreames commented on Aug 11, 2026

   

   zhoudreames

   on Aug 11, 2026

   Author

   More actions

   Can you provide some more info?

   On #261, you mentioned you have a large volume of RNA-seq, which we recommended you sample down quite a bit to get a handle on your HPC configuration. How many short read/long read runs are you using? When you say "The pipeline works normally," have you completed any EGAPx runs with your downsampled data?

   What other tasks are running concurrently with minimap2_wnode_fasta? The default minimap2 parameters are being set here which means more than two processes can be running at once, but it's possible there are no available hosts depending on what else is going on concurrently. You gave the example of bam_bin running many jobs concurrently, and since that is a task that runs on a single cpu the slurm scheduler might be prioritizing those jobs.

   If you are willing to share some of your configuration files (e.g. your process_resources.config and slurm.config files) and share some of the details of the processes that are going on concurrently that might help shed light on the problem.

   Thank you for your response. Here are some additional details:

   • I am using 140 long-read runs, each downsampled to 3 million reads.
   • I am using 200 short-read RNA-seq runs, each downsampled to 10 million read pairs.
   • I have successfully completed an EGAPx test run using a single chromosome with the downsampled data.

   My main goal is to increase the parallelism of minimap2_wnode_fasta, since my HPC cluster has sufficient available resources. Currently, this step uses less than 30% of the cluster’s available capacity.

   Interestingly, other EGAPx processes, such as bin_and_sort:merge and bam2asn:convert, can gradually submit up to approximately 100 concurrent jobs. In contrast, minimap2_wnode_fasta remains limited to around 16 concurrently running jobs. This also occurs after all other tasks have finished, when minimap2_wnode_fasta is the only active process. Therefore, resource competition from other EGAPx tasks does not appear to explain the limitation.

   For example:

   [2f/4869af] ega…_bin_and_sort:merge (1260) | 1260 of 1260, cached: 1260 ✔
   [08/1b852a] ega…ane:bam2asn:convert (1106) | 1260 of 1260, cached: 1259 ✔
   [de/83cd8b] ega…seq_collapse:generate_jobs | 1 of 1 ✔
   [57/ea0b4e] ega…se:run_rnaseq_collapse (1) | 10 of 11, failed: 1, retries: 1
   [2e/db10f9] ega…2_index:run_minimap2_index | 1 of 1, cached: 1 ✔
   [7b/0e94e8] ega…ane:rename_fasta_ids (142) | 144 of 144, cached: 144 ✔
   [9c/c600fd] ega…ta:split_fasta_files (142) | 144 of 144, cached: 144 ✔
   [da/9b3604] ega…minimap2_wnode_fasta (345) | 335 of 2304
   

   My cluster has enough resources to run approximately 100 minimap2_wnode_fasta jobs concurrently, so I would like to understand what is limiting this process to around 16 jobs and which configuration parameter should be adjusted to increase its parallelism.

   I have attached my process_resources.config and slurm.config files for reference. Please let me know if any additional logs or configuration files would be helpful.

   slurm.cg.txt

   process_resources.cg.txt
3. etvedte commented on Aug 11, 2026

   

   etvedte

   on Aug 11, 2026

   Last edited by etvedte

   Contributor

   More actions

   @zhoudreames - edited response below

   minimap2_wnode_fasta currently has a hard cap of 16 concurrent processes defined by maxForks here. You can test increasing that with your resources configuration to increase cluster allocation.

   Based on #261 (particularly the node count) you could have:

   params.threads = 32
   params.nodes = 10
   params.num_cpus_per_node = 96
   

   How the jobs are split within/across nodes are  'hopes', not 'promises'. EGAPx has a deterministic way of splitting up jobs, but what nextflow will do with those jobs is seemingly not deterministic.