Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 8 additions & 6 deletions CLAUDE.md
Original file line number Diff line number Diff line change
@@ -1,28 +1,30 @@
# EpiModelHPC

## Overview
EpiModelHPC is an R package that extends the core [EpiModel](https://github.com/EpiModel/EpiModel) package for running stochastic network epidemic models on high-performance computing (HPC) systems. It provides parallelization, checkpointing, scenario batching, and integration with [slurmworkflow](https://github.com/EpiModel/slurmworkflow) for Slurm-based job scheduling.
EpiModelHPC is an R package that extends the core [EpiModel](https://github.com/EpiModel/EpiModel) package for running stochastic network epidemic models on high-performance computing (HPC) systems. It provides scenario batching, merging of simulation output, cluster presets, and step templates for [slurmworkflow](https://github.com/EpiModel/slurmworkflow), which handles Slurm-based job scheduling. Parallelism and checkpointing come from EpiModel itself (`ncores`, `.checkpoint.dir` and `.checkpoint.steps` in `control.net()`).

## Package Structure
- `R/` - 13 source files with core functionality
- `man/` - roxygen2-generated documentation (32 .Rd files)
- `R/` - 5 source files with core functionality
- `man/` - roxygen2-generated documentation
- `vignettes/` - detailed vignette on slurmworkflow integration
- `tests/` - testthat tests

## Key Dependencies
- **EpiModel** (>= 2.5.0) - core epidemic modeling framework
- **slurmworkflow** - Slurm workflow management (GitHub: EpiModel/slurmworkflow)
- **swfcalib** - calibration framework (GitHub: EpiModel/swfcalib)
- **doParallel / foreach** - parallel execution
- **future / future.apply** - scenario-level parallelization

## Key Functions
- `netsim_hpc()` - parallel netsim with checkpointing
- `netsim_scenarios()` / `step_tmpl_netsim_scenarios()` - scenario-based simulation (local / HPC)
- `merge_netsim_scenarios()` / `step_tmpl_merge_netsim_scenarios()` - merge batch results
- `merge_netsim_scenarios_tibble()` - convert results to tibble
- `swf_configs_hyak()` / `swf_configs_rsph()` - cluster presets
- `swf_configs_hyak()` / `swf_configs_rsph()` - cluster presets (`swf_configs_rsph()` knows the current RSPH partitions; `preemptable` no longer exists)
- `step_tmpl_renv_restore()` - renv setup on HPC
- `hpc_doctor_script()`, `add_doctor_register_step()`, `add_doctor_teardown_step()` - workflow monitoring helpers

## Removed in 2.8.0
The pre-`slurmworkflow` HPC path is gone, so do not suggest these functions: `netsim_hpc()` (use `EpiModel::netsim()` with `ncores` in `control.net()`), `check_cp()`, `initialize_cp()`, `save_cpdata()` (use EpiModel's `.checkpoint.dir` and `.checkpoint.steps`), `savesim()`, `merge_simfiles()`, `process_simfiles()` (use `merge_netsim_scenarios()` or `merge_netsim_scenarios_tibble()`), `sbatch_master()` (use the slurmworkflow step templates), and `pull_env_vars()` (handled by slurmworkflow). See NEWS.md.

## Build & Check
```bash
Expand Down
4 changes: 2 additions & 2 deletions DESCRIPTION
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
Package: EpiModelHPC
Version: 2.9.2
Date: 2026-07-28
Version: 2.9.3
Date: 2026-10-12
Title: EpiModel Extensions for High-Performance Computing
Description: Extension package to EpiModel to run large-scale stochastic network
models on modern high-performance computing systems. Provides scenario-based
Expand Down
18 changes: 18 additions & 0 deletions NEWS.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,21 @@
# EpiModelHPC 2.9.3

## BREAKING CHANGES

- `swf_configs_rsph()` targets the current RSPH partitions. `partition` defaults to `"epimodel"`, the lab's dedicated nodes, instead of `"preemptable"`, which no longer exists on the cluster and now gives an error saying so. It accepts `epimodel`, `CIDMATH`, `short-cpu`, `day-long-cpu`, `week-long-cpu`, `month-long-cpu`, `interactive-cpu` and `largemem`, as a vector or a comma-separated string, so a step can start on whichever listed partition has room first.
- `swf_configs_rsph()` defaults `r_version` to 4.6.1, the current spack build. Projects whose lockfile records R 4.5 pass `r_version = "4.5.1"`.

## NEW FEATURES

- `swf_configs_rsph()` returns `max_cores`, the largest `cpus-per-task` a single task can request on every listed partition: 32, or 16 when `interactive-cpu` is listed and 31 when `largemem` is.
- `swf_configs_rsph()` warns when `r_version` is not a full version. `spack load r@4.3` fails because it matches three builds.
- `?swf_configs_rsph` documents the time and core limits of each partition, the 5 GB per CPU memory default, and the array size ceiling.

## OTHER

- The restore step from `swf_configs_rsph()` gets 180 minutes instead of 120, as in the EpiModelHIV-Template. A first restore under a new R version builds every package from source.
- The `epimodelhiv-slurmworkflow` vignette is rewritten for the current EpiModelHIV-Template: `make_em_workflow()` in `R/hpc_configs.R`, `renv.lock.hpc` restored on compute nodes by the first step, `data/run/`, the current partitions and `max_array_size = 1000`. It no longer passes `save_pattern` (removed in 2.5.0) or reserves whole nodes for simulation steps, and it points to the Template chapter READMEs and the wiki for the step-by-step instructions.

# EpiModelHPC 2.9.2

## NEW FEATURES
Expand Down
144 changes: 135 additions & 9 deletions R/slurmworkflow_helpers.R
Original file line number Diff line number Diff line change
Expand Up @@ -69,43 +69,112 @@ swf_configs_hyak <- function(hpc = "klone", partition = "csde",

#' Preset of Configuration for the RSPH Cluster
#'
#' @param partition Which partition to use on RSPH (either "compute" or
#' "epimodel")
#' @param partition The RSPH partition, or partitions, to submit to: a single
#' name, a character vector, or a comma-separated string such as
#' `"epimodel,CIDMATH"`. With several partitions, SLURM starts each job on
#' whichever can run it first. The default, `"epimodel"`, is the lab's
#' dedicated partition. See the "Partitions" section for the valid names.
#' @param r_version Which version of R to load with spack. Use the full version
#' (`"4.6.1"` or `"4.5.1"`) matching the R version in the project lockfile. A
#' partial version such as `"4.3"` can match several spack builds, which makes
#' `spack load` fail, so it gives a warning.
#' @param cleanup_workers If \code{TRUE} (the default), append
#' \code{\link{swf_cleanup_r_workers}} to \code{r_loader} so each step reaps
#' its own R workers on cancel or timeout. RSPH has no cgroup containment, so
#' without this, workers survive \code{scancel} and squat on cores SLURM has
#' re-allocated, starving later tasks. See \code{\link{swf_cleanup_r_workers}}.
#'
#' @inherit swf_configs_hyak return
#' @inheritParams swf_configs_hyak
#' @inheritSection swf_configs_hyak hpc_configs
#'
#' @return a list with four elements:
#' * `default_sbatch_opts`: sbatch options to pass to
#' `slurmworkflow::create_workflow()`. They set the partition list,
#' `mail-type = "FAIL"` and, when `mail_user` is given, `mail-user`.
#' * `renv_sbatch_opts`: sbatch options for the [step_tmpl_renv_restore()] step:
#' 16 GB of memory, 4 CPUs and 180 minutes, as in the EpiModelHIV-Template.
#' The first restore under a new R version builds every package from source
#' and takes longer than later ones.
#' * `r_loader`: bash lines that make R available on a compute node. Pass them
#' to the `setup_lines` argument of the `step_tmpl_` functions.
#' * `max_cores`: the largest `cpus-per-task` a single task can request on
#' every listed partition. It is 32, except 16 when `interactive-cpu` is
#' listed and 31 when `largemem` is listed.
#'
#' @section Partitions:
#' The RSPH partitions as of October 2026. All nodes have 32 cores.
#'
#' | Partition | Time limit | Max cores per task | Notes |
#' |---|---|---|---|
#' | `epimodel` | none | 32 | The lab's dedicated nodes; the default |
#' | `CIDMATH` | none | 32 | Dedicated nodes shared with three other groups |
#' | `short-cpu` | 8 hours | 32 | The cluster's default partition |
#' | `day-long-cpu` | 1 day | 32 | |
#' | `week-long-cpu` | 7 days | 32 | At most 6 nodes per job |
#' | `month-long-cpu` | 31 days | 32 | At most 4 nodes per job |
#' | `interactive-cpu` | 12 hours | 16 | Meant for `srun` sessions |
#' | `largemem` | 7 days | 31 | High-memory nodes |
#'
#' The `preemptable` partition no longer exists, and asking for it is an error.
#'
#' `default_sbatch_opts` sets no `time`, so a step without its own `time` runs
#' up to the limit of the partition it lands on. A step whose `time` exceeds a
#' partition's limit can only start on the other partitions in the list.
#'
#' RSPH gives each CPU 5 GB of memory by default (`DefMemPerCPU = 5120`) and
#' does not enforce memory requests. A step that uses more than it asked for is
#' not killed, but it can starve other jobs on the node, so request what each
#' step needs with `mem-per-cpu`. Keep `mem = "0"` (a whole node) for network
#' estimation.
#'
#' SLURM's `MaxArraySize` is 1001 on RSPH, so one array holds at most 1001
#' tasks (indices 0 to 1000). We use `max_array_size = 1000` in array steps.
#'
#' @section Node setup:
#' \code{r_loader} sources the shared spack environment, unloads everything, and
#' loads R. It deliberately does not load git: git is on the default path on
#' RSPH, and the project template has not loaded it for some time. The
#' \code{git_version} argument was removed accordingly.
#'
#' @examples
#' hpc_configs <- swf_configs_rsph(mail_user = "user@emory.edu")
#' hpc_configs$default_sbatch_opts
#'
#' # Start each step on whichever partition can run it first
#' swf_configs_rsph(c("epimodel", "CIDMATH", "short-cpu"))$default_sbatch_opts
#'
#' # A project whose lockfile records R 4.5
#' swf_configs_rsph(r_version = "4.5.1", cleanup_workers = FALSE)$r_loader
#'
#' @export
swf_configs_rsph <- function(partition = "preemptable",
r_version = "4.5.1",
swf_configs_rsph <- function(partition = "epimodel",
r_version = "4.6.1",
mail_user = NULL,
cleanup_workers = TRUE) {

if (!partition %in% c("preemptable", "epimodel"))
stop("On RSPH, partition must be one of \"preemptable\" or \"epimodel\"")
partition <- rsph_check_partition(partition)

if (!grepl("^[0-9]+\\.[0-9]+\\.[0-9]+$", r_version[1])) {
warning(
"`r_version = \"", r_version[1], "\"` is not a full version. ",
"It can match several spack builds and make `spack load` fail. ",
"Use a full version such as \"4.6.1\".",
call. = FALSE
)
}

partitions <- rsph_partitions()

hpc_configs <- list()
hpc_configs[["default_sbatch_opts"]] <- list(
"partition" = partition,
"partition" = paste(partition, collapse = ","),
"mail-type" = "FAIL"
)

if (!is.null(mail_user))
hpc_configs[["default_sbatch_opts"]][["mail-user"]] <- mail_user

hpc_configs[["renv_sbatch_opts"]] <- swf_renv_sbatch_opts()
hpc_configs[["renv_sbatch_opts"]][["time"]] <- 180

hpc_configs[["r_loader"]] <- c(
". /projects/epimodel/spack/share/spack/setup-env.sh",
Expand All @@ -120,9 +189,66 @@ swf_configs_rsph <- function(partition = "preemptable",
)
}

hpc_configs[["max_cores"]] <- min(
partitions$max_cores[match(partition, partitions$partition)]
)

return(hpc_configs)
}

#' RSPH partitions and the most cores a single task can request on each
#'
#' Checked with `scontrol show partition` on 2026-10-12.
#' @noRd
rsph_partitions <- function() {
data.frame(
partition = c(
"epimodel", "CIDMATH", "short-cpu", "day-long-cpu", "week-long-cpu",
"month-long-cpu", "interactive-cpu", "largemem"
),
max_cores = c(32, 32, 32, 32, 32, 32, 16, 31),
stringsAsFactors = FALSE
)
}

#' Validate and normalize the `partition` argument of `swf_configs_rsph`
#'
#' Accepts a vector or a comma-separated string, matches names without regard
#' to case, and returns the canonical names without duplicates.
#' @noRd
rsph_check_partition <- function(partition) {
if (!is.character(partition) || anyNA(partition))
stop("`partition` must be a character vector of RSPH partition names")

requested <- trimws(unlist(strsplit(partition, ",", fixed = TRUE)))
requested <- requested[nzchar(requested)]
if (length(requested) == 0)
stop("`partition` must name at least one RSPH partition")

if ("preemptable" %in% tolower(requested)) {
stop(
"The \"preemptable\" partition no longer exists on RSPH. ",
"Use \"epimodel\" (the lab's dedicated nodes), or a list such as ",
"c(\"epimodel\", \"short-cpu\", \"day-long-cpu\") to run on whichever ",
"partition starts first.",
call. = FALSE
)
}

valid <- rsph_partitions()$partition
idx <- match(tolower(requested), tolower(valid))
if (anyNA(idx)) {
stop(
"Unknown RSPH partition: ",
paste0("\"", requested[is.na(idx)], "\"", collapse = ", "), ". ",
"Valid partitions are ", paste0("\"", valid, "\"", collapse = ", "), ".",
call. = FALSE
)
}

unique(valid[idx])
}

#' @noRd
swf_renv_sbatch_opts <- function() {
list(
Expand Down
85 changes: 66 additions & 19 deletions man/swf_configs_rsph.Rd

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading
Loading