diff --git a/CLAUDE.md b/CLAUDE.md index 5b61320..4adf47c 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -1,11 +1,11 @@ # EpiModelHPC ## Overview -EpiModelHPC is an R package that extends the core [EpiModel](https://github.com/EpiModel/EpiModel) package for running stochastic network epidemic models on high-performance computing (HPC) systems. It provides parallelization, checkpointing, scenario batching, and integration with [slurmworkflow](https://github.com/EpiModel/slurmworkflow) for Slurm-based job scheduling. +EpiModelHPC is an R package that extends the core [EpiModel](https://github.com/EpiModel/EpiModel) package for running stochastic network epidemic models on high-performance computing (HPC) systems. It provides scenario batching, merging of simulation output, cluster presets, and step templates for [slurmworkflow](https://github.com/EpiModel/slurmworkflow), which handles Slurm-based job scheduling. Parallelism and checkpointing come from EpiModel itself (`ncores`, `.checkpoint.dir` and `.checkpoint.steps` in `control.net()`). ## Package Structure -- `R/` - 13 source files with core functionality -- `man/` - roxygen2-generated documentation (32 .Rd files) +- `R/` - 5 source files with core functionality +- `man/` - roxygen2-generated documentation - `vignettes/` - detailed vignette on slurmworkflow integration - `tests/` - testthat tests @@ -13,16 +13,18 @@ EpiModelHPC is an R package that extends the core [EpiModel](https://github.com/ - **EpiModel** (>= 2.5.0) - core epidemic modeling framework - **slurmworkflow** - Slurm workflow management (GitHub: EpiModel/slurmworkflow) - **swfcalib** - calibration framework (GitHub: EpiModel/swfcalib) -- **doParallel / foreach** - parallel execution - **future / future.apply** - scenario-level parallelization ## Key Functions -- `netsim_hpc()` - parallel netsim with checkpointing - `netsim_scenarios()` / `step_tmpl_netsim_scenarios()` - scenario-based simulation (local / HPC) - `merge_netsim_scenarios()` / `step_tmpl_merge_netsim_scenarios()` - merge batch results - `merge_netsim_scenarios_tibble()` - convert results to tibble -- `swf_configs_hyak()` / `swf_configs_rsph()` - cluster presets +- `swf_configs_hyak()` / `swf_configs_rsph()` - cluster presets (`swf_configs_rsph()` knows the current RSPH partitions; `preemptable` no longer exists) - `step_tmpl_renv_restore()` - renv setup on HPC +- `hpc_doctor_script()`, `add_doctor_register_step()`, `add_doctor_teardown_step()` - workflow monitoring helpers + +## Removed in 2.8.0 +The pre-`slurmworkflow` HPC path is gone, so do not suggest these functions: `netsim_hpc()` (use `EpiModel::netsim()` with `ncores` in `control.net()`), `check_cp()`, `initialize_cp()`, `save_cpdata()` (use EpiModel's `.checkpoint.dir` and `.checkpoint.steps`), `savesim()`, `merge_simfiles()`, `process_simfiles()` (use `merge_netsim_scenarios()` or `merge_netsim_scenarios_tibble()`), `sbatch_master()` (use the slurmworkflow step templates), and `pull_env_vars()` (handled by slurmworkflow). See NEWS.md. ## Build & Check ```bash diff --git a/DESCRIPTION b/DESCRIPTION index 2f59500..d2aa5e8 100644 --- a/DESCRIPTION +++ b/DESCRIPTION @@ -1,6 +1,6 @@ Package: EpiModelHPC -Version: 2.9.2 -Date: 2026-07-28 +Version: 2.9.3 +Date: 2026-10-12 Title: EpiModel Extensions for High-Performance Computing Description: Extension package to EpiModel to run large-scale stochastic network models on modern high-performance computing systems. Provides scenario-based diff --git a/NEWS.md b/NEWS.md index 11bb9d3..02c7a60 100644 --- a/NEWS.md +++ b/NEWS.md @@ -1,3 +1,21 @@ +# EpiModelHPC 2.9.3 + +## BREAKING CHANGES + +- `swf_configs_rsph()` targets the current RSPH partitions. `partition` defaults to `"epimodel"`, the lab's dedicated nodes, instead of `"preemptable"`, which no longer exists on the cluster and now gives an error saying so. It accepts `epimodel`, `CIDMATH`, `short-cpu`, `day-long-cpu`, `week-long-cpu`, `month-long-cpu`, `interactive-cpu` and `largemem`, as a vector or a comma-separated string, so a step can start on whichever listed partition has room first. +- `swf_configs_rsph()` defaults `r_version` to 4.6.1, the current spack build. Projects whose lockfile records R 4.5 pass `r_version = "4.5.1"`. + +## NEW FEATURES + +- `swf_configs_rsph()` returns `max_cores`, the largest `cpus-per-task` a single task can request on every listed partition: 32, or 16 when `interactive-cpu` is listed and 31 when `largemem` is. +- `swf_configs_rsph()` warns when `r_version` is not a full version. `spack load r@4.3` fails because it matches three builds. +- `?swf_configs_rsph` documents the time and core limits of each partition, the 5 GB per CPU memory default, and the array size ceiling. + +## OTHER + +- The restore step from `swf_configs_rsph()` gets 180 minutes instead of 120, as in the EpiModelHIV-Template. A first restore under a new R version builds every package from source. +- The `epimodelhiv-slurmworkflow` vignette is rewritten for the current EpiModelHIV-Template: `make_em_workflow()` in `R/hpc_configs.R`, `renv.lock.hpc` restored on compute nodes by the first step, `data/run/`, the current partitions and `max_array_size = 1000`. It no longer passes `save_pattern` (removed in 2.5.0) or reserves whole nodes for simulation steps, and it points to the Template chapter READMEs and the wiki for the step-by-step instructions. + # EpiModelHPC 2.9.2 ## NEW FEATURES diff --git a/R/slurmworkflow_helpers.R b/R/slurmworkflow_helpers.R index 0c93df5..b68e18c 100644 --- a/R/slurmworkflow_helpers.R +++ b/R/slurmworkflow_helpers.R @@ -69,17 +69,65 @@ swf_configs_hyak <- function(hpc = "klone", partition = "csde", #' Preset of Configuration for the RSPH Cluster #' -#' @param partition Which partition to use on RSPH (either "compute" or -#' "epimodel") +#' @param partition The RSPH partition, or partitions, to submit to: a single +#' name, a character vector, or a comma-separated string such as +#' `"epimodel,CIDMATH"`. With several partitions, SLURM starts each job on +#' whichever can run it first. The default, `"epimodel"`, is the lab's +#' dedicated partition. See the "Partitions" section for the valid names. +#' @param r_version Which version of R to load with spack. Use the full version +#' (`"4.6.1"` or `"4.5.1"`) matching the R version in the project lockfile. A +#' partial version such as `"4.3"` can match several spack builds, which makes +#' `spack load` fail, so it gives a warning. #' @param cleanup_workers If \code{TRUE} (the default), append #' \code{\link{swf_cleanup_r_workers}} to \code{r_loader} so each step reaps #' its own R workers on cancel or timeout. RSPH has no cgroup containment, so #' without this, workers survive \code{scancel} and squat on cores SLURM has #' re-allocated, starving later tasks. See \code{\link{swf_cleanup_r_workers}}. #' -#' @inherit swf_configs_hyak return #' @inheritParams swf_configs_hyak -#' @inheritSection swf_configs_hyak hpc_configs +#' +#' @return a list with four elements: +#' * `default_sbatch_opts`: sbatch options to pass to +#' `slurmworkflow::create_workflow()`. They set the partition list, +#' `mail-type = "FAIL"` and, when `mail_user` is given, `mail-user`. +#' * `renv_sbatch_opts`: sbatch options for the [step_tmpl_renv_restore()] step: +#' 16 GB of memory, 4 CPUs and 180 minutes, as in the EpiModelHIV-Template. +#' The first restore under a new R version builds every package from source +#' and takes longer than later ones. +#' * `r_loader`: bash lines that make R available on a compute node. Pass them +#' to the `setup_lines` argument of the `step_tmpl_` functions. +#' * `max_cores`: the largest `cpus-per-task` a single task can request on +#' every listed partition. It is 32, except 16 when `interactive-cpu` is +#' listed and 31 when `largemem` is listed. +#' +#' @section Partitions: +#' The RSPH partitions as of October 2026. All nodes have 32 cores. +#' +#' | Partition | Time limit | Max cores per task | Notes | +#' |---|---|---|---| +#' | `epimodel` | none | 32 | The lab's dedicated nodes; the default | +#' | `CIDMATH` | none | 32 | Dedicated nodes shared with three other groups | +#' | `short-cpu` | 8 hours | 32 | The cluster's default partition | +#' | `day-long-cpu` | 1 day | 32 | | +#' | `week-long-cpu` | 7 days | 32 | At most 6 nodes per job | +#' | `month-long-cpu` | 31 days | 32 | At most 4 nodes per job | +#' | `interactive-cpu` | 12 hours | 16 | Meant for `srun` sessions | +#' | `largemem` | 7 days | 31 | High-memory nodes | +#' +#' The `preemptable` partition no longer exists, and asking for it is an error. +#' +#' `default_sbatch_opts` sets no `time`, so a step without its own `time` runs +#' up to the limit of the partition it lands on. A step whose `time` exceeds a +#' partition's limit can only start on the other partitions in the list. +#' +#' RSPH gives each CPU 5 GB of memory by default (`DefMemPerCPU = 5120`) and +#' does not enforce memory requests. A step that uses more than it asked for is +#' not killed, but it can starve other jobs on the node, so request what each +#' step needs with `mem-per-cpu`. Keep `mem = "0"` (a whole node) for network +#' estimation. +#' +#' SLURM's `MaxArraySize` is 1001 on RSPH, so one array holds at most 1001 +#' tasks (indices 0 to 1000). We use `max_array_size = 1000` in array steps. #' #' @section Node setup: #' \code{r_loader} sources the shared spack environment, unloads everything, and @@ -87,18 +135,38 @@ swf_configs_hyak <- function(hpc = "klone", partition = "csde", #' RSPH, and the project template has not loaded it for some time. The #' \code{git_version} argument was removed accordingly. #' +#' @examples +#' hpc_configs <- swf_configs_rsph(mail_user = "user@emory.edu") +#' hpc_configs$default_sbatch_opts +#' +#' # Start each step on whichever partition can run it first +#' swf_configs_rsph(c("epimodel", "CIDMATH", "short-cpu"))$default_sbatch_opts +#' +#' # A project whose lockfile records R 4.5 +#' swf_configs_rsph(r_version = "4.5.1", cleanup_workers = FALSE)$r_loader +#' #' @export -swf_configs_rsph <- function(partition = "preemptable", - r_version = "4.5.1", +swf_configs_rsph <- function(partition = "epimodel", + r_version = "4.6.1", mail_user = NULL, cleanup_workers = TRUE) { - if (!partition %in% c("preemptable", "epimodel")) - stop("On RSPH, partition must be one of \"preemptable\" or \"epimodel\"") + partition <- rsph_check_partition(partition) + + if (!grepl("^[0-9]+\\.[0-9]+\\.[0-9]+$", r_version[1])) { + warning( + "`r_version = \"", r_version[1], "\"` is not a full version. ", + "It can match several spack builds and make `spack load` fail. ", + "Use a full version such as \"4.6.1\".", + call. = FALSE + ) + } + + partitions <- rsph_partitions() hpc_configs <- list() hpc_configs[["default_sbatch_opts"]] <- list( - "partition" = partition, + "partition" = paste(partition, collapse = ","), "mail-type" = "FAIL" ) @@ -106,6 +174,7 @@ swf_configs_rsph <- function(partition = "preemptable", hpc_configs[["default_sbatch_opts"]][["mail-user"]] <- mail_user hpc_configs[["renv_sbatch_opts"]] <- swf_renv_sbatch_opts() + hpc_configs[["renv_sbatch_opts"]][["time"]] <- 180 hpc_configs[["r_loader"]] <- c( ". /projects/epimodel/spack/share/spack/setup-env.sh", @@ -120,9 +189,66 @@ swf_configs_rsph <- function(partition = "preemptable", ) } + hpc_configs[["max_cores"]] <- min( + partitions$max_cores[match(partition, partitions$partition)] + ) + return(hpc_configs) } +#' RSPH partitions and the most cores a single task can request on each +#' +#' Checked with `scontrol show partition` on 2026-10-12. +#' @noRd +rsph_partitions <- function() { + data.frame( + partition = c( + "epimodel", "CIDMATH", "short-cpu", "day-long-cpu", "week-long-cpu", + "month-long-cpu", "interactive-cpu", "largemem" + ), + max_cores = c(32, 32, 32, 32, 32, 32, 16, 31), + stringsAsFactors = FALSE + ) +} + +#' Validate and normalize the `partition` argument of `swf_configs_rsph` +#' +#' Accepts a vector or a comma-separated string, matches names without regard +#' to case, and returns the canonical names without duplicates. +#' @noRd +rsph_check_partition <- function(partition) { + if (!is.character(partition) || anyNA(partition)) + stop("`partition` must be a character vector of RSPH partition names") + + requested <- trimws(unlist(strsplit(partition, ",", fixed = TRUE))) + requested <- requested[nzchar(requested)] + if (length(requested) == 0) + stop("`partition` must name at least one RSPH partition") + + if ("preemptable" %in% tolower(requested)) { + stop( + "The \"preemptable\" partition no longer exists on RSPH. ", + "Use \"epimodel\" (the lab's dedicated nodes), or a list such as ", + "c(\"epimodel\", \"short-cpu\", \"day-long-cpu\") to run on whichever ", + "partition starts first.", + call. = FALSE + ) + } + + valid <- rsph_partitions()$partition + idx <- match(tolower(requested), tolower(valid)) + if (anyNA(idx)) { + stop( + "Unknown RSPH partition: ", + paste0("\"", requested[is.na(idx)], "\"", collapse = ", "), ". ", + "Valid partitions are ", paste0("\"", valid, "\"", collapse = ", "), ".", + call. = FALSE + ) + } + + unique(valid[idx]) +} + #' @noRd swf_renv_sbatch_opts <- function() { list( diff --git a/man/swf_configs_rsph.Rd b/man/swf_configs_rsph.Rd index 924e514..87f91df 100644 --- a/man/swf_configs_rsph.Rd +++ b/man/swf_configs_rsph.Rd @@ -5,17 +5,23 @@ \title{Preset of Configuration for the RSPH Cluster} \usage{ swf_configs_rsph( - partition = "preemptable", - r_version = "4.5.1", + partition = "epimodel", + r_version = "4.6.1", mail_user = NULL, cleanup_workers = TRUE ) } \arguments{ -\item{partition}{Which partition to use on RSPH (either "compute" or -"epimodel")} +\item{partition}{The RSPH partition, or partitions, to submit to: a single +name, a character vector, or a comma-separated string such as +\code{"epimodel,CIDMATH"}. With several partitions, SLURM starts each job on +whichever can run it first. The default, \code{"epimodel"}, is the lab's +dedicated partition. See the "Partitions" section for the valid names.} -\item{r_version}{Which version of R to load} +\item{r_version}{Which version of R to load with spack. Use the full version +(\code{"4.6.1"} or \code{"4.5.1"}) matching the R version in the project lockfile. A +partial version such as \code{"4.3"} can match several spack builds, which makes +\verb{spack load} fail, so it gives a warning.} \item{mail_user}{The mail address to send the messages to, default is NULL (see 'sbatch --mail-type' argument)} @@ -27,12 +33,56 @@ without this, workers survive \code{scancel} and squat on cores SLURM has re-allocated, starving later tasks. See \code{\link{swf_cleanup_r_workers}}.} } \value{ -a list containing \code{default_sbatch_opts}, \code{renv_sbatch_opts} -and \code{r_loader} (see the "hpc_configs" section) +a list with four elements: +\itemize{ +\item \code{default_sbatch_opts}: sbatch options to pass to +\code{slurmworkflow::create_workflow()}. They set the partition list, +\code{mail-type = "FAIL"} and, when \code{mail_user} is given, \code{mail-user}. +\item \code{renv_sbatch_opts}: sbatch options for the \code{\link[=step_tmpl_renv_restore]{step_tmpl_renv_restore()}} step: +16 GB of memory, 4 CPUs and 180 minutes, as in the EpiModelHIV-Template. +The first restore under a new R version builds every package from source +and takes longer than later ones. +\item \code{r_loader}: bash lines that make R available on a compute node. Pass them +to the \code{setup_lines} argument of the \code{step_tmpl_} functions. +\item \code{max_cores}: the largest \code{cpus-per-task} a single task can request on +every listed partition. It is 32, except 16 when \code{interactive-cpu} is +listed and 31 when \code{largemem} is listed. +} } \description{ Preset of Configuration for the RSPH Cluster } +\section{Partitions}{ + +The RSPH partitions as of October 2026. All nodes have 32 cores.\tabular{llll}{ + Partition \tab Time limit \tab Max cores per task \tab Notes \cr + \code{epimodel} \tab none \tab 32 \tab The lab's dedicated nodes; the default \cr + \code{CIDMATH} \tab none \tab 32 \tab Dedicated nodes shared with three other groups \cr + \code{short-cpu} \tab 8 hours \tab 32 \tab The cluster's default partition \cr + \code{day-long-cpu} \tab 1 day \tab 32 \tab \cr + \code{week-long-cpu} \tab 7 days \tab 32 \tab At most 6 nodes per job \cr + \code{month-long-cpu} \tab 31 days \tab 32 \tab At most 4 nodes per job \cr + \code{interactive-cpu} \tab 12 hours \tab 16 \tab Meant for \code{srun} sessions \cr + \code{largemem} \tab 7 days \tab 31 \tab High-memory nodes \cr +} + + +The \code{preemptable} partition no longer exists, and asking for it is an error. + +\code{default_sbatch_opts} sets no \code{time}, so a step without its own \code{time} runs +up to the limit of the partition it lands on. A step whose \code{time} exceeds a +partition's limit can only start on the other partitions in the list. + +RSPH gives each CPU 5 GB of memory by default (\code{DefMemPerCPU = 5120}) and +does not enforce memory requests. A step that uses more than it asked for is +not killed, but it can starve other jobs on the node, so request what each +step needs with \code{mem-per-cpu}. Keep \code{mem = "0"} (a whole node) for network +estimation. + +SLURM's \code{MaxArraySize} is 1001 on RSPH, so one array holds at most 1001 +tasks (indices 0 to 1000). We use \code{max_array_size = 1000} in array steps. +} + \section{Node setup}{ \code{r_loader} sources the shared spack environment, unloads everything, and @@ -41,17 +91,14 @@ RSPH, and the project template has not loaded it for some time. The \code{git_version} argument was removed accordingly. } -\section{hpc_configs}{ +\examples{ +hpc_configs <- swf_configs_rsph(mail_user = "user@emory.edu") +hpc_configs$default_sbatch_opts -\enumerate{ -\item \code{default_sbatch_opts} is a list of sbatch options to be passed to -\code{slurmworkflow::create_workflow}. -\item \code{renv_sbatch_opts} is a list of sbatch options to be passed to -\code{slurmworkflow::step_tmpl_renv_restore}. It provides sane defaults for -building the dependencies of an R project using \code{renv} -\item \code{r_loader} is a set of bash lines to make the R software available. -This is passed to the \code{setup_lines} arguments of the -\code{slurmworkflow::step_tmpl_} functions that requires it. -} -} +# Start each step on whichever partition can run it first +swf_configs_rsph(c("epimodel", "CIDMATH", "short-cpu"))$default_sbatch_opts + +# A project whose lockfile records R 4.5 +swf_configs_rsph(r_version = "4.5.1", cleanup_workers = FALSE)$r_loader +} diff --git a/tests/testthat/test-swf_configs.R b/tests/testthat/test-swf_configs.R new file mode 100644 index 0000000..97032c9 --- /dev/null +++ b/tests/testthat/test-swf_configs.R @@ -0,0 +1,113 @@ +test_that("swf_configs_rsph defaults to the epimodel partition and R 4.6.1", { + cfg <- swf_configs_rsph() + expect_named( + cfg, + c("default_sbatch_opts", "renv_sbatch_opts", "r_loader", "max_cores") + ) + expect_identical( + cfg$default_sbatch_opts, + list("partition" = "epimodel", "mail-type" = "FAIL") + ) + expect_identical( + cfg$renv_sbatch_opts, + list("mem" = "16G", "cpus-per-task" = 4, "time" = 180) + ) + expect_identical(cfg$r_loader[3], "spack load r@4.6.1") + expect_identical(cfg$max_cores, 32) +}) + +test_that("swf_configs_rsph sets mail-user only when given", { + cfg <- swf_configs_rsph(mail_user = "user@emory.edu") + expect_identical(cfg$default_sbatch_opts[["mail-user"]], "user@emory.edu") + expect_null(swf_configs_rsph()$default_sbatch_opts[["mail-user"]]) +}) + +test_that("swf_configs_rsph accepts every current partition", { + for (p in c("epimodel", "CIDMATH", "short-cpu", "day-long-cpu", + "week-long-cpu", "month-long-cpu", "interactive-cpu", + "largemem")) { + expect_identical( + swf_configs_rsph(p)$default_sbatch_opts$partition, p + ) + } +}) + +test_that("swf_configs_rsph takes a vector or a comma-separated string", { + from_vec <- swf_configs_rsph(c("epimodel", "CIDMATH", "short-cpu")) + from_str <- swf_configs_rsph("epimodel, CIDMATH,short-cpu") + expect_identical( + from_vec$default_sbatch_opts$partition, "epimodel,CIDMATH,short-cpu" + ) + expect_identical(from_vec, from_str) + + # case-insensitive match to the canonical name, duplicates dropped + cfg <- swf_configs_rsph(c("cidmath", "epimodel", "CIDMATH")) + expect_identical(cfg$default_sbatch_opts$partition, "CIDMATH,epimodel") +}) + +test_that("swf_configs_rsph rejects the retired preemptable partition", { + expect_error(swf_configs_rsph("preemptable"), "no longer exists") + expect_error(swf_configs_rsph("epimodel,preemptable"), "no longer exists") +}) + +test_that("swf_configs_rsph rejects unknown or empty partitions", { + expect_error(swf_configs_rsph("compute"), "Unknown RSPH partition") + expect_error(swf_configs_rsph(""), "at least one") + expect_error(swf_configs_rsph(NA_character_), "character vector") + expect_error(swf_configs_rsph(1), "character vector") +}) + +test_that("max_cores is the smallest per-task limit of the listed partitions", { + expect_identical(swf_configs_rsph("short-cpu")$max_cores, 32) + expect_identical( + swf_configs_rsph(c("epimodel", "interactive-cpu"))$max_cores, 16 + ) + expect_identical(swf_configs_rsph("largemem")$max_cores, 31) +}) + +test_that("r_loader loads the requested R and the worker cleanup trap", { + with_trap <- swf_configs_rsph(r_version = "4.5.1")$r_loader + no_trap <- swf_configs_rsph(r_version = "4.5.1", cleanup_workers = FALSE) + expect_identical( + no_trap$r_loader, + c( + ". /projects/epimodel/spack/share/spack/setup-env.sh", + "spack unload -a", + "spack load r@4.5.1" + ) + ) + expect_identical(with_trap, c(no_trap$r_loader, swf_cleanup_r_workers())) +}) + +test_that("swf_configs_rsph warns on a partial R version", { + expect_warning(swf_configs_rsph(r_version = "4.3"), "not a full version") + expect_no_warning(swf_configs_rsph(r_version = "4.5.1")) +}) + +test_that("the default sbatch options reach the generated job scripts", { + wd <- file.path(tempdir(), "swf_configs_test") + unlink(wd, recursive = TRUE) + dir.create(wd, showWarnings = FALSE, recursive = TRUE) + old <- setwd(wd) + on.exit({ setwd(old); unlink(wd, recursive = TRUE) }, add = TRUE) + + cfg <- swf_configs_rsph(c("epimodel", "CIDMATH"), mail_user = "u@emory.edu") + wf <- slurmworkflow::create_workflow( + wf_name = "toy", + default_sbatch_opts = cfg$default_sbatch_opts + ) + wf <- slurmworkflow::add_workflow_step( + wf_summary = wf, + step_tmpl = step_tmpl_renv_restore( + git_branch = "main", + setup_lines = cfg$r_loader, + lockfile = "renv.lock.hpc" + ), + sbatch_opts = cfg$renv_sbatch_opts + ) + + job <- readLines(file.path("workflows", "toy", "SWF", "steps", "1", "job.sh")) + expect_true("#SBATCH --partition=epimodel,CIDMATH" %in% job) + expect_true("#SBATCH --mail-user=u@emory.edu" %in% job) + expect_true("#SBATCH --time=180" %in% job) +}) diff --git a/vignettes/epimodelhiv-slurmworkflow.Rmd b/vignettes/epimodelhiv-slurmworkflow.Rmd index 2de7ae6..5012014 100644 --- a/vignettes/epimodelhiv-slurmworkflow.Rmd +++ b/vignettes/epimodelhiv-slurmworkflow.Rmd @@ -18,735 +18,350 @@ knitr::opts_chunk$set( ## Introduction -`slurmworkflow` is a package to construct *workflows* on a -[SLURM](https://slurm.schedmd.com/overview.html) equipped High Performance -Computer (HPC). In this vignette, a *workflow* refers to a set of tasks to be -executed on the HPC, one after the other. +[`slurmworkflow`](https://epimodel.github.io/slurmworkflow/) builds *workflows* for a [SLURM](https://slurm.schedmd.com/overview.html) equipped high-performance computing (HPC) cluster. A workflow is a sequence of *steps*, each one a SLURM job, that run one after the other on the HPC. We build the workflow on our own computer, copy it to the HPC, and start it there with a single command. -We will describe how to construct and use *workflows* using the -[EpiModel/EpiModelHIV-Template](https://github.com/EpiModel/EpiModelHIV-Template) project. +This vignette explains the ideas behind the workflows we use in applied EpiModelHIV projects. It follows the [EpiModelHIV-Template](https://github.com/EpiModel/EpiModelHIV-Template) (branch `v3.3.x`) and uses the Rollins School of Public Health (RSPH) HPC at [Emory University](https://www.sph.emory.edu/) as the example cluster. It is not a step-by-step guide. For that, follow the README in each chapter of the Template and the wiki page [Running Project Workflows on the HPC](https://github.com/EpiModel/EpiModeling/wiki/Running-Project-Workflows-on-the-HPC). -This project uses [renv](https://rstudio.github.io/renv/index.html) and requires -access to the [EpiModelHIV-p](https://github.com/EpiModel/EpiModelHIV-p) private -repository. This vignette assumes that your project is hosted on a git -repository checked out on your local computer and on the HPC. +The Template uses [renv](https://rstudio.github.io/renv/) and needs access to the private [EpiModelHIV-p](https://github.com/EpiModel/EpiModelHIV-p) repository. We assume that the project is a git repository cloned both on your computer and on the HPC. -This vignette will use the Rollins School of Public Health (RSPH) High -Performance Computing cluster (HPC) from [Emory -University](https://www.sph.emory.edu/) as an example. +## Structure of an applied EpiModelHIV project -## Structure of an Applied EpiModelHIV Project +All the R code lives under `R/`: -The R scripts are all located in the "R/" subdirectory and are using the -following naming conventions: +- `R/00-setup.R`: one-time setup of the project library with renv. +- `R/shared_variables.R`: paths, time steps and other values shared by every script. The paths to generated files point inside `data/run/`: for example `est_dir` is `data/run/estimates/` and `scenarios_dir` is `data/run/scenarios/`. +- `R/hpc_configs.R`: the HPC settings and the `make_em_workflow()` function used by every workflow. +- `R/netsim_settings.R`: builds the `param` and `init` objects and the `path_to_est` and `path_to_restart` paths used by the simulations. +- One folder per chapter: `R/A-networks`, `R/B-model_dev`, `R/C-calibration` and `R/D-interventions`. Each chapter has a README, numbered scripts (`1-estimation.R`, `2-diagnostics.R`, ...) to run locally in order, `workflow-*.R` scripts that build HPC workflows, and a `z-context.R` file that switches between local and HPC settings. -- "01-snake_case_name.R": steps to be run locally in a given order. -- "workflow_01-snake_case_names.R": scripts creating *workflow* directories to - be sent on the HPC. -- "utils-snake_case_name.R": utility scripts to be `source`d by the steps or - *workflows*. They limit code repetition. +The `data/` folder contains: -The "data/" directory contains: +- `data/input/`: files the project needs before any code runs, such as model parameters and scenario definitions. Tracked by git. +- `data/run/`: everything the code produces: network estimates, calibration and scenario runs. Not tracked by git. +- `data/output/`: final tables and figures. Not tracked by git. -- "data/input/": files required by the project before any code is ran. These - files are tracked by `git`. -- "data/run/": raw files produced and used by the code. They are not tracked by `git`. -- "data/output/": final results, tables, graphs. Tracked by `git`. +The project has two renv lockfiles. `renv.lock` records the full local environment, analysis packages included. `renv.lock.hpc` records only what the model needs on the HPC: EpiModelHIV, EpiModelHPC, ARTnetData and their dependencies. Workflows are built in `workflows/`, which is not tracked by git either. -## General Steps in a Applied EpiModelHIV Project +Two per-user values go in the project `.Renviron` file (open it with `usethis::edit_r_environ("project")`): `EPIMODELHIV_DIR`, the path to your local clone of EpiModelHIV-p, and `HPC_MAIL_USER`, the address SLURM mails about your jobs. -These applied projects aim to accurately represent the population of Men who -have Sex with Men (MSM) from the Atlanta Metro area. And then simulate how -the HIV epidemic would behave under different *intervention scenarios*. +## Local and HPC contexts -A (massive) oversimplification of the project would be to see it as the -following 3 steps: +Every numbered script and every workflow sources `R/shared_variables.R` and the `z-context.R` file of its chapter. `z-context.R` looks for a variable named `hpc_context`. When it is `TRUE`, it sets `context <- "hpc"` and the full-scale settings; otherwise it sets `context <- "local"` and small, fast ones. In `R/A-networks/z-context.R`, for example, the local context estimates 10,000-node networks with the Stochastic-Approximation method on one core, and the HPC context estimates 100,000-node networks with MCMLE on 10 cores. -1. Estimate the social networks modeling the population interactions. -2. Calibrate the epidemic model to match key epidemiological targets. -3. Run the model under *intervention scenarios* to assess their effects on the - epidemic. +The context also goes into the file names, so local and HPC results never overwrite each other: the network estimates are saved as `data/run/estimates/netest-local.rds` or `data/run/estimates/netest-hpc.rds`. -## Local vs HPC Context +On the HPC, workflow steps run the same numbered scripts with `hpc_context = TRUE`. The goal is that a script that runs locally also runs on the HPC without modification, so test every script locally first. -All the numbered scripts ("R/01-snake_case_name.R") are meant to be executed on -your local computer. They allow you to explore all the steps and substeps on -small networks with few replications. Most of them define a `context` variable -at the top. This variable takes the value "local" or "hpc" and is use to set -the context of the computation: -- "local": Uses small networks and few replications, unfit for publication. - This is used to test the code and processes. -- "hpc": Uses full size networks and many replications. This is used to produce - the analysis for publication. Once the code has been validated locally. +**Note**: run each numbered script in a fresh R session (Ctrl+Shift+F10 in RStudio) to avoid starting from a polluted environment, which makes debugging very hard. Do not use `rm(list = ls())` instead: [it does not do the same thing](https://rstats.wtf/source-and-blank-slates). It also breaks the scripts run by `step_tmpl_do_call_script()`, because it deletes the variables the step defines before running the script. -A lot of the numbered scripts are re-used by the *workflows* with the `context` -variable set to "hpc". +## Workflows and steps -The goal is: if your scripts run locally, they should run on the HPC -without modification. +A workflow is a folder, `workflows//`, holding a `start_workflow.sh` script and one subfolder per step. Each `workflow-*.R` script builds one workflow in two moves: -## Network Estimation and Diagnostics +1. `make_em_workflow()` creates the workflow and its first step, which brings the project on the HPC up to date (see below). +2. Each call to `slurmworkflow::add_workflow_step()` appends a step. Its `step_tmpl` argument is a *step template*, which says what the step runs, and its `sbatch_opts` argument says which resources the step asks SLURM for. -### Overview +`sbatch_opts` accepts any long-form [`sbatch`](https://slurm.schedmd.com/sbatch.html) option without the leading `--` ("cpus-per-task" is valid, "c" is not). The options given to the workflow when it is created are the defaults for every step, and each step can override them. -This project simulates HIV dynamics on a population of 100 000 individuals. The -first step is to estimate 3 networks of 100 000 nodes representing respectively -*main*, *casual* and *one off* partnerships. This step will happen in the -script "R/01-networks_estimation.R". Afterwards we will want to diagnose the -estimations using the script "R/02-networks_diagnostics.R" and finally explore -these diagnostics interactively in the script -"R/03-networks_diagnostics_explore.R". +The steps are chained by a small controller job, so step 2 starts when step 1 is over, and so on. A step can also change which step runs next; this is how `swfcalib` loops over calibration iterations. -You should run these 3 scripts locally and make sure you understand what they do. -Without modification, the `context` variable will be set to "local" and produce -fast "Stochastic-Approximation" for 5k nodes networks. +The templates used in the Template come from two packages: -**NOTE**: You should run each script in a fresh R console each time to avoid -starting with a polluted environment that will lead to very complicated -debugging. In RStudio: Ctrl-Shift-F10 or `.rs.restartR()` (aliased to `rs()` -in the project). Do not do: `rm(list = ls())` instead, [it does not do the same -thing and should not be used](https://rstats.wtf/save-source.html#rm-list-ls). +- `slurmworkflow`: `step_tmpl_do_call_script()` runs an R script that is on the HPC, `step_tmpl_do_call()` runs an R function, and `step_tmpl_map()` runs a function over many inputs as a SLURM array. +- `EpiModelHPC`: `step_tmpl_renv_restore()` updates the project, `step_tmpl_netsim_scenarios()` runs scenario simulations, `step_tmpl_merge_netsim_scenarios_tibble()` merges their output, and `step_tmpl_netsim_swfcalib_output()` runs the model with calibrated parameters. -### Defining the "networks_estimation" *workflow* +## `make_em_workflow()`: the common first step -Now that you have run the 3 scripts locally, we will define an HPC *workflow* -to run the first 2 parts, networks estimation and networks diagnostics, with -100k nodes networks with the full MCMLE estimation method. Trying to run this -locally will take multiple days and probably crash your computer before ending. +Every workflow in the Template starts with `make_em_workflow()`, defined in `R/hpc_configs.R`. Here is a slightly shortened version: -Instead, we will create a *workflow* locally, send it to the HPC, run it there -and collect the results for analysis. +```{r hpc_configs, eval = FALSE} +current_git_branch <- "main" # <- USER: your git branch -The script "R/workflow_01-networks_estimation.R" is responsible to the creation -of the first *workflow*. We will walk through it block by block to understand -the basics of `slurmworkflow`. - -#### Setup - -A *workflow* exists on your computer as a directory inside the "workflows/" -directory of your project. Our first *workflow* is called "networks_estimation" -and will live in "workflows/networks_estimation/". +hpc_node_setup <- c( + ". /projects/epimodel/spack/share/spack/setup-env.sh", + "spack unload -a", + "spack load r@4.5.1" +) -First we load the required libraries and source the -"R/utils-0_project_settings.R". +make_em_workflow <- function(wf_name, override = FALSE, update_renv = TRUE) { + # The Template stops with instructions if this is not set in `.Renviron` + hpc_mail_user <- Sys.getenv("HPC_MAIL_USER") + + if (update_renv) { + renv::snapshot( + packages = c("EpiModelHIV", "EpiModelHPC", "ARTnetData"), + lockfile = "renv.lock.hpc", + prompt = FALSE + ) + } + + wf_path <- paste0("workflows/", wf_name) + if (override && fs::dir_exists(wf_path)) fs::dir_delete(wf_path) + + wf <- slurmworkflow::create_workflow( + wf_name = wf_name, + default_sbatch_opts = list( + "partition" = + "epimodel,short-cpu,day-long-cpu,week-long-cpu,month-long-cpu", + "mail-type" = "FAIL", + "mail-user" = hpc_mail_user + ) + ) -```{r wf_1_setup1, eval = FALSE} -# Libraries -------------------------------------------------------------------- -library("slurmworkflow") -library("EpiModelHPC") + # Update renv on the HPC + wf <- slurmworkflow::add_workflow_step( + wf_summary = wf, + step_tmpl = EpiModelHPC::step_tmpl_renv_restore( + git_branch = current_git_branch, + setup_lines = hpc_node_setup, + lockfile = if (update_renv) "renv.lock.hpc" else NULL + ), + sbatch_opts = list( + "mem" = "16G", + "cpus-per-task" = 4, + "time" = 180 + ) + ) -# Settings --------------------------------------------------------------------- -source("./R/utils-0_project_settings.R") + wf +} ``` -This last script contains variable used throughout the project. You should make -sure that the `current_git_branch` is correct and put your emai address in -`mail_user`. This way the HPC will send you a mail when the *workflow* is -finished. - -Then we set a `max_cores` variable to 10. This will be the number of CPU cores -to be used for the network estimations. 10 usually works fine. +It does four things. -```{r wf_1_setup2, eval = FALSE} -max_cores <- 10 -source("./R/utils-hpc_configs.R") # creates `hpc_configs` -``` +**It refreshes `renv.lock.hpc`** from the packages installed in your local project library. Install the EpiModelHIV-p version you want to run locally first, then build the workflow, then commit and push `renv.lock.hpc`. -The "R/utils-hpc_configs.R" script contains helper functions to simplify the HPC -setup. Here is the part that should be uncommented for using the RSPH cluster: +**It creates the workflow** in `workflows//`. `create_workflow()` stops if that folder already exists; `override = TRUE` deletes the old one first. -```{r wf_hpc_inputs, eval = FALSE} -# Must be sourced **AFTER** "./R/utils-0_project_settings.R" +**It sets the default sbatch options**: mail `HPC_MAIL_USER` when a step fails, and run on any of the listed partitions. SLURM starts each job on whichever listed partition can run it first, and it considers the lab's dedicated `epimodel` partition (15 nodes, no time limit) before the shared ones: `short-cpu` (8 hours), `day-long-cpu` (1 day), `week-long-cpu` (7 days) and `month-long-cpu` (31 days). A step that asks for more time than a partition allows only runs on the others. Lab members can also add `CIDMATH`, 8 dedicated nodes shared with three other groups. The `preemptable` partition that older projects used no longer exists. `interactive-cpu` (12 hours, at most 16 cores per job) is meant for `srun` sessions and `largemem` holds high-memory nodes; neither is needed for routine runs. -hpc_configs <- EpiModelHPC::swf_configs_rsph( - partition = "epimodel", - r_version = "4.5.1", - mail_user = mail_user -) -``` +**It adds the restore step**, which is always step 1. On a compute node, this step loads R with `hpc_node_setup`, checks that the HPC clone is on `current_git_branch` (and stops otherwise, to avoid running the wrong code), runs `git pull`, and restores the packages listed in `renv.lock.hpc`. This is how EpiModelHIV-p and the other packages get installed on the HPC: there is no need to run `renv::init()` or `renv::restore()` by hand. Do not run them on the login node either. The login node runs a different release of the operating system than the compute nodes, so renv keeps a separate library for it that the jobs never use. -We use the `EpiModelHPC::swf_configs_rsph` helper function to create the -`hpc_configs` objects that will holds some pre-defined configurations. We -specify that we want to work on the "epimodel" partition and that e-mails -telling us when the jobs are done should be sent to "user@emory.edu". +`hpc_node_setup` holds the bash lines that make R available in each step. On RSPH, R comes from [spack](https://spack.io/): `r@4.6.1` is the current build and `r@4.5.1` is still available. Load the version recorded in your lockfile, and always give the full version: `spack load r@4.3`, for example, fails because it matches several builds. -*Note*: the HPC is using the [Slurm workflow -manager](https://slurm.schedmd.com/overview.html) to allocate tasks to -computing nodes. The EpiModel partition allocate jobs to nodes *reserved* for -the EpiModel team. The other option is *preemptable*, where you can use any -empty node but may be kicked out if a *reserved* node is *preempted* by -someone. +### Outside the Template: `swf_configs_rsph()` -#### Creating the *workflow* +Projects that do not use `make_em_workflow()` can get the same settings from `swf_configs_rsph()`: -The `slurmworkflow::create_workflow` function takes 2 mandatory arguments: +```{r swf_configs, eval = FALSE} +library(slurmworkflow) +library(EpiModelHPC) -1. `wf_name`: the name of the new workflow -2. `default_sbatch_opts`: a list of default options for the - [`sbatch`](https://slurm.schedmd.com/sbatch.html) command. They will be - shared among all steps. +hpc_configs <- swf_configs_rsph( + partition = c("epimodel", "short-cpu", "day-long-cpu"), + r_version = "4.6.1", + mail_user = "@emory.edu" +) -```{r wf_1_creation, eval = FALSE} -# Workflow creation ------------------------------------------------------------ wf <- create_workflow( - wf_name = "networks_estimation", + wf_name = "my_workflow", default_sbatch_opts = hpc_configs$default_sbatch_opts ) -``` - -Here we created a *workflow* called "networks_estimation" and use the sbatch -options stored in `hpc_configs$default_sbatch_opts`. - -```{r wf_1_sbopts0, eval = FALSE} -hpc_configs$default_sbatch_opts -#> list( -#> "partition" = "epimodel", -#> "mail-type" = "FAIL" -#> "mail-user" = "user@emory.edu" -#> ) -``` - -It specifies that we want to use the "epimodel" partition and that an -e-mail should be sent if a task fails. - -With this we have created the directory "workflows/networks_estimation" and -stored a summary of it in the `wf` variable. For now our workflow has no steps. - -*notes*: SLURM configuration can vary, on [HYAK](https://hyak.uw.edu/) for instance - there is an accounting module and we would have to specify the "account" - option). An equivalent - `swf_configs_hyak` function exists for the HYAK ecosystem. -- `default_sbatch_opts` and `sbatch_opts` parameters accept all the options for - `sbatch` starting with "--". (e.g. "account" is valid but "A" is not, as it - corresponds to the "-A" shorthand) -- If a "workflows/networks_estimation" directory already exists, - `create_workflow` will throw an error. You have to delete the previous - versions of the workflow yourself if you want to overwrite them. -#### Adding of a `renv::restore` Step - -Before running the actual calculation, we want to make sure that the project on -the HPC is up to date with the right packages version. It translates to running -`git pull` on the HPC and `renv::restore()` from the project. - -To do this we will add a *step* to the workflow. The -`slurmworkflow::add_workflow_step` take 2 mandatory arguments: - -1. `wf_summary`: a summary of the workflow to edit (the `wf` variable) -2. `step_tmpl`: a *step template*. These are made by a special kind of functions - from `slurmworkflow`. - -Here we will also use the optional `sbatch_opts` arguments to override some -of the default options defined above. - -```{r wf_1_renv, eval = FALSE} -# Update RENV on the HPC ------------------------------------------------------- wf <- add_workflow_step( wf_summary = wf, step_tmpl = step_tmpl_renv_restore( - git_branch = current_git_branch, + git_branch = "main", setup_lines = hpc_configs$r_loader ), sbatch_opts = hpc_configs$renv_sbatch_opts ) - ``` -The *step template* here is from the function -`EpiModelHPC::step_tmpl_renv_restore` which takes two arguments: - -1. `git_branch`: the branch that the repository must follow. If the branch - followed (on the HPC) is not the right one, the step will stop there to - avoid potential data loss and undefined behaviors. Here we use the `current_git_branch` - variable defined in "R/utils-0_project_settings.R". -2. `setup_lines`: some boilerplate `bash` code to allow running R code on the HPC. +`hpc_configs` holds: -Internally this function sets up an `sbatch` task that will run `git pull` and -`renv::restore()` on the HPC. +- `default_sbatch_opts`: the partition list and the mail settings. +- `renv_sbatch_opts`: the resources for the restore step. +- `r_loader`: the spack lines, followed by a trap that stops a job's own R workers when the job is canceled or times out (see `?swf_cleanup_r_workers`). Pass it as `setup_lines` to every step template. +- `max_cores`: the most CPUs one task can request on every listed partition. -For this specific task we need to change some of the `sbatch` options using -`hpc_configs$renv_sbatch_opts`. - -```{r wf_1_sbopts1, eval = FALSE} -hpc_configs$renv_sbatch_opts -#> list( -#> "mem" = "16G", -#> "cpus-per-task" = 4, -#> "time" = 120 -#> ) -``` +See `?swf_configs_rsph` for the partitions it accepts and their limits. `swf_configs_hyak()` does the same for the HYAK cluster at the University of Washington. -It asks for 16GB of RAM, 4 CPUs and tell SLURM that the job should take less -than 120 minutes. +## Example: estimating the networks -We assigned the result of the call to `wf` (`wf <- add_workflow_step(...)`), and -the function modified the "workflows/networks_estimation/" folder. +`R/A-networks/workflow-networks.R` runs the network estimation and diagnostics at full scale. Estimating 100,000-node networks with MCMLE would take days on a laptop. -*notes*: on the MOX cluster from HYAK, `renv_sbatch_opts` would also changes the - `partition` to "build" as on MOX the "default" partition does not have - internet access. +```{r wf_networks, eval = FALSE} +library(slurmworkflow) -#### Addition of the *estimation* Step +hpc_context <- TRUE +source("R/shared_variables.R", local = TRUE) +source("R/A-networks/z-context.R", local = TRUE) +source("R/hpc_configs.R", local = TRUE) -Now that we ensured that the project will be up to date on the HPC, we want to -run the "R/01-networks_estimation.R" script there with `context <- "hpc"`. +wf <- make_em_workflow("networks", override = TRUE) -To do this we add another step with `slurmworkflow::add_workflow_step` but with -a different *step template*. - -```{r wf_1_est, eval = FALSE} -# Estimate the networks -------------------------------------------------------- wf <- add_workflow_step( wf_summary = wf, step_tmpl = step_tmpl_do_call_script( - r_script = "./R/01-networks_estimation.R", - args = list( - context = "hpc", - estimation_method = "MCMLE", - estimation_ncores = max_cores - ), - setup_lines = hpc_configs$r_loader + r_script = "R/A-networks/1-estimation.R", + args = list(hpc_context = TRUE), + setup_lines = hpc_node_setup ), sbatch_opts = list( - "cpus-per-task" = max_cores, + "cpus-per-task" = est_cores, "time" = "24:00:00", "mem" = "0" ) ) -``` -`slurmworkflow::step_tmpl_do_call_script` template sets up a step to run the -script located **on the HPC** under the path `r_script`, here -"R/01-networks_estimation.R", with some variables pre-defined. -We set the following variables: -- `context = "hpc"`: signals the script to use the "hpc" settings. -- `estimation_method = "MCMLE"`: we want the slower but more accurate - estimation method. -- `estimation_ncores = max_cores`: this estimation method benefits from being - parallelized. (more that 10 cores can slow things down significantly). - -If you take a look at the "R/01-networks_estimation.R" script, you will see that -`estimation_method` and `estimation_ncores` are set if `context == "local"` but -not for "hpc". Thanks to our *step template* they will be defined when the -script run as part of the workflow. - -*note*: The syntax of `step_tmpl_do_call_script` to pass arguments to a script -is similar to the one of `base::do.call`. - -**Important note**: Some users like to clear their R environment by placing -`rm(list = ls())` at the start of their scripts. In addition to [it being -discouraged generally](https://rstats.wtf/save-source.html#rm-list-ls), it will -actually prevent a script to work with `step_tmpl_do_call_script` as it -deletes the variable at the start of the script. Restarting the R session or -using the [`callr` package](https://callr.r-lib.org/index.html) are better -alternatives when working interactively. - -Finally, we also provide the `setup_lines` as before and some new -`sbatch_opts`. As no "partition" option is provided, it will default -to "epimodel" (using the values set in `create_workflow` at the beginning. - -This step will write 3 files on the HPC: (see the script itself for details) - -1. "data/intermediate/estimates/epistats-hpc.rds" -2. "data/intermediate/estimates/netstats-hpc.rds" -3. "data/intermediate/estimates/netest-hpc.rds" - -#### Addition of the *diagnostics* Step - -Finally we want to generate diagnostics for these networks with -"R/02-networks_diagnostics.R". - -```{r wf_1_diag, eval = FALSE} -# Generate the diagnostics data ------------------------------------------------ wf <- add_workflow_step( wf_summary = wf, step_tmpl = step_tmpl_do_call_script( - r_script = "./R/02-networks_diagnostics.R", - args = list( - context = "hpc", - ncores = max_cores, - nsims = 50 - ), - setup_lines = hpc_configs$r_loader + r_script = "R/A-networks/2-diagnostics.R", + args = list(hpc_context = TRUE), + setup_lines = hpc_node_setup ), sbatch_opts = list( - "cpus-per-task" = max_cores, - "time" = "04:00:00", - "mem-per-cpu" = "4G", - "mail-type" = "FAIL,END" + "mail-type" = "END", + "cpus-per-task" = est_cores, + "time" = "24:00:00", + "mem" = "0" ) ) ``` -This step uses the same template as before, with 3 variables passed to the -script: `context`, `ncores` and `nsteps`. - -As it is the last step of this *workflow* we override the "mail-type" `sbatch` -option to receive a mail when this step ends. We do so to be notified when the -workflow is finished. - -This step will write 3 files on the HPC: (see the script itself for details) - -1. "data/intermediate/calibration/netdx-main-hpc.rds" -2. "data/intermediate/calibration/netdx-casl-hpc.rds" -3. "data/intermediate/calibration/netdx-inst-hpc.rds" - -### Using the "estimation" *workflow* on the RSPH HPC - -Now that our *estimation workflow* is set up, we need to send it to the HPC, run -it and download the results. - -We assume that the "workflows/" and "data/intermediate/" directories are not -tracked by git (using ".gitignore" for example) and that the user has an SSH -access to the HPC. - -We will use `scp` to copy the folder over to the HPC as it is available on -Windows, MacOS and GNU/Linux. - -In this example, the "EpiModelHIV-Template" repository is located at -"~/projects/EpiModelHIV-Template" on the HPC. - -Before sending the workflow, make sure that the project on the HPC has `renv` -initialized. This means running `renv::init()` from the root of the project on -the HPC. - -#### Sending the *workflow* to the HPC - -If you have never used the command line before, we recommend using the terminal -from RStudio (not the R console). - -Everything written `` is to be replaced with the correct -value. - -You should make sure to understand what each part of the commands do before -running them. It will make your life easier. - -The following commands are to be run from your local computer. - -**MacOS or GNU/Linux** - -``` -# bash - local -scp -r workflows/networks_estimation @clogin01.sph.emory.edu:projects/EpiModelHIV-Template/workflows/ -``` - -**Windows** - -``` -# bash - local -set DISPLAY= -scp -r workflows\networks_estimation @clogin01.sph.emory.edu:projects/EpiModelHIV-Template/workflows/ -``` - -Forgetting `set DISPLAY=` will prevent `scp` from working correctly. - -Note that its `workflows\networks_estimation`. Windows uses back-slashes for -directories and Unix OSes uses forward-slashes. +Setting `hpc_context <- TRUE` before sourcing `z-context.R` gives us the HPC settings in this script too, including `est_cores` (10). -#### Running the *workflow* from the HPC +`step_tmpl_do_call_script()` runs a script that is already **on the HPC**, in the git clone of the project, after defining the variables listed in `args`. The syntax is similar to `base::do.call()`. Here the scripts start with `hpc_context = TRUE`, so they pick the HPC settings when they source `z-context.R`. -For this step, you must be at the command line on the HPC. This means that you -have run: `ssh @clogin01.sph.emory.edu` from your local computer. +The `sbatch_opts` of these two steps ask for `est_cores` CPUs, 24 hours, and `mem = "0"`, which means all the memory of the node. That reserves a whole node, which network estimation needs; do not use it for simulation steps. With 24 hours, these steps cannot run on `short-cpu`. The last step asks for a mail when it ends, so we know when the workflow is done. -*run `set DISPLAY=` on Windows before if you get this error: -`ssh_askpass: posix_spawnp: No such file or directory`* +The steps write `netest-hpc.rds`, `netstats-hpc.rds` and `epistats-hpc.rds` to `data/run/estimates/` on the HPC, and the diagnostics to `data/run/diagnostics/`. Once they are downloaded to the same folders locally, `R/A-networks/3-assess.R` reads them (see the chapter README). -You also need to be at the root directory of the project (where the ".git" -folder is as well as the "renv.lock" file". In this example you would get there -by running `$ cd ~/projects/EpiModelHIV-Template`. The following steps will not -work if you are not at the root of your project. - -Running the *workflow* is done by **executing** the file -"workflows/estimation/start_workflow.sh" with the following command: - -``` -# bash - hpc -./workflows/estimation/start_workflow.sh -``` - -If you are using Windows, the may not be executable. You can solve it with the -following command: - -``` -# bash - hpc -chmod +x workflows/estimation/start_workflow.sh` -``` +## Example: running intervention scenarios -The workflow will not work if you *source* the file (with `source