Restart pool

CLAUDE.md

@@ -21,6 +21,7 @@ EpiModelHPC is an R package that extends the core [EpiModel](https://github.com/
 - `netsim_scenarios()` / `step_tmpl_netsim_scenarios()` - scenario-based simulation (local / HPC)
 - `merge_netsim_scenarios()` / `step_tmpl_merge_netsim_scenarios()` - merge batch results
 - `merge_netsim_scenarios_tibble()` - convert results to tibble
+- `make_restart_pool()` / `validate_restart_pool()` / `netsim_path_wrapper()` - restart pools (directories of single-sim restart points) used as `path_to_x`
 - `swf_configs_hyak()` / `swf_configs_rsph()` - cluster presets
 - `step_tmpl_renv_restore()` - renv setup on HPC
 

DESCRIPTION

@@ -1,6 +1,6 @@
 Package: EpiModelHPC
-Version: 2.9.2
-Date: 2026-07-28
+Version: 2.10.0
+Date: 2026-10-05
 Title: EpiModel Extensions for High-Performance Computing
 Description: Extension package to EpiModel to run large-scale stochastic network
     models on modern high-performance computing systems. Provides scenario-based
@@ -15,7 +15,7 @@ URL: https://www.epimodel.org/, https://epimodel.github.io/EpiModelHPC/
 BugReports: https://github.com/EpiModel/EpiModelHPC/issues
 Depends:
     R (>= 4.1),
-    EpiModel (>= 2.5.0)
+    EpiModel (>= 2.7.0)
 Imports:
     callr,
     dplyr,
@@ -38,4 +38,4 @@ Remotes:
 VignetteBuilder: knitr
 Encoding: UTF-8
 Roxygen: list(markdown = TRUE)
-Config/roxygen2/version: 8.0.0
+Config/roxygen2/version: 8.1.1

NAMESPACE

@@ -5,8 +5,10 @@ export(add_doctor_teardown_step)
 export(get_scenarios_batches_infos)
 export(get_scenarios_tibble_infos)
 export(hpc_doctor_script)
+export(make_restart_pool)
 export(merge_netsim_scenarios)
 export(merge_netsim_scenarios_tibble)
+export(netsim_path_wrapper)
 export(netsim_scenarios)
 export(netsim_swfcalib_output)
 export(step_tmpl_merge_netsim_scenarios)
@@ -18,5 +20,6 @@ export(swf_cleanup_r_workers)
 export(swf_configs_hyak)
 export(swf_configs_rsph)
 export(swfcalib_proposal_to_scenario)
+export(validate_restart_pool)
 import(EpiModel)
 importFrom(rlang,.data)

NEWS.md

@@ -1,3 +1,25 @@
+# EpiModelHPC 2.10.0
+
+## NEW FEATURES
+
+- Restart pools: a directory of single simulation restart points (`1.rds`, `2.rds`, ...) can now be passed as `path_to_x` to `netsim_scenarios()`, `step_tmpl_netsim_scenarios()`, `netsim_swfcalib_output()` and `step_tmpl_netsim_swfcalib_output()`. Simulation `k` of a scenario (numbered over all its batches) starts from pool element `(k - 1) %% N + 1`, the same for every scenario, so scenarios can be compared simulation by simulation.
+- `make_restart_pool()` makes such a pool from the batch files of a previous run, with `EpiModel::make_restart_point()`. Pool element `i` is made from row `i` of `keep_sims`, and a `pool_index.csv` file records the batch and simulation each element comes from.
+- `validate_restart_pool()` checks a pool before use: files numbered `1` to `N`, each a single simulation restart point. It returns a table of the elements (number of simulations, time step made at).
+- `netsim_path_wrapper()` runs one `netsim()` call per simulation, from a pool or from a single file, and is now what every batch of `netsim_scenarios()` and `step_tmpl_netsim_scenarios()` runs. Batches are then merged with `merge()`.
+- `merge_netsim_scenarios()` and `step_tmpl_merge_netsim_scenarios()` gain the `keep.run`, `keep.cumulative.edgelist` and `keep.attr.history` arguments of `merge.netsim()`.
+
+## BREAKING CHANGES
+
+- Requires EpiModel 2.7.0: restart points holding a single simulation (`make_restart_point(sim_num = )`), the `keep.run`, `keep.cumulative.edgelist` and `keep.attr.history` arguments of `merge.netsim()`, and merges of batches in sequence.
+- Each simulation of a batch is now its own `netsim()` call (one per `future` worker). Restarting from a single file holding several simulations (e.g. a merged restart object) restarts every simulation from the first one; make a pool with `make_restart_pool()` instead. Runs from a fitted network model read it once per simulation.
+- With checkpointing, each simulation checkpoints in `<.checkpoint.dir>/sim__<scenario>__<batch>/sim_<k>`.
+- `step_tmpl_netsim_swfcalib_output()` now runs its batches with `future::plan("multicore", workers = n_cores)` by default, like `step_tmpl_netsim_scenarios()`. It previously used the `multisession` default of `EpiModel::netsim()`, whose PSOCK workers can stall at startup and outlive a cancelled job on the HPC. The local `netsim_scenarios()` and `netsim_swfcalib_output()` keep `multisession`, which also runs in parallel from RStudio and on Windows. `control$future.use.plan <- future::tweak(<your plan>)` still overrides the default.
+
+## BUG FIXES
+
+- `get_scenarios_batches_infos()` sorts the batch files by scenario, then batch number. They were listed in alphabetical order of their file names (batch `10` before batch `2`), so `merge_netsim_scenarios()` merged the batches of a scenario with 10 or more batches out of order, and simulation `k` of the merged object was not simulation `k` of the scenario.
+- `step_tmpl_merge_netsim_scenarios()` passed `keep.diss.stats` and `param.error` in swapped order to its inner function, and swapped them back when calling `merge_netsim_scenarios()`. Arguments are now passed by name.
+
 # EpiModelHPC 2.9.2
 
 ## NEW FEATURES

R/netsim_scenarios.R

@@ -3,9 +3,10 @@
 #' This step template is similar to `netsim_scenarios` but for the HPC. It uses
 #' `slurmworkflow::step_tmpl_map` internally and should be used as any
 #' `slurmworkflow` step. For details, see `netsim_scenarios` documentation.
-#' The inner parallelization is by default handled with
+#' The simulations of a batch are run in parallel, one `netsim` call per
+#' simulation (see `netsim_path_wrapper`), by default with
 #' `future::plan("multicore", workers = n_cores)`.
-#' Using `control$future.use.plan <- future::tweaked(<your plan>)` will bypass
+#' Using `control$future.use.plan <- future::tweak(<your plan>)` will bypass
 #' this setting.
 #'
 #' @inheritParams slurmworkflow::step_tmpl_map
@@ -59,6 +60,13 @@ step_tmpl_netsim_scenarios <- function(path_to_x, param, init, control,
 #' "sim__name_of_scenario__2.rds". Where the last number is the batch number
 #' for this particular scenario. Each scenario is therefore run over
 #' `ceiling(n_rep / n_cores)` batches.
+#' The simulations of a batch are run in parallel, one `netsim` call per
+#' simulation (see `netsim_path_wrapper`), by default with
+#' `future::plan("multisession", workers = n_cores)`. Unlike the `multicore`
+#' default of `step_tmpl_netsim_scenarios`, this also runs in parallel from
+#' RStudio and on Windows.
+#' Using `control$future.use.plan <- future::tweak(<your plan>)` will bypass
+#' this setting.
 #' This function is meant to mimic the behavior of
 #' `step_tmpl_netsim_scenarios` in your local machine. It should fail
 #' in a similar fashion an reciprocally, if it runs correctly locally, moving
@@ -131,13 +139,25 @@ netsim_scenarios_setup <- function(path_to_x, param, init, control,
   )
 }
 
-#' Run one `netsim` call with a scenario and saves the results deterministically
+#' Run one batch of `netsim` simulations with a scenario and saves the results
+#' deterministically
 #'
 #' This inner function is called by `netsim_scenarios` and
-#' `step_tmpl_netsim_scenarios`.
-#'
-#' @param path_to_x Path to a Fitted network model object saved with `saveRDS`.
-#'   (See the `x` argument to the `EpiModel::netsim` function)
+#' `step_tmpl_netsim_scenarios`. It runs the simulations of batch `batch_num`
+#' with `netsim_path_wrapper`, one `netsim` call per simulation, merges them
+#' and saves them as "sim__<scenario id>__<batch_num>.rds" in `output_dir`.
+#' The simulations of batch `b` are numbered `(b - 1) * n_cores + 1` to
+#' `b * n_cores` over all the batches of a scenario. With a restart pool of
+#' size `N`, simulation `k` starts from pool element `(k - 1) %% N + 1`, the
+#' same for every scenario.
+#' The simulations are run with the `future::tweak()` plan in
+#' `control$future.use.plan`, or with
+#' `future::plan("multisession", workers = n_cores)` if there is none. The step
+#' templates set it to `multicore` before calling this function.
+#'
+#' @param path_to_x Path to a fitted network model or a restart point saved
+#'   with `saveRDS` (See the `x` argument to the `EpiModel::netsim`
+#'   function), or to a restart pool directory (see `make_restart_pool`).
 #' @param scenario A single "`EpiModel` scenario" to be used in the simulation
 #' @param batch_num The batch number, calculated from the number of replications
 #'   and CPUs required.
@@ -151,36 +171,46 @@ netsim_scenarios_setup <- function(path_to_x, param, init, control,
 #'
 #' @section Checkpointing:
 #' This function takes care of editing `.checkpoint.dir` to create unique sub
-#' directories for each scenario. The `EpiModel::control.net` way of setting up
-#' checkpoints can be used transparently.
+#' directories for each scenario, batch and simulation:
+#' "<.checkpoint.dir>/sim__<scenario id>__<batch_num>/sim_<k>". The
+#' `EpiModel::control.net` way of setting up checkpoints can be used
+#' transparently.
 netsim_run_one_scenario <- function(scenario, batch_num,
                                     path_to_x, param, init, control,
                                     libraries, output_dir,
                                     n_batch, n_rep, n_cores) {
-  est <- readRDS(path_to_x)
   start_time <- Sys.time()
   lapply(libraries, function(l) library(l, character.only = TRUE))
 
   if (!fs::dir_exists(output_dir))
     fs::dir_create(output_dir, recurse = TRUE)
 
+  sim_nums_offset <- (batch_num - 1) * n_cores
   # On last batch, adjust the number of simulation to be run
   if (batch_num == n_batch)
     n_cores <- n_rep - n_cores * (n_batch - 1)
+  sim_nums <- sim_nums_offset + seq_len(n_cores)
 
   param_sc <- EpiModel::use_scenario(param, scenario)
-  control$nsims <- n_cores
-  control$ncores <- n_cores
 
   if (!is.null(control[[".checkpoint.dir"]])) {
     control[[".checkpoint.dir"]] <- paste0(
       control[[".checkpoint.dir"]], "/sim__", scenario[["id"]], "__", batch_num
     )
   }
 
+  # `multisession` plan with `n_cores` workers unless set in `future.use.plan`
+  # The step templates set a `multicore` plan before reaching here
+  sim_plan <- control$future.use.plan
+  if (!inherits(sim_plan, c("tweaked", "future"))) {
+    sim_plan <- future::tweak("multisession", workers = n_cores)
+  }
+
   print(paste0("Starting simulation for scenario: ", scenario[["id"]]))
   print(paste0("Batch number: ", batch_num, " / ", n_batch))
-  sim <- EpiModel::netsim(est, param_sc, init, control)
+  with(future::plan(sim_plan), local = TRUE)
+  sim <- netsim_path_wrapper(path_to_x, param_sc, init, control, sim_nums) |>
+    Reduce(f = merge)
 
   file_name <- paste0("sim__", scenario[["id"]], "__", batch_num, ".rds")
   print(paste0("Saving simulation in file: ", file_name))
@@ -201,7 +231,8 @@ netsim_run_one_scenario <- function(scenario, batch_num,
 #'
 #' @return a `tibble` with three columns: `file_path` - the full paths of
 #' the simulation file, `scenario_name` the associated scenario name,
-#' `batch_number` the associated batch number.
+#' `batch_number` the associated batch number. Sorted by scenario name, then
+#' batch number (`2` before `10`).
 #'
 #' @export
 get_scenarios_batches_infos <- function(scenario_dir) {
@@ -219,7 +250,8 @@ get_scenarios_batches_infos <- function(scenario_dir) {
     file_path = file_name_list,
     scenario_name = parts[, "scenario"],
     batch_number = as.integer(parts[, "batch"])
-  )
+  ) |>
+    dplyr::arrange(.data$scenario_name, .data$batch_number)
 }
 
 
@@ -237,8 +269,9 @@ merge_netsim_scenarios <- function(sim_dir, output_dir,
                                    keep.transmat = TRUE, keep.network = TRUE,
                                    keep.nwstats = TRUE, keep.other = TRUE,
                                    param.error = FALSE, keep.diss.stats = TRUE,
-                                   truncate.at = NULL) {
-
+                                   truncate.at = NULL, keep.run = TRUE,
+                                   keep.cumulative.edgelist = FALSE,
+                                   keep.attr.history = TRUE) {
   if (!fs::dir_exists(output_dir)) fs::dir_create(output_dir)
   batches_infos <- get_scenarios_batches_infos(sim_dir)
 
@@ -268,7 +301,10 @@ merge_netsim_scenarios <- function(sim_dir, output_dir,
             keep.nwstats = keep.nwstats,
             keep.other = keep.other,
             param.error = param.error,
-            keep.diss.stats = keep.diss.stats
+            keep.diss.stats = keep.diss.stats,
+            keep.run = keep.run,
+            keep.cumulative.edgelist = keep.cumulative.edgelist,
+            keep.attr.history = keep.attr.history
           )
         }
 
@@ -301,25 +337,32 @@ step_tmpl_merge_netsim_scenarios <- function(sim_dir, output_dir,
                                              param.error = FALSE,
                                              keep.diss.stats = TRUE,
                                              truncate.at = NULL, n_cores = 1,
-                                             setup_lines = NULL) {
+                                             setup_lines = NULL,
+                                             keep.run = TRUE,
+                                             keep.cumulative.edgelist = FALSE,
+                                             keep.attr.history = TRUE) {
 
-  merge_fun <- function(sim_dir, output_dir, keep.transmat, keep.network,
-                        keep.nwstats, keep.other, param.error, keep.diss.stats,
-                        truncate.at, n_cores) {
+  merge_fun <- function(n_cores, ...) {
     future::plan("multicore", workers = n_cores)
-    EpiModelHPC::merge_netsim_scenarios(
-      sim_dir, output_dir,
-      keep.transmat, keep.network, keep.nwstats, keep.other, keep.diss.stats,
-      param.error, truncate.at
-    )
+    EpiModelHPC::merge_netsim_scenarios(...)
   }
 
   slurmworkflow::step_tmpl_do_call(
     what = merge_fun,
     args = list(
-      sim_dir, output_dir,
-      keep.transmat, keep.network, keep.nwstats, keep.other, keep.diss.stats,
-      param.error, truncate.at, n_cores
+      n_cores = n_cores,
+      sim_dir = sim_dir,
+      output_dir = output_dir,
+      keep.transmat = keep.transmat,
+      keep.network = keep.network,
+      keep.nwstats = keep.nwstats,
+      keep.other = keep.other,
+      param.error = param.error,
+      keep.diss.stats = keep.diss.stats,
+      truncate.at = truncate.at,
+      keep.run = keep.run,
+      keep.cumulative.edgelist = keep.cumulative.edgelist,
+      keep.attr.history = keep.attr.history
     ),
     setup_lines = setup_lines
   )