Skip to content

Granular control of PISA fast-histogram threading and more robust numba thread setup - #939

Merged
thehrh merged 4 commits into
masterfrom
numba_multithreading_improvements
May 28, 2026
Merged

thehrh merged 4 commits into
masterfrom
numba_multithreading_improvements

Conversation

@thehrh

@thehrh thehrh commented May 22, 2026 •

Copy link
Copy Markdown
Collaborator

Changes

  • Set up numba with correct number of threads already upon pisa package import (see Is it merely a coincidence that certain modules' numba operations respect PISA_NUM_THREADS? #785)
  • For fast_histogram histogramming, setting PISA_HIST_THREADING now takes precedence over PISA_NUM_THREADS and PISA_TARGET, in order to enable granular histogram threading control.
    • New default behaviour will be no multi-threading for fast histogramming even when PISA_NUM_THREADS > 1. Instead, the user needs to opt in explicitly by setting PISA_HIST_THREADING='auto' (recovers current default) or any integer.
  • Update readme about global variables to reflect these changes (together with a few other fixes)

Motivation

Timing study

analysed script (no octant check to speed it up—still retaining hundreds of template calculations)

#!/usr/bin/env python3

import pisa
from pisa.analysis.analysis import Analysis
from pisa.core.distribution_maker import DistributionMaker
from pisa.core.pipeline import Pipeline

ana = Analysis()
minimizer_cfg = pisa.utils.fileio.from_file('settings/minimizer/slsqp_ftol1e-6_eps1e-4_maxiter1000.json')

data_maker = Pipeline("settings/pipeline/IceCube_3y_data.cfg")
data = data_maker.get_outputs()

model = DistributionMaker(["settings/pipeline/IceCube_3y_neutrinos.cfg", "settings/pipeline/IceCube_3y_muons.cfg"], profile=True)

result = ana.fit_hypo(data, model, metric="mod_chi2", minimizer_settings=minimizer_cfg, fit_octants_separately=False, check_octant=False, pprint=False) # no octant check for speed up

model.report_profile()

results

  • NB: all parallel operations configured to use 4 threads; no. of templates produced/minimiser iterations not constant + import/setup overhead included <-> speedup does not apply exactly to the average duration of get_outputs
Configuration Time Speedup
Single-threaded baseline (CPU) 261.43s -
Numba parallel only (no histogram threading) 137.10s 1.91x
Numba parallel + histogram threading 180.45s 1.45x
CPU single-threaded + forced 4-thread histogram 308.58s (1.18x slower)

Here we see that histogram threading always introduces overhead, whether relative to numba parallel or the baseline: the ThreadPoolExecutor overhead in _threaded_fh_histogramdd itself is the problem.

Following are two sets of neutrino Pipeline profile reports for more details, for configurations 2 and 4 above.

  • Numba parallel only (no histogram threading):
Pipeline: neutrinos
- setup:        Total time (s): 3.621, n calls: 1
- run:          Total time (s): 91.620, n calls: 427, time/call (s): mean 0.215, max. 21.641, min. 0.043
- get_outputs:  Total time (s): 93.104, n calls: 427, time/call (s): mean 0.218, max. 21.644, min. 0.046
Individual services:
data csv_loader
- setup:    Total time (s): 3.308, n calls: 1
- compute:  Total time (s): +0.000, n calls: 1
- apply:    Total time (s): 0.568, n calls: 427, time/call (s): mean 0.001, max. 0.004, min. 0.001
flux honda_ip
- setup:    Total time (s): 0.201, n calls: 1
- compute:  Total time (s): 20.187, n calls: 1
- apply:    Total time (s): +0.000, n calls: 427, time/call (s): mean +0.000, max. +0.000, min. +0.000
flux barr_simple
- setup:    Total time (s): +0.000, n calls: 1
- compute:  Total time (s): 11.531, n calls: 183, time/call (s): mean 0.063, max. 0.096, min. 0.057
- apply:    Total time (s): +0.000, n calls: 427, time/call (s): mean +0.000, max. +0.000, min. +0.000
osc prob3
- setup:    Total time (s): 0.081, n calls: 1
- compute:  Total time (s): 35.830, n calls: 161, time/call (s): mean 0.223, max. 0.297, min. 0.212
- apply:    Total time (s): 7.404, n calls: 427, time/call (s): mean 0.017, max. 1.062, min. 0.005
aeff aeff
- setup:    Total time (s): +0.000, n calls: 1
- compute:  Total time (s): +0.000, n calls: 161, time/call (s): mean +0.000, max. +0.000, min. +0.000
- apply:    Total time (s): 0.987, n calls: 427, time/call (s): mean 0.002, max. 0.005, min. 0.002
utils hist
- setup:    Total time (s): 0.025, n calls: 1
- compute:  Total time (s): +0.000, n calls: 1
- apply:    Total time (s): 13.537, n calls: 427, time/call (s): mean 0.032, max. 0.045, min. 0.031
discr_sys hypersurfaces
- setup:    Total time (s): +0.000, n calls: 1
- compute:  Total time (s): 0.124, n calls: 205, time/call (s): mean +0.000, max. +0.000, min. +0.000
- apply:    Total time (s): 0.193, n calls: 427, time/call (s): mean +0.000, max. 0.003, min. +0.000
  • CPU single-threaded + forced 4-thread histogram (flux.barr_simple, osc.prob3 and utils.hist times):
Pipeline: neutrinos
- setup:        Total time (s): 3.435, n calls: 1
- run:          Total time (s): 257.888, n calls: 499, time/call (s): mean 0.517, max. 24.587, min. 0.065
- get_outputs:  Total time (s): 260.161, n calls: 499, time/call (s): mean 0.521, max. 24.591, min. 0.069
Individual services:
data csv_loader
- setup:    Total time (s): 3.297, n calls: 1
- compute:  Total time (s): +0.000, n calls: 1
- apply:    Total time (s): 0.742, n calls: 499, time/call (s): mean 0.001, max. 0.004, min. 0.001
flux honda_ip
- setup:    Total time (s): 0.024, n calls: 1
- compute:  Total time (s): 22.872, n calls: 1
- apply:    Total time (s): +0.000, n calls: 499, time/call (s): mean +0.000, max. +0.000, min. +0.000
flux barr_simple
- setup:    Total time (s): +0.000, n calls: 1
- compute:  Total time (s): 47.262, n calls: 200, time/call (s): mean 0.236, max. 0.302, min. 0.198
- apply:    Total time (s): +0.000, n calls: 499, time/call (s): mean +0.000, max. +0.000, min. +0.000
osc prob3
- setup:    Total time (s): 0.082, n calls: 1
- compute:  Total time (s): 145.745, n calls: 173, time/call (s): mean 0.842, max. 0.899, min. 0.796
- apply:    Total time (s): 9.374, n calls: 499, time/call (s): mean 0.019, max. 0.613, min. 0.005
aeff aeff
- setup:    Total time (s): +0.000, n calls: 1
- compute:  Total time (s): +0.000, n calls: 173, time/call (s): mean +0.000, max. +0.000, min. +0.000
- apply:    Total time (s): 1.172, n calls: 499, time/call (s): mean 0.002, max. 0.004, min. 0.002
utils hist
- setup:    Total time (s): 0.026, n calls: 1
- compute:  Total time (s): +0.000, n calls: 1
- apply:    Total time (s): 28.313, n calls: 499, time/call (s): mean 0.057, max. 0.067, min. 0.050
discr_sys hypersurfaces
- setup:    Total time (s): +0.000, n calls: 1
- compute:  Total time (s): 0.211, n calls: 227, time/call (s): mean +0.000, max. 0.001, min. +0.000
- apply:    Total time (s): 0.327, n calls: 499, time/call (s): mean +0.000, max. +0.000, min. +0.000

@thehrh
thehrh merged commit c8100e7 into master May 28, 2026
@thehrh
thehrh deleted the numba_multithreading_improvements branch May 28, 2026 12:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Is it merely a coincidence that certain modules' numba operations respect PISA_NUM_THREADS?

2 participants