Repository navigation
Add ecRad test suite - #2637
Add ecRad test suite#2637matthias-ibach wants to merge 1 commit into
Conversation
Add ecRad (ECMWF radiation scheme, OpenMP offload) to the test suite: - manifests: add ecrad (master-omp) - build_supp.sh: add netcdf-c, netcdf-fortran, ecbuild and fiat (serial) - run_ecrad.sh: build ecRad, run the CTest suite and the benchmark with comparison against reference
| local _ecbuild_path=$AOMP_SUPP/ecbuild | ||
|
|
||
| declare -a MYCMAKEOPTS | ||
| MYCMAKEOPTS=(-DCMAKE_BUILD_TYPE="$BUILD_TYPE" |
There was a problem hiding this comment.
Is this a GPU build and if so, for which GPU arch does that build? All of them or the ones in the machine?
There was a problem hiding this comment.
No, the build_supp components are CPU-only. ecbuild does not compile anything (only CMake macros) and fiat is built without any --offload-arch. The only GPU build is ecRad in run_ecrad.sh, for the arch detected on the machine.
There was a problem hiding this comment.
Are there any concerns when there are multiple GPU archs on that machine? Eg. a CPU with iGPU and another dGPU? Will that still work and pick the correct target (given ROCR_VISIBLE_DEVICES is set?
There was a problem hiding this comment.
That is a good point, which I checked now. The arch comes from setaompgpu (same as for run_qe.sh and run_mglet.sh, I actually based this PR on the PRs where QE and MGLET were added!). It takes the first GPU that rocm_agent_enumerator lists. On an iGPU + dGPU machine it can pick the iGPU arch if that one comes first, ROCR_VISIBLE_DEVICES however will not change that.
Another note is that in run_ecrad.sh I set OMP_TARGET_OFFLOAD=MANDATORY (this was not done for QE and MGLET, so feel free to either keep it or omit it). I thought it might be good if a binary built for the wrong arch aborts (instead of silently just running on the host).
There was a problem hiding this comment.
Can we make it so that AOMP_GPU takes precedence for which arch to build, when it is set? This would make a couple of places easier in other areas where we might consume these scripts.
There was a problem hiding this comment.
Any libraries that are built for a specific GPU should be built as part of the test script itself (run_ecrad.sh). The tools & libraries that are built in build_supp.sh are built once and used across a wide variety of gfx targets on different test systems.
There was a problem hiding this comment.
You are right! Apologies, I got lost which script I'm commenting on. My comment about AOMP_GPU should be attached to the actual run_* script, not the build_supp.sh.
There was a problem hiding this comment.
@dpalermo:
Thank you for making that clear! I think my thought on it was the following your comment. Since ecbuild, fiat and netcdf are not GPU specific I put them in build_supp.sh. The GPU part (ecRad itself) is built in run_ecrad.sh.
Note: another reason why I put ecbuild and fiat into build_supp.sh is that I am working on putting ecTrans also in the CI (coming up soon, already working on it). It needs ecbuild and fiat too (+ LAPACK, also CPU-only), so this way they are only built once and both apps use the same ones.
Is that fine for you, or would you rather have ecbuild and fiat built inside the run scripts, so they would be built once per app?
@jplehr:
Not a problem, now I understand it :) AOMP_GPU should already take precedence (setaompgpu only uses the detected GPU if AOMP_GPU is not set) and then run_ecrad.sh uses GPU_ARCH=${GPU_ARCH:-$AOMP_GPU} (as mentioned I took QE and MGLET as reference).
The only thing that overrides AOMP_GPU is GPU_ARCH, if this is set explicitly.
Should I drop GPU_ARCH from run_ecrad.sh so AOMP_GPU is the only knob, or keep it (like in the MGLET script)?
There was a problem hiding this comment.
Sorry again for causing the confusion. :)
I am not concerned about GPU_ARCH in that it could overwrite AOMP_GPU. Thank you.
There was a problem hiding this comment.
No worries :)
Alright, then I will leave as is! Let me know if there is anything else.
Motivation
Add ecRad (ECMWF atmospheric radiation scheme) to the test suite. It is a Fortran OpenMP offload app.
Technical Details
manifests/test_24.0.xml: added ecrad (branch master-omp).bin/build_supp.sh: added netcdf-c 4.10.0, netcdf-fortran 4.6.2 (on top of the existing hdf5), ecbuild 3.16.0 (ECMWF CMake macros) and fiat 2.0.1 (ECMWF Fortran library, no MPI. Both are also needed by other ECMWF apps like ecTrans).bin/run_ecrad.sh: Builds ecRad, thenrun_ecrad_testsruns 1) "ecrad" (the CTest suite) and 2) "ecrad-benchmark" (bit-identity check against a reference. Can be skipped with ECRAD_BENCHMARK=OFF). Benchmark data directoy (input, reference, namelist) are not in this repo. If not found on the cluster (/path/to/ecrad-data), feel free to reach out to me.Test Plan
Tested from scratch on ringo and r14 with the srock nightly from 2026-10-19 and the cmake from the nightly (3.31.0):
Test Result
The code has been successfully cloned, build and the test suite has been executed.
Submission Checklist