Source test cases from other repositories by PranjalManhgaye · Pull Request #732 · precice/tutorials

PranjalManhgaye · 2026-03-01T03:00:12Z

What this is about

(closes #538)

right now, every entry in tests.yaml assumes the tutorial folder is already in this repo. you give a path like flow-over-heated-plate, and the runner finds it under the local precice/tutorials checkout.

Issue #538 asks for a way to run cases that live somewhere else — another git repo, a fork, or from a direct download link (e.g. a .tar.gz ), without copying the files into this repo first.

This PR adds an optional source block in tests.yaml for that.

What happens at runtime

I have not touched the local behaviour : if there is no source field, behaviour is the same as today.

If source is set:

The runner fetches the tutorial (git shallow clone or archive download) into a cache directory (~/.cache/precice-tutorials by default, or PRECICE_EXTERNAL_CACHE_DIR if you set it).
It loads metadata.yaml from that folder, same as for in-repo tutorials.
It copies the tutorial into the usual run directory and continues with Docker build, docker compose up, and fieldcompare.

TUTORIALS_REF / TUTORIALS_PR still only affect local tutorials. For external ones, the version is whatever you put in source.ref.

Example

Git:

- path: flow-over-heated-plate
  source:
    type: git
    url: https://github.com/precice/tutorials.git
    ref: develop
    subdir: .
  case_combination:
    - fluid-openfoam
    - solid-openfoam
  reference_result: ./flow-over-heated-plate/reference-results/fluid-openfoam_solid-openfoam.tar.gz

Archive:

- path: some-tutorial
  source:
    type: archive
    url: https://example.org/tutorial-bundle.tar.gz
  case_combination: [...]
  reference_result: ./some-tutorial/reference-results/...

type can be git or archive. If you omit source, it stays local.

For git, if clone --branch ref fails, there is a small fallback that tries fetching without a branch and then checks a few common default branch names.

How it works

flowchart LR
  A[tests.yaml entry] --> B{Has source?}
  B -->|No| C[Use local tutorial]
  B -->|Yes| D[Fetch git or archive]
  D --> E[Resolve tutorial folder]
  E --> F[Copy into run directory]
  F --> G[Docker build and run]
  G --> H[fieldcompare vs reference]

Files touched

tools/tests/systemtests/sources.py — new: fetch + cache + resolve_tutorial_root
tools/tests/systemtests/TestSuite.py — read source, load external metadata when needed
tools/tests/systemtests/Systemtest.py — copy from resolved path; ref/PR checkout only for local
tools/tests/metadata_parser/metdata.py — Tutorial.source, ReferenceResult.base_dir
tools/tests/README.md — short how-to
changelog-entries/732.md

I did not add a real source: entry to tests.yaml yet, so existing suites should behave as before until someone opts in.

Rebased onto current develop and fixed the merge conflicts in README / Systemtest.py / TestSuite.py.

testing

I ran the usual checks on external-source-fix after rebasing onto develop:
python3 -m compileall on tools/tests/systemtests and metadata_parser — no issues.

Loaded all 13 suites from tests.yaml — unchanged for entries without source.

For git, I used a smoke test that drops quickstart from the local index and points at precice/tutorials on develop. The tutorial was fetched into PRECICE_EXTERNAL_CACHE_DIR, source.type was git, and the quickstart systemtest (fluid-openfoam + solid-cpp) passed end to end. No tutorials_ref symlink in the run directory.

For archive, I built a quickstart-develop.tar.gz from the same develop checkout (symlinks resolved so scripts are real files), served it via a file:// URL, and ran the same smoke path with source.type: archive. Download, extract, cache, copy, Docker build/run, and fieldcompare all passed. Again, no tutorials_ref.

I also ran the normal quickstart entry from tests.yaml with TUTORIALS_REF:develop so local tutorials still behave as today when the path exists in the repo.

Docker images were already cached, so both external runs were on the order of ~15–20s each; the steps were still the full pipeline.

Checklist

Changelog entry in changelog-entries/732.md

PranjalManhgaye · 2026-03-01T09:24:44Z

@MakisH , sir happy to update anything needed before review .

precice-bot · 2026-03-08T13:11:59Z

This pull request has been mentioned on preCICE Forum on Discourse. There might be relevant details there:

https://precice.discourse.group/t/gsoc-2026-pranjal-manhgaye/2769/6

MakisH · 2026-05-17T16:01:44Z

@PranjalManhgaye this seems quite long. Is it still intended for review in its current form?

In the PR description, I would expect some explanation of the whole concept.

Please also resolve the conflicts.

PranjalManhgaye · 2026-05-17T16:21:46Z

thanks @MakisH , i will take an another look of this pr and will inform you if its indeed necessary to review it .

Allow tests.yaml entries to load tutorials from external git repos or archives. Rebased on develop and documented usage in the README.

Rebase onto latest precice/develop and resolve conflicts in systemtests files and README for external tutorial sources (PR precice#732).

PranjalManhgaye · 2026-05-18T11:31:38Z

@MakisH , i updated the description, rebased on develop, resolved the conflicts, and retested everything locally ,Existing suites are unchanged and both external git + archive quickstart paths are passing now. i think everything should be good now, but happy to change anything if you spot something ,,

MakisH

Thanks for including a configuration example and a Mermaid graph, this helps a lot!

Looking at the example, I think that defining a source: in tests.yaml makes sense. I also like that it works both for repositories and for archives.

Could you please update your branch to the latest state? I am through with the larger changes I wanted to do for now, and it would be a good time to review this in the next days.

Integrate latest develop; external tutorial source support unchanged.

PranjalManhgaye · 2026-06-16T19:06:03Z

@MakisH i have merged latest develop into the branch , also fixed the conflicts and retested locally , it is ready for a review when you get time in next days ,,

MakisH

I tried this locally with a concrete example (from #845) and it seems to work. 🎉

Still, I think the configuration looks a bit strange/unintuitive at the moment. Let's refine the configuration based on that example.

I have not yet tried generating reference results, which might be an issue, as it would not be easy/always possible for the runner to push to the external repository (some sources are even static archives). We should also not store those reference results in the tutorials repository, as they have nothing to do with the tutorials, and might not be public. For the specific test, I could push reference results to the same repository/branch and use a more relaxed tolerance. Other than that, I would just expect to skip the comparison now (both related to #847).

Use external: instead of tutorials: for external suites, put source before path in examples, and move docs under Extending.

PranjalManhgaye · 2026-06-24T17:08:25Z

@MakisH addressed the review points => tested locally, all existing suites still load fine and the external example parses correctly , for reference results on external tests i'll wire up skip_compare from #847 once that's in rather than storing refs here , let me know if the config shape looks better now ,,

Integrate latest develop (precice#847 tolerance/skip_compare, precice#848, precice#852, etc.) and resolve TestSuite.py conflict while keeping external: tutorial support. Set skip_compare on external suite entries now that precice#847 is merged.

PranjalManhgaye · 2026-06-30T02:47:45Z

@MakisH I have merged latest develop and pushed - now it should address the conflict and the external skip_compare about which we talked earlier. Ready for another look when you have time.

Integrate precice#724, precice#849, precice#854 and resolve TestSuite.py conflict (external sources + tolerance/skip_compare + run-before/run-after).

MakisH

I had another look and test, and while it generally works, I think some aspects can be simplified. My main doubts were regarding the cache, the restriction to not mix tutorials: and external:, and some issue with ZIP archives.

Getting there!

MakisH · 2026-07-01T05:30:17Z

+                    raise ValueError(
+                        f"Test suite '{test_suite_name}' must use either 'tutorials' or "
+                        f"'external', not both."
+                    )


Why are they not possible to combine? It would be convenient to be able to write:

release: tutorials: - *breaking-dam-2d_fluid-openfoam_solid-calculix - *channel-transport_fluid-openfoam_transport-nutils - ... external: - *partitioned-heat-conduction-robin_lrft-openfoam_right-openfoam - *partitioned-heat-conduction-mixedbc_left-openfoam_right-openfoam

which would also allow the external cases to be used in the openfoam-adapter test suite, for example.

I have dropped the mutual-exclusion check so a suite can list both tutorials and external. External cases can be defined once with anchors and referenced from release (and other suites). Also, I have updated the README example accordingly.

MakisH · 2026-07-01T05:33:45Z

+# Cache directory for fetched tutorials. Can be overridden via PRECICE_EXTERNAL_CACHE_DIR env.
+_DEFAULT_CACHE = Path(os.environ.get("XDG_CACHE_HOME", Path.home() / ".cache")) / "precice-tutorials"
+PRECICE_EXTERNAL_CACHE_DIR = Path(os.environ.get("PRECICE_EXTERNAL_CACHE_DIR", _DEFAULT_CACHE))


What is the motivation for this cache?
Similarly for the _cache_key function.

I have added a short explanation in the README: the cache avoids re-cloning/re-downloading the same external source on every run and in CI; And it is keyed by URL, ref, and optional subdir. Override with PRECICE_EXTERNAL_CACHE_DIR if needed.

MakisH · 2026-07-01T05:56:46Z

+                ["git", "-C", str(checkout), "fetch", "origin", ref, "--depth", "1"],
+                check=True,
+                capture_output=True,
+                timeout=120,


There is a short timeout defined globally somewhere else. Why not use that?
Similarly in different places.
No strong opinion on the one or the other approach, but it looks inconsistent.

I have replaced the hardcoded fetch timeouts with _FETCH_TIMEOUT, driven by the same PRECICE_SYSTEMTESTS_TIMEOUT env var as GLOBAL_TIMEOUT (could not import GLOBAL_TIMEOUT directly without a circular import).

MakisH · 2026-07-01T05:59:04Z

+            # Fallback: branch may not exist (e.g. repo uses develop/master instead of main)
+            # Clone without branch, then fetch and checkout ref (with common aliases)
+            shutil.rmtree(checkout, ignore_errors=True)
+            logging.debug(
+                f"git clone --branch {ref} failed ({result.stderr}), trying clone + fetch"
+            )


If the ref does not exist, shouldn't the user get an error to choose a different ref?

In general, I think that parts of this function can be simplified. Better throw an error / let it crash and let the user fix the configuration, rather than trying to guess.

I have removed the main/develop/master fallback. If git clone --branch fails, we now raise a clear error so the config can be fixed.

MakisH · 2026-07-01T06:19:10Z

+        else:
+            import zipfile
+
+            with zipfile.ZipFile(tmp_path, "r") as zf:
+                zf.extractall(extract_dir)


When downloading a zip file (https://github.com/vidulejs/tutorials/archive/refs/heads/partitioned-heat-conduction-mixed-robin.zip) I am getting a permission denied when trying to run the respective run.sh scripts in the system tests.

See the Partitioned heat conduction mixed BC (left-openfoam, right-openfoam) case (sourced from an archive), which reports success, even though the logs show this error: https://github.com/precice/tutorials/actions/runs/28495445035/attempts/5

However, if I just unzip and run the tutorial locally, it just works. So the ZIP file seems to have maintained the file permissions. This seems to be a common issue:

https://stackoverflow.com/q/76645832/2254346

https://www.w3reference.com/blog/python-zipfile-removes-execute-permissions-from-binaries/

After the zip/tar extract (and on cache hits), we restore execute bits on *.sh files. I have also reproduced the mixed-BC archive case locally; the external suite passes end-to-end with no permission denied on run.sh.

Allow mixed tutorials/external suites, fail fast on bad git refs, restore shell execute bits after ZIP extract, align fetch timeouts with GLOBAL_TIMEOUT, and document external cache behaviour in README.

PranjalManhgaye changed the title ~~Add external tutorial sources for systemtests (Closes #538)~~ Source test cases from other repositories (Closes #538) Mar 1, 2026

MakisH added systemtests GSoC Contributed in the context of the Google Summer of Code labels Mar 1, 2026

MakisH changed the title ~~Source test cases from other repositories (Closes #538)~~ Source test cases from other repositories May 17, 2026

Add external tutorial sources support (git/archive) for systemtests

57b1807

Allow tests.yaml entries to load tutorials from external git repos or archives. Rebased on develop and documented usage in the README.

PranjalManhgaye force-pushed the external-source-fix branch from 3ba6044 to 57b1807 Compare May 18, 2026 08:04

Merge branch 'develop' into external-source-fix

2a409bb

Rebase onto latest precice/develop and resolve conflicts in systemtests files and README for external tutorial sources (PR precice#732).

Fix pre-commit style checks for PR precice#732

4d68a31

MakisH moved this to Open in GSoC 2026: System tests improvements May 25, 2026

MakisH added this to GSoC 2026: System tests improvements May 25, 2026

MakisH moved this from Open to Needs review in GSoC 2026: System tests improvements May 25, 2026

MakisH reviewed Jun 16, 2026

View reviewed changes

Merge branch 'develop' into external-source-fix

55bb04b

Integrate latest develop; external tutorial source support unchanged.

PranjalManhgaye requested a review from MakisH June 16, 2026 19:06

MakisH added 2 commits June 24, 2026 15:07

Merge branch 'develop' into external-source-fix

ee526d4

Add a concrete example in the tests.yaml

a9e6c78

MakisH reviewed Jun 24, 2026

View reviewed changes

Comment thread tools/tests/README.md Outdated

Comment thread tools/tests/README.md Outdated

Refine external test config shape per review on PR precice#732

8424e57

Use external: instead of tutorials: for external suites, put source before path in examples, and move docs under Extending.

PranjalManhgaye mentioned this pull request Jun 24, 2026

Add per-test fieldcompare tolerance and skip_compare #847

Merged

PranjalManhgaye requested a review from MakisH June 30, 2026 03:50

Merge branch 'develop' into external-source-fix

567662a

Integrate precice#724, precice#849, precice#854 and resolve TestSuite.py conflict (external sources + tolerance/skip_compare + run-before/run-after).

MakisH added 9 commits July 1, 2026 07:08

Revert change in Quickstart

4b30bce

Rearrange test suites

026816c

Convert a test case from git to zip

87803f6

Fix type of archive

a07e6cb

Remove external from release

bd711a6

Set a longer timeous

c9b1b6e

Set longer timeous

1a46abd

Split test suite

a81795e

Improve documentation

0b29acb

MakisH reviewed Jul 1, 2026

View reviewed changes

Address Jul 1 review on external test sources

b1f07e4

Allow mixed tutorials/external suites, fail fast on bad git refs, restore shell execute bits after ZIP extract, align fetch timeouts with GLOBAL_TIMEOUT, and document external cache behaviour in README.

PranjalManhgaye requested a review from MakisH July 1, 2026 16:22

Uh oh!

Uh oh!

Conversation

PranjalManhgaye commented Mar 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What this is about

What happens at runtime

Example

How it works

Files touched

testing

Checklist

Uh oh!

PranjalManhgaye commented Mar 1, 2026

Uh oh!

precice-bot commented Mar 8, 2026

Uh oh!

MakisH commented May 17, 2026

Uh oh!

PranjalManhgaye commented May 17, 2026

Uh oh!

PranjalManhgaye commented May 18, 2026

Uh oh!

MakisH left a comment

Choose a reason for hiding this comment

Uh oh!

PranjalManhgaye commented Jun 16, 2026

Uh oh!

MakisH left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

PranjalManhgaye commented Jun 24, 2026

Uh oh!

PranjalManhgaye commented Jun 30, 2026

Uh oh!

MakisH left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

PranjalManhgaye commented Mar 1, 2026 •

edited

Loading