add many more tests and R ports

[danielnachun]

This PR should cover almost all testable notebooks save for twoone:

• STAR because the executable is broken on macOS (I'm still working on this)
• gregor because the database seems to be missing from the MWE

[jaempawi]

Tested this branch (015e513) end-to-end on the BU SCC (RHEL 8, glibc 2.28), 8 cores via the batch scheduler.

Results: test_scripts 60/60 green; test_notebooks 134/139.

Three findings, all environment-level rather than test bugs:

1. pixi.toml needs a newer pixi than the error suggests. The new inline form platforms = [{ platform = "linux-64", glibc = "2.17" }, "osx-arm64"] fails to parse in pixi 0.57.0 with a bare "expected a string, found table" pointing at line 5, which is a confusing failure mode. 0.75.0 parses it fine. Worth documenting a minimum pixi version in the README, since CI always gets the latest via setup-pixi and won't catch this.

2. Three tools can't run on glibc 2.28. This affects real analyses, not just tests. test_rnaseqc_call, test_grm_loco_workflow and test_rsem_index all fail at dynamic-link time before executing any pipeline logic. objdump shows rnaseqc, gcta and rsem-extract-reference-transcripts each require GLIBC_2.38.

Note the manifest declares glibc = "2.17", so these builds are being selected despite advertising a floor they don't honor (likely upstream packaging metadata). Either way, the protocol currently can't run RNA-SeQC, GCTA or RSEM on RHEL 8 clusters from this environment.

3. The preprocessCore threading workaround is needed outside CI too. test_sesame and test_minfi fail in quantile normalization with ERROR; return code from pthread_create() is 22. Minimal reproducer: Rscript -e 'library(preprocessCore); normalize.quantiles(matrix(rnorm(100), 10))'. Rebuilding with configure.args = "--disable-threading" makes both pass immediately (verified: 2 passed). Since ci.yml flags that rebuild as CI-only and unsafe for production, it'd be worth either pinning a build that works or documenting the constraint. Right now methylation calling is broken on this platform out of the box.

[danielnachun]

@jaempawi I identified the cause for the package issues you identified here and they should all be fixed now. I was wondering if the preprocessCore issue would show up in a real environment and since it did, I went ahead and repackaged it on my channel. It should work now because I remove the workaround in the CI commands and it passed here.

I will update the README to reflect that a newer pixi is needed. Thankfully pixi can be updated with pixi self-update, which I will add to the README.

Please go ahead and retest everything now and let's see if it all works this time.

[jaempawi]

Retested on SCC (glibc 2.28), fresh clone with a cleared cache. Your fixes work. gcta runs now, no pthread_create errors anywhere, and test_sesame/test_minfi pass. test_scripts 61/61.

One heads-up on the packages: you rebuilt over the same build strings, so anyone with a warm rattler cache will keep the old broken binaries and see no change. I only picked up the fix by forcing a fresh PIXI_CACHE_DIR. Worth a note in the PR, and bumping build numbers next time would keep the lock's hashes honest.

Under the batch scheduler I still get 7 failures + 2 errors, but they're not yours . All of them are ValueError("illegal environment variable name") from subprocess. SCC exports its shell functions as BASH_FUNC_module%%, BASH_FUNC_ml%%, BASH_FUNC_which%%, and the %% makes the name invalid. I tried unset -f module ml which in the job script and it made no difference. They come back before pytest runs, so it can't be worked around from outside. tests/helpers/sos_runner.py would need to filter the inherited env to well-formed keys. Happy to send a small PR if useful.

Documentation items from the review:

1. apa_calling.ipynb — the four sos run examples still pass --bam-dir, --chr-prefix, --dapars-path; none are declared parameters now. --chr-prefix looks like a real feature loss rather than a doc fix — the old step passed "F" if chr_prefix else "T" to Dapars2_Multi_Sample.py and I don't see prefix handling in dapars.R.

2. ems_prediction.ipynb — [predict] lost its output: and task:, so no signature tracking and no HPC submission, unlike every other workflow. The Output table still documents predictions_weighted_model_chr2.tsv.

3. intact.ipynb — hardcodes Rscript code/script/pecotmr_integration/intact.R instead of ${modular_script_dir}/..., and doesn't declare modular_script_dir, so it only works from the repo root.

4. code/snakemake/README.md — the py_compile snippet still lists three .py files this PR deletes.

5. Container support is inconsistent — removed from 7 pages, partially from 3 more (59→52 notebooks declaring it). Deliberate, or fallout from moving bodies into code/script?

6. Stale text: the <!-- BiCV requires the APEX tool... skipped --> comment (apex is declared and BiCV passes), methylation_calling's cached -h showing /opt/cross_reactive_probe..., and genotype_formatting's "npz format" FIXME though the output is .rds.

These don't show up in CI because the tests call the R/bash workers directly rather than the documented sos run commands.

Also, test_gwas_rss_plot and test_rss_analysis::test_gwas_finemapping fail here, but this branch doesn't touch any of the files or fixtures they use. They look like fallout from the susie_rss work in #1395/#1396, so main may be red on its own.

[danielnachun]

Retested on SCC (glibc 2.28), fresh clone with a cleared cache. Your fixes work. gcta runs now, no pthread_create errors anywhere, and test_sesame/test_minfi pass. test_scripts 61/61.

One heads-up on the packages: you rebuilt over the same build strings, so anyone with a warm rattler cache will keep the old broken binaries and see no change. I only picked up the fix by forcing a fresh PIXI_CACHE_DIR. Worth a note in the PR, and bumping build numbers next time would keep the lock's hashes honest.

Under the batch scheduler I still get 7 failures + 2 errors, but they're not yours . All of them are ValueError("illegal environment variable name") from subprocess. SCC exports its shell functions as BASH_FUNC_module%%, BASH_FUNC_ml%%, BASH_FUNC_which%%, and the %% makes the name invalid. I tried unset -f module ml which in the job script and it made no difference. They come back before pytest runs, so it can't be worked around from outside. tests/helpers/sos_runner.py would need to filter the inherited env to well-formed keys. Happy to send a small PR if useful.

This sounds like a cluster-specific issue we can't easily work around - I'll consider this for a follow up PR for now.

Documentation items from the review:

1. apa_calling.ipynb — the four sos run examples still pass --bam-dir, --chr-prefix, --dapars-path; none are declared parameters now. --chr-prefix looks like a real feature loss rather than a doc fix — the old step passed "F" if chr_prefix else "T" to Dapars2_Multi_Sample.py and I don't see prefix handling in dapars.R.

This was an oversight, I will fix this.

2. ems_prediction.ipynb — [predict] lost its output: and task:, so no signature tracking and no HPC submission, unlike every other workflow. The Output table still documents predictions_weighted_model_chr2.tsv.

This workflow was incomplete originally - I will ask on Slack if there is a plan to finish this.

3. intact.ipynb — hardcodes Rscript code/script/pecotmr_integration/intact.R instead of ${modular_script_dir}/..., and doesn't declare modular_script_dir, so it only works from the repo root.

This was a mistake and will be fixed.

4. code/snakemake/README.md — the py_compile snippet still lists three .py files this PR deletes.

I haven't touched the snakemake side of things (I want to replace it with nextflow anyway), so we can ignore this.

5. Container support is inconsistent — removed from 7 pages, partially from 3 more (59→52 notebooks declaring it). Deliberate, or fallout from moving bodies into code/script?

I will check this - we are dropping container support if possible - many of these references are probably to notebooks that will be archived.

6. Stale text: the <!-- BiCV requires the APEX tool... skipped --> comment (apex is declared and BiCV passes), methylation_calling's cached -h showing /opt/cross_reactive_probe..., and genotype_formatting's "npz format" FIXME though the output is .rds.

This will be easy to fix.

These don't show up in CI because the tests call the R/bash workers directly rather than the documented sos run commands.

Also, test_gwas_rss_plot and test_rss_analysis::test_gwas_finemapping fail here, but this branch doesn't touch any of the files or fixtures they use. They look like fallout from the susie_rss work in #1395/#1396, so main may be red on its own.

I will also fix these since this is all supposed to work.