Add X chr functionality#66
Conversation
Updated contact information and GitHub links.
Updated installation instructions and GitHub links in documentation.
Updated comments and mkdir use.
| pointEstimates[1, i] *= e | ||
| pointEstimates[2, i] *= 1 - e | ||
| pointEstimates[3, i] *= 1 - e | ||
| elif isXChrom: |
There was a problem hiding this comment.
I am struggling since I am doing this on the fly and can’t check. @XingerTang can you have a look too?
Also, we don’t need the isXChrom block for the second haplotype below?
There was a problem hiding this comment.
"ind.haplotypes[0][i] != 9" means this individual is female. If this individual is female, the segregation probability must be pointEstimates[0, i] *= e; pointEstimates[1, i] *= e; pointEstimates[2, i] *= 1 - e; pointEstimates[3, i] *= 1 - e. Don't need to change if sirehap0 != 9 and sirehap1 != 9 and sirehap0 != sirehap1: because it is not possible for isXChrom. If you think if isXChrom: is easy to understand, I can change elif to if.
gregorgorjanc
left a comment
There was a problem hiding this comment.
@AprilYUZhang great that you are pushing this functionality! I reviewed on the go without access to computer to look into the code, so I wasn’t sure in some places if the code is correct or not or it’s just my lack of deep understanding. @XingerTang your review would be also appreciated.
|
@AprilYUZhang I left some more comments at de2b6c8#r184944694 on your latest commit |
|
Github is behaving odd lately - you might need to toggle file by clicking the > symbol to hide and the unhide file contents to see the comments |
XingerTang
left a comment
There was a problem hiding this comment.
- Pre-commit not passed
- pytest failed on my end:
(phase) xtang3@S37-50MAQ05D AlphaImpute2 % pytest tests/functional_tests
======================================== test session starts =========================================
platform darwin -- Python 3.11.11, pytest-9.0.1, pluggy-1.6.0
benchmark: 5.2.3 (defaults: timer=time.perf_counter disable_gc=False min_rounds=5 min_time=0.000005 max_time=1.0 calibration_precision=10 warmup=False warmup_iterations=100000)
rootdir: /Users/xtang3/AlphaImpute2
configfile: pyproject.toml
plugins: benchmark-5.2.3
collected 2 items
tests/functional_tests/run_func_test.py F. [100%]
============================================== FAILURES ==============================================
_______________________________________________ test_1 _______________________________________________
def test_1():
"""basic functionality test with pedigree only mode"""
os.system(
"AlphaImpute2 -genotypes tests/functional_tests/test_1/genotypes.txt -pedigree tests/functional_tests/test_1/pedigree.txt -ped_only -phase_output -seg_output -out tests/functional_tests/outputs/test_1"
)
assert os.path.exists("tests/functional_tests/outputs/test_1.genotypes")
assert os.path.exists("tests/functional_tests/outputs/test_1.haplotypes")
assert os.path.exists("tests/functional_tests/outputs/test_1.segregation")
genotypes = read_file("tests/functional_tests/outputs/test_1.genotypes")
expected_genotypes = read_file("tests/functional_tests/test_1/true_genotypes.txt")
> assert genotypes == expected_genotypes
E AssertionError: assert [['1', 1.0, 2....0, 0.0, 0.0]] == [['1', 1.0, 2....0, 0.0, 0.0]]
E
E At index 2 diff: ['3', 1.0, 2.0, 0.0, 0.0, 1.0] != ['3', 0.0, 2.0, 0.0, 0.0, 1.0]
E Use -v to get more diff
tests/functional_tests/run_func_test.py:53: AssertionError
---------------------------------------- Captured stdout call ----------------------------------------
------------------------------------------
AlphaImpute2
------------------------------------------
Version: 0.0.3
Email: alphagenes.dev@gmail.com
Website: https://github.com/AlphaGenes
------------------------------------------
Reading in AlphaGenes format: tests/functional_tests/test_1/genotypes.txt
Read in: 0.01 seconds
Pedigree Imputation Only
------------------------------------------
Number of peeling cycles: 4
Final cutoff: 0.1
Imputation cycle 1
False 1
[[0.25 0.25 0.25 0.25 0.25]
[0.25 0.25 0.25 0.25 0.25]
[0.25 0.25 0.25 0.25 0.25]
[0.25 0.25 0.25 0.25 0.25]]
False -1
[[2.8239352e-01 3.0398920e-01 3.3998200e-01 3.9997000e-01 4.9994999e-01]
[2.1760647e-01 1.9601080e-01 1.6001801e-01 1.0003000e-01 4.9999999e-05]
[2.8239352e-01 3.0398920e-01 3.3998200e-01 3.9997000e-01 4.9994999e-01]
[2.1760647e-01 1.9601080e-01 1.6001801e-01 1.0003000e-01 4.9999999e-05]]
Peel down: 4.86 seconds
False -1
[[2.8239352e-01 3.0398920e-01 3.3998200e-01 3.9997000e-01 4.9994999e-01]
[2.1760647e-01 1.9601080e-01 1.6001801e-01 1.0003000e-01 4.9999999e-05]
[2.8239352e-01 3.0398920e-01 3.3998200e-01 3.9997000e-01 4.9994999e-01]
[2.1760647e-01 1.9601080e-01 1.6001801e-01 1.0003000e-01 4.9999999e-05]]
False 1
[[0.25 0.25 0.25 0.25 0.25]
[0.25 0.25 0.25 0.25 0.25]
[0.25 0.25 0.25 0.25 0.25]
[0.25 0.25 0.25 0.25 0.25]]
Peel up: 4.89 seconds
Imputation cycle 2
False 1
[[0.25 0.25 0.25 0.25 0.25]
[0.25 0.25 0.25 0.25 0.25]
[0.25 0.25 0.25 0.25 0.25]
[0.25 0.25 0.25 0.25 0.25]]
False -1
[[2.8239352e-01 3.0398920e-01 3.3998200e-01 3.9997000e-01 4.9994999e-01]
[2.1760647e-01 1.9601080e-01 1.6001801e-01 1.0003000e-01 4.9999999e-05]
[2.8239352e-01 3.0398920e-01 3.3998200e-01 3.9997000e-01 4.9994999e-01]
[2.1760647e-01 1.9601080e-01 1.6001801e-01 1.0003000e-01 4.9999999e-05]]
Peel down: 0.00 seconds
False -1
[[2.8239352e-01 3.0398920e-01 3.3998200e-01 3.9997000e-01 4.9994999e-01]
[2.1760647e-01 1.9601080e-01 1.6001801e-01 1.0003000e-01 4.9999999e-05]
[2.8239352e-01 3.0398920e-01 3.3998200e-01 3.9997000e-01 4.9994999e-01]
[2.1760647e-01 1.9601080e-01 1.6001801e-01 1.0003000e-01 4.9999999e-05]]
False 1
[[0.25 0.25 0.25 0.25 0.25]
[0.25 0.25 0.25 0.25 0.25]
[0.25 0.25 0.25 0.25 0.25]
[0.25 0.25 0.25 0.25 0.25]]
Peel up: 0.00 seconds
Imputation cycle 3
False 1
[[0.25 0.25 0.25 0.25 0.25]
[0.25 0.25 0.25 0.25 0.25]
[0.25 0.25 0.25 0.25 0.25]
[0.25 0.25 0.25 0.25 0.25]]
False -1
[[2.8239352e-01 3.0398920e-01 3.3998200e-01 3.9997000e-01 4.9994999e-01]
[2.1760647e-01 1.9601080e-01 1.6001801e-01 1.0003000e-01 4.9999999e-05]
[2.8239352e-01 3.0398920e-01 3.3998200e-01 3.9997000e-01 4.9994999e-01]
[2.1760647e-01 1.9601080e-01 1.6001801e-01 1.0003000e-01 4.9999999e-05]]
Peel down: 0.00 seconds
False -1
[[2.8239352e-01 3.0398920e-01 3.3998200e-01 3.9997000e-01 4.9994999e-01]
[2.1760647e-01 1.9601080e-01 1.6001801e-01 1.0003000e-01 4.9999999e-05]
[2.8239352e-01 3.0398920e-01 3.3998200e-01 3.9997000e-01 4.9994999e-01]
[2.1760647e-01 1.9601080e-01 1.6001801e-01 1.0003000e-01 4.9999999e-05]]
False 1
[[0.25 0.25 0.25 0.25 0.25]
[0.25 0.25 0.25 0.25 0.25]
[0.25 0.25 0.25 0.25 0.25]
[0.25 0.25 0.25 0.25 0.25]]
Peel up: 0.00 seconds
Imputation cycle 4
False 1
[[0.25 0.25 0.25 0.25 0.25]
[0.25 0.25 0.25 0.25 0.25]
[0.25 0.25 0.25 0.25 0.25]
[0.25 0.25 0.25 0.25 0.25]]
False -1
[[2.8239352e-01 3.0398920e-01 3.3998200e-01 3.9997000e-01 4.9994999e-01]
[2.1760647e-01 1.9601080e-01 1.6001801e-01 1.0003000e-01 4.9999999e-05]
[2.8239352e-01 3.0398920e-01 3.3998200e-01 3.9997000e-01 4.9994999e-01]
[2.1760647e-01 1.9601080e-01 1.6001801e-01 1.0003000e-01 4.9999999e-05]]
Peel down: 0.00 seconds
False -1
[[2.8239352e-01 3.0398920e-01 3.3998200e-01 3.9997000e-01 4.9994999e-01]
[2.1760647e-01 1.9601080e-01 1.6001801e-01 1.0003000e-01 4.9999999e-05]
[2.8239352e-01 3.0398920e-01 3.3998200e-01 3.9997000e-01 4.9994999e-01]
[2.1760647e-01 1.9601080e-01 1.6001801e-01 1.0003000e-01 4.9999999e-05]]
False 1
[[0.25 0.25 0.25 0.25 0.25]
[0.25 0.25 0.25 0.25 0.25]
[0.25 0.25 0.25 0.25 0.25]
[0.25 0.25 0.25 0.25 0.25]]
Peel up: 0.00 seconds
Core peeling cycles: 9.75 seconds
Heuristic Peeling: 11.28 seconds
Writing Out Results
------------------------------------------
Write out: 0.00 seconds
Full Program Run: 11.52 seconds
|
No tests are written for this code. Convert this PR to a draft. |
|
@AprilYUZhang Very nice plots! Could you try to implement this: #66 (comment) As that one is segregation-related. |
see 648b0cc |
|
I am slowly catching up. One thing I noticed in the summary from @XingerTang are high uncalled rates with X chr, say |
|
@AprilYUZhang that accuracy plot is very effective!!!!! How did you produce it? @XingerTang would that be a useful standard summary for the development!? Looking at the stats I can see clear expected difference, higher correlation in males, so that is good to see. Generation 2 is odd though, but I think you are looking into that. can I check what you do with any potential 9s in the output!? Hmm, what are you actually correlating here - allele dosage or called genotypes or genotype probabilities? All are valid targets, wit allele dosages being the most straightforward as there are no 9s in there. |
|
@AprilYUZhang Thank you for your update. It is a surprise to see that the accuracy is not changed by your updated code. Maybe the same limits have already been applied in other places in the code. Nevertheless, I now find that I cannot convince myself of the reason why the accuracy of segregation of females should be lower than that of males. Both females and males only have two options out of four. They should have the same base probability (a random choice should yield an accuracy of 0.5). For the X chromosome, the uncertainty always comes from the mother's side. And both males and females have a mother. So I would expect no difference in the accuracy of segregation. I tried different examples to see if the segregation is correctly inferred, and I think everything looks as expected. Later, I noticed that the correlation favours the evaluation of the segregation probabilities of the autosomal chromosome, For
This is not the sole example that "similar" segregation probabilities have higher correlation accuracy in the autosomal case. I suspect that the fact that the sex chromosome has only two options actually makes it less likely to achieve higher correlation accuracy than the autosomal chromosome. The later the generation, the more confident we are with the inheritance pattern, so a lot of segregation probabilities for X chromosomes are I think what you did earlier for the |
@gregorgorjanc The uncalled rate is not actually the uncalled rate; it is the proportion of males who always have their paternal haplotype written as |
These plots are nice and informative, maybe a better way to display than the tables. But the correlation is not the best metric here (my above comment explains a little about this). Here we are checking the correlation accuracy of segregation probabilities. So far, I think |
|
@AprilYUZhang, could you also add a format checker for the pedigree file? I input a pedigree file with no sex specified with |
b945593 to
0301d3a
Compare
|
I updated the accuracy test report table; it should now look like the following: Only the population values are preserved. For segregation probs, we only count the individuals that have meaningful values, that is, from generation 3 to generation 5. For the new metric bits can be improved:
|
|
|
||
| def generateGenoProbs(): | ||
| global geno_probs | ||
| error = 0.0001 |
There was a problem hiding this comment.
refactor the code so that the value assignment follows the order of 4x4x4
|
|
||
| def generateGenoProbs_Xchr_male(): | ||
| global geno_probs_Xchr_male | ||
| error = 0.0001 |
There was a problem hiding this comment.
refactor the code to follow the order as the autosomal
| (10, 10, 10, 4), 0.25, dtype=np.float32 | ||
| ) # Because 9 indexing for missing. | ||
| generateGenoProbs() | ||
| geno_probs_Xchr_male = np.full((10, 10, 10, 4), 0.25, dtype=np.float32) |
There was a problem hiding this comment.
We start with 0.25 and then in the next function, we change some of these but not all of them, do we ever wrongly use the unmodified 0.25s.
|
Updated accuracy report table:
|
|
Replaced the |
XingerTang
left a comment
There was a problem hiding this comment.
I reviewed the population imputation code and left some comments.
| ind.current_haplotypes[0][:] = pat_hap | ||
| ind.current_haplotypes[1][:] = mat_hap | ||
| if ind.isXChr and ind.sex == 0: | ||
| ind.current_haplotypes[0][:] = 0 |
There was a problem hiding this comment.
I understand the values won't be used, but is there a reason why we assign 0 instead of 9 here?
| @@ -529,10 +539,17 @@ def haplib_sample_alt(sample, bw_library, ind, random_samples): | |||
| tmp_mat_prob = hap_lib_prop | |||
| scale = rec_rate * rec_rate | |||
|
|
|||
| geno_probs[i, 0] = tmp_pat_prob[0] * tmp_mat_prob[0] * scale | |||
| geno_probs[i, 1] = tmp_pat_prob[0] * tmp_mat_prob[1] * scale | |||
| geno_probs[i, 2] = tmp_pat_prob[1] * tmp_mat_prob[0] * scale | |||
| geno_probs[i, 3] = tmp_pat_prob[1] * tmp_mat_prob[1] * scale | |||
| if isXChr and sex == 0: | |||
| # Force pat=0: only j=0 and j=1 are possible | |||
| geno_probs[i, 0] = tmp_mat_prob[0] * scale # pat fixed to 0 | |||
| geno_probs[i, 1] = tmp_mat_prob[1] * scale # pat fixed to 0 | |||
There was a problem hiding this comment.
For X chromosomes in males, there could possibly be only one recombination on its maternal haplotype.
It shouldn't be in this loop of for i in range(4), which iterates over four possible recombination states. And the value of scale multiplied by the tmp_mat_probcan only take values from either 1 - rec_rate or rec_rate.
Similarly, for the X chromosome of female individuals, its paternal haplotype cannot be recombined, so there should only be two cases to iterate through as well.
| elif count1 > 0: | ||
| mat_hap[i] = 1 | ||
|
|
||
| pat_hap = np.full(nLoci, 0, dtype=np.int8) |
There was a problem hiding this comment.
Same as the other comment, I understand the values won't be used, but is there a reason why we assign 0 instead of 9 here?
| mat_hap[i] = 1 | ||
|
|
||
| pat_hap = np.full(nLoci, 0, dtype=np.int8) | ||
| return pat_hap, mat_hap |
There was a problem hiding this comment.
The get_consensus for the autosomal chromosome assigns the genotype value as well.
| for i in range(nLoci): | ||
| count0 = 0 | ||
| count1 = 0 | ||
| for j in range(nHaps): | ||
| val = samples[j].haplotypes[1][i] | ||
| if val == 0: | ||
| count0 += 1 | ||
| elif val == 1: | ||
| count1 += 1 | ||
|
|
||
| if count0 > count1: | ||
| mat_hap[i] = 0 | ||
| elif count1 > 0: | ||
| mat_hap[i] = 1 |
There was a problem hiding this comment.
The original count_regional_rec returns rec_scores that are based on the region of the target marker, and later, the majority vote is based on the comparison against the values of rec_scores. But the calculation of rec_scores is absent here.
There was a problem hiding this comment.
Pull request overview
This PR adds X-chromosome (“-x_chr”) support across core imputation/phasing/peeling logic, along with new functional + accuracy tests and a substantial documentation refresh to describe usage and file formats.
Changes:
- Add sex-chromosome-specific handling for males (hemizygous) across phasing consensus, BW reference-library construction, genotype/phase alignment, and heuristic peeling segregation math.
- Add new functional test fixtures for sex-chromosome scenarios and extend accuracy tests to run X-chromosome simulations.
- Improve CLI/docs/examples (new Sphinx docs, version flag handling, example scripts) and expand CI Python-version matrix.
Reviewed changes
Copilot reviewed 49 out of 54 changed files in this pull request and generated 12 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/functional_tests/test_sex/true-sex.with_recom.haplotypes | Adds expected haplotype truth set for X-chr functional scenario (with recombination). |
| tests/functional_tests/test_sex/true-sex.with_recom.genotypes | Adds expected genotype truth set for X-chr functional scenario (with recombination). |
| tests/functional_tests/test_sex/true-sex.with_recom_missing.haplotypes | Adds expected haplotype truth set for X-chr functional scenario (with recombination + missingness). |
| tests/functional_tests/test_sex/true-sex.with_recom_missing.genotypes | Adds expected genotype truth set for X-chr functional scenario (with recombination + missingness). |
| tests/functional_tests/test_sex/true-sex.no_recom.haplotypes | Adds expected haplotype truth set for X-chr functional scenario (no recombination). |
| tests/functional_tests/test_sex/true-sex.no_recom.genotypes | Adds expected genotype truth set for X-chr functional scenario (no recombination). |
| tests/functional_tests/test_sex/true-sex.no_recom_missing.haplotypes | Adds expected haplotype truth set for X-chr functional scenario (no recombination + missingness). |
| tests/functional_tests/test_sex/true-sex.no_recom_missing.genotypes | Adds expected genotype truth set for X-chr functional scenario (no recombination + missingness). |
| tests/functional_tests/test_sex/ped_file-with_recom.txt | Adds pedigree inputs (with recombination) including sex coding. |
| tests/functional_tests/test_sex/ped_file-with_recom_missing.txt | Adds pedigree inputs (with recombination + missingness) including sex coding. |
| tests/functional_tests/test_sex/ped_file-no_recom.txt | Adds pedigree inputs (no recombination) including sex coding. |
| tests/functional_tests/test_sex/ped_file-no_recom_missing.txt | Adds pedigree inputs (no recombination + missingness) including sex coding. |
| tests/functional_tests/test_sex/geno_file-with_recom.txt | Adds genotype inputs (with recombination) for X-chr functional test. |
| tests/functional_tests/test_sex/geno_file-with_recom_missing.txt | Adds genotype inputs (with recombination + missingness) for X-chr functional test. |
| tests/functional_tests/test_sex/geno_file-no_recom.txt | Adds genotype inputs (no recombination) for X-chr functional test. |
| tests/functional_tests/test_sex/geno_file-no_recom_missing.txt | Adds genotype inputs (no recombination + missingness) for X-chr functional test. |
| tests/functional_tests/run_func_test.py | Adds parametrized X-chr functional test invoking AlphaImpute2 with -x_chr. |
| tests/accuracy_tests/sim_for_alphapeel_accu_test/X_chr_ped_file.txt | Adds simulated pedigree file for X-chr accuracy benchmarking. |
| tests/accuracy_tests/run_accu_test.py | Extends accuracy benchmarking to support X-chr mode and new metrics output format. |
| src/alphaimpute2/Imputation/PhasingObjects.py | Adds X-chr male consensus logic and plumbs sex/isXChr into consensus computation. |
| src/alphaimpute2/Imputation/ParticlePhasing.py | Updates reference library and phasing to handle X-chr males (single haplotype, special consensus and haplotype assignment). |
| src/alphaimpute2/Imputation/ParticleImputation.py | Passes x_chr into reference-library creation during imputation. |
| src/alphaimpute2/Imputation/ImputationIndividual.py | Adds sex and isXChr to jit views; adds X-chr male penetrance/genotype-prob handling. |
| src/alphaimpute2/Imputation/Imputation.py | Adds X-chr-specific genotype/phase fill/align helpers. |
| src/alphaimpute2/Imputation/Heuristic_Peeling.py | Adds X-chr segregation/anterior/posterior handling and specialized smoothing/transmission logic. |
| src/alphaimpute2/alphaimpute2.py | Adds -x_chr and -geno_prob CLI flags, improves version handling, and updates output writing. |
| pyproject.toml | Updates author email and adjusts supported Python range metadata. |
| example/simple_example/simple_pedigree.txt | Adds simple example pedigree input file to repo. |
| example/simple_example/simple_genotype.txt | Adds simple example genotype input file to repo. |
| example/run_examples.sh | Makes example runner more robust/documented and adds shebang. |
| example/data/X_chr/true_simple_segregation.txt | Adds expected segregation truth for X-chr example. |
| example/data/X_chr/true_simple_haplotype.txt | Adds expected haplotype truth for X-chr example. |
| example/data/X_chr/true_simple_genotype.txt | Adds expected genotype truth for X-chr example. |
| example/data/X_chr/simple_X_chr.haplotypes | Adds example X-chr phased haplotypes file. |
| example/data/X_chr/simple_X_chr.genotypes | Adds example X-chr genotypes file. |
| example/data/X_chr/simple_pedigree.txt | Adds example X-chr pedigree file including sex column. |
| example/data/X_chr/simple_genotype.txt | Adds example X-chr genotype file including missingness patterns. |
| example/check_accuracy.r | Improves formatting/output of R accuracy script and adds shebang. |
| docs/source/usage.rst | Adds new detailed usage/file-format documentation including X-chr encoding guidance. |
| docs/source/introduction.rst | Adds new introduction page for Sphinx docs. |
| docs/source/index.rst | Refactors docs index into toctree-driven structure. |
| docs/source/get_started.rst | Adds getting started/install/build/run guidance. |
| docs/source/conf.py | Adds global Sphinx substitution for ` |
| docs/source/changelog.rst | Adds initial changelog scaffold. |
| docs/source/algorithm.rst | Adds algorithm overview documentation (peeling terms, segregation, etc.). |
| conftest.py | Reworks how accuracy metrics are summarized at end of pytest run. |
| .github/workflows/tests.yml | Expands CI matrix to run across multiple Python versions. |
| .github/ISSUE_TEMPLATE/task-issue-template.md | Adds task issue template. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| for row in file_data: | ||
| value = row["value"] | ||
| bar_length = int( | ||
| max_length * (value - min_value) / (max_value - min_value) | ||
| ) |
| def run_command(cmd): | ||
| exit_code = os.system(cmd) | ||
| if exit_code == 11: | ||
| import glob | ||
|
|
||
| outputs = glob.glob(os.path.join(output_path, "test.*")) | ||
| if outputs: | ||
| return # output was written, crash was in cleanup only | ||
| assert exit_code == 0, f"AlphaImpute2 failed with exit code {exit_code}" |
| if g == 2: | ||
| raise ValueError( | ||
| f"Unexpected genotype value for male in sex chr: {g}" | ||
| ) |
| if g == 2: | ||
| raise ValueError( | ||
| f"Unexpected genotype value for male in sex chr: {g}" | ||
| ) |
| if isXChr: | ||
| if child.sex == 0: | ||
| if geno == 2: | ||
| print( | ||
| f"Warning: No possible genotype 2 in male at position {i} of {child.idn} " | ||
| ) | ||
| if geno == 1 and hap1 == 0: | ||
| print( | ||
| f"Warning: No possible genotype 1 and maternal haplotype is 0 in male at position {i} of {child.idn} " | ||
| ) | ||
| return logGenotypeSegregationTensor_XYChrom[seg0, seg1, hap0, hap1, geno] | ||
| else: | ||
| return logGenotypeSegregationTensor_XXChrom[seg0, seg1, hap0, hap1, geno] |

Issue #47