Skip to content

refactor(lcao): decouple basis indexing from block-cyclic layouts - #8295

Open
kirk0830 wants to merge 64 commits into
deepmodeling:developfrom
kirk0830:pr/06-explicit-layout-v2
Open

kirk0830 wants to merge 64 commits into
deepmodeling:developfrom
kirk0830:pr/06-explicit-layout-v2

Conversation

@kirk0830

@kirk0830 kirk0830 commented Nov 7, 2026 •

Copy link
Copy Markdown
Collaborator

Key design: the atom basis function layout (ityp, iat, l, izeta, m) := mu is not necessary to be coupled with the block-cyclic distribution mechanism: one is about the matrix layout, the other is about how matrix is distributed

Summary

  • Continue the longer Parallel_Orbitals dissolution plan by passing the block-cyclic basis and matrix layouts explicitly instead of relying on implicit inheritance or forwarding.
  • Introduce BasisFlattenedIndexing and AtomBasisView as a single basis-indexing boundary shared by HContainer, LCAO operators, and their consumers.
  • Migrate Hamiltonian, TDDFT, LR/BSE, density-matrix, HS I/O, solver, and MD test paths to use explicit para_2d() layouts.
  • Remove the obsolete per-element LCAO offset-vector path.
  • Fix issue Rename block_cyclic_distribution and normalize its BLACS constants #8299.

The PR-scoped diff is limited to this refactor. Upstream synchronization added no CSR-reader, workflow, documentation, or repository-local agent-memory changes to the PR.

Behavior

No INPUT, output-format, numerical, or public workflow behavior changes. Documentation updates are not required because this is an internal ownership and interface refactor; the affected contracts are documented inline.

Regression coverage

The first revision of this PR retained a duplicate inline nspin=4 Mulliken path that bypassed the already corrected collect_MW_nspin4 helper. Existing unit tests exercised only the helper, so they did not cover the production Output_Mulliken::collect_MW() path where the regression occurred. Integration and CUDA CI caught the resulting spin2/spin4 sign reversal.

The new OutputMullikenSpinTest.Nspin4ProductionPathPreservesMySign test calls Output_Mulliken::collect_MW() directly. It fails with the old implementation (My=-1 instead of +1) and passes after restoring the corrected helper path.

Verification

The final rename and interface-cleanup state was verified at adbb91943 with abacus-dev/1.0, GNU 11.4.0, and OMP_NUM_THREADS=1:

cmake --build build -j 8
ctest --test-dir build --output-on-failure \
  -R "MODULE_BASE_parallel_blkcyc|MODULE_LCAO_tddft_boundary_fix_test|MODULE_IO_mulliken_test|MODULE_IO_output_mat_sparse_test|MODULE_LR_state_ovlp|MODULE_LR_state_track|MODULE_LR_spectrum_mo_output_test"

cmake --build build-exx -j 8
ctest --test-dir build-exx --output-on-failure \
  -R "MODULE_BASE_parallel_blkcyc|MODULE_LCAO_tddft_boundary_fix_test|MODULE_IO_mulliken_test|MODULE_IO_output_mat_sparse_test|MODULE_LR_state_ovlp|MODULE_LR_state_track|MODULE_LR_spectrum_mo_output_test"

git diff --check upstream/develop..HEAD
python3 tools/03_code_analysis/agent_governance_check.py \
  --base upstream/develop --head HEAD --format text
  • adbb91943: default CPU+MPI+LCAO+testing build [100%]; 9/9 focused tests passed, including MODULE_BASE_parallel_blkcyc and its four-rank MPI test.
  • adbb91943: LibRI/LibXC build [100%]; the same 9/9 focused tests passed.
  • 1fd814ea2: pre-rename production integration case scf_out_mul_spin4 passed with both 1 and 4 MPI ranks.
  • adbb91943: governance check has no blockers and git diff --check is clean.
  • CI at adbb91943 covers CUDA, Makefile/Intel builds, no-LCAO configurations, and the remaining toolchain matrices.

Auto-merge is intentionally disabled.

kirk0830 and others added 30 commits October 28, 2026 06:34
Purely additive: no call site changes, inheritance still in place.
para_2d() (const + non-const) returns *this while Parallel_Orbitals derives
from Parallel_2D, giving the two escape hatches a const getter cannot serve
(LR_Util::setup_2d_division mutation, test-side descriptor writes).
get_blacs_ctxt() completes the inherited getter family, which covers the
other seven public data members.
…bitals, migrate module_operator_lcao call sites
… (getter-equiv, derived-layer sync, value-semantics static_assert, MPI+serial dual coverage)
…arallel_2D to using alias, full-repo getter-ization

- new class BlockCyclicDistributionInformation owns all Parallel_2D responsibilities:
  data members (snake_case + trailing underscore), initialize/set/set_serial/set_coord,
  move semantics, getter family, global_to_local_*, owner_*, owns_index
- Parallel_2D demoted to 'using Parallel_2D = BlockCyclicDistributionInformation;' (removal in S5)
  with DEPRECATED-ALIAS/DELETION-COMMIT markers
- all legacy Parallel_2D direct field accesses replaced by getters (A0 burn-down = 0)
- dual build (MPI+serial) zero error; parallel_orbitals + composition tests green;
  D0 four-case byte-compare ok=36 diff=0
…Information

- Parallel_Orbitals member Parallel_2D para_2d_ -> BlockCyclicDistributionInformation para_2d_
- para_2d() accessor return types updated to BlockCyclicDistributionInformation& / const&
- composition test binds para_2d() to const BlockCyclicDistributionInformation&
- Parallel_2D alias untouched (deletion in S5); setup_2d_division signature unchanged
- dual build zero error; MPI ctest 7/7, serial ctest 3/3; D0 four cases byte-identical (ok=36 diff=0)
- all content from codex S2 run
… / gtest / legacy make / pyabacus) - zero changes needed
…ll references to BlockCyclicDistributionInformation

- delete parallel_2d.{h,cpp} (alias header + empty TU) and parallel_2d_test.{cpp,sh}
- rewrite ~127 files: Parallel_2D whole-word refs -> BlockCyclicDistributionInformation, drop parallel_2d.h includes
- burn-down == 0: git grep Parallel_2D / parallel_2d.h / build entries all zero
- build verification performed by computer-agent (serial+MPI zero-error, related tests green, D0 4-case byte-diff clean)
kirk0830 and others added 9 commits November 5, 2026 08:52
Use the storage-order local extent as the leading dimension, and validate a non-serial BLACS descriptor against its column-major layout independently. Skip descriptor value checks for set_serial layouts.

Cover a wide row-major block by filling all entries before reading them back, register the block-cyclic test on four MPI ranks, and consistently name the leading dimension ld in code introduced by this branch.
Rename BlockCyclicDistributionInformation to BlockCyclicDistribution and DistributedNDArray to DistNDArray, including their headers, tests, build targets, and includes. Use compact ScaLAPACK descriptor element names such as DESC_ICTXT and DESC_LLD.

Pure identifier rename; no behavior change.
Use size_t accumulators for NDArray shape products and strides, with a reshape regression case whose product is 2^32 + 6. Remove the misleading DistNDArray local_block() view so column-major callers use at_local() or explicit major_order()/ld()/data().

Use i and j for scalar matrix indices in the new block-cyclic tests and adapter, leaving row and column for axis or object concepts.
DistNDArray has no production caller and was scheduled for removal in the Matrix2d migration. Keep the independent NDArray size_t fix, but do not land a speculative storage-order API that would be deleted by the next stage.
…ayout-v2

# Conflicts:
#	source/source_base/CMakeLists.txt
#	source/source_base/test_parallel/CMakeLists.txt
#	source/source_basis/module_ao/parallel_orbitals.cpp
#	source/source_basis/module_ao/parallel_orbitals.h
#	source/source_basis/module_ao/test/parallel_orbitals_test.cpp
#	source/source_esolver/esolver_ks_lcao_tddft.cpp
#	source/source_esolver/esolver_lr_lcao_bse.cpp
#	source/source_esolver/esolver_lr_orb.cpp
#	source/source_hamilt/module_hcontainer/hcontainer.h
#	source/source_hsolver/hsolver_lcao.cpp
#	source/source_hsolver/kernels/cuda/diag_cusolvermp.cu
#	source/source_hsolver/parallel_k2d.cpp
#	source/source_io/module_current/td_current_io_comm.cpp
#	source/source_io/module_hs/lat_r_csr.cpp
#	source/source_io/module_hs/pos_op_writer.cpp
#	source/source_io/module_hs/unittests/CMakeLists.txt
#	source/source_io/unittests/test_read_wfc_nao.cpp
#	source/source_io/unittests/test_write_wfc_nao.cpp
#	source/source_lcao/lcao_init_basis.cpp
#	source/source_lcao/module_bse/bse_util.cpp
#	source/source_lcao/module_bse/hamilt_bse.cpp
#	source/source_lcao/module_dftu/unittests/CMakeLists.txt
#	source/source_lcao/module_lr/lr_spectrum_velocity.cpp
#	source/source_lcao/module_lr/state_ovlp.h
#	source/source_lcao/module_lr/utils/spectrum_mo.h
#	source/source_lcao/module_operator_lcao/op_exx_lcao.cpp
#	source/source_lcao/module_operator_lcao/td_ekinetic_lcao.cpp
#	source/source_lcao/module_ri/module_exx_symmetry/test/symm_rotation_test.cpp
#	source/source_lcao/module_ri/module_exx_symmetry/test/test_symm_rotation.cpp
#	source/source_lcao/module_ri/ri_2d_comm.cpp
#	source/source_lcao/module_ri/ri_2d_comm.hpp
#	source/source_lcao/module_rpa/rpa_lri_eigenvector.cpp
#	source/source_lcao/module_rpa/rpa_lri_mpi_layout.cpp
#	source/source_lcao/module_rpa/rpa_lri_velocity.cpp
#	source/source_lcao/module_rpa/rpa_lri_wfc_mpi.cpp
#	source/source_lcao/module_rt/band_energy.cpp
#	source/source_lcao/module_rt/boundary_fix.cpp
#	source/source_lcao/module_rt/evolve_elec.cpp
#	source/source_lcao/module_rt/evolve_psi.cpp
#	source/source_lcao/module_rt/force_rt_overlap.cpp
#	source/source_lcao/module_rt/middle_hamilt.cpp
#	source/source_lcao/module_rt/norm_psi.cpp
#	source/source_lcao/module_rt/propagator_cn2.cpp
#	source/source_lcao/module_rt/propagator_etrs.cpp
#	source/source_lcao/module_rt/propagator_taylor.cpp
#	source/source_lcao/module_rt/solve_propagation.cpp
#	source/source_lcao/module_rt/test/norm_psi_test.cpp
#	source/source_lcao/module_rt/test/test_evolve_psi.cpp
#	source/source_lcao/module_rt/test/upsi_test1.cpp
#	source/source_lcao/module_rt/test/upsi_test2.cpp
#	source/source_lcao/module_rt/test/upsi_test3.cpp
#	source/source_lcao/module_rt/upsi.cpp
#	source/source_md/unittests/CMakeLists.txt
#	source/source_md/unittests/test_md_base.cpp
@kirk0830 kirk0830 changed the title refactor(lcao): pass block-cyclic layouts explicitly refactor(lcao): decouple basis indexing from block-cyclic layouts Nov 7, 2026

@ABACUS-PR-BOT ABACUS-PR-BOT left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The new governance regression expects an unimplemented checker rule and needs to be aligned with the checker. The other inline notes are non-blocking cleanup.

Static review at 1fd814e: git diff --check and the governance checker completed successfully (warnings only). Builds, numerical/runtime tests and CI diagnostics were not run.

@kirk0830

kirk0830 commented Nov 7, 2026 •

Copy link
Copy Markdown
Collaborator Author

@ABACUS-PR-BOT All problems have been solved, could you review again?

@AsTonyshment AsTonyshment left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM overall!

Just a heads-up: our current guidance on forward declarations is a bit contradictory. The ABACUS C++ guide recommends avoiding them, while agent_governance_check.py and even the review bot recommend the opposite.

This is also a debated topic in software engineering, with major projects taking different approaches. For example, Google generally discourages forward declarations, while LLVM encourages minimizing header inclusions and using forward declarations where appropriate. It's essentially a trade-off between maintainability (e.g., explicit dependencies and avoiding fragile declarations) and compile-time efficiency (e.g., reduced header coupling and unnecessary recompilation).

We're planning to put together an updated version of the ABACUS development guidelines to clear up inconsistencies like this. No need to change anything here over this, though :) I'm just pointing it out, and we can leave the discussion of whether forward declarations are preferable for another time.

@kirk0830

kirk0830 commented Nov 8, 2026 •

Copy link
Copy Markdown
Collaborator Author

@AsTonyshment Thanks for your review.
As for the forward declaration and header only, I can share part of my opinions here: personally, I use forward declaration with much care, and I think it should not be allowed in a wide range presently, because it may hide the real design problem, especially hide the coupling between modules. Even for now AI can help me code many, I still spend much time on answering questions from myself about:

  1. What is this name indicates?
  2. What it (a class/variable/struct/...) IS?

As for the compile time management, I am not sure about the preference of other developers, so I cannot give a clear suggestion.

C++ provides many language features allowing developers coding in many different ways, but maintainers should be cautious on this feature before the code is "dry" or atomic enough.

Cheers,
Yike

@ABACUS-PR-BOT ABACUS-PR-BOT left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Both verification filters omit the renamed layout tests; the inline note identifies the update.

Static incremental review at adbb919: full and incremental git diff --check passed. Builds, runtime tests, numerical probes and CI diagnostics were not run; the reported test counts remain author-reported.

Comment thread source/source_base/CMakeLists.txt Resolved
@kirk0830

kirk0830 commented Nov 8, 2026

Copy link
Copy Markdown
Collaborator Author

@AsTonyshment update: to enhance the readability of the forward declaration I have to make at the intermediate stage of the big refactor, I append the inline comments to point to the place where the classes are really declared. In the next PR, the number of forward declaration used will be decreased significantly because of the introduction of Matrix2d container.

void save_lat_r(std::ofstream& ofs,
const SparseRBlock<T>& XR,
const Parallel_Orbitals& pv,
const BlockCyclicDistribution& layout,

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not only the file name, I would suggest when using parallel scheme, the input parameter name starts from Parallel_*

@kirk0830 kirk0830 Nov 9, 2026 •

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, but what about the naming of this scheme? Will Parallel_Block_Cyclic be a good choice? Or maybe there is also a rule now about confining the number of words to be not more than two, such as Parallel_Blkcyclic/Parallel_BlkCyclic/Parallel_Blkcyc/Parallel_BlkCyc ?

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Rename block_cyclic_distribution and normalize its BLACS constants

4 participants