From c7c747fb3f906d13024ff17bb8937cb0d4039585 Mon Sep 17 00:00:00 2001 From: Liang Zhen Date: Wed, 12 Aug 2026 16:05:06 +0800 Subject: [PATCH 1/6] DAOS-9355 doc: first draft of DAOS 2.8 release notes 2.8 release notes document Signed-off-by: Liang Zhen --- docs/release/release_notes.md | 456 +++++++++++++++++++++++++++++++++- 1 file changed, 452 insertions(+), 4 deletions(-) diff --git a/docs/release/release_notes.md b/docs/release/release_notes.md index c0913e1bb47..30a8d77a9c5 100644 --- a/docs/release/release_notes.md +++ b/docs/release/release_notes.md @@ -1,6 +1,454 @@ # DAOS Version 2.8 Release Notes -DAOS 2.8 is under active development and has not been released yet. -In the meantime, please refer to the support document for the -[latest](https://docs.daos.io/latest/release/release_notes/) -stable DAOS release. +## DAOS Version 2.8.0 (2026-08-14) + +We are pleased to announce the release of DAOS version 2.8. + +### General Support + +The DAOS 2.8.0 release includes the daos-2.8.0 RPM packages and its +prerequisites. DAOS Version 2.8.0 supports the following environments: + +Architecture Support: + +* DAOS 2.8.0 supports the x86\_64 architecture. + +Operating System Support: + +* SLES 15 SP6 and SP7 +* openSUSE Leap 15.6 +* EL9.6/9.7 (RHEL, Rocky Linux, Alma Linux) + +Fabric and Network Provider Support: + +* libfabric support for the following fabrics and providers: + + - ofi+tcp on all networking hardware + - ofi+tcp;ofi\_rxm when RXM is explicitly requested + - ofi+verbs on fabrics that support the verbs API + - ofi+cxi on HPE Slingshot fabrics + +* [UCX](https://docs.daos.io/v2.8/admin/ucx/) support on NVIDIA InfiniBand and + RoCE fabrics: + + - ucx+dc\_x is the recommended UCX provider + +Storage Class Memory Support (PMem and non-PMem servers): + +* DAOS Servers without Intel Optane Persistent Memory, using the production + Metadata-on-SSD Phase2 code path with allocator v2 as the default for newly + created pools. +* Existing Metadata-on-SSD Phase1 pools remain supported and retain their + original allocator when opened. +* DAOS Servers with 2nd gen Intel Xeon Scalable processors and Intel Optane + Persistent Memory 100 Series. +* DAOS Servers with 3rd gen Intel Xeon Scalable processors and Intel Optane + Persistent Memory 200 Series. + +For a complete list of supported hardware and software, refer to the [Support +Matrix](https://docs.daos.io/v2.8/release/support_matrix/) + +### Key features and improvements + +#### Software Version Currency + +* See [above](#General-Support) for supported operating system levels. + +* Refer to the [Support + Matrix](https://docs.daos.io/v2.8/release/support_matrix/) for supported + fabric providers and vendor software combinations. + +* The following prerequisite software packages used by the DAOS build have been + updated: + + - Libfabric has been updated to 1.22.0 + - UCX has been updated to 1.20.0 + - Mercury has been updated to 2.4.1 + - SPDK has been updated to 26.01 + - PMDK has been updated to 2.1.3 + - ISA-L has been updated to 2.31.1 + - ISA-L Crypto has been updated to 2.26 + - Argobots has been updated to 1.2 + - Protobuf-C remains at 1.3.3 + - Fused remains at 1.0.0 + +* The Go module and RPM build metadata require Go 1.21. + +#### New Features and Usability Improvements + +##### MD-on-SSD Phase 2 + +* DAOS Version 2.8 makes Metadata-on-SSD Phase2 production ready when metadata + eviction is NOT enabled. The bucket-memory allocator v2 is now the default + metadata allocator for newly created Metadata-on-SSD pools, including pools + configured with a 100% memory-to-metadata ratio. + +* Making allocator v2 the production default is a primary objective of this + release. It allows all new Metadata-on-SSD deployments to use the Phase2 + allocator and establishes a common allocator path, so allocator v1 can be + retired in the longer term instead of maintaining two allocator + implementations indefinitely. + +* Existing Phase1 pools remain on their original allocator and are not converted + when opened. Allocator v2 supplies the durable format, bucket-aware + allocation, cache, pinning, object preload, garbage collection, DTX, CPD, and + accounting groundwork needed for future metadata eviction. + +* Allocator v2 also provides the foundation for future metadata eviction, which + is intended to support metadata capacity larger than the in-memory metadata + cache. Metadata eviction itself is not production ready and has not been + validated by the community; this limitation does not apply to Phase2 operation + without metadata eviction. + +* `dmg pool create` supports metadata and data sizing together with + `--mem-ratio`. Pool create, extend, reintegration, list, query, storage + display, DDB, recovery, and pool recreation paths preserve the Metadata-on-SSD + sizing and format state required by Phase2. + +* The VOS-file size is recorded independently from the metadata-blob size where + they differ. This improves recovery and recreation handling for configurations + in which the cache file and durable metadata allocation are not the same size. + +* Phase2 improves small-metadata allocation, empty-slab reclamation, + memory-bucket accounting, address conversion, and aggregation tuning under + metadata-space pressure. Low `--mem-ratio` configurations also receive + corrected sizing and accounting behavior. + +##### Check and Repair (CR) Utilities and Framework + +* Catastrophic recovery remains the `dmg check` workflow for checking and + repairing DAOS system metadata. DAOS 2.8 expands and hardens the Technology + Preview framework rather than presenting it as an general-purpose data repair + facility. + +* DAOS 2.8 adds Metadata-on-SSD mode support to CR utilities, allowing check and + repair workflows to operate on systems using that storage layout. + +* `--for-all` of `dmg check` has been removed. Administrators can use `dmg + check set-policy` to apply a repair decision to interactions of the same + class, making repeated repair behavior explicit and reviewable. + +* Repeated pool arguments are filtered. Orphan processing preserves shards that + are still needed by incremental reintegration rather than treating all such + shards as immediately disposable. + +* `dmg check query` reports unchecked pools without requiring verbose output, + identifies dry-run results as not applied, returns reports in stable order, + and manages stale findings from previous checker runs. + +* A checker dry run keeps checked pools in immutable maintenance mode. Rebuild, + aggregation, data movement, and write access remain blocked until the + maintenance condition is resolved. + +* Checker reporting, per-engine repair, parallel inconsistency processing, + engine shutdown, rank-death, and cleanup paths have been hardened. These + changes improve administrator feedback and reduce stranded checker state when + an inspection or repair is interrupted. + +##### Rebuild Performance and Stability + +* Rebuild migration now processes object IDs on main xstreams and uses a + per-target global migration-resource manager. ULT and DMA-buffer use is + limited across all pools on a target instead of being independently + overcommitted by each rebuilding pool. + +* Compatible multi-rank rebuild operations can be combined. EC scanning, + reclaim, aggregation, and partial-layout generation use improved pacing + and fewer resources, reducing contention with foreground I/O and other + recovery work. + +* Rebuild resilience is improved for recoverable network failures, pool service + leader retry, interruption, massive failures, and migration destinations that + depart while work is in progress. + +* EC rebuild and aggregation fixes cover partial-parity epochs, checksum + handling, partial overwrites, parity-target consistency, epoch-boundary + persistence, restart, discard completion, and degraded-fetch races. + +* Rebuild status and return-code fixes improve detection of completed global + scans and ensure migration setup failures are returned to the controlling + service rather than being reported as success. + +##### Interactive Rebuild Control + +* DAOS Version 2.8 adds exceptional recovery controls for stopping and + restarting rebuild for an individual pool: + + - `dmg pool rebuild stop [--force]` + - `dmg pool rebuild start` + +* These are not regular rebuild-administration commands. Use them only when + rebuild has gone wrong or an administrator must halt rebuild traffic to + diagnose and correct server issues before continuing recovery. + +* A stop halts rebuild traffic and discards current progress. After fixing the + server issue, `rebuild start` resumes recovery by launching a new rebuild; + this is not a checkpointed pause or persistent disable. + +* `dmg pool query --health-only ` reports a stopped rebuild as idle with + `-DER_OP_CANCELED`, allowing automation to distinguish an administrator stop + from successful completion. + +* Interactive stop is phase-aware. An ordinary stop works during rebuild, is + rejected during successful reclaim, and allows fail-reclaim cleanup to finish + while suppressing the next retry. + +* `--force` is reserved for a fail-reclaim retry loop after fail-reclaim has + failed at least once. Follow-on fixes prevent rejected stops from latching + state and preserve cleanup broadcasts required by stopped reintegration. + +##### System-Level Rebuild Control + +* System-scoped commands submit the interactive request to each applicable pool: + + - `dmg system rebuild stop [--force]` + - `dmg system rebuild start` + +* These commands do not establish a separate global rebuild pause state. They + are the system-wide administrative form of the per-pool interactive stop/start + operation and remain distinct from automatic self-healing policy. + +* Automatic recovery policy is independently configurable at system and pool + scope through the `self_heal` property. System policy accepts `none` or an + ordered subset of `exclude`, `pool_exclude`, and `pool_rebuild`. + +* Pool policy uses `exclude`, `rebuild`, and `delay_rebuild`. Administrators can + inspect and change policy with `dmg system set-prop` / `get-prop` and `dmg + pool set-prop` / `get-prop`. + +* `dmg system self-heal eval` re-evaluates the current system policy. System and + pool query output identifies whether automatic exclusion or rebuild is + disabled by system policy, pool policy, or both. + +* Query output also reports data redundancy as normal or degraded. + Administrative exclude, drain, reintegrate, extend, and rebuild-start + operations can still schedule recovery independently of automatic policy. + +##### Incremental Reintegration Technology Preview + +* Incremental reintegration is available as an opt-in Technology Preview through + the `reintegration:incremental` pool property. It is disabled by default and + is not production ready. + +* Incremental reintegration is not functionally complete. Punches occurring + after rebuild and before reintegration are not fully handled and may not be + reproduced correctly on the reintegrated target. + +* The implementation maintains a global stable epoch and reintegrates from that + point instead of rebuilding the complete prior history. This reduces the + intended reintegration scope but does not yet provide all semantics of the + default reintegration mode. + +* Production deployments should continue to use the default `data_sync` mode. + The preview is intended for evaluation of the new mechanism and its + administrative workflow. + +##### Control Plane and Administration + +* Pool exclude, drain, and reintegrate accept multiple ranks. System-level drain + and reintegrate apply the requested operation to affected pools throughout the + system. + +* Server configuration adds `control_interface` and deprecates `access_points`. + Generated configurations enable NVMe hotplug by default, validate SPDK + configuration at engine start, and expose SPDK I/O-buffer tuning options. + +* SPDK configuration updates that are not allowed require an explicit override. + This makes potentially disruptive configuration changes visible to + administrators rather than applying them implicitly. + +* Storage preparation and startup handling improve hugepage allocation and + permissions, SPDK lock-file cleanup, preparation with one missing SCM + namespace, and reporting for Metadata-on-SSD, non-PMem, and emulated NVMe + configurations. + +* Server startup rejects conflicting transparent-hugepage settings. Packaging + creates persistent writable server locations, protects existing configuration + files during installation, and moves daemon logs from `/tmp` to + `/var/log/daos`. + +* The control plane can automatically restart a configured excluded rank + following an engine suicide event, avoids starting administratively excluded + ranks, and supports graceful shutdown of selected ranks. + +* NVMe health output reports PCIe link speed and width and can emit a RAS event + for a downgraded link. Storage fault and replacement commands require an + unambiguous host, and blocklisted VMD devices are not unbound. + +* Container listing includes unlabeled containers and displays their UUIDs in + non-verbose output, so administrators can identify containers that do not have + user-assigned labels. + +##### DFS + +* DFS and dfuse can mount a container snapshot read-only by snapshot name or + epoch. This is a verified DAOS 2.8 DFS implementation, not merely a test or + documentation change. + +* Client-side DFS telemetry adds operation counters and read/write byte + histograms for POSIX containers, providing workload visibility at the DFS + interface. + +* DFS automatic layout selection can cycle object IDs non-sequentially and + chooses chunk sizes aligned with complete EC stripes. Selection includes a + practical minimum chunk size and RF3 handling. + +* `daos fs get-attr` reports the object OID, and `daos fs chmod` has been added. + Snapshot mounting, telemetry, layout selection, OID reporting, and chmod are + the principal verified new DFS implementations in this release. + +* Other DFS changes are fixes and hardening. They improve object-class + reporting, update open directory handles after class changes, prevent EC + classes from being assigned to directories, and reject unsafe force removal + from cached system mounts. + +* DFS error handling and memory ownership are hardened. dfuse reads in larger + chunks, associates pre-read state with inodes, and improves directory caching, + readdir, and shutdown behavior. + +##### Observability and Telemetry + +* Prometheus export supports histograms, and telemetry registration can be + selective. Administrators can limit registration to the metrics needed by + their monitoring environment. + +* Per-NUMA memory information and SPDK I/O monitoring are available. VOS, DFS, + networking, and storage metrics receive reporting and lifecycle corrections. + +* RPC origin addresses, Mercury counters, CaRT counter dumps, and standardized + timestamps improve correlation of client, transport, and server activity. + +* Log rotation retains the first `.old` file for diagnostics. `dmg support + collect-log` no longer requires optional rsync aggregation, simplifying + support collection on installations without that component. + +##### Reliability, Recovery, and Data Integrity + +* Object and VOS changes add stricter array-IOD extent validation, improved + client restart and conditional-operation retry, congestion pacing, and safer + sparse-layout and resent collective-RPC handling. + +* An N+3 EC object class is available for automatic class selection. EC fetch, + recovery, enumeration, aggregation, consistency verification, and rotated + enumeration receive correctness fixes. + +* VOS can mark corruption at object, dkey, and akey granularity. Scrub and + checker iteration better respect visibility, operation intent, and concurrent + container destruction. + +* DTX fixes address partial commit, orphan entries, refresh and resync, + aggregation, eviction, participant and leader races, duplicate batched + commits, closed containers, and rebuild visibility of uncommitted entries. + +* DTX collective-RPC load can be limited, large merged extents are capped, and + huge single values can use gang allocation. These changes reduce pathological + memory and transaction pressure. + +* SPDK hotplug and automatic faulty-device handling are hardened and enabled by + default in generated configurations. Engine handling avoids aborting when an + NVMe device is removed. + +* WAL flushing, transaction sizing, and BIO asynchronous-I/O defaults are + improved. Pool and container protocol handling, service startup and shutdown, + handle persistence, and RDB lease behavior receive corrections. + +* Raft node-ID and upgrade-validation fixes improve service recovery. + Unsupported pool-layout upgrade jumps are rejected instead of risking an + incompatible transition. + +##### Client APIs, Tools, and Compatibility + +* `libpil4dfs` adds or corrects interception for `creat`, `creat64`, `renameat`, + `fchown`, `fgetxattr`, `fsetxattr`, `fdatasync`, `FIOCLEX`, `dup3`, and + `utimensat`. + +* `libpil4dfs` also adds whitelist mode and dentry-cache garbage collection. + Safety is improved around `exec`, `fork`, MPI, dynamic loading, and + accelerator-runtime initialization. + +* `pydaos.torch` adds checkpoint integration, automatic checkpoint-path + creation, batch reads, and dataset directory-object caching. + +* Python error mapping adds `DaosErrorCode`, and dictionary cleanup has an + explicit destroy operation. + +* Go bindings cover core libdaos pool and container operations. + Container-property handling and system-name propagation are corrected. The + Java client supplies a default type identifier when none is specified. + +* CaRT accepts free-form UCX provider specifications, improves multi-interface + fabric-domain parsing, supports explicit client and agent interfaces, and + retries transient UCX/CXI initialization failures. + +* Mercury includes UCX and key-resolution fixes. OFI multi-receive is enabled + where supported, while bulk/RPC timeout and quota behavior are refined. + +* Client package installation refreshes the dynamic-linker cache, preventing + later package operations from leaving libdaos consumers with stale linker + state. + +#### Other notable changes + +* Generated server configurations enable the supported NVMe hotplug path by + default and expose storage tuning without changing the requirement to validate + hardware and provider combinations against the Support Matrix. + +* The dfuse single-threaded option has been removed. Deployments that still pass + the deprecated option must update their launch configuration. + +* Build, packaging, test, and functional-test maintenance updates supported + operating-system images, dependency tooling, fault injection, and release + qualification. Such maintenance is not presented as new product functionality. + +### Known Issues and Limitations + +* Metadata-on-SSD Phase2 with allocator v2 is production ready when metadata + eviction is not enabled. Metadata eviction itself is not production ready and + has not been validated by the community. It remains future work intended to + allow metadata capacity to exceed the in-memory metadata cache. + +* Existing Phase1 pools are not converted to allocator v2 when opened. Allocator + v2 is the default for newly created Metadata-on-SSD pools, but this release + does not provide automatic migration of existing pools. + +* Incremental reintegration is an opt-in Technology Preview, is not functionally + complete, and is not production ready. Punches occurring between rebuild and + reintegration are not fully supported. Use the default `data_sync` mode for + production reintegration. + +* Interactive rebuild stop/start is an exceptional recovery workflow for halting + rebuild traffic while server issues are corrected; it is not a regular or + persistent pause/resume facility. A stop discards current progress, and start + launches a new rebuild. System-level stop/start sends those operations to + pools and is distinct from disabling automatic self-healing. + +* Check and Repair remains a recovery-oriented preview workflow with maintenance + restrictions. A dry run leaves checked pools in immutable maintenance mode + until the condition is resolved. + +* DAOS Version 2.8.0 has not reached a stable general-availability release. This + draft describes the v2.8.0-rc5 endpoint and does not assign or imply a future + GA date. + +### Bug fixes + +The DAOS 2.8.0 release includes fixes for numerous defects in rebuild, object, +VOS, DTX, erasure coding, pool and container services, DFS, dfuse, `libpil4dfs`, +storage, networking, telemetry, and administrative utilities. For details, +please refer to the Github [release/2.8 commit +history](https://github.com/daos-stack/daos/commits/release/2.8) and the +associated [Jira tickets](https://daosio.atlassian.net/jira) as stated in the +commit messages. + +## Additional resources + +Visit the [online documentation](https://docs.daos.io/v2.8/) for more +information. All DAOS project source code is maintained in the +[https://github.com/daos-stack/daos](https://github.com/daos-stack/daos) +repository. Please visit this +[link](https://github.com/daos-stack/daos/blob/release/2.8/LICENSE) for more +information on the DAOS license. + +Refer to the [System Deployment](https://docs.daos.io/v2.8/admin/deployment/) +section of the [DAOS Administration +Guide](https://docs.daos.io/v2.8/admin/hardware/) for installation details. From 7abe40aef56b480cae3a25cb6292f2d23e30ea1f Mon Sep 17 00:00:00 2001 From: Michael Hennecke Date: Wed, 12 Aug 2026 12:21:30 +0200 Subject: [PATCH 2/6] DAOS-9355 doc: version 2.8 release notes update 2.8 relnotes draft with review comments Doc-only: true Signed-off-by: Michael Hennecke --- docs/release/release_notes.md | 136 +++++++++++++++++++--------------- 1 file changed, 75 insertions(+), 61 deletions(-) diff --git a/docs/release/release_notes.md b/docs/release/release_notes.md index 30a8d77a9c5..3995dc3728b 100644 --- a/docs/release/release_notes.md +++ b/docs/release/release_notes.md @@ -1,6 +1,6 @@ # DAOS Version 2.8 Release Notes -## DAOS Version 2.8.0 (2026-08-14) +## DAOS Version 2.8.0 (2026-08-12) We are pleased to announce the release of DAOS version 2.8. @@ -40,10 +40,10 @@ Storage Class Memory Support (PMem and non-PMem servers): created pools. * Existing Metadata-on-SSD Phase1 pools remain supported and retain their original allocator when opened. -* DAOS Servers with 2nd gen Intel Xeon Scalable processors and Intel Optane - Persistent Memory 100 Series. * DAOS Servers with 3rd gen Intel Xeon Scalable processors and Intel Optane Persistent Memory 200 Series. +* DAOS Servers with 2nd gen Intel Xeon Scalable processors and Intel Optane + Persistent Memory 100 Series. For a complete list of supported hardware and software, refer to the [Support Matrix](https://docs.daos.io/v2.8/release/support_matrix/) @@ -62,7 +62,7 @@ Matrix](https://docs.daos.io/v2.8/release/support_matrix/) updated: - Libfabric has been updated to 1.22.0 - - UCX has been updated to 1.20.0 + - UCX (in DOCA-OFED) has been updated to 1.20.0 - Mercury has been updated to 2.4.1 - SPDK has been updated to 26.01 - PMDK has been updated to 2.1.3 @@ -78,20 +78,27 @@ Matrix](https://docs.daos.io/v2.8/release/support_matrix/) ##### MD-on-SSD Phase 2 -* DAOS Version 2.8 makes Metadata-on-SSD Phase2 production ready when metadata - eviction is NOT enabled. The bucket-memory allocator v2 is now the default - metadata allocator for newly created Metadata-on-SSD pools, including pools - configured with a 100% memory-to-metadata ratio. - -* Making allocator v2 the production default is a primary objective of this - release. It allows all new Metadata-on-SSD deployments to use the Phase2 - allocator and establishes a common allocator path, so allocator v1 can be - retired in the longer term instead of maintaining two allocator - implementations indefinitely. - -* Existing Phase1 pools remain on their original allocator and are not converted - when opened. Allocator v2 supplies the durable format, bucket-aware - allocation, cache, pinning, object preload, garbage collection, DTX, CPD, and +* It is a primary feature of DAOS Version 2.8 that the Metadata-on-SSD + bucket-memory allocator v2 is now production ready. This allocator is now + the default metadata allocator for newly created Metadata-on-SSD pools. + It allows all new Metadata-on-SSD deployments to use the Phase2 allocator + and establishes a common allocator path, so the allocator v1 can be retired + in the longer term instead of maintaining two allocator implementations + indefinitely. + +* The production ready default in DAOS Version 2.8 is to create MD-on-SSD + pools with a 100% memory-to-metadata ratio (`--mem-ratio 100%`), + so metadata eviction is not enabled. + As a technoloy preview, `dmg pool create` supports metadata sizing + with a memory-to-metadata ratio smaller than 100% (for example, using + `--mem-ratio 25%`). + +* Existing MD-on-SSD Phase1 pools remain on their original allocator and are + not converted when opened. To migrate Phase1 pools, data in those pools + needs to be copied by the user to a new pool created with the v2 allocator. + +* Allocator v2 supplies the durable format, bucket-aware allocation, cache, + pinning, object preload, garbage collection, DTX, CPD, and accounting groundwork needed for future metadata eviction. * Allocator v2 also provides the foundation for future metadata eviction, which @@ -100,10 +107,9 @@ Matrix](https://docs.daos.io/v2.8/release/support_matrix/) validated by the community; this limitation does not apply to Phase2 operation without metadata eviction. -* `dmg pool create` supports metadata and data sizing together with - `--mem-ratio`. Pool create, extend, reintegration, list, query, storage - display, DDB, recovery, and pool recreation paths preserve the Metadata-on-SSD - sizing and format state required by Phase2. +* Pool create, extend, reintegration, list, query, storage display, DDB, + recovery, and pool recreation paths preserve the Metadata-on-SSD sizing + and format state required by Phase2. * The VOS-file size is recorded independently from the metadata-blob size where they differ. This improves recovery and recreation handling for configurations @@ -116,16 +122,16 @@ Matrix](https://docs.daos.io/v2.8/release/support_matrix/) ##### Check and Repair (CR) Utilities and Framework -* Catastrophic recovery remains the `dmg check` workflow for checking and - repairing DAOS system metadata. DAOS 2.8 expands and hardens the Technology +* The `dmg check` command is the workflow for checking and repairing + DAOS system metadata. DAOS 2.8 expands and hardens the CR Technology Preview framework rather than presenting it as an general-purpose data repair facility. * DAOS 2.8 adds Metadata-on-SSD mode support to CR utilities, allowing check and repair workflows to operate on systems using that storage layout. -* `--for-all` of `dmg check` has been removed. Administrators can use `dmg - check set-policy` to apply a repair decision to interactions of the same +* The `dmg check --for-all` option has been removed. Administrators can use + `dmg check set-policy` to apply a repair decision to interactions of the same class, making repeated repair behavior explicit and reviewable. * Repeated pool arguments are filtered. Orphan processing preserves shards that @@ -166,13 +172,13 @@ Matrix](https://docs.daos.io/v2.8/release/support_matrix/) persistence, restart, discard completion, and degraded-fetch races. * Rebuild status and return-code fixes improve detection of completed global - scans and ensure migration setup failures are returned to the controlling + scans, and ensure migration setup failures are returned to the controlling service rather than being reported as success. ##### Interactive Rebuild Control -* DAOS Version 2.8 adds exceptional recovery controls for stopping and - restarting rebuild for an individual pool: +* DAOS Version 2.8 adds controls for stopping and restarting rebuild for an + individual pool, to handle exceptional rebuild situations: - `dmg pool rebuild stop [--force]` - `dmg pool rebuild start` @@ -181,17 +187,17 @@ Matrix](https://docs.daos.io/v2.8/release/support_matrix/) rebuild has gone wrong or an administrator must halt rebuild traffic to diagnose and correct server issues before continuing recovery. -* A stop halts rebuild traffic and discards current progress. After fixing the - server issue, `rebuild start` resumes recovery by launching a new rebuild; - this is not a checkpointed pause or persistent disable. +* A `rebuild stop` halts rebuild traffic and discards current progress. + After fixing the server issue, `rebuild start` resumes recovery by launching + a new rebuild. This is not a checkpointed pause or persistent disable. * `dmg pool query --health-only ` reports a stopped rebuild as idle with `-DER_OP_CANCELED`, allowing automation to distinguish an administrator stop from successful completion. -* Interactive stop is phase-aware. An ordinary stop works during rebuild, is - rejected during successful reclaim, and allows fail-reclaim cleanup to finish - while suppressing the next retry. +* Interactive rebuild stop is phase-aware. An ordinary stop works during rebuild, + is rejected during successful reclaim, and allows fail-reclaim cleanup + to finish while suppressing the next retry. * `--force` is reserved for a fail-reclaim retry loop after fail-reclaim has failed at least once. Follow-on fixes prevent rejected stops from latching @@ -205,8 +211,8 @@ Matrix](https://docs.daos.io/v2.8/release/support_matrix/) - `dmg system rebuild start` * These commands do not establish a separate global rebuild pause state. They - are the system-wide administrative form of the per-pool interactive stop/start - operation and remain distinct from automatic self-healing policy. + are the system-wide administrative form of the per-pool interactive rebuild + stop/start operation and remain distinct from automatic self-healing policy. * Automatic recovery policy is independently configurable at system and pool scope through the `self_heal` property. System policy accepts `none` or an @@ -220,7 +226,7 @@ Matrix](https://docs.daos.io/v2.8/release/support_matrix/) pool query output identifies whether automatic exclusion or rebuild is disabled by system policy, pool policy, or both. -* Query output also reports data redundancy as normal or degraded. +* Pool query output also reports data redundancy as normal or degraded. Administrative exclude, drain, reintegrate, extend, and rebuild-start operations can still schedule recovery independently of automatic policy. @@ -240,7 +246,7 @@ Matrix](https://docs.daos.io/v2.8/release/support_matrix/) default reintegration mode. * Production deployments should continue to use the default `data_sync` mode. - The preview is intended for evaluation of the new mechanism and its + The technology preview is intended for evaluation of the new mechanism and its administrative workflow. ##### Control Plane and Administration @@ -249,11 +255,19 @@ Matrix](https://docs.daos.io/v2.8/release/support_matrix/) and reintegrate apply the requested operation to affected pools throughout the system. -* Server configuration adds `control_interface` and deprecates `access_points`. - Generated configurations enable NVMe hotplug by default, validate SPDK - configuration at engine start, and expose SPDK I/O-buffer tuning options. +* Notable changes in the `daos_server.yml` configuration file: -* SPDK configuration updates that are not allowed require an explicit override. + - `access_points` is now deprecated; use `mgmt_svc_replicas` instead. + - A `control_iface` option is added to set a specific network interface for + the control plane listener (by default, it will listen on all interfaces) + - Configurations generated by `dmg config generate` now enable NVMe hotplug + by default. + - SPDK I/O-buffer tuning options are exposed to allow customisation for + large NVMe disks: A new `spdk_iobuf` section allows to explicitly set + `small_pool_count` and `large_pool_count`. + +* The SPDK configuration is validated at engine start, and + SPDK configuration updates that are not allowed require an explicit override. This makes potentially disruptive configuration changes visible to administrators rather than applying them implicitly. @@ -262,10 +276,12 @@ Matrix](https://docs.daos.io/v2.8/release/support_matrix/) namespace, and reporting for Metadata-on-SSD, non-PMem, and emulated NVMe configurations. -* Server startup rejects conflicting transparent-hugepage settings. Packaging - creates persistent writable server locations, protects existing configuration - files during installation, and moves daemon logs from `/tmp` to - `/var/log/daos`. +* Server startup rejects conflicting transparent-hugepage settings. + +* Packaging creates persistent writable server locations, protects existing + configuration files during installation. + +* Default daemon logfile location has changed from `/tmp` to `/var/log/daos`. * The control plane can automatically restart a configured excluded rank following an engine suicide event, avoids starting administratively excluded @@ -273,7 +289,7 @@ Matrix](https://docs.daos.io/v2.8/release/support_matrix/) * NVMe health output reports PCIe link speed and width and can emit a RAS event for a downgraded link. Storage fault and replacement commands require an - unambiguous host, and blocklisted VMD devices are not unbound. + unambiguous host, and blacklisted VMD devices are not unbound. * Container listing includes unlabeled containers and displays their UUIDs in non-verbose output, so administrators can identify containers that do not have @@ -318,9 +334,10 @@ Matrix](https://docs.daos.io/v2.8/release/support_matrix/) * RPC origin addresses, Mercury counters, CaRT counter dumps, and standardized timestamps improve correlation of client, transport, and server activity. -* Log rotation retains the first `.old` file for diagnostics. `dmg support - collect-log` no longer requires optional rsync aggregation, simplifying - support collection on installations without that component. +* Log rotation retains the first `.old` file for diagnostics. + +* `dmg support collect-log` no longer requires optional rsync aggregation, + simplifying support collection on installations without that component. ##### Reliability, Recovery, and Data Integrity @@ -328,9 +345,10 @@ Matrix](https://docs.daos.io/v2.8/release/support_matrix/) client restart and conditional-operation retry, congestion pacing, and safer sparse-layout and resent collective-RPC handling. -* An N+3 EC object class is available for automatic class selection. EC fetch, - recovery, enumeration, aggregation, consistency verification, and rotated - enumeration receive correctness fixes. +* An N+3 EC object class is available for automatic class selection. + +* EC fetch, recovery, enumeration, aggregation, consistency verification, + and rotated enumeration receive correctness fixes. * VOS can mark corruption at object, dkey, and akey granularity. Scrub and checker iteration better respect visibility, operation intent, and concurrent @@ -340,7 +358,7 @@ Matrix](https://docs.daos.io/v2.8/release/support_matrix/) aggregation, eviction, participant and leader races, duplicate batched commits, closed containers, and rebuild visibility of uncommitted entries. -* DTX collective-RPC load can be limited, large merged extents are capped, and +* DTX collective-RPC load can be limited. Large merged extents are capped, and huge single values can use gang allocation. These changes reduce pathological memory and transaction pressure. @@ -378,10 +396,10 @@ Matrix](https://docs.daos.io/v2.8/release/support_matrix/) * CaRT accepts free-form UCX provider specifications, improves multi-interface fabric-domain parsing, supports explicit client and agent interfaces, and - retries transient UCX/CXI initialization failures. + retries transient UCX or CXI initialization failures. * Mercury includes UCX and key-resolution fixes. OFI multi-receive is enabled - where supported, while bulk/RPC timeout and quota behavior are refined. + where supported. Mercury bulk/RPC timeout and quota behavior are refined. * Client package installation refreshes the dynamic-linker cache, preventing later package operations from leaving libdaos consumers with stale linker @@ -426,10 +444,6 @@ Matrix](https://docs.daos.io/v2.8/release/support_matrix/) restrictions. A dry run leaves checked pools in immutable maintenance mode until the condition is resolved. -* DAOS Version 2.8.0 has not reached a stable general-availability release. This - draft describes the v2.8.0-rc5 endpoint and does not assign or imply a future - GA date. - ### Bug fixes The DAOS 2.8.0 release includes fixes for numerous defects in rebuild, object, From ee58ce98b46218de02c2e5108e53278255c44bbb Mon Sep 17 00:00:00 2001 From: Michael Hennecke Date: Wed, 12 Aug 2026 12:28:11 +0200 Subject: [PATCH 3/6] DAOS-9355 doc: version 2.8 release notes fix typo Doc-only: true Signed-off-by: Michael Hennecke --- docs/release/release_notes.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/release/release_notes.md b/docs/release/release_notes.md index 3995dc3728b..32c99f520c8 100644 --- a/docs/release/release_notes.md +++ b/docs/release/release_notes.md @@ -262,7 +262,7 @@ Matrix](https://docs.daos.io/v2.8/release/support_matrix/) the control plane listener (by default, it will listen on all interfaces) - Configurations generated by `dmg config generate` now enable NVMe hotplug by default. - - SPDK I/O-buffer tuning options are exposed to allow customisation for + - SPDK I/O-buffer tuning options are exposed to allow customization for large NVMe disks: A new `spdk_iobuf` section allows to explicitly set `small_pool_count` and `large_pool_count`. From 4e3a14673136c94410f62156aeb115e895183e11 Mon Sep 17 00:00:00 2001 From: Mohamad Chaarawi Date: Wed, 12 Aug 2026 14:21:05 +0000 Subject: [PATCH 4/6] fix a few things doc-only: true Signed-off-by: Mohamad Chaarawi --- docs/release/release_notes.md | 23 +++++++---------------- 1 file changed, 7 insertions(+), 16 deletions(-) diff --git a/docs/release/release_notes.md b/docs/release/release_notes.md index 32c99f520c8..f09898525d0 100644 --- a/docs/release/release_notes.md +++ b/docs/release/release_notes.md @@ -295,11 +295,10 @@ Matrix](https://docs.daos.io/v2.8/release/support_matrix/) non-verbose output, so administrators can identify containers that do not have user-assigned labels. -##### DFS +##### DFS and dfuse -* DFS and dfuse can mount a container snapshot read-only by snapshot name or - epoch. This is a verified DAOS 2.8 DFS implementation, not merely a test or - documentation change. +* As a Technology Preview, DFS and dfuse can mount a container snapshot + read-only by snapshot name or epoch. This feature is not production ready. * Client-side DFS telemetry adds operation counters and read/write byte histograms for POSIX containers, providing workload visibility at the DFS @@ -310,17 +309,14 @@ Matrix](https://docs.daos.io/v2.8/release/support_matrix/) practical minimum chunk size and RF3 handling. * `daos fs get-attr` reports the object OID, and `daos fs chmod` has been added. - Snapshot mounting, telemetry, layout selection, OID reporting, and chmod are - the principal verified new DFS implementations in this release. * Other DFS changes are fixes and hardening. They improve object-class reporting, update open directory handles after class changes, prevent EC classes from being assigned to directories, and reject unsafe force removal - from cached system mounts. + from cached system mounts. DFS error handling and memory ownership are hardened -* DFS error handling and memory ownership are hardened. dfuse reads in larger - chunks, associates pre-read state with inodes, and improves directory caching, - readdir, and shutdown behavior. +* dfuse reads in larger chunks, associates pre-read state with inodes, and + improves directory caching, readdir, and shutdown behavior. ##### Observability and Telemetry @@ -387,12 +383,7 @@ Matrix](https://docs.daos.io/v2.8/release/support_matrix/) * `pydaos.torch` adds checkpoint integration, automatic checkpoint-path creation, batch reads, and dataset directory-object caching. -* Python error mapping adds `DaosErrorCode`, and dictionary cleanup has an - explicit destroy operation. - -* Go bindings cover core libdaos pool and container operations. - Container-property handling and system-name propagation are corrected. The - Java client supplies a default type identifier when none is specified. +* PyDAOS dictionary cleanup has an explicit destroy operation. * CaRT accepts free-form UCX provider specifications, improves multi-interface fabric-domain parsing, supports explicit client and agent interfaces, and From 574fde44289396ee58faf2123658ce54e0b28435 Mon Sep 17 00:00:00 2001 From: Michael Hennecke Date: Wed, 12 Aug 2026 17:35:25 +0200 Subject: [PATCH 5/6] DAOS-9355 doc: add new defaults to version 2.8 release notes add new pool/cont defaults to 2.8 relnotes Doc-only: true Signed-off-by: Michael Hennecke --- docs/release/release_notes.md | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/docs/release/release_notes.md b/docs/release/release_notes.md index f09898525d0..e195cb722ed 100644 --- a/docs/release/release_notes.md +++ b/docs/release/release_notes.md @@ -396,6 +396,23 @@ Matrix](https://docs.daos.io/v2.8/release/support_matrix/) later package operations from leaving libdaos consumers with stale linker state. +#### Better pool and container property defaults + +* The default redundancy factor that is set on the pool level is now set to + `rd_fac=3`; the previous default was zero. Container create operations will + inherit this pool-level default unless a different redundancy factor is + explicitly specified at container creation time. This means that new + containers created with DAOS Version 2.8 will now have 3-fault-tolerance + by default. Note that if the number of fault domains in the pool + (servers that are participating in the pool) is smaller than three, + the `rd_fac` will be adjusted down accordingly. + +* The default fraction of pool space that is reserved for rebuild (`space_rb`) + is now 5%. In previous releases, no space was reserved by default. + +* The default `ec_cell_sz` has been increased from 64kiB to 128kiB. + This improves EC performance. + #### Other notable changes * Generated server configurations enable the supported NVMe hotplug path by From 35ec10e120668ac1230e874151bea40081fd0723 Mon Sep 17 00:00:00 2001 From: Michael Hennecke Date: Wed, 12 Aug 2026 22:51:18 +0200 Subject: [PATCH 6/6] DAOS-9355 doc: version 2.8 release notes review comments update 2.8 relnotes draft with review comments Doc-only: true Signed-off-by: Michael Hennecke --- docs/release/release_notes.md | 23 ++++++++++++++++++++++- 1 file changed, 22 insertions(+), 1 deletion(-) diff --git a/docs/release/release_notes.md b/docs/release/release_notes.md index e195cb722ed..71f516caf42 100644 --- a/docs/release/release_notes.md +++ b/docs/release/release_notes.md @@ -81,7 +81,7 @@ Matrix](https://docs.daos.io/v2.8/release/support_matrix/) * It is a primary feature of DAOS Version 2.8 that the Metadata-on-SSD bucket-memory allocator v2 is now production ready. This allocator is now the default metadata allocator for newly created Metadata-on-SSD pools. - It allows all new Metadata-on-SSD deployments to use the Phase2 allocator + It allows all new Metadata-on-SSD deployments to use the v2 allocator and establishes a common allocator path, so the allocator v1 can be retired in the longer term instead of maintaining two allocator implementations indefinitely. @@ -452,6 +452,27 @@ Matrix](https://docs.daos.io/v2.8/release/support_matrix/) restrictions. A dry run leaves checked pools in immutable maintenance mode until the condition is resolved. +* When downgrading from DAOS version 2.8 back to Version 2.6, the following + changes to the SPDK configuration file `daos_nvme.conf` have to be manually + applied on all servers, to revert a parameter name change that was introduced + by the newer SPDK version in DAOS version 2.8: + +```bash + # For PMem based servers, with 2 engines per server: + export PATH0=/mnt/daos0 # matching the scm_mount path for engine 0 in daos_server.yml + export PATH1=/mnt/daos1 # matching the scm_mount path for engine 1 in daos_server.yml + + # For MD-on-SSD based servers, with 2 engines per server: + export CONTROL_METADATA=/mnt # matching the control_metadata path in daos_server.yml + export PATH0=$CONTROL_METADATA/daos_control/engine0 + export PATH1=$CONTROL_METADATA/daos_control/engine1 + + sed -i 's/vmd_enable/enable_vmd/g' $PATH0/daos_nvme.conf + sed -i 's/vmd_enable/enable_vmd/g' $PATH1/daos_nvme.conf + sed -i 's/transport_retry_count/retry_count/g' $PATH0/daos_nvme.conf + sed -i 's/transport_retry_count/retry_count/g' $PATH1/daos_nvme.conf +``` + ### Bug fixes The DAOS 2.8.0 release includes fixes for numerous defects in rebuild, object,