Repository navigation
Disks keep waking up every 90 minutes - TrueNAS 25.10.1 #4
Description
TrueNAS Scale: 25.10.1
System Dataset Pool: boot-pool
latest spindown-fix.sh and v25 patch provided here.
Spindown time set to 5 minutes for all HDDs, power management set to 1.
Script triggered on boot:
Q: Should it really be Pre Init? I’m pretty sure i’ve seen it PostInit in previous fix revision in this repo.
RAIDZ pool named storage:
raidz1:
3x SATA Seagate EXOS X24 24TB
special vdev:
mirrored 2x SATA Intel SSD D3-S4610 960G
zil:
1x NVME Intel Optane 800p 118G
l2arc:
1x NVME Silicon Power P34A80 2T
Power consumption log during night, with running middleware service on:
and with middlewared service stopped:
Running hdd-wake-trace.sh on balanced profile logs wake-events.tsv:
1781145196 2026-06-11 04:33:16 partial disk wake 14478 /dev/sdd=ST24000NM000H-3KS103_ZYD0BVY4 /root/hdd-wake-trace-logs/2026-06-11_04-33-25_storage_partial-disk-wake_st24000nm000h-3ks103_zyd0bvy4.log
1781150600 2026-06-11 06:03:20 partial disk wake 5404 /dev/sdd=ST24000NM000H-3KS103_ZYD0BVY4 /root/hdd-wake-trace-logs/2026-06-11_06-03-25_storage_partial-disk-wake_st24000nm000h-3ks103_zyd0bvy4.log
1781155997 2026-06-11 07:33:17 partial disk wake 5397 /dev/sdd=ST24000NM000H-3KS103_ZYD0BVY4 /root/hdd-wake-trace-logs/2026-06-11_07-33-25_storage_partial-disk-wake_st24000nm000h-3ks103_zyd0bvy4.log
1781161396 2026-06-11 09:03:16 partial disk wake 5399 /dev/sdd=ST24000NM000H-3KS103_ZYD0BVY4 /root/hdd-wake-trace-logs/2026-06-11_09-03-25_storage_partial-disk-wake_st24000nm000h-3ks103_zyd0bvy4.log
Extra info:
cat spindown-fix.sh | grep SCRIPT_DIR/
PATCH="$SCRIPT_DIR/spindown-v25.patch"
BOOT_SCRIPT="$SCRIPT_DIR/spindown-overlay-mount.sh"
./spindown-fix.sh status
Overlay status:
File:
/usr/lib/python3/dist-packages/middlewared/plugins/pool_/dataset_encryption_info.py
Overlay:
present: /root/TrueNAS-Tricks/my-overlay/usr/lib/python3/dist-packages/middlewared/plugins/pool_/dataset_encryption_info.py
Mount:
/usr/lib/python3/dist-packages/middlewared/plugins/pool_/dataset_encryption_info.py boot-pool/ROOT/25.10.1/root[/TrueNAS-Tricks/my-overlay/usr/lib/python3/dist-packages/middlewared/plugins/pool_/dataset_encryption_info.py] zfs rw,nosuid,nodev,relatime,xattr,noacl,casesensitive
File:
/usr/lib/python3/dist-packages/middlewared/utils/disks_/disk_class.py
Overlay:
present: /root/TrueNAS-Tricks/my-overlay/usr/lib/python3/dist-packages/middlewared/utils/disks_/disk_class.py
Mount:
/usr/lib/python3/dist-packages/middlewared/utils/disks_/disk_class.py boot-pool/ROOT/25.10.1/root[/TrueNAS-Tricks/my-overlay/usr/lib/python3/dist-packages/middlewared/utils/disks_/disk_class.py] zfs rw,nosuid,nodev,relatime,xattr,noacl,casesensitive
File:
/usr/lib/python3/dist-packages/middlewared/alert/source/smart.py
Overlay:
present: /root/TrueNAS-Tricks/my-overlay/usr/lib/python3/dist-packages/middlewared/alert/source/smart.py
Mount:
/usr/lib/python3/dist-packages/middlewared/alert/source/smart.py boot-pool/ROOT/25.10.1/root[/TrueNAS-Tricks/my-overlay/usr/lib/python3/dist-packages/middlewared/alert/source/smart.py] zfs rw,nosuid,nodev,relatime,xattr,noacl,casesensitive
File:
/usr/lib/python3/dist-packages/middlewared/plugins/disk.py
Overlay:
present: /root/TrueNAS-Tricks/my-overlay/usr/lib/python3/dist-packages/middlewared/plugins/disk.py
Mount:
/usr/lib/python3/dist-packages/middlewared/plugins/disk.py boot-pool/ROOT/25.10.1/root[/TrueNAS-Tricks/my-overlay/usr/lib/python3/dist-packages/middlewared/plugins/disk.py] zfs rw,nosuid,nodev,relatime,xattr,noacl,casesensitive
So fixes are there, are properly mounted and in the right place.
running hdd-wake-trace.sh on balanced profile i see in event detail logs:
== Wake analysis summary ==
HIGH: recurrence Wake interval is approximately 90 minutes (5399s since previous event).
MEDIUM: SMART SMART or TrueNAS disk polling activity was seen around the wake. Check whether it appears before or after wake_epoch.
MEDIUM: ZFS ZFS kernel worker or txg_sync I/O appeared immediately after wake.
MEDIUM: ZFS ZFS workers were active around the wake. This shows the executor, but not necessarily the original caller.
== Human-readable interpretation ==
- The wake cadence is highly regular, which points toward a timer, polling cycle, middleware task, or scheduled client behaviour.
- ZFS workers were active. This usually means ZFS executed I/O, but it may not be the original initiator.
- SMART or disk polling activity was seen. Check whether it appears before or after the wake timestamp.
- The detection time is not the exact spin-up instant; the real trigger may be inside the previous poll interval.
== SMART / disk polling context ==
-- SMART processes currently visible --
278278 1130 root Thu Jun 11 09:03:36 2026 smartctl smartctl -x /dev/sdb -jc
which makes me think that patch isn’t working as pre-patch that line in /usr/lib/python3/dist-packages/middlewared/utils/disks_/disk_class.py file looked like this:
cmd = ["smartctl", "-x", self.devpath]
but for me it seems it’s correctly patched:
cat /usr/lib/python3/dist-packages/middlewared/utils/disks_/disk_class.py | grep smartctl
def __run_smartctl_cmd_impl(self, cmd: list[str], raise_alert: bool = True) -> str:
def smartctl_info(self, return_json: bool = False, raise_alert: bool = True) -> dict | str:
"""Return smartctl -x information.
cmd = ["smartctl", "-n", "standby", "-x"]
stdout = self.__run_smartctl_cmd_impl(cmd, raise_alert)
def smartctl_test(
self.__run_smartctl_cmd_impl(["smartctl", self.devpath, "-t", ttype], raise_alert)
Any ideas? Should it really be launched in Pre Init? I'm happy to provide more info if needed. Thanks.
Full logs:
2026-06-11_04-33-25_storage_partial-disk-wake_st24000nm000h-3ks103_zyd0bvy4.log
2026-06-11_06-03-25_storage_partial-disk-wake_st24000nm000h-3ks103_zyd0bvy4.log
2026-06-11_07-33-25_storage_partial-disk-wake_st24000nm000h-3ks103_zyd0bvy4.log
2026-06-11_09-03-25_storage_partial-disk-wake_st24000nm000h-3ks103_zyd0bvy4.log
Activity
itiligent commentedon Jun 13, 2026
I went through a similar issue in testing and refining the patch script....
Every 90 mins is the default schedule for a bunch of smarctl things that the scripts is supposed to disable, which points us in a fairly clear direction. Do the drives ever reach standby?
During testing, in some cases the overlay gets in a race condition at boot (even as a pre-int command.).
The 90 minute schedule points to the original config files being parsed at boot BEFORE the overlay patch takes effect... so its created, , but not applied.
Try this:
Restart middleware manually after the boot is settled and see what happens. If that works, see my notes in the readme around making this middleware restart a short time after boot with
ENABLE_DELAYED_RESTART=yes DELAY_SECONDS=300 bash /path/to/spindown-overlay-mount.shFailing that, have a deeper read of this to run my hdd-wake-trace script which will watch, log, and track sequences/patterns of wake events to help point you toward exactly what processes are touching the drives and when.
mepreston commentedon Jun 28, 2026
https://ixsystems.atlassian.net/browse/NAS-137887
zokzokzokzok commentedon Sep 11, 2026
Try this:
Restart middleware manually after the boot is settled and see what happens. If that works, see my notes in the readme around making this middleware restart a short time after boot with ENABLE_DELAYED_RESTART=yes DELAY_SECONDS=300 bash /path/to/spindown-overlay-mount.sh
can confirm this fixes 90min spinup for me