System log flooded with victoria-metrics warnings while an alert is active #1958

Activity

1. Tbaile

   added this to the 8.8.1 milestone on Oct 22, 2026
2. Tbaile

   self-assigned this

   on Oct 22, 2026
3. github-project-automation

   added this to NethSecurityon Oct 22, 2026
4. github-project-automation

   moved this to ToDo 🕐 in NethSecurityon Oct 22, 2026
5. Tbaile

   added the

   Bug

   issue type on Oct 22, 2026
6. github-project-automation

   moved this from ToDo 🕐 to In Progress 🛠 in NethSecurityon Oct 22, 2026
7. Tbaile commented on Oct 22, 2026

   

   Tbaile

   on Oct 22, 2026

   CollaboratorAuthor

   More actions

   Latest dev image

   QA: have an up-to-date machine with an alert, check that every 5 minutes you get the log showed in the issue.
   Flash over the new image, check that the issue is gone (but alerts in the UI are still there)
8. Tbaile

   added

   testingPackages are available from testing repositoriesPackages are available from testing repositories

   on Oct 22, 2026
9. Tbaile

   assigned

   m-dilorenzi

   and unassigned

   Tbaile

   on Oct 22, 2026
10. nethbot

    moved this from In Progress 🛠 to Testing in NethSecurityon Oct 22, 2026
11. m-dilorenzi commented on Oct 22, 2026

    

    m-dilorenzi

    on Oct 22, 2026

    Collaborator

    More actions

    1. Crash cron service

    root@NethSec:~# for i in 1 2 3 4 5 6 7; do kill -9 $(pidof crond); sleep 7; done
    kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec]
    root@NethSec:~# /usr/libexec/telegraf-services | grep -o '{[^}]*cron[^}]*}'
    {"service": "cron", "instance": "instance1", "running": 0, "pid": 0, "exit_code": 137}
    

    2. Check the alarm has been opened and the issue is reproducible

    

    root@NethSec:~# grep "rollup_result_cache.go" /var/log/messages
    Oct 22 16:20:16 NethSec victoria-metrics[6525]: 2026-10-22T16:20:16.137Z#011warn#011VictoriaMetrics/app/vmselect/promql/rollup_result_cache.go:77#011resetting rollup result cache because the metric ALERTS{service="cron",db="os-metrics",instance="instance1",alertgroup="services",alertname="ServiceDown",alertstate="pending",severity="critical"} (Timestamp=1792685700000, Value=1.000000) has a timestamp older than -search.cacheTimestampOffset=5m0s by 15.000s
    Oct 22 16:20:16 NethSec victoria-metrics[6525]: 2026-10-22T16:20:16.137Z#011info#011VictoriaMetrics/app/vmselect/promql/rollup_result_cache.go:217#011rollupResult cache has been cleared
    Oct 22 16:25:16 NethSec victoria-metrics[6525]: 2026-10-22T16:25:16.173Z#011warn#011VictoriaMetrics/app/vmselect/promql/rollup_result_cache.go:77#011resetting rollup result cache because the metric ALERTS{service="cron",db="os-metrics",instance="instance1",alertgroup="services",alertname="ServiceDown",alertstate="pending",severity="critical"} (Timestamp=1792686000000, Value=NaN) has a timestamp older than -search.cacheTimestampOffset=5m0s by 15.000s
    Oct 22 16:25:16 NethSec victoria-metrics[6525]: 2026-10-22T16:25:16.173Z#011info#011VictoriaMetrics/app/vmselect/promql/rollup_result_cache.go:217#011rollupResult cache has been cleared
    

    3. Flash with the new image

    

    4. Crash cron service again

    root@NethSec:~# for i in 1 2 3 4 5 6 7; do kill -9 $(pidof crond); sleep 7; done
    kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec]
    root@NethSec:~# /usr/libexec/telegraf-services | grep -o '{[^}]*cron[^}]*}'
    {"service": "cron", "instance": "instance1", "running": 0, "pid": 0, "exit_code": 137}
    root@NethSec:~# 
    

    5. Check the alarm has been opened again but the issue is no more reproducible (logs are not found inside /var/log/messages)

    

    root@NethSec:~# grep "rollup_result_cache.go" /var/log/messages
    root@NethSec:~#
    

    Verified
12. m-dilorenzi

    added

    verifiedAll test cases were verified successfullyAll test cases were verified successfully

    and removed

    testingPackages are available from testing repositoriesPackages are available from testing repositories

    on Oct 22, 2026
13. m-dilorenzi

    removed their assignment

    on Oct 22, 2026
14. nethbot

    moved this from Testing to Verified in NethSecurityon Oct 22, 2026
15. Tbaile

    closed this as completedon Oct 27, 2026
16. github-project-automation

    moved this from Verified to Done ✅ in NethSecurityon Oct 27, 2026