Skip to content

BUG REPORT : useLegacyNaming doesn't handle correctly PVC naming for vlstorage inside VLCluster object #2385

New issue
New issue
Open
Open
BUG REPORT : useLegacyNaming doesn't handle correctly PVC naming for vlstorage inside VLCluster object#2385
Assignees
AndrewChubatiuk
Labels
bugSomething isn't working

Description

@ccaillet1974
ccaillet1974
opened on Jul 8, 2026

Hi all,

Target : migration from Helm deployed victoria-logs-cluster to operator managed VLCluster without disruption

Issue : naming for vlselect and vlinsert deployment is correct and all objects are handled correctly as it seems but with the problem with PVC cannot confirm because operator failed at vlstorage stage

Descrpition :
problem with PVC name : iwith helm the pvc are build with the name : vlstorage-volume-<app_name>-vlstorage-<idx_sts> and operator even with useLegacyNaming flag build the PVC name like : vlstorage-db-<app_name>-vlstorage-<idx_sts>

Attended result : when useLegacyNaming is set PVC names are the same than with helm

Activity

  1. AndrewChubatiuk
    self-assigned this
    on Jul 8, 2026
  2. AndrewChubatiuk
    added
    bugSomething isn't working
    on Jul 8, 2026

  3. AndrewChubatiuk commented on Jul 8, 2026

    @AndrewChubatiuk
    AndrewChubatiuk
    on Jul 8, 2026
    Contributor

    added a PR with a fix, but for now you can override that value using spec.vlstorage.storage.volumeClaimTemplate.name

  4. ccaillet1974 commented on Jul 9, 2026

    @ccaillet1974
    ccaillet1974
    on Jul 9, 2026
    Author

    Andrew,

    I've tested and now i have the following error : seems it is more deeper thant only a naming issue :

    {"level":"info","ts":"2026-07-09T07:49:36Z","logger":"controller.VLCluster","msg":"object has changes with previous state, applying changes","vlcluster":"lyo0-int-mtls-ovl","namespace":"kube-test"}
{"level":"info","ts":"2026-07-09T07:49:36Z","logger":"controller.VLCluster","msg":"creating new ServiceAccount=kube-test/lyo0-int-mtls-ovl-vlcluster","vlcluster":"lyo0-int-mtls-ovl","namespace":"kube-test"}
{"level":"info","ts":"2026-07-09T07:49:36Z","logger":"controller.VLCluster","msg":"updating PDB name=kube-test/lyo0-int-mtls-ovl-vlstorage, is_prev_nil=true","vlcluster":"lyo0-int-mtls-ovl","namespace":"kube-test","spec_diff":{"spec.selector.matchLabels['app']":{"--":"vlstorage"},"spec.selector.matchLabels['app.kubernetes.io/component']":{"++":"monitoring"},"spec.selector.matchLabels['app.kubernetes.io/name']":{"--":"lyo0-int-mtls-ovl","++":"vlstorage"},"spec.selector.matchLabels['managed-by']":{"++":"vm-operator"}}}
{"level":"info","ts":"2026-07-09T07:49:36Z","logger":"controller.VLCluster","msg":"immutable StatefulSet=kube-test/lyo0-int-mtls-ovl-vlstorage field changed, spec_diff=map[spec.podManagementPolicy:{<nil> OrderedReady Parallel} spec.selector.matchLabels['app']:{<nil> vlstorage <nil>} spec.selector.matchLabels['app.kubernetes.io/component']:{<nil> <nil> monitoring} spec.selector.matchLabels['app.kubernetes.io/name']:{<nil> lyo0-int-mtls-ovl vlstorage} spec.selector.matchLabels['managed-by']:{<nil> <nil> vm-operator}]","vlcluster":"lyo0-int-mtls-ovl","namespace":"kube-test"}
{"level":"info","ts":"2026-07-09T07:49:36Z","logger":"controller.VLCluster","msg":"waiting for StatefulSet=kube-test/lyo0-int-mtls-ovl-vlstorage to be removed ","vlcluster":"lyo0-int-mtls-ovl","namespace":"kube-test"}
{"level":"info","ts":"2026-07-09T07:49:36Z","logger":"controller.VLCluster","msg":"waiting for StatefulSet=kube-test/lyo0-int-mtls-ovl-vlstorage to be removed (finalizers=[orphan])","vlcluster":"lyo0-int-mtls-ovl","namespace":"kube-test"}
{"level":"info","ts":"2026-07-09T07:49:36Z","logger":"controller.VLCluster","msg":"creating new StatefulSet=kube-test/lyo0-int-mtls-ovl-vlstorage","vlcluster":"lyo0-int-mtls-ovl","namespace":"kube-test"}
{"level":"info","ts":"2026-07-09T07:49:36Z","logger":"controller.VLCluster","msg":"possible BUG, cannot find target PVC=vlstorage-volume-lyo0-int-mtls-ovl-vlstorage-lyo0-int-mtls-ovl-vlstorage-0 in new statefulset by claim name=vlstorage-volume-lyo0-int-mtls-ovl-vlstorage-lyo0-int-mtls-ovl-vlstorage","vlcluster":"lyo0-int-mtls-ovl","namespace":"kube-test"}
{"level":"info","ts":"2026-07-09T07:49:36Z","logger":"controller.VLCluster","msg":"ignoring custom update behavior settings with update strategy=RollingUpdate on StatefulSet=kube-test/lyo0-int-mtls-ovl-vlstorage","vlcluster":"lyo0-int-mtls-ovl","namespace":"kube-test"}
{"level":"error","ts":"2026-07-09T07:50:56Z","logger":"manager","msg":"Reconciler error","controller":"vlcluster","controllerGroup":"operator.victoriametrics.com","controllerKind":"VLCluster","VLCluster":{"name":"lyo0-int-mtls-ovl","namespace":"kube-test"},"namespace":"kube-test","name":"lyo0-int-mtls-ovl","reconcileID":"acb40d4c-6361-4bfa-ba47-2e1637f2279a","error":"failed create or update vlcluster: cannot reconcile storage: cannot ensure that statefulset is ready with strategy=\"RollingUpdate\": cannot find any pod for selector=\"app.kubernetes.io/component=monitoring,app.kubernetes.io/instance=lyo0-int-mtls-ovl,app.kubernetes.io/name=vlstorage,managed-by=vm-operator\", check kubernetes events: cannot wait for statefulSet=lyo0-int-mtls-ovl-vlstorage to become ready: context deadline exceeded"}
{"level":"info","ts":"2026-07-09T07:50:56Z","logger":"controller.VLCluster","msg":"possible BUG, cannot find target PVC=vlstorage-volume-lyo0-int-mtls-ovl-vlstorage-lyo0-int-mtls-ovl-vlstorage-0 in new statefulset by claim name=vlstorage-volume-lyo0-int-mtls-ovl-vlstorage-lyo0-int-mtls-ovl-vlstorage","vlcluster":"lyo0-int-mtls-ovl","namespace":"kube-test"}
{"level":"error","ts":"2026-07-09T07:51:01Z","logger":"manager","msg":"Reconciler error","controller":"vlcluster","controllerGroup":"operator.victoriametrics.com","controllerKind":"VLCluster","VLCluster":{"name":"lyo0-int-mtls-ovl","namespace":"kube-test"},"namespace":"kube-test","name":"lyo0-int-mtls-ovl","reconcileID":"f3554dab-7af9-4ff7-88af-274cec78f292","error":"failed create or update vlcluster: cannot reconcile storage: cannot handle rolling-update on StatefulSet=kube-test/lyo0-int-mtls-ovl-vlstorage: actual pod count: 0 less than needed: 2, possible statefulset misconfiguration"}
{"level":"info","ts":"2026-07-09T07:51:01Z","logger":"controller.VLCluster","msg":"possible BUG, cannot find target PVC=vlstorage-volume-lyo0-int-mtls-ovl-vlstorage-lyo0-int-mtls-ovl-vlstorage-0 in new statefulset by claim name=vlstorage-volume-lyo0-int-mtls-ovl-vlstorage-lyo0-int-mtls-ovl-vlstorage","vlcluster":"lyo0-int-mtls-ovl","namespace":"kube-test"}
{"level":"error","ts":"2026-07-09T07:51:06Z","logger":"manager","msg":"Reconciler error","controller":"vlcluster","controllerGroup":"operator.victoriametrics.com","controllerKind":"VLCluster","VLCluster":{"name":"lyo0-int-mtls-ovl","namespace":"kube-test"},"namespace":"kube-test","name":"lyo0-int-mtls-ovl","reconcileID":"135c6dc5-7589-4191-886b-ca223a57b32a","error":"failed create or update vlcluster: cannot reconcile storage: cannot handle rolling-update on StatefulSet=kube-test/lyo0-int-mtls-ovl-vlstorage: actual pod count: 0 less than needed: 2, possible statefulset misconfiguration"}
{"level":"info","ts":"2026-07-09T07:51:14Z","logger":"controller.VLCluster","msg":"possible BUG, cannot find target PVC=vlstorage-volume-lyo0-int-mtls-ovl-vlstorage-lyo0-int-mtls-ovl-vlstorage-0 in new statefulset by claim name=vlstorage-volume-lyo0-int-mtls-ovl-vlstorage-lyo0-int-mtls-ovl-vlstorage","vlcluster":"lyo0-int-mtls-ovl","namespace":"kube-test"}
{"level":"error","ts":"2026-07-09T07:51:19Z","logger":"manager","msg":"Reconciler error","controller":"vlcluster","controllerGroup":"operator.victoriametrics.com","controllerKind":"VLCluster","VLCluster":{"name":"lyo0-int-mtls-ovl","namespace":"kube-test"},"namespace":"kube-test","name":"lyo0-int-mtls-ovl","reconcileID":"2f5c4ed4-5f7b-49d9-9956-e9d6ae489639","error":"failed create or update vlcluster: cannot reconcile storage: cannot handle rolling-update on StatefulSet=kube-test/lyo0-int-mtls-ovl-vlstorage: actual pod count: 0 less than needed: 2, possible statefulset misconfiguration"}
{"level":"error","ts":"2026-07-09T07:51:35Z","logger":"manager","msg":"Reconciler error","controller":"vlcluster","controllerGroup":"operator.victoriametrics.com","controllerKind":"VLCluster","VLCluster":{"name":"lyo0-int-mtls-ovl","namespace":"kube-test"},"namespace":"kube-test","name":"lyo0-int-mtls-ovl","reconcileID":"b3bfb0ff-76b7-405d-b22e-7f4bdc761dcd","error":"failed create or update vlcluster: cannot reconcile storage: got 0 pvcs under kube-test for selector map[app.kubernetes.io/component:monitoring app.kubernetes.io/instance:lyo0-int-mtls-ovl app.kubernetes.io/name:vlstorage managed-by:vm-operator], statefulset could not be working"}
{"level":"error","ts":"2026-07-09T07:51:35Z","logger":"manager","msg":"Reconciler error","controller":"vlcluster","controllerGroup":"operator.victoriametrics.com","controllerKind":"VLCluster","VLCluster":{"name":"lyo0-int-mtls-ovl","namespace":"kube-test"},"namespace":"kube-test","name":"lyo0-int-mtls-ovl","reconcileID":"b2e07bf1-fb59-4602-aed2-3a733b38eab8","error":"failed create or update vlcluster: cannot reconcile storage: got 0 pvcs under kube-test for selector map[app.kubernetes.io/component:monitoring app.kubernetes.io/instance:lyo0-int-mtls-ovl app.kubernetes.io/name:vlstorage managed-by:vm-operator], statefulset could not be working"}

    As I understand the selector are wrong for getting active PVC for vlstorage statefulset :
    Labels from statefulset produced by Helm

    labels:
  app.kubernetes.io/component: vlstorage
  app.kubernetes.io/instance: lyo0-int-mtls-ovl
  app.kubernetes.io/managed-by: Helm
  app.kubernetes.io/name: lyo0-int-mtls-ovl
  app.kubernetes.io/version: v1.51.0-enterprise
  helm.sh/chart: victoria-logs-cluster-0.2.7

    Labels for PVC produced by Helm :

    labels:
    app: vlstorage
    app.kubernetes.io/instance: lyo0-int-mtls-ovl
    app.kubernetes.io/name: lyo0-int-mtls-ovl

    EDIT1 : could it be possible to have an alignment on labels produced by Helm and labels produced nby operator like app.kubernetes.io/manged-by used by Helm and managed-by used by operator

  5. ccaillet1974 commented on Jul 9, 2026

    @ccaillet1974
    ccaillet1974
    on Jul 9, 2026
    Author

    added a PR with a fix, but for now you can override that value using spec.vlstorage.storage.volumeClaimTemplate.name

    The correct usage is : spec.vlstorage.storage.volumeClaimTemplate.metadata.name

  6. ccaillet1974 commented on Jul 9, 2026

    @ccaillet1974
    ccaillet1974
    on Jul 9, 2026
    Author

    Better isolation for having relevant logs from operator:

    {"level":"info","ts":"2026-07-09T14:42:54Z","logger":"controller.VLCluster","msg":"object has changes with previous state, applying changes","vlcluster":"lyo0-int-mtls-ovl","namespace":"kube-test"}
{"level":"info","ts":"2026-07-09T14:42:54Z","logger":"controller.VLCluster","msg":"creating new ServiceAccount=kube-test/lyo0-int-mtls-ovl-vlcluster","vlcluster":"lyo0-int-mtls-ovl","namespace":"kube-test"}
{"level":"info","ts":"2026-07-09T14:42:54Z","logger":"controller.VLCluster","msg":"updating PDB name=kube-test/lyo0-int-mtls-ovl-vlstorage, is_prev_nil=true","vlcluster":"lyo0-int-mtls-ovl","namespace":"kube-test","spec_diff":{"spec.selector.matchLabels['app']":{"--":"vlstorage"},"spec.selector.matchLabels['app.kubernetes.io/component']":{"++":"monitoring"},"spec.selector.matchLabels['app.kubernetes.io/name']":{"--":"lyo0-int-mtls-ovl","++":"vlstorage"},"spec.selector.matchLabels['managed-by']":{"++":"vm-operator"}}}
{"level":"info","ts":"2026-07-09T14:42:54Z","logger":"controller.VLCluster","msg":"immutable StatefulSet=kube-test/lyo0-int-mtls-ovl-vlstorage field changed, spec_diff=map[spec.podManagementPolicy:{<nil> OrderedReady Parallel} spec.selector.matchLabels['app']:{<nil> vlstorage <nil>} spec.selector.matchLabels['app.kubernetes.io/component']:{<nil> <nil> monitoring} spec.selector.matchLabels['app.kubernetes.io/name']:{<nil> lyo0-int-mtls-ovl vlstorage} spec.selector.matchLabels['managed-by']:{<nil> <nil> vm-operator}]","vlcluster":"lyo0-int-mtls-ovl","namespace":"kube-test"}
{"level":"info","ts":"2026-07-09T14:42:54Z","logger":"controller.VLCluster","msg":"waiting for StatefulSet=kube-test/lyo0-int-mtls-ovl-vlstorage to be removed ","vlcluster":"lyo0-int-mtls-ovl","namespace":"kube-test"}
{"level":"info","ts":"2026-07-09T14:42:54Z","logger":"controller.VLCluster","msg":"waiting for StatefulSet=kube-test/lyo0-int-mtls-ovl-vlstorage to be removed (finalizers=[orphan])","vlcluster":"lyo0-int-mtls-ovl","namespace":"kube-test"}
{"level":"info","ts":"2026-07-09T14:42:54Z","logger":"controller.VLCluster","msg":"waiting for StatefulSet=kube-test/lyo0-int-mtls-ovl-vlstorage to be removed (finalizers=[orphan])","vlcluster":"lyo0-int-mtls-ovl","namespace":"kube-test"}
{"level":"info","ts":"2026-07-09T14:42:54Z","logger":"controller.VLCluster","msg":"waiting for StatefulSet=kube-test/lyo0-int-mtls-ovl-vlstorage to be removed (finalizers=[orphan])","vlcluster":"lyo0-int-mtls-ovl","namespace":"kube-test"}
{"level":"info","ts":"2026-07-09T14:42:54Z","logger":"controller.VLCluster","msg":"creating new StatefulSet=kube-test/lyo0-int-mtls-ovl-vlstorage","vlcluster":"lyo0-int-mtls-ovl","namespace":"kube-test"}
{"level":"error","ts":"2026-07-09T14:42:55Z","logger":"manager","msg":"Reconciler error","controller":"vlcluster","controllerGroup":"operator.victoriametrics.com","controllerKind":"VLCluster","VLCluster":{"name":"lyo0-int-mtls-ovl","namespace":"kube-test"},"namespace":"kube-test","name":"lyo0-int-mtls-ovl","reconcileID":"81994d1a-5297-4650-8c5f-eb3c05bb5700","error":"failed create or update vlcluster: cannot reconcile storage: got 0 pvcs under kube-test for selector map[app.kubernetes.io/component:monitoring app.kubernetes.io/instance:lyo0-int-mtls-ovl app.kubernetes.io/name:vlstorage managed-by:vm-operator], statefulset could not be working"}
{"level":"error","ts":"2026-07-09T14:42:55Z","logger":"manager","msg":"Reconciler error","controller":"vlcluster","controllerGroup":"operator.victoriametrics.com","controllerKind":"VLCluster","VLCluster":{"name":"lyo0-int-mtls-ovl","namespace":"kube-test"},"namespace":"kube-test","name":"lyo0-int-mtls-ovl","reconcileID":"f84285ee-4309-4eca-8607-025ceca137db","error":"failed create or update vlcluster: cannot reconcile storage: got 0 pvcs under kube-test for selector map[app.kubernetes.io/component:monitoring app.kubernetes.io/instance:lyo0-int-mtls-ovl app.kubernetes.io/name:vlstorage managed-by:vm-operator], statefulset could not be working"}
{"level":"error","ts":"2026-07-09T14:42:55Z","logger":"manager","msg":"Reconciler error","controller":"vlcluster","controllerGroup":"operator.victoriametrics.com","controllerKind":"VLCluster","VLCluster":{"name":"lyo0-int-mtls-ovl","namespace":"kube-test"},"namespace":"kube-test","name":"lyo0-int-mtls-ovl","reconcileID":"5dc2e870-0135-41a4-92f9-baa94ead9dd5","error":"failed create or update vlcluster: cannot reconcile storage: got 0 pvcs under kube-test for selector map[app.kubernetes.io/component:monitoring app.kubernetes.io/instance:lyo0-int-mtls-ovl app.kubernetes.io/name:vlstorage managed-by:vm-operator], statefulset could not be working"}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions