Allow configuring vtbackup storage separately from tablet pools (#661)

[mcrauwel]

Summary

Fixes #661.

This change adds an optional vtbackup block alongside tabletPools in VitessShardTemplate. It allows vtbackup Pods to use scratch storage that differs from the tablet-pool storage they otherwise inherit.

Historically, vtbackup Pods inherited the first tablet pool's dataVolumeClaimTemplate. That behavior allocates a PVC and an underlying disk per shard, even for the initial backup of an empty database. Large clusters can therefore allocate substantial unused storage or fail to bootstrap when those PVCs cannot bind.

Configuration

Set useEmptyDirForInitialBackup to let the initial backup use ephemeral scratch space while tablet Pods retain their normal PVCs:

spec:
  keyspaces:
  - name: commerce
    partitionings:
    - equal:
        parts: 2
        shardTemplate:
          vtbackup:
            useEmptyDirForInitialBackup: true
          tabletPools:
          - cell: zone1
            type: replica
            replicas: 3
            dataVolumeClaimTemplate:
              accessModes: [ReadWriteOnce]
              resources:
                requests:
                  storage: 100Gi
            vttablet: {}
            mysqld: {}

A shard template can instead provide a dedicated PVC template for vtbackup Pods:

vtbackup:
  dataVolumeClaimTemplate:
    storageClassName: cheap-disk
    accessModes: [ReadWriteOnce]
    resources:
      requests:
        storage: 10Gi

useEmptyDirForInitialBackup: true and dataVolumeClaimTemplate are mutually exclusive. The CRD rejects configurations that set both.

Semantics

vtbackup configuration	Initial backup	Scheduled backup
Block omitted or empty	Inherits the first tablet pool's PVC template	Inherits the first tablet pool's PVC template
useEmptyDirForInitialBackup: true	Uses ephemeral emptyDir scratch space	Inherits the first tablet pool's PVC template
dataVolumeClaimTemplate set	Uses the vtbackup-specific PVC template	Uses the vtbackup-specific PVC template

Scheduled backups retain persistent scratch storage when useEmptyDirForInitialBackup is set because vtbackup restores the latest full database before taking a new backup. Using an unbounded emptyDir for that workload could cause node disk pressure or eviction.

Omitting the block, supplying an empty block, or explicitly setting useEmptyDirForInitialBackup: false preserves the historical behavior. Tablet Pod storage is unaffected.

Storage reconciliation

The operator records a hash of the desired vtbackup PVC template on the initial-backup Pod and PVC. It separately records a hash of the Pod's effective /vt/vtdataroot mount and referenced volume source on the Pod. This accounts for extraVolumes and extraVolumeMounts that override the default data mount without changing the PVC template.

When the storage configuration changes, the operator:

1. Preserves a running initial-backup Pod.
2. Recreates a non-running initial-backup Pod when its effective data mount or volume source changes.
3. Replaces the PVC, after the old Pod is gone, only when the desired PVC template changes.
4. Stops and retries if it cannot verify Pod absence before replacing a PVC, preventing a replacement Pod from starting against stale storage.

For upgrades, the controller adopts hashless resources and resources created by the earlier single-hash implementation when their effective Pod and PVC configuration already matches the desired configuration. The comparison accounts for Kubernetes-assigned defaults and bindings without conflating an explicit empty storage class or a user-prebound volume with controller-assigned values. If the controller cannot read a legacy PVC, it preserves the Pod and retries instead of making a destructive decision from incomplete state.

This allows the controller to converge across PVC-to-emptyDir, emptyDir-to-PVC, custom-PVC-template, and effective Pod-volume changes. In particular, setting useEmptyDirForInitialBackup: true can recover an initial backup that is Pending because its inherited PVC cannot bind.

Scope

This change only separates vtbackup data-volume configuration from tablet-pool storage. Broader vtbackup Pod overrides can build on the dedicated vtbackup block in follow-up work.

Tests

• Unit coverage for inherited, explicit emptyDir, and custom templates on initial and scheduled backups.
• Scheduled-backup coverage confirming that useEmptyDirForInitialBackup applies only to the initial backup.
• Reconciliation coverage for PVC-to-emptyDir, emptyDir-to-PVC, custom-template, and effective Pod-volume transitions.
• Coverage confirming that effective Pod-volume changes preserve an unchanged PVC.
• Legacy-adoption coverage for hashless and earlier single-hash resources, Kubernetes-defaulted PVC fields, explicit empty storage classes, user-prebound volumes, and transient PVC lookup failures.
• Fault-injection coverage confirming that uncertain Pod or PVC reads preserve existing resources and retry safely.
• Coverage confirming that storage changes do not interrupt a running initial backup.
• Integration coverage confirming that useEmptyDirForInitialBackup: true removes the initial-backup PVC while tablet Pods retain theirs.
• Generated CRDs, deepcopy code, API reference documentation, and the bundled operator manifest are updated.

Local verification:

• go test -count=1 ./pkg/...
• go test -race -count=1 ./pkg/controller/vitessshard ./pkg/controller/vitessbackupschedule ./pkg/operator/vttablet
• Changed-line golangci-lint: 0 issues
• git diff --check