Skip to content

quickstart.md

Latest commit

 

History

History
283 lines (245 loc) · 16.6 KB

quickstart.md

File metadata and controls

283 lines (245 loc) · 16.6 KB
Raw

Quickstart

Get a cache-aware inference setup running in about five minutes. This page shows the minimum CacheBackend you write today, what it wires up, and where to go next. It assumes the inference-cache operator (controller + policy server + CRDs) is already installed in the cluster.

5-minute quickstart

A CacheBackend binds to your inference engine pods by label and makes their KV cache reusable across requests. Here is the minimum-viable spec:

apiVersion: inferencecache.io/v1alpha1
kind: CacheBackend
metadata:
  name: my-cache
spec:
  runtime: VLLM                # inference runtime
  type: LMCache                # engine-local cache integration
  engineSelector:
    matchLabels:
      inferencecache.io/cache-domain: my-engine-cache
  lmCache:
    topology: PodLocal
    chunkSizeTokens: 256
    podLocal:
      server:
        image: docker.io/lmcache/standalone@sha256:b813bf0bb616d1012b6a6edcbd4a44f1576dbbdaa857962e56d48b9f7c127d13
        port: 5555
        l1Capacity: 4Gi
        maxWorkers: 4
        resources:
          requests:
            cpu: "1"
            memory: 5Gi
          limits:
            cpu: "2"
            memory: 6Gi
  observation:
    modelID: Qwen/Qwen2.5-0.5B-Instruct

That is the whole CacheBackend. The runtime/cache pair, ownership domain, model identity, and MP server contract are explicit; integration.role defaults to ReadWrite, the readiness gate's firstEventTimeout defaults to 5m, and integration.failOpen defaults to true. Omitting remoteStorage selects host-only MP: L1 is per engine Pod and there is no cross-Pod sharing. Add an explicit Redis L3 when sharing is required.

To share one MP L1 across several engine Pods on each GPU node, use the focused NodeLocal samples for vLLM or SGLang. Configure the controller's --node-local-shm-cleanup-image once with the cleanup helper digest published by the same inference-cache release. NodeLocal is an advanced host-bound topology. The inference system schedules each engine without CacheBackend changing its placement; the controller then creates one shared server Pod on every node that actually has an active selected engine. The server and engine mount only the backend's /dev/shm/inference-cache/<uid> host directory as /dev/shm and declare MP and HTTP host ports. Engines are held in an init gate until their own node's server reports the exact CacheBackend name/UID/generation and healthy config. Co-schedule only mutually trusted engines and enforce host-port access with node firewalls; hostNetwork bypasses Kubernetes NetworkPolicy. L1 capacity is per node, and maxGPUWorkers must cover the maximum engine instances on that node. The focused samples retain an idle per-node server and its L1 for 300 seconds so an engine restart can reuse them; set idleRetentionSeconds: 0 for immediate cleanup.

One label does the binding. Every non-empty selector contains only inferencecache.io/cache-domain; its namespace-scoped value must also appear on your engine pods' template labels. Other Pod labels do not participate in CacheBackend ownership. That domain match lets the mutating Pod webhook inject the cache wiring at pod CREATE. Drift them apart and the engine runs uncached — kubectl get cachebackend then shows MATCHED: 0.

The CacheBackend injects the MP server into matching engine Pods; it does not own or replace their image. To get a working end-to-end setup you need an engine image containing a compatible LMCache client plus Pods carrying that label and publishing KV events. A paired shape is available at:

kubectl apply -f config/samples/cachebackend-with-engine.yaml

That file ships the typed host-only CacheBackend plus a matching vLLM Deployment. Repository admission validation does not prove the supplied engine image contains the connector; normal engine startup is the authoritative check. Acting on LookupRoute hints remains the gateway's responsibility.

One install-time prerequisite for observability. The piece that publishes KV events — the kvevent-subscriber sidecar — is only auto-attached when the controller runs with --kvevent-subscriber-image set, which is empty by default. Engine↔cache wiring (KV reuse) works without it, but until it is set no KV events are reported: a managed backend holds at Ready=False (AwaitingFirstKVEvent) and then, once the firstEventTimeout window (default 5m) elapses, flips to Ready=False (NoKVEventsObserved) with Degraded=True; PREFIXES stays 0 throughout. Set that flag on the controller to get the Ready/observability surface below. (Externally owned backends are exempt from this gate — they go Ready as soon as admission accepts the endpoint.)

A second readiness gate composes on top. Once the KV-event gate above clears, the controller runs a synthetic functional self-test (ingest a synthetic prefix through the server's in-process index.Ingest → look it up → optional tier-2 round-trip; the gRPC PublishEvent / ReportCacheState subscriber path is NOT exercised) and publishes a FunctionalProbeOK condition on the CR. Every stage ok or skipped (the passing states ProbeResult.AllPassed accepts; today's clean installs report t2=skipped because no T2Prober is wired) → Ready=True stays; a per-stage failure downgrades Ready=False with a stage-specific reason (ProbeIngestFailed / ProbeRoutingFailed / ProbeT2Failed). The gate is cascade-prevented — FunctionalProbeOK does not appear while upstream readiness is not Ready=True (KV-event-pending, rollout in progress, replicas unavailable, scaled-to-zero, …). Disabled by passing --server-probe-url="" to the controller (the condition then never appears); externally owned backends are exempt from this gate too. See Troubleshooting for the per-reason runbook.

What you get

Once the backend is Ready and engine pods are bound, three things are live:

  • Cache-aware routing. The policy server records which engine replicas hold which prompt prefixes warm and answers LookupRoute with that hint, so a gateway can route a request to a replica that already has its prefix cached — lower time-to-first-token, less recompute. (inference-cache provides the hint; the gateway owns the routing decision.)

  • KV reuse. Matched engine pods get the LMCache wiring injected automatically, so their KV cache is offloaded to the per-Pod MP server and reused instead of being recomputed per request.

  • Observability. kubectl get cachebackend surfaces the live state:

    $ kubectl get cachebackend
NAME            TYPE      READY   MATCHED   ENDPOINT   PREFIXES   LASTEVENT   AGE
my-cache        LMCache   True    1         <none>     128        12s         3m

    READY flips to True only after the MP connector/server coverage baseline and the KV-event gate (a real event observed, not merely the pod being reachable). When functional probing is enabled and not bypassed, a per-stage failed probe outcome additionally downgrades Ready=False. On a /probe transport/HTTP error the gate is fail-soft AND sticky in two distinct ways depending on prior state:

    • If no FunctionalProbeOK is present, or it's currently True, a transport error publishes FunctionalProbeOK=Unknown/ProbeError and leaves Ready alone — a transient server outage does NOT hold an otherwise-Ready backend out of Ready=True.
    • If FunctionalProbeOK=False/Probe*Failed is already published, a transport error preserves the False condition AND keeps Ready=False with the prior stage reason. The False is sticky until a successful probe explicitly resolves it — a transient server outage must NOT mask a known per-stage regression by fading the condition to Unknown and then to Ready=True.

    The functional-probe gate can also be disabled cluster-wide (--server-probe-url="") or skipped per-CR via the inferencecache.io/skip-functional-probe: "true" annotation. Any active gate that reports a per-stage failure can hold the backend at Ready=False with a stage-specific reason on .status.conditions[] — see Troubleshooting. ENDPOINT is empty for host-only PodLocal or NodeLocal MP and contains only a configured remote L3 endpoint, never a local MP connector address. MATCHED is the engine-pod count the selector binds, and PREFIXES / LASTEVENT show the cache actually receiving state.

Next steps

  • Recipe catalog — copy-paste a curated scenario: config/samples/README.md. Includes CPU dev, GPU production, external cache, multi-tenant, and engine tuning.
  • Engine binding mental model — how the selector → webhook → injection lifecycle works and its failure modes: concepts/cachebackend-engine-binding.md.
  • Tuning engine injection — the engineOverrides surface (amend the injected args/env without losing the integration): concepts/cachebackend-engine-overrides.md.
  • Per-namespace lookup/eviction tuning — configured on CachePolicy (evictionTTL, minimumPrefixTokens, lookupTimeoutMs). Until a dedicated concept doc lands, kubectl explain cachepolicy.spec is the field reference; the recipe-gpu-production.yaml recipe shows a production CachePolicy.
  • Full field reference — kubectl explain cachebackend.spec (and cachepolicy.spec, cachetenant.spec) for every field and its defaults.

Troubleshooting

A managed backend's Ready status is the composition of three gates (managed-readiness → KV-event gate → functional-probe gate), so the READY column of kubectl get cachebackend only tells half the story. kubectl get cachebackend <name> -o yaml and scan .status.conditions[] for which gate is unhappy. The condition .reason is the actionable string.

Ready=False / AwaitingFirstKVEvent

The KV-event gate is still waiting for the first KV event. Common causes:

  • The controller is running with --kvevent-subscriber-image unset (the default). No subscriber sidecar is injected, so no events ever flow. Set the flag on the controller Deployment.
  • The subscriber sidecar is present but cannot reach the engine's KV-event publisher. Check the subscriber pod's logs for ZMQ connect errors; verify the engine container has --kv-events-config set (see the recipe-* samples for the canonical shape).
  • The engine container is running but has not received any prompts yet (the first BlockStored is published on the first request). Send a single chat completion through the engine and re-check.

If firstEventTimeout elapses (default 5m) without an event, the condition flips to Ready=False / NoKVEventsObserved and Degraded=True; diagnostic steps are the same.

Ready=False / Probe*Failed (with FunctionalProbeOK=False)

The KV-event gate cleared (real engine pods reported state) but the controller's synthetic functional round-trip failed. Read the condition's .message first — the controller embeds the server's stage-specific diagnostic. By .reason:

.reason Stage What it means First-response
ProbeIngestFailed ingest The server's in-process index Ingest path is dropping writes. Does not indicate a subscriber problem — that path isn't exercised by the probe (the gRPC PublishEvent / ReportCacheState subscriber surface is bypassed by design; see the stage description at the top of this section). Read the FunctionalProbeOK condition's .message for the server's stage diagnostic; check inferencecache_backend_probe_result_total{backend="<namespace>/<name>", stage="ingest", result="failed"} for the trend. The server-side inferencecache_index_entries gauge is fine as background index-health context but cannot confirm whether the synthetic probe entry landed — probe entries live under the reserved inferencecache.io/probe tenant, which is excluded from the cap-accounting gauge by design. Inspect server logs for internal/index errors. Confirm inferencecache_server_up == 1.
ProbeRoutingFailed lookup LookupRoute did not return a clean PREFIX_MATCH for the probe's reserved replica. Two failure modes share this reason and are disambiguated by the condition .message: (a) the lookup returned a non-PREFIX_MATCH index strategy — NO_HINT (the cleanly-missing case), TENANT_HOT, UNKNOWN_TENANT, UNKNOWN_MODEL, or UNKNOWN_HASH_SCHEME — likely an internal hash_scheme regression that dropped the probe's scheme on ingest (an empty scheme fails open and produces NO_HINT) or a lookup-filter regression in internal/index/internal/server; (b) the lookup did return PREFIX_MATCH but the probe's reserved replica isn't among the scored replicas — a probe-id-derivation or reserved-replica-collision regression. (Note: TIMEOUT is produced by the gRPC LookupRoute handler's deadline path and is NOT reachable here — the probe calls index.LookupRoute directly through an in-process seam.) The probe's hashScheme derives from spec.runtime for canonical resources (admission rejects unsupported runtime/cache pairs). Read the FunctionalProbeOK condition's .message first — the server names which failure mode hit. Check inferencecache_backend_probe_result_total{backend="<namespace>/<name>", stage="routing", result="failed"} for the trend. (The server-side inferencecache_lookup_route_calls_total is NOT the right surface here — same reason: the probe bypasses the gRPC handler that emits that metric.) Inspect server logs for internal/index lookup-path errors.
ProbeT2Failed tier-2 The tier-2 put/get cycle failed (LMCache, today). Only reachable when a T2Prober is wired into the server — none is registered in the current revision, so this condition does not appear on a clean install. Will be applicable once a T2Prober ships; not actionable today.

FunctionalProbeOK=Unknown / ProbeError

The controller could not reach the server's /probe endpoint at all (transport error, 5xx, audience-bound TokenReview rejected the call). A brief server outage produces this state without flapping Ready — unless FunctionalProbeOK is already False/Probe*Failed, in which case the prior stage failure is sticky and Ready stays downgraded (a transient server outage must not mask a real regression by fading the condition back to Unknown and then to Ready=True).

First-response:

  • Check that --server-probe-url is reachable from the controller pod (intra-cluster inference-cache-server:8081 by default).
  • Verify the projected SA token is mounted at /var/run/secrets/inferencecache.io/controller-token/token.
  • Confirm the audience-bound TokenReview accepts the controller SA — the server's --controller-audience flag must equal inferencecache.io/controller. A mismatch surfaces as repeated ProbeError on every reconcile.

FunctionalProbeOK=True / ProbeBypassed

An operator has annotated the CR with inferencecache.io/skip-functional-probe: "true". The probe is skipped entirely and the gate does not downgrade Ready. Remove the annotation when you no longer need the bypass — a bypassed backend with a real regression still ships broken cache state.

FunctionalProbeOK is missing from .status.conditions[]

Four possible causes, in order of how common they are on a healthy install:

  • Any upstream gate is holding Ready != True. The probe gate is cascade-prevented from running whenever the composed upstream verdict (managed-readiness baseline + KV-event gate) is not Ready=True. That includes rollout-in-progress, replicas unavailable, scaled-to-zero, and AwaitingFirstKVEvent / NoKVEventsObserved — not only KV-event-pending. Resolve the upstream condition first; the probe condition appears on the next reconcile after the upstream clears.
  • Functional probing is disabled on the controller. The --server-probe-url="" flag turns the gate off entirely. Any stale FunctionalProbeOK left over from a previous wiring is also cleared on the next reconcile in this mode.
  • The CR uses spec.remoteStorage.ownership: External. Externally owned backends are wholly exempt from the probe gate — the controller never drives a round-trip against a cache it does not manage.
  • The CR is on an Unmanaged path (for example, an unsupported runtime or provider binding that bypassed admission). The probe gate is exempt from these paths and any prior FunctionalProbeOK condition is removed on transition.