RecoverFromPage throws NullReferenceException on non-inline keys, preventing checkpoint recovery

[nixxcz]

Describe the bug

Garnet accepts and successfully checkpoints a non-inline key, but cannot recover that checkpoint. Restart fails with a NullReferenceException from LogRecord.get_KeyBytes() in RecoverFromPage, and the server exits without listening.

This reproduces with ordinary SET and HSET, with or without --storage-tier. It does not require cluster mode, custom modules, concurrent requests, or injected corruption. All writes and both SAVE commands return success before restart.

The failed server also exits with code 0 after logging Unable to initialize server due to exception, which may hide the startup failure from a service manager.

Steps to reproduce the bug

The script uses fresh containers, preserves the server's writable layer across docker restart, and cleans up everything it creates.

#!/usr/bin/env bash
set -euo pipefail

suffix="$$"
network="garnet-overflow-key-${suffix}"
server="garnet-overflow-key-server-${suffix}"
client="garnet-overflow-key-client-${suffix}"

cleanup() {
  docker rm -f "$client" "$server" >/dev/null 2>&1 || true
  docker network rm "$network" >/dev/null 2>&1 || true
}
trap cleanup EXIT

docker network create "$network" >/dev/null
docker run -d --name "$client" --network "$network" redis:7-alpine sleep 300 >/dev/null
docker run -d --name "$server" --network "$network" --network-alias garnet \
  ghcr.io/microsoft/garnet:2.1.8 \
  --aof --recover --fail-on-recovery-error true \
  --index 16m --memory 64m --page 1m --segment 16m \
  --checkpointdir /data >/dev/null

for attempt in $(seq 1 50); do
  if [ "$(docker exec "$client" redis-cli -h garnet PING 2>/dev/null)" = PONG ]; then
    break
  fi
  sleep 0.1
done

docker exec "$client" redis-cli -h garnet SET seed value
docker exec "$client" redis-cli -h garnet SAVE
key="$(python3 -c 'print("k" * 1024)')"
docker exec "$client" redis-cli -h garnet HSET "$key" field value
docker exec "$client" redis-cli -h garnet SAVE

docker restart --time 2 "$server"
sleep 2
docker inspect --format 'status={{.State.Status}} exitCode={{.State.ExitCode}}' "$server"
docker logs "$server" 2>&1 | grep -E 'Error during recovery|get_KeyBytes|RecoverFromPage|Unable to initialize'

Observed result:

OK
OK
1
OK
status=exited exitCode=0

Error during recovery of store; storeVersion = 0;
System.NullReferenceException: Object reference not set to an instance of an object.
   at Tsavorite.core.LogRecord.get_KeyBytes() in .../Allocator/LogRecord.cs:line 1033
   at Tsavorite.core.TsavoriteKV`2.RecoverFromPage(...) in .../Recovery/Recovery.cs:line 1310
   at Tsavorite.core.TsavoriteKV`2.ProcessReadSnapshotPage(...) in .../Recovery/Recovery.cs:line 1221
   at Tsavorite.core.TsavoriteKV`2.RecoverSnapshotPages(...) in .../Recovery/Recovery.cs:line 1142
   at Tsavorite.core.TsavoriteKV`2.RecoverHybridLogFromSnapshotFileAsync(...) in .../Recovery/Recovery.cs:line 983
Unable to initialize server due to exception: Object reference not set to an instance of an object.

Additional isolated cells:

Version	Command	Key bytes	Storage tier	Restart result
2.1.8	HSET	16	enabled	Server runs; HGET returns value
2.1.8	HSET	1,024	enabled	Same recovery exception
2.1.8	HSET	65,536	enabled	Same recovery exception
2.1.8	SET	1,024	enabled	Same recovery exception
2.1.8	HSET	1,024	disabled	Same recovery exception
2.1.7	HSET	1,024	enabled	Same recovery exception

The exact key-length boundary was not measured.

Expected behavior

The key and value survive checkpoint recovery and are readable after restart. If recovery prevents startup, the server process exits with a nonzero status.

Release version

2.1.8 (ghcr.io/microsoft/garnet@sha256:20d9f41b02bcdf8aae1ad734fb7e31ef98b06efffdfa487eeea49687bdd3ff55); also reproduced on 2.1.7.

OS version

Docker Desktop 29.7.2, Linux arm64 containers.

Additional context

The failure appears to come from using an address-only LogRecord while rebuilding the hash index. RecoverFromPage constructs new LogRecord(pageStartPhysicalAddress + recordOffset) and then calls storeFunctions.GetKeyHashCode64(logRecord). The address-only constructor explicitly says it must not be used when an overflow allocator is needed. It leaves objectIdMap unset, while the non-inline key getter dereferences objectIdMap.GetOverflowByteArray(...).

This explains the stack and the short-key/long-key contrast. Recovery needs the page's object map when hashing a non-inline key; skipping the record would lose data. The same address-only construction is present on main at c323bbf7aa2783c1e9981edd387437d2618ff750 (source inspection only; that revision was not executed).