Fix JSONDecodeError when a layer's comfy_quant marker is empty

[chelsealong]

Fixes #16472

Problem

Loading the Qwen 2.1 image-edit template's text encoder
(qwen3vl_8b_int8_convrot.safetensors) fails with:

[ERROR] !!! Exception during processing !!! Expecting value: line 1 column 1 (char 0)
...
File "comfy\ops.py", line 1628, in _load_from_state_dict
    layer_conf = json.loads(layer_conf.numpy().tobytes())
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

Root cause

comfy/ops.py reads each quantized layer's config from a
<prefix>.comfy_quant tensor and always JSON-decodes it once the key
is present:

layer_conf = state_dict.pop(f"{prefix}comfy_quant", None)
if layer_conf is not None:
    layer_conf = json.loads(layer_conf.numpy().tobytes())

Some quantized checkpoints store this marker for a layer that isn't
actually quantized (e.g. the token embedding table in this
int8_convrot text encoder) as a present-but-empty tensor rather
than omitting the key entirely. json.loads(b"") raises
JSONDecodeError: Expecting value: line 1 column 1 (char 0) — exactly
the error in the traceback — which aborts loading the whole CLIP
model.

This code path exists twice in comfy/ops.py: once in the shared
_load_quantized_module helper used by Linear/Conv/etc., and once
in MixedPrecisionOps.Embedding._load_from_state_dict.

Fix

Treat an empty comfy_quant tensor the same as a missing one: skip
the JSON decode and fall through to the existing "not quantized" path
that loads the layer as a plain full-precision weight.

Test plan

Added two tests to tests-unit/comfy_quant/test_mixed_precision.py,
one per affected code path:

• test_empty_comfy_quant_marker_treated_as_unquantized covers the
  Linear/Conv path (_load_quantized_module).
• test_empty_comfy_quant_marker_on_embedding_treated_as_unquantized
  covers MixedPrecisionOps.Embedding._load_from_state_dict directly
  — the actual path in the reported traceback (the token embedding
  table of the int8_convrot text encoder).

Each builds a state dict with an empty comfy_quant tensor and
asserts the layer loads without error and stays a plain
(non-quantized) weight.

Verified the Embedding test reproduces the exact reported error
without the fix, matching the issue's traceback function and line
number:

$ git checkout HEAD~1 -- comfy/ops.py   # (fix removed)
$ python -m pytest tests-unit/comfy_quant/test_mixed_precision.py -k test_empty_comfy_quant_marker_on_embedding_treated_as_unquantized
...
comfy/ops.py:1628: in _load_from_state_dict
    layer_conf = json.loads(layer_conf.numpy().tobytes())
...
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
1 failed in 2.22s

$ git checkout HEAD -- comfy/ops.py     # (fix restored)
$ python -m pytest tests-unit/comfy_quant/test_mixed_precision.py -q
............
12 passed in 2.22s

ruff check comfy/ops.py tests-unit/comfy_quant/test_mixed_precision.py passes with no findings.

AI disclosure

This change was written by an autonomous Claude-based coding agent, with the diff reviewed for correctness before submission.

[coderabbitai]

Navigate logical layers of code changes, visualize relationships, and explore their blast radius.

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Repository: Comfy-Org/ComfyUI/.coderabbit.yaml

Review profile: ASSERTIVE

Plan: Advanced

Run ID: 9344060b-64d4-4e67-8894-8615c22441ac

📥 Commits

Reviewing files that changed from the base of the PR and between b33e2b5 and f7c3fd2.

📒 Files selected for processing (2)

• comfy/ops.py
• tests-unit/comfy_quant/test_mixed_precision.py

Included review availability: Your plan provides up to 10 included reviews per hour; 9 remain after this review.

📜 Recent review details

⏰ Context from checks skipped due to timeout. (6)

• GitHub Check: test (ubuntu-latest)
• GitHub Check: test (windows-2022)
• GitHub Check: test (macos-latest)
• GitHub Check: test (ubuntu-latest)
• GitHub Check: test (windows-latest)
• GitHub Check: test (macos-latest)

🧰 Additional context used

📓 Path-based instructions (3)

Core ML/diffusion engine.

⚙️ CodeRabbit configuration file

Files:

• comfy/ops.py

IMPORTANT: Only comment on issues directly introduced by this PR's code changes.

⚙️ CodeRabbit configuration file

Files:

• comfy/ops.py
• tests-unit/comfy_quant/test_mixed_precision.py

Documentation and README edits should be concise, factual, and tied to the changed behavior.

📄 CodeRabbit inference engine (AGENTS.md)

Files:

• comfy/ops.py
• tests-unit/comfy_quant/test_mixed_precision.py

🪛 ast-grep (0.45.3)

comfy/ops.py

[warning] 1347-1709: Do not use an empty list as a default parameter
Context: def mixed_precision_ops(quant_config={}, compute_dtype=torch.bfloat16, full_precision_mm=False, disabled=[]):
class MixedPrecisionOps(manual_cast):
_quant_config = quant_config
_compute_dtype = compute_dtype
_full_precision_mm = full_precision_mm
_disabled = disabled

    class Linear(torch.nn.Module, MixedPrecisionOp):
        _disabled_formats = disabled

        def __init__(self, in_features: int, out_features: int, bias: bool = True, device=None, dtype=None):
            super().__init__()

            self.factory_kwargs = {"device": device, "dtype": MixedPrecisionOps._compute_dtype}

            self.in_features = in_features
            self.out_features = out_features
            self._orig_shape = (out_features, in_features)
            if bias:
                self.bias = torch.nn.Parameter(torch.empty(out_features, **self.factory_kwargs))
            else:
                self.register_parameter("bias", None)

            self.tensor_class = None
            self._full_precision_mm = MixedPrecisionOps._full_precision_mm
            self._full_precision_mm_config = False

        def reset_parameters(self):
            return None

        def _load_from_state_dict(self, *args):
            _load_quantized_module(self, super()._load_from_state_dict, *args, load_extra_params=True)

        def state_dict(self, *args, destination=None, prefix="", **kwargs):
            sd = destination if destination is not None else {}
            return _quantized_weight_state_dict(self, sd, prefix, extra_quant_params=("input_scale", "pre_quant_scale"))

        def _forward(self, input, weight, bias):
            return torch.nn.functional.linear(input, weight, bias)

        def forward_comfy_cast_weights(
            self,
            input,
            compute_dtype=None,
            want_requant=False,
            weight_only_quant=False,
        ):
            if not weight_only_quant:
                with CastBiasWeightContext(
                    self,
                    input,
                    offloadable=True,
                    compute_dtype=compute_dtype,
                    want_requant=want_requant,
                ) as (weight, bias):
                    if self._full_precision_mm and isinstance(weight, QuantizedTensor):
                        weight = weight.dequantize()
                    return self._forward(input, weight, bias)

            with CastBiasWeightContext(
                self,
                input=None,
                dtype=self.weight.dtype,
                device=input.device,
                bias_dtype=input.dtype,
                offloadable=True,
                compute_dtype=compute_dtype,
                want_requant=True,
            ) as (weight, bias):
                weight = weight.to(dtype=input.dtype)
                return self._forward(input, weight, bias)

        def forward(self, input, *args, **kwargs):
            run_every_op()

            # ModelOpt AWQ-style smoothing
            pre_quant_scale = getattr(self, 'pre_quant_scale', None)
            if pre_quant_scale is not None:
                input = input * comfy.model_management.cast_to_device(pre_quant_scale, input.device, input.dtype)

            input_shape = input.shape
            reshaped_nd = False
            `#If` cast needs to apply lora, it should be done in the compute dtype
            compute_dtype = input.dtype

            _use_quantized = (
                getattr(self, 'layout_type', None) is not None and
                not isinstance(input, QuantizedTensor) and not self._full_precision_mm and
                not getattr(self, 'comfy_force_cast_weights', False) and
                len(self.weight_function) == 0 and len(self.bias_function) == 0
            )
            quantize_input = QUANT_ALGOS.get(getattr(self, 'quant_format', None), {}).get("quantize_input", True)

            # Training path: quantized forward with compute_dtype backward via autograd function
            if (input.requires_grad and _use_quantized and quantize_input):
                with CastBiasWeightContext(
                    self,
                    input,
                    offloadable=True,
                    compute_dtype=compute_dtype,
                    want_requant=True
                ) as (weight, bias):
                    scale = getattr(self, 'input_scale', None)
                    if scale is not None:
                        scale = comfy.model_management.cast_to_device(scale, input.device, None)

                    return QuantLinearFunc.apply(
                        input, weight, bias, self.layout_type, scale, compute_dtype
                    )

            # Inference path (unchanged)
            if _use_quantized and quantize_input:

                # Reshape >=3D tensors to 2D for quantization (needed for NVFP4 and others)
                input_reshaped = input.reshape(-1, input_shape[-1]) if input.ndim >= 3 else input

                # Fall back to non-quantized for non-2D tensors
                if input_reshaped.ndim == 2:
                    reshaped_nd = input.ndim >= 3
                    # dtype is now implicit in the layout class
                    scale = getattr(self, 'input_scale', None)
                    if scale is not None:
                        scale = comfy.model_management.cast_to_device(scale, input.device, None)
                    input = QuantizedTensor.from_float(input_reshaped, self.layout_type, scale=scale)

            weight_only_quant = _use_quantized and not quantize_input and isinstance(self.weight, QuantizedTensor)
            output = self.forward_comfy_cast_weights(
                input,
                compute_dtype,
                want_requant=isinstance(input, QuantizedTensor),
                weight_only_quant=weight_only_quant,
            )

            # Reshape output back to original rank if input was >2D
            if reshaped_nd:
                output = output.reshape((*input_shape[:-1], self.weight.shape[0]))

            return output

        def convert_weight(self, weight, inplace=False, **kwargs):
            if isinstance(weight, QuantizedTensor):
                return weight.dequantize()
            else:
                return weight

        def set_weight(self, weight, inplace_update=False, seed=None, return_weight=False, **kwargs):
            if getattr(self, 'layout_type', None) is not None:
                weight = self.weight.requantize_from_float(weight, scale="recalculate", stochastic_rounding=seed, inplace_ops=True).to(self.weight.dtype)
            else:
                weight = weight.to(self.weight.dtype)
            if return_weight:
                return weight

            assert inplace_update is False  # TODO: eventually remove the inplace_update stuff
            self.weight = torch.nn.Parameter(weight, requires_grad=False)

        def _apply(self, fn, recurse=True):  # This is to get torch.compile + moving weights to another device working
            return _quantized_apply(self, fn, recurse)

    class MoEExperts(torch.nn.Module, MixedPrecisionOp):
        """Container for E quantized expert weights, indexed via expert_weight(i).

        The bank lives on self.weight as a single 3D tensor — either a
        compute_dtype Parameter or a Parameter wrapping a QuantizedTensor
        with leading expert dim.

        State-dict layout matches mixed_precision_ops.Linear with a leading
        expert dim:
            {prefix}.weight          quant data (storage_t), leading dim = E
            {prefix}.weight_scale    block / per-tensor scale
            {prefix}.weight_scale_2  [E] or scalar           NVFP4 only
            {prefix}.bias            [E, out_features]       optional, compute_dtype
            {prefix}.comfy_quant     json -> {{"format": "...", "num_experts": E}}

        Without comfy_quant the weight loads as a plain compute_dtype 3D Parameter [E, out, in].
        """

        _disabled_formats = disabled

        def __init__(self, num_experts: int, in_features: int, out_features: int, bias: bool = True, device=None, dtype=None):
            super().__init__()
            self.num_experts = num_experts
            self.in_features = in_features
            self.out_features = out_features
            self._orig_shape = (num_experts, out_features, in_features)
            self.factory_kwargs = {"device": device, "dtype": MixedPrecisionOps._compute_dtype}
            if bias:
                self.bias = torch.nn.Parameter(torch.empty(num_experts, out_features, **self.factory_kwargs))
            else:
                self.register_parameter("bias", None)

            # Populated by _load_from_state_dict:
            self.weight = None
            self.quant_format = None
            self.layout_type = None
            self._full_precision_mm = MixedPrecisionOps._full_precision_mm
            self._full_precision_mm_config = False
            self._resident_bank = None

        def reset_parameters(self):
            return None

        def _apply(self, fn, recurse=True):
            return _quantized_apply(self, fn, recurse)

        def _load_from_state_dict(self, *args):
            _load_quantized_module(self, super()._load_from_state_dict, *args, load_extra_params=False)

        def expert_weight(self, i: int):
            """Expert i's weight (Tensor or per-expert QuantizedTensor view)."""
            if isinstance(self.weight, QuantizedTensor):
                return self._expert_qt_from(self.weight, i)
            return self.weight[i]

        `@contextlib.contextmanager`
        def bank_resident(self, input):
            """Cast the whole bank once; expert_linear inside reuses the cast.
            Not re-entrant — do not nest calls on the same instance.
            """
            with CastBiasWeightContext(self, input, offloadable=True) as self._resident_bank:
                try:
                    yield self
                finally:
                    self._resident_bank = None

        def expert_linear(self, input: torch.Tensor, i: int) -> torch.Tensor:
            """Linear against expert i's weight (with optional bias)."""
            resident = getattr(self, "_resident_bank", None)
            if resident is not None:
                weight, bias = resident
                return self._expert_linear_impl(input, weight, bias, i)
            with CastBiasWeightContext(self, input, offloadable=True) as (weight, bias):
                return self._expert_linear_impl(input, weight, bias, i)

        def _expert_linear_impl(self, input, weight, bias, i):
            if isinstance(weight, QuantizedTensor):
                qw = self._expert_qt_from(weight, i)
            else:
                qw = weight[i]
            b = cast_to_input(bias[i], input, copy=False) if bias is not None else None

            if isinstance(qw, QuantizedTensor):
                use_fast = (
                    not self._full_precision_mm
                    and qw.layout_cls.supports_fast_matmul()
                    and input.dim() == 2
                )
                if use_fast:
                    qin = QuantizedTensor.from_float(input, self.layout_type)
                    return torch.nn.functional.linear(qin, qw, b)
                out = input @ qw.dequantize().t()
                return out + b if b is not None else out
            return torch.nn.functional.linear(input, qw, b)

        def _expert_qt_from(self, weight: QuantizedTensor, i: int) -> QuantizedTensor:
            """Build a per-expert QuantizedTensor by indexing into a resident bank."""
            params = weight._params
            kwargs = {
                "scale": params.scale[i] if params.scale.dim() else params.scale,
                "orig_dtype": params.orig_dtype,
                "orig_shape": (self.out_features, self.in_features),
            }
            if hasattr(params, "block_scale"): # NVFP4
                kwargs["block_scale"] = params.block_scale[i]
            if hasattr(params, "quant_group_size"):
                kwargs["quant_group_size"] = params.quant_group_size
            if hasattr(params, "convrot_groupsize"):
                kwargs["convrot_groupsize"] = params.convrot_groupsize
            if hasattr(params, "linear_dtype"):
                kwargs["linear_dtype"] = params.linear_dtype
            return QuantizedTensor(weight._qdata[i], weight._layout_cls, type(params)(**kwargs))

        def state_dict(self, *args, destination=None, prefix="", **kwargs):
            sd = destination if destination is not None else {}
            return _quantized_weight_state_dict(self, sd, prefix, extra_quant_conf={"num_experts": self.num_experts})

    class Embedding(manual_cast.Embedding):
        def _load_from_state_dict(self, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs):
            weight_key = f"{prefix}weight"
            layer_conf = state_dict.pop(f"{prefix}comfy_quant", None)
            if layer_conf is not None:
                # An empty marker means "not quantized" rather than malformed JSON.
                layer_conf = json.loads(layer_conf.numpy().tobytes()) if layer_conf.numel() > 0 else None

            # Only fp8 and int8_tensorwise support per-row dequant via index select.
            # Block-scaled formats (NVFP4, MXFP8) can't do per-row lookup efficiently.
            quant_format = layer_conf.get("format") if layer_conf is not None else None
            manually_loaded_keys = []

            if quant_format in ("float8_e4m3fn", "float8_e5m2", "int8_tensorwise") and weight_key in state_dict:
                self.quant_format = quant_format
                qconfig = QUANT_ALGOS[quant_format]
                self.layout_type = qconfig["comfy_tensor_layout"]
                layout_cls = get_layout_class(self.layout_type)
                weight = state_dict.pop(weight_key)
                manually_loaded_keys.append(weight_key)

                scale_key = f"{prefix}weight_scale"
                scale = state_dict.pop(scale_key, None)
                if scale is not None:
                    scale = scale.float()
                    manually_loaded_keys.append(scale_key)

                extra = {}
                if quant_format == "int8_tensorwise" and layer_conf.get("convrot", False):
                    # rotated embedding table: record it so the forward un-rotates after lookup
                    extra["convrot"] = True
                    extra["convrot_groupsize"] = int(layer_conf.get("convrot_groupsize", 256))
                params = layout_cls.Params(
                    scale=scale if scale is not None else torch.ones((), dtype=torch.float32),
                    orig_dtype=MixedPrecisionOps._compute_dtype,
                    orig_shape=(self.num_embeddings, self.embedding_dim),
                    **extra,
                )
                self.weight = torch.nn.Parameter(
                    QuantizedTensor(weight.to(dtype=qconfig["storage_t"]), qconfig["comfy_tensor_layout"], params),
                    requires_grad=False)
            elif layer_conf is not None:
                # Unsupported format — restore the marker so it round-trips; fall through to default load.
                state_dict[f"{prefix}comfy_quant"] = torch.tensor(
                    list(json.dumps(layer_conf).encode('utf-8')), dtype=torch.uint8)

            super()._load_from_state_dict(state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs)
            for k in manually_loaded_keys:
                if k in missing_keys:
                    missing_keys.remove(k)

        def state_dict(self, *args, destination=None, prefix="", **kwargs):
            sd = destination if destination is not None else {}
            return _quantized_weight_state_dict(self, sd, prefix)

        def forward_comfy_cast_weights(self, input, out_dtype=None):
            weight = self.weight

            # Optimized path: lookup in fp8/int8, dequantize only the selected rows.
            if isinstance(weight, QuantizedTensor) and len(self.weight_function) == 0:
                with CastBiasWeightContext(self, device=input.device, dtype=weight.dtype, offloadable=True) as (qdata, _bias):
                    if isinstance(qdata, QuantizedTensor):
                        params = qdata._params
                        scale = params.scale
                        qdata = qdata._qdata
                    else:
                        params = weight._params
                        scale = None

                    # int8: per-row scale possible ConvRot, so let the layout do the gather
                    if self.quant_format == "int8_tensorwise":
                        x = get_layout_class(self.layout_type).dequantize_embedding(qdata, params, input)
                        return x if out_dtype is None else x.to(dtype=out_dtype)

                    x = torch.nn.functional.embedding(
                        input, qdata, self.padding_idx, self.max_norm,
                        self.norm_type, self.scale_grad_by_freq, self.sparse)
                target_dtype = out_dtype if out_dtype is not None else weight._params.orig_dtype
                x = x.to(dtype=target_dtype)
                if scale is not None:
                    x = x * scale.to(dtype=target_dtype)
                return x

            # Fallback for non-quantized or weight_function (LoRA) case
            return super().forward_comfy_cast_weights(input, out_dtype=out_dtype)

return MixedPrecisionOps

Note: [CWE-710] Improper Adherence to Coding Standards (mutable default argument).

(no-empty-list-as-parameter)

🔇 Additional comments (4)

comfy/ops.py (2)

1191-1192: LGTM!

---

1629-1630: LGTM!

tests-unit/comfy_quant/test_mixed_precision.py (2)

233-250: LGTM!

---

252-264: LGTM!

---

📝 Walkthrough

Walkthrough

The quantized module loader and MixedPrecisionOps.Embedding loader now treat an empty comfy_quant tensor as an unquantized configuration. They avoid passing empty bytes to json.loads. Two tests verify plain bfloat16 weights load without JSONDecodeError and are not wrapped in QuantizedTensor.

Priority: ➖ Normal

Severity of issue fixed: Medium

Merge Risk: ⚪ Minimal · up to f7c3f

Empty quantization markers now load as unquantized weights, preventing the Qwen text-encoder loading failure. The focused regression coverage supports merging this change.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 71.43% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 7 functions across 2 files.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly and concisely describes the main fix: preventing JSONDecodeError when a layer's comfy_quant marker is empty.
Description check	✅ Passed	The description directly explains the reported failure, root cause, implementation, affected code paths, regression tests, and verification results.
Linked Issues check	✅ Passed	The changes address issue #16472. In comfy/ops.py, _load_quantized_module and MixedPrecisionOps.Embedding._load_from_state_dict treat zero-element comfy_quant markers as unquantized layers and…
Out of Scope Changes check	✅ Passed	The changes are limited to handling empty comfy_quant markers in the two affected loading paths and adding regression tests for those paths. These changes directly support issue #16472 and contain n…

• Fix all pre-merge checks with AI

---

_{Comment @coderabbitai help to get the list of available commands.}

[kijai]

I don't understand how to reproduce this, with what model file does this happen?

[chelsealong]

From the linked issue (#16472), the model is qwen3vl_8b_int8_convrot.safetensors (the Qwen 2.1 image-edit template's text encoder), loaded via the CLIP loader path in comfy/sd.py:2092 → load_text_encoder_state_dicts. Repro: load the Qwen 2.1 image-edit workflow template and run it — the token embedding layer's comfy_quant marker is a present-but-empty tensor, so json.loads(b"") raised JSONDecodeError in MixedPrecisionOps.Embedding._load_from_state_dict (comfy/ops.py:1627, pre-fix; same pattern also existed in _load_quantized_module at comfy/ops.py:1189). The reporter's full traceback and log are in issue #16472.

[kijai]

Well I can't reproduce it, and that's the only text encoder I've ever used with this model.

[chelsealong]

Checked the actual file the Qwen 2.1 template uses: Comfy-Org/Qwen-Image-2.1/text_encoders/qwen3vl_8b_int8_convrot.safetensors, uploaded 2026-09-15 and unchanged since (same revision both you and the reporter would be pulling). Reading just the safetensors header (no full download needed): model.embed_tokens.comfy_quant is not empty there — it's a valid 72-byte value, {"format": "int8_tensorwise", "convrot": true, "convrot_groupsize": 256}.

So the officially-hosted file doesn't hit this path, which lines up with @kijai not being able to reproduce it. The reporter's traceback is real, but the likely cause is a truncated/incomplete local download rather than the file itself shipping an empty marker — that comfy_quant tensor sits right at the tail of the ~9.35GB file (offsets 9350639144–9350639216 of 9350798360), exactly where an interrupted download would leave zero bytes while everything read up to that point still succeeds.

The code change itself is still a reasonable no-op safety net for a malformed/truncated marker (it only changes behavior when the tensor is actually empty), but it isn't reproducible against the current, complete model file.