Skip to content

(fix): handle none/dict rewards in _compute_zero_std_metrics#2157

Open
atoniolo76 wants to merge 2 commits into
THUDM:mainfrom
atoniolo76:alessio/filter-non-numeric-rewards-zero-std-metrics
Open

(fix): handle none/dict rewards in _compute_zero_std_metrics#2157
atoniolo76 wants to merge 2 commits into
THUDM:mainfrom
atoniolo76:alessio/filter-non-numeric-rewards-zero-std-metrics

Conversation

@atoniolo76

@atoniolo76 atoniolo76 commented Jul 1, 2026

Copy link
Copy Markdown

In OPD or multi-turn cases, the sample.reward can come back as None or a dict of teacher log probs.

The _compute_zero_std_metrics function returns 1 if all samples in a group have the same reward, 0 otherwise. If a sample is aborted (common in multi-turn training) and sample.reward is None, Slime fails with TypeError: type NoneType doesn't define __round__ method at the round() call in interesting_rewards. For OPD, the sample.reward may be a custom dict containing a payload of teacher log probs for post_process to handle. This function intercepts the reward and fails with the same TypeError: type dict doesn't define __round__ method. Note that the original failure mode only happens when a group of rewards are all None or contain a dict. Now, when any rewards are None or dict, _is_zero_std is not run for the whole group.

To reproduce:

from argparse import Namespace
from slime.ray.rollout import _compute_zero_std_metrics
from slime.utils.types import Sample
 
args = Namespace(advantage_estimator="grpo", reward_key=None)
samples = [Sample(group_index=0, reward=None), Sample(group_index=0, reward=None)]
_compute_zero_std_metrics(args, samples) 

Will fail with # TypeError: type NoneType doesn't define __round__ method

@atoniolo76 atoniolo76 changed the title Filter None or dict rewards from _compute_zero_std_metrics in rollout.py. Add unit tests. (fix): handle none/dict rewards in _compute_zero_std_metrics Jul 1, 2026
@nanjiangwill nanjiangwill requested a review from zhuzilin July 1, 2026 04:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant