OpenAI has published details of a new safety training method called Rule-Based Rewards (RBR), which uses explicit rules to guide model behavior during reinforcement learning. The approach was developed after the company observed that traditional reinforcement learning from human feedback (RLHF) sometimes produced inconsistent safety outcomes [source].

Under RBR, human reviewers write explicit rules describing desired and undesired model behaviors. These rules are then converted into reward signals during training. OpenAI reports that the method improved refusal accuracy on internal safety benchmarks while reducing over-refusal on benign prompts. The company tested RBR on models in the GPT-4 family.

The disclosure follows a pattern of providers acknowledging that standard RLHF can lead to unpredictable safety behavior. OpenAI states that rule-based approaches offer more transparency than purely learned reward models, allowing engineers to inspect and modify specific behavioral constraints. The company notes that RBR does not eliminate all safety failures but provides a more controllable training signal.

OpenAI has not specified whether RBR is currently deployed in production models or remains experimental. The blog post describes the method as part of ongoing research into alignment techniques. The company indicates that rule-based methods may complement rather than replace existing RLHF pipelines.

The announcement comes as providers face increasing scrutiny over inconsistent content moderation and safety filtering. Multiple reports have documented cases where models refuse benign requests or permit harmful ones. OpenAI's disclosure suggests the company is exploring alternatives to address these reliability gaps, though no deployment timeline or performance metrics for production systems were provided.