OpenAI published research on 5 November 2018 describing a model-based reinforcement learning approach that separates planning from learning [source]. The method, termed "plan online, learn offline," uses a learned dynamics model to generate synthetic experience during deployment, then updates the policy offline using that data.

The research addresses a known limitation in model-based control: compounding prediction errors when models are rolled out over long horizons. OpenAI's approach mitigates this by performing short-horizon planning online with model-predictive control, then using the resulting trajectories to train a policy network offline. Testing was conducted on continuous control tasks in MuJoCo simulation environments.

Results showed the method achieved sample efficiency comparable to model-free algorithms while requiring fewer real environment interactions. On tasks including Swimmer, Hopper, and HalfCheetah, the approach reached target performance with 10–100 times fewer environment samples than baseline model-free methods. The learned policies generalised to unseen states encountered during online deployment.

The work represents an incremental advance in sample-efficient reinforcement learning rather than a production system deployment. OpenAI noted the method's performance depends on model accuracy and the quality of offline data aggregation. No claims were made about deployment in user-facing products.

The research was published as part of OpenAI's ongoing investigation into reducing the sample complexity of deep reinforcement learning. The findings contribute to understanding how learned world models can accelerate policy learning in simulated control tasks, though practical application to real-world robotics or production AI systems was not demonstrated in the published work.