Premium Only Content

Reinforced Self-Training (ReST) for Language Modeling (Paper Explained)
#ai #rlhf #llm
ReST uses a bootsrap-like method to produce its own extended dataset and trains on ever higher-quality subsets of it to improve its own reward. The method allows for re-using the same generated data multiple times and thus has an efficiency advantage with respect to Online RL techniques like PPO.
Paper: https://arxiv.org/abs/2308.08998
Abstract:
Reinforcement learning from human feedback (RLHF) can improve the quality of large language model's (LLM) outputs by aligning them with human preferences. We propose a simple algorithm for aligning LLMs with human preferences inspired by growing batch reinforcement learning (RL), which we call Reinforced Self-Training (ReST). Given an initial LLM policy, ReST produces a dataset by generating samples from the policy, which are then used to improve the LLM policy using offline RL algorithms. ReST is more efficient than typical online RLHF methods because the training dataset is produced offline, which allows data reuse. While ReST is a general approach applicable to all generative learning settings, we focus on its application to machine translation. Our results show that ReST can substantially improve translation quality, as measured by automated metrics and human evaluation on machine translation benchmarks in a compute and sample-efficient manner.
Authors: Caglar Gulcehre, Tom Le Paine, Srivatsan Srinivasan, Ksenia Konyushkova, Lotte Weerts, Abhishek Sharma, Aditya Siddhant, Alex Ahern, Miaosen Wang, Chenjie Gu, Wolfgang Macherey, Arnaud Doucet, Orhan Firat, Nando de Freitas
Links:
Homepage: https://ykilcher.com
Merch: https://ykilcher.com/merch
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
Discord: https://ykilcher.com/discord
LinkedIn: https://www.linkedin.com/in/ykilcher
If you want to support me, the best thing to do is to share out the content :)
If you want to support me financially (completely optional and voluntary, but a lot of people have asked for this):
SubscribeStar: https://www.subscribestar.com/yannickilcher
Patreon: https://www.patreon.com/yannickilcher
Bitcoin (BTC): bc1q49lsw3q325tr58ygf8sudx2dqfguclvngvy2cq
Ethereum (ETH): 0x7ad3513E3B8f66799f507Aa7874b1B0eBC7F85e2
Litecoin (LTC): LQW2TRyKYetVC8WjFkhpPhtpbDM4Vw7r9m
Monero (XMR): 4ACL8AGrEo5hAir8A9CeVrW8pEauWvnp1WnSDZxW7tziCDLhZAGsgzhRQABDnFy8yuM9fWJDviJPHKRjV4FWt19CJZN9D4n
-
24:24
Jasmin Laine
6 hours ago“Their BIGGEST Mistake Yet”—U.S. Officials TORCH Carney in Brutal Report
29K26 -
LIVE
Rallied
4 hours ago $0.12 earnedWARZONE SOLO CHALLENGES ALL DAY
453 watching -
2:00:33
Joker Effect
3 hours agoSoo @MrBeast charity, is it legit? XenaTheWitch got attested? Lets talk about it! HOW TO GO VIRAL.
18.8K1 -
27:01
TheRoyaltyAutoService
10 hours ago $0.06 earnedAnother Shop Replaced The Timing Belt & Fuel Injectors On This Honda Pilot… Now It Has No Power!
31.9K3 -
10:14:25
GritsGG
12 hours agoRanked Crim 2! Most Wins 3390+ 🧠
62.7K -
3:12:37
IsaiahLCarter
1 day ago $1.69 earnedJonah O. Wheeler: In Defense of Reality || APOSTATE RADIO #026
31.4K1 -
LIVE
Cewpins
6 hours agoSmoke Sesh🔥Gaming???💨420🍃!MJ !giveaway
166 watching -
LIVE
Phyxicx
5 hours agoShin Megami Tensei - Part 5 - 8/13/2025 | Chaos Route | Expert Difficulty
107 watching -
1:59:32
vivafrei
15 hours agoEp. 277: Russia Peace Talks! Trump D.C. Takeover Leads to Lawsuit! Heat Wave Lawsuit? AND MORE!
84.5K114 -
2:17:57
Nerdrotic
7 hours ago $0.22 earnedTransient Lunar Phenomenon: Deeper into the Mysteries of the Moon | Forbidden Frontier #112
27.1K3