Reinforced Self-Training (ReST) for Language Modeling (Paper Explained)
#ai #rlhf #llm
ReST uses a bootsrap-like method to produce its own extended dataset and trains on ever higher-quality subsets of it to improve its own reward. The method allows for re-using the same generated data multiple times and thus has an efficiency advantage with respect to Online RL techniques like PPO.
Paper: https://arxiv.org/abs/2308.08998
Abstract:
Reinforcement learning from human feedback (RLHF) can improve the quality of large language model's (LLM) outputs by aligning them with human preferences. We propose a simple algorithm for aligning LLMs with human preferences inspired by growing batch reinforcement learning (RL), which we call Reinforced Self-Training (ReST). Given an initial LLM policy, ReST produces a dataset by generating samples from the policy, which are then used to improve the LLM policy using offline RL algorithms. ReST is more efficient than typical online RLHF methods because the training dataset is produced offline, which allows data reuse. While ReST is a general approach applicable to all generative learning settings, we focus on its application to machine translation. Our results show that ReST can substantially improve translation quality, as measured by automated metrics and human evaluation on machine translation benchmarks in a compute and sample-efficient manner.
Authors: Caglar Gulcehre, Tom Le Paine, Srivatsan Srinivasan, Ksenia Konyushkova, Lotte Weerts, Abhishek Sharma, Aditya Siddhant, Alex Ahern, Miaosen Wang, Chenjie Gu, Wolfgang Macherey, Arnaud Doucet, Orhan Firat, Nando de Freitas
Links:
Homepage: https://ykilcher.com
Merch: https://ykilcher.com/merch
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
Discord: https://ykilcher.com/discord
LinkedIn: https://www.linkedin.com/in/ykilcher
If you want to support me, the best thing to do is to share out the content :)
If you want to support me financially (completely optional and voluntary, but a lot of people have asked for this):
SubscribeStar: https://www.subscribestar.com/yannickilcher
Patreon: https://www.patreon.com/yannickilcher
Bitcoin (BTC): bc1q49lsw3q325tr58ygf8sudx2dqfguclvngvy2cq
Ethereum (ETH): 0x7ad3513E3B8f66799f507Aa7874b1B0eBC7F85e2
Litecoin (LTC): LQW2TRyKYetVC8WjFkhpPhtpbDM4Vw7r9m
Monero (XMR): 4ACL8AGrEo5hAir8A9CeVrW8pEauWvnp1WnSDZxW7tziCDLhZAGsgzhRQABDnFy8yuM9fWJDviJPHKRjV4FWt19CJZN9D4n
-
2:48:55
Price of Reason
13 hours agoDESPERATE Hollywood Blames Customers For Losses, AGAIN! Can Ubisoft Survive Amidst ANOTHER Decline?
29.6K7 -
1:33:11
Man in America
15 hours ago🔴 LIVE: Asheville HORROR—DISTURBING Reports Reveal SICKENING Agenda
115K284 -
3:56:18
Alex Zedra
9 hours agoLIVE! Scary Game Night AGAIN
69.3K5 -
1:25:47
Glenn Greenwald
17 hours agoProf. John Mearsheimer On Israel, Iran, Lebanon, and Widening War in the Middle East | SYSTEM UPDATE #346
122K143 -
2:13:40
Roseanne Barr
12 hours ago $66.13 earnedThe Deep State Hit List with Ivan Raiklin | The Roseanne Barr Podcast #68
146K301 -
9:35
Colion Noir
12 hours agoViral Tiktok Lady Claims To Have The Solution To Gun Violence, You Won't Believe This
74.5K121 -
1:59:08
Danny Haiphong
14 hours ago $6.04 earnedBen Norton: Hezbollah REPELS Israel's Ground War, IDF STRIKES Russia Air Base–Is WW3 Next?
67.7K64 -
58:47
Flyover Conservatives
1 day agoPort Shutdowns Paralyzing Global Trade: What You Need to Know - Clay Clark | FOC Show
75.9K20 -
55:19
LFA TV
1 day agoKamala’s Triple Whammy: Flooding, Port Strikes and War | Trumpet Daily 10.3.24 9PM EST
61.5K17 -
46:23
Donald Trump Jr.
18 hours agoJD Shows why He was the Perfect VP Pick, my Full Reaction | TRIGGERED Ep.179
154K281