Premium Only Content

Decision Transformer: Reinforcement Learning via Sequence Modeling (Research Paper Explained)
#decisiontransformer #reinforcementlearning #transformer
Proper credit assignment over long timespans is a fundamental problem in reinforcement learning. Even methods designed to combat this problem, such as TD-learning, quickly reach their limits when rewards are sparse or noisy. This paper reframes offline reinforcement learning as a pure sequence modeling problem, with the actions being sampled conditioned on the given history and desired future rewards. This allows the authors to use recent advances in sequence modeling using Transformers and achieve competitive results in Offline RL benchmarks.
OUTLINE:
0:00 - Intro & Overview
4:15 - Offline Reinforcement Learning
10:10 - Transformers in RL
14:25 - Value Functions and Temporal Difference Learning
20:25 - Sequence Modeling and Reward-to-go
27:20 - Why this is ideal for offline RL
31:30 - The context length problem
34:35 - Toy example: Shortest path from random walks
41:00 - Discount factors
45:50 - Experimental Results
49:25 - Do you need to know the best possible reward?
52:15 - Key-to-door toy experiment
56:00 - Comments & Conclusion
Paper: https://arxiv.org/abs/2106.01345
Website: https://sites.google.com/berkeley.edu...
Code: https://github.com/kzl/decision-trans...
Abstract:
We present a framework that abstracts Reinforcement Learning (RL) as a sequence modeling problem. This allows us to draw upon the simplicity and scalability of the Transformer architecture, and associated advances in language modeling such as GPT-x and BERT. In particular, we present Decision Transformer, an architecture that casts the problem of RL as conditional sequence modeling. Unlike prior approaches to RL that fit value functions or compute policy gradients, Decision Transformer simply outputs the optimal actions by leveraging a causally masked Transformer. By conditioning an autoregressive model on the desired return (reward), past states, and actions, our Decision Transformer model can generate future actions that achieve the desired return. Despite its simplicity, Decision Transformer matches or exceeds the performance of state-of-the-art model-free offline RL baselines on Atari, OpenAI Gym, and Key-to-Door tasks.
Authors: Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Michael Laskin, Pieter Abbeel, Aravind Srinivas, Igor Mordatch
Links:
TabNine Code Completion (Referral): http://bit.ly/tabnine-yannick
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
Discord: https://discord.gg/4H8xxDF
BitChute: https://www.bitchute.com/channel/yann...
Minds: https://www.minds.com/ykilcher
Parler: https://parler.com/profile/YannicKilcher
LinkedIn: https://www.linkedin.com/in/yannic-ki...
BiliBili: https://space.bilibili.com/1824646584
If you want to support me, the best thing to do is to share out the content :)
If you want to support me financially (completely optional and voluntary, but a lot of people have asked for this):
SubscribeStar: https://www.subscribestar.com/yannick...
Patreon: https://www.patreon.com/yannickilcher
Bitcoin (BTC): bc1q49lsw3q325tr58ygf8sudx2dqfguclvngvy2cq
Ethereum (ETH): 0x7ad3513E3B8f66799f507Aa7874b1B0eBC7F85e2
Litecoin (LTC): LQW2TRyKYetVC8WjFkhpPhtpbDM4Vw7r9m
Monero (XMR): 4ACL8AGrEo5hAir8A9CeVrW8pEauWvnp1WnSDZxW7tziCDLhZAGsgzhRQABDnFy8yuM9fWJDviJPHKRjV4FWt19CJZN9D4n
-
2:01
WTMJMilwaukee
4 years agoMixed reaction to MPS' decision to return to in-person learning
2 -
2:07
KMTV
4 years agoOPS Parents React to Remote Learning Decision
51 -
1:55
KMGH
4 years agoJeffCo schools to make decision on in-person learning by end of week
3 -
6:37
KGUN
4 years agoLearning Loss
15 -
7:02
China Uncensored
15 hours agoWell, I Guess Now We Know...
16.5K21 -
46:10
Members Club
19 hours ago $0.94 earnedThe WNBA Has Demands, TSA Loosens Up, and NYC Has a Whale Whisperer - MC04
14K4 -
1:35:15
Man in America
15 hours ago🚨 ALERT: Hospitals in the U.S. Are KILLING Patients… for Their Organs!
88.9K39 -
8:00
DropItLikeItsScott
15 hours ago $1.06 earnedIs This The BEST SIG P365? SIG P365 FUSE
11.8K -
11:55
The Shannon Joy Show
14 hours ago🔥Babies Aborted for Cash.🔥
10.7K -
6:24
Zach Humphries
15 hours agoBreaking Crypto News on Crypto Regulation!
10.7K4