Premium Only Content

Retentive Network: A Successor to Transformer for Large Language Models (Paper Explained)
#ai #retnet #transformers
Retention is an alternative to Attention in Transformers that can both be written in a parallel and in a recurrent fashion. This means the architecture achieves training parallelism while maintaining low-cost inference. Experiments in the paper look very promising.
OUTLINE:
0:00 - Intro
2:40 - The impossible triangle
6:55 - Parallel vs sequential
15:35 - Retention mechanism
21:00 - Chunkwise and multi-scale retention
24:10 - Comparison to other architectures
26:30 - Experimental evaluation
Paper: https://arxiv.org/abs/2307.08621
Abstract:
In this work, we propose Retentive Network (RetNet) as a foundation architecture for large language models, simultaneously achieving training parallelism, low-cost inference, and good performance. We theoretically derive the connection between recurrence and attention. Then we propose the retention mechanism for sequence modeling, which supports three computation paradigms, i.e., parallel, recurrent, and chunkwise recurrent. Specifically, the parallel representation allows for training parallelism. The recurrent representation enables low-cost O(1) inference, which improves decoding throughput, latency, and GPU memory without sacrificing performance. The chunkwise recurrent representation facilitates efficient long-sequence modeling with linear complexity, where each chunk is encoded parallelly while recurrently summarizing the chunks. Experimental results on language modeling show that RetNet achieves favorable scaling results, parallel training, low-cost deployment, and efficient inference. The intriguing properties make RetNet a strong successor to Transformer for large language models. Code will be available at this https URL.
Authors: Yutao Sun, Li Dong, Shaohan Huang, Shuming Ma, Yuqing Xia, Jilong Xue, Jianyong Wang, Furu Wei
Links:
Homepage: https://ykilcher.com
Merch: https://ykilcher.com/merch
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
Discord: https://ykilcher.com/discord
LinkedIn: https://www.linkedin.com/in/ykilcher
If you want to support me, the best thing to do is to share out the content :)
If you want to support me financially (completely optional and voluntary, but a lot of people have asked for this):
SubscribeStar: https://www.subscribestar.com/yannickilcher
Patreon: https://www.patreon.com/yannickilcher
Bitcoin (BTC): bc1q49lsw3q325tr58ygf8sudx2dqfguclvngvy2cq
Ethereum (ETH): 0x7ad3513E3B8f66799f507Aa7874b1B0eBC7F85e2
Litecoin (LTC): LQW2TRyKYetVC8WjFkhpPhtpbDM4Vw7r9m
Monero (XMR): 4ACL8AGrEo5hAir8A9CeVrW8pEauWvnp1WnSDZxW7tziCDLhZAGsgzhRQABDnFy8yuM9fWJDviJPHKRjV4FWt19CJZN9D4n
-
10:17
MattMorseTV
14 hours ago $9.39 earnedTrump's DOJ just DROPPED a NUKE.
56.5K63 -
2:09:32
Side Scrollers Podcast
17 hours agoStreamer DIES Live On Air + Your Food is Poison + Xbox Announces $900 Handheld | Side Scrollers Live
4.17K6 -
15:32
GritsGG
13 hours agoFull Auto ABR Sniper Support! Most Winning Quad Win Streaking!
1.07K1 -
7:42
The Pascal Show
12 hours agoBREAKING! Police Provide UPDATE In Emmanuel Haro's Case! Is Jake's Lawyer Lying To Us?!
1.31K -
2:29:46
FreshandFit
5 hours agoAfter Hours w/ Girls
96.5K61 -
5:28
Zach Humphries
11 hours ago $0.33 earnedNEAR PROTCOL AND STELLAR TEAM UP!
5.03K1 -
1:09:57
Brandon Gentile
1 day ago10,000 Hour BITCOIN Expert Reveals Why $13.5M Is Just The Start
11.7K -
2:03:55
Badlands Media
6 hours agoDevolution Power Hour Ep. 382: DOJ Coverups, Clapper’s Team Sport & Trump’s Countermoves
121K18 -
2:06:30
Inverted World Live
9 hours agoDon't Approach the Zombie Rabbits | Ep. 95
46.7K23 -
3:26:45
Drew Hernandez
5 hours agoISRAEL PLANNING POSSIBLE DRAFT IN USA & TRUMP'S VIEW ON ETERNAL LIFE ANALYZED PT 2
36.7K25