Fastformer: Additive Attention Can Be All You Need (Machine Learning Research Paper Explained)
#attention #transformer #fastformer
Transformers have become the dominant model class in the last few years for large data, but their quadratic complexity in terms of sequence length has plagued them until now. Fastformer claims to be the fastest and most performant linear attention variant, able to consume long contexts at once. This is achieved by a combination of additive attention and elementwise products. While initial results look promising, I have my reservations...
OUTLINE:
0:00 - Intro & Outline
2:15 - Fastformer description
5:20 - Baseline: Classic Attention
10:00 - Fastformer architecture
12:50 - Additive Attention
18:05 - Query-Key element-wise multiplication
21:35 - Redundant modules in Fastformer
25:00 - Problems with the architecture
27:30 - Is this even attention?
32:20 - Experimental Results
34:50 - Conclusion & Comments
Paper: https://arxiv.org/abs/2108.09084
Abstract:
Transformer is a powerful model for text understanding. However, it is inefficient due to its quadratic complexity to input sequence length. Although there are many methods on Transformer acceleration, they are still either inefficient on long sequences or not effective enough. In this paper, we propose Fastformer, which is an efficient Transformer model based on additive attention. In Fastformer, instead of modeling the pair-wise interactions between tokens, we first use additive attention mechanism to model global contexts, and then further transform each token representation based on its interaction with global context representations. In this way, Fastformer can achieve effective context modeling with linear complexity. Extensive experiments on five datasets show that Fastformer is much more efficient than many existing Transformer models and can meanwhile achieve comparable or even better long text modeling performance.
Authors: Chuhan Wu, Fangzhao Wu, Tao Qi, Yongfeng Huang
Links:
TabNine Code Completion (Referral): http://bit.ly/tabnine-yannick
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
Discord: https://discord.gg/4H8xxDF
BitChute: https://www.bitchute.com/channel/yann...
Minds: https://www.minds.com/ykilcher
Parler: https://parler.com/profile/YannicKilcher
LinkedIn: https://www.linkedin.com/in/yannic-ki...
BiliBili: https://space.bilibili.com/1824646584
If you want to support me, the best thing to do is to share out the content :)
If you want to support me financially (completely optional and voluntary, but a lot of people have asked for this):
SubscribeStar: https://www.subscribestar.com/yannick...
Patreon: https://www.patreon.com/yannickilcher
Bitcoin (BTC): bc1q49lsw3q325tr58ygf8sudx2dqfguclvngvy2cq
Ethereum (ETH): 0x7ad3513E3B8f66799f507Aa7874b1B0eBC7F85e2
Litecoin (LTC): LQW2TRyKYetVC8WjFkhpPhtpbDM4Vw7r9m
Monero (XMR): 4ACL8AGrEo5hAir8A9CeVrW8pEauWvnp1WnSDZxW7tziCDLhZAGsgzhRQABDnFy8yuM9fWJDviJPHKRjV4FWt19CJZN9D4n
-
44:19
ykilcher
2 years agoPonderNet: Learning to Ponder (Machine Learning Research Paper Explained)
93 -
1:15:01
Squaring The Circle w/ Randall Carlson
20 hours ago#004 Exploring the Mind ft. Sean Webb - Squaring the Circle: A Randall Carlson Podcast
1.19K9 -
9:38
Adam Does Movies
1 day agoFallout TV Series Review - Did Amazon Make Something Good?
3496 -
6:37
Tactical Advisor
23 hours agoDream Shotgun Build - Remington 870
624 -
21:16
MYLUNCHBREAK CHANNEL PAGE
14 hours agoThe Old World is in The Catacombs?
2.07K13 -
13:16
JoBlo Originals
19 hours agoWhat Happened to JEFFREY JONES?
5.87K16 -
6:37
scoutthedoggie
23 hours agoNovritsch SSX23 Pistol In Action - Scotland
15K5 -
11:47
ParisDemers
15 hours agoTrain Like DareDevil To Get Jacked FAST And Gain Enhanced Senses!
17.6K5 -
42:17
CarlCrusher
1 day agoSkinwalker Ranch Ancient Relic Artifacts - Part 6
19.7K7 -
14:45
GeekyNerdyTechy
21 hours agoZHIYUN MOLUS X60 RGB LED Combo Review & Tutorial
22.8K