PonderNet: Learning to Ponder (Machine Learning Research Paper Explained)
#pondernet #deepmind #machinelearning
Humans don't spend the same amount of mental effort on all problems equally. Instead, we respond quickly to easy tasks, and we take our time to deliberate hard tasks. DeepMind's PonderNet attempts to achieve the same by dynamically deciding how many computation steps to allocate to any single input sample. This is done via a recurrent architecture and a trainable function that computes a halting probability. The resulting model performs well in dynamic computation tasks and is surprisingly robust to different hyperparameter settings.
OUTLINE:
0:00 - Intro & Overview
2:30 - Problem Statement
8:00 - Probabilistic formulation of dynamic halting
14:40 - Training via unrolling
22:30 - Loss function and regularization of the halting distribution
27:35 - Experimental Results
37:10 - Sensitivity to hyperparameter choice
41:15 - Discussion, Conclusion, Broader Impact
Paper: https://arxiv.org/abs/2107.05407
Abstract:
In standard neural networks the amount of computation used grows with the size of the inputs, but not with the complexity of the problem being learnt. To overcome this limitation we introduce PonderNet, a new algorithm that learns to adapt the amount of computation based on the complexity of the problem at hand. PonderNet learns end-to-end the number of computational steps to achieve an effective compromise between training prediction accuracy, computational cost and generalization. On a complex synthetic problem, PonderNet dramatically improves performance over previous adaptive computation methods and additionally succeeds at extrapolation tests where traditional neural networks fail. Also, our method matched the current state of the art results on a real world question and answering dataset, but using less compute. Finally, PonderNet reached state of the art results on a complex task designed to test the reasoning capabilities of neural networks.1
Authors: Andrea Banino, Jan Balaguer, Charles Blundell
Links:
TabNine Code Completion (Referral): http://bit.ly/tabnine-yannick
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
Discord: https://discord.gg/4H8xxDF
BitChute: https://www.bitchute.com/channel/yann...
Minds: https://www.minds.com/ykilcher
Parler: https://parler.com/profile/YannicKilcher
LinkedIn: https://www.linkedin.com/in/yannic-ki...
BiliBili: https://space.bilibili.com/1824646584
If you want to support me, the best thing to do is to share out the content :)
If you want to support me financially (completely optional and voluntary, but a lot of people have asked for this):
SubscribeStar: https://www.subscribestar.com/yannick...
Patreon: https://www.patreon.com/yannickilcher
Bitcoin (BTC): bc1q49lsw3q325tr58ygf8sudx2dqfguclvngvy2cq
Ethereum (ETH): 0x7ad3513E3B8f66799f507Aa7874b1B0eBC7F85e2
Litecoin (LTC): LQW2TRyKYetVC8WjFkhpPhtpbDM4Vw7r9m
Monero (XMR): 4ACL8AGrEo5hAir8A9CeVrW8pEauWvnp1WnSDZxW7tziCDLhZAGsgzhRQABDnFy8yuM9fWJDviJPHKRjV4FWt19CJZN9D4n
-
1:24
laurastephens99
3 years agoLove learning
19 -
1:17:27
Dexerto League of Legends
21 hours agoLEC Final Review! The REAL Reason G2 Is Better! Does Fnatic Need a NEW Roster?!
4.35K -
1:10
Maicy Renee
1 day agoGet to Know Me!
6.32K19 -
8:15
Dr. Nick Zyrowski
2 days ago5 Foods Destroying Your Liver - Your Liver Is Dying
8.5K6 -
12:48
Fascinating Horror
1 day agoNew Zealand's Worst Fire: Ballantynes Department Store | Fascinating Horror
10.3K5 -
7:59
JoBlo Originals
18 hours agoIs Deadpool 2 BETTER Than the Original?!
10.9K4 -
29:41
Melonie Mac
20 hours agoNetflix is Erasing Lara Croft's Legacy
10.2K19 -
8:13
Hannah Barron
20 hours agoPRECIATE U-HAUL...Disaster Situation Turns Into Surprise Fish Trip
12.7K9 -
11:54
GameLeap RAID Shadow Legends
1 day agoULTIMATE GEM GUIDE (2024) BEST Way to Spend Gems in Raid: Shadow Legends
11.3K3 -
2:07:53
Price of Reason
16 hours agoDisney & Apple CAUGHT Discriminating! Deadpool & Wolverine Makes Bank! Ubisoft GASLIGHTS Gamers!
29.9K2