Premium Only Content

ActInf GuestStream 115.1 ~ Energy-Based Transformers and the Future of Scaling
"Energy-Based Transformers are Scalable Learners and Thinkers"
Alexi Gladstone, Ganesh Nanduru, Md Mofijul Islam, Peixuan Han, Hyeonjeong Ha, Aman Chadha, Yilun Du, Heng Ji, Jundong Li, Tariq Iqbal
https://arxiv.org/abs/2507.02092
Inference-time computation techniques, analogous to human System 2 Thinking, have recently become popular for improving model performances. However, most existing approaches suffer from several limitations: they are modality-specific (e.g., working only in text), problem-specific (e.g., verifiable domains like math and coding), or require additional supervision/training on top of unsupervised pretraining (e.g., verifiers or verifiable rewards). In this paper, we ask the question "Is it possible to generalize these System 2 Thinking approaches, and develop models that learn to think solely from unsupervised learning?" Interestingly, we find the answer is yes, by learning to explicitly verify the compatibility between inputs and candidate-predictions, and then re-framing prediction problems as optimization with respect to this verifier. Specifically, we train Energy-Based Transformers (EBTs) -- a new class of Energy-Based Models (EBMs) -- to assign an energy value to every input and candidate-prediction pair, enabling predictions through gradient descent-based energy minimization until convergence. Across both discrete (text) and continuous (visual) modalities, we find EBTs scale faster than the dominant Transformer++ approach during training, achieving an up to 35% higher scaling rate with respect to data, batch size, parameters, FLOPs, and depth. During inference, EBTs improve performance with System 2 Thinking by 29% more than the Transformer++ on language tasks, and EBTs outperform Diffusion Transformers on image denoising while using fewer forward passes. Further, we find that EBTs achieve better results than existing models on most downstream tasks given the same or worse pretraining performance, suggesting that EBTs generalize better than existing approaches. Consequently, EBTs are a promising new paradigm for scaling both the learning and thinking capabilities of models.
Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Computer Vision and Pattern Recognition (cs.CV)
Cite as: arXiv:2507.02092 [cs.LG]
(or arXiv:2507.02092v1 [cs.LG] for this version)
https://doi.org/10.48550/arXiv.2507.02092
Active Inference Institute information:
Website: https://www.activeinference.institute/
Activities: https://activities.activeinference.institute/
Discord: https://discord.activeinference.institute/
Donate: http://donate.activeinference.institute/
YouTube: https://www.youtube.com/c/ActiveInference/
X: https://x.com/InferenceActive
Active Inference Livestreams: https://video.activeinference.institute/
-
1:43:29
Active Inference Institute
9 days agoActInf GuestStream 121.1 ~ Conscious active inference (Wiest and Puniani)
111 -
LIVE
MattMorseTV
5 hours ago $2.02 earned🔴Antifa RIOT vs. Federal OFFICERS.🔴
7,362 watching -
1:05:28
Man in America
8 hours agoLIVE: Digital ID & the DEATH of Freedom—An URGENT Warning
18.8K14 -
40:24
The Connect: With Johnny Mitchell
1 day ago $0.03 earnedInside The WORST Drug-Infested Slums Of Medellin, Colombia
4.29K3 -
Tundra Tactical
4 hours ago $0.05 earned🛑LIVE NOW!! FBI Gets Caught LYING About Good Guys With Guns For 10 YEARS!!!!
13.1K -
LIVE
BlackDiamondGunsandGear
2 days agoAFTER HOURS ARMORY / Antifa / Lies/ Prison time
80 watching -
LIVE
DLDAfterDark
4 hours ago $0.02 earnedThe After Hours Armory! Tonight is The Chat's Chat! God, Guns, and Gear!
199 watching -
3:32:18
Mally_Mouse
7 hours ago🌶️ 🥵Spicy BITE Saturday!! 🥵🌶️- Let's Play: Phasmophobia
28.1K2 -
1:13:19
iCkEdMeL
3 hours ago $0.31 earnedChaos Explodes in Chicago & Portland | Feds Clash with Protesters!
18.8K4 -
21:54
Exploring With Nug
1 day ago $6.64 earnedScuba Diving Missing Person Search Leads to Discovery of Classic Cars!
44.1K8