LLaMA: Open and Efficient Foundation Language Models (Paper Explained)
#ai #meta #languagemodel
LLaMA is a series of large language models from 7B to 65B parameters, trained by Meta AI. They train for longer on more data and show that something like gpt-3 can be outperformed by significantly smaller models when trained like this. Meta also releases the trained models to the research community.
OUTLINE:
0:00 - Introduction & Paper Overview
4:30 - Rant on Open-Sourcing
8:05 - Training Data
12:40 - Training Hyperparameters
14:50 - Architecture Modifications
17:10 - Optimizer
19:40 - Efficient Implementation
26:15 - Main Results
38:00 - Some more completions
40:00 - Conclusion
Paper: https://arxiv.org/abs/2302.13971
Website: https://ai.facebook.com/blog/large-language-model-llama-meta-ai/
Abstract:
We introduce LLaMA, a collection of foundation language models ranging from 7B to 65B parameters. We train our models on trillions of tokens, and show that it is possible to train state-of-the-art models using publicly available datasets exclusively, without resorting to proprietary and inaccessible datasets. In particular, LLaMA-13B outperforms GPT-3 (175B) on most benchmarks, and LLaMA-65B is competitive with the best models, Chinchilla-70B and PaLM-540B. We release all our models to the research community.
Authors: Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, Guillaume Lample
Links:
Homepage: https://ykilcher.com
Merch: https://ykilcher.com/merch
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
Discord: https://ykilcher.com/discord
LinkedIn: https://www.linkedin.com/in/ykilcher
If you want to support me, the best thing to do is to share out the content :)
If you want to support me financially (completely optional and voluntary, but a lot of people have asked for this):
SubscribeStar: https://www.subscribestar.com/yannickilcher
Patreon: https://www.patreon.com/yannickilcher
Bitcoin (BTC): bc1q49lsw3q325tr58ygf8sudx2dqfguclvngvy2cq
Ethereum (ETH): 0x7ad3513E3B8f66799f507Aa7874b1B0eBC7F85e2
Litecoin (LTC): LQW2TRyKYetVC8WjFkhpPhtpbDM4Vw7r9m
Monero (XMR): 4ACL8AGrEo5hAir8A9CeVrW8pEauWvnp1WnSDZxW7tziCDLhZAGsgzhRQABDnFy8yuM9fWJDviJPHKRjV4FWt19CJZN9D4n
-
58:42
The StoneZONE with Roger Stone
1 hour agoLBJ + CIA + Mob + Texas Oil = JFK Murder | The StoneZONE w/ Roger Stone
14.1K2 -
58:00
Donald Trump Jr.
9 hours agoBreaking News on Deadly Plane Crash, Plus Hearing on the Hill, Live with Rep Cory Mills & Sen Marsha Blackburn | TRIGGERED Ep.212
148K108 -
52:03
Kimberly Guilfoyle
7 hours agoLatest Updates on Deadly Air Collision, Plus Major Hearings on Capitol Hill,Live with Marc Beckman & Steve Friend | Ep.192
82.3K28 -
Josh Pate's College Football Show
6 hours ago $0.85 earnedMichigan vs NCAA | ESPN’s ACC Deal | Season Grades: UGA & Miami | Notre Dame Losses
27.3K2 -
1:26:50
Redacted News
6 hours agoWhat happened? Trump DESTROYS the Pete Buttigieg run FAA for tragic airline crash | Redacted News
211K147 -
LIVE
VOPUSARADIO
1 day agoPOLITI-SHOCK! Hero Angela Stanton King & Wrongfully Imprisoned J6 Prisoner Josh Pruitt!
58 watching -
43:37
Candace Show Podcast
6 hours agoThe Taylor Swift Plot Thickens | Candace Ep 142
131K109 -
1:37:33
Common Threads
4 hours agoLIVE DEBATE: Trump's Wild Handling of Plane Tragedy
21.4K5 -
54:45
LFA TV
10 hours agoConfirmation Chaos | TRUMPET DAILY 1.30.25 7pm
34.2K2 -
1:57:53
Revenge of the Cis
7 hours agoEpisode 1437: Kash Me Outside
67.3K23