LLaMA: Open and Efficient Foundation Language Models (Paper Explained)
#ai #meta #languagemodel
LLaMA is a series of large language models from 7B to 65B parameters, trained by Meta AI. They train for longer on more data and show that something like gpt-3 can be outperformed by significantly smaller models when trained like this. Meta also releases the trained models to the research community.
OUTLINE:
0:00 - Introduction & Paper Overview
4:30 - Rant on Open-Sourcing
8:05 - Training Data
12:40 - Training Hyperparameters
14:50 - Architecture Modifications
17:10 - Optimizer
19:40 - Efficient Implementation
26:15 - Main Results
38:00 - Some more completions
40:00 - Conclusion
Paper: https://arxiv.org/abs/2302.13971
Website: https://ai.facebook.com/blog/large-language-model-llama-meta-ai/
Abstract:
We introduce LLaMA, a collection of foundation language models ranging from 7B to 65B parameters. We train our models on trillions of tokens, and show that it is possible to train state-of-the-art models using publicly available datasets exclusively, without resorting to proprietary and inaccessible datasets. In particular, LLaMA-13B outperforms GPT-3 (175B) on most benchmarks, and LLaMA-65B is competitive with the best models, Chinchilla-70B and PaLM-540B. We release all our models to the research community.
Authors: Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, Guillaume Lample
Links:
Homepage: https://ykilcher.com
Merch: https://ykilcher.com/merch
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
Discord: https://ykilcher.com/discord
LinkedIn: https://www.linkedin.com/in/ykilcher
If you want to support me, the best thing to do is to share out the content :)
If you want to support me financially (completely optional and voluntary, but a lot of people have asked for this):
SubscribeStar: https://www.subscribestar.com/yannickilcher
Patreon: https://www.patreon.com/yannickilcher
Bitcoin (BTC): bc1q49lsw3q325tr58ygf8sudx2dqfguclvngvy2cq
Ethereum (ETH): 0x7ad3513E3B8f66799f507Aa7874b1B0eBC7F85e2
Litecoin (LTC): LQW2TRyKYetVC8WjFkhpPhtpbDM4Vw7r9m
Monero (XMR): 4ACL8AGrEo5hAir8A9CeVrW8pEauWvnp1WnSDZxW7tziCDLhZAGsgzhRQABDnFy8yuM9fWJDviJPHKRjV4FWt19CJZN9D4n
-
2:41:03
I_Came_With_Fire_Podcast
12 hours ago🔥🔥Suing CHINA, Hillary AIDED RUSSIA, and DEI REMOVED from Military🔥🔥
64.2K12 -
LIVE
SoniCentric
13 hours agoCozy Up With SNOWY Lakeside Cabin Jazz Vibes
165 watching -
1:36:16
PMG
1 day ago $0.69 earnedSPECIAL: JUSTICE FOR JEREMY - NOW!
26.8K4 -
1:01:01
TheTapeLibrary
16 hours ago $3.05 earnedThe Horrifying True Story of Summerwind Mansion
37.9K7 -
29:28
Afshin Rattansi's Going Underground
1 day agoMax Blumenthal on US’ Ukraine Aid Corruption, 'Psychotic' Israel Turning the West Bank into Gaza
42.3K4 -
57:12
Flyover Conservatives
1 day agoCovid, Control, & Corruption —Dr. Stella Immanuel’s Plan to BEAT the System! | FOC Show
29.8K -
57:13
Sarah Westall
8 hours agoUnited States in a Two Front War, Identify Military Psyops and Special Operations w/ Jeffrey Prather
68.2K19 -
2:14:05
Quite Frankly
12 hours ago"Open Phones: Philly & D.C. Crashes" 1/31/25
53.3K21 -
1:45:32
2 MIKES LIVE
11 hours ago2 MIKES LIVE #173 Open Mike Friday!
52K1 -
54:53
LFA TV
15 hours agoDemocrats’ Greatest Fear Is Accountability | TRUMPET DAILY 1.31.25 7pm
38.5K17