Premium Only Content

Compiler From Scratch: Phase 1 - Tokenizer Generator 021: Using tokenizer in the tokenizer generator
Streamed on 2024-12-06 (https://www.twitch.tv/thediscouragerofhesitancy)
Zero Dependencies Programming!
The tokenizer generator has to parse two files (so far): the project file and the tokenizer definition file. If we can generate a tokenizer, why not use that to parse those files? There's no reason not to so I started that task today. There is a bit of back and forth as you update your file parsing and the supported tokens and keeping them in sync enough to continue to build and run while making the switch.
Things were going reasonably smoothly until I ran into a bug. I thought it was a bug that I had been anticipating for quite a while, but it turned out to be something else. I anticipated it being an ordering problem, but instead it looks like our DFA isn't quite formed correctly. There are states that should be merged, like if two keywords start with the same letter. Also, there are states that should have reasonable fallbacks if they don't complete (or if they continue after the expected end) as in the case of a KEYWORD being a subset of an identifier. If the KEYWORD doesn't match exactly, it can and should still skip over into the identifier track. But that isn't happening right now.
I started looking into this bug, but didn't have time to finish it. Today's stream was a bit short and the code of interest is stuff I haven't looked at in quite a long time, so there was quite a bit of time spent trying to remember how it works and reason about what the fix should be. We'll finish debugging this next week.
-
LIVE
Spartan
6 hours agoPro Halo Player | Short scrim at 3 EST, Pro League Match vs CoL at 5 EST
43 watching -
1:27:58
Glenn Greenwald
4 hours agoWhy are Big Tech Execs Integrating into the US Military? CBS Pays Trump $15 Million: With FIRE's Nico Perrino; Diddy Verdict Locals Exclusive | SYSTEM UPDATE #480
54.8K12 -
LIVE
Badlands Media
16 hours agoAltered State S3 Ep. 35
1,745 watching -
28:30
World2Briggs
6 hours ago $0.05 earnedPortland’s 25-Year Collapse: The Timeline Nobody Wanted to Believe
77 -
6:44
WhaddoYouMeme
1 hour agoThe TRUTH About The God Diddy Worships
18 -
LIVE
Joker Effect
20 minutes agoRUMBLE AMBASSADOR INTERVIEW - VAPINGAMERS... RumbleGaming is growing... we find out how and why!
546 watching -
LIVE
Melonie Mac
2 hours agoGo Boom Live Ep 53!
236 watching -
55:21
RiftTV
4 hours agoIs The Catholic Church the TRUE CHURCH of Jesus Christ? | The Rift | Guest: Timothy Gordon
18.8K8 -
1:51:26
Andres Meneses
14 hours agoInside the Mind of Zuby: The Future of Bitcoin and Free Thought
6.02K1 -
35:50
The Kevin Trudeau Show Limitless
11 hours agoHow to Start a Small Business or Side Hustle in 2025 Mark Kohler's Secrets!
4.93K1