Premium Only Content

Compiler From Scratch: Phase 1 - Tokenizer Generator 021: Using tokenizer in the tokenizer generator
Streamed on 2024-12-06 (https://www.twitch.tv/thediscouragerofhesitancy)
Zero Dependencies Programming!
The tokenizer generator has to parse two files (so far): the project file and the tokenizer definition file. If we can generate a tokenizer, why not use that to parse those files? There's no reason not to so I started that task today. There is a bit of back and forth as you update your file parsing and the supported tokens and keeping them in sync enough to continue to build and run while making the switch.
Things were going reasonably smoothly until I ran into a bug. I thought it was a bug that I had been anticipating for quite a while, but it turned out to be something else. I anticipated it being an ordering problem, but instead it looks like our DFA isn't quite formed correctly. There are states that should be merged, like if two keywords start with the same letter. Also, there are states that should have reasonable fallbacks if they don't complete (or if they continue after the expected end) as in the case of a KEYWORD being a subset of an identifier. If the KEYWORD doesn't match exactly, it can and should still skip over into the identifier track. But that isn't happening right now.
I started looking into this bug, but didn't have time to finish it. Today's stream was a bit short and the code of interest is stuff I haven't looked at in quite a long time, so there was quite a bit of time spent trying to remember how it works and reason about what the fix should be. We'll finish debugging this next week.
-
LIVE
ZWOGs
10 hours ago🔴LIVE IN 1440p! - EFT w/ crgoodw1n, Kingdom Come Deliverance, Splitgate 2, & More! - Come Hang Out!
626 watching -
30:09
Afshin Rattansi's Going Underground
1 day agoCurtis Yarvin: ‘Trump 47 is 10x More Powerful than Trump 45’ & the TOXIC US-Israel Relationship
10.3K25 -
LIVE
Drew Hernandez
7 hours agoTRUMP HINTS DEMS COULD HAVE PLANTED FAKE EVIDENCE & MANIPULATED EPSTEIN FILES
992 watching -
LIVE
Eternal_Spartan
10 hours agoThe Legend of Zelda: Majoras Mask Ep. 5 | USMC Vet | Come Join the Best Chat on Rumble!!!!
84 watching -
3:44:24
Barry Cunningham
5 hours agoBREAKING NEWS: LIVE COVERAGE OF NEW YORK POLICE OFFICERS UNDER SIEGE!
56.8K56 -
DVR
The Pascal Show
3 hours ago $0.41 earnedBREAKING! Active Shooter In Midtown Manhattan NYC Multiple People Shot!
3.36K -
10:25
MattMorseTV
8 hours ago $7.47 earnedVance just DROPPED a NUKE.
15.2K33 -
LIVE
Jokeuhl Gaming and Chat
5 hours agoDARKTIDE - Warhammer 40k w/ Nubes and AoA
89 watching -
LIVE
Shoriantrax
2 hours agoLIVE: Hardcore Chaos in Tarkov – Loot, Die, Repeat!
17 watching -
LIVE
John_Goetz
2 hours agoJohn Gets Gaming - Medal of Honor Vanguard Part 2
11 watching