Premium Only Content

Compiler From Scratch: Phase 1 - Tokenizer Generator 008: Code generating Context and Token
Streamed on 2024-09-06 (https://www.twitch.tv/thediscouragerofhesitancy)
Zero Dependencies Programming!
Today we generated more code to support the tokenizer we will be generating. We generated a class to handle Context. This is a simple object that keeps all the shared state and intermediate results which are in use throughout the different stages. For starters it contains our text buffer and a list of error and warning messages. This class will grow as we need more features.
We also generated the Token class that the Tokenizer will generate. This class contains all the information about where to find the token in the text buffer and a token identifier.
Then we got a start on generating the Tokenizer itself. We have stubbed in some of the simple parts, but there are two main processing methods we could go through: lazy or batch evaluation. Both have different performance implications, so we will generate both inside of ifdef checks so we can test them both.
In all this coding, we are writing out text, and large sections are just text with no modification which we could get from a separate file and not clutter our main code generation. The idea of having code snippets in an asset folder, then an asset pipeline script which embeds the code into our code would mean we could edit our snippets easier and also not count as much of that code towards our line count. We might work on this next week ... unless I get ambitious and excited about it and do it on my own before the next stream. That happens sometimes. Especially when I don't get to work on my compiler more than once a week and start to get the shakes.
-
LIVE
The Mike Schwartz Show
1 hour agoTHE MIKE SCHWARTZ SHOW with DR. MICHAEL J SCHWARTZ 07-10-2025
4,124 watching -
LIVE
LFA TV
14 hours agoLFA TV ALL DAY STREAM - THURSDAY 7/10/25
3,833 watching -
DVR
Welcome to the Rebellion Podcast
13 hours ago $0.97 earnedSorry for the Confusion - WTTR Podcast Live 7/10
22.1K -
1:22:57
Game On!
14 hours ago $2.56 earnedHere's Why The Eagles Are The Betting Favorites To REPEAT As Super Bowl CHAMPS!
52.1K1 -
28:25
The Rad Factory
13 hours ago $3.28 earnedFULL TOUR of My $10k Car & Bike Collection
57K5 -
13:24
James Klüg
1 day agoWhy Are So Many Democrats Not Proud To Be American?
25.2K6 -
1:23:26
Dad Dojo Podcast
1 day ago $2.02 earnedEP39: Josh Fighting Nate Diaz?
19.9K2 -
53:00
The Dr. Ardis Show
1 day ago $7.67 earnedThe Dr. Ardis Show | Is There An Answer to Lyme Disease w/Kerri Simpson | Episode 07.09.2025
28.5K14 -
2:00:02
BEK TV
1 day agoTrent Loos in the Morning 7/10/2025
20.4K1 -
8:25
The Car Guy Online
21 hours ago $3.31 earnedRAMs Latest BLUNDER & The PLASTIFICATION of Trucks. RAM 1500, 2500 & 3500
60.9K17