Premium Only Content

Compiler From Scratch: Phase 1 - Tokenizer Generator 008: Code generating Context and Token
Streamed on 2024-09-06 (https://www.twitch.tv/thediscouragerofhesitancy)
Zero Dependencies Programming!
Today we generated more code to support the tokenizer we will be generating. We generated a class to handle Context. This is a simple object that keeps all the shared state and intermediate results which are in use throughout the different stages. For starters it contains our text buffer and a list of error and warning messages. This class will grow as we need more features.
We also generated the Token class that the Tokenizer will generate. This class contains all the information about where to find the token in the text buffer and a token identifier.
Then we got a start on generating the Tokenizer itself. We have stubbed in some of the simple parts, but there are two main processing methods we could go through: lazy or batch evaluation. Both have different performance implications, so we will generate both inside of ifdef checks so we can test them both.
In all this coding, we are writing out text, and large sections are just text with no modification which we could get from a separate file and not clutter our main code generation. The idea of having code snippets in an asset folder, then an asset pipeline script which embeds the code into our code would mean we could edit our snippets easier and also not count as much of that code towards our line count. We might work on this next week ... unless I get ambitious and excited about it and do it on my own before the next stream. That happens sometimes. Especially when I don't get to work on my compiler more than once a week and start to get the shakes.
-
LIVE
The Shannon Joy Show
1 hour ago🔥🔥ALERT!! Medical Freedom Doc Kirk Moore Is Facing 35 Years In Federal Prison, Persecuted By Trump’s DOJ And Forgotten By #MAHA!🔥🔥
178 watching -
14:37
Michael Button
3 days ago $1.37 earnedEvidence For LOST Ancient Civilization is Growing 🤯
13.8K5 -
1:01:23
VINCE
3 hours agoExplosive New Info Against The Deep State Emerges | Episode 82 - 07/10/25
204K168 -
1:22:09
theoriginalmarkz
2 hours agoCoffee with MarkZ. 07/10/2025
22.8K1 -
DVR
Bannons War Room
4 months agoWarRoom Live
27.4M6.92K -
1:14:29
The Big Migâ„¢
3 hours agoThe Biden Mush Brain Cover-Up Saga Continues
14.6K8 -
1:39:47
Badlands Media
8 hours agoBadlands Daily - July 10, 2025
57.8K3 -
1:15:03
Dear America
4 hours agoWhy Is EVERYWHERE Flooding!! + I Am DONE With All Of The LIES!!!
125K260 -
2:18:22
Matt Kohrs
11 hours agoMARKETS ARE SET TO SKYROCKET 🚀🚀🚀 || Live Trading Options & Futures
29.7K1 -
2:59:32
Wendy Bell Radio
7 hours agoTiming Is Everything
62.9K47