Premium Only Content

Compiler From Scratch: Phase 1 - Tokenizer Generator 008: Code generating Context and Token
Streamed on 2024-09-06 (https://www.twitch.tv/thediscouragerofhesitancy)
Zero Dependencies Programming!
Today we generated more code to support the tokenizer we will be generating. We generated a class to handle Context. This is a simple object that keeps all the shared state and intermediate results which are in use throughout the different stages. For starters it contains our text buffer and a list of error and warning messages. This class will grow as we need more features.
We also generated the Token class that the Tokenizer will generate. This class contains all the information about where to find the token in the text buffer and a token identifier.
Then we got a start on generating the Tokenizer itself. We have stubbed in some of the simple parts, but there are two main processing methods we could go through: lazy or batch evaluation. Both have different performance implications, so we will generate both inside of ifdef checks so we can test them both.
In all this coding, we are writing out text, and large sections are just text with no modification which we could get from a separate file and not clutter our main code generation. The idea of having code snippets in an asset folder, then an asset pipeline script which embeds the code into our code would mean we could edit our snippets easier and also not count as much of that code towards our line count. We might work on this next week ... unless I get ambitious and excited about it and do it on my own before the next stream. That happens sometimes. Especially when I don't get to work on my compiler more than once a week and start to get the shakes.
-
2:03:39
Pop Culture Crisis
4 hours agoWTF Happened to Miley Cyrus? HARRY POTTER Set Images, Sydney Sweeney Hater EXPOSED | Ep. 901
18.4K18 -
10:17
MattMorseTV
4 hours ago $5.37 earnedTrump's DOJ just DROPPED a NUKE.
23.8K33 -
11:07
Politibrawl
12 days agoRepublican rising star DESTROYS theatrical Democrat in front of the entire world
66.7K30 -
4:42:30
StoneMountain64
5 hours agoOnly game with BETTER desctruction than Battlefield?
45.7K -
4:51:37
Viss
7 hours ago🔴LIVE - Viss & Dr Disrespect Take on The 5 Win Minimum PUBG Challenge!
21.7K2 -
LIVE
sophiesnazz
3 hours ago $0.52 earnedLETS TALK ABOUT BO7 !socials !specs
42 watching -
1:12:05
The Quartering
5 hours agoToday's Breaking News!
82.3K43 -
8:20:24
GritsGG
9 hours agoWin Streaking! Most Wins 3390+ ðŸ§
53.6K2 -
2:20:00
Tucker Carlson
6 hours agoDave Collum: Financial Crisis, Diddy, Energy Weapons, QAnon, and the Deep State’s Digital Evolution
121K115 -
1:06:56
Sean Unpaved
21 hours agoSwitch-Hitting Stories: Chipper on Baseball, Football, & the Game's Future
29.1K1