Premium Only Content

Compiler From Scratch: Phase 1 - Tokenizer Generator 008: Code generating Context and Token
Streamed on 2024-09-06 (https://www.twitch.tv/thediscouragerofhesitancy)
Zero Dependencies Programming!
Today we generated more code to support the tokenizer we will be generating. We generated a class to handle Context. This is a simple object that keeps all the shared state and intermediate results which are in use throughout the different stages. For starters it contains our text buffer and a list of error and warning messages. This class will grow as we need more features.
We also generated the Token class that the Tokenizer will generate. This class contains all the information about where to find the token in the text buffer and a token identifier.
Then we got a start on generating the Tokenizer itself. We have stubbed in some of the simple parts, but there are two main processing methods we could go through: lazy or batch evaluation. Both have different performance implications, so we will generate both inside of ifdef checks so we can test them both.
In all this coding, we are writing out text, and large sections are just text with no modification which we could get from a separate file and not clutter our main code generation. The idea of having code snippets in an asset folder, then an asset pipeline script which embeds the code into our code would mean we could edit our snippets easier and also not count as much of that code towards our line count. We might work on this next week ... unless I get ambitious and excited about it and do it on my own before the next stream. That happens sometimes. Especially when I don't get to work on my compiler more than once a week and start to get the shakes.
-
2:46:21
TimcastIRL
10 hours agoTexas GOP Greenlights ARREST WARRANTS For Dems Who FLED State | Timcast IRL
257K93 -
3:16:50
Laura Loomer
10 hours agoEP137: EXPOSED: How Tucker Carlson Became Hunter Biden's Wingman
82.6K98 -
4:22:34
Akademiks
9 hours agoNicki Minaj vs Dez Bryant. Trump Calls out Charlamagne. Diddy Denied Bail Again! ICEMAN soon?
68.1K3 -
3:16:28
Nerdrotic
11 hours ago $11.71 earnedNerdrotic at Night 504
93.7K3 -
8:20:30
Dr Disrespect
19 hours ago🔴LIVE - DR DISRESPECT - WARZONE - RANDOMLY GENERATED LOADOUTS EVENT
241K9 -
1:36:07
Glenn Greenwald
13 hours agoTrump Admin Unleashes More Policies That Prioritize Israel Over American Citizens; The Smear Campaign Against Gaza Aid Whistleblower with Journalist Mel Witte | SYSTEM UPDATE #497
152K198 -
58:14
MattMorseTV
12 hours ago $16.09 earned🔴Hakeem just lost EVERYTHING.🔴
90.6K64 -
9:21:36
Rallied
15 hours ago $3.20 earnedWARZONE CHALLENGES WITH DRDISRESPECT & BOB
90.1K5 -
7:15:14
SpartakusLIVE
12 hours agoMonday MOTIVATION leaves viewers ENERGIZED and MAXIMIZED to conquer the week
40.9K1 -
3:19:17
Barry Cunningham
11 hours agoPRESIDENT TRUMP IS THE SUM OF THEIR FEARS! IT HAS BEGUN!
69K77