Compiler From Scratch: Phase 1 - Tokenizer Generator 010: Dispatching DFAStates with switch/case

Streamed on 2024-09-20 (https://www.twitch.tv/thediscouragerofhesitancy)

Zero Dependencies Programming!

Work on the Tokenizer Generator continues. I plan to implement it with several different processing options so we can compare the performance implications, which leads to the permutations of code to generate a bit unwieldy. There are 96 permutations for the Tokenizer Generator right now. To narrow the scope for the days work we settled on the following three options to get the process working end-to-end in one way. I set the defines in my Makefile for using a C++ style iterator to get each character, to process all the tokens in a single batch, and to dispatch DFAStates with a switch/case statement.

With that scope set for the day I got to work. It didn't take long to dispatch the DFAStates, and there was a bit of work to initialize the tokenization process correctly. Instead of generating the tokens when accepting a token, I just logged the information the tokens would need. This worked pretty well and showed that the flow was working as expected.

Then it was time to actually generate the tokens themselves. I decided to just put the list of Tokens in the Context and not keep them in the Tokenizer. This shouldn't have been too hard, but we ended the day with a crash that I wasn't able to pin down before the end of the stream. The debugger was useless as it wasn't breaking on the lines specified and wouldn't do anything with the stackdump. I am sorely tempted to debug it this weekend off-stream. We'll see.