Compiler From Scratch: Phase 1 - Tokenizer Generator 011: Dispatching DFAStates in multiple ways

Streamed on 2024-09-27 (https://www.twitch.tv/thediscouragerofhesitancy)

Zero Dependencies Programming!

Last time It mostly implemented dispatching our DFAStates in a switch/case statement. It almost worked, but was blocked by a strange segmentation fault. Over the weekend I debugged off stream, which I don't feel that bad about because it wasn't a problem with my code. It was a problem with the gcc compiler. With no other changes, I changed an std::vector to a simple array and it worked. I changed the simple array to std::list and it worked. Then just for fun I put it back to std::vector and ... it worked! I circled all the way back to where I started with no meaningful change and it works. That is the third time I've run into it where the compiler won't generated correct code unless a very particular line of code is removed or modified, but then it works fine after it is put right back in the same way. Very strange. With that bug "fixed" I was able to generate a sequence of tokens.

Therefore, today we implemented DFAState dispatching using an array of function pointers, and another where we used an array pointer directly. for both cases, I added another #define flag to create the function pointers by lambda and by std::bind. We now have almost 200 permutations of performance flags that we can play with when we get to performance testing. In all, the new dispatch methods went it pretty easy.

After that I updated the "macros" branch of code generation. There are small helper subprocesses that could be either methods or macros, so I generate both. With a few updates I was able to test that the macro helper subprocesses work as well. And all throughout I was making small refactors as I went.

Toward the end of stream, I started adding line tracking to TxtBuf. It more or less went in without a hitch ... except that I couldn't test it because something in the NFA to DFA transformation is broken. So I sidelined the line tracking in TxtBuf to examine the bug. Removing the "GROUP" concept from NFA helped clean up the code a bit, but didn't fix the issue. We'll have to pick up there next week.