Where does tokenization and analysis occur? #1535
Unanswered
MatthewWilliamson4040
asked this question in
Q&A
Replies: 1 comment
-
Okay I partially found my own answer here. I did finally find under |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
-
I've been looking for where the magic happens. Tokenization and the analysis of tokens. In simpler terms, where's the code that determines whether the token in question is a comment, a variable, and likely falls through to just plain text?
I'm trying to locate this for Issue #1534 I've located most other bits of code I need to make it happen, but I am going to need to modify the behavior to teach the program how to recognize some new types of tokens.
I've been digging for a while and was hoping someone more familiar with the project could just point me right to it. I feel I've spent enough time digging to justify asking at this point. Thank you in advance.
Beta Was this translation helpful? Give feedback.
All reactions