tokenizer.texi

The Clang C Tokenizer component offers the ability to construct a sequence of
tokens that represent a software object.

@subsection API Support


The @refapigeneric{sel/cp/clang-tokens:clang-tokens} method takes a @refapiclass{sel:software} object and an
optional list of roots indicating the elements that should be
tokenized. It returns a list of symbols representing all the tokens
under each of the roots.

The Clang C tokenizer uses the following conventions:

@multitable {} {}

@headitem Token Kind @tab Represented By

@item identifier
@tab interned string ``identifier''.

@item special character @* @w{(operator, parenthesis, brace, etc) }
@tab the interned string of the special character, e.g., @code{(intern "+")}.

@item primitive type
@tab the interned string for that type, e.g., @code{(intern "int")}.

@item keyword
@tab the interned string for that keyword, e.g., @code{(intern "if")}.

@item macros
@tab the interned string ``macro'' (macros are generally not expanded in the AST).

@item literals
@tab character, float, imaginary, integer, and string literals
are represented by the following interned strings, respectively:
``char-literal'', ``float-literal'', ``i-literal'', ``int-literal'',
``string-literal''.
@end multitable