unknown characters ignored rather than rejected

[AbigailBrady]

It appears that characters that are not in the valid lexical set (outside of string literals) are often ignored rather than rejected. An error is printed, but no exception is thrown and an AST is returned.

For example, this input file

print(!h!)
results in these error messages
line 1:6 token recognition error at: '!' line 1:8 token recognition error at: '!'
and this ast.

Chunk: {} 2 keys
  body: {} 2 keys
    Block: {} 2 keys
      body: [] 1 item
        0: {} 1 key          
          Call: {} 5 keys
            wrapped: False
            func: {} 4 keys
              Name: {} 4 keys
                wrapped: False
                id: 'print'
            args: [] 1 item
              0: {} 1 key                
                Name: {} 4 keys
                  wrapped: False
                  id: 'h'
            style: DEFAULT

I would expect in this case SyntaxError to have been thrown.

this appears to be a consistent issue with a bunch of characters which are not part of lua syntax. in addtional to !, i also see this behaviour with spurious backticks, dollar signs, backslash and the question mark (which do not form part of lua syntax), but not with tilde, percentage or pipe (which do). i discovered this bug when adding some tests to my prototype downstream static validator to ensure that it is rejecting roblox forked lua (luau) string interpolation, which uses backticks as string delimeters.