Parser choices

[jonmeow]

I'm noting this down here for parser options, since the question came up on #1423, and we'll probably need to update bison versions per #2561.

Right now we use flex/bison for parsing. We're starting to see some edge cases in the grammar where we end up duplicating syntax to get the parser working, and also we've been using an old version, so I was looking at alternatives before spending time figuring out how to upgrade bison.

I've looked at tree-sitter in #2503, but it doesn't provide error messages. While it's likely we'll want to provide a tree-sitter grammar in the future (for IDEs and syntax highlighting in particular), we're generally in agreement that parser error messages are important and we're hesitant to adopt tree-sitter if it means figuring out how to write appropriate error messages for various parser states.

I've looked at antlr4 in #2573. A naive translation of the bison grammar is substantially slower: parsing prelude.carbon in fastbuild currently takes about 15s on my machine (antlr only, without generating an AST), versus sub-second performance with bison (including AST generation). opt is substantially better, but still 1-2s. We might be able to tailor the grammar for antlr, but right now it seems like that'd be difficult and may result in a fairly tangled-looking grammar. The prelude will also keep growing, so unless performance came pretty close to bison we probably wouldn't want to switch. Maybe someone else will be able to provide a significantly better antlr4 grammar, which is why I'm mentioning it here.

If these issues with the alternatives remain, it seems likely that we'll stick with flex/bison for explorer.