WIP: Started working on the Parser

[Michael-F-Bryan]

This is the first instalment towards #13. I'm planning to do a quick high-level overview of the parser and libsyntax and mentioning the major players (Parser, ParseSess, etc) and what they do.

[nikomatsakis]

Here I think I would link to the relevant chapter of the guide -- oh, wait, do we not have a chapter for the AST? We should add one =)

[Michael-F-Bryan]

I was thinking the AST would fall under the parser section, maybe as a short sub-chapter.

[nikomatsakis]

Link to wikipedia? Maybe pointless?

[Michael-F-Bryan]

I just added another commit which points out a couple of (what I perceive to be) the key players in libsyntax. I wasn't really sure what to link to when mentioning the specific modules/types, so I linked to their source files.

What things would someone need to know when either working on the parser or trying to use the parser (e.g. because you're using rustc as a library)? Also, are there any easy-ish issues relating to the parser I can work on to get a better feel for libsyntax and how it is used?

[nikomatsakis]

I wasn't really sure what to link to when mentioning the specific modules/types, so I linked to their source files.

Yeah, that's tricky. For now let's do that. Eventually rustdoc would probably be better.

What things would someone need to know when either working on the parser or trying to use the parser

Hmm, good question. I imagine it'd be good to look at a few PRs that modified the parser to see what they had to do. Actually, it might be worth just including a link to those PRs in the guide to help people get oriented.

rust-lang/rust#45047 is such a PR, and here is a link to some of the relevant content.

Some things immediately jump out as worth explaining:

• What is this eat function?
  • It consumes a token, and queues up the next token as lookahead
• How do you handle identifiers and keywords?
  • Actually, this PR didn't have to do anything around that. I'd have to find a good example (cc @petrochenkov -- thoughts?)
• What about constructing spans?
  • I have to bring this back into cache, but in general there is this last_span thing that stores the span of the last token, and many things follow the idiom of saving the "lo" point of the span, parsing some stuff, then extracting the "hi" point and combining them. This would be used to make a span that encompasses, for example, an entire trait definition (the lo point would come from the trait keyword, but the end point comes after having parsed a bunch of other things).
• What is this crazy pretty printer?
  • It dumps out the AST in human readable form.
  • We have tests that will parse each file in run-pass, pretty print it, and then test that the resulting pretty printed file type checks, so you have to implement this.