Return a single token for OpenQASM version statement from lexer

This commit changes lexing the OpenQASM version statement. Flags
recording failure to recognize major and minor versions are included.

An example of the statement is
```
OPENQASM 3.1;
```

Here are some choices for lexing the openqasm version statement.
This commit changes the implementaton from the first choice to the second.

This was first implemented by doing nothing in the lexer to treat this
statement. This is the way almost all keywords are lexed. All identifiers are
recognized as keywords when parsing, rather than lexing. The number `3.1` is
tokenized as a float literal (actually an integer literal, converted to float
somewhere before parsing. This is inherited from r-a) But parsing does not have
access to the input text, so this defers checking that `3.1` is a valid to the
semantic analysis. This last step is easy enough to do, but was not done.

Another way to implement lexing version statement is to recognize the entire
statement as a token. This means recognizing the character sequence`OPENQASM` +
whitespace + valid version number. If this fails, it will be lexed as an
invalid identifier, or an invalid version statement, depending on where the
error is. This allows catching errors where they ought to be caught, at the
lexing and/or parsing stages. The version is then extracted from the
token. This can be done with an api call created from the ungrammar, and/or
coded by hand.

Another way is to have the lexer recognize `OPENQASM` as a token, then a
whitespace token, and finally a version number tokenkind, say
`OPENQASM_VERSION`. This would be easier to parse and consume in semantic
analysis. But the version number text would ordinarily be lexed as a float
literal. Furthermore, the lexer produces a stream of tokens and maintains as
little state a possible. We would need to break this abstraction in some way to
accommodate this way of parsing the version string. For example, saving tokens
and emitting all three when done. Or entering a special mode. The reference
parser enters a "mode" enabled by the antlr parser generator. The lexer is
currently very simple and fast. I don't want to experiment with adding
complexity just for this purpose.

Author: jlapeyre

crates/oq3_lexer/src/lib.rs

@@ -76,6 +76,11 @@ pub enum TokenKind {
     /// Like the above, but containing invalid unicode codepoints.
     InvalidIdent,
 
+    OpenQasmVersionStmt {
+        major: bool,
+        minor: bool,
+    },
+
     Pragma,
 
     Dim,
@@ -301,6 +306,22 @@ impl Cursor<'_> {
 
             'p' => self.pragma_or_ident_or_unknown_prefix(),
 
+            // This implementation stores the entire "OPENQASM x.y" string as a single token.
+            // There are several options, but others seem to come with a cost.
+            // At present the tokenizer has in a sense no context, very little state.
+            // There are no "modes". It might be possible to do this in a way that does not
+            // affect performance. But I don't want to take the time to experiment. Breaking
+            // an abstraction for one small feature seems unwise.
+            'O' => {
+                if self.have_openqasm() {
+                    self.eat_while(is_whitespace);
+                    let (major, minor) = self.openqasm_version();
+                    OpenQasmVersionStmt { major, minor }
+                } else {
+                    self.ident_or_unknown_prefix()
+                }
+            }
+
             // Identifier (this should be checked after other variant that can
             // start as identifier).
             c if is_id_start(c) => self.ident_or_unknown_prefix(),
@@ -483,6 +504,7 @@ impl Cursor<'_> {
         }
     }
 
+    /// Yikes!! assumes the *previous* token was also whitespace.
     fn whitespace(&mut self) -> TokenKind {
         debug_assert!(is_whitespace(self.prev()));
         self.eat_while(is_whitespace);
@@ -528,6 +550,55 @@ impl Cursor<'_> {
         false
     }
 
+    /// This is called if we just consumed 'O'
+    /// If we consume "OPENQASM" + whitespace, then is is a version statement.
+    /// Otherwise, an identifier, valid or not.
+    fn have_openqasm(&mut self) -> bool {
+        if self.first() == 'P' {
+            self.bump();
+            if self.first() == 'E' {
+                self.bump();
+                if self.first() == 'N' {
+                    self.bump();
+                    if self.first() == 'Q' {
+                        self.bump();
+                        if self.first() == 'A' {
+                            self.bump();
+                            if self.first() == 'S' {
+                                self.bump();
+                                if self.first() == 'M' {
+                                    self.bump();
+                                    return is_whitespace(self.first());
+                                }
+                            }
+                        }
+                    }
+                }
+            }
+        }
+        false
+    }
+
+    fn openqasm_version(&mut self) -> (bool, bool) {
+        if !self.eat_decimal_digits() {
+            return (false, false);
+        }
+        let c = self.first();
+        if c == '.' {
+            self.bump();
+
+            if !self.eat_decimal_digits() {
+                // Do not allow "3."
+                return (true, false);
+            }
+        }
+        let c = self.first();
+        if c != ';' && !is_whitespace(c) {
+            return (false, false);
+        }
+        (true, true)
+    }
+
     fn pragma_or_ident_or_unknown_prefix(&mut self) -> TokenKind {
         if self.have_pragma() {
             Pragma
@@ -856,6 +927,7 @@ impl Cursor<'_> {
     }
 
     fn eat_decimal_digits(&mut self) -> bool {
+        //
         let mut has_digits = false;
         loop {
             match self.first() {

crates/oq3_parser/src/grammar/expressions.rs

@@ -101,6 +101,14 @@ pub(crate) fn stmt(p: &mut Parser<'_>) {
         return q_or_c_reg_declaration(p, m);
     }
 
+    if p.at(VERSION_STRING) {
+        p.bump_any();
+        if !p.eat(T![;]) {
+            p.error("Expecting semicolon terminating statement");
+        }
+        m.complete(p, VERSION_STRING);
+        return;
+    }
     // FIXME: straighten out logic
     if !(p.current().is_classical_type() && (p.nth(1) == T!['('] || p.nth(1) == T!['[']))
         && !p.at_ts(EXPR_FIRST)

crates/oq3_parser/src/lexed_str.rs

@@ -225,6 +225,19 @@ fn inner_extend_token<'a>(
                 COMMENT
             }
 
+            oq3_lexer::TokenKind::OpenQasmVersionStmt { major, minor } => {
+                if *major {
+                    if *minor {
+                        // All good
+                    } else {
+                        err = "Invalid minor version in OpenQASM version statement";
+                    }
+                } else {
+                    err = "Invalid version number in OpenQASM version statement";
+                }
+                VERSION_STRING
+            }
+
             oq3_lexer::TokenKind::Whitespace => WHITESPACE,
             oq3_lexer::TokenKind::Ident if token_text == "_" => UNDERSCORE,
 

crates/oq3_semantics/src/syntax_to_semantics.rs

@@ -536,8 +536,10 @@ fn stmt_to_asg_stmt(stmt: synast::Stmt, context: &mut Context) -> Option<asg::St
         synast::Stmt::ExprStmt(expr_stmt) => expr_stmt_to_asg_stmt(expr_stmt, context),
 
         synast::Stmt::VersionString(version_string) => {
-            let version = version_string.version().unwrap().version().unwrap();
-            let _ = version.split_into_parts();
+            // let version = version_string.version().unwrap().version().unwrap();
+            //            not_impl!(context, version_string)
+            context.insert_error(NotImplementedError, &version_string);
+            // Better None than NullStmt.
             None
         }
 

crates/oq3_semantics/tests/from_string_tests.rs

@@ -515,7 +515,7 @@ bit[2] out;
 out[0] = measure $0;
 "#;
     let (program, errors, _symbol_table, _have_syntax_errors) = parse_string(code);
-    assert!(errors.is_empty());
+    assert_eq!(errors.len(), 1);
     assert_eq!(program.len(), 2);
 }
 

crates/oq3_semantics/tests/spec.rs

@@ -56,7 +56,7 @@ include "stdgates.inc";
 // Rest of QASM program
 "#;
     let (program, errors, _symbol_table) = parse_string(code);
-    assert!(errors.is_empty());
+    assert_eq!(errors.len(), 1);
     assert!(program.is_empty());
 }
 

crates/pipeline-tests/tests/snapshots/runner__tests__snippets__invalid__statements__headers.qasm-lex.snap

@@ -18,43 +18,38 @@ include "hello;
 ok: false
 errors: 1
 [0] Whitespace "\n" @0..1
-[1] Ident "OPENQASM" @1..9
-[2] Whitespace " " @9..10
-[3] Ident "int" @10..13
-[4] Semi ";" @13..14
-[5] Whitespace "\n" @14..15
-[6] Ident "OPENQASM" @15..23
-[7] Whitespace " " @23..24
-[8] Literal { kind: Str { terminated: true }, suffix_start: 14 } "'hello, world'" @24..38
-[9] Semi ";" @38..39
-[10] Whitespace "\n" @39..40
-[11] Ident "OPENQASM" @40..48
-[12] Whitespace " " @48..49
-[13] Literal { kind: Int { base: Decimal, empty_int: false }, suffix_start: 1 } "3" @49..50
-[14] Whitespace " " @50..51
-[15] Literal { kind: Int { base: Decimal, empty_int: false }, suffix_start: 1 } "3" @51..52
-[16] Semi ";" @52..53
-[17] Whitespace "\n" @53..54
-[18] Ident "OPENQASM" @54..62
-[19] Whitespace " " @62..63
-[20] Literal { kind: Float { base: Decimal, empty_exponent: false }, suffix_start: 2 } "3.x" @63..66
-[21] Semi ";" @66..67
-[22] Whitespace "\n" @67..68
-[23] Ident "include" @68..75
-[24] Whitespace " " @75..76
-[25] Literal { kind: Int { base: Decimal, empty_int: false }, suffix_start: 1 } "3" @76..77
-[26] Semi ";" @77..78
-[27] Whitespace "\n" @78..79
-[28] Ident "include" @79..86
-[29] Whitespace " " @86..87
-[30] Ident "include" @87..94
-[31] Semi ";" @94..95
-[32] Whitespace "\n" @95..96
-[33] Ident "include" @96..103
-[34] Whitespace " " @103..104
-[35] Ident "def" @104..107
-[36] Semi ";" @107..108
-[37] Whitespace "\n" @108..109
-[38] Ident "include" @109..116
-[39] Whitespace " " @116..117
-[40] Literal { kind: Str { terminated: false }, suffix_start: 8 } ""hello;\n" @117..125
+[1] OpenQasmVersionStmt { major: false, minor: false } "OPENQASM " @1..10
+[2] Ident "int" @10..13
+[3] Semi ";" @13..14
+[4] Whitespace "\n" @14..15
+[5] OpenQasmVersionStmt { major: false, minor: false } "OPENQASM " @15..24
+[6] Literal { kind: Str { terminated: true }, suffix_start: 14 } "'hello, world'" @24..38
+[7] Semi ";" @38..39
+[8] Whitespace "\n" @39..40
+[9] OpenQasmVersionStmt { major: true, minor: true } "OPENQASM 3" @40..50
+[10] Whitespace " " @50..51
+[11] Literal { kind: Int { base: Decimal, empty_int: false }, suffix_start: 1 } "3" @51..52
+[12] Semi ";" @52..53
+[13] Whitespace "\n" @53..54
+[14] OpenQasmVersionStmt { major: true, minor: false } "OPENQASM 3." @54..65
+[15] Ident "x" @65..66
+[16] Semi ";" @66..67
+[17] Whitespace "\n" @67..68
+[18] Ident "include" @68..75
+[19] Whitespace " " @75..76
+[20] Literal { kind: Int { base: Decimal, empty_int: false }, suffix_start: 1 } "3" @76..77
+[21] Semi ";" @77..78
+[22] Whitespace "\n" @78..79
+[23] Ident "include" @79..86
+[24] Whitespace " " @86..87
+[25] Ident "include" @87..94
+[26] Semi ";" @94..95
+[27] Whitespace "\n" @95..96
+[28] Ident "include" @96..103
+[29] Whitespace " " @103..104
+[30] Ident "def" @104..107
+[31] Semi ";" @107..108
+[32] Whitespace "\n" @108..109
+[33] Ident "include" @109..116
+[34] Whitespace " " @116..117
+[35] Literal { kind: Str { terminated: false }, suffix_start: 8 } ""hello;\n" @117..125

crates/pipeline-tests/tests/snapshots/runner__tests__snippets__invalid__statements__headers.qasm-parse.snap

@@ -7,6 +7,6 @@ expect-parse: Diag
 --- parser ---
 ok: false
 panicked: false
-errors: 1
+errors: 4
 --- ast ---
 (no ast)

crates/pipeline-tests/tests/snapshots/runner__tests__snippets__reference__comments__comments.qasm-lex.snap

@@ -21,57 +21,55 @@ cx q[0], q[1];
 ok: true
 errors: 0
 [0] Whitespace "\n" @0..1
-[1] Ident "OPENQASM" @1..9
-[2] Whitespace " " @9..10
-[3] Literal { kind: Float { base: Decimal, empty_exponent: false }, suffix_start: 3 } "3.0" @10..13
-[4] Semi ";" @13..14
-[5] Whitespace "\n" @14..15
-[6] LineComment "// Line comment before include" @15..45
-[7] Whitespace "\n" @45..46
-[8] Ident "include" @46..53
-[9] Whitespace " " @53..54
-[10] Literal { kind: Str { terminated: true }, suffix_start: 14 } ""stdgates.inc"" @54..68
-[11] Semi ";" @68..69
-[12] Whitespace " " @69..70
-[13] LineComment "// Inline comment" @70..87
-[14] Whitespace "\n" @87..88
-[15] BlockComment { terminated: true } "/* Block comment before declaration */" @88..126
-[16] Whitespace "\n" @126..127
-[17] Ident "qubit" @127..132
-[18] OpenBracket "[" @132..133
-[19] Literal { kind: Int { base: Decimal, empty_int: false }, suffix_start: 1 } "2" @133..134
-[20] CloseBracket "]" @134..135
-[21] Whitespace " " @135..136
-[22] Ident "q" @136..137
-[23] Semi ";" @137..138
-[24] Whitespace " " @138..139
-[25] BlockComment { terminated: true } "/* Inline block comment */" @139..165
-[26] Whitespace "\n\n" @165..167
-[27] LineComment "// Comment before gate" @167..189
-[28] Whitespace "\n" @189..190
-[29] Ident "h" @190..191
-[30] Whitespace " " @191..192
-[31] Ident "q" @192..193
-[32] OpenBracket "[" @193..194
-[33] Literal { kind: Int { base: Decimal, empty_int: false }, suffix_start: 1 } "0" @194..195
-[34] CloseBracket "]" @195..196
-[35] Semi ";" @196..197
-[36] Whitespace " " @197..198
-[37] LineComment "// Gate with comment" @198..218
-[38] Whitespace "\n" @218..219
-[39] BlockComment { terminated: true } "/* Multi-line block comment\n   spanning multiple lines */" @219..276
-[40] Whitespace "\n" @276..277
-[41] Ident "cx" @277..279
-[42] Whitespace " " @279..280
-[43] Ident "q" @280..281
-[44] OpenBracket "[" @281..282
-[45] Literal { kind: Int { base: Decimal, empty_int: false }, suffix_start: 1 } "0" @282..283
-[46] CloseBracket "]" @283..284
-[47] Comma "," @284..285
-[48] Whitespace " " @285..286
-[49] Ident "q" @286..287
-[50] OpenBracket "[" @287..288
-[51] Literal { kind: Int { base: Decimal, empty_int: false }, suffix_start: 1 } "1" @288..289
-[52] CloseBracket "]" @289..290
-[53] Semi ";" @290..291
-[54] Whitespace "\n" @291..292
+[1] OpenQasmVersionStmt { major: true, minor: true } "OPENQASM 3.0" @1..13
+[2] Semi ";" @13..14
+[3] Whitespace "\n" @14..15
+[4] LineComment "// Line comment before include" @15..45
+[5] Whitespace "\n" @45..46
+[6] Ident "include" @46..53
+[7] Whitespace " " @53..54
+[8] Literal { kind: Str { terminated: true }, suffix_start: 14 } ""stdgates.inc"" @54..68
+[9] Semi ";" @68..69
+[10] Whitespace " " @69..70
+[11] LineComment "// Inline comment" @70..87
+[12] Whitespace "\n" @87..88
+[13] BlockComment { terminated: true } "/* Block comment before declaration */" @88..126
+[14] Whitespace "\n" @126..127
+[15] Ident "qubit" @127..132
+[16] OpenBracket "[" @132..133
+[17] Literal { kind: Int { base: Decimal, empty_int: false }, suffix_start: 1 } "2" @133..134
+[18] CloseBracket "]" @134..135
+[19] Whitespace " " @135..136
+[20] Ident "q" @136..137
+[21] Semi ";" @137..138
+[22] Whitespace " " @138..139
+[23] BlockComment { terminated: true } "/* Inline block comment */" @139..165
+[24] Whitespace "\n\n" @165..167
+[25] LineComment "// Comment before gate" @167..189
+[26] Whitespace "\n" @189..190
+[27] Ident "h" @190..191
+[28] Whitespace " " @191..192
+[29] Ident "q" @192..193
+[30] OpenBracket "[" @193..194
+[31] Literal { kind: Int { base: Decimal, empty_int: false }, suffix_start: 1 } "0" @194..195
+[32] CloseBracket "]" @195..196
+[33] Semi ";" @196..197
+[34] Whitespace " " @197..198
+[35] LineComment "// Gate with comment" @198..218
+[36] Whitespace "\n" @218..219
+[37] BlockComment { terminated: true } "/* Multi-line block comment\n   spanning multiple lines */" @219..276
+[38] Whitespace "\n" @276..277
+[39] Ident "cx" @277..279
+[40] Whitespace " " @279..280
+[41] Ident "q" @280..281
+[42] OpenBracket "[" @281..282
+[43] Literal { kind: Int { base: Decimal, empty_int: false }, suffix_start: 1 } "0" @282..283
+[44] CloseBracket "]" @283..284
+[45] Comma "," @284..285
+[46] Whitespace " " @285..286
+[47] Ident "q" @286..287
+[48] OpenBracket "[" @287..288
+[49] Literal { kind: Int { base: Decimal, empty_int: false }, suffix_start: 1 } "1" @288..289
+[50] CloseBracket "]" @289..290
+[51] Semi ";" @290..291
+[52] Whitespace "\n" @291..292

crates/pipeline-tests/tests/snapshots/runner__tests__snippets__reference__comments__comments.qasm-parse.snap

@@ -23,7 +23,6 @@ h q[0]; // Gate with comment
 cx q[0], q[1];
 
 VERSION_STRING@1..14: OPENQASM 3.0;
-VERSION@10..14: 3.0;
 INCLUDE@46..69: include "stdgates.inc";
 FILE_PATH@54..68: "stdgates.inc"
 QUANTUM_DECLARATION_STATEMENT@127..138: qubit[2] q;