Constexpr parsing: non-constexpr containers

[JPenuchot]

I'm currently facing an issue while using ctpg::parser:

• libstdc++ doesn't have a constexpr-ready version of std::optional
• libc++ doesn't have a constexpr-ready version of std::vector

This makes ctpg::parser currently unusable in constexpr functions since both are required but no standard library implementation has them both constexpr-ready.

There could several ways to make constexpr ctpg happen, being:

• Using @TartanLlama's std::optional implementation (libstdc++ compatible)
• Using cest's std::vector implementation (libc++ compatible)
• Re-implementing std::optional (libstdc++ compatible)

I will probably go for one of these options on my side because I need ctpg to work in constexpr contexts for my research, but I would like to know if merging one of these changes on your side sounds reasonable to you, and eventually which one you prefer so I can put my focus on it.

Regards,
Jules

[peter-winter]

Hold on. There is an example of constexpr usage of ctpg, it works just fine.
There is no requirement on std::vector being constexpr at all, constexpr parsing uses my own cvector from ctpg.

Did you examine constexpr parsing used in simple-expr-parser.cpp? (lines 65 - 69)
In fact i'm using ctpg to parse regexes in regex_term class in compile time.

Constexpr vector is a c++20 thing, ctpg relies entirely on c++17

[JPenuchot]

Hold on. There is an example of constexpr usage of ctpg, it works just fine. There is no requirement on std::vector being constexpr at all, constexpr parsing uses my own cvector from ctpg.

In my case I'm getting this error becaused ctpg uses std::vector in my constexpr function because the underlying type isn't compatible with cvector (probably because it's a unique_ptr structure):

/usr/local/include/ctpg/ctpg.hpp:1803:90: note: non-literal type 'detail::parser_value_stack_type_t<cstring_buffer<7>, empty_rules_count, value_variant_type>' (aka 'vector<std::variant<nullptr_t, ctpg::no_type, cest::unique_ptr<asciimath::expr_node_interface_t>, ctpg::term_value<char>, ctpg::term_value<std::string_view>>>') cannot be used in a constant expression
        detail::parser_value_stack_type_t<Buffer, empty_rules_count, value_variant_type> value_stack{};
                                                                                         ^

Did you examine constexpr parsing used in simple-expr-parser.cpp? (lines 65 - 69) In fact i'm using ctpg to parse regexes in regex_term class in compile time.

I did, I'm trying to make a similar example where expressions are stored as a regular AST.

[peter-winter]

Yes. So this is not an issue with std::vector or std::optional.
In fact your suggestions would not fix anything, the main requirement for the constexpr parsing is that all of the value types for terminals and nonterminals need to be literal types, which unique_ptr is clearly not.

This is true, CTPG examines the value types of symbols and all of them need to be cvector_compatible.

What you need is an AST representation as a literal type.
Maybe take a look how I do the constexpr json parsing, it may contain some clues (ctjs.cpp in examples)

By the way std::optional in c++17 is partially constexpr already, and CTPG uses just these parts, and they are all implemented in all standard libraries that CTPG supports.

[JPenuchot]

I figured I got confused while working on making ctpg compatible with non-literal types in constexpr functions: the fact that ctpg::parser variables can be constexpr variables has stronger implications than ctpg::parser being usable in constexpr functions.

constexpr nterm<int> root("root");

constexpr parser p1(root, terms('0', '1', '2'), nterms(root),
                    rules(root('0', '1', '2') >= val(42)));

p1 can't support non-literal types because it's a constexpr variable,

constexpr auto foo() {
  constexpr nterm<int> root("root");

  parser p2(root, terms('0', '1', '2'), nterms(root),
            rules(root('0', '1', '2') >= val(42)));
}

p2 can support non-literal types because it's in a constexpr function, but isn't a constexpr variable.

The use cases that interest me the most are the latter, therefore I will be working on a fork of ctpg that aims to support non-literal types in constexpr functions. It requires compiler support for constexpr new (P0784R7), and I'm not sure all compilers have proper support for that. Also GCC doesn't support virtual constexpr destructors atm.

Is it something you would like mainline ctpg to support someday? I would happily bring those changes to your repo under your own terms.

[peter-winter]

You would have to relax the requirement for the cvector compatibility ad switch to the std vector (or implement once yourself).
The vectors used in parsing are only temporaries and are gathered in a parse_state stucture, the parse result is a std optional, something which is constexpr compatible in c++20 for sure (partialy even in c++17).
Virtual constexpr destructors is not something required to implement what you wan't to achieve, but maybe your application of ctpg requires it.

I would gladly take a look at the progress when you have something.

Such feature would be very nice. I will consider moving it into mainline once you actually have something supported by 3 major compilers (gcc, clang and msvc). I can see a ctpg v2.0 moved to c++20 entirely.

There is a possibility to allow your feature conditionally if compiled with appropriate standard (c++20) options.
C++17 will still require vector implementation without heap allocation (similar to cvector) and require literal types as value types.