[cpp] Marshalling Extern Types (#11981)

* Initial class, interface, and enum separation

* fold init stuff

* Move interface header generation into its own module

* reduce some duplication

* managed interface implementation gen in its own module

* cache dependencies in the ctx

* move to gencpp with the other context data func

* remove interface code from class gen modules

* Use option for reference wildcard

* path Map

* Add back objc guard

* calculate IDs up front

* separate id lookup table

* Object IDs module to hide cache

* remap enum constructors

* more interface dead code removal

* tcpp interface type

* name, flags, and debug level in tcpp_class

* remove unused constructor var list return item

* separate out header field generation

* separate managed and native header gen functions

* move header stuff into shared function

* break class impl field gen function down

* add converted fields to tcpp_class

* Initial split for managed and native class impl

* map for haxe and native implementations

* use a list for haxe and native implementations

* use tcpp_class field variables

* field rename and meta / rtti filtering

* create fields and default functions at class transform

* use container flag

* Mark and visit using variables and fix inverted container flag

* static mark and visit use static variables list

* print reflective fields

* don't duplicate boot generation code

* don't duplicate init function generation

* don't duplicate dynamic function allocation generation

* generate properties and __Field function

* fold right member get fields

* static get fields

* shared member and static get fold functions

* static set function uses ordered fields

* member set uses organised fields

* turn abstract functions into normal functions

* cppia gen uses organised functions

* simplify ScriptNamedFunction generation

* bit of simplification of more cppia stuff

* GetFields uses organised fields

* move class retyping into retyper

* remap interface functions

* store interface hash

* store meta and rtti fields in remapped interface

* dead code removal

* tcpp interfaces only store their functions

* remapped enum fields rename for consistency

* calculate interface slots ahead of time

* remap interface arg names

* retype the interfaces stored on retyped classes

* retype tcpp_class supers

* find_class_implementation uses tcpp_class

* folds for class interface code

* other minor cleanup

* use flags in class generation to selectively generate some funcs

* retype class functions

* retype class variables

* dedicated var and dyn function boot functions

* dead code cleanup

* use bindings instead of to_list

* own implementation of of_list

* replace another to_list

* Fix pointer and struct wrapping

* fix debug level not being reset between classes

* add space after static

* remove unused variable

* ensure SourceFile paths are made absolute

* some debugging

* remove debugging prints and store class params

* don't use already mangled name when generating the getter name in reflection

* same for setters

* dont use new gc references function

* better handling of static and virtual attributes

* generate scriptable class using tcpp class function lists

* go back to type cant be null

* Container flag refers to entire inheritance tree

* allow code to easily distinguish between the different container types

* remove accidentally tracked file

* pass a context object around the retyper

* closures tracked in retyper ctx

* injection stored in retyper ctx

* declarations track by retyper ctx

* this handling moved into ctx

* gc stack handled by retyper ctx

* function return type handled by retyper ctx

* move loop goto management into retyper ctx

* consistent retyper ctx variable name

* move remaining counter into the fold

* shared string_map_of_list function

* initial value type support

* generic templates and correct return types

* Boxing of captured locals and use marshalling struct helper

* cast dynamic to a reference object

* always use a reference object instead of sometimes directly a boxed object

* cast variant to value type

* error on to value type field closures

* Use dedicated new marshal types

* Update forgotten about old cpp reference

* use _hx_vt as a prefix

* support namespace field

* omit brackets for stack constructors which have no arguments

* Fix alignment of reference declarations in closures

* default to creating boxed value types and deal with the few special cases

* some nullable support

* move auto cast into a separate file

* retype tvar to tcpp specific type

* retype tvar to simplify code and fix closure argument assignment

* promote function and lambda args if captured

* unify closure and function arg generation

* initial attempt at assigning value types a state

* some cleanup surrounding value type state assignment

* remove the two variable creation system

* implement class fields and statics

* wrap value type arguments for new non based struct value type

* cleanup of code and default values

* support enums

* treat abstract this variable as a reference and cast with a promoted type

* pointer type interop support

* ensure arrays of value types are boxed

* treat nullable value types as promoted

* default to heap construction for references

* treat extern value type fields of extern value types as refences

* avoid double :: when no namespace is specified

* another pass to catch some missing :: prefixes

* Initial enum support and comparison operator support

* correct enum casting

* handle extern value type field access and assignment correctly

* fallback to stack allocation instead of heap

* Allow creating reference state values types on the stack

* require reference semantics metadata

* Don't use the package if no namespace is provided

* add inheritance and casting tests

* support star of and dereference native functions

* Initial support for pointer types

* Add forgotten meta json change

* make sure all boxed pointer vars are allocated

* handle adding pointer stars once

* Use custom marshal pointer type for pointer references

* pointer and value specific reference cpp types

* don't retype args with dynamic

* bits of cleanup and specifying some types more

* Remove need for assignment filter, now fixed at source

* remove useless argument

* retype null promoted pointers into boxed objects

* inheritance casting with pointer types as well

* Look for include meta on extern abstracts

* don't add * to pointer type and references

* Don't use subscript operator for pointers of marshal types

* less code duplication and unwrap marshal type params

* do something for cppia

* require reference semantics on pointer types

* Don't allow constructors on pointer types

* Initial support for managed externs

* first pass of the _obj handling

* Handle static functions and variable

* Fix incorrect template handling with standard naming

* another standard naming fix

* allow static fields on extern pointers

* better static handling and helper function name consistency

* Exclude marshal types from having debug info generated

* rename marshal type enum for consistency

* add managed marshal type to cppia enum

* don't follow away marshalling enum abstracts to ensure we see meta

* More generic extern abstract following

* Add a bunch of errors

* Fix error position

* Move meta up and fill in doc comment

* try a different way of dealing with extern abstracts metadata

* actually fix include and abstract issue

* Ensure all marshal types are object returns in cppia wrapper classes

* I knew arguments were hiding in there somewhere...

* use proper marshalling state

* deal with args in the return call as well

* do same for interfaces

* forgotten bracket

* fix call dump

* rework cppia glue generation

* do similar cleanup for interfaces

* store the interface return type as a tcpp

* store interface args as retyped versions

* bloody submodules

* require value semantics instead

* Deal with generated temporaries which need to be value types

Also a quick fix for templated return types

* less string maps

* Support array access on marshaling types

* initial templated function support

* Guard against unresolved extern parameters

* reinterpreting to value types

* Fix pointer to pointer types, and add view extern

* static function template types

* Document some marshal functions

* Document marshal toString functions

* ViewExtensions convenience functions

* view compare and marshal read / write helpers

* bit of code shuffling and checking for stack only types in classes and enums

* store errors in one place

* make view stack only and use size_t for length

* Guard again promoting stack only value type in expressions

* endian specific read and write functions, plus ref to view

* view based FPHelper

* Dodge pointer vs user var fusion issue

* Change the public api to use int64 over size_t

* throw when converting big view to haxe type

* view based byte implementation

* test using hxcpp fork

* set docgen to use hxcpp fork as well

* Revert "set docgen to use hxcpp fork as well"

This reverts commit fcd3c1a.

* Revert "test using hxcpp fork"

This reverts commit 31ecd0b.

Author: Aidan63

src-json/meta.json

@@ -1013,6 +1013,33 @@
 		"platforms": ["cpp"],
 		"targets": ["TClass"]
 	},
+	{
+		"name": "NativeArrayAccess",
+		"metadata": ":nativeArrayAccess",
+		"doc": "When used on an extern class which implements haxe.ArrayAccess native array access syntax will be generated",
+		"platforms": ["cpp"]
+	},
+	{
+		"name": "CppValueType",
+		"metadata": ":cpp.ValueType",
+		"doc": "Indicates that the externed type should be treated as a value type",
+		"platforms": ["cpp"],
+		"targets": [ "TClass", "TAbstract" ]
+	},
+	{
+		"name": "CppPointerType",
+		"metadata": ":cpp.PointerType",
+		"doc": "Indicates that the externed type should be treated as a pointer",
+		"platforms": ["cpp"],
+		"targets": [ "TClass" ]
+	},
+	{
+		"name": "CppManagedType",
+		"metadata": ":cpp.ManagedType",
+		"doc": "Indicates that the externed type is a sub class of ::hx::Object",
+		"platforms": ["cpp"],
+		"targets": ["TClass"]
+	},
 	{
 		"name": "StructInit",
 		"metadata": ":structInit",
@@ -1138,11 +1165,5 @@
 		"metadata": ":exceptionTypeCheck",
 		"doc": "Internally used for transformed exception type check wrt to `catch`.",
 		"internal": true
-	},
-	{
-		"name": "NativeArrayAccess",
-		"metadata": ":nativeArrayAccess",
-		"doc": "When used on an extern class which implements haxe.ArrayAccess native array access syntax will be generated",
-		"platforms": ["cpp"]
 	}
 ]

src/generators/cpp/cppAst.ml

@@ -40,7 +40,17 @@ module InterfaceSlots = struct
     StringMap.find_opt name slots.hash
 end
 
-type tcpp =
+type marshal_type_state =
+  | Stack
+  | Promoted
+  | Reference
+
+and native_type = 
+  | ValueClass of tclass * tcpp list
+  | ValueEnum of tabstract
+  | Pointer of tclass * tcpp list
+
+and tcpp =
   | TCppDynamic
   | TCppUnchanged
   | TCppObject
@@ -58,6 +68,8 @@ type tcpp =
   | TCppRest of tcpp
   | TCppReference of tcpp
   | TCppStruct of tcpp
+  | TCppMarshalNativeType of native_type * marshal_type_state
+  | TCppMarshalManagedType of tclass * tcpp list
   | TCppStar of tcpp * bool
   | TCppVoidStar
   | TCppVarArg
@@ -78,12 +90,19 @@ type tcpp =
 
 and tcppexpr = { cppexpr : tcpp_expr_expr; cpptype : tcpp; cpppos : pos }
 
+and tcppvar = {
+  tcppv_type : tcpp;
+  tcppv_var : tvar;
+  tcppv_name : string;
+  tcppv_debug_name : string;
+}
+
 and tcpp_closure = {
   close_type : tcpp;
-  close_args : (tvar * texpr option) list;
+  close_args : (tcppvar * texpr option) list;
   close_expr : tcppexpr;
   close_id : int;
-  close_undeclared : tvar StringMap.t;
+  close_undeclared : tcppvar IntMap.t;
   close_this : tcppthis option;
 }
 
@@ -92,8 +111,8 @@ and tcppunop = CppNeg | CppNegBits | CppNot
 and tcppthis = ThisReal | ThisFake | ThisDynamic
 
 and tcppvarloc =
-  | VarLocal of tvar
-  | VarClosure of tvar
+  | VarLocal of tcppvar
+  | VarClosure of tcppvar
   | VarThis of tclass_field * tcpp
   | VarInstance of tcppexpr * tclass_field * string * string
   | VarInterface of tcppexpr * tclass_field
@@ -104,8 +123,8 @@ and tcppinst = InstPtr | InstObjC | InstStruct
 
 and tcppfuncloc =
   | FuncThis of tclass_field * tcpp
-  | FuncInstance of tcppexpr * tcppinst * tclass_field
-  | FuncStatic of tclass * bool * tclass_field
+  | FuncInstance of tcppexpr * tcppinst * tclass_field * tcpp list
+  | FuncStatic of tclass * bool * tclass_field * tcpp list
   | FuncTemplate of tclass * tclass_field * path * bool
   | FuncInterface of tcppexpr * tclass * tclass_field
   | FuncEnumConstruct of tenum * tenum_field
@@ -166,15 +185,15 @@ and tcpp_expr_expr =
   | CppPosition of string * int32 * string * string
   | CppArrayDecl of tcppexpr list
   | CppUnop of tcppunop * tcppexpr
-  | CppVarDecl of tvar * tcppexpr option
+  | CppVarDecl of tcppvar * tcppexpr option
   | CppBlock of tcppexpr list * tcpp_closure list * bool
-  | CppFor of tvar * tcppexpr * tcppexpr
+  | CppFor of tcppvar * tcppexpr * tcppexpr
   | CppIf of tcppexpr * tcppexpr * tcppexpr option
   | CppWhile of tcppexpr * tcppexpr * Ast.while_flag * int
   | CppIntSwitch of tcppexpr * (Int32.t list * tcppexpr) list * tcppexpr option
   | CppSwitch of
       tcppexpr * tcpp * (tcppexpr list * tcppexpr) list * tcppexpr option * int
-  | CppTry of tcppexpr * (tvar * tcppexpr) list
+  | CppTry of tcppexpr * (tcppvar * tcppexpr) list
   | CppBreak
   | CppContinue
   | CppClassOf of path * bool
@@ -211,6 +230,8 @@ and tcpp_class_function = {
   tcf_field : tclass_field;
   tcf_name : string;
   tcf_func : tfunc;
+  tcf_args : (tcppvar * texpr option) list;
+  tcf_return : tcpp;
 
   tcf_is_virtual : bool;
   tcf_is_reflective : bool;
@@ -240,6 +261,7 @@ and tcpp_class = {
   tcl_debug_level : int;
   tcl_super : tcpp_class option;
   tcl_container : tcpp_class_container option;
+  tcl_constructor : tcpp_class_function option;
 
   tcl_haxe_interfaces : tcpp_interface list;
   tcl_native_interfaces : tcpp_interface list;
@@ -262,8 +284,8 @@ and tcpp_class = {
 and tcpp_interface_function = {
   iff_field : tclass_field;
   iff_name : string;
-  iff_args : (string * bool * t) list;
-  iff_return : t;
+  iff_args : tcpp_tfun_arg list;
+  iff_return : tcpp;
   iff_script_slot : int option;
 }
 
@@ -280,9 +302,16 @@ and tcpp_interface = {
   if_scriptable : bool;
 }
 
+and tcpp_tfun_arg = {
+  tfa_name : string;
+  tfa_type : tcpp;
+  tfa_optional : bool;
+}
+
 and tcpp_enum_field = {
   tef_field : tenum_field;
   tef_name : string;
+  tef_args : tcpp_tfun_arg list option;
   tef_hash : string;
 }