Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[gui] Metal GGUI support & run metal ggui tests #8343

Merged
merged 28 commits into from
Sep 3, 2023

Conversation

bobcao3
Copy link
Collaborator

@bobcao3 bobcao3 commented Aug 31, 2023

Issue: #

Brief Summary

Implemented by @AntonioFerreras

Walkthrough

🤖 Generated by Copilot at 4d41154

  • Remove unnecessary dependencies and linking of taichi_ui and taichi_ui_vulkan to taichi_core, avoiding circular linking issues and errors (link, link, link, link, link)
  • Add a new member variable ggui_arch to AppConfig and PyWindow, storing the desired RHI backend for GGUI, either vulkan or metal, based on the user input or the ti_arch parameter (link, link, link)
  • Add a new pure virtual method prepare_for_next_frame to GuiBase, which is needed for the metal backend of GGUI to prepare the drawable and the render pass descriptor before rendering the next frame (link)
  • Rewrite the imread function in image.py, using the new _ti_core.imfree function to free the raw pointer returned by _ti_core.imread, and using ctypes to create a numpy array from the pointer without using C_memcpy, which is removed (link, link, link)
  • Modify the conversion of floating point images to uint8 images in cook_image_to_bytes, removing the unnecessary addition of 0.5, which causes rounding errors and inconsistent results (link)
  • Remove the unused header files common_metal.h and simd/simd.h from cpp_examples/rhi_examples, which are not part of the RHI interface and cause compilation errors (link, link)
  • Fix a typo in the comment of the SampleApp class in sample_3_textured_triangle.cpp, changing "voronoi" to "worley", which is the correct name of the noise function (link)
  • Modify the arch parameter for the taichi.init function in mass_spring_game_ggui.py and mpm3d_ggui.py, using metal as the preferred backend for GGUI on macOS, and ti.gpu as the generic value for the best GPU backend available (link, link)
  • Modify the include list in export_ggui.cpp, replacing the vulkan subdirectory with the ggui subdirectory, which contains the new unified interface for GGUI that supports both vulkan and metal backends (link)
  • Add the GLFW_EXPOSE_NATIVE_COCOA macro and the QuartzCore/QuartzCore.h header file to window_system.h, which are needed to access the native window handle and the CAMetalLayer class for the metal backend on macOS (link)
  • Add a new helper class NonAssignable in impl_support.h, which deletes the copy constructor and the copy assignment operator of any derived class, preventing accidental copies of non-copyable resources such as metal objects (link)
  • Make various metal structs and classes inherit from the NonAssignable helper class, deleting their copy constructors and copy assignment operators, and preventing accidental copies of the metal objects (link, link, link, link)
  • Explicitly delete the copy constructor of the MetalSampler struct, which is redundant after inheriting from the NonAssignable helper class, but added for clarity and consistency (link)
  • Add default constructors to the MetalRasterLibraries and MetalRasterFunctions structs, initializing the metal library and function members to nil, which avoids potential undefined behavior when using the structs without explicit initialization (link, link, link)
  • Modify the constructor of the MetalPipeline class, changing the parameters of the raster libraries, functions, and params from const references to non-const references, and using std::move to transfer ownership of the metal objects to the pipeline object, avoiding unnecessary retain and release calls (link, link)
  • Remove the destroy method and the is_destroyed_ member variable from the MetalPipeline class, which are no longer needed after using std::move to transfer ownership of the metal objects, and move the release logic to the destructor of the class, which is more idiomatic and safe (link, link, link)
  • Modify the raster_params_ member variable of the MetalPipeline class, changing it from a const reference to a value, which avoids potential dangling references when the original parameter goes out of scope, and makes the class more consistent with the other member variables (link)
  • Add two new methods to the MetalCommandList class, create_render_pass_desc and is_renderpass_active, which are needed for the new metal implementation of GGUI that supports resuming the current render pass without clearing the attachments, and setting the appropriate load actions for the render pass descriptor (link, link, link, link)
  • Add a new method set_renderpass_active to the MetalCommandList class, which sets the value of the is_renderpass_active_ member variable to true, indicating that a render pass has been started (link)
  • Rewrite the begin_renderpass method of the MetalCommandList class, removing the redundant assignments of the current viewport, and using the metal image objects directly to get the render target textures and formats, instead of querying the device with the allocation ids. This change also flips the framebuffer Y coordinate to match the metal coordinate system, and stores the render target height for later use (link)
  • Modify the pre_draw_setup method of the MetalPipeline class, removing the conditional check for the depth attachment format, and always setting the depth attachment pixel format to the one stored in the current render pass details, which simplifies the logic and avoids potential errors when the format is unknown (link)
  • Modify the pre_draw_setup method of the MetalCommandList class, calling the new create_render_pass_desc method to get the render pass descriptor, and passing the is_renderpass_active_ flag to it, to determine the load actions for the render pass. This change also sets the depth stencil state for the render command encoder, based on the depth test and depth write flags of the raster params, and sets the is_renderpass_active_ flag to true after starting the render pass (link)
  • Remove the get_depth_data and get_image_data methods from the RHI_DLL_EXPORT, VulkanSurface, and MetalSurface classes, which are no longer part of the RHI interface and are replaced by the new screenshot and depth_screenshot methods in the Window class (link, link, link, link)
  • Remove the depth_buffer_ and screenshot_buffer_ member variables from the VulkanSurface class, which are no longer needed after removing the get_depth_data and get_image_data methods, and simplify the logic of the class (link)
  • Modify the TAICHI_IMGUI_SOURCE list in CMakeLists.txt, adding a new conditional branch for macOS, and appending the imgui_impl_metal.mm source file to the list, which is needed for the metal backend of ImGUI. This change also adds a new compile option to enable automatic reference counting for Objective-C objects, which is required by the metal backend of ImGUI (link)
  • Delete the file scene.cpp, which is no longer needed after the new unified interface for GGUI that supports both vulkan and metal backends (link)

bobcao3 and others added 13 commits August 29, 2023 12:25
Issue: #

<!--
copilot:summary
-->

This pull request updates the RHI and GGUI modules to support both
Vulkan and Metal backends for graphics and user interface
functionalities. It adds new examples, shaders, and headers for the RHI
module, and replaces the `taichi_ui_vulkan` library with the
`taichi_ui_ggui` library. It also fixes some bugs, removes some unused
code, and improves the code formatting and compatibility for the RHI and
GGUI modules.

<!--
copilot:walkthrough
-->

* Replace `taichi_ui_vulkan` library with `taichi_ui_ggui` library to
use the new GGUI library that supports both Vulkan and Metal backends
for the Taichi UI module
([link](https://github.com/taichi-dev/taichi/pull/8338/files?diff=unified&w=0#diff-ddbed628f91a9ee8bb2f6af3cb6f4b793746b0cc8182bcd4513b6793e816f347L386-R386))
* Add conditional include of `common_metal.h` or `common_vulkan.h`
depending on the CMake system name to use the appropriate header file
for the Metal or Vulkan backend for the RHI examples
([link](https://github.com/taichi-dev/taichi/pull/8338/files?diff=unified&w=0#diff-ce18c0b2ae4fee335806357f23ceb248ed7b32ab121fdd4f750e5856bfc65328L5-R9))
* Add a new example `sample_3_textured_triangle` to the CMake file to
include the new example in the build process
([link](https://github.com/taichi-dev/taichi/pull/8338/files?diff=unified&w=0#diff-ce18c0b2ae4fee335806357f23ceb248ed7b32ab121fdd4f750e5856bfc65328L28-R39))
* Add new header and source files `common_metal.h` and `common_metal.mm`
to define and implement some common types and utilities for the Metal
backend for the RHI examples
([link](https://github.com/taichi-dev/taichi/pull/8338/files?diff=unified&w=0#diff-c2200488f0fb2fc5f9b2d852cdf9dab73f8a03911dcfa0e854d6f97b43c1cbefR1-R30),
[link](https://github.com/taichi-dev/taichi/pull/8338/files?diff=unified&w=0#diff-b8acd5141334e03814080758436245c5f2e18e8c35b507214d37e91d8f50133eR1-R77))
* Modify the include statements in `sample_1_window.cpp` and
`sample_2_triangle.cpp` to use the `common_vulkan.h` and
`common_metal.h` header files instead of the `common.h` header file,
which are more specific to the Vulkan and Metal backends respectively
([link](https://github.com/taichi-dev/taichi/pull/8338/files?diff=unified&w=0#diff-f5e80908e1912664728078f2b0a50511d4c807b2738e12bca6e5a210ce609a2eL1-R2),
[link](https://github.com/taichi-dev/taichi/pull/8338/files?diff=unified&w=0#diff-49805b429c6c67e22573214b30ccc9657d8caa0348082e56427a85318d672d1bL1-R2))
* Add new source, header, and shader files
`sample_3_textured_triangle.cpp`, `3_triangle.frag`,
`3_triangle.frag.spv.h`, `3_triangle.vert`, and `3_triangle.vert.spv.h`
to implement a new example that shows how to use textures with the RHI
module
([link](https://github.com/taichi-dev/taichi/pull/8338/files?diff=unified&w=0#diff-928a01d05781b44e3a14ade72e1cded1397fcb6296e5b032230f450f04c477dbR1-R277),
[link](https://github.com/taichi-dev/taichi/pull/8338/files?diff=unified&w=0#diff-e4ebde697c4962b56a040f309a93e1564b13d3e994895aaf9404811a147d3f41R1-R12),
[link](https://github.com/taichi-dev/taichi/pull/8338/files?diff=unified&w=0#diff-bbfd5e1adf18b3fb417282df1b87c5d83a3d750e9b9f0a757e146d2c656e5584R1-R41),
[link](https://github.com/taichi-dev/taichi/pull/8338/files?diff=unified&w=0#diff-bb661ff55d0c5f1c2cfe41f62b9d15f75524d01b72dc9e087532d5bc823a3875R1-R19),
[link](https://github.com/taichi-dev/taichi/pull/8338/files?diff=unified&w=0#diff-1b18019e7fad40aa42466d3798b8e1d43c6403063bb8a8b893b27412fbc89fe9R1-R63))
* Modify the SPIRV binary data in `2_triangle.frag.spv.h` to update the
SPIRV version from 0x000d000a to 0x000d000b, which is required by the
Metal backend
([link](https://github.com/taichi-dev/taichi/pull/8338/files?diff=unified&w=0#diff-00e93cf0c366583599b5939d339f4b54098426859fc59ef4d2761cb76cfa4534L2-R2))
* Modify the `ti.init` call in `mass_spring_game_ggui.py` to use the
Metal backend instead of the Vulkan backend for the GGUI module
([link](https://github.com/taichi-dev/taichi/pull/8338/files?diff=unified&w=0#diff-e2756c236761fac4f823a8b6c11470369599a6189de1b8d1bb24edee2406b4c5L3-R4))
* Replace the `vulkan` namespace with the `ggui` namespace in the
include statements in `export_ggui.cpp` to use the new GGUI library that
supports both Vulkan and Metal backends for the Taichi UI module
([link](https://github.com/taichi-dev/taichi/pull/8338/files?diff=unified&w=0#diff-de0a9a1fb6c6210ac5653025b79a982b6721b1c66e17d2dca1db90fa82613bddL16-R20))
* Add code to determine the GGUI backend based on the Taichi backend and
to verify that Vulkan is available if the GGUI backend is Vulkan in the
`PyWindow` struct in `export_ggui.cpp`
([link](https://github.com/taichi-dev/taichi/pull/8338/files?diff=unified&w=0#diff-de0a9a1fb6c6210ac5653025b79a982b6721b1c66e17d2dca1db90fa82613bddR524-R536))
* Modify the `WindowConfig` constructor call in the `PyWindow` struct in
`export_ggui.cpp` to pass the `ggui_arch` variable as an argument and to
remove the redundant check for Vulkan availability
([link](https://github.com/taichi-dev/taichi/pull/8338/files?diff=unified&w=0#diff-de0a9a1fb6c6210ac5653025b79a982b6721b1c66e17d2dca1db90fa82613bddL533-R548))
* Add code to expose the native Cocoa window handle in `window_system.h`
to enable the Metal backend to access the native window handle for
creating a Metal layer
([link](https://github.com/taichi-dev/taichi/pull/8338/files?diff=unified&w=0#diff-c0447b54db4ef2688c36b301c669203c6276ae622a4950271188e449b2c1af4bR5-R6))
* Add new Metal types to `metal_device.h` to define some Metal types
that are used by the Metal backend for the RHI module
([link](https://github.com/taichi-dev/taichi/pull/8338/files?diff=unified&w=0#diff-5b304e188996abd217fad85fd8b2434e729ad9c089eb9ef48a848c0fcc74d614L38-R64))
* Add a new method declaration to the `MetalCommandList` class in
`metal_device.h` to declare a new method that creates a Metal render
pass descriptor
([link](https://github.com/taichi-dev/taichi/pull/8338/files?diff=unified&w=0#diff-5b304e188996abd217fad85fd8b2434e729ad9c089eb9ef48a848c0fcc74d614R392))
* Remove some method declarations from the `MetalSurface` and
`RHI_DLL_EXPORT` classes in `metal_device.h` and `public_device.h` to
remove some methods that are not implemented or needed by the Metal and
RHI modules
([link](https://github.com/taichi-dev/taichi/pull/8338/files?diff=unified&w=0#diff-5b304e188996abd217fad85fd8b2434e729ad9c089eb9ef48a848c0fcc74d614L443-L449),
[link](https://github.com/taichi-dev/taichi/pull/8338/files?diff=unified&w=0#diff-0290f4565dcad1ce884dc8e6377a55a36bdf13bc3914c2e9b9da039cd22dcdf1L889-L892))
* Modify the order of some statements in the `pre_draw_setup` method of
the `MetalCommandList` class in `metal_device.mm` to avoid accessing the
pixel format of a null texture pointer
([link](https://github.com/taichi-dev/taichi/pull/8338/files?diff=unified&w=0#diff-c350a27659df1ebcf2c854e2c21a80f82de87d3b325a80e00f6a6dd6ca53303bL538-R540))
* Split the `pre_draw_setup` method of the `MetalCommandList` class in
`metal_device.mm` into two methods to separate the logic of creating a
Metal render pass descriptor from the logic of setting up the Metal
render command encoder and the render pipeline state
([link](https://github.com/taichi-dev/taichi/pull/8338/files?diff=unified&w=0#diff-c350a27659df1ebcf2c854e2c21a80f82de87d3b325a80e00f6a6dd6ca53303bL614-R615))
* Modify the parameter of the `create_render_pass_desc` method of the
`MetalCommandList` class in `metal_device.mm` to use the `depth_write`
parameter instead of the `raster_params` parameter, which is more
specific and consistent with the method name
([link](https://github.com/taichi-dev/taichi/pull/8338/files?diff=unified&w=0#diff-c350a27659df1ebcf2c854e2c21a80f82de87d3b325a80e00f6a6dd6ca53303bL630-R630))
* Add code to release the Metal render pass descriptor and to return it
from the `create_render_pass_desc` method of the `MetalCommandList`
class in `metal_device.mm` to avoid memory leaks and to return the
created Metal render pass descriptor to the caller
([link](https://github.com/taichi-dev/taichi/pull/8338/files?diff=unified&w=0#diff-c350a27659df1ebcf2c854e2c21a80f82de87d3b325a80e00f6a6dd6ca53303bL639-R651))
* Add code to set the Metal depth stencil state in the `pre_draw_setup`
method of the `MetalCommandList` class in `metal_device.mm` to enable
depth testing and depth writing for the Metal backend
([link](https://github.com/taichi-dev/taichi/pull/8338/files?diff=unified&w=0#diff-c350a27659df1ebcf2c854e2c21a80f82de87d3b325a80e00f6a6dd6ca53303bR669-R676))
* Remove some method definitions from the `VulkanSurface` class in
`vulkan_device.cpp` to remove some methods that are not implemented or
needed by the Vulkan backend
([link](https://github.com/taichi-dev/taichi/pull/8338/files?diff=unified&w=0#diff-4479f242ce99cd434c015531997155aa7faa38fb3ead6b0997caf097b05cecb8L2850-L2914))
* Add a newline character in the `MAKE_ENUM_FLAGS` macro in
`public_device.h` to improve the code formatting and readability
([link](https://github.com/taichi-dev/taichi/pull/8338/files?diff=unified&w=0#diff-0290f4565dcad1ce884dc8e6377a55a36bdf13bc3914c2e9b9da039cd22dcdf1L58-R60))

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Bob Cao <[email protected]>
@netlify
Copy link

netlify bot commented Aug 31, 2023

Deploy Preview for docsite-preview ready!

Name Link
🔨 Latest commit 69b7f47
🔍 Latest deploy log https://app.netlify.com/sites/docsite-preview/deploys/64f2a2a17c767d0008ee4114
😎 Deploy Preview https://deploy-preview-8343--docsite-preview.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

@bobcao3 bobcao3 changed the title [ui] Metal GGUI support & run metal ggui tests [gui] Metal GGUI support & run metal ggui tests Aug 31, 2023
@AntonioFerreras AntonioFerreras merged commit aa0619f into master Sep 3, 2023
25 checks passed
AntonioFerreras added a commit that referenced this pull request Sep 3, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: Done
Development

Successfully merging this pull request may close these issues.

2 participants