[gui] Metal GGUI support & run metal ggui tests (#8343)

Issue: #

### Brief Summary

Implemented by @AntonioFerreras 

### Walkthrough

<!--
copilot:walkthrough
-->
### <samp>🤖 Generated by Copilot at 4d41154</samp>

* Remove unnecessary dependencies and linking of taichi_ui and
taichi_ui_vulkan to taichi_core, avoiding circular linking issues and
errors
([link](https://github.com/taichi-dev/taichi/pull/8343/files?diff=unified&w=0#diff-ddbed628f91a9ee8bb2f6af3cb6f4b793746b0cc8182bcd4513b6793e816f347L352),
[link](https://github.com/taichi-dev/taichi/pull/8343/files?diff=unified&w=0#diff-ddbed628f91a9ee8bb2f6af3cb6f4b793746b0cc8182bcd4513b6793e816f347L386),
[link](https://github.com/taichi-dev/taichi/pull/8343/files?diff=unified&w=0#diff-e33fc28fc67040566898b7816ffdf7a8fcd04f0613c7bfb58d4f1ea176d10921L3-R3),
[link](https://github.com/taichi-dev/taichi/pull/8343/files?diff=unified&w=0#diff-e33fc28fc67040566898b7816ffdf7a8fcd04f0613c7bfb58d4f1ea176d10921L27),
[link](https://github.com/taichi-dev/taichi/pull/8343/files?diff=unified&w=0#diff-e33fc28fc67040566898b7816ffdf7a8fcd04f0613c7bfb58d4f1ea176d10921L58-R66))
* Add a new member variable ggui_arch to AppConfig and PyWindow, storing
the desired RHI backend for GGUI, either vulkan or metal, based on the
user input or the ti_arch parameter
([link](https://github.com/taichi-dev/taichi/pull/8343/files?diff=unified&w=0#diff-de0a9a1fb6c6210ac5653025b79a982b6721b1c66e17d2dca1db90fa82613bddR524-R536),
[link](https://github.com/taichi-dev/taichi/pull/8343/files?diff=unified&w=0#diff-de0a9a1fb6c6210ac5653025b79a982b6721b1c66e17d2dca1db90fa82613bddL533-R548),
[link](https://github.com/taichi-dev/taichi/pull/8343/files?diff=unified&w=0#diff-e7752d866d98da134724e52b83c9b0e5221ed8ae8b1119a4cf28ab285d67ff5dR21))
* Add a new pure virtual method prepare_for_next_frame to GuiBase, which
is needed for the metal backend of GGUI to prepare the drawable and the
render pass descriptor before rendering the next frame
([link](https://github.com/taichi-dev/taichi/pull/8343/files?diff=unified&w=0#diff-348ff7237682de53727d98a47ac4f414a1937b18c1f1639606003c43382e21b8R29))
* Rewrite the imread function in image.py, using the new _ti_core.imfree
function to free the raw pointer returned by _ti_core.imread, and using
ctypes to create a numpy array from the pointer without using C_memcpy,
which is removed
([link](https://github.com/taichi-dev/taichi/pull/8343/files?diff=unified&w=0#diff-2bee848f6799bd6db46adca1834b0f4e6e976014eb4a579afb0f739b91e8cbe9R6),
[link](https://github.com/taichi-dev/taichi/pull/8343/files?diff=unified&w=0#diff-2bee848f6799bd6db46adca1834b0f4e6e976014eb4a579afb0f739b91e8cbe9L84-R88),
[link](https://github.com/taichi-dev/taichi/pull/8343/files?diff=unified&w=0#diff-153c1d3856e2e55553451efe0eab141bb2ef3bfc2a94158049bd6d629968051bL113-R113))
* Modify the conversion of floating point images to uint8 images in
cook_image_to_bytes, removing the unnecessary addition of 0.5, which
causes rounding errors and inconsistent results
([link](https://github.com/taichi-dev/taichi/pull/8343/files?diff=unified&w=0#diff-2bee848f6799bd6db46adca1834b0f4e6e976014eb4a579afb0f739b91e8cbe9L19-R20))
* Remove the unused header files common_metal.h and simd/simd.h from
cpp_examples/rhi_examples, which are not part of the RHI interface and
cause compilation errors
([link](https://github.com/taichi-dev/taichi/pull/8343/files?diff=unified&w=0#diff-c2200488f0fb2fc5f9b2d852cdf9dab73f8a03911dcfa0e854d6f97b43c1cbefL9-L31),
[link](https://github.com/taichi-dev/taichi/pull/8343/files?diff=unified&w=0#diff-b8acd5141334e03814080758436245c5f2e18e8c35b507214d37e91d8f50133eL4))
* Fix a typo in the comment of the SampleApp class in
sample_3_textured_triangle.cpp, changing "voronoi" to "worley", which is
the correct name of the noise function
([link](https://github.com/taichi-dev/taichi/pull/8343/files?diff=unified&w=0#diff-928a01d05781b44e3a14ade72e1cded1397fcb6296e5b032230f450f04c477dbL107-R107))
* Modify the arch parameter for the taichi.init function in
mass_spring_game_ggui.py and mpm3d_ggui.py, using metal as the preferred
backend for GGUI on macOS, and ti.gpu as the generic value for the best
GPU backend available
([link](https://github.com/taichi-dev/taichi/pull/8343/files?diff=unified&w=0#diff-e2756c236761fac4f823a8b6c11470369599a6189de1b8d1bb24edee2406b4c5L3-R4),
[link](https://github.com/taichi-dev/taichi/pull/8343/files?diff=unified&w=0#diff-c180380281cbad6c17bf08be1986c785e0ac20b22f96f35eeda90091498a0642L5-R5))
* Modify the include list in export_ggui.cpp, replacing the vulkan
subdirectory with the ggui subdirectory, which contains the new unified
interface for GGUI that supports both vulkan and metal backends
([link](https://github.com/taichi-dev/taichi/pull/8343/files?diff=unified&w=0#diff-de0a9a1fb6c6210ac5653025b79a982b6721b1c66e17d2dca1db90fa82613bddL16-R20))
* Add the GLFW_EXPOSE_NATIVE_COCOA macro and the QuartzCore/QuartzCore.h
header file to window_system.h, which are needed to access the native
window handle and the CAMetalLayer class for the metal backend on macOS
([link](https://github.com/taichi-dev/taichi/pull/8343/files?diff=unified&w=0#diff-c0447b54db4ef2688c36b301c669203c6276ae622a4950271188e449b2c1af4bR5-R6))
* Add a new helper class NonAssignable in impl_support.h, which deletes
the copy constructor and the copy assignment operator of any derived
class, preventing accidental copies of non-copyable resources such as
metal objects
([link](https://github.com/taichi-dev/taichi/pull/8343/files?diff=unified&w=0#diff-20ba241db159f1497f5fc05dbe871118791a0d7f3ca87a933a772fad6a0466e7R182-R190))
* Make various metal structs and classes inherit from the NonAssignable
helper class, deleting their copy constructors and copy assignment
operators, and preventing accidental copies of the metal objects
([link](https://github.com/taichi-dev/taichi/pull/8343/files?diff=unified&w=0#diff-5b304e188996abd217fad85fd8b2434e729ad9c089eb9ef48a848c0fcc74d614L69-R76),
[link](https://github.com/taichi-dev/taichi/pull/8343/files?diff=unified&w=0#diff-5b304e188996abd217fad85fd8b2434e729ad9c089eb9ef48a848c0fcc74d614L87-R94),
[link](https://github.com/taichi-dev/taichi/pull/8343/files?diff=unified&w=0#diff-5b304e188996abd217fad85fd8b2434e729ad9c089eb9ef48a848c0fcc74d614L102-R109),
[link](https://github.com/taichi-dev/taichi/pull/8343/files?diff=unified&w=0#diff-5b304e188996abd217fad85fd8b2434e729ad9c089eb9ef48a848c0fcc74d614L168-R187))
* Explicitly delete the copy constructor of the MetalSampler struct,
which is redundant after inheriting from the NonAssignable helper class,
but added for clarity and consistency
([link](https://github.com/taichi-dev/taichi/pull/8343/files?diff=unified&w=0#diff-5b304e188996abd217fad85fd8b2434e729ad9c089eb9ef48a848c0fcc74d614R115-R117))
* Add default constructors to the MetalRasterLibraries and
MetalRasterFunctions structs, initializing the metal library and
function members to nil, which avoids potential undefined behavior when
using the structs without explicit initialization
([link](https://github.com/taichi-dev/taichi/pull/8343/files?diff=unified&w=0#diff-5b304e188996abd217fad85fd8b2434e729ad9c089eb9ef48a848c0fcc74d614L145-R159),
[link](https://github.com/taichi-dev/taichi/pull/8343/files?diff=unified&w=0#diff-5b304e188996abd217fad85fd8b2434e729ad9c089eb9ef48a848c0fcc74d614L151-R168),
[link](https://github.com/taichi-dev/taichi/pull/8343/files?diff=unified&w=0#diff-c350a27659df1ebcf2c854e2c21a80f82de87d3b325a80e00f6a6dd6ca53303bL56-R66))
* Modify the constructor of the MetalPipeline class, changing the
parameters of the raster libraries, functions, and params from const
references to non-const references, and using std::move to transfer
ownership of the metal objects to the pipeline object, avoiding
unnecessary retain and release calls
([link](https://github.com/taichi-dev/taichi/pull/8343/files?diff=unified&w=0#diff-5b304e188996abd217fad85fd8b2434e729ad9c089eb9ef48a848c0fcc74d614L179-R202),
[link](https://github.com/taichi-dev/taichi/pull/8343/files?diff=unified&w=0#diff-c350a27659df1ebcf2c854e2c21a80f82de87d3b325a80e00f6a6dd6ca53303bL74-R109))
* Remove the destroy method and the is_destroyed_ member variable from
the MetalPipeline class, which are no longer needed after using
std::move to transfer ownership of the metal objects, and move the
release logic to the destructor of the class, which is more idiomatic
and safe
([link](https://github.com/taichi-dev/taichi/pull/8343/files?diff=unified&w=0#diff-5b304e188996abd217fad85fd8b2434e729ad9c089eb9ef48a848c0fcc74d614L193),
[link](https://github.com/taichi-dev/taichi/pull/8343/files?diff=unified&w=0#diff-5b304e188996abd217fad85fd8b2434e729ad9c089eb9ef48a848c0fcc74d614L221-L222),
[link](https://github.com/taichi-dev/taichi/pull/8343/files?diff=unified&w=0#diff-c350a27659df1ebcf2c854e2c21a80f82de87d3b325a80e00f6a6dd6ca53303bL220-L234))
* Modify the raster_params_ member variable of the MetalPipeline class,
changing it from a const reference to a value, which avoids potential
dangling references when the original parameter goes out of scope, and
makes the class more consistent with the other member variables
([link](https://github.com/taichi-dev/taichi/pull/8343/files?diff=unified&w=0#diff-5b304e188996abd217fad85fd8b2434e729ad9c089eb9ef48a848c0fcc74d614L234-R250))
* Add two new methods to the MetalCommandList class,
create_render_pass_desc and is_renderpass_active, which are needed for
the new metal implementation of GGUI that supports resuming the current
render pass without clearing the attachments, and setting the
appropriate load actions for the render pass descriptor
([link](https://github.com/taichi-dev/taichi/pull/8343/files?diff=unified&w=0#diff-5b304e188996abd217fad85fd8b2434e729ad9c089eb9ef48a848c0fcc74d614R401-R406),
[link](https://github.com/taichi-dev/taichi/pull/8343/files?diff=unified&w=0#diff-5b304e188996abd217fad85fd8b2434e729ad9c089eb9ef48a848c0fcc74d614L402-R430),
[link](https://github.com/taichi-dev/taichi/pull/8343/files?diff=unified&w=0#diff-c350a27659df1ebcf2c854e2c21a80f82de87d3b325a80e00f6a6dd6ca53303bL614-R691),
[link](https://github.com/taichi-dev/taichi/pull/8343/files?diff=unified&w=0#diff-c350a27659df1ebcf2c854e2c21a80f82de87d3b325a80e00f6a6dd6ca53303bR709-R718))
* Add a new method set_renderpass_active to the MetalCommandList class,
which sets the value of the is_renderpass_active_ member variable to
true, indicating that a render pass has been started
([link](https://github.com/taichi-dev/taichi/pull/8343/files?diff=unified&w=0#diff-c350a27659df1ebcf2c854e2c21a80f82de87d3b325a80e00f6a6dd6ca53303bR734-R735))
* Rewrite the begin_renderpass method of the MetalCommandList class,
removing the redundant assignments of the current viewport, and using
the metal image objects directly to get the render target textures and
formats, instead of querying the device with the allocation ids. This
change also flips the framebuffer Y coordinate to match the metal
coordinate system, and stores the render target height for later use
([link](https://github.com/taichi-dev/taichi/pull/8343/files?diff=unified&w=0#diff-c350a27659df1ebcf2c854e2c21a80f82de87d3b325a80e00f6a6dd6ca53303bL527-R584))
* Modify the pre_draw_setup method of the MetalPipeline class, removing
the conditional check for the depth attachment format, and always
setting the depth attachment pixel format to the one stored in the
current render pass details, which simplifies the logic and avoids
potential errors when the format is unknown
([link](https://github.com/taichi-dev/taichi/pull/8343/files?diff=unified&w=0#diff-c350a27659df1ebcf2c854e2c21a80f82de87d3b325a80e00f6a6dd6ca53303bL172-R198))
* Modify the pre_draw_setup method of the MetalCommandList class,
calling the new create_render_pass_desc method to get the render pass
descriptor, and passing the is_renderpass_active_ flag to it, to
determine the load actions for the render pass. This change also sets
the depth stencil state for the render command encoder, based on the
depth test and depth write flags of the raster params, and sets the
is_renderpass_active_ flag to true after starting the render pass
([link](https://github.com/taichi-dev/taichi/pull/8343/files?diff=unified&w=0#diff-c350a27659df1ebcf2c854e2c21a80f82de87d3b325a80e00f6a6dd6ca53303bL1357-R1426))
* Remove the get_depth_data and get_image_data methods from the
RHI_DLL_EXPORT, VulkanSurface, and MetalSurface classes, which are no
longer part of the RHI interface and are replaced by the new screenshot
and depth_screenshot methods in the Window class
([link](https://github.com/taichi-dev/taichi/pull/8343/files?diff=unified&w=0#diff-0290f4565dcad1ce884dc8e6377a55a36bdf13bc3914c2e9b9da039cd22dcdf1L889-L892),
[link](https://github.com/taichi-dev/taichi/pull/8343/files?diff=unified&w=0#diff-4479f242ce99cd434c015531997155aa7faa38fb3ead6b0997caf097b05cecb8L2850-L2914),
[link](https://github.com/taichi-dev/taichi/pull/8343/files?diff=unified&w=0#diff-36ad983c49527946c94e9e0e8334d9a7de3e50f66d88f7611e5458168ff6b703L493-L495),
[link](https://github.com/taichi-dev/taichi/pull/8343/files?diff=unified&w=0#diff-5b304e188996abd217fad85fd8b2434e729ad9c089eb9ef48a848c0fcc74d614L457-L463))
* Remove the depth_buffer_ and screenshot_buffer_ member variables from
the VulkanSurface class, which are no longer needed after removing the
get_depth_data and get_image_data methods, and simplify the logic of the
class
([link](https://github.com/taichi-dev/taichi/pull/8343/files?diff=unified&w=0#diff-36ad983c49527946c94e9e0e8334d9a7de3e50f66d88f7611e5458168ff6b703L514-L517))
* Modify the TAICHI_IMGUI_SOURCE list in CMakeLists.txt, adding a new
conditional branch for macOS, and appending the imgui_impl_metal.mm
source file to the list, which is needed for the metal backend of ImGUI.
This change also adds a new compile option to enable automatic reference
counting for Objective-C objects, which is required by the metal backend
of ImGUI
([link](https://github.com/taichi-dev/taichi/pull/8343/files?diff=unified&w=0#diff-e33fc28fc67040566898b7816ffdf7a8fcd04f0613c7bfb58d4f1ea176d10921L45-R54))
* Delete the file scene.cpp, which is no longer needed after the new
unified interface for GGUI that supports both vulkan and metal backends
([link](https://github.com/taichi-dev/taichi/pull/8343/files?diff=unified&w=0#diff-e6eb99341e80689aaa3a2cc10977828d021719b47be26427220ba0fd74c7de91))

---------

Co-authored-by: Antonio Ferreras <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

cmake/TaichiCore.cmake

@@ -349,7 +349,6 @@ if(TI_WITH_PYTHON)
     # This requires refactoring on the python/export_*.cpp as well as better
     # error message on the Python side.
     add_subdirectory(taichi/ui)
-    target_link_libraries(taichi_ui PUBLIC ${CORE_LIBRARY_NAME})
 
     message("PYTHON_LIBRARIES: " ${PYTHON_LIBRARIES})
     set(CORE_WITH_PYBIND_LIBRARY_NAME taichi_python)
@@ -383,7 +382,6 @@ if(TI_WITH_PYTHON)
 
     if(TI_WITH_GGUI)
         target_compile_definitions(${CORE_WITH_PYBIND_LIBRARY_NAME} PRIVATE -DTI_WITH_GGUI)
-        target_link_libraries(${CORE_WITH_PYBIND_LIBRARY_NAME} PRIVATE taichi_ui_vulkan)
     endif()
 
     target_link_libraries(${CORE_WITH_PYBIND_LIBRARY_NAME} PRIVATE taichi_ui)

cpp_examples/rhi_examples/CMakeLists.txt

@@ -1,4 +1,4 @@
-macro(make_sample executable_name src_file)
+macro(make_sample executable_name src_file flags)
 add_executable(${executable_name})
 set_property(TARGET ${executable_name} PROPERTY CXX_STANDARD 17)
 set_property(TARGET ${executable_name} PROPERTY C_STANDARD 17)
@@ -32,8 +32,15 @@ target_link_libraries(${executable_name} taichi_core glfw)
 if (${CMAKE_SYSTEM_NAME} MATCHES "Darwin")
   target_link_libraries(${executable_name} "-framework QuartzCore")
 endif()
+target_compile_definitions(${executable_name} PRIVATE ${flags})
 endmacro()
 
-make_sample(sample_1_window sample_1_window.cpp)
-make_sample(sample_2_triangle sample_2_triangle.cpp)
-make_sample(sample_3_textured_triangle sample_3_textured_triangle.cpp)
+if (CMAKE_SYSTEM_NAME MATCHES "Darwin")
+  set(RHI_EXAMPLE_FLAGS "RHI_EXAMPLE_BACKEND_METAL")
+else()
+  set(RHI_EXAMPLE_FLAGS "RHI_EXAMPLE_BACKEND_VULKAN")
+endif()
+
+make_sample(sample_1_window sample_1_window.cpp ${RHI_EXAMPLE_FLAGS})
+make_sample(sample_2_triangle sample_2_triangle.cpp ${RHI_EXAMPLE_FLAGS})
+make_sample(sample_3_textured_triangle sample_3_textured_triangle.cpp ${RHI_EXAMPLE_FLAGS})

cpp_examples/rhi_examples/common_metal.h

@@ -6,29 +6,6 @@
 #import <GLFW/glfw3native.h>
 #include "glm/glm.hpp"
 
-#if defined(__APPLE__) && defined(__OBJC__)
-#import <Metal/Metal.h>
-#import <QuartzCore/QuartzCore.h>
-#define DEFINE_METAL_ID_TYPE(x) typedef id<x> x##_id;
-#else
-#define DEFINE_METAL_ID_TYPE(x) typedef struct x##_t *x##_id;
-#endif
-
-DEFINE_METAL_ID_TYPE(MTLDevice);
-DEFINE_METAL_ID_TYPE(MTLBuffer);
-DEFINE_METAL_ID_TYPE(MTLTexture);
-DEFINE_METAL_ID_TYPE(MTLSamplerState);
-DEFINE_METAL_ID_TYPE(MTLLibrary);
-DEFINE_METAL_ID_TYPE(MTLFunction);
-DEFINE_METAL_ID_TYPE(MTLComputePipelineState);
-DEFINE_METAL_ID_TYPE(MTLCommandQueue);
-DEFINE_METAL_ID_TYPE(MTLCommandBuffer);
-DEFINE_METAL_ID_TYPE(MTLBlitCommandEncoder);
-DEFINE_METAL_ID_TYPE(MTLComputeCommandEncoder);
-DEFINE_METAL_ID_TYPE(CAMetalDrawable);
-
-#undef DEFINE_METAL_ID_TYPE
-
 #include "taichi/rhi/metal/metal_api.h"
 #include "taichi/rhi/metal/metal_device.h"
 

cpp_examples/rhi_examples/common_metal.mm

@@ -1,7 +1,6 @@
 #include "common_metal.h"
 
 #include <assert.h>
-#import <simd/simd.h>
 #include <stdio.h>
 #include <stdlib.h>
 

cpp_examples/rhi_examples/sample_1_window.cpp

@@ -1,5 +1,10 @@
+#ifdef RHI_EXAMPLE_BACKEND_VULKAN
 #include "common_vulkan.h"
-// #include "common_metal.h"
+#endif  // RHI_EXAMPLE_BACKEND_VULKAN
+
+#ifdef RHI_EXAMPLE_BACKEND_METAL
+#include "common_metal.h"
+#endif
 
 class SampleApp : public App {
  public:

cpp_examples/rhi_examples/sample_2_triangle.cpp

@@ -1,5 +1,10 @@
-// #include "common_vulkan.h"
+#ifdef RHI_EXAMPLE_BACKEND_VULKAN
+#include "common_vulkan.h"
+#endif  // RHI_EXAMPLE_BACKEND_VULKAN
+
+#ifdef RHI_EXAMPLE_BACKEND_METAL
 #include "common_metal.h"
+#endif
 
 std::vector<uint32_t> frag_spv =
 #include "shaders/2_triangle.frag.spv.h"

cpp_examples/rhi_examples/sample_3_textured_triangle.cpp

@@ -1,5 +1,10 @@
-// #include "common_vulkan.h"
+#ifdef RHI_EXAMPLE_BACKEND_VULKAN
+#include "common_vulkan.h"
+#endif  // RHI_EXAMPLE_BACKEND_VULKAN
+
+#ifdef RHI_EXAMPLE_BACKEND_METAL
 #include "common_metal.h"
+#endif
 
 std::vector<uint32_t> frag_spv =
 #include "shaders/3_triangle.frag.spv.h"
@@ -104,7 +109,7 @@ class SampleApp : public App {
     {
       constexpr uint32_t tex_size = 256;
 
-      // Just a little 8 bit, 256x256 voronoi texture
+      // Just a little 8 bit, 256x256 worley texture
       std::vector<std::pair<double, double>> random_points;
       for (int i = 0; i < 100; i++) {
         double x = tex_size * static_cast<double>(std::rand()) / RAND_MAX;

python/taichi/examples/ggui_examples/mass_spring_game_ggui.py

@@ -1,7 +1,7 @@
 import taichi as ti
 
-arch = ti.vulkan if ti._lib.core.with_vulkan() else ti.cuda
-ti.init(arch=arch)
+# arch = ti.vulkan if ti._lib.core.with_vulkan() else ti.cuda
+ti.init(arch=ti.metal)
 
 spring_Y = ti.field(dtype=ti.f32, shape=())  # Young's modulus
 paused = ti.field(dtype=ti.i32, shape=())

python/taichi/examples/ggui_examples/mpm3d_ggui.py

@@ -2,8 +2,7 @@
 
 import taichi as ti
 
-arch = ti.vulkan if ti._lib.core.with_vulkan() else ti.cuda
-ti.init(arch=arch)
+ti.init(arch=ti.gpu)
 
 # dim, n_grid, steps, dt = 2, 128, 20, 2e-4
 # dim, n_grid, steps, dt = 2, 256, 32, 1e-4

python/taichi/shaders/SetImage_vk.frag

@@ -7,12 +7,15 @@ layout(location = 0) in vec2 frag_texcoord;
 layout(location = 0) out vec4 out_color;
 
 layout(binding = 1) uniform UBO {
+  vec2 lower_bound;
+  vec2 upper_bound;
   float x_factor;
   float y_factor;
   int is_transposed;
 } ubo;
 
 void main() {
   vec2 coord = frag_texcoord * vec2(ubo.x_factor,ubo.y_factor);
-  out_color = texture(texSampler, ubo.is_transposed != 0 ? coord.yx : coord);
+  coord = clamp(coord, ubo.lower_bound, ubo.upper_bound);
+  out_color = textureLod(texSampler, ubo.is_transposed != 0 ? coord.yx : coord, 0);
 }

python/taichi/tools/image.py

@@ -3,6 +3,8 @@
 
 import taichi as ti
 
+import ctypes
+
 
 def cook_image_to_bytes(img):
     """
@@ -16,7 +18,7 @@ def cook_image_to_bytes(img):
     if img.dtype in [np.uint16, np.uint32, np.uint64]:
         img = (img // (np.iinfo(img.dtype).max // 256)).astype(np.uint8)
     elif img.dtype in [np.float32, np.float64]:
-        img = (np.clip(img, 0, 1) * 255.0 + 0.5).astype(np.uint8)
+        img = (np.clip(img, 0, 1) * 255.0).astype(np.uint8)
     elif img.dtype != np.uint8:
         raise ValueError(f"Data type {img.dtype} not supported in ti.tools.imwrite")
 
@@ -81,11 +83,10 @@ def imread(filename, channels=0):
         np.ndarray : An output image loaded from given filename.
     """
     ptr, resx, resy, comp = _ti_core.imread(filename, channels)
-    img = np.ndarray(shape=(resy, resx, comp), dtype=np.uint8)
-    img = np.ascontiguousarray(img)
-    # TODO(archibate): Figure out how np.ndarray constructor works and replace:
-    _ti_core.C_memcpy(img.ctypes.data, ptr, resx * resy * comp)
-    # Discussion: https://github.com/taichi-dev/taichi/issues/802
+    img = np.copy(np.ctypeslib.as_array((ctypes.c_uint8 * resx * resy * comp).from_address(ptr))).reshape(
+        resy, resx, comp
+    )
+    _ti_core.imfree(ptr)
     return img.swapaxes(0, 1)[:, ::-1, :]
 
 

taichi/python/export_ggui.cpp

@@ -13,11 +13,11 @@ namespace py = pybind11;
 
 #include "taichi/ui/utils/utils.h"
 #include "taichi/ui/common/window_base.h"
-#include "taichi/ui/backends/vulkan/window.h"
+#include "taichi/ui/ggui/window.h"
 #include "taichi/ui/common/canvas_base.h"
 #include "taichi/ui/common/camera.h"
-#include "taichi/ui/backends/vulkan/canvas.h"
-#include "taichi/ui/backends/vulkan/scene.h"
+#include "taichi/ui/ggui/canvas.h"
+#include "taichi/ui/ggui/scene.h"
 #include "taichi/rhi/vulkan/vulkan_loader.h"
 #include "taichi/rhi/arch.h"
 #include "taichi/program/field_info.h"
@@ -521,6 +521,19 @@ struct PyWindow {
            double fps_limit,
            std::string package_path,
            Arch ti_arch) {
+    Arch ggui_arch = Arch::vulkan;
+
+    if (ti_arch == Arch::metal) {
+      ggui_arch = Arch::metal;
+    }
+
+    if (ggui_arch == Arch::vulkan) {
+      // Verify vulkan available
+      if (!lang::vulkan::is_vulkan_api_available()) {
+        throw std::runtime_error("Vulkan must be available for GGUI");
+      }
+    }
+
     AppConfig config = {name,
                         res[0].cast<int>(),
                         res[1].cast<int>(),
@@ -530,10 +543,9 @@ struct PyWindow {
                         show_window,
                         fps_limit,
                         package_path,
-                        ti_arch};
-    if (!lang::vulkan::is_vulkan_api_available()) {
-      throw std::runtime_error("Vulkan must be available for GGUI");
-    }
+                        ti_arch,
+                        ggui_arch};
+
     window = std::make_unique<vulkan::Window>(prog, config);
   }
 

taichi/python/export_visual.cpp

@@ -110,10 +110,7 @@ void export_visual(py::module &m) {
            py::return_value_policy::reference);
   m.def("imwrite", &imwrite);
   m.def("imread", &imread);
-  // TODO(archibate): See misc/image.py
-  m.def("C_memcpy", [](size_t dst, size_t src, size_t size) {
-    std::memcpy((void *)dst, (void *)src, size);
-  });
+  m.def("imfree", &imfree);
 }
 
 }  // namespace taichi

taichi/rhi/common/window_system.h

@@ -2,6 +2,10 @@
 
 #ifdef TI_WITH_GLFW
 #include "GLFW/glfw3.h"
+#ifdef TI_WITH_METAL
+#define GLFW_EXPOSE_NATIVE_COCOA
+#import <GLFW/glfw3native.h>
+#endif
 #endif  // TI_WITH_GLFW
 
 namespace taichi::lang::window_system {

taichi/rhi/impl_support.h

@@ -179,5 +179,16 @@ inline void hash_combine(std::size_t &seed, const T &v) {
   seed ^= hasher(v) + 0x9e3779b9 + (seed << 6) + (seed >> 2);
 }
 
+// A helper to remove copy constructor
+class NonAssignable {
+ private:
+  NonAssignable(NonAssignable const &);
+  NonAssignable &operator=(NonAssignable const &);
+
+ public:
+  NonAssignable() {
+  }
+};
+
 }  // namespace rhi_impl
 }  // namespace taichi::lang