CUDA Impl INT4 input quantization with padding #3595
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Adds CUDA context management files (cuda_context.h and cuda_context.cpp) that provide similar functionality to the existing OpenCL context. The changes include: - CudaContext class inheriting from Context and Singleton - CUDA kernel management and execution interfaces - Build system updates to support CUDA with enable-cuda option - Conditional linking of CUDA runtime library for both Windows and Linux - Addition of enable-cuda option in meson_options.txt Signed-off-by: Daekyoung Jung <[email protected]>
This commit adds CUDA context management files (cuda_context.h and cuda_context.cpp) that provide similar functionality to the existing OpenCL context. The changes include: - Implementation of CudaContext class inheriting from Context and Singleton - CUDA kernel management and execution interface - Build system updates to support CUDA with enable-cuda meson_options - Conditional linking of CUDA runtime library for both Windows and Linux - Addition of enable-cuda option in meson_options.txt - Implementation of RMSNorm CUDA kernel and build configuration Signed-off-by: Daekyoung Jung <[email protected]>
This commit includes the following changes: 1. Add new CUDA unit test file (unittest_cuda.cpp) with RMSNorm CUDA kernel tests 2. Reorganize CUDA operations directory structure by moving subdir inclusion from nntrainer/meson.build to nntrainer/tensor/meson.build 3. Add CUDA test target in test/unittest/meson.build 4. Fix CUDA linking issues by adding proper link arguments (-NOIMPLIB, -NOEXP) to prevent generation of unnecessary .lib and .exp files 5. Add CUDA dependencies handling in unit test build configuration The changes ensure proper CUDA support in the build system and add comprehensive unit tests for CUDA operations. Signed-off-by: Daekyoung Jung <[email protected]>
This commit introduces CUDA support for addition operations:
1. Added new CUDA files:
- `nntrainer/tensor/cuda_operations/addition_cuda.cu`: Implementation
of CUDA addition kernel
- `nntrainer/tensor/cuda_operations/addition_cuda.h`: Header for CUDA
addition functions
- `nntrainer/tensor/cuda_operations/cuda_interface.cpp`: Implementation
of CUDA interface functions
- `nntrainer/tensor/cuda_operations/cuda_interface.h`: Header for CUDA
interface
2. Updated build configuration:
- Modified meson.build to include new CUDA files in the build
- Updated test/unittest/meson.build to add unittest_cuda_addition target
3. Added unit test:
- `test/unittest/unittest_cuda_addition.cpp`: Unit test for CUDA addition
operations with timing measurements
The new implementation provides:
- CUDA kernel for element-wise addition operations
- CUDA interface functions for tensor operations
- Comprehensive unit test with performance timing
Signed-off-by: Daekyoung Jung <[email protected]>
- Format all CUDA files in nntrainer/tensor/cuda_operations with clang-format - Add GGML Q8_1 quantization/dequantization implementation for CUDA - Include CPU fallback functions for quantization operations - Add unit tests for CUDA Q8_1 quantization functionality - Update meson build files to include new CUDA operations Signed-off-by: Daekyoung Jung <[email protected]>
- Move int4 quantization test from unittest_blas_kernels_cl.cpp to new unittest_quantize_cl.cpp for better organization - Create shared test utilities (unittest_util.h/cpp) with: * generate_random_vector template function * allocateSVM/freeSVM helper functions - Add unittest_util.cpp to OpenCL test targets in meson.build - Update blas_kernels to support shared test utilities - Add CUDA int4 GEMM kernel implementation (gemm_int4_cuda.cu/h) - Update GGML quantization headers and implementations
This commit adds CUDA impl for INT4 quantization with padding. It matches the existing OpenCL kernel behavior for compatibility. Changes: - Add quantize_input_int4_pad_kernel CUDA kernel function - Add quantize_input_int4_pad_cuda wrapper function - Update unit tests to use new CPU reference implementation - Add round_half_to_even helper function for rounding to nearest even - Add cpu_quantize_input_int4_pad CPU reference implementation - Add unittest_util.cpp to unittest_cuda_quantize target - Add new test case for M=63, K=3072, G=32 Signed-off-by: Daekyoung Jung <[email protected]>
dkjung
requested review from
DonghakPark,
EunjuYang,
SeoHyungjun,
again4you,
anyj0527,
baek2sm,
djeong20,
gichan-jang,
haehun,
jaeyun-jung,
jihochu,
jijoongmoon,
leemgs,
lhs8928,
myungjoo,
skykongkong8,
songgot and
wooksong
as code owners
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This commit adds CUDA impl for INT4 quantization with padding.
It matches the existing OpenCL kernel behavior for compatibility.
Changes:
Signed-off-by: Daekyoung Jung [email protected]