Replies: 25 comments 149 replies
-
|
I'm happy to support moving the repo into the bytedeco org (or into a dedicated org if that makes more sense). I could also transfer the original repo currently under my personal namespace once it's up to date, as otherwise we would have a forked repo which is often treated as a second class citizen by GitHub compared to a non-forked one. I haven't had the time to really develop Storch further this year, and I doubt this will change much in the near future due other responsibilities and unforeseen things happening in my life. That said, I'll try to help as much as possible. Storch has come a long way and you've put an amazing amount of work into it @mullerhai! |
Beta Was this translation helpful? Give feedback.
-
|
@sbrunk If it's OK with you, please do transfer your repository to @bytedeco. We can update later as necessary. Thanks! @mullerhai Well, if we can find sponsors that sounds great! |
Beta Was this translation helpful? Give feedback.
-
@saudet @sbrunk ，wait we transfer the whole latest storch repo to bytedeco ，I think I will try to introduce storch as bytedeco organization incubate top level【🎖️】？project ，and let these company support us， these days I am prepare for Pytorch on Scala3 Online lesson on Youtube and Bilibili【China】，I think we could make big impact for world |
Beta Was this translation helpful? Give feedback.
-
|
I've just transferred the original repo, which is now at: https://github.com/bytedeco/storch Now we can look into getting the changes from bytedeco/storch#88 merged. |
Beta Was this translation helpful? Give feedback.
-
Perfectly! let me do more fix and clean code ,then make a pr |
Beta Was this translation helpful? Give feedback.
-
|
Beta Was this translation helpful? Give feedback.
-
|
@saudet today I have talk with HUAWEI company AI leader Mr.Cheng , He is very interesting with Storch and javacpp-pytorch. it we could develop CANN[ like cuda] and [TensorRT-llm] with javacpp-pytorch. they want to support us |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for feedback! Yes, this is something that we can do. What would that support look like? |
Beta Was this translation helpful? Give feedback.
-
some Money sponsor and Gpu pc ，but gpu is HUAWEI NPU shengteng |
Beta Was this translation helpful? Give feedback.
-
|
This sounds interesting. The easiest thing for me is if they could become a sponsor through GitHub here: |
Beta Was this translation helpful? Give feedback.
-
|
Hi, @saudet @sbrunk , I want to transfer Storch-numpy Storch-pandas ,storch-opencv ,storch-ffmpeg to Bytedeco repo, to make more people use them . storch-opencv ,storch-ffmpeg is base on javacpp-opencv and javacpp-ffmpeg. how do you think ? |
Beta Was this translation helpful? Give feedback.
-
|
Sure, why not? |
Beta Was this translation helpful? Give feedback.
-
|
@sbrunk I have update storch-core code to the version 0.7.5-1.5.12 , you could check and validate review the code , in my test on pytorch_on_scala3_lesson is could correctly work |
Beta Was this translation helpful? Give feedback.
-
ohh，you know wechat？maybe use wechat or Tencent Meeting ，but they want to meet us in Shenzhen or Beijing to talk about storch and CANN [HuaWei cuda], the develop scheduler maybe is first generate Huawei CANN java glue layer code from cpp ,and try to suit with Huawei NPU clips, We could serach [pytorch-npu] it is build pytorch for huawei clips use python. second , storch invoke CANN Java glue layer code to run huawei gpu. finally they will use storch for Huawei cloud machine pc |
Beta Was this translation helpful? Give feedback.
-
|
I'm fine doing remote meetings, but I'm not going to go to China for that for now anyway. You could go in person, and others like me can join remotely though. |
Beta Was this translation helpful? Give feedback.
-
@saudet If Huawei makes any new progress, I will share the latest updates with you. I will continue to communicate and follow up with them. Currently, I've asked them to internally evaluate the suitability of javacpp-pytorch and Storch. However, it's important to note that Huawei is just one company. While their chip market share in China is currently around 5%, NVIDIA's CUDA still dominates with approximately 90% market share. Regardless, we must prioritize robust support for NVIDIA CUDA and NVIDIA NCCL. First, When do you plan to release javacpp-pytorch 2.9.0-1.5.13? These questions are not specifically aimed at the Huawei collaboration but rather to enhance the competitiveness of our project." |
Beta Was this translation helpful? Give feedback.
-
|
Regardless of what happens with CUDA and NVIDIA in China, I'm not aware of anyone anywhere doing distributed training on the Java platform. If Huawei wants to do that, that's interesting, but we're going to need a team to basically port all the tools required from the Python platform. NCCL is only a very small piece. But it's something you could ask them for starters. Could Huawei have one of their engineers spend some time to add support for NCCL to PyTorch on Java? |
Beta Was this translation helpful? Give feedback.
-
yeah ，no one use java for distribute training llm ，but just we want to make first ，it is should us aim，or we can not instead of python ai env，Huawei chips CANN gpu driven it also support distribute training |
Beta Was this translation helpful? Give feedback.
-
|
@sbrunk Hi Sbrunk, |
Beta Was this translation helpful? Give feedback.
-
|
Hi @saudet @sbrunk ,good news, I attended Huawei's chip-driven CANN salon meetup in Beijing today 2025-11-15. The event was vibrant and engaging, with many Chinese companies already having completed adaptations with them. I communicated with the event organizer—their next salon will feature presentations on javacpp-pytorch and storch, and I’ll probably get a 10-minute slot to speak. Most of the participating companies today still primarily use C++ and Python, and the venue was quite crowded. Their current research depth has reached the level of transformer Attention Matrix multiplication, tile blocking, and CUDA kernel optimization—essentially focusing on matrix blocking strategies, more efficient model inference communication across multiple GPUs and machines, and so on. Compared to them, our innovative javacpp-pytorch and storch are still in the shallow end, not yet diving into the kernel layer. We have a lot of work ahead. They’ll contact me next week to discuss the details of cooperation between CANN and Bytedeco. Do you have any questions? I’ll ask Huawei for their thoughts about your behalf. the CANN repo https://gitcode.com/cann |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for the update! I guess my only question would be, given that we would need to invest a lot of effort, time and money to get all this working with Java, what are the use cases that are worth it? Why not just keep doing those tasks with Python? |
Beta Was this translation helpful? Give feedback.
-
|
I'm writing to provide a quick update on my communication with an engineering team from Huawei's chip division. The conversation was very positive, and they are very interested in collaborating with us. They are currently working on an internal budget and assessment, with a projected start for our official collaboration in 2026. As a first step, they want to evaluate the performance of Storch. I've already demonstrated some of its capabilities to them, and they were impressed, noting that it is very close to Python PyTorch. They then asked for a demonstration of Storch's training capabilities with a GPU. To show this, I had to demonstrate Storch integrated with CUDA. However, I've run into a significant technical issue. Initially, I tried on my Windows 11 machine with an NVIDIA 3060 GPU, but javacpp-pytorch was unable to communicate with CUDA. I then switched to a different laptop running Ubuntu with an NVIDIA 4060 GPU. I had to uninstall CUDA 12.4, as this machine had CUDA 13.0 installed. Even with the Ubuntu setup, javacpp-pytorch (version 2.7.1-1.5.12) still fails to recognize and use CUDA properly. Because of this, I had to reschedule the demonstration with them for either next week or in about two weeks. This is very time-sensitive for us, as the Huawei partnership is a key opportunity. Therefore, I am hoping a stable release of javacpp-pytorch 2.9-1.5.13 can be made available soon. This would allow me to use javacpp-cuda 1.3.0-1.5.13 to successfully demonstrate Storch's CUDA capabilities to them. Summary of Issues and Requests: Urgent Need for New Release: I am encountering CUDA recognition issues on Ubuntu with the current version (2.7.1-1.5.12). Your prompt release of version 1.5.13 would be crucial for this partnership. |
Beta Was this translation helpful? Give feedback.
-
|
What is the error message you're getting when you try to use the snapshots? |
Beta Was this translation helpful? Give feedback.
-
|
Beta Was this translation helpful? Give feedback.
-
|
Please try to use the precompiled binaries |
Beta Was this translation helpful? Give feedback.
-
[INFO] Targeting /home/muller/Documents/code/javacpp-presets/pytorch/src/main/java/../../gen/java/org/bytedeco/pytorch/AllToAllOptions.java |
Beta Was this translation helpful? Give feedback.
-
|
Please try to build the pom.xml file in the samples directory here: |
Beta Was this translation helpful? Give feedback.
-
|
(base) muller@muller-Dell-G16-7630:~/Documents/code/javacpp-presets/pytorch$ mvn --version jdk 21 |
Beta Was this translation helpful? Give feedback.
-
|
hi, @saudet ,huawei company Ai engineer tell me ,they also want to try javacpp-pytorch with cuda 1.13.0 ,they are write python ,they don't want to like me build the all project ,just want to make dependency the jar, I think you need really release and publish to maven center repo |
Beta Was this translation helpful? Give feedback.
-
|
What error do you get with the pom.xml file in this samples directory? |
Beta Was this translation helpful? Give feedback.
-
|
@saudet the javacpp-pytorch all part build is so slowly and complex and often meet error , not suit for all people ,please consider ,make easier for most beginer |
Beta Was this translation helpful? Give feedback.
-
|
What error do you get with the pom.xml file in this samples directory? |
Beta Was this translation helpful? Give feedback.
-
(base) muller@muller-Dell-G16-7630: |
Beta Was this translation helpful? Give feedback.
-
|
Please follow the instructions here and try again: |
Beta Was this translation helpful? Give feedback.
-
|
the Huawei company ai engineer is eagerly try to use javacpp-pytorch with cuda 1.13.0, two choices,publish 1.5.13 snapshot version all to maven snap repo ,please not build by user , another release 1.5.13 version to maven center repo this week. |
Beta Was this translation helpful? Give feedback.
-
this RPC module is important, HUAWEI cloud user need use parameter server to do online training ,we need implement RPC |
Beta Was this translation helpful? Give feedback.
-
|
for EmbeddingImpl EmbeddingBagImpl both not found Embedding::from_pretrained , only have Options for the Embedding::from_pretrained ,does from_pretrained method implement by javacpp is hard ?@saudet. for llm we really need it |
Beta Was this translation helpful? Give feedback.
-
|
ProcessGroupNCCL I think we need implement in new version, I think the gloo problem meet is similar with nccl , |
Beta Was this translation helpful? Give feedback.
-
|
@saudet ,Mnist really have bug, I let other people try to load the dataset, also error |
Beta Was this translation helpful? Give feedback.
-
|
The MNIST dataset works just fine for me. I can't fix what isn't broken! As for the rest, I'm sure we can get everything running as per the C++ API. We just need someone to spend enough time on it. |
Beta Was this translation helpful? Give feedback.
-
|
Hi @saudet ,I have write one gloo example with pytorch , after javacpp-pytorch gloo could work fun, I will implement these code in scala3, these code also like https://github.com/pytorch/examples/blob/main/cpp/distributed/dist-mnist.cpp we need to open two terminal windows then the output |
Beta Was this translation helpful? Give feedback.
-
|
I've added FileStore and TCPStore in commit bbf8c5b so please try again with that. |
Beta Was this translation helpful? Give feedback.
-
perfect I will try |
Beta Was this translation helpful? Give feedback.
-
oh, I delete the pytorch snapshot jar on my disk ,then I try to make new pytorch snapshot jar with mvn compile cmd, but download file is OT/pytorch-2.9.0-1.5.13-20251023.024851-17-* version ? have you publish the new commit and deploy new jar to the maven snapshot repo ? and I see the commit log you add two line in pytorch/src/main/java/org/bytedeco/pytorch/presets/torch.java file to compile so could you add ProcessGroupGlooCuda.cpp |
Beta Was this translation helpful? Give feedback.
-
|
NCCL isn't available on Windows, so that makes the build for Linux more complicated, but let me see... |
Beta Was this translation helpful? Give feedback.
-
Let's not hesitate; let's work and push to make it happen. I firmly believe the goal for javacpp-pytorch is to build an enterprise-grade framework. Now, it's rare for anyone to deploy deep learning frameworks on Windows. The deployment is almost exclusively done on Linux distributions, including: Debian, Ubuntu, Fedora, CentOS, and Red Hat. This is precisely the main battleground where NCCL shines. Alternatively, we could issue an additional declaration to users that certain classes are not available on Windows. This is determined by the underlying CUDA dependencies, which hopefully,maybe NVIDIA will support nccl on windows one day. NVIDIA, in fact, doesn't support Windows for many of its tools, including their own Spark DataFrame plugin called cuDF. This isn't our fault, but it means we should maximize the effectiveness of javacpp-pytorch in an enterprise Linux environment. I recently reached out to a tool called executorch, hoping they would support Linux with java, but they only support Android on the JVM. This is a significant drawback for its growth. On another front, we are working on implementing some of the APIs for huggingface Transformers model api and VLLM apinatively in Java. In the future, this will allow for seamless interaction using javacpp-pyorch's tensors. Also, regarding the pytorch.distributed.rpc module: could we consider implementing it? This would give us an additional channel for deploying distributed models for user invoke,will make more powerful. I have read Huawei NPU PYTORCH code, they also use pytorch rpc to inference and deploy the model Thank you for your hard work and consideration. |
Beta Was this translation helpful? Give feedback.
-
|
I write a gloo cuda example pytorch , it could work ,I think future in scala3 we just need implement like this @saudet |
Beta Was this translation helpful? Give feedback.
-
|
I've added FileStore and TCPStore in commit bbf8c5b so please try again with that. |
Beta Was this translation helpful? Give feedback.
-
|
@saudet I'm scheduled to give another demonstration to Huawei this weekend with the new javacpp-pytorch version 2.9.1-1.5.13, but gloo verification keeps failing, which is making me extremely anxious. Currently, the ProcessGroupGloo creation is failing and causing a JVM crash. I'm not sure if it's a bug in the software itself, due to my incorrect usage, or if I'm passing parameters incorrectly. the final I try ,also failed |
Beta Was this translation helpful? Give feedback.
-
the output error |
Beta Was this translation helpful? Give feedback.
-
|
after I change FileStore also can't connect to the rank java code output |
Beta Was this translation helpful? Give feedback.
-
|
I think there have some tiny bug , some logical code sometime could run , sometime cannot run ,get |
Beta Was this translation helpful? Give feedback.
-
|
There are at least issues with the wrappers for c10::intrusive_ptr. I'll let you know if I manage to fix them. |
Beta Was this translation helpful? Give feedback.
-
|
I've pushed a fix for c10::intrusive_ptr in commit a0dca3b and the following code doesn't crash anymore at least, so please try again with this update. FileStore store = new FileStore("/tmp/foo", 2);
ProcessGroupGloo.Options options = ProcessGroupGloo.Options.create();
options.timeout(new Milliseconds(50));
options.devices().push_back(
ProcessGroupGloo.createDeviceForHostname("127.0.0.1"));
ProcessGroupGloo processGroup = new ProcessGroupGloo(store, 1, 2, options); |
Beta Was this translation helpful? Give feedback.
-
|
now try to use TCPStore and FileStore with ProcessGroupGloo, I think if we could make the gloo multi ranks could discover each other ,the gloo maybe could wok , I have try how to set ,but all failed ,maybe is the gloo.Device cause? I am not sure, but the ProcessGroupGloo should already is running . |
Beta Was this translation helpful? Give feedback.
-
|
Please try to translate working C++ code. I can't guess how we're supposed to use the API |
Beta Was this translation helpful? Give feedback.
-
Sorry, I didn't quite understand what you meant. Are you saying that there's no need to fix ProcessGroup or ProcessGroupNCCL? Or is it that we're still unclear on how we'll use these APIs? My understanding is that ProcessGroup seems to be the parent class of ProcessGroupGloo, ProcessGroupCCL, ProcessGroupMPI, and ProcessGroupUCC. Also, ProcessGroup acts as a factory method; by specifying the backend and device, it can delegate to subclasses like ProcessGroupGLOO and ProcessGroupNCCL to run. I imagine that |
Beta Was this translation helpful? Give feedback.
-
|
I mean, for example, I asked Gemini, "Write a very small application that demonstrates how to use c10d::ProcessGroupGloo from the C++ API of PyTorch using CUDA as backend", and it came up with the following. It doesn't call setBackend(), so I don't think we need to call setBackend(), among other things. #include <torch/torch.h>
#include <c10d/GlooProcessGroup.hpp>
#include <c10d/FileStore.hpp>
#include <c10d/ProcessGroup.hpp>
#include <iostream>
#include <memory>
#include <chrono>
// Helper function to create the communication store and process group
std::shared_ptr<c10d::ProcessGroup> init_gloo_pg(int rank, int world_size, const std::string& master_path) {
// 1. Initialize the shared store (using FileStore for simplicity)
// FileStore uses a shared file path for synchronization. In a real multi-node
// setting, you would use c10d::TCPStore for network-based coordination.
auto store = std::make_shared<c10d::FileStore>(master_path, rank);
// 2. Set up Gloo Process Group Options
auto options = c10d::GlooOptions::create();
options->timeout = std::chrono::seconds(10); // Set a reasonable timeout
// 3. Create the Gloo Process Group
// Gloo handles the actual low-level communication and synchronization.
// Since the tensors are on CUDA, Gloo manages the peer-to-peer data transfers on the GPU.
return std::make_shared<c10d::ProcessGroupGloo>(
store,
rank,
world_size,
std::move(options)
);
}
int main(int argc, char* argv[]) {
// --- Argument Parsing ---
if (argc != 3) {
std::cerr << "Usage: " << argv[0] << " <RANK> <WORLD_SIZE>" << std::endl;
std::cerr << "Example for Rank 0: " << argv[0] << " 0 2" << std::endl;
return 1;
}
const int rank = std::stoi(argv[1]);
const int world_size = std::stoi(argv[2]);
// This file path must be accessible by both processes (e.g., in /tmp/ or an NFS mount)
const std::string master_file_path = "/tmp/pytorch_gloo_store";
std::cout << "Starting process [Rank " << rank << "/" << world_size << "]" << std::endl;
// --- CUDA Device Check ---
if (!torch::cuda::is_available()) {
std::cerr << "CUDA is NOT available. This example requires a CUDA device." << std::endl;
return 1;
}
// Use the first CUDA device
torch::Device device(torch::kCUDA, 0);
// --- 1. Initialize Gloo Process Group ---
auto process_group = init_gloo_pg(rank, world_size, master_file_path);
std::cout << "Process Group initialized successfully." << std::endl;
// --- 2. Create CUDA Tensor ---
// Create a tensor on the CUDA device, initialized based on the rank.
torch::Tensor tensor = torch::ones({2, 2}, device) * (rank + 1);
std::cout << "\nRank " << rank << " initial tensor on CUDA:\n" << tensor << std::endl;
// Rank 0 starts with: [[1, 1], [1, 1]]
// Rank 1 starts with: [[2, 2], [2, 2]]
// --- 3. Perform AllReduce Collective Operation ---
// AllReduce sums all tensors and sends the result back to all ranks.
// The operation is performed on the CUDA device.
std::vector<torch::Tensor> tensors_to_reduce = {tensor};
c10d::ProcessGroup::Options allreduce_opts(c10d::OpType::SUM);
// Start the collective operation
auto work = process_group->allreduce(tensors_to_reduce, allreduce_opts);
// Wait for the collective operation to complete
work->wait();
// --- 4. Verify Result ---
// Expected result (1 + 2 = 3): [[3, 3], [3, 3]]
std::cout << "\nRank " << rank << " final tensor after AllReduce (SUM):\n" << tensor << std::endl;
// Verification check
torch::Tensor expected_result = torch::ones({2, 2}, device) * (1 + world_size);
if (tensor.equal(expected_result)) {
std::cout << "\nRank " << rank << ": SUCCESS! AllReduce result is correct." << std::endl;
} else {
std::cerr << "\nRank " << rank << ": ERROR! AllReduce result is incorrect." << std::endl;
}
return 0;
} |
Beta Was this translation helpful? Give feedback.
-
I think you should be clear that in the init_gloo_pg method within the Gemini-generated code you provided, you actually used ProcessGroupGloo to create an instance of the parent class ProcessGroup, which is equivalent to an internal operation, essentially calling setBackend(torch.DeviceType.cpu, ProcessGroup.BackendType.GLOO, BackendOptional). I have also tested with Gemini, and I believe it can have hallucinations; I couldn't find any direct C++ usage of creating a gloo instance with ProcessGroup on GitHub, as it's mostly done with ProcessGroupGloo, ProcessGroupMPI, or ProcessGroupNCCL. However, I did see in C++ that ProcessGroup is a parent class with factory methods to create gloo and NCCL subclasses. Now, I believe that python pytorch's dist.init_process_group(backend, rank=rank, world_size=world_size) is a call to the ProcessGroup or Backend factory method. |
Beta Was this translation helpful? Give feedback.
-
HI @saudet ，I review my test code and try to run it, Today, I obtained some important error logs; we can analyze the reasons for the connection rejection together. I'm not sure if it's caused by my local computer, the code library, or the way I wrote the code example. I've checked my computer, and the port should not be in use. The VPN is also turned off, which suggests that some of its dependent tools may not have started, and the port is not taking effect. ProcessGroupGloo likely still requires some adjustments and checks. output 3ranks the javacode |
Beta Was this translation helpful? Give feedback.
-
|
I agree that gloo distributed training will be a key challenge we need to tackle before our new version release. After resolving the distributed communication and mutual rank discovery issues, gloo should be able to function normally. The error log shows "connection refused." I suggest you could run the Java code I wrote, starting three separate terminals and specifying rank=0, rank=1, and rank=2 respectively, to see what results you get. If you also get a connection refused error, it might be a problem with the javacpp-pytorch library; if the connection is successful, then it's an issue with my computer's network configuration. I tried again today and still couldn't connect on my computer. |
Beta Was this translation helpful? Give feedback.
-
Perhaps what you said is also correct; it might be that I'm using these APIs incorrectly. I haven't set up a C++ LibTorch environment on my computer yet, but I can try to set it up. However, I'm a bit rusty with C++. If I get it set up and try running some code, also, the PyTorch on GitHub indeed doesn't provide any usable gloo usage examples, but there seems to be a test file called However, it has to be admitted that the normal implementation and operation of Gloo in Python PyTorch definitely rely on Gloo in C++ LibTorch. Currently, though, we are not quite sure about the specific operations and configurations made in Python that enable Gloo to run properly. Actually, I would prefer it to be a configuration issue, but the failure of TCP socket connection should not be caused by the too-short timeout setting. Even when I set the timeout to 100 times the original value, it still remains in the waiting phase. It should be noted that this "wait allreduce" message is the print output I set myself, not from Gloo. Personally, I think the incorrect TCP socket connection is still the main cause, but I am not sure about the exact reason. [W1201 14:32:51.794314750 socket.cpp:472] [c10d] waitForInput: poll for socket SocketImpl(fd=80, addr=[localhost]:36124, remote=[localhost]:12455) returned 0, likely a timeout |
Beta Was this translation helpful? Give feedback.
-
|
FileStore doesn't work either, so there's definitely something wrong with PyTorch when running inside the JVM. That's going to be hard to debug... |
Beta Was this translation helpful? Give feedback.
-
|
Ah, no, after fixing up Gemini's output some more, this works just fine for me: import org.bytedeco.javacpp.*;
import org.bytedeco.javacpp.chrono.*;
import org.bytedeco.pytorch.*;
import org.bytedeco.pytorch.global.torch;
import org.bytedeco.pytorch.global.torch_cuda;
import static org.bytedeco.pytorch.global.gloo.*;
import static org.bytedeco.pytorch.global.torch.*;
import static org.bytedeco.pytorch.global.torch_cuda.*;
public class GlooCudaExample {
public static void main(String[] args) {
// Use a PointerScope to manage memory automatically when objects go out of scope
try (PointerScope scope = new PointerScope()) {
// --- Argument Parsing ---
if (args.length != 2) {
System.err.println("Usage: java GlooCudaExample <RANK> <WORLD_SIZE>");
System.err.println("Example for Rank 0: java GlooCudaExample 0 2");
return;
}
final int rank = Integer.parseInt(args[0]);
final int worldSize = Integer.parseInt(args[1]);
// This file path must be accessible by all processes for synchronization
final String masterFilePath = "/tmp/pytorch_gloo_store";
System.out.println("Starting process [Rank " + rank + "/" + worldSize + "]");
// --- CUDA Device Check ---
if (!torch_cuda.is_available()) {
System.err.println("CUDA is NOT available. This example requires a CUDA device.");
return;
}
// Use the first CUDA device
Device device = new Device(DeviceType.CUDA, (byte)0);
// --- 1. Initialize Gloo Process Group (Conceptual Placeholder) ---
FileStore store = new FileStore(masterFilePath, rank);
ProcessGroupGloo.Options options = ProcessGroupGloo.Options.create();
options.timeout(new Milliseconds(new Seconds(10)));
options.devices().push_back(
ProcessGroupGloo.createDeviceForHostname("127.0.0.1"));
ProcessGroupGloo processGroup = new ProcessGroupGloo(store, rank, worldSize, options);
// --- 2. Create CUDA Tensor ---
// Create a tensor on the CUDA device, initialized based on the rank (rank + 1).
// Note the use of .mul_() for in-place multiplication to match the C++ initialization.
Tensor tensor = torch.ones(new long[]{2, 2}, new TensorOptions(device)).mul_(new Scalar((float)(rank + 1)));
System.out.println("\nRank " + rank + " initial tensor on CUDA:\n" + tensor);
// Rank 0 starts with: [[1.0, 1.0], [1.0, 1.0]]
// Rank 1 starts with: [[2.0, 2.0], [2.0, 2.0]]
// --- 3. Perform AllReduce Collective Operation ---
// AllReduce sums all tensors and sends the result back to all ranks.
// The operation is performed on the CUDA device.
TensorVector tensorsToReduce = new TensorVector(tensor);
AllreduceOptions allreduceOps = new AllreduceOptions().reduceOp(new ReduceOp(ReduceOp.RedOpType.SUM));
// Start the collective operation
Work work = processGroup.allreduce(tensorsToReduce, allreduceOps);
// Wait for the collective operation to complete
work._wait();
// --- 4. Verify Result ---
// Expected result (1 + 2 = 3): [[3.0, 3.0], [3.0, 3.0]]
System.out.println("\nRank " + rank + " final tensor after AllReduce (SUM) Simulation:\n" + tensor);
// Verification check
Tensor expectedResult = torch.ones(new long[]{2, 2}, new TensorOptions(device)).mul_(new Scalar(1 + worldSize));
if (tensor.equal(expectedResult)) {
System.out.println("\nRank " + rank + ": SUCCESS! Simulated AllReduce result is correct (Expected value: " + (1 + worldSize) + ").");
} else {
System.err.println("\nRank " + rank + ": ERROR! Simulated AllReduce result is incorrect.");
}
} catch (Exception e) {
e.printStackTrace();
}
}
} |
Beta Was this translation helpful? Give feedback.
-
|
very nice ,very NB! yeah ,it is really work for me !!!! you are the HERO ,SO PERFECT. Now TcpStore and FileStore all could work. The ProcessGroupGloo is validate pass!!! it is magical , so just my fault that bad gloo use case before |
Beta Was this translation helpful? Give feedback.
-
|
First, it did succeed, with tcpstore and filestore. I looked at my previous code, and it would also succeed if the timeout was set longer. However, it seems that sometimes using filestore, initializing gloo would throw an error. The error was java.lang.RuntimeException: Gloo connectFullMesh failed with timed out connecting: SO_ERROR: Connection refused. But this is related to our programming code and should have nothing to do with javacpp. |
Beta Was this translation helpful? Give feedback.
-
|
Hi @saudet I'm so eager to see the Linux version of ProcessGroupNCCL implemented with javacpp. Regarding suspending gloo and other issues, is it something we can implement? The javacpp pytorch on scala3 meetup will be held in Beijing, China on Saturday, December 13th. I will likely be the second speaker and will join via remote video conference. I'm really keen on recommending to the attendees that they try using javacpp. For the issue they care about most, NCCL, I'm truly looking forward to announcing to them the feasibility of our ProcessGroupNCCL support. This would be a huge draw for them to try javacpp-pytorch. They are currently making the event promotional poster, and I will share it with you all once it's done. I predict that if our implementation is robust enough, they will choose ours. |
Beta Was this translation helpful? Give feedback.
-
I have try to use the mvn compile exec:java to run the nccl example ,also failed, 
|
Beta Was this translation helpful? Give feedback.
-
|
I see, I think I found the issue. This should be fixed with commit 8a1a114. Please download the latest binaries and try again! |
Beta Was this translation helpful? Give feedback.
-
you are right , now nccl is working ,it could init but not sure nccl module all working correctly, I need time to validate , |
Beta Was this translation helpful? Give feedback.
-
|
one thing ,I think HashStore and ChunkSequentialDataLoader need to implement , how do you think |
Beta Was this translation helpful? Give feedback.
-
|
something make me confused , after create processgroupnccl ,but when I invoke ''pgnccl.isInitialized()" get false console log another things ,I think we should solve the processGroup the backendtype can not detect problem console log |
Beta Was this translation helpful? Give feedback.
-
|
Hi ，@saudet we. need to create autoCast context. like that and we. need create the GradScaler now also. torchSharp. C# binding. have implement these api some scala binding |
Beta Was this translation helpful? Give feedback.
-
Don't worry, there are other ways. For example, you can link some European and American bank cards, such as Citibank and HSBC, to WeChat and Alipay, although there is likely to be a handling fee of about 0.5% when withdrawing. I am very willing to assist you with the process, and we can all figure out a way to make it happen for you. Additionally, it's also possible through virtual currency, such as USDT. |
Beta Was this translation helpful? Give feedback.
-
|
I need something that works for Japanese banks, not American banks, but anyway let's not worry about this for now. Please feel free to open accounts for "Bytedeco" on WeChat, Alipay, and what not, and if you ever receive over say $1000, then let's start thinking about what to do next at that time. If you receive less than $1000, then just keep it all for yourself, that's fine. Thanks! |
Beta Was this translation helpful? Give feedback.
-
|
Beta Was this translation helpful? Give feedback.
-
|
I have a PayPal account registered to my GMail account as shown here: https://github.com/saudet |
Beta Was this translation helpful? Give feedback.
-
I will tell the javacpp-pytorch. user in China |
Beta Was this translation helpful? Give feedback.
-
|
if. we.could. implement thses torch autocast method ,we will could do that |
Beta Was this translation helpful? Give feedback.
-
|
hi @saudet Additionally, I have some more to share. I need to sync some new updates with you. Over the past couple of days, I've tested exporting computer vision models, yolo11n and yolo11x, from Python to TorchScript and running inference with javacpp-pytorch. The results are fantastic; we've essentially bridged the last mile. I also used translation and language models from the Python transformers library—minimind, qwen3:05b, and qwen3:4b—and managed to load them all into javacpp-pytorch for normal chat conversations. Honestly, it was quite challenging, and it took me two days of debugging to get it working. I need to sync this with sbrunk. The tokenizer sbrunk developed for Huggingface is excellent, and I've used it. It would be even better if we could develop our own tokenization libraries in Java in the future. @sbrunk Furthermore, I loaded the qwen3:05b model into four different JVM languages—Kotlin, Scala, Groovy, and Clojure—and tested it. The results were excellent, and it ran smoothly in all of them. I have to say, javacpp-pytorch is incredibly powerful. It's bringing a new spring to Java and the JVM. these test demo code project will later share with bytedeco Also, regarding the graph neural network based on javacpp-pytorch, I estimate it can be updated to the bytedeco organization by January 1st, 2026. I've added some new graph transformation code, but it's still in the debugging phase. |
Beta Was this translation helpful? Give feedback.
-
|
hi @saudet However, starting from Pytorch 2*, AOT mode for loading models is supported, compiling them into machine code library files like .so or .pt2 files. An increasing number of models are beginning to support this mode. I tried it on a Mac M5 chip, and a simple Python PyTorch model's exported .so file runs, but the exported .so file for the popular torchvision ResNet18 failed. We primarily use AOTIModelContainerRunnerCpu, and on Windows or Ubuntu, we can also use AOTIModelContainerRunnerCuda. However, we currently cannot load .pt2 format files. This .pt2 file is actually a compressed package containing the .so file and other metadata files like schema and configuration files. I later looked into it and found that the AOTIModelPackageLoader class is needed, corresponding to the C++ libtorch AOTIModelPackageLoader.h: https://docs.pytorch.org/docs/stable/torch.compiler_aot_inductor.html. I also saw that C#'s TorchSharp has already introduced and implemented it: dotnet/TorchSharp#1498, dotnet/TorchSharp#1499. java aot. so file load |
Beta Was this translation helpful? Give feedback.
-
|
HI @saudet |
Beta Was this translation helpful? Give feedback.
-
hi @saudet |
Beta Was this translation helpful? Give feedback.
-
|
oh ，my fault ，javacpp-pytorch already implement part method about ProfilerResult, just use @namespace("torch::autograd::profiler") |
Beta Was this translation helpful? Give feedback.
-
|
hi @saudet ,the @namespace("at") @NoOffset @properties(inherit = org.bytedeco.pytorch.presets.torch.class) // before functions initialize RecordFunction members and call |
Beta Was this translation helpful? Give feedback.
-
|
hi @saudet 。 |
Beta Was this translation helpful? Give feedback.
-
|
Yes, yes, yes, we can all fix that and support everything except I do not have the resources to do it! It won't get done, period, stop asking! |
Beta Was this translation helpful? Give feedback.
-
I know you've contributed a great deal to javacpp-pyorch and expended a lot of energy. Please don't be angry. I think you should take a break recently, and I don't want to occupy your time and energy for free. For the past couple of days, I've been testing a graph neural network framework and a deep reinforcement learning framework based on javacpp-pytorch. It's incredibly powerful; I've used it to implement the training of many graph neural network models and reinforcement learning training. First and foremost, I want to thank the power of javacpp-pytorch. I plan to organize and release them around the New Year. I am promoting the use of javacpp-pytorch to people in China. I've always believed that the more people use the javacpp-pytorch framework and the more it is recognized and trusted, the more willing everyone will be to donate to bytedeco. Here is the technical blog I wrote for promoting javacpp-pytorch. |
Beta Was this translation helpful? Give feedback.
-
|
It doesn't matter how many presentations you do, if there are no users, there are no users! So let's do it this way. I will try to implement anything that has at least a single request from another user out that isn't you. Because at this point in time, the only person interested in the world in all those features is you, and you alone. Unless I hear from at least one other interested user, I will not implement anything that you request. Is this clear? |
Beta Was this translation helpful? Give feedback.
-
https://mp.weixin.qq.com/s/A-vaVjKLSA-lEJlD9Z5XFQ I have already posted the donation information for javacpp in my latest article. You can open it to see that I have also provided a detailed explanation at the end. We are all moved by your contribution and admire your perseverance. Bytedeco and javacpp are like your children, and you have dedicated over ten years to them. Everyone hopes they will be excellent and used by more and more people. I believe you do this out of passion, which is why you have built it to such a massive scale. If it were purely for salary and money, no one could selflessly dedicate a decade to it. We truly respect your contribution. We are expanding the use of javacpp-pytorch, and I have been looking for other communities to integrate it with, such as lancedb, daft, ruoyi-ai, and some commercial companies. I hope everyone can see the bright future. The dawn is before our eyes. javacpp-pytorch has become the most comprehensive deep learning framework after Python PyTorch and C++ libtorch, and we are very excited about it. Regarding the users you are worried about, don't be distressed. I am about to launch a series of video courses on PyTorch on Java for the New Year, which will be available on YouTube and Bilibili. We believe it will attract more and more users. Here is the English translation of the content from my blog. Conclusion: Behind every line of code, there are people carrying the weight for us. In today's sharing, we witnessed the ultimate performance of JavaCPP-PyTorch under AOT compilation. But what you may not know is that to allow Java developers to so smoothly call C++ layer operators, it is an open-source organization called Bytedeco that has been building this "cross-language ladder" alone and firmly for over a decade. Why is Bytedeco worthy of our support? A true lone warrior in infrastructure: In an era where Python monopolizes AI discourse, he has been silently integrating the latest CUDA versions and the latest PyTorch operators, ensuring that Java does not fall behind in the industrial AI field. Zero commercialization, pure contribution: Bytedeco has no commercial funding. All feature updates and bug fixes stem from saudet's love for the Java ecosystem. Not just a tool, but a source of dignity: Because of it, we dare to write "Java deep learning" on our resumes and dare to say "no" to the instability of Python in production environments. The future of AI Infra 3.0 needs your help Your every donation will be directly converted into: A faster update frequency for the operator library (keeping pace with PyTorch/TensorFlow). 💌 How to support Bytedeco? Direct PayPal donation: GitHub Sponsors: Visit the Bytedeco GitHub Sponsors page to choose a subscription-based sponsorship. Corporate sponsorship: If your project is benefiting from JavaCPP, please encourage your company to contribute to this underlying infrastructure. "We never lack people who use packages, but we are extremely short of people who build roads." Supporting Bytedeco is casting a crucial vote for the future of the Java ecosystem. Let's work together to keep Java shining in the AI era! #SupportOpenSource #Bytedeco #JavaCPP #AIInfra #ThankYouDevelopers |
Beta Was this translation helpful? Give feedback.
-
|
HI @saudet You seem to think I'm making demands as if I'm the only user of javacpp-pytorch, as if these features are being developed just for me. That's not the case. There's a logical cause and effect here. I am an algorithm engineer and also a R&D personnel. I have been using Python to write neural networks, and I know which features are of concern to everyone. When I communicate with you, I feel more like a product manager and a tester. I believe that if we don't implement the features that many algorithm professionals care about, it will become our shortcoming, causing people to abandon its use and leading to user attrition. If we believe that a better product, better service, and better features are what attract users, then what we are doing is actually the right approach. I don't think you should worry about the donation issue. People trust ByteDeco, javacpp-pytorch, and you. I even believe that if we had a market team, we could approach venture capitalists and investors, similar to what Manu did, to seek acquisition by a major company like Meta to solve our financial difficulties. Trust us, javacpp-pytorch will have many users in the future. You are not working just for me. I am also developing many frameworks to extend the use cases of javacpp-pytorch, such as course code, graph neural network frameworks, reinforcement learning frameworks, and numerous upstream and downstream components. We are all doing this to make it more powerful and widespread, to break the AI barrier of Python, and to allow more people to use javacpp-pytorch. I will continue to work on the donation issue, please trust me. |
Beta Was this translation helpful? Give feedback.
-
|
@saudet @sbrunk happy new year ，I have try to use China bank card sponsor bytedeco for test ，now it is successful ，so I think more China users could do like that to sponsor ， and I find one javacpp China user @SW-Fantastic. he have made three javacpp. compile project live2d llama.cpp . mariadb, to java https://github.com/SW-Fantastic/cpp-presets , I. think we could. do more things. |
Beta Was this translation helpful? Give feedback.
-
|
anyway ，We must resolve the urgent feature requirement of AOTIModelPackageLoader supporting PT2 and fix the critical bugs before the official release of javacpp-pytorch 2.9.1-1.5.13 — specifically the issues with MemoryProfiler for model training and VRAM usage tracking, and the inference error of AOTIModelContainerRunnerCuda.This is not an emotional request, but a practical issue that will directly impact our subsequent user adoption and promotion. PyTorch is already widely used by the vast majority of PyTorch developers in China, and these problems will severely hinder our rollout. We hope you can understand the urgency of this matter. @saudet |
Beta Was this translation helpful? Give feedback.
-
|
If you care that much about Java, you're going to need to talk with other people than me, so please go do that! The main problem is with Java. No one cares about Java. There is no one investing in Java anymore, not Oracle, not IBM, not Microsoft, not Google, not Amazon, and I don't know of any Chinese companies investing in Java either. There absolutely is no one investing in Java anymore! To be able to do everything we can do with PyTorch with Java like you want it to be we need to make changes to the JDK, a lot of changes. However, there is nothing I can do about that. I am not able to make changes to the JDK. So please stop talking like I can do something about this. I cannot do anything about this. So please stop asking. It's not going to happen |
Beta Was this translation helpful? Give feedback.
-
HI @saudet Don't be discouraged. We just implemented the DeepSeek V3 network structure using javacpp-pytorch today. I just bound my credit card on GitHub, so donations are now possible. I want to make a transaction in my personal name. It's too much for me to take a thousand dollars myself. So, I want to make a deal with you: to purchase these features on behalf of future javacpp-pytorch users. I'm not a rich man, but I'm willing to pay $200 for these features. If you agree, let's make the deal. Of course, if you refuse and think it's simply not implementable, that's fine too. I will still donate in the future to continue supporting the development of javacpp-pytorch. These features include the bugs I've found in the past and feature requests:
To be honest, I'm reluctant to have these transaction records with you, because I have great respect and admiration for your abilities and character, and I'm worried that my transactions might affect your perception of certain things. Personally, it doesn't matter to me as long as we can continue to advance and improve the functionality of javacpp-pytorch. Right now, the most important thing for me is the implementation of the high-performance inference service modules for javacpp-pytorch, namely AOTIModelPackageLoader and AOTIModelContainerRunnerCuda. I appreciate your hard work. If this issue cannot be resolved even by spending money, unless it requires modifying the JDK, then it's fine for us to abandon these for now. |
Beta Was this translation helpful? Give feedback.
-
|
Like I said, I'll try to implement things that other users than yourself request. So if AOTIModelPackageLoader is most important to you, create a new issue for AOTIModelPackageLoader, and if I see other users saying they need that, then I'll try to work on that. Right now, no other users than yourself wants either AOTIModelPackageLoader or any of the other stuff. It's not worth my free time or anyone's free time, including yourself, if no one uses anything that we make. |
Beta Was this translation helpful? Give feedback.
-
Hi @saudet , I'm very happy to see your reply. As you know, the past decade has been a mediocre one for Java. With the rise of AI, Python, C++, and Rust have all developed very well, and many new languages have emerged. Java has entered an era of百花齐鸣 (a hundred schools of thought contending) in deep learning, but none have completeness that surpasses C++ libtorch and PyTorch. Javacpp-pytorch is the only hope, and the only one that can fully compile libtorch. Since you implemented the NCCL and Gloo distributed training engines and mixed precision, javacpp-pytorch has already surpassed other Java deep learning frameworks, whose ecosystems are fundamentally incomplete. From 2021 until now, after using javacpp-pytorch, I have conducted many challenging experiments with it, and it has been able to implement and run them well. For me, it is already quite complete, and I am full of anticipation and hope for it. I have some new good news to share with you and @sbrunk
For me, AOTIModelPackageLoader, AOT Cuda, and profiler are critical factors for future inference acceleration. I am confident that we can use javacpp-pytorch to rebuild a large model inference engine with performance comparable to vLLM, SGLang, and DeepSpeed. A large model inference engine based on Java will become a tool on par with Apache Spark and Flink in the future. However, if you think this is too ahead of its time, of course, if you look at some Java programmers, they are still quite junior, stuck at the level of using tensors or matrix calculations. In any case, in terms of ecosystem completeness, AOTIModelPackageLoader, AOT Cuda, and profiler are indispensable for javacpp-pytorch. If you can implement them, I am willing to pay $200. This is not just for me; it is truly for the Java, Scala, and Kotlin ecosystems. I believe that in the future, more major companies will also consider these. With AOTIModelPackageLoader, AOT Cuda, and profiler, I even feel that javacpp-pytorch would not need updates for the next five years because it is truly complete, and there are no other new breakthrough features. I have already started recording video courses for javacpp-pytorch. I have recorded three lessons so far, the first one in Chinese. Afterwards, I will translate the subtitles and audio into English. By 2026, it will attract many people to turn to learning javacpp-pytorch. Promoting the adoption of Java deep learning in enterprise production is currently difficult for most people to understand and believe in, and they might even think it's a joke. Of course, I am not the only one trying; I have seen some very amusing efforts. |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
-
Hi @saudet
The latest version of Storch is now basically stable. Since all the lessons in the "PyTorch on Scala" course have been successfully completed, I believe I have verified most of the basic functionalities of PyTorch. It’s safe to say that Storch works properly and follows a coding style close to Python’s PyTorch. I highly recommend that everyone learns the Scala 3 version of PyTorch because it appears simpler and allows users to focus more on neural network construction, training, and inference.
After three years of verification and nearly 110,000 lines of Scala code, I can confirm that javacpp-pytorch is indeed an excellent glue layer. However, directly programming with javacpp-pytorch comes with many risks and drawbacks it not easy to use :
first maybe The code is quite verbose, especially when written directly in Java.
second It’s error-prone, with many parameters using types like Pointer and cpp...Vector, and vague names like var1 or var2. Without checking PyTorch’s official documentation, it’s hard to know the exact data types and actual meanings of these parameters.
thrid some Javacpp-pytorch still has some unresolved bugs, such as some padding type layers not working correctly and some hyperparameters failing to take effect use put() method .
Since Python’s PyTorch extends libtorch significantly, many features implemented in Python PyTorch are not available in libtorch, making it impossible for javacpp-pytorch to generate corresponding bindings.
Storch addresses most of these risks and issues, greatly improving the efficiency of using PyTorch. Based on the above, I believe that to promote more widespread use of PyTorch on the JVM platform, it’s better to encourage everyone to try Storch, which is built on javacpp-pytorch. What do you think?
Of course, I also anticipate that there might be a Java-based wrapper library for javacpp-pytorch called JTorch in the future. If necessary, I would like to transfer my extended Storch repository (based on @sbrunk Storch main branch code ) to the bytedeco organization. I also plan to consult @sbrunk sbrunk’s opinions, as I believe hosting it under bytedeco would attract more contributors to optimize Storch and foster ecosystem growth.
On another note, we have high expectations for the 1.5.13 version of pytorch. Looking through the code, I noticed some new additions:
For ProcessGroupGloo, new methods like:
The new class ProcessGroupStatus.
Additions to the global torch class:
I believe these updates show that Gloo support is quite robust. However, based on my understanding of large model training, most users still rely on NCCL-based distributed frameworks with DDP and DSFP modes. Therefore, I would like to request adding the following includes to the presets/torch_include.h generation script:
This should generate corresponding bindings after compilation. I understand this is challenging and may not produce correct code immediately, requiring further debugging. However, without ProcessGroupNCLL, we can only work on small single-machine models, making it difficult to optimize model quantization and CUDA memory usage. I sincerely hope that the 1.5.13 version of javacpp-pytorch will include functional ProcessGroupNCLL, enabling us to develop libraries like DeepSpeed, FairScale, or Colossal-AI to accelerate distributed training. I’d like to hear your thoughts and current challenges—I’m willing to assist with implementation.
Additionally, could the EmbeddingImpl layer include the Embedding::from_pretrained() method? This is crucial for future large-model fine-tuning, especially since we already have the EmbeddingFromPretrainedOptions class. Regarding JIT, I’m unsure if libtorch includes a method similar to torch::trace(jitmodule) and torch.jit.is_scripting in Python.
Beta Was this translation helpful? Give feedback.
All reactions