Issue with surfel_tracer

[azureovo]

作者您好，感谢您的时间。
我在运行训练的第二步时，出现了Segmentation fault (core dumped)：

通过断点检测，发现是在pip下载的surfel_tracer包的raytracer.py中的from surfel_tracer import _C出现了问题：

之后调用self.impl = _C.create_gaussiantracer()时报的段错误：

请问有什么解决问题surfel_tracer包的思路吗，非常希望能复现成功您的工作，感谢您的指导！

[SuLvXiangXin]

optix对显卡和环境有一些要求，你可以参考1, 2

[azureovo]

非常感谢您！想知道：本项目依赖的Optix最低版本是到7.5吗？
fudan-zvg/gaussian-raytracing#1 (comment)
或者能支持本项目的最低Optix是多少呢？

[tangjm24]

你解决了这个问题了吗，我也遇到了同样的问题

[tangjm24]

我的情况是：
nvidia驱动版本是550.67.
cuda: 11.8
os: ubuntu 22.04

[SuLvXiangXin]

@azureovo 本项目文件中已经包含了optix7.5，不需要你自己下载，如果你改成其他版本的话某些optix的函数调用可能会不一样。

[SuLvXiangXin]

@tangjm24 你的版本应该没问题，optix似乎需要rtx系列显卡，且不是wsl

[tangjm24]

@tangjm24 你的版本应该没问题，optix似乎需要rtx系列显卡，且不是wsl

似乎我的硬件并没有问题也不是wsl. 现在没有segment fault了，但是多了另外一个错误：）

[SuLvXiangXin]

你看看安装流程的命令是否全部照做了

[tangjm24]

你看看安装流程的命令是否全部照做了

感谢您的回复，我是正常安装您给的安装命令make编译并安装的，这个过程也没有任何报错。

现在似乎是optixDeviceContextCreate这个部分失败了

[Start1er]

Hi, I run successfully in my windows docker desktop. Your issues can be solved in Nvidia docker for Windows desktop. Thanks for author sharing their awesome work.😎

[ArisPan]

Hello everyone, thank you for your insights so far! I am dealing with the same Segmentation fault (core dumped) error during the second part of training.

I'm on Windows 11 running Docker Desktop with WSL2 (with Ubuntu 22.04). GPU is RTX 4080. NVIDIA driver version is 560.94 (Optix 7.5 minimum requirement is 515.48). The Dockerfile I used is the following:

FROM nvidia/cuda:11.8.0-cudnn8-devel-ubuntu22.04

ENV DEBIAN_FRONTEND=noninteractive
RUN apt-get update && apt-get install -y \
	git \
	wget \
	curl \
	build-essential \
	cmake \
	gcc-11 g++-11 \
	libgl1-mesa-glx \
	libglib2.0-0 \
	libx11-dev \
	&& rm -rf /var/lib/apt/lists/*

RUN curl -o ~/miniconda.sh https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh && \
	bash ~/miniconda.sh -b -p /opt/conda && \
	rm ~/miniconda.sh && \
	/opt/conda/bin/conda clean -ya
ENV PATH=/opt/conda/bin:$PATH
RUN conda init

ENV TORCH_CUDA_ARCH_LIST="8.9"
ENV NVIDIA_DRIVER_CAPABILITIES=compute,utility,graphics
ENV NVIDIA_VISIBLE_DEVICES=all
ENV FORCE_CUDA=1

WORKDIR /workspace
COPY . .

RUN conda update -n base -c defaults conda && \
	conda env create -f environment.yml

SHELL ["conda", "run", "-n", "irgs", "/bin/bash", "-c"]

RUN pip install torch==2.0.0+cu118 torchvision==0.15.0+cu118 torchaudio==2.0.0+cu118 --extra-index-url https://download.pytorch.org/whl/cu118

RUN git clone https://github.com/NVlabs/nvdiffrast.git && \
	cd nvdiffrast && \
	git checkout tags/v0.3.1 && \
	pip install .

RUN pip install submodules/diff-surfel-rasterization \
	submodules/simple-knn \
	submodules/raytracing

RUN cd submodules/surfel_tracer && rm -rf ./build && mkdir build && cd build && cmake .. && make && cd ../ && cd ../../ && \
	pip install submodules/surfel_tracer

While the installation process completes successfully and the first part of training (Ref-Gaussian) produces the expected results, the second part always results in Segmentation fault (core dumped). I have tried on both CUDA 11.7 (as proposed by Ref-Gaussian) and 11.8. @Start1er could you kindly provide your Dockerfile or any additional insight?

Thank you very much!

[ArisPan]

I have finally gotten IRGS to complete both stages of training. The environment is:

• Graphics Card: NVIDIA RTX4080 16GB
• OS: Ubuntu 22.04 LTS
• NVIDIA Driver: 535.230.02
• CUDA Toolkit: 11.8

I have found that the safest and most frictionless way to install the different NVIDIA driver version is via the ubuntu-drivers tool. You can find more information here. A simple:

sudo ubuntu-drivers install nvidia:535

was all it took (and some reboots). ubuntu-drivers takes care of upgrading and cleaning up (no need for purging older versions etc.).

An alternative is to use the .run installer (although I have found this bothersome and prone to errors). You can get the English NVIDIA Driver version 535.183.01 here. This is generally advised against.

The issue revolved around the driver compatibility with OptiX 7.5 (required by Surfel Tracer). Although I have tried to manually install OptiX 7.5 along my default driver version (570.133.07) and change the environment variables to point to the 7.5 runtime, the OptiX pipeline always failed to initialize. Leaving everything as is and completely changing the NVIDIA Driver to 535 solved the issue. You can check your OptiX runtime version via:

ls /usr/lib/x86_64-linux-gnu | grep optix

With NVIDIA Driver 535.230.02, the output is:

libnvoptix.so.1
libnvoptix.so.535.230.02

I hope this helps!