Skip to content

Commit 02299d5

Browse files
BrummiBrummi
committed
Inference code for MonoRec.
1 parent 883c79a commit 02299d5

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

46 files changed

+4369
-2
lines changed

‎.gitignore

+112
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,112 @@
1+
# Byte-compiled / optimized / DLL files
2+
__pycache__/
3+
*.py[cod]
4+
*$py.class
5+
6+
# C extensions
7+
*.so
8+
9+
# Distribution / packaging
10+
.Python
11+
build/
12+
develop-eggs/
13+
dist/
14+
downloads/
15+
eggs/
16+
.eggs/
17+
lib/
18+
lib64/
19+
parts/
20+
sdist/
21+
var/
22+
wheels/
23+
*.egg-info/
24+
.installed.cfg
25+
*.egg
26+
27+
# PyInstaller
28+
# Usually these files are written by a python script from a template
29+
# before PyInstaller builds the exe, so as to inject date/other infos into it.
30+
*.manifest
31+
*.spec
32+
33+
# Installer logs
34+
pip-log.txt
35+
pip-delete-this-directory.txt
36+
37+
# Unit test / coverage reports
38+
htmlcov/
39+
.tox/
40+
.coverage
41+
.coverage.*
42+
.cache
43+
nosetests.xml
44+
coverage.xml
45+
*.cover
46+
.hypothesis/
47+
48+
# Translations
49+
*.mo
50+
*.pot
51+
52+
# Django stuff:
53+
*.log
54+
local_settings.py
55+
56+
# Flask stuff:
57+
instance/
58+
.webassets-cache
59+
60+
# Scrapy stuff:
61+
.scrapy
62+
63+
# Sphinx documentation
64+
docs/_build/
65+
66+
# PyBuilder
67+
target/
68+
69+
# Jupyter Notebook
70+
.ipynb_checkpoints
71+
72+
# pyenv
73+
.python-version
74+
75+
# celery beat schedule file
76+
celerybeat-schedule
77+
78+
# SageMath parsed files
79+
*.sage.py
80+
81+
# dotenv
82+
.env
83+
84+
# virtualenv
85+
.venv
86+
venv/
87+
ENV/
88+
89+
# Spyder project settings
90+
.spyderproject
91+
.spyproject
92+
93+
# Rope project settings
94+
.ropeproject
95+
96+
# mkdocs documentation
97+
/site
98+
99+
# mypy
100+
.mypy_cache/
101+
102+
# input data, saved log, checkpoints, configs
103+
data/
104+
input/
105+
saved/
106+
datasets/
107+
configs/
108+
109+
# editor, os cache directory
110+
.vscode/
111+
.idea/
112+
__MACOSX/

‎LICENSE

+21
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
MIT License
2+
3+
Copyright (c) 2020 Felix Wimbauer
4+
5+
Permission is hereby granted, free of charge, to any person obtaining a copy
6+
of this software and associated documentation files (the "Software"), to deal
7+
in the Software without restriction, including without limitation the rights
8+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9+
copies of the Software, and to permit persons to whom the Software is
10+
furnished to do so, subject to the following conditions:
11+
12+
The above copyright notice and this permission notice shall be included in all
13+
copies or substantial portions of the Software.
14+
15+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21+
SOFTWARE.

‎README.md

+121-2
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,122 @@
1-
# MonoRec: Semi-Supervised Dense Reconstruction in Dynamic Environments from a Single Moving Camera
1+
# MonoRec
2+
[**Paper**](https://arxiv.org/abs/2011.11814) | [**Video** (CVPR)](https://youtu.be/XimdlXUamo0) | [**Video** (Reconstruction)](https://youtu.be/-gDSBIm0vgk) | [**Project Page**](https://vision.in.tum.de/research/monorec)
23

3-
Official repository. Code will be published soon.
4+
This repository is the official implementation of the paper:
5+
6+
> **MonoRec: Semi-Supervised Dense Reconstruction in Dynamic Environments from a Single Moving Camera**
7+
>
8+
> [Felix Wimbauer*](https://www.linkedin.com/in/felixwimbauer), [Nan Yang*](https://vision.in.tum.de/members/yangn), [Lukas Von Stumberg](https://vision.in.tum.de/members/stumberg), [Niclas Zeller](https://vision.in.tum.de/members/zellern) and [Daniel Cremers](https://vision.in.tum.de/members/cremers)
9+
>
10+
> [**CVPR 2021** (arXiv)](https://arxiv.org/abs/2011.11814)
11+
12+
<a href="https://youtu.be/-gDSBIm0vgk"><div style="text-align:center"><img src="./pictures/frames.gif" style="height:auto;width:50%"/><img src="./pictures/pointcloud.gif" style="height:auto;width:50%"/></div></a>
13+
14+
If you find our work useful, please consider citing our paper:
15+
```
16+
@InProceedings{wimbauer2020monorec,
17+
title = {{MonoRec}: Semi-Supervised Dense Reconstruction in Dynamic Environments from a Single Moving Camera},
18+
author = {Wimbauer, Felix and Yang, Nan and von Stumberg, Lukas and Zeller, Niclas and Cremers, Daniel},
19+
booktitle = {IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
20+
year = {2021},
21+
}
22+
```
23+
24+
## 🏗️️ Setup
25+
26+
The `conda` environment for this project can be setup by running the following command:
27+
28+
```shell
29+
conda env create -f environment.yml
30+
```
31+
32+
## 🏃 Running the Example Script
33+
34+
We provide a sample from the KITTI Odometry test set and a script to run MonoRec on it in ``example/``.
35+
To download the pretrained model and put it into the right place, run ``download_model.sh``.
36+
You can manually do this by can by downloading the weights from [here](https://vision.in.tum.de/_media/research/monorec/monorec_depth_ref.pth.zip)
37+
and unpacking the file to ``saved/checkpoints/monorec_depth_ref.pth``.
38+
The example script will plot the keyframe, depth prediction and mask prediction.
39+
40+
```shell
41+
cd example
42+
python test_monorec.py
43+
```
44+
45+
## 🗃️ Data
46+
47+
In all of our experiments we used the KITTI Odometry dataset for training. For additional evaluations, we used the KITTI, Oxford RobotCar,
48+
TUM Mono-VO and TUM RGB-D datasets. All datapaths can be specified in the respective configuration files. In our experiments, we put all datasets into a seperate folder ```../data```.
49+
50+
### KITTI Odometry
51+
52+
To setup KITTI Odometry, download the color images and calibration files from the
53+
[official website](http://www.cvlibs.net/datasets/kitti/eval_odometry.php) (around 145 GB). Instead of the given
54+
velodyne laser data files, we use the improved ground truth depth for evaluation, which can be downloaded from
55+
[here](http://www.cvlibs.net/datasets/kitti/eval_depth_all.php).
56+
57+
Unzip the color images and calibration files into ```../data```. The lidar depth maps can be extracted into the given
58+
folder structure by running ```data_loader/scripts/preprocess_kitti_extract_annotated_depth.py```.
59+
60+
For training and evaluation, we use the poses estimated by [Deep Virtual Stereo Odometry (DVSO)](https://vision.in.tum.de/research/vslam/dvso). They can be downloaded
61+
from [here](https://vision.in.tum.de/_media/research/monorec/poses_dvso.zip) and should be placed under ``../data/{kitti_path}/poses_dso``. This folder structure is ensured when
62+
unpacking the zip file in the ``{kitti_path}`` directory.
63+
64+
The auxiliary moving object masks can be downloaded from [here](https://vision.in.tum.de/_media/research/monorec/mvobj_mask.zip). They should be placed under
65+
``../data/{kitti_path}/sequences/{seq_num}/mvobj_mask``. This folder structure is ensured when
66+
unpacking the zip file in the ``{kitti_path}`` directory.
67+
68+
### Oxford RobotCar
69+
70+
71+
To setup Oxford RobotCar, download the camera model files and the large sample from
72+
[the official website](https://robotcar-dataset.robots.ox.ac.uk/downloads/). Code, as well as, camera extrinsics need to be downloaded
73+
from the [official GitHub repository](https://github.com/ori-mrg/robotcar-dataset-sdk).
74+
Please move the content of the ``python`` folder to ``data_loaders/oxford_robotcar/``.
75+
``extrinsics/``, ``models/`` and ``sample/`` need to be moved to ``../data/oxford_robotcar/``. Note that for poses we
76+
use the official visual odometry poses, which are not provided in the large sample. They need to be downloaded manually from
77+
[the raw dataset](http://mrgdatashare.robots.ox.ac.uk/download/?filename=datasets/2014-12-12-10-45-15/2014-12-12-10-45-15_vo.tar)
78+
and unpacked into the sample folder.
79+
80+
### TUM Mono-VO
81+
82+
Unfortunately, TUM Mono-VO images are provided only in the original, distorted form. Therefore, they need to be undistorted
83+
first before fed into MonoRec. To obtain poses for the sequences, we run the publicly available version
84+
of [Direct Sparse Odometry](https://github.com/JakobEngel/dso).
85+
86+
### TUM RGB-D
87+
88+
The official sequences can be downloaded from [the official website](https://vision.in.tum.de/data/datasets/rgbd-dataset/download)
89+
and need to be unpacked under ``../data/tumrgbd/{sequence_name}``. Note that our provided dataset implementation assumes
90+
intrinsics from ``fr3`` sequences. Note that the data loader for this dataset also relies on the code from the Oxford Robotcar dataset.
91+
92+
## 🏋️ Training & Evaluation
93+
94+
**Please stay tuned! Training code will be published soon!**
95+
96+
We provide checkpoints for each training stage:
97+
98+
| Training stage | Download |
99+
| --- | --- |
100+
| Depth Bootstrap | [Link](https://vision.in.tum.de/_media/research/monorec/monorec_depth.pth.zip) |
101+
| Mask Bootstrap | [Link](https://vision.in.tum.de/_media/research/monorec/monorec_mask.pth.zip) |
102+
| Mask Refinement | [Link](https://vision.in.tum.de/_media/research/monorec/monorec_mask_ref.pth.zip) |
103+
| Depth Refinement (**final model**) | [Link](https://vision.in.tum.de/_media/research/monorec/monorec_depth_ref.pth.zip) |
104+
105+
Run ``download_model.sh`` to download the final model. It will automatically get moved to ``saved/checkpoints``.
106+
107+
To reproduce the evaluation results on different datasets, run the following commands:
108+
109+
```shell
110+
python evaluate.py --config configs/evaluate/eval_monorec.json # KITTI Odometry
111+
python evaluate.py --config configs/evaluate/eval_monorec_oxrc.json # Oxford Robotcar
112+
```
113+
114+
## ☁️ Pointclouds
115+
116+
To reproduce the pointclouds depicted in the paper and video, use the following commands:
117+
118+
```shell
119+
python create_pointcloud.py --config configs/test/pointcloud_monorec.json # KITTI Odometry
120+
python create_pointcloud.py --config configs/test/pointcloud_monorec_oxrc.json # Oxford Robotcar
121+
python create_pointcloud.py --config configs/test/pointcloud_monorec_tmvo.json # TUM Mono-VO
122+
```

‎base/__init__.py

+3
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
from .base_data_loader import *
2+
from .base_model import *
3+
from .base_trainer import *

‎base/base_data_loader.py

+61
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,61 @@
1+
import numpy as np
2+
from torch.utils.data import DataLoader
3+
from torch.utils.data.dataloader import default_collate
4+
from torch.utils.data.sampler import SubsetRandomSampler
5+
6+
7+
class BaseDataLoader(DataLoader):
8+
"""
9+
Base class for all data loaders
10+
"""
11+
def __init__(self, dataset, batch_size, shuffle, validation_split, num_workers, collate_fn=default_collate):
12+
self.validation_split = validation_split
13+
self.shuffle = shuffle
14+
15+
self.batch_idx = 0
16+
self.n_samples = len(dataset)
17+
18+
self.sampler, self.valid_sampler = self._split_sampler(self.validation_split)
19+
20+
self.init_kwargs = {
21+
'dataset': dataset,
22+
'batch_size': batch_size,
23+
'shuffle': self.shuffle,
24+
'collate_fn': collate_fn,
25+
'num_workers': num_workers
26+
}
27+
super().__init__(sampler=self.sampler, **self.init_kwargs)
28+
29+
def _split_sampler(self, split):
30+
if split == 0.0:
31+
return None, None
32+
33+
idx_full = np.arange(self.n_samples)
34+
35+
np.random.seed(0)
36+
np.random.shuffle(idx_full)
37+
38+
if isinstance(split, int):
39+
assert split > 0
40+
assert split < self.n_samples, "validation set size is configured to be larger than entire dataset."
41+
len_valid = split
42+
else:
43+
len_valid = int(self.n_samples * split)
44+
45+
valid_idx = idx_full[0:len_valid]
46+
train_idx = np.delete(idx_full, np.arange(0, len_valid))
47+
48+
train_sampler = SubsetRandomSampler(train_idx)
49+
valid_sampler = SubsetRandomSampler(valid_idx)
50+
51+
# turn off shuffle option which is mutually exclusive with sampler
52+
self.shuffle = False
53+
self.n_samples = len(train_idx)
54+
55+
return train_sampler, valid_sampler
56+
57+
def split_validation(self):
58+
if self.valid_sampler is None:
59+
return None
60+
else:
61+
return DataLoader(sampler=self.valid_sampler, **self.init_kwargs)

‎base/base_model.py

+25
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
import torch.nn as nn
2+
import numpy as np
3+
from abc import abstractmethod
4+
5+
6+
class BaseModel(nn.Module):
7+
"""
8+
Base class for all models
9+
"""
10+
@abstractmethod
11+
def forward(self, *inputs):
12+
"""
13+
Forward pass logic
14+
15+
:return: Model output
16+
"""
17+
raise NotImplementedError
18+
19+
def __str__(self):
20+
"""
21+
Model prints with number of trainable parameters
22+
"""
23+
model_parameters = filter(lambda p: p.requires_grad, self.parameters())
24+
params = sum([np.prod(p.size()) for p in model_parameters])
25+
return super().__str__() + '\nTrainable parameters: {}'.format(params)

0 commit comments

Comments
 (0)