Dev/finishing (#174)

* Reformat according to the new rules.

* `git` now uses 8 or fewer jobs for cloning PyTorch and TorchVision.
This was necessary as newer versions of `git` no longer have the `--jobs 0` option.

* Add `deepspeed` as a dependency.

* Remove the `threads` flag as it cannot automatically detect the number of architectures.

Author: veritas9872

Dockerfile

@@ -178,9 +178,10 @@ FROM ${GIT_IMAGE} AS clone-torch
 ARG PYTORCH_VERSION_TAG
 ARG TORCH_URL=https://github.com/pytorch/pytorch.git
 # Minimize downloads by only cloning shallow branches and not the full `git` history.
-RUN git clone --jobs 0 --depth 1 --single-branch --shallow-submodules \
-        --recurse-submodules --branch ${PYTORCH_VERSION_TAG} \
-        ${TORCH_URL} /opt/pytorch
+# Use at most 8 jobs for cloning the repository and its submodules.
+RUN git clone --jobs $(( 8 < $(nproc) ? 8: $(nproc) )) --depth 1 \
+        --single-branch --shallow-submodules --recurse-submodules \
+        --branch ${PYTORCH_VERSION_TAG} ${TORCH_URL} /opt/pytorch
 
 ########################################################################
 FROM build-base AS build-torch
@@ -211,9 +212,8 @@ ARG BUILD_CAFFE2
 ARG BUILD_CAFFE2_OPS
 ARG USE_PRECOMPILED_HEADERS
 ARG TORCH_CUDA_ARCH_LIST
-# ARG CMAKE_PREFIX_PATH=/opt/conda -> Less portable binary.
-# The `--threads` option is only available for CUDA 11.2+.
-ARG TORCH_NVCC_FLAGS="-Xfatbin -compress-all --threads"
+#ARG CMAKE_PREFIX_PATH=/opt/conda -> Less portable binary.
+ARG TORCH_NVCC_FLAGS="-Xfatbin -compress-all"
 # Build wheel for installation in later stages.
 # Install PyTorch for subsidiary libraries (e.g., TorchVision).
 RUN --mount=type=cache,target=/opt/ccache \
@@ -282,9 +282,9 @@ FROM ${GIT_IMAGE} AS clone-vision
 
 ARG TORCHVISION_VERSION_TAG
 ARG VISION_URL=https://github.com/pytorch/vision.git
-RUN git clone --jobs 0 --depth 1 --single-branch --shallow-submodules \
-        --recurse-submodules --branch ${TORCHVISION_VERSION_TAG} \
-        ${VISION_URL} /opt/vision
+RUN git clone --jobs $(( 8 < $(nproc) ? 8: $(nproc) )) --depth 1 \
+        --single-branch --shallow-submodules --recurse-submodules \
+        --branch ${TORCHVISION_VERSION_TAG} ${VISION_URL} /opt/vision
 
 ########################################################################
 FROM build-torch AS build-vision

README.md

@@ -44,7 +44,7 @@ If this is your first time using this project, follow these steps:
    Cresset has been tested on Windows 11 WSL2 with the Windows CUDA driver
    using Docker Desktop for Windows. There is no need to install a separate
    WSL CUDA driver or Docker for Linux inside WSL.
-   Note that only Docker Desktop is under a commercial EULA and Docker Engine 
+   Note that only Docker Desktop is under a commercial EULA and Docker Engine
    (for Linux) and Lima Docker (for Mac) are still both open-source.
    _N.B._ Windows Security real-time protection causes significant slowdown if enabled.
    Disable any active antivirus programs on Windows for best performance.
@@ -442,8 +442,8 @@ See tutorials for [Docker](https://www.jetbrains.com/help/pycharm/docker.html) a
 for details. JetBrains [Gateway](https://www.jetbrains.com/remote-development/gateway)
 can also be used to connect to running containers.
 
-When using the `ngc` service, add `/usr/local/lib/python3/dist-packages` and 
-`/opt/conda/lib/python3/site-packages` to the interpreter search paths via 
+When using the `ngc` service, add `/usr/local/lib/python3/dist-packages` and
+`/opt/conda/lib/python3/site-packages` to the interpreter search paths via
 the GUI to enable code assistance on the packages installed with `conda`.
 
 _N.B._ PyCharm Professional and other JetBrains IDEs are available

reqs/simple-environment.yaml

@@ -19,3 +19,7 @@ dependencies: # Use conda packages if possible.
   - pytest
   - tmux==3.2a
   - tqdm
+  - pip
+  - pip:
+      - deepspeed
+      - pyre-check

reqs/train-environment.yaml

@@ -60,4 +60,5 @@ dependencies: # Use conda packages if possible.
   # Source compiled packages are installed via pip.
   - pip
   - pip:
+      - deepspeed
       - pyre-check