Poetry build system (#2725)

* remove generated front end bundle

* add poetry and front end dist to vcs ignore

* update build system manifest for poetry

- Modify pyproject.toml file for poetry build system
- Adds build script to generate front end as part of poetry build
- Adds renaming script to remove architecture info from wheel file name
- Updates Makefile to call the correct build steps with `make build`

* chmod on release renaming script

* re-add wheel extension to changed file name

* ensure vsc autoversioning plugin

* ensure vcs auto versioning plugin

* revert vscode settings

* add local dev install command

* add version file to package output

* only include poetry lock in sdist

* run build setup as one shell

* check if yarn is installed before building

* deny poetry venvs creation

* remove poetry installation and just check for poetry and yarn

* update URLs for related info

* Ensure spawning_complete happens only once on workers.

* Update various test cases for new spawning_complete behavior.

* Simplify additional test cases using the new get_messages function.

* Set worker_state to 'running' in start_worker (this used to be by spawning_complete before #2728, but now that only happens once *all* workers have finished)

* Simplify semaphore_wait example

* Reverse accidental change

* Update WorkerRunner.state when spawn cycle is done (it used to rely on the event happening)

* Further simplify example

* Rewrite confusing error messages if someone accesses an HTTP Web UI using https

* Convert report time to locale time

* Update HTML report tests

* Use requests 2.32.2 or higher for Python 3.12, fixes #2555

* Upgrade Sphinx to latest version

* Add to extensions also?

Mentioned in https://sphinx-rtd-theme.readthedocs.io/en/stable/installing.html#how-to-install-and-use-the-theme

* hash anchors in setup docs pages

* testing version of pep440 generator

* remove dist

* ignore dist contents

* include changes to pyproject from main - requests version

* Remove unnecessary python

https://sphinx-rtd-theme.readthedocs.io/en/stable/installing.html

* Downgrade docutils to officially supported

* Restore theme name

* Fix ruff error

* Rename css options

* Conform to style

* Add timestamp to chart tooltip

* Add zoom to echarts

* Update test

* Fix empty timestamp formatting

* Round total_avg_response_time from history

* Reduce CSV_STATS_FLUSH_INTERVAL_SEC to 5 seconds by default (more reasonable in the time of SSD:s :) and remove it from the docs (because it will only rarely need to be adjusted, and increasing it is a very unusual need)

* Dial back the API TOC

* doc formatting fix

* Modify timestamp generation to remove deprecation warning

* Add newline at the end of the file

* Fix incorrect error message detection for https requests introduced in 84ada19

* update build system manifest for poetry

- Modify pyproject.toml file for poetry build system
- Adds build script to generate front end as part of poetry build
- Adds renaming script to remove architecture info from wheel file name
- Updates Makefile to call the correct build steps with `make build`

* remove merge object

* test commit distance

* jinja templating for format number

* simple vesion formatting

* take a stab at updating workflows for poetry

* might as well add discussions forum while I'm here

* add docs serve command to makefile

* developing locust docs page

* make readme and pyproject align in terms of URLs

* remove unused py version in wheel

* adjust dockerfile for poetry build

* add dev dependency group

* docs dependencies

* docs dependencies and readme

* venv for tests

* RTD config for versioning

* rtd step

* don't run the build script on the remote

* rtd

* tests and tox

* tox fixes

* progress on CI runners locally

* skip front end build

* in-progress

* run on remotes not my poor laptop

* run on remotes not my poor laptop

* ruff and mypy version

* yarn timeout for windows, update mypy

* windows plz

* windows plz

* add mypy as a dev dependency

* tests

* testing if this is actually a shell thing

* increase fetch depth for dunamai versioning

* I know what the problem is at least

* test windows

* tox install

* revert vs code settings

* remove oneshell directive from old installation process

* Update maintainers and authors

* move web ui to correct location

* update file exclusions

* skip fe build in fail-fast test

* pin some version dependencies

* update tox version

* add wheel renaming to prepublishing steps

---------

Co-authored-by: Lars Holmberg <[email protected]>
Co-authored-by: Andrew Baldwin <[email protected]>
Co-authored-by: Sam Wright <[email protected]>
Co-authored-by: Lars Holmberg <[email protected]>
Co-authored-by: JavierUhagón <[email protected]>

.github/ISSUE_TEMPLATE/bug.yml

@@ -18,6 +18,7 @@ body:
 
          * [Check the documentation](https://docs.locust.io/en/stable/)
          * [Check the FAQ in the wiki](https://github.com/locustio/locust/wiki/FAQ) 
+         * [Check the Github Discussions](https://github.com/orgs/locustio/discussions) 
          * Search [Stack Overflow](https://stackoverflow.com/questions/tagged/locust), or [ask there](https://stackoverflow.com/questions/ask) yourself. If you tag your question with `locust` we will see it.
          * Ask on [Locust's slack](https://locustio.slack.com) [(sign up here)](https://communityinviter.com/apps/locustio/locust)
 

.github/ISSUE_TEMPLATE/feature_request.yml

@@ -18,6 +18,7 @@ body:
 
          * [Check the documentation](https://docs.locust.io/en/stable/)
          * [Check the FAQ in the wiki](https://github.com/locustio/locust/wiki/FAQ) 
+         * [Check the Github Discussions](https://github.com/orgs/locustio/discussions) 
          * Search [Stack Overflow](https://stackoverflow.com/questions/tagged/locust), or [ask there](https://stackoverflow.com/questions/ask) yourself. If you tag your question with `locust` we will see it.
          * Ask on [Locust's slack](https://locustio.slack.com) [(sign up here)](https://communityinviter.com/apps/locustio/locust)
 

.github/workflows/tests.yml

@@ -26,25 +26,45 @@ jobs:
           - { name: "3.9", python: "3.9", os: ubuntu-latest, tox: py39 }
 
     steps:
-      - uses: actions/checkout@v3
+      - uses: actions/checkout@v4
+        with:
+          fetch-depth: 0
       - uses: actions/setup-python@v4
         with:
           python-version: ${{ matrix.python }}
-      - name: Update pip
+      - name: Install poetry
+        run: |
+          python -m pip install poetry
+      - name: Configure poetry
+        run: |
+          python -m poetry config virtualenvs.in-project true
+      - name: Cache the virtualenv
+        uses: actions/cache@v3
+        with:
+          path: ./.venv
+          key: ${{ runner.os }}-venv-${{ hashFiles('pyproject.toml') }}
+      - name: Install poetry plugin
+        run: python -m poetry self add "poetry-dynamic-versioning[plugin]"
+      # Install test dependencies (tox) to the root (non-package install)
+      - name: Install dependencies
         run: |
-          python -m pip install --upgrade pip
-      - name: Get pip cache dir
-        id: pip-cache
-        run: echo "dir=$(pip cache dir)" >> $GITHUB_OUTPUT
-      - uses: actions/cache@v3
-        with:
-          path: ${{ steps.pip-cache.outputs.dir }}
-          key: pip|${{ runner.os }}|${{ matrix.python }}|${{ hashFiles('pyproject.toml') }}
+          python -m poetry install --no-root --only test
       - name: set full Python version in PY env var
         # See https://pre-commit.com/#github-actions-example
         run: echo "PY=$(python -VV | sha256sum | cut -d' ' -f1)" >> $GITHUB_ENV
-      - run: python -m pip install tox
-      - run: python -m tox -e ${{ matrix.tox }}
+      # Install node and yarn in order to build the front end during packaging  
+      - name: Set Node.js 18.x
+        uses: actions/setup-node@v3
+        with:
+          node-version: 18.x
+      - name: Install Yarn
+        run: npm install -g yarn
+      # Build and install the package if it's Windows, to service shell-based tests
+      - if: ${{ matrix.os }} == "windows-latest"
+        run: |
+          python -m poetry build
+          pip install --find-links=dist locust
+      - run: python -m poetry run tox -e ${{ matrix.tox }}
 
   lint_typecheck_test_webui:
     runs-on: ubuntu-latest
@@ -145,10 +165,13 @@ jobs:
           python-version: "3.x"
       - name: Install dependencies
         run: |
-          python -m pip install --upgrade pip build
+          python -m pip install --upgrade pip poetry
+          python -m poetry config virtualenvs.create false
+          python -m poetry self add "poetry-dynamic-versioning[plugin]"
       - name: Build + set TAG env var for later use
         run: |
-          python -m build
+          python -m poetry build
+          ./rename-wheel.sh
           echo "TAG=${GITHUB_REF#refs/*/}" | tee -a $GITHUB_ENV
       - name: Publish tagged version to PyPI
         uses: pypa/gh-action-pypi-publish@release/v1
@@ -183,11 +206,14 @@ jobs:
       - name: Install dependencies
         if: ${{ env.NOT_MERGE_COMMIT == '' }}
         run: |
-          python -m pip install --upgrade pip build
+          python -m pip install --upgrade pip poetry
+          python -m poetry config virtualenvs.create false
+          python -m poetry self add "poetry-dynamic-versioning[plugin]"
       - name: Build
         if: ${{ env.NOT_MERGE_COMMIT == '' }}
         run: |
-          python -m build
+          python -m poetry build
+          ./rename-wheel.sh
       - name: Publish prerelease version to PyPI
         if: ${{ env.NOT_MERGE_COMMIT == '' }}
         uses: pypa/gh-action-pypi-publish@release/v1

.gitignore

@@ -30,3 +30,5 @@ yarn-error.log
 .venv
 .DS_Store
 .python-version
+poetry.lock
+locust/webui/dist/**

.readthedocs.yaml

@@ -5,12 +5,12 @@ build:
   os: ubuntu-22.04
   tools:
     python: "3.11"
+  jobs:
+    post_create_environment:
+      - python -m pip install poetry
+      - python -m poetry self add "poetry-dynamic-versioning[plugin]"
+    post_install:
+      - SKIP_PRE_BUILD=true VIRTUAL_ENV=$READTHEDOCS_VIRTUALENV_PATH poetry install --with docs
 
 sphinx:
   configuration: docs/conf.py
-
-python:
-  install:
-    - method: pip
-      path: .
-    - requirements: docs/requirements.txt

Dockerfile

@@ -1,12 +1,14 @@
+# Stage 1: Build web front end
 FROM node:18.0.0-alpine as webui-builder
 
 ADD locust/webui locust/webui
 ADD package.json .
 
+# long yarn timeout necessary in certain network environments
 RUN yarn webui:install --production --network-timeout 60000
-
 RUN yarn webui:build
 
+# Stage 2: Build Locust package
 FROM python:3.11-slim as base
 
 FROM base as builder
@@ -15,12 +17,20 @@ RUN apt-get update && apt-get install -y git
 RUN if [ -n "$(arch | grep 'arm64\|aarch64')" ]; then apt install -y --no-install-recommends gcc python3-dev; fi
 RUN python -m venv /opt/venv
 ENV PATH="/opt/venv/bin:$PATH"
+ENV SKIP_PRE_BUILD="true"
 COPY . /build
-RUN pip install /build/
+WORKDIR /build
+# bring in the prebuilt front-end before package installation
+COPY --from=webui-builder locust/webui/dist locust/webui/dist
+RUN pip install poetry && \
+    poetry config virtualenvs.create false && \
+    poetry self add "poetry-dynamic-versioning[plugin]" && \
+    poetry build -f wheel && \
+    pip install dist/*.whl
 
+# Stage 3: Runtime image
 FROM base
 COPY --from=builder /opt/venv /opt/venv
-COPY --from=webui-builder locust/webui/dist locust/webui/dist
 ENV PATH="/opt/venv/bin:$PATH"
 # turn off python output buffering
 ENV PYTHONUNBUFFERED=1

Makefile

@@ -1,23 +1,46 @@
 test:
 	tox
 
-build:
-	rm -f dist/* && python3 -m pip install --upgrade build && python3 -m build .
+.PHONY: build
+build: setup_dependencies
+	rm -rf dist/* && poetry build && ./rename-wheel.sh
+
+install: setup_dependencies
+	poetry install --with dev
+
+setup_dependencies: check-poetry check-yarn
+	poetry config virtualenvs.create false
+	poetry self add "poetry-dynamic-versioning[plugin]"
+
+.SILENT:
+.PHONY: check-poetry
+check-poetry:
+	command -v poetry >/dev/null 2>&1 || { echo >&2 "Locust requires the poetry binary to be available in this shell to build the Python package.\nSee: https://docs.locust.io/en/stable/developing-locust.html#install-locust-for-development"; exit 1; }
+
+.SILENT:
+.PHONY: check-yarn
+check-yarn:
+	command -v yarn >/dev/null 2>&1 || { echo >&2 "Locust requires the yarn binary to be available in this shell to build the web front-end.\nSee: https://docs.locust.io/en/stable/developing-locust.html#making-changes-to-locust-s-web-ui"; exit 1; }
 
 frontend_build:
 	yarn webui:install && yarn webui:build
 
 release: build
 	twine upload dist/*
 
-build_docs:
+setup_docs_dependencies:
+	poetry install --with docs
+
+build_docs: setup_docs_dependencies
 	sphinx-build -b html docs/ docs/_build/
 
+# This command can be used to serve the built documentation at http://localhost for
+# easier offline viewing
+.SILENT:
+.PHONY: serve_docs
+serve_docs:
+	echo "Serving docs at http://localhost:80"
+	python -m http.server 80 -d docs/_build
+
 changelog:
 	@echo "Not supported any more. Run ./generate_changelog.py <version_number> instead!"
-
-sass_watch:
-	sass --watch locust/static/sass:locust/static/css
-
-sass_build:
-	sass --update locust/static/sass:locust/static/css

README.md

@@ -68,6 +68,7 @@ Locust's code base is intentionally kept small and doesn't solve everything out
 
 * Documentation: [docs.locust.io](https://docs.locust.io)
 * Support/Questions: [StackOverflow](https://stackoverflow.com/questions/tagged/locust)
+* Github Discussions: [Github Discussions](https://github.com/orgs/locustio/discussions)
 * Chat/discussion: [Slack](https://locustio.slack.com) [(signup)](https://communityinviter.com/apps/locustio/locust)
 
 ## Authors

docs/developing-locust.rst

@@ -11,10 +11,19 @@ Install Locust for development
 
 Fork Locust on `GitHub <https://github.com/locustio/locust/>`_ and then run
 
-.. code-block:: console
+.. code-block:: sh
+
+    # clone the repo
+    $ git clone git://github.com/<YourName>/locust.git
+
+    # install the poetry build system
+    $ pip3 install poetry 
+
+    # install the dynamic versioning plugin for poetry
+    $ pip3 -m poetry self add "poetry-dynamic-versioning[plugin]"
 
-    $ git clone git://github.com/<YourName>/locust.git # clone the repo
-    $ pip3 install -e locust/                          # install in editable mode
+    # perform an editable install of the "locust" package
+    $ pip3 -m poetry install --with dev
 
 Now the ``locust`` command will run *your* code with no need for reinstalling after making changes.
 
@@ -78,7 +87,7 @@ The documentation source is in the `docs/ <https://github.com/locustio/locust/tr
 
     .. code-block:: console
 
-        $ pip3 install -r docs/requirements.txt
+        $ pip3 -m poetry install --with docs
 
 #. Build the documentation locally:
 

locust/contrib/fasthttp.py

@@ -483,7 +483,7 @@ class FastResponse(CompatResponse):
 
     _response: HTTPSocketPoolResponse | None = None
 
-    encoding: str | None = None
+    encoding: str | float | None = None
     """In some cases setting the encoding explicitly is needed. If so, do it before calling .text"""
 
     request: FastRequest | None = None

locust/webui/.yarnrc

@@ -0,0 +1 @@
+network-timeout 600000

pre_build.py

@@ -0,0 +1,14 @@
+import os
+import subprocess
+
+
+def build() -> None:
+    if os.environ.get("SKIP_PRE_BUILD", "") == "true":
+        print("Skipping front end build...")
+        return
+    print("Building front end...")
+    subprocess.run(["make", "frontend_build"])
+
+
+if __name__ == "__main__":
+    build()