Improved docker compose usage and README.md

.dockerignore

@@ -1,5 +1,10 @@
+.DS_STORE
 .git
-dist
+.idea
+__pycache__
+.flake8
 .pytest_cache
+dist
 volumes
 .env*
+compose*.yaml

.github/workflows/test-python.yaml

@@ -5,5 +5,5 @@ jobs:
   test-python:
     uses: winds-mobi/winds-mobi-cicd/.github/workflows/test-python.yaml@main
     with:
-      python_version: 3.10.6
+      python_version: 3.10.11
       poetry_version: 1.5.1

Dockerfile

@@ -22,5 +22,4 @@ ENV PATH=/.venv/bin:$PATH
 
 COPY . /opt/project/
 WORKDIR /opt/project/
-
-ENTRYPOINT ["python", "run_scheduler.py"]
+CMD ["./docker-cmd.sh"]

README.md

@@ -1,44 +1,44 @@
-winds.mobi - real-time weather observations
-===========================================
-
-[![Follow us on https://www.facebook.com/WindsMobi/](https://img.shields.io/badge/facebook-follow_us-blue)](https://www.facebook.com/WindsMobi/)
-
-[winds.mobi](http://winds.mobi): Paraglider pilot, kitesurfer, check real-time weather conditions of your favorite spots
-on your smartphone, your tablet or your computer.
-
 winds-mobi-providers
---------------------
+====================
 
 Python scripts that get the weather data from different providers and save it in a common format into mongodb. 
 This project use Google Cloud APIs to compute any missing station details (altitude, name, timezone, ...).
 Google Cloud API results are cached with redis.
 
+## Run the project with docker compose (simple way)
 ### Dependencies
-
-- python 3.10 and [poetry](https://python-poetry.org) 
-- mongodb 4.4
-- redis
+- [docker](https://docs.docker.com/get-docker/)
 - Google Cloud API key
+- Providers secrets (optional)
 
-See [settings.py](https://github.com/winds-mobi/winds-mobi-providers/blob/main/settings.py)
-
-### Run the project with docker compose (simple way)
-
-Create a `.env` file from `.env.template` which will be read by docker compose:
-
+Create an `.env` file from `.env.template` read by docker compose:
 - fill GOOGLE_API_KEY with you own [Google Cloud API key](https://cloud.google.com/docs/authentication/api-keys#creating_an_api_key)
-- optionally fill the missing secrets for each provider
+- optionally fill the missing secrets for each provider you want to test
 
-Then start the external services and the providers scheduler:
+### Start the databases
+- `docker compose -f compose.services.yaml up`
+
+### Run the providers
+- `docker compose up`
 
-- `docker compose --profile=scheduler up --build`
+Or, run only a specific provider:
+- `PROVIDER=ffvl docker compose up`
 
 Some providers need [winds-mobi-admin](https://github.com/winds-mobi/winds-mobi-admin#run-the-project-with-docker-compose-simple-way) running to get stations metadata.
 
-### Run the project locally on macOS
+## Run the project locally on macOS
+### Dependencies
+- [homebrew](https://brew.sh)
+- python 3.10
+- [poetry](https://python-poetry.org)
+- Google Cloud API key
+- Providers secrets (optional)
 
-#### Install dependencies
+Create an `.env.localhost` file from `.env.localhost.template` read by `dotenv` for our local commands:
+- fill GOOGLE_API_KEY with you own [Google Cloud API key](https://cloud.google.com/docs/authentication/api-keys#creating_an_api_key)
+- optionally fill the missing secrets for each provider you want to test
 
+Install libraries with homebrew:
 - `brew install openssl`
 - `export LDFLAGS=-L/usr/local/opt/openssl/lib`
 
@@ -48,41 +48,31 @@ Some providers need [winds-mobi-admin](https://github.com/winds-mobi/winds-mobi-
 - `brew install mysql-client`
 - `export PATH=/usr/local/opt/mysql-client/bin:$PATH`
 
-#### Python environment
-
+### Python virtual environment
 - `poetry install`
 - `poetry shell`
 
-Create a `.env.localhost` file from `.env.localhost.template` which will be read by `dotenv` for our local commands:
-
-- fill GOOGLE_API_KEY with you own [Google Cloud API key](https://cloud.google.com/docs/authentication/api-keys#creating_an_api_key)
-- optionally fill the missing secrets for each provider
-
-#### External services with docker compose
-
-Create a `.env` file from `.env.template` which will be read by docker compose.
-
-Then start the external services:
-
-- `docker compose up`
-
-#### Run the scheduler
+### Start the databases
+You must already have the `.env` file created in the [previous section](#run-the-project-with-docker-compose-simple-way)
+- `docker compose -f compose.services.yaml up`
 
+### Run the providers
 - `dotenv -f .env.localhost run python run_providers.py`
 
-#### Run only a provider
-
+Or, run only a specific provider:
 - `dotenv -f .env.localhost run python providers/ffvl.py`
 
-### Contributing
+## Contributing
+### Checking the code style
 
-#### Checking the code style
+Format your code:
+- `poetry run black .`
 
-Format your code: `poetry run black .`
+Run the linters:
+- `poetry run flake8 .`
+- `poetry run isort .`
 
-Run the linter: `poetry run flake8 .`
-
-#### Add a new provider to winds.mobi
+### Add a new provider to winds.mobi
 
 You know a good weather station that would be useful for many paraglider pilots or kitesurfers? 
 
@@ -172,26 +162,5 @@ if __name__ == "__main__":
     my_provider()
 ```
 
-##### And test it
-
-Start the external services:
-
-- `docker compose up`
-
-Build a Docker image containing your new provider `providers/my_provider.py`:
-
-- `docker build --tag=winds.mobi/my_provider .`
-
-Then run your provider inside a container with:
-
-- `docker run -it --rm --env-file=.env --network=winds-mobi-providers --entrypoint=python winds.mobi/my_provider -m providers.my_provider`
-
-To avoid building a new image on every change, you can mount your local source to the container directory `/opt/project` 
-with a Docker volume:
-
-- `docker run -it --rm --env-file=.env --network=winds-mobi-providers --volume=$(pwd):/opt/project --entrypoint=python winds.mobi/my_provider -m providers.my_provider`
-
-Licensing
----------
-
+## Licensing
 Please see the file called [LICENSE.txt](https://github.com/winds-mobi/winds-mobi-providers/blob/main/LICENSE.txt)

compose.services.yaml

@@ -0,0 +1,14 @@
+services:
+  mongodb:
+    image: mongo:4.4.19
+    ports:
+      - "8011:27017"
+    volumes:
+      - ./volumes/mongodb:/data/db
+  redis:
+    image: redis:7.0.4
+    ports:
+      - "8012:6379"
+networks:
+  default:
+    name: winds-mobi

compose.yaml

@@ -0,0 +1,22 @@
+services:
+  providers:
+    build:
+      context: .
+    volumes:
+      - .:/opt/project
+    environment:
+      - MONGODB_URL
+      - REDIS_URL
+      - ADMIN_DB_URL
+      - GOOGLE_API_KEY
+      - WINDLINE_SQL_URL
+      - CHECKWX_API_KEY
+      - ROMMA_KEY
+      - IWEATHAR_KEY
+      - BORN_TO_FLY_VENDOR_ID
+      - BORN_TO_FLY_DEVICE_ID
+      - WINDY_API_KEY
+      - PROVIDER
+networks:
+  default:
+    name: winds-mobi

docker-cmd.sh

@@ -0,0 +1,7 @@
+#!/usr/bin/env bash
+
+if [[ $PROVIDER ]]; then
+  python -m "providers.${PROVIDER}"
+else
+  python run_scheduler.py
+fi