Merge branch 'develop' into u/tjr/terraform

Author: hernandezc1

.github/workflows/BrokerTest.yml

@@ -0,0 +1,66 @@
+name: Test Broker
+
+on:
+  workflow_dispatch:
+  workflow_call:
+  push:
+
+env:
+  PYTHON_VERSION: 3.11
+
+jobs:
+  discover:
+    name: Discover Microservices
+    runs-on: ubuntu-latest
+
+    outputs:
+      matrix: ${{ steps.find_tests.outputs.matrix }}
+
+    steps:
+      - name: Checkout source
+        uses: actions/checkout@v4
+
+      - name: Find testable apps
+        id: find_tests
+        run: |
+          # Find application directories containing a 'tests' subdirectory nd format the result as JSON
+          test_dirs=$(find broker/cloud_functions -type d -name "tests" -exec dirname {} \; | jq -R . | jq -s . | tr -d '\n')
+          echo "found $test_dirs"
+          echo "matrix={\"app_dir\":$test_dirs}" >> $GITHUB_OUTPUT
+
+  test:
+    name: Test Microservices
+    runs-on: ubuntu-latest
+    needs: [ discover ]
+
+    strategy:
+      fail-fast: false
+      matrix: ${{ fromJSON(needs.discover.outputs.matrix) }}
+
+    steps:
+      - name: Checkout source
+        uses: actions/checkout@v4
+
+      - name: Setup Python
+        uses: actions/setup-python@v4
+        with:
+          python-version: ${{ env.PYTHON_VERSION }}
+
+      - name: Install dependencies
+        run: |
+          pip install coverage
+          pip install -r ${{ matrix.app_dir }}/requirements.txt
+
+      - name: Test ${{ matrix.app_dir }} app
+        run: coverage run -m unittest discover  ${{ matrix.app_dir }}
+
+  report-status:
+    name: Report Test Status
+    if: always()
+    needs: [ test ]
+    runs-on: ubuntu-latest
+
+    steps:
+      - name: Check build status
+        if: ${{ contains(needs.*.result, 'failure') }}
+        run: exit 1

.gitignore

@@ -13,6 +13,7 @@ pitt-reader.user.keytab
 
 # IDE files
 .idea/
+.vscode/
 
 # Byte-compiled / optimized / DLL files
 __pycache__/

.readthedocs.yml renamed to .readthedocs.yaml

@@ -4,6 +4,12 @@
 # Required
 version: 2
 
+# Set the OS, Python version and other tools you might need
+build:
+  os: ubuntu-22.04
+  tools:
+    python: "3.12"
+
 # Build documentation in the docs/ directory with Sphinx
 sphinx:
   configuration: docs/source/conf.py
@@ -13,10 +19,5 @@ formats: []
 
 # Set the version of Python and requirements required to build the docs
 python:
-  version: 3.7
-  system_packages: true
   install:
     - requirements: docs/requirements.txt
-    - method: setuptools
-      path: .
-

CHANGELOG.md

@@ -1,4 +1,5 @@
 # Changelog
+
 All notable changes to this project will be documented in this file.
 
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
@@ -10,4 +11,6 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
 ## [0.6.0] - 2021-00-00
 
 ### Changed
-- Creates a "Working Notes" section in RTD and moves the following directories there: notes/, version_tracking/, troy/ (from the `troy` branch).
+
+- Creates a "Working Notes" section in RTD and moves the following directories there:
+  notes/, version_tracking/, troy/ (from the `troy` branch).

README.md

@@ -1,13 +1,16 @@
-[![python](https://img.shields.io/badge/python-3.7-g.svg)]()
-[![Build Status](https://travis-ci.com/mwvgroup/Pitt-Google-Broker.svg?branch=master)](https://travis-ci.com/mwvgroup/Pitt-Google-Broker)
-[![Documentation Status](https://readthedocs.org/projects/pitt-broker/badge/?version=latest)](https://pitt-broker.readthedocs.io/en/latest/?badge=latest)
-
 # Pitt-Google Alert Broker
 
-The Pitt-Google broker is an astronomical alert broker that is being developed for large scale surveys of the night sky, particularly the upcoming [Vera Rubin Observatory's Legacy Survey of Space and Time](https://www.lsst.org/) (LSST).
-We currently process and serve the [Zwicky Transient Facility](https://www.ztf.caltech.edu/)'s (ZTF) nightly alert stream.
+[![Codacy Badge](https://app.codacy.com/project/badge/Grade/93e282e9d5784de6a46e6b87ab6d0afa)](https://app.codacy.com/gh/mwvgroup/Pitt-Google-Broker/dashboard?utm_source=gh&utm_medium=referral&utm_content=&utm_campaign=Badge_grade)
+[![Documentation Status](https://readthedocs.org/projects/pitt-broker/badge/?version=latest)](https://pitt-broker.readthedocs.io/en/latest/?badge=latest)
+
+The Pitt-Google broker is an astronomical alert broker that is being developed
+for large scale surveys of the night sky, particularly the upcoming
+[Vera Rubin Observatory's Legacy Survey of Space and Time](https://www.lsst.org/) (LSST).
+We currently process and serve the [Zwicky Transient Facility](https://www.ztf.caltech.edu/)'s
+(ZTF) nightly alert stream.
 The broker runs on the [Google Cloud Platform](https://cloud.google.com) (GCP).
 
 Documentation is at [pitt-broker.readthedocs.io](https://pitt-broker.readthedocs.io/).
 
-If you run into issues or need assistance, please [open an Issue](https://github.com/mwvgroup/Pitt-Google-Broker/issues).
+If you run into issues or need assistance, please
+[open an Issue](https://github.com/mwvgroup/Pitt-Google-Broker/issues).

broker/__init__.py

@@ -8,16 +8,16 @@
 import os
 from warnings import warn
 
-if 'PGB_OFFLINE' not in os.environ:
+if "PGB_OFFLINE" not in os.environ:
     from google.cloud import logging as cloud_logging
 
     cloud_logging.Client().setup_logging()
 
-for _var in ('GOOGLE_CLOUD_PROJECT', 'GOOGLE_APPLICATION_CREDENTIALS'):
+for _var in ("GOOGLE_CLOUD_PROJECT", "GOOGLE_APPLICATION_CREDENTIALS"):
     if _var not in os.environ:
         warn(
-            f'Environmental variable ``{_var}`` not found. '
-            f'Some package functionality may not be available.'
+            f"Environmental variable ``{_var}`` not found. "
+            f"Some package functionality may not be available."
         )
 
-__version__ = 'development'
+__version__ = "development"

broker/broker_utils/CHANGELOG.md

@@ -0,0 +1,227 @@
+# Changelog<a name="changelog"></a>
+
+All notable changes to this project will be documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
+
+<!-- uncomment the following when we're out of alpha and actually following it -->
+
+<!-- and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). -->
+
+## \[Unreleased\]<a name="unreleased"></a>
+
+## \[0.2.47\] - 2023-06-20
+
+### Changed
+- `broker_utils/avro_schemas` now contains the alert schemas and broker classification schema for ELAsTiCC schema version 0.9.1
+- `avro_schemas/load.py` registers ELAsTiCC schema version 0.9.1
+- Improve error handling and logging in `data_utils.open_alert`. The function uses a lot of
+  broad try/excepts in a brute-force attempt to open an alert packet that could be in many
+  different formats. These changes do some moderate clean up of the error handling and
+  provide more detail in the logs about a known issue with some versions of fastavro.
+- Sort all imports.
+- Cleanup syntax issues identified by Codacy.
+
+### Fixed
+
+- Bugfix the IDs in `testing.Mock`.
+
+## \[0.2.46\] - 2023-04-09
+
+### Changed
+
+- Alert filename "objectId.sourceId.topic.avro" -> "topic/objectId/sourceId.avro"
+
+## \[0.2.45\] - 2022-10-22
+
+### Fixed
+
+- Import `pandas` for `data_utils.alert_lite_to_dataframe()`.
+- Remove the `xmatch` dict from the dataframe created by `data_utils.alert_lite_to_dataframe()`.
+
+## \[0.2.44\] - 2022-10-20
+
+### Added
+
+- `data_utils.alert_lite_to_dataframe()`
+
+## \[0.2.43\] - 2022-08-25
+
+### Fixed
+
+- Updates `data_utils.open_alert()` to properly handle json-serialized alerts in a Cloud Function.
+
+## \[0.2.42\] - 2022-08-24
+
+### Fixed
+
+- Fix bug introduced by allowing `data_utils.open_alert()` to blindly send `**kwargs` to the
+  functions it calls. `data_utils.open_alert()` now sends only the arguments that are appropriate
+  for the function it's calling.
+
+## \[0.2.41\] - 2022-08-19
+
+### Fixed
+
+- Fix bug preventing data files from being included with the package.
+
+## \[0.2.40\] - 2022-08-19
+
+### Added
+
+- `avro_schemas` module to handle schemaless alerts. Includes schema data files.
+- `data_utils.open_alert()` which uses a brute-force strategy to try to open alerts of all
+  different formats.
+
+### Changed
+
+- In `data_utils`, deprecate functions `load_alert()` and `decode_alert()` in favor of
+  `open_alert()`
+
+## \[0.2.39\] - 2022-06-23
+
+### Fixed
+
+- Bug fix: don't use `dataclasses.dataclass` decorator on `collections.namedtuple` modules.
+
+## \[0.2.38\] - 2022-06-23
+
+### Added
+
+- Use `dataclasses.dataclass` decorator on `types` and `testing` modules
+
+### Fixed
+
+- Bug fix in `types.AlertIds.extract_ids()`: access `self.id_keys` as a property, not a function call.
+
+## \[0.2.37\] - 2022-06-21
+
+### Changed
+
+- Minor internal improvements and bug fixes to `testing`.
+- Rename attributes `testing.Mock.results` to `testing.Mock.module_results` and
+  `testing.TestAlert.publish_as` to `testing.TestAlert.serialize`.
+
+## \[0.2.36\] - 2022-06-20
+
+### Added
+
+- Adds `mock` as a keyword argument in `testing.TestAlert`.
+
+## \[0.2.35\] - 2022-06-20
+
+### Added
+
+- `types` module containing `_AlertIds`, `AlertIds`, and `AlertFilename`. These were rewritten and
+  moved out of `data_utils`.
+
+### Changed
+
+- Lazy load `pandas`. It hogs memory on Cloud Functions.
+- Remove `data_utils.AlertIds`, `data_utils.idUtils`, and `data_utils.AlertFilename`. They moved to
+  the new `types` module.
+- Rename `tests` module to `testing` and rewrite the included classes to give them more
+  functionality and make them easier to use.
+
+## \[0.2.34\] - 2022-06-17
+
+### Added
+
+- add `AlertFilename` class in `data_utils`
+
+### Changed
+
+- updated `tests.TestValidator` to accept keyword argument `ids_in` that determines whether
+  alert IDs are extracted from message attributes or the Avro filename.
+
+## \[0.2.33\] - 2022-06-17
+
+### Added
+
+- add "kafka.topic" to mocked msg attributes in `tests.TestAlert`
+
+## \[0.2.32\] - 2022-05-19
+
+### Changed
+
+- fix bug when creating avro payload in `tests.TestAlert`
+
+## \[0.2.31\] - 2022-05-19
+
+### Added
+
+- `tests` module and `tests.TestValidator` class
+- `data_utils.idUtils` class
+
+### Changed
+
+- moved `data_utils.TestAlert` to `tests.TestAlert`
+
+## \[0.2.30\] - 2022-05-18
+
+### Added
+
+- `data_utils.AlertIds` and `data_utils.TestAlert`
+- `gcp_utils.purge_subscription()`
+
+## \[0.2.29\] - 2022-05-17
+
+### Added
+
+- `schema_maps.get_key()` return the key name or the final element in the list corresponding to a
+  schema map and key.
+- `schema_maps.get_value()` return the alert value corresponding to a schema map and key.
+
+### Changed
+
+- Modules updated to use the added functions.
+- `data_utils.load_yaml()` moved to `schema_maps.load_yaml()` to avoid circular imports.
+
+## \[0.2.28\] - 2022-03-27
+
+### Added
+
+- `math` module - new home for math functions (currently `jd_to_mjd` and `mag_to_flux`)
+- `data_utils.load_yaml`
+
+### Changed
+
+- Move functions `jd_to_mjd` and `mag_to_flux` from `data_utils` to the new `math` module. Update
+  SuperNNova Cloud Function to accommodate (outside `broker_utils` at
+  broker/cloud_functions/classify_snn).
+- Update `schema_maps.load_schema_map` to accept a Path and load from a local file
+
+### Removed
+
+- `data_utils._get_schema_map`
+
+## \[0.2.27\] - 2022-03-26
+
+### Added<a name="added-1"></a>
+
+- `data_utils._get_schema_map` function to load schema map from file if needed and check that it
+  is a dict.
+- `data_utils.load_alert` to load an alert from file.
+- Support for passing `alert_avro` as a path to `data_utils.decode_alert`.
+
+### Changed<a name="changed-1"></a>
+
+- `data_utils` now passes `schema_map` and `drop_cutouts` as kwargs.
+
+### Fixed<a name="fixed-1"></a>
+
+- Rename variable `pgb_project_id` to `PROJECT_ID`.
+
+## \[0.2.26\] - 2022-03-26<a name="0226---2022-03-26"></a>
+
+### Added<a name="added"></a>
+
+- Add `project_id` as a kwarg to `gcp_utils` bigquery functions.
+
+### Changed<a name="changed"></a>
+
+- Set GCP project ID by an environment variable, fall back to the production broker project.
+  Formerly this was hardcoded to production project.
+- Unpin `numpy`
+- Unpin `google.cloud.bigquery`
+- Unpin `google.cloud.pubsub`. Update `consumer_sim` and `gcp_utils` to accommodate.

broker/broker_utils/MANIFEST.in

@@ -1 +1,3 @@
 include requirements.txt
+include requirements_pinned.txt
+include broker_utils/avro_schemas/*.avsc

broker/broker_utils/README.md

@@ -5,5 +5,6 @@
 
 Note that the schema maps are stored in a Cloud Storage bucket and are not packaged with this module.
 This has the following benefits:
+
 1. Schema maps can be updated easily by uploading new files to the bucket.
 2. Broker instances can use different schema maps; each instance loads them from their own `broker_files` bucket.