Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Snow 1065172 gh workflow optimization freeze #481

Closed
wants to merge 19 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
95e86c4
SNOW-1065172-gh-workflow-optimization: update gh actions versions to …
sfc-gh-mraba Mar 12, 2024
c1c208f
SNOW-1065172-gh-workflow-optimization: run linting without tox
sfc-gh-mraba Mar 12, 2024
2eca636
SNOW-1065172-gh-workflow-optimization: move project definition to pyp…
sfc-gh-mraba Mar 13, 2024
1b93062
SNOW-1065172-gh-workflow-optimization: fix typo in os matrix
sfc-gh-mraba Mar 13, 2024
dc8e517
SNOW-1065172-gh-workflow-optimization: fix typo in os matrix
sfc-gh-mraba Mar 13, 2024
232fd11
SNOW-1065172-gh-workflow-optimization: remove displaying python version
sfc-gh-mraba Mar 13, 2024
8af09f3
SNOW-1065172-gh-workflow-optimization: unique names for coverage repo…
sfc-gh-mraba Mar 13, 2024
b123105
SNOW-1065172-gh-workflow-optimization: update checkout rules
sfc-gh-mraba Mar 13, 2024
db1fc2e
SNOW-1065172-gh-workflow-optimization: full tests jobs
sfc-gh-mraba Mar 13, 2024
4616f5c
SNOW-1065172-gh-workflow-optimization: markers added
sfc-gh-mraba Mar 13, 2024
9e3549d
SNOW-1065172-gh-workflow-optimization: gh-cache-sum script
sfc-gh-mraba Mar 13, 2024
2f7007e
SNOW-1065172-gh-workflow-optimization: dependencies order
sfc-gh-mraba Mar 13, 2024
c249291
SNOW-1065172-gh-workflow-optimization: fix VERSIONS parsing after mig…
sfc-gh-mraba Mar 13, 2024
17c3762
SNOW-1065172-gh-workflow-optimization: test_pandas_rewrite mark timeo…
sfc-gh-mraba Mar 14, 2024
07b7fb9
Added engine with numpy fixture
sfc-gh-astus Mar 14, 2024
4b69f5a
SNOW-1065172-gh-workflow-optimization: replace table if exists test_p…
sfc-gh-mraba Mar 14, 2024
328d193
Run single test
sfc-gh-astus Mar 14, 2024
2285829
Run single test
sfc-gh-astus Mar 14, 2024
8ee415f
Run single test
sfc-gh-astus Mar 14, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
248 changes: 143 additions & 105 deletions .github/workflows/build_test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -24,122 +24,160 @@ jobs:
name: Check linting
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- uses: actions/checkout@v4
with:
persist-credentials: false
- name: Set up Python
uses: actions/setup-python@v2
uses: actions/setup-python@v5
with:
python-version: '3.8'
- name: Display Python version
run: python -c "import sys; import os; print(\"\n\".join(os.environ[\"PATH\"].split(os.pathsep))); print(sys.version); print(sys.executable);"
- name: Upgrade setuptools, pip and wheel
run: python -m pip install -U setuptools pip wheel
- name: Install tox
run: python -m pip install tox
- name: Upgrade and install tools
run: |
python -m pip install -U pip
python -m pip install -U hatch
python -m hatch env create default
python -m hatch env find
- name: Set PY
run: echo "PY=$(python -VV | sha256sum | cut -d' ' -f1)" >> $GITHUB_ENV
- uses: actions/cache@v1
run: echo "PY=$(hatch run gh-cache-sum)" >> $GITHUB_ENV
- uses: actions/cache@v4
with:
path: ~/.cache/pre-commit
key: pre-commit|${{ env.PY }}|${{ hashFiles('.pre-commit-config.yaml') }}
- name: Run fix_lint
run: python -m tox -e fix_lint

test:
name: Test ${{ matrix.os.download_name }}-${{ matrix.python-version }}-${{ matrix.cloud-provider }}
needs: lint
runs-on: ${{ matrix.os.image_name }}
strategy:
fail-fast: false
matrix:
os:
- image_name: ubuntu-latest
download_name: manylinux_x86_64
- image_name: macos-latest
download_name: macosx_x86_64
- image_name: windows-2019
download_name: win_amd64
python-version: ["3.8"]
cloud-provider: [aws, azure, gcp]
steps:
- uses: actions/checkout@v2
- name: Set up Python
uses: actions/setup-python@v2
with:
python-version: ${{ matrix.python-version }}
- name: Display Python version
run: python -c "import sys; print(sys.version)"
- name: Setup parameters file
shell: bash
env:
PARAMETERS_SECRET: ${{ secrets.PARAMETERS_SECRET }}
run: |
gpg --quiet --batch --yes --decrypt --passphrase="$PARAMETERS_SECRET" \
.github/workflows/parameters/parameters_${{ matrix.cloud-provider }}.py.gpg > tests/parameters.py
- name: Upgrade setuptools, pip and wheel
run: python -m pip install -U setuptools pip wheel
- name: Install tox
run: python -m pip install tox
- name: List installed packages
run: python -m pip freeze
- name: Run tests
run: python -m tox -e "py${PYTHON_VERSION/\./}" --skip-missing-interpreters false
env:
PYTHON_VERSION: ${{ matrix.python-version }}
PYTEST_ADDOPTS: -vvv --color=yes --tb=short
- name: Combine coverages
run: python -m tox -e coverage --skip-missing-interpreters false
shell: bash
- uses: actions/upload-artifact@v2
with:
name: coverage_${{ matrix.os.download_name }}-${{ matrix.python-version }}-${{ matrix.cloud-provider }}
path: |
.tox/.coverage
.tox/coverage.xml
- name: Run lint checks
run: hatch run check

combine-coverage:
if: ${{ success() || failure() }}
name: Combine coverage
needs: [test]
runs-on: ubuntu-latest
test-dialect:
name: Test dialect ${{ matrix.os }}-${{ matrix.python-version }}-${{ matrix.cloud-provider }}
# needs: lint
runs-on: ${{ matrix.os }}
strategy:
fail-fast: false
matrix:
os: [
macos-latest,
]
python-version: ["3.8"]
cloud-provider: [
gcp
]
steps:
- uses: actions/checkout@v2
- uses: actions/download-artifact@v2
- uses: actions/checkout@v4
with:
path: artifacts
persist-credentials: false
- name: Set up Python
uses: actions/setup-python@v2
uses: actions/setup-python@v5
with:
python-version: '3.8'
- name: Display Python version
run: python -c "import sys; print(sys.version)"
- name: Upgrade setuptools and pip
run: python -m pip install -U setuptools pip wheel
- name: Install tox
run: python -m pip install tox
- name: Collect all coverages to one dir
python-version: ${{ matrix.python-version }}
- name: Upgrade pip and prepare environment
run: |
python -c '
from pathlib import Path
import shutil
src_dir = Path("artifacts")
dst_dir = Path(".") / ".tox"
dst_dir.mkdir()
for src_file in src_dir.glob("*/.coverage"):
dst_file = dst_dir / f".coverage.{src_file.parent.name[9:]}"
print(f"{src_file} copy to {dst_file}")
shutil.copy(str(src_file), str(dst_file))'
- name: Combine coverages
run: python -m tox -e coverage
- name: Publish html coverage
uses: actions/upload-artifact@v2
with:
name: overall_cov_html
path: .tox/htmlcov
- name: Publish xml coverage
uses: actions/upload-artifact@v2
with:
name: overall_cov_xml
path: .tox/coverage.xml
- uses: codecov/codecov-action@v1
python -m pip install -U pip
python -m pip install -U hatch
python -m hatch env create default
- name: Setup parameters file
shell: bash
env:
PARAMETERS_SECRET: ${{ secrets.PARAMETERS_SECRET }}
run: |
gpg --quiet --batch --yes --decrypt --passphrase="$PARAMETERS_SECRET" \
.github/workflows/parameters/parameters_${{ matrix.cloud-provider }}.py.gpg > tests/parameters.py
- name: Run tests
run: hatch run test-dialect
- uses: actions/upload-artifact@v4
with:
file: .tox/coverage.xml
name: coverage.xml_dialect-${{ matrix.os }}-${{ matrix.python-version }}-${{ matrix.cloud-provider }}
path: |
./coverage.xml

# test-dialect-compatibility:
# name: Test dialect compatibility ${{ matrix.os }}-${{ matrix.python-version }}-${{ matrix.cloud-provider }}
# # needs: test-dialect
# needs: lint
# runs-on: ${{ matrix.os }}
# strategy:
# fail-fast: false
# matrix:
# os: [
# ubuntu-latest,
# macos-latest,
# windows-latest,
# ]
# python-version: ["3.8"]
# cloud-provider: [
# aws,
# azure,
# gcp,
# ]
# steps:
# - uses: actions/checkout@v4
# with:
# persist-credentials: false
# - name: Set up Python
# uses: actions/setup-python@v5
# with:
# python-version: ${{ matrix.python-version }}
# - name: Upgrade pip and install hatch
# run: |
# python -m pip install -U pip
# python -m pip install -U hatch
# python -m hatch env create default
# - name: Setup parameters file
# shell: bash
# env:
# PARAMETERS_SECRET: ${{ secrets.PARAMETERS_SECRET }}
# run: |
# gpg --quiet --batch --yes --decrypt --passphrase="$PARAMETERS_SECRET" \
# .github/workflows/parameters/parameters_${{ matrix.cloud-provider }}.py.gpg > tests/parameters.py
# - name: Run tests
# run: hatch run test-dialect-compatibility
# - uses: actions/upload-artifact@v4
# with:
# name: coverage.xml_dialect-compatibility-${{ matrix.os }}-${{ matrix.python-version }}-${{ matrix.cloud-provider }}
# path: |
# ./coverage.xml

# test-dialect-run-v20:
# name: Test dialect run v20 ${{ matrix.os }}-${{ matrix.python-version }}-${{ matrix.cloud-provider }}
# # needs: test-dialect
# needs: lint
# runs-on: ${{ matrix.os }}
# strategy:
# fail-fast: false
# matrix:
# os: [
# ubuntu-latest,
# macos-latest,
# windows-latest,
# ]
# python-version: ["3.8"]
# cloud-provider: [
# aws,
# azure,
# gcp,
# ]
# steps:
# - uses: actions/checkout@v4
# with:
# persist-credentials: false
# - name: Set up Python
# uses: actions/setup-python@v5
# with:
# python-version: ${{ matrix.python-version }}
# - name: Setup parameters file
# shell: bash
# env:
# PARAMETERS_SECRET: ${{ secrets.PARAMETERS_SECRET }}
# run: |
# gpg --quiet --batch --yes --decrypt --passphrase="$PARAMETERS_SECRET" \
# .github/workflows/parameters/parameters_${{ matrix.cloud-provider }}.py.gpg > tests/parameters.py
# - name: Upgrade pip and install hatch
# run: |
# python -m pip install -U pip
# python -m pip install -U hatch
# python -m hatch env create default
# - name: Run tests
# run: hatch run test-run_v20
# - uses: actions/upload-artifact@v4
# with:
# name: coverage.xml_dialect-run-20-${{ matrix.os }}-${{ matrix.python-version }}-${{ matrix.cloud-provider }}
# path: |
# ./coverage.xml
114 changes: 114 additions & 0 deletions pyproject.toml
Original file line number Diff line number Diff line change
@@ -1,5 +1,119 @@
[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"

[project]
name = "snowflake-sqlalchemy"
dynamic = ["version"]
description = "Snowflake SQLAlchemy Dialect"
readme = "README.md"
license = "Apache-2.0"
requires-python = ">=3.8"
authors = [
{ name = "Snowflake Inc.", email = "[email protected]" },
]
keywords = ["Snowflake", "analytics", "cloud", "database", "db", "warehouse"]
classifiers = [
"Development Status :: 5 - Production/Stable",
"Environment :: Console",
"Environment :: Other Environment",
"Intended Audience :: Developers",
"Intended Audience :: Education",
"Intended Audience :: Information Technology",
"Intended Audience :: System Administrators",
"License :: OSI Approved :: Apache Software License",
"Operating System :: OS Independent",
"Programming Language :: Python :: 3",
"Programming Language :: Python :: 3 :: Only",
"Programming Language :: Python :: 3.8",
"Programming Language :: Python :: 3.9",
"Programming Language :: Python :: 3.10",
"Programming Language :: Python :: 3.11",
"Programming Language :: Python :: 3.12",
"Programming Language :: SQL",
"Topic :: Database",
"Topic :: Scientific/Engineering :: Information Analysis",
"Topic :: Software Development",
"Topic :: Software Development :: Libraries",
"Topic :: Software Development :: Libraries :: Application Frameworks",
"Topic :: Software Development :: Libraries :: Python Modules",
]
dependencies = ["snowflake-connector-python", "SQLAlchemy"]

[tool.hatch.version]
path = "src/snowflake/sqlalchemy/version.py"

[project.optional-dependencies]
development = [
"pre-commit",
"pytest",
"pytest-cov",
"pytest-timeout",
"pytest-rerunfailures",
"pytz",
"numpy",
"mock",
]
pandas = ["snowflake-connector-python[pandas]"]

[project.entry-points."sqlalchemy.dialects"]
snowflake = "snowflake.sqlalchemy:dialect"

[project.urls]
Changelog = "https://github.com/snowflakedb/snowflake-sqlalchemy/blob/main/DESCRIPTION.md"
Documentation = "https://docs.snowflake.com/en/user-guide/sqlalchemy.html"
Homepage = "https://www.snowflake.com/"
Issues = "https://github.com/snowflakedb/snowflake-sqlalchemy/issues"
Source = "https://github.com/snowflakedb/snowflake-sqlalchemy"

[tool.hatch.build.targets.sdist]
exclude = ["/.github"]

[tool.hatch.build.targets.wheel]
packages = ["src/snowflake"]

[tool.hatch.envs.default]
extra-dependencies = ["SQLAlchemy<2.0.0,>=1.4.19"]
features = ["development", "pandas"]
python = "3.8"

[tool.hatch.envs.sa20]
extra-dependencies = ["SQLAlchemy>=2.0.0"]
python = "3.8"

[tool.hatch.envs.default.env-vars]
COVERAGE_FILE = "coverage.xml"
SQLACHEMY_WARN_20 = "1"

[tool.hatch.envs.default.scripts]
check = "pre-commit run --all-files"
test-dialect = "pytest -ra -vvv --tb=short --cov snowflake.sqlalchemy --cov-append --junitxml ./junit.xml --ignore=tests/sqlalchemy_test_suite"
test-dialect-compatibility = "pytest -ra -vvv --tb=short --cov snowflake.sqlalchemy --cov-append --junitxml ./junit.xml tests/sqlalchemy_test_suite"
test-run_v20 = "pytest -ra -vvv --tb=short --cov snowflake.sqlalchemy --cov-append --junitxml ./junit.xml --ignore=tests/sqlalchemy_test_suite --run_v20_sqlalchemy"
gh-cache-sum = "python -VV | sha256sum | cut -d' ' -f1"

[tool.ruff]
line-length = 88

[tool.black]
line-length = 88

[tool.pytest.ini_options]
markers = [
# Optional dependency groups markers
"lambda: AWS lambda tests",
"pandas: tests for pandas integration",
"sso: tests for sso optional dependency integration",
# Cloud provider markers
"aws: tests for Amazon Cloud storage",
"azure: tests for Azure Cloud storage",
"gcp: tests for Google Cloud storage",
# Test type markers
"integ: integration tests",
"unit: unit tests",
"skipolddriver: skip for old driver tests",
# Other markers
"timeout: tests that need a timeout time",
"internal: tests that could but should only run on our internal CI",
"external: tests that could but should only run on our external CI",
]
Loading
Loading