Merge pull request #8 from singlestore-labs/modernize

Modernize the SQLAlchemy dialect. This includes adding many SingleStoreDB-specific features to the SQL reflection parser, support for vector data types, and additional singlestoredb keyword parameters to SQLAlchemy objects for SingleStoreDB-specific options.

Author: kesmit13

.github/workflows/coverage.yml

@@ -11,58 +11,54 @@ jobs:
   test-coverage:
     runs-on: ubuntu-latest
 
-    services:
-      singlestore:
-        image: ghcr.io/singlestore-labs/singlestoredb-dev:latest
-        ports:
-          - 3307:3306
-          - 8081:8080
-          - 9080:9080
-        env:
-          SINGLESTORE_LICENSE: ${{ secrets.SINGLESTORE_LICENSE }}
-          ROOT_PASSWORD: "root"
-
     steps:
       - uses: actions/checkout@v3
 
       - name: Set up Python
         uses: actions/setup-python@v4
         with:
-          python-version: "3.10"
+          python-version: "3.11"
           cache: "pip"
 
       - name: Install dependencies
         run: |
           python -m pip install --upgrade pip
           pip install -e ".[dev]"
 
-      - name: Install SingleStore package
+      - name: Install SingleStore package and SQLAlchemy 2.0.x
         run: |
+          pip install 'sqlalchemy~=2.0.0'
           pip install .
 
       - name: Run MySQL protocol tests
         run: |
           pytest -v --cov=sqlalchemy_singlestoredb --pyargs sqlalchemy_singlestoredb.tests
         env:
-          COVERAGE_FILE: "coverage-mysql.cov"
-          SINGLESTOREDB_URL: "root:[email protected]:3307"
+          COVERAGE_FILE: "coverage-mysql-2-0.cov"
           SINGLESTOREDB_MANAGEMENT_TOKEN: ${{ secrets.CLUSTER_API_KEY }}
 
       - name: Run HTTP protocol tests
         run: |
           pytest -v --cov=sqlalchemy_singlestoredb --pyargs sqlalchemy_singlestoredb.tests
         env:
           COVERAGE_FILE: "coverage-http.cov"
-          SINGLESTOREDB_URL: "http://root:[email protected]:9080"
           SINGLESTOREDB_MANAGEMENT_TOKEN: ${{ secrets.CLUSTER_API_KEY }}
-          # Can not change databases using HTTP API. The URL below will be
-          # used to create the database and the generated database name will
-          # be applied to the above URL.
-          SINGLESTOREDB_INIT_DB_URL: "root:[email protected]:3307"
+          USE_DATA_API: "1"
+
+      - name: Install SingleStore package and SQLAlchemy 1.4.x
+        run: |
+          pip install 'sqlalchemy~=1.4.0'
+
+      - name: Run MySQL protocol tests
+        run: |
+          pytest -v --cov=sqlalchemy_singlestoredb --pyargs sqlalchemy_singlestoredb.tests
+        env:
+          COVERAGE_FILE: "coverage-mysql-1-4.cov"
+          SINGLESTOREDB_MANAGEMENT_TOKEN: ${{ secrets.CLUSTER_API_KEY }}
 
       - name: Generate report
         run: |
-          coverage combine coverage-mysql.cov coverage-http.cov
+          coverage combine coverage-mysql-2-0.cov coverage-mysql-1-4.cov
           coverage report
           coverage xml
           coverage html

.github/workflows/publish.yml

@@ -34,16 +34,16 @@ jobs:
 
     runs-on: ubuntu-latest
 
-    services:
-      singlestore:
-        image: ghcr.io/singlestore-labs/singlestoredb-dev:latest
-        ports:
-          - 3307:3306
-          - 8081:8080
-          - 9080:9000
-        env:
-          SINGLESTORE_LICENSE: ${{ secrets.SINGLESTORE_LICENSE }}
-          ROOT_PASSWORD: "root"
+#   services:
+#     singlestore:
+#       image: ghcr.io/singlestore-labs/singlestoredb-dev:latest
+#       ports:
+#         - 3307:3306
+#         - 8081:8080
+#         - 9080:9000
+#       env:
+#         SINGLESTORE_LICENSE: ${{ secrets.SINGLESTORE_LICENSE }}
+#         ROOT_PASSWORD: "root"
 
     steps:
       - uses: actions/checkout@v3

.gitignore

@@ -1,4 +1,9 @@
 ## Editor temporary/working/backup files
+.hypothesis
+.serena
+.claude
+CLAUDE.md
+dev-docs
 .#*
 *\#*\#
 [#]*#
@@ -57,6 +62,7 @@ dist
 # wheel files
 *.whl
 **/wheelhouse/*
+**/.hypothesis
 # coverage
 .coverage
 swat.egg-info/
@@ -73,6 +79,20 @@ ehthumbs.db
 Icon?
 Thumbs.db
 
+**/CLAUDE.md
+.claude
+.serena
+.vscode
+.venv
+**/.hypothesis
+dev-docs
+**/.ipynb_checkpoints
+**/.benchmarks
+*.ipynb
+certs
+**/*.prof
+**/*.pprof
+
 ## Documentation generated files
 docs/src/_build/doctrees
 docs/src/generated

CONTRIBUTING.md

@@ -31,16 +31,64 @@ pip install -e ".[dev]"
 
 This installs the package in editable mode along with all testing dependencies (pytest, pytest-cov, coverage).
 
-The tests must be run in multiple modes: standard protocol and Data API (HTTP).
-If you have Docker installed, you can simply run the following:
+#### Test Connection Methods
+
+The test infrastructure supports three connection methods:
+
+**1. Docker Auto-Start (Recommended for Development)**
+
+If Docker is installed and running, the test fixtures will automatically start a SingleStoreDB container:
+```bash
+# Docker container will auto-start, run tests, then stop
+pytest -v sqlalchemy_singlestoredb/tests
+```
+
+**2. Using an Existing Server**
+
+Set `SINGLESTOREDB_URL` to use an existing SingleStoreDB server instead of Docker:
+```bash
+# Direct connection using MySQL protocol
+SINGLESTOREDB_URL=user:[email protected]:3306 pytest -v sqlalchemy_singlestoredb/tests
+
+# Data API connection using HTTP protocol
+SINGLESTOREDB_INIT_DB_URL=user:[email protected]:3306 \
+    SINGLESTOREDB_URL=http://user:[email protected]:8090 \
+    pytest -v sqlalchemy_singlestoredb/tests
 ```
+
+#### Required Test Modes
+
+The tests must be run in **two connection modes** to ensure compatibility:
+
+**1. Direct Connection (MySQL Protocol)** - Default mode:
+```bash
+# With Docker auto-start
 pytest -v sqlalchemy_singlestoredb/tests
-USE_DATA_API=1 pytest -v sqlalchemy_singlestoredb/tests
+
+# With existing server
+SINGLESTOREDB_URL=user:pw@host:port pytest -v sqlalchemy_singlestoredb/tests
 ```
 
-You can also specify a SingleStoreDB server if you need to run against a
-specific server type:
+**2. Data API (HTTP Protocol)** - Use `USE_DATA_API=1`:
+```bash
+# With Docker auto-start (will use HTTP Data API)
+USE_DATA_API=1 pytest -v sqlalchemy_singlestoredb/tests
+
+# With existing Data API server
+SINGLESTOREDB_INIT_DB_URL=user:[email protected]:3306 \
+    SINGLESTOREDB_URL=http://user:pw@host:port \
+    pytest -v sqlalchemy_singlestoredb/tests
 ```
-SINGLESTOREDB_URL=user:[email protected]:3306 pytest -v sqlalchemy_singlestoredb/tests
-SINGLESTOREDB_URL=http://user:[email protected]:8090 pytest -v sqlalchemy_singlestoredb/tests
+
+#### Test Output and Logging
+
+The test fixtures include detailed logging that is **enabled by default**, showing:
+- Which connection method is being used (Docker vs existing server)
+- API type (Data API/HTTP vs Direct/MySQL protocol)
+- Connection URLs
+- Database creation/cleanup operations
+
+This logging is configured in `pyproject.toml` and appears automatically when running tests. To disable it, use:
+```bash
+pytest --log-cli-level=WARNING sqlalchemy_singlestoredb/tests
 ```

PLANNING.md

@@ -0,0 +1,22 @@
+# SingleStoreDB SQLAlchemy dialect
+
+This project is an SQLAlchemy dialect for the SingleStoreDB database. The goal
+is to support as many of the SingleStoreDB features in the SQLAlchemy
+framework. The SingleStoreDB dialect is based on the MySQL dialect that is
+part of SQLAlchemy. SingleStoreDB uses the same protocol as MySQL, but does
+add a number of new features not supported by MySQL. Those need to be added
+to the SingleStoreDB SQLAlchemy dialect.
+
+## Resources
+
+- SingleStoreDB docs - https://docs.singlestore.com/
+- SQLAlchemy project web page - https://www.sqlalchemy.org/
+- SQLAlchemy project github repo - https://github.com/sqlalchemy/sqlalchemy/
+- SingleStoreDB Python SDK repo - https://github.com/singlestore-labs/singlestoredb-python/
+- Python language - https://python.org/doc/
+
+
+## Requirements
+
+- Still requires SQLAlchemy v1.4 compatilibity
+- Must work for Python 3.9 or greater