Version 0.2.0

.env.template

@@ -0,0 +1,2 @@
+SYNAPSE_AUTH_TOKEN=
+LOG_LEVEL=DEBUG

.gitignore

@@ -21,6 +21,9 @@ var/
 .venv
 .vscode
 .coverage
+.coverage.*
+.tox
 htmlcov
 tests/private.*
 *.log
+.env

.travis.yml

@@ -1,6 +1,7 @@
 language: python
 python:
-  - "3.7"
+  - "3.10"
+  - "3.11"
 install:
   - pip install pipenv
   - pipenv install --dev

CHANGELOG.md

@@ -1,5 +1,14 @@
 # Change Log
 
+## Version 0.2.0 (2023-11-07)
+
+### Changes
+
+- Only supports Python 3.10+.
+- Refactor code to improve speed.
+- Added `sync-from-synapse` option.
+- Changed command line arguments.
+
 ## Version 0.0.6 (2021-07-28)
 
 ### Added

Makefile

@@ -27,16 +27,6 @@ install_local:
 	pip install -e .
 
 
-.PHONY: install_test
-install_test:
-	pip install --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple synapse-downloader
-
-
-.PHONY: publish_test
-publish_test: build
-	twine upload --repository-url https://test.pypi.org/legacy/ dist/*
-
-
 .PHONY: publish
 publish: build
 	twine upload dist/*

Pipfile

@@ -12,14 +12,16 @@ coverage = "*"
 coveralls = "*"
 pytest-mock = "*"
 pytest-pylint = "*"
-pylint = "*"
+pylint = "<3.0.0"
 wheel = "*"
 autopep8 = "*"
+tox = "*"
+python-dotenv = "*"
+synapse_test_helper = ">=0.0.3"
 
 [packages]
-synapseclient = "*"
-aiohttp = ">=3.7.4"
-aiofiles = "*"
+synapseclient = ">=2.3.1,<3.0.0"
+synapsis = ">=0.0.7"
 
 [requires]
-python_version = "3.7"
+python_version = "3.10"

README.md

@@ -7,8 +7,8 @@ Utility for downloading large datasets from Synapse.
 
 ## Dependencies
 
-- [Python3.7](https://www.python.org/)
-- A [Synapse](https://www.synapse.org/) account with a username/password. Authentication through a 3rd party (.e.g., Google) will not work, you must have a Synapse user/pass for the [API to authenticate](http://docs.synapse.org/python/#connecting-to-synapse).
+- [Python3.10+](https://www.python.org/)
+- A [Synapse](https://www.synapse.org/) account with an auth token.
 
 ## Install
 
@@ -18,58 +18,161 @@ pip install synapse-downloader
 
 ## Configuration
 
-Your Synapse credential can be provided on the command line (`--username`, `--password`) or via environment variables.
+### Environment Variables
 
-```bash
-SYNAPSE_USERNAME=your-synapse-username
-SYNAPSE_PASSWORD=your-synapse-password
+No configuration is necessary if using environment variables or the default synapse config file.
+
+For user/pass, set:
+
+```shell
+SYNAPSE_USERNAME=
+SYNAPSE_PASSWORD=
+```
+
+For auth token, set:
+
+```shell
+SYNAPSE_AUTH_TOKEN=
+```
+
+For Synapse Config file:
+
+Have a valid config file in: `~/.synapseConfig`
+
+Or, have the environment variable set: `SYNAPSE_CONFIG_FILE=`
+
+### Command Line Arguments
+
+```text
+options:
+  -u USERNAME, --username USERNAME
+                        Synapse username.
+  -p PASSWORD, --password PASSWORD
+                        Synapse password.
+  --auth-token AUTH_TOKEN
+                        Synapse auth token.
+  --synapse-config SYNAPSE_CONFIG
+                        Path to Synapse configuration file.
 ```
 
 ## Usage
 
 ```text
-usage: synapse-downloader [-h] [-e [EXCLUDE]] [-u USERNAME] [-p PASSWORD]
-                          [-ll LOG_LEVEL] [-ld LOG_DIR] [-dt DOWNLOAD_TIMEOUT]
-                          [-w] [-wc] [-c] [-ci [COMPARE_IGNORE]]
-                          entity-id download-path
+usage: synapse-downloader [-h] [--version] {download,compare,sync-from-synapse} ...
+
+Synapse Downloader
+
+options:
+  -h, --help            show this help message and exit
+  --version             show program's version number and exit
+
+Commands:
+  {download,compare,sync-from-synapse}
+    download            Download items from Synapse to a local directory. Default command.
+    compare             Compare items in Synapse to a local directory.
+    sync-from-synapse   Download items from Synapse to a local directory using the syncFromSynapse method.
+```
+
+### Download
+
+```text
+usage: synapse-downloader download [-h] [-u USERNAME] [-p PASSWORD] [--auth-token AUTH_TOKEN]
+                                   [--synapse-config SYNAPSE_CONFIG] [-ll LOG_LEVEL] [-ld LOG_DIR] [-e [EXCLUDE]] [-wc]
+                                   entity-id local-path
+
+positional arguments:
+  entity-id             The ID of the Synapse entity to download (Project, Folder or File).
+  local-path            The local path to save the files to.
+
+options:
+  -h, --help            show this help message and exit
+  -u USERNAME, --username USERNAME
+                        Synapse username.
+  -p PASSWORD, --password PASSWORD
+                        Synapse password.
+  --auth-token AUTH_TOKEN
+                        Synapse auth token.
+  --synapse-config SYNAPSE_CONFIG
+                        Path to Synapse configuration file.
+  -ll LOG_LEVEL, --log-level LOG_LEVEL
+                        Set the logging level.
+  -ld LOG_DIR, --log-dir LOG_DIR
+                        Set the directory where the log file will be written.
+  -e [EXCLUDE], --exclude [EXCLUDE]
+                        Items to exclude from download. Synapse IDs, names, or filenames (names are case-sensitive).
+  -wc, --with-compare   Run compare after downloading everything.
+
+```
+
+### Compare
+
+```text
+usage: synapse-downloader compare [-h] [-u USERNAME] [-p PASSWORD] [--auth-token AUTH_TOKEN] [--synapse-config SYNAPSE_CONFIG]
+                                  [-ll LOG_LEVEL] [-ld LOG_DIR] [-e [EXCLUDE]]
+                                  entity-id local-path
 
 positional arguments:
-  entity-id             The ID of the Synapse entity to download or compare
-                        (Project, Folder or File).
-  download-path         The local path to save the files to or to compare.
+  entity-id             The ID of the Synapse entity to compare (Project, Folder or File).
+  local-path            The local path to compare.
 
-optional arguments:
+options:
   -h, --help            show this help message and exit
+  -u USERNAME, --username USERNAME
+                        Synapse username.
+  -p PASSWORD, --password PASSWORD
+                        Synapse password.
+  --auth-token AUTH_TOKEN
+                        Synapse auth token.
+  --synapse-config SYNAPSE_CONFIG
+                        Path to Synapse configuration file.
+  -ll LOG_LEVEL, --log-level LOG_LEVEL
+                        Set the logging level.
+  -ld LOG_DIR, --log-dir LOG_DIR
+                        Set the directory where the log file will be written.
   -e [EXCLUDE], --exclude [EXCLUDE]
-                        Items to exclude from download. Synapse IDs or names
-                        (names are case-sensitive).
+                        Items to exclude from compare. Synapse IDs, names, or filenames (names are case-sensitive).
+```
+
+### Sync From Synapse
+
+```text
+usage: synapse-downloader sync-from-synapse [-h] [-u USERNAME] [-p PASSWORD] [--auth-token AUTH_TOKEN]
+                                            [--synapse-config SYNAPSE_CONFIG] [-ll LOG_LEVEL] [-ld LOG_DIR]
+                                            entity-id local-path
+
+positional arguments:
+  entity-id             The ID of the Synapse entity to download (Project, Folder or File).
+  local-path            The local path to save the files to.
+
+options:
+  -h, --help            show this help message and exit
   -u USERNAME, --username USERNAME
                         Synapse username.
   -p PASSWORD, --password PASSWORD
                         Synapse password.
+  --auth-token AUTH_TOKEN
+                        Synapse auth token.
+  --synapse-config SYNAPSE_CONFIG
+                        Path to Synapse configuration file.
   -ll LOG_LEVEL, --log-level LOG_LEVEL
                         Set the logging level.
   -ld LOG_DIR, --log-dir LOG_DIR
                         Set the directory where the log file will be written.
-  -dt DOWNLOAD_TIMEOUT, --download-timeout DOWNLOAD_TIMEOUT
-                        Set the maximum time (in seconds) a file can download
-                        before it is canceled.
-  -w, --with-view       Use an entity view for loading file info. Fastest for
-                        large projects.
-  -wc, --with-compare   Run the comparison after downloading everything.
-  -c, --compare         Compare a local directory against a remote project or
-                        folder.
-  -ci [COMPARE_IGNORE], --compare-ignore [COMPARE_IGNORE]
-                        Path to directories or files to ignore when comparing.
 ```
 
 ## Development Setup
 
 ```bash
-pipenv --three
+pipenv --python 3.10
 pipenv shell
 make pip_install
 make build
 make install_local
 ```
+
 See [Makefile](Makefile) for all commands.
+
+Run tests:
+
+1. Rename `.env.template` to `.env` and set the variables in the file.
+2. Run `make test` or `tox`

pytest.ini

@@ -0,0 +1,6 @@
+[pytest]
+asyncio_mode = auto
+pythonpath = src
+testpaths =
+    tests
+

setup.py

@@ -7,8 +7,6 @@
 setuptools.setup(
     name="synapse-downloader",
     version=__version__,
-    author="Patrick Stout",
-    author_email="[email protected]",
     license="Apache2",
     description="Utility for downloading large datasets from Synapse.",
     long_description=long_description,
@@ -18,7 +16,8 @@
     packages=setuptools.find_packages(where="src"),
     classifiers=(
         "Programming Language :: Python",
-        "Programming Language :: Python :: 3.7",
+        "Programming Language :: Python :: 3.10",
+        "Programming Language :: Python :: 3.11",
         "License :: OSI Approved :: Apache Software License",
         "Operating System :: OS Independent",
     ),
@@ -28,8 +27,7 @@
         ]
     },
     install_requires=[
-        "synapseclient>=2.1.0,<3.0.0",
-        "aiohttp>=3.7.4",
-        "aiofiles"
+        "synapseclient>=2.3.1,<3.0.0",
+        "synapsis>=0.0.7"
     ]
 )

src/synapse_downloader/_version.py

@@ -1 +1 @@
-__version__ = '0.0.6'
+__version__ = '0.2.0'