Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Debug #913

Closed
wants to merge 95 commits into from
Closed

Debug #913

wants to merge 95 commits into from

Conversation

merelcht
Copy link
Member

Description

Development notes

Checklist

  • Opened this PR as a 'Draft Pull Request' if it is work-in-progress
  • Updated the documentation to reflect the code changes
  • Added a description of this change in the relevant RELEASE.md file
  • Added tests to cover my changes
  • Received approvals from at least half of the TSC (required for adding a new, non-experimental dataset)

tdhooghe and others added 30 commits October 21, 2024 17:22
@tdhooghe
check and convert pd df to snowflake df
2305c3d 
Signed-off-by: tdhooghe <[email protected]>
@tdhooghe
add tests
07ef67b 
Signed-off-by: tdhooghe <[email protected]>
@tdhooghe
revert import to previous state
04928ba 
Signed-off-by: tdhooghe <[email protected]>
@tdhooghe
relax snowpark version
77551d6 
Signed-off-by: tdhooghe <[email protected]>
@tdhooghe
revert pyproject.toml
fbb3374 
Signed-off-by: tdhooghe <[email protected]>
@tdhooghe
bump snowpark version
fb621ae 
Signed-off-by: tdhooghe <[email protected]>
@tdhooghe
relax python version for snowpark
cb320bd 
Signed-off-by: tdhooghe <[email protected]>
@tdhooghe
downgrade snowpark to 1.22
628def6 
Signed-off-by: tdhooghe <[email protected]>
@tdhooghe
upgrade cloudpickle to 2.2.1
136fb09 
Signed-off-by: tdhooghe <[email protected]>
@deepyaman @tdhooghe
feat(datasets): create separate ibis.FileDataset (#842)
fe3bfbe 
* feat(datasets): create separate `ibis.FileDataset`

Signed-off-by: Deepyaman Datta <[email protected]>

* chore(datasets): deprecate `TableDataset` file I/O

Signed-off-by: Deepyaman Datta <[email protected]>

* feat(datasets): implement `FileDataset` versioning

Signed-off-by: Deepyaman Datta <[email protected]>

* chore(datasets): try `os.path.exists`, for Windows

Signed-off-by: Deepyaman Datta <[email protected]>

* revert(datasets): use pathlib, ignore Windows test

Refs: b7ff0c7

Signed-off-by: Deepyaman Datta <[email protected]>

* docs(datasets): add `ibis.FileDataset` to contents

Signed-off-by: Deepyaman Datta <[email protected]>

* chore(datasets): add docstring for `hashable` func

Signed-off-by: Deepyaman Datta <[email protected]>

* chore(datasets): add docstring for `hashable` func

Signed-off-by: Deepyaman Datta <[email protected]>

* feat(datasets)!: expose `load` and `save` publicly

Signed-off-by: Deepyaman Datta <[email protected]>

* chore(datasets): remove second filepath assignment

Signed-off-by: Deepyaman Datta <[email protected]>

---------

Signed-off-by: Deepyaman Datta <[email protected]>
Signed-off-by: tdhooghe <[email protected]>
@ankatiyar @tdhooghe
ci(docker): Update error code in e2e test (#888)
fce2974 
Update error code in e2e test

Signed-off-by: Ankita Katiyar <[email protected]>
Signed-off-by: tdhooghe <[email protected]>
@tdhooghe
update docstring
60b4a86 
Signed-off-by: tdhooghe <[email protected]>
@tdhooghe
test ci/cd without python version 3.12
3e578c6 
Signed-off-by: tdhooghe <[email protected]>
@tdhooghe
fix linting error
a161191 
Signed-off-by: tdhooghe <[email protected]>
@tdhooghe
add max python version for compatibility
e4509bc 
Signed-off-by: tdhooghe <[email protected]>
@tdhooghe
fix error
c98aef8 
Signed-off-by: tdhooghe <[email protected]>
@merelcht @deepyaman @tdhooghe
chore: Update PR template with checkbox for core dataset contribution (…
e4f1c3c 
…#891)

* Update PR template with checkbox for core dataset contribution

Signed-off-by: Merel Theisen <[email protected]>

* Update .github/PULL_REQUEST_TEMPLATE.md

Co-authored-by: Deepyaman Datta <[email protected]>
Signed-off-by: Merel Theisen <[email protected]>

* Fix lint

Signed-off-by: Merel Theisen <[email protected]>

---------

Signed-off-by: Merel Theisen <[email protected]>
Signed-off-by: Merel Theisen <[email protected]>
Co-authored-by: Deepyaman Datta <[email protected]>
Signed-off-by: tdhooghe <[email protected]>
@ankatiyar @tdhooghe
ci(datasets): Fix mypy errors (#893)
bcec735 
Signed-off-by: tdhooghe <[email protected]>
@deepyaman @tdhooghe
fix(datasets): default to DuckDB in in-memory mode (#897)
c243c6e 
* fix(datasets): default to DuckDB in in-memory mode

Signed-off-by: Deepyaman Datta <[email protected]>

* test(datasets): use `object()` sentinel as default

Signed-off-by: Deepyaman Datta <[email protected]>

* docs(datasets): add default database to RELEASE.md

Signed-off-by: Deepyaman Datta <[email protected]>

---------

Signed-off-by: Deepyaman Datta <[email protected]>
Signed-off-by: tdhooghe <[email protected]>
@deepyaman @tdhooghe
build(datasets): make Kedro-Datasets 5.1.0 release (#887)
388f455 
Signed-off-by: Deepyaman Datta <[email protected]>
Signed-off-by: tdhooghe <[email protected]>
@merelcht @tdhooghe
ci: Add GH action to check for TSC votes on core dataset changes (#896)
3c85b1f 
* Add GH action to check for TSC votes on core dataset changes
* Ignore TSC vote action in gatekeeper
* Trigger TSC vote action only on changes in core dataset

---------

Signed-off-by: Merel Theisen <[email protected]>
Signed-off-by: tdhooghe <[email protected]>
@tdhooghe
update testing framework and docstrings
101696c 
Signed-off-by: tdhooghe <[email protected]>
@tdhooghe
Update kedro-datasets/kedro_datasets/snowflake/snowpark_dataset.py
c1eeb77 
Signed-off-by: Thomas <[email protected]>
Signed-off-by: tdhooghe <[email protected]>
@tdhooghe
validate and check table name for nones
b9bf73e 
Signed-off-by: tdhooghe <[email protected]>
@tdhooghe
remove breakpoint
2c804c9 
Signed-off-by: tdhooghe <[email protected]>
@tdhooghe
change doc type hinting for tests to pass
7b7811d 
Signed-off-by: tdhooghe <[email protected]>
@deepyaman @tdhooghe
feat(datasets): create separate ibis.FileDataset (#842)
23b056a 
* feat(datasets): create separate `ibis.FileDataset`

Signed-off-by: Deepyaman Datta <[email protected]>

* chore(datasets): deprecate `TableDataset` file I/O

Signed-off-by: Deepyaman Datta <[email protected]>

* feat(datasets): implement `FileDataset` versioning

Signed-off-by: Deepyaman Datta <[email protected]>

* chore(datasets): try `os.path.exists`, for Windows

Signed-off-by: Deepyaman Datta <[email protected]>

* revert(datasets): use pathlib, ignore Windows test

Refs: b7ff0c7

Signed-off-by: Deepyaman Datta <[email protected]>

* docs(datasets): add `ibis.FileDataset` to contents

Signed-off-by: Deepyaman Datta <[email protected]>

* chore(datasets): add docstring for `hashable` func

Signed-off-by: Deepyaman Datta <[email protected]>

* chore(datasets): add docstring for `hashable` func

Signed-off-by: Deepyaman Datta <[email protected]>

* feat(datasets)!: expose `load` and `save` publicly

Signed-off-by: Deepyaman Datta <[email protected]>

* chore(datasets): remove second filepath assignment

Signed-off-by: Deepyaman Datta <[email protected]>

---------

Signed-off-by: Deepyaman Datta <[email protected]>
Signed-off-by: tdhooghe <[email protected]>
@ankatiyar @tdhooghe
ci(datasets): Fix mypy errors (#893)
27e58cd 
Signed-off-by: tdhooghe <[email protected]>
@tdhooghe
more strict type hinting and casting for linting
178f0f4 
Signed-off-by: tdhooghe <[email protected]>
@tdhooghe
add back python 3.12 to see if it is compatible
0133025 
Signed-off-by: tdhooghe <[email protected]>
tdhooghe and others added 28 commits October 23, 2024 09:18
@tdhooghe
revert matplotlib version bump
435159b 
Signed-off-by: tdhooghe <[email protected]>
@tdhooghe
exclude lines from coverage
ba6975c 
Signed-off-by: tdhooghe <[email protected]>
@tdhooghe
only test python 3.12
52ce490 
Signed-off-by: tdhooghe <[email protected]>
@tdhooghe
omit snowflake testing from python 3.12
71a3801 
Signed-off-by: tdhooghe <[email protected]>
@tdhooghe
add python version check
b89a552 
Signed-off-by: tdhooghe <[email protected]>
@tdhooghe
fix bug in ignoreopts and add back python versions 3.10 and 3.11 in m…
d967ff5 
…atrix

Signed-off-by: tdhooghe <[email protected]>
@tdhooghe
fix bug in ignoreopts and add back python versions 3.10 and 3.11 in m…
233b4f0 
…atrix

Signed-off-by: tdhooghe <[email protected]>
@tdhooghe
revert requires-python to 3.10
7349fac 
Signed-off-by: tdhooghe <[email protected]>
@tdhooghe
ignore entire folder
e64551e 
Signed-off-by: tdhooghe <[email protected]>
@tdhooghe
add back omit for snowflake as python 3.12 is not supported, hence ci…
ceacc8c 
…cd fails on coverage

Signed-off-by: tdhooghe <[email protected]>
@tdhooghe
revert makefile to original state
e0735ad 
Signed-off-by: tdhooghe <[email protected]>
@tdhooghe
remove commented out setters
d17b8a6 
Signed-off-by: tdhooghe <[email protected]>
@tdhooghe
remove commented out fixtures
c5c9e4d 
Signed-off-by: tdhooghe <[email protected]>
@tdhooghe
Merge branch 'main' into feature/save-pd-to-snowflaketable
fbc2545
@tdhooghe
remove commented out fixtures
67f24c0 
Signed-off-by: tdhooghe <[email protected]>
@tdhooghe
skip test if import is missing
be5fb6f 
Signed-off-by: tdhooghe <[email protected]>
@merelcht @tdhooghe
build: Bump matplotlib test dependency (#904)
217e60b 
* Bump matplotlib test dependency

Signed-off-by: Merel Theisen <[email protected]>

* Fix matplotlib test

Signed-off-by: Merel Theisen <[email protected]>

---------

Signed-off-by: Merel Theisen <[email protected]>
Signed-off-by: tdhooghe <[email protected]>
@tdhooghe
remove snowpark from omit
f288ab7 
Signed-off-by: tdhooghe <[email protected]>
@tdhooghe
change snowpark to snowflake
6a21d1f 
Signed-off-by: tdhooghe <[email protected]>
@tdhooghe
fix error in import or skip and add noqa for ruff
0e2abe3 
Signed-off-by: tdhooghe <[email protected]>
@tdhooghe
implement pr feedback to raise if data is not pandas or snowpark data…
c0c42a8 
…frame

Signed-off-by: tdhooghe <[email protected]>
@tdhooghe
add back omit for snowflake as python 3.12 is not supported, hence ci…
41dfa22 
…cd fails on coverage

Signed-off-by: tdhooghe <[email protected]>
@tdhooghe
add test for new raising error
0f3d838 
Signed-off-by: tdhooghe <[email protected]>
@tdhooghe
process final pr comments
2a688d8 
Signed-off-by: tdhooghe <[email protected]>
@merelcht
Merge branch 'main' into feature/save-pd-to-snowflaketable
a974010
@merelcht
Merge branch 'main' into feature/save-pd-to-snowflaketable
660a428
@merelcht
Merge branch 'feature/save-pd-to-snowflaketable' of https://github.co…
ecabf7d 
…m/tdhooghe/kedro-plugins into feature/save-pd-to-snowflaketable
@merelcht
Fix lint
af4eff7 
Signed-off-by: Merel Theisen <[email protected]>
@merelcht merelcht closed this Oct 28, 2024
@merelcht merelcht deleted the debug branch October 28, 2024 14:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@merelcht @tdhooghe @deepyaman @ankatiyar