Skip to content

Commit

Permalink
merge
Browse files Browse the repository at this point in the history
  • Loading branch information
@sfc-gh-aalam
sfc-gh-aalam committed Oct 4, 2024
2 parents 54fe228 + 39a07d4 commit 1e4dcad
Show file tree
Hide file tree
Showing 398 changed files with 12,747 additions and 12,979 deletions.
41 changes: 39 additions & 2 deletions CHANGELOG.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,23 +2,60 @@

## 1.23.0 (TBD)

### Snowpark pandas API Updates
### Snowpark Python API Updates

#### New Features

- Added the following new functions in `snowflake.snowpark.functions`:
- `make_interval`
- Added support for using Snowflake Interval constants with `Window.range_between()` when the order by column is TIMESTAMP or DATE type.
- Added support for file writes. This feature is currently in private preview.
- Added support for `DataFrameGroupBy.fillna` and `SeriesGroupBy.fillna`.
- Added support for constructing `Series` and `DataFrame` objects with the lazy `Index` object as `data`, `index`, and `columns` arguments.
- Added support for constructing `Series` and `DataFrame` objects with `index` and `column` values not present in `DataFrame`/`Series` `data`.
- Added `thread_id` to `QueryRecord` to track the thread id submitting the query history.
-
#### Improvements

- Improved `to_pandas` to persist the original timezone offset for TIMESTAMP_TZ type.
#### Bug Fixes

### Snowpark pandas API Updates

#### New Features

- Added support for `TimedeltaIndex.mean` method.
- Added support for some cases of aggregating `Timedelta` columns on `axis=0` with `agg` or `aggregate`.
- Added support for `by`, `left_by`, `right_by`, `left_index`, and `right_index` for `pd.merge_asof`.
- Added support for passing parameter `include_describe` to `Session.query_history`.
- Added support for `DatetimeIndex.mean` and `DatetimeIndex.std` methods.
- Added support for `Resampler.asfreq`.
- Added support for `resample` frequency `W`, `ME`, `YE` with `closed = "left"`.
- Added support for `DataFrame.rolling.corr` and `Series.rolling.corr` for `pairwise = False` and int `window`.
- Added support for string time-based `window` and `min_periods = None` for `Rolling`.
- Added support for `pd.read_sas` (Uses native pandas for processing).
- Added suppport for applying `rolling().count()` and `expanding().count()` to `Timedelta` series and columns.
- Added support for `tz` in both `pd.date_range` and `pd.bdate_range`.

#### Improvements

- Improved `to_pandas` to persist the original timezone offset for TIMESTAMP_TZ type.
- Improved `dtype` results for TIMESTAMP_TZ type to show correct timezone offset.
- Improved `dtype` results for TIMESTAMP_LTZ type to show correct timezone.
- Improved error message when passing non-bool value to `numeric_only` for groupby aggregations.
- Removed unnecessary warning about sort algorithm in `sort_values`.
- Use SCOPED object for internal create temp tables. The SCOPED objects will be stored sproc scoped if created within stored sproc, otherwise will be session scoped, and the object will be automatically cleaned at the end of the scope.
- Improved warning messages for operations that lead to materialization with inadvertent slowness.
- Removed unnecessary warning message about `convert_dtype` in `Series.apply`.

#### Bug Fixes

- Fixed a bug where an `Index` object created from a `Series`/`DataFrame` incorrectly updates the `Series`/`DataFrame`'s index name after an inplace update has been applied to the original `Series`/`DataFrame`.
- Suppressed an unhelpful `SettingWithCopyWarning` that sometimes appeared when printing `Timedelta` columns.
- Fixed `inplace` argument for `Series` objects derived from other `Series` objects.
- Fixed a bug where `Series.sort_values` failed if series name overlapped with index column name.
- Fixed a bug where transposing a dataframe would map `Timedelta` index levels to integer column levels.
- Fixed a bug where `Resampler` methods on timedelta columns would produce integer results.
- Fixed a bug where `pd.to_numeric()` would leave `Timedelta` inputs as `Timedelta` instead of converting them to integers.

## 1.22.1 (2024-09-11)
This is a re-release of 1.22.0. Please refer to the 1.22.0 release notes for detailed release content.
Expand Down
2 changes: 1 addition & 1 deletion ci/check_standalone_function_snowpark_pandas_telemetry_decorator.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -64,7 +64,7 @@ def check_standalone_function_snowpark_pandas_telemetry_decorator(
telemetry_decorator_name="snowpark_pandas_telemetry_standalone_function_decorator",
)
check_standalone_function_snowpark_pandas_telemetry_decorator(
target_file="src/snowflake/snowpark/modin/pandas/general.py",
target_file="src/snowflake/snowpark/modin/plugin/extensions/general_overrides.py",
telemetry_decorator_name="snowpark_pandas_telemetry_standalone_function_decorator",
)
check_standalone_function_snowpark_pandas_telemetry_decorator(
Expand Down
2 changes: 1 addition & 1 deletion docs/source/modin/general_functions.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
General functions
=============================

.. currentmodule:: snowflake.snowpark.modin.pandas.general
.. currentmodule:: modin.pandas
.. rubric:: :doc:`All supported general functions <supported/general_supported>`

.. rubric:: Data manipulations
Expand Down
2 changes: 1 addition & 1 deletion docs/source/modin/groupby.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
GroupBy
=============================

.. currentmodule:: snowflake.snowpark.modin.pandas.groupby
.. currentmodule:: snowflake.snowpark.modin.plugin.extensions.groupby_overrides
.. rubric:: :doc:`All supported groupby APIs <supported/groupby_supported>`

.. rubric:: Indexing, iteration
Expand Down
2 changes: 1 addition & 1 deletion docs/source/modin/indexing.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ Index objects

Index
-----
.. currentmodule:: snowflake.snowpark.modin.pandas
.. currentmodule:: modin.pandas
.. rubric:: :doc:`All supported Index APIs <supported/index_supported>`

.. rubric:: Constructor
Expand Down
3 changes: 2 additions & 1 deletion docs/source/modin/io.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
Input/Output
=============================

.. currentmodule:: snowflake.snowpark.modin.pandas
.. currentmodule:: modin.pandas

.. rubric:: Flat file

Expand All @@ -13,6 +13,7 @@ Input/Output
read_excel
read_json
read_parquet
read_sas

.. rubric:: SQL

Expand Down
2 changes: 1 addition & 1 deletion docs/source/modin/resampling.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
Resampling
=============================

.. currentmodule:: snowflake.snowpark.modin.pandas.resample
.. currentmodule:: snowflake.snowpark.modin.plugin.extensions.resample_overrides
.. rubric:: :doc:`All supported resampling APIs <supported/resampling_supported>`

.. rubric:: Indexing, iteration
Expand Down
14 changes: 8 additions & 6 deletions docs/source/modin/supported/dataframe_supported.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -91,8 +91,9 @@ Methods
+-----------------------------+---------------------------------+----------------------------------+----------------------------------------------------+
| ``applymap`` | P | | ``N`` if ``na_action == "ignore"`` |
+-----------------------------+---------------------------------+----------------------------------+----------------------------------------------------+
| ``asfreq`` | P | ``how``, ``normalize``, | See ``resample`` |
| | | ``fill_value`` | |
| ``asfreq`` | P | ``how``, ``normalize``, | Only DatetimeIndex is supported and its ``freq`` |
| | | ``fill_value`` | will be lost. Only ``rule`` frequencies 's', 'min',|
| | | | 'h', and 'D' are supported. |
+-----------------------------+---------------------------------+----------------------------------+----------------------------------------------------+
| ``asof`` | N | | |
+-----------------------------+---------------------------------+----------------------------------+----------------------------------------------------+
Expand Down Expand Up @@ -347,10 +348,11 @@ Methods
| ``replace`` | P | ``copy`` is ignored, ``method``, | |
| | | ``limit`` | |
+-----------------------------+---------------------------------+----------------------------------+----------------------------------------------------+
| ``resample`` | P | ``axis``, ``closed``, ``label``, | Only DatetimeIndex is supported and its ``freq`` |
| | | ``convention``, ``kind``, ``on`` | will be lost. Only ``rule`` frequencies 's', 'min',|
| | | , ``level``, ``origin``, | 'h', and 'D' are supported. |
| | | , ``offset``, ``group_keys`` | |
| ``resample`` | P | ``axis``, ``label``, | Only DatetimeIndex is supported and its ``freq`` |
| | | ``convention``, ``kind``, ``on`` | will be lost. ``rule`` frequencies 's', 'min', |
| | | , ``level``, ``origin``, | 'h', and 'D' are supported. ``rule`` frequencies |
| | | , ``offset``, ``group_keys`` | 'W', 'ME', and 'YE' are supported with |
| | | | `closed = "left"` |
+-----------------------------+---------------------------------+----------------------------------+----------------------------------------------------+
| ``reset_index`` | Y | | |
+-----------------------------+---------------------------------+----------------------------------+----------------------------------------------------+
Expand Down
4 changes: 2 additions & 2 deletions docs/source/modin/supported/datetime_index_supported.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -100,7 +100,7 @@ Methods
+-----------------------------+---------------------------------+----------------------------------+----------------------------------------------------+
| ``day_name`` | P | ``locale`` | |
+-----------------------------+---------------------------------+----------------------------------+----------------------------------------------------+
| ``mean`` | N | | |
| ``mean`` | Y | | |
+-----------------------------+---------------------------------+----------------------------------+----------------------------------------------------+
| ``std`` | N | | |
| ``std`` | P | ``ddof`` | |
+-----------------------------+---------------------------------+----------------------------------+----------------------------------------------------+
6 changes: 4 additions & 2 deletions docs/source/modin/supported/general_supported.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -99,6 +99,8 @@ Data manipulations
| | | | Obtaining results from stored procedures is also |
| | | | supported via CALL queries. |
+-----------------------------+---------------------------------+----------------------------------+----------------------------------------------------+
| ``read_sas`` | Y | | Uses native pandas to read sas files. |
+-----------------------------+---------------------------------+----------------------------------+----------------------------------------------------+
| ``read_table`` | N | | |
+-----------------------------+---------------------------------+----------------------------------+----------------------------------------------------+
| ``to_pandas`` | Y | | |
Expand Down Expand Up @@ -143,9 +145,9 @@ Top-level dealing with datetime-like data
+-----------------------------+---------------------------------+----------------------------------+----------------------------------------------------+
| Method | Snowpark implemented? (Y/N/P/D) | Missing parameters | Notes for current implementation |
+-----------------------------+---------------------------------+----------------------------------+----------------------------------------------------+
| ``bdate_range`` | P | ``tz`` | ``N`` for custom frequencies |
| ``bdate_range`` | P | | ``N`` for custom frequencies |
+-----------------------------+---------------------------------+----------------------------------+----------------------------------------------------+
| ``date_range`` | P | ``tz`` | ``N`` for custom frequencies |
| ``date_range`` | P | | ``N`` for custom frequencies |
+-----------------------------+---------------------------------+----------------------------------+----------------------------------------------------+
| ``infer_freq`` | N | | |
+-----------------------------+---------------------------------+----------------------------------+----------------------------------------------------+
Expand Down
3 changes: 2 additions & 1 deletion docs/source/modin/supported/groupby_supported.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -106,7 +106,8 @@ Computations/descriptive stats
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``ffill`` | N | |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``fillna`` | N | |
| ``fillna`` | P | GroupBy axis = 0 is supported. |
| | | Does not support ``downcast`` parameter |
+-----------------------------+---------------------------------+----------------------------------------------------+
| ``first`` | P | Does not support ``min_count`` parameter |
+-----------------------------+---------------------------------+----------------------------------------------------+
Expand Down
2 changes: 1 addition & 1 deletion docs/source/modin/supported/resampling_supported.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,7 @@ Upsampling
+-----------------------------+---------------------------------+----------------------------------+----------------------------------------------------+
| Resampler method | Snowpark implemented? (Y/N/P/D) | Missing parameters | Notes for current implementation |
+-----------------------------+---------------------------------+----------------------------------+----------------------------------------------------+
| ``asfreq`` | N | | |
| ``asfreq`` | P | ``fill_value`` | |
+-----------------------------+---------------------------------+----------------------------------+----------------------------------------------------+
| ``bfill`` | P | ``limit`` | |
+-----------------------------+---------------------------------+----------------------------------+----------------------------------------------------+
Expand Down
Loading

0 comments on commit 1e4dcad

Please sign in to comment.