[FSTORE-1411] On-Demand Transformation Functions #1371

manu-sj · 2024-07-10T07:21:29Z

This PR adds functionality for On-demand transformation function as per the design log https://hopsworks.atlassian.net/wiki/spaces/FST/pages/461733903/On-Demand+Transformation+Functions+-+User+API+and+Implementation

Changes Done:

Allow custom UDF's to drop features after transformations. The feature is implemented by adding a parameter drop in the decorator.

@hopsworks.udf(return_type=float, drop=["arg1", "arg2"])
def custom_udf(arg1, arg2, arg3):
    ...
    return value

Removed TransformationFunctionsAttached from the python client
Added ability to attach transformation functions to feature groups. (On-Demand Transformations)
Added to feature view ability to recompute on-demand features using request parameters.
Added to feature view ability to manually compute on-demand features using the function compute_on_demand_features
Added to feature view ability to access manually apply model-dependent transformations to the feature vector transform.
Added the ability to access executable model-dependent transformation function using a dictionary with the key was the output column name. (Only output column name is unique for a model dependent transformation function).
Added the ability to access executable on-demand transformation function using a dictionary with the key was the on-demand feature name. (Only output column name is unique for a model dependent transformation function).

JIRA Issue: https://hopsworks.atlassian.net/browse/FSTORE-1411

Priority for Review: -

Related PRs:
https://github.com/logicalclocks/hopsworks-ee/pull/1862
logicalclocks/hopsworks-chef#1148
https://github.com/logicalclocks/loadtest/pull/394

How Has This Been Tested?

Unit Tests
Integration Tests
Manual Tests on VM

Checklist For The Assigned Reviewer:

- [ ] Checked if merge conflicts with master exist
- [ ] Checked if stylechecks for Java and Python pass
- [ ] Checked if all docstrings were added and/or updated appropriately
- [ ] Ran spellcheck on docstring
- [ ] Checked if guides & concepts need to be updated
- [ ] Checked if naming conventions for parameters and variables were followed
- [ ] Checked if private methods are properly declared and used
- [ ] Checked if hard-to-understand areas of code are commented
- [ ] Checked if tests are effective
- [ ] Built and deployed changes on dev VM and tested manually
- [x] (Checked if all type annotations were added and/or updated appropriately)

kennethmhc

Comments about the user facing API.

python/hsfs/hopsworks_udf.py

kennethmhc · 2024-07-12T07:16:59Z

python/hsfs/hopsworks_udf.py

@@ -519,6 +629,8 @@ def get_udf(self, force_python_udf: bool = False) -> Callable:
        # Returns
            `Callable`: Pandas UDF in the spark engine otherwise returns a python function for the UDF.
        """
+        if self.udf_type is None:


why is it needed?

kennethmhc · 2024-07-12T07:25:47Z

python/hsfs/hopsworks_udf.py

@@ -31,7 +32,18 @@
 from hsfs.transformation_statistics import TransformationStatistics


-def udf(return_type: Union[List[type], type]) -> "HopsworksUdf":
+class UDFType(Enum):


This is actually transformation type. The UDF is actually the same and can be used for both ODT and MDT. I understand this type is needed for the udf to return different column names, but imo this is better to be done in transformation_function class and also the validation of the transformation type. But it is not critical, can be refactored later.

You are right. I will move it out to transformation function class.

kennethmhc · 2024-07-12T07:50:55Z

python/hsfs/feature_view.py

+        request_parameters: Optional[
+            Union[List[Dict[str, Any]], Dict[str, Any]]
+        ] = None,
+        external: Optional[bool] = None,


why is it needed? this function won't fetch features from online store right?

yes you are right. There is no real need for a Vector server to be initlized. I think should probably refactor the code to move the functionality out of VectorServer

kennethmhc · 2024-07-12T07:51:04Z

python/hsfs/feature_view.py

+    def transform(
+        self,
+        feature_vector: Union[List[Any], List[List[Any]], pd.DataFrame, pl.DataFrame],
+        external: Optional[bool] = None,


why is it needed? this function won't fetch features from online store right?

Yes you are right it won't fetch features from the online store, but MDT's might require statistics of features. The statistics used while transforming depends on the training_dataset_version set in init_serving so if init_serving had not been done we don't know the statistics to be used for applying the transformation function. So I basically copied the code done in get_feature_vector for initializing vector server.

I could be wrong here but I think there should be some check to see if init_serving or init_batch_scoring has been performed.

python/hsfs/feature_group.py

python/hsfs/core/vector_server.py

python/hsfs/engine/python.py

python/hsfs/feature_view.py

…d and adapting changes for save functions feature group

kennethmhc

LGTM (the rest of the comment will be addressed in a separate PR)

manu-sj requested a review from kennethmhc July 10, 2024 07:27

manu-sj added 3 commits July 11, 2024 09:00

on-demand transformations

7bcdb27

adding api for computing and transforming feature vectors

44dc976

setting request parameters to {} if None

ed8a6f3

manu-sj force-pushed the FSTORE-1411 branch from a6e3fb2 to ed8a6f3 Compare July 11, 2024 07:01

kennethmhc reviewed Jul 12, 2024

View reviewed changes

python/hsfs/feature_group.py Show resolved Hide resolved

kennethmhc requested changes Jul 12, 2024

View reviewed changes

python/hsfs/core/vector_server.py Outdated Show resolved Hide resolved

kennethmhc reviewed Jul 12, 2024

View reviewed changes

python/hsfs/engine/python.py Show resolved Hide resolved

python/hsfs/feature_view.py Show resolved Hide resolved

checking all element to check if multiple feature vectors are provide…

459d097

…d and adapting changes for save functions feature group

kennethmhc approved these changes Jul 12, 2024

View reviewed changes

manu-sj merged commit b066aa9 into logicalclocks:master Jul 12, 2024
13 checks passed

manu-sj mentioned this pull request Aug 1, 2024

[FSTORE-1411][APPEND] On-Demand Transformations logicalclocks/hopsworks-api#255

Closed

3 tasks

manu-sj mentioned this pull request Aug 16, 2024

[FSTORE-1411][APPEND] On-Demand Transformations logicalclocks/hopsworks-api#284

Merged

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[FSTORE-1411] On-Demand Transformation Functions #1371

[FSTORE-1411] On-Demand Transformation Functions #1371

manu-sj commented Jul 10, 2024 •

edited

Loading

kennethmhc left a comment

kennethmhc Jul 12, 2024

kennethmhc Jul 12, 2024

manu-sj Jul 12, 2024

kennethmhc Jul 12, 2024

manu-sj Jul 12, 2024

kennethmhc Jul 12, 2024

manu-sj Jul 12, 2024

kennethmhc left a comment

[FSTORE-1411] On-Demand Transformation Functions #1371

[FSTORE-1411] On-Demand Transformation Functions #1371

Conversation

manu-sj commented Jul 10, 2024 • edited Loading

kennethmhc left a comment

Choose a reason for hiding this comment

kennethmhc Jul 12, 2024

Choose a reason for hiding this comment

kennethmhc Jul 12, 2024

Choose a reason for hiding this comment

manu-sj Jul 12, 2024

Choose a reason for hiding this comment

kennethmhc Jul 12, 2024

Choose a reason for hiding this comment

manu-sj Jul 12, 2024

Choose a reason for hiding this comment

kennethmhc Jul 12, 2024

Choose a reason for hiding this comment

manu-sj Jul 12, 2024

Choose a reason for hiding this comment

kennethmhc left a comment

Choose a reason for hiding this comment

manu-sj commented Jul 10, 2024 •

edited

Loading