Cost logs for API Deployments #1167

pk-zipstack · 2025-03-05T07:29:46Z

What

Cost logs for API deployments

Why

Makes it easier for customers to check the cost easily in the UI instead of looking them up in the API like earlier.

How

Used the usage's existing aggregate token function to collect the cost for a particular file.
Created a new function in usage to aggregate the cost for a particular execution_id. This function was used to gather the total cost for an execution, which helps to identify the average cost for a particular execution.

Can this PR break any existing features? If yes, please list possible items. If no, please explain why. (PS: Admins do not merge the PR without this section filled)

No, this is just a log change which includes the already existing functions.

Database Migrations

No

Env Config

Relevant Docs

Related Issues or PRs

https://zipstack.atlassian.net/browse/UN-1799

Dependencies Versions

Notes on Testing

Tested with multiple files and calculated the cost per file and average cost manually to check if it's accurate.

Screenshots

Checklist

I have read and understood the Contribution Guidelines.

for more information, see https://pre-commit.ci

chandrasekharan-zipstack

Looks okay for the most part, left some minor comments. Do add some screenshots as well if possible

backend/sample.env

backend/workflow_manager/workflow_v2/execution.py

backend/usage_v2/helper.py

backend/workflow_manager/workflow_v2/execution.py

backend/usage_v2/helper.py

backend/workflow_manager/workflow_v2/execution.py

backend/workflow_manager/workflow_v2/workflow_helper.py

Co-authored-by: Chandrasekharan M <[email protected]> Signed-off-by: Praveen Kumar <[email protected]>

Signed-off-by: Praveen Kumar <[email protected]>

for more information, see https://pre-commit.ci

backend/workflow_manager/workflow_v2/execution.py

Co-authored-by: Chandrasekharan M <[email protected]> Signed-off-by: Praveen Kumar <[email protected]>

Signed-off-by: Praveen Kumar <[email protected]>

for more information, see https://pre-commit.ci

backend/workflow_manager/workflow_v2/execution.py

backend/workflow_manager/workflow_v2/models/execution.py

backend/workflow_manager/workflow_v2/workflow_helper.py

backend/workflow_manager/workflow_v2/execution.py

Co-authored-by: Chandrasekharan M <[email protected]> Signed-off-by: Praveen Kumar <[email protected]>

for more information, see https://pre-commit.ci

Signed-off-by: Praveen Kumar <[email protected]>

backend/workflow_manager/workflow_v2/execution.py

tools/structure/src/main.py

backend/workflow_manager/workflow_v2/models/execution.py

backend/workflow_manager/workflow_v2/execution.py

backend/workflow_manager/workflow_v2/models/execution.py

backend/workflow_manager/workflow_v2/execution.py

for more information, see https://pre-commit.ci

Signed-off-by: Praveen Kumar <[email protected]>

github-actions · 2025-04-03T09:51:03Z

filepath	function	$$\textcolor{#23d18b}{\tt{passed}}$$	SUBTOTAL
$$\textcolor{#23d18b}{\tt{runner/src/unstract/runner/clients/test\_docker.py}}$$	$$\textcolor{#23d18b}{\tt{test\_logs}}$$	$$\textcolor{#23d18b}{\tt{1}}$$	$$\textcolor{#23d18b}{\tt{1}}$$
$$\textcolor{#23d18b}{\tt{runner/src/unstract/runner/clients/test\_docker.py}}$$	$$\textcolor{#23d18b}{\tt{test\_cleanup}}$$	$$\textcolor{#23d18b}{\tt{1}}$$	$$\textcolor{#23d18b}{\tt{1}}$$
$$\textcolor{#23d18b}{\tt{runner/src/unstract/runner/clients/test\_docker.py}}$$	$$\textcolor{#23d18b}{\tt{test\_cleanup\_skip}}$$	$$\textcolor{#23d18b}{\tt{1}}$$	$$\textcolor{#23d18b}{\tt{1}}$$
$$\textcolor{#23d18b}{\tt{runner/src/unstract/runner/clients/test\_docker.py}}$$	$$\textcolor{#23d18b}{\tt{test\_client\_init}}$$	$$\textcolor{#23d18b}{\tt{1}}$$	$$\textcolor{#23d18b}{\tt{1}}$$
$$\textcolor{#23d18b}{\tt{runner/src/unstract/runner/clients/test\_docker.py}}$$	$$\textcolor{#23d18b}{\tt{test\_get\_image\_exists}}$$	$$\textcolor{#23d18b}{\tt{1}}$$	$$\textcolor{#23d18b}{\tt{1}}$$
$$\textcolor{#23d18b}{\tt{runner/src/unstract/runner/clients/test\_docker.py}}$$	$$\textcolor{#23d18b}{\tt{test\_get\_image}}$$	$$\textcolor{#23d18b}{\tt{1}}$$	$$\textcolor{#23d18b}{\tt{1}}$$
$$\textcolor{#23d18b}{\tt{runner/src/unstract/runner/clients/test\_docker.py}}$$	$$\textcolor{#23d18b}{\tt{test\_get\_container\_run\_config}}$$	$$\textcolor{#23d18b}{\tt{1}}$$	$$\textcolor{#23d18b}{\tt{1}}$$
$$\textcolor{#23d18b}{\tt{runner/src/unstract/runner/clients/test\_docker.py}}$$	$$\textcolor{#23d18b}{\tt{test\_get\_container\_run\_config\_without\_mount}}$$	$$\textcolor{#23d18b}{\tt{1}}$$	$$\textcolor{#23d18b}{\tt{1}}$$
$$\textcolor{#23d18b}{\tt{runner/src/unstract/runner/clients/test\_docker.py}}$$	$$\textcolor{#23d18b}{\tt{test\_run\_container}}$$	$$\textcolor{#23d18b}{\tt{1}}$$	$$\textcolor{#23d18b}{\tt{1}}$$
$$\textcolor{#23d18b}{\tt{TOTAL}}$$		$$\textcolor{#23d18b}{\tt{9}}$$	$$\textcolor{#23d18b}{\tt{9}}$$

sonarqubecloud · 2025-04-03T09:51:36Z

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

muhammad-ali-e

LGTM...

Added logs for cost per file

0919c34

pk-zipstack requested review from Deepak-Kesavan, chandrasekharan-zipstack and muhammad-ali-e March 5, 2025 07:29

pk-zipstack self-assigned this Mar 5, 2025

pre-commit-ci bot and others added 2 commits March 5, 2025 07:30

[pre-commit.ci] auto fixes from pre-commit.com hooks

dccb7a0

for more information, see https://pre-commit.ci

Fixed pre-sommit issues

b29a261

pk-zipstack requested a review from a team March 5, 2025 07:43

pre-commit-ci bot and others added 2 commits March 5, 2025 07:43

[pre-commit.ci] auto fixes from pre-commit.com hooks

12937cb

for more information, see https://pre-commit.ci

fixed pre commit issues

f52b1e9

chandrasekharan-zipstack reviewed Mar 5, 2025

View reviewed changes

pk-zipstack and others added 8 commits March 5, 2025 17:37

Update backend/workflow_manager/workflow_v2/execution.py

d0061a0

Co-authored-by: Chandrasekharan M <[email protected]> Signed-off-by: Praveen Kumar <[email protected]>

Update backend/workflow_manager/workflow_v2/execution.py

4fabd40

Co-authored-by: Chandrasekharan M <[email protected]> Signed-off-by: Praveen Kumar <[email protected]>

Made changes to log the total cost in get_aggregated_cost function

2c83d08

added error handling to prevent TypeError and ZeroDivisionError

6fdbe5b

Removed devision by zero error for publish_average_cost_log

d4755ed

Updated the way the error logs are displayed to the user.

a819e84

Merge branch 'main' into UN-1799-Cost-log-for-ETL

a3ee996

Signed-off-by: Praveen Kumar <[email protected]>

[pre-commit.ci] auto fixes from pre-commit.com hooks

8a138cb

for more information, see https://pre-commit.ci

chandrasekharan-zipstack reviewed Mar 7, 2025

View reviewed changes

backend/workflow_manager/workflow_v2/execution.py Outdated Show resolved Hide resolved

chandrasekharan-zipstack reviewed Mar 7, 2025

View reviewed changes

backend/workflow_manager/workflow_v2/execution.py Outdated Show resolved Hide resolved

pk-zipstack and others added 8 commits March 7, 2025 15:49

Moved publish_average_cost_log inside publish_final_workflow_logs

4108df8

Update backend/workflow_manager/workflow_v2/execution.py

3394549

Co-authored-by: Chandrasekharan M <[email protected]> Signed-off-by: Praveen Kumar <[email protected]>

Merge branch 'main' into UN-1799-Cost-log-for-ETL

e745bd7

Signed-off-by: Praveen Kumar <[email protected]>

[pre-commit.ci] auto fixes from pre-commit.com hooks

f940d33

for more information, see https://pre-commit.ci

Updated execution.py to fix line too long issue

7fc7b3e

Converted log type "error" to "warning" in publish_average_cost_log

1f33197

Moved get_aggregated_cost to WorkflowExectuion class as a property

56dcf64

[pre-commit.ci] auto fixes from pre-commit.com hooks

965a3b7

for more information, see https://pre-commit.ci

chandrasekharan-zipstack reviewed Mar 12, 2025

View reviewed changes

pk-zipstack and others added 12 commits March 12, 2025 23:55

Update backend/workflow_manager/workflow_v2/models/execution.py

fad083a

Co-authored-by: Chandrasekharan M <[email protected]> Signed-off-by: Praveen Kumar <[email protected]>

Update backend/workflow_manager/workflow_v2/models/execution.py

dce0255

Co-authored-by: Chandrasekharan M <[email protected]> Signed-off-by: Praveen Kumar <[email protected]>

[pre-commit.ci] auto fixes from pre-commit.com hooks

899638e

for more information, see https://pre-commit.ci

Merge branch 'main' into UN-1799-Cost-log-for-ETL

9f77f90

removed NoneType error handling in execution.py

061db21

[pre-commit.ci] auto fixes from pre-commit.com hooks

948942c

for more information, see https://pre-commit.ci

Merge branch 'main' into UN-1799-Cost-log-for-ETL

f48b23d

Merge branch 'main' into UN-1799-Cost-log-for-ETL

6d62694

Signed-off-by: Praveen Kumar <[email protected]>

Merge branch 'main' into UN-1799-Cost-log-for-ETL

550019d

Updated tool structure version

8e5fd12

Signed-off-by: Praveen Kumar <[email protected]>

Update sample.env

1a065cf

Signed-off-by: Praveen Kumar <[email protected]>

Merge branch 'main' into UN-1799-Cost-log-for-ETL

471e6b9

chandrasekharan-zipstack approved these changes Mar 18, 2025

View reviewed changes

backend/workflow_manager/workflow_v2/execution.py Outdated Show resolved Hide resolved

muhammad-ali-e reviewed Mar 18, 2025

View reviewed changes

Deepak-Kesavan approved these changes Mar 26, 2025

View reviewed changes

pk-zipstack and others added 9 commits April 3, 2025 11:30

Added .exists() for getting aggregated cost in execution.py

e71386c

[pre-commit.ci] auto fixes from pre-commit.com hooks

babad72

for more information, see https://pre-commit.ci

fixed line to long in execution.py

64729ad

[pre-commit.ci] auto fixes from pre-commit.com hooks

94e3289

for more information, see https://pre-commit.ci

Removed redundant rounding of cost in exexution.py

82c423b

Added indexing to execution_id in Usage model

b6d95b8

Merge branch 'main' into UN-1799-Cost-log-for-ETL

9f9323a

Deleted a file

3ac473c

Signed-off-by: Praveen Kumar <[email protected]>

Updated structure tool version to 0.0.70

d7d65c0

muhammad-ali-e approved these changes Apr 3, 2025

View reviewed changes

muhammad-ali-e merged commit 9662435 into main Apr 3, 2025
7 checks passed

muhammad-ali-e deleted the UN-1799-Cost-log-for-ETL branch April 3, 2025 10:08

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Cost logs for API Deployments #1167

Cost logs for API Deployments #1167

pk-zipstack commented Mar 5, 2025 •

edited

Loading

chandrasekharan-zipstack left a comment

github-actions bot commented Apr 3, 2025

sonarqubecloud bot commented Apr 3, 2025

muhammad-ali-e left a comment •

edited

Loading

Cost logs for API Deployments #1167

Cost logs for API Deployments #1167

Conversation

pk-zipstack commented Mar 5, 2025 • edited Loading

What

Why

How

Can this PR break any existing features? If yes, please list possible items. If no, please explain why. (PS: Admins do not merge the PR without this section filled)

Database Migrations

Env Config

Relevant Docs

Related Issues or PRs

Dependencies Versions

Notes on Testing

Screenshots

Checklist

chandrasekharan-zipstack left a comment

Choose a reason for hiding this comment

github-actions bot commented Apr 3, 2025

sonarqubecloud bot commented Apr 3, 2025

Quality Gate passed

muhammad-ali-e left a comment • edited Loading

Choose a reason for hiding this comment

pk-zipstack commented Mar 5, 2025 •

edited

Loading

muhammad-ali-e left a comment •

edited

Loading