Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Pebble functions to help troubleshoot if file exists, is empty and inspecting its file size #6888

Closed
anna-geller opened this issue Jan 22, 2025 · 3 comments · Fixed by #7191
Assignees
Labels
area/backend Needs backend code changes enhancement New feature or request

Comments

@anna-geller
Copy link
Member

Feature description

Proposed pebble functions:

{{ fileSize(output.download.uri)  }}
{{ fileExists(output.download.uri)  }}
{{ isEmpty(output.download.uri)  }} # true or false
@anna-geller anna-geller added area/backend Needs backend code changes enhancement New feature or request labels Jan 22, 2025
@github-project-automation github-project-automation bot moved this to Backlog in Issues Jan 22, 2025
@deepPublicGit
Copy link
Contributor

deepPublicGit commented Feb 1, 2025

Hi @anna-geller few questions:

  1. Is the proposed filesize function different from the existing one:

https://github.com/kestra-io/kestra/blob/develop/core/src/main/java/io/kestra/core/runners/pebble/functions/FileSizeFunction.java

  1. What should the return type of fileExists function ? (true/false ?)

  2. Can we reuse fileSize function for isEmpty function, if fileSize==0 return true or should it check the file content?

  3. Would it better to rename isEmpty to isFileEmpty to avoid potential confusion with any future function that would check if the content of a variable/string is empty?

@anna-geller
Copy link
Member Author

  1. You are right, I wasn't aware this function already exists (see flow below), so it seems that no action is needed then ✅
  2. Yup, exactly, a boolean with true if file exists is great!
  3. I believe checking the file content seems like a more reliable way to check if the file is empty
  4. We can name it fileEmpty() for consistency with fileExists() 👍
id: etl
namespace: company.team

tasks:
  - id: download
    type: io.kestra.plugin.core.http.Download
    uri: https://huggingface.co/datasets/kestra/datasets/raw/main/csv/orders.csv
  
  - id: log
    type: io.kestra.plugin.core.log.Log
    message: "{{ fileSize(outputs.download.uri) }}"

@deepPublicGit
Copy link
Contributor

Hi @anna-geller , can you review these changes:
#7191

@anna-geller anna-geller moved this from In progress to In review in Issues Feb 5, 2025
@github-project-automation github-project-automation bot moved this from In review to Done in Issues Feb 11, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/backend Needs backend code changes enhancement New feature or request
Projects
Status: Done
Development

Successfully merging a pull request may close this issue.

2 participants