feat(spark): Populate time variables for log links (#6328)

[demoncoder-crypto]

This pull request modifies the getEventInfoForSpark function within flyteplugins/go/tasks/plugins/k8s/spark/spark.go.
Extract Timestamps: Retrieves the SubmissionTime, CompletionTime, and TerminationTime from the sparkOp.SparkApplicationStatus (sj.Status).
Format Timestamps: The retrieved metav1.Time values are formatted into RFC3339 strings and standard Unix timestamps (int64). SubmissionTime is used for the start time, and CompletionTime (or TerminationTime as a fallback) is used for the end time. Checks are included to handle cases where these timestamps might be zero (e.g., job hasn't started/finished).
Populate tasklog.Input: The formatted timestamps are used to populate the PodRFC3339StartTime, PodRFC3339FinishTime, PodUnixStartTime, and PodUnixFinishTime fields within the tasklog.Input struct.
Targeted Application: This population is specifically done for the calls to p.GetTaskLogs that fetch logs associated with the Spark driver pod (sj.Status.DriverInfo.PodName), namely the "Driver Logs" (using the "Mixed" log config) and "User Logs" (using the "User" log config). Calls for "System" and "AllUser" logs remain unchanged as they use the application name rather than a specific pod name.

Summary by Bito

This PR enhances Spark logging by extracting time metrics from SparkApplicationStatus, computing start/finish timestamps based on submission and completion times. The timestamps are formatted in RFC3339 and Unix formats to improve the tasklog.Input structure for both driver and user logs, enabling more accurate log tracking and troubleshooting.

Unit tests added: False

Estimated effort to review (1-5, lower is better): 1

[welcome]

Thank you for opening this pull request! 🙌

These tips will help get your PR across the finish line:

• Most of the repos have a PR template; if not, fill it out to the best of your knowledge.
• Sign off your commits (Reference: DCO Guide).

[flyte-bot]

Code Review Agent Run #adc73a

Actionable Suggestions - 0

Review Details

• Files reviewed - 1 · Commit Range: b23de8d..b23de8d

  • flyteplugins/go/tasks/plugins/k8s/spark/spark.go
• Files skipped - 0
• Tools

  • Whispers (Secret Scanner) - ✔︎ Successful
  • Detect-secrets (Secret Scanner) - ✔︎ Successful

---

Bito Usage Guide

Commands

Type the following command in the pull request comment and save the comment.

• /review - Manually triggers a full AI review.

Refer to the documentation for additional commands.

Configuration

This repository uses code_review_bito You can customize the agent settings here or contact your Bito workspace admin at [email protected].

Documentation & Help

• Review rules
• General documentation
• FAQ

AI Code Review powered by

[demoncoder-crypto]

Not yet tested I just want to make sure my implementation and my track is correct as this is my first contribution to flyte.

[flyte-bot]

Changelist by Bito

This pull request implements the following key changes.

Key Change	Files Impacted
New Feature - Spark Log Time Population	- spark.go - Enhanced getEventInfoForSpark by extracting, formatting, and populating SubmissionTime and Completion/TerminationTime into both RFC3339 and Unix timestamp fields for improved log link functionality.