Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Trends report #765

Open
wants to merge 2 commits into
base: main
Choose a base branch
from
Open

Trends report #765

wants to merge 2 commits into from

Conversation

erfanio
Copy link
Collaborator

@erfanio erfanio commented Jan 17, 2025

Trends report is available to collaborators at https://llm-exp.oss-fuzz.com/trend-reports/index.html

This PR

  • Adds trends report web page
  • Adds a cloud run function to update trend-reports/index.json in GCS
  • Adds a cloud run function to update web page files (html, js, css)
  • Adds a script to deploy cloud run functions (this script needs to be run manually after every change to the code)
  • Moves trends report summary generation tool to a subdirectory so everything related to trends report is together

@DavidKorczynski @DonggeLiu please let me know if you want anything else added to the trends report

@erfanio erfanio force-pushed the trends-report branch 2 times, most recently from 56fd529 to 5e01694 Compare January 17, 2025 07:00
@erfanio
Copy link
Collaborator Author

erfanio commented Jan 17, 2025

/gcbrun exp -n erfanio

@DonggeLiu
Copy link
Collaborator

Wow this is truly amazing, thanks @erfanio!
It makes comparing metrics across experiments so much easier, e.g., we can easily see the build rate improvements in 2024 Nov.

I just returned to work and might need a bit more time to read the code, but by looking at the trend report I think it's already great!

@DonggeLiu
Copy link
Collaborator

Some minor suggestions as discussed:

  1. Include the results of incomplete experiments (e.g., 2024-11-08-weekly-all).

  2. Use build rate (% values) instead of absolute values in the Overview plot, e.g., 30.30%.

  3. Use % for build rate, percentage coverage gain, etc. For example 0.1% coverage gain (instead of 0.001). Some small numbers might be rounded to 0 because of this:
    image

  4. Maybe remove reportName in plots? It can be misleading given it has to be distant from the long x-axis labels, which are self-explanatory.

  5. Plot line color indicator for function signatures, e.g., mention that in the table beneath the plot? Omit this if it's too complex.

  6. Add copyright headers with the correct year.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants