Split up job data dimension #1050
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jjnesbitt
force-pushed
the
split-up-job-data-dimension
branch
from
7529776 to
6aca90c
Compare
mvandenburgh
requested review from
mvandenburgh
and removed request for
mvandenburgh
mvandenburgh
previously approved these changes
May 14, 2025
|
Also, we'll want to be careful about how we deploy this since it will mess up all our metabase dashboards. I think we can update all the dashboards in metabase locally and apply them after this is deployed. |
mvandenburgh
dismissed
their stale review
Actually, I'll dismiss this for now so it doesn't get merged accidentally since you mentioned you still have to review migrations + we should prep the metabase dashboards beforehand
mvandenburgh
approved these changes
May 27, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR accomplishes the following:
Split up
JobDataDimensioninto the following dimensions:SpackJobDataDimensionGitlabJobDataDimensionJobResultDimensionJobRetryDimensionThis data was all bundled into the
JobDataDimension, which necessitated an entry in that dimensional table for every job. This meant that theJobDataDimensiontable was as large as theJobFacttable, which is undesirable, and results in poor join performance. With this dimension split up, each table is much smaller, which results in more re-used rows, as well as faster joins.The storage of gitlab section timers on the
JobFacttable. Since this is numeric, and will have aggregations performed on it, it's the correct place to store it.Setting the
job_idas the primary key for theJobFacttable. This is really how we expect these facts to exist anyway (only one job fact per job). As a result of the old primary key for this table, there were actually several duplicate job entries, which this PR cleans up.The removal of the
JobDataDimensiontable. This is due to all of its constituent data being split into smaller dimension tables. However, this also results in the removal of that foreign key from theTimerFactandTimerPhaseFacttables, although they still store thejob_id. I plan to evaluate if and what new relations should be introduced, and make those changes in the future.I have run these migrations locally and can confirm there are no errors that arise. I also have tested some webhook payloads and it seems the job processor runs without error. The migration downtime for this should be ~6 hours in production.