Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: SQLite-based package cache #26608

Merged
merged 47 commits into from
Jan 30, 2024
Merged

feat: SQLite-based package cache #26608

merged 47 commits into from
Jan 30, 2024

Conversation

zharinov
Copy link
Collaborator

@zharinov zharinov commented Jan 12, 2024 •
edited
Loading

Changes

Additional useSqliteCache option could enable usage of SQLite for package cache, instead of file-based one (cacache)

Context

Documentation (please check one with an [x])

  • I have updated the documentation, or
  • No documentation update is required

How I've tested my work (please select one)

I have verified these changes via:

  • Code inspection only, or
  • Newly added/modified unit tests, or
  • No unit tests but ran on a real repository, or
  • Both unit tests + ran on a real repository

@zharinov
Copy link
Collaborator Author

Cold Start

Backend Count (r/w) Total Ms (r/w) Avg Ms (r/w) Median ms (r/w) Max Ms (r/w)
Redis 660/641 1106/10422 2/16 0/4 136/1238
File 660/641 508/12657 1/20 0/6 20/1244
SQLite 660/641 74/165 0/0 0/0 4/12

Warm Start

Backend Count (r/w) Total Ms (r/w) Avg Ms (r/w) Median ms (r/w) Max Ms (r/w)
Redis 499/194 8797/984 18/5 1/2 392/217
File 499/194 9360/1179 19/6 1/3 389/218
SQLite 499/194 220/19 0/0 0/0 13/1

(Apple M1 16Gb)

@zharinov zharinov mentioned this pull request Jan 12, 2024
Merged
6 tasks
@zharinov zharinov requested review from viceice and rarkins January 12, 2024 01:34
@zharinov
Copy link
Collaborator Author

viceice
viceice reviewed Jan 12, 2024
View reviewed changes
Copy link
Member

@viceice viceice left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like it, how to distribute it for multiple runners? or when a single runner is running on different hosts? 🤔

lib/util/cache/package/sqlite.ts Outdated Show resolved Hide resolved
@rarkins
Copy link
Collaborator

rarkins commented Jan 12, 2024

I think this can be for one instance only

@zharinov zharinov marked this pull request as ready for review January 12, 2024 10:47
rarkins
rarkins reviewed Jan 12, 2024
View reviewed changes
lib/config/types.ts Outdated Show resolved Hide resolved
rarkins
rarkins requested changes Jan 12, 2024
View reviewed changes
lib/util/cache/package/sqlite.ts Outdated Show resolved Hide resolved
lib/util/cache/package/sqlite.ts Outdated Show resolved Hide resolved
@zharinov zharinov marked this pull request as draft January 12, 2024 13:24
@zharinov
Copy link
Collaborator Author

Use sqlite-native time function instead of Date.now

@viceice
Copy link
Member

viceice commented Jan 12, 2024

I think this can be for one instance only

normally sqlite can handle multiple processes accessing the same database

@zharinov zharinov marked this pull request as ready for review January 12, 2024 20:50
rarkins
rarkins reviewed Jan 13, 2024
View reviewed changes
lib/util/cache/package/sqlite.ts Show resolved Hide resolved
lib/util/cache/package/sqlite.ts Outdated Show resolved Hide resolved
rarkins
rarkins requested changes Jan 19, 2024
View reviewed changes
Copy link
Collaborator

@rarkins rarkins left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Log when initializing too. Ideally logging if an existing one was found or new one created

viceice
viceice reviewed Jan 19, 2024
View reviewed changes
lib/util/cache/package/index.spec.ts Outdated Show resolved Hide resolved
lib/workers/global/initialize.ts Outdated Show resolved Hide resolved
docs/usage/self-hosted-experimental.md Show resolved Hide resolved
@zharinov zharinov requested review from viceice and rarkins January 23, 2024 19:58
@viceice viceice added this pull request to the merge queue Jan 30, 2024
Merged via the queue into renovatebot:main with commit 0378bf7 Jan 30, 2024
36 checks passed
@viceice viceice deleted the feat/sqlite-package-cache branch January 30, 2024 08:02
@renovate-release
Copy link
Collaborator

🎉 This PR is included in version 37.160.0 🎉

The release is available on:

Your semantic-release bot 📦🚀

rvanbutselaar pushed a commit to rvanbutselaar/renovate that referenced this pull request Feb 2, 2024
@zharinov @viceice
feat: SQLite-based package cache (renovatebot#26608)
2f84df6 
Co-authored-by: Michael Kriese <[email protected]>
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Mar 1, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers

@rarkins rarkins rarkins approved these changes

@viceice viceice viceice approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@zharinov @rarkins @viceice @renovate-release