-
Notifications
You must be signed in to change notification settings - Fork 740
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SYCL] Implement eviction for in-memory program cache #16062
[SYCL] Implement eviction for in-memory program cache #16062
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@maarquitos14 Since your last review, I have also converted the E2E test case that I added earlier into unit tests and fixed a few data races.
It took me some time to understand which program it was. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM.
@intel/dpcpp-doc-reviewers / @steffenlarsen gentle ping. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM!
@intel/llvm-gatekeepers the PR is ready to be merged. |
Fixes: CMPLRLLVM-27640, #2517
The PR implements LRU cache eviction policy for in-memory program caches.
The high-level idea is to store programs in a linked-list, called eviction list. When the program is first added to the cache, it is also added to the eviction list. When a program is fetched from cache, we move the program to the end of the eviction list. So, that the programs at the beginning of the eviction list are always least recently used.
When adding a new program to cache, we check if the size of the program cache exceeds the threshold, if so, we evict the program from cache and corresponding kernels from Kernel and fast kernel cache.
This PR also adds a new environment variable,
SYCL_IN_MEM_CACHE_EVICTION_THRESHOLD
that user can use to control the size of in-memory cache. By default, cache eviction is disabled.