Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SYCL] Implement eviction for in-memory program cache #16062

Merged
merged 14 commits into from
Nov 20, 2024

Conversation

uditagarwal97
Copy link
Contributor

@uditagarwal97 uditagarwal97 commented Nov 12, 2024

Fixes: CMPLRLLVM-27640, #2517

The PR implements LRU cache eviction policy for in-memory program caches.
The high-level idea is to store programs in a linked-list, called eviction list. When the program is first added to the cache, it is also added to the eviction list. When a program is fetched from cache, we move the program to the end of the eviction list. So, that the programs at the beginning of the eviction list are always least recently used.
When adding a new program to cache, we check if the size of the program cache exceeds the threshold, if so, we evict the program from cache and corresponding kernels from Kernel and fast kernel cache.

This PR also adds a new environment variable, SYCL_IN_MEM_CACHE_EVICTION_THRESHOLD that user can use to control the size of in-memory cache. By default, cache eviction is disabled.

sycl/source/detail/config.hpp Outdated Show resolved Hide resolved
sycl/source/detail/config.hpp Show resolved Hide resolved
sycl/source/detail/kernel_program_cache.hpp Outdated Show resolved Hide resolved
sycl/source/detail/kernel_program_cache.hpp Outdated Show resolved Hide resolved
sycl/source/detail/kernel_program_cache.hpp Outdated Show resolved Hide resolved
sycl/source/detail/kernel_program_cache.hpp Outdated Show resolved Hide resolved
sycl/source/detail/kernel_program_cache.hpp Outdated Show resolved Hide resolved
sycl/source/detail/kernel_program_cache.hpp Outdated Show resolved Hide resolved
sycl/unittests/config/ConfigTests.cpp Outdated Show resolved Hide resolved
sycl/unittests/config/ConfigTests.cpp Outdated Show resolved Hide resolved
Copy link
Contributor Author

@uditagarwal97 uditagarwal97 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@maarquitos14 Since your last review, I have also converted the E2E test case that I added earlier into unit tests and fixed a few data races.

sycl/source/detail/config.hpp Show resolved Hide resolved
sycl/source/detail/kernel_program_cache.hpp Outdated Show resolved Hide resolved
sycl/source/detail/kernel_program_cache.hpp Outdated Show resolved Hide resolved
sycl/source/detail/kernel_program_cache.hpp Show resolved Hide resolved
sycl/source/detail/kernel_program_cache.hpp Outdated Show resolved Hide resolved
sycl/source/detail/kernel_program_cache.hpp Show resolved Hide resolved
@uditagarwal97 uditagarwal97 marked this pull request as ready for review November 14, 2024 04:09
@keryell
Copy link
Contributor

keryell commented Nov 14, 2024

It took me some time to understand which program it was.
While you are on it, could you please add some top-comments to sycl/source/detail/kernel_program_cache.hpp to give an idea of what it is about?

Copy link
Contributor

@maarquitos14 maarquitos14 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

@uditagarwal97
Copy link
Contributor Author

@intel/dpcpp-doc-reviewers / @steffenlarsen gentle ping.

sycl/doc/EnvironmentVariables.md Outdated Show resolved Hide resolved
sycl/source/detail/kernel_program_cache.hpp Outdated Show resolved Hide resolved
sycl/source/detail/kernel_program_cache.hpp Show resolved Hide resolved
sycl/source/detail/kernel_program_cache.hpp Outdated Show resolved Hide resolved
sycl/source/detail/program_manager/program_manager.cpp Outdated Show resolved Hide resolved
sycl/unittests/assert/assert.cpp Outdated Show resolved Hide resolved
sycl/source/detail/program_manager/program_manager.cpp Outdated Show resolved Hide resolved
sycl/source/detail/kernel_program_cache.hpp Outdated Show resolved Hide resolved
Copy link
Contributor

@steffenlarsen steffenlarsen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@uditagarwal97
Copy link
Contributor Author

@intel/llvm-gatekeepers the PR is ready to be merged.

@steffenlarsen steffenlarsen merged commit 0e2094d into intel:sycl Nov 20, 2024
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants