[SYCL] Implement eviction in program and kernel caches

[kbobrovs]

Programs and kernels created by SYCL runtime are recorded in internal cache-like data structures. But cache entries are never evicted, which makes it possible to exhaust resources in big long-running apps. So SYCL runtime needs to implement some eviction scheme - like LRU.

[romanovvlad]

@kbobrovs AFAIK program and kernel caches are attached to the sycl::context object which program is compiled for. So, cached values are freed when the context object is released. I still agree that we might need some additional mechanism for cache eviction during the context lifetime. Maybe we could have manual one, so a user explicitly says which program/kernel is not needed anymore?
@s-kanaev Please, correct me if I'm wrong.

[s-kanaev]

AFAIK program and kernel caches are attached to the sycl::context object which program is compiled for. So, cached values are freed when the context object is released.

That's correct. We have this code in KernelProgramCache::~KernelProgramCache

Maybe we could have manual one, so a user explicitly says which program/kernel is not needed anymore?

I think this one may be combined with general policy and the policy should be enforced/lifted with API or environment variable.

[kbobrovs]

@romanovvlad, manual management can do the job in some cases. Users can delete/re-create context. But there will always be scenarios where this is not possible or practical, I believe. I don't think this is high-priority until we hit this.

[romanovvlad]

I think this one may be combined with general policy and the policy should be enforced/lifted with API or environment variable.

@romanovvlad, manual management can do the job in some cases. Users can delete/re-create context. But there will always be scenarios where this is not possible or practical, I believe. I don't think this is high-priority until we hit this.

Just to clarify, I was thinking about adding a new method in the sycl::context class(may not only) which removes cached program for a given kernel name(or some other identification).

[kbobrovs]

I see. That would be a SYCL extension to workaround our implementation, as SYCL spec does not mandate caches.

[uditagarwal97]

Fixed in #16062