[Task] rsx: Fix hacky stencil mirror samplers

[kd-11]

These are really hacky right now. A proper implementation should be added taking advantage of custom border colors and clamping modes.

[digant73]

is this task related to some reported bugs/broken features or just an improvement/better code implementation?

[kd-11]

is this task related to some reported bugs/broken features or just an improvement/better code implementation?

More of "I can't believe this is still in the code, no idea how many games are broken by it".
It's one of those subtle things though, so in practice maybe just creates broken shadows in some games or some other broken effects. The technique is mostly used in postprocessing shaders anyway.