Loads of issues resulting from use of VkFFT

[inducer]

Issues showed up after #114:

• Catastrophic slowness: Loads of issues resulting from use of VkFFT #129 (comment)
• Missing global barrier in some cases (FYI @isuruf this is not #564, it's a problem I haven't talked about yet)
  • This was a un-upgraded loopy
• Crashes, e.g. https://github.com/inducer/pytential/runs/7625741382?check_suite_focus=true
  • pocl crash in macos only with out of bounds access pocl/pocl#1084
  • Add a fallback for FFT when VkFFT is not found or broken #130
• VkFFT errors on Mac: Loads of issues resulting from use of VkFFT #129 (comment)
• Intermittent VkFFT build failures: https://gitlab.tiker.net/inducer/sumpy/-/jobs/443459

cc @isuruf

[inducer]

Pytential's Mac CI on main is failing now, too: https://github.com/inducer/pytential/runs/7694158164?check_suite_focus=true#step:3:1057

A number of @alexfikl's PRs had picked up similar failures, and this establishes that the issue is not specific to them.

I conjecture that this is connected to the #114 merge. To help back up that conjecture, here's a run without that merge: inducer/pytential#175. If there are no failures there, then that is further evidence that #114 is causing problems.

The CI runs don't necessarily back up the slowness claim.

• Here's a CI summary page pre-merge: https://github.com/inducer/pytential/actions/runs/2724593425
• and post merge: https://github.com/inducer/pytential/actions/runs/2770799383

At least the overall CI times are relatively similar. @alexfikl, could you provide a way to reproduce your Stokes run that got substantially slower?

The missing barrier is something I encountered in work with @rckirby. I still have to work out what's happening, I will report back here.

[inducer]

The first run of inducer/pytential#175 passed. I've just started another run as an additional data point.

[inducer]

Second run passed as well.

[alexfikl]

Just to add more information here from my side.

First, for the slowdown, I noticed that on the 3D Stokes operator from inducer/pytential#29 (slightly modified and merged with main). That uses the Laplace formulation from
https://github.com/isuruf/pytential/blob/feec35e928767dd6add0cba55883cd15bed93445/pytential/symbolic/stokes.py#L578-L639
I'll try to reproduce something and come back with some numbers.

Then, for the warnings on sumpy, they were

/mnt/data/code/inducer/loopy/loopy/kernel/creation.py:1909: LoopyWarning: in kernel m2l_generate_translation_classes_dependent_data: The single-writer dependency heuristic added dependencies on instruction ID(s) 'set_d' to instruction ID 'm2l__insn_2' after kernel creation is complete. This is deprecated and may stop working in the future. To fix this, ensure that instruction dependencies are added/resolved as soon as possible, ideally at kernel creation time. (add 'single_writer_after_creation' to silenced_warnings kernel attribute to disable)

with different m2l__insn_XX in there. From what I can tell, that comes from here somewhere

sumpy/sumpy/e2e.py

Lines 560 to 565 in 4ace8ea

	[idata]: m2l_translation_classes_dependent_data[
	itr_class, idata] = \
	m2l_data(
	src_rscale,
	[idim]: d[idim],
	) {id=update,dep=set_d}

but the dependencies seem correctly declared at least.

[alexfikl]

For a reproducer of my slowness claim, this should work
https://gist.github.com/alexfikl/d16750b2407ccbc064151976c5473c53
It just evaluates the 3D Stokeslet using the Laplace kernel (Tornberg).

EDIT: Just started running in pre-114 and it seems to be equally slow. I'll post the numbers once it finishes 😢

EDIT2: I take that back, something definitely seems off. Pre-114 it was giving

elapsed: 2094.80s wall 0.91x CPU
elapsed: 1917.34s wall 1.01x CPU

so about 30min to compile and evaluate the Stokeslet. But now it's over an hour. I'm running this on dunkel with everything installed through conda.

EDIT3: Ok, something is very fishy, just ran that script on main and

elapsed: 23082.66s wall 1.00x CPU

which is somewhere around 6h! Let me know if you get a change to take a look or trying it out, maybe there's something silly in there :\

[isuruf]

Warnings about write races are false positives. See inducer/loopy#564

[isuruf]

macos failure is a segfault in pocl.

[inducer]

Are you able to get a backtrace? Is it in the pocl runtime or in a kernel?

[isuruf]

(lldb) bt
* thread #4, stop reason = EXC_BAD_ACCESS (code=2, address=0x70000ff87240)
  * frame #0: 0x000000014f69227e VkFFT_main.so`_pocl_kernel_VkFFT_main_workgroup + 7886
    frame #1: 0x0000000136ce7c47 libpocl-devices-pthread.so`work_group_scheduler + 727
    frame #2: 0x0000000136ce723e libpocl-devices-pthread.so`pocl_pthread_driver_thread + 334
    frame #3: 0x00007fff5137c661 libsystem_pthread.dylib`_pthread_body + 340
    frame #4: 0x00007fff5137c50d libsystem_pthread.dylib`_pthread_start + 377
    frame #5: 0x00007fff5137bbf9 libsystem_pthread.dylib`thread_start + 13

[inducer]

So does pocl miscompile vkfft?

[inducer]

Is there an easy way to reproduce this? Will it reproduce on the CEESD M1? Should we start thinking about reverting #114 for now?

[inducer]

Or maybe turn off vkfft on Mac?

[isuruf]

I can reproduce on appletini.

[isuruf]

Running the test with oclgrind passes and shows no out of bounds accesses.