discussion about buffers and views

[SimeonEhrig]

For alpaka 1, we never really defined a design idea for buffers, views and so one, means memory which can be copied/accessed between host and device.

In the alpaka 1 repository we collect some ideas to improve the current state of art without breaking to much stuff: alpaka-group/alpaka#2417

But I think for alpaka 3 we should be more radical to clarify a clear design and implement it for this kind of memory. Therefore I open this issue for discussion. So the alpaka 1 issue is for improving the current implementation and this issue is for new ideas.

cc @fwyzard @psychocoderHPC

[SimeonEhrig]

In my opinion, first we should distinguished between memory access and ownership. I wrote down the current state, because this makes some discussion easier later.

In the standard library it is soft separated with std::span with std::mdspan since C++20. So std::span and std::mdspan only describes how to access memory but never owns memory. On the other site we have std::vector and std::array. Each object of a std::vector and std::array owns memory, which is not shared with any other object. Means a std::vector will delete it's own memory without checking if something else if using it (except the memory was moved). The soft separation is, because you can also access the memory of std::vector via member functions. Since C++ 20, if your function fulfill the requirements (e.g. accessing continuous memory, do not add or delete elements ...), you should use std::span instead passing a std::vector or std::array as function arguments because this avoids unintended memory copies and also the function is fully independent how the storage was created (does not matter if the storage was allocated with std::vector or std::array).

In alpaka1, the current state is that we have views and buffers. The views works like std::span and std::mdspan. The buffers are different compare to std::vector and std::array. They works like in Python. A buffer can allocate memory or reference to existing memory. Therefore a buffer has a shared memory. If we pass a buffer via reference or copy, it will never allocate new memory. Allocating new storage from a existing buffer always requires an active allocation. The actual memory is also some kind of anonymous. You have to ask the buffer to which storage it is pointing. It is not implicit defined that the buffer is pointing to it's own memory. Also the memory does not know it's buffers. It only know how many active buffers are pointing to the storage. Therefore all buffers know, to which memory they pointing to but non of the buffers know, which other buffer points to the same memory.

[SimeonEhrig]

So first discussion point (and the reason why I wrote the big preamble), how do we define const memory access.

std::vector works like the following:

// mutable access
vector<int>[] -> int &
// const access
const vector<int>[] -> int const &
// not allowed because adding new elements would not work
vector<int const>[] -> X

std::span works like the following

// mutable access
span<int>[] -> int &
// mutable access
const span<int>[] -> int &
// const access
span<int const>[] -> int const &
// const access
const span<int const>[] -> int const &

The advantage of the std::span approach is, that we can easy keep const correctness. The user cannot accidentally remove the constnes by forgetting the const keyword in an argument. At the moment we implemented this approach in alpaka 3 and we should test it with alpaka::onHost::transform.

Nice side effect, passing temporary object via const reference is also possible with std::span. But actual not relevant because constructing an view is cheap and therefore we can also use pass as value.

void foo(std::span<int> const & s){
 s[0] = 3;
}

int main(){
  foo(std::span{ptr, 1});
}

FYI @AntonReinhard

[SimeonEhrig]

A discussion point for the next VC. Should stick on the model of view + python like buffer or should we maybe move the current C++20 approach with view + storage with exclusive owning memory. Alpaka 1 was develop before C++ introduced std::span. At the moment I have no good to point to change it, except it works more like the standard library, but I have the personal feeling that we should think about it, because it could solve some problems.

[AntonReinhard]

I think the C++20 way is preferable over what we had in alpaka 1, since it provides nice separation of concerns. Also the const-correctness, as you say, should be much easier to represent and also easier to make fool-proof.

[fwyzard]

I am not too concerned about having a Buffer that implements shared ownership or not; however I would avoid the case where copying a buffer creates a new buffer and copies its content (like std::vector does): these operations are usually costly (even only on CPU), and rarely what people intend¹.

Note: for some other "prior art" that uses shared ownership you can look a SYCL Buffers [1] and in general at SYCL "Common Reference Semantics" [2].

[1] https://github.khronos.org/SYCL_Reference/iface/buffer.html
[2] https://github.khronos.org/SYCL_Reference/iface/common-reference.html

So, I would prefer either the current alpaka approach where buffers behave like shared_ptr<T[]>, or an approach where buffers are non-copyable - only movable - and behave like unique_ptr<T[]>.

If there is the need to make copies of a buffer we could provide a utility function, like for example

auto buf2 = alpaka::copy(buf1)

that makes the copy explicit, and optionally queue ordered (including the buffer allocation)

auto buf2 = alpaka::copy(queue, buf1)

?

---

¹ and I have the impression that, as more and more people arrive to C++ from a python background, they don't consider the impact of memory operations; for example, while reviewing code changes for the CMS software that are unrelated to GPUs and heterogeneous programming, I've seen vectors being copied freely around and passed by value unnecessarily.

[fwyzard]

Instead I am more concerned about how to guarantee the correct lifetime of the memory buffers, in the case of multi-threading, or at least asynchronous operations.

See alpaka-group/alpaka#2417 and alpaka-group/alpaka#2371 for more background information.

Simple CPU-only example:

alpaka::PlatformCpu platform{};
alpaka::DevCpu device = alpaka::getDevByIdx(platform, 0);
alpaka::Queue<alpaka::DevCpu, alpaka::NonBlocking> queue(device);
{
  auto buf = alpaka::allocBuf<float, size_t>(device, 1000ul);
  alpaka::fill(queue, buf, 3.14159265f);
  alpaka::enqueue(queue, [&buf]() {
      std::this_thread::sleep_for(1000ms);
      printf("%f\n", buf[42]);
  });
}
{ 
  auto buf = alpaka::allocBuf<float, size_t>(device, 1000ul);
  alpaka::memset(queue, buf, 0x00);
}

Without any explicit alpaka::wait(queue) this prints 0.000000 instead of 3.141593 and the raises a segmentation fault.

IMHO it would be good if the alpaka memory objects (buffers and maybe views) could be robust enough to prevent this class of error.

[psychocoderHPC]

@fwyzard alpaka3 solved the dependency issue by

• allocAsync(queue) - The deallocation is enqueued into the creation queue, and all previous work in the queue is guaranteed to be executed before the memory is destroyed

      onHost::Queue queue0 = device.makeQueue();
  
      onHost::fill(queue0, dViewResults, 1);
      onHost::wait(queue0);
      {
          // Asynchronous allocation memory is in the destructor waiting for 
          // all work enqueued in the creator queue.
          auto managedView = onHost::allocAsync<float>(queue0, 10ul);
          onHost::fill(queue0, managedView, 3.14159265f);
          queue0.enqueue(
              exec,
              getFrameSpec<float>(queue0.getDevice(), managedView.getExtents()),
              KernelBundle{RaceCheckKErnel{}, dViewResults, managedView});
      }
      {
          auto managedView = onHost::allocAsync<float>(queue0, 10ul);
          onHost::fill(queue0, managedView, 42.0f);
      }

• If the work is executed in another queue and not in the creator queue the user can add an action to the ManagedViews via
  addDestructorAction() or destructorWaitFor(). This can be used for ManagedView created with onHost::alloc() too.

      onHost::Queue queue0 = device.makeQueue();
      onHost::Queue queue1 = device.makeQueue();
  
      onHost::fill(queue0, dViewResults, 1);
      onHost::wait(queue0);
      {
          auto managedView = onHost::allocAsync<float>(queue1, 10ul);
          // The destructor will wait for all work enqueued to queue0 before the destructor call
          // Can be called multiple times with different queues, but should not be 
          // called with the same queue again to avoid deallocation overhead.
          managedView.destructorWaitFor(queue0);
          // wait for the allocation
          onHost::wait(queue1);
          onHost::fill(queue0, managedView, 3.14159265f);
          queue0.enqueue(
              exec,
              getFrameSpec<float>(queue0.getDevice(), managedView.getExtents()),
              KernelBundle{RaceCheckKErnel{}, dViewResults, managedView});
      }
      {
          auto managedView = onHost::allocAsync<float>(queue0, 10ul);
          onHost::fill(queue0, managedView, 42.0f);
      }

[fwyzard]

Does it mean that alpaka3 has only allocAsync ?

[psychocoderHPC]

No it has a blocking onHost::alloc() too but there is only one MangedView implementaion therefore you can add actions to ManagesViews allocated with alloc to.
In the second example allocAsync can be replaced by alloc(device, 10ul) other code stays the same.

[fwyzard]

No it has a blocking onHost::alloc() too

Do the buffer allocated with alloc() always synchronise the device when they are destroyed ?
If they don't, they have the same issues as the current alpaka buffers.

[psychocoderHPC]

Do the buffer allocated with alloc() always synchronise the device when they are destroyed ?
If they don't, they have the same issues as the current alpaka buffers.

No, it is not implicit synchronization of the device. And if the user is not explicitly guarding against the issue and does not wait, it could lead to the data race you pointed out.
Implicit device synchronization would introduce performance implications.
As soon as unified memory is implemented, it will also mean that we need to synchronize the entire system.

You can think of destructorWaitFor() as a free() function for memory you allocated with malloc, but with the difference that you do not say when in the control flow the synchronization happened, you only say what must be fulfilled before the memory is destroyed.

If the user like, he can simply write it's own alloc.

auto myAlloc(auto device,auto extent)
{
   auto managedView = onHost::allloc(device,extent);
   managedView.destructorWaitFor(device);
   return managedView;
}

Now you would have a managed view that synchronizes the device before the memory is destroyed.

[SimeonEhrig]

I am not too concerned about having a Buffer that implements shared ownership or not; however I would avoid the case where copying a buffer creates a new buffer and copies its content (like std::vector does): these operations are usually costly (even only on CPU), and rarely what people intend¹.

¹ and I have the impression that, as more and more people arrive to C++ from a python background, they don't consider the impact of memory operations; for example, while reviewing code changes for the CMS software that are unrelated to GPUs and heterogeneous programming, I've seen vectors being copied freely around and passed by value unnecessarily.

This argumentation makes sense.

So, I would prefer either the current alpaka approach where buffers behave like shared_ptr<T[]>, or an approach where buffers are non-copyable - only movable - and behave like unique_ptr<T[]>.

Actual I would prefer the std::shared_ptr<T[]> approach, because otherwise we are bounded to references to the buffers, instead using simply values (alpaka::Buffer& vs. alpaka::Buffer).

But maybe it is a good idea to not pass around buffers. Instead the default way is to create a buffer and pass around views of it by default. At the moment, I have no strong opinion on this but this is the way, how C++20 works. Passing around std::vector is expensive, std::span is cheap (maybe not in each case, but it is a good practice ;-)).

[fwyzard]

Actual I would prefer the std::shared_ptr<T[]> approach, because otherwise we are bounded to references to the buffers, instead using simply values (alpaka::Buffer& vs alpaka::Buffer).

That's that alpaka 2 approach, so OK for me.
If we could add "null" buffers we could simplify our code, but by now we have buffers wrapped in std::optional anyway, so I won't insist.

However, in general we do pass buffers (and queues, events, etc) by reference (alpaka::Buffer&) or const reference anyway, to avoid the increment/decrement of the reference counters.

But maybe it is a good idea to not pass around buffers. Instead the default way is to create a buffer and pass around views of it by default. At the moment, I have no strong opinion on this but this is the way, how C++20 works. Passing around std::vector is expensive, std::span is cheap (maybe not in each case, but it is a good practice ;-)).

Mhm, OK, maybe ?

Because a View (like an std::span) does not guarantee that the memory remains valid 🤷🏻

[SimeonEhrig]

Requirements (from VC on 12/08)

• At least one of the storage objects should be able to be passed by value and not affect performance too much.
• Provision of storage objects with lifetime management and storage objects without lifetime management for external storage resources.
• Memory objects have a fixed size and cannot be resized.
• Lifetime
  • If a full lifetime were to be implemented, this would only be possible if the memory were used with Alpaca queue functions -> no access in raw loops.
    • Not implemented because we cannot even track the entire buffer usage in the alpaka API -> It is permissible to pass an mdspan of a memory object to a kernel exec instead of the memory object itself.
  • if synchronized allocate memory with a device, wait for the device before destroying the memory
  • if asynchronously allocate memory with a queue, wait for the queue before destroying the memory
  • for all memory objects, it is possible to add queues that must be waited for before the memory is released