Issues preventing running Go applications in Gramine

[meithecatte]

Go (also known as Golang) is one of the languages currently popular for writing applications in. Ideally it would be possible to run those within Gramine. However, some issues within the Go runtime make that unreliable and slow.

Syscall interface

Gramine needs to be in charge of handling all syscalls made by the application. The recommended way to achieve this is to use a specific call instruction instead of syscall. We carry patches for glibc and musl to handle this for most applications. However, because of an interesting decision made by the Go team, Go doesn't use the libc to issue syscalls. This is not the first time this has caused issues.

For Gramine, the impact is that the fallback that traps the syscall instruction will be used. This causes a significant performance degradation, as many more context switches are required for each syscall. This is signalled by the following message printed during startup:

Emulating a raw syscall instruction. This degrades performance, consider patching your application to use Gramine syscall API.

As Go binaries are fully statically linked, the only solution seems to be patching the Go toolchain to create binaries with the Gramine-specific syscall ABI. Apart from the overhead of maintaining the patches, this would be somewhat annoying for users, as for most other stacks, the build system doesn't need to know about Gramine, and the binaries produced can be run on Linux directly for testing.

Go runtime thread count

Due to SGX limitations, we need to specify the total number of threads each process in the enclave might use (this is the sgx.thread_num manifest key). However, the number of threads the Go runtime may use is unbounded. The closest we have to limiting this is the GOMAXPROCS environment variable, however this doesn't count the threads that are blocked on a syscall, so it doesn't solve the problem.

Possible solutions:

• patch Go to introduce a proper thread count limit
• as above, but campaign to have it included upstream to ease the maintenance burden
• set sgx.thread_num to a somewhat high number, cross your fingers and hope for the best (this is what's happening in practice right now)
• wait for EDMM/SGXv2, which will allow dynamically allocating more threads (requires support in hardware, the Linux kernel and Gramine)

[lejunzhu]

Just my two cents, for the syscall interface, there is another possibility: to patch the ELF file after the Go toolchain produces it. And this could be more convenient if the user only has the binary.
For example, a tool could scan the whole ELF and find the pattern "mov ...; syscall", and rewrite it as a long jump to a new piece of code, to call Gramine then jump back.

[mkow]

@lejunzhu: This method is super unreliable and hard to maintain, we don't plan to go in this direction.

[StanPlatinum]

• wait for EDMM/SGXv2, which will allow dynamically allocating more threads (requires support in hardware, the Linux kernel and Gramine)

Are there any updates on this "waiting for EDMM/SGXv2 solution"?

And AFAIK, SGX2 only allows you to manage memory dynamically. I am not sure it can allow dynamically allocating more threads.

[kailun-qin]

• wait for EDMM/SGXv2, which will allow dynamically allocating more threads (requires support in hardware, the Linux kernel and Gramine)

Are there any updates on this "waiting for EDMM/SGXv2 solution"?

Yes, it's been supported since Gramine v1.6. Pls see the related manifest option, issue #1223 and PR #1451 for details.

And AFAIK, SGX2 only allows you to manage memory dynamically. I am not sure it can allow dynamically allocating more threads.

Dynamic threading is achieved based on SGX2 that allows TCS to be added at runtime (specifically, by changing regular enclave pages into TCS pages). Pls see above and Section 3.4 of this EDMM paper.

[dimakuv]

@mkow @kailun-qin Should this be moved to Discussions?

[kailun-qin]

Should this be moved to Discussions?

Yes, make sense to me.