test: add timeout_on_sleeping_server interop test

[plaflamme]

Motivation

I'm dealing with a conformance test that assumes that when you set (and hit) a timeout in a gRPC client, the status code you see is DEADLINE_EXCEEDED. Currently, when setting a timeout (say by using request.set_timeout), I sometimes see a CANCELLED code instead.

As far as I can tell, the gRPC interop test timeout_on_sleeping_server checks this and asserts that the status code is DEADLINE_EXCEEDED. This part of the gRPC spec isn't particularly clear, but it looks like grpc-go sometimes converts CANCELLED into DEADLINE_EXCEEDED (see here).

This PR adds this particular test to the interop suite, which I believe is equivalent to this on in grpc-go here.

Solution

I don't know how to solve this. It appears to me like the test case is flaky. I believe this is due to the fact that sometimes, the server is responding with DEADLINE_EXCEED and other times, the error is produced internally in the client as CANCELLED.

I'm hoping that by adding the test case, this can be nailed down.

[LucioFranco]

TimeoutOnSleepingServer:
  x code must be DeadlineExceeded
  in `e.code() == Code::DeadlineExceeded`: error=Status { code: Cancelled, message: "Timeout expired", source: Some(tonic::transport::Error(Transport, TimeoutExpired(()))) }

Seems like it failed like you said :D

[plaflamme]

@LucioFranco Right, as far as I can tell, the new test is correct. Though, I'm not sure why it's flaky and I sometimes see DEADLINE_EXCEEDED.

According to this guide a timeout observed in a client should lead to DEADLINE_EXCEEDED status code which in turn would lead to CANCELLED within the server.

If you have some guidance around how this can be fixed, I can take a stab at it

[LucioFranco]

I am closing this for now, if you would like to continue this PR feel free to reopen.