grpc_test: add tests for cardinality violation #8120

eshitachandwani · 2025-02-25T09:59:02Z

Fixes: #7570

Added the following tests for client streaming RPC :

Client should ensure a status internal error is returned for client-streaming RPCs if the server doesn't send a message before returning status OK. This is a cardinality violation. Skipping this test for now till the issues are fixed: Cardinaltiy violation in stream interface implementation #8119
Server can continue to receive messages after calling SendAndClose.
Recv() on client streaming client blocks till the server handler returns even after calling SendAndClose from the server handler
Client receives nil message and non-nil error when server handler returns an error after calling SendAndClose.

RELEASE NOTES: N/A

test/end2end_test.go

codecov · 2025-02-25T10:05:08Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 82.19%. Comparing base (ae2a04f) to head (fc205e8).
Report is 82 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #8120      +/-   ##
==========================================
- Coverage   82.29%   82.19%   -0.10%     
==========================================
  Files         387      417      +30     
  Lines       38967    41344    +2377     
==========================================
+ Hits        32066    33983    +1917     
- Misses       5586     5939     +353     
- Partials     1315     1422     +107

see 142 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

dfawley · 2025-02-27T21:27:31Z

test/end2end_test.go

+	earlyNil           bool // whether to return nil without calling SendAndClose().
+	recvAfterClose     bool // whether to call Recv() after calling SendAndClose().


We really really don't want to extend testServer. Could you use the stubserver package instead? testServer has a lot of bad properties (especially: it splits the testing code into two different places: the test and the testServer), and this only complicates it further by adding more ways to conditionalize its behavior.

You can follow this kind of example to use stubserver instead:

grpc-go/test/end2end_test.go

Line 4485 in d48317f

ss := &stubserver.StubServer{

Also please add a comment on testServer that it should not be used for any new tests so that this doesn't happen again.

Sorry!

purnesh42H

Looks like we need one more test case when "Server Doesn't RST_STREAM"

Server-Side:

Create a client-streaming RPC handler.
The handler calls SendAndClose with a valid response message.
The handler then attempts to call Recv.

Client-Side:

Make a client-streaming RPC call.
Continuously send messages.

Expected Outcome:

The server's Recv call should return an error indicating that the stream is closed.
The client's Send calls should eventually return an error indicating that the stream has been reset.

test/end2end_test.go

eshitachandwani · 2025-03-06T16:19:11Z

Looks like we need one more test case when "Server Doesn't RST_STREAM"

Server-Side:

Create a client-streaming RPC handler.

The handler calls SendAndClose with a valid response message.

The handler then attempts to call Recv.

Client-Side:

Make a client-streaming RPC call.

Continuously send messages.

Expected Outcome:

The server's Recv call should return an error indicating that the stream is closed.

The client's Send calls should eventually return an error indicating that the stream has been reset.

This looks like the test I have written already , the TestClientStreamingRecvAfterCloseError test , can you tell me what is the difference between the two?

purnesh42H · 2025-03-07T06:42:30Z

Looks like we need one more test case when "Server Doesn't RST_STREAM"
Server-Side:

Create a client-streaming RPC handler.

The handler calls SendAndClose with a valid response message.

The handler then attempts to call Recv.

Client-Side:

Make a client-streaming RPC call.

Continuously send messages.

Expected Outcome:

The server's Recv call should return an error indicating that the stream is closed.

The client's Send calls should eventually return an error indicating that the stream has been reset.

This looks like the test I have written already , the TestClientStreamingRecvAfterCloseError test , can you tell me what is the difference between the two?

yeah looks close. I think the for loop is not clear. We should do t.LogF when receiving error from client.Send indicating rst_stream. Is it possible to verify if rst stream was received in trailers. If its too complicated we can skip but let's see if we can. Other thing is the for loop shouldn't be indefinite. We should also do t.Fatalf if context is done. That way its clear that we are expecting an error for this test.

Having said that, then I think then we need a separate test case where client ignores the error after SendAndClose. It will slightly shorter where the handler return a static error and client just ignores it and status is ok.

test/end2end_test.go

eshitachandwani

Looks like I had not published the earlier comments.

purnesh42H

LGTM

test/end2end_test.go

purnesh42H · 2025-04-01T17:15:44Z

@dfawley for second review

dfawley · 2025-04-01T21:32:37Z

test/end2end_test.go

+// Tests  to verify that client receive a cardinality violation error for
+// client-streaming RPCs if the server doesn't send a message before returning
+// status OK.


Suggested change

// Tests to verify that client receive a cardinality violation error for

// client-streaming RPCs if the server doesn't send a message before returning

// status OK.

// Tests that a client receives a cardinality violation error for

// client-streaming RPCs if the server doesn't send a message before returning

// status OK.

(Please rewrap)

dfawley · 2025-04-01T21:36:22Z

test/end2end_test.go

+			})
+			_, err := stream.Recv()
+			if err == nil {
+				logger.Fatalf("stream.Recv() = %v, want an error", err)


You want t.Errorf not logger.Fatalf here. logger.Fatalf kills the whole process. t.Fatalf kills the current goroutine. This won't run in the main test goroutine, so t.Fatalf would just cause problems, not terminate the test.

dfawley · 2025-04-01T21:38:01Z

test/end2end_test.go

+			})
+			_, err := stream.Recv()
+			if err == nil {
+				logger.Fatalf("stream.Recv() = %v, want an error", err)


Nit: "= %v", err -- err is nil. Just put nil in the string.

dfawley · 2025-04-01T21:38:31Z

test/end2end_test.go

+// Test to verify for client-streaming RPCs, when SendAndClose is called, the server
+// should reset stream after sending the response message successfully.


You don't appear to be validating RST_STREAM is written to the connection. You'll need to use the serverTester for that.

dfawley · 2025-04-01T21:39:05Z

test/end2end_test.go

+	if status.Code(err) != codes.OK {
+		t.Fatalf("stream.CloseAndRecv() = %v, want error %s", err, codes.OK)
+	}


status.Code(nil) is always codes.OK so this doesn't do anything.

dfawley · 2025-04-01T21:41:30Z

test/end2end_test.go

+	// TODO : https://github.com/grpc/grpc-go/issues/8119 - remove `t.Skip()`
+	// after this is fixed.
+	t.Skip()


Based on the test description above...why does this fail today? I don't think this should be a problem -- I was expecting that it's impossible for the server to do anything else.

I am not sure what you mean , because if I send an error code , example Unimplemented , according to the test , it is being reflected in the client when CloseAndRecv() is called by client.
Also in the the code, SendAndCLose() doesnt look like its doing anything to close , just calling SendMsg() . It also says to return nil after calling SendAndCLose() or return an error ,look here so I would think that returning an error after SendAndClose() would surely propogate to client and this is the intended behavior and this test is not needed.

I see. As mentioned offline, I'm worried that we won't be able to ever change the behavior of SendAndClose to end the RPC, even though that is what it claims to do. So we should instead confirm that SendAndClose doesn't send the TRAILERS on success. And that returning an error from the handler after SendAndClose results in a TRAILERS frame being sent that contains the error.

How the client handles that may depend on the type of stream from the client's view. E.g. if the client is using it as a bidi streaming RPC, then it should see the error trailers. But if it's a client streaming call, then it might get some other kind of error, since a message and an error is not expected for client streaming RPCs.

dfawley · 2025-04-01T21:41:53Z

test/end2end_test.go

+			sum := 0
+			in, _ := stream.Recv()
+			p := in.GetPayload().GetBody()
+			sum += len(p)


sum := len(p) ? But why is it a sum? Probably this was left over from an old iteration. You can just delete it and return len(p) directly in the response message, too. Or, better, just return 0 in the response. The value is unimportant.

dfawley · 2025-04-01T21:44:16Z

test/end2end_test.go

+	payload, err := newPayload(testpb.PayloadType_COMPRESSABLE, 1)
+	if err != nil {
+		t.Fatal(err)
+	}


Seems easier to just delete this. Send empty requests instead...

Less code is usually better code.

dfawley · 2025-04-01T21:45:00Z

test/end2end_test.go

+		select {
+		case <-ctx.Done():
+			t.Fatal("timed out waiting for error from server")
+		default:
+		}


This can be replaced by doing:

for ctx.Err() == nil { if err := stream.Send().... }

dfawley · 2025-04-01T21:45:48Z

test/end2end_test.go

+		}
+		if err != nil {


If you use an else if here, then you can use err := in the if. That's more natural unless it's important to keep err beyond the loop -- but it isn't.

dfawley

Looks great, thanks! Just a bunch of little things.

dfawley · 2025-04-24T21:43:05Z

test/end2end_test.go

+	t.Skip()
+	ss := &stubserver.StubServer{
+		StreamingInputCallF: func(_ testgrpc.TestService_StreamingInputCallServer) error {
+			return nil


There should be comments here and there explaining why the test is doing what it is. Like "// Immediately return a success without first sending a response message, which is illegal."

dfawley · 2025-04-24T21:45:30Z

test/end2end_test.go

+			if resp, err := stream.Recv(); err != nil || resp == nil {
+				t.Fatalf("stream.Recv() = %v, %v, want non-nil empty response, <nil>", resp, err)
+			}


Can you assert that the response matches some expectation, not just "not nil"? I.e. proto.Equal(&testpb.StreamingInputCallRequest{})?

dfawley · 2025-04-24T21:47:20Z

test/end2end_test.go

+	if err := stream.Send(&testpb.StreamingInputCallRequest{}); err != nil {
+		t.Fatalf("stream.Send(_) = %v, want <nil>", err)
+	}
+	if resp, err := stream.CloseAndRecv(); err != nil || resp == nil {


Similar here.

dfawley · 2025-04-24T21:49:50Z

test/end2end_test.go

+// Tests the behavior where Recv() can be called after SendAndClose(). Although
+// this is unexpected, we retain it for backward compatibility.


The test description shouldn't simply say "tests this scenario", it should say "tests that in this scenario this thing happens". Here it should say something like "Tests that the server can continue to receive messages after calling SendAndClose".

dfawley · 2025-04-24T21:53:12Z

test/end2end_test.go

+	ss := stubserver.StubServer{
+		StreamingInputCallF: func(stream testgrpc.TestService_StreamingInputCallServer) error {
+			if err := stream.SendAndClose(&testpb.StreamingInputCallResponse{}); err != nil {
+				t.Fatalf("stream.SendAndClose(_) = %v, want <nil>", err)


This needs to be Errorf since it's not in the main test goroutine. Otherwise it just kills the current goroutine, and keeps the test running. (I see the same thing above.)

dfawley · 2025-04-24T21:55:00Z

test/end2end_test.go

+	if err != nil {
+		t.Fatalf(".StreamingInputCall(_) = _, %v, want <nil>", err)
+	}
+	if resp, err := stream.CloseAndRecv(); status.Code(err) != codes.Internal || resp != nil {


You should also check status.Message(err) == status.Message(wantError). And it would be better to use status.Code(wantError) instead of repeating the code in it here.

dfawley

Looks great, thanks! Just a bunch of little things.

eshitachandwani added 4 commits February 17, 2025 13:09

rough tests

a38ba9a

add tests

68d25fe

skipped tests

2811ac5

remove

d551123

eshitachandwani added the Type: Testing label Feb 25, 2025

eshitachandwani added this to the 1.72 Release milestone Feb 25, 2025

eshitachandwani requested a review from dfawley February 25, 2025 09:59

eshitachandwani assigned dfawley Feb 25, 2025

arjan-bal reviewed Feb 25, 2025

View reviewed changes

test/end2end_test.go Outdated Show resolved Hide resolved

remove

6eaf44d

dfawley reviewed Feb 27, 2025

View reviewed changes

dfawley assigned eshitachandwani and unassigned dfawley Feb 27, 2025

eshitachandwani added 2 commits March 3, 2025 12:10

use stub server

988dad9

remove unused var

40abccd

eshitachandwani requested a review from purnesh42H March 3, 2025 07:42

eshitachandwani assigned purnesh42H Mar 3, 2025

change

ad7ff22

purnesh42H reviewed Mar 4, 2025

View reviewed changes

purnesh42H removed their assignment Mar 4, 2025

add test

c49420e

eshitachandwani requested a review from purnesh42H March 13, 2025 09:20

eshitachandwani assigned purnesh42H and unassigned eshitachandwani Mar 13, 2025

purnesh42H reviewed Mar 13, 2025

View reviewed changes

purnesh42H assigned eshitachandwani and unassigned purnesh42H Mar 13, 2025

eshitachandwani commented Apr 1, 2025

View reviewed changes

eshitachandwani requested a review from purnesh42H April 1, 2025 06:39

eshitachandwani assigned purnesh42H and unassigned eshitachandwani Apr 1, 2025

purnesh42H approved these changes Apr 1, 2025

View reviewed changes

test/end2end_test.go Outdated Show resolved Hide resolved

purnesh42H requested a review from dfawley April 1, 2025 17:15

purnesh42H assigned dfawley and unassigned purnesh42H Apr 1, 2025

dfawley reviewed Apr 1, 2025

View reviewed changes

dfawley assigned eshitachandwani and unassigned dfawley Apr 1, 2025

eshitachandwani modified the milestones: 1.72 Release, 1.73 Release Apr 3, 2025

eshitachandwani added 2 commits April 7, 2025 15:42

comments

a953b40

rst stream

cb1e1c8

eshitachandwani requested a review from dfawley April 11, 2025 09:12

eshitachandwani assigned dfawley and unassigned eshitachandwani Apr 11, 2025

eshitachandwani added 4 commits April 14, 2025 11:29

RST_STREAM test

dc11504

sendAndCLoseTest

b69699d

3 tests

0b57b15

test

2f99af7

dfawley approved these changes Apr 24, 2025

View reviewed changes

dfawley assigned eshitachandwani and unassigned dfawley Apr 24, 2025

address comments

fc205e8

eshitachandwani merged commit 4cedec4 into grpc:master Apr 25, 2025
15 checks passed

purnesh42H pushed a commit to purnesh42H/grpc-go that referenced this pull request May 4, 2025

grpc_test: add tests for client streaming (grpc#8120)

38fc564

		earlyNil bool // whether to return nil without calling SendAndClose().
		recvAfterClose bool // whether to call Recv() after calling SendAndClose().

		// Test to verify for client-streaming RPCs, when SendAndClose is called, the server
		// should reset stream after sending the response message successfully.

		// Tests the behavior where Recv() can be called after SendAndClose(). Although
		// this is unexpected, we retain it for backward compatibility.

grpc_test: add tests for cardinality violation #8120

grpc_test: add tests for cardinality violation #8120

Uh oh!

Conversation

eshitachandwani commented Feb 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

codecov bot commented Feb 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Choose a reason for hiding this comment

Uh oh!

purnesh42H left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

eshitachandwani commented Mar 6, 2025

Uh oh!

purnesh42H commented Mar 7, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

eshitachandwani left a comment

Choose a reason for hiding this comment

Uh oh!

purnesh42H left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

purnesh42H commented Apr 1, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

eshitachandwani commented Feb 25, 2025 •

edited

Loading

codecov bot commented Feb 25, 2025 •

edited

Loading