Feature Request: Custom hook for assertion failure

[filzrev]

I want to gets actual/expected instance data when xUnit assertion failed.
And report custom diffs as test attachments.

Currently it need to add try-catch block capture these information.
So I want extensibility point to hook assertion failure event.

Expected Usage

public class ReportDiffOnFailedAttribute : BeforeAfterTestAttribute
{
    public override void Before(MethodInfo methodUnderTest, IXunitTest test)
    {
        // Add custom hook that is ivoked when assertion failed
        TestContext.Current.OnAssertionFailed = (string assertionName, object? actual, object? expected) =>
        {
            // Generate actual/expected diffs and add results to attachments.
        };
    }
}

Similar Issues

• Post failure diagnostic hook #1613
• please provide an API to get the entire expected and actual results #1713

[dmytro-moment]

@bradwilson Would you be don't mind if I'll grab the feature ?

[bradwilson]

@dmytro-moment I'm not 100% sure what I think this should look like yet, so I'd rather not have this picked up yet.

[bradwilson]

I don't think putting this on TestContext makes sense from a technical perspective, because the core functionality lives inside Assert. Even the notion of Current would imply "subscribing for just this one test" which the assertion library cannot do, as a purely static class. Further, we wouldn't want any strong tie between xunit.v3.assert (where Assert lives) and xunit.v3.core (where TestContext lives) due to the fact that these libraries are 100% independent today by design, so that developers can choose to use alternative assertion libraries without being forced to include ours.

I'm still considering designs based on subscribing to Assert, but it would be assembly-wide subscription and thus likely to be something done during assembly initialization, not as a BeforeAfterTest attribute.

[bradwilson]

I may be able to get something that's TestContext.Current level by translating a global registration by TestContext into something that's subscribed to and then dispatched locally. I don't want to promise this yet until I understand how reliable this would be to achieve.

[bradwilson]

As I'm diving into this, it's going to require breaking changes to the exception classes to be useful.

One example:

Calling Assert.Collection allows you to provide inspectors of the items. When there is an item "mismatch" it's the result of some exception thrown in the inspector lambda. This is an example test that exists today:

var list = new List<int> { 42, 2112 };

var ex = Record.Exception(() =>
	Assert.Collection(list,
		item => Assert.Equal(42, item),
		item => Assert.Equal(2113, item)
	)
);

var collEx = Assert.IsType<CollectionException>(ex);
Assert.StartsWith(
	"Assert.Collection() Failure: Item comparison failure" + Environment.NewLine +
	"                 ↓ (pos 1)" + Environment.NewLine +
	"Collection: [42, 2112]" + Environment.NewLine +
	"Error:      Assert.Equal() Failure: Values differ" + Environment.NewLine +
	"            Expected: 2113" + Environment.NewLine +
	"            Actual:   2112" + Environment.NewLine +
	"            Stack Trace:",
	ex.Message
);

What kind of metadata would you expect to be available here for the failure? Right now it looks like:

• Assertion name = Collection
• Failure type = MismatchedItem
• Collection = [42, 2112]
• Index = 1
• Exception = an instance of EqualException

There's no way to get a top-level "expected" or "actual" value, because there's no guarantee that the exception thrown even has those types of values available. (It's a coincidence that our test uses Assert.Equal here.) Further, none of the exception types have any values in them other than the fully formatted message string, so inspecting the exception isn't useful as-is today.

So the real question I'm left with here is: who is responsible for storing the information about the failure in a way that's useful to the end user? One thought would be to convert all the exceptions to have a mechanism to return the appropriate metadata, such that every time we're throwing an exception from an assertion, we can query that exception for its metadata.

Some exceptions are not set up for this today. EqualException is a great example, as it turns out. If you call EqualException.ForMismatchedCollections, then you're expected to pass already-string-formatted collections for expected and actual (formatting them involves a lot of logic like determining which part of the collection to display). So methods like this would be required instead to pass in a lot more information, to that we could both make the collection available as well as format the result for display in the exception string. That would obviously be breaking changes to the factory methods for the exceptions.

This is just my way of saying "this is not a trivial feature request" so you have expectations set correctly as to whether it gets prioritized and how long that effort will take.