Support dynamic tests

[lounres]

Is your feature request related to a problem? Please describe.
Yes. I have a use case of creating dynamically fuzzing tests. For example, I have a lot of implementations of the same interface and I want to test them all using the same set of instructions. It would be nice to have a way to provide generated test to the kotlinx.fuzz framework.

Describe the solution you'd like
For example, special annotation like TestFactory from JUnit 5.

Additional context
No. Sorry, I guess there is nothing I can add.

[AbdullinAM]

@lounres Hey! Can you please elaborate a bit more on your idea, maybe with a concrete example? Currently I don't fully understand it

[lounres]

Hi, @AbdullinAM! Here is my use case.

I have my own library named "Kone". (So don't be scared of "Kone-" prefixes in names of interfaces, classes, and functions. It only means that these declarations are part of the Kone library.) It has its own collections. For example, there is an interface for its own lists:

public interface KoneList<out Element> {
    public operator fun get(index: UInt): Element
    public fun iteratorFrom(index: UInt): KoneListIterator<Element>
    public fun iterator(): KoneListIterator<Element> = iteratorFrom(0u)
    
    public companion object
}

Also, there are a lot of implementations of the interface: array lists, connected lists, tree lists, and so on.

The problem is that there are bugs in the implementations of the KoneList's methods. The only solution is to write tests on the interface's methods. Here kotlinx.fuzz appears and solves the problem by generating fuzz tests for the methods (of course, with special configuration, but it is not important right now). For example:

object KoneListImplementationsTest {
    @KFuzzTest
    fun `test SomeKoneListImplementation`(data: KFuzzer) {
        // 1. Create list of squares of arbitrary length.
        val list = SomeKoneListImplementation(size = data.int()) { it * it }
        // 2. Perform some operations on it. For example,
        assertEquals(25, list[5], "Sixth element in the list is not 25 for some reason")
    }
}

But here is another problem: we have to write the same test for each implementation:

object KoneListImplementationsTest {
    @KFuzzTest
    fun `test AnotherKoneListImplementation1`(data: KFuzzer) {
        // 1. Create list of squares of arbitrary length.
        val list = AnotherKoneListImplementation1(size = data.int()) { it * it }
        // 2. Perform some operations on it. For example,
        assertEquals(25, list[5], "Sixth element in the list is not 25 for some reason")
    }

    @KFuzzTest
    fun `test AnotherKoneListImplementation2`(data: KFuzzer) {
        // 1. Create list of squares of arbitrary length.
        val list = AnotherKoneListImplementation2(size = data.int()) { it * it }
        // 2. Perform some operations on it. For example,
        assertEquals(25, list[5], "Sixth element in the list is not 25 for some reason")
    }

    ...
}

For usual tests, testing frameworks like JUnit 5 or Kotest solve the problem by providing a way to describe so-called test factories. For example, there are @TestFactory annotation in JUnit 5 (see example) and withData function in Kotest (see docs). Unfortunately, these solutions don't work with kotlinx.fuzz, because kotlinx.fuzz tests depend on @KFuzzTest annotation.

That's why I would like to use some alternative to @TestFactory but for fuzz tests.

[AbdullinAM]

@lounres Got it, thanks!

We actually have this feature as one of our future plans: ability to construct objects and method call chains dynamically. Right now I can't give you an exact timeline for this, but I will try to work on it in the (relatively) near future.

In the meantime, I can suggest some workarounds for your case. We actually used something similar for our kotlinx-html tests, you can check them out here.

1. Manually encode all the available list options in the test:

@KFuzzTest
fun `test KoneList get`(data: KFuzzer) {
    // 1. Create list of squares of arbitrary length.
    val list: KoneList<Int> = when (data.int() % 10 /* number of impls we want to test */) {
        SomeKoneListImplementation(size = data.int()) { it * it }
        AnotherKoneListImplementation2(size = data.int()) { it * it }
        SomeKoneListImplementation(size = data.int()) { it * it }
        ...
    }
    // 2. Perform some operations on it. For example,
    assertEquals(25, list[5], "Sixth element in the list is not 25 for some reason")
}

2. Use some reflection library to find all objects (this is pseudocode):

@KFuzzTest
fun `test KoneList get`(data: KFuzzer) {
    // 1. Create list of squares of arbitrary length.
    val allImpls = Reflections.getAllSubclassesOf<KoneList>()
    val randomImpl = data.pick(allImpls)
    val list: KoneList<Int> = randomImpl.getConstructor(Int::class.java, kotlin.Function1::class.java).invoke(data.int(), {it * it}) as KoneList<Int>
    // 2. Perform some operations on it. For example,
    assertEquals(25, list[5], "Sixth element in the list is not 25 for some reason")
}

I understand that this is not very pleasant-looking code, but it will allow you to deal with the problem for now :)

[lounres]

I think I'll use workaround # 1. Thank you for your response!