Multiple time out tests are generated by fuzzer #2595

IlyaMuravjov · 2023-09-12T19:47:03Z

Description

Since #2583 fuzzer is able to construct quite complex arguments for method under test, that can take quite some time to actually create, while instrumnted-process marks executions as timed out even if the thing that actually timed out is the argument creation prior to method under test invocation.

To Reproduce

Use UtBot version from Introduce object creation using builders for fuzzers #2583
Generate tests with fuzzing set to 100% for the following method

public static boolean isNull(CharSequence charSequence) {
    if (charSequence == null) return true;
    else return false;
}

Expected behavior

Two successful test and no time out test.

Actual behavior

Two successful test and 10 time out test.

Visual proofs

public final class ClassUnderTestTest {
    ///region Test suites for executable org.example.ClassUnderTest.isNull

    ///region FUZZER: SUCCESSFUL EXECUTIONS for method isNull(java.lang.CharSequence)

    /**
     * @utbot.classUnderTest {@link ClassUnderTest}
     * @utbot.methodUnderTest {@link ClassUnderTest#isNull(CharSequence)}
     */
    @Test
    @DisplayName("isNull: arg_0 = null -> return true")
    public void testIsNullReturnsTrue() {
        boolean actual = ClassUnderTest.isNull(null);

        assertTrue(actual);
    }

    /**
     * @utbot.classUnderTest {@link ClassUnderTest}
     * @utbot.methodUnderTest {@link ClassUnderTest#isNull(CharSequence)}
     */
    @Test
    @DisplayName("isNull: arg_0 = '10\uFFFC' (mutated from '10') -> return false")
    public void testIsNullReturnsFalseWithNonEmptyString() {
        boolean actual = ClassUnderTest.isNull("10\uFFFC");

        assertFalse(actual);
    }
    ///endregion

    ///region FUZZER: TIMEOUTS for method isNull(java.lang.CharSequence)

    /**
     * @utbot.classUnderTest {@link ClassUnderTest}
     * @utbot.methodUnderTest {@link ClassUnderTest#isNull(CharSequence)}
     */
    @Test
    @DisplayName("isNull: arg_0 = InetAddress.getCanonicalHostName()")
    @Timeout(value = 1000L, unit = TimeUnit.MILLISECONDS)
    public void testIsNull() throws UnknownHostException {
        byte[] byteArray = {Byte.MIN_VALUE, Byte.MIN_VALUE, Byte.MIN_VALUE, Byte.MIN_VALUE};
        InetAddress inetAddress = getByAddress(byteArray);
        byte[] byteArray1 = {Byte.MIN_VALUE, Byte.MIN_VALUE, Byte.MIN_VALUE, Byte.MIN_VALUE, Byte.MIN_VALUE};
        inetAddress.getByAddress(byteArray1);
        inetAddress.getLocalHost();
        inetAddress.getByName("");
        byte[] byteArray2 = {Byte.MIN_VALUE, (byte) 0, (byte) -1};
        inetAddress.getByAddress("-3", byteArray2);
        inetAddress.getLoopbackAddress();
        String charSequence = inetAddress.getCanonicalHostName();
        
        /* This execution may take longer than the 1000 ms timeout
         and therefore fail due to exceeding the timeout. */
        assertTimeoutPreemptively(Duration.ofMillis(1000L), () -> ClassUnderTest.isNull(charSequence));
    }

    /**
     * @utbot.classUnderTest {@link ClassUnderTest}
     * @utbot.methodUnderTest {@link ClassUnderTest#isNull(CharSequence)}
     */
    @Test
    @DisplayName("isNull: arg_0 = InetAddress.getCanonicalHostName()")
    @Timeout(value = 1000L, unit = TimeUnit.MILLISECONDS)
    public void testIsNull1() throws UnknownHostException {
        byte[] byteArray = {Byte.MIN_VALUE, Byte.MIN_VALUE, Byte.MIN_VALUE, Byte.MIN_VALUE};
        InetAddress inetAddress = getByAddress(byteArray);
        byte[] byteArray1 = {Byte.MIN_VALUE, Byte.MIN_VALUE, Byte.MIN_VALUE, Byte.MIN_VALUE, Byte.MIN_VALUE};
        inetAddress.getByAddress(byteArray1);
        inetAddress.getLocalHost();
        inetAddress.getByName("");
        byte[] byteArray2 = {Byte.MIN_VALUE, (byte) 0, (byte) -1};
        inetAddress.getByAddress("-3", byteArray2);
        inetAddress.getLoopbackAddress();
        String charSequence = inetAddress.getCanonicalHostName();
        
        /* This execution may take longer than the 1000 ms timeout
         and therefore fail due to exceeding the timeout. */
        assertTimeoutPreemptively(Duration.ofMillis(1000L), () -> ClassUnderTest.isNull(charSequence));
    }

    /**
     * @utbot.classUnderTest {@link ClassUnderTest}
     * @utbot.methodUnderTest {@link ClassUnderTest#isNull(CharSequence)}
     */
    @Test
    @DisplayName("isNull: arg_0 = InetAddress.getCanonicalHostName()")
    @Timeout(value = 1000L, unit = TimeUnit.MILLISECONDS)
    public void testIsNull2() throws UnknownHostException {
        byte[] byteArray = {Byte.MIN_VALUE, Byte.MIN_VALUE, Byte.MIN_VALUE, Byte.MIN_VALUE};
        InetAddress inetAddress = getByAddress(byteArray);
        inetAddress.getByName("");
        String charSequence = inetAddress.getCanonicalHostName();
        
        /* This execution may take longer than the 1000 ms timeout
         and therefore fail due to exceeding the timeout. */
        assertTimeoutPreemptively(Duration.ofMillis(1000L), () -> ClassUnderTest.isNull(charSequence));
    }

    /**
     * @utbot.classUnderTest {@link ClassUnderTest}
     * @utbot.methodUnderTest {@link ClassUnderTest#isNull(CharSequence)}
     */
    @Test
    @DisplayName("isNull: arg_0 = InetAddress.getCanonicalHostName()")
    @Timeout(value = 1000L, unit = TimeUnit.MILLISECONDS)
    public void testIsNull3() throws UnknownHostException {
        byte[] byteArray = {Byte.MIN_VALUE, Byte.MIN_VALUE, Byte.MIN_VALUE, Byte.MIN_VALUE};
        InetAddress inetAddress = getByAddress(byteArray);
        String charSequence = inetAddress.getCanonicalHostName();
        
        /* This execution may take longer than the 1000 ms timeout
         and therefore fail due to exceeding the timeout. */
        assertTimeoutPreemptively(Duration.ofMillis(1000L), () -> ClassUnderTest.isNull(charSequence));
    }

    /**
     * @utbot.classUnderTest {@link ClassUnderTest}
     * @utbot.methodUnderTest {@link ClassUnderTest#isNull(CharSequence)}
     */
    @Test
    @DisplayName("isNull: arg_0 = InetAddress.getCanonicalHostName()")
    @Timeout(value = 1000L, unit = TimeUnit.MILLISECONDS)
    public void testIsNull4() throws UnknownHostException {
        byte[] byteArray = {Byte.MIN_VALUE, Byte.MIN_VALUE, Byte.MIN_VALUE, Byte.MIN_VALUE};
        InetAddress inetAddress = getByAddress(byteArray);
        String charSequence = inetAddress.getCanonicalHostName();
        
        /* This execution may take longer than the 1000 ms timeout
         and therefore fail due to exceeding the timeout. */
        assertTimeoutPreemptively(Duration.ofMillis(1000L), () -> ClassUnderTest.isNull(charSequence));
    }

    /**
     * @utbot.classUnderTest {@link ClassUnderTest}
     * @utbot.methodUnderTest {@link ClassUnderTest#isNull(CharSequence)}
     */
    @Test
    @DisplayName("isNull: arg_0 = InetAddress.getCanonicalHostName()")
    @Timeout(value = 1000L, unit = TimeUnit.MILLISECONDS)
    public void testIsNull5() throws UnknownHostException {
        byte[] byteArray = {Byte.MIN_VALUE, Byte.MIN_VALUE, Byte.MIN_VALUE, Byte.MIN_VALUE};
        InetAddress inetAddress = getByAddress(byteArray);
        String charSequence = inetAddress.getCanonicalHostName();
        
        /* This execution may take longer than the 1000 ms timeout
         and therefore fail due to exceeding the timeout. */
        assertTimeoutPreemptively(Duration.ofMillis(1000L), () -> ClassUnderTest.isNull(charSequence));
    }

    /**
     * @utbot.classUnderTest {@link ClassUnderTest}
     * @utbot.methodUnderTest {@link ClassUnderTest#isNull(CharSequence)}
     */
    @Test
    @DisplayName("isNull: arg_0 = InetAddress.getCanonicalHostName()")
    @Timeout(value = 1000L, unit = TimeUnit.MILLISECONDS)
    public void testIsNull6() throws UnknownHostException {
        byte[] byteArray = {Byte.MIN_VALUE, Byte.MIN_VALUE, Byte.MIN_VALUE, Byte.MIN_VALUE};
        InetAddress inetAddress = getByAddress(byteArray);
        String charSequence = inetAddress.getCanonicalHostName();
        
        /* This execution may take longer than the 1000 ms timeout
         and therefore fail due to exceeding the timeout. */
        assertTimeoutPreemptively(Duration.ofMillis(1000L), () -> ClassUnderTest.isNull(charSequence));
    }

    /**
     * @utbot.classUnderTest {@link ClassUnderTest}
     * @utbot.methodUnderTest {@link ClassUnderTest#isNull(CharSequence)}
     */
    @Test
    @DisplayName("isNull: arg_0 = InetAddress.getCanonicalHostName()")
    @Timeout(value = 1000L, unit = TimeUnit.MILLISECONDS)
    public void testIsNull7() throws UnknownHostException {
        byte[] byteArray = {Byte.MIN_VALUE, Byte.MIN_VALUE, Byte.MIN_VALUE, Byte.MIN_VALUE};
        InetAddress inetAddress = getByAddress(byteArray);
        String charSequence = inetAddress.getCanonicalHostName();
        
        /* This execution may take longer than the 1000 ms timeout
         and therefore fail due to exceeding the timeout. */
        assertTimeoutPreemptively(Duration.ofMillis(1000L), () -> ClassUnderTest.isNull(charSequence));
    }

    /**
     * @utbot.classUnderTest {@link ClassUnderTest}
     * @utbot.methodUnderTest {@link ClassUnderTest#isNull(CharSequence)}
     */
    @Test
    @DisplayName("isNull: arg_0 = InetAddress.getCanonicalHostName()")
    @Timeout(value = 1000L, unit = TimeUnit.MILLISECONDS)
    public void testIsNull8() throws UnknownHostException {
        byte[] byteArray = {Byte.MIN_VALUE, Byte.MIN_VALUE, Byte.MIN_VALUE, Byte.MIN_VALUE};
        InetAddress inetAddress = getByAddress(byteArray);
        String charSequence = inetAddress.getCanonicalHostName();
        
        /* This execution may take longer than the 1000 ms timeout
         and therefore fail due to exceeding the timeout. */
        assertTimeoutPreemptively(Duration.ofMillis(1000L), () -> ClassUnderTest.isNull(charSequence));
    }

    /**
     * @utbot.classUnderTest {@link ClassUnderTest}
     * @utbot.methodUnderTest {@link ClassUnderTest#isNull(CharSequence)}
     */
    @Test
    @DisplayName("isNull: arg_0 = InetAddress.getCanonicalHostName()")
    @Timeout(value = 1000L, unit = TimeUnit.MILLISECONDS)
    public void testIsNull9() throws UnknownHostException {
        byte[] byteArray = {Byte.MIN_VALUE, Byte.MIN_VALUE, Byte.MIN_VALUE, Byte.MIN_VALUE};
        InetAddress inetAddress = getByAddress(byteArray);
        String charSequence = inetAddress.getCanonicalHostName();
        
        /* This execution may take longer than the 1000 ms timeout
         and therefore fail due to exceeding the timeout. */
        assertTimeoutPreemptively(Duration.ofMillis(1000L), () -> ClassUnderTest.isNull(charSequence));
    }
    ///endregion

    ///endregion
}

The text was updated successfully, but these errors were encountered:

alisevych · 2023-09-19T14:55:45Z

Probably related issue:

Timeout tests are sometimes generated by Fuzzing for simple functions #2262

(#2532)

IlyaMuravjov added ctg-bug Issue is a bug spec-regression Regression comp-fuzzing Issue is related to the fuzzing comp-instrumented-process Issue is related to Instrumented process labels Sep 12, 2023

korifey added this to UTBot Java Sep 12, 2023

github-project-automation bot moved this to Todo in UTBot Java Sep 12, 2023

IlyaMuravjov mentioned this issue Sep 12, 2023

Introduce object creation using builders for fuzzers #2583

Open

6 tasks

alisevych mentioned this issue Sep 20, 2023

Fuzzer ignores generics of class StringMap extends HashMap<String, String> #2571

Open

olganaumenko pushed a commit that referenced this issue Oct 13, 2023

Save already created mock model in models to variables cache #2494 #2595

ca7df7c

(#2532)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Multiple time out tests are generated by fuzzer #2595

Multiple time out tests are generated by fuzzer #2595

IlyaMuravjov commented Sep 12, 2023

alisevych commented Sep 19, 2023

Multiple time out tests are generated by fuzzer #2595

Multiple time out tests are generated by fuzzer #2595

Comments

IlyaMuravjov commented Sep 12, 2023

alisevych commented Sep 19, 2023