How to use createValue method of org.testng.collections.ListMultiMap class

I had the same problem, and found that it helps to set the field storing test case name in the method annotated with @BeforeMethod, using native injection of TestNG to provide method name and test parameters. The test name is taken from test parameters supplied by the DataProvider. If your test method does not have parameters, just report the method name.

//oversimplified for demontration purposes
public class TestParameters {
    private String testName = null;
    private String testDescription = null;

    public TestParameters(String name,
                          String description) {
        this.testName = name;
        this.testDescription = description;
    }

    public String getTestName() {
        return testName;
    }
    public String getTestDescription() {
        return testDescription;
    }
}

public class SampleTest implements ITest {
    // Has to be set to prevent NullPointerException from reporters
    protected String mTestCaseName = "";

    @DataProvider(name="BasicDataProvider")
    public Object[][] getTestData() {
        Object[][] data = new Object[][] {
                { new TestParameters("TestCase1", "Sample test 1")},
                { new TestParameters("TestCase2", "Sample test 2")},
                { new TestParameters("TestCase3", "Sample test 3")},
                { new TestParameters("TestCase4", "Sample test 4")},
                { new TestParameters("TestCase5", "Sample test 5") }
        };
        return data;
    }

    @BeforeMethod(alwaysRun = true)
    public void testData(Method method, Object[] testData) {
        String testCase = "";
        if (testData != null && testData.length > 0) {
            TestParameters testParams = null;
            //Check if test method has actually received required parameters
            for (Object testParameter : testData) {
                if (testParameter instanceof TestParameters) {
                    testParams = (TestParameters)testParameter;
                    break;
                }
            }
            if (testParams != null) {
                testCase = testParams.getTestName();
            }
        }
        this.mTestCaseName = String.format("%s(%s)", method.getName(), testCase);
    }

    @Override
    public String getTestName() {
        return this.mTestCaseName;
    }

    @Test(dataProvider="BasicDataProvider")
    public void testSample1(TestParameters testParams){
        //test code here
    }

    @Test(dataProvider="BasicDataProvider")
    public void testSample2(TestParameters testParams){
        //test code here
    }

    @Test
    public void testSample3(){
        //test code here
    }
}

EDIT: Based on the comments below, I realized a sample from report will be useful.

Extract from the report from running code above:

<testng-results skipped="0" failed="0" total="5" passed="5">
  <suite name="SampleTests" duration-ms="2818" started-at="<some-time>" finished-at="<some-time>">
    <test name="Test1" duration-ms="2818" started-at="<some-time>" finished-at="<some-time>">
        <test-method 
            status="PASS" 
            signature="testSample1(org.example.test.TestParameters)[pri:0, instance:org.example.test.TimeTest@c9d92c]"
            test-instance-name="testSample1(TestCase5)"
            name="testSample1" 
            duration-ms="1014"
            started-at="<some-time-before>" 
            data-provider="BasicDataProvider" 
            finished-at="<some-time-later>" >
            <!-- excluded for demonstration purposes -->
        </test-method>
        <!-- the rest of test results excluded for brevity -->
    </test>
  </suite>
</testng-result>

Note, that the value returned from getTestName() method is in the test-instance-name attribute, and not in the name attribute.

Allure has built-in aggregation functionality, just put two allure results bundles in one folder and use it as a source for your report. You will get report with two suites in it.

It looks like the difference between path and location is many entries vs one. A location is a file or directory, a path can be a list.

From the manual

The location attribute specifies a single file or directory relative to the project's base directory (or an absolute filename), while the path attribute accepts colon- or semicolon-separated lists of locations. The path attribute is intended to be used with predefined paths - in any other case, multiple elements with location attributes should be preferred.

Note that the JVM used by ant has just about no relation to the JVM used by the java task. By default the environment of ant isn't the same as that of things started with the java task via ant. This is actually helpful when you want to use a different JVM from the one ant wants to use and makes things explicit, helping avoid surprises later on.

Check out the docs for the java task, particularly clonevm

clonevm: If set to true, then all system properties and the bootclasspath of the forked Java Virtual Machine will be the same as those of the Java VM running Ant. Default is "false" (ignored if fork is disabled). since Ant 1.7

You need PowerMock to do mock static methods directly. See https://github.com/jayway/powermock/wiki/TestNG_usage

You have to implement ISuiteLisnter like below:

public ISuiteListener implements ISuiteLisnter {

 private long testCount = 0 ; 
 private List<ITestNGMethod> testMethods = null;

 public void setTestCount(long testCount){
      this.testCount = testCount;
 }

 public long getTestCount(){
     return this.testCount;
 }

 @Override
 public void onStart( ISuite suite){
       testMethods  = suite.getAllMethods();
       this.testCount = testMethods.size();
 }

 @Override 
 public void onFinish( ISuite suite){

 }
}

This will give you count of all the test methods in suite which is currently running along this you can look around utility method which is present in ITestNGMethod to process.

All independent methods (that do not have @dependsOnMethods dependency) will be executed first. Then methods with dependency will be executed. If there is ambiguity in execution order even after this ordering, priority comes into picture.

This is the ordering scheme:

1. Execute all independent methods (methods without @dependsOnMethods annotation)
2. If there is ambiguity in this ordering use priority to resolve ambiguity for independent methods
3. Execute dependent methods in order of dependency
4. If there is ambiguity in this ordering use priority to resolve ambiguity for dependent methods
5. If there is still ambiguity (due to not using priority or two methods having same priority) order them based on alphabetical order.

Now all ambiguity is resolved since no two methods can have the same name.

If you're using the API, your main class is no longer a TestNG test class, just a regular Java class, so it's not surprising that Surefire no longer sees it as a test.

If you want to make some asynchronous tests, take a look at
Awaitility.

I have used it, and it does the job quite well :)

In IntelliJ, go to file -> settings -> plugins -> choose the plugin Cucumber for Java

Normally you should be able to use all annotations and IntelliJ will recognize the file-extensions