How to use isParallel method of org.testng.Interface IDataProviderMethod class

A naive but working solution:

List<String> received = insert_code_for_get_call_here();
List<String> sorted = received.stream().sorted().collect(Collectors.toList());
assertEquals(sorted, received);

What does this do? It takes the received values, sorts them again, and then checks if the resulting list is equal to the original list. If the original list was already sorted, this is should be true. If the received list was not sorted, they will differ.

You can create a listener implementing IMethodInterceptor interface. Which will give you an ability to access groups list from your @Test and manage your "tests to execute list" as you need. At same time ITestContext parameter allows you to access the data from testNg xml. So, you can set groups to run in default testNg manner (suite xml file); but run them depending on algorithm you implement. Something like:

import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;

import org.testng.IMethodInstance;
import org.testng.IMethodInterceptor;
import org.testng.ITestContext;
import org.testng.annotations.Test;

public class Interceptor implements IMethodInterceptor
{

    @Override
    public List<IMethodInstance> intercept(List<IMethodInstance> methods, ITestContext context)
    {
        int methCount = methods.size();
        List<IMethodInstance> result = new ArrayList<IMethodInstance>();

        for (int i = 0; i < methCount; i++)
        {
            IMethodInstance instns = methods.get(i);
            List<String> grps = Arrays.asList(instns.getMethod().getConstructorOrMethod().getMethod().getAnnotation(Test.class).groups());
//get these groups from testng.xml via context method parameter                         
            if (grps.contains("A") && grps.contains("B"))
            {
                result.add(instns);
            }                       
        }                       
        return result;
    }
}

Maven has a '-o' switch which allows you to build offline:

 -o,--offline                           Work offline

Of course, you will need to have your dependencies already cached into your $HOME/.m2/repository for this to build without errors. You can load the dependencies with:

mvn dependency:go-offline

I tried this process and it doesn't seem to fully work. I did a:

rm -rf $HOME/.m2/repository
mvn dependency:go-offline       # lot of stuff downloaded
# unplugged my network
# develop stuff
mvn install                     # errors from missing plugins

What did work however is:

rm -rf $HOME/.m2/repository
mvn install                     # while still online
# unplugged my network
# develop stuff
mvn install

Without knowing precisely which logging library you are using, I would guess that the answer is almost certainly not unless you want to explicitly specify a newline character everywhere else.

The java.util.logging (and Log4J) frameworks allow you to plugin formatters to format your statements. You'd need to specify a format which did not end with a newline character. For example, in Log4J. they give an example pattern as

 %-5p [%t]: %m%n

So if you removed everything except the %m (the %n is the newline and the other stuff includes the time and log-level; %t and %p respectively, I think), newlines would not be automatically appended to each statement.

However, this means that everywhere else, your other log statements would have to look like this:

log.info("I want this on one line\n");

No JUnit does not support test ordering, except in the manner that you say, through Suite. This only defines the order in which the test classes are executed. This has been there for a long time, including JUnit 3 Suite classes.

For a more complete explanation, there are three things we need to talk about here:

1. the ordering of tests classes within a test suite
2. the ordering of test classes when they are found by reflection by Eclipse or Maven
3. the ordering of test methods (annotated with @Test) within a test class.

The ordering of tests classes within a test suite

When you specify the list of classes to be executed in a test suite, you're defining an array, and these test classes will be executed in order, except when you're doing parallel execution. Unfortunately, this allows the introduction of dependencies between your test classes.

The ordering of test classes when they are found by reflection

When searching the classpath for classes, the order in which they are found is not guaranteed, so cannot be depended upon. It isn't actually JUnit which is doing the searching, but the Eclipse Junit plugin, or maven surefire or failsafe.

The ordering of test methods within a test class

JUnit does not guarantee the order of execution of tests within a class. Most of the time, on most JVMs pre version 7, the order in which they are found using reflection is in declaration order, ie the order they are in the file. This is the order in which they are executed. However, with JVM 7, this is no longer guaranteed, so there won't be a consistent order. There is a github issue #293 Sort test methods for predictability open with suggested solutions, and there is a thread on the junit mailing list: Alphabetizing test method run order?. So you cannot depend upon the order that tests will be executed using JUnit, but this is currently under discussion.

You could also check imports for imported Test annotation, should be:

import org.testng.annotations.Test;
@Test
public myTest(){ ... }

and not for example:

import org.junit.Test;

What about to use one data provider which returns different data - based on current test group:

@DataProvider(name = "myDataProvider")
public Object[][] testDataProvider(ITestContext context) {
    List<String> includedGroups = Arrays.asList(context.getIncludedGroups());

    if(includedGroups.contains("myGroup")) {
        return dataA;
    }
    else if (includedGroups.contains("myOtherGroup")) {
        return dataBC;
    }

    //...
}

With selenium-java dependency you get all java bindings for all browsers. So, you do not need to change anything in your pom from what you already have. The only thing that would change is your driver initialization code, which is driver = new FirefoxDriver() instead of OperaDriver.

Finally, I figured it out myself. I use javadoc combined with a little bit of annotation processing (just to distinguish the testcase groups). I use a custom doclet, which first builds the list of testcases in the following way:

private MethodDoc[] getTestMethods(RootDoc root) {
    List<MethodDoc> result = new ArrayList<MethodDoc>();
    for (ClassDoc c : root.classes()) {
        for(MethodDoc m : c.methods()) {
            if (isTest(m))
                result.add(m);
        }
    }
    return result.toArray(new MethodDoc[0]);
}

// simplified test detection
private static boolean isTest(MethodDoc m) {
    for(AnnotationDesc a: m.annotations()) {
        if (a.annotationType().name().equals("Test"))
            return true;
    }
    return false;
}

Then, for each test I retrieve the set of groups:

static Set<String> getMethodGroups(MethodDoc m) {
    Set<String> result = getGroups(m);
    result.addAll(getGroups(m.containingClass()));
    return result;
}

static Set<String> getGroups(ProgramElementDoc m) {
    Set<String> result = new HashSet<String>();
    for(AnnotationDesc a: m.annotations()) {
        if (a.annotationType().name().equals("Test")) {
            for(ElementValuePair ev : a.elementValues()) {
                if (ev.element().name().equals("groups")) {
                    String value = ev.value().toString();
                    StringTokenizer tokenizer = new StringTokenizer(value, "{}\", ");
                    while (tokenizer.hasMoreElements()) {
                        result.add(tokenizer.nextToken());
                    }
                }
            }
        }
    }
    return result;
}

The rest is just standard doclet processing. In addition, I found out that I can use custom tags directly in javadoc, e.g.

/**
 * Some description
 * @Step Step 1
 * @Step Step 2
*/
void someMethod() {}