How to use getStackTraceOutputLevelForPassedTests method of org.testng.reporters.XMLReporterConfig class

All singletons either eagerly or lazily instantiated are anti-pattern. Usage of singletons makes unit testing harder because there is no easy way to mock singleton.

Mock static method

A workaround is to use PowerMock to mock static method returning singleton instance.

Use dependency injection

A better solution is to use dependency injection. If you already use a dependency injection framework (e.g. Spring, CDI), refactor the code to make ResourceLoader a managed bean with scope singleton.

If you don't use a dependency injection framework, an easy refactoring will be to make changes to all classes using the singleton ResourceLoader:

public class MyService {

  public MyService() {
    this(ResourceLoader.getInstance());
  }

  public MyService(ResourceLoader resourceLoader) {
    this.resourceLoader = resourceLoader;
  }
}

And then in unit tests mock ResourceLoader using Mockito

ResourceLoader resourceLoader = mock(ResourceLoader.class);
when(ResourceLoader.getProperty("my-property")).thenReturn("10");
MyService myService = new MyService(resourceLoader);

Externalise configuration

Another approach is to place a file with test settings under src/test/resources. If you store settings in the src/main/resources/application.properties, a file src/test/resources/application.properties will override it.

Also, externalising configuration to a file not packaged in a JAR is a good idea. This way, file src/main/resources/application.properties will contain default properties and a file passed using command-line parameter will override these properties. So, a file with test properties will be passed as a command line parameter too. See how Spring handles externalised configuration.

Use Java System Properties

Even easier approach is to allow overriding of default properties with System Properties in the method ResourceLoader.getInstance().getProperty() and pass test properties this way

public String getProperty(String name) {
  // defaultProperties are loaded from a file on a file system:
  // defaultProperties.load(new FileInputStream(new File(filePath)));
  // or from a file in the classpath:
  // defaultProperties.load(ResourceLoader.class.getResourceAsStream(filePath));
  return System.getProperty(name, defaultProperties.get(name));
}

You can use REST Assured which makes it very easy to test and validate REST services in Java from JUnit or TestNG. E.g. let's say that a GET request to a service called "/lotto" returns the following JSON

{ "lotto":{ "lottoId":5, "winning-numbers":[2,45,34,23,7,5,3], "winners":[{ "winnerId":23, "numbers":[2,45,34,23,3,5] },{ "winnerId":54, "numbers":[52,3,12,11,18,22] }] } }

then you can make the request and validate the response (in this case that lotto id equal to 5) with REST Assured like this:

expect().body("lotto.lottoId", equalTo(5)).when().get("/lotto");

The problem is a dependency issue. In my pom.xml I had a dependency on some classes which itself depended on TestNG 5.11. This worked pretty good until I figured out that I was not able to use the @Listeners annotation.

With the command

mvn dependency:tree

I was able to figure out that I had the wrong dependencies. To get rid of them I had to change the pom.xml around the artifact dependent on TestNG 5.11:

<dependency>
    <groupId>some.group.id</groupId>
    <artifactId>the.artifact.id</artifactId>
    <vers‌​ion>1.0</version>
    <exclusions>
        <exclusion>
            <groupId>org.testng</groupId>
            <artifactId>‌​testng</artifactId>
        </exclusion>
    </exclusions>
</dependency>

After that the correct TestNG dependency was working fine and I was able to use my Listener.

On a side note: The behavior of when e.g. retryAnalyzers and Listeners are called changed dramatically over the versions from 5.11 to 6.9.6, and I guess it did so quite a few times. There are for example many questions on STO dealing with "how to fix reports for retried tests which eventually pass" - none of those approaches I found worked fine for me, so I figured out one myself. This is not exactly related to the problem, but I just want you to make sure that you double check if your listeners and analyzers are called as expected and work as intended after fixing the dependencies.

Just to let you know that i have found a solution as provided in this How to programmatically call a Maven-task

as i mentioned I did use the main class. So i did the below

-Added the maven-invoker dependency in pom.xml

<dependency>
    <groupId>org.apache.maven.shared</groupId>
    <artifactId>maven-invoker</artifactId>
  <version>2.1.1</version>
</dependency>

• Updated my main class class as below using this link- http://maven.apache.org/shared/maven-invoker/usage.html

  public class MainOne {
      public static void main(String[] args) {
          InvocationRequest request = new DefaultInvocationRequest();
          request.setPomFile( new File( "pom.xml" ));
          request.setGoals( Collections.singletonList( "install" ));
          Invoker invoker = new DefaultInvoker();
          invoker.setMavenHome(new File(System.getenv("MAVEN_HOME")));
          try {
              invoker.execute( request );
          } catch (MavenInvocationException e) {
              e.printStackTrace();
          } 
      } 
  }
  

• Then from command prompt ran the command mvn clean package shade:shade

• This created the jar in the target folder.
• Copied the jar into Project folder and ran the command java -jar myproj-0.0.1-SNAPSHOT.jar, my tests starting running perfectly!!.

You can ignore test method using the annotation @Test(enabled = false)

Instead of using your surefire configuration with a property, you can:

1. Remove the surefire configuration and replace mvn test -DtestSuite=myCustomSuite.xml by mvn test -Dsurefire.suiteXmlFiles=myCustomSuite.xml. See Surefire documentation
2. Continue to use the mvn test -Dgroups=myGroup.

As the surefire configuration will be removed, the error testSuiteXmlFiles0 has null value won't be present with the -Dgroup option.

You can also use maven profiles which will configure surefire plugin depending on which property you pass to maven.

<profiles>
  <profile>
    <id>suite</id>
    <activation>
      <property>
        <name>testSuite</name>
      </property>
    </activation>
    <plugin>
      <groupId>org.apache.maven.plugins</groupId>
      <artifactId>maven-surefire-plugin</artifactId>
      <configuration>
        <suiteXmlFiles>
          <suiteXmlFile>${testSuite}</suiteXmlFile>
        </suiteXmlFiles>
      </configuration>
    </plugin>
  </profile>
</profiles>

You need to manually setup an Authentication object and set it in the global SecurityContextHolder. You can move initialization before class' test methods, before each method or as the part of each test method.

@Before
public void setupAuth(){
    //password actually doesn't matter, meanwhile GrantedAuthorities should be necessary,
    //if you are using built-in checking functions such as "hasAnyRole" etc.
    List<GrantedAuthority> grantedAuthorities = Collections.emptyList();
    authentication = new UsernamePasswordAuthenticationToken(name, null, grantedAuthorities);

    //set authentication into static security context for proper handling by annotations
    SecurityContextHolder.getContext().setAuthentication(authentication);   
}

EDIT: OP provided interesting resource, where approach described above is working along with IInvokedMethodListener, which allows annotation-based control for Authentication object.

One way can be to use @BeforeGroups and @AfterGroups. Make your test fall in a group and then use the Before/After groups annotation to do setup/teardown once for the tests.

This is actually by design in TestNG. I see two potential options:

First, look at using a DataProvider with your tests so they can be executed repeatedly with different parameters.

Or, alter the XML layout so that each method is part of its own tag. In this configuration, the XML will be longer, but you'll be able to invoke a test method as many times as you like as part of a suite.

EDIT: More details now.

In response to your comment, here's a sample XML layout that worked for me when I was up against this same problem. Try this:

<suite name="WebServiceTestSuite">
<test name="webServiceTest">
    <classes>
        <class name="com.abcco.webservice.DivisionTest">
            <methods>
                <include name="registrationCreateTest"></include>
            </methods>
        </class>
    </classes>
</test>
<test name="webServiceTest">
    <classes>
        <class name="com.abcco.webservice.DivisionTest">
            <methods>
                <include name="registrationActivateTest"></include>
            </methods>
        </class>
    </classes>
</test>
<test name="webServiceTest">
    <classes>
        <class name="com.abcco.webservice.DivisionTest">
            <methods>
                <include name="appUserLoginTest"></include>
            </methods>
        </class>
    </classes>
</test>
<test name="webServiceTest">
    <classes>
        <class name="com.abcco.webservice.DivisionTest">
            <methods>
                <include name="appUserLogoutTest"></include>
            </methods>
        </class>
    </classes>
</test>
<test name="webServiceTest">
    <classes>
        <class name="com.abcco.webservice.DivisionTest">
            <methods>
                <include name="appUserLoginTest"></include>
            </methods>
        </class>
    </classes>
</test>
<test name="webServiceTest">
    <classes>
        <class name="com.abcco.webservice.DivisionTest">
            <methods>
                <include name="sessionValidateTest"></include>
            </methods>
        </class>
    </classes>
</test>
<test name="webServiceTest">
    <classes>
        <class name="com.abcco.webservice.DivisionTest">
            <methods>
                <include name="sessionExtendTest"></include>
            </methods>
        </class>
    </classes>
</test>
</suite>

I removed the verbose="10" parallel="none" from the <test> tags, as the layout that worked for me didn't have this... You may be able to add the verbose="10" to the <suite> tag to get any specific functionality there that you need. Have a go with this and post your results and I'll be happy to help further. Without seeing your runner configuration or knowing too much about the rest of your setup, there may be some other things to consider.

You'll find that this is a much more verbose XML layout, but it solved this same problem for me at the time. You could also consider re-arranging your code a bit and making an @Test method that calls all the methods on this XML as a single test, allowing you to reduce the length of your XML file.