How to use getSkippedTestResults method of org.testng.reporters.EmailableReporter2.TestResult class

I had a similar requirement wherein I had to run the tests as testng jar and not through the maven test phase. Steps documented here.

HTH

You can ignore test method using the annotation @Test(enabled = false)

Known Issue : If you keep Junit & TestNG both dependencies in parallel, Then TestNG dependency causes Surefire to ignore JUnit wrapper class.

Solution : There could be multiple way of handling this like we can define 2 execution, each for TestNG & JUnit and disable one as per your need.

Can you please try with this : Please remove any direct/indirect TestNG dependency.

org.testng testng 6.14.3 test

And try to add below one -

io.cucumber cucumber-testng 3.0.0

Also, I would suggest you one more thing to keep your pom.xml clean.

Key Point :

• We shall not mix direct & transitive dependencies specially their versions! Doing so can cause unpredictable outcome.

Cucumber Execution via JUnit

You shall remove cucumber-core, cucumber-java, cucumber-jvm, cucumber-jvm-deps, gherkin as these are transitive dependencies which would get added by Maven when you add below direct (main) dependencies. Just add below 2 and one for testng shared above.

 <dependency>
    <groupId>io.cucumber</groupId>
    <artifactId>cucumber-junit</artifactId>
    <version>3.0.0</version>
</dependency>

<dependency>
    <groupId>io.cucumber</groupId>
    <artifactId>cucumber-picocontainer</artifactId>
    <version>3.0.0</version>
</dependency>

Follow the steps below:

Step# 1:

Create a custom annotation in a separate file (ie: SetTestName.java)

@Retention(RetentionPolicy.RUNTIME)
public @interface SetTestName {
    int idx() default 0;
}

Step# 2:

Create a base class implementing ITest interface of TestNG (TestNameSetter.java).

public class TestNameSetter implements ITest{
    private String newTestName = "";

    private void setTestName(String newTestName){
        this.newTestName = newTestName;
    }

    public String getTestName() {

        return newTestName;
    }


    @BeforeMethod(alwaysRun=true)
    public void getTheNameFromParemeters(Method method, Object [] parameters){
        SetTestName setTestName = method.getAnnotation(SetTestName.class);
        String testCaseName = (String) parameters[setTestName.idx()];
        setTestName(testCaseName);
    }
}

Step# 3:

Use your DataProvider like in the code snippet:

@DataProvider(name="userData")
 public Object[][] sampleDataProvider()
 {
  Object[][] data = {
    {"loginTestUS_Username","loginTestUSPass"}, 
    {"loginTestIN_Username","loginTestINPass"},
    {"loginTestJP_UserName","loginTestJPPass"}
  };

  return data;
 }



 @SetTestName(idx=0)
 @Test(dataProvider="userData")
 public void test1(String userName, String pass)
 {
     System.out.println("Testcase 1");
 }

 @SetTestName(idx=1)
 @Test(dataProvider="userData")
 public void test2(String userName, String pass)
 {
     System.out.println("Testcase 2");
 } 

That's all. Now, you will see your test name changed accordingly in the console.

Follow the link below for your query. I hope, you might get your desired answer here:

http://biggerwrench.blogspot.com/2014/02/testng-dynamically-naming-tests-from.html

Additionally, until this issue is fixed, if things still aren't working for you after following the previous advice OR you do not wish your code to be executed before every single method, then add the following code to your test class:

@Override
@BeforeSuite
protected void springTestContextPrepareTestInstance() throws Exception {
    super.springTestContextPrepareTestInstance();
}

This ensures that the Spring Context will be prepared before your @BeforeClass methods are executed.

*note, I posted this answer since in the title you're asking about @BeforeClass, even though there is no usage of @BeforeClass in your sample code.

The documentation says:

If static method mocks are used, it is required to close the initialization. Additionally, if using third-party mock-makers, other life-cycles might be handled by the open-release routine.

So no, tests won't fail under all circumstances if you don't close the mocks, but there are cases where it's necessary, so any generic test framework should make sure to close().

You shouldn't see a NPE as a test failure. You should actually assert it is not null and provide an error message,

assertNotNull(someVal, "Some descriptive message saying why value should not be null");

If you are using Maven then i would simply use the surefire plugin and run tests as part of the build:

<plugins>

    ...

      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-surefire-plugin</artifactId>
        <version>RELEASE</version>
        <configuration>
          <suiteXmlFiles>
            <suiteXmlFile>testng.xml</suiteXmlFile>
          </suiteXmlFiles>
        </configuration>
      </plugin>

    ...

</plugins>

Just specify the path to your testng.xml and take advantage of it if you can use this kind of configuration. It allows for tons of parametrization and i have used it extensively in my projects.

Check out this tutorial to get a hang of it: http://maven.apache.org/surefire/maven-surefire-plugin/examples/testng.html

Update: 4.0.0 version is available at maven central repository with bunch of changes.for more details go here.

Update: 2.2.0 version is available at maven central repository.

You can use opensource plugin cucumber-jvm-parallel-plugin which has many advantages over existing solutions. Available at maven repository

   <dependency>
     <groupId>com.github.temyers</groupId>
     <artifactId>cucumber-jvm-parallel-plugin</artifactId>
     <version>2.1.0</version>
   </dependency>

1. First you need to add this plugin with required configuration in your project pom file.

   <plugin>
     <groupId>com.github.temyers</groupId>
     <artifactId>cucumber-jvm-parallel-plugin</artifactId>
     <version>2.1.0</version>
     <executions>
        <execution>
        <id>generateRunners</id>
        <phase>generate-test-sources</phase>
        <goals>
          <goal>generateRunners</goal>
        </goals>
        <configuration>
            <!-- Mandatory -->
            <!-- comma separated list of package names to scan for glue code -->
            <glue>foo, bar</glue>
            <outputDirectory>${project.build.directory}/generated-test-sources/cucumber</outputDirectory>
             <!-- The directory, which must be in the root of the runtime classpath, containing your feature files.  -->
              <featuresDirectory>src/test/resources/features/</featuresDirectory>
             <!-- Directory where the cucumber report files shall be written  -->
             <cucumberOutputDir>target/cucumber-parallel</cucumberOutputDir>
             <!-- comma separated list of output formats json,html,rerun.txt -->
             <format>json</format>
             <!-- CucumberOptions.strict property -->
             <strict>true</strict>
             <!-- CucumberOptions.monochrome property -->
             <monochrome>true</monochrome>
             <!-- The tags to run, maps to CucumberOptions.tags property you can pass ANDed tags like "@tag1","@tag2" and ORed tags like "@tag1,@tag2,@tag3" -->
            <tags></tags>
            <!-- If set to true, only feature files containing the required tags shall be generated. -->
            <filterFeaturesByTags>false</filterFeaturesByTags>
            <!-- Generate TestNG runners instead of default JUnit ones. --> 
            <useTestNG>false</useTestNG>
            <!-- The naming scheme to use for the generated test classes.  One of 'simple' or 'feature-title' --> 
           <namingScheme>simple</namingScheme>
           <!-- The class naming pattern to use.  Only required/used if naming scheme is 'pattern'.-->
           <namingPattern>Parallel{c}IT</namingPattern>
           <!-- One of [SCENARIO, FEATURE]. SCENARIO generates one runner per scenario.  FEATURE generates a runner per feature. -->
           <parallelScheme>SCENARIO</parallelScheme>
           <!-- This is optional, required only if you want to specify a custom template for the generated sources (this is a relative path) -->
           <customVmTemplate>src/test/resources/cucumber-custom-runner.vm</customVmTemplate>
           </configuration>
          </execution>
        </executions>
      </plugin>
   

2. Now add below plugin just below above plugin which will invoke runner classes generated by above plugin

       <plugin>
           <groupId>org.apache.maven.plugins</groupId>
           <artifactId>maven-surefire-plugin</artifactId>
           <version>2.19</version>
           <configuration> 
               <forkCount>5</forkCount>
               <reuseForks>true</reuseForks>
               <includes>
                   <include>**/*IT.class</include>
               </includes>
           </configuration>
       </plugin>
   

3. Above two plugins will do magic for cucumber test running in parallel (provided you machine also have advanced hardware support).

4. Strictly provided <forkCount>n</forkCount> here 'n' is directly proportional to 1) Advanced Hardware support and 2) you available nodes i.e. registered browser instances to HUB.

5. One major and most important changes is your WebDriver class must be SHARED and you should not implement driver.quit() method, as closing is take care by shutdown hook.

   import cucumber.api.Scenario;
   import cucumber.api.java.After;
   import cucumber.api.java.Before;
   import org.openqa.selenium.OutputType;
   import org.openqa.selenium.WebDriver;
   import org.openqa.selenium.WebDriverException;
   import org.openqa.selenium.firefox.FirefoxDriver;
   import org.openqa.selenium.support.events.EventFiringWebDriver;
   
   public class SharedDriver extends EventFiringWebDriver {
       private static WebDriver REAL_DRIVER = null;
   
   
   
       private static final Thread CLOSE_THREAD = new Thread() {
           @Override
           public void run() {
               REAL_DRIVER.close();
           }
       };
   
       static {
           Runtime.getRuntime().addShutdownHook(CLOSE_THREAD);
       }
   
       public SharedDriver() {
           super(CreateDriver());
       }
   
       public static WebDriver CreateDriver() {
           WebDriver webDriver;
           if (REAL_DRIVER == null)
               webDriver = new FirefoxDriver();
           setWebDriver(webDriver);
           return webDriver;
       }
   
       public static void setWebDriver(WebDriver webDriver) {
           this.REAL_DRIVER = webDriver;
       }
   
       public static WebDriver getWebDriver() {
           return this.REAL_DRIVER;
       }
   
       @Override
       public void close() {
           if (Thread.currentThread() != CLOSE_THREAD) {
               throw new UnsupportedOperationException("You shouldn't close this WebDriver. It's shared and will close when the JVM exits.");
           }
           super.close();
       }
   
       @Before
       public void deleteAllCookies() {
           manage().deleteAllCookies();
       }
   
       @After
       public void embedScreenshot(Scenario scenario) {
           try {
               byte[] screenshot = getScreenshotAs(OutputType.BYTES);
               scenario.embed(screenshot, "image/png");
           } catch (WebDriverException somePlatformsDontSupportScreenshots) {
               System.err.println(somePlatformsDontSupportScreenshots.getMessage());
           }
       }
   }
   

6. Considering you want to execute more than 50 threads i.e. same no of browser instances are registered to HUB but Hub will die if it doesn't get enough memory therefore to avoid this critical situation you should start hub with -DPOOL_MAX=512 (or larger) as stated in grid2 documentation.

   Really large (>50 node) Hub installations may need to increase the jetty threads by setting -DPOOL_MAX=512 (or larger) on the java command line.

   java -jar selenium-server-standalone-<version>.jar -role hub -DPOOL_MAX=512