How to use testRunStarted method of org.junit.runner.notification.RunListener class

Indeed, Derkeiler.com suggests:

• Why System.exit() ?

Instead of terminating with System.exit(whateverValue), why not throw an unchecked exception? In normal use it will drift all the way out to the JVM's last-ditch catcher and shut your script down (unless you decide to catch it somewhere along the way, which might be useful someday).

In the JUnit scenario it will be caught by the JUnit framework, which will report that such-and-such test failed and move smoothly along to the next.

• Prevent System.exit() to actually exit the JVM:

Try modifying the TestCase to run with a security manager that prevents calling System.exit, then catch the SecurityException.

public class NoExitTestCase extends TestCase 
{

    protected static class ExitException extends SecurityException 
    {
        public final int status;
        public ExitException(int status) 
        {
            super("There is no escape!");
            this.status = status;
        }
    }

    private static class NoExitSecurityManager extends SecurityManager 
    {
        @Override
        public void checkPermission(Permission perm) 
        {
            // allow anything.
        }
        @Override
        public void checkPermission(Permission perm, Object context) 
        {
            // allow anything.
        }
        @Override
        public void checkExit(int status) 
        {
            super.checkExit(status);
            throw new ExitException(status);
        }
    }

    @Override
    protected void setUp() throws Exception 
    {
        super.setUp();
        System.setSecurityManager(new NoExitSecurityManager());
    }

    @Override
    protected void tearDown() throws Exception 
    {
        System.setSecurityManager(null); // or save and restore original
        super.tearDown();
    }

    public void testNoExit() throws Exception 
    {
        System.out.println("Printing works");
    }

    public void testExit() throws Exception 
    {
        try 
        {
            System.exit(42);
        } catch (ExitException e) 
        {
            assertEquals("Exit status", 42, e.status);
        }
    }
}

Update December 2012:

Will proposes in the comments using System Rules, a collection of JUnit(4.9+) rules for testing code which uses java.lang.System.
This was initially mentioned by Stefan Birkner in his answer in December 2011.

System.exit(…)

Use the ExpectedSystemExit rule to verify that System.exit(…) is called.
You could verify the exit status, too.

For instance:

public void MyTest {
    @Rule
    public final ExpectedSystemExit exit = ExpectedSystemExit.none();

    @Test
    public void noSystemExit() {
        //passes
    }

    @Test
    public void systemExitWithArbitraryStatusCode() {
        exit.expectSystemExit();
        System.exit(0);
    }

    @Test
    public void systemExitWithSelectedStatusCode0() {
        exit.expectSystemExitWithStatus(0);
        System.exit(0);
    }
}

Mockito 2 now supports final classes and methods!

But for now that's an "incubating" feature. It requires some steps to activate it which are described in What's New in Mockito 2:

Mocking of final classes and methods is an incubating, opt-in feature. It uses a combination of Java agent instrumentation and subclassing in order to enable mockability of these types. As this works differently to our current mechanism and this one has different limitations and as we want to gather experience and user feedback, this feature had to be explicitly activated to be available ; it can be done via the mockito extension mechanism by creating the file src/test/resources/mockito-extensions/org.mockito.plugins.MockMaker containing a single line:

mock-maker-inline

After you created this file, Mockito will automatically use this new engine and one can do :

 final class FinalClass {
   final String finalMethod() { return "something"; }
 }

 FinalClass concrete = new FinalClass(); 

 FinalClass mock = mock(FinalClass.class);
 given(mock.finalMethod()).willReturn("not anymore");

 assertThat(mock.finalMethod()).isNotEqualTo(concrete.finalMethod());

In subsequent milestones, the team will bring a programmatic way of using this feature. We will identify and provide support for all unmockable scenarios. Stay tuned and please let us know what you think of this feature!

The interesting thing about assertAll is that it always checks all of the assertions that are passed to it, no matter how many fail. If all pass, all is fine - if at least one fails you get a detailed result of all that went wrong (and right for that matter).

It is best used for asserting a set of properties that belong together conceptionally. Something where your first instinct would be, "I want to assert this as one".

Example

Your specific example is not an optimal use case for assertAll because checking isPrime with a prime and a non-prime is independent of each other - so much so that I would recommend writing two test methods for that.

But assume you have a simple class like an address with fields city, street, number and would like to assert that those are what you expect them to be:

Address address = unitUnderTest.methodUnderTest();
assertEquals("Redwood Shores", address.getCity());
assertEquals("Oracle Parkway", address.getStreet());
assertEquals("500", address.getNumber());

Now, as soon as the first assertion fails, you will never see the results of the second, which can be quite annoying. There are many ways around this and JUnit Jupiter's assertAll is one of them:

Address address = unitUnderTest.methodUnderTest();
assertAll("Should return address of Oracle's headquarter",
    () -> assertEquals("Redwood Shores", address.getCity()),
    () -> assertEquals("Oracle Parkway", address.getStreet()),
    () -> assertEquals("500", address.getNumber())
);

If the method under test returns the wrong address, this is the error you get:

org.opentest4j.MultipleFailuresError:
    Should return address of Oracle's headquarter (3 failures)
    expected: <Redwood Shores> but was: <Walldorf>
    expected: <Oracle Parkway> but was: <Dietmar-Hopp-Allee>
    expected: <500> but was: <16>

In a general way, what you did with @Rule and @ClassRule in JUnit 4 should be done with @ExtendWith and Extension that associated provide a very close feature in JUnit 5.
It works as standards JUnit lifecycle hooks but that it is extracted in a Extension class. And similarly to @Rule, as many Extensions as required may be added for a test class.

To handle the issue you have several possible approaches among :

• keep the JUnit 4 way (JUnit 5 owns the JUnit Vintage part that allows to execute JUnit 3 or 4 tests).
• rewrite the @Rule as an Extension.
• do the actual processing done by WireMockRule (start the server, execute your tests and stop the server) in each test of class with @BeforeEach and @AfterEach hook methods.
• use a third library that implements the equivalent of WireMockRule in the JUnit 5 Extension way such as https://github.com/lanwen/wiremock-junit5

Note that your issue already discussed in the JUnit 5 Issues.

I found out one possible way to achieve what I want, but I don't consider this to be the best possible solution as it relies on ClassPathSuite library that is not part of JUnit.

I define the test suite for slow tests like this:

@RunWith(Categories.class)
@Categories.IncludeCategory(SlowTests.class)
@Suite.SuiteClasses( { AllTests.class })
public class SlowTestSuite {
}

AllTests class is defined like this:

@RunWith(ClasspathSuite.class)
public class AllTests {
}

I had to use ClassPathSuite class from ClassPathSuite project here. It will find all the classes with tests.

Mockito 2 now supports final classes and methods!

But for now that's an "incubating" feature. It requires some steps to activate it which are described in What's New in Mockito 2:

Mocking of final classes and methods is an incubating, opt-in feature. It uses a combination of Java agent instrumentation and subclassing in order to enable mockability of these types. As this works differently to our current mechanism and this one has different limitations and as we want to gather experience and user feedback, this feature had to be explicitly activated to be available ; it can be done via the mockito extension mechanism by creating the file src/test/resources/mockito-extensions/org.mockito.plugins.MockMaker containing a single line:

mock-maker-inline

After you created this file, Mockito will automatically use this new engine and one can do :

 final class FinalClass {
   final String finalMethod() { return "something"; }
 }

 FinalClass concrete = new FinalClass(); 

 FinalClass mock = mock(FinalClass.class);
 given(mock.finalMethod()).willReturn("not anymore");

 assertThat(mock.finalMethod()).isNotEqualTo(concrete.finalMethod());

In subsequent milestones, the team will bring a programmatic way of using this feature. We will identify and provide support for all unmockable scenarios. Stay tuned and please let us know what you think of this feature!

You project class-path has to include a version of junit-jupiter-params-xxx.jar, like junit-jupiter-params-5.0.0.jar from http://central.maven.org/maven2/org/junit/jupiter/junit-jupiter-params/5.0.0/

The blog post from codefx.org you link to says (edited to the current 5.0.0 release):

Getting started with parameterized tests is pretty easy but before the fun can begin you have to add the following dependency to your project:

Group ID: org.junit.jupiter
Artifact ID: junit-jupiter-params
Version: 5.0.0

Either download and add it manually, or if you're using a build tool with dependency management (Gradle, Maven, ...) configure the build script (build.gradle, pom.xml, ...) accordingly.

Find some generic samples here: https://github.com/junit-team/junit5-samples

Starting with version 5.4.0-M1 JUnit Jupiter provides an aggregator artifact that bundles all available Jupiter-defining artifacts for easy consumption. See https://sormuras.github.io/blog/2018-12-26-junit-jupiter-aggregator.html for details.

The junitreport task uses XSLT to produce the report from the XML files generated by the junittask.

You can customize the output by specifying your own XSLT using the styledir attribute of the nested report element:

<!-- use reportstyle/junit-frames.xsl to produce the report -->
<report styledir="reportstyle" format="frames" todir="testreport"/>

For customizing the the output, one option would be to make a copy of the default XSLT and modify that. Or you could look for an alternative XSLT which is more easy to customize for your purposes.

For small changes, it might be easiest to just import the default XSLT and override whatever templates you need to customize. For example, to add a column for each test, you would need to override the template which produces the table header and the template which produces a table row. Below, I have just copied those templates and modified them a bit to add one column (look for two additions marked with <!-- ADDED -->).

<xsl:stylesheet version="1.0"
                xmlns:xsl="http://www.w3.org/1999/XSL/Transform">

  <!-- import the default stylesheet -->
  <xsl:import href="jar:file:lib/ant-junit.jar!/org/apache/tools/ant/taskdefs/optional/junit/xsl/junit-frames.xsl"/>

  <!-- override the template producing the test table header --> 
  <xsl:template name="testcase.test.header">
    <xsl:param name="show.class" select="''"/>
    <tr valign="top">
      <xsl:if test="boolean($show.class)">
        <th>Class</th>
      </xsl:if>
      <th>Name</th>
      <th>Status</th>
      <th width="80%">Type</th>
      <th nowrap="nowrap">Time(s)</th>

      <!-- ADDED -->
      <th>Screenshot</th>

    </tr>
  </xsl:template>

  <!-- override the template producing a test table row -->
  <xsl:template match="testcase" mode="print.test">
    <xsl:param name="show.class" select="''"/>
    <tr valign="top">
      <xsl:attribute name="class">
        <xsl:choose>
          <xsl:when test="error">Error</xsl:when>
          <xsl:when test="failure">Failure</xsl:when>
          <xsl:otherwise>TableRowColor</xsl:otherwise>
        </xsl:choose>
      </xsl:attribute>
      <xsl:variable name="class.href">
        <xsl:value-of select="concat(translate(../@package,'.','/'), '/', ../@id, '_', ../@name, '.html')"/>
      </xsl:variable>
      <xsl:if test="boolean($show.class)">
        <td><a href="{$class.href}"><xsl:value-of select="../@name"/></a></td>
      </xsl:if>
      <td>
        <a name="{@name}"/>
        <xsl:choose>
          <xsl:when test="boolean($show.class)">
            <a href="{concat($class.href, '#', @name)}"><xsl:value-of select="@name"/></a>
          </xsl:when>
          <xsl:otherwise>
            <xsl:value-of select="@name"/>
          </xsl:otherwise>
        </xsl:choose>
      </td>
      <xsl:choose>
        <xsl:when test="failure">
          <td>Failure</td>
          <td><xsl:apply-templates select="failure"/></td>
        </xsl:when>
        <xsl:when test="error">
          <td>Error</td>
          <td><xsl:apply-templates select="error"/></td>
        </xsl:when>
        <xsl:otherwise>
          <td>Success</td>
          <td></td>
        </xsl:otherwise>
      </xsl:choose>
      <td>
        <xsl:call-template name="display-time">
          <xsl:with-param name="value" select="@time"/>
        </xsl:call-template>
      </td>

      <!-- ADDED -->
      <td>
        <a href="link/to/screenshot/for/test/{@name}">screenshot</a>
      </td>

    </tr>
  </xsl:template>

</xsl:stylesheet>

Here's how the result looks like:

Make sure the hamcrest jar is higher on the import order than your JUnit jar.

JUnit comes with its own org.hamcrest.Matcher class that is probably being used instead.

You can also download and use the junit-dep-4.10.jar instead which is JUnit without the hamcrest classes.

mockito also has the hamcrest classes in it as well, so you may need to move\reorder it as well