How to use onTestFailedButWithinSuccessPercentage method of org.testng.reporters.JUnitXMLReporter class

The common solution is to use an @BeforeMethod method to setup test state,

@BeforeMethod
public void setup() {
   i = 0; 
}

To run iOS tests, you can follow these steps :

(Note : I am using Java language here in Eclipse IDE and using Appium app):

1. Create a new java project in Eclipse.
2. Import jar files : Selenium Server (formerly the Selenium RC Server) version and required client driver (according to your language choice) which can be downloaded here. (To import : Right click on your project -> Properties -> Libraries -> Add External JARs. Add all three selenium jar files here.)
3. Download Appium app and launch.
4. You can run your test scripts either in simulator or on real device. To run scripts on real iOS device, you will need 'deviceName', 'platformVersion', 'UDID' (Device ID) and 'Bundle ID' (Application Bundle ID) and absolute path to the .ipa.
5. To run scripts on iOS simulator , you will need 'deviceName', 'platformVersion', path to .app and 'Bundle ID' of your app.
6. Choose capabilties in Appium app and mention the same in your script based on whether you are testing on simulator or on device.
7. Launch Appium server and then run your script.

All Appium server capabilities which can be used can be found here.

You can refer to my blog post here as well for more details to execute a sample basic script.

Are you able to set up a TestNG run configuration and try ticking the Group radio button, as per the image below?

http://www.jetbrains.com/idea/features/screenshots/70/TestNG/tngRunConfig.gif

You mentioned that you are able to run the tests from within eclipse via maven surefire plugin and you are not able to run it only when executing your main method.

This can only happen when you class LaunchTests resides in src/main/java and your suite xml file testng.xml is referring to classes within src/test/java. Java classes within src/main/java donot have visibility into java classes that reside within src/test/java. That explains why TestNG throws the exception Cannot find class in classpath

And if your LaunchTests moves into src/test/java then by default it wont be included in the uber jar (executable jar) that you are building because test classes are omitted.

So if you really want to be able to do this, you have two options

1. Move all your test classes from src/test/java into src/main/java and then work with it (Surefire plugin configuration in your pom file may need some tweaks to let it know that your test sources is not src/test/java but src/main/java.
2. You make use of something like maven jar plugin to create a jar of test classes by referring to here.

We investigated with the team and right now, our suspicion is that the XML parser is not honoring 301 redirects.

I just removed the redirection and restored the file at http://testng.org/testng-1.0.dtd, can you try again?

If this works, I'd be curious to find out why your application is fetching this DTD from the network instead of using the one shipped inside TestNG's jar file, but we can discuss that in a separate issue.

Your buildJson method is not using the parameter at all. Replace the values part with

" \"values\":[" + values + "\"]" +

Mockito supports changing the returned value; this support extends to PowerMockito. Just use OngoingStubbing.thenReturn(T value, T... values)

OngoingStubbing<T> thenReturn(T value, T... values)

Sets consecutive return values to be returned when the method is called.
E.g:

when(mock.someMethod()).thenReturn(1, 2, 3);

Last return value in the sequence (in example: 3) determines the behavior of further consecutive calls.

So, in this case, you would do:

PowerMockito.when(DBUtil.getCurrentCount()).thenReturn(100, 150);

Note: this answer assumes you already know how to mock static methods. If you do not, see this question.

Summary: Modern white-box testing uses sophisticated automated tools to help developers design good tests and measure code coverage in meaningful ways. Widely used (if rudimentary) examples include Cobertura, EMMA, EclEmma and JaCoCo.

How is white box testing done on java applications?

Traditionally, black-box testing meant that another team (not the developer) was responsible for testing the software, and they did not look at the source code while doing so.

Even with the move to unit testing, a common approach to designing tests is to use a more-or-less black-box input domain characterization, that is, just test the boundary conditions and a few arbitrary "normal" input values.

Clearly, however, this approach may fail to exercise some branches of the function. A good developer will open up the code and say "hey, I need to add an input to exercise that branch."

Modern white-box testing takes that idea a step further by automating the process, producing output like this to show which parts of a method have not yet been exercised by, say, JUnit (from the NetBeans plugin TikiOne):

What should the white box testing look like?

The code coverage tools mentioned in the summary attempt to automate the process of making sure you exercise every line or branch of code. When integrated into a continuous integration environment like Jenkins, it allows your team to keep a constant eye on not only whether your tests are passing, but whether you are meeting your coverage goals (from the NetBeans plugin TikiOne):

In software engineering research, white-box testing can get far more sophisticated than counting lines. A wide variety of tools and mathematical techniques have been developed to help you make sure that the inputs you give to your tests properly cover all the possible behaviors of your code.

For instance, warning systems for an airplane auto-pilot system that uses complex Boolean predicates might use white-box testing criteria rooted in formal logic to ensure that we cover all logically possible paths through the controller.

Analyzing line and branch coverage, the most common criteria, is a special case of using graph representations of software, such as a control flow graph or a data flow diagram. A good tool will read your code to generate the graph, and then you can choose from various graph coverage criteria to automatically recommend a set of paths through the method's control logic (branches) that you need to exercise in order to fulfill your testing goals.

Lines and branches are the simplest and most-used (control) graph coverage criteria (corresponding to covering nodes and edges) -- more thorough (and difficult to satisfy) metrics like edge-pairs and prime paths are available in some tools. These ask not just "did I go down every choice at each branch at least once," but "did I go through every branch via every way I was able to get there."

It's still up to the programmer to find a set of inputs to exercise these paths, and to design code small enough and simple enough that the number of required tests to get good coverage with the more thorough criteria doesn't explode exponentially. But now you have formal, quantitative ways to A) tell you what needs tested, and B) tell you how well you have tested your project.

Does it mean that I have to test class by class?

White-box testing is a method for designing tests. It can apply at all testing levels, including unit tests, integration tests, and system tests.

For more information on test coverage criteria, see

• Beizer, Boris. Software testing techniques. Dreamtech Press, 2003.

or see the more up-to-date

• Ammann, Paul, and Jeff Offutt. Introduction to software testing. Cambridge University Press, 2008.

TestNG or any other testing tool decides success or failure of a test based on assertion.

Assert.assertEquals(actualVal, expectedVal);

So if actualVal and expectedVal are same then test will pass else it will fail.

Similarly you will find other assertion options if you using any IDE like Eclipse.