To remove noise from our testing, tests that are expected to fail are quarantined or excluded. A quarantined test is removed from standard runs, but run in a separate job. Excluded tests are not run. Test quarantines are done in the file ProblemList.txt while exclusions shall be done in the source code using @ignore. This enables adhoc runs on local machines, it makes it possible to go back in time and get expected results, and (for good and bad) by being part of the standard code review process the visibility increases. For each quarantine and exclude, there should of course be a corresponding bug, which is referenced in the quarantine. Quarantine jtreg testsUse ProblemList.txt and exclude the test in the known issues ignore list. Example where MyTest.java is excluded on windows, tracked by bug JDK-4711: sun.tools.jcmd.MyTest.java 4711 windows-all
|
Exclude jtreg testsUse @Ignore with a bug reference in the test case, to prevent the test from being run. Example where MyTest.java is excluded, tracked by bug JDK-4711: /**
* @test
* @ignore 4711
*
|
| Info |
|---|
@ignore should always be placed directly before the first @run line in the test.
|
| Tip |
|---|
titleDealing with JBS bugs for test exclusionWith the quarantine/exclude mechanism in the source code, a check-in into the repository is needed, which means a unique JBS issue and a code review is needed. This is a good thing since it makes the test problems visible. - Code review: Since it is a trivial change, it only needs a review from one official Reviewer and don't need to wait 24 hour before commit.
- JBS issue: A JBS issue is created for the bug. Create a subtask for the test exclusion checkin.
| Note |
|---|
| Removing tests should not be the standard thing to do. A failing test is expected to be a regression, and should be handled promptly with high priority. Preferably it is fixed, possibly it is backed out. |
| Tip |
|---|
| title | Triage quarantined issues |
|---|
| | After a failure is handled by a test quarantine, the JBS issue for the remaining product/test bug should be re-triaged and possibly given a new priority. This should be handled by the standard triage process. When the triage team does re-triage the issue, it should not only consider the impact of the bug itself, but also the outage on the testing the quarantine of the test results in. |
| Info |
|---|
| icon | false |
|---|
| title | Who looks at quarantine batches? |
|---|
| | The gatekeeper does not review the quarantine batches as part of daily work, the point is to reduce the load for the gatekeeper. It is the responsibility of the team who owns the bugs to keep an eye at this in general. It is for a team to be aware of the risk of quarantining tests and fix it accordingly. The batches are still run to verify the state of the quarantine batches and supply statistics on the results. It is a good idea to make sure the state of the batches don't get worse. |
|