When a change is identified that causes a regression and the best way to handle it is to back out the change, anyone can do so. It will still go through the standard code review process, but is considered a trivial change and thus it requires only one Reviewer and will avoid the 24h code review window. The idea here is to save time by not having a broken change hindering others. There is also the rationale that the change itself is automatically created by hg, and reviewed by the person who is performing the backout, so only one additional reviewer is required.

There are two parts to this task, how to do the bookkeeping in JBS, and how to do the actual backout in mercurial.

How to work with JBS when a change is backed out

1. Close the original JBS issue and mark it with "Fix Failed", which is an alternative on the "Verify" action.
2. If the intention is to fix the change and submit it again, create a redo-issue to track that the work still needs to be done. Clone the original JBS issue and use prefix [REDO] on the summary.
   • Make sure relevant information is brought to the clone.
3. Create a backout-issue:
   • Alternative 1 - a regression is identified directly. Create a Sub-Task to the redo-issue with the same summary, but prefix with [BACKOUT].
   • Alternative 2 - an investigation issue is created, and during the investigation backing out the change is identified as the best solution.
     1. Use the investigation issue for the backout.
     2. Change summary to the same as the issue to back out and prefix with [BACKOUT].
     3. Link the redo-issue and the backout-issue.
   • Alternative 3 - no redo issue was created. Create a backout-issue with the same summary, but prefix with [BACKOUT].
     1. Link the backout-issue and the original issue.

How to work with mercurial when a change is backed out

In order to backout a change, the hg backout command is recommended, which essentially applies the anti delta of the change. Make sure you perform the backout in the most upstream repostiory the change has escaped to.

hg backout [OPTION]... [-r] REV

reverse effect of earlier changeset

    Prepare a new changeset with the effect of REV undone in the current
    working directory.

    If REV is the parent of the working directory, then this new changeset is
    committed automatically. Otherwise, hg needs to merge the changes and the
    merged result is left uncommitted.

To remove noise from our testing, tests that are expected to fail are quarantined or excluded. A quarantined test is removed from standard runs, but run in a separate job. Excluded tests are not run.

Test quarantines are done in the file ProblemList.txt while exclusions shall be done in the source code using @ignore. This enables adhoc runs on local machines, it makes it possible to go back in time and get expected results, and (for good and bad) by being part of the standard code review process the visibility increases. For each quarantine and exclude, there should of course be a corresponding bug, which is referenced in the quarantine.

Quarantine jtreg tests

Use ProblemList.txt and exclude the test in the known issues ignore list.

Example where MyTest.java is excluded on windows, tracked by bug JDK-4711:

Any rule, any process has exceptions. And in some cases it is better to live with a potential failure and integrate anyway. It should of course be used in rare cases only, for example when it is more important to continue to run the test, and thus at the same time accept the failure. A typical example is an intermittent failure like an intermittent crash (say once a month) in a large system test, or if more info is needed, possibly diagnostics are added to the test, and the risk of not running it is high.

Label the bug with hs-nightly-quarantine-exception.

There are a few ways to figure out if a problem has already escaped a repository:

• The easiest way is of course if you know which change that caused the problem. Just have a look at the hg log of the upstream repository to see if the change is in there.
• Searching JBS can give information as well. See if there is more than one bug filed for the issue. If it escaped it is likely that someone else has found it and maybe filed a new bug.