Sanity Testing

Sanity Testing

To improve quality, there will be a greater emphasis on testing.  As a team, we are responsible for the quality of FX and in addition to the usual unit and automated testing, we will be performing weekly sanity tests.

Every Monday morning for about an hour or two at most, every team members will sanity test the release.  Team members will catch up on all changes from the different components that make up FX and ensure that they can build the product (or test using the latest Hudson build).  Monday morning testing is a time bounded exercise that should not interfere with regular work but will give the team a sense of over all quality in the product

Testing will involve running for Ensemble8, Modena, toys and other applications on Windows, Mac and Linux and the other platforms.  The goal is to catch regressions quickly at the time they go in.  For example, a recent change to better support the Input Method on Linux caused selected text to be deleted every time a text control lost focus.  This is the kind of thing that should be found immediately by anyone running the Linux platform.

Weekly Code Freeze / Push Rules / Milestone Week

Code will be frozen every Monday at 1:00 AM Pacific Time. The repo will remain effectively locked for most changes until 1:00 PM Pacific Time. During that time, only fixes for "stopper" bugs or test failures found during sanity testing are permitted. For some committers that are not located in the US time zones, this may result in "no push Mondays" where changesets will need to wait until the very late in the day or the next day (Tuesday) to be pushed.

Monday 1:00 AM Pacific -- repo is locked, testing can begin as soon as a build is available (or you can build it yourself) 
Monday 1:00 PM Pacific -- repo is unlocked, changesets can be pushed normally again 

During the week before a Milestone, we need to be extra careful about destabilizing the release. Changes can go in, but they require the approval of one of the Project Lead, Technical Lead(s) or Release Manager(s) (Steve, Daniel, Jasper, Kevin, Lisa, Dave) along with a pre-commit review. Approval takes the form of '+1' in JIRA.

Changes that do not affect the "bits that we ship" do not require approval.  For example, changes to example code that is not part of the deliverable or JavaDoc changes do not require approval.  Changes to things like config files, jfxrt.jar and the build process require approval.  The idea is to use common sense and ask if in doubt.

The idea behind ramping down during a milestone is to stabilize the code base but not stop all changes and disrupt work.  It is important for the leads to understand which changes are going into a milestone in order to manage risk.  As you might expect, we will be more restrictive as we approach the ship date.

Fixing Bugs Found as a Result of Sanity Testing

Serious bugs found during sanity testing are almost always regressions.  When a "stopper" bug is found, the default action is that the change set that caused the problem will be reverted.  Obviously, each particular case needs to be evaluated and the engineer who made the change will be consulted.  If reverting the fix is likely to make things more unstable, then reverting is not the best answer.  If it is a milestone week, then extra care will need to be taken when any changes are made (either reverting or fixing).  

Monday 10:00 AM Pacific - the deadline for fixing a bug, if no plan is in place for a fix, the change set will be reverted

Bugs found as a result of sanity testing that are not stoppers are expected to be fixed before the next sanity test pass.

Assignments

The following table describes sanity test assignments: