Sanity Testing

Sanity Testing

To improve quality, there will be a greater emphasis on testing.  As a team, we are responsible for the quality of FX and in addition to the usual unit and automated testing, we will be performing weekly sanity tests.

Every Thursday evening / Friday morning for about an hour or two at most, every team member will sanity test the release. Team members will catch up on all changes from the different components that make up FX and ensure that they can build the product (or test using the latest Hudson build). Friday morning testing is a time bounded exercise that should not interfere with regular work but will give the team a sense of over all quality in the product.

Testing will involve running for Ensemble8, Modena, toys and other applications on Windows, Mac and Linux. The goal is to catch regressions quickly at the time they go in.

Weekly Code Freeze / Push Rules / Milestone Week

A snapshot of the source code repository will be taken every Friday shortly after 1:00 AM Pacific Time for testing and as a candidate for integration to master for the following week's build. The repo will be locked (frozen) every Friday at 1:00 AM Pacific for one hour. No changes are allowed to be pushed to the 9-dev repo until 2:00 AM Pacific. Further, developers are strongly encouraged to avoid pushing intrusive changes (e.g., large scale refactoring, WebKit upgrades, etc.) or risky fixes from Thursday 5:00 PM Pacific through Friday 5:00 PM Pacific. Intrusive changes and risky fixes should go in prior to Thursday afternoon (or over the weekend as long as you don't break the build).

Thursday 5:00 PM Pacific -- no more risky or intrusive changes for this week
Friday 1:00 AM Pacific -- repo is locked, testing can begin as soon as a build is available (or you can build it yourself)
Friday 2:00 AM Pacific -- repo is unlocked, safe, non-intrusive changesets can be pushed again, likely for next week's integration
Friday 5:00 PM Pacific -- changesets can be pushed normally again (including intrusive or risky fixes) for next week's integration

Please see the 9 release page for more information about pushing changes into 9-dev.

Fixing Bugs Found as a Result of Sanity Testing

Serious bugs found during sanity testing are almost always regressions.  When a "stopper" bug is found, the default action is that the change set that caused the problem will be reverted.  Obviously, each particular case needs to be evaluated and the engineer who made the change will be consulted.  If reverting the fix is likely to make things more unstable, then reverting is not the best answer.

Friday 10:00 AM Pacific - the deadline for fixing a stopper bug; if no plan is in place for a fix, the changeset will be reverted

Bugs found as a result of sanity testing that are not stoppers are expected to be fixed before the next week's sanity test run.

Assignments

The following table describes sanity test assignments (* for has machine, ~ for has Vbox):