Introduction

Accessibility is a key feature of any widget toolkit and is very often a mandatory requirement before a technology can be adopted by some governments and businesses.  Unfortunately, for one reason or another, accessibility has remained unimplemented in OpenFX.  By implementing a simple but comprehensive accessibility prototype, we hope to explore the problem area and get our feet wet so that accessibility can be quickly and efficiently implemented in the near future.

The Challenge of Accessibility

Part of the reason that accessibility is hard to implement is that is touches many different layers from top to bottom within the toolkit.  

Native code is required to talk to the operating system accessibility API’s.  This native API is different between the different platforms.  This requires a layer of abstraction either in native code or in Java and an API to talk to native code.  In JavaFX terms, the same way that Glass wraps native concepts such as a top level window, it must also wrap accessible concepts like “the selection in a control has changed”.  However, the situation is worse for Accessibility.  

Each JavaFX control and the various different concepts it contains need to be surfaced to the screen reader.  This causes an explosion of API.  To put this in Glass terms, Glass wrapped native top level windows but did not attempt to wrap any native controls.  Accessibility essentially must do this by exposing concepts and behavior that are common to all controls.  For example, all table-like controls whether they are implemented by toolkits such as SWT, AWT/Swing or JavaFX must be selectable, traversable and indicate when interesting properties such as their selection has changed.  In essence, FX is providing a portable API to native functionality which causes complexity (see the implementation of AWT and SWT).  Fortunately, the accessibility API is smaller than every API possible in a control.

Accessibility has the further requirement that, not only must the built-in JavaFX controls be accessible, custom controls must also be accessible.  This requires FX API to indicate which parts of a custom control are things like the row in a table or the selection in a text control.  The API is used both by built-in and custom controls.

To mitigate risk and to get engineers using the tools and thinking about API, an Accessibility prototype was implemented on Windows and Mac.

Design Goals

These are the design goals for the exploration:

• Explore a minimal JavaFX API that is complete, full featured and is easily extended
• Develop rules and patterns and ensure that they work on different platforms
• Understand the low level platform accessibility API on Mac, Windows and Linux
• Explore simple low level abstractions that will help reduce native code
• Mitigate risk by exploring well known problem areas in depth

It is not a goal of this exploration to have a complete JavaFX API for Accessibility in place.  In fact, many of the names and some of the behavior will not be in the final code.  However, in each area investigated, a portable API will be in place, controls will be working and functionality explored.

Towards a Minimal Accessibility API

The underlying operating systems have many of the same concepts around accessibility.  For example, all platforms have the concepts of:

• Roles
• Attributes
• Actions
• Notifications

In many cases, accessible concepts map one-to-one with concepts in JavaFX.  For example, every accessible control has a role that the screen reader uses to tell the user what kind of control is active.  The screen reader also uses the role to decide how the control be addressed. 

There is a one-to-one mapping from the role to the class of a control.  Roles are predefined by the operating system, although some platforms have a mechanism to mix and match behavior.  Roles are static.  For example, once a control has declared that it is a button, it will not suddenly switch roles and declare itself to be a list.

Controls have attributes which correspond roughly to Java FX properties.  When the screen reader needs to discover the selection, it will query for the selection attribute.  When the selection is changed by the program, it notifies the screen reader that this has happened.  This suggests that a get/set API might be a good match, at least at the lowest level in the implementation.  Further up in stack, some attributes might map directly to FX properties.

Screen readers can request that a control perform an action.  For example, a screen reader might ask that a button be selected.  This is in addition to the normal key sequence used to activate a control (for example, press a push button).  Actions are used by voice recognition software instead of requiring a keyboard.

When an attribute has changed and the change was caused by the program, the screen reader requires notification.  This is very similar to a property change, however, notifications are always one way.  For example, when the focus changes, the screen reader must be notified so that it can read the new focus control.

Implementation

For each FX node, there is a corresponding operating system accessible object.  Also, there is a one-to-one mapping between the hierarchy of operating system accessible objects and the FX scene graph.  

While it is possibility is to have a hierarchy of accessible objects for each control (for example ButtonAccessible), this leads to an explosion of classes, extra layering, code indirection and a parallel hierarchy.  Rather than capture inheritance in another hierarchy, inheritance is captured by the regular inheritance hierarchy and runtime hierarchy of JavaFX nodes.  Methods to call out to the screen reader and methods that the screen reader calls are defined on Node.

Roles and Attributes

A natural expression for an attribute in Java is an enum.

public enum AccessibleAttribute {
     /**
     * Returns the role for the Node.
     * Type: Roles (enum)
     */
    ROLE("Role"), 
    // more here ...

}
 
// this method is in Node, it needs Javadoc
public Object queryAccessibleAttribute(AccessibleAttribute attribute, Object... parameters) {

An attribute must have a value and that value depends on the attribute itself.  Sometimes the value is a simple Java type like a Boolean.  Other times, it can be an object like a Node.  In the case of roles, a role is best expressed as another enum:

public enum AccessibleRole {
    /**
     * Node
     * Attributes: Parent, Role, Scene, Bounds, Enabled, Focused
     * Actions: (none)
     */
    NODE,
    // more here ...
}

Role is the most important attribute a control can have.  Like classes, roles behavior is inherited.  For example, all controls are nodes and all must be able to answer their parent node.

Actions and Notifications

Actions are used when the operating system calls your program to take some action on behalf of the user.  Actions are enums and a method in Node is overridden to respond to an action.

public enum AccessibleAction {
    FIRE,
    INCREMENT,
    DECREMENT,
    ...
}

@Override public void executeAccessibleAction(AccessibleAction action) {
    ...
}

Notifications are used when you are notifying the operating system that you have changed state.  This will often trigger the operating system to ask for the new state. 

Why is this not implemented as an attribute set method?  When the operating system calls you with a set method, it calls you with the new value and expects you to do the work to update the value.  In the case of a notification, you have already done the work and you need to notify the operating system that the attribute has changed.

public void notifyAccessibleAttributeChanged(AccessibleAttribute notification) {

A perfect example of notification is focus.  The operating system and native controls have the concept of a focus control.  JavaFX has the concept of focus that is different from the platform.  In JavaFX, the operating system focus is always in the top level window, while the JavaFX focus is on a control inside.  When JavaFX focus changes, the operating system must be notified.

Rules and Patterns

The following rules and patterns are being explored:

• One to one mapping between Node and Accessible:
• As much as possible, try to implement Accessibility in the Control
• Accessible code may be in Control or Control Skin where appropriate
• Some controls have “lightweights” (ie Tabs a tab folder)
• Minimal FX Accessibility API based on get/set and values (no custom API)
• Maximizing working done in Java (JNI methods call-in rather than C code)
• Minimizing state stored in C (all state stored in Java, native peer is stateless)

Low Level Abstractions to Minimize Native Code

//TODO – describe architecture

Well Known Problem Areas

The following are well known areas that will be explored:

• Focus/Navigation:  Nodes have focus inside Controls that are not reflected in FX API.  For example, “lightweight” Tabs in a TabPane do not have FX focus but need to have the Accessibility concept of focus.
• Selection:  All screen readers have the concept of selection (ie selected items in a list or the selected tab in a tab pane).  How is this manifested in a portable manner?
• Lightweights and Virtual Items:  
  • Tabs are lightweight.  Tables may have millions of items and only a small percentage is visible on the screen.  This breaks the one-to-one node mapping.
  • Parent/Child Relationship:   In complex controls, FX may have a different parent/child relationship within the skin.  Screen readers won’t work in some cases.  For example, Accessible objects representing Tabs must have the accessible object for the TabPane as their parent, pushing the implementation of accessibility into the skin.  In complex cases such as tables with rows and columns, will this be a problem?

The following are will known areas that will be explored (time permitting):

• Relations:  FX already has a labledFor property.  Can this be used?  IDEA: FX has sophisticated traversal code to detect previous and next controls for tab traversal.  Labels do not take part in traversal, but could the traversal code be tweaked such that FX could automatically identify the labelFor control dynamically and automatically for the programmer.  We would still require the API, but programmers would need to use it much less ofte.
• Menu Bar:  The Mac has a native menu bar that is part of the desktop.  Does this work with FX Accessibility?  FX allows a menu bar per window.  How can this be mapped on the Mac?

Phase I: Critical Controls

In order to explore these areas, we will implement the following kinds of controls:

• Static Control
  • Implement Text and ImageView
• Interactive Control
  • Implement Button
• Simple Control with State
  • Implement CheckBox and RadioButton
  • Implement  Slider
• Simple Text Control
  • Implement TextField and/or TextArea
• Simple Control with “LightWeights”
  • Implement TabPane and Tabs
• Complex Control with State and Virtual Items
  • Implement TableView, TreeView and/or ListView

The following controls will be explored (time permitting):

• Simple Popup Control
  • Implement ComboBox

Phase II: Important Controls

These controls need to be explored and have been identified as important, however, they are unlikely to force dramatic design changes:

• Combo
• Tree
• TableTree
• MenuBar (Mac...)
• ToolBar

Phase III: Remaining Controls

Each of these controls must be made accessible. They will require changes to the low level native code, but these changes are likely in the form of adding constants rather than implementing native methods. For tracking purposes there is a complete list of controls and their native mappings (for Windows and Mac).

Conclusion

The result of this exploration will be working code and understanding.  For example, if a certain code pattern cannot work, we will understand the pattern that will.  The actual code may or may not form the basis of accessibility in JavaFX in the future, however naming and concepts are likely to change.