Proposal: Provides Dynamic AutomationPeer

[licanhua]

Proposal: Provides Dynamic AutomationPeer

Summary

In Microsoft.UI.Xaml, we need a lot of code to create an automation peer for each Control even if the real change away from an template is just one line of code.
Below is an example of accessibility implementation in react-native, and it's very easy to use by customer.

<View
    accessible={true}
    accessibilityLabel="Accessibility label."
    accessibilityRole="button"
    accessibilityStates={['selected', 'checked']}
    accessibilityHint="Accessibility hint.">
</View>

It's possible for WinUI to provides dynamic automation peer which can allows customer to change the automation properties in an declarative way in XAML without a customized automationPeer implementation.

Note:
Microsoft has an internal proposal before on 'XAML developer can use Controls that support UIA patterns conditionally, based on their context of use'. This proposal is not used to address that problem.

<local:MyCustomControl x:Name="Title2" Grid.RowSpan="2" Content="Title2" … > 
	<local:MyCustomControl.Patterns > 
		<InvokePattern OnInvoke="InvokePattern_OnInvoke" /> 
		<ValuePattern 
			IsReadOnly="False" 
			Value="{Binding MyValue}" 
			OnSetValue="ValuePattern_OnSetValue" /> 
	</local:MyCustomControl.Patterns> 
</local:MyCustomControl> 

References:

1. Accessibility
2. accessibilitystates
3. React Native Windows Dynamic AutomationPeer

Rationale

Painful experience on RadioButtons' implementation

RadioButtons where in order to change the ListViewItem automation control type to RadioButton, To support it:

1. Subclass ListViewItemAutomationPeer just so I could return AutomationControlType::RadioButton from GetAutomationControlTypeCore()
2. Subclass ListViewItem just so I could return my peer from OnCreateAutomationPeer
3. Subclass ListView so that I could make it spin out my subclassed ListViewItems

6 files and 3 new public types all for one line of code!!

When do we requires a new automation peer?

• Change control types. For example, change Control type to a MenuBar.

    winrt::AutomationControlType MenuBarAutomationPeer::GetAutomationControlTypeCore()
    {
        return winrt::AutomationControlType::MenuBar;
    }

• Implement Control Patterns. For example, provides IExpandCollapseProvided for TreeViewAutomationPeer

class TreeViewItemAutomationPeer :
    ...
{
    ...
    // IExpandCollapseProvider
    void Collapse();
    void Expand();

• UI Automation Tree structure change. For example, Filter out unrealized peers in a collection control.

winrt::IVector<winrt::AutomationPeer> RepeaterAutomationPeer::GetChildrenCore()
{
    ...
    // Filter out unrealized peers.
    {
        for (unsigned i = 0u; i < peerCount; ++i)
        {
            auto childPeer = childrenPeers.GetAt(i);
            if (auto childElement = GetElement(childPeer, repeater))
            {
                auto virtInfo = ItemsRepeater::GetVirtualizationInfo(childElement);
                if (virtInfo->IsRealized())
                {
                    realizedPeers.push_back(std::make_pair(virtInfo->Index(), childPeer));
                }
            }
        }
    }

    // Sort peers by index.
    std::sort(realizedPeers.begin(), realizedPeers.end(), [](const auto& lhs, const auto& rhs) { return lhs.first < rhs.first; });

    // Select peers.
    ...
}

• Automation properties exposed to Narrator

winrt::ExpandCollapseState MenuBarItemAutomationPeer::ExpandCollapseState()
{
    winrt::UIElement owner = Owner();
    auto menuBarItem = owner.as<winrt::MenuBarItem>();
    if (winrt::get_self<MenuBarItem>(menuBarItem)->IsFlyoutOpen())
    {
        return winrt::ExpandCollapseState::Expanded;
    }
    else
    {
        return winrt::ExpandCollapseState::Collapsed;
    }
} 

what we don't need an new AutomationPeer, but needs code change in Control

• Localization

        winrt::AutomationProperties::SetName(m_alphaSlider, ResourceAccessor::GetLocalizedStringResource(SR_AutomationNameAlphaSlider));

• Property Change event (I didn't see we can improved it here)

void ColorPickerSlider::OnValueChangedEvent(winrt::IInspectable const& /*sender*/, winrt::RangeBaseValueChangedEventArgs const& args)
{
 ...
        winrt::ColorPickerSliderAutomationPeer peer = winrt::FrameworkElementAutomationPeer::FromElement(*this).as<winrt::ColorPickerSliderAutomationPeer>();
        winrt::get_self<ColorPickerSliderAutomationPeer>(peer)->RaisePropertyChangedEvent(oldColor, newColor, static_cast<int>(round(args.OldValue())), static_cast<int>(round(args.NewValue())));
    }
}

What's the problem of current implementation?

1. Code duplication between Control and it's automation peer. Although control and its AutomationPeer are one on one binding, in implementation, they are in separate classes. So most of time, actual implementation is in Control itself, and automation peer is just a delegated object. In AutomationPeer, we always get its owner, then query the actual control to get the

winrt::ExpandCollapseState MenuBarItemAutomationPeer::ExpandCollapseState()
{
    winrt::UIElement owner = Owner();
    auto menuBarItem = owner.as<winrt::MenuBarItem>();
    if (winrt::get_self<MenuBarItem>(menuBarItem)->IsFlyoutOpen())
    {
        return winrt::ExpandCollapseState::Expanded;
    }
    else
    {
        return winrt::ExpandCollapseState::Collapsed;
    }
} 

2. Redundant code to supports simple change like changing Control type.

Proposal

• Supports default setting for Control type, automation Name, Localization and Patterns in Control

TreeViewItem() 
{
    Super.defaultRole(TreeViewItem);
    Super.defaultLabel('TreeViewItem');
    Super.defaultLocalizedKey('TreeViewItem');
}

• Supports declarative control type, automation name, and Localization and Patterns change

<TreeViewItem accessbility.accessibilityRole="button" />
<TreeViewItem accessbility.accessibilityLabel="MyClickButton" />

• Allow customized dictionary to replace the system defined localized string and supports customized localization

Customer registers a dictionary which provides localization for any key

ResourceDictionary.registerLocalization(customizedDictionary)

then customer declare the key in XAML to request a customized value in above dictionary.

<TreeViewItem accessbility.accessibilityCustomizedLabelKey="MyClickButton" />

In control, Dynamic AutomationPeer would look up the customizedDictionary to provides the localized automation name.

• Supports Pattern implemented in Control.
  Currently, Pattern are implemented in AutomationPeer, now we can move it to control.
  For example, IExpandCollapseProvider is implemented in TreeViewItemAutomationDataPeer, it's possible to just implement it in Control. Dynamic automation peer has the ability to figure out its control and invoke the patterns in the control.

Scope

Capability	Priority
Allow control to define default settings for Automation Name, control Type, supported Pattern, and localized Key	Must
Allow declarative accessibility change in XAML for AutomationName, ControlType, supported patterns and localized Key	Must
Allow pattern implementation in Control itself	Should
Allow customer to intercept the localization look up process and provides customized dictionary	Should

Important Notes

Open Questions

[jevansaks]

@licanhua sounds great! Next step would be to grab a pitch slot on the calendar and pitch it!

The email thread we had about this also brought up an old spec, did you read through that and look at the open questions from that previous spec review?

[licanhua]

spec: microsoft/microsoft-ui-xaml-specs#50