Mobile Testing

Introduction

The ICT Accessibility Testing Symposium has developed a methodology for evaluating the accessibility of mobile websites and native apps. These documents are an amalgamation of accepted mobile site accessibility testing standards from around the world, including additional developments from the ICT Accessibility Testing Symposium’s Mobile Site Sub-Committee.

WCAG2 success criteria are applicable to mobile sites and native apps; however, not all aspects of mobile site or native app accessibility are specifically covered by WCAG2. It is the opinion of this committee that merely conforming to WCAG2 (or WCAG 2.1) does not provide for a fully accessible experience for users with disabilities.

Please note that this methodology does not include those issues already included in WCAG2, however does include issues identified in WCAG2.1. This guide was written with the intent to clarify the unique needs of users with disabilities who use mobile websites and native apps and to raise the bar for the web development community. This is a work-in-progress, and, as such, we do not make a claim that conforming to these requirements will ensure that your mobile site is fully accessible to all users.

A note on WCAG2.1

The Committee decided that it was important to also include issues added in WCAG2.1 so that testers who were testing against WCAG2 would also benefit from the mobile-related errors published in WCAG2.1. This methodology is more stringent than WCAG2 and WCAG2.1 in some areas, and these are detailed in the Test Cases documents.

Mobile sites versus native apps

There is a great difference between mobile sites and native apps – native apps utilize a completely different codebase. Therefore, the ICT Accessibility Testing Symposium has decided to separate the native app methodology from the mobile site testing methodology.

A note on hybrid native apps

Hybrid mobile apps consist of both HTML and native code. The HTML may be included as a specific page within an application, or within a specific container. When testing a hybrid app, you will need to use this methodology and the Mobile Site Accessibility Testing Methodology.

The Methodologies

Unfortunately, when developing WCAG2, the W3C WCAG Working Group did not envision the current world where mobile is almost ubiquitous. For example, on a mobile device there is no continual access to a keyboard (unless someone is using it as an add-on to the device – or using a Blackberry Classic!). WCAG2 requires that all content be accessible to the keyboard interface, but it does not require that all content be accessible to a mouse or to a touchscreen user, which is essential on a mobile device. WCAG2.1 does include some mobile accessibility requirements but doesn’t go far enough.

Gian Wild chaired the Mobile Site Sub-Committee to develop a set of Mobile Site Testing Guidelines that are available under Creative Commons. These documents are an amalgamation of accepted mobile site accessibility testing standards from around the world, including additional developments from the ICT Accessibility Testing Symposium’s Mobile Site Sub-Committee. These guidelines are meant to be used in conjunction with WCAG2 (and WCAG2.1) to ensure that sites are accessible to people with disabilities using mobile and tablet devices. There are two sets of documents: one for Mobile Sites and one for Native Apps.

Mobile Site Testing Methodology Overview

The steps for Mobile Site accessibility are:

  1. Identify devices: we recommend testing with iPhone (Safari), iPad (Safari) and Android (Chrome), but also consider alternative devices such as an Android tablet, especially if this is indicated in analytics or expected usage
  2. Identify site type and variations: identify whether the site is a desktop site (one view whether on a desktop or mobile device), responsive site (resize as the screen size changes) or m.dot site (a completely separate site for mobile devices). Where the site is a responsive site, identify all the different variations of a page
  3. Test critical issues: test for any traps in the web site (exit trap, swipe / scroll trap and layer trap)
  4. Test mobile issues: test cases are broken into the following categories: Alternatives, Display, Actionable items, Navigational aids, Audio and video, Forms and Mobile / desktop interaction
  5. Test mobile assistive technology and feature support: assistive technology / mobile features that are recommended to be tested on iPhone and iPad are:
    • VoiceOver
    • Keyboard
    • Switch
    • Zoom
    • Reduce Motion
    • Invert colors
    • Grayscale
    • Reader view

On Android:

    • TalkBack
    • Keyboard
    • Switch
    • Magnification
    • Remove Animations
    • Color Inversion
    • Grayscale
    • Color correction
    • Increase display size
    • Increase text size with Android Chrome
    • Simplified view

There are two overview documents:

There are three sets of test cases documents, which detail how to test a particular requirement in the methodology, why it is important and example passes of the requirement:

Native App Testing Methodology Overview

The steps for Native App accessibility are:

  1. Identify devices: as per mobile site accessibility
  2. Define application functionality: identify the purpose of the native app and detail the required user journeys and test for common elements such as:
    • Navigation – Menus, header, footer
    • Landing screen(s)
    • Emergency sections and content
    • Login flows
    • Settings
    • Account and profile
    • Contact Us
    • Real-time updates (eBay, Uber)
    • Privacy policy, Terms and Conditions
    • Interactional functionality (adding items to a shopping cart, payment details, live chat, selections for a product in a catalogue, scanning a barcode, VR, QR code)
    • Help section
    • Widgets (calendars, date pickers)
    • Third-party integrations (geo-locational maps)
    • High-traffic areas
  3. Test critical issues: test for any traps in the native app (exit trap, swipe / scroll trap and layer trap, text-to-speech trap, headset trap)
  4. Test mobile issues: test cases are broken into the following categories: Alternatives, Display, Actionable items, Navigational aids, Audio and video and Forms
  5. Test mobile assistive technology and feature support: assistive technology / mobile features that are recommended to be tested on iPhone and iPad are:
    • VoiceOver
    • Keyboard
    • Switch
    • Zoom
    • Reduce Motion
    • Invert colors
    • Grayscale
    • Larger Text

On Android:

    • TalkBack
    • Keyboard
    • Switch
    • Magnification
    • Remove Animations
    • Color Inversion
    • Grayscale
    • Color correction
    • Increase display size
    • Increase font size

There are two overview documents:

There are three sets of test cases documents, which detail how to test a particular requirement in the methodology, why it is important and example passes of the requirement: