Web Testing Options: An Overview

By Chad Lamb, Senior Test Engineer

This past year, GSA and OMB sent self-assessments to federal agencies, asking the agencies to evaluate the maturity of their implementation of Section 508.  Section 508 offices and their respective agencies were asked to evaluate and report on their 508 implementation efforts. One area of focus was ICT Testing.  After the assessment, some agencies that had not been testing, are looking at what approach to take to perform these tasks. This question is not new and agencies need to weigh the pros and cons of each option to determine the best fit for their agencies

There are really only three options to perform web testing: automated testing, manual testing, or a hybrid approach combining the first two.  Below is a discussion of each option.

Automated Testing:

Automated testing requires an agency to purchase an automated testing tool. There are many tools that need to be evaluated to determine which tool best meets the agency’s needs.  Most tools give the option to test against WCAG standards, Section 508, and to modify the test standards to meet the test process. For example, on the New Editions’ contract with DHS, we did our best to align the automated tool sets with the Trusted Tester test process.  This is great because you can align scans to meet the needs of your agency.

An automated tool’s greatest benefit is how quickly a website can be scanned as compared to a human tester. However, you still need a human to verify results. These tools are helpful for teams of all sizes because you can monitor websites and see if the sites’ accessibility scores go up or down after updates to the site. It is also very useful for smaller teams that do not have enough manual testers to perform tests on all agency sites.

Automated tools are limited. For instance, the tools scan the code to determine accessibility. In the case of images, it can determine if there is an <IMG> tag and whether or not there is accessible markup like ALT, TITLE, or ARIA being used to provide a description of the Image. It cannot, however, tell if the description of the image is accurate. This is the case for other standards as well. It can identify when a table is being used. If there is no markup on the table, it cannot determine if it is a layout table or a data table that needs accessible markup. Typically, these types of issues are marked as alerts and the user can go in and determine if it should be a failure or not. Some systems also allow users to remove them from the issues list if it is determined that it is a pass.

Manual testing:

This form of testing requires an agency to have a test process. At DHS and other agencies, New Editions testers have used the Trusted Tester process. As long as the process tests against WCAG 2.0 A and AA standards, it will meet the minimum level of accessibility for the Federal Government. Only testers trained in the agency’s test process, for example DHS Trusted Tester Test Process for Web Applications, should perform manual testing.  It is important that your tester’s know and understand the applicable Section 508 standards as well as how to test for each of them.  The standards only apply when the elements they reference are on a page.  Once a tester determines the applicability of the standards, they then must use their agencies test process to evaluate whether or not the element is in compliance with Section 508.  Test processes will typically outline tools that need to be used and how to use them to determine the whether an element meets compliance.

Testers will evaluate each page of the application against the agency’s test process. A trained human tester will be able to make the determinations on the accessibility of images and tables. Testers will also be able to discuss their findings with developers to help them better understand how to remediate the issues that are found while testing.  A manual tester can also perform assistive technology (AT) validation to make sure the application works with various types of AT. 

The bonus to this form of testing is it allows validation of all of the applicable standards. The downside is that a team of testers is needed to effectively implement this within the agency.  There are ways to increase efficiency. One way is to have testers only test a sampling of pages from a site (i.e. different page templates, pages with different elements, or high traffic pages). Another method is to require development teams to have a certified tester on the team to perform testing as the application is being built. 

Hybrid Testing:

Hybrid testing takes the best of both options and increases accessibility. Agencies run the scan with their automated tool. A section 508 subject matter expert that conducts manual assessments then evaluates the “alerts” to verify that they are compliant. The 508 SME has the knowledge to work with the development team to explain the report and offer suggestions to fix the code. Manual testing can still be performed as needed. There may be applications that need to have each page evaluated. Agencies may choose to implement sample testing along with the full site scan. This option can work with large and small teams.

Conclusion:

There is no perfect solution that works for every team because there is so much variation in teams. Every agency has a different number of support employees, different mandates, different budgets. Each agency needs to look at the options available to determine which works best for them.