Exploring Accessibility CLI tools

Note: this blog is an archive and not actively maintained. Some information may be out of date. If you'd like to see what I am working on or work with me in a consulting capacity, visit my website lindseykopacz.com.

April 15, 2019

Happy Tax day (at least if you’re American)!

My last post was about my accessibility testing process. However, this week, I wanted to examine automated testing. I have my initial process of what I do while I am working and what I make sure I manually test. I also wrote about all the things I manually test.

These days Continuous Integration is essential for large teams. To your surprise, I haven’t configured accessibility CLI tools for my projects. I was the primary front end most of the time and I always manually test things out of habit, so it never felt necessary.

It’s time to call myself out. I absolutely should be integrating these into my tests into my projects. I used to be the only front end developer, and I took it for granted. I am now on a team where I am the accessibility expert, but I have other front end developers on the team. I review many pull requests, but it would be way easier if Continuous Integration could catch new errors. Isn’t that the point of Continuous Integration? To catch things that we miss? To make our lives easier?

Heads up. This is going to be a “learn with me” post.

I’m going to be going over 2 of the CLI tools I am familiar with - pa11y and aXe-cli. Feel free to Tweet at me any tools you would like to check out.

Pa11y

Pa11y is the first accessibility testing tool I used a couple of years back. The first time I tried it, it was a gulp package. Admittedly, I never did any of the configuring because at the time I found the tool to be too restrictive.

I’m coming at this fresh, so I am going to go through installing it in this blog post and see how I like it.

npm install pa11y --save

Now I am going to create an npm script to run pa11y. In my package.json file:

"scripts": {
  "test-a11y": "pa11y ./index.html"
}

Then I went to my command line and typed npm run test-a11y and got errors. I got a SyntaxError that I remember seeing 2 years ago, and I happened to remember that it was a node version issue.

$ npm run test-a11y

> egghead-focus-event@1.0.0 test-a11y /Users/lkopacz/Sites/egghead-focus-event
> pa11y ./index.html

/Users/lkopacz/Sites/egghead-focus-event/node_modules/pa11y/bin/pa11y.js:92
async function runProgram() {
      ^^^^^^^^

When I checked what node version I was using, I learned that somehow, I had switched from node v.8 to node v.6. Embarrassing to admit I didn’t catch this and I was on an older version of node.

nvm use v8.9.1

Now the script appears to run properly! I ran it and got an accessibility error.

$ npm run test-a11y
> egghead-focus-event@1.0.0 test-a11y /Users/lkopacz/Sites/egghead-focus-event
> pa11y ./index.html


Welcome to Pa11y

 > Running Pa11y on URL file:///Users/lkopacz/Sites/egghead-focus-event/index.html

Results for URL: file:///Users/lkopacz/Sites/egghead-focus-event/index.html

 • Error: The html element should have a lang or xml:lang attribute which describes the language of the document.
   ├── WCAG2AA.Principle3.Guideline3_1.3_1_1.H57.2
   ├── html
   └── <html class="no-js"><!--<![endif]--><head> <met...</html>

1 Errors

Welp that’s embarrassing. However, that’s precisely why this exists. Going to fix this and see what happens.

<html class="no-js" lang="en">
  <!-- MOAR HTML-->
</html>

Now let’s run that test again and see what happens.

$ npm run test-a11y

> egghead-focus-event@1.0.0 test-a11y /Users/lkopacz/Sites/egghead-focus-event
> pa11y ./index.html


Welcome to Pa11y

 > Running Pa11y on URL file:///Users/lkopacz/Sites/egghead-focus-event/index.html

No issues found!

Well, that was easy to set up! So far, I like pa11y. I would love to play around more with this and look into all their other tools, such as the pa11y dashboard. However, the CLI seems to be the most beneficial for the point of this blog post.

aXe-cli

Deque Systems, a leading accessibility firm, developed the aXe-cli tool. I haven’t checked out many Deque tools, which is pretty shocking since they are leaders in this space. I am going to install the module the same way I did above with pa11y. This time I am going to test it against my form labels lesson code. This particular lesson has accessibility errors because I fixed them in the video. I know there are form label errors.

npm install axe-cli --save

Then I am going to create the same npm script that I did with the pa11y task.

"scripts": {
  "test-a11y": "axe ./index.html"
}

When I ran the command, this is what I received:

$ npm run test-a11y

> egghead-30-second@1.0.0 test-a11y /Users/lkopacz/Sites/egghead-form-labels
> axe ./index.html

Running axe-core 3.2.2 in chrome-headless

Testing http://./index.html ... please wait, this may take a minute.
axe-webdriverjs deprecated Error must be handled as the first argument of axe.analyze. See: #83 node_modules/axe-cli/lib/axe-test-urls.js:85:8

  Violation of "document-title" with 1 occurrences!
    Ensures each HTML document contains a non-empty <title> element. Correct invalid elements at:
     - html
    For details, see: https://dequeuniversity.com/rules/axe/3.2/document-title

  Violation of "html-has-lang" with 1 occurrences!
    Ensures every HTML document has a lang attribute. Correct invalid elements at:
     - html
    For details, see: https://dequeuniversity.com/rules/axe/3.2/html-has-lang

  Violation of "landmark-one-main" with 1 occurrences!
    Ensures the document has only one main landmark and each iframe in the page has at most one main landmark. Correct invalid elements at:
     - html
    For details, see: https://dequeuniversity.com/rules/axe/3.2/landmark-one-main

  Violation of "page-has-heading-one" with 1 occurrences!
    Ensure that the page, or at least one of its frames contains a level-one heading. Correct invalid elements at:
     - html
    For details, see: https://dequeuniversity.com/rules/axe/3.2/page-has-heading-one

4 Accessibility issues detected.

Please note that only 20% to 50% of all accessibility issues can automatically be detected.
Manual testing is always required. For more information see:
https://dequeuniversity.com/curriculum/courses/testingmethods

Ok. So, at first glance I like some things a ton. Mostly because of the disclaimer at the end.

Please note that only 20% to 50% of all accessibility issues can automatically be detected. Manual testing is always required.

Comparing the two

I like both these tools. Let’s first go to the Pros and Cons of both of them!

pa11y

PROS:

Setting it up was pretty simple.
They tell you where the offending code is in the HTML

CONS:

The only reason I knew how to debug the initial error is because I remembered it from a few years back. It does have it on its documentation, but I missed it. Deque’s requirements were way higher up on the README.md
The error messages aren’t super verbose. Sometimes this is trickier for newcomers

aXe-cli

PROS:

As I stated above, I love the disclaimer that you should be manually testing to catch other issues. Please see my last post about my accessibility testing process. I also have a checklist of what I manually test.
I also like that it gives you some links to the rules that failed.

CONS:

What I like better about pa11y is that it tells you exactly where the offending elements are. Note that I am not saying it’s not possible. I’ve only tested the most basic command I could.

Conclusion

Well, I got started in automated accessibility testing late, but I am a convert. If you noticed, I forgot a couple of things even as an expert. We are not perfect. If you’re working with teams newer to accessibility, it’s a powerful tool. It ensures that other team members do not introduce new issues or errors. I think both of these tools are super solid out of the box. I’m planning on going through both their documentation more and seeing how I can further configure it!

For now, I am planning on implementing pa11y on my projects. I prefer that one out of the box for my team because I can explain to my team why the errors are occurring. I want to explore aXe-cli more because I think the verbosity was much clearer. I’m sure with more configuration, I’d be able to have aXe work more to my liking.

About Lindsey

Lindsey is an accessibility expert, JavaScript lover, and Front End Developer who's passionate about inclusivity both inside and outside the web. Read more about her on the About Page