Abstract blue and purple diamond grid pattern

Design Audit

From Weeks to Minutes: Rebuilding the Floto Accessibility Audit

The Floto accessibility audit tests a live web page against 46 of the 55 Level A and AA success criteria of WCAG 2.2. Here is exactly what it covers, what it leaves to a person, and how it scores on public benchmarks.

Akshay

•

Engineer

•

At Floto we treat every product question as a research question first. When we set out to make the Floto Accessibility Audit genuinely useful, we didn't start by adding rules. We started by asking a simpler question:

How much of a WCAG accessibility audit can be automated and completed in minutes, leaving auditors only a small part to review?

A manual accessibility audit takes weeks. An auditor works through every WCAG criterion, page by page, with a keyboard, a screen reader and a checklist. For a company with hundreds of pages, that's months of work, and it starts again with every release.

Our goal was simple: let the Floto Accessibility Audit do the bulk of that work in minutes, so auditors and product teams only need a few more minutes to complete the accessibility review.

The Floto Accessibility Audit started where most tools do: a Quick Audit that reads a page's code and checks it against a rule engine in seconds. It's useful, and it still runs whenever you want a fast pass. But reading code is only part of an audit. So we built a Deep Audit version. It doesn't just read the code. It uses the page the way people do, judges meaning as well as markup, and follows the page as it changes.

Then we benchmarked the new Floto Accessibility Deep Audit against five public accessibility test suites. Each one is published by an independent team, with a documented answer for every barrier or test case in it. The rest of this post explains how we chose them, what they showed us, and what we changed. Here's how it performs today:

Benchmark

Result

GOV.UK accessibility tool audit (142 barriers)

132 found (93%)

Deque Mars Commuter (57 barriers)

48 found (84%)

Accessible University (22 barriers)

22 found (100%)

Accessible U, Duke (19 barriers)

19 found (100%)

W3C ACT Rules test cases (1,213 cases)

≈83% of failures caught

Why automation hasn't solved this yet

Accessibility is now an obligation, not a nice-to-have: the European Accessibility Act has applied since June 2025, with EN 301 549 as its standard for the web. Yet the WebAIM Million 2026 report found detectable WCAG failures on 95.9% of the top one million home pages.

Manual audits don't scale to that, and automated checkers don't go far enough. When the UK Government Digital Service planted 142 barriers on one page and tested 13 popular tools, the best found 40%, or 50% counting issues flagged for review. The weakest found 13%. So teams end up running a checker, then paying for weeks of manual work anyway.

How we measured

We didn't want to grade our own homework, so we looked for public test suites with documented answers, written by people independent of us:

Benchmark

Published by

What it tests

W3C ACT Rules test cases

W3C

1,213 small pages, each a precise pass or fail case written by the standards body

GOV.UK accessibility tool audit

UK Government Digital Service

142 deliberate barriers, built specifically to compare tools

Accessible University

University of Washington

A realistic university site, inaccessible by design, with a fixed version to compare against

Accessible U

Duke / Open Assessment Toolkit

A modernised take on the same idea, used for training

Mars Commuter

Deque

A travel site full of interactive barriers: tabs, widgets and forms

Then we built a test harness that runs the exact same audit our users run against every suite, matches each finding to the documented barriers, and produces a scorecard. Three rules kept us honest:

  • Baseline first. We scored the audit before changing anything, so every improvement has a "before".

  • The whole suite counts, including barriers no automated tool could reasonably find.

  • Clean pages too. We also run the accessible versions of the same sites, so an improvement can't come from simply reporting more.

Every idea after that followed the same loop: change one thing, rerun everything, keep it only if the numbers improve without breaking anything else.

The gaps, and what we did about each

The barriers that checkers miss fall into five gaps. Each of these is something the Quick Audit, like most checkers, can't see, and what the Deep Audit was built to close.


Animation 1. Each finding points at the exact element on the page: stepping through results from a Floto Accessibility Audit of Deque's Mars Commuter demo site.

1. It looks right, but it isn't built right

The gap. Paragraphs marked up as headings, and headings whose levels jump around, so the page's outline makes no sense. Text that looks like a heading but isn't marked up as one. A form section with no group label. A list that's really just lines of text. Sighted users never notice; screen-reader users lose the page's structure. Checkers that only read the markup can't see the mismatch, because they never look at the page.

What we did. The audit reads the page twice. First as a screen reader would: the headings, their levels, labels and lists. Then as it's drawn on screen: size, weight and position. Wherever the two tell different stories, like a whole paragraph marked up as a heading, or a level-6 heading straight under a level-3 one, it reports the gap. Outline problems are reported as best practice; only what WCAG requires is called a failure.


Figure 1. On Mars Commuter, a 32-word introduction is marked up as a level-2 heading. It looks like a paragraph, but someone jumping through the page by headings hears the whole paragraph as a section title.

2. It's there, but does it make sense?

The gap. A rule can confirm an image has alt text, but not that "Beautiful baboon, blowing bubbles" describes a photo of a Martian sunrise. It can confirm a link has text, but not that "Learn more" says where it goes.

What we did. Where a question needs judgement, the audit makes the call and shows its reasoning. Clear-cut cases are settled by fixed rules; only the genuinely ambiguous ones get a judgement call, and when the audit isn't sure, it asks a person to confirm.


Figure 2. A photo of a Martian sunrise with the alt text "Beautiful baboon, blowing bubbles, biking backward". The alt text exists, so a basic check passes it; it just describes something else.

3. Checkers read pages; people use them

The gap. Can you reach every control with a keyboard? Can you see where you are? Does the page still work when text is enlarged? These barriers only appear when someone actually uses the page.

What we did. The audit moves through the page by keyboard, the way a keyboard user would, and re-lays it at larger text sizes and narrow widths. Browsers deliberately show focus differently to mouse and keyboard users. So the audit asks for a single key press before this test, to see exactly what a keyboard user sees.


Figure 3. On Mars Commuter, nothing visibly changes when the "Login" button receives keyboard focus, so keyboard users can't tell where they are.


Animation 2. Pressing Tab from the top of Mars Commuter: the first presses land on elements with nothing visible, so focus seems to vanish. The audit reports each invisible tab stop.


Animation 3. At 200% text size, the lines of the "A trip to Mars" introduction are drawn over each other, because their line height is fixed in pixels.

4. A page is not a snapshot

The gap. Menus open, dialogs appear, forms show errors, logo strips scroll. A check that looks once, as the page loads, never sees the dialog that traps you or the error that's never announced.

What we did. The audit opens what can be opened, submits empty forms, and watches the page over time for content that moves on its own. The hard part was doing this without changing anything. A safety layer lets every interaction happen, but holds back anything that would leave the page, send data, or change something real.


Figure 4. Mars Commuter's special offers scroll by themselves, with no way to pause them.


Animation 4. The special offers move on their own; the audit watches the page over time and reports that they can't be paused.

5. Modern sites rebuild themselves

The gap. Sites made with visual builders rewrite parts of the page while you're on it. A finding that pointed at an element a second ago can end up pointing at nothing. No benchmark tests for this, but we hit it on real sites, starting with our own.

What we did. Every finding gets a stable fingerprint, based on what the element is and where it sits. After the page rebuilds, the audit finds the element again, so the card still points at the right place. On some builder sites, large parts of the page are replaced while you scroll; without this, cards pointed at nothing.


Figure 5. A contrast finding ("Countdown…", 1.87:1 against a 4.5:1 minimum) highlighted on the live page, next to its card.

Underneath all five sits one rule: an audit either finishes every check or stops and returns your credits. It never hands over a report with silent holes.

From findings to fixes

Finding barriers is only half of an audit; the other half is getting them fixed. Every finding in the Floto Accessibility Audit comes with a suggested fix, and the results are built to move straight into a team's workflow:

  • Fix prompt: a ready-made prompt for a coding agent, for one issue or the whole page.

  • Create issue: send a finding to Linear or Jira with one click.

  • Download report: a PDF or Excel sheet for auditors and compliance records.

A worked example: "Be Bold…", "Countdown…", "Blast Off!"

The Mars Commuter home page has three columns, each headed by a link: "Be Bold…", "Countdown…" and "Blast Off!". Is that an accessibility problem?

It depends on the context. WCAG's AA rule asks whether a link's purpose is clear from its context: the link text together with its sentence, paragraph or heading. A vague link is fine if the words around it say where it goes. Here they don't. Each column's text talks about the trip, but nothing says what clicking the link opens.

Our first version flagged every vague-sounding link, even where a heading or sentence made the destination clear, which cluttered reports with issues pages didn't have. Now the audit reads the context first: clear in context passes, clearly unclear is flagged, and anything in between gets a judgement call with its reasoning. These three are unclear even in context, so they're flagged.


Figure 6. Three links on Mars Commuter that don't say where they go, and neither does the text around them, so the audit flags them under WCAG 2.4.4.

What didn't work

Some of our best lessons came from changes we threw away.

  • Waiting longer made things worse. Giving pages extra time to settle seemed safe, but on interactive sites it meant missing barriers that only exist for a moment. We removed it.

  • A fix for one site broke two others. Handling one kind of site that swaps its content cost us pop-up detection on two university sites. Running every change against every suite caught it the same day.

  • New checks were too eager at first. Our first structure checks flagged problems on the clean, accessible versions of test sites. Those clean pages are exactly why we run them.

  • Our own home page fooled us. Comparing runs on floto.ai showed the logo strip was sometimes never checked at all: the images loaded, but in some browser setups the audit couldn't read them. Reading them a different way fixed it, and it's a reminder to test on real pages, not just benchmarks.

Results

Figure 7. Share of each suite's documented barriers the Floto Accessibility Audit finds. Whole suite counted.

All results here are for the Deep Audit. A full Deep Audit of one page takes under a minute: between 31 and 56 seconds on the benchmark sites.

On the GOV.UK suite, the one designed to show where tools fall short, the audit finds 93% of the barriers. For context, the best tool in GOV.UK's original study found 40%, or 50% including issues flagged for review. Results gathered under different rules aren't a league table, but the gap is large.

What we still miss is mostly what a single-page audit shouldn't claim to settle: flashing content, gestures, consistency across many pages, and legal or financial confirmation steps. We say so plainly and leave those to a person.

What's next

Figure 8. The Floto Accessibility Audit covers 46 of WCAG 2.2's 55 Level A and AA criteria.

The same checks map onto EN 301 549, the European standard: 41 of the 49 web criteria in its clause 9 are covered. The audit also checks 9 of the 28 AAA criteria, reported as suggestions rather than failures, for teams that want to go beyond the legal minimum.

Next we're researching wider WCAG coverage, testing journeys across several pages, and deeper checks on images.

The goal stays the same: an audit that takes weeks today should take minutes, with people spending their time only where judgement is needed. So does the method: find independent ground truth, measure before changing anything, and let the evidence decide. That's how we work at Floto, on accessibility and on every part of the product.

Related Research Papers

Scaling UI Navigation Graph Generation with Multimodal Embeddings and Incremental LLM Classification

As software products grow in complexity, maintaining an accurate understanding of their navigational structure becomes increasingly difficult. Designers and product managers routinely lose track of how screens connect, which paths are reachable, and where the flow breaks down. Existing tools either require manual documentation or are limited to small frame sets due to the cost of large language model (LLM) inference.

We present a scalable pipeline for automatically generating product navigation graphs from UI design files. Given a Figma page containing an arbitrary number of screen designs, our system produces a directed flow graph representing the navigational structure of the product—identifying screens, transitions, entry points, exit points, and user goals. The core contribution is a three-pass architecture that uses multimodal image embeddings to deduplicate visually similar screens before LLM classification, reducing the number of expensive inference calls from O(N) to O(k), where k is the number of visually distinct screens. Graph synthesis is performed by a single LLM call that receives only the k deduplicated representative frames rather than all N frames, achieving scalability through input reduction while preserving the holistic reasoning that makes single-call synthesis produce coherent graphs. We evaluated an embedding-based edge inference alternative and found it unreliable due to ambiguity between visually similar screens with identical labels and the absence of global context in per-frame matching. We further introduce an incremental update strategy that combines file-level change detection with per-frame content hashing, enabling re-runs to skip unchanged frames entirely and reuse their stored classifications and embeddings.

We evaluate the system on real product design files ranging from 20 to 130 frames, demonstrating a 60–80% reduction in LLM calls compared to naive per-frame classification, and near-instant re-runs when designs have not changed. The resulting graphs are structurally equivalent to hand-authored flows and serve as a foundation for downstream tasks including automated flow testing and design quality auditing.

Read paper →