When we think about software products, we naturally think in terms of features, workflows, and user journeys. Yet the artifacts we use to design these products tell a very different story. Design tools such as Figma organize products as collections of individual screens rather than as connected systems. A mature enterprise application may contain well over a hundred screens spread across multiple pages, representing onboarding flows, settings, administrative interfaces, error states, edge cases, and feature variations. Although each screen is carefully designed, the relationships between those screens are rarely captured in a structured and maintainable way.

This creates an interesting paradox. Designers understand how the product is supposed to behave, product managers understand the intended user journeys, and engineers understand the implementation. However, very few teams possess an explicit representation of the product's navigational structure. Instead, this understanding exists almost entirely as institutional knowledge distributed across the team. As products evolve, that shared mental model gradually diverges, documentation becomes outdated, prototype connections become incomplete, and the true structure of the application becomes increasingly difficult to reason about.

At Floto, we encountered this problem while building AI-powered design validation tools. Many of the analyses we wanted to perform—such as identifying unreachable screens, validating user journeys, testing navigation flows, and understanding the intent behind a design—required something surprisingly fundamental: an accurate graph describing how the product was connected. We realized that before AI could meaningfully evaluate a product's design, it first needed to understand the product itself.

Why Existing Approaches Fall Short

At first glance, solving this problem appears straightforward. Modern vision-language models are remarkably capable of interpreting user interfaces. Given an image of a screen, they can identify whether it is a login page, a dashboard, a form, or a settings screen. They can summarize the screen, identify interactive elements, infer user intent, and even describe likely user actions.

The obvious solution, therefore, is to run an LLM on every screen in a design file, classify each one independently, and then connect them into a navigation graph.

Unfortunately, this approach does not scale.

Large products frequently contain between fifty and two hundred screens. Processing each screen independently requires an equivalent number of expensive multimodal inference calls, resulting in several minutes of processing time even for moderately sized applications. More importantly, it ignores a simple characteristic shared by almost every software product: visual redundancy.

Most products contain numerous variations of essentially the same screen. Login pages appear in multiple states. Lists have empty, loading, populated, and error variants. Settings pages often differ only in a handful of options. Confirmation dialogs are repeated throughout the application. From a semantic perspective, these screens represent the same navigational concept, yet a naïve pipeline treats every variation as an entirely independent reasoning problem.

This meant that our system was repeatedly asking the language model to understand information it had effectively already seen.

Our First Attempt: Can Embeddings Replace the LLM?

Rather than immediately optimizing the LLM pipeline, we first explored whether it could be avoided altogether.

Recent multimodal embedding models produce rich semantic representations of images, making it tempting to frame navigation inference as a similarity problem. The proposed architecture was elegant in its simplicity. Every screen would be converted into an embedding describing its visual and semantic content. Each interactive element would similarly produce a destination hint, which could then be matched against screen embeddings using cosine similarity. If successful, this would eliminate the need for expensive graph reasoning and replace it with deterministic nearest-neighbor search.

On paper, the idea was compelling.

In practice, it failed.

The difficulty became obvious when we examined real products rather than idealized examples. Consider a multi-step onboarding wizard where every screen is titled "Create Project." Although each step serves a different purpose, the screens are intentionally designed to look almost identical. Their layouts, typography, and visual hierarchy are nearly indistinguishable, producing embedding vectors that occupy almost the same region of semantic space. Similarity search simply does not contain enough information to distinguish which "Create Project" screen a particular interaction should navigate to.

A second, more subtle issue emerged during evaluation. Similarity-based approaches make navigation decisions independently for every screen. Without visibility into the broader application, each screen optimizes its own local decisions, often producing graphs that are excessively connected and structurally unrealistic. Human designers, by contrast, think globally. They understand complete workflows, happy paths, branching decisions, and overall product structure before deciding how individual screens relate to one another.

This experiment led us to an important realization. Navigation inference is fundamentally not a similarity problem. It is a reasoning problem.

A Different Question

Once we accepted that holistic reasoning was necessary, our perspective shifted.

Instead of asking how to replace the language model, we began asking how to reduce the amount of information it needed to reason about.

This distinction may seem subtle, but it ultimately shaped the entire architecture of our system.

Visual redundancy, which had previously been treated as an inconvenience, suddenly became an opportunity. If multiple screens conveyed essentially the same navigational meaning, then perhaps only one of them actually required semantic interpretation. The remaining screens could simply inherit that understanding.

This observation led us to a three-stage pipeline in which multimodal embeddings and language models work together rather than competing with one another. Embeddings are responsible for identifying redundancy, while the LLM performs the higher-level reasoning that embeddings cannot.

Pass One: Understanding Visual Redundancy

The first stage of the pipeline generates multimodal embeddings for every screen requiring analysis. These embeddings capture both visual appearance and semantic information, allowing visually similar screens to be clustered together using cosine similarity.

Rather than treating every frame as unique, the system identifies representative screens for each cluster. A login page with slightly different copy, an empty-state list, and a populated list often collapse into a single representative because their navigational role remains unchanged despite their visual differences.

As a result, a design file containing one hundred screens might contain only thirty or forty genuinely distinct navigational concepts.

This dramatically changes the economics of the problem. Instead of requiring one hundred LLM classification calls, only the representative screens need to be analyzed. Every remaining screen simply inherits the semantic classification of its representative. In our production datasets, this approach reduced LLM inference calls by anywhere from roughly 40 to 80 percent, with the largest gains on products—like component libraries—built from highly repetitive screens, and smaller gains on products with mostly distinct, non-redundant screens.

Pass Two: Teaching the AI What Each Screen Represents

Once representative screens have been identified, a vision-language model analyzes each one in depth.

Rather than producing a simple label, the classifier generates structured metadata describing the screen's purpose, its primary user intent, visible interface states, interactive elements, likely navigation destinations, and other contextual information required for downstream reasoning.

One design decision proved particularly valuable. Instead of relying on short labels such as "Dashboard" or "Settings," the model also generates a detailed natural-language description of every screen. These descriptions serve as rich semantic representations during graph synthesis and are significantly more reliable than raw Figma layer names, which are often generic labels such as "Frame 47" or "Copy of Screen."

By enriching each representative screen with meaningful semantic context, the system creates a foundation upon which higher-level reasoning can operate.

Pass Three: Reasoning About the Entire Product

The final stage is where the navigation graph is constructed.

Rather than asking the model to infer individual edges independently, we provide the LLM with the complete set of representative screen classifications and ask it to synthesize the entire navigation graph in a single reasoning step.

This distinction is important.

Because the model sees every significant screen simultaneously, it can reason about the application as a cohesive system rather than as isolated fragments. It can recognize likely entry points, identify exit conditions, distinguish primary user journeys from secondary flows, and avoid creating unrealistic connections that emerge when each screen is considered independently.

Ironically, the embedding stage that initially appeared to compete with the language model ultimately became the mechanism that enabled this holistic reasoning. By removing redundant screens before graph synthesis, the representative set comfortably fits within the language model's context window while preserving the information required to reconstruct the complete product graph.

Rather than replacing reasoning with algorithms, we reduced the amount of reasoning that was necessary.

Making the System Practical

Scalability alone is insufficient for production systems. Design files evolve continuously, and rerunning the entire pipeline after every small modification would still be unnecessarily expensive.

To address this, we introduced an incremental execution strategy.

Before processing begins, the system first checks whether the Figma file has changed since the previous execution. If nothing has changed, the cached navigation graph is returned immediately. When modifications are detected, individual screen images are hashed so that only modified screens proceed through embedding generation and classification. Previously analyzed screens reuse their stored embeddings and classifications without requiring additional model inference.

This incremental architecture transforms repeated executions from minutes into fractions of a second whenever designs remain unchanged, making the system practical for everyday design workflows rather than occasional batch processing.

What We Learned

Evaluating the system across production design files produced encouraging results. Depending on the product, visual deduplication reduced the number of LLM classification calls by anywhere from 38 to 78 percent, with the biggest gains showing up on products built from highly repetitive components. Re-executing the pipeline on unchanged design files became effectively instantaneous, and partially modified files required only the affected screens to be reprocessed.

Preliminary evaluations also showed strong screen-identification accuracy: node-level precision (whether a screen was correctly understood) came in between 91 and 96 percent. Edge-level precision (whether the graph drew the right connections between screens) was lower, in the 74 to 83 percent range. Most of those connection errors traced back to icon-only interactive elements whose destination is genuinely ambiguous from the image alone—not to the system misunderstanding what a screen was.

However, perhaps the most valuable outcome was not the graph itself, but what the graph enabled.

Once a product possesses a structured representation of its navigational intent, entirely new categories of AI-assisted product validation become possible. Navigation graphs provide context for automated flow testing, identify unreachable states and dead ends, support design quality auditing, and create a semantic bridge between design artifacts and implemented software. Rather than analyzing isolated screenshots, AI systems can reason about complete products.

A Broader Lesson About Building AI Systems

Although this work focuses specifically on user interface understanding, the underlying principle extends far beyond design tooling.

When faced with scaling challenges, our instinct is often to replace large language models with deterministic algorithms. Our experience suggests that this is not always the right direction. In many cases, language models remain the best available mechanism for solving problems that genuinely require contextual reasoning. The more effective optimization is often to reduce the amount of information the model must process before that reasoning begins.

In our case, multimodal embeddings did not replace the LLM. Instead, they eliminated redundancy, allowing the language model to focus exclusively on decisions that required genuine semantic understanding. The result was a system that was simultaneously faster, less expensive, and more accurate than either approach could achieve independently.

As AI systems become increasingly integrated into software engineering workflows, we believe this pattern will become more common. The future may not belong to systems that eliminate reasoning altogether, but to architectures that carefully combine efficient algorithms with powerful reasoning models, ensuring that each component solves the problem it is best suited to address.

For us, automatically generating product navigation graphs is only the beginning. We see these graphs as the foundational representation upon which a much broader ecosystem of AI-powered design validation, product understanding, and autonomous quality assurance can be built—enabling software teams to reason about their products long before a single line of production code is written.