The Evolving Face of GUI (III) – Computer-Use Agents

Preface: Are Grounding Tasks All They’re Grounded to Be?

Research on GUI agents has advanced rapidly since the release of early grounding datasets such as the ScreenSpot series, with these benchmarks reaching saturation within a remarkably short time. However, despite the rapid advancement of GUI grounding techniques, current approaches often fail to capture the inherent complexity of real-world GUI interactions, reducing them to oversimplified element grounding tasks. This limitation has prompted a fundamental shift in focus—from grounding individual elements to developing end-to-end agents capable of performing complete GUI-based tasks. In practical scenarios, comprehensive computer-use tasks demand a multifaceted approach that extends far beyond simple grounding, encompassing long-horizon planning, sophisticated action space modeling, and multi-step reasoning. To address these complex requirements, robust systems may incorporate memory modules for context retention across interactions and leverage API integration to enhance operational efficiency, marking an evolution from isolated grounding capabilities to holistic task execution frameworks.

1. Datasets and Benchmarks

In order to train and evaluate robust computer-use agents, the community has built both open trajectory datasets (for agent training / imitation) and benchmarks / evaluation suites (for measuring task success, generalization, and robustness). Below are expanded lists of each, incorporating the newest works as of 2023–2025.

Open-source Datasets

Over recent years, several groups have released open datasets of GUI interaction traces (screenshots, states, actions, reasoning traces) across desktop, mobile, and hybrid interfaces. These datasets serve as the raw substrate for training and pretraining agents—from imitation to reinforcement learning.

• OS-Genesis — A pioneering data construction pipeline for generating large-scale, high-quality GUI interaction trajectories. By combining agent synthesis, human correction, and iterative refinement, OS-Genesis enables scalable data generation across multiple operating systems, significantly reducing the manual annotation burden while maintaining trajectory quality.
• AgentNet (OpenCUA) — The flagship desktop trajectory dataset in the OpenCUA project. It contains 22.5K human-annotated computer-use tasks across Windows, macOS, and Ubuntu (12K from Windows, 5K macOS, 5K Ubuntu), spanning 140+ applications and 190 websites. Each trace includes state, action, and chain-of-thought reasoning per step.
• Computer-Browser-Phone-Use Agent Datasets (Khang-9966 repo) — A community-curated catalog of datasets used for GUI agents, covering web, mobile, and desktop interactions (click/typing/navigation) with screenshots, element maps, and action logs.
• GUI-World — A multimodal GUI-oriented video dataset containing over 12,000 annotated video clips across desktop, mobile, XR contexts. It includes temporally grounded annotations (keyframes, operation history) to assess dynamic GUI understanding.
• ScaleCUA (2025) — A newly proposed dataset scaling open-source GUI agent training; it spans 6 operating systems and 3 task domains using a closed-loop pipeline of agent synthesis + human correction. It reports strong gains over baselines (e.g. +26.6 on WebArena-Lite, +10.7 on ScreenSpot-Pro) and sets new state-of-the-art open benchmarks.
• macOSWorld — Designed specifically for the macOS domain with 202 multilingual interactive tasks across 30 apps (28 of which are macOS-exclusive). It also includes a safety/deception-testing subset and multilingual instructions in 5 languages.

These datasets help agents learn cross-platform behaviors and multimodal reasoning. The inclusion of multiple OS types, diversity of applications, and chain-of-thought annotations are key trends pushing toward generalist agents. OS-Genesis represents a significant step toward automating dataset creation at scale.

Core Benchmarks for Computer-Use Agents

Benchmarks provide standard tasks and metrics to compare agent capabilities. They are broadly categorized into online benchmarks, which evaluate agents in live, interactive environments, and offline benchmarks, which test an agent’s ability to predict actions or understand states based on static, pre-recorded datasets.

Together, this suite of online and offline benchmarks provides a robust framework for assessing agent capabilities. Offline benchmarks are crucial for evaluating action prediction and representation learning on large-scale data, while online benchmarks test an agent’s true ability to plan, act, and recover from errors in dynamic environments.

2. E2E Training

With the advent of multimodal large language models (MLLMs) and agentic reasoning frameworks like ReAct, many works now exploit an LLM’s visual perception and planning capabilities to build computer-use agents. Early pioneers include OpenAI’s Operator and Anthropic’s Computer-Use Agent. In open-source terrain, UI-TARS and others have led the push for fully transparent agent models.

Below is an overview of representative E2E agent systems and model families (2023–2025), along with their training strategies and architectural contributions.

Training techniques & trends:

• Many recent works adopt a pipeline of supervised fine-tuning (SFT) → preference tuning / DPO → RL or self-improvement, blending imitation and reward-based learning.
• Trajectory augmentation (as in PC Agent-E) is increasingly popular: using LLMs to expand or diversify human trajectories before training.
• Joint training for grounding + reasoning modules (rather than separate isolated modules) is gaining traction.
• Self-improving loops (agent executes, reviews, refines its own data) are a rising paradigm (Mobile-Agent-v3 uses this).
• Models often treat the screen + mouse/keyboard interface as universal, making the same core agent usable on both mobile and desktop.

3. Multi-Agent Architectures

Relying on a single monolithic model for all GUI capabilities can limit specialization and adaptability as tasks vary wildly (e.g. form-filling vs. file management vs. app-specific flows). To address this, several recent works decompose agents into collaborating submodules or specialists.

• Agent S2 exemplifies this: it uses a generalist planner plus specialist grounding / reasoning modules, with a custom Mixture-of-Grounding that dispatches to the best specialist per UI context.
• UFO / Desktop AgentOS (Microsoft) employs a HostAgent + AppAgents architecture: HostAgent decomposes tasks, and AppAgents (per app) handle execution via vision + native UI Automation. This hybrid approach improves reliability and error recovery.
• Module-based pipelines in UI agent projects: many models internally split perception, grounding, plan generation, and action modules — though not always as independent agents, the conceptual modularization helps debugging, scaling, and flexibility.

Modular and multi-agent designs improve flexibility and robustness. For instance, if a grounding module fails, a fallback specialist can intervene; or a planning module can be replaced or updated without retraining perception.

4. Hybrid Architectures: Combining End-to-End RL with Multi-Agent Systems

Though multi-agent frameworks harness context engineering, role separation, and task decomposition, their cascaded or pipelined structure can subtly accumulate errors and constrict the performance ceiling of the underlying model. In those designs, mistakes in grounding or planning propagate downstream agents, amplifying deviations. This limitation has motivated hybrid architectures that combine an end-to-end, GUI-trained foundational model with multi-agent orchestration for optimization, error correction, and task decomposition.

One prominent example is Mobile-Agent-v3, which uses GUI-Owl as its foundational backbone and wraps a multi-agent system (MAS) around it for enhanced control and robustness. GUI-Owl is trained as an end-to-end multimodal agent (vision → grounding → planning → action), capable of operating directly on desktop and mobile interfaces. Then Mobile-Agent-v3 orchestrates specialized agents (e.g. Manager, Worker, Reflector, Notetaker) to supervise, reflect, and guide the execution of long-horizon tasks using dynamic replanning, memory, and error recovery. This hybrid approach mitigates cascading error drift while retaining the capacity and flexibility of a single, learned core model.

5. Cross-Platform Approaches

Achieving universality — i.e. one agent that works well across desktop, mobile, and web — is a key ambition. Hybrid strategies help bridge domain gaps and make cross-platform generalization possible.

• Unified interface abstraction: Many agents (CUA, PC Agent-E, UI-TARS) treat the GUI as a rendering + mouse/keyboard action space, hiding OS-specific APIs under a universal interface. This allows the same core agent to act on both Android, Windows, macOS, and web.
• Hybrid control (vision + native APIs): For example, Microsoft’s UFO leverages native UI Automation where possible and falls back to vision-based control otherwise. This increases robustness and reduces brittleness.
• Cross-OS trajectory synthesis / self-evolution: Mobile-Agent-v3’s training pipeline runs agents in Android, Ubuntu, Windows, macOS environments in cloud infrastructure, generating trajectories that span OS types.
• ScaleCUA’s cross-platform dataset: By training agents on diverse operating systems (six OSes) and domains, ScaleCUA encourages models that generalize across devices.
• macOSWorld: macOSWorld fills a gap by explicitly focusing on macOS GUI tasks (including multilingual and safety scenarios), helping to adapt agents to that domain.

By combining shared modeling, API fallback, cross-platform data, and modular interfaces, hybrid architectures approach the goal of a general GUI agent.

6. Challenges in Current Research

Even with all these advances, computer-use agent research faces several persistent and emerging challenges. Below I elaborate each with concrete examples and recent mitigation techniques.

Generalization to Unseen Interfaces and Apps

Agents often overfit to UI layouts or app families seen in training. When presented with new designs, rare applications, or multilingual interfaces, they fail to adapt. For example, current models struggle with specialized domain applications such as hardware control utilities (e.g., Lenovo Legion Zone), vendor-specific OEM software, or niche productivity tools that rarely appear in training corpora.

• Mitigation: Employ diverse cross-platform datasets (ScaleCUA, AgentNet, GUI-World) and multilingual benchmarks (macOSWorld’s 5-language support) to broaden exposure. Data augmentation techniques such as LLM-based trajectory blending (e.g., PC Agent-E’s Trajectory Boost) help synthesize diverse interaction patterns. Training on long-tail app coverage and domain-specific UI patterns (including OEM software and niche utilities) improves robustness. Additionally, designing specialized pre-training tasks focused on UI element recognition, layout understanding, and cross-domain transfer can enhance the model’s foundational visual comprehension.
• Open problem: true zero-shot adaptation to wholly new domain types (e.g., VR apps, games, OEM utilities, novel UI style languages) and low-resource language interfaces remains largely unsolved.

Long-Horizon Planning & Sparse Rewards

Complex tasks (e.g. “organize files, then email results”) require tens of steps and delayed feedback.

• Mitigation: Hierarchical planning, semi-online RL (UI-S1 style), reward shaping, Curriculum Learning.
• Open problem: maintaining robustness and recovery in long sequences with branching branches.

Safety, Robustness & Error Recovery

Agents interacting with GUI risk performing unsafe or destructive actions (e.g. closing important documents, clicking malicious popups) and prompt injection from Pop-ups.

• Mitigation: Action masking, human-in-the-loop confirmation, state rollback / undo modules, safety benchmarks (macOSWorld includes deception tests).
• Open problem: guaranteeing safety in open environments with adversarial UIs or unseen popups.

Efficiency, Memory & On-Device Inference

Running giant models is expensive, particularly on mobile or desktop inferences, and inference time for each step might be intolerable.

• Mitigation: Quantization, smaller models (AgentCPM, 8B), vector memory modules (e.g. UFO’s memory), caching embeddings, state compression.
• Open problem: balancing model capacity vs latency vs reliability in real-world GUI tasks.

Proactivity and Autonomy

Most agents are reactive: they wait for instructions and then act. Proactive agent behaviors (anticipating user needs, automating routine flows) is barely explored.

• Mitigation: Some modular frameworks (AgentKit, Agent S2) allow agent chaining or planning ahead; feedback loops and user profiling modules may help.
• Open problem: trust, interpretability, autonomy governance — how to let the agent act without unwanted surprises.

References