Engineering Autonomous LLM-Based Desktop Agents: A Modular Approach

Large Language Models (LLMs) are increasingly used as decision-making components in autonomous agents operating in real-world environments. A particularly challenging setting is autonomous computer use, where agents must interact with graphical user interfaces under partial observability and without task-specific APIs. Despite recent progress, it remains unclear which architectural components are most critical for reliable performance in such environments. This article presents a modular architecture for LLM-based desktop agents and a controlled empirical study of two key design dimensions: multimodal perception and revision-aware short-term memory. The architecture integrates perception, short-horizon planning, grounded execution, explicit post-execution evaluation, and a revisable task-state memory. We evaluate the system on the OSWorld benchmark under four ablation settings to isolate the contribution of each component. Results show that visual perception is the dominant factor, improving grounding and task success compared with using accessibility metadata alone. Revision-aware short-term memory further enhances performance by improving recovery from execution failures and maintaining task continuity, especially when combined with visual perception.