VisualSkills

Code and skills for the paper VisualSkill: Multimodal Skills for Computer-Use Agents.

VisualSkill packages application knowledge (LibreOffice Writer / Calc / Impress, GIMP, QGIS, Zotero, Chrome, ...) as a topic-indexed hierarchy of markdown guides with cropped UI screenshots, and exposes it to a Claude Code agent through a load_topic MCP tool. Skills are built with a two-stage pipeline (Stage 1 from official docs, Stage 2 from live UI exploration) and evaluated end-to-end on CUA-World and OSExpert-Eval.

visualskills/
├── skills/                       # Generated skills (Stage 1 + Stage 2, text + multimodal)
├── plugins/                      # Claude Code --plugin-dir bundles
│   ├── cua-world-text/
│   ├── cua-world-multimodal/
│   ├── osexpert-text/
│   └── osexpert-multimodal/
├── tools/
│   ├── cua-world-controller/  # GUI action MCP (screenshot / click / type / hotkey)
│   ├── osexpert-controller/       # Same, targeting the OSExpert-Eval VM
│   └── skill_server.py           # load_topic / list_topics MCP (atomic prose+figures load)
├── preprocess/skill-pipeline/
│   ├── stage1/                   # Stage 1: docs-driven skill generation
│   └── stage2/                   # Stage 2: live UI explorer
├── scripts/
│   ├── run-cua-world/         # Inference on cua-world envs
│   └── run-osexpert/              # Inference on OSExpert-Eval
└── vendor/
    ├── gym-anything/            # CUA-World benchmark envs (submodule)
    └── OSWorld/                 # OSExpert-Eval benchmark (submodule; OSWorld fork with evaluation_examples_expert/)

Both inference and Stage 2 skill construction depend on the two benchmark submodules under vendor/ and on a Docker host with KVM + sysbox for live Ubuntu desktop containers.

git clone --recurse-submodules https://github.com/XMHZZ2018/VisualSkills.git
cd VisualSkills
# or, if already cloned:
git submodule update --init --recursive

vendor/gym-anything and vendor/OSWorld are pinned to specific commits.

# Python packages (host)
pip install --break-system-packages \
    mcp httpx pymupdf Pillow PyYAML \
    numpy jsonschema paramiko pycryptodome requests docker anthropic \
    beautifulsoup4 markdownify

# OSExpert-Eval's own deps
bash scripts/run-osexpert/setup.sh

The benchmarks run each task in a fresh Ubuntu + GNOME + systemd container. Docker must be configured with the sysbox runtime so systemd works inside containers:

docker info | grep -i runtime   # must list sysbox-runc

Install sysbox from https://github.com/nestybox/sysbox. On GCP, use an n2-standard-16 VM (or larger) with --enable-nested-virtualization for OSExpert-Eval's KVM guest.

Both runners drive the agent through the Claude Code CLI (not the raw API), because plugins/skills are only surfaced through --plugin-dir:

npm install -g @anthropic-ai/claude-code
claude login          # writes ~/.claude/.credentials.json

~/.claude/.credentials.json is mounted read-only into every Claude container, so this only needs to be done once per host.

Each runner has its own thin image that pins the CLI + the corresponding MCP server:

# cua-world
docker build -t cw-claude-cli \
    -f scripts/run-cua-world/Dockerfile.claude-cli scripts/run-cua-world/

# OSExpert-Eval
docker build -t osexpert-claude-cli \
    -f scripts/run-osexpert/Dockerfile.claude-cli scripts/run-osexpert/

Both runners share the same three skill modes:

Five domains × five skill conditions = 25 pre-built eval configs, matching the paper's Table 1 exactly. Run any one of them with:

bash scripts/run-cua-world/run.sh \
    --config scripts/run-cua-world/experiments/configs/<domain>_<mode>.yaml

Config matrix — every cell is a ready-to-run YAML in scripts/run-cua-world/experiments/configs/:

Every config uses model: claude-opus-4-6, num_parallel: 4, rerun: true, and task_timeout: 5400 as a safety net — the actual per-task step budget is CUA-World's own init.max_steps, capped at 120 steps by the runner (see run_task.py:404-417) to keep trajectories inside the Anthropic 20 MB request-payload envelope.

Per-task outputs land under workspaces/{model}/skill-{mode}/{env}/{task_id}/ with result.json, score.txt, screenshots/, and the full Claude event log. See scripts/run-cua-world/README.md for the config schema, parallel-worker layout, and the trajectory viewer.

# Baseline
python3 scripts/run-osexpert/run.py \
    --provider_name docker --domain chrome --skill_mode none

# Multimodal skill (4 parallel workers, each with its own VM + Claude container)
python3 scripts/run-osexpert/run.py \
    --provider_name docker --domain chrome --skill_mode multimodal --parallel 4

Claude Code runs inside osexpert-claude-cli with no Docker socket and no direct network path to the VM — the only channel is a bridge that forwards actions to the QEMU guest at 1280×720 (matching OSExpert-Eval's original Anthropic agent). See scripts/run-osexpert/README.md for provider options (docker, vmware, virtualbox), GCP VM setup, and the full argument reference.

Every domain skill in skills/<domain>-{text,multimodal}-stage{1,2}/ is produced by the same two-stage pipeline. Each stage emits a matched pair of artifacts — a multimodal form with cropped UI figures, and a text-only form in which each figure has been replaced by a verbal description of the same image — so any inference-score gap between them isolates the contribution of visual evidence.

(PDF: assets/skill_generation_new.pdf)

At inference the multimodal artifact is loaded through tools/skill_server.py, which exposes load_topic(topic) and list_topics(). The agent gets a topic's prose and figures atomically in one MCP call instead of issuing separate Reads.

Mines a skill from authored sources (a PDF user guide, a docs website, or a clustered list of CUA-World task descriptions). Runs entirely on the host. Lives at preprocess/skill-pipeline/stage1/.

Five phases, per YAML config:

1. Taxonomy — build a category → topic tree from the PDF ToC, HTML headings, or task clustering.
2. Pages — render (PDF) or fetch (HTML) each topic's source pages.
3. Figures — extract per-topic UI figures using PyMuPDF's embedded image rects (PDF) or inline <img> tags (HTML).
4. Guides — one Claude call per topic to write the prose; a second call anchors each figure back into the prose.
5. Use-when + Index — one keyword line per topic, then write SKILL.md.

An optional Phase 6 derives the matched text-Stage-1 artifact by verbalizing every figNN.png reference in the multimodal guides.

# Full pipeline, both modalities
./preprocess/skill-pipeline/stage1/run.sh \
    --config preprocess/skill-pipeline/stage1/configs/libreoffice_writer.yaml \
    --mode both

# Derive text-Stage-1 from multimodal-Stage-1
./preprocess/skill-pipeline/stage1/run.sh \
    --config preprocess/skill-pipeline/stage1/configs/libreoffice_writer.yaml \
    --phase 6

Configs for LibreOffice Writer / Calc / Impress, GIMP, QGIS, and Zotero live under preprocess/skill-pipeline/stage1/configs/.

Augments the Stage 1 skill with knowledge that only exists in the running application. Runs on the GCP osexpert VM because phases 1–2 spin up CUA-World Docker containers. Lives at preprocess/skill-pipeline/stage2/.

Two exploration sub-passes:

• Free UI explorer. An Opus planner inspects the idle app and proposes ~8 UI regions ("drawing toolbar", "properties sidebar", ...). ~8 Sonnet workers each drive the live app in parallel, scoped to one region, capturing screenshots and per-control notes.
• Training-task-targeted explorer. Failed train-task trajectories are reviewed to surface UI regions the agent measurably struggled with; additional workers are dispatched against those targets. Targets are scoped to UI regions, not specific tasks, so patches transfer to any test task touching the same UI surface.

Assembly is deterministic: an Opus assembler reconciles worker notes into per-region reference sections, an Opus mapper decides which Stage 1 guide each region belongs to, and the sections are inlined into a fresh copy of multimodal-Stage-1 with ui-*.png crops.

The matched text-Stage-2 artifact can be produced two ways, controlled by --text-source:

• derived (default, matches paper): verbalize each figure ref in the multimodal-Stage-2 guide (same mechanic as Stage 1 Phase 6). Prose is identical to multimodal-Stage-2 up to those refs, giving a tight ablation of what the visual modality contributes.
• independent: a separate text-only Claude pass over the same worker notes + screenshots, mirroring the mm region set. Prose is genuinely independent between the two artefacts.

# Stage 2 free explorer — full chain (paper default: mm-stage2 + derived text)
./preprocess/skill-pipeline/stage2/run.sh \
    --config preprocess/skill-pipeline/stage2/configs/writer.yaml --phase all

# Independent text prose (Option 1) alongside mm
./preprocess/skill-pipeline/stage2/run.sh \
    --config preprocess/skill-pipeline/stage2/configs/writer.yaml --phase all \
    --mode both --text-source independent

# Training-task-targeted explorer (after Stage 2a workers finish),
# chained straight into assembler + mapper + inline(s) so the augmented
# 2a+2b worker set produces the final skill pair in one command.
python3 preprocess/skill-pipeline/stage2/run_phase_2b.py \
    --v3-config preprocess/skill-pipeline/stage2/configs/writer.yaml \
    --rollouts-config scripts/run-cua-world/experiments/configs/writer_train16_mm_skill.yaml \
    --app-name "LibreOffice Writer" \
    --then-inline --mode both --text-source derived

@article{jiang2026visualskill,
  title   = {VisualSkill: Multimodal Skills for Computer-Use Agents},
  author  = {Jiang, Ziyan and An, Li and Liu, Yujian and Ji, Jiabao and
             Wu, Qiucheng and Andreas, Jacob and Zhang, Yang and Chang, Shiyu},
  journal = {arXiv preprint arXiv:2606.18448},
  year    = {2026}
}