VisualEvolve: A Skill Self-Evolution
Framework for Visual Reasoning

Jiawei Wang¹, Yucun Li², Xueqi Li³, Shenghao Wang², Xinquan Gong², Dan Zhang¹,

Zheqi Lv⁴, Yuquan Le⁵, Shaozu Yuan⁶, An Zhang⁷, Da Cao², Tat-Seng Chua¹

¹National University of Singapore ²Hunan University ³Shenzhen University

⁴Cornell University ⁵Hunan Normal University ⁶Meituan

⁷University of Science and Technology of China

Circular comparison across six visual-reasoning evaluation sets. The dashed circle marks each set’s best reported reference. The lighter outer segments show the positive accuracy differences of four VisualEvolve backbones; labels give the original accuracy scores. — **VisualEvolve versus training-based agents on six visual-reasoning evaluation sets.** VisualEvolve uses our framework with four backbones; References are agents trained with SFT and/or RL. The dashed circle marks the highest reported reference accuracy in each set. VisualEvolve: Acc@1; †: Avg@32; ‡: Avg@8.

Abstract

Visual tool use enables vision-language models (VLMs) to acquire task-relevant evidence, but its effectiveness can be limited by predefined visual operations and the tool-use strategies learned during training. Representing these operations and strategies as reusable external skills allows them to be refined through self-evolution without updating model parameters. However, skill refinement can introduce performance regressions. Meanwhile, inspecting accumulated task execution trajectories together with skill evolution history may lead to evolution evidence overload. To address these challenges, we propose VisualEvolve, a self-evolving framework that evolves executable visual operations and tool-use strategies for VLMs through a Select–Execute–Evolve–Evaluate cycle. Skill-Library Snapshot Selection uses performance on a fixed validation set disjoint from execution tasks to select an effective historical snapshot for subsequent evolution, limiting the propagation of performance regressions. Progressive Evidence Access (PEA) guides capability-gap diagnosis through a coarse-to-fine hierarchy comprising the Skill-Performance Report, Evolution Log, Trajectory Summaries, and relevant raw trajectory segments. Across four VLM backbones and six evaluation sets from three benchmarks, VisualEvolve improves over the Initial Skill Library in 23 of 24 settings, with a maximum gain of 15.1 percentage points on VisualProbe Hard using Qwen3.5-Plus. Ablation studies support the contributions of Snapshot Selection and PEA, while further experiments show that skills can be improved using experience from different data settings and directly reused across backbones. Moreover, VisualEvolve enables effective skill self-evolution with over an order of magnitude fewer task-solving trajectories than estimated for representative RL-based methods.

Framework Overview

VisualEvolve evolves a skill library through multi-round Select–Execute–Evolve–Evaluate cycles. Skill-Library Snapshot Selection chooses the starting skill library, Execute applies it to the execution set to collect execution evidence, Evolve diagnoses capability gaps and updates the skill library, and Evaluate assesses and archives the updated skill library.

The VisualEvolve framework showing the frozen Worker, Self-Evolving Agent, skill library, snapshot selection, and progressive evidence access.

To limit the propagation of performance regressions, Skill-Library Snapshot Selection guides the agent to choose a starting library from archived snapshots based on validation performance.

Main Results

Macro-average

68.877.6+8.8 pp

Full results · Table 1

Table 1. Accuracy (%) on six evaluation sets from the three benchmarks.
Backbone	Method	V*Bench	HRBench		VisualProbe			Macro Avg.
Backbone	Method	V*Bench	4K	8K	Easy	Medium	Hard	Macro Avg.
Training-based methods
Qwen2.5-VL-7B	Pixel Reasoner^R	86.3^†	74.0^‡	66.9^‡	58.4^†	29.6^†	28.8^†	57.3
	DeepEyes^R	83.3^†	73.2^‡	69.5^‡	60.1^†	29.8^†	35.1^†	58.5
	Mini-o3	88.2^†	77.5^‡	73.3^‡	67.0^†	50.4^†	48.0^†	67.4
	DeepEyesV2	81.8	77.9	73.8	—	—	—	—
Qwen3-VL-32B	CodeVision-32B	86.2	84.3	76.1	—	—	—	—
Within-backbone comparisons
Qwen3.5-Plus	Single-Pass	83.8	84.4	80.0	66.7	37.7	35.8	64.7
	Initial Skill Library	86.4	87.0	83.3	68.1	48.1	39.6	68.8
	VisualEvolve	92.1	91.0	90.0	77.3	60.4	54.7	77.6
Qwen3.6-Plus	Single-Pass	88.0	87.8	84.1	58.9	34.7	31.1	64.1
	Initial Skill Library	90.6	88.5	87.4	66.7	53.4	36.8	70.6
	VisualEvolve	91.1	92.0	90.9	73.8	58.2	50.0	76.0
Kimi-K2.5	Single-Pass	82.7	79.8	75.8	50.4	32.1	24.5	57.6
	Initial Skill Library	85.3	88.5	81.5	70.9	54.1	35.8	69.4
	VisualEvolve	89.0	89.5	85.5	70.9	60.1	49.1	74.0
Doubao-Seed-2.0-Pro	Single-Pass	88.5	83.3	76.5	60.3	36.2	29.2	62.3
	Initial Skill Library	89.5	86.8	86.0	75.9	59.0	42.5	73.3
	VisualEvolve	90.6	89.4	89.0	79.4	63.8	50.9	77.2

For training-based methods, results in rows marked with ^R are taken from the Mini-o3 paper, while results for the remaining methods are taken from their respective original papers. Sources: Mini-o3, DeepEyesV2, and CodeVision.

†: Avg@32; ‡: Avg@8; our runs: Acc@1. Macro Avg.: unweighted mean of all six sets, shown only for complete rows. “—”: unavailable.

Bold/underline: best/second-best per column within training-based methods or each of our backbone groups; ties share a rank.

Ablation Study

Framework stages. All three framework stages contribute substantially to multi-round skill evolution. Removing Skill-Library Snapshot Selection, Execute, or Evaluate reduces macro-average accuracy by 5.4, 7.7, and 6.4 percentage points, respectively. In particular, w/o Execute reaches only 61.5% best-so-far validation accuracy, highlighting the importance of execution evidence for effective skill updates.

Progressive Evidence Access. The ablations support the effectiveness of the coarse-to-fine evidence-access mechanism. Removing any individual PEA component reduces macro-average accuracy by 2.6–4.6 percentage points. Jointly removing the Skill-Performance Report and Evolution Log reduces accuracy by 5.8 percentage points, while removing PEA entirely reduces accuracy by 6.3 percentage points.

Table 2 · Component ablations
Variant	Macro Avg.	Δ vs. full
VisualEvolve	77.6	—
Evolution stages
w/o Snapshot Selection	72.2	−5.4
w/o Execute	69.9	−7.7
w/o Evaluate	71.2	−6.4
Progressive Evidence Access
w/o SPR	73.6	−4.0
w/o EL	75.0	−2.6
w/o TS	73.0	−4.6
w/o SPR + EL	71.8	−5.8
w/o PEA	71.3	−6.3
Baseline
Single-Pass	64.7	−12.9

Execute ablation validation curves for Qwen3.5-Plus. — Figure 3. Execute ablation over 16 evolution rounds. Lines show best-so-far validation accuracy; points show each round.

All ablation variants use Qwen3.5-Plus as the backbone model. Macro Avg. is the mean Acc@1 (%) across six evaluation sets. Δ vs. full denotes the change in percentage points relative to the full VisualEvolve framework.

Notation: PEA: Progressive Evidence Access; SPR: Skill-Performance Report; EL: Evolution Log; TS: Trajectory Summary.

For w/o Evaluate, all snapshots are evaluated on the fixed validation set only after evolution, then the highest-scoring snapshot is tested on held-out benchmarks.

Generalization

(a) Evolution-data sources. VisualEvolve consistently improves macro-average accuracy across all three evolution-data sources, with gains of 8.8, 7.1, and 5.0 percentage points over the Initial Skill Library under Default, Mini-o3, and CodeVision, respectively. These results support the robustness of skill evolution across the data sources.

(b) Cross-backbone transfer. The skill library evolved with Kimi-K2.5 is reused without modification and improves over the Initial Skill Library on all three other backbones. Its macro-average accuracy is only 2.7, 1.4, and 1.1 percentage points below backbone-specific evolution on Qwen3.5-Plus, Qwen3.6-Plus, and Doubao-Seed-2.0-Pro, respectively, retaining most of the performance gains from backbone-specific evolution.

Generalization across data sources and target VLM backbones. — Figure 4. Generalization across evolution-data sources and cross-backbone skill transfer. Values are macro-average accuracy over six evaluation sets.

Cost Analysis

(a) Trajectory budget. VisualEvolve uses 3,912 Worker task-solving trajectories, whereas the training-based baselines use an estimated 63–209× as many RL trajectories.

(b) Evolution cost. The mean API cost of each evolution round is $2.55–$4.12 across the evaluated backbones.

(c) Inference overhead. The evolved skill library consistently improves accuracy with only modest additional inference cost. On VisualProbe Hard, an additional cost of $0.0268 per example improves accuracy by 15.1 percentage points over the Initial Skill Library, using Qwen3.5-Plus.

Trajectory budgets, evolution API costs, and inference costs from the paper. — Figure 5. Trajectory budget, evolution cost, and added inference cost versus accuracy gain. RL trajectory counts are estimates; inference results use Qwen3.5-Plus. Costs reflect the reported experiments.

Qualitative Examples

Follow two recorded trajectories to see how evolved skills guide visual reasoning.

Find the right book, one crop at a time.

The evolved library combines read_text guidance with progressively tighter crops to locate and verify a book title in a crowded bookshelf image.

01
Locate
Use the read_text skill to guide the search toward the relevant bookshelf region.
02
Refine
Crop progressively tighter regions to gather readable visual evidence.
03
Verify
Check the target book spine before returning the answer.

View in paper ↗

Figure 9. The complete recorded bookshelf trajectory, including iterative crops, verification, and the final answer.

Using VisualEvolve

Run skill evolution with your choice of model by updating the model and API settings. To adapt to a new domain, prepare domain-specific data, initial skills, and task instructions.

Same domain, a different model.

Switch the task-solving model through configuration.

01
Choose the task-solving model
Set worker_model to the model ID supplied by your API provider. Choose a model that supports the target task’s inputs and tool use.
02
Connect the API
For a new provider, update base_url and api_key_env, set compatible model IDs for the agent roles, and add the required API keys to .env.
03
Run skill evolution
Create a configuration from the released defaults, then run the same evolution entry point with the new configuration.

Explore the code ↗

PYTHON · CREATE MODEL CONFIG

import json
from pathlib import Path

cfg = json.loads(Path("configs/default.json").read_text())
cfg["worker_model"] = "qwen3.6-plus"

# If changing API providers, also configure:
# cfg["base_url"] = "https://<provider>/v1"
# cfg["api_key_env"] = "MODEL_API_KEY"

Path("configs/model.json").write_text(
    json.dumps(cfg, indent=2) + "\n"
)

Example model ID: qwen3.6-plus. Replace it with a model available at your endpoint. The other role settings are inherited from configs/default.json.

Run from the repository root

python run_evolution.py --config configs/model.json

VisualEvolve: A Skill Self-Evolution Framework for Visual Reasoning