GPT-5.6 Sol leads colored-pencil model showdown

The four vision models produced 28 drawings in a colored-pencil arena, working from either a blank canvas and a text prompt or one of two reference images: the Mona Lisa and Van Gogh’s Starry Night. Each model used the same tools to set color, brush width, and pressure; draw layered strokes; smudge, erase, and inspect its canvas.

The open-source harness is available at github.com/hershalb/canvas-arena. The authors emphasize that this is not an objective measure of artistic ability, but argue that open-ended tasks reveal differences that standard benchmarks can miss.

Tool use, cost, and speed

The models developed sharply different working styles:

• GPT-5.6 Sol never called set_color, set_brush, or set_pressure, instead specifying those settings inside draw calls.
• Grok 4.5 used those setting tools for 65% of its 1,349 tool calls, averaging 99 steps per drawing.
• Claude Fable 5 relied heavily on smudge, with 123 calls, and repeatedly reviewed its work.
• Gemini 3.6 Flash devoted nearly one-third of its calls to 'view_canvas, averaging about 23 self-reviews per drawing**.

Claude was by far the most expensive model in the test, costing an estimated $160.58 for seven drawings—roughly 20 times the cost of GPT-5.6 Sol ($7.74), Grok 4.5 ($9.21), or Gemini 3.6 Flash ($12.87). Grok processed the most tokens, at 34 million, but about 98% were cached reads. Gemini used 27.7 million tokens, also benefiting heavily from cached reads.

Target scores and model performance

Similarity was measured with SSIM, where 0 means less similar and 1 means more similar. Images were resized to 256×256 before calculating SSIM and RMSE; the scores measure structural and pixel closeness, not artistic quality.

Recommended reading

Google’s Gemini 3.6 Flash cuts tokens, trails rivals

| Model and target | Tool calls | Time | Reviews | Final SSIM | Peak SSIM | RMSE | |---|---:|---:|---:|---:|---:|---:| | GPT-5.6 Sol — Mona Lisa | 448 | 4m 41s | 7 | 0.325 | 0.352 | 54.2 | | Claude Fable 5 — Mona Lisa | 86 | 18m 58s | 27 | 0.286 | 0.289 | 56.9 | | Grok 4.5 — Mona Lisa | 106 | 4m 30s | 5 | 0.151 | 0.152 | 95.6 | | Gemini 3.6 Flash — Mona Lisa | 60 | 5m 31s | 22 | 0.337 | 0.449 | 51.1 | | GPT-5.6 Sol — Starry Night | 235 | 5m 47s | 4 | 0.130 | 0.179 | 37.8 | | Claude Fable 5 — Starry Night | 43 | 11m 30s | 12 | 0.153 | 0.176 | 40.9 | | Grok 4.5 — Starry Night | 58 | 5m 51s | 3 | 0.039 | 0.039 | 104.9 | | Gemini 3.6 Flash — Starry Night | 74 | 6m 36s | 24 | 0.172 | 0.190 | 42.8 |

The models generally plateaued early. In all eight target runs, the final image scored below the best intermediate version. Gemini reached the highest peak—0.449 on the Mona Lisa—but ended at 0.337 after continuing to edit. GPT-5.6 Sol peaked at 0.352 on the Mona Lisa and finished at 0.325; on Starry Night, it fell from 0.179 to 0.130.

The authors judged GPT-5.6 Sol the strongest overall, naming its Starry Night and rose drawings their favorites. They described Grok 4.5 as largely unusable, while Claude Fable 5 ranked second on quality but was much slower and roughly 20 times more expensive than the other leading options.