GenClaw: Code-Driven Agentic Image Generation

Rethinking Image Generation: From Prompting to Planning

The landscape of artificial intelligence has transitioned rapidly from simple text-to-pixel translation to sophisticated agentic systems. Early diffusion models and Generative Adversarial Networks (GANs) functioned primarily as "black boxes"—users provided a text prompt, and the model generated an image in a single, end-to-end pass. While these models produce high-quality aesthetics, they often struggle with precision, such as exact object counts, complex spatial layouts, or specific typographical requirements.

GenClaw introduces a paradigm shift by treating image generation as a multi-staged, code-driven process. Instead of asking a model to "guess" the entire scene at once, GenClaw decomposes the task into conceptualization, sketching, and coloring. This approach mirrors the human artistic workflow: an artist first thinks about the subject, sketches the structure (the "skeleton"), and finally applies color and detail. By using executable code—such as SVG, HTML, and Three.js—as an intermediate representation, the system gains a "digital brush" that allows for deterministic control over the visual structure before a single pixel is rendered.

The Limitations of Black-Box Generation

Traditional Large Multimodal Models (LMMs) and text-to-image agents typically rely on iterative prompt refinement. If a model fails to generate exactly five apples, the agent might rewrite the prompt to emphasize the number five. This "trial-and-error" cycle is inherently stochastic; because the final synthesis is hidden within the model's weights, the agent acts more as an optimizer of words than a direct creator of visual structure.

Natural language is often too ambiguous for precise technical tasks. For example, describing the exact coordinates of a menu item or the physical trajectory of a water spray is difficult in prose. This ambiguity leads to "hallucinations" where models ignore constraints, miscount objects, or garble text. GenClaw addresses this by offloading structural planning to code, which is unambiguous and logically rigorous.

The Three-Layer Agentic Framework

The core of GenClaw is its three-layer architecture: Cognitive Structuring, Executable Canvas Construction, and Visual Generation and Review. This framework decouples the "thinking" from the "drawing," allowing for specialized tools to handle different aspects of the creative process.

Layer 1: Cognitive Structuring (Conceptualize)

The process begins with the agent parsing the user's intent. If the request involves real-world facts (e.g., "Design a poster for the 2026 World Cup"), the agent invokes search tools to retrieve current data, such as tournament dates and host cities. For tasks requiring reasoning, such as physics simulations or geometric problems, the agent performs explicit calculations. The output of this layer is a "Structured Record" (often in JSONL format), which prioritizes executability and logical consistency over descriptive flair.

Layer 2: Executable Canvas (Sketch)

This layer transforms the structured plan into a visual "sketch" using programming languages. By using code, the agent defines the deterministic structure of the image:

• SVG (Scalable Vector Graphics): Used for compositional scenes where object counts ($N$) and spatial coordinates are critical. If the user wants thirteen donuts, the agent generates thirteen distinct SVG nodes.
• HTML/CSS: Used for text-heavy tasks like posters or menus. This ensures that text is rendered with perfect spelling and alignment, offloading the difficult task of character generation from the image model.
• Three.js/Python: Used for physical simulations. The agent can model physical laws, such as Archimedes' principle for buoyancy or Hooke's Law for springs, creating a "symbolic world model" that serves as the visual draft.

Layer 3: Visual Generation and Review (Color)

In the final stage, the executable sketch is passed to a high-fidelity image generation model. The model's role is no longer to plan the scene but to "color" it—adding textures, lighting, shadows, and photorealism. After generation, a review module (powered by a Vision-Language Model or user feedback) checks the output against the initial structured record. Because the process is staged, errors are traceable: if a count is wrong, the system can check if the error occurred in the SVG code (sketch) or the rendering stage (color).

Precise Control Through Structured Sketching

One of the most significant advantages of the code-driven approach is its ability to handle complex layouts that traditional models often fail. In tasks involving precise positioning, GenClaw uses the SVG backend to define exact bounding boxes and layer orders.

As shown in the evaluation, direct generators often suffer from attribute-binding issues—for instance, coloring the wrong object or placing items in the wrong relative positions. By generating a sketch first, GenClaw "locks in" these attributes. The image model then follows this structural guide, significantly reducing the likelihood of spatial hallucinations.

Bridging the Gap with Physical Reasoning

GenClaw extends the capabilities of image agents into the realm of physical simulation. Traditional generative models often produce physically implausible scenes because they lack an internal model of physics. GenClaw solves this by having the agent write code to solve the relevant physical equations before drawing.

For example, if asked to depict three balls with different densities in water, the agent identifies the relevant law (Archimedes' Principle) and calculates the submerged fraction for each ball. If a ball has density $\rho$, the submerged volume is proportional to the ratio of its density to the water's density. The agent then translates these calculations into a visual draft where the vertical position (the $z$-axis or $y$-coordinate) is mathematically determined.

Similarly, for a spring system, the agent uses Hooke's Law:

To find the elongation $\Delta L$, it calculates $\Delta L = mg/k$, where $m$ is the mass, $g$ is the gravitational constant, and $k$ is the spring constant. This ensures that the visual representation of the spring's length is physically consistent with the weight attached to it.

Layered Image Editing and Knowledge Grounding

Beyond generation from scratch, GenClaw demonstrates high performance in image editing. Most current editors modify the entire image, which can lead to unintended changes in unedited regions (measured by metrics like PSNR and SSIM). GenClaw adopts a "Layered Editing" approach: it decomposes the image into discrete layers using tools like Segment Anything (SAM). By targeting only the relevant layer in the code, the agent ensures that the rest of the canvas remains untouched.

For tasks requiring deep world knowledge, GenClaw uses a search-and-grounding workflow. When asked to draw a specific real-world location or a future event, the agent searches for reference images and facts, incorporating them into the structured plan.

Empirical Findings and Performance

The researchers evaluated GenClaw across several benchmarks, comparing it to both open-source models (like Qwen-Image) and proprietary systems (like GPT-Image-1.5).

1. Compositional Control: On the GenEval++ benchmark, which tests instruction following for complex layouts, GenClaw achieved a score of $0.878$, outperforming Qwen-Image ($0.668$) and GPT-Image-1.5 ($0.750$). The deterministic nature of SVG code contributed significantly to its superior performance in object counting and spatial relations.
2. Text Rendering: On LongText-Bench, GenClaw achieved scores of $0.989$ for English and $0.988$ for Chinese. This highlights the effectiveness of using HTML/SVG text layers, as the model no longer needs to "draw" characters but simply renders the code.
3. Image Editing: In editing tasks, GenClaw maintained a high VLM-Score of $4.29$ while achieving a PSNR of $27.87$ and an SSIM of $0.718$ for unedited regions. This indicates that its layered approach preserves the integrity of the original image much better than global editing methods.
4. Knowledge Integration: On Mind-Bench, a benchmark for reasoning-driven generation, GenClaw scored $0.57$, surpassing the $0.31$ and $0.38$ scores of other agentic models.

Conclusion and Future Directions

GenClaw demonstrates that the path to more controllable and intelligent image generation lies in decoupling high-level reasoning from low-level pixel synthesis. By using code as a "digital brush," the system provides agents with the tools to plan, calculate, and sketch with mathematical precision. This transparency not only improves the quality of the output but also makes the generation process interpretable and easier to debug.

Future work in this area may focus on reducing the dependency on high-end proprietary models for the final rendering stage and improving the robustness of the code generation itself. As Large Language Models continue to improve in their coding and reasoning abilities, the potential for code-driven visual agents to handle increasingly complex and specialized creative tasks will continue to grow.