Open AI models painting an isometric city

Why

Our isometric map at iso.slopcouver.ca is baked from the game's own 3D scene: an orthographic camera renders the city in 512-pixel tiles. That gives a true-to-data map, but only one look. Hand-illustrated city maps (watercolour, lithograph, vector) are expensive to commission and can't be redrawn when the data changes. If a model could restyle our render while keeping every building where the data puts it, a map could be re-illustrated automatically after each data update.

The hard part is not the style; it is consistency. A map is hundreds of tiles painted separately, and image models don't know what their neighbours painted. Any drift shows as a grid of seams, and any "creativity" moves streets that should not move.

Hypothesis

An open-licence image model running on a 6 GB consumer GPU can repaint a block of our isometric render so that:

1. seams don't show: the seam ratio at every tile edge (defined below) is no higher than our own render's, 1.26; and
2. the geometry stays: at least 80% of the render's strong edges (buildings, kerbs, shorelines) are still within 2 px in the painting;
3. at a speed that could paint a neighbourhood overnight (under 2 minutes per tile).

The hypothesis fails for a model if any of the three fails.

Method

Test patch

Cambie Street and West 6th Avenue, a mixed block of offices, low-rise housing, a park and the False Creek shore, rendered by the game at heading 315°. The test patch is a 2 × 2 block of 512 px tiles (1024 × 1024 px) cut from a 4 × 2 render, with a 16-bit depth map rendered alongside it. Every model paints the same patch with the same five style prompts and seed 7.

Painting a block without seams

Our first trial (on a 3 × 3 block at False Creek) painted each tile on its own. That was the obvious approach and it failed. We then built the method all later runs use:

• Overlapping windows: each 512 px tile is painted inside a 640 px window (64 px of overlap with its neighbours), in row order.
• Colour match: before blending, each new window gets a per-channel gain and offset fitted over the band it shares with tiles already painted.
• Feather: a linear blend across the shared band.
• Conditioning: the model sees our render (image 1) and its depth map (image 2). Klein takes both as references; SD 1.5 uses the render for img2img (strength 0.65) and the depth map through a depth ControlNet.

Measures

• Seam ratio: the mean absolute colour jump across a tile edge, divided by the median jump between neighbouring pixel columns (or rows) inside the tiles. About 1 means the edge can't be told from the picture around it; well above 1 is a visible seam. We report the worst edge of the block. Our own render scores 1.26 on its vertical edge (0.98 horizontal), so that is the bar.
• Geometry kept (edge recall): take the strongest 10% of Sobel edges in the render; the share of those pixels that still have a strong edge within 2 px in the painting.
• Speed: wall-clock seconds per 512 px tile, excluding the one-off model load.

Results

1. The seam problem, and how far the method fixes it

Klein on a 3 × 3 block of False Creek. Same model, same prompt; only the way the block is painted changes.

2. SD 1.5 against FLUX.2 Klein 4B, five styles

Both models with the full method on the Cambie & 6th patch. Green rows pass all three criteria.

3. Speed and hardware notes

• Klein, one 512 px tile on its own: 46.5 s on the GPU; 329.5 s on the CPU (7× slower). A single 1024 px tile took 267.8 s, so painting at 512 px and blending is faster per pixel than painting large tiles.
• The overlap costs time: a 640 px window is 56% more pixels than the 512 px it keeps.
• Arithmetic, not a measurement: at 66–88 s per tile, 1,000 tiles is 18–24 GPU-hours on this card.

Source: the restyle trial's run logs (one tile on GPU and on CPU, a 1024 px tile, the 3 × 3 runs).

4. The overnight bake-off (in progress)

To see whether newer or larger open models do better, we set up a bake-off of six model setups over the same patch and five styles, with the same window method, seed and measures. Results so far:

The bake-off queue was halted after the SD 1.5 runs because it shared the one GPU with other work. So the comparison with bigger models is not done; we will fill this table in as runs finish. The ISO map's Style menu at Cambie & 6th reads the list of finished runs live, so new runs appear there before this page is updated.

5. Compared with paid image models

We did not test hosted, paid image models, so we have no numbers for them and make no quality claim. What we can say is structural: our method needs a model that accepts a reference image (and ideally a depth map), is run thousands of times with a fixed seed, and can be re-run whenever the data changes. A local open model makes that a fixed hardware cost with no per-image fee, no upload of our renders and a licence we can read; a hosted model trades that for speed and, often, quality. A fair comparison would run the same patch, prompts and measures through a hosted model; that is a natural next step.

Limitations and threats to validity

• One patch, one seed. Five styles on one 2 × 2 block is a small sample. A different neighbourhood (dense towers, open water) could rank the models differently.
• The seam ratio is a proxy. It catches colour and brightness steps at the edge, not a road that changes width or a roof that changes material across an edge. Values well below 1 (0.80) mean the painting is smoother at the edge than elsewhere, which is the feather at work, not a quality win.
• Edge recall rewards keeping edges, not keeping the right content. A painting can keep outlines and still recolour a park as a car park. We checked the images by eye as well; the 0.80 bar is our choice, not a standard.
• Prompts were tuned on Klein first. SD 1.5 got shortened versions of the same prompts (its text encoder reads 77 tokens), which may disadvantage it.
• Timings are single runs on a shared machine; Klein's photoreal run had a slow model load (172 s; the cause wasn't recorded).

Verdict

Practical application

• For the game and the ISO map: a "Style" switch over the same data. Five Klein styles are live at Cambie & 6th as a trial.
• For city maps and tourism: illustrated maps that are regenerated from data rather than redrawn by hand; a new building appears in the illustration at the next rebuild.
• For other cities: the input is any 3D city render plus a depth map; the method and measures don't depend on Vancouver.
• What it would take: the whole ISO map is thousands of tiles at several zoom levels and four headings, which is days of GPU time on one GTX 1060 per style. A newer card with more memory would, we expect, cut time per tile substantially (not measured). The seam and geometry checks should run on every tile automatically, with failing tiles repainted or flagged.

Reproducibility

Scripts (paths in the Slopcouver repository, which is private for now; we'll share them on request):

• tools/iso/paint-patch.py: the window loop (640 px windows, 64 px overlap, colour match, feather).
• tools/iso/seam-check.py: seam ratio and edge recall.
• The bake-off driver (bakeoff.py), its comparison sheet (sheet.py) and a per-model README writer that records licences and verifies file hashes.

python bakeoff.py --model sd15 --styles watercolour,photoreal,vector,litho,pixel --out runs
python seam-check.py src-mid.png runs/sd15__watercolour/mid.png --tile 512 --json seams.json

Data and credits: the source render is Slopcouver's own scene, built from City of Vancouver Open Data (OGL–Vancouver) and OpenStreetMap (ODbL). Models: FLUX.2 Klein 4B (Black Forest Labs, Apache-2.0); Stable Diffusion 1.5 (CreativeML Open RAIL-M); ControlNet 1.1 depth (lllyasviel, OpenRAIL).