Summary
Human–AI Design (HAD) is a prototype for designing in conversation with an AI. You sketch; the assistant sees your plan, asks what you are designing, talks it through with you, and can act on it. Like the assistant in an operating room who hands the surgeon the right instrument, it can switch you to the tool you need, hand you a part to place, or act itself: place, move, rotate, or swap things, and explain why, in words you can check and undo. I developed it to explore how people and AI can design together when models interpret and computational tools decide.
One design model serves every surface: a 2D drawing canvas, a 3D view you can walk or fly through on screen, and a VR walkthrough on a headset. This page follows one request through that collaboration: how the plan becomes something a model can understand, how intent becomes a form its tools can act on, and how the person stays the author. Placing a bed is the worked example; the subject is the hand-off.
Human–AI design process Conversational assistant Structured scene context Typed intent Deterministic placement 2D · 3D walk & fly · VR
Picture people designing in conversation: they describe, point, reconsider. “Bed against the long wall, nightstands either side, desk facing the window” is a complete instruction for a person who can see the room. A drafting tool still needs positions and angles before those words can become part of the plan.
Language models can now hold that conversation fluently, which sharpens the question: who decides where the bed actually goes? I developed HAD to find out what a design conversation becomes when the AI on the other side can act: let the model interpret, and give it tools that can place. HAD is research and development rather than a product line: two experiments on the same prototype decided how that hand-off works.
Experiment one gave the realtime voice model the whole job. It could discuss the plan, but its only tool named an item, so it had to tell the person where to click. It never emitted coordinates, yet it described positions nobody could draw, sent a desk and chair to opposite corners, and knew nothing of wall lengths or door swings. The finding was clear: the realtime model is weak at spatial math. It was doing its job as a conversational partner; the tool surface was the bug.
Experiment two stopped asking it to do any. The model passes intent in a small vocabulary to a deterministic solver: a tool that computes positions and angles from the plan itself. When people said that confirming each proposal by click felt like the assistant was suggesting rather than placing, I let placements commit on their own, with undo. Before any of that, the solver needed a room it could reason about.
The model was doing its job. The tool was the bug.
To understand the plan it discusses, the assistant needs rooms. People draw walls as strokes at any angle, including partitions that simply end against another wall. Nobody draws a room. HAD joins wall ends that nearly meet, splits a wall where another lands on it, and walks the resulting network to find every closed face: a room with its area, centre, and surrounding walls.
Take a bedroom 4.0 by 3.6 m, with a door on its south wall and a window on its east wall. The room is recomputed whenever the plan changes, never stored. Each opening remembers its wall and its position along it, so it moves or disappears with that wall.
Together, these relationships form the design model, the structured description of the plan that the rest of HAD works from. Its knowledge of rooms comes from the walls people draw and the openings attached to them. The next question was how to describe that information to a language model.
I prepared scene context, a task-specific description of the current plan, so the assistant can understand what it is discussing. For conversation, HAD tells the plan as an architect might: which room, how large, which openings, what is inside. That description connects the conversation to the drawing.
Layout planning needs a survey in 10 cm cells, including the free space on each side of existing objects. A precise move needs a smaller, exact brief about one object, its neighbours, and the room’s safe interior rectangle. Each model also receives a canvas screenshot.
The door’s swing arcs, the window’s view cones, and the rules about what may stand against a wall stay with the solver, because it is the part that can be held to them. Tell each task what it needs, and no more. What the model may say back is just as deliberate.
A typed instruction is a form with a limited vocabulary for expressing intent. It can ask for a wall, choosing the longest, shortest, or a specific one; a corner; the centre; a named side of another piece; somewhere near something; or a position between two things. There is no field for coordinates.
Placement can face into the room, toward the door, toward the window, toward another object, or match its anchor. References identify a piece by name, by the current selection, or as “the piece before this one.” These choices let the model express relationships that the solver can resolve.
“Put a queen bed in the bedroom” becomes an item, a room, the longest unobstructed wall, and a direction into the room. “And a nightstand on the left” refers back to that bed and names its left side. The model has interpreted; it has not measured. Measuring is the solver’s job.
The solver ranks candidate walls by length, longest first. Ties are broken by preferences such as being away from the door, then by a stable order, so the same request in the same room gives the same answer. It poses this bed against the wall, facing into the room.
It checks that the bed stays inside the room without overlapping walls or furniture. For wall-anchored items, it tests the door’s swing arc and protects the window’s view from tall pieces. Here, south and north tie in length, but the door’s swing takes the southern spot; the bed goes north.
If a wall fails, the solver tries the next. For a kit, pieces are solved in dependency order against a scene already holding the earlier ones, and anything that does not fit is reported by name. Each placement changes what is available to the next.
A single item that fits nowhere on its requested anchor instead goes to the nearest open spot, and the assistant says so. That fallback matters: the result can differ from the requested position. From the person’s side, the machinery gives way to something simpler: a placement they can see and undo.
Before placing anything, the assistant asks what you are designing and which room to focus on. It keeps that goal in view as you talk about the room, whether by voice or text. The request grows out of that conversation.
When you ask for the bed, it appears with a small AI badge that fades after a few seconds. The assistant explains its position in ordinary words: against the longest clear wall, facing into the room. Select it later and the inspector shows which tier and tool placed it, and when.
“Set up the office” arrives as one batch, and one undo removes the whole kit. The bed and the nightstand, requested separately, take two undos. I wanted the unit of reversal to follow the action, so a room kit remains something a person can readily take back.
I kept structure on the slower path: walls, doors, windows, and labels are placed by your click. Here the assistant works like a surgical assistant handing over instruments. Ask for a partition and it activates the interior wall tool and tells you where to start and end; ask for a door and it hands you one that snaps onto the wall it names; ask for the bed, or ask it to move or turn something, and it acts itself. The assistant furnishes; you draw. Both work on the same design model, which makes the next part possible.
01A brief, not a layout
An empty sheet, a brief, and an order of work proposed before any wall is drawn.
YouI want a two-bedroom apartment, a separate home office, and a comfortable shared living space.
HADLet’s define the shell and rooms first. Then we can make the furnishing decisions together.
02The first wall, drawn by hand
The first ten-metre wall of a ten-by-eight outline is on the sheet, begun with the exterior wall tool; the plan keeps room to the south for later changes.
YouThe first wall is ten metres long.
HADContinue around the outline; the enclosure will define the initial space.
03Four rooms from drawn walls
The shell and two partitions are drawn by hand; the editor detects four enclosed rooms, and the assistant asks for openings before furniture.
YouThe partitions are drawn. What comes next?
HADDoors and windows before furniture. Let’s establish access and daylight openings.
04A plan with names
Doors and windows placed by the person, and four labels, Bedroom 1, Bedroom 2, Office, Living Room, that make later requests specific.
YouAnd the large southeast space is Living Room.
HADNow we can make decisions room by room.
05We forgot the kitchen
A new requirement arrives after the plan is settled; the answer is a south extension off Living Room, drawn by hand, and a fifth region the editor detects the moment its boundary closes. The bedroom count stays.
YouWe’ve forgotten the kitchen. Can we add it without losing a bedroom?
HADWe could extend south from Living Room, with a direct connection to the shared space.
06Five named spaces
The extension joined by a double door, lit by a south window, labeled Kitchen.
YouAdd a south window and label it Kitchen.
HADFive named spaces are ready for the furnishing study.
07Reasons before furniture
The empty Office with its real east window and south door; the arrangement is argued from them before anything is placed.
YouI want to face the window while I work. Put the desk along that wall.
HADThe desk can sit just below the window, with the chair on the room side and a clear approach from the door.
08The office as a whole
Desk and chair at the window, the library on the solid wall, and a reading corner as a second zone; eight furnishings, door and window kept clear.
YouHow does the whole office work?
HADThe desk faces the east window. The library uses the solid wall, while the lounge chair and ottoman define a quieter second zone.
09A guest room that is also a study
A different brief for the second bedroom: the desk takes the west window, the bed sits opposite and beyond the door swing.
YouFinish the guest bedroom so it also works as a study.
HADA single bed, bedside table, wardrobe and library complete the room around its window-side desk.
10The whole arrangement
Forty furnishings across five rooms, including the living and dining groups and the fitted kitchen, placed one decision at a time.
YouLet’s review every room together.
HADWe have two furnished bedrooms, an office with a reading area, a living and dining room, and the connected kitchen.
11A chair outside the conversation
Before the revision: the armchair sits low in the room, away from the seating group.
YouThe Living Room armchair feels detached. Move it beside the coffee table; keep the table fixed.
HADWe can bring just that chair into the seating group.
12One chair, one change
The armchair moved beside the coffee table and turned toward the sofa; its neighbours keep their positions, sizes and angles.
YouNow turn that armchair toward the sofa without moving its centre.
HADThe seat turns toward the sofa while its position stays fixed.
13A wider window, the same wall
The Office window widened from 120 to 160 cm about the same centre; the desk and the rest of the room do not move.
YouThat is the width I wanted.
HADThe Office window is now 160 cm wide at the same centre. The desk and furniture remain fixed.
14The office at eye level
Desk and chair at the window, the library behind, the reading chair in the foreground; the same positions as the plan.
YouThe office now feels like a complete interior.
HADThe desk and chair face the east window; the reading chair and storage occupy the other part of the office.
15The main bedroom
Bed, two bedside tables, wardrobe and window bench, with a clear zone at the foot of the bed.
YouThe main bedroom should feel calm rather than empty.
HADThe bed, two bedside tables, wardrobe and window bench give the room distinct places to rest, store and sit.
16From the living room into the kitchen
The double opening gives the south extension a direct relationship to the shared space.
YouThe new kitchen should feel connected to the living area.
HADThe double opening gives the south extension a direct relationship to Living Room.
17One design, three ways
The finished model from above: five rooms, one design, inspected from the plan, from above, and from inside.
YouThis feels like one home, with different places to work, rest and gather.
HADThese views let us inspect one layout from the plan, from above, and from inside each room.
One click turns the design model into a 3D view you can walk or fly through on screen: each wall becomes boxes split around its openings, and each detected room becomes a floor. Nothing is modelled twice, so the bed placed on the plan is the bed you walk around. The voice session started in 2D stays open in 3D, so you can keep talking while you walk the plan.
On a headset, the same scene is a walkthrough, not an editing surface: you can still undo from the controller and keep talking to the assistant, but there are no tools for drawing or moving things. I kept editing on the desktop on purpose, where the canvas supports precision. Three surfaces share one model, and one question follows them: what did the system actually check?
The assistant’s actions take different roads: solver-backed placement and relational moves check containment and overlap, with swing and view checks for wall anchors. “Turn the desk to face into the room” expresses a move the solver can resolve. If the proposed move fails, the model gets one more attempt with the exact violation; success is not guaranteed.
Small nudges apply the model’s own change with lighter checks: a rotation is refused if it would hit a wall, a resize is clamped, and a nudge simply moves. Walls and openings follow your click. For catalog items, the model draws the icon and 3D shape but never chooses their position in a room.
The session recorder keeps every prompt, every instruction, and every solver verdict in order. A recording of a desk rotated into a wall without a retry is why rotations are now checked first. I used those traces to hold claims to implemented behaviour, the same standard this page needs to meet.
Checked against what is implemented, not promised against everything.
The person draws, judges, and decides; the model listens and interprets; structured tools resolve and check. That division is what Human–AI Design names. I developed HAD around it so that discussing a design can lead to an action on the plan, while the person remains its author.
For the person, the design conversation continues through actions they can question, inspect, and reverse. For me, it is a pattern I keep reaching for: models interpret, structured tools decide, wherever geometry, rules, or records deserve precision. “Can you just put the bed in the bedroom?” now has an answer in the plan, with the placement explained and the decision reversible.
Designing in conversation with an AI requires an assistant that understands the plan, acts without guessing geometry, and explains actions people can reverse.