A robot that clears an obstacle in a simulator has solved a useful engineering problem only if the real machine can do the same job. A new MIT study puts that gap at the center of RoboBuilder, a benchmark for coding agents that design both a robot and its motor program from available parts. Every simulated design that passes is built and tested on a physical course. The agents then receive video of real failures and can redesign. RoboBuilder paper

Our view is that the closed loop matters more than the novelty of asking an agent to draw a robot. Earlier software agents could receive fast, exact test results. Physical machines demand fabrication, materials, assembly, power and imperfect measurement. RoboBuilder connects design to that slower reality. The evidence is promising as a research benchmark, not proof that general-purpose AI can reliably engineer robots for arbitrary environments. The authors tested a bounded parts inventory, ten obstacles and a particular laboratory setup.

Simulation is a proposal about the physical world#

The study starts with an ordinary engineering difficulty. A robot must be assembled from real parts, joints must hold, gears must mesh, and motors must deliver enough force. A simulator can approximate these conditions, but its simplifications matter. In RoboBuilder, 15 early designs passed simulation while none crossed the real course. Wheels slipped, joints separated, gears disengaged and loads shifted. This is not a small error bar around performance. It is the difference between a design that passes and one that fails.

The gap appears because physical parts do not behave exactly like their catalog descriptions. The benchmark initially used public part data and estimates. The authors report one friction-pin connection holding about 4.5 newton-millimeters, while the simulator had assumed 24.5. That mismatch can change a design decision. A simulated wheel assembly may appear to transmit torque reliably while a real connection slips under load. A simulator that has not been checked against real runs can reward an agent for exploiting the model rather than solving the obstacle.

The benchmark uses standard brick-compatible pieces, motors, electronics and a limited custom 3D-printed component. It contains 132 part types and roughly 3,000 pieces in stock. A robot that passes uses a median of 114 pieces. Each agent also writes an open-loop motor program. The machine is not steered during a run. This gives the experiment a tangible design space while keeping the challenge bounded enough to build repeated physical prototypes. Project overview and benchmark

The tasks comprise ten obstacles at three difficulty levels. A robot must reach a goal within a time limit and remain upright there for five seconds. Some tasks allow a printed part, with limits on the number and size of custom parts. The obstacles make different demands. A gap rewards reach and stability, a rope may favor a light chassis, and a wall calls for clearance or a mechanism that can lift the frame. The same set of parts supports many strategies.

These constraints make the study closer to a controlled engineering benchmark than an attempt to build a commercially useful robot. That is a strength for examining the design loop. It is also a limit on generalization. Results in a brick-based course do not directly describe robots moving among people, handling unknown objects or operating for months without a researcher nearby. A clear account of the benchmark boundary lets readers value the experiment without asking it to answer a broader question than it measured.

The agents search a constrained design space#

RoboBuilder asks coding agents to design hardware and software together. They select parts and connections, create an optional custom geometry and specify timed motor commands. The task is not merely to generate a CAD image. The physical assembly has to obey inventory and building rules, and its program has to drive it over the obstacle. This pairing matters because the same body can behave differently under another motor schedule, while a better schedule may not rescue a weak connection or unstable frame.

The design space is still extremely large. The authors estimate that even assemblies of 25 pieces from the inventory create around 10 to the 68 possible combinations. Passing robots have a median of 114 parts, and each motor program can vary in timing and power. Those figures illustrate why manually enumerating every plausible machine is infeasible. They do not mean the agents explored anything close to all possible robots. The experiment relies on a benchmark harness that gives each run a bounded number of simulations and a structured set of tasks.

Four coding agents were tested. The authors report 16,427 simulations across 12,774 distinct robots, and 135 robots were built by hand. The agents solved nearly all easy obstacles and produced mechanisms the search baselines did not reach. At harder difficulty levels, their success fell. When asked for one robot to cross every obstacle, the all-in-one designs passed at most six of ten level-one courses. A machine optimized for each obstacle could use a different structure. One universal design had to compromise.

This contrast is an important result for practical engineering. Real products often need to satisfy multiple conditions at once, not win one isolated task. A robot that solves a single benchmark course is not automatically a useful system. The all-in-one task asks a more demanding question, and performance drops. The study therefore contains evidence both for agent capability and for the limits of transferring that capability into a consolidated product. Benchmark and results overview

The paper also reports that broader exploration did not reliably predict better designs. The strongest agent often passed a task with its first concept. That finding is specific to this task distribution and budget. It may reflect useful learned engineering priors, a benchmark with patterns the model recognizes, or both. The paper studies idea diversity and search behavior, but readers should avoid turning one benchmark’s result into a general law that more exploration never helps robot design.

Video feedback changes what the agent can learn#

The key move in RoboBuilder is returning a real failure to the agent as video. The agent can inspect how a prototype behaved, update its understanding of the simulator and redesign the machine. The workflow connects three activities: run a simulation, build and test a passing design, then learn from a failure on the physical course. It is a more meaningful loop than treating the simulator as a final authority.

Stylized loop diagram linking a simulated robot, a hand-built brick prototype on an obstacle, and a camera feeding a failure video back to the simulator.

This process begins with a mismatch. None of the 15 initial simulator-passing robots crossed the course using the model built from public part data. The researchers and agents then requested physical measurements, such as friction and connection strength, and adjusted the simulator. After learning from real runs, two thirds of simulated passes crossed in reality across the evaluated levels. The paper reports 66 percent for the broad all-level comparison. This number has a particular denominator and should not be mixed with the later level-specific results.

The study reports a higher figure for the easier subset. For levels one and two, 73 percent of simulated passes crossed in the real world after simulator learning. When shown video of a real failure and allowed to redesign, agents removed 72 percent of level-one and level-two failures in one round. The project page summarizes the resulting rate as 93 percent after the loop. These figures refer to different stages and subsets. The initial 15 designs, all-level pass rate and easier-level post-redesign rate are not interchangeable measures.

A particular failure can teach more than a score. A robot that spins its wheels against a bump may indicate that the friction model is wrong, that the wheel placement is poor or that the control schedule applies force at the wrong time. Video gives the agent observations with which to distinguish possibilities. In one described case, a spoked wheel that worked in simulation spun against a ridge in reality. The agent adjusted its model of plastic friction on the rods and later designed a worm-driven wheel mechanism that crossed. This is a case study within the benchmark, not a guarantee that every failure has an equally legible video explanation.

The loop changes the role of the simulator. It remains useful for trying many designs cheaply, but its predictions become hypotheses tested against the machine. That is a sensible engineering relation between model and hardware. It can also be expensive. The paper estimates 240 hours for hand-building and testing 135 machines, with 461 real videos paired to simulations. Physical feedback has more value than a simulated score, and it costs more to obtain. A benchmark that makes this cost visible helps readers understand what scaling the method would require.

Real-world success has a careful denominator#

The most memorable statistic can also be the easiest to misuse. The abstract says 66 percent of simulated passes hold in the real world across all levels. The project page separately reports that two in three simulated passes cross and that most remaining easier-level failures can be removed after a video loop. In the paper, the post-learning 66 percent counts 57 of 86 simulated passes for levels one and two? The exact paper overview reports 57/86 across the learned simulator sample, while later reports 73 percent for the subset of level one and two simulated passes and 93 percent after one loop. Readers need to keep these denominators attached to the claims.

The initial 0 of 15 result is also conditional. It applies to early designs that passed a simulator based on public part data before the simulator had been calibrated from physical observations. It is not a universal failure rate for simulation in robotics. The result demonstrates how badly this particular model represented parts and courses before measurement. The correction is valuable precisely because the authors show the transition from an inadequate model to a more informative one.

The post-loop 93 percent should likewise not be read as 93 percent of arbitrary robot designs or tasks becoming real. It concerns level-one and level-two failures after a bounded redesign process in this benchmark. The agents were shown videos, allowed to change up to two parameters for a failure under the protocol and had to pass the revised simulator. The study also built and filmed designs by hand. Those constraints make the result interpretable, while limiting how far it can be generalized.

The paper’s authors identify limitations in the supplementary material. The simulator is fitted from a fixed camera and bench tests, so not every physical parameter can be identified. Each correction is fitted to a particular failure. The lightest and fastest designs were not built. The benchmark uses specified obstacles and parts. These disclosures matter because they establish where real validation happened and where the results remain simulation-side. Readers should not assume that every figure on the leaderboard represents a physical robot.

Run-to-run variation also matters. Full agent runs across all tasks were not repeated because each pass took nearly 4,000 simulations per agent. The authors did repeat the four one-robot-for-all-obstacles designs under perturbed starting position, command timing and battery level. Some obstacles remained inconsistent across repeated trials. This is useful evidence about robustness, and it also reinforces that a single successful crossing is not a complete reliability characterization.

The laboratory loop is an expensive form of evidence#

Physical robotics imposes a different cost structure from software testing. A software agent can run a test in seconds and receive an exact verdict. Building a robot requires parts, assembly, cameras, courses and people who can inspect what happened. RoboBuilder’s reported 240 hours of build and test work make that cost visible. The agents can generate simulations at scale, but the real feedback channel is the scarce resource that grounds the simulation in physical behavior.

This has a consequence for adoption. A research lab can use the benchmark to study agent design and sim-to-real adaptation under a controlled inventory. A company designing a product needs additional assurance around durability, variation between components, safety and use outside a prepared course. The study does not claim to solve those problems. Its value is to provide an experimental structure for one of them, with a dataset of designs, simulations and paired real videos that others can inspect.

The distinction between laboratory and product testing is not a criticism of the research. It is how evidence accumulates. A benchmark can establish that an agent can propose physical designs, learn from a particular simulator mismatch and improve performance on specified courses. It cannot by itself establish safe deployment in open environments. Those conclusions would require new tasks, different hardware, repeated reliability trials and clear safety criteria.

There is also a question of who does the fabrication. The robots in the study were built by hand. That creates a human role in turning a file into a functioning prototype. An agent may design parts and programs, but the process still depends on fabrication and assembly. A future system could automate more of that work, yet this study does not test a fully autonomous factory. The honest description is agent-assisted robot co-design with human-built physical trials.

The paper and project page make the benchmark artifacts available, including simulator, courses, designs and real video pairs. Open materials can help other researchers check definitions, reproduce selected tests and identify cases where the simulator misses reality. Reproduction still requires compatible equipment, parts, tools and time. A dataset lowers the barrier to examining the work. It does not erase the practical resources needed to repeat physical runs.

For developers of robot tooling, the most reusable idea may be that a failure video can be part of a structured engineering interface. A video is not just a demonstration; it can become evidence used to update a model and design. That requires the video to be paired with the simulated run, configuration and failure record. The benchmark’s structure makes these relations explicit. Whether the same approach works for other robots depends on what their sensors reveal and how well an agent can distinguish design, simulation and fabrication errors.

An all-in-one robot reveals the cost of competing requirements#

The study’s all-in-one challenge asks one machine to cross all ten level-one obstacles. It succeeds on at most six. This is a compact way to show why isolated wins do not automatically add up to a useful general-purpose machine. A bump course benefits from clearance. A bridge favors a narrow track. A rope obstacle rewards a light chassis. A wall may need a mechanism to lift or vault the vehicle. One hardware design must balance incompatible demands.

That result connects the benchmark to a broader engineering question. A design optimizer can improve a selected score, yet product requirements often include several criteria at once: size, speed, reliability, cost and adaptability. The paper includes light and fast follow-up tasks for already accepted designs, but those designs were not physically built. The distinction is important. Simulated optimization of mass or speed is not the same as showing that a lighter or faster robot remains robust on the real course.

The team's own benchmark also shows that passing standards matter. A design must reach the goal within a time limit and remain upright for five seconds. It is validated against inventory and assembly rules. A simulator pass is only one gate, followed by physical testing. If a different project changes the task, acceptance criteria should change too. A robot that reaches a goal once may not be adequate for repetitive use. The metric should reflect what an eventual user needs the machine to do.

This is why the paper’s progress is stronger as a research procedure than a product claim. It describes how agents can compose hardware and programs, how physical observations can revise the simulator and how failures can guide redesign. The all-in-one shortfall keeps the scope honest. Generality is not achieved by pointing one agent at more obstacles. It requires testing how well the design handles trade-offs and how performance behaves under variation.

What future work would make the claim stronger#

A useful next evaluation would repeat the full agent process enough times to estimate variability. The study notes that a full run is costly, taking thousands of simulations per agent. That cost is itself part of the research question. Teams could compare the value of more simulated search against the value of additional real tests. A measured allocation would show where each hour or fabrication cycle contributes the most information.

The benchmark could also expand beyond its current inventory and fixed camera setup. Different materials, connection types and sensing modalities would test how portable the learned simulator is. New obstacle layouts could test whether agents learn general physical relationships or adapt to a familiar course. Those are future research directions, not capabilities already shown by the paper. The authors’ own limitations give a clear starting point for such follow-up work.

For engineers considering similar systems, the immediate practical lesson is to treat simulation as a model with a maintenance plan. Record assumptions, request physical measurements where they matter and pair observed failures with the simulation that predicted them. If the real robot disagrees, decide whether the design, model, assembly or test conditions changed. This keeps the diagnosis from collapsing into “the AI failed” or “the simulator was wrong” without examining evidence.

The assistant’s role should also be inspectable. The study reports four coding agents, compares them with search baselines and makes distinctions between task levels and all-in-one performance. Future evaluations could compare against experienced human designers under matched time and material budgets, if the goal is to assess labor substitution. That comparison is not present here. The current study demonstrates agent capability within a defined benchmark and exposes failure cases, which is already a useful contribution.

Our assessment would become more favorable if independent labs reproduce the loop, achieve similar results on new courses and report the labor needed for calibration and fabrication. It would become more cautious if gains disappear under different parts or if real failure rates remain high when robots operate repeatedly. These conditions are observable. They tie confidence to the next evidence rather than to the visual appeal of an agent-built machine.

The important design is the loop around the robot#

RoboBuilder shows agents designing more than renderings. They assemble available parts, write motor programs, test in simulation and, in some cases, adjust their understanding after seeing a real failure. The measurements are substantial for a research benchmark: 16,427 simulations, 12,774 distinct designs and 135 machines built by hand. They show both inventive capability and a clear drop between simulation and physical performance.

The most useful interpretation keeps that drop visible. A simulator is valuable because it lets an agent explore cheaply. It is dangerous when its predictions are treated as the world. RoboBuilder’s contribution is to return physical evidence to the design process and let that evidence reshape both the model and the machine. The project provides an example of how AI-assisted engineering can become more accountable to reality.

It does not establish a robot designer ready for arbitrary environments, a fully automated manufacturing line or a human-equivalent engineering process. The obstacles, inventory, cameras and open-loop controls bound the experiment. The all-in-one task passes at most six of ten courses. Some optimized simulated designs were not built. These facts do not diminish the result; they make its scope legible.

The evidence should therefore change how developers think about the word “agent.” A useful system does not merely issue a design. It works through an interface that exposes observations, records assumptions and makes failures available for correction. The harness, simulator and fabrication loop are not secondary plumbing. They define what the agent can learn and what a successful result means.

For now, RoboBuilder is a strong case for experiments that connect digital design to physical tests. Its most important machine may be the loop that builds a robot, watches it fail and asks what the failure teaches. The next step is to repeat that cycle across designs, materials and tasks, while accounting for the people and time it still requires. A machine that works in the world earns that description from the world itself.