Optical neural networks process information with light. Their appeal includes the possibility of fast, energy-efficient computation, but a physical device rarely matches its ideal design exactly. A model trained in simulation can lose accuracy once misalignment, aberrations and component behavior enter the experiment. On November 18, researchers at Stanford and EPFL posted a paper describing an agent that learns an optical network directly from camera measurements. The important result is a structured experiment loop, not a general demonstration that language models understand every physical system. Research paper

The paper introduces “optics as code.” A coding agent proposes programs and phase masks, a harness displays them on a spatial light modulator and returns camera images, and the agent records hypotheses and observations. The team reports that its optical classifier reached 96.8 percent test accuracy in its best run with notes from an earlier session, compared with 52.5 percent for the strongest gradient-free baseline and 96.3 percent for backpropagation through a calibrated device model. Those results make the method worth examining. They come from one laboratory setup and defined tasks. Full paper and methods

The device departs from its ideal model#

The experiment uses an optical system that reflects laser light across several regions of a spatial light modulator. Some regions encode the input, while two display trainable phase masks. A camera measures output intensity. To classify handwritten digits zero, one and two, the system assigns the class whose detector region receives the most light. The physical interactions are sensitive to the device's actual geometry and response, even though the simulated model starts from an idealized propagation design.

The calibration problem appears clearly in the paper. The initial phase masks classified 98 percent of test digits in the ideal simulation but only 35 percent on the real apparatus. Small physical deviations can change how light travels, where spots land and how much signal reaches each detector. A conventional approach measures and fits a digital model of the particular device, then trains through that calibrated model. In this experiment, the calibration reference used roughly 67,000 camera images. It reached 96.3 percent on the hardware after calibration and training.

A different route uses gradient-free trial and error on the device. The paper compared against proximal policy optimization and policy gradient. In the reported setup, those methods improved accuracy from 35 percent to around 52 percent. This gives the agent a meaningful baseline to beat, but it is not every possible model-free training method or every optical network. Comparisons depend on the design, measurement budget, search settings and physical apparatus. The authors describe the optimizer settings and used the same starting masks and harness budget.

The practical question is whether a system can help understand the mismatch rather than simply search for a better score. A candidate mask might improve one class while hiding why the change worked. If the agent can examine spatial measurements, it may form a more useful model of the hardware. That shifts the evaluation from “did it produce a good output?” to “what information helped it reach a design, and can another run inspect the reasoning?” The paper gives the agent access to more than a scalar reward. Experiment and agent harness

This problem is relevant beyond optics because physical systems contain imperfections that ideal models leave out. Still, optical propagation has its own mathematical structure, and the agent used prior language-model knowledge about optics and programming. The study does not establish that the same approach will work for arbitrary instruments or physical processes. It shows how a well-designed interface can let a model use measurements to learn about one physical device while staying within a controlled experiment.

A harness makes measurement into a question#

The agent did not directly operate the optics. It ran in an isolated workspace and interacted through software the researchers built. It could display phase masks on the modulator, request a camera image, evaluate candidate masks and validate them on practice digits. The task required each request to state a hypothesis and expected outcome. That turns a measurement into a test of the agent's current understanding, rather than another blind adjustment to a score.

Each image counted against a shared measurement budget. The agent kept a notes file, which could carry observations into a later session. It ran on CPU only. The task prohibited numerical models of light propagation, and access to the network and outside files was blocked. These restrictions matter. They rule out some ways of learning the answer from a simulation or outside source and make camera measurements the route to understanding the device. The harness defines the evidence available to the agent.

The researchers report that the agent noticed class-dependent shifts in focused spots, not merely brightness changes. It divided one phase mask into blocks, probed their responses and designed a “spot router” that directed different digit classes toward different detector regions. The paper describes the first version of this design as surpassing the final accuracy of the gradient-free methods after about one seventh of the measurement budget. This is an informative story of a specific design path. It should not be mistaken for proof that every agent run discovers the same physical principle.

Editorial diagram of an agent probe loop inside a harness, with a blocked phase mask routing colored beams to separate detectors.

Information access mattered in the ablation experiments. Removing camera images, probe tools or the requirement to state hypotheses affected the system's ability. The paper reports that adding more rules about how the agent should decide or allocate its budget did not improve accuracy. The researchers interpret this as evidence that better observations helped more than additional instructions. That is a useful design lesson for experimental-agent interfaces. It remains a finding from this harness and evaluation, not a universal law that prompts never help.

The experiment also lets a later run use notes and code from an earlier one. Notes did not include phase masks or test data. Instead, they recorded observations about the apparatus. The best classifier results came in runs with notes from prior sessions. This raises an important possibility: an agent might learn how one device behaves, then pass that knowledge to another session, reducing repeated characterization. Whether such knowledge transfers across devices, setups or laboratory teams is a separate question. The reported experiment provides early evidence within a tightly defined setting.

A measurement budget makes the trade-off visible. The classifier had 51,200 camera measurements available. The encoder task had 25,600. A camera measurement took about 0.107 seconds, so acquiring the classifier budget took around 1.5 hours, separate from other time and costs. The harness let the researchers cap physical experimentation. A real deployment would also need to account for model inference, device setup, researcher oversight and repeated runs. The paper compares data and training time with calibration, but does not provide a general dollar-cost comparison for many deployed devices.

One classifier result is strong, but narrow#

The classifier's best agent run with earlier-session notes reached 96.8 percent accuracy on a sealed test set of 3,147 handwritten digits from three classes. Without those notes, the reported best agent run from the task description alone reached 91.1 percent. The gradient-free baselines remained near 52 percent in the physical experiment. Backpropagation through a calibrated model reached 96.3 percent. In this particular setup, the agent came close to and slightly exceeded that reference in its best reported run.

The authors also report three classifier runs that received notes or general lessons from earlier sessions. Their test accuracies were 89.2, 91.3 and 96.8 percent, with a mean of 92.4 percent and a standard deviation of 3.9 percentage points. That spread is useful context. It shows variation across runs rather than a single stable 96.8 result. The strongest result is notable, and the run-to-run range deserves to sit beside it. The experiment did not establish a performance guarantee for a new device.

The measurement process also has a statistical limit. The test set is finite, and reported accuracy has uncertainty. The paper estimates a maximum standard error of 0.9 percentage points for 3,147 samples at 50 percent accuracy and 0.3 points for the best run. Those values describe sampling uncertainty under the test set, not systematic variation across new optical hardware, fabrication batches, environmental conditions or model versions. A future deployment study would need those sources of variation to make a broader reliability claim.

The benchmark was deliberately controlled. The agent ran without a human typing during each headless session. It worked in isolation and could not access a GPU, network or external files. The researchers also sealed test digits from the agent. These controls reduce some forms of leakage and make the comparison easier to interpret. They do not turn the paper into an independent replication. The work is a preprint from the researchers who built the experiment and harness, and its conclusions should be read at that stage of evidence.

The result is still meaningful. Calibrating a separate digital model for every physical device can require substantial expertise and data. An agent that can probe and tune a device from measurements could eventually reduce some of that work. The current evidence supports a narrower statement. In this lab setup, a coding agent used a designed harness to train an optical classifier with fewer measurements than the calibration reference required, reaching comparable accuracy in the best run. Scaling and cost remain questions for further study.

A second task tests a different use of light#

The paper also studies an optical encoder. Instead of classifying digits directly, the optical system compresses Fashion-MNIST images into 16 detector signals. A small digital decoder reconstructs the images from those signals. The phase masks begin with random values, and performance is measured by peak signal-to-noise ratio on held-out images. This task asks whether the same agent-and-harness setup can train a compact optical representation, not only route a few digit classes.

With notes from an earlier session, the agent reached 16.1 dB in the reported experiment. PPO reached 15.0 dB and the random starting point 13.8 dB. Backpropagation through a calibrated model produced the best result at 16.9 dB, putting the agent within 0.8 dB in that run. These numbers support the idea that the approach can apply to another task in the same apparatus. They remain results from one optical encoder setup and a limited collection of held-out images.

The optical encoder could be useful because information processed optically is often transferred, stored or further processed electronically. Compressing an image to fewer detector outputs could reduce that step's cost. The article should not treat this experiment as a demonstrated commercial benefit. The study measures image reconstruction quality and compares training approaches. It does not provide a deployment cost analysis, production throughput test or full system evaluation that would establish an economic advantage.

The comparison across language models provides another signal. The paper says Claude Fable 5.2, described as the more capable model in the tested group, outperformed Opus 5.5 and Sonnet 5 on this encoder task under the same task text. The authors infer that the approach may improve with more capable models. That is a plausible reading of these results, not a guarantee for other systems. Model capability changes quickly, and the experiment tests a small set in one harness.

The second task also illustrates why claims should follow the evaluated output. A classifier accuracy score and reconstruction PSNR measure different outcomes. It would be misleading to compress both into a vague statement that the agent “trained optical networks better” without naming what was measured. A stronger account keeps the tasks separate, identifies the baselines and gives the experimental context. This precision lets optical researchers decide whether the result relates to their own instruments and objectives.

The harness may be the more reusable contribution#

The researchers frame “optics as code” around an agent that writes and tests programs to control optical elements. It represents components as interpretable elements such as lenses, prisms and aberrations. Instead of adjusting hundreds of thousands of pixels independently, the agent can form a compact hypothesis about how a component changes the light and test it on hardware. The paper reports that this approach was used to characterize relationships in the apparatus with far fewer measurements than the calibrated model's full campaign.

That characterization is an intermediate accomplishment, not just a way to reach the final task score. The agent examined where optical planes appeared on the camera, how phase masks redirected light and how small alignments differed from nominal positions. It recorded the observations in notes and used them to inform designs. An optical scientist can inspect the described measurements and judge whether they support the reasoning. The value of the system partly lies in exposing a chain of observations rather than returning a mysterious weight update.

This is why the harness deserves attention from tool builders. Each probe had a defined relationship to the physical device. The interface returned images and required a hypothesis. The shared budget made the cost of experiments visible. Notes provided a memory surface that could be passed between sessions. These choices shaped what the agent could learn. A different interface that returned only a scalar reward would provide a different learning environment. The paper's ablation results support the importance of these observations in its experiments.

For laboratories, the lesson is to make instruments legible through their software interfaces. An agent needs safe, bounded actions and meaningful feedback. Researchers need to see what was requested and why. A harness can enforce budgets, isolate files and preserve logs. These are proposed design principles that follow from the study. The paper does not show a universal plug-and-play platform for arbitrary lab equipment. It presents a carefully designed interface for a particular optical bench.

There is a safety dimension even in a modest laboratory setting. The agent was limited to a controlled workspace and predefined measurement tools. It did not operate an unconstrained instrument with unrestricted network or file access. This makes the experiment easier to inspect and safer to evaluate. Extending to other devices would require identifying actions that can be automated, defining safe ranges and deciding which measurements require human approval. Those questions belong to the deployment design, not just the language model.

The note-transfer result is also promising but early. A later session that received earlier notes achieved the strongest classifier performance. That suggests an agent can avoid relearning some properties of a device. A laboratory would need to test whether notes remain accurate after hardware changes, maintenance or realignment. The notes themselves may need versioning alongside the instrument. Without that, a once-true measurement could guide a later session toward the wrong adjustment. Reusable memory is useful only when its validity can be checked.

Evidence should separate one apparatus from many devices#

Optical neural networks are attractive partly because light can carry signals in parallel and propagate with low loss. Those properties motivate research into fast and energy-efficient computing. But each assembled device has its own imperfections. An optical setup that works in one lab may require different calibration when components are repositioned, replaced or fabricated with variation. The paper's argument for agentic in situ training is motivated by this device-specific reality.

The experiment uses a particular phase-only spatial light modulator, a mirror, a laser and a camera. It trains two masks over several hundred thousand pixels. Those choices create a meaningful physical system but limit generalization. The result does not show that the same procedure will work unchanged on another modulator, optical architecture, wavelength, detector layout or nonlinear material. The authors discuss broader potential, while the measured evidence remains tied to their apparatus.

The sealed test set and shared comparison harness help prevent some unfairness between methods. Gradient-free baselines and the agent used the same start, budget and validation structure. Each method was scored on held-out digits. This is careful experiment design. More confidence would come from repeating the whole process with new physical units, independent researchers and varied task sets. That would reveal whether performance follows the design of the method or a favorable property of this one setup.

The costs also need broader measurement. The calibrated model required about 67,000 camera images in this experiment, while the agent's approach used fewer measurements. That is a useful data-efficiency comparison. An operational cost comparison should also include model inference, device access, setup, maintenance, repeated trials and expertise required to review output. The paper does not claim a universal economic result. A deployment case for many devices must measure the complete cost of keeping them calibrated and working.

There is a related question of model change. The paper suggests more capable models may improve the method, based on its encoder experiments. If an organization relies on a hosted coding agent, a model update could alter behavior. A reproducible scientific workflow needs to record the model and reasoning configuration along with hardware and measurement budget. This is an evaluation recommendation, not an issue reported by the authors. It lets later users explain why two sessions produced different designs.

A strong follow-up might compare device-by-device calibration with agent probing across a set of hardware units. It could measure accepted accuracy, number of measurements, time, failures and how much expert intervention each route required. It should also inspect whether the agent's notes transfer safely to a similar unit. Such an evaluation would directly test the paper's scaling motivation. Until then, “potentially scalable” remains a research direction rather than a deployment outcome.

The most persuasive result is a testable sequence#

The paper's strongest contribution is a sequence a reader can inspect: start from an ideal model, measure how it misses the hardware, give an agent tools to probe the device, record hypotheses and compare the resulting design against controlled baselines. This does not make every step correct. It makes each important step more visible. That is a useful standard for AI-assisted experimental work, where a polished final number alone hides how the system reached it.

An optical scientist reading the study can ask whether the probes are informative, whether the physical interpretation matches known optics and whether the comparison is fair. A systems engineer can inspect the harness boundary and measurement budget. A developer of scientific agents can examine how notes and hypotheses affect the loop. The article makes different kinds of scrutiny possible by preserving technical details. Accessibility of the paper and its experimental description are part of the value.

This does not eliminate the need for replication. The study has a small number of physical runs, one hardware configuration and a narrow classification and encoding task. The three classifier sessions with prior notes range from 89.2 to 96.8 percent, a spread worth showing next to the best outcome. Readers can appreciate the result and still ask how reliably it repeats. The authors report uncertainty for the held-out sample, but that is distinct from variation across devices and future sessions.

Our assessment would become more positive if independent laboratories reproduce the gains on different devices, preserve test separation and report total experimental effort. It would become more cautious if calibration needs return under hardware variation or if performance depends heavily on one model session. These are concrete evidence conditions. They connect the conclusion to what future work can check rather than to broad excitement about AI doing science.

For now, “optics as code” is a promising method in one physical experimental setup. Its agent learned from camera images, wrote hypotheses, received feedback and built a classifier that approached a calibrated-model result. That is a meaningful demonstration. It is not a universal optical engineer or proof of energy-efficient production hardware. The difference between those claims matters because frontier research becomes useful when its results and boundaries are both legible.

A laboratory assistant should expose its reasoning path#

The broader idea extends beyond optical networks. Many scientific instruments expose parameters that can be adjusted and produce measurements that help explain their response. An agent could in principle propose an adjustment, predict an outcome, run a bounded measurement and revise its model. The usefulness depends on a reliable harness, safe action limits and output that a domain expert can interpret. That is a research direction suggested by the experiment, not a demonstrated general capability across laboratories.

The harness also shapes incentives. A system optimizing validation accuracy can learn to pursue that measure. The sealed test set helps protect the final assessment, and the authors tested several harness components. Future work can examine metrics that better reflect desired physical behavior and resist reward hacking. For this optics study, the available evidence consists of camera measurements, a hypothesis protocol and returned images. Evaluation metrics remain specific to the classification and encoding tasks.

The need to state a hypothesis per request is an interesting interface choice. It may improve the record of why a measurement was taken and how the agent interpreted it. In a laboratory, a hypothesis can be wrong, and that is useful if it is falsifiable. Logging a prediction and observed outcome allows a reviewer to understand whether the agent revised its belief or merely changed parameters. This can make experimental automation easier to audit and may offer researchers a better view of the agent's method.

Memory between sessions introduces a related scientific responsibility. Notes should preserve what the instrument showed, what conditions were used and which conclusions were tentative. A later agent can verify an earlier observation with probes. A human team can also determine whether a mechanical realignment invalidated it. This suggests that memory in physical research should be tied to the instrument state and session metadata, not just retained as free-form text. Those are implementation implications for future systems, not functions the paper claims to have solved.

For adoption, the first question is not whether an agent can replace a domain expert. It is whether it can help the expert obtain informative measurements and test a clearly scoped idea. The paper demonstrates a route for that kind of collaboration. The scientist remains responsible for interpreting evidence and deciding whether a result matters. This modest framing makes the technology more credible than claims of autonomous discovery, because it connects the system to observable work.

The result deserves interest with the bench still in view#

The authors are researchers at Stanford and EPFL. Their arXiv paper is a preprint, and their public project materials provide a readable overview of the benchmark. The work reports a careful experiment with a real optical setup and controlled baselines. Public discussion available at launch does not amount to independent replication. The evidence should be treated as a first-party research result awaiting the ordinary scrutiny and reproduction that follow a new method.

The optical classifier result is striking because it crosses the gap between an idealized design and an imperfect device. The harness gives the agent meaningful observations instead of only a score. The encoder task suggests the approach may serve another optical computation. These are solid reasons for attention. They do not warrant turning a two-task study into a promise that any optical processor can be trained cheaply without calibration.

A sound development path would publish further test configurations, measurement logs and independent runs across hardware units. Researchers could assess whether the method learns stable physical properties or adapts only to one session. They could compare the expert time required to calibrate a model with the time required to supervise the agent. Those comparisons would show where the approach saves resources and where it transfers work to a different part of the process.

The paper also models transparency as a technical feature. Notes, hypotheses, camera images and a visible measurement budget allow people to inspect the agent's path. A production-grade research tool could keep that auditability while adding stronger experiment management, versioned calibration and repeatable test sets. The exact design will differ among instruments. The principle is that physical automation should show why it acts and what evidence changed its next step.

Our view would change with evidence that the approach repeats across distinct optical devices, maintains the measured gains under new sessions and accounts for total deployment cost. It would weaken if those gains vanish under modest misalignment or if the notes fail to transfer. These are not predictions. They define what a researcher or buyer can check before extending the claims beyond the paper's bench.

For now, the study offers a useful frontier result: a coding agent, constrained by an experiment harness, can characterize and train a real optical network from measurements. The scientifically valuable part is not merely that the answer reached 96.8 percent. It is that the paper shows what the agent measured, how it interpreted the apparatus and which controls shaped the result. Keeping that experimental path visible is how a promising demonstration becomes a method others can assess.